3560X Crash and Reboot Issue on IOS 15.0(2)SE

[woodc83]

Hi All,

I have 3560X-24T-S switches with IP Services loaded at remote sites that I have been having troubles with.  Originally, they had IOS 12.2(58)SE1 on them.  I have up to four diverse paths via point-to-point microwave at the remote sites.  The Microwave equipment is by Microwave Networks and is a Prodeus M series which Ethernet support.  The original issue manifested itself as hardware loopback errors on some of the ports that were connected to the microwave links.  

My experience in the past has been that when a hardware loopback error occurred it is usually a bad switch port.  In this case however, if I disconnected all of the microwave Ethernet links and rebooted the switch and then connected the Ethernet connections to the microwave links again, everything works fine.  No hardware loopback errors.  That is, until the next switch reboot.  Then the hardware loopback error would return.  Interestingly, it would come back on different ports connected to the microwave links every time.  So if a reboot was done without disconnecting the microwave Ethernet links the hardware loopback error would change from one microwave link to another after each reboot.

I then went through and read the lengthy release notes for IOS version 15.0(2)SE and found several fixes that I thought could fix my issue.  So I downloaded it and updated a couple of the offending switches (not all of them were having this problem).  After going through the second update required to resolve the 'open file error' that happens going between 12.2(58)SE1 and 15.0(2)SE the problem seemed to be resolved on the offending switches.  So, I went ahead and updated the IOS on all of the switches with point-to-point microwave connections.

I now have one switch that was updated that is crashing and rebooting continuously when the Ethernet links for the point-to-point microwave are connected.  Again, if I disconnect all the microwave links and reboot, it comes up fine ands stays fine when the microwave links are connected back up.  It will work fine until the next reboot and then the crash and reboot loop starts over again.  Below is a portion of the putty log when the crash occurs:

----------------------------------------------------------------------------------------------------------------------

previous memory block, bp = 0x59BF838,

memorypool type is Processor

data check, ptr = 0x59BF860

========= Dump bp = 0x59BF87C ======================

59BF77C:        0        0        0 FD0110DF AB1234CD FFFE0000       56  383FDB4

59BF79C:  212AC68  59BF838  59BF6F4 80000042        1        0  A504F53 543A2050

59BF7BC: 6F727441 53494320 506F7274 204C6F6F 70626163 6B205465 73747320 3A20456E

59BF7DC: 642C2053 74617475 73205061 73736564        0        0        0        0

59BF7FC:        0        0        0        0        0        0        0        0

59BF81C:        0        0        0        0        0        0 FD0110DF AB1234CD

59BF83C:   540000       56  372ADBC  244C060  59BF904  59BF7A0       52        0

59BF85C:  244B964 DEADBEEF  244B964        0  D0D0D0D  42FFC88  42FFC84  D0D0D0D

59BF87C: DC5678BB FFFE0000        0  383FDB4  1D1A000  59BF904  59BF84C       30

59BF89C:        0  213116C BEEFBACC  213116C        0  D0D0D0D  42FFC4C  42FFC48

59BF8BC:  D0D0D0D  D0D0D0D  D0D0D0D  D0D0D0D  D0D0D0D  D0D0D0D  D0D0D0D  D0D0D0D

59BF8DC:  D0D0D0D  D0D0D0D  D0D0D0D  D0D0D0D  D0D0D0D  D0D0D0D  D0D0D0D  D0D0D0D

59BF8FC:  D0D0D0D  D0D0D0D AB1234CD FFFE0000        0  383FDB4  212AC68  59BF9B0

59BF91C:  59BF84C 80000042        1        0  A000000        0        0        0

59BF93C:        0        0        0        0        0        0        0        0

59BF95C:        0        0        0        0        0        0        0        0

========= Dump bp->next = 0x59BF904 ======================

59BF804:        0        0        0        0        0        0        0        0

59BF824:        0        0        0        0 FD0110DF AB1234CD   540000       56

59BF844:  372ADBC  244C060  59BF904  59BF7A0       52        0  244B964 DEADBEEF

59BF864:  244B964        0  D0D0D0D  42FFC88  42FFC84  D0D0D0D DC5678BB FFFE0000

59BF884:        0  383FDB4  1D1A000  59BF904  59BF84C       30        0  213116C

59BF8A4: BEEFBACC  213116C        0  D0D0D0D  42FFC4C  42FFC48  D0D0D0D  D0D0D0D

59BF8C4:  D0D0D0D  D0D0D0D  D0D0D0D  D0D0D0D  D0D0D0D  D0D0D0D  D0D0D0D  D0D0D0D

59BF8E4:  D0D0D0D  D0D0D0D  D0D0D0D  D0D0D0D  D0D0D0D  D0D0D0D  D0D0D0D  D0D0D0D

59BF904: AB1234CD FFFE0000        0  383FDB4  212AC68  59BF9B0  59BF84C 80000042

59BF924:        1        0  A000000        0        0        0        0        0

59BF944:        0        0        0        0        0        0        0        0

59BF964:        0        0        0        0        0        0        0        0

59BF984:        0        0        0        0        0        0        0        0

59BF9A4:        0        0 FD0110DF AB1234CD FFFE0000        0  383FDB4  212AC68

59BF9C4:  59BFA5C  59BF918 80000042        1        0  A504F53 543A2050 6F727441

59BF9E4: 53494320 4D616373 6563204C 6F6F7062 61636B20 54657374 73203A20 42656769

========== Dump bp->previous = 0x59BF84C =====================

59BF74C:        0        0        0        0        0        0        0        0

59BF76C:        0        0        0        0        0        0        0 FD0110DF

59BF78C: AB1234CD FFFE0000       56  383FDB4  212AC68  59BF838  59BF6F4 80000042

59BF7AC:        1        0  A504F53 543A2050 6F727441 53494320 506F7274 204C6F6F

59BF7CC: 70626163 6B205465 73747320 3A20456E 642C2053 74617475 73205061 73736564

59BF7EC:        0        0        0        0        0        0        0        0

59BF80C:        0        0        0        0        0        0        0        0

59BF82C:        0        0 FD0110DF AB1234CD   540000       56  372ADBC  244C060

59BF84C:  59BF904  59BF7A0       52        0  244B964 DEADBEEF  244B964        0

59BF86C:  D0D0D0D  42FFC88  42FFC84  D0D0D0D DC5678BB FFFE0000        0  383FDB4

59BF88C:  1D1A000  59BF904  59BF84C       30        0  213116C BEEFBACC  213116C

59BF8AC:        0  D0D0D0D  42FFC4C  42FFC48  D0D0D0D  D0D0D0D  D0D0D0D  D0D0D0D

59BF8CC:  D0D0D0D  D0D0D0D  D0D0D0D  D0D0D0D  D0D0D0D  D0D0D0D  D0D0D0D  D0D0D0D

59BF8EC:  D0D0D0D  D0D0D0D  D0D0D0D  D0D0D0D  D0D0D0D  D0D0D0D AB1234CD FFFE0000

59BF90C:        0  383FDB4  212AC68  59BF9B0  59BF84C 80000042        1        0

59BF92C:  A000000        0        0        0        0        0        0        0

============================================

%Software-forced reload

00:01:09 UTC Mon Mar 1 1993: Unexpected exception to CPUvector 2000, PC = 25DFF84

-Traceback= 0x25DFF84z 0x25DFF20z 0x1B885F0z 0x2739F64z 0x2130F94z 0x2132284z 0x1F6F1ECz 0x1B6B78Cz 0x1B6B94Cz 0x1B6BA64z 0x4E28FCz 0x4E2B3Cz 0x25E1240z 0x25DB990z

Writing crashinfo to flash:/crashinfo_ext/crashinfo_ext_110

00:01:11 UTC Mon Mar 1 1993: Unexpected exception to CPUvector 2000, PC = 1BAAE74

-Traceback= 0x1BAAE74z 0x1B81BD0z 0x25E1240z 0x25DB990z

----------------------------------------------------------------------------------------------------------------------

I have attached one of the crashinfo files.  Any help you can provide would be greatly appreciated.

woodc83

[Leo Laohoo]

Unless there are features found exclusively on the 12.2(58)SE or 15.0(1)SE or 15.0(2)SE, avoide using these versions.  Try 12.2(55)SE6.