Re: Our 4506 switch is crashing once in a while

belicns · ‎10-18-2010

We have 2 Cisco 4506 switches configured in redundant configuration (HSRP). Primary of them is crashing once in 6-7 days. Here is his show version:

Cisco IOS Software, Catalyst 4000 L3 Switch Software (cat4000-I5K91S-M), Version 12.2(25)EWA7, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2006 by Cisco Systems, Inc.
Compiled Mon 16-Oct-06 18:43 by dchih
Image text-base: 0x10000000, data-base: 0x115FE9A4

ROM: 12.1(20r)EW1
Dagobah Revision 95, Swamp Revision 4

VS1_39_SW01 uptime is 2 days, 17 hours, 59 minutes
System returned to ROM by abort at PC 0x0
System image file is "bootflash:"

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C4506 (MPC8245) processor (revision 7) with 524288K bytes of memory.
Processor board ID FOX081905AK
MPC8245 CPU at 400Mhz, Supervisor V
Last reset from Abort
55 Virtual Ethernet interfaces
44 Gigabit Ethernet interfaces
511K bytes of non-volatile configuration memory.

Configuration register is 0x2101

CRASHDATA

more crashinfo:data

Last powerfail: 09/24/2010 23:02:24

Current time: 10/18/2010 04:09:58

Last reload status: 00000C00 03890000

Last crash: 10/15/2010 10:00:41

Build: 12.2(25)EWA7
buildversion addr: 11F6277C

pc=102EA098 lr=102EB0F8 msr=00089030 vector=00000200
cr=80000024 ctr=103D1448 xer=20000000
r0=00000000 r1=12C1C478 r2=0000C000 r3=205B1970
r4=11FB6948 r5=00009030 r6=1244C604 r7=205B19A0
r8=12C1C498 r9=FD0000DF r10=205B1AB0 r11=80000088
r12=11ED243C r13=FFFFFFFF r14=102ED4EC r15=00000000
r16=00000000 r17=00000000 r18=00000000 r19=00000000
r20=00000000 r21=00000000 r22=00000000 r23=12130000
r24=11F70000 r25=12130000 r26=12C1C4A8 r27=205B1AA8
r28=11FB6948 r29=11FB7A1C r30=11FB6948 r31=205B1970
dec=0031AAEA tbu=0000091E tbl=19AC7104
dar=00000000 dsisr=00000000 hid0=8000C000

ErrDr1=00 ErrDr2=08 ErrStsCpu=72 ErrStsPci=12
BusErrAddr=105B1AA8
Single bit ECC count 0

Traceback: 102EA098 102EB0F8 102ED384 102ED5A4 1030F26C 10305C08

Stack frames:
Frame 1: pc=102EB0F8 stack=12C1C4A0
Frame 2: pc=102ED384 stack=12C1C4C8
Frame 3: pc=102ED5A4 stack=12C1C4E0
Frame 4: pc=1030F26C stack=12C1C518
Frame 5: pc=10305C08 stack=12C1C520

Pushed stack:
12C1C470: 12C1C4A0 12130000
12C1C480: 12C1C488 0000003C 12C1C4A0 0094587F
12C1C490: 11FB6948 203FCFB0 203FCE40 205B1970
12C1C4A0: 12C1C4C8 102EB0F8 00000000 11ED243C
12C1C4B0: 11F70000 12130000 11F70000 121294E8
12C1C4C0: 00000001 11FB6948 12C1C4E0 102ED384
12C1C4D0: 12C1C4E0 12130000 12129350 12C1C4E8
12C1C4E0: 12C1C518 102ED5A4 00000000 11ED0854
12C1C4F0: FFFFFFFF 00000000 00000000 00000000
12C1C500: 00000000 00000000 00000000 00000000
12C1C510: 00000000 00000000 12C1C520 1030F26C
12C1C520: 00000000 10305C08

Popped stack:
12C1C270: FFFFFFFF FFFFFFFF
12C1C280: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
12C1C290: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
12C1C2A0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
12C1C2B0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
12C1C2C0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
12C1C2D0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
12C1C2E0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
12C1C2F0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
12C1C300: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
12C1C310: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
12C1C320: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
12C1C330: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
12C1C340: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
12C1C350: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
12C1C360: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
12C1C370: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
12C1C380: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
12C1C390: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
12C1C3A0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
12C1C3B0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
12C1C3C0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
12C1C3D0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
12C1C3E0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
12C1C3F0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
12C1C400: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
12C1C410: FFFFFFFF FFFFFFFF 12C1C420 FFFFFFFF
12C1C420: 12C1C460 102CC588 00000000 11ED22E0
12C1C430: 00000000 00000000 FFFFFFFF 40000022
12C1C440: 11F70000 12130000 12C1C4A8 0094648A
12C1C450: 12C1C6F8 12C1C470 1F736DC0 0000003C
12C1C460: 12C1C468 102CC770 12C1C488 102CC884
12C1C470: 00000000 11ED22E0

Log buffer:

Vlan251 Grp 251 state Active -> Speak
*Oct 15 10:00:25: %HSRP-6-STATECHANGE: Vlan54 Grp 54 state Standby -> Active
*Oct 15 10:00:25: %HSRP-6-STATECHANGE: Vlan251 Grp 251 state Standby -> Active
*Oct 15 10:00:26: %HSRP-6-STATECHANGE: Vlan251 Grp 251 state Active -> Speak
*Oct 15 10:00:26: %SEC-6-IPACCESSLOGP: list ServeriOut permitted udp 192.168.3.120(137) -> 192.168.2.2(137), 1 packet
*Oct 15 10:00:27: %HSRP-6-STATECHANGE: Vlan54 Grp 54 state Active -> Speak
*Oct 15 10:00:27: %HSRP-6-STATECHANGE: Vlan251 Grp 251 state Standby -> Active
*Oct 15 10:00:27: %HSRP-6-STATECHANGE: Vlan251 Grp 251 state Active -> Speak
*Oct 15 10:00:27: %SEC-6-IPACCESSLOGP: list ServeriIn permitted tcp 192.168.2.30(4001) -> 192.168.3.31(1045), 1 packet
*Oct 15 10:00:28: %HSRP-6-STATECHANGE: Vlan54 Grp 54 state Standby -> Active
*Oct 15 10:00:28: %SEC-6-IPACCESSLOGP: list ServeriIn permitted udp 192.168.2.19(1985) -> 224.0.0.2(1985), 1 packet
*Oct 15 10:00:29: %HSRP-6-STATECHANGE: Vlan251 Grp 251 state Standby -> Active
*Oct 15 10:00:29: %SEC-6-IPACCESSLOGP: list ServeriOut permitted udp 192.168.3.120(137) -> 192.168.2.2(137), 1 packet
*Oct 15 10:00:30: %HSRP-6-STATECHANGE: Vlan251 Grp 251 state Active -> Speak
*Oct 15 10:00:31: %SEC-6-IPACCESSLOGP: list ServeriIn permitted tcp 192.168.2.30(4001) -> 192.168.3.31(1045), 1 packet
*Oct 15 10:00:31: %HSRP-6-STATECHANGE: Vlan54 Grp 54 state Active -> Speak
*Oct 15 10:00:31: %HSRP-6-STATECHANGE: Vlan251 Grp 251 state Standby -> Active
*Oct 15 10:00:32: %SEC-6-IPACCESSLOGP: list ServeriOut permitted tcp 208.100.61.2(80) -> 192.168.2.3(47241), 1 packet
*Oct 15 10:00:32: %HSRP-6-STATECHANGE: Vlan54 Grp 54 state Standby -> Active
*Oct 15 10:00:33: %SEC-6-IPACCESSLOGP: list ServeriIn permitted tcp 192.168.2.4(1194) -> 192.168.252.20(2572), 1 packet
*Oct 15 10:00:34: %SEC-6-IPACCESSLOGP: list ServeriOut permitted tcp 192.168.244.116(2274) -> 192.168.2.3(8080), 1 packet
*Oct 15 10:00:34: %HSRP-6-STATECHANGE: Vlan251 Grp 251 state Active -> Speak
*Oct 15 10:00:35: %HSRP-6-STATECHANGE: Vlan54 Grp 54 state Active -> Speak
*Oct 15 10:00:35: %SEC-6-IPACCESSLOGP: list ServeriOut permitted tcp 192.168.244.116(1658) -> 192.168.2.3(8080), 1 packet
*Oct 15 10:00:35: %HSRP-6-STATECHANGE: Vlan251 Grp 251 state Standby -> Active
*Oct 15 10:00:35: %C4K_EBM-4-HOSTFLAPPING: Host 00:40:8C:95:21:54 in vlan 152 is flapping between port Gi2/15 and port Gi2/8
*Oct 15 10:00:36: %SEC-6-IPACCESSLOGP: list ServeriOut permitted udp 192.168.244.102(137) -> 192.168.2.2(137), 1 packet
*Oct 15 10:00:36: %HSRP-6-STATECHANGE: Vlan54 Grp 54 state Standby -> Active
*Oct 15 10:00:38: %SEC-6-IPACCESSLOGP: list ServeriOut permitted udp 192.168.244.102(138) -> 192.168.2.2(138), 1 packet
*Oct 15 10:00:38: %HSRP-6-STATECHANGE: Vlan54 Grp 54 state Active -> Speak
*Oct 15 10:00:39: %HSRP-6-STATECHANGE: Vlan251 Grp 251 state Active -> Speak
*Oct 15 10:00:39: %C4K_EBM-4-HOSTFLAPPING: Host 00:40:8C:93:BD:61 in vlan 152 is flapping between port Gi2/4 and port Gi2/6
*Oct 15 10:00:39: %C4K_EBM-4-HOSTFLAPPING: Host 00:40:8C:93:BD:64 in vlan 152 is flapping between port Gi2/1 and port Gi2/18
*Oct 15 10:00:39: %C4K_EBM-4-HOSTFLAPPING: Host 00:40:8C:93:B6:FF in vlan 152 is flapping between port Gi2/15 and port Gi2/10
*Oct 15 10:00:39: %C4K_EBM-4-HOSTFLAPPING: Host 00:40:8C:93:BD:1A in vlan 152 is flapping between port Gi2/10 and port Po1
*Oct 15 10:00:39: %SEC-6-IPACCESSLOGP: list ServeriIn permitted udp 192.168.2.31(138) -> 192.168.2.255(138), 1 packet
*Oct 15 10:00:40: %HSRP-6-STATECHANGE: Vlan251 Grp 251 state Standby -> Active
*Oct 15 10:00:40: %HSRP-6-STATECHANGE: Vlan251 Grp 251 state Active -> Speak
*Oct 15 10:00:40: %HSRP-6-STATECHANGE: Vlan54 Grp 54 state Standby -> Activ

Malloc / Free trace:
1 . pc=1025F4C0 addr=205BB3B0
2 . pc=27D22626 addr=205BB3B0
3 . pc=1025F354 addr=205BB068
4 . pc=27D227CA addr=205BB068
5 . pc=1025F4C0 addr=205BAF28
6 . pc=27D2286A addr=205BAF28
7 . pc=1025F354 addr=205BABE0
8 . pc=27D22A0E addr=205BABE0
9 . pc=1025F4C0 addr=205BAAA0
10. pc=27D22AAE addr=205BAAA0
11. pc=1025F354 addr=205BA758
12. pc=27D22C52 addr=205BA758
13. pc=103767B4 addr=192AA8D0
14. pc=103767B4 addr=192AA8D0
15. pc=103767B4 addr=192AA8D0
16. pc=103767B4 addr=192AA8D0
17. pc=103767B4 addr=12C58F88
18. pc=103767B4 addr=12C58F88
19. pc=103767B4 addr=12C58F88
20. pc=103767B4 addr=12C58F88
21. pc=103767B4 addr=12C58F88
22. pc=103767B4 addr=12C58F88
23. pc=103767B4 addr=12C58F88
24. pc=103767B4 addr=12C58F88
25. pc=103767B4 addr=12C58F88
26. pc=103767B4 addr=12C58F88
27. pc=103767B4 addr=12C58F88
28. pc=103767B4 addr=12C58F88
29. pc=103767B4 addr=12C58F88
30. pc=103767B4 addr=12C58F88
31. pc=103767B4 addr=12C58F88
32. pc=103767B4 addr=12C58F88
33. pc=1025F4C0 addr=205BA618
34. pc=27D22CF2 addr=205BA618
35. pc=1025F354 addr=205BA2D0
36. pc=27D22E96 addr=205BA2D0
37. pc=1025F4C0 addr=205BA190
38. pc=27D22F36 addr=205BA190
39. pc=1025F354 addr=205B9E48
40. pc=27D230DA addr=205B9E48
41. pc=1025F4C0 addr=205B9D08
42. pc=27D2317A addr=205B9D08
43. pc=1025F354 addr=205B99C0
44. pc=27D2331E addr=205B99C0
45. pc=1025F4C0 addr=205B9880
46. pc=27D233BE addr=205B9880
47. pc=1025F354 addr=205B9538
48. pc=27D23562 addr=205B9538
49. pc=1025F4C0 addr=205B93F8
50. pc=27D23602 addr=205B93F8
51. pc=1025F354 addr=205B90B0
52. pc=27D237A6 addr=205B90B0
53. pc=1025F4C0 addr=205B8F70
54. pc=27D23846 addr=205B8F70
55. pc=1025F354 addr=205B8C28
56. pc=27D239EA addr=205B8C28
57. pc=1025F4C0 addr=205B8AE8
58. pc=27D23A8A addr=205B8AE8
59. pc=1025F354 addr=205B87A0
60. pc=27D23C2E addr=205B87A0
61. pc=1025F4C0 addr=205B8660
62. pc=27D23CCE addr=205B8660
63. pc=1025F354 addr=205B8318
64. pc=27D23E72 addr=205B8318

Every time HSRP states is start to flap between two switches for minute or two and than I think bridging loop is formed, and after that primary switch is rebooted all by itself. After reboot all works again.

Anyone have an idea?

Calin C. · ‎10-18-2010

Hello!

Apparently, it's a software bug:

CSCsi17158 Bug Details

Multiple invalid ssh attempts crashes switch

Symptoms: Devices running Cisco IOS may reload with the error message "System returned to ROM by
abort at PC 0x0" when processing SSHv2 sessions. A switch crashes. We have a script running that will
continuously ssh-v2 into the 3560 then close the session normally. If the vty line that is being used by
SSHv2 sessions to the device is cleared while the SSH session is being processed, the next time an ssh
into the device is done, the device will crash.

Conditions: This problem is platform independent, but it has been seen on Cisco Catalyst 3560, Cisco
Catalyst 3750 and Cisco Catalyst 4948 series switches. The issue is specific to SSH version 2, and its
seen only when the box is under brute force attack. This crash is not seen under normal conditions.

Workaround: There are mitigations to this vulnerability:
For Cisco IOS, the SSH server can be disabled by applying the command crypto key
zeroize rsa while in configuration mode. The SSH server is enabled automatically upon
generating an RSA key pair. Zeroing the RSA keys is the only way to completely disable the SSH server.

Access to the SSH server on Cisco IOS may also be disabled via removing SSH as a valid transport
protocol. This can be done by reapplying the transport input command
with 'ssh' removed from the list of permitted transports on VTY lines while in configuration mode. For
example:
line vty 0 4
transport input telnet
end

If SSH server functionality is desired, access to the server can be restricted to specific source IP
addresses or blocked entirely using Access Control Lists (ACLs) on the VTY lines as shown in the
following URL:

http://www.cisco.com/en/US/docs/switches/lan/catalyst2950/software/release/12.1_9_ea1/configur
ation/guide/swacl.html#xtocid14

More information on configuring ACLs can be found on the Cisco public website:
http://www.cisco.com/warp/public/707/confaccesslists.html

Among affected versions:

12.2(25)EWA
12.2(25)EWA1
12.2(25)EWA2
12.2(25)EWA3
12.2(25)EWA4
12.2(25)EWA5
12.2(25)EWA6
12.2(25)EWA7
12.2(25)EWA8
12.2(25)EWA9
12.2(25)EWA10
12.2(25)EWA11

It's fixed in 12.2(25)EWA12

Personal advice, if you can, do an IOS upgrade!

Good luck!

Calin

belicns · ‎10-18-2010

Calin, thanks for replay but how can you be sure that this is SSHv2 issue. Nobody is connected to terminal lines at that time. There is no SSHv2 session at that time.

Calin C. · ‎10-18-2010

Actually, in case of a software bug you cannot be 100% sure, but according to

what to BUG description that I've posted, you don't need to have somebody logged in through SSH, it's enough an attack like brute-force to crash your device. I don't know your environment, if this devices are exposed to Internet so I cannot give you a security advice.

Searching more on the internet I've found this:

"If the address indicated is an invalid address out of the memory range, it is a software bug"

http://www.cisco.com/en/US/products/hw/switches/ps708/products_tech_note09186a008072c406.shtml

So, it's not only related to C4500 series, but in general.

Can't you run an update and see if the problems are fixed? As I understood you have HSRP there, so there is an alternate path if you put one device in maintenance mode.

Cheers,

Calin

Leo Laohoo · ‎10-18-2010

It's fixed in 12.2(25)EWA12

Personal advice, if you can, do an IOS upgrade!

12.2(25) ... wow. That's pretty ... ancient.

gephelps · ‎10-18-2010

If you have a SmartNet contract then open up a TAC Service Request and get the supervisor replaced. The crash here was not due to software. In the crashinfo data we can see the exception vector was set to 0x200. A value of 0x200 is machine check exception which indicates a hardware failure.