2960S 12.2(58)SE Crash on SSH

[jan.b.brooks]

I just upgraded several of my C2960S-48FPD-L from 12.2(55)SE to 12.2(58)SE and now almost every time I try to SSH to it the switch reboots and gives me these tracebacks ...

07:58:52 CST Thu Apr 14 2011: Unexpected exception to CPUvector 200, PC = 1EADA28

-Traceback= 0x1EADA28z 0x1CDC324z 0x1EA5520z 0x1CDC35Cz 0x1CDD0E0z 0x1CDF0C4z 0x1CE06CCz 0x1CE07F8z 0x1D67348z 0x1D611BCz

Writing crashinfo to flash:/crashinfo_ext/crashinfo_ext_3

07:58:56 CST Thu Apr 14 2011: Unexpected exception to CPUvector 1100, PC = 121BB98

-Traceback= 0x121BB98z 0x12250FCz 0xFA7540z 0xA807B8z 0xA75DF4z 0xA78080z 0x94B708z 0x9717C0z 0x918DD8z 0x936520z 0x9344BCz 0x9345FCz 0x93484Cz 0x1D67348z 0x1D611BCz

=== Flushing messages (07:58:56 CST Thu Apr 14 2011) ===

Buffered messages: (last 4096 bytes only)

0x00000005  R6 = 0x01224F44  R7 = 0x02A23C6C

000089: *Feb 28 18:03:13.577 CST: %PLATFORM-1-CRASHED: R8 = 0x00000080  R9 = 0x00000000  R10 = 0x00000000  R11 = 0x00000071

000090: *Feb 28 18:03:13.577 CST: %PLATFORM-1-CRASHED: R12 = 0x00034855  R13 = 0x00000280  R14 = 0x01CE0778  R15 = 0x00000000

000091: *Feb 28 18:03:13.577 CST: %PLATFORM-1-CRASHED: R16 = 0x00000000  R17 = 0x00000000  R18 = 0x00000000  R19 = 0x00000000

000092: *Feb 28 18:03:13.577 CST: %PLATFORM-1-CRASHED: R20 = 0x00000000  R21 = 0x00000000  R22 = 0x00000000  R23 = 0x00000000

000093: *Feb 28 18:03:13.577 CST: %PLATFORM-1-CRASHED: R24 = 0x030E94A4  R25 = 0x030E9448  R26 = 0x02A20000  R27 = 0x02E70000

000094: *Feb 28 18:03:13.577 CST: %PLATFORM-1-CRASHED: R28 = 0x00000000  R29 = 0x04830AC0  R30 = 0x01CDC308  R31 = 0x6F6B7300

000095: *Feb 28 18:03:13.577 CST: %PLATFORM-1-CRASHED:

000096: *Feb 28 18:03:13.577 CST: %PLATFORM-1-CRASHED: Stack trace:

000097: *Feb 28 18:03:13.577 CST: %PLATFORM-1-CRASHED: PC = 0x01EADA20, SP = 0x04830A88

000098: *Feb 28 18:03:13.577 CST: %PLATFORM-1-CRASHED: Frame 00: SP = 0x04830A90    PC = 0x00034850

000099: *Feb 28 18:03:13.582 CST: %PLATFORM-1-CRASHED: Frame 01: SP = 0x04830AA8    PC = 0x01EA5520

000100: *Feb 28 18:03:13.582 CST: %PLATFORM-1-CRASHED: Frame 02: SP = 0x04830AB0    PC = 0x01CDC35C

000101: *Feb 28 18:03:13.582 CST: %PLATFORM-1-CRASHED: Frame 03: SP = 0x04830AE8    PC = 0x01CDD0E0

000102: *Feb 28 18:03:13.582 CST: %PLATFORM-1-CRASHED: Frame 04: SP = 0x04830B00    PC = 0x01CDF0C4

000103: *Feb 28 18:03:13.582 CST: %PLATFORM-1-CRASHED: Frame 05: SP = 0x04830B40    PC = 0x01CE06CC

000104: *Feb 28 18:03:13.582 CST: %PLATFORM-1-CRASHED: Frame 06: SP = 0x04830B78    PC = 0x01CE07F8

000105: *Feb 28 18:03:13.582 CST: %PLATFORM-1-CRASHED: Frame 07: SP = 0x04830B80    PC = 0x01D67348

000106: *Feb 28 18:03:13.582 CST: %PLATFORM-1-CRASHED: Frame 08: SP = 0x00000000    PC = 0x01D611BC

000107: *Feb 28 18:03:13.582 CST: %PLATFORM-1-CRASHED:

000070: *Feb 28 18:03:13.603 CST: %PLATFORM-1-CRASHED: System previously crashed with the following message:

000071: *Feb 28 18:03:13.603 CST: %PLATFORM-1-CRASHED: Cisco IOS Software, C2960S Software (C2960S-UNIVERSALK9-M), Version 12.2(58)SE, RELEASE SOFTWARE (fc1)

000072: *Feb 28 18:03:13.603 CST: %PLATFORM-1-CRASHED: Technical Support: http://www.cisco.com/techsupport

000073: *Feb 28 18:03:13.603 CST: %PLATFORM-1-CRASHED: Copyright (c) 1986-2011 by Cisco Systems, Inc.

000074: *Feb 28 18:03:13.603 CST: %PLATFORM-1-CRASHED: Compiled Tue 05-Apr-11 08:59 by prod_rel_team

000075: *Feb 28 18:03:13.603 CST: %PLATFORM-1-CRASHED: 

000076: *Feb 28 18:03:13.603 CST: %PLATFORM-1-CRASHED: Machine Check Exception (0x0200)!

000077: *Feb 28 18:03:13.603 CST: %PLATFORM-1-CRASHED: 

000078: *Feb 28 18:03:13.603 CST: %PLATFORM-1-CRASHED: ESR: 0x00000000

000079: *Feb 28 18:03:13.603 CST: %PLATFORM-1-CRASHED: SRR0: 0x0122203C SRR1: 0x00029230 SRR2: 0x01EADA28 SRR3: 0x00029230

000080: *Feb 28 18:03:13.603 CST: %PLATFORM-1-CRASHED: PLB4A0 - ESRH: 0x00000000 ESRL: 0x08000000 EARH: 0x00000000 EARL: 0x10000000

000081: *Feb 28 18:03:13.603 CST: %PLATFORM-1-CRASHED: PLB4A1 - ESRH: 0x00000000 ESRL: 0x00000000 EARH: 0x9EF5EDE3 EARL: 0xF9B28B67

000082: *Feb 28 18:03:13.603 CST: %PLATFORM-1-CRASHED: SDRAM0 - BESR: 0x00000000 BEARH: 0x00000000 BEARL: 0x00000000

000083: *Feb 28 18:03:13.603 CST: %PLATFORM-1-CRASHED: PEGPL0 - ESR: 0x92000000 EARH: 0x00000000 EARL: 0xDEADBEFB EATR: 0x82000000

000084: *Feb 28 18:03:13.603 CST: %PLATFORM-1-CRASHED: PEGPL1 - ESR: 0x00000000 EARH: 0x00000000 EARL: 0x00000000 EATR: 0x00000000

000085: *Feb 28 18:03:13.603 CST: %PLATFORM-1-CRASHED: PLB42OPB0 - BESR0: 0x00000000 BESR1: 0x00000000

000086: *Feb 28 18:03:13.603 CST: %PLATFORM-1-CRASHED:             BEARH: 0x0000000F BEARL: 0x102B540D

000087: *Feb 28 18:03:13.603 CST: %PLATFORM-1-CRASHED: OPB2PLB40 - BSTAT: 0x00000000 BEARH: 0x0000000F BEARL: 0xFFFFFFE7

000088: *Feb 28 18:03:13.603 CST: %PLATFORM-1-CRASHED: AHB - SESR: 0x00000000 SEARU: 0x00000000 SEARL: 0x00000000

000089: *Feb 28 18:03:13.603 CST: %PLATFORM-1-CRASHED: 

000090: *Feb 28 18:03:13.603 CST: %PLATFORM-1-CRASHED: CPU Register Context:

000091: *Feb 28 18:03:13.603 CST: %PLATFORM-1-CRASHED: Vector = 0x00000200  PC = 0x01EADA28  MSR = 0x00029230  CR = 0x50000035

000092: *Feb 28 18:03:13.609 CST: %PLATFORM-1-CRASHED: LR = 0x01CDC324  CTR = 0x00000000  XER = 0xC0000000

000093: *Feb 28 18:03:13.609 CST: %PLATFORM-1-CRASHED: R0 = 0x00000000  R1 = 0x04836D88  R2 = 0x00000000  R3 = 0xDEADBEFC

000094: *Feb 28 18:03:13.609 CST: %PLATFORM-1-CRASHED: R4 = 0x04836DCD  R5 = 0x00000005  R6 = 0x01224F44  R7 = 0x02A23C6C

000095: *Feb 28 18:03:13.609 CST: %PLATFORM-1-CRASHED: R8 = 0x00000080  R9 = 0x00000004  R10 = 0x00000000  R11 = 0x00000039

000096: *Feb 28 18:03:13.609 CST: %PLATFORM-1-CRASHED: R12 = 0x000BAAA8  R13 = 0x00000280  R14 = 0x01CE0778  R15 = 0x00000000

000097: *Feb 28 18:03:13.609 CST: %PLATFORM-1-CRASHED: R16 = 0x00000000  R17 = 0x00000000  R18 = 0x00000000  R19 = 0x00000000

000098: *Feb 28 18:03:13.609 CST: %PLATFORM-1-CRASHED: R20 = 0x00000000  R21 = 0x00000000  R22 = 0x00000000  R23 = 0x00000000

000099: *Feb 28 18:03:13.609 CST: %PLATFORM-1-CRASHED: R24 = 0x030E94A4  R25 = 0x030E9448  R26 = 0x02A20000  R27 = 0x02E70000

000100: *Feb 28 18:03:13.609 CST: %PLATFORM-1-CRASHED: R28 = 0x00000000  R29 = 0x04836DC0  R30 = 0x01CDC308  R31 = 0xDEADBEEF

000101: *Feb 28 18:03:13.609 CST: %PLATFORM-1-CRASHED: 

000102: *Feb 28 18:03:13.609 CST: %PLATFORM-1-CRASHED: Stack trace:

000103: *Feb 28 18:03:13.609 CST: %PLATFORM-1-CRASHED: PC = 0x01EADA28, SP = 0x04836D88

000104: *Feb 28 18:03:13.609 CST: %PLATFORM-1-CRASHED: Frame 00: SP = 0x04836D90    PC = 0x000BAA9E

000105: *Feb 28 18:03:13.609 CST: %PLATFORM-1-CRASHED: Frame 01: SP = 0x04836DA8    PC = 0x01EA5520

000106: *Feb 28 18:03:13.609 CST: %PLATFORM-1-CRASHED: Frame 02: SP = 0x04836DB0    PC = 0x01CDC35C

000107: *Feb 28 18:03:13.609 CST: %PLATFORM-1-CRASHED: Frame 03: SP = 0x04836DE8    PC = 0x01CDD0E0

000108: *Feb 28 18:03:13.609 CST: %PLATFORM-1-CRASHED: Frame 04: SP = 0x04836E00    PC = 0x01CDF0C4

000109: *Feb 28 18:03:13.609 CST: %PLATFORM-1-CRASHED: Frame 05: SP = 0x04836E40    PC = 0x01CE06CC

000110: *Feb 28 18:03:13.609 CST: %PLATFORM-1-CRASHED: Frame 06: SP = 0x04836E78    PC = 0x01CE07F8

000111: *Feb 28 18:03:13.609 CST: %PLATFORM-1-CRASHED: Frame 07: SP = 0x04836E80    PC = 0x01D67348

000112: *Feb 28 18:03:13.609 CST: %PLATFORM-1-CRASHED: Frame 08: SP = 0x00000000    PC = 0x01D611BC

000113: *Feb 28 18:03:13.609 CST: %PLATFORM-1-CRASHED: 

[Antonio Knox]

Sounds like a bug to me.  You should check out the Bug Toolkit.

http://www.cisco.com/cgi-bin/Support/Bugtool/launch_bugtool.pl

[Martin Parry · Accepted solution]

Hi

I believe you may have hit the following caveat:

CSCto62631

A switch running Cisco IOS Release 12.2(58)SE might reload if:

–SSH version 2 is configured on the switch, and

–a customized login banner was configured by using the banner login message global configuration command

Use one of these workarounds:

–Disable the login banner by entering the no login banner command.

–Disable SSH on the switch.

–Downgrade to a software version prior to Cisco IOS Release 12.2(58)SE.

For more details on caveats within the code please see the release notes:

http://www.cisco.com/en/US/partner/docs/switches/lan/catalyst3750/software/release/12.2_58_se/release/notes/OL24337.html#wp1042774

Hope this helps

Martin

[redman]

I have experienced this issue as well on multiple C2960G-24TC-L switches. Back-reving to 12.2.52SE worked for us.

[jan.b.brooks]

Looks like this fixed it ... Thanks for the help

[brharden]

12.2(58)SE has been pulled from Cisco.com due to the bug that Martin mentioned, CSCto62631.  Basically the switch will crash every other SSH session into the switch. The first SSH session will work fine, the second will crash the device.

A release with the fix for this issue should be posted to Cisco.com in mid-May as 12.2(58)SE1

-Brian

PS Another workaround is to use SSHv1

[Leo Laohoo]

Thanks for the update Brian.

Nuts.  This means I've got to stop upgrading the 2960/2960S.  I've already done >200 units out of my 600. 

[Leo Laohoo]

Right.  12.2(58)SE1 has been released, as scheduled.

Release Notes for the Catalyst 3750, 3560, 2960-S, and 2960 Switches, Cisco IOS Release 12.2(58)SE1

Cisco IOS Release 12.2(58)SE1 and later does NOT support all the Catalyst 3750 and 3560 switches. The models listed below are NOT supported in this release. For ongoing maintenance rebuilds for these switches, use Cisco IOS Release 12.2(55)SE and later (SE1, SE2, and so on).

• WS-C3560-24TS

•WS-C3560-24PS

•WS-C3560-48PS

•WS-C3560-48TS

•WS-C3750-24PS

•WS-C3750-24TS

•WS-C3750-48PS

•WS-C3750-48TS

•WS-3750G-24T

•WS-C3750G-12

•WS-C3750G-24TS

•WS-C3750G-16TD

[John Trumbell]

I'm having a similiar issue, on our 3750 on 12.2.(25)SE IOS. Does any one know if there was a similiar issue with that IOS?

thanks

John

[Leo Laohoo]

Use IOS version 12.2(55)SE5 or SE6.