2960 Switch 48TC-L crashed

[Phuc Duong]

Hi All,

I had a Switch 2960 48TCL that had crashed without any reason. I found out some log information of this switch on log server as below. Is this a software issue. Do I need an IOS upgrade? 

Mar 4 01:26:52 192.168.9.72 : %TCP-2-INVALIDTCB: Invalid TCB pointer: 0x2723CFC -Process= "HTTP CORE", ipl= 0, pid= 204
Mar 4 08:28:58 192.168.9.72 -Traceback= E6A370z 10212E4z 1002288z 1009C10z C750Cz C7918z CEF14z C1640z 14A3F88z 149E538z
Mar 4 01:26:52 192.168.9.72 : %TCP-2-INVALIDTCB: Invalid TCB pointer: 0x31BF4B4 -Process= "HTTP CORE", ipl= 0, pid= 204
Mar 4 08:28:58 192.168.9.72 -Traceback= E6A370z 10212E4z 1002288z 1009C10z C750Cz C7918z CEF14z C1640z 14A3F88z 149E538z
Mar 4 01:26:52 192.168.9.72 : %TCP-2-INVALIDTCB: Invalid TCB pointer: 0x320155C -Process= "HTTP CORE", ipl= 0, pid= 204
Mar 4 08:28:58 192.168.9.72 -Traceback= E6A370z 10212E4z 1002288z 1009C10z C750Cz C7918z CEF14z C1640z 14A3F88z 149E538z
Mar 4 01:26:52 192.168.9.72 : %TCP-2-INVALIDTCB: Invalid TCB pointer: 0x409C678 -Process= "HTTP CORE", ipl= 0, pid= 204
Mar 4 08:28:58 192.168.9.72 -Traceback= E6A370z 10212E4z 1002288z 1009C10z C750Cz C7918z CEF14z C1640z 14A3F88z 149E538z
Mar 4 01:26:52 192.168.9.72 : %TCP-2-INVALIDTCB: Invalid TCB pointer: 0x412BD98 -Process= "HTTP CORE", ipl= 0, pid= 204
Mar 4 08:28:58 192.168.9.72 -Traceback= E6A370z 10212E4z 1002288z 1009C10z C750Cz C7918z CEF14z C1640z 14A3F88z 149E538z
Mar 4 01:26:52 192.168.9.72 : %TCP-2-INVALIDTCB: Invalid TCB pointer: 0x2709258 -Process= "HTTP CORE", ipl= 0, pid= 204
Mar 4 08:28:58 192.168.9.72 -Traceback= E6A370z 10212E4z 1002288z 1009C10z C750Cz C7918z CEF14z C1640z 14A3F88z 149E538z
Mar 4 01:28:39 192.168.9.72 : %TCP-2-INVALIDTCB: Invalid TCB pointer: 0x409B9AC -Process= "SSH Process", ipl= 0, pid= 245
Mar 4 08:30:49 192.168.9.72 -Traceback= E6A370z 10212E4z 10252CCz 1028AF4z 173041Cz 17398A8z 173A0C4z 173AC28z 1737D44z 1736190z 14A3F88z 149E538z
Mar 4 01:28:39 192.168.9.72 : %TCP-2-INVALIDTCB: Invalid TCB pointer: 0x409B9AC -Process= "SSH Process", ipl= 0, pid= 245
Mar 4 08:30:49 192.168.9.72 -Traceback= E6A370z 10212E4z 1026B94z 1026F58z 1730474z 17398A8z 173A0C4z 173AC28z 1737D44z 1736190z 14A3F88z 149E538z
Mar 4 08:33:17 192.168.9.72 Cisco IOS Software, C2960 Software (C2960-LANBASEK9-M), Version 15.0(2)SE6, RELEASE SOFTWARE (fc2)
Mar 4 08:33:17 192.168.9.72 Technical Support: http://www.cisco.com/techsupport
Mar 4 08:33:17 192.168.9.72 Copyright (c) 1986-2014 by Cisco Systems, Inc.
Mar 4 08:33:17 192.168.9.72 Compiled Wed 09-Apr-14 03:40 by prod_rel_team
Mar 4 01:31:58 192.168.9.72 : %PLATFORM-1-CRASHED: System previously crashed with the following message:
Mar 4 01:31:58 192.168.9.72 : %PLATFORM-1-CRASHED: Cisco IOS Software, C2960 Software (C2960-LANBASEK9-M), Version 15.0(2)SE6, RELEASE SOFTWARE (fc2)
Mar 4 01:31:58 192.168.9.72 : %PLATFORM-1-CRASHED: Technical Support: http://www.cisco.com/techsupport
Mar 4 01:31:58 192.168.9.72 : %PLATFORM-1-CRASHED: Copyright (c) 1986-2014 by Cisco Systems, Inc.
Mar 4 01:31:58 192.168.9.72 : %PLATFORM-1-CRASHED: Compiled Wed 09-Apr-14 03:40 by prod_rel_team
Mar 4 01:31:58 192.168.9.72 : %PLATFORM-1-CRASHED:
Mar 4 01:31:58 192.168.9.72 : %PLATFORM-1-CRASHED: Data TLB Miss Exception (0x1100)!
Mar 4 01:31:58 192.168.9.72 : %PLATFORM-1-CRASHED:
Mar 4 01:31:58 192.168.9.72 : %PLATFORM-1-CRASHED: SRR0 = 0x00D8F480 SRR1 = 0x00029230 SRR2 = 0x007D439C SRR3 = 0x00029230
Mar 4 01:31:58 192.168.9.72 : %PLATFORM-1-CRASHED: ESR = 0x00000000 DEAR = 0x0D0D1405 TSR = 0x84000000 DBSR = 0x00000000
Mar 4 01:31:58 192.168.9.72 : %PLATFORM-1-CRASHED:
Mar 4 01:31:58 192.168.9.72 : %PLATFORM-1-CRASHED: CPU Register Context:
Mar 4 01:31:58 192.168.9.72 : %PLATFORM-1-CRASHED: Vector = 0x00001100 PC = 0x00D8F480 MSR = 0x00029230 CR = 0x53000059
Mar 4 01:31:58 192.168.9.72 : %PLATFORM-1-CRASHED: LR = 0x01736274 CTR = 0x005DEE38 XER = 0xA0000075
Mar 4 01:31:58 192.168.9.72 : %PLATFORM-1-CRASHED: R0 = 0x01736274 R1 = 0x027629A8 R2 = 0x00000000 R3 = 0x0D0D0D0D
Mar 4 01:31:58 192.168.9.72 : %PLATFORM-1-CRASHED: R4 = 0x0321143C R5 = 0x03E5208C R6 = 0x00000000 R7 = 0x7A8FD377
Mar 4 01:31:58 192.168.9.72 : %PLATFORM-1-CRASHED: R8 = 0x00000000 R9 = 0x02230000 R10 = 0x00000000 R11 = 0x00000000
Mar 4 01:31:58 192.168.9.72 : %PLATFORM-1-CRASHED: R12 = 0x33000055 R13 = 0x00110000 R14 = 0x01735F00 R15 = 0x00000000
Mar 4 01:31:58 192.168.9.72 : %PLATFORM-1-CRASHED: R16 = 0x00000000 R17 = 0x00000000 R18 = 0x00000000 R19 = 0x00000000
Mar 4 01:31:58 192.168.9.72 : %PLATFORM-1-CRASHED: R20 = 0x00000000 R21 = 0x00000000 R22 = 0x00000000 R23 = 0x00000000
Mar 4 01:31:58 192.168.9.72 : %PLATFORM-1-CRASHED: R24 = 0x00000000 R25 = 0x00000000 R26 = 0x00000000 R27 = 0x00000000
Mar 4 01:31:58 192.168.9.72 : %PLATFORM-1-CRASHED: R28 = 0x00000000 R29 = 0x0172B6A8 R30 = 0x0409B9AC R31 = 0x0D0D0D0D
Mar 4 01:31:58 192.168.9.72 : %PLATFORM-1-CRASHED:
Mar 4 01:31:58 192.168.9.72 : %PLATFORM-1-CRASHED: Stack trace:
Mar 4 01:31:58 192.168.9.72 : %PLATFORM-1-CRASHED: PC = 0x00D8F480, SP = 0x027629A8
Mar 4 01:31:58 192.168.9.72 : %PLATFORM-1-CRASHED: Frame 00: SP = 0x027629D0 PC = 0x00000014
Mar 4 01:31:58 192.168.9.72 : %PLATFORM-1-CRASHED: Frame 01: SP = 0x027629F8 PC = 0x01736274
Mar 4 01:31:58 192.168.9.72 : %PLATFORM-1-CRASHED: Frame 02: SP = 0x02762A00 PC = 0x014A3F88
Mar 4 01:31:58 192.168.9.72 : %PLATFORM-1-CRASHED: Frame 03: SP = 0x00000000 PC = 0x0149E538
Mar 4 01:31:58 192.168.9.72 : %PLATFORM-1-CRASHED:

-------------------

Version of this switch

Cisco IOS Software, C2960 Software (C2960-LANBASEK9-M), Version 15.0(2)SE6, RELE ASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2014 by Cisco Systems, Inc.
Compiled Wed 09-Apr-14 03:40 by prod_rel_team

ROM: Bootstrap program is C2960 boot loader
BOOTLDR: C2960 Boot Loader (C2960-HBOOT-M) Version 15.0(2r)EZ1, RELEASE SOFTWARE (fc1)

Lab2_Core uptime is 4 days, 2 hours, 28 minutes
System returned to ROM by bus error at PC 0xD8F480, address 0x0
System restarted at 08:30:08 VietNam Sun Mar 4 2018
System image file is "flash:/c2960-lanbasek9-mz.150-2.SE6/c2960-lanbasek9-mz.150 -2.SE6.bin"

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C2960+48TC-L (PowerPC405) processor (revision G0) with 131072K bytes of memory.
Processor board ID FOC1939Y1JP
Last reset from bus error
4 Virtual Ethernet interfaces
48 FastEthernet interfaces
2 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

64K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address : 9C:57:AD:D8:91:00
Motherboard assembly number : 73-15619-01
Power supply part number : 341-0097-03
Motherboard serial number : FOC19393P71
Power supply serial number : DCA193480FY
Model revision number : G0
Motherboard revision number : B0
Model number : WS-C2960+48TC-L
System serial number : FOC1939Y1JP
Top Assembly Part Number : 800-40260-02
Top Assembly Revision Number : B0
Version ID : V02
CLEI Code Number : CMMKR00ARB
Hardware Board Revision Number : 0x0C

Switch Ports Model SW Version SW Image
------ ----- ----- ---------- ----------
* 1 50 WS-C2960+48TC-L 15.0(2)SE6 C2960-LANBASEK9-M

[Mark Malone]

System returned to ROM by bus error at PC 0xD8F480, address 0x0

Yes you have hit a bug first i would try get the ID , is the device under support with TAC ?
if not can you post the show tech ill see if i can get it from analyzer or just take a chance and upgarde but you may hit it again without the bug id

[Phuc Duong]

Hi Mark,

Thanks for your help.

if not can you post the show tech ill see if i can get it from analyzer or just take a chance and upgarde but you may hit it again without the bug id

=> Is it true that you mean the command " show tech-support"??

[Mark Malone]

Yes the full show tech then you dump it into the analyzer if the caveat is known it will provide the bug id then you can see which ios release its not in to upgrade too , you can do it yourself too download analyzer in tools in Cisco website its free , if analyzer doesn't see anything you could TAC it then

[Phuc Duong]

Hi Mark,

Thanks for your help. I tried it but my Cisco account is limited.

[Mark Malone · Accepted solution]

ok if you want to post he show tech here i will run it for you and provide the results

[Phuc Duong]

Hi Mark,

Here is what I got from command " show tech". Thanks a lot for your greate support. 

[Mark Malone · Accepted solution]

Unfortunately it did not give a bug id , it has advised to open with TAC as below output , is there a crash file in flash too ?

if no support with Cisco i would upgrade the switch to this image version , this is what Cisco currently recommends this switch uses , the image your on is from 2014, this is from July 2017 c2960-lanbasek9-mz.152-2.E7.bin

https://sds-prod.cisco.com/download/home/284776533/type/280805680/release/15.2.2E7


The system encounters a bus error when the processor tries to access a memory location that either does not exist (a software error) or does not respond properly (a hardware problem). A bus error can be identified from the output of the show version command provided by the router if not power-cycled or manually reloaded.

Please collect the following and engage the TAC:
- show tech
- all crashinfo files (instructions for collecting the crashinfo files can be found here.)
For details please refer to: http://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/ios-software-releases-121-mainline/7949-crashes-buserror-troubleshooting.html

[Phuc Duong]

Hi Mark,

Here are 2 crash files. Thanks 

[Mark Malone · Accepted solution]

Hi
start with the upgrade there seems to be multiple TCB software bugs when i search for it , i would be nearly sure now its software issue triggering it and not hardware

[Phuc Duong]

Hi Mark,

I'll upgrade IOS to the newest version at this weekend as your suggestion. Thanks for your great support.

[Georg Pauwen]

Hello,

on a side note, check if you have hung TCP sessions. The command 'sh tcp brief' will generate an output as the one below:

2960#sh tcp brief
TCB            Local Address     Foreign Address       (state)
6153D5B8 192.168.1.11.22 192.168.3.12.12871 SYNRCVD
64565M12 192.168.1.11.22 192.168.3.12.12814 SYNRCVD

Anything in the SYNRCVD state is hung. Clear the session(s) that are hung with the command:

2960#clear tcp tcb 6153D5B8

[Phuc Duong]

Hi Georg,

Thanks for your help. I've already tried this command but there is only my telnet session at this time.

Lab2_Core#sh tcp brief
TCB Local Address Foreign Address (state)
03E4ACA0 192.168.9.72.23 192.168.9.210.47730 ESTAB

[ysotocob]

I reviewed the information on the crashfiles and found that the root cause of the crash is a CPU hog on the switch due to HTTP and SSH sessions not properly closed on the switch.

%TCP-2-INVALIDTCB: Invalid TCB pointer: 0x2709258 -Process= "HTTP CORE", ipl= 0, pid= 204

%TCP-2-INVALIDTCB: Invalid TCB pointer: 0x409B9AC -Process= "SSH Process", ipl= 0, pid= 245

I tracked a bug that was released in order to properly address this CPU hog situation:

Cisco device unexpectedly reloads after TCP session timeout

Unfortunally the IOS you are using 15.0(2)SE6 was out of support when the bug was identified and the the development team did not provide a fix for this version. Right now you have the following configuration:

ip http server
ip http authentication local
ip http secure-server

If that was enable by default please disable the http commands in order to avoid another crash related to this situation. Proactivly you can upgrade the device to the most recent IOS version 15.0.2-SE11.

https://software.cisco.com/download/home/279963472/type/280805680/release/15.0.2-SE11

- Regarding the source of those HTTP connections, you should track down source and destination IP addresses in your network and verify if there is some unauthorized application or any kind of virus causing this behaviour.

- Suggested commands:

   *Show tcp brief 

   *Cear tcp tcb TCB_ADDRESS (for sessions closed)

Can be helpfull in case you notice the previous TCB error logs in the swtich during its normal operation for the TCP sessions (HTTP & SSH in your case).