Problems with hanging MSFC2

[alanbcameron]

I have a Multilayer Switch Feature (WS-F6K-MSFC2) installed in a 6509 chassis.

The MSFC hangs periodically where no traffic traverses the switch and from the supervisor I must issue a reset to the device.

It will recover after a fashion.

It has a redundant 6509 which handles BGP duties as well.

I am not certain if this is related, but after issuing the reset  on the primary switch, I must log on to the

backup switch's router and issue a clear ip bgp * command to resume WAN access.

I thought this was a hardware problem. I am being told though by my hardware support this

maybe more a software problem.

I have not made configuration changes to either of these switches in at least a year.

But in reviewing the supervisor log, I see my MSFC has reset a few time in the past

30 days anyway.

Is this a behavior anyone has observed and if so what was the resolution?

Thanks

[sachinraja]

Hi

Can you post the "show log" of your MSFC ? Can you also let us know what is the IOS version running on the MSFC ? Did you check the release notes to see if there are any open caveats related to hardware reset ? Is the other SUP working good always ?

Raj

[acameron12]

Do you mean the show log or the show logging command?

[acameron12]

This is in my show logging command:

cat6500A> show logging buffer 1023
2010 Jan 06 14:11:46 %PAGP-5-PORTTOSTP:Port 4/22 joined bridge port 4/22
2010 Jan 06 14:11:46 %PAGP-5-PORTTOSTP:Port 4/23 joined bridge port 4/23
2010 Jan 06 14:12:17 %PAGP-5-PORTFROMSTP:Port 2/13 left bridge port 2/13
2010 Jan 06 14:12:33 %PAGP-5-PORTTOSTP:Port 2/13 joined bridge port 2/13
2010 Jan 06 17:27:31 %PAGP-5-PORTTOSTP:Port 5/11 joined bridge port 5/11
2010 Jan 09 19:27:09 %SYS-5-MOD_OK:Module 15 is online
2010 Jan 14 10:58:11 %SYS-5-MOD_OK:Module 15 is online
2010 Jan 14 11:32:09 %SYS-5-MOD_OK:Module 15 is online
2010 Jan 16 13:31:40 %PAGP-5-PORTFROMSTP:Port 3/28 left bridge port 3/28
2010 Jan 16 13:31:40 %PAGP-5-PORTFROMSTP:Port 4/10 left bridge port 4/10
2010 Jan 16 13:31:57 %PAGP-5-PORTTOSTP:Port 3/28 joined bridge port 3/28
2010 Jan 16 13:31:57 %PAGP-5-PORTTOSTP:Port 4/10 joined bridge port 4/10
2010 Jan 16 13:32:10 %PAGP-5-PORTFROMSTP:Port 4/10 left bridge port 4/10
2010 Jan 16 13:32:27 %PAGP-5-PORTTOSTP:Port 4/10 joined bridge port 4/10
2010 Jan 16 13:32:50 %PAGP-5-PORTFROMSTP:Port 3/28 left bridge port 3/28
2010 Jan 16 13:32:51 %PAGP-5-PORTFROMSTP:Port 4/10 left bridge port 4/10
2010 Jan 16 13:33:07 %PAGP-5-PORTTOSTP:Port 3/28 joined bridge port 3/28
2010 Jan 16 13:33:08 %PAGP-5-PORTTOSTP:Port 4/10 joined bridge port 4/10
2010 Jan 16 13:38:48 %PAGP-5-PORTFROMSTP:Port 3/28 left bridge port 3/28
2010 Jan 16 13:38:49 %PAGP-5-PORTFROMSTP:Port 4/10 left bridge port 4/10
2010 Jan 16 13:39:06 %PAGP-5-PORTTOSTP:Port 3/28 joined bridge port 3/28
2010 Jan 16 13:39:06 %PAGP-5-PORTTOSTP:Port 4/10 joined bridge port 4/10
2010 Jan 16 13:39:18 %PAGP-5-PORTFROMSTP:Port 4/10 left bridge port 4/10
2010 Jan 16 13:39:35 %PAGP-5-PORTTOSTP:Port 4/10 joined bridge port 4/10
2010 Jan 16 13:39:58 %PAGP-5-PORTFROMSTP:Port 3/28 left bridge port 3/28
2010 Jan 16 13:39:59 %PAGP-5-PORTFROMSTP:Port 4/10 left bridge port 4/10
2010 Jan 16 13:40:16 %PAGP-5-PORTTOSTP:Port 4/10 joined bridge port 4/10
2010 Jan 16 13:40:27 %PAGP-5-PORTTOSTP:Port 3/28 joined bridge port 3/28
2010 Jan 16 13:43:37 %PAGP-5-PORTTOSTP:Port 2/44 joined bridge port 2/44
2010 Jan 16 13:43:37 %PAGP-5-PORTTOSTP:Port 4/36 joined bridge port 4/36
2010 Jan 16 13:46:29 %PAGP-5-PORTFROMSTP:Port 4/36 left bridge port 4/36
2010 Jan 16 13:46:32 %PAGP-5-PORTFROMSTP:Port 2/44 left bridge port 2/44
2010 Jan 16 13:46:50 %PAGP-5-PORTTOSTP:Port 4/36 joined bridge port 4/36
2010 Jan 16 13:46:50 %PAGP-5-PORTTOSTP:Port 2/44 joined bridge port 2/44
2010 Jan 17 13:04:11 %SYS-5-MOD_OK:Module 15 is online
2010 Jan 20 00:38:34 %PAGP-5-PORTFROMSTP:Port 5/43 left bridge port 5/43
2010 Jan 20 00:39:21 %PAGP-5-PORTTOSTP:Port 5/43 joined bridge port 5/43
2010 Jan 20 00:40:36 %PAGP-5-PORTFROMSTP:Port 5/43 left bridge port 5/43
2010 Jan 20 00:40:53 %PAGP-5-PORTTOSTP:Port 5/43 joined bridge port 5/43
2010 Jan 20 00:44:01 %PAGP-5-PORTFROMSTP:Port 5/46 left bridge port 5/46
2010 Jan 20 00:45:02 %PAGP-5-PORTTOSTP:Port 5/46 joined bridge port 5/46
2010 Jan 20 00:46:15 %PAGP-5-PORTFROMSTP:Port 5/46 left bridge port 5/46
2010 Jan 20 00:46:32 %PAGP-5-PORTTOSTP:Port 5/46 joined bridge port 5/46
2010 Jan 29 07:50:25 %SYS-5-MOD_OK:Module 15 is online
2010 Jan 29 19:29:06 %PAGP-5-PORTFROMSTP:Port 4/1 left bridge port 4/1
2010 Jan 29 19:29:26 %PAGP-5-PORTTOSTP:Port 4/1 joined bridge port 4/1
2010 Jan 29 19:29:34 %PAGP-5-PORTFROMSTP:Port 4/1 left bridge port 4/1
2010 Jan 29 19:29:55 %PAGP-5-PORTTOSTP:Port 4/1 joined bridge port 4/1
2010 Jan 29 19:29:59 %PAGP-5-PORTFROMSTP:Port 4/1 left bridge port 4/1
2010 Jan 29 19:30:17 %PAGP-5-PORTTOSTP:Port 4/1 joined bridge port 4/1
2010 Jan 29 19:47:20 %PAGP-5-PORTFROMSTP:Port 8/47 left bridge port 8/47
2010 Jan 29 19:47:40 %PAGP-5-PORTTOSTP:Port 8/47 joined bridge port 8/47
2010 Jan 29 19:47:48 %PAGP-5-PORTFROMSTP:Port 8/47 left bridge port 8/47
2010 Jan 29 19:48:09 %PAGP-5-PORTTOSTP:Port 8/47 joined bridge port 8/47
2010 Jan 29 19:48:14 %PAGP-5-PORTFROMSTP:Port 8/47 left bridge port 8/47
2010 Jan 29 19:48:31 %PAGP-5-PORTTOSTP:Port 8/47 joined bridge port 8/47
2010 Feb 03 01:44:43 %SYS-5-MOD_NOSCPPINGRESPONSE:Module 15 not responding... resetting module
2010 Feb 03 01:44:43 %SYS-5-MOD_RESET:Module 15 reset from Software
2010 Feb 03 01:55:30 %SYS-5-MOD_OK:Module 15 is online
2010 Feb 03 03:53:49 %SYS-5-MOD_OK:Module 15 is online
2010 Feb 03 03:54:46 %SYS-5-MOD_OK:Module 15 is online
2010 Feb 03 08:31:45 %SYS-5-MOD_OK:Module 15 is online
2010 Feb 03 23:47:00 %SPANTREE-2-RX_1QNON1QTRUNK: Rcved 1Q-BPDU on non-1Q-trunk port 8/43 vlan 210
2010 Feb 03 23:47:15 %SPANTREE-2-PORTUNBLK: Unblock previously inc port 8/43 on VLAN 210
2010 Feb 04 01:34:57 %PAGP-5-PORTFROMSTP:Port 6/32 left bridge port 6/32
2010 Feb 04 01:35:14 %PAGP-5-PORTTOSTP:Port 6/32 joined bridge port 6/32
2010 Feb 04 01:36:18 %PAGP-5-PORTFROMSTP:Port 6/32 left bridge port 6/32
2010 Feb 04 01:36:36 %PAGP-5-PORTTOSTP:Port 6/32 joined bridge port 6/32
2010 Feb 04 01:40:12 %PAGP-5-PORTFROMSTP:Port 6/32 left bridge port 6/32
2010 Feb 04 01:40:37 %PAGP-5-PORTTOSTP:Port 6/32 joined bridge port 6/32
2010 Feb 04 04:38:57 %SYS-5-MOD_RESET:Module 15 reset from telnet/172.16.199.62/
2010 Feb 04 04:40:03 %SYS-5-MOD_OK:Module 15 is online
2010 Feb 04 08:58:18 %PAGP-5-PORTFROMSTP:Port 5/4 left bridge port 5/4
2010 Feb 04 09:07:35 %PAGP-5-PORTFROMSTP:Port 8/43 left bridge port 8/43
2010 Feb 05 05:26:47 %SYS-5-MOD_RESET:Module 15 reset from telnet/172.16.199.62/
2010 Feb 05 05:29:00 %SYS-5-MOD_OK:Module 15 is online
2010 Feb 05 05:35:01 %SYS-5-MOD_RESET:Module 15 reset from telnet/172.16.199.62/
2010 Feb 05 05:35:38 %SYS-3-MOD_FAIL:Module 15 failed to come online
2010 Feb 05 05:40:21 %SYS-5-MOD_RESET:Module 15 reset from telnet/172.16.199.62/
2010 Feb 05 05:45:18 %PAGP-5-PORTFROMSTP:Port 3/35 left bridge port 3/35
2010 Feb 05 05:49:39 %PAGP-5-PORTTOSTP:Port 3/35 joined bridge port 3/35
2010 Feb 05 05:49:58 %SYS-5-MOD_RESET:Module 15 reset from telnet/172.16.199.62/
2010 Feb 05 05:57:24 %SYS-5-MOD_OK:Module 15 is online
2010 Feb 05 06:22:46 %PAGP-5-PORTFROMSTP:Port 3/35 left bridge port 3/35
2010 Feb 05 06:23:01 %SYS-5-MOD_OK:Module 15 is online
2010 Feb 05 07:30:48 %SYS-5-MOD_RESET:Module 15 reset from telnet/172.16.199.62/
2010 Feb 05 07:35:29 %SYS-5-MOD_OK:Module 15 is online
2010 Feb 05 08:01:55 %SYS-5-MOD_RESET:Module 15 reset from telnet/172.16.199.62/
2010 Feb 05 08:12:47 %SYS-5-MOD_RESET:Module 15 reset from telnet/172.16.199.62/
2010 Feb 05 08:15:05 %SYS-5-MOD_OK:Module 15 is online
2010 Feb 05 14:31:46 %SYS-5-MOD_OK:Module 15 is online
2010 Feb 05 15:03:54 %SYS-5-MOD_OK:Module 15 is online
cat6500A>

[sachinraja]

Hi Alan

I just went through the release notes, and found the following bug.. might be related to your issue: You can probably update your software to 8.4(4) to solve this issue.. this is more related to flapping of interface counters, which is causing your SUP to reset...

CSCeh84332 Bug Details

Line card reset due to excessive Link changes

Under Certain Circumstances constellation classic cards with certain port asic type (WS-X6148-45AF, WS-X6348-RJ-45,...) may reset with following Error. 2005 May 06 13:02:24 %SYS-5-MOD_NOSCPPINGRESPONSE:Module 9 not responding... resetting module 2005 May 06 13:02:24 %SYS-5-MOD_RESET:Module 9 reset from Software 2005 May 06 13:04:07 %SYS-5-MOD_OK:Module 9(WS-X6148-45AF,SAL08175TMG) is online This reset is not because of BAD hardware and is caused if some port/ports on the particular Line card are flapping abnormally with high rate. Work Around: Check the linkChange counter in the out put of Sh counters. If this has huge number of link changes then it could cause line card to get reset. Fixing the frequent Link change issue will avoid reset of line card. Also new software will have the fix where system will not reset the line card if their is rapid link changes

Hope this helps.. all the best..

Raj

[acameron12]

This is output from show log:

cat6500A> sho log

Network Management Processor (ACTIVE NMP) Log:
  Reset count:   8
  Re-boot History:   Oct 29 2005 05:38:21 0, Oct 29 2005 05:27:38 0
                     Oct 29 2005 05:21:34 0, Nov 14 2001 14:58:15 0
                     Nov 12 2001 07:24:41 0, Oct 02 2001 08:39:43 0
                     Oct 02 2001 07:44:20 0, Oct 01 2001 11:22:05 0
  Bootrom Checksum Failures:      0   UART Failures:                  0
  Flash Checksum Failures:        0   Flash Program Failures:         0
  Power Supply 1 Failures:        1   Power Supply 2 Failures:        4
  Swapped to CLKA:                0   Swapped to CLKB:                0
  Swapped to Processor 1:         0   Swapped to Processor 2:         0
  DRAM Failures:                  0

  Exceptions:                     0

  Loaded NMP version:            6.1(4)
  Reload same NMP version count: 5

  Last software reset by user: 11/14/2001,14:57:07

  EOBC Exceptions/Hang:            0

Heap Memory Log:
Corrupted Block = none

NVRAM log:

01. 10/2/2001,08:39:45: convert_post_SAC_CiscoMIB:Block 0 converted from version 4 to 5
02. 10/2/2001,08:39:54: supVersion:Nmp version 6.1(4)

Module 2 Log:
  Reset Count:   1
  Reset History: Tue Nov 11 2008, 05:55:37
               

Module 3 Log:
  Reset Count:   9
  Reset History: Sat Oct 29 2005, 05:40:01
                 Sat Oct 29 2005, 05:29:18
                 Sat Oct 29 2005, 05:23:14
                 Thu Jan 16 2003, 08:00:15
               

Module 4 Log:
  Reset Count:   6
  Reset History: Sat Oct 29 2005, 05:40:03
                 Sat Oct 29 2005, 05:29:20
                 Sat Oct 29 2005, 05:23:16
                 Sat Sep 24 2005, 00:00:43
               

Module 5 Log:
  Reset Count:   5
  Reset History: Sat Oct 29 2005, 05:40:01
                 Sat Oct 29 2005, 05:29:18
                 Sat Oct 29 2005, 05:23:14
                 Fri Oct 17 2003, 15:03:29
               

Module 6 Log:
  Reset Count:   5
  Reset History: Sat Oct 29 2005, 05:40:03
                 Sat Oct 29 2005, 05:29:20
                 Sat Oct 29 2005, 05:23:16
                 Sun Jul 10 2005, 20:51:39
               

Module 7 Log:
  Reset Count:   4
  Reset History: Sat Oct 29 2005, 05:40:02
                 Sat Oct 29 2005, 05:29:19
                 Sat Oct 29 2005, 05:23:14
                 Thu Jul 1 2004, 22:31:22
               

Module 8 Log:
  Reset Count:   4
  Reset History: Sat Oct 29 2005, 05:40:02
                 Sat Oct 29 2005, 05:29:19
                 Sat Oct 29 2005, 05:23:14
                 Thu Apr 14 2005, 22:11:32
               

Module 9 Log:
  Reset Count:   15
  Reset History: Sat Oct 29 2005, 05:40:00
                 Sat Oct 29 2005, 05:29:17
                 Sat Oct 29 2005, 05:23:12
                 Mon Dec 10 2001, 12:28:34
               

Module 15 Log:
  Reset Count:   28
  Reset History: Fri Feb 5 2010, 15:03:52
                 Fri Feb 5 2010, 14:31:43
                 Fri Feb 5 2010, 08:15:01
                 Fri Feb 5 2010, 07:35:26
                
cat6500A>

[Reza Sharifi]

Hello Alan,

How many BGP routes do you have on the MSFC?

What IOS ver are you using?

Reza

[acameron12]

Hello,

   This is the IOS version on the MSFC2

Cisco Internetwork Operating System Software
IOS (tm) MSFC2 Software (C6MSFC2-IS-M), Version 12.1(2)E, EARLY DEPLOYMENT RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2000 by cisco Systems, Inc.
Compiled Mon 19-Jun-00 22:17 by linda
Image text-base: 0x30008930, data-base: 0x30D14000

ROM: System Bootstrap, Version 12.1(3r)E2, RELEASE SOFTWARE (fc1)
BOOTFLASH: MSFC2 Software (C6MSFC2-BOOT-M), Version 12.1(2)E, EARLY DEPLOYMENT RELEASE SOFTWARE (fc1)

cat6509A uptime is 2 minutes
System returned to ROM by error - a Software forced crash, PC 0x3021945C at 05:31:20 EST Fri Feb 5 2010
Running default software

cisco Cat6k-MSFC2 (R7000) processor with 114688K/16384K bytes of memory.
Processor board ID SAD05030F6P
R7000 CPU at 300Mhz, Implementation 39, Rev 2.1, 256KB L2, 1024KB L3 Cache
Last reset from power-on
Bridging software.
X.25 software, Version 3.0.0.
7 Virtual Ethernet/IEEE 802.3  interface(s)
509K bytes of non-volatile configuration memory.

16384K bytes of Flash internal SIMM (Sector size 512K).
Configuration register is 0x102

There are maybe 12 entries in my routing table.

[Reza Sharifi]

Hello Alan,

Looking at the "sh ver" command you provided, your switch is experiencing software crashes

Identify a Software-forced Crash

A software-forced crash occurs when the router detects a severe, unrecoverable error, and reloads itself so that it does not transmit corrupted data. A vast majority of software-forced crashes are caused by Cisco IOS® software bugs, although some platforms (such as the old Cisco 4000) can report a hardware problem as a software-forced crash.

You should open a TAC case and send them crash file to be analyzed.

http://www.cisco.com/en/US/customer/products/hw/routers/ps359/products_tech_note09186a008010260c.shtml

HTH

Reza

[acameron12]

Thanks for your prompt and helpful insights.

I will pursue this path of resolution.

[sachinraja]

Hello Alan

as told in my previous post, you might want to look at the bug that I sent to you.. Have a look at that and see if it matches your issue.. you might then need to upgrade the OS to 8.4(4) to solve this..

Raj

[Reza Sharifi]

Hi Raj,

He is running native IOS.

Cheers,

Reza