Re: 6506 Supervisor 720 Problem

jpl861 · ‎09-20-2009

Hi,

I'm trying to determine the problem of our 6500 switch. One is using Sup720 and the other one is Sup720-3bxl. They are using the same IOS image. That's not an issue right? As there is no specific image only for Sup720 and Sup720-3bxl. Correct me if I'm wrong.

Next, I don't know what's wrong with the show logging output. The switch rebooted unexpectedly and we are in doubt that this might happen again. Please see logs below and please let me know if there's something interesting. If it's a hardware or software issue.

00:01:03: curr is 0x20000

00:01:03: RP: Currently running ROMMON from F2 region

00:01:27: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan

00:01:30: %SYS-5-CONFIG_I: Configured from memory by console

00:01:32: %SYS-5-RESTART: System restarted --

Cisco Internetwork Operating System Software

IOS (tm) s72033_rp Software (s72033_rp-PSV-M), Version 12.2(18)SXD7b, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Compiled Fri 08-Dec-06 12:38 by ccai

00:01:32: %SYS-6-BOOTTIME: Time taken to reboot after reload = 1617643 seconds

00:00:12: %SYS-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.

00:00:13: %SYS-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.

Firmware compiled 24-Oct-06 16:18 by integ Build [100]

- RELIABILITY DRIVER: wrong signature on NVFLASH

00:01:23: SP: SP: Currently running ROMMON from F2 region

00:01:27: %SPANTREE-SP-5-EXTENDED_SYSID: Extended SysId enabled for type vlan

00:01:29: %C6K_PROCMIB-SP-3-IPC_PORTOPEN_FAIL: Failed to open port while connecting to process statistics: error code = no such port

-Traceback= 4087F6D8 4087F810

00:01:32: %C6K_PROCMIB-SP-3-IPC_PORTOPEN_FAIL: Failed to open port while connecting to process statistics: error code = no such port

-Traceback= 4087F6D8 4087F830

00:01:32: %SYS-SP-5-RESTART: System restarted --

Cisco Internetwork Operating System Software

IOS (tm) s72033_sp Software (s72033_sp-PSV-M), Version 12.2(18)SXD7b, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Compiled Fri 08-Dec-06 12:31 by ccai

00:01:34: %OIR-SP-6-INSPS: Power supply inserted in slot 1

00:01:34: %C6KPWR-SP-4-PSOK: power supply 1 turned on.

00:01:34: %OIR-SP-6-INSPS: Power supply inserted in slot 2

00:01:34: %MLSM-6-MMLS_MODE_CHANGE: MMLS: Replication Mode Change Detected because oir event

00:01:34: %C6KPWR-SP-4-PSOK: power supply 2 turned on.

00:01:34: %C6KPWR-SP-4-PSREDUNDANTBOTHSUPPLY: in power-redundancy mode, system is operating on both power supplies.

00:01:36: %C6KENV-SP-4-FANHIOUTPUT: Version 2 high-output fan-tray is in effect

00:01:39: %FABRIC-SP-5-CLEAR_BLOCK: Clear block option is off for the fabric in slot 5.

00:01:40: %FABRIC-SP-5-FABRIC_MODULE_ACTIVE: The Switch Fabric Module in slot 5 became active.

00:01:41: %DIAG-SP-6-RUN_MINIMUM: Module 5: Running Minimum Diagnostics...

00:01:53: %DIAG-SP-6-DIAG_OK: Module 5: Passed Online Diagnostics

00:01:54: %OIR-SP-6-INSCARD: Card inserted in slot 5, interfaces are now online

00:01:57: %DIAG-SP-6-RUN_MINIMUM: Module 3: Running Minimum Diagnostics...

00:01:58: %DIAG-SP-6-RUN_MINIMUM: Module 2: Running Minimum Diagnostics...

00:02:02: %MLSM-6-MMLS_MODE_CHANGE: MMLS: Replication Mode Change Detected because oir event

Thanks in advance.

John

Giuseppe Larosa · ‎09-20-2009

Hello John,

the lines you have posted show the reboot but not the reason for reboot.

you can use

sh ver

sh stack

and you can check with dir /all if there is a crashinfo file for the reload.

these can provide more useful information

Edit:

only interesting lines are :

00:01:29: Ã†K_PROCMIB-SP-3-IPC_PORTOPEN_FAIL: Failed to open port while connecting to process statistics: error code = no such port

but these can be during the reboot not the cause of it.

Hope to help

Giuseppe

jpl861 · ‎09-20-2009

I rebooted the switch with the reload command but the reason for the reload was different in show version and show stack.

sh stack

Minimum process stacks:

Free/Size Name

5704/6000 OIR IOS Process

5536/6000 IPC Zone Manager

5688/6000 ICC Retry Q

4544/6000 IPC delayed init

4888/6000 IPC delayed funcs

5704/6000 CDP BLOB

5648/6000 FM HA Sync

5656/6000 L3 Manager HA

5616/6000 Draco FIB process

5616/6000 QM OIR Proc

3528/6000 eobc_init_process

5232/6000 ICC Slave Comp. Up

5592/6000 PM MP Process

1936/3000 EARL INFO CAPABILITY process

5400/6000 C6K ENV RP init

5048/6000 SPAN Subsystem

11496/12000 Router Init

7920/12000 Init

8704/9000 cdp init process

5112/6000 Draco DFS Port Registation Proc

5408/6000 RADIUS INITCONFIG

Minimum process stacks:

Free/Size Name

5016/6000 LCC Configure

4032/6000 draco-oir-process:slot 5

2328/3000 L3 MGR Send ARP Req

4584/6000 SLB RF Active Proc

4240/6000 draco-oir-process:slot 3

5240/6000 CEF Reloader

4240/6000 draco-oir-process:slot 2

5600/6000 BGP Accepter

4928/6000 BGP Open

8296/12000 Exec

5560/6000 Link Status process

34936/36000 TCP Command

19616/24000 show techsupport huge_stack

8264/12000 Virtual Exec

Interrupt level stacks:

Level Called Unused/Size Name

1 2257087 7664/9000 Inband Interrupt

2 4042541 7560/9000 EOBC Interrupt

3 55932 8456/9000 Management Interrupt

4 73879 8600/9000 Console Uart

5 0 9000/9000 Mistral Error Interrupt

7 13981851 8584/9000 NMI Interrupt Handler

Spurious interrupts: 229

System was restarted by bus error at PC 0x41D8590C, address 0x0 at 12:48:04 UTC Wed Feb 2 2000

s72033_rp Software (s72033_rp-PSV-M), Version 12.2(18)SXD7b, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Compiled Fri 08-Dec-06 12:38 by ccai (current version)

Image text-base: 0x4002100C, data-base: 0x42040000

Stack trace from system failure:

FP: 0x5000BB28, RA: 0x41D8590C

FP: 0x5000BB28, RA: 0x41D83858

FP: 0x5000BB48, RA: 0x41B32454

FP: 0x5000BB60, RA: 0x41B32480

FP: 0x5000BB78, RA: 0x41A050D8

FP: 0x5000BBB0, RA: 0x41A46324

FP: 0x5000BBD0, RA: 0x41A3C6BC

FP: 0x5000BBF0, RA: 0x41A3C714

***************************************************

******* Information of Last System Crash **********

***************************************************

Using bootflash:crashinfo_20000112-140324.

%Error opening bootflash:crashinfo_20000112-140324 (File not found)

***************************************************

****** Information of Last System Crash - SP ******

***************************************************

Using sup-bootflash:crashinfo_20000217-140158.

%Error opening sup-bootflash:crashinfo_20000217-140158 (File not found)

jpl861 · ‎09-20-2009

xxxxxxxx-6500-standby#sh version

Cisco Internetwork Operating System Software

IOS (tm) s72033_rp Software (s72033_rp-PSV-M), Version 12.2(18)SXD7b, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Compiled Fri 08-Dec-06 12:38 by ccai

Image text-base: 0x4002100C, data-base: 0x42040000

ROM: System Bootstrap, Version 12.2(17r)SX5, RELEASE SOFTWARE (fc1)

BOOTLDR: s72033_rp Software (s72033_rp-PSV-M), Version 12.2(18)SXD7b, RELEASE SOFTWARE (fc1)

xxxxxxxx-6500-standby uptime is 15 hours, 33 minutes

Time since xxxxxxxx-6500-standby switched to active is 15 hours, 32 minutes

System returned to ROM by bus error at PC 0x41D8590C, address 0x0 at 12:48:04 UTC Wed Feb 2 2000 (SP by reload)

System restarted at 05:22:12 edt Sun Sep 20 2009

System image file is "sup-bootflash:s72033-psv-mz.122-18.SXD7b.bin"

cisco WS-C6506 (R7000) processor (revision 2.0) with 983008K/65536K bytes of memory.

Processor board ID TBA04171663

SR71000 CPU at 600Mhz, Implementation 0x504, Rev 1.2, 512KB L2 Cache

Last reset from s/w reset

X.25 software, Version 3.0.0.

Bridging software.

9 Virtual Ethernet/IEEE 802.3 interface(s)

26 Gigabit Ethernet/IEEE 802.3 interface(s)

1917K bytes of non-volatile configuration memory.

8192K bytes of packet buffer memory.

65536K bytes of Flash internal SIMM (Sector size 512K).

Configuration register is 0x2102

Leo Laohoo · ‎09-20-2009

What is the system clock of your switch? According to your post, the last crashinfo file was made on December 2000.

Giuseppe Larosa · ‎09-21-2009

Hello John,

there was a bus error at program counter position 0x41D8590C

System was restarted by bus error at PC 0x41D8590C, address 0x0 at 12:48:04 UTC Wed Feb 2 2000

s72033_rp Software (s72033_rp-PSV-M), Version 12.2(18)SXD7b, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Compiled Fri 08-Dec-06 12:38 by ccai (current version)

Image text-base: 0x4002100C, data-base: 0x42040000

the IOS image is of December 2006.

the crash declared to be happened in Feb 2000.

Use the following document to go on with the analysis of crash

http://www.cisco.com/en/US/partner/products/hw/switches/ps708/products_tech_note09186a008072c406.shtml

or

http://www.cisco.com/en/US/products/hw/switches/ps708/products_tech_note09186a008072c406.shtml

Hope to help

Giuseppe