Urgent Help Required on 7609 SUP engine abrupt reboot

jatinder sharma · ‎09-28-2015

Hello All,

we have one Cisco router 7609 and yesterday early morning it's SUP engine's got rebooted ,logs are below.

actually there was no power issue and only SUP cards were rebooted.

Sep 27 00:35:03: %PFREDUN-SP-STDBY-6-ACTIVE: Initializing as ACTIVE processor
Sep 27 00:35:03: %PFREDUN-SP-STDBY-6-ACTIVE: Initializing as ACTIVE processor
Sep 27 00:35:03: SP-STDBY: oir_scp_swover_msg_handler: invalid slot 16
Sep 27 00:35:04: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host x.x.x.x port 514 started - CLI initiated
Sep 27 00:35:04: %OIR-SP-3-PWRCYCLE: Card in module 5, is being power-cycled (Switchover)
Sep 27 00:35:05: %ENT_ALARM-6-INFO: ASSERT CRITICAL Physical Slot 5 Active Card Removed OIR Alarm
Sep 27 00:35:05: %SNMP-5-MODULETRAP: Module 2 [Up] Trap
Sep 27 00:35:05: %SNMP-5-MODULETRAP: Module 6 [Up] Trap
Sep 27 00:35:05: %C6KPWR-SP-4-PSOK: power supply 1 turned on.
Sep 27 00:35:05: %C6KPWR-SP-4-PSOK: power supply 2 turned on.
Sep 27 00:35:05: %OIR-SP-6-INSCARD: Card inserted in slot 6, interfaces are now online
Sep 27 00:35:05: %OIR-SP-6-INSCARD: Card inserted in slot 2, interfaces are now online
Sep 27 00:35:07: %SNMP-5-MODULETRAP: Module 5 [Down] Trap
Sep 27 00:35:07: %SNMP-5-MODULETRAP: Module 1 [Up] Trap
Sep 27 00:35:07: %OIR-SP-6-INSCARD: Card inserted in slot 1, interfaces are now online
Sep 27 00:35:07: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet6/1, changed state to up
Sep 27 00:35:07: %LINK-3-UPDOWN: Interface GigabitEthernet6/1, changed state to up
Sep 27 00:35:08: %PIM-5-DRCHG: DR change from neighbor 0.0.0.0 to x.x.x.x on interface FastEthernet2/0/0
Sep 27 00:35:08: %PIM-5-DRCHG: DR change from neighbor 0.0.0.0 to x.x.x.x on interface FastEthernet2/0/2
Sep 27 00:35:09: %LINK-5-CHANGED: Interface GigabitEthernet6/2, changed state to administratively down
Sep 27 00:35:09: %LINK-5-CHANGED: Interface GigabitEthernet6/5, changed state to administratively down
Sep 27 00:35:09: %LINK-5-CHANGED: Interface GigabitEthernet6/6, changed state to administratively down
Sep 27 00:35:09: %PIM-5-DRCHG: DR change from neighbor 0.0.0.0 to x.x.x.x on interface GigabitEthernet6/1
Sep 27 00:35:09: %PIM-5-NBRCHG: neighbor x.x.x.x UP on interface FastEthernet2/0/0
Sep 27 00:35:09: %PIM-5-DRCHG: DR change from neighbor x.x.x.x to x.x.x.x on interface FastEthernet2/0/0
Sep 27 00:35:09: %PIM-5-NBRCHG: neighbor x.x.x.x UP on interface FastEthernet2/0/2
Sep 27 00:35:09: %NTP-6-RESTART: NTP process starts
Sep 27 00:35:14: %BGP-5-ADJCHANGE: neighbor x.x.x.x Up
Sep 27 00:35:14: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host x.x.x.x port 514 started - reconnection
Sep 27 00:35:36: %HSRP-5-STATECHANGE: Vlan2 Grp 2 state Standby -> Active
Sep 27 00:39:46: %PFREDUN-SP-6-ACTIVE: Standby initializing for SSO mode
Sep 27 00:39:49: %SYS-SP-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.
Sep 27 00:40:01: %CHKPT-SP-4-GET_HUGE_BUF: Client 163 buffer requested (size = 6160) is too large
Sep 27 00:42:14: %ICC-SP-5-HUGE_BUFFER: Class [L2-DRV(FC)] with Request id 35 requested a huge buffer of Size 24574.
Sep 27 00:42:14: %DIAG-SP-6-RUN_MINIMUM: Module 5: Running Minimal Diagnostics...
Sep 27 00:42:19: %DIAG-SP-6-DIAG_OK: Module 5: Passed Online Diagnostics
Sep 27 00:42:19: %SNMP-5-MODULETRAP: Module 5 [Up] Trap
Sep 27 00:42:19: %ICC-SP-5-HUGE_BUFFER: Class [L2-DRV(FC)] with Request id 38 requested a huge buffer of Size 47280.
Sep 27 00:42:20: %LINK-3-UPDOWN: Interface GigabitEthernet5/5, changed state to down
Sep 27 00:42:21: %ENT_ALARM-6-INFO: BLANK CRITICAL Physical Slot 5 Active Card Removed OIR Alarm
Sep 27 00:42:21: %OIR-SP-6-INSCARD: Card inserted in slot 5, interfaces are now online
Sep 27 00:42:29: %SVCLC-5-FWTRUNK: Firewalled VLANs configured on trunks
Sep 27 00:43:01: %HSRP-5-STATECHANGE: Vlan2 Grp 2 state Active -> Speak
Sep 27 00:43:11: %HSRP-5-STATECHANGE: Vlan2 Grp 2 state Speak -> Standby
*Sep 26 19:09:45.923: %SYS-SP-STDBY-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.
*Sep 26 19:10:14.732: %SPANTREE-SP-STDBY-5-EXTENDED_SYSID: Extended SysId enabled for type vlan
Sep 26 19:11:44.817: %DIAG-SP-STDBY-6-RUN_MINIMUM: Module 5: Running Minimal Diagnostics...
Sep 26 19:12:13.097: %DIAG-SP-STDBY-6-DIAG_OK: Module 5: Passed Online Diagnostics
Sep 26 19:13:29.258: %SYS-SP-STDBY-5-RESTART: System restarted --
Cisco IOS Software, c7600s3223_sp Software (c7600s3223_sp-ADVENTERPRISEK9-M), Version 12.2(33)SRE12, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Wed 11-Mar-15 23:28 by prod_rel_team
Sep 26 19:13:29.298: SP-STDBY: Currently running ROMMON from S (Gold) region
Sep 27 00:43:30: %SYS-SP-STDBY-6-BOOTTIME: Time taken to reboot after reload = 497 seconds
Sep 27 00:43:45: %CHKPT-SP-4-GET_HUGE_BUF: Client 39 buffer requested (size = 4232) is too large
Sep 27 00:43:53: TSPTUN-HA: UNEXPECTED MESSAGE

please throw some lights on this.

Regards

Jatinder Sharma

Mark Malone · ‎09-28-2015

Did you get a crash file in Flash?

Vinit Jain · ‎09-28-2015

Could you please share the following logs:

- show fabric channel

- show fabric error

- show redundancy

Also, as Mark suggested, check for crashinfo file in bootflash or sup-bootflash file system. If you have two Supervisor cards, then you will have to check the same in the standby (assuming the active SUP crashed and became standby).

Regards

Vinit

Thanks
--Vinit

jatinder sharma · ‎09-30-2015

hello Vinit,

PFB required output.

#show fabric channel
^
% Invalid input detected at '^' marker.

#show fa
#show fabric error
^
% Invalid input detected at '^' marker.

#show redundancy
Redundant System Information :
------------------------------
Available system uptime = 2 days, 15 hours, 52 minutes
Switchovers system experienced = 1
Standby failures = 0
Last switchover reason = active unit failed

Hardware Mode = Duplex
Configured Redundancy Mode = sso
Operating Redundancy Mode = sso
Maintenance Mode = Disabled
Communications = Up

Current Processor Information :
-------------------------------
Active Location = slot 6
Current Software state = ACTIVE
Uptime in current state = 2 days, 5 hours, 12 minutes
Image Version = Cisco IOS Software, c7600s3223_rp Software (c7600s3223_rp-ADVENTERPRISEK9-M), Version 12.2(33)SRE12, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Wed 11-Mar-15 23:19 by prod_rel_team
BOOT = sup-bootdisk:c7600s3223-adventerprisek9-mz.122-33.SRE12.bin,1;
CONFIG_FILE =
BOOTLDR =
Configuration register = 0x2102

Peer Processor Information :
----------------------------
Standby Location = slot 5
Current Software state = STANDBY HOT
Uptime in current state = 2 days, 5 hours, 4 minutes
Image Version = Cisco IOS Software, c7600s3223_rp Software (c7600s3223_rp-ADVENTERPRISEK9-M), Version 12.2(33)SRE12, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Wed 11-Mar-15 23:19 by prod_rel_team
BOOT = sup-bootdisk:c7600s3223-adventerprisek9-mz.122-33.SRE12.bin,1;
CONFIG_FILE =
BOOTLDR =
Configuration register = 0x2102

jatinder sharma · ‎10-03-2015

Hi All,

Please help on raised output.

Vinit Jain · ‎10-03-2015

The crashinfo file that you shared is from the bootflash: and seems to be an old one, Thus not relevant.

*Nov 26 07:08:34.679: %CPU_MONITOR-STDBY-3-PEER_EXCEPTION: CPU_MONITOR peer has

%Software-forced reload


 07:08:34 UTC Wed Nov 26 2008: Breakpoint exception, CPU signal 23, PC = 0x42A41

Could you please check the crashinfo file in slavebootdisk: and slavesup-bootdisk:

Check for the latest crashinfo file generated.

Thanks

Vinit

Thanks
--Vinit