7606 S Router software restart once a day

samsain2014 · ‎05-23-2014

Please help to find the root cause of this issue

sh fabric channel-counters

slot channel rxErrors txErrors txDrops lbusDrops

1 0 0 0 0 0

1 1 0 0 0 0

2 0 0 0 0 0

2 1 0 0 0 0

3 0 0 0 0 0

3 1 0 0 0 0

5 0 0 0 0 0

6 0 0 0 0 0

01. 05/19/2014 07:14:10: pf_redun_announce_standby_init:Initialized as STANDBY processor

02. 05/21/2014 11:01:59: pf_redun_negotiate:Initialized as ACTIVE processor
03. 05/21/2014 14:02:57: fabric_error_handle:Error (2): fabric in slot 6, FPOE 4 connected to slot 6
04. 05/21/2014 14:02:57: nvlog_crash_hook:System crashed on slot 6. Check value of Rommon variable CRASHINFO
05. 05/21/2014 11:04:30: pf_redun_announce_standby_init:Initialized as STANDBY processor
06. 05/21/2014 11:06:30: pf_redun_announce_standby_init:Initialized as STANDBY processor
07. 05/21/2014 11:08:40: pf_redun_status_process:Failure of ACTIVE processor detected, resetting STANDBY
08. 05/21/2014 11:10:25: pf_redun_negotiate:Initialized as ACTIVE processor
09. 05/21/2014 11:41:19: pf_redun_announce_standby_init:Initialized as STANDBY processor
10. 05/21/2014 15:04:50: pf_redun_track_switchover:STANDBY Supervisor switching-over to ACTIVE
11. 05/21/2014 15:04:50: oir_reload_peer:Card in slot 5, is being power-cycled (5)
12. 05/21/2014 12:48:32: pf_redun_announce_standby_init:Initialized as STANDBY processor
13. 05/23/2014 07:13:29: pf_redun_negotiate:Initialized as ACTIVE processor
14. 05/23/2014 10:14:28: fabric_error_handle:Error (2): fabric in slot 6, FPOE 4 connected to slot 6
15. 05/23/2014 10:14:29: hb_sp_reset_by_rp:SP is being reset by the RP via hardware heartbeat
16. 05/23/2014 10:14:54: nvlog_crash_hook:System crashed on slot 6. Check value of Rommon variable CRASHINFO
17. 05/23/2014 07:16:27: pf_redun_announce_standby_init:Initialized as STANDBY processor
18. 05/23/2014 07:18:13: pf_redun_negotiate:Initialized as ACTIVE processor
19. 05/23/2014 10:35:30: fabric_error_handle:Error (2): fabric in slot 6, FPOE 10 connected to slot 2
20. 05/23/2014 10:35:36: fabric_error_handle:Error (2): fabric in slot 6, FPOE 2 connected to slot 3
21. 05/23/2014 10:35:42: fabric_error_handle:Error (2): fabric in slot 6, FPOE 0 connected to slot 6
22. 05/23/2014 10:35:47: fabric_error_handle:Error (2): fabric in slot 6, FPOE 4 connected to slot 6
23. 05/23/2014 10:35:47: nvlog_crash_hook:System crashed on slot 6. Check value of Rommon variable CRASHINFO
24. 05/23/2014 07:37:28: pf_redun_negotiate:Initialized as ACTIVE processor
25. 05/23/2014 10:38:27: fabric_error_handle:Error (2): fabric in slot 6, FPOE 4 connected to slot 6
26. 05/23/2014 10:38:28: nvlog_crash_hook:System crashed on slot 6. Check value of Rommon variable CRASHINFO
27. 05/23/2014 07:39:58: pf_redun_announce_standby_init:Initialized as STANDBY processor
28. 05/23/2014 07:41:58: pf_redun_announce_standby_init:Initialized as STANDBY processor
29. 05/23/2014 07:44:11: pf_redun_status_process:Failure of ACTIVE processor detected, resetting STANDBY
30. 05/23/2014 07:45:52: pf_redun_negotiate:Initialized as ACTIVE processor

--------------------------------------------------------------------------------------------------------------------------

od Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
1 20 7600 ES+T 76-ES+T-20G
2 4 7600 ES+T 76-ES+T-4TG
3 4 7600 ES+T 76-ES+T-4TG
5 2 Route Switch Processor 720 (Hot) RSP720-3C-GE
6 2 Route Switch Processor 720 (Active) RSP720-3C-GE

Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
1 503d.e5f6.07c0 to 503d.e5f6.07ff 1.4 12.2(33r)SRD 12.2(33)SRE5 Ok
2 44d3.ca51.0510 to 44d3.ca51.051f 1.5 12.2(33r)SRD 12.2(33)SRE5 Ok
3 44d3.ca51.0520 to 44d3.ca51.052f 1.5 12.2(33r)SRD 12.2(33)SRE5 Ok
5 001c.584c.87d4 to 001c.584c.87d7 5.13 12.2(33r)SRD 12.2(33)SRE5 Ok
6 c89c.1dfb.a350 to c89c.1dfb.a353 5.12 12.2(33r)SRD 12.2(33)SRE5 Ok

Mod Sub-Module Model Serial Hw Status
---- --------------------------- ------------------ ----------- ------- -------
1 7600 ES+ DFC XL 7600-ES+3CXL 1.2 Ok
1 7600 ES+T 20x1GE SFP 76-ES+T-20GQ 1.1 Ok
2 7600 ES+ DFC XL 7600-ES+3CXL 1.2 Ok
2 7600 ES+T 4x10GE XFP ITU 76-ES+T-4TGQ 1.2 Ok
3 7600 ES+ DFC XL 7600-ES+3CXL 1.2 Ok
3 7600 ES+T 4x10GE XFP ITU 76-ES+T-4TGQ 1.2 Ok
5 Policy Feature Card 3 7600-PFC3C 1.2 Ok
5 C7600 MSFC4 Daughterboard 7600-MSFC4 1.5 Ok
6 Policy Feature Card 3 7600-PFC3C 1.2 Ok
6 C7600 MSFC4 Daughterboard 7600-MSFC4 1.6 Ok

Mod Online Diag Status
---- -------------------
1 Pass
2 Pass
3 Pass
5 Pass
6 Pass

--------------------------------------------------------------------------------------------------------------------------

isco IOS Software, c7600rsp72043_rp Software (c7600rsp72043_rp-ADVIPSERVICESK9-M), Version 12.2(33)SRE5, RELEASE SOF
TWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2011 by Cisco Systems, Inc.
Compiled Thu 15-Sep-11 03:20 by prod_rel_team

ROM: System Bootstrap, Version 12.2(33r)SRD5, RELEASE SOFTWARE (fc1)
BOOTLDR: Cisco IOS Software, c7600rsp72043_rp Software (c7600rsp72043_rp-ADVIPSERVICESK9-M), Version 12.2(33)SRE5, RE
LEASE SOFTWARE (fc1)

uptime is 46 minutes
Uptime for this control processor is 46 minutes
System returned to ROM by s/w reset at 13:37:56 GMT Fri May 23 2014 (SP by error - a Software forced crash, PC 0x835A
F10)
System restarted at 13:39:58 GMT Fri May 23 2014
System image file is "bootdisk:c7600rsp72043-advipservicesk9-mz.122-33.SRE5.bin"
Last reload type: Normal Reload

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Cisco CISCO7606-S (M8500) processor (revision 1.1) with 1900544K/131072K bytes of memory.
Processor board ID FOX1522GQAB
BASEBOARD: RSP720
CPU: MPC8548_E, Version: 2.1, (0x80390021)
CORE: E500, Version: 2.2, (0x80210022)
CPU:1200MHz, CCB:400MHz, DDR:200MHz,
L1: D-cache 32 kB enabled
I-cache 32 kB enabled

Last reset from s/w reset
8 Virtual Ethernet interfaces
34 Gigabit Ethernet interfaces
8 Ten Gigabit Ethernet interfaces
3964K bytes of non-volatile configuration memory.

500472K bytes of Internal ATA PCMCIA card (Sector size 512 bytes).
Configuration register is 0x2102

--------------------------------------------------------------------------------------------------------------------------

*May 23 10:27:14.852 UTC: %OIR-SP-6-INSPS: Power supply inserted in slot 1

*May 23 10:27:14.852 UTC: %C7600_PWR-SP-4-PSOK: power supply 1 turned on.
*May 23 10:27:14.924 UTC: %OIR-SP-6-INSPS: Power supply inserted in slot 2
*May 23 10:27:14.928 UTC: %C7600_PWR-SP-4-PSOK: power supply 2 turned on.
*May 23 10:27:14.936 UTC: %C7600_PWR-SP-4-PSREDUNDANTBOTHSUPPLY: in power-redundancy mode, system is operating on both power supplies.
May 23 13:27:22.803 GMT: %FABRIC-SP-5-CLEAR_BLOCK: Clear block option is off for the fabric in slot 6.
May 23 13:27:22.887 GMT: %FABRIC-SP-5-FABRIC_MODULE_ACTIVE: The Switch Fabric Module in slot 6 became active.
May 23 13:27:24.344 GMT: %DIAG-SP-6-RUN_MINIMUM: Module 6: Running Minimal Diagnostics...
May 23 13:27:30.965 GMT: %DIAG-SP-6-DIAG_OK: Module 6: Passed Online Diagnostics
May 23 13:27:31.385 GMT: %OIR-SP-6-INSCARD: Card inserted in slot 6, interfaces are now online
May 23 13:27:32.782 GMT: %PFREDUN-SP-4-PHYSMEM_MISMATCH: Asymmetrical redundant configuration: Active SP has (2097152/131072K) memory, Standby has (1048576/65536K).
May 23 13:27:32.782 GMT: %PFREDUN-SP-4-PHYSMEM_MISMATCH: Asymmetrical redundant configuration: Active RP has (2097152/131072K) memory, Standby has (1048576/65536K).
May 23 13:27:33.786 GMT: %PFREDUN-SP-6-ACTIVE: Standby initializing for SSO mode
Buffered messages:

*May 23 10:26:25.383: %SYS-SP-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.

Firmware compiled 30-Jun-11 13:52 by integ Build [100]

*May 23 10:27:02.303: %SPANTREE-SP-5-EXTENDED_SYSID: Extended SysId enabled for type vlan
*May 23 10:27:02.503: SP: SP: Currently running ROMMON from S (Gold) region
*May 23 10:27:10.955: %SW_VLAN-SP-6-VTP_DOMAIN_NAME_CHG: VTP domain name changed to QPMPLS.
*May 23 10:27:12.624: %SYS-SP-5-RESTART: System restarted --
Cisco IOS Software, c7600rsp72043_sp Software (c7600rsp72043_sp-ADVIPSERVICESK9-M), Version 12.2(33)SRE5, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2011 by Cisco Systems, Inc.
Compiled Thu 15-Sep-11 03:27 by prod_rel_team
*May 23 10:27:14.852 UTC: %OIR-SP-6-INSPS: Power supply inserted in slot 1
*May 23 10:27:14.852 UTC: %C7600_PWR-SP-4-PSOK: power supply 1 turned on.
*May 23 10:27:14.924 UTC: %OIR-SP-6-INSPS: Power supply inserted in slot 2
*May 23 10:27:14.928 UTC: %C7600_PWR-SP-4-PSOK: power supply 2 turned on.
*May 23 10:27:14.936 UTC: %C7600_PWR-SP-4-PSREDUNDANTBOTHSUPPLY: in power-redundancy mode, system is operating on both power supplies.
May 23 13:27:22.803 GMT: %FABRIC-SP-5-CLEAR_BLOCK: Clear block option is off for the fabric in slot 6.
May 23 13:27:22.887 GMT: %FABRIC-SP-5-FABRIC_MODULE_ACTIVE: The Switch Fabric Module in slot 6 became active.
May 23 13:27:24.344 GMT: %DIAG-SP-6-RUN_MINIMUM: Module 6: Running Minimal Diagnostics...
May 23 13:27:30.965 GMT: %DIAG-SP-6-DIAG_OK: Module 6: Passed Online Diagnostics
May 23 13:27:31.385 GMT: %OIR-SP-6-INSCARD: Card inserted in slot 6, interfaces are now online
May 23 13:27:32.782 GMT: %PFREDUN-SP-4-PHYSMEM_MISMATCH: Asymmetrical redundant configuration: Active SP has (2097152/131072K) memory, Standby has (1048576/65536K).
May 23 13:27:32.782 GMT: %PFREDUN-SP-4-PHYSMEM_MISMATCH: Asymmetrical redundant configuration: Active RP has (2097152/131072K) memory, Standby has (1048576/65536K).
Queued messages:
May 23 13:27:36.096 GMT: %SYS-SP-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

May 23 13:27:33.786 GMT: %PFREDUN-SP-6-ACTIVE: Standby initializing for SSO mode
May 23 13:27:36.312 GMT: %SYS-SP-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.

May 23 13:27:46.792 GMT: %CHKPT-SP-4-GET_HUGE_BUF: Client 163 buffer requested (size = 6160) is too large
May 23 13:28:03.552 GMT: %FABRIC-SP-5-CLEAR_BLOCK: Clear block option is off for the fabric in slot 5.
May 23 13:28:03.632 GMT: %FABRIC-SP-5-FABRIC_MODULE_BACKUP: The Switch Fabric Module in slot 5 became standby
May 23 13:28:04.424 GMT: %DIAG-SP-6-RUN_MINIMUM: Module 5: Running Minimal Diagnostics...
May 23 13:28:04.925 GMT: %DIAG-SP-6-DIAG_OK: Module 5: Passed Online Diagnostics
May 23 13:28:05.428 GMT: %ICC-SP-5-HUGE_BUFFER: Class [L2-DRV(FC)] with Request id 38 requested a huge buffer of Size 47280.
May 23 13:28:05.672 GMT: %OIR-SP-6-INSCARD: Card inserted in slot 5, interfaces are now online
May 23 13:28:06.808 GMT: %LINK-SP-3-UPDOWN: Interface GigabitEthernet5/1, changed state to up
May 23 13:28:09.796 GMT: %LINEPROTO-SP-5-UPDOWN: Line protocol on Interface GigabitEthernet5/1, changed state to up
May 23 13:28:12.005 GMT: %DIAG-SP-6-RUN_MINIMUM: Module 1: Running Minimal Diagnostics...
May

Leo Laohoo · ‎05-23-2014

a Software forced crash, PC 0x835AF10)

A bug in the IOS caused the switch processor to crash.

You may need to upgrade to another IOS or get TAC to translate the debug for your benefit.

samsain2014 · ‎05-27-2014

Thanks Leo for replay

Following RP crash info taken from bootflash, it is very helpful if we can find the related bug details.

000008: May 23 13:29:10.385 GMT: %SYS-STDBY-5-RESTART: System restarted --
Cisco IOS Software, c7600rsp72043_rp Software (c7600rsp72043_rp-ADVIPSERVICESK9-M), Version 12.2(33)SRE5, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2011 by Cisco Systems, Inc.
Compiled Thu 15-Sep-11 03:20 by prod_rel_team
000009: May 23 13:29:10.413 GMT: %SSH-STDBY-5-ENABLED: SSH 2.0 has been enabled
000010: .May 23 13:29:32.465 GMT: %SSH-STDBY-5-DISABLED: SSH 2.0 has been disabled
000011: .May 23 13:29:32.465 GMT: %SSH-STDBY-5-ENABLED: SSH 2.0 has been enabled
000012: May 23 13:37:57.750 GMT: %PFREDUN-SP-STDBY-6-ACTIVE: Initializing as ACTIVE processor
000013: May 23 13:37:57.750 GMT: %PFREDUN-SP-STDBY-6-ACTIVE: Initializing as ACTIVE processor
000014: .May 23 13:37:58.710 GMT: %CPU_MONITOR-3-PEER_EXCEPTION: CPU_MONITOR peer has failed due to exception , reset by [5/0]

%Software-forced reload

13:37:58 GMT Fri May 23 2014: Unexpected exception to CPU: vector 1500, PC = 0xAFB2914 , LR = 0xAFB28D8

-Traceback= 0xAFB2914 0xAFB28D8 0xAC75778 0xAB31B64 0xAB7D128 0xAB7D18C 0x8BE8CA4 0x8BE9E60 0x8BEA8F0 0x8BEAE80 0xAA18FC0 0xAA19760 0xB00ACFC 0xB00B03C 0xB00CA7C 0xAFA466C