Cisco C6905-E Dual Sup Engine Communication (SYNC) Issue

[PRASOBH SUNDARESAN]

Sup Engine is not able to sync each other and standby Sup restart when the active Sup is detach

Please let me know the existing IOS is fine and provide the procedure to fix this issue.

Switch Model : Cisco C6905-E

Sup Engine : VS-SUP-2T-10G

                      VS-SUP-2T-10G

=====================================ACTIVE SUP============================================================

CORE_SWITCH#sh redundancy
Redundant System Information :
------------------------------
Available system uptime = 1 minute
Switchovers system experienced = 0
Standby failures = 0
Last switchover reason = none

Hardware Mode = Simplex
Configured Redundancy Mode = sso
Operating Redundancy Mode = sso
Maintenance Mode = Disabled
Communications = Down Reason: Failure

Current Processor Information :
-------------------------------
Active Location = slot 1/5
Current Software state = ACTIVE
Uptime in current state = 1 minute
Image Version = Cisco IOS Software, s2t54 Software (s2t54-IPSERVICESK9-M), Version 15.1(1)SY5, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Thu 19-Mar-15 12:24 by prod_rel_team
BOOT = bootdisk:,1;,1;
CONFIG_FILE =
BOOTLDR =
Configuration register = 0x2102

Peer (slot: unavailable) information is not available because it is in 'DISABLED' state

----------------------------CORE_SWITCH_SUP_1# dir-------------------Same for SUP1&2---------------------------------------------------------------------
Directory of bootdisk:/

1 -rw- 33554432 Nov 3 2015 13:56:42 +00:00 sea_console.dat
2 -rw- 93668792 Nov 3 2015 14:02:26 +00:00 s2t54-ipservicesk9-mz.SPA.151-1.SY5.bin
3 -rw- 9629 Nov 15 2016 13:14:26 +00:00 startup-config.converted_vs-20161115-131407
4 -rw- 33554432 Nov 3 2015 14:12:02 +00:00 sea_log.dat

1025589248 bytes total (864321536 bytes free)
CORE_SWITCH#
=======================================================SUP2============================================

1 -rw- 33554432 Nov 3 2015 14:07:58 +00:00 sea_console.dat
2 -rw- 93668792 Nov 3 2015 14:13:38 +00:00 s2t54-ipservicesk9-mz.SPA.151-1.SY5.bin
3 -rw- 33554432 Nov 3 2015 14:11:18 +00:00 sea_log.dat

CORE_SWITCH#sh redundancy
Redundant System Information :
------------------------------
Available system uptime = 3 minutes
Switchovers system experienced = 0
Standby failures = 0
Last switchover reason = none

Hardware Mode = Simplex
Configured Redundancy Mode = sso
Operating Redundancy Mode = sso
Maintenance Mode = Disabled
Communications = Down Reason: Failure

Current Processor Information :
-------------------------------
Active Location = slot 1/6
Current Software state = ACTIVE
Uptime in current state = 3 minutes
Image Version = Cisco IOS Software, s2t54 Software (s2t54-IPSERVICESK9-M), Version 15.1(1)SY5, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Thu 19-Mar-15 12:24 by prod_rel_team
BOOT = bootdisk:,1;,1;
CONFIG_FILE =
BOOTLDR =
Configuration register = 0x2102

Peer (slot: unavailable) information is not available because it is in 'DISABLED' state

[Palani Mohan]

Hi Prasobh

Looks like the entire node went down. When it came back up, the hot/standby is down still. It is possible that the hot/standby has been down for quire sometime?

Please look at the output of:

• show mod
• show redundancy history | include my state
• remote command switch show nvlog

Another thing to consider doing is to connect to the console of the standby and see what it displays. Share this output along with the output of:

• dev
• dir bootdisk:
• boot
• boot bootdisk:s2t54-ipservicesk9-mz.SPA.151-1.SY5.bin

Share this with us and we can definitely help you, to make progress.

Kind regards ... Palani

[PRASOBH SUNDARESAN]

CORE_SWITCH#sh mod
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
1 24 CEF720 24 port 1000mb SFP WS-X6824-SFP SAL1943RNND
5 5 Supervisor Engine 2T 10GE w/ CTS (Acti VS-SUP2T-10G SAL1943RS7U
6 5 Supervisor Engine 2T 10GE w/ CTS (CSSO VS-SUP2T-10G SAL1943RSEL
9 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6848-GE-TX SAL1944S503

Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
1 e865.49f1.0514 to e865.49f1.052b 2.4 Unknown Unknown Other
5 d072.dcbb.ef52 to d072.dcbb.ef59 2.1 12.2(50r)SYS 15.1(1)SY5 Ok
6 503d.e531.8978 to 503d.e531.897f 2.1 12.2(50r)SYS 15.1(1)SY5 Ok
9 a46c.2ae7.f47c to a46c.2ae7.f4ab 1.4 Unknown Unknown Other

Mod Sub-Module Model Serial Hw Status
---- --------------------------- ------------------ ----------- ------- -------
1 Distributed Forwarding Card WS-F6K-DFC4-A SAL1943RNND 2.4 Other
5 Policy Feature Card 4 VS-F6K-PFC4 SAL1944RZBJ 3.0 Ok
5 CPU Daughterboard VS-F6K-MSFC5 SAL1943RSSA 3.1 Ok
6 Policy Feature Card 4 VS-F6K-PFC4 SAL1944RZFB 3.0 Ok
6 CPU Daughterboard VS-F6K-MSFC5 SAL1943RSGV 3.1 Ok
9 Distributed Forwarding Card WS-F6K-DFC4-A SAL1944S503 1.4 Other

Mod Online Diag Status
---- -------------------
1 Unknown
5 Pass
6 Pass
9 Unknown
==========================================================================
CORE_SWITCH#show redundancy history | include my state
00:00:15 *my state = INITIALIZATION(2) peer state = DISABLED(1)
00:00:15 *my state = NEGOTIATION(3) peer state = DISABLED(1)
00:02:33 *my state = ACTIVE-FAST(9) peer state = DISABLED(1)
00:02:33 *my state = ACTIVE-DRAIN(10) peer state = DISABLED(1)
00:02:33 *my state = ACTIVE_PRECONFIG(11) peer state = DISABLED(1)
00:02:33 *my state = ACTIVE_POSTCONFIG(12) peer state = DISABLED(1)
00:02:33 *my state = ACTIVE(13) peer state = DISABLED(1)
==========================================================================
CORE_SWITCH#remote command switch show nvlog
NVRAM log:

01. 10/28/2015 07:56:58.119: pf_redun_negotiate:Initialized as ACTIVE processor
02. 11/03/2015 13:43:17.143: pf_redun_negotiate:Initialized as ACTIVE processor
03. 11/03/2015 14:06:49.087: oir_rf_reload_self:Manual Switch-over requested
04. 11/03/2015 14:10:41.163: pf_redun_announce_standby_init:Initialized as STANDBY processor
05. 11/03/2015 14:31:14.103: pf_redun_negotiate:Initialized as ACTIVE processor
06. 11/03/2015 14:34:50.207: oir_reload_chassis:Reload Chassis requested
07. 11/03/2015 14:54:41.103: pf_redun_negotiate:Initialized as ACTIVE processor
08. 11/03/2015 07:24:02.247: oir_reload_chassis:Reload Chassis requested
09. 11/04/2015 01:47:37.103: pf_redun_negotiate:Initialized as ACTIVE processor
10. 11/03/2015 17:59:33.111: oir_reload_chassis:Reload Chassis requested
11. 11/04/2015 02:21:56.103: pf_redun_negotiate:Initialized as ACTIVE processor
12. 11/03/2015 18:41:53.063: oir_reload_chassis:Reload Chassis requested
13. 11/04/2015 02:43:10.199: pf_redun_negotiate:Initialized as ACTIVE processor
14. 11/04/2015 02:58:40.099: pf_redun_negotiate:Initialized as ACTIVE processor
15. 11/04/2015 03:19:11.103: pf_redun_negotiate:Initialized as ACTIVE processor
16. 11/04/2015 03:36:03.091: oir_reload_chassis:Reload Chassis requested
17. 11/14/2016 05:37:49.139: pf_redun_negotiate:Initialized as ACTIVE processor
18. 11/15/2016 06:31:57.139: pf_redun_negotiate:Initialized as ACTIVE processor
19. 11/15/2016 13:14:27.238: oir_reload_chassis:Reload Chassis requested
20. 11/15/2016 13:17:28.215: pf_redun_negotiate:Initialized as ACTIVE processor
21. 11/16/2016 06:37:17.051: oir_reload_chassis:Reload Chassis requested
22. 11/16/2016 06:37:17.175: oir_reload_chassis:Reload Chassis requested
23. 11/16/2016 06:37:19.187: oir_reload_chassis:Reload Chassis requested
24. 11/16/2016 06:39:10.187: pf_redun_negotiate:Initialized as ACTIVE processor
25. 11/16/2016 06:54:20.207: pf_redun_announce_standby_init:Initialized as STANDBY processor
26. 11/16/2016 06:54:22.003: pf_redun_announce_standby_init:Initialized as STANDBY processor
27. 11/16/2016 06:54:25.079: pf_redun_announce_standby_init:Initialized as STANDBY processor
28. 11/16/2016 07:00:59.095: oir_reload_chassis:Reload Chassis requested
29. 11/16/2016 07:04:22.023: pf_redun_negotiate:Initialized as ACTIVE processor
30. 11/16/2016 07:15:12.095: pf_redun_negotiate:Initialized as ACTIVE processor
31. 11/16/2016 07:23:12.207: pf_redun_announce_standby_init:Initialized as STANDBY processor
32. 11/16/2016 07:23:14.003: pf_redun_announce_standby_init:Initialized as STANDBY processor
33. 11/16/2016 07:23:54.087: pf_redun_announce_standby_init:Initialized as STANDBY processor
34. 11/16/2016 07:26:04.207: pf_redun_announce_standby_init:Initialized as STANDBY processor
35. 11/16/2016 07:26:06.003: pf_redun_announce_standby_init:Initialized as STANDBY processor
36. 11/16/2016 07:26:09.191: pf_redun_announce_standby_init:Initialized as STANDBY processor
37. 11/16/2016 07:50:01.135: oir_reload_chassis:Reload Chassis requested
38. 11/16/2016 07:54:23.023: pf_redun_negotiate:Initialized as ACTIVE processor
39. 11/16/2016 08:57:14.207: pf_redun_announce_standby_init:Initialized as STANDBY processor
40. 11/16/2016 08:59:39.087: pf_redun_announce_standby_init:Initialized as STANDBY processor
41. 11/16/2016 08:59:44.239: pf_redun_announce_standby_init:Initialized as STANDBY processor
42. 11/17/2016 07:30:06.103: pf_redun_negotiate:Initialized as ACTIVE processor
43. 11/17/2016 07:35:39.103: pf_redun_negotiate:Initialized as ACTIVE processor
44. 11/17/2016 08:51:35.107: pf_redun_negotiate:Initialized as ACTIVE processor
45. 11/17/2016 09:07:22.207: pf_redun_announce_standby_init:Initialized as STANDBY processor
46. 11/17/2016 09:07:24.007: pf_redun_announce_standby_init:Initialized as STANDBY processor
47. 11/17/2016 09:07:52.179: pf_redun_announce_standby_init:Initialized as STANDBY processor
48. 11/17/2016 09:10:04.207: pf_redun_announce_standby_init:Initialized as STANDBY processor
49. 11/17/2016 09:10:06.003: pf_redun_announce_standby_init:Initialized as STANDBY processor
50. 11/17/2016 09:10:09.191: pf_redun_announce_standby_init:Initialized as STANDBY processor
51. 11/17/2016 09:29:47.207: pf_redun_negotiate:Initialized as ACTIVE processor
52. 11/17/2016 09:46:50.071: oir_reload_chassis:Reload Chassis requested
53. 11/17/2016 09:51:13.203: pf_redun_announce_standby_init:Initialized as STANDBY processor
54. 11/17/2016 09:53:38.091: pf_redun_announce_standby_init:Initialized as STANDBY processor
55. 11/17/2016 09:53:42.171: pf_redun_announce_standby_init:Initialized as STANDBY processor
56. 11/17/2016 10:08:48.211: pf_redun_negotiate:Initialized as ACTIVE processor
57. 11/17/2016 10:30:07.247: oir_reload_chassis:Reload Chassis requested
58. 11/17/2016 10:36:13.207: pf_redun_announce_standby_init:Initialized as STANDBY processor
59. 11/17/2016 10:38:32.003: pf_redun_announce_standby_init:Initialized as STANDBY processor
60. 11/17/2016 10:38:36.091: pf_redun_announce_standby_init:Initialized as STANDBY processor
61. 11/17/2016 10:47:08.047: pf_redun_negotiate:Initialized as ACTIVE processor
62. 11/17/2016 11:54:47.055: oir_reload_chassis:Reload Chassis requested
63. 11/17/2016 11:54:47.155: oir_reload_chassis:Reload Chassis requested
64. 11/17/2016 11:54:50.079: oir_reload_chassis:Reload Chassis requested
65. 11/17/2016 11:58:15.103: pf_redun_negotiate:Initialized as ACTIVE processor
66. 11/17/2016 12:00:40.087: oir_reload_chassis:Reload Chassis requested
67. 11/17/2016 12:02:20.211: pf_redun_announce_standby_init:Initialized as STANDBY processor
68. 11/17/2016 12:04:50.107: pf_redun_announce_standby_init:Initialized as STANDBY processor
69. 11/17/2016 12:04:55.239: pf_redun_announce_standby_init:Initialized as STANDBY processor
70. 11/19/2016 07:52:17.103: pf_redun_negotiate:Initialized as ACTIVE processor
71. 11/19/2016 08:02:19.103: pf_redun_negotiate:Initialized as ACTIVE processor
==========================================================================
CORE_SWITCH#sh run
Building configuration...

Current configuration : 22527 bytes
!
! Last configuration change at 08:07:34 UTC Sat Nov 19 2016
!
version 15.1
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
service counters max age 5
!
hostname CORE_SWITCH
!
boot-start-marker
boot system flash
boot system flash bootdisk:s2t54-ipservicesk9-mz.SPA.151-1.SY5.bin
boot-end-marker
!
!
no logging on
!
no aaa new-model
platform ip cef load-sharing ip-only
!
diagnostic bootup level minimal
!
redundancy
main-cpu
auto-sync running-config
mode sso

[Palani Mohan]

Hi Prasobh

Putting pieces together, taken days apart is kinda hard. Also, the CLI info is hard to view in the forum post. Please consider capturing the data as a txt file and upload to the thread.

The show mod out seem to indicate both Sup/2Ts are in OK state. What is the status? Are both Sup-2Ts operational?

Sincerely .... Palani

[PRASOBH SUNDARESAN]

Both are operational as standalone...

Regards,

Prasobh Sundaresan

[Palani Mohan]

Prasobh

In the show mod you shared, both Sups are located in the same chassis and the status is ok. Please help me understand what you mean by "Both are operational as standalone".

Please write/explain what you expect and what is not working.

Also, upload the show tech, please.

Thanks .... Palani

[PRASOBH SUNDARESAN]

======================CORE=====================
CORE_SWITCH#dir /all
Directory of bootdisk:/

1 -rw- 33554432 Nov 3 2015 13:56:42 +00:00 sea_console.dat
2 -rw- 93668792 Nov 3 2015 14:02:26 +00:00 s2t54-ipservicesk9-mz.SPA.151-1.SY5.bin
3 -rw- 9629 Nov 15 2016 13:14:26 +00:00 startup-config.converted_vs-20161115-131407
4 -rw- 33554432 Nov 3 2015 14:12:02 +00:00 sea_log.dat

1025589248 bytes total (864256000 bytes free)

----
rommon 2 > dev
Devices in device table:
id name
bootdisk: PCMCIA boot disk
disk0: PCMCIA disk 0
tftp: TFTP boot
usbflash0: USB boot

rommon 3 > dir bootdisk:

bootdisk:%s72044-atafslib-m: Digitally Signed Release Software with key version A

Initializing ATA monitor library...
Directory of bootdisk:

2050 93668792 -rw- s2t54-ipservicesk9-mz.SPA.151-1.SY5.bin
2 33554432 -rw- sea_console.dat
7775 9629 -rw- startup-config.converted_vs-20161115-131407 This file is not in my SUP2 standby
10410 33554432 -rw- sea_log.dat

============================STANDBY===============================
rommon 6 > dev
Devices in device table:
id name
bootdisk: PCMCIA boot disk
disk0: PCMCIA disk 0
tftp: TFTP boot
usbflash0: USB boot
rommon 7 > dir bootdisk:

monitor: command "dir" not found

rommon 8 > dir bootdisk:

bootdisk:%s72044-atafslib-m: Digitally Signed Release Software with key version A

Initializing ATA monitor library...
Directory of bootdisk:

2050 93668792 -rw- s2t54-ipservicesk9-mz.SPA.151-1.SY5.bin
2 33554432 -rw- sea_console.dat
4249 33554432 -rw- sea_log.dat

============================ERROR ON BOOTING : SUP1>=========

%PFREDUN-4-BOOTSTRING_INVALID: The bootfile is present on the chassis active supervisor but not on the chassis standby
CORE_SWITCH>
%CONST_DIAG-6-DIAG_PORT_SKIPPED: Module 5 port 4 is skipped in TestL2CTSLoopback due to: the port is used as a VSL link.
CORE_SWITCH>
CORE_SWITCH>
CORE_SWITCH>
%CONST_DIAG-6-DIAG_PORT_SKIPPED: Module 5 port 4 is skipped in TestL3CTSLoopback due to: the port is used as a VSL link.
CORE_SWITCH>
CORE_SWITCH>
CORE_SWITCH>
CORE_SWITCH>
%RF-5-RF_TERMINAL_STATE: Terminal state reached for (SSO)
%CONST_DIAG-6-DIAG_PORT_SKIPPED: Module 5 port 4 is skipped in TestChannel due to: the port is used as a VSL link.
%DIAG-6-DIAG_OK: Switch 1 Module 5: Passed Online Diagnostics
%FABRIC-5-FABRIC_MODULE_BACKUP: The Switch Fabric Module in slot 6 became standby
%C6KENV-4-LOWER_SLOT_EMPTY: The lower adjacent slot of module 5 might be empty. Airdam must be installed in that slot to be NEBS compliant
%OIR-6-SP_INSCARD: Card inserted in Switch_number = 1, physical slot 5, interfaces are now online
%SYS-SW1-6_STBY-5-RESTART: System restarted --
Cisco IOS Software, s2t54 Software (s2t54-IPSERVICESK9-M), Version 15.1(1)SY5, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Thu 19-Mar-15 12:24 by prod_rel_team
%SYS-SW1-6_STBY-6-BOOTTIME: Time taken to reboot after reload = 528 seconds
%C6KENV-4-HIGHER_SLOT_EMPTY: The higher adjacent slot of module 6 might be empty. Airdam must be installed in that slot to be NEBS compliant
%OIR-6-SP_INSCARD: Card inserted in Switch_number = 1, physical slot 6, interfaces are now online
==========================ERROR ON STANDBY SUP2 While Detach the SUP1=====
*Nov 19 09:02:44.751: %PFREDUN-6-STANDBY: Initializing as STANDBY processor for this switch
*Nov 19 09:04:37.215: %SYS-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

*Nov 19 09:02:45.047: %SYS-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.

*Nov 19 09:04:37.215: %PFREDUN-6-STANDBY: Active not responding, will be takendown in 120 sec
*Nov 19 09:04:52.575: %SYS-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

*Nov 19 09:04:37.571: %SYS-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.

*Nov 19 09:04:52.575: %PFREDUN-6-STANDBY: Active not responding, will be takendown in 105 sec
*Nov 19 09:05:07.967: %SYS-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

*Nov 19 09:04:52.931: %SYS-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.

*Nov 19 09:05:07.967: %PFREDUN-6-STANDBY: Active not responding, will be takendown in 90 sec
*Nov 19 09:05:23.323: %SYS-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

*Nov 19 09:05:08.319: %SYS-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.

*Nov 19 09:05:23.323: %PFREDUN-6-STANDBY: Active not responding, will be takendown in 75 sec
*Nov 19 09:05:38.679: %SYS-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

*Nov 19 09:05:23.675: %SYS-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.

*Nov 19 09:05:38.679: %PFREDUN-6-STANDBY: Active not responding, will be takendown in 60 sec
*Nov 19 09:05:54.035: %SYS-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

*Nov 19 09:05:39.031: %SYS-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.

*Nov 19 09:05:54.035: %PFREDUN-6-STANDBY: Active not responding, will be takendown in 45 sec
*Nov 19 09:06:09.467: %SYS-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

*Nov 19 09:05:54.387: %SYS-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.

*Nov 19 09:06:09.467: %PFREDUN-6-STANDBY: Active not responding, will be takendown in 30 sec
*Nov 19 09:06:24.823: %SYS-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

*Nov 19 09:06:09.819: %SYS-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.

*Nov 19 09:06:24.823: %PFREDUN-6-STANDBY: Active not responding, will be takendown in 15 sec
*Nov 19 09:06:40.179: %SYS-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

*Nov 19 09:06:25.175: %SYS-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.

*Nov 19 09:06:40.179: %PFREDUN-6-STANDBY: Failure of ACTIVE processor detected, resetting
*Nov 19 09:06:40.527: %SYS-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.

*Nov 19 09:06:42.631: %SYS-5-SWITCHOVER: Switchover requested by PF Redun Status. Reload Reason: Reason unspecified.
Resetting .......