ASR1002 redundancy failure, state "init, standby"

[Steffen Pontiac]

Hi Community,

that is my first post here so please be patient. ;-)

On many of our ASR1002-X and -HX router I see the following:

Router1#sh platform
Chassis type: ASR1002-X

Slot Type State Insert time (ago)
--------- ------------------- --------------------- -----------------
0 ASR1002-X ok 17:14:40
0/0 6XGE-BUILT-IN ok 17:13:31
R0 ASR1002-X ok 17:14:40
R0/0 ok, active 17:14:40
R0/1 init, standby 17:13:01
F0 ASR1002-X ok, active 17:14:40
P0 ASR1002-PWR-AC ok 17:14:20
P1 ASR1002-PWR-AC ok 17:14:20

Slot CPLD Version Firmware Version
--------- ------------------- ---------------------------------------
0 14012203 16.2(1r)
R0 14012203 16.2(1r)
F0 14012203 16.2(1r)

Router1#sh redundancy states
my state = 13 -ACTIVE
peer state = 1 -DISABLED
Mode = Duplex
Unit = Primary
Unit ID = 48

Redundancy Mode (Operational) = sso
Redundancy Mode (Configured) = sso
Redundancy State = sso
Maintenance Mode = Disabled
Manual Swact = disabled (the peer unit is still initializing)
Communications = Down Reason: Failure

client count = 119
client_notification_TMR = 30000 milliseconds
RF debug mask = 0x0

####################################################################

Router2#sh platform
Chassis type: ASR1002-HX

Slot Type State Insert time (ago)
--------- ------------------- --------------------- -----------------
0 ASR1002-HX ok 2y11w
0/0 BUILT-IN-EPA-8x1G ok 2y11w
0/1 BUILT-IN-EPA-8x10G ok 2y11w
R0 ASR1002-HX ok 2y11w
R0/0 ok, active 2y11w
R0/1 init, standby 2y11w
F0 ASR1002-HX ok, active 2y11w
P0 ASR1000X-AC-750W ok 2y11w
P1 ASR1000X-AC-750W ok 2y11w
P2 ASR1002HX-FAN ok 2y11w

Slot CPLD Version Firmware Version
--------- ------------------- ---------------------------------------
0 15100715 16.2(2r)
R0 15100715 16.2(2r)
F0 15100715 16.2(2r)

######################################################################

Router2#sh redundancy states
my state = 13 -ACTIVE
peer state = 1 -DISABLED
Mode = Duplex
Unit = Primary
Unit ID = 48

Redundancy Mode (Operational) = sso
Redundancy Mode (Configured) = sso
Redundancy State = sso
Maintenance Mode = Disabled
Manual Swact = disabled (the peer unit is still initializing)
Communications = Down Reason: Failure

client count = 119
client_notification_TMR = 30000 milliseconds
RF debug mask = 0x0

##################################################################

Do you have any commands for me to find out what is producing the failure?

I tried using the following debug command but did not catch any event:

debug redundancy kpa
debug redundancy errors
debug redundancy status
debug redundancy info
debug redundancy idb-sync-history

Router1 uses Version 16.3.5

Router2 uses Version 16.3.3

Thanks guys!

Steffen

[Mark Elsen]

 - Please post the output of show platform diag 

  M.

[Steffen Pontiac]

Router1#sh platform diag
Chassis type: ASR1002-HX

Slot: 0, ASR1002-HX
Running state : ok
Internal state : online
Internal operational state : ok
Physical insert detect time : 00:00:37 (2y11w ago)
Software declared up time : 00:01:08 (2y11w ago)
CPLD version : 15100715
Firmware version : 16.2(2r)

Sub-slot: 0/0, BUILT-IN-EPA-8x1G
Operational status : ok
Internal state : inserted
Physical insert detect time : 00:01:33 (2y11w ago)
Logical insert detect time : 00:01:33 (2y11w ago)

Sub-slot: 0/1, BUILT-IN-EPA-8x10G
Operational status : ok
Internal state : inserted
Physical insert detect time : 00:01:34 (2y11w ago)
Logical insert detect time : 00:01:34 (2y11w ago)

Slot: R0, ASR1002-HX
Running state : ok
Internal state : online
Internal operational state : ok
Physical insert detect time : 00:00:37 (2y11w ago)
Software declared up time : 00:00:37 (2y11w ago)
CPLD version : 15100715
Firmware version : 16.2(2r)

Sub-slot: R0/0,
Running state : ok, active
Logical insert detect time : 00:00:37 (2y11w ago)

Sub-slot: R0/1,
Running state : init, standby
Logical insert detect time : 00:02:01 (2y11w ago)

Slot: F0, ASR1002-HX
Running state : ok, active
Internal state : online
Internal operational state : ok
Physical insert detect time : 00:00:37 (2y11w ago)
Software declared up time : 00:01:03 (2y11w ago)
Hardware ready signal time : 00:01:02 (2y11w ago)
Packet ready signal time : 00:01:35 (2y11w ago)
CPLD version : 15100715
Firmware version : 16.2(2r)

Slot: P0, ASR1000X-AC-750W
State : ok
Physical insert detect time : 00:00:53 (2y11w ago)

Slot: P1, ASR1000X-AC-750W
State : ok
Physical insert detect time : 00:00:53 (2y11w ago)

Slot: P2, ASR1002HX-FAN
State : ok
Physical insert detect time : 00:00:52 (2y11w ago)

#############################################################

Router2#sh platform diag
Chassis type: ASR1002-X

Slot: 0, ASR1002-X
Running state : ok
Internal state : online
Internal operational state : ok
Physical insert detect time : 00:00:20 (19:32:09 ago)
Software declared up time : 00:00:55 (19:31:34 ago)
CPLD version : 14012203
Firmware version : 16.2(1r)

Sub-slot: 0/0, 6XGE-BUILT-IN
Operational status : ok
Internal state : inserted
Physical insert detect time : 00:01:29 (19:31:00 ago)
Logical insert detect time : 00:01:29 (19:31:00 ago)

Slot: R0, ASR1002-X
Running state : ok
Internal state : online
Internal operational state : ok
Physical insert detect time : 00:00:20 (19:32:09 ago)
Software declared up time : 00:00:20 (19:32:09 ago)
CPLD version : 14012203
Firmware version : 16.2(1r)

Sub-slot: R0/0,
Running state : ok, active
Logical insert detect time : 00:00:20 (19:32:09 ago)

Sub-slot: R0/1,
Running state : init, standby
Logical insert detect time : 00:01:59 (19:30:30 ago)

Slot: F0, ASR1002-X
Running state : ok, active
Internal state : online
Internal operational state : ok
Physical insert detect time : 00:00:20 (19:32:09 ago)
Software declared up time : 00:01:14 (19:31:15 ago)
Hardware ready signal time : 00:00:48 (19:31:41 ago)
Packet ready signal time : 00:01:28 (19:31:01 ago)
CPLD version : 14012203
Firmware version : 16.2(1r)

Slot: P0, ASR1002-PWR-AC
State : ok
Physical insert detect time : 00:00:40 (19:31:49 ago)

Slot: P1, ASR1002-PWR-AC
State : ok
Physical insert detect time : 00:00:40 (19:31:49 ago)

[Mark Elsen]

  - If you can easily identify the faulty router-module by for instance a led indicator , try re-seating it, but attach a console to your device first, followup on console messages.

 M.

[Steffen Pontiac]

Hi marce1000, the routers are not locally. BR,Steffen

[Cristian Matei]

Hi,

    What does "show redundancy history" say?

Regards,

Cristian Matei.

[Steffen Pontiac]

Hi,

I attached both outputs to this post.

BR,

Steffen

[Cristian Matei]

Hi,

    Somehow i missed the second command: " show redundancy".

Regards,

Cristian Matei.

[Steffen Pontiac]

NP, thanks for helping christian.

Router1#sh redundancy
Redundant System Information :
------------------------------
Available system uptime = 2 years, 12 weeks, 21 hours, 19 minutes
Switchovers system experienced = 0
Standby failures = 0
Last switchover reason = none

Hardware Mode = Duplex
Configured Redundancy Mode = sso
Operating Redundancy Mode = sso
Maintenance Mode = Disabled
Communications = Down Reason: Simplex mode

Current Processor Information :
-------------------------------
Active Location = slot 6
Current Software state = ACTIVE
Uptime in current state = 2 years, 12 weeks, 21 hours, 19 minutes
Image Version = Cisco IOS Software [Denali], ASR1000 Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.3.5, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2017 by Cisco Systems, Inc.
Compiled Thu 05-Oct-17 01:20 by mcpre
BOOT = bootflash:asr1000-universalk9.16.03.05.SPA.bin,1;bootflash:,1;
CONFIG_FILE =
Configuration register = 0x2102

Peer (slot: 7) information is not available because it is in 'DISABLED' state

Router2#sh redundancy
Redundant System Information :
------------------------------
Available system uptime = 3 days, 15 hours, 35 minutes
Switchovers system experienced = 0
Standby failures = 0
Last switchover reason = none

Hardware Mode = Duplex
Configured Redundancy Mode = sso
Operating Redundancy Mode = sso
Maintenance Mode = Disabled
Communications = Down Reason: Simplex mode

Current Processor Information :
-------------------------------
Active Location = slot 6
Current Software state = ACTIVE
Uptime in current state = 3 days, 15 hours, 35 minutes
Image Version = Cisco IOS Software [Denali], ASR1000 Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.3.3, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2017 by Cisco Systems, Inc.
Compiled Tue 28-Feb-17 06:09 by mcpre
Configuration register = 0x2102

Peer (slot: 7) information is not available because it is in 'DISABLED' state

[Cristian Matei]

Hi,

    I see "Peer (slot: 7) information is not available because it is in 'DISABLED' state". Usually this is a hardware issue (needs RMA), or a proper reseat of the module.

Regards,

Cristian Matei.

[Steffen Pontiac]

Hi Christian,

sorry for the late replay.

Since we have the "Init" state message on a couple of ASR1K Router it looks more like an Software issue to me.

I have to check them to get an overview of the affected routers and open a TAC case to let them investigate the issue.

@christian and Marce

Thanks for participating to my question. ;-)

BR,
Steffen

[terencepowell]

Hey Steffen,

I know it's been several years since dealing with this matter but did you ever find a solution? I have the exact same issue as you describe above.

Thanks!

Terence