C9404R Standby Sudden Reboot

[Noah L]

May I ask why the Standby on the C9404R rebooted suddenly. Please see logs below.

May 30 17:18:51 PST: %IOSXE_OIR-6-REMCARD: Card (cc) removed from chassis 2 slot 1
May 30 17:18:51 PST: %IOSXE_OIR-6-REMCARD: Card (cc) removed from chassis 2 slot 2
May 30 17:18:51 PST: %IOSXE_OIR-6-REMCARD: Card (cc) removed from chassis 2 slot 3
May 30 17:18:51 PST: %IOSXE_OIR-6-REMCARD: Card (cc) removed from chassis 2 slot 4
May 30 17:18:51 PST: %IOSXE_PEM-6-REMPEM_FM: PEM/FM chassis 2 slot P1 removed
May 30 17:18:51 PST: %IOSXE_PEM-6-REMPEM_FM: PEM/FM chassis 2 slot P2 removed
May 30 17:18:51 PST: %IOSXE_PEM-6-REMPEM_FM: PEM/FM chassis 2 slot P3 removed
May 30 17:18:51 PST: %IOSXE_PEM-6-REMPEM_FM: PEM/FM chassis 2 slot P4 removed
May 30 17:18:51 PST: %IOSXE_PEM-6-REMPEM_FM: PEM/FM chassis 2 slot P9 removed
May 30 17:18:51 PST: %IOSXE_OIR-6-REMSPA: SPA removed from chassis 2 subslot 1/0, interfaces disabled
May 30 17:18:51 PST: %IOSXE_OIR-6-REMSPA: SPA removed from chassis 2 subslot 2/0, interfaces disabled
May 30 17:18:51 PST: %IOSXE_OIR-6-REMSPA: SPA removed from chassis 2 subslot 4/0, interfaces disabled
May 30 17:18:51 PST: %REDUNDANCY-3-STANDBY_LOST: Standby processor fault (PEER_NOT_PRESENT)
May 30 17:18:51 PST: %REDUNDANCY-3-STANDBY_LOST: Standby processor fault (PEER_DOWN)
May 30 17:18:51 PST: %REDUNDANCY-3-STANDBY_LOST: Standby processor fault (PEER_REDUNDANCY_STATE_CHANGE)
May 30 17:18:51 PST: %SPA_OIR-6-OFFLINECARD: SPA (C9400-LC-48S) offline in chassis 2 subslot 1/0
May 30 17:18:51 PST: %SPA_OIR-6-OFFLINECARD: SPA (C9400-SUP-1XL) offline in chassis 2 subslot 2/0
May 30 17:18:51 PST: %STACKMGR-6-CHASSIS_REMOVED: Chassis 1 R0/0: stack_mgr: Chassis 2 has been removed from the stack.
May 30 17:18:52 PST: %SPA_OIR-6-OFFLINECARD: SPA (C9400-LC-48S) offline in chassis 2 subslot 4/0
May 30 17:18:52 PST: %RF-5-RF_RELOAD: Peer reload. Reason: EHSA standby down
May 30 17:18:52 PST: %IOSXE_REDUNDANCY-6-PEER_LOST: Active detected switch 2 is no longer standby

show version

Cisco IOS XE Software, Version 16.09.03
Cisco IOS Software [Fuji], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.9.3, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2019 by Cisco Systems, Inc.

[Flavio Miranda]

Hi

 Are this logs from the Primary switch? Can you show the logs from the Standby?

[Noah L]

Hi Flavio,

Switches are in VSS. Yes this is from the Primary switch.

[Flavio Miranda]

Alright, they are in VSS.  But still you have two switches, right?

"Active detected switch 2 is no longer standby"

What I meant is, did you plugged a console cable on the second box to see if any log could give a clue about what happened?

[Noah L]

Let me check on this.

Wont the both switches output the same logs since they are in VSS?

[Flavio Miranda]

If we had a split brain scenario, the switch must store the logs somewhere.

[marce1000]

 - FYI : https://bst.cloudapps.cisco.com/bugsearch?pf=prdNm&kw=IOSXE_OIR-6-REMCARD&bt=custV&sb=anfr

 Apparently the full bug report is not accessible for the general public , but if you hoover over the link you can see a summary description of it on the right. Some further suggestion   .  consider using advisory release :  https://software.cisco.com/download/home/286320244/type/282046477/release/Amsterdam-17.3.5 , if not yet done.

                         These might be related too :

  https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvs30569

  https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvy18431

 M.

 M.

[Noah L]

I dont think this is the bug we are hitting

Cat9400 standalone chassis with dual supervisor inserted , after issue the "redundancy force-switchover" command, the previous active supervisor doesn't come e up.

Supervisor is up

"show module" from the new active supervisor doesn't see the new standby supervisor, and the console output of the new standby supervisor stuck at loading the IOS image.

On show module, Standby supervisor status is OK.

"show log" on new active supervisor shows below l..

I cant see past this line.

[marce1000]

            - I would still advise to go for the mentioned advisory release , if not yet installed

 M.

[Noah L]

Noted on this.

I would like to determine the root cause to give as a reason for the upgrade.

[marce1000]

          - That's the point , if the bug reports are not determining the cause (fully) , it is always good to go for the advisory release and check if that can resolve the issue.

 M.

[Leo Laohoo]

---

@Noah L wrote:

May 30 17:18:52 PST: %RF-5-RF_RELOAD: Peer reload. Reason: EHSA standby down

---

EHSA Standby Down is a "generic" term.  If the cause of the crash is not "coded" or not found in the coded list of reasons, it will fall into this generic term.  

Please post the complete output to the following commands: 

1. dir crashinfo-2:
2. dir bootflash-2:core
3. sh platform resources
4. sh platform software status con brief