Adding second RSP on ASR9010 failed

Lyphiard · ‎12-20-2018

Today, I've just tried adding a second RSP to a running ASR9010 on IOS XR 5.3.4. However, the card's state does not show as XR RUN and there are several error messages in the debug logs.

Does this mean that this card has a hardware issue?

shelfmgr[400]: %PLATFORM-SHELFMGR_HAL-7-CAN_INFO : Can API: slot 5 online event. 
shelfmgr[400]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/RSP1/CPU0 A9K-RSP440-SE state:PRESENT 
invmgr[255]: %PLATFORM-INV-6-OIRIN : OIR: Node 0/RSP1/CPU0 Sn: <redacted> inserted 
shelfmgr[400]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/RSP1/CPU0 A9K-RSP440-SE state:BRINGDOWN 
shelfmgr[400]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/RSP1/CPU0 A9K-RSP440-SE state:NOT-PRESENT 
invmgr[255]: %PLATFORM-INV-6-NODE_STATE_CHANGE : Node: 0/RSP1/CPU0, state: BRINGDOWN 
invmgr[255]: %PLATFORM-INV-6-OIROUT : OIR: Node 0/RSP1/CPU0 Sn: <redacted> removed 
shelfmgr[400]: %PLATFORM-SHELFMGR_HAL-7-CAN_INFO : Can API: slot 5 online event. 
shelfmgr[400]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/RSP1/CPU0 A9K-RSP440-SE state:PRESENT 
invmgr[255]: %PLATFORM-INV-6-OIRIN : OIR: Node 0/RSP1/CPU0 Sn: <redacted> inserted 
canb-server[153]: %PLATFORM-CANB_SERVER-3-ALARM_INDICATION : Raise alarm from CBC in slot 0/RSP1/CPU0, alarm code CBC_ALRM_PWR_A_FUSE_FAILED 
envmon[209]: %PLATFORM-ENVMON-3-ALARM_RAISED : Raise alarm from CBC in slot 0/RSP1/CPU0, alarm code CBC_ALRM_PWR_SEQ_FAILED Reason: IBV FAULT Bus:0x1 Addresss:0xb8 Chan:0x0 STATUS_INPUT 0x8 
shelfmgr[400]: %PLATFORM-SHELFMGR-3-FSMTIMEOUT_RESET : Node 0/RSP1/CPU0 is reset due to failed bootup. Node state was: 1 Timeout ID: 10 
canb-server[153]: %PLATFORM-CANB_SERVER-7-CBC_PRE_RESET_NOTIFICATION : Node 0/RSP1/CPU0 , Power Cycle (0x05000000) 
shelfmgr[400]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/RSP1/CPU0 A9K-RSP440-SE state:ROMMON 
shelfmgr[400]: %PLATFORM-SHELFMGR-3-FSMTIMEOUT_RESET : Node 0/RSP1/CPU0 is reset due to failed bootup. Node state was: 3 Timeout ID: 10 
canb-server[153]: %PLATFORM-CANB_SERVER-7-CBC_PRE_RESET_NOTIFICATION : Node 0/RSP1/CPU0 , Power Cycle (0x05000000) 
shelfmgr[400]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/RSP1/CPU0 A9K-RSP440-SE state:BRINGDOWN 
invmgr[255]: %PLATFORM-INV-6-NODE_STATE_CHANGE : Node: 0/RSP1/CPU0, state: BRINGDOWN 
shelfmgr[400]: %PLATFORM-SHELFMGR-3-FSMTIMEOUT_RESET : Node 0/RSP1/CPU0 is reset due to failed bootup. Node state was: 7 Timeout ID: 10 
canb-server[153]: %PLATFORM-CANB_SERVER-7-CBC_PRE_RESET_NOTIFICATION : Node 0/RSP1/CPU0 , Power Cycle (0x05000000)

I did notice that the two RSP's were different "versions" (V06 vs V05). Would this make any difference?

RP/0/RSP0/CPU0:router(admin)#show platform summary location all
Thu Dec 20 13:23:00.705 EST
-------------------------------------------------------------------------------
     Platform Node : 0/RSP0/CPU0 (slot 4) 
               PID : A9K-RSP440-SE 
         Card Type : ASR9K Fabric, Controller, 12G memory
            VID/SN : V06 / <redacted>
        Oper State : IOS XR RUN
        Last Reset : User Initiated reload                                                  Process: reload      
                   : Fri Nov  2 07:28:44 2018
     Configuration : Power is enabled 
                     Bootup enabled.  
                     Monitoring enabled
        Rommon Ver : Version 0.75 
        IOS SW Ver : 5.3.4
        Main Power : Power state Enabled. Estimate power 350 Watts of power required.
            Faults : N/A
-------------------------------------------------------------------------------
     Platform Node : 0/RSP1/CPU0 (slot 5) 
               PID : A9K-RSP440-SE 
         Card Type : ASR9K Fabric, Controller, 12G memory
            VID/SN : V05 / <redacted>
        Oper State : BRINGDOWN
        Last Reset : N/A
                   : N/A
     Configuration : Power is enabled 
                     Bootup enabled.  
                     Monitoring enabled
        Rommon Ver : N/A
        IOS SW Ver : 5.3.4
        Main Power : Power state Enabled. Estimate power 350 Watts of power required.
            Faults : N/A
-------------------------------------------------------------------------------

Aleksandar Vidakovic · ‎12-21-2018

If CBC_ALRM_PWR_A_FUSE_FAILED is repeatedly reported on every OIR, this RSP440 should be replaced.

Lyphiard · ‎12-25-2018

This is unrelated, but I know a secondary RSP added to a chassis will automatically sync up to the primary RSP, even if the contents of the disk/version of IOS XR is different.

Would there be a way for me to inspect and retrieve the files off a secondary RSP before they are overwritten? I only have one ASR chassis that cannot be shut down and the primary RSP cannot be disabled/removed.

Elvin Arias Soto · ‎12-24-2018

As Aleksandar mentioned, if after OIR the RSP continues with the same issue, open a TAC case, share outputs and HW troubleshooting done and you should get an RMA of your part accordingly.

Elvin