Re: ASR 9001 Line card Bringdown

larbi farouk · ‎10-19-2019

Dear,

we running cluster of Cisco ASR9001 on IOS XR 6.1.4, and suddenly rack1 crash down and reboot with led 'LC fail' on

you can see below the output of show platform command:

Node Type State Config State
-----------------------------------------------------------------------------
1/RSP0/CPU0 ASR9001-RP(Active) IOS XR RUN PWR,NSHUT,MON
1/FT0/SP ASR-9001-FAN READY
1/0/CPU0 ASR9001-LC BRINGDOWN PWR,NSHUT,MON
1/PS0/M0/SP A9K-750W-AC READY PWR,NSHUT,MON
1/PS0/M1/SP A9K-750W-AC READY PWR,NSHUT,MON

In the booting process log we found '' CBC_ALRM_PWR_A_FUSE_FAILED''

RP/1/RSP0/CPU0:Oct 19 10:50:45.213 : dhcpd[1091]: %SUBSCRIBER-SUB_UTIL-5-SESSION_THROTTLE : Subscriber Infra is ready. Reason: [V4 Subscriber infra process(es) is available].
RP/1/RSP0/CPU0:Oct 19 10:50:45.213 : ipsub_ma[286]: %SUBSCRIBER-SUB_UTIL-5-SESSION_THROTTLE : Subscriber Infra is ready. Reason: [V4 Subscriber infra process(es) is available].
RP/1/RSP0/CPU0:Oct 19 10:50:45.214 : iedged[248]: %SUBSCRIBER-SUB_UTIL-5-SESSION_THROTTLE : Subscriber Infra is ready. Reason: [V4 Subscriber infra process(es) is available].
RP/1/RSP0/CPU0:Oct 19 10:50:49.852 : dsc[190]: %PLATFORM-DSC_CTRL-6-RACK_STATE : Rack 0 is now Down
RP/1/RSP0/CPU0:Oct 19 10:50:49.857 : gsp[244]: gsp_rack_down_handler: for node:0x0
RP/1/RSP0/CPU0:Oct 19 10:50:59.036 : ifmgr[250]: %PKT_INFRA-LINK-3-UPDOWN : Interface MgmtEth1/RSP0/CPU0/0, changed state to Down
RP/1/RSP0/CPU0:Oct 19 10:50:59.039 : ifmgr[250]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface MgmtEth1/RSP0/CPU0/0, changed state to Down
RP/1/RSP0/CPU0:Oct 19 10:50:59.039 : ifmgr[250]: %PKT_INFRA-LINK-3-UPDOWN : Interface MgmtEth1/RSP0/CPU0/1, changed state to Down
RP/1/RSP0/CPU0:Oct 19 10:50:59.039 : ifmgr[250]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface MgmtEth1/RSP0/CPU0/1, changed state to Down
RP/1/RSP0/CPU0:Oct 19 10:50:59.045 : cfgmgr-rp[167]: %MGBL-CONFIG-6-OIR_RESTORE : Configuration for node '1/RSP0/0' has been restored.
RP/1/RSP0/CPU0:Oct 19 10:50:59.062 : ifmgr[250]: %PKT_INFRA-LINK-3-UPDOWN : Interface MgmtEth1/RSP0/CPU0/0, changed state to Up
RP/1/RSP0/CPU0:Oct 19 10:50:59.077 : ifmgr[250]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface MgmtEth1/RSP0/CPU0/0, changed state to Up
RP/1/RSP0/CPU0:Oct 19 10:51:29.825 : canb-server[157]: %PLATFORM-CANB_SERVER-7-CBC_PRE_RESET_NOTIFICATION : Node 1/0/CPU0 , Power Sequencer Failure (0x09000000)
RP/1/RSP0/CPU0:Oct 19 10:51:29.830 : shelfmgr[421]: %PLATFORM-SHELFMGR-6-NODE_CPU_RESET : Node 1/0/CPU0 CPU reset detected.
RP/1/RSP0/CPU0:Oct 19 10:51:29.832 : shelfmgr[421]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 1/0/CPU0 ASR9001-LC state:BRINGDOWN
RP/1/RSP0/CPU0:Oct 19 10:51:29.841 : invmgr[257]: %PLATFORM-INV-6-NODE_STATE_CHANGE : Node: 1/0/CPU0, state: BRINGDOWN
RP/1/RSP0/CPU0:Oct 19 10:51:30.277 : envmon[213]: %PLATFORM-ENVMON-3-ALARM_RAISED : Raise alarm from CBC in slot 1/0/CPU0, alarm code CBC_ALRM_PWR_SEQ_FAILED Reason: OUTPUT VOLTAGE FAULT Bus:0x18 Addresss:0xc4 Chan:0x1 STATUS_VOUT 0x10
RP/1/RSP0/CPU0:Oct 19 10:51:32.081 : canb-server[157]: %PLATFORM-CANB_SERVER-3-ALARM_INDICATION : Raise alarm from CBC in slot 1/0/CPU0, alarm code CBC_ALRM_PWR_A_FUSE_FAILED
RP/1/RSP0/CPU0:Oct 19 10:52:34.451 : envmon[213]: %PLATFORM-ENVMON-3-ALARM_CLEARED : Clear alarm from CBC in slot 1/0/CPU0, alarm code CBC_ALRM_PWR_SEQ_FAILED
RP/1/RSP0/CPU0:Oct 19 10:53:59.327 : shelfmgr[421]: %PLATFORM-SHELFMGR-3-FSMTIMEOUT_RESET : Node 1/0/CPU0 is reset due to failed bootup. Node state was: 7 Timeout ID: 10
RP/1/RSP0/CPU0:Oct 19 10:53:59.451 : canb-server[157]: %PLATFORM-CANB_SERVER-7-CBC_PRE_RESET_NOTIFICATION : Node 1/0/CPU0 , Power Cycle (0x05000000)
RP/1/RSP0/CPU0:Oct 19 10:54:38.435 : canb-server[157]: %PLATFORM-CANB_SERVER-3-ALARM_CLEARED : Clear alarm from CBC in slot 1/0/CPU0, alarm code CBC_ALRM_PWR_A_FUSE_FAILED
RP/1/RSP0/CPU0:Oct 19 10:55:42.257 : envmon[213]: %PLATFORM-ENVMON-3-ALARM_RAISED : Raise alarm from CBC in slot 1/0/CPU0, alarm code CBC_ALRM_PWR_SEQ_FAILED Reason: OUTPUT VOLTAGE FAULT Bus:0x2 Addresss:0xc2 Chan:0x0 STATUS_VOUT 0x10

I dont know if its hardware problem in power supply or line card, or it can be bug in Operating system ???

larbi farouk · ‎10-23-2019

Update !

the problem persists after doing turboboot with 5.1.2, line card still in bringdown state

with error: RP/0/RSP0/CPU0:Oct 22 17:17:47.871 : canb-server[152]: %PLATFORM-CANB_SERVER-3-ALARM_INDICATION : Raise alarm from CBC in slot 0/0/CPU0, alarm code CBC_ALRM_PWR_A_FUSE_FAILED

Do you have any suggestion

smilstea · ‎10-24-2019

This is indicating that the power sequencer has failed, you will need to replace the 9001.

The bad thing about the 9001 is that it doesn't come with an image, the good thing is you have a cluster, so all you need to do is modify the admin config with the new chassis serial number, optionally set the RACK_NUM rommon variable on the new 9001, cable up the EOBC and IRL ports and then it will sync over the OS / bake it to disk and form the cluster again.

Sam

philclemens1835 · ‎10-24-2019

Larbi,

This looks very much like a hardware issue. Especially if there were no other changes around this event.

We see similar messages when we lose a line card in a 9006:

RP/0/RSP0/CPU0:Apr 16 09:53:01.217 : canb-server[154]: %PLATFORM-CANB_SERVER-7-CBC_PRE_RESET_NOTIFICATION : Node 0/0/CPU0 , Power Cycle (0x05000000)

If I were you, I would proceed with an RMA. Make sure you are aware of the license process when RMA'ing the box, which I assume will be the case here.

-Phil