Line cards issue with UNPOWERED error

[behzadho]

Hello everyone

Our both 24X10 TR Line cards got powered off  because of following error messages and the only way we could bring them up was rebooting ASR9904 router manually . I do not know the reason if it is a power problem or a bug or something else. We did not have any electricity problem at all.

I want to know what  this type of problem is  and how to fix it for the future

 ce_switch_srv[54]: %PLATFORM-CE_SWITCH-6-UPDN : Interface 3 (NC_3) is down
ce_switch_srv[54]: %PLATFORM-CE_SWITCH-6-UPDN : Interface 4 (NC_4) is down
 ce_switch_srv[54]: %PLATFORM-CE_SWITCH-6-UPDN : Interface 3 (NC_3) is down
ce_switch_srv[54]: %PLATFORM-CE_SWITCH-6-UPDN : Interface 4 (NC_4) is down

shelfmgr[407]: %PLATFORM-SHELFMGR-1-NO_POWER_AVAIL : Node 0/0/CPU0 is kept in unpowered state because of insufficient power.
shelfmgr[407]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/0/CPU0 A9K-24x10GE-TR state:UNPOWERED
 shelfmgr[407]: %PLATFORM-SHELFMGR-1-NO_POWER_AVAIL : Node 0/1/CPU0 is kept in unpowered state because of insufficient power.

pfm_node_rp[362]: %PLATFORM-CROSSBAR-1-SERDES_ERROR_LNK1 : Set|fab_xbar[213109]|0x101702f|XBAR_1_Slot_0
pfm_node_rp[362]: %PLATFORM-CROSSBAR-1-SERDES_ERROR_LNK1 :Set|fab_xbar[213109]|0x101702e|XBAR_1_Slot_3
pfm_node_rp[362]: %PLATFORM-CROSSBAR-1-SERDES_ERROR_LNK2 : Set|fab_xbar[213109]|0x101702a|XBAR_1_Slot_3
pfm_node_rp[362]: %PLATFORM-CROSSBAR-1-SERDES_ERROR_LNK2 : Set|fab_xbar[213109]|0x1017027|XBAR_1_Slot_0
pfm_node_rp[362]: %PLATFORM-CROSSBAR-1-SERDES_ERROR_LNK0 : Set|fab_xbar[213109]|0x101700e|XBAR_0_Slot_3
pfm_node_rp[362]: %PLATFORM-CROSSBAR-1-SERDES_ERROR_LNK0 : Set|fab_xbar[213109]|0x101700b|XBAR_0_Slot_0
pfm_node_rp[362]: %PLATFORM-CROSSBAR-1-SERDES_ERROR_LNK1 : Set|fab_xbar[213109]|0x1017007|XBAR_0_Slot_0
pfm_node_rp[362]: %PLATFORM-CROSSBAR-1-SERDES_ERROR_LNK0 : Set|fab_xbar[213109]|0x1017030|XBAR_1_Slot_3
pfm_node_rp[362]: %PLATFORM-CROSSBAR-1-SERDES_ERROR_LNK0 : Clear|fab_xbar[213109]|0x101700b|XBAR_0_Slot_0
pfm_node_rp[362]: %PLATFORM-CROSSBAR-1-SERDES_ERROR_LNK1 : Clear|fab_xbar[213109]|0x1017007|XBAR_0_Slot_0
pfm_node_rp[362]: %PLATFORM-CROSSBAR-1-SERDES_ERROR_LNK0 : Clear|fab_xbar[213109]|0x1017030|XBAR_1_Slot_3
pfm_node_rp[362]: %PLATFORM-CROSSBAR-1-SERDES_ERROR_LNK1 : Clear|fab_xbar[213109]|0x101702f|XBAR_1_Slot_0
pfm_node_rp[362]: %PLATFORM-CROSSBAR-1-SERDES_ERROR_LNK1 : Clear|fab_xbar[213109]|0x101702e|XBAR_1_Slot_3
pfm_node_rp[362]: %PLATFORM-CROSSBAR-1-SERDES_ERROR_LNK0 : Clear|fab_xbar[213109]|0x101700e|XBAR_0_Slot_3

Thank you for your help and kindness

[tkarnani]

Hi,

can we check

admin

show environment power

this would tell us if the system is providing enough power for the line cards.

%PLATFORM-CROSSBAR-1-SERDES_ERROR_LNK1

this tells us there are errors on the fabric links from the LC towards the RSP, i would re-seat this card to see if these errors clear " %PLATFORM-CROSSBAR-1-SERDES_ERROR_LNK1" if they do not, it would be best to open a TAC case.

possibly we may need to re-place the card if its still not clearing.

[behzadho]

Thank you for your response

This the result for power:

(admin)#show environment power-supply

Thu Aug 12 10:19:57.689 EDT
R/S/I Modules Capacity Status
(W)
0/PS0/M0/*
host PM 3000 Ok

0/PS0/M1/*
host PM 3000 Ok

0/PS0/M2/*
host PM 3000 Ok

0/PS0/M3/*
host PM 3000 Ok

R/S/I Power Supply Voltage Current
(W) (V) (A)
0/PS0/M0/* 431.3 55.3 7.8
0/PS0/M1/* 447.9 55.3 8.1
0/PS0/M2/* 447.9 55.3 8.1
0/PS0/M3/* 436.8 55.3 7.9
--------------
Total: 1763.9

R/S/I Power Draw Voltage Current
(W) (V) (A)
0/FT0/* 380 **
--------------
Total: 380.0

R/S/I Power Draw Voltage Current
(W) (V) (A)
0/RSP0/* 231.0 55.0 4.2
0/RSP1/* 209.7 55.2 3.8
--------------
Total: 440.7

R/S/I Power Draw Voltage Current
(W) (V) (A)
0/0/* 697.2 54.9 12.7
0/1/* 708.2 54.9 12.9
--------------
Total: 1405.4
** Indicates worst case power draw (No power measuring sensor)

Chassis Power Draw for rack 0: 1763.9 (W)

Power Budget Summary for Rack 0
--------------------------------

Power Shelves Type: AC
Power Budget Strict Mode: Disabled

Total Power Capacity: 12000W
Usable Power Capacity: 9000W
N+1 Supply Failure Protected Capacity: 9000W
Worst Case Power Used: 2780W

Slot Max Watts
---- ---------
0/0/CPU0 850
0/RSP0/CPU0 350
0/RSP1/CPU0 350
0/1/CPU0 850
0/FT0/SP 380

Worst Case Power Available: 6220W
N+1 Supply Protected Capacity Available: 6220W

[tkarnani]

the router seems to have enough power now.

did we check the output during the time of the issue?

thanks