A9K-4X100GE-TR, FIA-0 UP and FIA-1,2,3 are DOWN

[Volgash]

Hello everyone,

I have a A9K-4X100GE-TR that has problems with the N. is there anybody has same experience and have any idea.

RP/0/RSP0/CPU0:February(admin)#sh platform
Thu Feb 15 15:49:05.009 UTC
Node Type State Config State
-----------------------------------------------------------------------------
0/RSP0/CPU0 A9K-RSP880-LT-TR(Active) February XR RUN PWR,NSHUT,MON
0/FT0/SP ASR-9904-FAN READY
0/1/CPU0 A9K-4X100GE-TR February XR RUN PWR,NSHUT,MON
0/PS0/M0/SP PWR-3KW-AC-V2 READY PWR,NSHUT,MON
0/PS0/M1/SP PWR-3KW-AC-V2 READY PWR,NSHUT,MON
0/PS0/M2/SP PWR-3KW-AC-V2 READY PWR,NSHUT,MON
0/PS0/M3/SP PWR-3KW-AC-V2 READY PWR,NSHUT,MON
RP/0/RSP0/CPU0:February(admin)#
RP/0/RSP0/CPU0:February(admin)#
RP/0/RSP0/CPU0:February(admin)#
RP/0/RSP0/CPU0:February(admin)#exit
RP/0/RSP0/CPU0:February#
RP/0/RSP0/CPU0:February#
RP/0/RSP0/CPU0:February#
RP/0/RSP0/CPU0:February#
RP/0/RSP0/CPU0:February#
RP/0/RSP0/CPU0:February#show controller np sum all
Thu Feb 15 15:51:46.572 UTC

Node: 0/1/CPU0:
----------------------------------------------------------------
[total 4 NP] Driver Version 14.46a Build 8
NP 0 : Hardware Revision: v2 A1
: Ucode - Version: 1.0 Build Date: ( Sep 28 2017, 21:32:00 )
: NP core frequency: 575 MHz
: DDR frequency : 933 MHz
: State: UP
NP 1 : Hardware Revision: Not available
: Ucode - Version: Not available
: NP core frequency: Not available
: DDR frequency: Not available
: State: DOWN
NP 2 : Hardware Revision: Not available
: Ucode - Version: Not available
: NP core frequency: Not available
: DDR frequency: Not available
: State: DOWN
NP 3 : Hardware Revision: Not available
: Ucode - Version: Not available
: NP core frequency: Not available
: DDR frequency: Not available
: State: DOWN
RP/0/RSP0/CPU0:February#
RP/0/RSP0/CPU0:February#
RP/0/RSP0/CPU0:February#
RP/0/RSP0/CPU0:February#
RP/0/RSP0/CPU0:February#
RP/0/RSP0/CPU0:February#show controllers fabric fia link-status loc 0/1/CPU0
Thu Feb 15 15:52:32.516 UTC

********** FIA-0 **********
Category: link-0
spaui link-0 Up
spaui link-1 Up
arb link-0 Up
xbar link-0 Up
xbar link-1 Up
xbar link-2 Up

********** FIA-1 **********
Category: link-1
spaui link-0 Down
spaui link-1 Down
arb link-0 Up
xbar link-0 Up
xbar link-1 Up
xbar link-2 Up

********** FIA-2 **********
Category: link-2
spaui link-0 Down
spaui link-1 Down
arb link-0 Up
xbar link-0 Up
xbar link-1 Up
xbar link-2 Up

********** FIA-3 **********
Category: link-3
spaui link-0 Down
spaui link-1 Down
arb link-0 Up
xbar link-0 Up
xbar link-1 Up
xbar link-2 Up
RP/0/RSP0/CPU0:February#
RP/0/RSP0/CPU0:February#
RP/0/RSP0/CPU0:February#
RP/0/RSP0/CPU0:February#

[anpetit]

Hello @Volgash ,
Hope your are doing well. 
Regarding this faulty linecard, which release are you running on ?
Would it be possible to share:  

show logging (from 0/1/CPU0 event)
show controllers np ports all location 0/1/CPU0

Slices are down which might indicate a "possible" hardware issue.

In my opinion, the best way to treat your thread will be to open a TAC case to gather all necessary logs. Then, if RCA is to replace the card, then you will already have the TAC Case opened to proceed with a RMA. 
Let me know if it helps you.

Kind Regards,
Antoine

[Volgash · Accepted solution]

Hi Antoine

Thank you so much for your help. I got it.

P/0/RSP0/CPU0:February#show controllers np ports all location 0/1/CPU0
Fri Feb 16 10:09:13.798 UTC

Node: 0/1/CPU0:
----------------------------------------------------------------

NP Bridge Fia Ports
-- ------ --- ---------------------------------------------------
0 -- 0 HundredGigE0/1/0/0

NP1 is down.

NP2 is down.

NP3 is down.

RP/0/RSP0/CPU0:February#show logging location 0/1/CPU0
Fri Feb 16 10:13:56.547 UTC
Syslog logging: enabled (0 messages dropped, 0 flushes, 0 overruns)
Console logging: level warnings, 102 messages logged
Monitor logging: level debugging, 0 messages logged
Trap logging: level informational, 0 messages logged
Buffer logging: level debugging, 237 messages logged

Log Buffer (2097152 bytes):

LC/0/1/CPU0:Jan 1 00:01:21.426 : init[65540]: %OS-INIT-7-MBI_STARTED : total time 26.270 seconds
LC/0/1/CPU0:Feb 16 08:47:47.528 : sysmgr[95]: %OS-SYSMGR-5-NOTICE : Card is COLD started
LC/0/1/CPU0:Feb 16 08:47:47.734 : init[65540]: %OS-INIT-7-INSTALL_READY : total time 51.539 seconds
LC/0/1/CPU0:Feb 16 08:47:48.021 : sysmgr[350]: %OS-SYSMGR-6-INFO : Backup system manager is ready
LC/0/1/CPU0:Feb 16 08:47:59.761 : prm_server_to[302]: %PLATFORM-NP-3-UNEXPECTED : Logfile storage error
LC/0/1/CPU0:Feb 16 08:48:17.256 : prm_server_to[302]: %PLATFORM-QOSHAL-6-FLOW_QOS_CONFIG : Flow Aware QoS is disabled. max flow count set to 0
LC/0/1/CPU0:Feb 16 08:48:17.278 : prm_server_to[302]: %PLATFORM-QOSHAL-6-QOS_MODE_CHANGE : Line-card QoS mode set to child-shaping-enabled.
LC/0/1/CPU0:Feb 16 08:48:17.278 : prm_server_to[302]: %PLATFORM-QOSHAL-6-QOS_MODE_CHANGE : Line-card QoS mode set to lowburst-disabled.
LC/0/1/CPU0:Feb 16 08:48:17.278 : prm_server_to[302]: %PLATFORM-QOSHAL-6-QOS_MODE_CHANGE : Line-card QoS mode set to l4-wred-buffer-mode-disabled.
LC/0/1/CPU0:Feb 16 08:48:17.278 : prm_server_to[302]: %PLATFORM-QOSHAL-6-QOS_MODE_CHANGE : Line-card QoS mode set to per-priority-limit-disabled.
LC/0/1/CPU0:Feb 16 08:48:21.881 : pciesvr[77]: %PLATFORM-PCIE-1-ERROR : B/D/F 0/1/0 dev name:IB 1a & 1b: AER| UES: 0x00004000 CES: 0x00000000 RES: 0x00000024 IDS: 0x00080a00| HDRS: 0x00000000 0x00000000 0x00000000 0x00000000
LC/0/1/CPU0:Feb 16 08:48:21.882 : pciesvr[77]: %PLATFORM-PCIE-1-ERROR : B/D/F 0/1/0 dev name:IB 1a & 1b: PCIE| DevCap 0x00008001, DevCntl 0x00000007, DevStat 0x00000002| LinkCap 0x017a7043, LinkCntl 0x00000000, LinkStat 0x00003042| SlotCap 0x00180040, SlotCntl 0x000007c0, SlotStat 0x00000148
LC/0/1/CPU0:Feb 16 08:48:21.890 : pciesvr[77]: %PLATFORM-PCIE-6-ROOT_DEVICE_BUS_INFO : PCI/PCIe Device: bus/device/function 0/1/0 - Uncorrectable Non-Fatal Error: Completion Timeout
LC/0/1/CPU0:Feb 16 08:48:21.890 : pciesvr[77]: %PLATFORM-PCIE-1-ERROR : B/D/F 0/1/0 dev name:IB 1a & 1b: AER| UES: 0x00004000 CES: 0x00000000 RES: 0x00000000 IDS: 0x00080a00| HDRS: 0x00000000 0x00000000 0x00000000 0x00000000
LC/0/1/CPU0:Feb 16 08:48:21.890 : pciesvr[77]: %PLATFORM-PCIE-1-ERROR : B/D/F 0/1/0 dev name:IB 1a & 1b: PCIE| DevCap 0x00008001, DevCntl 0x00000007, DevStat 0x00000000| LinkCap 0x017a7043, LinkCntl 0x00000000, LinkStat 0x00003042| SlotCap 0x00180040, SlotCntl 0x000007c0, SlotStat 0x00000148
LC/0/1/CPU0:Feb 16 08:48:21.892 : pciesvr[77]: %PLATFORM-PCIE-1-ERROR : B/D/F 0/1/0 dev name:IB 1a & 1b: AER| UES: 0x00004000 CES: 0x00000000 RES: 0x0000002c IDS: 0x00080a00| HDRS: 0x00000000 0x00000000 0x00000000 0x00000000
LC/0/1/CPU0:Feb 16 08:48:21.892 : pciesvr[77]: %PLATFORM-PCIE-1-ERROR : B/D/F 0/1/0 dev name:IB 1a & 1b: PCIE| DevCap 0x00008001, DevCntl 0x00000007, DevStat 0x00000000| LinkCap 0x017a7043, LinkCntl 0x00000000, LinkStat 0x00003042| SlotCap 0x00180040, SlotCntl 0x000007c0, SlotStat 0x00000148
LC/0/1/CPU0:Feb 16 08:48:21.894 : pciesvr[77]: %PLATFORM-PCIE-1-ERROR : B/D/F 0/1/0 dev name:IB 1a & 1b: AER| UES: 0x00004000 CES: 0x00000000 RES: 0x0000002c IDS: 0x00080a00| HDRS: 0x00000000 0x00000000 0x00000000 0x00000000
LC/0/1/CPU0:Feb 16 08:48:21.894 : pciesvr[77]: %PLATFORM-PCIE-1-ERROR : B/D/F 0/1/0 dev name:IB 1a & 1b: PCIE| DevCap 0x00008001, DevCntl 0x00000007, DevStat 0x00000002| LinkCap 0x017a7043, LinkC