ASR9010 LC 2X100GE-TR Issue

Joester Brondani · ‎01-23-2024

Hello, good afternoon everyone. I would like some advice because I have an atypical problem: I have an ASR9010, with the following specifications:

0/RSP0/CPU0 A9K-RSP880-TR(Standby) IOS XR RUN PWR,NSHUT,MON
0/RSP1/CPU0 A9K-RSP880-TR(Active) IOS XR RUN PWR,NSHUT,MON
0/2/CPU0 A9K-2x100GE-TR IOS XR RUN PWR,NSHUT,MON
0/3/CPU0 A9K-2x100GE-TR IOS XR RUN PWR,NSHUT,MON
0/4/CPU0 A9K-2x100GE-TR IOS XR RUN PWR,NSHUT,MON
0/5/CPU0 A9K-2x100GE-TR IOS XR RUN PWR,NSHUT,MON
0/7/CPU0 A9K-2x100GE-TR IOS XR RUN PWR,NSHUT,MON

The ios version is 6.7.1, cards typhoon (a9k-2x100GE-TR)

I can't use more than 4 line cards, regardless of the slot (I've already tested changing the positions of the line cards between slots) and one of them always doesn't work. The interesting thing is that if I leave just one RSP, everyone goes up. When I move up both RSPs, one slot is left out. Apparently I have a redundancy problem, but I believe it's in the configuration since I already changed the RSP, then I also changed the LC and finally I even changed the chassis :S. The problem persists.

Some errors:

show pfm location all

Dec 7 09:09:59 2023|9 |SERDES_ERROR_LNK0 |E/A|168018 |Crossbar Switc|0x1017011

RP/0/RSP1/CPU0:SPO-EQUINIX-SP4-EDGE01#show controllers fabric crossbar link-status instance 0 location 0/7/CPU0
Tue Jan 23 15:35:06.048 Brazil
PORT Remote Slot Remote Inst Logical ID Status
======================================================
00 0/7/CPU0 01 1 Up
01 0/7/CPU0 03 1 Up
02 0/7/CPU0 03 0 Up
03 0/7/CPU0 02 0 Up
04 0/7/CPU0 02 1 Up
05 0/7/CPU0 00 1 Up
07 0/RSP1/CPU0 00 1 Up
08 0/7/CPU0 00 0 Up
09 0/RSP0/CPU0 00 1 Up
11 0/RSP1/CPU0 00 0 Up
12 0/RSP0/CPU0 00 0 Up
14 0/RSP0/CPU0 01 1 Up
15 0/RSP1/CPU0 01 1 Up
16 0/RSP0/CPU0 01 0 Up
17 0/RSP1/CPU0 01 0 Down <----------------------------
24 0/7/CPU0 01 0 Up

-----------------------------------------------------------

ec 7 03:01:29.632 crossbar 0/RSP1/CPU0 t1 init xbar_fmlc_handle_chico_phase1: fab_fti_get_asic_type: 9,4,4
Dec 7 03:01:29.632 crossbar 0/RSP1/CPU0 t1 error xbar_check_run_chico_phase1: xbar:1 got DO_PHASE1 port:8 in invalid state:5
Dec 7 03:01:29.632 crossbar 0/RSP1/CPU0 t1 error xbar_fmlc_handle_chico_phase1: phase1 failed: (5,1,1),(9,1,0),0 (sp:4)(nasic:9:4) xbar:1,port:8, invalid state
Dec 7 03:01:29.637 crossbar 0/RSP1/CPU0 t1 init xbar_fmlc_handle_chico_phase1: fab_fti_get_asic_type: 9,4,4
Dec 7 03:01:29.637 crossbar 0/RSP1/CPU0 t1 error xbar_check_run_chico_phase1: xbar:1 got DO_PHASE1 port:8 in invalid state:5
Dec 7 03:01:29.637 crossbar 0/RSP1/CPU0 t1 error xbar_fmlc_handle_chico_phase1: phase1 failed: (5,1,1),(9,1,0),0 (sp:4)(nasic:9:4) xbar:1,port:8, invalid state
Dec 7 03:01:29.643 crossbar 0/RSP1/CPU0 t1 init xbar_fmlc_handle_chico_phase1: fab_fti_get_asic_type: 9,4,4
Dec 7 03:01:29.643 crossbar 0/RSP1/CPU0 t1 error xbar_check_run_chico_phase1: xbar:1 got DO_PHASE1 port:8 in invalid state:5
Dec 7 03:01:29.643 crossbar 0/RSP1/CPU0 t1 error xbar_fmlc_handle_chico_phase1: phase1 failed: (5,1,1),(9,1,0),0 (sp:4)(nasic:9:4) xbar:1,port:8, invalid state
Dec 7 03:01:29.647 crossbar 0/RSP1/CPU0 t1 init xbar_fmlc_handle_chico_phase1: fab_fti_get_asic_type: 9,4,4
Dec 7 03:01:29.647 crossbar 0/RSP1/CPU0 t1 error xbar_check_run_chico_phase1: xbar:1 got DO_PHASE1 port:8 in invalid state:5
Dec 7 03:01:29.647 crossbar 0/RSP1/CPU0 t1 error xbar_fmlc_handle_chico_phase1: phase1 failed: (5,1,1),(9,1,0),0 (sp:4)(nasic:9:4) xbar:1,port:8, invalid state
Dec 7 03:01:29.653 crossbar 0/RSP1/CPU0 t1 init xbar_fmlc_handle_chico_phase1: fab_fti_get_asic_type: 9,4,4
Dec 7 03:01:29.653 crossbar 0/RSP1/CPU0 t1 error xbar_check_run_chico_phase1: xbar:1 got DO_PHASE1 port:8 in invalid state:5
Dec 7 03:01:29.653 crossbar 0/RSP1/CPU0 t1 error xbar_fmlc_handle_chico_phase1: phase1 failed: (5,1,1),(9,1,0),0 (sp:4)(nasic:9:4) xbar:1,port:8, invalid state
detail xbar_handle_20msec_timer:xbar:1 port:8 CHSTS:DOWN
Dec 7 03:01:29.659 crossbar 0/RSP1/CPU0 t1 init xbar_fmlc_handle_chico_phase1: fab_fti_get_asic_type: 9,4,4

error xbar_disable_slot: attempt to disable lport:11 state:0 before training
Dec 7 04:59:21.250 crossbar 0/RSP1/CPU0 t1 error xbar_disable_slot: attempt to disable lport:11 state:0 before training
Dec 7 04:59:21.250 crossbar 0/RSP1/CPU0 t1 error xbar_disable_slot: attempt to disable lport:11 state:0 before training
Dec 7 04:59:31.323 crossbar 0/RSP1/CPU0 t1 error xbar_disable_slot: attempt to disable lport:11 state:0 before training
Dec 7 04:59:31.323 crossbar 0/RSP1/CPU0 t1 error xbar_disable_slot: attempt to disable lport:11 state:0 before training
Dec 7 04:59:41.402 crossbar 0/RSP1/CPU0 t1 error xbar_disable_slot: attempt to disable lport:11 state:0 before training
Dec 7 04:59:41.402 crossbar 0/RSP1/CPU0 t1 error xbar_disable_slot: attempt to disable lport:11 state:0 before training
Dec 7 04:59:51.482 crossbar 0/RSP1/CPU0 t1 error xbar_disable_slot: attempt to disable lport:11 state:0 before training
Dec 7 04:59:51.482 crossbar 0/RSP1/CPU0 t1 error xbar_disable_slot: attempt to disable lport:11 state:0 before training

-------------------------------

Dec 7 09:11:34.941 crossbar 0/RSP1/CPU0 t1 event xbar_shelfmgr_notif_handler: Got shelfmgr event for slot:9 type:2 nodestate:6
Dec 7 09:11:52.971 crossbar 0/RSP1/CPU0 t1 detail xbar_fmlc_cbfn_cmn: FIA_OFFLINE:slot:9
Dec 7 09:11:52.974 crossbar 0/RSP1/CPU0 t1 detail xbar_fmlc_cbfn_cmn: FIA_OFFLINE:slot:9
Dec 7 09:11:52.978 crossbar 0/RSP1/CPU0 t1 detail xbar_fmlc_cbfn_cmn: FIA_OFFLINE:slot:9
Dec 7 09:11:52.981 crossbar 0/RSP1/CPU0 t1 detail xbar_fmlc_cbfn_cmn: FIA_OFFLINE:slot:9

Any help is very welcome, thank you

balaji.bandi · ‎01-23-2024

The interesting thing is that if I leave just one RSP, everyone goes up

Not sure - but this sounds like for me - i had similar issue with Power Supply- when we change to higher power supply all the cards should be working as expected.

check what PSU were installed.

https://content.cisco.com/chapter.sjs?uri=/searchable/chapter/content/en/us/td/docs/iosxr/asr9000/hardware-install/overview-reference/b-asr9k-overview-reference-guide/b-asr9k-hardware-installation-guide_chapter_011.html.xml

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

Joester Brondani · ‎01-24-2024

Hello, thank you for your response. I forgot to also post the electrical specifications, apparently I have enough power. But still, your observation makes a lot of sense, I can try installing v3 fonts. Thank you again

Wed Jan 24 08:20:11.645 Brazil
R/S/I Modules Capacity Status
(W)
0/PS0/M0/*
host PM 2100 Ok

0/PS0/M1/*
host PM 2100 Ok

0/PS0/M2/*
host PM 2100 Ok

0/PS0/M3/*
host PM 2100 Ok

0/PS1/M0/*
host PM 2100 Ok

0/PS1/M1/*
host PM 2100 Ok

0/PS1/M2/*
host PM 2100 Ok

0/PS1/M3/*
host PM 2100 Ok

R/S/I Power Supply Voltage Current
(W) (V) (A)
0/PS0/M0/* 504.1 54.8 9.2
0/PS0/M1/* 516.0 54.9 9.4
0/PS0/M2/* 528.0 55.0 9.6
0/PS0/M3/* 541.5 54.7 9.9
0/PS1/M0/* 503.2 54.7 9.2
0/PS1/M1/* 517.0 55.0 9.4
0/PS1/M2/* 526.0 54.8 9.6
0/PS1/M3/* 470.4 54.7 8.6
--------------
Total: 4106.2

R/S/I Power Draw Voltage Current
(W) (V) (A)
0/FT0/* 141.7 54.5 2.6
0/FT1/* 164.1 54.7 3.0
--------------
Total: 305.8

R/S/I Power Draw Voltage Current
(W) (V) (A)
0/RSP0/* 272.5 54.5 5.0
0/RSP1/* 278.4 54.6 5.1
--------------
Total: 550.9

R/S/I Power Draw Voltage Current
(W) (V) (A)
0/2/* 724.4 55.3 13.1
0/3/* 690.4 54.8 12.6
0/4/* 704.3 54.6 12.9
0/5/* 698.8 54.6 12.8
0/7/* 622.4 54.6 11.4
--------------
Total: 3440.3

Power Redundancy Mode: N + 1

Total Power Capacity: 16800W
Usable Power Capacity: 14700W
N+1 Supply Failure Protected Capacity: 14700W
Worst Case Power Used: 6435W

Slot Max Watts
---- ---------
0/2/CPU0 875
0/3/CPU0 875
0/RSP0/CPU0 430
0/RSP1/CPU0 430
0/4/CPU0 875
0/5/CPU0 875
0/7/CPU0 875
0/FT0/SP 600
0/FT1/SP 600

Worst Case Power Available: 8265W
N+1 Supply Protected Capacity Available: 8265W