ASR9906, 2 interfaces didn't come back up after SMU installation

[ThomasD86]

Hi,

we're in the process of installing several SMUs on our 9906 chassis running 7.3.2. We've already upgraded several and, the process went without a hitch. Tonight, however after upgrading a note 2 interfaces didn't come back up. Trying to shut down the controller, the interface or, reconfiguring it altogether didn't help.

With the "show controllers internal" CLI the error reported is the following:

H/W Link Defects : (0x0000000000000800) FIA_SHUTDOWN
H/W Raw Link Defects : (0x0000000000000800) FIA_SHUTDOWN

In the PFM alarms, the LC (A9k-48x10GE-1G-SE) raises the following ones:

node: node0_0_CPU0
---------------------
CURRENT TIME: Jan 11 05:32:22 2023
PFM TOTAL: 4 EMERGENCY/ALERT(E/A): 4 CRITICAL(CR): 0 ERROR(ER): 0
-------------------------------------------------------------------------------------------------
Raised Time         |S#|Fault Name  |Sev|Proc_ID|Dev/Path Name |Handle
--------------------+--+------------+---+-------+--------------+----------
Jan 11 03:36:24 2023|2 |RX_LINK_ERR |E/A|4852   |Fabric Arbiter|0x1001001
Jan 11 03:36:24 2023|2 |SYNC_ERR    |E/A|4852   |Fabric Arbiter|0x1001001
Jan 11 03:36:24 2023|2 |RX_LINK_ERR |E/A|4852   |Fabric Arbiter|0x1001000
Jan 11 03:36:24 2023|2 |SYNC_ERR    |E/A|4852   |Fabric Arbiter|0x1001000

Can anyone advise what are the best practices to troubleshoot this further? 

Thanks

[smilstea · Accepted solution]

From the outputs you have shared it seems like the FIA is shutdown. If you check show controller np ports all loc 0/0/cpu0 you should see the ports sharing a fia.

This could be hardware / software / or transient. When an ASIC like a FIA is shutdown there will be corresponding syslogs and logs in show tech fabric. I would need to see those to know what kind of fault it is. You can try to reload the LC to recover if its transient. But again without logs I cannot tell you which of those 3 problem types it is.

Sam

[ThomasD86]

Hi @smilstea

after troubleshooting the issue further, we ended up opening SR694880116 where we shared all the evidences we've gathered. Reloading the LC, did fix the issue.