ASR9K - Node feature uptime is not available on onboard

AmyEST · ‎01-12-2023

I've run into this problem and can't seem to find how to resolve it. We're getting this error with a new LC that uptime feature not available on A9K-2x100GE-TR. I hope someone could please help clarify and if the board is faulty or if there is a way how to resolve it.

RP/0/RSP0/CPU0:router#admin show logging onboard uptime detail location 0/0/CPU0

Node: node0_0_CPU0 feature uptime is not available

Thank you,

AmyEST.

diefierr · ‎01-12-2023

AmyEST, why dont you share "show version brief" + "show platform"
is this happening only for this LC, or for all LCs and RSPs?
Any other "admin show logging onboard ...." showing information?

Diego

smilstea · ‎01-12-2023

Can you share show reboot history for the LC, as well as show logging? We have seen this issue of OBFL logs not showing when there is a fault on the card such as a CANBUS error, which would show in the logging. Diego has a good point to check if other cards see the same problem or if other OBFL (show logging onboard) commands work for the same LC). Checking show pfm loc all is good too to see if any active alarms exist on the router.

Sam

AmyEST · ‎01-12-2023

Thank you all for your quick responses. This is a new LC has that problem, but we decided to reinsert the old card in the slot 0.

Here what we found in the log and not other cards showing that.

C/0/0/CPU0:Jan 12 01:54:52.427 EST: ifmgr[216]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface HundredGigE0/0/0/1, changed state to Up
LC/0/0/CPU0:Jan 12 01:54:55.656 EST: envmon_lc[170]: %PLATFORM-ENVMON-3-OPERATION_FAIL : Failed in init_obfl_card_syslog.5399, error code OBFL syslog initialization failed
LC/0/0/CPU0:Jan 12 01:58:02.772 EST: obfl_agent-lc[295]: %PLATFORM-OBFL-3-ERR_CREATE : Failed to create the OBFL environment record
LC/0/0/CPU0:Jan 12 02:03:02.909 EST: obfl_agent-lc[295]: %PLATFORM-OBFL-3-ERR_CREATE : Failed to create the OBFL uptime record

---

RP/0/RSP0/CPU0:router#show pfm loc all
Thu Jan 12 13:35:43.225 EST

node: node0_RSP0_CPU0
---------------------
CURRENT TIME: Jan 12 13:35:43 2023
PFM TOTAL: 0 EMERGENCY/ALERT(E/A): 0 CRITICAL(CR): 0 ERROR(ER): 0
-------------------------------------------------------------------------------------------------
Raised Time |S#|Fault Name |Sev|Proc_ID|Dev/Path Name |Handle
--------------------+--+-----------------------------------+---+-------+--------------+----------
Mon Jan 01 00:00:000|--|NONE |NO |0000000|NONE |0x00000000

node: node0_3_CPU0
---------------------
CURRENT TIME: Jan 12 13:35:43 2023
PFM TOTAL: 0 EMERGENCY/ALERT(E/A): 0 CRITICAL(CR): 0 ERROR(ER): 0
-------------------------------------------------------------------------------------------------
Raised Time |S#|Fault Name |Sev|Proc_ID|Dev/Path Name |Handle
--------------------+--+-----------------------------------+---+-------+--------------+----------
Mon Jan 01 00:00:000|--|NONE |NO |0000000|NONE |0x00000000

node: node0_1_CPU0
---------------------
CURRENT TIME: Jan 12 13:35:43 2023
PFM TOTAL: 0 EMERGENCY/ALERT(E/A): 0 CRITICAL(CR): 0 ERROR(ER): 0
-------------------------------------------------------------------------------------------------
Raised Time |S#|Fault Name |Sev|Proc_ID|Dev/Path Name |Handle
--------------------+--+-----------------------------------+---+-------+--------------+----------
Mon Jan 01 00:00:000|--|NONE |NO |0000000|NONE |0x00000000

node: node0_2_CPU0
---------------------
CURRENT TIME: Jan 12 13:35:43 2023
PFM TOTAL: 0 EMERGENCY/ALERT(E/A): 0 CRITICAL(CR): 0 ERROR(ER): 0
-------------------------------------------------------------------------------------------------
Raised Time |S#|Fault Name |Sev|Proc_ID|Dev/Path Name |Handle
--------------------+--+-----------------------------------+---+-------+--------------+----------
Mon Jan 01 00:00:000|--|NONE |NO |0000000|NONE |0x00000000

node: node0_RSP1_CPU0
---------------------
CURRENT TIME: Jan 12 13:35:43 2023
PFM TOTAL: 0 EMERGENCY/ALERT(E/A): 0 CRITICAL(CR): 0 ERROR(ER): 0
-------------------------------------------------------------------------------------------------
Raised Time |S#|Fault Name |Sev|Proc_ID|Dev/Path Name |Handle
--------------------+--+-----------------------------------+---+-------+--------------+----------
Mon Jan 01 00:00:000|--|NONE |NO |0000000|NONE |0x00000000

node: node0_0_CPU0
---------------------
CURRENT TIME: Jan 12 13:35:43 2023
PFM TOTAL: 0 EMERGENCY/ALERT(E/A): 0 CRITICAL(CR): 0 ERROR(ER): 0
-------------------------------------------------------------------------------------------------
Raised Time |S#|Fault Name |Sev|Proc_ID|Dev/Path Name |Handle
--------------------+--+-----------------------------------+---+-------+--------------+----------
Mon Jan 01 00:00:000|--|NONE |NO |0000000|NONE |0x00000000

Thank you again,

AmyEST

smilstea · ‎01-12-2023

It looks like the fabric crossbar keeps getting a serdes error and then clearing:

01/10/2023 11:52:32 sev:1 0/0/CPU0 pfm_node_lc[308]: %PLATFORM-CROSSBAR-1-SERDES_ERROR_LNK1 : Set|fab_xbar[168017]|0x1017009|Slot_0_XBAR_0
01/10/2023 11:52:42 sev:1 0/0/CPU0 pfm_node_lc[308]: %PLATFORM-CROSSBAR-1-SERDES_ERROR_LNK1 : Clear|fab_xbar[168017]|0x1017009|Slot_0_XBAR_0
01/10/2023 11:53:38 sev:1 0/0/CPU0 pfm_node_lc[308]: %PLATFORM-CROSSBAR-1-SERDES_ERROR_LNK1 : Set|fab_xbar[168017]|0x1017009|Slot_0_XBAR_0
01/10/2023 11:53:48 sev:1 0/0/CPU0 pfm_node_lc[308]: %PLATFORM-CROSSBAR-1-SERDES_ERROR_LNK1 : Clear|fab_xbar[168017]|0x1017009|Slot_0_XBAR_0
01/10/2023 11:56:09 sev:1 0/0/CPU0 pfm_node_lc[308]: %PLATFORM-CROSSBAR-1-SERDES_ERROR_LNK1 : Set|fab_xbar[168017]|0x1017009|Slot_0_XBAR_0
01/10/2023 11:56:19 sev:1 0/0/CPU0 pfm_node_lc[308]: %PLATFORM-CROSSBAR-1-SERDES_ERROR_LNK1 : Clear|fab_xbar[168017]|0x1017009|Slot_0_XBAR_0
01/10/2023 11:59:20 sev:1 0/0/CPU0 pfm_node_lc[308]: %PLATFORM-CROSSBAR-1-SERDES_ERROR_LNK1 : Set|fab_xbar[168017]|0x1017009|Slot_0_XBAR_0
01/10/2023 11:59:41 sev:1 0/0/CPU0 pfm_node_lc[308]: %PLATFORM-CROSSBAR-1-SERDES_ERROR_LNK1 : Clear|fab_xbar[168017]|0x1017009|Slot_0_XBAR_0
01/10/2023 12:01:51 sev:1 0/0/CPU0 pfm_node_lc[308]: %PLATFORM-CROSSBAR-1-SERDES_ERROR_LNK1 : Set|fab_xbar[168017]|0x1017009|Slot_0_XBAR_0
01/10/2023 12:02:01 sev:1 0/0/CPU0 pfm_node_lc[308]: %PLATFORM-CROSSBAR-1-SERDES_ERROR_LNK1 : Clear|fab_xbar[168017]|0x1017009|Slot_0_XBAR_0
01/10/2023 12:08:58 sev:1 0/0/CPU0 pfm_node_lc[308]: %PLATFORM-CROSSBAR-1-SERDES_ERROR_LNK1 : Set|fab_xbar[168017]|0x1017009|Slot_0_XBAR_0
01/10/2023 12:09:08 sev:1 0/0/CPU0 pfm_node_lc[308]: %PLATFORM-CROSSBAR-1-SERDES_ERROR_LNK1 : Clear|fab_xbar[168017]|0x1017009|Slot_0_XBAR_0

i would look for any pins/backplane issues such as bent pins etc. You will need a good flashflight, not a phone, and a good quality camera to get a high resolution photo of the slot and back of the two LCs. Do not insert any further cards into this slot or these two cards in any other slots until we can say they are good.

Unfortunately the OBFL data doesn't tell me everything so I recommend opening a TAC case with a full show log, show version, show install active summary, show platform, admin show diagnostic trace error loc <both rsps>, and the pictures from the backplane of the slot and the back of the two LCs.

The problem with OBFL is likely a symptom of whatever fabric problem or backplane problem you are experiencing, yes you can format the bootflash to initialize the bootflash and have the results show up for OBFL but I fear you have a much bigger problem that needs to be solved that likely impacts traffic; not seeing uptime is a minor issue we can easily fix and likely will be fixed once the main issue is resolved.

Sam