cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
434
Views
0
Helpful
2
Replies
satish.txt1
Beginner

Cisco nexus 9396 rebooted with reason xbar_client

Today my one of cisco nexus 9396PX switch got rebooted and this is what i found, based on following logs what could be wrong? We don't have Cisco support for this switch, trying to see if i can get help from community otherwise last option is upgrade software.

Software
  BIOS: version 07.41
  NXOS: version 7.0(3)I4(7)
  BIOS compile time:  10/12/2015
  NXOS image file is: bootflash:///nxos.7.0.3.I4.7.bin
  NXOS compile time:  6/28/2017 14:00:00 [06/28/2017 16:53:29]


Hardware
  cisco Nexus9000 C9396PX Chassis
  Intel(R) Core(TM) i3- CPU @ 2.50GHz with 16401396 kB of memory.
  Processor Board ID SAL2006Y9CQ

reset reason

N9K# show system reset-reason
----- reset reason for module 1 (from Supervisor in slot 1) ---
1) At 310613 usecs after Thu May  9 18:06:25 2019
    Reason: Reset Requested due to Fatal Module Error
    Service: System manager
    Version: 7.0(3)I4(7)

logs-1

  N9K# show cores
    VDC  Module  Instance  Process-name     PID       Date(Year-Month-Day Time)
    ---  ------  --------  ---------------  --------  -------------------------
    1    1       1         xbar_client      15716     2019-05-09 18:01:23
    1    1       1         xbar_client      25312     2019-05-09 18:06:18

logs-2

 N9K# show module internal exceptionlog module 1
    ********* Exception info for module 1 ********

    exception information --- exception instance 1 ----
    Module Slot Number: 1
    Device Id         : 134
    Device Name       : System Manager
    Device Errorcode  : 0x00000049
    Device ID         : 00 (0x00)
    Device Instance   : 00 (0x00)
    Dev Type (HW/SW)  : 00 (0x00)
    ErrNum (devInfo)  : 73 (0x49)
    System Errorcode  : 0x401e008a Service on linecard had a hap-reset
    Error Type        : FATAL error
    PhyPortLayer      : 0x0
    Port(s) Affected  :
    Error Description : xbar_client hap reset
    DSAP              : 0 (0x0)
    UUID              : 1 (0x1)
    Time              : Thu May  9 18:06:25 2019
                        (Ticks: 5CD4B271 jiffies)

    exception information --- exception instance 2 ----
    Module Slot Number: 1
    Device Id         : 241
    Device Name       : BCM5685X
    Device Errorcode  : 0xcf130200
    Device ID         : 241 (0xf1)
    Device Instance   : 48 (0x30)
    Dev Type (HW/SW)  : 02 (0x02)
    ErrNum (devInfo)  : 00 (0x00)
    System Errorcode  : 0x40390047 internal link between forwarding ASICs down
    Error Type        : Minor error
    PhyPortLayer      : Ethernet
    Port(s) Affected  : Ethernet2/1
    DSAP              : 0 (0x0)
    UUID              : 0 (0x0)
    Time              : Thu May  9 17:59:52 2019
                        (Ticks: 5CD4B0E8 jiffies)

logs-3

N9K# show system internal xbar sw
======= Global Information =========
         db_restored = 0
         xbm_iam_almost_active = 0
         modules_lock_bmap = 0
         global_lock = 0
         global_lock gwrap = (nil)
         chassis type = 34
         fabric mode  = 1
         fabric speed mode  = 42g
         fabric speed sequence in progress  = False
         xbar is fully connected
         xbar libdrv_xlink_is_t2_speed_40g() is : FALSE
======= Module Information =========
Module in module 1 (present = 1)

rid 0x2000000 type 0 state 0 sub_type 0 node_id 0x102
sw_card_id 0x12a lc_node_addr 0x102 feature_bits 0x0
xlink_index 0x0
locked_gwrap: (nil)
timer: hdl 0x10763d5c rid 0x2000000 ev_id 0xffff timer_id 0x0 tim_type 0x0


Module in module 2 (present = 0)

Module in module 3 (present = 0)

Module in module 4 (present = 0)

logs-4

N9K# show processes log
VDC Process          PID     Normal-exit  Stack  Core   Log-create-time
--- ---------------  ------  -----------  -----  -----  ---------------
  1 xbar_client      15716             N      N      N Thu May  9 18:01:25 2019
  1 xbar_client      24518             N      N      N Thu May  9 18:03:03 2019
  1 xbar_client      24933             N      N      N Thu May  9 18:04:41 2019
  1 xbar_client      25312             N      N      N Thu May  9 18:06:19 2019

logs-5

N9K# show processes log pid 25312
Service: xbar_client
Description: Xbar Client
Executable: /lc/isan/bin/xbar_client

Started at Thu May  9 18:04:41 2019 (663071 us)
Stopped at Thu May  9 18:06:19 2019 (474464 us)
Uptime: 1 minutes 38 seconds

Start type: SRV_OPTION_RESTART_STATEFUL (24)
Death reason: SYSMGR_DEATH_REASON_FAILURE_HEARTBEAT (9)
Last heartbeat 95.72 secs ago
System image name:
System image version: 7.0(3)I4(7)

PID: 25375
Exit code: signal 6 (no core)

Threads: 25312

CWD: /var/sysmgr/work

RLIMIT_AS:      4294967295

Virtual Memory:

    CODE      100CB000 - 101B3CA8
    DATA      101B4000 - 101B57C8
    BRK       115DD000 - 11771000
    STACK     FF9C7AC0
    TOTAL     618916 KB

Memory Map: 100CB000 xbar_clien 101B4000 xbar_clien D30DE000 mts E30DE000 libmtsdlutils.s E30DF000 libmtsdlutils.s E30E5000 libstathash.s E30E7000 libstathash.s E30E8000 libqosmgr.s
 E30EE000 libqosmgr.s E30EF000 libfm.s E30F7000 libfm.s E30FA000 liburiparse.s E30FD000 liburiparse.s E317A000 libvdc_capability.s E317D000 libvdc_capability.s E317E000 libvdc_mgr_c
mn.s E3181000 libvdc_mgr_cmn.s E3183000 libz.so.1.2. E3198000 libz.so.1.2. E3199000 libifmgr.s E31FF000 libifmgr.s E3248000 libuspace_utils.s E324A000 libuspace_utils.s E324B000 lib
pcm_sdb.s E3272000 libpcm_sdb.s E3278000 libltlmap.s E3284000 libltlmap.s E3287000 libsdwraphist.s E3293000 libsdwraphist.s E3295000 libcmd.s E32BE000 libcmd.s E32C1000 libdleft.s E
32C5000 libdleft.s E32C7000 liburi_map.s E32C9000 liburi_map.s E32CA000 libvsh.s E32E8000 libvsh.s E32EA000 libipfibutils.s E32EE000 libipfibutils.s EAAB5000 libbios.s EAADC000 libb
ios.s EAADE000 libavl.s EAAE1000 libavl.s EAAE2000 libutils_cli_callback.s EAAE3000 libutils_cli_callback.s EAAE4000 libexec.s EAAE7000 libexec.s EAAE8000 libvdb.s EAAEA000 libvdb.s
 EAAF4000 libdll_obj.s EAAF7000 libdll_obj.s EAAF8000 libsysstr.s EAAFA000 libsysstr.s EAAFB000 libsysmgrcmn.s EAB07000 libsysmgrcmn.s EAB08000 libif_index.s EAB47000 libif_index.s
EAB9D000 librt-2.15.s EABA4000 librt-2.15.s EABA5000 librt-2.15.s EABA6000 libbmp.s EAD81000 libbmp.s EAD84000 libeventseq.s EAD98000 libeventseq.s EAD9E000 libsystem_vdc.s EAD9F000
--More--

logs-6

N9K# show system internal xbar event-history lock

1) Event:E_FU_UNLOCK, length:32, at 444695 usecs after Thu May  9 18:09:09 2019
    Status: 0x0
    Gwrap: 0x108cd04c  Cat: 0x0
      Opc:MTS_OPC_LC_INSERTED(1081)
      Msg id: 0X00006B8E
    Lock type: 1
    RID Size: 4
        Val :  0x 2000000


2) Event:E_FU_LOCK, length:32, at 444635 usecs after Thu May  9 18:09:09 2019
    Status: 0x0
    Gwrap: 0x108cd04c  Cat: 0x0
      Opc:MTS_OPC_LC_INSERTED(1081)
      Msg id: 0X00006B8E
    Lock type: 1
    RID Size: 4
        Val :  0x 2000000


3) Event:E_FU_UNLOCK, length:32, at 139624 usecs after Thu May  9 18:09:06 2019
    Status: 0x0
    Gwrap: 0x108cd04c  Cat: 0x0
      Opc:MTS_OPC_LC_INSERTED(1081)
      Msg id: 0X00004EB4
    Lock type: 1
    RID Size: 4
        Val :  0x 2000000


4) Event:E_FU_LOCK, length:32, at 136670 usecs after Thu May  9 18:09:06 2019
    Status: 0x0
    Gwrap: 0x108cd04c  Cat: 0x0
      Opc:MTS_OPC_LC_INSERTED(1081)
      Msg id: 0X00004EB4
    Lock type: 1
    RID Size: 4
        Val :  0x 2000000
2 REPLIES 2
balaji.bandi
VIP Master

May be you been hit with bug, Upgrade and test is advised here.

 

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvi80392/?rfs=iqvred

 

Known Fixed Releases:
(3)

BB

***** Rate All Helpful Responses *****

How to Ask The Community for Help

Thanks you, i read that bug but it was about "bcm_usd" so i thought may be not my case. but yes worth upgrade