It maybe hit this bug, please refer to workaround to fix it.
Memory leak of the statsAG process on UCS Fabric Interconnects
CSCuw36128
Symptom:
- No statsAG core/restart
- Major Fault "F0885: Fabric Interconnect B inventory is not complete card-inventory,eth-pc-inventory,eth-port-inventory,fc-pc-inventory,fc-port-inventory,mgmt-port-inventory,remote-eth-port-inventory,switch-fru" seen. (It could be for either A or B side, or even both)
- svc_sam_statsAG logfiles will have these (and more) messages:
[MAJOR][0x7544eb90][Jul 16 07:21:23.806][app_sam_statsAG:pollF] Error getting switch FRU inventory; details: SC_Send_tlv(3640) cmd_ucsm_req_send_recv() failed(-1)
[MAJOR][0x7544eb90][Jul 16 07:21:53.809][app_sam_statsAG:getSw] Error getting Line-Card inventory; details: Internal Error: SC_Execute_show_command() returned -1
[MAJOR][0x7544eb90][Jul 16 07:21:53.811][app_sam_statsAG:getPh] Error getting physical Ethernet port inventory; details: Error in pm_get_if_index_listing(), errno: 16
[MAJOR][0x7544eb90][Jul 16 07:21:53.811][app_sam_statsAG:getRe] Error getting remote physical Ethernet port inventory; details: Error in pm_get_if_index_listing(), errno: 16
[MAJOR][0x7544eb90][Jul 16 07:21:53.811][app_sam_statsAG:getPh] Error getting physical FC port inventory; details: Error in pm_get_if_index_listing(), errno: 16
- sw_techsupport file will show the following output for the command "show logging log":
%KERN-2-SYSTEM_MSG: [12960970.669225] [sap 2896][pid 5828][comm:svc_sam_statsAG] WARNING: possible memory leak is detected on non-pers queue (len=191,bytes=22046904) - kernel
-----AND/OR-----
%KERN-2-SYSTEM_MSG: [23213620.524664] mts_do_msg_input() failing since no space available in 2779 (src_sap = 2779, opc = 7679) - kernel
Conditions:
Normal Cisco UCS Manager operation
Workaround:
- Will need to restart UCSM processes to recover statsAG mts buffer space.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Rps-Cheers | If it solves your problem, please mark as answer. Thanks !