ASR9902 - problems with snmp polling and timeout

[RBordais]

The ether_ctrl_msg_client process would be blocked.

The commands below concerning this blocked process which would certainly generate these polling/timeout problems on snmp requests in my opinion. So, with the following, the ether_ctrl_msg client process would be blocked could it generate these logs?

#show process blocked location 0/0/CPU0
Jid Pid Tid ProcessName State TimeInState Blocked-on

204 5708 5867 ether_ctrl_msg_client Mutex 0481:39:57.0202 0x7fc7aced39d8 5863 ether_ctrl_msg_client ui_aipc_cb_handler:1320
204 5708 5863 ether_ctrl_msg_client Reply 0481:40:24.0572 16609 hvic_7

#show processes blocked
Jid Pid Tid ProcessName State TimeInState Blocked-on
255 7026 7228 lpts_fm Reply 0000:00:06.0348 6050 lpts_pa
354 5139 5483 sysdb_mc Reply 0000:00:00.0001 5121 gsp

#show process memory location 0/0/CPU0
JID Text(KB) Data(KB) Stack(KB) Dynamic(KB) Process
------ ---------- ---------- ---------- ----------- ------------------------------
...
126 72 794388 136 152180 hvic_7
...
373 180 408212 136 2290 psa

#show process placement ether_ctrl_msg_client
1 process found
NODE PID JID #THR TYPE PROGRAM
0/0/CPU0 5708 204 12 LC ether_ctrl_msg_client

#show process log location 0/0/CPU0 | i hvic_7
05/24 02:18:18.894 140358869931840 sysmgr_set_startup_path:Setting startup path:/opt/cisco/XR/packages/asr9k-base-64-4.0.0.0-r741/lc/instances/SUPERLUMINAL-LC/startup/hvic_7.startup for Processhvic_7
05/24 02:18:18.895 140358869931840 sysmgr_add_pcb_to_list: pcb hvic_7(1) (jid 126)
05/24 02:21:01.345 140357786400512 sysmgr_start_pcb: hvic_7(1) (jid 126) requested by vic_ls
05/24 02:21:01.345 140357786400512 sysmgr_spawn_enqueue: hvic_7(1) (jid 126) state 1 curlevel 0 respawn 0 capable 0 level 0 placement 0 forced stop 0
05/24 02:21:01.346 140357785347840 hvic_7(1) (jid 126) State Change, spawning process, (Spawn-Queued)==>(Spawning)
05/24 02:21:01.346 140357785347840 hvic_7(1) (jid 126) Starting with args '-i 0x7 -b 0'
05/24 02:21:01.350 140357785347840 hvic_7(1) (jid 126) State Change, process spawned, (Spawning)==>(Initializing)
05/24 02:21:01.350 140357786400512 sysmgr_start_pcb: hvic_7(1) (jid 126) spawn done
05/24 02:21:01.564 140358869931840 OK-sysmgr-ready: hvic_7 1 [126] Args: -i 0x7 -b 0 READY time 0.217
05/24 02:21:01.564 140358869931840 sysmgr_mark_pcb_ready: pcb hvic_7(1) jid=126 Handle EOI eoi=1
05/24 02:21:01.564 140358869931840 hvic_7(1) (jid 126) State Change, mark ready, (Initializing)==>(Run)
01/25 21:40:36.436 140357786400512 sysmgr_control_pcb: type PROC_RESTART state Run name hvic_7(1) jid 126 caller sysmgr_control
01/25 21:40:36.436 140357786400512 sysmgr_kill_proc: hvic_7 requested by sysmgr_control
01/25 21:40:36.436 140357786400512 sysmgr_kill_pcb: hvic_7(1) (jid 126) is being killed with SIGTERM
01/25 21:40:36.436 140357786400512 sysmgr_kill_pcb: hvic_7(1) (jid 126) start the sigterm timer
01/25 21:40:36.436 140357786400512 hvic_7(1) (jid 126) State Change, kill signal sent, (Run)==>(Terminating)
01/25 21:40:36.448 140358257784576 hvic_7 (jid 126 pid 6922) exited. Code=0, Signal=15.sysmgr_coid_death:0, restart_by_cmd:1
01/25 21:40:36.448 140358257784576 hvic_7(1) (jid 126) State Change, reset pcb, (Terminating)==>(Exited)
01/25 21:40:36.448 140358257784576 sysmgr_reset_pcb: hvic_7(1) (jid 126) Delete kill timer
01/25 21:40:36.448 140358257784576 sysmgr_respawn_now: dead_pcb = hvic_7, fail_count 0, eoi_received=1
01/25 21:40:36.448 140358257784576 sysmgr_respawn_now: try again for hvic_7 now fail_count= 0
01/25 21:40:36.448 140358257784576 sysmgr_spawn_proc_if_allowed: hvic_7(1) jid 126 placement 0
01/25 21:40:36.448 140358257784576 sysmgr_spawn_enqueue: hvic_7(1) (jid 126) state 1 curlevel 0 respawn 1 capable 0 level 0 placement 0 forced stop 0
01/25 21:40:36.448 140358257784576 sysmgr_proc_death: sysmgr_notify_proc_death for pcb=hvic_7[126]exit_status=1
01/25 21:40:36.450 140357785347840 hvic_7(1) (jid 126) State Change, spawning process, (Spawn-Queued)==>(Spawning)
01/25 21:40:36.450 140357785347840 hvic_7(1) (jid 126) Starting with args '-i 0x7 -b 0'
01/25 21:40:36.454 140357785347840 hvic_7(1) (jid 126) State Change, process spawned, (Spawning)==>(Initializing)
01/25 21:40:36.634 140357779896064 OK-sysmgr-ready: hvic_7 1 [126] Args: -i 0x7 -b 0 READY time 0.183
01/25 21:40:36.634 140357779896064 sysmgr_mark_pcb_ready: pcb hvic_7(1) jid=126 Handle EOI eoi=1
01/25 21:40:36.634 140357779896064 hvic_7(1) (jid 126) State Change, mark ready, (Initializing)==>(Run)

#show process log location 0/0/CPU0 | i psa
05/24 02:18:19.128 140358869931840 sysmgr_set_startup_path:Setting startup path:/opt/cisco/XR/packages/asr9k-os-supp-64-4.0.0.0-r741/lc/instances/SUPERLUMINAL-LC/startup/psa.startup for Processpsa
05/24 02:18:19.129 140358869931840 sysmgr_add_pcb_to_list: pcb psa(1) (jid 373)
05/24 02:18:19.157 140358258878208 sysmgr_spawn_enqueue: psa(1) (jid 373) state 1 curlevel 9000 respawn 0 capable 0 level 9000 placement 0 forced stop 0
05/24 02:18:19.245 140357785347840 psa(1) (jid 373) State Change, spawning process, (Spawn-Queued)==>(Spawning)
05/24 02:18:19.254 140357785347840 psa(1) (jid 373) State Change, process spawned, (Spawning)==>(Initializing)
05/24 02:18:28.050 140357778843392 OK-sysmgr-ready: psa 1 [373] Args: None READY time 8.805
05/24 02:18:28.050 140357778843392 sysmgr_mark_pcb_ready: pcb psa(1) jid=373 Handle EOI eoi=1
05/24 02:18:28.050 140357778843392 psa(1) (jid 373) State Change, mark ready, (Initializing)==>(Run)
05/24 02:18:28.050 140357778843392 sysmgr_mark_pcb_ready(psa 373): pgroup not ready for level 9000.

#show process log location 0/0/CPU0 | i ether_ctrl_msg_client
05/24 02:18:18.955 140358869931840 sysmgr_set_startup_path:Setting startup path:/opt/cisco/XR/packages/asr9k-base-64-4.0.0.0-r741/lc/startup/ether_ctrl_msg_client.startup for Processether_ctrl_msg_client
05/24 02:18:18.955 140358869931840 sysmgr_add_pcb_to_list: pcb ether_ctrl_msg_client(1) (jid 204)
05/24 02:20:09.433 140358258878208 sysmgr_spawn_enqueue: ether_ctrl_msg_client(1) (jid 204) state 1 curlevel 10000 respawn 0 capable 0 level 10000 placement 0 forced stop 0
05/24 02:20:09.789 140357785347840 ether_ctrl_msg_client(1) (jid 204) State Change, spawning process, (Spawn-Queued)==>(Spawning)
05/24 02:20:09.792 140357785347840 ether_ctrl_msg_client(1) (jid 204) State Change, process spawned, (Spawning)==>(Initializing)
05/24 02:20:14.679 140357778843392 OK-sysmgr-ready: ether_ctrl_msg_client 1 [204] Args: None READY time 4.885
05/24 02:20:14.679 140357778843392 sysmgr_mark_pcb_ready: pcb ether_ctrl_msg_client(1) jid=204 Handle EOI eoi=1
05/24 02:20:14.680 140357778843392 ether_ctrl_msg_client(1) (jid 204) State Change, mark ready, (Initializing)==>(Run)
05/24 02:20:14.680 140357778843392 sysmgr_mark_pcb_ready(ether_ctrl_msg_client 204): pgroup not ready for level 10000.

2#show process location 0/0/CPU0 | i 126
JID TID Stack pri state NAME rt_pri
126 16609 0K 20 Sleeping hvic 0
126 16610 0K 20 Sleeping lwm_service_thr 0
126 16611 0K 20 Sleeping qsm_service_thr 0
126 16612 0K 20 Sleeping hvic 0
126 16613 0K 20 Sleeping hvic_ls_dwdm_ms 0
126 16614 0K 20 Sleeping evm_signal_thre 0
126 16616 0K 20 Sleeping chkpt_evm 0
126 16618 0K 20 Sleeping hvic_pm_thread 0
126 16619 0K 20 Sleeping hvic_pm_thread 0
126 16620 0K 20 Sleeping telemetry_event 0
126 16621 0K 20 Sleeping async 0

#show process location 0/0/CPU0 | i 373
JID TID Stack pri state NAME rt_pri
373 4729 0K 20 Sleeping psa 0
373 5065 0K 20 Sleeping lwm_service_thr 0
373 5067 0K 20 Sleeping qsm_service_thr 0
373 5076 0K 20 Sleeping psa 0
373 5102 0K 20 Sleeping psa 0
373 5108 0K 20 Sleeping async 0
373 5109 0K 20 Sleeping evm_signal_thre 0
220 6373 0K 20 Sleeping request 0

#show process location 0/0/CPU0 | i 204
JID TID Stack pri state NAME rt_pri
204 5708 0K 20 Sleeping ether_ctrl_msg_ 0
204 5827 0K 20 Sleeping lwm_service_thr 0
204 5830 0K 20 Sleeping qsm_service_thr 0
204 5838 0K 20 Sleeping ether_ctrl_msg_ 0
204 5860 0K 20 Sleeping ether_ctrl_msg_ 0
204 5863 0K 20 Sleeping ether_ctrl_msg_ 0
204 5865 0K 20 Sleeping async 0
204 5866 0K 20 Sleeping evm_signal_thre 0
204 5867 0K 20 Sleeping pm_ui thread 0
204 5868 0K 20 Sleeping aipc-lrd_thread 0
204 6910 0K 20 Sleeping pm_ui thread 0
204 6911 0K 20 Sleeping pm_ui thread 0

#show process blocked location all
node: node0_0_CPU0
------------------------------------------
Jid Pid Tid ProcessName State TimeInState Blocked-on
204 5708 5867 ether_ctrl_msg_client Mutex 0509:31:47.0412 0x7fc7aced39d8 5863 ether_ctrl_msg_client ui_aipc_cb_handler:1320
204 5708 5863 ether_ctrl_msg_client Reply 0509:32:14.0782 16609 hvic_7

node: node0_RP0_CPU0
------------------------------------------
Jid Pid Tid ProcessName State TimeInState Blocked-on
0 8411 8411 sh_proc_ng_blocked Reply 0000:00:00.0264 5163 procfs_server
373 13438 13516 mibd_interface Reply 0000:01:18.0619 5139 sysdb_mc
255 7026 7228 lpts_fm Reply 0000:00:00.0040 6050 lpts_pa

node: node0_RP1_CPU0
------------------------------------------
Jid Pid Tid ProcessName State TimeInState Blocked-on
358 7317 7506 lpts_fm Reply 6432:54:59.0506 6050 node0_RP0_CPU0 PID:6050

#show processes placement ether_ctrl_msg_client
1 process found
NODE PID JID #THR TYPE PROGRAM
0/0/CPU0 5708 204 12 LC ether_ctrl_msg_client

#show process location 0/RP1/CPU0 | i 358
JID TID Stack pri state NAME rt_pri
358 7317 0K 20 Sleeping lpts_fm 0
358 7489 0K 20 Sleeping lwm_service_thr 0
358 7491 0K 20 Sleeping qsm_service_thr 0
358 7498 0K 20 Sleeping lpts_fm 0
358 7499 0K 20 Sleeping evm_signal_thre 0
358 7506 0K 20 Sleeping lpts_fm 0
358 7507 0K 20 Sleeping lpts_fm 0

#show processes dynamic | i 358
JID LAST STARTED STATE RE- PLACE- MANDA- MAINT- NAME(IID) ARGS
START MENT TORY MODE
-------------------------------------------------------------------------------
358 05/24/2022 13:30:23.707 Run 1 mibd_infra(1) -g infra

What could I do/execute on these ASRs in order to resolve this overflowing process/log issue?

Thank you very much and have a good day.