Re: c3850 Stack Switch: CPUmem is periodically rising/falling triggered by cpumemd and snmp_subagen...

Netmart · ‎04-30-2018

Hello,

I noticed the periodic appearance of the following log messages:

01:57:21: %CPUMEM-5-FALLING_THRESHOLD: 1 CPUMEMd[6274]: Threshold: :2, Total CPU Utilzation(Total/Intr): :1/0

01:57:36: %CPUMEM-5-RISING_THRESHOLD: 1 CPUMEMd[6274]: Threshold: : 2, Total CPU Utilzation(total/Intr) :2/0, Top 3 processes(Pid/Util) : 12424/0, 2/0, 3/0

01:57:51: %CPUMEM-5-FALLING_THRESHOLD: STANDBY:2 CPUMEMd[6260]: Threshold: :2, Total CPU Utilzation(Total/Intr): :2/0 (njo1p-rtr-core-2)

01:58:06: %CPUMEM-5-RISING_THRESHOLD: STANDBY:2 CPUMEMd[6260]: Threshold: : 2, Total CPU Utilzation(total/Intr) :4/0, Top 3 processes(Pid/Util) : 5711/1, 12861/0, 3/0 (c3850-2)

c3850#sh version | i INSTALL
* 1 56 WS-C3850-48T 03.06.08E cat3k_caa-universalk9 INSTALL
2 56 WS-C3850-48T 03.06.08E cat3k_caa-universalk9 INSTALL

As workaround, I adjusted the "rising percentage" of the cpu threshold from 5 to 20%.

However, I was wondering, if there is anything to be worried about or can a certain spike in CPU usage be considered as "normal".

process cpu threshold type total rising 20 interval 5 switch 1

process cpu threshold type total rising 20 interval 5 switch 2

Please advise.

Thanks.

===========================================================================

c3850# sh processes cpu switch 2 det process cpumemd sorted
Core 0: CPU utilization for five seconds: 12%; one minute: 7%; five minutes: 6%
Core 1: CPU utilization for five seconds: 0%; one minute: 0%; five minutes: 0%
Core 2: CPU utilization for five seconds: 2%; one minute: 1%; five minutes: 1%
Core 3: CPU utilization for five seconds: 0%; one minute: 0%; five minutes: 1%
PID T C TID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
(%) (%) (%)
6260 L 15640 26404 592 0.05 0.05 0.05 0 cpumemd
6260 L 0 6260 15460 9423 0 0.05 0.05 0.05 0 cpumemd
6260 L 0 7185 180 9435 0 0.00 0.00 0.00 0 cpumemd
6260 L 0 7191 0 2107 0 0.00 0.00 0.00 0 cpumemd
6260 L 2 7196 10 1540 0 0.00 0.00 0.00 0 CMI default xdm
6260 L 0 7932 0 1303 0 0.00 0.00 0.00 0 eicore_bdn
6260 L 1 7933 0 1303 0 0.00 0.00 0.00 0 eicore_bdn
6260 L 3 7934 0 1303 0 0.00 0.00 0.00 0 eicore_bdn

c3850# sh processes cpu switch 2 det process snmp_subagent
Core 0: CPU utilization for five seconds: 8%; one minute: 6%; five minutes: 6%
Core 1: CPU utilization for five seconds: 1%; one minute: 0%; five minutes: 0%
Core 2: CPU utilization for five seconds: 1%; one minute: 0%; five minutes: 1%
Core 3: CPU utilization for five seconds: 0%; one minute: 1%; five minutes: 1%
PID T C TID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
(%) (%) (%)
6274 L 8470 110111 76 0.00 0.00 0.00 0 snmp_subagent
6274 L 1 6274 7040 1734 0 0.00 0.00 0.00 0 snmp_subagent
6274 L 0 7589 250 9393 0 0.00 0.00 0.00 0 snmp_subagent
6274 L 0 7600 0 2103 0 0.00 0.00 0.00 0 snmp_subagent
6274 L 0 7624 10 1310 0 0.00 0.00 0.00 0 snmp dcm sa thr
6274 L 1 7635 490 3298 0 0.00 0.00 0.00 0 CMI default xdm
6274 L 0 7779 40 1551 0 0.00 0.00 0.00 0 snmp_subagent
6274 L 2 7785 0 1325 0 0.00 0.00 0.00 0 HALib Dispatch
6274 L 2 7786 0 1325 0 0.00 0.00 0.00 0 HALib Dispatch
6274 L 2 7787 0 1325 0 0.00 0.00 0.00 0 HALib Dispatch
6274 L 3 7788 0 1681 0 0.00 0.00 0.00 0 RF Slave Main T
6274 L 1 7936 0 1325 0 0.00 0.00 0.00 0 eicore_bdn
6274 L 3 7937 0 1325 0 0.00 0.00 0.00 0 eicore_bdn
6274 L 1 7938 0 1325 0 0.00 0.00 0.00 0 eicore_bdn
6274 L 0 9032 530 1627 0 0.00 0.00 0.00 0 eicore_ipc
6274 L 3 9727 60 1868 0 0.00 0.00 0.00 0 eicore_ipc
6274 L 0 9734 10 1741 0 0.00 0.00 0.00 0 snmp_subagent
6274 L 3 9735 0 1734 0 0.00 0.00 0.00 0 SNMP SA CACHE
6274 L 1 9736 0 1734 0 0.00 0.00 0.00 0 SNMP SA CACHE
6274 L 3 9737 0 1734 0 0.00 0.00 0.00 0 SNMP SA CACHE
6274 L 1 9738 0 1734 0 0.00 0.00 0.00 0 xos_thread
6274 L 2 13029 10 17150 0 0.00 0.00 0.00 0 COMET_SUB_AGENT
6274 L 2 13043 0 17247 0 0.00 0.00 0.00 0 COMET_SUB_AGENT
6274 L 0 13056 30 17294 0 0.00 0.00 0.00 0 COMET_SUB_AGENT
6274 L 3 13059 10 17284 0 0.00 0.00 0.00 0 COMET_SUB_AGENT

Leo Laohoo · ‎04-30-2018

Post the complete output to the command "sh proc cpu sort | ex 0.00".

Netmart · ‎05-01-2018

Please see output below:

c3850# sh proc cpu sort | ex 0.00
Core 0: CPU utilization for five seconds: 4%; one minute: 6%; five minutes: 6%
Core 1: CPU utilization for five seconds: 2%; one minute: 1%; five minutes: 1%
Core 2: CPU utilization for five seconds: 1%; one minute: 1%; five minutes: 1%
Core 3: CPU utilization for five seconds: 0%; one minute: 0%; five minutes: 1%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
5721 1018300 7423331 137 0.78 1.03 1.05 1088 fed
10206 1082620 14504658 74 0.68 0.70 0.62 0 iosd
5723 86620 2394133 36 0.10 0.07 0.06 0 stack-mgr
5722 88290 5503720 16 0.05 0.06 0.07 0 platform_mgr
6268 18320 101722 180 0.05 0.02 0.05 0 mem_mgmt
6274 88530 106331 832 0.05 0.05 0.05 0 cpumemd
10202 272740 20422059 13 0.05 0.14 0.15 0 wcm

Leo Laohoo · ‎05-01-2018

Uhhhh ... I don't see any signs of high CPU.

Netmart · ‎05-01-2018

Please see attachment, catching the spike in cpu utilization.
It seems that core 3 is the one indicating the high cpu utilization.

I already upgrade c3850 to recommended version:

* 1 56 WS-C3850-48T 03.06.08E cat3k_caa-universalk9 INSTALL
2 56 WS-C3850-48T 03.06.08E cat3k_caa-universalk9 INSTALL

c3850 Stack Switch: CPUmem is periodically rising/falling triggered by cpumemd and snmp_subagent