Re: WS-C3850-48T CPU utilization for five seconds: 99%

CSCO11227946 · ‎04-02-2021

Hi all, i need help with sorting out CPU utilization.

We received this alert from monitoring tool > CPU (Switch 2) has exceeded threshold: 90% currently 99%

Upon checking on switch, show process cpu / sho process cpu history command returns normal values:

sw-1#sho processes cpu
CPU utilization for five seconds: 7%/3%; one minute: 6%; five minutes: 6%

but upon checking show process cpu platform this is output:

sw-1#sho processes cpu platform sorted
CPU utilization for five seconds: 99%, one minute: 99%, five minutes: 99%
Core 0: CPU utilization for five seconds: 100%, one minute: 99%, five minutes: 99%
Core 1: CPU utilization for five seconds: 99%, one minute: 99%, five minutes: 99%
Core 2: CPU utilization for five seconds: 100%, one minute: 100%, five minutes: 99%
Core 3: CPU utilization for five seconds: 99%, one minute: 99%, five minutes: 99%
Pid PPid 5Sec 1Min 5Min Status Size Name
--------------------------------------------------------------------------------
5103 4247 1% 1% 1% S 749555712 repm
32546 1 0% 0% 0% S 8921088 rotee
31924 15517 0% 0% 0% S 5992448 pman.sh
29765 2 0% 0% 0% S 0 kworker/3:1
29756 29755 0% 0% 0% Z 0 in.telnetd
29755 29587 0% 0% 0% R 14508032 in.telnetd
29587 10700 0% 0% 0% S 6201344 in.telnetd.sh

How can I locate what process is causing this behavior, or is it some bug?

Switch is in stack and model is:

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
1 56 WS-C3850-48T 16.6.4a CAT3K_CAA-UNIVERSALK9 INSTALL
* 2 56 WS-C3850-48T 16.6.4a CAT3K_CAA-UNIVERSALK9 INSTALL

this error is from log:
*Apr 2 02:56:07: %PLATFORM-4-ELEMENT_WARNING: Switch 2 R0/0: smand: 2/RP/0: 5-Minute Load Average value 5.41 exceeds warning level 5.00

sw-1#sho platform software status control-processor brief
Load Average
Slot Status 1-Min 5-Min 15-Min
1-RP0 Healthy 0.19 0.32 0.31
2-RP0 Warning 5.68 5.53 5.50

Memory (kB)
Slot Status Total Used (Pct) Free (Pct) Committed (Pct)
1-RP0 Healthy 3983060 2432608 (61%) 1550452 (39%) 3368452 (85%)
2-RP0 Healthy 3983060 3192868 (80%) 790192 (20%) 3978136 (100%)

CPU Utilization
Slot CPU User System Nice Idle IRQ SIRQ IOwait
1-RP0 0 1.60 0.80 0.00 97.50 0.00 0.10 0.00
1 2.20 0.70 0.00 97.10 0.00 0.00 0.00
2 3.39 0.99 0.00 95.50 0.00 0.09 0.00
3 3.70 0.40 0.00 95.90 0.00 0.00 0.00
2-RP0 0 19.64 80.25 0.00 0.00 0.00 0.09 0.00
1 23.92 76.07 0.00 0.00 0.00 0.00 0.00
2 21.23 78.76 0.00 0.00 0.00 0.00 0.00
3 20.73 79.16 0.00 0.09 0.00 0.00 0.00

sw-1#sho processes cpu platform history location switch active r0
5 seconds ago, CPU utilization: 99%
10 seconds ago, CPU utilization: 99%
15 seconds ago, CPU utilization: 100%
20 seconds ago, CPU utilization: 99%
25 seconds ago, CPU utilization: 99%
....
375 seconds ago, CPU utilization: 100%
380 seconds ago, CPU utilization: 99%
385 seconds ago, CPU utilization: 99%

Any help is appreciated.

Reza Sharifi · ‎04-02-2021

Hi,

What is the output of "sh process cpu sport | exc 0.00"

HTH

CSCO11227946 · ‎04-04-2021

Here is the output>
sw-1# sho processes cpu sorted | exclude 0.00
CPU utilization for five seconds: 9%/3%; one minute: 7%; five minutes: 7%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
210 656577057 2331169970 0 1.35% 1.14% 1.12% 0 Spanning Tree
275 54117647 2802715 19309 1.19% 0.14% 0.07% 0 Per-minute Jobs
406 54407902 90052295 604 1.03% 0.37% 0.15% 0 SNMP ENGINE
211 111645112 733376500 152 0.23% 0.17% 0.16% 0 UDLD
74 147545001 492178068 299 0.23% 0.26% 0.24% 0 IOSD ipc task
185 77717138 1144743121 67 0.15% 0.10% 0.09% 0 VRRS Main thread
199 99197159 2270399209 0 0.15% 0.13% 0.13% 0 IP ARP Retry Age
115 77613820 789243832 98 0.15% 0.18% 0.14% 0 IOSXE-RP Punt Se
485 395 589 670 0.07% 0.15% 0.09% 2 SSH Process
138 28283245 39098619 723 0.07% 0.05% 0.05% 0 PLFM-MGR IPC pro
200 53791357 571723921 94 0.07% 0.07% 0.07% 0 IP Input
291 76884922 1144742340 67 0.07% 0.10% 0.09% 0 MMA DB TIMER
110 26012481 560362125 46 0.07% 0.05% 0.04% 0 100ms check
67 98883250 381753746 259 0.07% 0.16% 0.15% 0 Net Background
37 44058260 83218079 529 0.07% 0.07% 0.07% 0 ARP Input
322 77812208 1144740598 67 0.07% 0.10% 0.09% 0 MMA DP TIMER
139 35631029 18337576 1943 0.07% 0.05% 0.06% 0 FEP background p

Leo Laohoo · ‎04-03-2021

@CSCO11227946 wrote:

2-RP0 Healthy 3983060 3192868 (80%) 790192 (20%) 3978136 (100%)

This is not good.

Can you post the PAGE 1 of the following output:

sh process memory platform sort location switch 2 r0

Do you have Dot1X enabled?

CSCO11227946 · ‎04-04-2021

Hello Leo,

dot1x is not enabled, here is requested output:

sw-1#sho processes memory platform sorted location switch 2 r0
System memory: 3983060K total, 3195256K used, 787804K free,
Lowest: 787804K
   Pid    Text      Data   Stack   Dynamic       RSS     Total              Name
--------------------------------------------------------------------------------
 12330  177183   1436060     136       120   1436060   2816108   linux_iosd-imag
 18619     116    349280     136     71656    349280   2552040    fed main event
 20537     304    180700     136      2964    180700   1563796           sif_mgr
 19448    1011    187212     136      6316    187212   1294624      platform_mgr
 10767     287    187372     136      7688    187372    971728         cli_agent
 14215     649    182632     136     27048    182632    872228             smand
 11265     170    191244     136      3484    191244    863712               dbm
   987    8817    161844     136      9016    161844    825832     fman_fp_image
 21627    1538     27084     136      1204     27084    794528             nginx
 20475    1538    118168     136      1204    118168    788472             nginx
  5674     127    175248       0       268    175248    772416               smd
 11542    8482    160396     136      2680    160396    732288           fman_rp
  5103     430    130648     136      1760    130648    731988              repm
 15120     250    128956     136      8692    128956    714772               tms
 13959      38    120864     136      1328    120864    710632         bt_logger
 15065     519    125648     136      2460    125648    705944              hman
 13428      40    121256     136      1152    121256    702052               psd
 18677     200    113784     136       400    113784    701524           nif_mgr
 21046     418    122408     136      1792    122408    701500         stack_mgr
 10873     114    117524     136      1224    117524    701028               cmm
 16502     147    120860     136      1912    120860    700964              lman
  4534      45    118160     136      2080    118160    698752             plogd
 16064      73    114588     136       432    114588    692856            keyman
  3499     604      3788     132       132      3788    212372          libvirtd
  3477     754      1648     132       132      1648     17172          virtlogd
 13658       7      1804     136       148      1804     16592   auto_upgrade_se
 29755      56      2044     132       172      2044     14168        in.telnetd
 24290      56      2044     132       172      2044     14168        in.telnetd
 15758      56      2052     132       172      2052     14168        in.telnetd
 11631      56      2044     132       172      2044     14168        in.telnetd
  7142      56      2044     132       172      2044     14168        in.telnetd
  6054      56      2044     132       172      2044     14168        in.telnetd
  4474      56      2044     132       172      2044     14168        in.telnetd
    88     314      5212     132       132      5212     12432   systemd-journal
 16835     974      7520     136      5784      7520     10128            ncd.sh
 15507     974      7448     136      5784      7448     10128     issu_stack.sh
 27858     974      6460     136      5652      6460      9996     issu_stack.sh
 27850     974      6360     136      5652      6360      9996     issu_stack.sh
 13178     974      7224     136      5524      7224      9868   auto_upgrade_cl
 19027     974      6176     136      4516      6176      8860       periodic.sh
 32546       7      1524     136       148      1524      8712             rotee
 20643       7      1524     136       148      1524      8712             rotee
 20092       7      1384     136       148      1384      8712             rotee
 19631       7      1524     136       148      1524      8712             rotee
 18849       7      1524     136       148      1524      8712             rotee
 18498       7      1420     136       148      1420      8712             rotee
 18261       7      1460     136       148      1460      8712             rotee
 17691       7      1524     136       148      1524      8712             rotee
 17223       7      1536     136       148      1536      8712             rotee
 16640       7      1360     136       148      1360      8712             rotee
 16231       7      1356     136       148      1356      8712             rotee
 15713       7      1524     136       148      1524      8712             rotee
 14945       7      1352     136       148      1352      8712             rotee
 14781       7      1560     136       148      1560      8712             rotee
 14700       7      1452     136       148      1452      8712             rotee
 14437       7      1524     136       148      1524      8712             rotee
 13863       7      1560     136       148      1560      8712             rotee
 13677       7      1448     136       148      1448      8712             rotee
 11936       7      1536     136       148      1536      8712             rotee
 11202       7      1560     136       148      1560      8712             rotee
 10780       7      1560     136       148      1560      8712             rotee
 10309       7      1560     136       148      1560      8712             rotee
  9953       7      1560     136       148      1560      8712             rotee
  4589       7      1524     136       148      1524      8712             rotee
 16209       7      1376     132       148      1376      8708             rotee
 12163       7      1376     132       148      1376      8708             rotee
 12033       7      1348     132       148      1348      8708             rotee
 20060       7      1396     136       148      1396      8592             rotee
 19659       7      1348     136       148      1348      8592             rotee
 18541       7      1348     136       148      1348      8592             rotee
 17848       7      1348     136       148      1348      8592             rotee
 15264       7      1348     136       148      1348      8592             rotee
 14192       7      1364     136       148      1364      8592             rotee
 13303       7      1524     136       148      1524      8592             rotee
 13073       7      1560     136       148      1560      8592             rotee
 12934       7      1352     136       148      1352      8592             rotee
 12621       7      1360     136       148      1360      8592             rotee
  5256       7      1524     136       148      1524      8592             rotee
  4105       7      1524     136       148      1524      8592             rotee
 24676       7      1384     132       148      1384      8588             rotee
 11376       7      1348     132       148      1348      8588             rotee
  7546       7      1384     132       148      1384      8588             rotee
  4344       7      1348     132       148      1348      8588             rotee
  3996       7      1376     132       148      1376      8588             rotee
  3988       7      1376     132       148      1376      8588             rotee
  3820       7      1348     132       148      1348      8588             rotee
     1    1400      4472     132      1436      4472      8016           systemd
 10702     974      5180     132      3472      5180      7812   rollback_timer.
 12045     974      4424     132      2676      4424      7016            pvp.sh
  3434     974      4116     132      2460      4116      6800      reflector.sh
 11922     974      4052     132      2432      4052      6772           psvp.sh
  3468     974      4088     132      2432      4088      6772       droputil.sh
 14141     974      4064     136      2404      4064      6748   btrace_rotate.s
 17725     974      4060     136      2364      4060      6708   btrace_rotate.s
 15517     974      4000     132      2288      4000      6628            pvp.sh
  3483      49      1112     132       132      1112      6180           rpcbind
 24460     974      3320     132      1728      3320      6068       bexecute.sh
  7312     974      3320     132      1728      3320      6068       bexecute.sh
 29587     974      3380     132      1716      3380      6056     in.telnetd.sh
 24291     974      3376     132      1716      3376      6056         brelay.sh
 24124     974      3380     132      1716      3380      6056     in.telnetd.sh
 15589     974      3380     132      1716      3380      6056     in.telnetd.sh
 11464     974      3380     132      1716      3380      6056     in.telnetd.sh
  7143     974      3376     132      1716      3376      6056         brelay.sh
  6976     974      3380     132      1716      3380      6056     in.telnetd.sh
  5888     974      3380     132      1716      3380      6056     in.telnetd.sh
  4305     974      3380     132      1716      3380      6056     in.telnetd.sh
 19511     974      3364     136      1648      3364      5992           pman.sh
 14866     974      3348     136      1648      3348      5992           pman.sh
 17842     974      3364     136      1644      3364      5988           pman.sh
 17114     974      3348     136      1644      3348      5988           pman.sh
 14548     974      3364     136      1644      3364      5988           pman.sh
 15593     974      3364     136      1640      3364      5984           pman.sh
 20131     974      3352     136      1636      3352      5980           pman.sh
 19830     974      3352     136      1636      3352      5980           pman.sh
 19241     974      3352     136      1636      3352      5980           pman.sh
 14526     974      3352     136      1636      3352      5980           pman.sh
 13599     974      3348     136      1636      3348      5980           pman.sh
 13138     974      3368     136      1636      3368      5980           pman.sh
 12786     974      3352     136      1636      3352      5980           pman.sh
 12516     974      3352     136      1636      3352      5980           pman.sh
 12336     974      3364     136      1636      3364      5980           pman.sh
 10370     974      3348     136      1636      3348      5980           pman.sh
  9809     974      3348     136      1636      3348      5980           pman.sh
  4926     974      3344     136      1636      3344      5980           pman.sh
  4247     974      3352     136      1636      3352      5980           pman.sh
  3835     974      3348     136      1636      3348      5980           pman.sh
 18333     974      3352     136      1632      3352      5976           pman.sh
 15196     974      3348     136      1632      3348      5976           pman.sh
 13990     974      3348     136      1632      3348      5976           pman.sh

Leo Laohoo · ‎04-04-2021

@CSCO11227946 wrote:

sho platform software status control-processor brief

Is this output still showing Switch 2 memory as >80%?

CSCO11227946 · ‎04-04-2021

Here is current state:

sw-1#sho platform software status control-processor brief
Load Average
 Slot  Status  1-Min  5-Min 15-Min
1-RP0 Healthy   0.04   0.12   0.13
2-RP0 Warning   5.46   5.55   5.52

Memory (kB)
 Slot  Status    Total     Used (Pct)     Free (Pct) Committed (Pct)
1-RP0 Healthy  3983060  2433696 (61%)  1549364 (39%)   3367616 (85%)
2-RP0 Healthy  3983060  3195780 (80%)   787280 (20%)   3981304 (100%)

CPU Utilization
 Slot  CPU   User System   Nice   Idle    IRQ   SIRQ IOwait
1-RP0    0   1.90   1.10   0.00  97.00   0.00   0.00   0.00
         1   3.60   0.40   0.00  96.00   0.00   0.00   0.00
         2   2.60   0.60   0.00  96.80   0.00   0.00   0.00
         3   2.69   0.79   0.00  96.50   0.00   0.00   0.00
2-RP0    0  20.02  79.87   0.00   0.00   0.00   0.10   0.00
         1  20.75  79.14   0.00   0.09   0.00   0.00   0.00
         2  19.70  80.30   0.00   0.00   0.00   0.00   0.00
         3  25.97  74.02   0.00   0.00   0.00   0.00   0.00

Leo Laohoo · ‎04-04-2021

What is the uptime of this stack?

Could you consider upgrading to the latest 16.6.X train like, say, 16.6.9?

CSCO11227946 · ‎04-05-2021

Hi,

sw-1 uptime is 2 years, 17 weeks, 17 minutes
Uptime for this control processor is 2 years, 17 weeks, 25 minutes

If it is an IOS bug than it will be upgraded. But just need to know if it is some bug or not?