cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements
Join Customer Connection to register!
531
Views
5
Helpful
13
Replies
Highlighted
Beginner

Catalyst 3650 high cpu utilization

Hello, Cisco Community!

We've encountered high CPU utilization issue on our 2x3650 stack.

Processes eating CPU are: ARP Input, SISF Switcher Th, SISF Main Thread
Issue is happening only on working hours, so likely that connected to user activity.

We tried disabling ip dhcp snooping on user VLANs and adding deny all rule for ipv6 but nothing helps.

 

Switch firmware is 16.6.6, haven't seen any related bugs for this version.

 

Do you have any ideas what else to check?

Thanks a lot!

13 REPLIES 13
Highlighted
VIP Expert

Highlighted

Hello,

thanks for pointing to that, I'll take a look. 

 

Thanks

Highlighted
Hall of Fame Community Legend


@KirillKO wrote:

Switch firmware is 16.6.6, haven't seen any related bugs for this version.


There are, a minimum of, seven bugs attributed for "SISF Main Thread".  

Can you post the complete output to the following commands: 

sh platform software status con brief
sh version
show processes cpu platform sorted location switch <1 & 2> r0
Highlighted

Hello, thanks for the prompt answer!

See comands output below, however output might be irrelevant cause of non-working hours as I described earlier

#sh platform software status con brief
Load Average
 Slot  Status  1-Min  5-Min 15-Min
1-RP0 Healthy   1.42   1.67   1.63
2-RP0 Healthy   0.24   0.21   0.23

Memory (kB)
 Slot  Status    Total     Used (Pct)     Free (Pct) Committed (Pct)
1-RP0 Healthy  3983088  2769832 (70%)  1213256 (30%)   3552132 (89%)
2-RP0 Healthy  3983088  2189824 (55%)  1793264 (45%)   2981732 (75%)

CPU Utilization
 Slot  CPU   User System   Nice   Idle    IRQ   SIRQ IOwait
1-RP0    0  24.70  10.60   0.00  64.50   0.00   0.20   0.00
         1  35.40   8.40   0.00  56.20   0.00   0.00   0.00
         2  31.40  10.90   0.00  57.70   0.00   0.00   0.00
         3  34.30   7.80   0.00  57.80   0.00   0.10   0.00
2-RP0    0   3.00   0.80   0.00  96.20   0.00   0.00   0.00
         1   4.30   0.60   0.00  95.10   0.00   0.00   0.00
         2   2.80   0.80   0.00  96.40   0.00   0.00   0.00
         3   2.60   1.10   0.00  96.29   0.00   0.00   0.00
#sh ver
Cisco IOS XE Software, Version 16.06.06
Cisco IOS Software [Everest], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.6.6, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2019 by Cisco Systems, Inc.
Compiled Thu 11-Apr-19 02:24 by mcpre

ROM: IOS-XE ROMMON
BOOTLDR: CAT3K_CAA Boot Loader (CAT3K_CAA-HBOOT-M) Version 3.56, RELEASE SOFTWARE (P)

sw-c3650 uptime is 1 year, 34 weeks, 5 days, 55 minutes
Uptime for this control processor is 1 year, 34 weeks, 5 days, 1 hour, 0 minutes
System returned to ROM by Reload Command
System image file is "flash:packages.conf"
Last reload reason: Reload Command

Technology Package License Information:
-----------------------------------------------------------------
Technology-package                   Technology-package
Current             Type             Next reboot
------------------------------------------------------------------
ipservicesk9        Permanent        ipservicesk9

cisco WS-C3650-24TD (MIPS) processor (revision N0) with 852917K/6147K bytes of memory.
Processor board ID #########
11 Virtual Ethernet interfaces
52 Gigabit Ethernet interfaces
4 Ten Gigabit Ethernet interfaces
2048K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
250456K bytes of Crash Files at crashinfo:.
1609272K bytes of Flash at flash:.
0K bytes of WebUI ODM Files at webui:.
250456K bytes of Crash Files at crashinfo-2:.
1609272K bytes of Flash at flash-2:.

Base Ethernet MAC Address          : xx:xx:xx:xx:xx:xx
Motherboard Assembly Number        : xx-xxxxx-xx
Motherboard Serial Number          : XXXX
Model Revision Number              : N0
Motherboard Revision Number        : B0
Model Number                       : WS-C3650-24TD
System Serial Number               : XXXX
Switch Ports Model              SW Version        SW Image              Mode
------ ----- -----              ----------        ----------            ----
*    1 28    WS-C3650-24TD      16.6.6            CAT3K_CAA-UNIVERSALK9 INSTALL
     2 28    WS-C3650-24TD      16.6.6            CAT3K_CAA-UNIVERSALK9 INSTALL

Switch 02
---------
Switch uptime                      : 1 year, 32 weeks, 4 hours, 29 minutes

Base Ethernet MAC Address          : xx:xx:xx:xx:xx:xx
Motherboard Assembly Number        : xx-xxxxx-xx
Motherboard Serial Number          : XXXX
Model Revision Number              : K0
Motherboard Revision Number        : B0
Model Number                       : WS-C3650-24TD
System Serial Number               : XXXX

Configuration register is 0x102
#show processes cpu platform sorted location switch 1 r0
CPU utilization for five seconds:  9%, one minute:  9%, five minutes:  9%
Core 0: CPU utilization for five seconds:  9%, one minute:  9%, five minutes: 10%
Core 1: CPU utilization for five seconds: 15%, one minute:  9%, five minutes:  9%
Core 2: CPU utilization for five seconds: 10%, one minute:  9%, five minutes:  9%
Core 3: CPU utilization for five seconds:  9%, one minute:  9%, five minutes:  9%
   Pid    PPid    5Sec    1Min    5Min  Status         Size  Name
---------------------------------------------------------------------------------
 31690   30179      1%      1%      1%  S         749502464  repm
 20547   19205      1%      0%      0%  S        1433403392  sif_mgr
 32656   31341      0%      0%      0%  S         892858368  smand
 32359       1      0%      0%      0%  S           8794112  rotee
 31889       1      0%      0%      0%  S           8916992  rotee
 31783   12036      0%      0%      0%  S           6197248  pman.sh
 31392   30191      0%      0%      0%  S         846893056  fman_fp_image
 31341   12036      0%      0%      0%  S           6197248  pman.sh
 30863       2      0%      0%      0%  S                 0  kworker/3:1
 30741       1      0%      0%      0%  S           8916992  rotee
 30730       1      0%      0%      0%  S           8916992  rotee
 30206   30182      0%      0%      0%  S           4362240  sntp
 30191   15779      0%      0%      0%  S           5931008  pman.sh
 30182   16724      0%      0%      0%  S           4579328  stack_sntp.sh
 30179   12036      0%      0%      0%  S           6189056  pman.sh
 28065    3531      0%      0%      0%  S           2842624  inotifywait
 27832   15476      0%      0%      0%  S           2846720  inotifywait
 27824   15476      0%      0%      0%  S          10272768  issu_stack.sh
 27822   15476      0%      0%      0%  S           2846720  inotifywait
 27816   15476      0%      0%      0%  S          10272768  issu_stack.sh
 27813   15476      0%      0%      0%  S           2846720  inotifywait
 27799   16724      0%      0%      0%  S           2846720  inotifywait
 26778   20579      0%      0%      0%  S           2682880  sleep
 26666   14169      0%      0%      0%  S           2846720  inotifywait
 26568   19001      0%      0%      0%  S           2682880  sleep
 26255   17723      0%      0%      0%  S           2846720  inotifywait
 25272       2      0%      0%      0%  S                 0  kworker/u8:1
 25022   24372      0%      0%      0%  S         714989568  epc_ws_liaison
 24754       1      0%      0%      0%  S           8794112  rotee
 24372   15779      0%      0%      0%  S           5931008  pman.sh
 23730       2      0%      0%      0%  S                 0  kworker/u8:0
 21135   20083      0%      0%      0%  S         718180352  stack_mgr
 20757   19854      0%      0%      0%  S         709263360  tams_proc
 20649       1      0%      0%      0%  S           8916992  rotee
 20579   19478      0%      0%      0%  S           5103616  sort_files_by_i
 20567   19569      0%      0%      0%  S         708685824  tamd_proc
 20323       1      0%      0%      0%  S           8794112  rotee
 20083   12036      0%      0%      0%  S           6197248  pman.sh
 20022       1      0%      0%      0%  S           8794112  rotee
 19974       1      0%      0%      0%  S           8794112  rotee
 19854   12036      0%      0%      0%  S           6193152  pman.sh
 19645       1      0%      0%      0%  S           8794112  rotee
 19626       1      0%      0%      0%  S           8916992  rotee
 19569   12036      0%      0%      0%  S           6193152  pman.sh
 19478   12036      0%      0%      0%  S           6189056  pman.sh
 19406       1      0%      0%      0%  S           8790016  rotee
 19387   18300      0%      0%      0%  S        1082458112  platform_mgr
 19287   19188      0%      0%      0%  S           5693440  iosd_console_at
 19205   12036      0%      0%      0%  S           6197248  pman.sh
 19188   19185      0%      0%      0%  S           6287360  bexecute.sh
 19185   19019      0%      0%      0%  S           4571136  bexec.sh
 19019   19018      0%      0%      0%  S           6270976  brelay.sh
 19018   18850      0%      0%      0%  S          14503936  in.telnetd
 19001   17711      0%      0%      0%  S           9072640  periodic.sh
 18852       1      0%      0%      0%  S           8916992  rotee
 18850   10621      0%      0%      0%  S           6270976  in.telnetd.sh
 18753   17036      0%      0%      0%  S        2429480960  fed main event
 18660   17010      0%      0%      0%  S         718204928  nif_mgr
 18537       1      0%      0%      0%  S           8916992  rotee
 18457       1      0%      0%      0%  S           8794112  rotee
 18300   12036      0%      0%      0%  S           6189056  pman.sh
 18157       1      0%      0%      0%  S           8916992  rotee
 17801       1      0%      0%      0%  S           8916992  rotee
 17756       1      0%      0%      0%  S           8794112  rotee
 17723   16725      0%      0%      0%  S           6905856  btrace_rotate.s
 17711   12036      0%      0%      0%  S           6193152  pman.sh
 17241       1      0%      0%      0%  S           8916992  rotee
 17036   15779      0%      0%      0%  S           5926912  pman.sh
 17010   12036      0%      0%      0%  S           6193152  pman.sh
 16725   15779      0%      0%      0%  S           5926912  pman.sh
 16724   15573      0%      0%      0%  S          10412032  ncd.sh
 16500       1      0%      0%      0%  S           8916992  rotee
 16478   15779      0%      0%      0%  S           2842624  inotifywait
 16423   15172      0%      0%      0%  S         717787136  lman
 16386       1      0%      0%      0%  S           8912896  rotee
 16157       1      0%      0%      0%  S           8916992  rotee
 15931   14827      0%      0%      0%  S         709464064  keyman
 15779       1      0%      0%      0%  S           6750208  pvp.sh
 15666       1      0%      0%      0%  S           8916992  rotee
 15573   12036      0%      0%      0%  S           6193152  pman.sh
 15476   14507      0%      0%      0%  S          10407936  issu_stack.sh
 15225       1      0%      0%      0%  S           8794112  rotee
 15172   12036      0%      0%      0%  S           6189056  pman.sh
 15062   13934      0%      0%      0%  R         722673664  hman
 14936       1      0%      0%      0%  S           8916992  rotee
 14827   12036      0%      0%      0%  S           6189056  pman.sh
 14715       1      0%      0%      0%  S           8916992  rotee
 14507   12036      0%      0%      0%  S           6193152  pman.sh
 14428       1      0%      0%      0%  S           8916992  rotee
 14211       1      0%      0%      0%  S           8794112  rotee
 14169   13132      0%      0%      0%  S           6893568  btrace_rotate.s
 13948   12771      0%      0%      0%  S         727834624  bt_logger
 13934   12036      0%      0%      0%  S           6189056  pman.sh
 13665       1      0%      0%      0%  S           8916992  rotee
 13617   12505      0%      0%      0%  S          16986112  auto_upgrade_se
 13264       1      0%      0%      0%  S           8794112  rotee
 13192   12327      0%      0%      0%  S          10141696  auto_upgrade_cl
 13132   12036      0%      0%      0%  S           6189056  pman.sh
 12929       1      0%      0%      0%  S           8794112  rotee
 12771   12036      0%      0%      0%  S           6197248  pman.sh
 12612       1      0%      0%      0%  S           8794112  rotee
 12505   12036      0%      0%      0%  S           6189056  pman.sh
 12327   12036      0%      0%      0%  S           6189056  pman.sh
 12225   12036      0%      0%      0%  S           2842624  inotifywait
 12154       1      0%      0%      0%  S           8912896  rotee
 12036       1      0%      0%      0%  S           7241728  pvp.sh
 12031   11914      0%      0%      0%  S           2842624  inotifywait
 12024       1      0%      0%      0%  S           8912896  rotee
 11914       1      0%      0%      0%  S           6877184  psvp.sh
 11336   10629      0%      0%      0%  S           2842624  inotifywait
 11314       1      0%      0%      0%  S           8790016  rotee
 10629       1      0%      0%      0%  S           7999488  rollback_timer.
 10621       1      0%      0%      0%  S           3452928  xinetd
 10619       1      0%      0%      0%  S           3452928  xinetd
  8714   13192      0%      0%      0%  S           2846720  inotifywait
  7939       2      0%      0%      0%  S                 0  kworker/3:2
  6139    5333      0%      0%      0%  S         718807040  psd
  5934    5153      0%      0%      0%  S         715366400  plogd
  5747       1      0%      0%      0%  S           8794112  rotee
  5444       1      0%      0%      0%  S           8794112  rotee
  5443       1      0%      0%      0%  S           3452928  xinetd
  5333   12036      0%      0%      0%  S           6189056  pman.sh
  5153   12036      0%      0%      0%  S           6201344  pman.sh
  4820    3951      0%      0%      0%  S        2829021184  linux_iosd-imag
  4686       2      0%      0%      0%  S                 0  lsmpi-rx
  4684       2      0%      0%      0%  S                 0  lsmpi-xmit
  4683       2      0%      0%      0%  S                 0  lsmpi-refill
  4450    4126      0%      0%      0%  S           2842624  inotifywait
  4426       1      0%      0%      0%  S           8790016  rotee
  4226       1      0%      0%      0%  S           8916992  rotee
  4126       1      0%      0%      0%  S           5812224  iptbl.sh
  4098    3534      0%      0%      0%  S           2842624  inotifywait
  4085       1      0%      0%      0%  S           8790016  rotee
  4051       1      0%      0%      0%  S           8790016  rotee
  3964    3530      0%      0%      0%  S           2674688  sleep
  3951   12036      0%      0%      0%  S           6197248  pman.sh
  3945       1      0%      0%      0%  S           8790016  rotee
  3808    3022      0%      0%      0%  S         751104000  fman_rp
  3654       1      0%      0%      0%  S           5607424  rpc.mountd
  3614       2      0%      0%      0%  S                 0  nfsd
  3613       2      0%      0%      0%  S                 0  nfsd
  3612       2      0%      0%      0%  S                 0  nfsd
  3611       2      0%      0%      0%  S                 0  nfsd
  3610       2      0%      0%      0%  S                 0  nfsd
  3609       2      0%      0%      0%  S                 0  nfsd
  3608       2      0%      0%      0%  S                 0  nfsd
  3607       2      0%      0%      0%  S                 0  nfsd
  3606       2      0%      0%      0%  S                 0  lockd
  3570       1      0%      0%      0%  S           3936256  rpc.statd
  3562    3549      0%      0%      0%  S           2842624  inotifywait
  3554    3535      0%      0%      0%  S         217464832  libvirtd
  3549       1      0%      0%      0%  S           4575232  boothelper_evt.
  3535       1      0%      0%      0%  S           4575232  libvirtd.sh
  3534       1      0%      0%      0%  S           6959104  droputil.sh
  3531       1      0%      0%      0%  S           6971392  reflector.sh
  3530       1      0%      0%      0%  S           4980736  oom.sh
  3524       1      0%      0%      0%  S          17580032  virtlogd
  3522       1      0%      0%      0%  S           6324224  rpcbind
  3521       1      0%      0%      0%  S           5267456  klogd
  3520       1      0%      0%      0%  S           3452928  xinetd
  3496    2486      0%      0%      0%  S         884219904  dbm
  3395       1      0%      0%      0%  S           8916992  rotee
  3022   12036      0%      0%      0%  S           6189056  pman.sh
  3009    2052      0%      0%      0%  S         717688832  cmm
  2975       1      0%      0%      0%  S           8916992  rotee
  2833    1872      0%      0%      0%  S        1145745408  cli_agent
  2486   12036      0%      0%      0%  S           6189056  pman.sh
  2463       1      0%      0%      0%  S           8916992  rotee
  2148       1      0%      0%      0%  S           8916992  rotee
  2052   12036      0%      0%      0%  S           6189056  pman.sh
  1872   12036      0%      0%      0%  S           6189056  pman.sh
  1608    1021      0%      0%      0%  S         730210304  tms
  1275       1      0%      0%      0%  S           8916992  rotee
  1021   12036      0%      0%      0%  S           6189056  pman.sh
   899       2      0%      0%      0%  S                 0  loop7
   878       2      0%      0%      0%  S                 0  loop6
   813       2      0%      0%      0%  S                 0  loop5
   795   31783      0%      0%      0%  S         793092096  smd
   783       2      0%      0%      0%  S                 0  loop4
   762       2      0%      0%      0%  S                 0  loop3
   676       2      0%      0%      0%  S                 0  loop2
   655       2      0%      0%      0%  S                 0  loop1
   609       2      0%      0%      0%  S                 0  loop0
   268       2      0%      0%      0%  S                 0  kworker/2:2
   266       2      0%      0%      0%  S                 0  kworker/2:1H
   197       2      0%      0%      0%  S                 0  kworker/1:1H
   196       2      0%      0%      0%  S                 0  kworker/0:1H
   195       2      0%      0%      0%  S                 0  kworker/3:1H
   136       2      0%      0%      0%  S                 0  kworker/0:2
   104       1      0%      0%      0%  S           5742592  systemd-udevd
    99       2      0%      0%      0%  S                 0  kauditd
    92       2      0%      0%      0%  S                 0  kworker/1:2
    88       1      0%      0%      0%  S          12726272  systemd-journal
    81       2      0%      0%      0%  S                 0  deferwq
    70       2      0%      0%      0%  S                 0  irq/97-octeon-h
    69       2      0%      0%      0%  S                 0  octeon-ethernet
    64       2      0%      0%      0%  S                 0  scsi_eh_0
    57       2      0%      0%      0%  S                 0  nfsiod
    56       2      0%      0%      0%  S                 0  fsnotify_mark
    55       2      0%      0%      0%  S                 0  ksmd
    54       2      0%      0%      0%  S                 0  kswapd0
    51       2      0%      0%      0%  S                 0  kworker/0:1
    50       2      0%      0%      0%  S                 0  kworker/1:1
    49       2      0%      0%      0%  S                 0  rpciod
    48       2      0%      0%      0%  S                 0  edac-poller
    47       2      0%      0%      0%  S                 0  md
    46       2      0%      0%      0%  S                 0  khubd
    45       2      0%      0%      0%  S                 0  ata_sff
    44       2      0%      0%      0%  S                 0  kblockd
    43       2      0%      0%      0%  S                 0  crypto
    42       2      0%      0%      0%  S                 0  bioset
    41       2      0%      0%      0%  S                 0  writeback
    40       2      0%      0%      0%  S                 0  irq/99-octeon-h
    39       2      0%      0%      0%  S                 0  irq/98-octeon-h
    38       2      0%      0%      0%  S                 0  irq/96-octeon-h
    37       2      0%      0%      0%  S                 0  irq/127-octeon-
    36       2      0%      0%      0%  S                 0  irq/95-octeon-h
    35       2      0%      0%      0%  S                 0  irq/93-octeon-h
    34       2      0%      0%      0%  S                 0  irq/92-octeon-h
    33       2      0%      0%      0%  S                 0  irq/91-octeon-h
    32       2      0%      0%      0%  S                 0  irq/94-octeon-h
    31       2      0%      0%      0%  S                 0  irq/90-octeon-h
    30       2      0%      0%      0%  S                 0  irq/89-octeon-h
    29       2      0%      0%      0%  S                 0  irq/60-octeon-h
    28       2      0%      0%      0%  S                 0  netns
    27       2      0%      0%      0%  S                 0  kdevtmpfs
    26       2      0%      0%      0%  S                 0  khelper
    25       2      0%      0%      0%  S                 0  kworker/3:0H
    23       2      0%      0%      0%  S                 0  ksoftirqd/3
    22       2      0%      0%      0%  S                 0  migration/3
    21       2      0%      0%      0%  S                 0  rcuc/3
    20       2      0%      0%      0%  S                 0  kworker/2:0H
    19       2      0%      0%      0%  S                 0  kworker/2:0
    18       2      0%      0%      0%  S                 0  ksoftirqd/2
    17       2      0%      0%      0%  S                 0  migration/2
    16       2      0%      0%      0%  S                 0  rcuc/2
    15       2      0%      0%      0%  S                 0  kworker/1:0H
    13       2      0%      0%      0%  S                 0  ksoftirqd/1
    12       2      0%      0%      0%  S                 0  migration/1
    11       2      0%      0%      0%  S                 0  rcuc/1
    10       2      0%      0%      0%  S                 0  rcuc/0
     9       2      0%      0%      0%  S                 0  rcu_sched
     8       2      0%      0%      0%  S                 0  rcu_bh
     7       2      0%      0%      0%  S                 0  migration/0
     5       2      0%      0%      0%  S                 0  kworker/0:0H
     3       2      0%      0%      0%  S                 0  ksoftirqd/0
     2       0      0%      0%      0%  S                 0  kthreadd
     1       0      0%      0%      0%  S           8114176  systemd

#show processes cpu platform sorted location switch 2 r0
CPU utilization for five seconds:  3%, one minute:  4%, five minutes:  4%
Core 0: CPU utilization for five seconds:  3%, one minute:  3%, five minutes:  3%
Core 1: CPU utilization for five seconds:  3%, one minute:  3%, five minutes:  4%
Core 2: CPU utilization for five seconds:  4%, one minute:  4%, five minutes:  4%
Core 3: CPU utilization for five seconds:  3%, one minute:  4%, five minutes:  4%
   Pid    PPid    5Sec    1Min    5Min  Status         Size  Name
---------------------------------------------------------------------------------
 10646   10000      2%      3%      3%  S        2130436096  linux_iosd-imag
 20414   19142      1%      0%      0%  S        1433403392  sif_mgr
   789   30873      1%      1%      1%  S         749408256  repm
 32577       2      0%      0%      0%  S                 0  kworker/2:2
 32545   11975      0%      0%      0%  S           6197248  pman.sh
 32390   32289      0%      0%      0%  S           6250496  btelnet.sh
 32289   30799      0%      0%      0%  S           4575232  bexec.sh
 31890       1      0%      0%      0%  S           8916992  rotee
 31661   29478      0%      0%      0%  S         715309056  plogd
 31532   29501      0%      0%      0%  S         846295040  fman_fp_image
 30873   11975      0%      0%      0%  S           6189056  pman.sh
 30799   29767      0%      0%      0%  S           6275072  brelay.sh
 30723       1      0%      0%      0%  S           9830400  rotee
 30480       1      0%      0%      0%  S           8916992  rotee
 30391       1      0%      0%      0%  S           8794112  rotee
 29767       1      0%      0%      0%  S           6107136  brelay_console.
 29638       1      0%      0%      0%  S           9830400  rotee
 29501   15680      0%      0%      0%  S           5931008  pman.sh
 29478   11975      0%      0%      0%  S           6189056  pman.sh
 29180       2      0%      0%      0%  S                 0  kworker/0:2
 27208       2      0%      0%      0%  S                 0  kworker/1:2
 26462   15400      0%      0%      0%  S           2846720  inotifywait
 26452   15400      0%      0%      0%  S          10272768  issu_stack.sh
 26450   15400      0%      0%      0%  S           2846720  inotifywait
 26443   15400      0%      0%      0%  S          10272768  issu_stack.sh
 26442   15400      0%      0%      0%  S           2846720  inotifywait
 26376   16629      0%      0%      0%  S           2846720  inotifywait
 26298    3459      0%      0%      0%  S           2842624  inotifywait
 24602   23956      0%      0%      0%  S         714915840  epc_ws_liaison
 24334       1      0%      0%      0%  S           8794112  rotee
 23956   15680      0%      0%      0%  S           5931008  pman.sh
 21127       2      0%      0%      0%  S                 0  kworker/2:1
 20969   20048      0%      0%      0%  S         718155776  stack_mgr
 20597       1      0%      0%      0%  S           8916992  rotee
 20507   19426      0%      0%      0%  S           5103616  sort_files_by_i
 20048   11975      0%      0%      0%  S           6197248  pman.sh
 19947       1      0%      0%      0%  S           8794112  rotee
 19571       1      0%      0%      0%  S           8794112  rotee
 19556       1      0%      0%      0%  S           8916992  rotee
 19426   11975      0%      0%      0%  S           6189056  pman.sh
 19383   18195      0%      0%      0%  S        1080860672  platform_mgr
 19142   11975      0%      0%      0%  S           6197248  pman.sh
 18935   17545      0%      0%      0%  S           9072640  periodic.sh
 18786       1      0%      0%      0%  S           8916992  rotee
 18744   17022      0%      0%      0%  S        2429480960  fed main event
 18641       1      0%      0%      0%  S           8916992  rotee
 18600   16888      0%      0%      0%  S         718204928  nif_mgr
 18377       1      0%      0%      0%  S           8794112  rotee
 18195   11975      0%      0%      0%  S           6189056  pman.sh
 18088       1      0%      0%      0%  S           8916992  rotee
 17869   16733      0%      0%      0%  S           6901760  btrace_rotate.s
 17812       1      0%      0%      0%  S           8916992  rotee
 17608       1      0%      0%      0%  S           8794112  rotee
 17545   11975      0%      0%      0%  S           6193152  pman.sh
 17274       1      0%      0%      0%  S           8916992  rotee
 17022   15680      0%      0%      0%  S           5926912  pman.sh
 16888   11975      0%      0%      0%  S           6193152  pman.sh
 16733   15680      0%      0%      0%  S           5926912  pman.sh
 16629   15492      0%      0%      0%  S          10412032  ncd.sh
 16501       1      0%      0%      0%  S           8916992  rotee
 16470   15680      0%      0%      0%  S           2842624  inotifywait
 16378       1      0%      0%      0%  S           8912896  rotee
 16317   15119      0%      0%      0%  S         717631488  lman
 16057       1      0%      0%      0%  S           8916992  rotee
 15992   13142      0%      0%      0%  S           2846720  inotifywait
 15920   14790      0%      0%      0%  S         709464064  keyman
 15680       1      0%      0%      0%  S           6750208  pvp.sh
 15602       1      0%      0%      0%  S           8916992  rotee
 15492   11975      0%      0%      0%  S           6193152  pman.sh
 15400   14451      0%      0%      0%  S          10407936  issu_stack.sh
 15177       1      0%      0%      0%  S           8794112  rotee
 15119   11975      0%      0%      0%  S           6189056  pman.sh
 14954   13866      0%      0%      0%  R         722579456  hman
 14843       1      0%      0%      0%  S           8916992  rotee
 14790   11975      0%      0%      0%  S           6189056  pman.sh
 14668       1      0%      0%      0%  S           8916992  rotee
 14482       2      0%      0%      0%  S                 0  kworker/3:0
 14451   11975      0%      0%      0%  S           6193152  pman.sh
 14340       1      0%      0%      0%  S           8916992  rotee
 14161       1      0%      0%      0%  S           8794112  rotee
 14140   13055      0%      0%      0%  S           6893568  btrace_rotate.s
 13906   12709      0%      0%      0%  S         727535616  bt_logger
 13866   11975      0%      0%      0%  S           6189056  pman.sh
 13623       1      0%      0%      0%  S           8916992  rotee
 13596   12444      0%      0%      0%  S          16986112  auto_upgrade_se
 13473   12751      0%      0%      0%  S         729792512  tms
 13211       1      0%      0%      0%  S           8794112  rotee
 13142   12266      0%      0%      0%  S          10141696  auto_upgrade_cl
 13055   11975      0%      0%      0%  S           6189056  pman.sh
 13010       1      0%      0%      0%  S           8916992  rotee
 12871       1      0%      0%      0%  S           8794112  rotee
 12751   11975      0%      0%      0%  S           6197248  pman.sh
 12709   11975      0%      0%      0%  S           6197248  pman.sh
 12553       1      0%      0%      0%  S           8794112  rotee
 12444   11975      0%      0%      0%  S           6197248  pman.sh
 12439   11820      0%      0%      0%  S         884977664  smand
 12266   11975      0%      0%      0%  S           6189056  pman.sh
 12164   11975      0%      0%      0%  S           2842624  inotifywait
 12093       1      0%      0%      0%  S           8912896  rotee
 12071       1      0%      0%      0%  S           8916992  rotee
 11975       1      0%      0%      0%  S           7254016  pvp.sh
 11970   11853      0%      0%      0%  S           2842624  inotifywait
 11963       1      0%      0%      0%  S           8912896  rotee
 11853       1      0%      0%      0%  S           6877184  psvp.sh
 11820   11975      0%      0%      0%  S           6189056  pman.sh
 11671   11066      0%      0%      0%  S         718884864  psd
 11321       1      0%      0%      0%  S           8794112  rotee
 11251   10546      0%      0%      0%  S           2842624  inotifywait
 11232       1      0%      0%      0%  S           8790016  rotee
 11066   11975      0%      0%      0%  S           6189056  pman.sh
 10613   20507      0%      0%      0%  S           2682880  sleep
 10568   18935      0%      0%      0%  S           2682880  sleep
 10552       1      0%      0%      0%  S           3452928  xinetd
 10548       1      0%      0%      0%  S           3452928  xinetd
 10546       1      0%      0%      0%  S           7999488  rollback_timer.
 10247       1      0%      0%      0%  S           8916992  rotee
 10185   17869      0%      0%      0%  S           2846720  inotifywait
 10148   14140      0%      0%      0%  S           2846720  inotifywait
 10000   11975      0%      0%      0%  S           6189056  pman.sh
  9877    9171      0%      0%      0%  S         748216320  fman_rp
  9623    8668      0%      0%      0%  S         882077696  dbm
  9513       1      0%      0%      0%  S           8916992  rotee
  9381       2      0%      0%      0%  S                 0  kworker/1:0
  9177    8224      0%      0%      0%  S         717688832  cmm
  9171   11975      0%      0%      0%  S           6189056  pman.sh
  9088       1      0%      0%      0%  S           8916992  rotee
  9077    7870      0%      0%      0%  S         917106688  cli_agent
  8668   11975      0%      0%      0%  S           6189056  pman.sh
  8610       1      0%      0%      0%  S           8916992  rotee
  8339       1      0%      0%      0%  S           8916992  rotee
  8224   11975      0%      0%      0%  S           6189056  pman.sh
  7870   11975      0%      0%      0%  S           6189056  pman.sh
  7398    7382      0%      0%      0%  S           4362240  sntp
  7382   16629      0%      0%      0%  S           4579328  stack_sntp.sh
  7100       2      0%      0%      0%  S                 0  kworker/u8:0
  5408       1      0%      0%      0%  S           3452928  xinetd
  5098       2      0%      0%      0%  S                 0  kworker/3:2
  4629       2      0%      0%      0%  S                 0  lsmpi-rx
  4628       2      0%      0%      0%  S                 0  lsmpi-xmit
  4627       2      0%      0%      0%  S                 0  lsmpi-refill
  4396    4069      0%      0%      0%  S           2842624  inotifywait
  4371       1      0%      0%      0%  S           8790016  rotee
  4069       1      0%      0%      0%  S           5812224  iptbl.sh
  4049    3479      0%      0%      0%  S           2842624  inotifywait
  4019       1      0%      0%      0%  S           8790016  rotee
  4003       1      0%      0%      0%  S           8790016  rotee
  3869    3461      0%      0%      0%  S           2674688  sleep
  3846       1      0%      0%      0%  S           8790016  rotee
  3629       1      0%      0%      0%  S           5607424  rpc.mountd
  3577       2      0%      0%      0%  S                 0  nfsd
  3574       2      0%      0%      0%  S                 0  nfsd
  3560       2      0%      0%      0%  S                 0  nfsd
  3558       2      0%      0%      0%  S                 0  nfsd
  3557       2      0%      0%      0%  S                 0  nfsd
  3556       2      0%      0%      0%  S                 0  nfsd
  3554       2      0%      0%      0%  S                 0  nfsd
  3553       2      0%      0%      0%  S                 0  nfsd
  3552       2      0%      0%      0%  S                 0  lockd
  3542       1      0%      0%      0%  S           3936256  rpc.statd
  3494       1      0%      0%      0%  S           3452928  xinetd
  3484    3477      0%      0%      0%  S           2842624  inotifywait
  3482       1      0%      0%      0%  S           5267456  klogd
  3479       1      0%      0%      0%  S           6959104  droputil.sh
  3477       1      0%      0%      0%  S           4575232  boothelper_evt.
  3470       1      0%      0%      0%  S           6324224  rpcbind
  3465    3456      0%      0%      0%  S         217464832  libvirtd
  3463       1      0%      0%      0%  S          17580032  virtlogd
  3461       1      0%      0%      0%  S           4980736  oom.sh
  3459       1      0%      0%      0%  S           6967296  reflector.sh
  3456       1      0%      0%      0%  S           4575232  libvirtd.sh
  2427       2      0%      0%      0%  S                 0  kworker/u8:1
  1889   32545      0%      0%      0%  S         790482944  smd
  1377   32390      0%      0%      0%  S          12963840  btelnet
  1261       1      0%      0%      0%  S           8794112  rotee
   843       2      0%      0%      0%  S                 0  loop7
   822       2      0%      0%      0%  S                 0  loop6
   757       2      0%      0%      0%  S                 0  loop5
   727       2      0%      0%      0%  S                 0  loop4
   706       2      0%      0%      0%  S                 0  loop3
   620       2      0%      0%      0%  S                 0  loop2
   599       2      0%      0%      0%  S                 0  loop1
   553       2      0%      0%      0%  S                 0  loop0
   223       2      0%      0%      0%  S                 0  kworker/3:1H
   222       2      0%      0%      0%  S                 0  kworker/1:1H
   206       2      0%      0%      0%  S                 0  kworker/2:1H
   188       2      0%      0%      0%  S                 0  kworker/0:1H
   106       1      0%      0%      0%  S           5607424  systemd-udevd
   100       2      0%      0%      0%  S                 0  kauditd
    94       1      0%      0%      0%  S          37826560  systemd-journal
    81       2      0%      0%      0%  S                 0  deferwq
    70       2      0%      0%      0%  S                 0  irq/97-octeon-h
    69       2      0%      0%      0%  S                 0  octeon-ethernet
    64       2      0%      0%      0%  S                 0  scsi_eh_0
    57       2      0%      0%      0%  S                 0  nfsiod
    56       2      0%      0%      0%  S                 0  fsnotify_mark
    55       2      0%      0%      0%  S                 0  ksmd
    54       2      0%      0%      0%  S                 0  kswapd0
    49       2      0%      0%      0%  S                 0  rpciod
    48       2      0%      0%      0%  S                 0  edac-poller
    47       2      0%      0%      0%  S                 0  md
    46       2      0%      0%      0%  S                 0  khubd
    45       2      0%      0%      0%  S                 0  ata_sff
    44       2      0%      0%      0%  S                 0  kblockd
    43       2      0%      0%      0%  S                 0  crypto
    42       2      0%      0%      0%  S                 0  bioset
    41       2      0%      0%      0%  S                 0  writeback
    40       2      0%      0%      0%  S                 0  irq/99-octeon-h
    39       2      0%      0%      0%  S                 0  irq/98-octeon-h
    38       2      0%      0%      0%  S                 0  irq/96-octeon-h
    37       2      0%      0%      0%  S                 0  irq/127-octeon-
    36       2      0%      0%      0%  S                 0  irq/95-octeon-h
    35       2      0%      0%      0%  S                 0  irq/93-octeon-h
    34       2      0%      0%      0%  S                 0  irq/92-octeon-h
    33       2      0%      0%      0%  S                 0  irq/91-octeon-h
    32       2      0%      0%      0%  S                 0  irq/94-octeon-h
    31       2      0%      0%      0%  S                 0  irq/90-octeon-h
    30       2      0%      0%      0%  S                 0  irq/89-octeon-h
    29       2      0%      0%      0%  S                 0  irq/60-octeon-h
    28       2      0%      0%      0%  S                 0  netns
    27       2      0%      0%      0%  S                 0  kdevtmpfs
    26       2      0%      0%      0%  S                 0  khelper
    25       2      0%      0%      0%  S                 0  kworker/3:0H
    23       2      0%      0%      0%  S                 0  ksoftirqd/3
    22       2      0%      0%      0%  S                 0  migration/3
    21       2      0%      0%      0%  S                 0  rcuc/3
    20       2      0%      0%      0%  S                 0  kworker/2:0H
    18       2      0%      0%      0%  S                 0  ksoftirqd/2
    17       2      0%      0%      0%  S                 0  migration/2
    16       2      0%      0%      0%  S                 0  rcuc/2
    15       2      0%      0%      0%  S                 0  kworker/1:0H
    13       2      0%      0%      0%  S                 0  ksoftirqd/1
    12       2      0%      0%      0%  S                 0  migration/1
    11       2      0%      0%      0%  S                 0  rcuc/1
    10       2      0%      0%      0%  S                 0  rcuc/0
     9       2      0%      0%      0%  S                 0  rcu_sched
     8       2      0%      0%      0%  S                 0  rcu_bh
     7       2      0%      0%      0%  S                 0  migration/0
     5       2      0%      0%      0%  S                 0  kworker/0:0H
     4       2      0%      0%      0%  S                 0  kworker/0:0
     3       2      0%      0%      0%  S                 0  ksoftirqd/0
     2       0      0%      0%      0%  S                 0  kthreadd
     1       0      0%      0%      0%  S           8413184  systemd

Thanks a lot!

Highlighted
Hall of Fame Community Legend

I had a look at the output and, you are right, it really is nothing.  

When there is sufficient "load" on the stack, run the command to give a snapshot of the CPU and the memory.

sh platform software status con brief

NOTE:  Do not bother with the classic IOS command "sh proc cpu history" because the output provides an "average" of all the cores.  The output is very useful for single-core appliances that run classic IOS.  For appliances running IOS-XE, it does not provide any useful information as to which CPU core is "hot spinning".  

If the Free Memory or Idle CPU drops to, say <30%, run the other two commands.  I made a mistake and forgot to mention that the "first page" of the two CPU/Memory commands would suffice. 

Highlighted

Here's the commands output while CPU cores are on load

#sh platform software status con brief
Load Average
 Slot  Status  1-Min  5-Min 15-Min
1-RP0 Healthy   2.96   2.83   2.40
2-RP0 Healthy   0.52   0.30   0.26

Memory (kB)
 Slot  Status    Total     Used (Pct)     Free (Pct) Committed (Pct)
1-RP0 Healthy  3983088  2774200 (70%)  1208888 (30%)   3556028 (89%)
2-RP0 Healthy  3983088  2159008 (54%)  1824080 (46%)   2966520 (74%)

CPU Utilization
 Slot  CPU   User System   Nice   Idle    IRQ   SIRQ IOwait
1-RP0    0  36.20  15.30   0.00  48.40   0.00   0.10   0.00
         1  36.53  16.71   0.00  46.74   0.00   0.00   0.00
         2  31.33  14.51   0.00  54.15   0.00   0.00   0.00
         3  37.76  13.08   0.00  49.05   0.00   0.09   0.00
2-RP0    0   3.10   1.10   0.00  95.70   0.00   0.10   0.00
         1   3.39   0.99   0.00  95.50   0.00   0.09   0.00
         2   3.29   0.89   0.00  95.70   0.00   0.09   0.00
         3   2.99   1.09   0.00  95.90   0.00   0.00   0.00

#show processes cpu platform sorted location switch 1 r0
CPU utilization for five seconds: 46%, one minute: 49%, five minutes: 48%
Core 0: CPU utilization for five seconds: 41%, one minute: 51%, five minutes: 49%
Core 1: CPU utilization for five seconds: 46%, one minute: 49%, five minutes: 49%
Core 2: CPU utilization for five seconds: 45%, one minute: 49%, five minutes: 49%
Core 3: CPU utilization for five seconds: 45%, one minute: 47%, five minutes: 48%
   Pid    PPid    5Sec    1Min    5Min  Status         Size  Name
---------------------------------------------------------------------------------
  4686       2      4%      5%      5%  S                 0  lsmpi-rx
  4684       2      4%      6%      5%  S                 0  lsmpi-xmit
 31690   30179      1%      1%      1%  S         749502464  repm
 32656   31341      0%      0%      5%  R         892981248  smand
 32359       1      0%      0%      0%  S           8794112  rotee
 31889       1      0%      0%      0%  S           8916992  rotee
 31783   12036      0%      0%      0%  S           6197248  pman.sh
 31392   30191      0%      0%      0%  S         846893056  fman_fp_image
 31341   12036      0%      0%      0%  S           6197248  pman.sh
 30863       2      0%      0%      0%  S                 0  kworker/3:1
 30741       1      0%      0%      0%  S           8916992  rotee
 30730       1      0%      0%      0%  S           8916992  rotee
 30206   30182      0%      0%      0%  S           4362240  sntp
 30191   15779      0%      0%      0%  S           5931008  pman.sh
 30182   16724      0%      0%      0%  S           4579328  stack_sntp.sh
 30179   12036      0%      0%      0%  S           6189056  pman.sh
 28065    3531      0%      0%      0%  S           2842624  inotifywait
 27832   15476      0%      0%      0%  S           2846720  inotifywait
 27824   15476      0%      0%      0%  S          10272768  issu_stack.sh
 27822   15476      0%      0%      0%  S           2846720  inotifywait
 27816   15476      0%      0%      0%  S          10272768  issu_stack.sh
 27813   15476      0%      0%      0%  S           2846720  inotifywait
 27799   16724      0%      0%      0%  S           2846720  inotifywait
 25022   24372      0%      0%      0%  S         714989568  epc_ws_liaison
 24754       1      0%      0%      0%  S           8794112  rotee
 24372   15779      0%      0%      0%  S           5931008  pman.sh
 21135   20083      0%      0%      0%  S         718180352  stack_mgr
 20757   19854      0%      0%      0%  S         709263360  tams_proc
 20649       1      0%      0%      0%  S           8916992  rotee
 20579   19478      0%      0%      0%  S           5103616  sort_files_by_i
 20567   19569      0%      0%      0%  S         708685824  tamd_proc
 20547   19205      0%      1%      0%  S        1433403392  sif_mgr
 20323       1      0%      0%      0%  S           8794112  rotee
 20083   12036      0%      0%      0%  S           6197248  pman.sh
 20022       1      0%      0%      0%  S           8794112  rotee
 19974       1      0%      0%      0%  S           8794112  rotee
 19854   12036      0%      0%      0%  S           6193152  pman.sh
 19645       1      0%      0%      0%  S           8794112  rotee
 19626       1      0%      0%      0%  S           8916992  rotee
 19569   12036      0%      0%      0%  S           6193152  pman.sh
 19478   12036      0%      0%      0%  S           6189056  pman.sh
 19412   20579      0%      0%      0%  S           2682880  sleep
 19406       1      0%      0%      0%  S           8790016  rotee
 19387   18300      0%      0%      0%  S        1082458112  platform_mgr
 19365   17723      0%      0%      0%  S           2846720  inotifywait
 19287   19188      0%      0%      0%  S           5693440  iosd_console_at
 19205   12036      0%      0%      0%  S           6197248  pman.sh
 19188   19185      0%      0%      0%  S           6287360  bexecute.sh

#show processes cpu platform sorted location switch 2 r0
CPU utilization for five seconds:  4%, one minute:  4%, five minutes:  4%
Core 0: CPU utilization for five seconds:  4%, one minute:  4%, five minutes:  4%
Core 1: CPU utilization for five seconds:  3%, one minute:  4%, five minutes:  4%
Core 2: CPU utilization for five seconds:  5%, one minute:  4%, five minutes:  4%
Core 3: CPU utilization for five seconds:  4%, one minute:  3%, five minutes:  4%
   Pid    PPid    5Sec    1Min    5Min  Status         Size  Name
---------------------------------------------------------------------------------
 10646   10000      3%      3%      3%  S        2130436096  linux_iosd-imag
 20414   19142      1%      0%      0%  S        1433403392  sif_mgr
   789   30873      1%      1%      1%  S         749408256  repm
 32577       2      0%      0%      0%  S                 0  kworker/2:2
 32545   11975      0%      0%      0%  S           6197248  pman.sh
 32390   32289      0%      0%      0%  S           6250496  btelnet.sh
 32289   30799      0%      0%      0%  S           4575232  bexec.sh
 31890       1      0%      0%      0%  S           8916992  rotee
 31661   29478      0%      0%      0%  S         715309056  plogd
 31532   29501      0%      0%      0%  S         846295040  fman_fp_image
 30873   11975      0%      0%      0%  S           6189056  pman.sh
 30799   29767      0%      0%      0%  S           6275072  brelay.sh
 30723       1      0%      0%      0%  S           9830400  rotee
 30480       1      0%      0%      0%  S           8916992  rotee
 30391       1      0%      0%      0%  S           8794112  rotee
 29767       1      0%      0%      0%  S           6107136  brelay_console.
 29638       1      0%      0%      0%  S           9830400  rotee
 29501   15680      0%      0%      0%  S           5931008  pman.sh
 29478   11975      0%      0%      0%  S           6189056  pman.sh
 29180       2      0%      0%      0%  S                 0  kworker/0:2
 27208       2      0%      0%      0%  S                 0  kworker/1:2
 26462   15400      0%      0%      0%  S           2846720  inotifywait
 26452   15400      0%      0%      0%  S          10272768  issu_stack.sh
 26450   15400      0%      0%      0%  S           2846720  inotifywait
 26443   15400      0%      0%      0%  S          10272768  issu_stack.sh
 26442   15400      0%      0%      0%  S           2846720  inotifywait
 26376   16629      0%      0%      0%  S           2846720  inotifywait
 26298    3459      0%      0%      0%  S           2842624  inotifywait
 24602   23956      0%      0%      0%  S         714915840  epc_ws_liaison
 24334       1      0%      0%      0%  S           8794112  rotee
 23956   15680      0%      0%      0%  S           5931008  pman.sh
 22455   17869      0%      0%      0%  S           2846720  inotifywait
 22423   20507      0%      0%      0%  S           2682880  sleep
 22232   18935      0%      0%      0%  S           2682880  sleep
 22002   14140      0%      0%      0%  S           2846720  inotifywait
 21127       2      0%      0%      0%  S                 0  kworker/2:1
 20969   20048      0%      0%      0%  S         718155776  stack_mgr
 20597       1      0%      0%      0%  S           8916992  rotee
 20507   19426      0%      0%      0%  S           5103616  sort_files_by_i
 20048   11975      0%      0%      0%  S           6197248  pman.sh
 19947       1      0%      0%      0%  S           8794112  rotee
 19571       1      0%      0%      0%  S           8794112  rotee
 19556       1      0%      0%      0%  S           8916992  rotee
 19426   11975      0%      0%      0%  S           6189056  pman.sh
 19383   18195      0%      0%      0%  S        1080860672  platform_mgr
 19142   11975      0%      0%      0%  S           6197248  pman.sh
 18935   17545      0%      0%      0%  S           9072640  periodic.sh
 18786       1      0%      0%      0%  S           8916992  rotee

Thanks for your note on "sh proc cpu history" command, I'll consider to switch from AvgBusy1 SNMP OID monitoring to more relevant one.

Highlighted
Hall of Fame Community Legend

Other than the CPU average going >30%, I do not see anything suspicious. 

Here is another command to check the memory: 

sh processes memory platform sorted location switch <1 - 8> r0
Highlighted

Posting an update:
During the past weeks we decided to upgrade sw to 16.09.06 to understand if that's a bug or not.

It seems that High CPU utilization events are happening little less often than before, but we still observing them sometimes.

What's strange for me is that when I see alerts that Prometheus is unable to scrape SNMP metrics from the switch, 'sh proc cpu history' shows 90+% CPU usage but 'show processes cpu platform sorted location switch 2 r0' shows ~40% on all cores.

So why it's not answering to SNMP if CPU cores load only on ~40%. It seems like that sh proc cpu history for some reason is more precise for us, somehow.

 

However, pasting last outputs when the issue was encountered

#show processes cpu platform sorted location switch 2 r0
CPU utilization for five seconds: 47%, one minute: 46%, five minutes: 46%
Core 0: CPU utilization for five seconds: 44%, one minute: 42%, five minutes: 44%
Core 1: CPU utilization for five seconds: 49%, one minute: 47%, five minutes: 47%
Core 2: CPU utilization for five seconds: 43%, one minute: 46%, five minutes: 47%
Core 3: CPU utilization for five seconds: 45%, one minute: 46%, five minutes: 47%
   Pid    PPid    5Sec    1Min    5Min  Status        Size  Name
--------------------------------------------------------------------------------
 12080    8383     95%     92%     93%  R       2178207744  linux_iosd-imag
 12480   10330     79%     82%     84%  S       1836642304  fed main event
  4436       2      6%      6%      6%  S                0  lsmpi-xmit
  4437       2      5%      5%      5%  S                0  lsmpi-rx
  4435       2      1%      1%      1%  S                0  lsmpi-refill
    21       2      1%      1%      1%  S                0  rcuc/3
    16       2      1%      1%      1%  S                0  rcuc/2
 29266   11642      0%      0%      0%  S          2908160  sleep
 28993       2      0%      0%      0%  S                0  kworker/u8:0
 28992    9810      0%      0%      0%  S          2908160  sleep
 27742       2      0%      0%      0%  S                0  kworker/u8:2
 25677   25673      0%      0%      0%  S        180260864  python2.7
 25673   25375      0%      0%      0%  S          5136384  rdope.sh
 25492       1      0%      0%      0%  S          9498624  rotee
 25375       1      0%      0%      0%  S          5943296  pman.sh
 23052   22113      0%      0%      0%  S        311644160  fman_rp
 22309       1      0%      0%      0%  S          9498624  rotee
 22293   21747      0%      0%      0%  S        254238720  dbm
 22113    6431      0%      0%      0%  S          5951488  pman.sh
 21950       1      0%      0%      0%  S          9498624  rotee
 21873   21377      0%      0%      0%  S         42532864  cmm
 21747    6431      0%      0%      0%  S          5951488  pman.sh
 21725   19718      0%      0%      0%  S        342634496  sessmgrd
 21598   20970      0%      0%      0%  S        505753600  cli_agent
 21565       1      0%      0%      0%  S          9498624  rotee
 21377    6431      0%      0%      0%  S          5951488  pman.sh
 21216       1      0%      0%      0%  S          9498624  rotee
 20970    6431      0%      0%      0%  S          5951488  pman.sh
 20799   20370      0%      0%      0%  S         53084160  tms
 20527       1      0%      0%      0%  S          9498624  rotee
 20370    6431      0%      0%      0%  S          5951488  pman.sh
 20262   19962      0%      0%      0%  S        235200512  smand
 20072       1      0%      0%      0%  S          9498624  rotee
 19962    6431      0%      0%      0%  S          5951488  pman.sh
 19857   19471      0%      0%      0%  S        148779008  repm
 19850       1      0%      0%      0%  S          9498624  rotee
 19718    6431      0%      0%      0%  S          5951488  pman.sh
 19610       1      0%      0%      0%  S          9498624  rotee
 19506   18957      0%      0%      0%  S         55529472  psd
 19471    6431      0%      0%      0%  S          5951488  pman.sh
 19426       2      0%      0%      0%  S                0  kworker/u8:1
 19326       1      0%      0%      0%  S          9498624  rotee
 19250   19163      0%      0%      0%  S         10207232  journalctl
 19163   18501      0%      0%      0%  S         39899136  plogd
 18957    6431      0%      0%      0%  S          5951488  pman.sh
 18760       1      0%      0%      0%  S          9498624  rotee

#sh platform software status con brief
Load Average
 Slot  Status  1-Min  5-Min 15-Min
2-RP0 Healthy   3.11   2.81   2.55

Memory (kB)
 Slot  Status    Total     Used (Pct)     Free (Pct) Committed (Pct)
2-RP0 Healthy  3977740  1660184 (42%)  2317556 (58%)   2519316 (63%)

CPU Utilization
 Slot  CPU   User System   Nice   Idle    IRQ   SIRQ IOwait
2-RP0    0  34.43   9.90   0.00  55.65   0.00   0.00   0.00
         1  38.26   8.19   0.00  53.54   0.00   0.00   0.00
         2  35.90   6.20   0.00  57.90   0.00   0.00   0.00
         3  41.25   9.39   0.00  49.35   0.00   0.00   0.00
#sh processes memory platform sorted location switch 2 r0
System memory: 3977740K total, 1660932K used, 2316808K free,
Lowest: 2316808K
   Pid    Text      Data   Stack   Dynamic       RSS     Total              Name
--------------------------------------------------------------------------------
 12080  171507    629616       0       296    629616   2127156   linux_iosd-imag
 12480     206    162188     132     71832    162188   1793596    fed main event
 12003     396     57144     132      2536     57144    778160           sif_mgr
 21598     344     48244     132      6264     48244    493900         cli_agent
 12105    1004     60212     132      2328     60212    448596      platform_mgr
 21725     178     71328     132      7364     71328    334604          sessmgrd
 23052    8562     63592     132      2564     63592    304340           fman_rp
 18550    9018     59584     132     10116     59584    290648     fman_fp_image
 22293     197     53588     132      4168     53588    248280               dbm
 20262     799     72468     132     24580     72468    230948             smand
  3714     545      4428     132       132      4428    214820          libvirtd
 25677       8     25280     136      7380     25280    176036         python2.7
 19857     578     27248     132      1872     27248    145292              repm
  7498      99     13348     132      1416     13348     63896         bt_logger
  9083     200     12424     132      1800     12424     58352              lman
 11926     483     13036     132      1428     13036     56088         stack_mgr
 19506     100     13564     132      1444     13564     54228               psd
 20799     305     15084     132      6704     15084     51840               tms
  8031     589     13828     352      2336     13828     51780              hman
  7741     166     10260     132      1464     10260     46448             btman
 10612     166     10024     132      1380     10024     45980             btman
  9564     230      6420     132       272      6420     45688           nif_mgr
  8491     138     10004     132      1232     10004     45580            keyman
 21873     170      7612     132      1108      7612     41536               cmm
 19163     112      8272     132      1848      8272     38964             plogd
 15728     119      7440     132      1028      7440     38204    epc_ws_liaison
 14269      88      6984     136      1464      6984     28816         tams_proc
 14169     126      6576     136       868      6576     28308         tamd_proc
 13982      76      5860     136       680      5860     27224   tam_svcs_ng3k_c
  3705     687      1968     132       132      1968     17784          virtlogd
  7249      68      2532     136       132      2532     13196   auto_upgrade_se
  6992    1096      8576     244      6568      8576     11560   auto_upgrade_cl
  9200    1096      8592     244      6560      8592     11552            ncd.sh
 16126    1096      7572     244      6412      7572     11404     issu_stack.sh
 16122    1096      7472     244      6412      7472     11404     issu_stack.sh
  8240    1096      8472     244      6412      8472     11404     issu_stack.sh
 19250     509      2116     136      1204      2116      9968        journalctl
  4312      68      1996     132       252      1996      9596             rotee
  3997      68      1996     132       252      1996      9596             rotee
  9810    1096      6436     248      4460      6436      9456       periodic.sh
  6173    1096      6420     244      4436      6420      9428   rollback_timer.
  6475      68      1996     132       252      1996      9392             rotee
 25492      68      1972     136       132      1972      9276             rotee
 22309      68      1980     136       132      1980      9276             rotee
 21950      68      1980     136       132      1980      9276             rotee
 21565      68      1980     136       132      1980      9276             rotee
 21216      68      1980     136       132      1980      9276             rotee
 20527      68      1980     136       132      1980      9276             rotee
 20072      68      1980     136       132      1980      9276             rotee
sw-c3650-6-4#show platform software status control-processor brief
Load Average
 Slot  Status  1-Min  5-Min 15-Min
2-RP0 Healthy   2.35   2.76   2.62

Memory (kB)
 Slot  Status    Total     Used (Pct)     Free (Pct) Committed (Pct)
2-RP0 Healthy  3977740  1660740 (42%)  2317000 (58%)   2519276 (63%)

CPU Utilization
 Slot  CPU   User System   Nice   Idle    IRQ   SIRQ IOwait
2-RP0    0  33.96  11.88   0.00  54.14   0.00   0.00   0.00
         1  39.20   7.40   0.00  53.30   0.00   0.10   0.00
         2  31.30   6.50   0.00  62.20   0.00   0.00   0.00
         3  37.16   8.19   0.00  54.64   0.00   0.00   0.00

Any ideas?

Thanks

Highlighted

Hello,

 

post the full running configuration (sh run) of the 3650 switch...

Highlighted
Hall of Fame Community Legend


@KirillKO wrote:
 12080    8383     95%     92%     93%  R       2178207744  linux_iosd-imag
 12480   10330     79%     82%     84%  S       1836642304  fed main event

Oh, FFS!  Not this one again!

Yes, I know of these two "clowns".  FED = Forwarding Engine Driver.  

Can you have a look at Troubleshoot High CPU Usage in Catalyst Switch Platforms Running IOS-XE 16.x?

There is a very big chance that you may need to downgrade to 16.3.X.

 

Highlighted

Hello, Leo.


@Leo Laohoo wrote:

@KirillKO wrote:
 12080    8383     95%     92%     93%  R       2178207744  linux_iosd-imag
 12480   10330     79%     82%     84%  S       1836642304  fed main event

Oh, FFS!  Not this one again!

Yes, I know of these two "clowns".  FED = Forwarding Engine Driver.  

Can you have a look at Troubleshoot High CPU Usage in Catalyst Switch Platforms Running IOS-XE 16.x?

There is a very big chance that you may need to downgrade to 16.3.X.

 


Could you please clarify why it's the case? 

Looked into the link you provided, will try to follow the steps from it, next time it appears.

Thanks a lot!

Highlighted
Hall of Fame Community Legend


@KirillKO wrote:

Could you please clarify why it's the case? 


Because this bug is not present/reported in 16.3.X.

Highlighted
VIP Mentor

Hello
As you mention disabling dhcp snooping i am assuming this is a layer2 host stack with ip routing disabled with just a single svi for mgt purposes and is that svi enabled for fast switching?

Do you have any active:

span or tap sessions 

igmp snooping 

port-security 

arp inspection  

Check if you have a full stack ring

Any multiple spanning- tree transisions

your access- ports have port-fast enabled 

any interface errors



kind regards
Paul

Please rate and mark posts accordingly if you have found any of the information provided useful.
It will hopefully assist others with similar issues in the future