01-22-2021 02:29 AM
Hello, Cisco Community!
We've encountered high CPU utilization issue on our 2x3650 stack.
Processes eating CPU are: ARP Input, SISF Switcher Th, SISF Main Thread
Issue is happening only on working hours, so likely that connected to user activity.
We tried disabling ip dhcp snooping on user VLANs and adding deny all rule for ipv6 but nothing helps.
Switch firmware is 16.6.6, haven't seen any related bugs for this version.
Do you have any ideas what else to check?
Thanks a lot!
Solved! Go to Solution.
05-18-2021 04:46 AM
As a conclusion, we've found the root cause description here https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvw92906/?rfs=iqvred
Packet captures showed us that CPU utilization is getting high when the ARP flood was happening and luckily we were able to identify Apple Watch devices having this behaviour from time to time while connected to the office Wi-Fi.
Hopefully, after we decided to upgrade WLC software to the latest 8.10.151.0 version the issue hasn't been encountered again(it's almost 3 weeks after the upgrade).
Thanks to all the community members who helped us here!
01-22-2021 02:51 AM
check this thread may be bug :
01-22-2021 08:37 AM
Hello,
thanks for pointing to that, I'll take a look.
Thanks
01-22-2021 02:52 AM
@KirillKO wrote:
Switch firmware is 16.6.6, haven't seen any related bugs for this version.
There are, a minimum of, seven bugs attributed for "SISF Main Thread".
Can you post the complete output to the following commands:
sh platform software status con brief sh version
show processes cpu platform sorted location switch <1 & 2> r0
01-22-2021 08:34 AM
Hello, thanks for the prompt answer!
See comands output below, however output might be irrelevant cause of non-working hours as I described earlier
#sh platform software status con brief Load Average Slot Status 1-Min 5-Min 15-Min 1-RP0 Healthy 1.42 1.67 1.63 2-RP0 Healthy 0.24 0.21 0.23 Memory (kB) Slot Status Total Used (Pct) Free (Pct) Committed (Pct) 1-RP0 Healthy 3983088 2769832 (70%) 1213256 (30%) 3552132 (89%) 2-RP0 Healthy 3983088 2189824 (55%) 1793264 (45%) 2981732 (75%) CPU Utilization Slot CPU User System Nice Idle IRQ SIRQ IOwait 1-RP0 0 24.70 10.60 0.00 64.50 0.00 0.20 0.00 1 35.40 8.40 0.00 56.20 0.00 0.00 0.00 2 31.40 10.90 0.00 57.70 0.00 0.00 0.00 3 34.30 7.80 0.00 57.80 0.00 0.10 0.00 2-RP0 0 3.00 0.80 0.00 96.20 0.00 0.00 0.00 1 4.30 0.60 0.00 95.10 0.00 0.00 0.00 2 2.80 0.80 0.00 96.40 0.00 0.00 0.00 3 2.60 1.10 0.00 96.29 0.00 0.00 0.00
#sh ver Cisco IOS XE Software, Version 16.06.06 Cisco IOS Software [Everest], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.6.6, RELEASE SOFTWARE (fc1) Technical Support: http://www.cisco.com/techsupport Copyright (c) 1986-2019 by Cisco Systems, Inc. Compiled Thu 11-Apr-19 02:24 by mcpre ROM: IOS-XE ROMMON BOOTLDR: CAT3K_CAA Boot Loader (CAT3K_CAA-HBOOT-M) Version 3.56, RELEASE SOFTWARE (P) sw-c3650 uptime is 1 year, 34 weeks, 5 days, 55 minutes Uptime for this control processor is 1 year, 34 weeks, 5 days, 1 hour, 0 minutes System returned to ROM by Reload Command System image file is "flash:packages.conf" Last reload reason: Reload Command Technology Package License Information: ----------------------------------------------------------------- Technology-package Technology-package Current Type Next reboot ------------------------------------------------------------------ ipservicesk9 Permanent ipservicesk9 cisco WS-C3650-24TD (MIPS) processor (revision N0) with 852917K/6147K bytes of memory. Processor board ID ######### 11 Virtual Ethernet interfaces 52 Gigabit Ethernet interfaces 4 Ten Gigabit Ethernet interfaces 2048K bytes of non-volatile configuration memory. 4194304K bytes of physical memory. 250456K bytes of Crash Files at crashinfo:. 1609272K bytes of Flash at flash:. 0K bytes of WebUI ODM Files at webui:. 250456K bytes of Crash Files at crashinfo-2:. 1609272K bytes of Flash at flash-2:. Base Ethernet MAC Address : xx:xx:xx:xx:xx:xx Motherboard Assembly Number : xx-xxxxx-xx Motherboard Serial Number : XXXX Model Revision Number : N0 Motherboard Revision Number : B0 Model Number : WS-C3650-24TD System Serial Number : XXXX Switch Ports Model SW Version SW Image Mode ------ ----- ----- ---------- ---------- ---- * 1 28 WS-C3650-24TD 16.6.6 CAT3K_CAA-UNIVERSALK9 INSTALL 2 28 WS-C3650-24TD 16.6.6 CAT3K_CAA-UNIVERSALK9 INSTALL Switch 02 --------- Switch uptime : 1 year, 32 weeks, 4 hours, 29 minutes Base Ethernet MAC Address : xx:xx:xx:xx:xx:xx Motherboard Assembly Number : xx-xxxxx-xx Motherboard Serial Number : XXXX Model Revision Number : K0 Motherboard Revision Number : B0 Model Number : WS-C3650-24TD System Serial Number : XXXX Configuration register is 0x102
#show processes cpu platform sorted location switch 1 r0 CPU utilization for five seconds: 9%, one minute: 9%, five minutes: 9% Core 0: CPU utilization for five seconds: 9%, one minute: 9%, five minutes: 10% Core 1: CPU utilization for five seconds: 15%, one minute: 9%, five minutes: 9% Core 2: CPU utilization for five seconds: 10%, one minute: 9%, five minutes: 9% Core 3: CPU utilization for five seconds: 9%, one minute: 9%, five minutes: 9% Pid PPid 5Sec 1Min 5Min Status Size Name --------------------------------------------------------------------------------- 31690 30179 1% 1% 1% S 749502464 repm 20547 19205 1% 0% 0% S 1433403392 sif_mgr 32656 31341 0% 0% 0% S 892858368 smand 32359 1 0% 0% 0% S 8794112 rotee 31889 1 0% 0% 0% S 8916992 rotee 31783 12036 0% 0% 0% S 6197248 pman.sh 31392 30191 0% 0% 0% S 846893056 fman_fp_image 31341 12036 0% 0% 0% S 6197248 pman.sh 30863 2 0% 0% 0% S 0 kworker/3:1 30741 1 0% 0% 0% S 8916992 rotee 30730 1 0% 0% 0% S 8916992 rotee 30206 30182 0% 0% 0% S 4362240 sntp 30191 15779 0% 0% 0% S 5931008 pman.sh 30182 16724 0% 0% 0% S 4579328 stack_sntp.sh 30179 12036 0% 0% 0% S 6189056 pman.sh 28065 3531 0% 0% 0% S 2842624 inotifywait 27832 15476 0% 0% 0% S 2846720 inotifywait 27824 15476 0% 0% 0% S 10272768 issu_stack.sh 27822 15476 0% 0% 0% S 2846720 inotifywait 27816 15476 0% 0% 0% S 10272768 issu_stack.sh 27813 15476 0% 0% 0% S 2846720 inotifywait 27799 16724 0% 0% 0% S 2846720 inotifywait 26778 20579 0% 0% 0% S 2682880 sleep 26666 14169 0% 0% 0% S 2846720 inotifywait 26568 19001 0% 0% 0% S 2682880 sleep 26255 17723 0% 0% 0% S 2846720 inotifywait 25272 2 0% 0% 0% S 0 kworker/u8:1 25022 24372 0% 0% 0% S 714989568 epc_ws_liaison 24754 1 0% 0% 0% S 8794112 rotee 24372 15779 0% 0% 0% S 5931008 pman.sh 23730 2 0% 0% 0% S 0 kworker/u8:0 21135 20083 0% 0% 0% S 718180352 stack_mgr 20757 19854 0% 0% 0% S 709263360 tams_proc 20649 1 0% 0% 0% S 8916992 rotee 20579 19478 0% 0% 0% S 5103616 sort_files_by_i 20567 19569 0% 0% 0% S 708685824 tamd_proc 20323 1 0% 0% 0% S 8794112 rotee 20083 12036 0% 0% 0% S 6197248 pman.sh 20022 1 0% 0% 0% S 8794112 rotee 19974 1 0% 0% 0% S 8794112 rotee 19854 12036 0% 0% 0% S 6193152 pman.sh 19645 1 0% 0% 0% S 8794112 rotee 19626 1 0% 0% 0% S 8916992 rotee 19569 12036 0% 0% 0% S 6193152 pman.sh 19478 12036 0% 0% 0% S 6189056 pman.sh 19406 1 0% 0% 0% S 8790016 rotee 19387 18300 0% 0% 0% S 1082458112 platform_mgr 19287 19188 0% 0% 0% S 5693440 iosd_console_at 19205 12036 0% 0% 0% S 6197248 pman.sh 19188 19185 0% 0% 0% S 6287360 bexecute.sh 19185 19019 0% 0% 0% S 4571136 bexec.sh 19019 19018 0% 0% 0% S 6270976 brelay.sh 19018 18850 0% 0% 0% S 14503936 in.telnetd 19001 17711 0% 0% 0% S 9072640 periodic.sh 18852 1 0% 0% 0% S 8916992 rotee 18850 10621 0% 0% 0% S 6270976 in.telnetd.sh 18753 17036 0% 0% 0% S 2429480960 fed main event 18660 17010 0% 0% 0% S 718204928 nif_mgr 18537 1 0% 0% 0% S 8916992 rotee 18457 1 0% 0% 0% S 8794112 rotee 18300 12036 0% 0% 0% S 6189056 pman.sh 18157 1 0% 0% 0% S 8916992 rotee 17801 1 0% 0% 0% S 8916992 rotee 17756 1 0% 0% 0% S 8794112 rotee 17723 16725 0% 0% 0% S 6905856 btrace_rotate.s 17711 12036 0% 0% 0% S 6193152 pman.sh 17241 1 0% 0% 0% S 8916992 rotee 17036 15779 0% 0% 0% S 5926912 pman.sh 17010 12036 0% 0% 0% S 6193152 pman.sh 16725 15779 0% 0% 0% S 5926912 pman.sh 16724 15573 0% 0% 0% S 10412032 ncd.sh 16500 1 0% 0% 0% S 8916992 rotee 16478 15779 0% 0% 0% S 2842624 inotifywait 16423 15172 0% 0% 0% S 717787136 lman 16386 1 0% 0% 0% S 8912896 rotee 16157 1 0% 0% 0% S 8916992 rotee 15931 14827 0% 0% 0% S 709464064 keyman 15779 1 0% 0% 0% S 6750208 pvp.sh 15666 1 0% 0% 0% S 8916992 rotee 15573 12036 0% 0% 0% S 6193152 pman.sh 15476 14507 0% 0% 0% S 10407936 issu_stack.sh 15225 1 0% 0% 0% S 8794112 rotee 15172 12036 0% 0% 0% S 6189056 pman.sh 15062 13934 0% 0% 0% R 722673664 hman 14936 1 0% 0% 0% S 8916992 rotee 14827 12036 0% 0% 0% S 6189056 pman.sh 14715 1 0% 0% 0% S 8916992 rotee 14507 12036 0% 0% 0% S 6193152 pman.sh 14428 1 0% 0% 0% S 8916992 rotee 14211 1 0% 0% 0% S 8794112 rotee 14169 13132 0% 0% 0% S 6893568 btrace_rotate.s 13948 12771 0% 0% 0% S 727834624 bt_logger 13934 12036 0% 0% 0% S 6189056 pman.sh 13665 1 0% 0% 0% S 8916992 rotee 13617 12505 0% 0% 0% S 16986112 auto_upgrade_se 13264 1 0% 0% 0% S 8794112 rotee 13192 12327 0% 0% 0% S 10141696 auto_upgrade_cl 13132 12036 0% 0% 0% S 6189056 pman.sh 12929 1 0% 0% 0% S 8794112 rotee 12771 12036 0% 0% 0% S 6197248 pman.sh 12612 1 0% 0% 0% S 8794112 rotee 12505 12036 0% 0% 0% S 6189056 pman.sh 12327 12036 0% 0% 0% S 6189056 pman.sh 12225 12036 0% 0% 0% S 2842624 inotifywait 12154 1 0% 0% 0% S 8912896 rotee 12036 1 0% 0% 0% S 7241728 pvp.sh 12031 11914 0% 0% 0% S 2842624 inotifywait 12024 1 0% 0% 0% S 8912896 rotee 11914 1 0% 0% 0% S 6877184 psvp.sh 11336 10629 0% 0% 0% S 2842624 inotifywait 11314 1 0% 0% 0% S 8790016 rotee 10629 1 0% 0% 0% S 7999488 rollback_timer. 10621 1 0% 0% 0% S 3452928 xinetd 10619 1 0% 0% 0% S 3452928 xinetd 8714 13192 0% 0% 0% S 2846720 inotifywait 7939 2 0% 0% 0% S 0 kworker/3:2 6139 5333 0% 0% 0% S 718807040 psd 5934 5153 0% 0% 0% S 715366400 plogd 5747 1 0% 0% 0% S 8794112 rotee 5444 1 0% 0% 0% S 8794112 rotee 5443 1 0% 0% 0% S 3452928 xinetd 5333 12036 0% 0% 0% S 6189056 pman.sh 5153 12036 0% 0% 0% S 6201344 pman.sh 4820 3951 0% 0% 0% S 2829021184 linux_iosd-imag 4686 2 0% 0% 0% S 0 lsmpi-rx 4684 2 0% 0% 0% S 0 lsmpi-xmit 4683 2 0% 0% 0% S 0 lsmpi-refill 4450 4126 0% 0% 0% S 2842624 inotifywait 4426 1 0% 0% 0% S 8790016 rotee 4226 1 0% 0% 0% S 8916992 rotee 4126 1 0% 0% 0% S 5812224 iptbl.sh 4098 3534 0% 0% 0% S 2842624 inotifywait 4085 1 0% 0% 0% S 8790016 rotee 4051 1 0% 0% 0% S 8790016 rotee 3964 3530 0% 0% 0% S 2674688 sleep 3951 12036 0% 0% 0% S 6197248 pman.sh 3945 1 0% 0% 0% S 8790016 rotee 3808 3022 0% 0% 0% S 751104000 fman_rp 3654 1 0% 0% 0% S 5607424 rpc.mountd 3614 2 0% 0% 0% S 0 nfsd 3613 2 0% 0% 0% S 0 nfsd 3612 2 0% 0% 0% S 0 nfsd 3611 2 0% 0% 0% S 0 nfsd 3610 2 0% 0% 0% S 0 nfsd 3609 2 0% 0% 0% S 0 nfsd 3608 2 0% 0% 0% S 0 nfsd 3607 2 0% 0% 0% S 0 nfsd 3606 2 0% 0% 0% S 0 lockd 3570 1 0% 0% 0% S 3936256 rpc.statd 3562 3549 0% 0% 0% S 2842624 inotifywait 3554 3535 0% 0% 0% S 217464832 libvirtd 3549 1 0% 0% 0% S 4575232 boothelper_evt. 3535 1 0% 0% 0% S 4575232 libvirtd.sh 3534 1 0% 0% 0% S 6959104 droputil.sh 3531 1 0% 0% 0% S 6971392 reflector.sh 3530 1 0% 0% 0% S 4980736 oom.sh 3524 1 0% 0% 0% S 17580032 virtlogd 3522 1 0% 0% 0% S 6324224 rpcbind 3521 1 0% 0% 0% S 5267456 klogd 3520 1 0% 0% 0% S 3452928 xinetd 3496 2486 0% 0% 0% S 884219904 dbm 3395 1 0% 0% 0% S 8916992 rotee 3022 12036 0% 0% 0% S 6189056 pman.sh 3009 2052 0% 0% 0% S 717688832 cmm 2975 1 0% 0% 0% S 8916992 rotee 2833 1872 0% 0% 0% S 1145745408 cli_agent 2486 12036 0% 0% 0% S 6189056 pman.sh 2463 1 0% 0% 0% S 8916992 rotee 2148 1 0% 0% 0% S 8916992 rotee 2052 12036 0% 0% 0% S 6189056 pman.sh 1872 12036 0% 0% 0% S 6189056 pman.sh 1608 1021 0% 0% 0% S 730210304 tms 1275 1 0% 0% 0% S 8916992 rotee 1021 12036 0% 0% 0% S 6189056 pman.sh 899 2 0% 0% 0% S 0 loop7 878 2 0% 0% 0% S 0 loop6 813 2 0% 0% 0% S 0 loop5 795 31783 0% 0% 0% S 793092096 smd 783 2 0% 0% 0% S 0 loop4 762 2 0% 0% 0% S 0 loop3 676 2 0% 0% 0% S 0 loop2 655 2 0% 0% 0% S 0 loop1 609 2 0% 0% 0% S 0 loop0 268 2 0% 0% 0% S 0 kworker/2:2 266 2 0% 0% 0% S 0 kworker/2:1H 197 2 0% 0% 0% S 0 kworker/1:1H 196 2 0% 0% 0% S 0 kworker/0:1H 195 2 0% 0% 0% S 0 kworker/3:1H 136 2 0% 0% 0% S 0 kworker/0:2 104 1 0% 0% 0% S 5742592 systemd-udevd 99 2 0% 0% 0% S 0 kauditd 92 2 0% 0% 0% S 0 kworker/1:2 88 1 0% 0% 0% S 12726272 systemd-journal 81 2 0% 0% 0% S 0 deferwq 70 2 0% 0% 0% S 0 irq/97-octeon-h 69 2 0% 0% 0% S 0 octeon-ethernet 64 2 0% 0% 0% S 0 scsi_eh_0 57 2 0% 0% 0% S 0 nfsiod 56 2 0% 0% 0% S 0 fsnotify_mark 55 2 0% 0% 0% S 0 ksmd 54 2 0% 0% 0% S 0 kswapd0 51 2 0% 0% 0% S 0 kworker/0:1 50 2 0% 0% 0% S 0 kworker/1:1 49 2 0% 0% 0% S 0 rpciod 48 2 0% 0% 0% S 0 edac-poller 47 2 0% 0% 0% S 0 md 46 2 0% 0% 0% S 0 khubd 45 2 0% 0% 0% S 0 ata_sff 44 2 0% 0% 0% S 0 kblockd 43 2 0% 0% 0% S 0 crypto 42 2 0% 0% 0% S 0 bioset 41 2 0% 0% 0% S 0 writeback 40 2 0% 0% 0% S 0 irq/99-octeon-h 39 2 0% 0% 0% S 0 irq/98-octeon-h 38 2 0% 0% 0% S 0 irq/96-octeon-h 37 2 0% 0% 0% S 0 irq/127-octeon- 36 2 0% 0% 0% S 0 irq/95-octeon-h 35 2 0% 0% 0% S 0 irq/93-octeon-h 34 2 0% 0% 0% S 0 irq/92-octeon-h 33 2 0% 0% 0% S 0 irq/91-octeon-h 32 2 0% 0% 0% S 0 irq/94-octeon-h 31 2 0% 0% 0% S 0 irq/90-octeon-h 30 2 0% 0% 0% S 0 irq/89-octeon-h 29 2 0% 0% 0% S 0 irq/60-octeon-h 28 2 0% 0% 0% S 0 netns 27 2 0% 0% 0% S 0 kdevtmpfs 26 2 0% 0% 0% S 0 khelper 25 2 0% 0% 0% S 0 kworker/3:0H 23 2 0% 0% 0% S 0 ksoftirqd/3 22 2 0% 0% 0% S 0 migration/3 21 2 0% 0% 0% S 0 rcuc/3 20 2 0% 0% 0% S 0 kworker/2:0H 19 2 0% 0% 0% S 0 kworker/2:0 18 2 0% 0% 0% S 0 ksoftirqd/2 17 2 0% 0% 0% S 0 migration/2 16 2 0% 0% 0% S 0 rcuc/2 15 2 0% 0% 0% S 0 kworker/1:0H 13 2 0% 0% 0% S 0 ksoftirqd/1 12 2 0% 0% 0% S 0 migration/1 11 2 0% 0% 0% S 0 rcuc/1 10 2 0% 0% 0% S 0 rcuc/0 9 2 0% 0% 0% S 0 rcu_sched 8 2 0% 0% 0% S 0 rcu_bh 7 2 0% 0% 0% S 0 migration/0 5 2 0% 0% 0% S 0 kworker/0:0H 3 2 0% 0% 0% S 0 ksoftirqd/0 2 0 0% 0% 0% S 0 kthreadd 1 0 0% 0% 0% S 8114176 systemd #show processes cpu platform sorted location switch 2 r0 CPU utilization for five seconds: 3%, one minute: 4%, five minutes: 4% Core 0: CPU utilization for five seconds: 3%, one minute: 3%, five minutes: 3% Core 1: CPU utilization for five seconds: 3%, one minute: 3%, five minutes: 4% Core 2: CPU utilization for five seconds: 4%, one minute: 4%, five minutes: 4% Core 3: CPU utilization for five seconds: 3%, one minute: 4%, five minutes: 4% Pid PPid 5Sec 1Min 5Min Status Size Name --------------------------------------------------------------------------------- 10646 10000 2% 3% 3% S 2130436096 linux_iosd-imag 20414 19142 1% 0% 0% S 1433403392 sif_mgr 789 30873 1% 1% 1% S 749408256 repm 32577 2 0% 0% 0% S 0 kworker/2:2 32545 11975 0% 0% 0% S 6197248 pman.sh 32390 32289 0% 0% 0% S 6250496 btelnet.sh 32289 30799 0% 0% 0% S 4575232 bexec.sh 31890 1 0% 0% 0% S 8916992 rotee 31661 29478 0% 0% 0% S 715309056 plogd 31532 29501 0% 0% 0% S 846295040 fman_fp_image 30873 11975 0% 0% 0% S 6189056 pman.sh 30799 29767 0% 0% 0% S 6275072 brelay.sh 30723 1 0% 0% 0% S 9830400 rotee 30480 1 0% 0% 0% S 8916992 rotee 30391 1 0% 0% 0% S 8794112 rotee 29767 1 0% 0% 0% S 6107136 brelay_console. 29638 1 0% 0% 0% S 9830400 rotee 29501 15680 0% 0% 0% S 5931008 pman.sh 29478 11975 0% 0% 0% S 6189056 pman.sh 29180 2 0% 0% 0% S 0 kworker/0:2 27208 2 0% 0% 0% S 0 kworker/1:2 26462 15400 0% 0% 0% S 2846720 inotifywait 26452 15400 0% 0% 0% S 10272768 issu_stack.sh 26450 15400 0% 0% 0% S 2846720 inotifywait 26443 15400 0% 0% 0% S 10272768 issu_stack.sh 26442 15400 0% 0% 0% S 2846720 inotifywait 26376 16629 0% 0% 0% S 2846720 inotifywait 26298 3459 0% 0% 0% S 2842624 inotifywait 24602 23956 0% 0% 0% S 714915840 epc_ws_liaison 24334 1 0% 0% 0% S 8794112 rotee 23956 15680 0% 0% 0% S 5931008 pman.sh 21127 2 0% 0% 0% S 0 kworker/2:1 20969 20048 0% 0% 0% S 718155776 stack_mgr 20597 1 0% 0% 0% S 8916992 rotee 20507 19426 0% 0% 0% S 5103616 sort_files_by_i 20048 11975 0% 0% 0% S 6197248 pman.sh 19947 1 0% 0% 0% S 8794112 rotee 19571 1 0% 0% 0% S 8794112 rotee 19556 1 0% 0% 0% S 8916992 rotee 19426 11975 0% 0% 0% S 6189056 pman.sh 19383 18195 0% 0% 0% S 1080860672 platform_mgr 19142 11975 0% 0% 0% S 6197248 pman.sh 18935 17545 0% 0% 0% S 9072640 periodic.sh 18786 1 0% 0% 0% S 8916992 rotee 18744 17022 0% 0% 0% S 2429480960 fed main event 18641 1 0% 0% 0% S 8916992 rotee 18600 16888 0% 0% 0% S 718204928 nif_mgr 18377 1 0% 0% 0% S 8794112 rotee 18195 11975 0% 0% 0% S 6189056 pman.sh 18088 1 0% 0% 0% S 8916992 rotee 17869 16733 0% 0% 0% S 6901760 btrace_rotate.s 17812 1 0% 0% 0% S 8916992 rotee 17608 1 0% 0% 0% S 8794112 rotee 17545 11975 0% 0% 0% S 6193152 pman.sh 17274 1 0% 0% 0% S 8916992 rotee 17022 15680 0% 0% 0% S 5926912 pman.sh 16888 11975 0% 0% 0% S 6193152 pman.sh 16733 15680 0% 0% 0% S 5926912 pman.sh 16629 15492 0% 0% 0% S 10412032 ncd.sh 16501 1 0% 0% 0% S 8916992 rotee 16470 15680 0% 0% 0% S 2842624 inotifywait 16378 1 0% 0% 0% S 8912896 rotee 16317 15119 0% 0% 0% S 717631488 lman 16057 1 0% 0% 0% S 8916992 rotee 15992 13142 0% 0% 0% S 2846720 inotifywait 15920 14790 0% 0% 0% S 709464064 keyman 15680 1 0% 0% 0% S 6750208 pvp.sh 15602 1 0% 0% 0% S 8916992 rotee 15492 11975 0% 0% 0% S 6193152 pman.sh 15400 14451 0% 0% 0% S 10407936 issu_stack.sh 15177 1 0% 0% 0% S 8794112 rotee 15119 11975 0% 0% 0% S 6189056 pman.sh 14954 13866 0% 0% 0% R 722579456 hman 14843 1 0% 0% 0% S 8916992 rotee 14790 11975 0% 0% 0% S 6189056 pman.sh 14668 1 0% 0% 0% S 8916992 rotee 14482 2 0% 0% 0% S 0 kworker/3:0 14451 11975 0% 0% 0% S 6193152 pman.sh 14340 1 0% 0% 0% S 8916992 rotee 14161 1 0% 0% 0% S 8794112 rotee 14140 13055 0% 0% 0% S 6893568 btrace_rotate.s 13906 12709 0% 0% 0% S 727535616 bt_logger 13866 11975 0% 0% 0% S 6189056 pman.sh 13623 1 0% 0% 0% S 8916992 rotee 13596 12444 0% 0% 0% S 16986112 auto_upgrade_se 13473 12751 0% 0% 0% S 729792512 tms 13211 1 0% 0% 0% S 8794112 rotee 13142 12266 0% 0% 0% S 10141696 auto_upgrade_cl 13055 11975 0% 0% 0% S 6189056 pman.sh 13010 1 0% 0% 0% S 8916992 rotee 12871 1 0% 0% 0% S 8794112 rotee 12751 11975 0% 0% 0% S 6197248 pman.sh 12709 11975 0% 0% 0% S 6197248 pman.sh 12553 1 0% 0% 0% S 8794112 rotee 12444 11975 0% 0% 0% S 6197248 pman.sh 12439 11820 0% 0% 0% S 884977664 smand 12266 11975 0% 0% 0% S 6189056 pman.sh 12164 11975 0% 0% 0% S 2842624 inotifywait 12093 1 0% 0% 0% S 8912896 rotee 12071 1 0% 0% 0% S 8916992 rotee 11975 1 0% 0% 0% S 7254016 pvp.sh 11970 11853 0% 0% 0% S 2842624 inotifywait 11963 1 0% 0% 0% S 8912896 rotee 11853 1 0% 0% 0% S 6877184 psvp.sh 11820 11975 0% 0% 0% S 6189056 pman.sh 11671 11066 0% 0% 0% S 718884864 psd 11321 1 0% 0% 0% S 8794112 rotee 11251 10546 0% 0% 0% S 2842624 inotifywait 11232 1 0% 0% 0% S 8790016 rotee 11066 11975 0% 0% 0% S 6189056 pman.sh 10613 20507 0% 0% 0% S 2682880 sleep 10568 18935 0% 0% 0% S 2682880 sleep 10552 1 0% 0% 0% S 3452928 xinetd 10548 1 0% 0% 0% S 3452928 xinetd 10546 1 0% 0% 0% S 7999488 rollback_timer. 10247 1 0% 0% 0% S 8916992 rotee 10185 17869 0% 0% 0% S 2846720 inotifywait 10148 14140 0% 0% 0% S 2846720 inotifywait 10000 11975 0% 0% 0% S 6189056 pman.sh 9877 9171 0% 0% 0% S 748216320 fman_rp 9623 8668 0% 0% 0% S 882077696 dbm 9513 1 0% 0% 0% S 8916992 rotee 9381 2 0% 0% 0% S 0 kworker/1:0 9177 8224 0% 0% 0% S 717688832 cmm 9171 11975 0% 0% 0% S 6189056 pman.sh 9088 1 0% 0% 0% S 8916992 rotee 9077 7870 0% 0% 0% S 917106688 cli_agent 8668 11975 0% 0% 0% S 6189056 pman.sh 8610 1 0% 0% 0% S 8916992 rotee 8339 1 0% 0% 0% S 8916992 rotee 8224 11975 0% 0% 0% S 6189056 pman.sh 7870 11975 0% 0% 0% S 6189056 pman.sh 7398 7382 0% 0% 0% S 4362240 sntp 7382 16629 0% 0% 0% S 4579328 stack_sntp.sh 7100 2 0% 0% 0% S 0 kworker/u8:0 5408 1 0% 0% 0% S 3452928 xinetd 5098 2 0% 0% 0% S 0 kworker/3:2 4629 2 0% 0% 0% S 0 lsmpi-rx 4628 2 0% 0% 0% S 0 lsmpi-xmit 4627 2 0% 0% 0% S 0 lsmpi-refill 4396 4069 0% 0% 0% S 2842624 inotifywait 4371 1 0% 0% 0% S 8790016 rotee 4069 1 0% 0% 0% S 5812224 iptbl.sh 4049 3479 0% 0% 0% S 2842624 inotifywait 4019 1 0% 0% 0% S 8790016 rotee 4003 1 0% 0% 0% S 8790016 rotee 3869 3461 0% 0% 0% S 2674688 sleep 3846 1 0% 0% 0% S 8790016 rotee 3629 1 0% 0% 0% S 5607424 rpc.mountd 3577 2 0% 0% 0% S 0 nfsd 3574 2 0% 0% 0% S 0 nfsd 3560 2 0% 0% 0% S 0 nfsd 3558 2 0% 0% 0% S 0 nfsd 3557 2 0% 0% 0% S 0 nfsd 3556 2 0% 0% 0% S 0 nfsd 3554 2 0% 0% 0% S 0 nfsd 3553 2 0% 0% 0% S 0 nfsd 3552 2 0% 0% 0% S 0 lockd 3542 1 0% 0% 0% S 3936256 rpc.statd 3494 1 0% 0% 0% S 3452928 xinetd 3484 3477 0% 0% 0% S 2842624 inotifywait 3482 1 0% 0% 0% S 5267456 klogd 3479 1 0% 0% 0% S 6959104 droputil.sh 3477 1 0% 0% 0% S 4575232 boothelper_evt. 3470 1 0% 0% 0% S 6324224 rpcbind 3465 3456 0% 0% 0% S 217464832 libvirtd 3463 1 0% 0% 0% S 17580032 virtlogd 3461 1 0% 0% 0% S 4980736 oom.sh 3459 1 0% 0% 0% S 6967296 reflector.sh 3456 1 0% 0% 0% S 4575232 libvirtd.sh 2427 2 0% 0% 0% S 0 kworker/u8:1 1889 32545 0% 0% 0% S 790482944 smd 1377 32390 0% 0% 0% S 12963840 btelnet 1261 1 0% 0% 0% S 8794112 rotee 843 2 0% 0% 0% S 0 loop7 822 2 0% 0% 0% S 0 loop6 757 2 0% 0% 0% S 0 loop5 727 2 0% 0% 0% S 0 loop4 706 2 0% 0% 0% S 0 loop3 620 2 0% 0% 0% S 0 loop2 599 2 0% 0% 0% S 0 loop1 553 2 0% 0% 0% S 0 loop0 223 2 0% 0% 0% S 0 kworker/3:1H 222 2 0% 0% 0% S 0 kworker/1:1H 206 2 0% 0% 0% S 0 kworker/2:1H 188 2 0% 0% 0% S 0 kworker/0:1H 106 1 0% 0% 0% S 5607424 systemd-udevd 100 2 0% 0% 0% S 0 kauditd 94 1 0% 0% 0% S 37826560 systemd-journal 81 2 0% 0% 0% S 0 deferwq 70 2 0% 0% 0% S 0 irq/97-octeon-h 69 2 0% 0% 0% S 0 octeon-ethernet 64 2 0% 0% 0% S 0 scsi_eh_0 57 2 0% 0% 0% S 0 nfsiod 56 2 0% 0% 0% S 0 fsnotify_mark 55 2 0% 0% 0% S 0 ksmd 54 2 0% 0% 0% S 0 kswapd0 49 2 0% 0% 0% S 0 rpciod 48 2 0% 0% 0% S 0 edac-poller 47 2 0% 0% 0% S 0 md 46 2 0% 0% 0% S 0 khubd 45 2 0% 0% 0% S 0 ata_sff 44 2 0% 0% 0% S 0 kblockd 43 2 0% 0% 0% S 0 crypto 42 2 0% 0% 0% S 0 bioset 41 2 0% 0% 0% S 0 writeback 40 2 0% 0% 0% S 0 irq/99-octeon-h 39 2 0% 0% 0% S 0 irq/98-octeon-h 38 2 0% 0% 0% S 0 irq/96-octeon-h 37 2 0% 0% 0% S 0 irq/127-octeon- 36 2 0% 0% 0% S 0 irq/95-octeon-h 35 2 0% 0% 0% S 0 irq/93-octeon-h 34 2 0% 0% 0% S 0 irq/92-octeon-h 33 2 0% 0% 0% S 0 irq/91-octeon-h 32 2 0% 0% 0% S 0 irq/94-octeon-h 31 2 0% 0% 0% S 0 irq/90-octeon-h 30 2 0% 0% 0% S 0 irq/89-octeon-h 29 2 0% 0% 0% S 0 irq/60-octeon-h 28 2 0% 0% 0% S 0 netns 27 2 0% 0% 0% S 0 kdevtmpfs 26 2 0% 0% 0% S 0 khelper 25 2 0% 0% 0% S 0 kworker/3:0H 23 2 0% 0% 0% S 0 ksoftirqd/3 22 2 0% 0% 0% S 0 migration/3 21 2 0% 0% 0% S 0 rcuc/3 20 2 0% 0% 0% S 0 kworker/2:0H 18 2 0% 0% 0% S 0 ksoftirqd/2 17 2 0% 0% 0% S 0 migration/2 16 2 0% 0% 0% S 0 rcuc/2 15 2 0% 0% 0% S 0 kworker/1:0H 13 2 0% 0% 0% S 0 ksoftirqd/1 12 2 0% 0% 0% S 0 migration/1 11 2 0% 0% 0% S 0 rcuc/1 10 2 0% 0% 0% S 0 rcuc/0 9 2 0% 0% 0% S 0 rcu_sched 8 2 0% 0% 0% S 0 rcu_bh 7 2 0% 0% 0% S 0 migration/0 5 2 0% 0% 0% S 0 kworker/0:0H 4 2 0% 0% 0% S 0 kworker/0:0 3 2 0% 0% 0% S 0 ksoftirqd/0 2 0 0% 0% 0% S 0 kthreadd 1 0 0% 0% 0% S 8413184 systemd
Thanks a lot!
01-22-2021 03:59 PM
I had a look at the output and, you are right, it really is nothing.
When there is sufficient "load" on the stack, run the command to give a snapshot of the CPU and the memory.
sh platform software status con brief
NOTE: Do not bother with the classic IOS command "sh proc cpu history" because the output provides an "average" of all the cores. The output is very useful for single-core appliances that run classic IOS. For appliances running IOS-XE, it does not provide any useful information as to which CPU core is "hot spinning".
If the Free Memory or Idle CPU drops to, say <30%, run the other two commands. I made a mistake and forgot to mention that the "first page" of the two CPU/Memory commands would suffice.
01-26-2021 04:43 AM
Here's the commands output while CPU cores are on load
#sh platform software status con brief Load Average Slot Status 1-Min 5-Min 15-Min 1-RP0 Healthy 2.96 2.83 2.40 2-RP0 Healthy 0.52 0.30 0.26 Memory (kB) Slot Status Total Used (Pct) Free (Pct) Committed (Pct) 1-RP0 Healthy 3983088 2774200 (70%) 1208888 (30%) 3556028 (89%) 2-RP0 Healthy 3983088 2159008 (54%) 1824080 (46%) 2966520 (74%) CPU Utilization Slot CPU User System Nice Idle IRQ SIRQ IOwait 1-RP0 0 36.20 15.30 0.00 48.40 0.00 0.10 0.00 1 36.53 16.71 0.00 46.74 0.00 0.00 0.00 2 31.33 14.51 0.00 54.15 0.00 0.00 0.00 3 37.76 13.08 0.00 49.05 0.00 0.09 0.00 2-RP0 0 3.10 1.10 0.00 95.70 0.00 0.10 0.00 1 3.39 0.99 0.00 95.50 0.00 0.09 0.00 2 3.29 0.89 0.00 95.70 0.00 0.09 0.00 3 2.99 1.09 0.00 95.90 0.00 0.00 0.00 #show processes cpu platform sorted location switch 1 r0 CPU utilization for five seconds: 46%, one minute: 49%, five minutes: 48% Core 0: CPU utilization for five seconds: 41%, one minute: 51%, five minutes: 49% Core 1: CPU utilization for five seconds: 46%, one minute: 49%, five minutes: 49% Core 2: CPU utilization for five seconds: 45%, one minute: 49%, five minutes: 49% Core 3: CPU utilization for five seconds: 45%, one minute: 47%, five minutes: 48% Pid PPid 5Sec 1Min 5Min Status Size Name --------------------------------------------------------------------------------- 4686 2 4% 5% 5% S 0 lsmpi-rx 4684 2 4% 6% 5% S 0 lsmpi-xmit 31690 30179 1% 1% 1% S 749502464 repm 32656 31341 0% 0% 5% R 892981248 smand 32359 1 0% 0% 0% S 8794112 rotee 31889 1 0% 0% 0% S 8916992 rotee 31783 12036 0% 0% 0% S 6197248 pman.sh 31392 30191 0% 0% 0% S 846893056 fman_fp_image 31341 12036 0% 0% 0% S 6197248 pman.sh 30863 2 0% 0% 0% S 0 kworker/3:1 30741 1 0% 0% 0% S 8916992 rotee 30730 1 0% 0% 0% S 8916992 rotee 30206 30182 0% 0% 0% S 4362240 sntp 30191 15779 0% 0% 0% S 5931008 pman.sh 30182 16724 0% 0% 0% S 4579328 stack_sntp.sh 30179 12036 0% 0% 0% S 6189056 pman.sh 28065 3531 0% 0% 0% S 2842624 inotifywait 27832 15476 0% 0% 0% S 2846720 inotifywait 27824 15476 0% 0% 0% S 10272768 issu_stack.sh 27822 15476 0% 0% 0% S 2846720 inotifywait 27816 15476 0% 0% 0% S 10272768 issu_stack.sh 27813 15476 0% 0% 0% S 2846720 inotifywait 27799 16724 0% 0% 0% S 2846720 inotifywait 25022 24372 0% 0% 0% S 714989568 epc_ws_liaison 24754 1 0% 0% 0% S 8794112 rotee 24372 15779 0% 0% 0% S 5931008 pman.sh 21135 20083 0% 0% 0% S 718180352 stack_mgr 20757 19854 0% 0% 0% S 709263360 tams_proc 20649 1 0% 0% 0% S 8916992 rotee 20579 19478 0% 0% 0% S 5103616 sort_files_by_i 20567 19569 0% 0% 0% S 708685824 tamd_proc 20547 19205 0% 1% 0% S 1433403392 sif_mgr 20323 1 0% 0% 0% S 8794112 rotee 20083 12036 0% 0% 0% S 6197248 pman.sh 20022 1 0% 0% 0% S 8794112 rotee 19974 1 0% 0% 0% S 8794112 rotee 19854 12036 0% 0% 0% S 6193152 pman.sh 19645 1 0% 0% 0% S 8794112 rotee 19626 1 0% 0% 0% S 8916992 rotee 19569 12036 0% 0% 0% S 6193152 pman.sh 19478 12036 0% 0% 0% S 6189056 pman.sh 19412 20579 0% 0% 0% S 2682880 sleep 19406 1 0% 0% 0% S 8790016 rotee 19387 18300 0% 0% 0% S 1082458112 platform_mgr 19365 17723 0% 0% 0% S 2846720 inotifywait 19287 19188 0% 0% 0% S 5693440 iosd_console_at 19205 12036 0% 0% 0% S 6197248 pman.sh 19188 19185 0% 0% 0% S 6287360 bexecute.sh #show processes cpu platform sorted location switch 2 r0 CPU utilization for five seconds: 4%, one minute: 4%, five minutes: 4% Core 0: CPU utilization for five seconds: 4%, one minute: 4%, five minutes: 4% Core 1: CPU utilization for five seconds: 3%, one minute: 4%, five minutes: 4% Core 2: CPU utilization for five seconds: 5%, one minute: 4%, five minutes: 4% Core 3: CPU utilization for five seconds: 4%, one minute: 3%, five minutes: 4% Pid PPid 5Sec 1Min 5Min Status Size Name --------------------------------------------------------------------------------- 10646 10000 3% 3% 3% S 2130436096 linux_iosd-imag 20414 19142 1% 0% 0% S 1433403392 sif_mgr 789 30873 1% 1% 1% S 749408256 repm 32577 2 0% 0% 0% S 0 kworker/2:2 32545 11975 0% 0% 0% S 6197248 pman.sh 32390 32289 0% 0% 0% S 6250496 btelnet.sh 32289 30799 0% 0% 0% S 4575232 bexec.sh 31890 1 0% 0% 0% S 8916992 rotee 31661 29478 0% 0% 0% S 715309056 plogd 31532 29501 0% 0% 0% S 846295040 fman_fp_image 30873 11975 0% 0% 0% S 6189056 pman.sh 30799 29767 0% 0% 0% S 6275072 brelay.sh 30723 1 0% 0% 0% S 9830400 rotee 30480 1 0% 0% 0% S 8916992 rotee 30391 1 0% 0% 0% S 8794112 rotee 29767 1 0% 0% 0% S 6107136 brelay_console. 29638 1 0% 0% 0% S 9830400 rotee 29501 15680 0% 0% 0% S 5931008 pman.sh 29478 11975 0% 0% 0% S 6189056 pman.sh 29180 2 0% 0% 0% S 0 kworker/0:2 27208 2 0% 0% 0% S 0 kworker/1:2 26462 15400 0% 0% 0% S 2846720 inotifywait 26452 15400 0% 0% 0% S 10272768 issu_stack.sh 26450 15400 0% 0% 0% S 2846720 inotifywait 26443 15400 0% 0% 0% S 10272768 issu_stack.sh 26442 15400 0% 0% 0% S 2846720 inotifywait 26376 16629 0% 0% 0% S 2846720 inotifywait 26298 3459 0% 0% 0% S 2842624 inotifywait 24602 23956 0% 0% 0% S 714915840 epc_ws_liaison 24334 1 0% 0% 0% S 8794112 rotee 23956 15680 0% 0% 0% S 5931008 pman.sh 22455 17869 0% 0% 0% S 2846720 inotifywait 22423 20507 0% 0% 0% S 2682880 sleep 22232 18935 0% 0% 0% S 2682880 sleep 22002 14140 0% 0% 0% S 2846720 inotifywait 21127 2 0% 0% 0% S 0 kworker/2:1 20969 20048 0% 0% 0% S 718155776 stack_mgr 20597 1 0% 0% 0% S 8916992 rotee 20507 19426 0% 0% 0% S 5103616 sort_files_by_i 20048 11975 0% 0% 0% S 6197248 pman.sh 19947 1 0% 0% 0% S 8794112 rotee 19571 1 0% 0% 0% S 8794112 rotee 19556 1 0% 0% 0% S 8916992 rotee 19426 11975 0% 0% 0% S 6189056 pman.sh 19383 18195 0% 0% 0% S 1080860672 platform_mgr 19142 11975 0% 0% 0% S 6197248 pman.sh 18935 17545 0% 0% 0% S 9072640 periodic.sh 18786 1 0% 0% 0% S 8916992 rotee
Thanks for your note on "sh proc cpu history" command, I'll consider to switch from AvgBusy1 SNMP OID monitoring to more relevant one.
01-26-2021 01:51 PM
Other than the CPU average going >30%, I do not see anything suspicious.
Here is another command to check the memory:
sh processes memory platform sorted location switch <1 - 8> r0
02-12-2021 06:54 AM
Posting an update:
During the past weeks we decided to upgrade sw to 16.09.06 to understand if that's a bug or not.
It seems that High CPU utilization events are happening little less often than before, but we still observing them sometimes.
What's strange for me is that when I see alerts that Prometheus is unable to scrape SNMP metrics from the switch, 'sh proc cpu history' shows 90+% CPU usage but 'show processes cpu platform sorted location switch 2 r0' shows ~40% on all cores.
So why it's not answering to SNMP if CPU cores load only on ~40%. It seems like that sh proc cpu history for some reason is more precise for us, somehow.
However, pasting last outputs when the issue was encountered
#show processes cpu platform sorted location switch 2 r0 CPU utilization for five seconds: 47%, one minute: 46%, five minutes: 46% Core 0: CPU utilization for five seconds: 44%, one minute: 42%, five minutes: 44% Core 1: CPU utilization for five seconds: 49%, one minute: 47%, five minutes: 47% Core 2: CPU utilization for five seconds: 43%, one minute: 46%, five minutes: 47% Core 3: CPU utilization for five seconds: 45%, one minute: 46%, five minutes: 47% Pid PPid 5Sec 1Min 5Min Status Size Name -------------------------------------------------------------------------------- 12080 8383 95% 92% 93% R 2178207744 linux_iosd-imag 12480 10330 79% 82% 84% S 1836642304 fed main event 4436 2 6% 6% 6% S 0 lsmpi-xmit 4437 2 5% 5% 5% S 0 lsmpi-rx 4435 2 1% 1% 1% S 0 lsmpi-refill 21 2 1% 1% 1% S 0 rcuc/3 16 2 1% 1% 1% S 0 rcuc/2 29266 11642 0% 0% 0% S 2908160 sleep 28993 2 0% 0% 0% S 0 kworker/u8:0 28992 9810 0% 0% 0% S 2908160 sleep 27742 2 0% 0% 0% S 0 kworker/u8:2 25677 25673 0% 0% 0% S 180260864 python2.7 25673 25375 0% 0% 0% S 5136384 rdope.sh 25492 1 0% 0% 0% S 9498624 rotee 25375 1 0% 0% 0% S 5943296 pman.sh 23052 22113 0% 0% 0% S 311644160 fman_rp 22309 1 0% 0% 0% S 9498624 rotee 22293 21747 0% 0% 0% S 254238720 dbm 22113 6431 0% 0% 0% S 5951488 pman.sh 21950 1 0% 0% 0% S 9498624 rotee 21873 21377 0% 0% 0% S 42532864 cmm 21747 6431 0% 0% 0% S 5951488 pman.sh 21725 19718 0% 0% 0% S 342634496 sessmgrd 21598 20970 0% 0% 0% S 505753600 cli_agent 21565 1 0% 0% 0% S 9498624 rotee 21377 6431 0% 0% 0% S 5951488 pman.sh 21216 1 0% 0% 0% S 9498624 rotee 20970 6431 0% 0% 0% S 5951488 pman.sh 20799 20370 0% 0% 0% S 53084160 tms 20527 1 0% 0% 0% S 9498624 rotee 20370 6431 0% 0% 0% S 5951488 pman.sh 20262 19962 0% 0% 0% S 235200512 smand 20072 1 0% 0% 0% S 9498624 rotee 19962 6431 0% 0% 0% S 5951488 pman.sh 19857 19471 0% 0% 0% S 148779008 repm 19850 1 0% 0% 0% S 9498624 rotee 19718 6431 0% 0% 0% S 5951488 pman.sh 19610 1 0% 0% 0% S 9498624 rotee 19506 18957 0% 0% 0% S 55529472 psd 19471 6431 0% 0% 0% S 5951488 pman.sh 19426 2 0% 0% 0% S 0 kworker/u8:1 19326 1 0% 0% 0% S 9498624 rotee 19250 19163 0% 0% 0% S 10207232 journalctl 19163 18501 0% 0% 0% S 39899136 plogd 18957 6431 0% 0% 0% S 5951488 pman.sh 18760 1 0% 0% 0% S 9498624 rotee
#sh platform software status con brief Load Average Slot Status 1-Min 5-Min 15-Min 2-RP0 Healthy 3.11 2.81 2.55 Memory (kB) Slot Status Total Used (Pct) Free (Pct) Committed (Pct) 2-RP0 Healthy 3977740 1660184 (42%) 2317556 (58%) 2519316 (63%) CPU Utilization Slot CPU User System Nice Idle IRQ SIRQ IOwait 2-RP0 0 34.43 9.90 0.00 55.65 0.00 0.00 0.00 1 38.26 8.19 0.00 53.54 0.00 0.00 0.00 2 35.90 6.20 0.00 57.90 0.00 0.00 0.00 3 41.25 9.39 0.00 49.35 0.00 0.00 0.00
#sh processes memory platform sorted location switch 2 r0 System memory: 3977740K total, 1660932K used, 2316808K free, Lowest: 2316808K Pid Text Data Stack Dynamic RSS Total Name -------------------------------------------------------------------------------- 12080 171507 629616 0 296 629616 2127156 linux_iosd-imag 12480 206 162188 132 71832 162188 1793596 fed main event 12003 396 57144 132 2536 57144 778160 sif_mgr 21598 344 48244 132 6264 48244 493900 cli_agent 12105 1004 60212 132 2328 60212 448596 platform_mgr 21725 178 71328 132 7364 71328 334604 sessmgrd 23052 8562 63592 132 2564 63592 304340 fman_rp 18550 9018 59584 132 10116 59584 290648 fman_fp_image 22293 197 53588 132 4168 53588 248280 dbm 20262 799 72468 132 24580 72468 230948 smand 3714 545 4428 132 132 4428 214820 libvirtd 25677 8 25280 136 7380 25280 176036 python2.7 19857 578 27248 132 1872 27248 145292 repm 7498 99 13348 132 1416 13348 63896 bt_logger 9083 200 12424 132 1800 12424 58352 lman 11926 483 13036 132 1428 13036 56088 stack_mgr 19506 100 13564 132 1444 13564 54228 psd 20799 305 15084 132 6704 15084 51840 tms 8031 589 13828 352 2336 13828 51780 hman 7741 166 10260 132 1464 10260 46448 btman 10612 166 10024 132 1380 10024 45980 btman 9564 230 6420 132 272 6420 45688 nif_mgr 8491 138 10004 132 1232 10004 45580 keyman 21873 170 7612 132 1108 7612 41536 cmm 19163 112 8272 132 1848 8272 38964 plogd 15728 119 7440 132 1028 7440 38204 epc_ws_liaison 14269 88 6984 136 1464 6984 28816 tams_proc 14169 126 6576 136 868 6576 28308 tamd_proc 13982 76 5860 136 680 5860 27224 tam_svcs_ng3k_c 3705 687 1968 132 132 1968 17784 virtlogd 7249 68 2532 136 132 2532 13196 auto_upgrade_se 6992 1096 8576 244 6568 8576 11560 auto_upgrade_cl 9200 1096 8592 244 6560 8592 11552 ncd.sh 16126 1096 7572 244 6412 7572 11404 issu_stack.sh 16122 1096 7472 244 6412 7472 11404 issu_stack.sh 8240 1096 8472 244 6412 8472 11404 issu_stack.sh 19250 509 2116 136 1204 2116 9968 journalctl 4312 68 1996 132 252 1996 9596 rotee 3997 68 1996 132 252 1996 9596 rotee 9810 1096 6436 248 4460 6436 9456 periodic.sh 6173 1096 6420 244 4436 6420 9428 rollback_timer. 6475 68 1996 132 252 1996 9392 rotee 25492 68 1972 136 132 1972 9276 rotee 22309 68 1980 136 132 1980 9276 rotee 21950 68 1980 136 132 1980 9276 rotee 21565 68 1980 136 132 1980 9276 rotee 21216 68 1980 136 132 1980 9276 rotee 20527 68 1980 136 132 1980 9276 rotee 20072 68 1980 136 132 1980 9276 rotee
sw-c3650-6-4#show platform software status control-processor brief Load Average Slot Status 1-Min 5-Min 15-Min 2-RP0 Healthy 2.35 2.76 2.62 Memory (kB) Slot Status Total Used (Pct) Free (Pct) Committed (Pct) 2-RP0 Healthy 3977740 1660740 (42%) 2317000 (58%) 2519276 (63%) CPU Utilization Slot CPU User System Nice Idle IRQ SIRQ IOwait 2-RP0 0 33.96 11.88 0.00 54.14 0.00 0.00 0.00 1 39.20 7.40 0.00 53.30 0.00 0.10 0.00 2 31.30 6.50 0.00 62.20 0.00 0.00 0.00 3 37.16 8.19 0.00 54.64 0.00 0.00 0.00
Any ideas?
Thanks
02-12-2021 08:17 AM
Hello,
post the full running configuration (sh run) of the 3650 switch...
02-12-2021 03:32 PM
@KirillKO wrote:
12080 8383 95% 92% 93% R 2178207744 linux_iosd-imag 12480 10330 79% 82% 84% S 1836642304 fed main event
Oh, FFS! Not this one again!
Yes, I know of these two "clowns". FED = Forwarding Engine Driver.
Can you have a look at Troubleshoot High CPU Usage in Catalyst Switch Platforms Running IOS-XE 16.x?
There is a very big chance that you may need to downgrade to 16.3.X.
02-17-2021 11:44 PM
Hello, Leo.
@Leo Laohoo wrote:
@KirillKO wrote:12080 8383 95% 92% 93% R 2178207744 linux_iosd-imag 12480 10330 79% 82% 84% S 1836642304 fed main eventOh, FFS! Not this one again!
Yes, I know of these two "clowns". FED = Forwarding Engine Driver.
Can you have a look at Troubleshoot High CPU Usage in Catalyst Switch Platforms Running IOS-XE 16.x?
There is a very big chance that you may need to downgrade to 16.3.X.
Could you please clarify why it's the case?
Looked into the link you provided, will try to follow the steps from it, next time it appears.
Thanks a lot!
02-18-2021 02:58 AM
@KirillKO wrote:
Could you please clarify why it's the case?
Because this bug is not present/reported in 16.3.X.
03-05-2021 07:33 AM
Thanks for the info, Leo.
As a quick update, we went through "Troubleshooting High CPU usage" manual, and encountered some kind of ARP flooding to our core switch. Presumably only from Apple Watch devices, but not yet 100% confirmed as we still collecting packets capture.
Also, found this bug report https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvi87866/?rfs=iqvred but it stated that its fixed for our current firmware.
Did you heard smth relevant about Apple Watch devices?
Thanks
03-05-2021 03:07 PM
Apple Watches talk BLE. I do not see any relations between BLE and this switch.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide