Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
- Cisco Community
- Technology and Support
- Networking
- Routing
- Re: Catalyst 3650 high cpu utilization
Options
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
7264
Views
5
Helpful
17
Replies
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-22-2021 02:29 AM
Hello, Cisco Community!
We've encountered high CPU utilization issue on our 2x3650 stack.
Processes eating CPU are: ARP Input, SISF Switcher Th, SISF Main Thread
Issue is happening only on working hours, so likely that connected to user activity.
We tried disabling ip dhcp snooping on user VLANs and adding deny all rule for ipv6 but nothing helps.
Switch firmware is 16.6.6, haven't seen any related bugs for this version.
Do you have any ideas what else to check?
Thanks a lot!
Solved! Go to Solution.
Labels:
1 Accepted Solution
Accepted Solutions
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-18-2021 04:46 AM
As a conclusion, we've found the root cause description here https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvw92906/?rfs=iqvred
Packet captures showed us that CPU utilization is getting high when the ARP flood was happening and luckily we were able to identify Apple Watch devices having this behaviour from time to time while connected to the office Wi-Fi.
Hopefully, after we decided to upgrade WLC software to the latest 8.10.151.0 version the issue hasn't been encountered again(it's almost 3 weeks after the upgrade).
Thanks to all the community members who helped us here!
17 Replies 17
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-22-2021 02:51 AM
check this thread may be bug :
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-22-2021 08:37 AM
Hello,
thanks for pointing to that, I'll take a look.
Thanks
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-22-2021 02:52 AM
@KirillKO wrote:
Switch firmware is 16.6.6, haven't seen any related bugs for this version.
There are, a minimum of, seven bugs attributed for "SISF Main Thread".
Can you post the complete output to the following commands:
sh platform software status con brief sh version
show processes cpu platform sorted location switch <1 & 2> r0
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-22-2021 08:34 AM
Hello, thanks for the prompt answer!
See comands output below, however output might be irrelevant cause of non-working hours as I described earlier
#sh platform software status con brief
Load Average
Slot Status 1-Min 5-Min 15-Min
1-RP0 Healthy 1.42 1.67 1.63
2-RP0 Healthy 0.24 0.21 0.23
Memory (kB)
Slot Status Total Used (Pct) Free (Pct) Committed (Pct)
1-RP0 Healthy 3983088 2769832 (70%) 1213256 (30%) 3552132 (89%)
2-RP0 Healthy 3983088 2189824 (55%) 1793264 (45%) 2981732 (75%)
CPU Utilization
Slot CPU User System Nice Idle IRQ SIRQ IOwait
1-RP0 0 24.70 10.60 0.00 64.50 0.00 0.20 0.00
1 35.40 8.40 0.00 56.20 0.00 0.00 0.00
2 31.40 10.90 0.00 57.70 0.00 0.00 0.00
3 34.30 7.80 0.00 57.80 0.00 0.10 0.00
2-RP0 0 3.00 0.80 0.00 96.20 0.00 0.00 0.00
1 4.30 0.60 0.00 95.10 0.00 0.00 0.00
2 2.80 0.80 0.00 96.40 0.00 0.00 0.00
3 2.60 1.10 0.00 96.29 0.00 0.00 0.00#sh ver
Cisco IOS XE Software, Version 16.06.06
Cisco IOS Software [Everest], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.6.6, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2019 by Cisco Systems, Inc.
Compiled Thu 11-Apr-19 02:24 by mcpre
ROM: IOS-XE ROMMON
BOOTLDR: CAT3K_CAA Boot Loader (CAT3K_CAA-HBOOT-M) Version 3.56, RELEASE SOFTWARE (P)
sw-c3650 uptime is 1 year, 34 weeks, 5 days, 55 minutes
Uptime for this control processor is 1 year, 34 weeks, 5 days, 1 hour, 0 minutes
System returned to ROM by Reload Command
System image file is "flash:packages.conf"
Last reload reason: Reload Command
Technology Package License Information:
-----------------------------------------------------------------
Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------
ipservicesk9 Permanent ipservicesk9
cisco WS-C3650-24TD (MIPS) processor (revision N0) with 852917K/6147K bytes of memory.
Processor board ID #########
11 Virtual Ethernet interfaces
52 Gigabit Ethernet interfaces
4 Ten Gigabit Ethernet interfaces
2048K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
250456K bytes of Crash Files at crashinfo:.
1609272K bytes of Flash at flash:.
0K bytes of WebUI ODM Files at webui:.
250456K bytes of Crash Files at crashinfo-2:.
1609272K bytes of Flash at flash-2:.
Base Ethernet MAC Address : xx:xx:xx:xx:xx:xx
Motherboard Assembly Number : xx-xxxxx-xx
Motherboard Serial Number : XXXX
Model Revision Number : N0
Motherboard Revision Number : B0
Model Number : WS-C3650-24TD
System Serial Number : XXXX
Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 28 WS-C3650-24TD 16.6.6 CAT3K_CAA-UNIVERSALK9 INSTALL
2 28 WS-C3650-24TD 16.6.6 CAT3K_CAA-UNIVERSALK9 INSTALL
Switch 02
---------
Switch uptime : 1 year, 32 weeks, 4 hours, 29 minutes
Base Ethernet MAC Address : xx:xx:xx:xx:xx:xx
Motherboard Assembly Number : xx-xxxxx-xx
Motherboard Serial Number : XXXX
Model Revision Number : K0
Motherboard Revision Number : B0
Model Number : WS-C3650-24TD
System Serial Number : XXXX
Configuration register is 0x102#show processes cpu platform sorted location switch 1 r0
CPU utilization for five seconds: 9%, one minute: 9%, five minutes: 9%
Core 0: CPU utilization for five seconds: 9%, one minute: 9%, five minutes: 10%
Core 1: CPU utilization for five seconds: 15%, one minute: 9%, five minutes: 9%
Core 2: CPU utilization for five seconds: 10%, one minute: 9%, five minutes: 9%
Core 3: CPU utilization for five seconds: 9%, one minute: 9%, five minutes: 9%
Pid PPid 5Sec 1Min 5Min Status Size Name
---------------------------------------------------------------------------------
31690 30179 1% 1% 1% S 749502464 repm
20547 19205 1% 0% 0% S 1433403392 sif_mgr
32656 31341 0% 0% 0% S 892858368 smand
32359 1 0% 0% 0% S 8794112 rotee
31889 1 0% 0% 0% S 8916992 rotee
31783 12036 0% 0% 0% S 6197248 pman.sh
31392 30191 0% 0% 0% S 846893056 fman_fp_image
31341 12036 0% 0% 0% S 6197248 pman.sh
30863 2 0% 0% 0% S 0 kworker/3:1
30741 1 0% 0% 0% S 8916992 rotee
30730 1 0% 0% 0% S 8916992 rotee
30206 30182 0% 0% 0% S 4362240 sntp
30191 15779 0% 0% 0% S 5931008 pman.sh
30182 16724 0% 0% 0% S 4579328 stack_sntp.sh
30179 12036 0% 0% 0% S 6189056 pman.sh
28065 3531 0% 0% 0% S 2842624 inotifywait
27832 15476 0% 0% 0% S 2846720 inotifywait
27824 15476 0% 0% 0% S 10272768 issu_stack.sh
27822 15476 0% 0% 0% S 2846720 inotifywait
27816 15476 0% 0% 0% S 10272768 issu_stack.sh
27813 15476 0% 0% 0% S 2846720 inotifywait
27799 16724 0% 0% 0% S 2846720 inotifywait
26778 20579 0% 0% 0% S 2682880 sleep
26666 14169 0% 0% 0% S 2846720 inotifywait
26568 19001 0% 0% 0% S 2682880 sleep
26255 17723 0% 0% 0% S 2846720 inotifywait
25272 2 0% 0% 0% S 0 kworker/u8:1
25022 24372 0% 0% 0% S 714989568 epc_ws_liaison
24754 1 0% 0% 0% S 8794112 rotee
24372 15779 0% 0% 0% S 5931008 pman.sh
23730 2 0% 0% 0% S 0 kworker/u8:0
21135 20083 0% 0% 0% S 718180352 stack_mgr
20757 19854 0% 0% 0% S 709263360 tams_proc
20649 1 0% 0% 0% S 8916992 rotee
20579 19478 0% 0% 0% S 5103616 sort_files_by_i
20567 19569 0% 0% 0% S 708685824 tamd_proc
20323 1 0% 0% 0% S 8794112 rotee
20083 12036 0% 0% 0% S 6197248 pman.sh
20022 1 0% 0% 0% S 8794112 rotee
19974 1 0% 0% 0% S 8794112 rotee
19854 12036 0% 0% 0% S 6193152 pman.sh
19645 1 0% 0% 0% S 8794112 rotee
19626 1 0% 0% 0% S 8916992 rotee
19569 12036 0% 0% 0% S 6193152 pman.sh
19478 12036 0% 0% 0% S 6189056 pman.sh
19406 1 0% 0% 0% S 8790016 rotee
19387 18300 0% 0% 0% S 1082458112 platform_mgr
19287 19188 0% 0% 0% S 5693440 iosd_console_at
19205 12036 0% 0% 0% S 6197248 pman.sh
19188 19185 0% 0% 0% S 6287360 bexecute.sh
19185 19019 0% 0% 0% S 4571136 bexec.sh
19019 19018 0% 0% 0% S 6270976 brelay.sh
19018 18850 0% 0% 0% S 14503936 in.telnetd
19001 17711 0% 0% 0% S 9072640 periodic.sh
18852 1 0% 0% 0% S 8916992 rotee
18850 10621 0% 0% 0% S 6270976 in.telnetd.sh
18753 17036 0% 0% 0% S 2429480960 fed main event
18660 17010 0% 0% 0% S 718204928 nif_mgr
18537 1 0% 0% 0% S 8916992 rotee
18457 1 0% 0% 0% S 8794112 rotee
18300 12036 0% 0% 0% S 6189056 pman.sh
18157 1 0% 0% 0% S 8916992 rotee
17801 1 0% 0% 0% S 8916992 rotee
17756 1 0% 0% 0% S 8794112 rotee
17723 16725 0% 0% 0% S 6905856 btrace_rotate.s
17711 12036 0% 0% 0% S 6193152 pman.sh
17241 1 0% 0% 0% S 8916992 rotee
17036 15779 0% 0% 0% S 5926912 pman.sh
17010 12036 0% 0% 0% S 6193152 pman.sh
16725 15779 0% 0% 0% S 5926912 pman.sh
16724 15573 0% 0% 0% S 10412032 ncd.sh
16500 1 0% 0% 0% S 8916992 rotee
16478 15779 0% 0% 0% S 2842624 inotifywait
16423 15172 0% 0% 0% S 717787136 lman
16386 1 0% 0% 0% S 8912896 rotee
16157 1 0% 0% 0% S 8916992 rotee
15931 14827 0% 0% 0% S 709464064 keyman
15779 1 0% 0% 0% S 6750208 pvp.sh
15666 1 0% 0% 0% S 8916992 rotee
15573 12036 0% 0% 0% S 6193152 pman.sh
15476 14507 0% 0% 0% S 10407936 issu_stack.sh
15225 1 0% 0% 0% S 8794112 rotee
15172 12036 0% 0% 0% S 6189056 pman.sh
15062 13934 0% 0% 0% R 722673664 hman
14936 1 0% 0% 0% S 8916992 rotee
14827 12036 0% 0% 0% S 6189056 pman.sh
14715 1 0% 0% 0% S 8916992 rotee
14507 12036 0% 0% 0% S 6193152 pman.sh
14428 1 0% 0% 0% S 8916992 rotee
14211 1 0% 0% 0% S 8794112 rotee
14169 13132 0% 0% 0% S 6893568 btrace_rotate.s
13948 12771 0% 0% 0% S 727834624 bt_logger
13934 12036 0% 0% 0% S 6189056 pman.sh
13665 1 0% 0% 0% S 8916992 rotee
13617 12505 0% 0% 0% S 16986112 auto_upgrade_se
13264 1 0% 0% 0% S 8794112 rotee
13192 12327 0% 0% 0% S 10141696 auto_upgrade_cl
13132 12036 0% 0% 0% S 6189056 pman.sh
12929 1 0% 0% 0% S 8794112 rotee
12771 12036 0% 0% 0% S 6197248 pman.sh
12612 1 0% 0% 0% S 8794112 rotee
12505 12036 0% 0% 0% S 6189056 pman.sh
12327 12036 0% 0% 0% S 6189056 pman.sh
12225 12036 0% 0% 0% S 2842624 inotifywait
12154 1 0% 0% 0% S 8912896 rotee
12036 1 0% 0% 0% S 7241728 pvp.sh
12031 11914 0% 0% 0% S 2842624 inotifywait
12024 1 0% 0% 0% S 8912896 rotee
11914 1 0% 0% 0% S 6877184 psvp.sh
11336 10629 0% 0% 0% S 2842624 inotifywait
11314 1 0% 0% 0% S 8790016 rotee
10629 1 0% 0% 0% S 7999488 rollback_timer.
10621 1 0% 0% 0% S 3452928 xinetd
10619 1 0% 0% 0% S 3452928 xinetd
8714 13192 0% 0% 0% S 2846720 inotifywait
7939 2 0% 0% 0% S 0 kworker/3:2
6139 5333 0% 0% 0% S 718807040 psd
5934 5153 0% 0% 0% S 715366400 plogd
5747 1 0% 0% 0% S 8794112 rotee
5444 1 0% 0% 0% S 8794112 rotee
5443 1 0% 0% 0% S 3452928 xinetd
5333 12036 0% 0% 0% S 6189056 pman.sh
5153 12036 0% 0% 0% S 6201344 pman.sh
4820 3951 0% 0% 0% S 2829021184 linux_iosd-imag
4686 2 0% 0% 0% S 0 lsmpi-rx
4684 2 0% 0% 0% S 0 lsmpi-xmit
4683 2 0% 0% 0% S 0 lsmpi-refill
4450 4126 0% 0% 0% S 2842624 inotifywait
4426 1 0% 0% 0% S 8790016 rotee
4226 1 0% 0% 0% S 8916992 rotee
4126 1 0% 0% 0% S 5812224 iptbl.sh
4098 3534 0% 0% 0% S 2842624 inotifywait
4085 1 0% 0% 0% S 8790016 rotee
4051 1 0% 0% 0% S 8790016 rotee
3964 3530 0% 0% 0% S 2674688 sleep
3951 12036 0% 0% 0% S 6197248 pman.sh
3945 1 0% 0% 0% S 8790016 rotee
3808 3022 0% 0% 0% S 751104000 fman_rp
3654 1 0% 0% 0% S 5607424 rpc.mountd
3614 2 0% 0% 0% S 0 nfsd
3613 2 0% 0% 0% S 0 nfsd
3612 2 0% 0% 0% S 0 nfsd
3611 2 0% 0% 0% S 0 nfsd
3610 2 0% 0% 0% S 0 nfsd
3609 2 0% 0% 0% S 0 nfsd
3608 2 0% 0% 0% S 0 nfsd
3607 2 0% 0% 0% S 0 nfsd
3606 2 0% 0% 0% S 0 lockd
3570 1 0% 0% 0% S 3936256 rpc.statd
3562 3549 0% 0% 0% S 2842624 inotifywait
3554 3535 0% 0% 0% S 217464832 libvirtd
3549 1 0% 0% 0% S 4575232 boothelper_evt.
3535 1 0% 0% 0% S 4575232 libvirtd.sh
3534 1 0% 0% 0% S 6959104 droputil.sh
3531 1 0% 0% 0% S 6971392 reflector.sh
3530 1 0% 0% 0% S 4980736 oom.sh
3524 1 0% 0% 0% S 17580032 virtlogd
3522 1 0% 0% 0% S 6324224 rpcbind
3521 1 0% 0% 0% S 5267456 klogd
3520 1 0% 0% 0% S 3452928 xinetd
3496 2486 0% 0% 0% S 884219904 dbm
3395 1 0% 0% 0% S 8916992 rotee
3022 12036 0% 0% 0% S 6189056 pman.sh
3009 2052 0% 0% 0% S 717688832 cmm
2975 1 0% 0% 0% S 8916992 rotee
2833 1872 0% 0% 0% S 1145745408 cli_agent
2486 12036 0% 0% 0% S 6189056 pman.sh
2463 1 0% 0% 0% S 8916992 rotee
2148 1 0% 0% 0% S 8916992 rotee
2052 12036 0% 0% 0% S 6189056 pman.sh
1872 12036 0% 0% 0% S 6189056 pman.sh
1608 1021 0% 0% 0% S 730210304 tms
1275 1 0% 0% 0% S 8916992 rotee
1021 12036 0% 0% 0% S 6189056 pman.sh
899 2 0% 0% 0% S 0 loop7
878 2 0% 0% 0% S 0 loop6
813 2 0% 0% 0% S 0 loop5
795 31783 0% 0% 0% S 793092096 smd
783 2 0% 0% 0% S 0 loop4
762 2 0% 0% 0% S 0 loop3
676 2 0% 0% 0% S 0 loop2
655 2 0% 0% 0% S 0 loop1
609 2 0% 0% 0% S 0 loop0
268 2 0% 0% 0% S 0 kworker/2:2
266 2 0% 0% 0% S 0 kworker/2:1H
197 2 0% 0% 0% S 0 kworker/1:1H
196 2 0% 0% 0% S 0 kworker/0:1H
195 2 0% 0% 0% S 0 kworker/3:1H
136 2 0% 0% 0% S 0 kworker/0:2
104 1 0% 0% 0% S 5742592 systemd-udevd
99 2 0% 0% 0% S 0 kauditd
92 2 0% 0% 0% S 0 kworker/1:2
88 1 0% 0% 0% S 12726272 systemd-journal
81 2 0% 0% 0% S 0 deferwq
70 2 0% 0% 0% S 0 irq/97-octeon-h
69 2 0% 0% 0% S 0 octeon-ethernet
64 2 0% 0% 0% S 0 scsi_eh_0
57 2 0% 0% 0% S 0 nfsiod
56 2 0% 0% 0% S 0 fsnotify_mark
55 2 0% 0% 0% S 0 ksmd
54 2 0% 0% 0% S 0 kswapd0
51 2 0% 0% 0% S 0 kworker/0:1
50 2 0% 0% 0% S 0 kworker/1:1
49 2 0% 0% 0% S 0 rpciod
48 2 0% 0% 0% S 0 edac-poller
47 2 0% 0% 0% S 0 md
46 2 0% 0% 0% S 0 khubd
45 2 0% 0% 0% S 0 ata_sff
44 2 0% 0% 0% S 0 kblockd
43 2 0% 0% 0% S 0 crypto
42 2 0% 0% 0% S 0 bioset
41 2 0% 0% 0% S 0 writeback
40 2 0% 0% 0% S 0 irq/99-octeon-h
39 2 0% 0% 0% S 0 irq/98-octeon-h
38 2 0% 0% 0% S 0 irq/96-octeon-h
37 2 0% 0% 0% S 0 irq/127-octeon-
36 2 0% 0% 0% S 0 irq/95-octeon-h
35 2 0% 0% 0% S 0 irq/93-octeon-h
34 2 0% 0% 0% S 0 irq/92-octeon-h
33 2 0% 0% 0% S 0 irq/91-octeon-h
32 2 0% 0% 0% S 0 irq/94-octeon-h
31 2 0% 0% 0% S 0 irq/90-octeon-h
30 2 0% 0% 0% S 0 irq/89-octeon-h
29 2 0% 0% 0% S 0 irq/60-octeon-h
28 2 0% 0% 0% S 0 netns
27 2 0% 0% 0% S 0 kdevtmpfs
26 2 0% 0% 0% S 0 khelper
25 2 0% 0% 0% S 0 kworker/3:0H
23 2 0% 0% 0% S 0 ksoftirqd/3
22 2 0% 0% 0% S 0 migration/3
21 2 0% 0% 0% S 0 rcuc/3
20 2 0% 0% 0% S 0 kworker/2:0H
19 2 0% 0% 0% S 0 kworker/2:0
18 2 0% 0% 0% S 0 ksoftirqd/2
17 2 0% 0% 0% S 0 migration/2
16 2 0% 0% 0% S 0 rcuc/2
15 2 0% 0% 0% S 0 kworker/1:0H
13 2 0% 0% 0% S 0 ksoftirqd/1
12 2 0% 0% 0% S 0 migration/1
11 2 0% 0% 0% S 0 rcuc/1
10 2 0% 0% 0% S 0 rcuc/0
9 2 0% 0% 0% S 0 rcu_sched
8 2 0% 0% 0% S 0 rcu_bh
7 2 0% 0% 0% S 0 migration/0
5 2 0% 0% 0% S 0 kworker/0:0H
3 2 0% 0% 0% S 0 ksoftirqd/0
2 0 0% 0% 0% S 0 kthreadd
1 0 0% 0% 0% S 8114176 systemd
#show processes cpu platform sorted location switch 2 r0
CPU utilization for five seconds: 3%, one minute: 4%, five minutes: 4%
Core 0: CPU utilization for five seconds: 3%, one minute: 3%, five minutes: 3%
Core 1: CPU utilization for five seconds: 3%, one minute: 3%, five minutes: 4%
Core 2: CPU utilization for five seconds: 4%, one minute: 4%, five minutes: 4%
Core 3: CPU utilization for five seconds: 3%, one minute: 4%, five minutes: 4%
Pid PPid 5Sec 1Min 5Min Status Size Name
---------------------------------------------------------------------------------
10646 10000 2% 3% 3% S 2130436096 linux_iosd-imag
20414 19142 1% 0% 0% S 1433403392 sif_mgr
789 30873 1% 1% 1% S 749408256 repm
32577 2 0% 0% 0% S 0 kworker/2:2
32545 11975 0% 0% 0% S 6197248 pman.sh
32390 32289 0% 0% 0% S 6250496 btelnet.sh
32289 30799 0% 0% 0% S 4575232 bexec.sh
31890 1 0% 0% 0% S 8916992 rotee
31661 29478 0% 0% 0% S 715309056 plogd
31532 29501 0% 0% 0% S 846295040 fman_fp_image
30873 11975 0% 0% 0% S 6189056 pman.sh
30799 29767 0% 0% 0% S 6275072 brelay.sh
30723 1 0% 0% 0% S 9830400 rotee
30480 1 0% 0% 0% S 8916992 rotee
30391 1 0% 0% 0% S 8794112 rotee
29767 1 0% 0% 0% S 6107136 brelay_console.
29638 1 0% 0% 0% S 9830400 rotee
29501 15680 0% 0% 0% S 5931008 pman.sh
29478 11975 0% 0% 0% S 6189056 pman.sh
29180 2 0% 0% 0% S 0 kworker/0:2
27208 2 0% 0% 0% S 0 kworker/1:2
26462 15400 0% 0% 0% S 2846720 inotifywait
26452 15400 0% 0% 0% S 10272768 issu_stack.sh
26450 15400 0% 0% 0% S 2846720 inotifywait
26443 15400 0% 0% 0% S 10272768 issu_stack.sh
26442 15400 0% 0% 0% S 2846720 inotifywait
26376 16629 0% 0% 0% S 2846720 inotifywait
26298 3459 0% 0% 0% S 2842624 inotifywait
24602 23956 0% 0% 0% S 714915840 epc_ws_liaison
24334 1 0% 0% 0% S 8794112 rotee
23956 15680 0% 0% 0% S 5931008 pman.sh
21127 2 0% 0% 0% S 0 kworker/2:1
20969 20048 0% 0% 0% S 718155776 stack_mgr
20597 1 0% 0% 0% S 8916992 rotee
20507 19426 0% 0% 0% S 5103616 sort_files_by_i
20048 11975 0% 0% 0% S 6197248 pman.sh
19947 1 0% 0% 0% S 8794112 rotee
19571 1 0% 0% 0% S 8794112 rotee
19556 1 0% 0% 0% S 8916992 rotee
19426 11975 0% 0% 0% S 6189056 pman.sh
19383 18195 0% 0% 0% S 1080860672 platform_mgr
19142 11975 0% 0% 0% S 6197248 pman.sh
18935 17545 0% 0% 0% S 9072640 periodic.sh
18786 1 0% 0% 0% S 8916992 rotee
18744 17022 0% 0% 0% S 2429480960 fed main event
18641 1 0% 0% 0% S 8916992 rotee
18600 16888 0% 0% 0% S 718204928 nif_mgr
18377 1 0% 0% 0% S 8794112 rotee
18195 11975 0% 0% 0% S 6189056 pman.sh
18088 1 0% 0% 0% S 8916992 rotee
17869 16733 0% 0% 0% S 6901760 btrace_rotate.s
17812 1 0% 0% 0% S 8916992 rotee
17608 1 0% 0% 0% S 8794112 rotee
17545 11975 0% 0% 0% S 6193152 pman.sh
17274 1 0% 0% 0% S 8916992 rotee
17022 15680 0% 0% 0% S 5926912 pman.sh
16888 11975 0% 0% 0% S 6193152 pman.sh
16733 15680 0% 0% 0% S 5926912 pman.sh
16629 15492 0% 0% 0% S 10412032 ncd.sh
16501 1 0% 0% 0% S 8916992 rotee
16470 15680 0% 0% 0% S 2842624 inotifywait
16378 1 0% 0% 0% S 8912896 rotee
16317 15119 0% 0% 0% S 717631488 lman
16057 1 0% 0% 0% S 8916992 rotee
15992 13142 0% 0% 0% S 2846720 inotifywait
15920 14790 0% 0% 0% S 709464064 keyman
15680 1 0% 0% 0% S 6750208 pvp.sh
15602 1 0% 0% 0% S 8916992 rotee
15492 11975 0% 0% 0% S 6193152 pman.sh
15400 14451 0% 0% 0% S 10407936 issu_stack.sh
15177 1 0% 0% 0% S 8794112 rotee
15119 11975 0% 0% 0% S 6189056 pman.sh
14954 13866 0% 0% 0% R 722579456 hman
14843 1 0% 0% 0% S 8916992 rotee
14790 11975 0% 0% 0% S 6189056 pman.sh
14668 1 0% 0% 0% S 8916992 rotee
14482 2 0% 0% 0% S 0 kworker/3:0
14451 11975 0% 0% 0% S 6193152 pman.sh
14340 1 0% 0% 0% S 8916992 rotee
14161 1 0% 0% 0% S 8794112 rotee
14140 13055 0% 0% 0% S 6893568 btrace_rotate.s
13906 12709 0% 0% 0% S 727535616 bt_logger
13866 11975 0% 0% 0% S 6189056 pman.sh
13623 1 0% 0% 0% S 8916992 rotee
13596 12444 0% 0% 0% S 16986112 auto_upgrade_se
13473 12751 0% 0% 0% S 729792512 tms
13211 1 0% 0% 0% S 8794112 rotee
13142 12266 0% 0% 0% S 10141696 auto_upgrade_cl
13055 11975 0% 0% 0% S 6189056 pman.sh
13010 1 0% 0% 0% S 8916992 rotee
12871 1 0% 0% 0% S 8794112 rotee
12751 11975 0% 0% 0% S 6197248 pman.sh
12709 11975 0% 0% 0% S 6197248 pman.sh
12553 1 0% 0% 0% S 8794112 rotee
12444 11975 0% 0% 0% S 6197248 pman.sh
12439 11820 0% 0% 0% S 884977664 smand
12266 11975 0% 0% 0% S 6189056 pman.sh
12164 11975 0% 0% 0% S 2842624 inotifywait
12093 1 0% 0% 0% S 8912896 rotee
12071 1 0% 0% 0% S 8916992 rotee
11975 1 0% 0% 0% S 7254016 pvp.sh
11970 11853 0% 0% 0% S 2842624 inotifywait
11963 1 0% 0% 0% S 8912896 rotee
11853 1 0% 0% 0% S 6877184 psvp.sh
11820 11975 0% 0% 0% S 6189056 pman.sh
11671 11066 0% 0% 0% S 718884864 psd
11321 1 0% 0% 0% S 8794112 rotee
11251 10546 0% 0% 0% S 2842624 inotifywait
11232 1 0% 0% 0% S 8790016 rotee
11066 11975 0% 0% 0% S 6189056 pman.sh
10613 20507 0% 0% 0% S 2682880 sleep
10568 18935 0% 0% 0% S 2682880 sleep
10552 1 0% 0% 0% S 3452928 xinetd
10548 1 0% 0% 0% S 3452928 xinetd
10546 1 0% 0% 0% S 7999488 rollback_timer.
10247 1 0% 0% 0% S 8916992 rotee
10185 17869 0% 0% 0% S 2846720 inotifywait
10148 14140 0% 0% 0% S 2846720 inotifywait
10000 11975 0% 0% 0% S 6189056 pman.sh
9877 9171 0% 0% 0% S 748216320 fman_rp
9623 8668 0% 0% 0% S 882077696 dbm
9513 1 0% 0% 0% S 8916992 rotee
9381 2 0% 0% 0% S 0 kworker/1:0
9177 8224 0% 0% 0% S 717688832 cmm
9171 11975 0% 0% 0% S 6189056 pman.sh
9088 1 0% 0% 0% S 8916992 rotee
9077 7870 0% 0% 0% S 917106688 cli_agent
8668 11975 0% 0% 0% S 6189056 pman.sh
8610 1 0% 0% 0% S 8916992 rotee
8339 1 0% 0% 0% S 8916992 rotee
8224 11975 0% 0% 0% S 6189056 pman.sh
7870 11975 0% 0% 0% S 6189056 pman.sh
7398 7382 0% 0% 0% S 4362240 sntp
7382 16629 0% 0% 0% S 4579328 stack_sntp.sh
7100 2 0% 0% 0% S 0 kworker/u8:0
5408 1 0% 0% 0% S 3452928 xinetd
5098 2 0% 0% 0% S 0 kworker/3:2
4629 2 0% 0% 0% S 0 lsmpi-rx
4628 2 0% 0% 0% S 0 lsmpi-xmit
4627 2 0% 0% 0% S 0 lsmpi-refill
4396 4069 0% 0% 0% S 2842624 inotifywait
4371 1 0% 0% 0% S 8790016 rotee
4069 1 0% 0% 0% S 5812224 iptbl.sh
4049 3479 0% 0% 0% S 2842624 inotifywait
4019 1 0% 0% 0% S 8790016 rotee
4003 1 0% 0% 0% S 8790016 rotee
3869 3461 0% 0% 0% S 2674688 sleep
3846 1 0% 0% 0% S 8790016 rotee
3629 1 0% 0% 0% S 5607424 rpc.mountd
3577 2 0% 0% 0% S 0 nfsd
3574 2 0% 0% 0% S 0 nfsd
3560 2 0% 0% 0% S 0 nfsd
3558 2 0% 0% 0% S 0 nfsd
3557 2 0% 0% 0% S 0 nfsd
3556 2 0% 0% 0% S 0 nfsd
3554 2 0% 0% 0% S 0 nfsd
3553 2 0% 0% 0% S 0 nfsd
3552 2 0% 0% 0% S 0 lockd
3542 1 0% 0% 0% S 3936256 rpc.statd
3494 1 0% 0% 0% S 3452928 xinetd
3484 3477 0% 0% 0% S 2842624 inotifywait
3482 1 0% 0% 0% S 5267456 klogd
3479 1 0% 0% 0% S 6959104 droputil.sh
3477 1 0% 0% 0% S 4575232 boothelper_evt.
3470 1 0% 0% 0% S 6324224 rpcbind
3465 3456 0% 0% 0% S 217464832 libvirtd
3463 1 0% 0% 0% S 17580032 virtlogd
3461 1 0% 0% 0% S 4980736 oom.sh
3459 1 0% 0% 0% S 6967296 reflector.sh
3456 1 0% 0% 0% S 4575232 libvirtd.sh
2427 2 0% 0% 0% S 0 kworker/u8:1
1889 32545 0% 0% 0% S 790482944 smd
1377 32390 0% 0% 0% S 12963840 btelnet
1261 1 0% 0% 0% S 8794112 rotee
843 2 0% 0% 0% S 0 loop7
822 2 0% 0% 0% S 0 loop6
757 2 0% 0% 0% S 0 loop5
727 2 0% 0% 0% S 0 loop4
706 2 0% 0% 0% S 0 loop3
620 2 0% 0% 0% S 0 loop2
599 2 0% 0% 0% S 0 loop1
553 2 0% 0% 0% S 0 loop0
223 2 0% 0% 0% S 0 kworker/3:1H
222 2 0% 0% 0% S 0 kworker/1:1H
206 2 0% 0% 0% S 0 kworker/2:1H
188 2 0% 0% 0% S 0 kworker/0:1H
106 1 0% 0% 0% S 5607424 systemd-udevd
100 2 0% 0% 0% S 0 kauditd
94 1 0% 0% 0% S 37826560 systemd-journal
81 2 0% 0% 0% S 0 deferwq
70 2 0% 0% 0% S 0 irq/97-octeon-h
69 2 0% 0% 0% S 0 octeon-ethernet
64 2 0% 0% 0% S 0 scsi_eh_0
57 2 0% 0% 0% S 0 nfsiod
56 2 0% 0% 0% S 0 fsnotify_mark
55 2 0% 0% 0% S 0 ksmd
54 2 0% 0% 0% S 0 kswapd0
49 2 0% 0% 0% S 0 rpciod
48 2 0% 0% 0% S 0 edac-poller
47 2 0% 0% 0% S 0 md
46 2 0% 0% 0% S 0 khubd
45 2 0% 0% 0% S 0 ata_sff
44 2 0% 0% 0% S 0 kblockd
43 2 0% 0% 0% S 0 crypto
42 2 0% 0% 0% S 0 bioset
41 2 0% 0% 0% S 0 writeback
40 2 0% 0% 0% S 0 irq/99-octeon-h
39 2 0% 0% 0% S 0 irq/98-octeon-h
38 2 0% 0% 0% S 0 irq/96-octeon-h
37 2 0% 0% 0% S 0 irq/127-octeon-
36 2 0% 0% 0% S 0 irq/95-octeon-h
35 2 0% 0% 0% S 0 irq/93-octeon-h
34 2 0% 0% 0% S 0 irq/92-octeon-h
33 2 0% 0% 0% S 0 irq/91-octeon-h
32 2 0% 0% 0% S 0 irq/94-octeon-h
31 2 0% 0% 0% S 0 irq/90-octeon-h
30 2 0% 0% 0% S 0 irq/89-octeon-h
29 2 0% 0% 0% S 0 irq/60-octeon-h
28 2 0% 0% 0% S 0 netns
27 2 0% 0% 0% S 0 kdevtmpfs
26 2 0% 0% 0% S 0 khelper
25 2 0% 0% 0% S 0 kworker/3:0H
23 2 0% 0% 0% S 0 ksoftirqd/3
22 2 0% 0% 0% S 0 migration/3
21 2 0% 0% 0% S 0 rcuc/3
20 2 0% 0% 0% S 0 kworker/2:0H
18 2 0% 0% 0% S 0 ksoftirqd/2
17 2 0% 0% 0% S 0 migration/2
16 2 0% 0% 0% S 0 rcuc/2
15 2 0% 0% 0% S 0 kworker/1:0H
13 2 0% 0% 0% S 0 ksoftirqd/1
12 2 0% 0% 0% S 0 migration/1
11 2 0% 0% 0% S 0 rcuc/1
10 2 0% 0% 0% S 0 rcuc/0
9 2 0% 0% 0% S 0 rcu_sched
8 2 0% 0% 0% S 0 rcu_bh
7 2 0% 0% 0% S 0 migration/0
5 2 0% 0% 0% S 0 kworker/0:0H
4 2 0% 0% 0% S 0 kworker/0:0
3 2 0% 0% 0% S 0 ksoftirqd/0
2 0 0% 0% 0% S 0 kthreadd
1 0 0% 0% 0% S 8413184 systemdThanks a lot!
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-22-2021 03:59 PM
I had a look at the output and, you are right, it really is nothing.
When there is sufficient "load" on the stack, run the command to give a snapshot of the CPU and the memory.
sh platform software status con brief
NOTE: Do not bother with the classic IOS command "sh proc cpu history" because the output provides an "average" of all the cores. The output is very useful for single-core appliances that run classic IOS. For appliances running IOS-XE, it does not provide any useful information as to which CPU core is "hot spinning".
If the Free Memory or Idle CPU drops to, say <30%, run the other two commands. I made a mistake and forgot to mention that the "first page" of the two CPU/Memory commands would suffice.
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-26-2021 04:43 AM
Here's the commands output while CPU cores are on load
#sh platform software status con brief
Load Average
Slot Status 1-Min 5-Min 15-Min
1-RP0 Healthy 2.96 2.83 2.40
2-RP0 Healthy 0.52 0.30 0.26
Memory (kB)
Slot Status Total Used (Pct) Free (Pct) Committed (Pct)
1-RP0 Healthy 3983088 2774200 (70%) 1208888 (30%) 3556028 (89%)
2-RP0 Healthy 3983088 2159008 (54%) 1824080 (46%) 2966520 (74%)
CPU Utilization
Slot CPU User System Nice Idle IRQ SIRQ IOwait
1-RP0 0 36.20 15.30 0.00 48.40 0.00 0.10 0.00
1 36.53 16.71 0.00 46.74 0.00 0.00 0.00
2 31.33 14.51 0.00 54.15 0.00 0.00 0.00
3 37.76 13.08 0.00 49.05 0.00 0.09 0.00
2-RP0 0 3.10 1.10 0.00 95.70 0.00 0.10 0.00
1 3.39 0.99 0.00 95.50 0.00 0.09 0.00
2 3.29 0.89 0.00 95.70 0.00 0.09 0.00
3 2.99 1.09 0.00 95.90 0.00 0.00 0.00
#show processes cpu platform sorted location switch 1 r0
CPU utilization for five seconds: 46%, one minute: 49%, five minutes: 48%
Core 0: CPU utilization for five seconds: 41%, one minute: 51%, five minutes: 49%
Core 1: CPU utilization for five seconds: 46%, one minute: 49%, five minutes: 49%
Core 2: CPU utilization for five seconds: 45%, one minute: 49%, five minutes: 49%
Core 3: CPU utilization for five seconds: 45%, one minute: 47%, five minutes: 48%
Pid PPid 5Sec 1Min 5Min Status Size Name
---------------------------------------------------------------------------------
4686 2 4% 5% 5% S 0 lsmpi-rx
4684 2 4% 6% 5% S 0 lsmpi-xmit
31690 30179 1% 1% 1% S 749502464 repm
32656 31341 0% 0% 5% R 892981248 smand
32359 1 0% 0% 0% S 8794112 rotee
31889 1 0% 0% 0% S 8916992 rotee
31783 12036 0% 0% 0% S 6197248 pman.sh
31392 30191 0% 0% 0% S 846893056 fman_fp_image
31341 12036 0% 0% 0% S 6197248 pman.sh
30863 2 0% 0% 0% S 0 kworker/3:1
30741 1 0% 0% 0% S 8916992 rotee
30730 1 0% 0% 0% S 8916992 rotee
30206 30182 0% 0% 0% S 4362240 sntp
30191 15779 0% 0% 0% S 5931008 pman.sh
30182 16724 0% 0% 0% S 4579328 stack_sntp.sh
30179 12036 0% 0% 0% S 6189056 pman.sh
28065 3531 0% 0% 0% S 2842624 inotifywait
27832 15476 0% 0% 0% S 2846720 inotifywait
27824 15476 0% 0% 0% S 10272768 issu_stack.sh
27822 15476 0% 0% 0% S 2846720 inotifywait
27816 15476 0% 0% 0% S 10272768 issu_stack.sh
27813 15476 0% 0% 0% S 2846720 inotifywait
27799 16724 0% 0% 0% S 2846720 inotifywait
25022 24372 0% 0% 0% S 714989568 epc_ws_liaison
24754 1 0% 0% 0% S 8794112 rotee
24372 15779 0% 0% 0% S 5931008 pman.sh
21135 20083 0% 0% 0% S 718180352 stack_mgr
20757 19854 0% 0% 0% S 709263360 tams_proc
20649 1 0% 0% 0% S 8916992 rotee
20579 19478 0% 0% 0% S 5103616 sort_files_by_i
20567 19569 0% 0% 0% S 708685824 tamd_proc
20547 19205 0% 1% 0% S 1433403392 sif_mgr
20323 1 0% 0% 0% S 8794112 rotee
20083 12036 0% 0% 0% S 6197248 pman.sh
20022 1 0% 0% 0% S 8794112 rotee
19974 1 0% 0% 0% S 8794112 rotee
19854 12036 0% 0% 0% S 6193152 pman.sh
19645 1 0% 0% 0% S 8794112 rotee
19626 1 0% 0% 0% S 8916992 rotee
19569 12036 0% 0% 0% S 6193152 pman.sh
19478 12036 0% 0% 0% S 6189056 pman.sh
19412 20579 0% 0% 0% S 2682880 sleep
19406 1 0% 0% 0% S 8790016 rotee
19387 18300 0% 0% 0% S 1082458112 platform_mgr
19365 17723 0% 0% 0% S 2846720 inotifywait
19287 19188 0% 0% 0% S 5693440 iosd_console_at
19205 12036 0% 0% 0% S 6197248 pman.sh
19188 19185 0% 0% 0% S 6287360 bexecute.sh
#show processes cpu platform sorted location switch 2 r0
CPU utilization for five seconds: 4%, one minute: 4%, five minutes: 4%
Core 0: CPU utilization for five seconds: 4%, one minute: 4%, five minutes: 4%
Core 1: CPU utilization for five seconds: 3%, one minute: 4%, five minutes: 4%
Core 2: CPU utilization for five seconds: 5%, one minute: 4%, five minutes: 4%
Core 3: CPU utilization for five seconds: 4%, one minute: 3%, five minutes: 4%
Pid PPid 5Sec 1Min 5Min Status Size Name
---------------------------------------------------------------------------------
10646 10000 3% 3% 3% S 2130436096 linux_iosd-imag
20414 19142 1% 0% 0% S 1433403392 sif_mgr
789 30873 1% 1% 1% S 749408256 repm
32577 2 0% 0% 0% S 0 kworker/2:2
32545 11975 0% 0% 0% S 6197248 pman.sh
32390 32289 0% 0% 0% S 6250496 btelnet.sh
32289 30799 0% 0% 0% S 4575232 bexec.sh
31890 1 0% 0% 0% S 8916992 rotee
31661 29478 0% 0% 0% S 715309056 plogd
31532 29501 0% 0% 0% S 846295040 fman_fp_image
30873 11975 0% 0% 0% S 6189056 pman.sh
30799 29767 0% 0% 0% S 6275072 brelay.sh
30723 1 0% 0% 0% S 9830400 rotee
30480 1 0% 0% 0% S 8916992 rotee
30391 1 0% 0% 0% S 8794112 rotee
29767 1 0% 0% 0% S 6107136 brelay_console.
29638 1 0% 0% 0% S 9830400 rotee
29501 15680 0% 0% 0% S 5931008 pman.sh
29478 11975 0% 0% 0% S 6189056 pman.sh
29180 2 0% 0% 0% S 0 kworker/0:2
27208 2 0% 0% 0% S 0 kworker/1:2
26462 15400 0% 0% 0% S 2846720 inotifywait
26452 15400 0% 0% 0% S 10272768 issu_stack.sh
26450 15400 0% 0% 0% S 2846720 inotifywait
26443 15400 0% 0% 0% S 10272768 issu_stack.sh
26442 15400 0% 0% 0% S 2846720 inotifywait
26376 16629 0% 0% 0% S 2846720 inotifywait
26298 3459 0% 0% 0% S 2842624 inotifywait
24602 23956 0% 0% 0% S 714915840 epc_ws_liaison
24334 1 0% 0% 0% S 8794112 rotee
23956 15680 0% 0% 0% S 5931008 pman.sh
22455 17869 0% 0% 0% S 2846720 inotifywait
22423 20507 0% 0% 0% S 2682880 sleep
22232 18935 0% 0% 0% S 2682880 sleep
22002 14140 0% 0% 0% S 2846720 inotifywait
21127 2 0% 0% 0% S 0 kworker/2:1
20969 20048 0% 0% 0% S 718155776 stack_mgr
20597 1 0% 0% 0% S 8916992 rotee
20507 19426 0% 0% 0% S 5103616 sort_files_by_i
20048 11975 0% 0% 0% S 6197248 pman.sh
19947 1 0% 0% 0% S 8794112 rotee
19571 1 0% 0% 0% S 8794112 rotee
19556 1 0% 0% 0% S 8916992 rotee
19426 11975 0% 0% 0% S 6189056 pman.sh
19383 18195 0% 0% 0% S 1080860672 platform_mgr
19142 11975 0% 0% 0% S 6197248 pman.sh
18935 17545 0% 0% 0% S 9072640 periodic.sh
18786 1 0% 0% 0% S 8916992 roteeThanks for your note on "sh proc cpu history" command, I'll consider to switch from AvgBusy1 SNMP OID monitoring to more relevant one.
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-26-2021 01:51 PM
Other than the CPU average going >30%, I do not see anything suspicious.
Here is another command to check the memory:
sh processes memory platform sorted location switch <1 - 8> r0
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-12-2021 06:54 AM
Posting an update:
During the past weeks we decided to upgrade sw to 16.09.06 to understand if that's a bug or not.
It seems that High CPU utilization events are happening little less often than before, but we still observing them sometimes.
What's strange for me is that when I see alerts that Prometheus is unable to scrape SNMP metrics from the switch, 'sh proc cpu history' shows 90+% CPU usage but 'show processes cpu platform sorted location switch 2 r0' shows ~40% on all cores.
So why it's not answering to SNMP if CPU cores load only on ~40%. It seems like that sh proc cpu history for some reason is more precise for us, somehow.
However, pasting last outputs when the issue was encountered
#show processes cpu platform sorted location switch 2 r0
CPU utilization for five seconds: 47%, one minute: 46%, five minutes: 46%
Core 0: CPU utilization for five seconds: 44%, one minute: 42%, five minutes: 44%
Core 1: CPU utilization for five seconds: 49%, one minute: 47%, five minutes: 47%
Core 2: CPU utilization for five seconds: 43%, one minute: 46%, five minutes: 47%
Core 3: CPU utilization for five seconds: 45%, one minute: 46%, five minutes: 47%
Pid PPid 5Sec 1Min 5Min Status Size Name
--------------------------------------------------------------------------------
12080 8383 95% 92% 93% R 2178207744 linux_iosd-imag
12480 10330 79% 82% 84% S 1836642304 fed main event
4436 2 6% 6% 6% S 0 lsmpi-xmit
4437 2 5% 5% 5% S 0 lsmpi-rx
4435 2 1% 1% 1% S 0 lsmpi-refill
21 2 1% 1% 1% S 0 rcuc/3
16 2 1% 1% 1% S 0 rcuc/2
29266 11642 0% 0% 0% S 2908160 sleep
28993 2 0% 0% 0% S 0 kworker/u8:0
28992 9810 0% 0% 0% S 2908160 sleep
27742 2 0% 0% 0% S 0 kworker/u8:2
25677 25673 0% 0% 0% S 180260864 python2.7
25673 25375 0% 0% 0% S 5136384 rdope.sh
25492 1 0% 0% 0% S 9498624 rotee
25375 1 0% 0% 0% S 5943296 pman.sh
23052 22113 0% 0% 0% S 311644160 fman_rp
22309 1 0% 0% 0% S 9498624 rotee
22293 21747 0% 0% 0% S 254238720 dbm
22113 6431 0% 0% 0% S 5951488 pman.sh
21950 1 0% 0% 0% S 9498624 rotee
21873 21377 0% 0% 0% S 42532864 cmm
21747 6431 0% 0% 0% S 5951488 pman.sh
21725 19718 0% 0% 0% S 342634496 sessmgrd
21598 20970 0% 0% 0% S 505753600 cli_agent
21565 1 0% 0% 0% S 9498624 rotee
21377 6431 0% 0% 0% S 5951488 pman.sh
21216 1 0% 0% 0% S 9498624 rotee
20970 6431 0% 0% 0% S 5951488 pman.sh
20799 20370 0% 0% 0% S 53084160 tms
20527 1 0% 0% 0% S 9498624 rotee
20370 6431 0% 0% 0% S 5951488 pman.sh
20262 19962 0% 0% 0% S 235200512 smand
20072 1 0% 0% 0% S 9498624 rotee
19962 6431 0% 0% 0% S 5951488 pman.sh
19857 19471 0% 0% 0% S 148779008 repm
19850 1 0% 0% 0% S 9498624 rotee
19718 6431 0% 0% 0% S 5951488 pman.sh
19610 1 0% 0% 0% S 9498624 rotee
19506 18957 0% 0% 0% S 55529472 psd
19471 6431 0% 0% 0% S 5951488 pman.sh
19426 2 0% 0% 0% S 0 kworker/u8:1
19326 1 0% 0% 0% S 9498624 rotee
19250 19163 0% 0% 0% S 10207232 journalctl
19163 18501 0% 0% 0% S 39899136 plogd
18957 6431 0% 0% 0% S 5951488 pman.sh
18760 1 0% 0% 0% S 9498624 rotee
#sh platform software status con brief
Load Average
Slot Status 1-Min 5-Min 15-Min
2-RP0 Healthy 3.11 2.81 2.55
Memory (kB)
Slot Status Total Used (Pct) Free (Pct) Committed (Pct)
2-RP0 Healthy 3977740 1660184 (42%) 2317556 (58%) 2519316 (63%)
CPU Utilization
Slot CPU User System Nice Idle IRQ SIRQ IOwait
2-RP0 0 34.43 9.90 0.00 55.65 0.00 0.00 0.00
1 38.26 8.19 0.00 53.54 0.00 0.00 0.00
2 35.90 6.20 0.00 57.90 0.00 0.00 0.00
3 41.25 9.39 0.00 49.35 0.00 0.00 0.00#sh processes memory platform sorted location switch 2 r0 System memory: 3977740K total, 1660932K used, 2316808K free, Lowest: 2316808K Pid Text Data Stack Dynamic RSS Total Name -------------------------------------------------------------------------------- 12080 171507 629616 0 296 629616 2127156 linux_iosd-imag 12480 206 162188 132 71832 162188 1793596 fed main event 12003 396 57144 132 2536 57144 778160 sif_mgr 21598 344 48244 132 6264 48244 493900 cli_agent 12105 1004 60212 132 2328 60212 448596 platform_mgr 21725 178 71328 132 7364 71328 334604 sessmgrd 23052 8562 63592 132 2564 63592 304340 fman_rp 18550 9018 59584 132 10116 59584 290648 fman_fp_image 22293 197 53588 132 4168 53588 248280 dbm 20262 799 72468 132 24580 72468 230948 smand 3714 545 4428 132 132 4428 214820 libvirtd 25677 8 25280 136 7380 25280 176036 python2.7 19857 578 27248 132 1872 27248 145292 repm 7498 99 13348 132 1416 13348 63896 bt_logger 9083 200 12424 132 1800 12424 58352 lman 11926 483 13036 132 1428 13036 56088 stack_mgr 19506 100 13564 132 1444 13564 54228 psd 20799 305 15084 132 6704 15084 51840 tms 8031 589 13828 352 2336 13828 51780 hman 7741 166 10260 132 1464 10260 46448 btman 10612 166 10024 132 1380 10024 45980 btman 9564 230 6420 132 272 6420 45688 nif_mgr 8491 138 10004 132 1232 10004 45580 keyman 21873 170 7612 132 1108 7612 41536 cmm 19163 112 8272 132 1848 8272 38964 plogd 15728 119 7440 132 1028 7440 38204 epc_ws_liaison 14269 88 6984 136 1464 6984 28816 tams_proc 14169 126 6576 136 868 6576 28308 tamd_proc 13982 76 5860 136 680 5860 27224 tam_svcs_ng3k_c 3705 687 1968 132 132 1968 17784 virtlogd 7249 68 2532 136 132 2532 13196 auto_upgrade_se 6992 1096 8576 244 6568 8576 11560 auto_upgrade_cl 9200 1096 8592 244 6560 8592 11552 ncd.sh 16126 1096 7572 244 6412 7572 11404 issu_stack.sh 16122 1096 7472 244 6412 7472 11404 issu_stack.sh 8240 1096 8472 244 6412 8472 11404 issu_stack.sh 19250 509 2116 136 1204 2116 9968 journalctl 4312 68 1996 132 252 1996 9596 rotee 3997 68 1996 132 252 1996 9596 rotee 9810 1096 6436 248 4460 6436 9456 periodic.sh 6173 1096 6420 244 4436 6420 9428 rollback_timer. 6475 68 1996 132 252 1996 9392 rotee 25492 68 1972 136 132 1972 9276 rotee 22309 68 1980 136 132 1980 9276 rotee 21950 68 1980 136 132 1980 9276 rotee 21565 68 1980 136 132 1980 9276 rotee 21216 68 1980 136 132 1980 9276 rotee 20527 68 1980 136 132 1980 9276 rotee 20072 68 1980 136 132 1980 9276 rotee
sw-c3650-6-4#show platform software status control-processor brief
Load Average
Slot Status 1-Min 5-Min 15-Min
2-RP0 Healthy 2.35 2.76 2.62
Memory (kB)
Slot Status Total Used (Pct) Free (Pct) Committed (Pct)
2-RP0 Healthy 3977740 1660740 (42%) 2317000 (58%) 2519276 (63%)
CPU Utilization
Slot CPU User System Nice Idle IRQ SIRQ IOwait
2-RP0 0 33.96 11.88 0.00 54.14 0.00 0.00 0.00
1 39.20 7.40 0.00 53.30 0.00 0.10 0.00
2 31.30 6.50 0.00 62.20 0.00 0.00 0.00
3 37.16 8.19 0.00 54.64 0.00 0.00 0.00Any ideas?
Thanks
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-12-2021 08:17 AM
Hello,
post the full running configuration (sh run) of the 3650 switch...
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-12-2021 03:32 PM
@KirillKO wrote:
12080 8383 95% 92% 93% R 2178207744 linux_iosd-imag 12480 10330 79% 82% 84% S 1836642304 fed main event
Oh, FFS! Not this one again!
Yes, I know of these two "clowns". FED = Forwarding Engine Driver.
Can you have a look at Troubleshoot High CPU Usage in Catalyst Switch Platforms Running IOS-XE 16.x?
There is a very big chance that you may need to downgrade to 16.3.X.
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-17-2021 11:44 PM
Hello, Leo.
@Leo Laohoo wrote:
@KirillKO wrote:12080 8383 95% 92% 93% R 2178207744 linux_iosd-imag 12480 10330 79% 82% 84% S 1836642304 fed main eventOh, FFS! Not this one again!
Yes, I know of these two "clowns". FED = Forwarding Engine Driver.
Can you have a look at Troubleshoot High CPU Usage in Catalyst Switch Platforms Running IOS-XE 16.x?
There is a very big chance that you may need to downgrade to 16.3.X.
Could you please clarify why it's the case?
Looked into the link you provided, will try to follow the steps from it, next time it appears.
Thanks a lot!
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-18-2021 02:58 AM
@KirillKO wrote:
Could you please clarify why it's the case?
Because this bug is not present/reported in 16.3.X.
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-05-2021 07:33 AM
Thanks for the info, Leo.
As a quick update, we went through "Troubleshooting High CPU usage" manual, and encountered some kind of ARP flooding to our core switch. Presumably only from Apple Watch devices, but not yet 100% confirmed as we still collecting packets capture.
Also, found this bug report https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvi87866/?rfs=iqvred but it stated that its fixed for our current firmware.
Did you heard smth relevant about Apple Watch devices?
Thanks
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-05-2021 03:07 PM
Apple Watches talk BLE. I do not see any relations between BLE and this switch.
Learn, share, save
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide
Customers Also Viewed These Support Documents
