cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
998
Views
0
Helpful
4
Replies

cisco 3850 High rate app CPU queue

paul amaral
Level 4
Level 4

Does anyone know what is considered traffic that is classified as High rate app? I notice that the FED on one of my cisco 3850's is putting this traffic to the CP. 

 

show platform software fed switch active cpu-interface
queue retrieved dropped invalid hol-block
------------------------------------------------------------------------
High rate app 994752082 994752082 0 0

This traffic ends up on queue 23

 

show platform software fed switch active punt cpuq al

CPU Q Id : 23
CPU Q Name : CPU_Q_HIGH_RATE_APP
Packets received from ASIC : 994786736
Send to IOSd total attempts : 0
Send to IOSd failed count : 0
RX suspend count : 0
RX unsuspend count : 0
RX unsuspend send count : 0
RX unsuspend send failed count : 0
RX consumed count : 0
RX dropped count : 994786736
RX non-active dropped count : 0
RX conversion failure dropped : 994786736
RX INTACK count : 742033258
RX packets dq'd after intack : 29234641
Active RxQ event : 745886762
RX spurious interrupt : 92117945

 

But I can't find anything on what this is. On my other 3650 switches I'm not seeing this counter increase just on the this 3850. 

 

Cisco IOS XE Software, Version 16.09.04
Cisco IOS Software [Fuji], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.9.4, REL
EASE SOFTWARE (fc2)

 

 

TIA, Paul 

 

4 Replies 4

Leo Laohoo
Hall of Fame
Hall of Fame

Please post the complete output to the following commands: show processes cpu platform sorted location switch <all switch members> r0

Hi Leo, below is what you requested

 

CPU utilization for five seconds: 8%, one minute: 8%, five minutes: 8%
Core 0: CPU utilization for five seconds: 7%, one minute: 7%, five minutes: 8%
Core 1: CPU utilization for five seconds: 6%, one minute: 8%, five minutes: 8%
Core 2: CPU utilization for five seconds: 6%, one minute: 9%, five minutes: 8%
Core 3: CPU utilization for five seconds: 10%, one minute: 8%, five minutes: 9%
Core 4: CPU utilization for five seconds: 11%, one minute: 8%, five minutes: 8%
Core 5: CPU utilization for five seconds: 6%, one minute: 6%, five minutes: 8%
Pid PPid 5Sec 1Min 5Min Status Size Name
--------------------------------------------------------------------------------
13308 12822 41% 40% 40% S 2182856704 fed main event
12859 9030 7% 8% 9% S 2177421312 linux_iosd-imag
31985 2 0% 0% 0% S 0 kworker/1:0
30849 11661 0% 0% 0% S 2908160 sleep
30597 10404 0% 0% 0% S 2908160 sleep
27480 2 0% 0% 0% S 0 kworker/u12:0
24904 24901 0% 0% 0% S 313520128 python2.7
24901 24775 0% 0% 0% S 5136384 rdope.sh
24823 1 0% 0% 0% S 9502720 rotee
24775 1 0% 0% 0% S 5931008 pman.sh
24285 2 0% 0% 0% S 0 kworker/u12:1
22458 21932 0% 0% 0% S 313970688 fman_rp
22087 21690 0% 0% 0% S 249655296 dbm
22051 1 0% 0% 0% S 9502720 rotee
21932 7003 0% 0% 0% S 5931008 pman.sh
21844 1 0% 0% 0% S 9502720 rotee
21797 21445 0% 0% 0% S 42508288 cmm
21690 7003 0% 0% 0% S 5931008 pman.sh
21607 21252 0% 0% 0% S 429998080 cli_agent
21588 1 0% 0% 0% S 9502720 rotee
21445 7003 0% 0% 0% S 5931008 pman.sh
21444 20154 0% 0% 0% S 342212608 sessmgrd
21356 1 0% 0% 0% S 9502720 rotee
21252 7003 0% 0% 0% S 5931008 pman.sh
21158 20881 0% 0% 0% S 52965376 tms
20962 1 0% 0% 0% S 9502720 rotee
20881 7003 0% 0% 0% S 5931008 pman.sh
20796 20410 0% 0% 0% S 236515328 smand
20529 1 0% 0% 0% S 9502720 rotee
20410 7003 0% 0% 0% S 5931008 pman.sh
20287 1 0% 0% 0% S 9502720 rotee
20260 19890 0% 0% 0% S 148680704 repm
20198 7582 0% 0% 0% S 3076096 inotifywait
20154 7003 0% 0% 0% S 5931008 pman.sh
20010 1 0% 0% 0% S 9502720 rotee
19913 19474 0% 0% 0% S 55672832 psd
19890 7003 0% 0% 0% S 5922816 pman.sh
19787 19692 0% 0% 0% S 10207232 journalctl
19694 1 0% 0% 0% S 9502720 rotee
19692 19068 0% 0% 0% S 39837696 plogd
19474 7003 0% 0% 0% S 5931008 pman.sh
19290 1 0% 0% 0% S 9502720 rotee
19068 7003 0% 0% 0% S 5931008 pman.sh
19064 19058 0% 0% 0% S 154062848 nginx
19058 18927 0% 0% 0% S 142594048 nginx
18976 1 0% 0% 0% S 9502720 rotee
18964 17943 0% 0% 0% S 300392448 fman_fp_image
18927 1 0% 0% 0% S 5931008 pman.sh
18202 18185 0% 0% 0% S 4763648 sntp
18185 1 0% 0% 0% S 5136384 stack_sntp.sh
18132 1 0% 0% 0% S 9502720 rotee
17943 12130 0% 0% 0% S 5931008 pman.sh
16485 9879 0% 0% 0% S 3076096 inotifywait
16414 8883 0% 0% 0% S 3076096 inotifywait
16407 8883 0% 0% 0% S 11661312 issu_stack.sh
16406 8883 0% 0% 0% S 3076096 inotifywait
16400 8883 0% 0% 0% S 11661312 issu_stack.sh
16398 8883 0% 0% 0% S 3076096 inotifywait
16292 4307 0% 0% 0% S 3067904 inotifywait
16224 2 0% 0% 0% S 0 kworker/3:1
16223 2 0% 0% 0% S 0 irq/100-octeon-
15440 2 0% 0% 0% S 0 kworker/3:2
14667 14311 0% 0% 0% S 39120896 epc_ws_liaison
14463 1 0% 0% 0% S 9379840 rotee
14311 12130 0% 0% 0% S 5931008 pman.sh
14111 13913 0% 0% 0% S 29503488 tams_proc
14014 13663 0% 0% 0% S 28958720 tamd_proc
14012 1 0% 0% 0% S 9379840 rotee
13913 1 0% 0% 0% S 5931008 pman.sh
13836 13493 0% 0% 0% S 27938816 tam_svcs_ng3k_c
13831 1 0% 0% 0% S 9379840 rotee
13663 1 0% 0% 0% S 5931008 pman.sh
13606 1 0% 0% 0% S 9379840 rotee
13493 1 0% 0% 0% S 5931008 pman.sh
13047 12621 0% 0% 0% S 47083520 btman
13004 1 0% 0% 0% S 9502720 rotee
12822 12130 0% 0% 0% S 5931008 pman.sh
12791 1 0% 0% 0% S 9502720 rotee
12621 12130 0% 0% 0% S 5931008 pman.sh
12410 12130 0% 0% 0% S 3067904 inotifywait
12342 10915 0% 0% 0% S 977035264 sif_mgr
12315 1 0% 0% 0% S 9498624 rotee
12130 1 0% 0% 0% S 7983104 pvp.sh
12016 10300 0% 0% 0% S 710819840 platform_mgr
11961 11592 0% 0% 0% S 57540608 stack_mgr
11746 1 0% 0% 0% S 9502720 rotee
11661 11303 0% 0% 0% S 5660672 sort_files_by_i
11592 7003 0% 0% 0% S 5931008 pman.sh
11494 1 0% 0% 0% S 9379840 rotee
11303 7003 0% 0% 0% S 5931008 pman.sh
11151 1 0% 0% 0% S 9502720 rotee
10915 7003 0% 0% 0% S 5931008 pman.sh
10842 1 0% 0% 0% S 9379840 rotee
10625 1 0% 0% 0% S 9502720 rotee
10489 1 0% 0% 0% S 9502720 rotee
10404 10039 0% 0% 0% S 10063872 periodic.sh
10300 7003 0% 0% 0% S 5931008 pman.sh
10218 9811 0% 0% 0% S 46739456 nif_mgr
10213 1 0% 0% 0% S 9379840 rotee
10039 7003 0% 0% 0% S 5931008 pman.sh
9966 1 0% 0% 0% S 9379840 rotee
9879 9534 0% 0% 0% S 11886592 ncd.sh
9811 7003 0% 0% 0% S 5931008 pman.sh
9727 9298 0% 0% 0% S 59686912 lman
9698 1 0% 0% 0% S 9502720 rotee
9534 7003 0% 0% 0% S 5931008 pman.sh
9461 1 0% 0% 0% S 9502720 rotee
9450 1 0% 0% 0% S 9502720 rotee
9298 7003 0% 0% 0% S 5931008 pman.sh
9189 1 0% 0% 0% S 9625600 rotee
9143 8732 0% 0% 0% S 31469568 keyman
9030 7003 0% 0% 0% S 5931008 pman.sh
8907 1 0% 0% 0% S 9379840 rotee
8883 8521 0% 0% 0% S 11661312 issu_stack.sh
8732 7003 0% 0% 0% S 5931008 pman.sh
8674 8216 0% 0% 0% R 53133312 hman
8662 1 0% 0% 0% S 9502720 rotee
8521 7003 0% 0% 0% S 5931008 pman.sh
8388 1 0% 0% 0% S 9502720 rotee
8344 7870 0% 0% 0% S 47505408 btman
8265 1 0% 0% 0% S 9502720 rotee
8216 7003 0% 0% 0% S 5931008 pman.sh
8112 7618 0% 0% 0% S 65429504 bt_logger
8061 1 0% 0% 0% S 9502720 rotee
7870 7003 0% 0% 0% S 5931008 pman.sh
7842 7376 0% 0% 0% S 13520896 auto_upgrade_se
7804 1 0% 0% 0% S 9379840 rotee
7618 7003 0% 0% 0% S 5931008 pman.sh
7582 7228 0% 0% 0% S 11825152 auto_upgrade_cl
7552 1 0% 0% 0% S 9379840 rotee
7376 7003 0% 0% 0% S 5931008 pman.sh
7341 1 0% 0% 0% S 9379840 rotee
7228 7003 0% 0% 0% S 5931008 pman.sh
7118 7003 0% 0% 0% S 3067904 inotifywait
7046 1 0% 0% 0% S 9621504 rotee
7003 1 0% 0% 0% S 8253440 pvp.sh
6998 6949 0% 0% 0% S 3067904 inotifywait
6992 1 0% 0% 0% S 9498624 rotee
6949 1 0% 0% 0% S 8122368 psvp.sh
6879 6725 0% 0% 0% S 3067904 inotifywait
6871 1 0% 0% 0% S 9375744 rotee
6736 1 0% 0% 0% S 3776512 xinetd
6732 1 0% 0% 0% S 3776512 xinetd
6725 1 0% 0% 0% S 9637888 rollback_timer.
5709 1 0% 0% 0% S 3776512 xinetd
5037 2 0% 0% 0% S 0 lsmpi-rx
5036 2 0% 0% 0% S 0 lsmpi-xmit
5035 2 0% 0% 0% S 0 lsmpi-refill
4914 4707 0% 0% 0% S 3067904 inotifywait
4815 1 0% 0% 0% S 9375744 rotee
4707 1 0% 0% 0% S 6782976 iptbl.sh
4693 4327 0% 0% 0% S 3067904 inotifywait
4648 1 0% 0% 0% S 9830400 rotee
4645 1 0% 0% 0% S 9830400 rotee
4575 4301 0% 0% 0% S 2899968 sleep
4533 1 0% 0% 0% S 9375744 rotee
4448 2 0% 0% 0% S 0 nfsd
4440 2 0% 0% 0% S 0 nfsd
4439 2 0% 0% 0% S 0 nfsd
4438 2 0% 0% 0% S 0 nfsd
4437 2 0% 0% 0% S 0 nfsd
4436 2 0% 0% 0% S 0 nfsd
4435 2 0% 0% 0% S 0 nfsd
4434 2 0% 0% 0% S 0 nfsd
4431 2 0% 0% 0% S 0 lockd
4406 1 0% 0% 0% S 4620288 rpc.mountd
4376 1 0% 0% 0% S 4255744 rpc.statd
4373 4364 0% 0% 0% S 3067904 inotifywait
4364 1 0% 0% 0% S 5132288 boothelper_evt.
4351 1 0% 0% 0% S 3776512 xinetd
4330 1 0% 0% 0% S 6950912 rpcbind
4327 1 0% 0% 0% S 7999488 droputil.sh
4321 4300 0% 0% 0% S 219979776 libvirtd
4307 1 0% 0% 0% S 7987200 reflector.sh
4305 1 0% 0% 0% S 18214912 virtlogd
4301 1 0% 0% 0% S 5537792 oom.sh
4300 1 0% 0% 0% S 5132288 libvirtd.sh
979 2 0% 0% 0% S 0 loop7
958 2 0% 0% 0% S 0 loop6
887 2 0% 0% 0% S 0 loop5
854 2 0% 0% 0% S 0 loop4
833 2 0% 0% 0% S 0 loop3
738 2 0% 0% 0% S 0 loop2
717 2 0% 0% 0% S 0 loop1
668 2 0% 0% 0% S 0 loop0
565 2 0% 0% 0% S 0 kworker/4:2
363 1 0% 0% 0% S 8134656 dbus-daemon
349 2 0% 0% 0% S 0 kworker/1:2
309 2 0% 0% 0% S 0 kworker/0:1H
308 2 0% 0% 0% S 0 kworker/1:1H
291 2 0% 0% 0% S 0 kworker/3:1H
279 2 0% 0% 0% S 0 kworker/2:2
275 2 0% 0% 0% S 0 kworker/4:1H
223 2 0% 0% 0% S 0 kworker/5:2
222 2 0% 0% 0% S 0 kworker/5:1H
221 2 0% 0% 0% S 0 kworker/2:1H
202 2 0% 0% 0% S 0 kworker/0:2
200 2 0% 0% 0% S 0 usb-storage
199 2 0% 0% 0% S 0 scsi_eh_1
120 1 0% 0% 0% S 6189056 systemd-udevd
111 2 0% 0% 0% S 0 kauditd
106 1 0% 0% 0% S 8896512 systemd-journal
91 2 0% 0% 0% S 0 kworker/0:1
90 2 0% 0% 0% S 0 deferwq
79 2 0% 0% 0% S 0 irq/97-octeon-h
78 2 0% 0% 0% S 0 octeon-ethernet
73 2 0% 0% 0% S 0 scsi_eh_0
66 2 0% 0% 0% S 0 nfsiod
65 2 0% 0% 0% S 0 fsnotify_mark
64 2 0% 0% 0% S 0 ksmd
63 2 0% 0% 0% S 0 kswapd0
60 2 0% 0% 0% S 0 kworker/2:1
59 2 0% 0% 0% S 0 rpciod
58 2 0% 0% 0% S 0 edac-poller
57 2 0% 0% 0% S 0 md
56 2 0% 0% 0% S 0 khubd
55 2 0% 0% 0% S 0 ata_sff
54 2 0% 0% 0% S 0 kblockd
53 2 0% 0% 0% S 0 crypto
52 2 0% 0% 0% S 0 bioset
51 2 0% 0% 0% S 0 writeback
50 2 0% 0% 0% S 0 irq/99-octeon-h
49 2 0% 0% 0% S 0 irq/98-octeon-h
48 2 0% 0% 0% S 0 irq/96-octeon-h
47 2 0% 0% 0% S 0 irq/127-octeon-
46 2 0% 0% 0% S 0 irq/95-octeon-h
45 2 0% 0% 0% S 0 irq/93-octeon-h
44 2 0% 0% 0% S 0 irq/92-octeon-h
43 2 0% 0% 0% S 0 irq/91-octeon-h
42 2 0% 0% 0% S 0 irq/94-octeon-h
41 2 0% 0% 0% S 0 irq/90-octeon-h
40 2 0% 0% 0% S 0 irq/89-octeon-h
39 2 0% 0% 0% S 0 irq/60-octeon-h
38 2 0% 0% 0% S 0 netns
37 2 0% 0% 0% S 0 kdevtmpfs
36 2 0% 0% 0% S 0 khelper
35 2 0% 0% 0% S 0 kworker/5:0H
34 2 0% 0% 0% S 0 kworker/5:0
33 2 0% 0% 0% S 0 ksoftirqd/5
32 2 0% 0% 0% S 0 migration/5
31 2 0% 0% 0% S 0 rcuc/5
30 2 0% 0% 0% S 0 kworker/4:0H
29 2 0% 0% 0% S 0 kworker/4:0
28 2 0% 0% 0% S 0 ksoftirqd/4
27 2 0% 0% 0% S 0 migration/4
26 2 0% 0% 0% R 0 rcuc/4
25 2 0% 0% 0% S 0 kworker/3:0H
23 2 0% 0% 0% S 0 ksoftirqd/3
22 2 0% 0% 0% S 0 migration/3
21 2 0% 0% 0% S 0 rcuc/3
20 2 0% 0% 0% S 0 kworker/2:0H
18 2 0% 0% 0% S 0 ksoftirqd/2
17 2 0% 0% 0% S 0 migration/2
16 2 0% 0% 0% S 0 rcuc/2
15 2 0% 0% 0% S 0 kworker/1:0H
13 2 0% 0% 0% S 0 ksoftirqd/1
12 2 0% 0% 0% S 0 migration/1
11 2 0% 0% 0% S 0 rcuc/1
10 2 0% 0% 0% S 0 rcuc/0
9 2 0% 0% 0% S 0 rcu_sched
8 2 0% 0% 0% S 0 rcu_bh
7 2 0% 0% 0% S 0 migration/0
5 2 0% 0% 0% S 0 kworker/0:0H
3 2 0% 0% 0% S 0 ksoftirqd/0
2 0 0% 0% 0% S 0 kthreadd
1 0 0% 0% 0% S 9007104 systemd

 

 

 

 

CPU utilization for five seconds: 8%, one minute: 8%, five minutes: 8%
Core 0: CPU utilization for five seconds: 7%, one minute: 7%, five minutes: 8%
Core 1: CPU utilization for five seconds: 6%, one minute: 8%, five minutes: 8%
Core 2: CPU utilization for five seconds: 6%, one minute: 9%, five minutes: 8%
Core 3: CPU utilization for five seconds: 10%, one minute: 8%, five minutes: 9%
Core 4: CPU utilization for five seconds: 11%, one minute: 8%, five minutes: 8%
Core 5: CPU utilization for five seconds: 6%, one minute: 6%, five minutes: 8%
Pid PPid 5Sec 1Min 5Min Status Size Name
--------------------------------------------------------------------------------
13308 12822 41% 40% 40% S 2182856704 fed main event
12859 9030 7% 8% 9% S 2177421312 linux_iosd-imag

(Oooops.  Me bad.  I forgot to mention to post only the "first page".  My apologies.)  

Top bit tells me the each CPU are working normally.  No CPU is running "hot" (or as TAC want to call it, "hot spinning").  

Bottom bit tells me the CPU is mostly being used by the process called "fed" (forwarding engine driver) and to be honest, 40% isn't high.  

Do you have IPv6 running?

One last thing:  Make sure to run the command 

show processes cpu platform sorted location switch <all switch members> r0

on all switch members and compare the "first page" to the master.  Same thing as I've mentioned above: 

  • Any of the individual cores running "hot"?  (60% and above is a "worry", 80% and above is BAD.)
  • Any processing other than the one found in the master hogging the CPU? 

Those are the things to watch out.  

NOTE:  The command "sh proc cpu cpu" does not "apply" to switches running 16.X.X (and later).  The command above is the new "norm".  

This particular switch is not part of a stack, I'm just worried about all these punts to the CPU which are being dropped and have no idea what they are and if that is a problem. The process using the most cpu is fed main event at 40% but overall CPU avg is around 10% not sure if the FED process is related to the CPU queue drops on the High Rate App queue. 

 

 

Paul 

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Review Cisco Networking products for a $25 gift card