cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2163
Views
0
Helpful
2
Replies

cisco catalyst 3850 high cpu due to xcvr in fed

sina.naser
Level 1
Level 1

Hi All

 

help me with this issue on my cisco 3850's cpu.

 

MD-3850-14#sh processes cpu sorted
Core 0: CPU utilization for five seconds: 84%; one minute: 81%; five minutes: 80%
Core 1: CPU utilization for five seconds: 99%; one minute: 99%; five minutes: 99%
Core 2: CPU utilization for five seconds: 17%; one minute: 21%; five minutes: 22%
Core 3: CPU utilization for five seconds: 3%; one minute: 6%; five minutes: 6%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
5826 949985 39322385 856 49.97 50.25 50.08 1088 fed
10317 1599961 27788751 102 1.68 1.48 1.36 34816 iosd
5828 3861166 32443121 107 0.25 0.21 0.20 0 stack-mgr
10313 2991654 28831246 12 0.20 0.10 0.10 0 wcm
10314 2605600 14925581 1 0.10 0.02 0.01 0 table_mgr
5827 533184 16117641 17 0.05 0.09 0.10 0 platform_mgr
6357 4267720 5636742 757 0.05 0.01 0.00 0 oom_poll.sh
6365 2926460 27283230 42 0.05 0.04 0.05 0 pdsd
6376 2894102 11927537 852 0.05 0.04 0.05 0 cpumemd
1 1680 1134 1481 0.00 0.00 0.00 0 init
2 0 131 0 0.00 0.00 0.00 0 kthreadd
3 4250 677393 6 0.00 0.00 0.00 0 migration/0
4 0 3 0 0.00 0.00 0.00 0 sirq-high/0
5 3292665 56959555 2 0.00 0.00 0.00 0 sirq-timer/0
6 0 8 0 0.00 0.00 0.00 0 sirq-net-tx/0
7 623550 9990435 62 0.00 0.00 0.00 0 sirq-net-rx/0
8 13090 2538975 5 0.00 0.00 0.00 0 sirq-block/0
9 0 8 0 0.00 0.00 0.00 0 sirq-block-iopo
10 0 497 0 0.00 0.00 0.00 0 sirq-tasklet/0
11 2890660 21567049 1 0.00 0.00 0.00 0 sirq-sched/0
12 71580 12810102 5 0.00 0.00 0.00 0 sirq-hrtimer/0
13 1414125 25130870 2 0.00 0.00 0.00 0 sirq-rcu/0
14 12730 2175546 5 0.00 0.00 0.00 0 desched/0
15 3010 441491 6 0.00 0.00 0.00 0 migration/1
16 0 15 0 0.00 0.00 0.00 0 sirq-high/1
17 3948730 28334872 1 0.00 0.01 0.02 0 sirq-timer/1
18 0 17 0 0.00 0.00 0.00 0 sirq-net-tx/1
19 2183290 55489226 39 0.00 0.00 0.00 0 sirq-net-rx/1
20 1510 354463 4 0.00 0.00 0.00 0 sirq-block/1
21 0 20 0 0.00 0.00 0.00 0 sirq-block-iopo
22 0 21 0 0.00 0.00 0.00 0 sirq-tasklet/1
23 1877100 16446361 1 0.00 0.00 0.00 0 sirq-sched/1
24 0 582109 0 0.00 0.00 0.00 0 sirq-hrtimer/1
25 3278910 26183044 1 0.00 0.00 0.00 0 sirq-rcu/1
26 10580 1104766 9 0.00 0.00 0.00 0 desched/1
27 8720 1381923 6 0.00 0.00 0.00 0 migration/2
28 0 27 0 0.00 0.00 0.00 0 sirq-high/2
29 4077222 25141316 3 0.00 0.01 0.03 0 sirq-timer/2
30 0 30 0 0.00 0.00 0.00 0 sirq-net-tx/2
31 3376090 66539079 50 0.00 0.02 0.02 0 sirq-net-rx/2
32 880 191478 4 0.00 0.00 0.00 0 sirq-block/2
33 0 32 0 0.00 0.00 0.00 0 sirq-block-iopo
34 0 33 0 0.00 0.00 0.00 0 sirq-tasklet/2
35 1650670 15886101 1 0.00 0.00 0.00 0 sirq-sched/2
36 0 330142 0 0.00 0.00 0.00 0 sirq-hrtimer/2
37 4291530 24228060 1 0.00 0.00 0.00 0 sirq-rcu/2
38 8740 1651178 5 0.00 0.00 0.00 0 desched/2
39 10530 1196517 8 0.00 0.00 0.00 0 migration/3
40 0 39 0 0.00 0.00 0.00 0 sirq-high/3
41 1060 23688145 0 0.00 0.00 0.00 0 sirq-timer/3
42 0 43 0 0.00 0.00 0.00 0 sirq-net-tx/3
43 2843280 63981496 44 0.00 0.00 0.02 0 sirq-net-rx/3
44 260 49708 5 0.00 0.00 0.00 0 sirq-block/3
45 0 44 0 0.00 0.00 0.00 0 sirq-block-iopo
46 0 45 0 0.00 0.00 0.00 0 sirq-tasklet/3
47 20 14732362 0 0.00 0.00 0.00 0 sirq-sched/3
48 0 237658 0 0.00 0.00 0.00 0 sirq-hrtimer/3
49 20 23452514 0 0.00 0.00 0.00 0 sirq-rcu/3
50 11450 1890186 6 0.00 0.00 0.00 0 desched/3
51 1766540 11567484 15 0.00 0.00 0.00 0 events/0
52 68750 56841774 1 0.00 0.00 0.00 0 events/1
53 82240 57230754 1 0.00 0.00 0.03 0 events/2
54 1400 59243969 0 0.00 0.00 0.00 0 events/3
55 0 54 0 0.00 0.00 0.00 0 cpuset
56 10 3150 3 0.00 0.00 0.00 0 khelper
59 0 56 0 0.00 0.00 0.00 0 netns
60 0 61 0 0.00 0.00 0.00 0 async/mgr
165 107530 9475776 11 0.00 0.00 0.00 0 sync_supers
167 90 189591 0 0.00 0.00 0.00 0 bdi-default
169 7760 2045551 3 0.00 0.00 0.00 0 kblockd/0
170 390 291513 1 0.00 0.00 0.00 0 kblockd/1
171 220 156946 1 0.00 0.00 0.00 0 kblockd/2
172 30 40412 0 0.00 0.00 0.00 0 kblockd/3
178 0 67 0 0.00 0.00 0.00 0 ata/0
179 0 67 0 0.00 0.00 0.00 0 ata/1
180 0 66 0 0.00 0.00 0.00 0 ata/2
181 0 67 0 0.00 0.00 0.00 0 ata/3
182 0 68 0 0.00 0.00 0.00 0 ata_aux
191 0 71 0 0.00 0.00 0.00 0 khubd
221 50 262 190 0.00 0.00 0.00 0 rpciod/0
222 10 239 41 0.00 0.00 0.00 0 rpciod/1
223 80 652 122 0.00 0.00 0.00 0 rpciod/2
224 10 671 14 0.00 0.00 0.00 0 rpciod/3
258 0 78 0 0.00 0.00 0.00 0 kswapd0
259 0 79 0 0.00 0.00 0.00 0 aio/0
260 0 80 0 0.00 0.00 0.00 0 aio/1
261 0 81 0 0.00 0.00 0.00 0 aio/2
262 0 82 0 0.00 0.00 0.00 0 aio/3
263 0 83 0 0.00 0.00 0.00 0 nfsiod
265 0 84 0 0.00 0.00 0.00 0 crypto/0
266 0 85 0 0.00 0.00 0.00 0 crypto/1
267 0 86 0 0.00 0.00 0.00 0 crypto/2
268 0 87 0 0.00 0.00 0.00 0 crypto/3
814 0 99 0 0.00 0.00 0.00 0 scsi_eh_0
823 259230 34986 7409 0.00 0.00 0.00 0 mtdblockd
898 0 96 0 0.00 0.00 0.00 0 kstriped
900 0 97 0 0.00 0.00 0.00 0 ksnapd
903 32520 11303748 0 0.00 0.00 0.00 0 edac-poller
938 99110 66721427 1 0.00 0.01 0.02 0 octeon-ethernet
1003 450 1308 344 0.00 0.00 0.00 0 udevd
1399 8460 10616795 0 0.00 0.00 0.00 0 flush-8:0
1654 60 3693 16 0.00 0.00 0.00 0 loop0
1672 60 4509 13 0.00 0.00 0.00 0 loop1
1690 130 2785 46 0.00 0.00 0.00 0 loop2
1708 220 4095 53 0.00 0.00 0.00 0 loop3
1726 80 1847 43 0.00 0.00 0.00 0 loop4
1811 0 399 0 0.00 0.00 0.00 0 loop5
2445 10 1096 9 0.00 0.00 0.00 0 udevd
2447 0 1060 0 0.00 0.00 0.00 0 udevd
3429 120 1174 102 0.00 0.00 0.00 0 dbus-daemon
3809 120 1603 74 0.00 0.00 0.00 0 klogd
3821 20 1089 18 0.00 0.00 0.00 0 portmap
3889 0 1018 0 0.00 0.00 0.00 0 sshd
3901 0 1055 0 0.00 0.00 0.00 0 xinetd
3905 0 1046 0 0.00 0.00 0.00 0 xinetd
3939 0 112 0 0.00 0.00 0.00 0 lockd
3940 10 15906 0 0.00 0.00 0.00 0 nfsd
3941 10 15907 0 0.00 0.00 0.00 0 nfsd
3942 0 15908 0 0.00 0.00 0.00 0 nfsd
3943 0 15909 0 0.00 0.00 0.00 0 nfsd
3944 10 15995 0 0.00 0.00 0.00 0 nfsd
3945 10 15912 0 0.00 0.00 0.00 0 nfsd
3946 0 15997 0 0.00 0.00 0.00 0 nfsd
3947 10 15912 0 0.00 0.00 0.00 0 nfsd
3953 20 1113 17 0.00 0.00 0.00 0 rpc.mountd
4353 10 1175 8 0.00 0.00 0.00 0 sh
4356 0 1175 0 0.00 0.00 0.00 0 inotifywait
4429 0 5636743 0 0.00 0.00 0.00 34816 sleep
4910 0 1233 0 0.00 0.00 0.00 0 sh
4913 150 1243 120 0.00 0.00 0.00 0 epc_sm_liaison
5561 3768238 11367633 146 0.00 0.01 0.04 0 system_mgr
5784 1052580 68222145 15 0.00 0.00 0.00 1088 slproc
6303 527445 11356514 80 0.00 0.01 0.04 0 idope.py
6355 894410 62529312 14 0.00 0.00 0.00 0 ns_oir_proxy
6356 899920 62534370 14 0.00 0.01 0.03 0 profiled
6358 789692 26353634 19 0.00 0.01 0.01 0 obfld
6359 899230 62534333 14 0.00 0.04 0.05 0 consoled
6360 227015 12022979 73 0.00 0.01 0.00 0 console_relay
6361 907250 62534364 14 0.00 0.00 0.00 0 csprovider
6362 892470 62534359 14 0.00 0.01 0.01 0 system_mgr_cli
6363 4106835 29493208 43 0.00 0.01 0.03 0 plogd
6364 896700 62534424 14 0.00 0.00 0.00 0 psdprov
6366 2587250 70830417 36 0.00 0.01 0.03 0 gold_slave
6367 2144355 80834115 132 0.00 0.02 0.15 0 osinfo-provider
6368 895170 62534401 14 0.00 0.00 0.00 0 oscore_p
6369 2157130 69250212 31 0.00 0.00 0.00 0 netd
6370 268343 11362490 191 0.00 0.01 0.01 0 mem_mgmt
6371 2284250 11937368 19 0.00 0.01 0.03 0 mgmte_tap
6372 892510 62589466 14 0.00 0.00 0.00 0 installer
6373 351702 13540572 34 0.00 0.00 0.02 0 ffm
6374 1828102 82078170 1284 0.00 1.27 0.97 0 eicored
6375 915190 64432473 14 0.00 0.00 0.00 0 dtmgr
6382 2253880 12043103 18 0.00 0.02 0.00 0 liin_tap
6388 1355760 63822324 21 0.00 0.00 0.00 0 cli_agent
6390 298297 14231168 142 0.00 0.11 0.06 0 snmp_subagent
6391 1834000 11937801 15 0.00 0.00 0.01 0 erm_p
6393 1632790 11937845 13 0.00 0.00 0.00 0 ngdumper_provid
6395 1620410 11937844 13 0.00 0.00 0.02 0 sysinfo_p
6405 898530 62527965 14 0.00 0.00 0.00 0 fs_cmd
8002 480 1429 335 0.00 0.00 0.00 0 rollback_timer.
8197 0 1419 0 0.00 0.00 0.00 0 rollback_timer.
8199 160 1419 112 0.00 0.00 0.00 0 xlogger
8204 0 1423 0 0.00 0.00 0.00 0 inotifywait
8384 170 1241 136 0.00 0.00 0.00 0 nova_sntp.sh
8609 0 1239 0 0.00 0.00 0.00 0 nova_sntp.sh
8613 160 1239 129 0.00 0.00 0.00 0 xlogger
8615 10 1245 8 0.00 0.00 0.00 0 sntp
10315 70 1455 48 0.00 0.00 0.00 0 wnweb.sh
10316 897290 62535770 14 0.00 0.01 0.03 0 iifd
10318 243472 85820938 52 0.00 0.03 0.04 0 ha_mgr
11223 2051195 11370923 93 0.00 0.00 0.00 34816 wnweb_paster.py
11224 420 1455 288 0.00 0.00 0.00 34816 xlogger
27563 100 1053 94 0.00 0.00 0.00 34816 in.telnetd
27564 110 1051 104 0.00 0.00 0.00 34816 bexec.sh
27567 10 1051 9 0.00 0.00 0.00 34816 mcp_chvrf.sh
27568 110 1061 103 0.00 0.00 0.00 34816 vty_attach.sh

 

 

then i realized it's about fed and issued the below command

 

MD-3850-14#sh processes cpu detailed process fed sorted | ex 0.0
Core 0: CPU utilization for five seconds: 85%; one minute: 94%; five minutes: 92%
Core 1: CPU utilization for five seconds: 100%; one minute: 79%; five minutes: 56%
Core 2: CPU utilization for five seconds: 17%; one minute: 32%; five minutes: 55%
Core 3: CPU utilization for five seconds: 3%; one minute: 9%; five minutes: 7%
PID T C TID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
(%) (%) (%)
5826 L 0 10038 3167205 3338213 0 21.39 21.39 21.27 0 Xcvr
5826 L 2 11202 2203540 2563599 0 3.49 3.44 3.46 0 XcvrPoll
5826 L 2 6269 501152 5616607 0 0.29 0.26 0.25 0 fed-ots-nfl
5826 L 2 6266 3304577 7233632 0 0.19 0.21 0.19 0 fed-ots-main

 

how can I terminate xcvr process?

this is the template: 

MD-3850-14#sh sdm prefer
Showing SDM Template Info

This is the Advanced (high scale) template.
Number of VLANs: 4094
Unicast MAC addresses: 32768
Overflow Unicast MAC addresses: 512
IGMP and Multicast groups: 8192
Overflow IGMP and Multicast groups: 512
Directly connected routes: 16384
Indirect routes: 7168
Security Access Control Entries: 3072
QoS Access Control Entries: 3072
Policy Based Routing ACEs: 1024
Netflow ACEs: 768
Wireless Input Microflow policer ACEs: 256
Wireless Output Microflow policer ACEs: 256
Flow SPAN ACEs: 512
Tunnels: 256
Control Plane Entries: 512
Input Netflow flows: 8192
Output Netflow flows: 16384
SGT/DGT entries: 4096
SGT/DGT Overflow entries: 512
These numbers are typical for L2 and IPv4 features.
Some features such as IPv6, use up double the entry size;
so only half as many entries can be created.

tnx in advance

2 Replies 2

Seb Rupik
VIP Alumni
VIP Alumni

Hi there,

Could be this bug:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuz57493

 

What version of code are you running?

 

Cheers,

Seb.

Hello,

 

recommended release for the 3850 is Denali-16.3.7 MD...

Review Cisco Networking for a $25 gift card