High CPu on 3850 switches

[q-le]

Hi All,

We experience high CPU usage on Stack 3850 switches and currently running

 WS-C3850-48P 03.06.04.E cat3k_caa-universalk9 INSTALL.

Any recommendation for new stable IOS please.

Thanks

PL

[Leo Laohoo]

Post the complete output to the command "sh proc cpu sort | ex 0.00". 

What happens if configure the command "ipv6 mld snooping"? 

[q-le]

S-LAS-B2Utility#sh proc cpu sort | ex 0.00
Core 0: CPU utilization for five seconds: 20%; one minute: 51%; five minutes: 61%
Core 1: CPU utilization for five seconds: 99%; one minute: 88%; five minutes: 89%
Core 2: CPU utilization for five seconds: 92%; one minute: 82%; five minutes: 81%
Core 3: CPU utilization for five seconds: 99%; one minute: 98%; five minutes: 93%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
5848 1697219 13106202 1620 49.26 50.01 49.93 0 stack-mgr
5846 3930459 20739688 103 24.97 26.33 26.42 0 fed
6396 2218868 17349166 595 15.92 1.36 0.70 0 eicored
8769 2306305 24576228 199 5.09 3.84 4.22 0 iosd
6425 1719735 68369062 109 0.99 0.08 0.06 0 snmp_subagent
8765 3469983 41688429 13 0.20 0.11 0.10 0 wcm
31 4226208 57483114 29 0.05 0.03 0.03 0 sirq-net-rx/2
6387 2351556 27692969 33 0.05 0.05 0.05 0 pdsd
6398 1928093 83181685 849 0.05 0.04 0.05 0 cpumemd

After we config "ipv6 mld snooping"

SOH-LAS-B2Utility#sh proc cpu sort | ex 0.00
Core 0: CPU utilization for five seconds: 34%; one minute: 44%; five minutes: 58%
Core 1: CPU utilization for five seconds: 98%; one minute: 94%; five minutes: 90%
Core 2: CPU utilization for five seconds: 97%; one minute: 96%; five minutes: 87%
Core 3: CPU utilization for five seconds: 99%; one minute: 87%; five minutes: 89%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
5848 2409699 13106869 1620 49.80 49.85 49.90 0 stack-mgr
5846 15101 20743310 103 26.13 26.45 26.40 0 fed
8769 2344755 24577481 199 4.29 3.25 3.84 0 iosd
6396 2230118 17349199 595 1.23 0.38 0.68 0 eicored
6425 1720455 66904267 111 0.29 0.03 0.04 0 snmp_subagent
5847 3984827 22451742 17 0.15 0.06 0.09 0 platform_mgr
8765 3472043 41689857 13 0.15 0.13 0.11 0 wcm
43 2857248 56668588 27 0.10 0.07 0.01 0 sirq-net-rx/3
6387 2352136 27693187 33 0.05 0.04 0.05 0 pdsd
6398 1928713 83182432 849 0.05 0.05 0.05 0 cpumemd
9541 3629082 79317841 99 0.05 0.05 0.03 0 wnweb_paster.py

[Leo Laohoo]

(I hope interface-level link state logging is enabled.)

Check the ports of the switch/stack if there is any that is flapping (CSCuo14511).

[q-le]

HI ,

These 7 switches stacked and only 1 port is flapping and should not cause CPU to almost 100% - i think the IOS has some bugs ?

[Leo Laohoo · Accepted solution]

IOS has bugs but the high CPU can be easily triggered by one flapping port.

[q-le]

found a flapping port- disconnected and CPU seems to be stable.

Please advise what is the best IOS for this model 3850.

Apparently we experienced some PoE (power in line) for our phones too  and after reboot the switch it fix the issues.

[Leo Laohoo]

Please advise what is the best IOS for this model 3850.

Stay with 3.6.X and upgrade to the latest.  Don't be tempted to go 3.7.X or 16.X.

found a flapping port- disconnected and CPU seems to be stable.

Happy to hear it's been identified.