Layer 3 Switch Utilization is High

[maxblaze1992]

Just wondering if it can cause slow down on my production network since 1 out of 4 cpu is on 99% utilization?

but the rest still quite low. See screenshot for your reference. 

Model Number : WS-C3850-24T

[Reza Sharifi]

That is not normal and can eventually cause problems. I would open a ticket with TAC and have them investigate the high CPU utilization. It may a bug in the version of the software you are running.

HTH

[Utkarsh Dewangan]

Hi,

Which Is the IOS Version you are running?

There had been Some instances of buggy behaviour on 3850 running 3.6.x code and 3.7.x code.

there could be a hit to these below

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuz57493/

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCux39490/

check with following

sh proc cpu sort | e 0.0                   

! check if you are seeing fed, pdsd, stack-mgr as top processes

sh proc cpu det proc fed sort | e 0.0

! check if you are seeing fed-ots-main ,fed as top processes

!check with these output continuesly for the cpu queues traffic punting

show platform punt statistics port-asic 0 cpuq -1 direction rx | beg CPU_Q_FORUS_TRAFFIC

If the above is the case, consider upgrading the switch.

you can also refer to following if its specific packet which is hitting CPU

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/117594-technote-hicpu3850-00.html#anc8

https://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/ios-xe-16/213549-troubleshoot-high-cpu-usage-in-catalyst.html

[maxblaze1992]

Hi Mr. Utkarsh Please see screenshot after executing the sh command you provided. can you recommend what should be done next, base on the result? Thanks,

[maxblaze1992]

sh version
Cisco IOS Software, IOS-XE Software, Catalyst L3 Switch Software (CAT3K_CAA-UNIV ERSALK9-M), Version 03.06.05E RELEASE SOFTWARE (fc2)

[Mark Malone]

Hi
dump the show tech into the Cisco analyzer to see if its a bug , if you dont have it setup , post the show tech ill run it for you and provide the output, should confirm if its a bug your hitting

[maxblaze1992]

Hi Mr. Malone will it cause downtime running show tech command?

[brselzer]

Hello,

You are probably hitting one of the issues that Utkarsh mentioned:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuz57493/

It is not fixed until 3.6.6. I believe 3.6.10 is out if you want to stay on the same train or move to the recommended code of 16.6.5. I would upgrade this device when you get a chance.

Hope that helps!

[Mark Malone]

Just to add as ive gone through some of these jumps recently , If your upgrading from bundle to install (all 16 versions are install now) and from 3.6 to 16 make sure to read the release notes for the image your jumping too , there's definitely some points you need to follow , their will be microcode and bootloader upgrades jumping that high but it works fine .You dont want to brick the switch.IT will be fine once you follow the upgrade path on the release notes its well documented in each guide and easy to follow

[paraina]

I don't think it will cause the network super slow as only one core at 99%, And the rest of core are below 20%, It could be due to one single instance. </p><p class="p1">But i would say keep eyes on CPU if it constantly shows 99% even-though for one core, In that case you might need to look, if it due to legitimate reason or is something else which may eventually impact other core as well and ended up with slow network or other intrusive impacts.

[Joseph W. Doherty]

Although mitigating this issue is likely worthwhile for other reasons, normally data plane forwarding, in a switch, is done with dedicated ASICs so normal data forwarding shouldn't be much impacted even if all 4 CPU were running at 100% all the time.

Where the CPU load can become an issue is for any traffic "punted" to the CPU and/or for control plane functions that modify ASIC programming (e.g. a change in routing topology).