HIGH CPU on 3560 L2

[Steve Coady]

OUTSIDE-DIST#sh proc cpu | ex 0.0
Core 0: CPU utilization for five seconds: 85%; one minute: 86%; five minutes: 88%
Core 1: CPU utilization for five seconds: 83%; one minute: 93%; five minutes: 83%
Core 2: CPU utilization for five seconds: 100%; one minute: 94%; five minutes: 83%
Core 3: CPU utilization for five seconds: 43%; one minute: 36%; five minutes: 54%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
5673 4265548 378885900 7185 24.7 24.6 24.6 0 stack-mgr
6203 2759041 271351187 3285 0.39 0.24 0.23 0 pdsd
10145 2526533 613459530 110 2.56 1.71 1.94 0 iosd

I need assistance determining why this is occurring. 

The switch/os are "old"

    OUTSIDE-DIST uptime is 5 years, 17 weeks, 4 days, 9 minutes

    cat3k_caa-universalk9

[balaji.bandi]

5 years too big uptime required to reboot and check

also post show version to see any bugs.

[Steve Coady]

Thank you Balaji

OUTSIDE-DIST#sh ver
Cisco IOS Software, IOS-XE Software, Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 03.03.05SE RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2014 by Cisco Systems, Inc.
Compiled Thu 30-Oct-14 13:12 by prod_rel_team

Cisco IOS-XE software, Copyright (c) 2005-2014 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0.
(http://www.gnu.org/licenses/gpl-2.0.html) For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: IOS-XE ROMMON
BOOTLDR: CAT3K_CAA Boot Loader (CAT3K_CAA-HBOOT-M) Version 1.2, RELEASE SOFTWARE (P)

OUTSIDE-DIST uptime is 5 years, 17 weeks, 4 days, 1 hour, 47 minutes
Uptime for this control processor is 5 years, 17 weeks, 4 days, 1 hour, 50 minutes
System returned to ROM by reload
System image file is "flash:packages.conf"
Last reload reason: reload

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

License Level: Lanbase
License Type: Permanent
Next reload license Level: Lanbase

cisco WS-C3650-24TD (MIPS) processor with 4194304K bytes of physical memory.
Processor board ID
1 Virtual Ethernet interface
52 Gigabit Ethernet interfaces
4 Ten Gigabit Ethernet interfaces
2048K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
257008K bytes of Crash Files at crashinfo:.
257008K bytes of Crash Files at crashinfo-2:.
1550272K bytes of Flash at flash:.
1550272K bytes of Flash at flash-2:.
0K bytes of Dummy USB Flash at usbflash0:.
0K bytes of Dummy USB Flash at usbflash0-2:.
0K bytes of at webui:.

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 28 WS-C3650-24TD 03.03.05SE cat3k_caa-universalk9 INSTALL
2 28 WS-C3650-24TD 03.03.05SE cat3k_caa-universalk9 INSTALL

Configuration register is 0x102

OUTSIDE-DIST#

[Reza Sharifi]

These processes are not running high for 1 min and 5 min regardless of how long the switch has been up and running.

5673 4265548 378885900 7185 24.7 24.6 24.6 0 stack-mgr
6203 2759041 271351187 3285 0.39 0.24 0.23 0 pdsd
10145 2526533 613459530 110 2.56 1.71 1.94 0 iosd

So, it could be that the switch was always running high and you just noticed it or there is a different bug causing a high CPU.

Is this causing any production issues?

HTH

[Steve Coady]

We are experiencing some issues with audio/vid during Microsoft TEAMS and Webex meetings for a large group (65) participants. Only a handful of the 65 experience problems.  The problems do occur each meeting for these (65) participants. It is not the same participants each time.

The topo is:  Internet->ASR->(3560 switch) ASA->Core switches->LAN

[Reza Sharifi · Accepted solution]

Is this a stack of 2 3650s? because the post title is:

"HIGH CPU on 3560 L2" and also in this line

The topo is:  Internet->ASR->(3560 switch) ASA->Core switches->LAN

you still have 3560.

This was confusing, as I have never seen an old 3560 with 4 core CPUs.

Anyway, both the Cisco Webex and MS Team are cloud-based solutions. If you have a SIP trunk with a provider, you may want to check and make sure you have no audio issues with them. Also, was this working before?  Did anyone make any changes to the switch or other devices especially QoS, configs? We had so many issues with Webex and had to switch to Zoom to stop the complaints about audio delays. Don't use Team much but I hear it is pretty solid. Anyway, if you think the problem is in the 3650s, in a maintenance window you can reboot the stack and see if the problem disappears.

HTH

[MHM Cisco World · Accepted solution]