Hi all,
I have a stack of 4 Catalyst 3850-24X working as a distribution switch, lying in between a Nexus 7K core switch and 34 C3850 access switches/stacks. I was struggling with high CPU utilization problems happening on the distribution switch. I upgraded the firmware from 16.03.06 to 16.12.5b without a noticeable change in CPU levels.
Mainly, the processes that eat the CPU are SISF Switcher Th, Spanning Tree, and Crimson flush tr. Sometimes, MATM RP Shim Pro and VMATM Callback spark enormously causing the switch to hit 100% and eventually leading to a network outage for a considerable amount of time.
I reviewed the STP configuration on the entire network to make sure there isn't a misconfig somewhere.
Here is show version output:
Cisco IOS XE Software, Version 16.12.05b
Cisco IOS Software [Gibraltar], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.12.5b, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2021 by Cisco Systems, Inc.
Compiled Thu 25-Mar-21 13:09 by mcpre
Cisco IOS-XE software, Copyright (c) 2005-2021 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.
ROM: IOS-XE ROMMON
BOOTLDR: CAT3K_CAA Boot Loader (CAT3K_CAA-HBOOT-M) Version 4.78, RELEASE SOFTWARE (P)
Switch uptime is 15 hours, 12 minutes
Uptime for this control processor is 15 hours, 15 minutes
System returned to ROM by Reload Command at 19:44:44 UTC Sun Nov 28 2021
System restarted at 19:50:28 UTC Sun Nov 28 2021
System image file is "flash:cat3k_caa-universalk9.16.12.05b.SPA.bin"
Last reload reason: Reload Command
This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.
A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
If you require further assistance please contact us by sending email to
export@cisco.com.
Technology Package License Information:
------------------------------------------------------------------------------
Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------------------
ipservicesk9 Smart License ipservicesk9
None Subscription Smart License None
Smart Licensing Status: UNREGISTERED/EVAL MODE
cisco WS-C3850-24XS (MIPS) processor (revision J0) with 794888K/6147K bytes of memory.
Processor board ID FCW2025F017
4 Virtual Ethernet interfaces
128 Ten Gigabit Ethernet interfaces
8 Forty Gigabit Ethernet interfaces
2048K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
255037K bytes of Crash Files at crashinfo:.
255037K bytes of Crash Files at crashinfo-2:.
255037K bytes of Crash Files at crashinfo-3:.
255037K bytes of Crash Files at crashinfo-4:.
3417161K bytes of Flash at flash:.
3417161K bytes of Flash at flash-2:.
3417161K bytes of Flash at flash-3:.
3417161K bytes of Flash at flash-4:.
0K bytes of WebUI ODM Files at webui:.
Base Ethernet MAC Address : 00:56:2b:d9:18:00
Motherboard Assembly Number : 73-16649-06
Motherboard Serial Number : FOC20237ZEH
Model Revision Number : J0
Motherboard Revision Number : A0
Model Number : WS-C3850-24XS
System Serial Number : FCW2025F017
Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 34 WS-C3850-24XS 16.12.05b CAT3K_CAA-UNIVERSALK9 BUNDLE
2 34 WS-C3850-24XS 16.12.05b CAT3K_CAA-UNIVERSALK9 BUNDLE
3 34 WS-C3850-24XS 16.12.05b CAT3K_CAA-UNIVERSALK9 BUNDLE
4 34 WS-C3850-24XS 16.12.05b CAT3K_CAA-UNIVERSALK9 BUNDLE
Switch 02
---------
Switch uptime : 15 hours, 15 minutes
Base Ethernet MAC Address : 00:56:2b:fb:b3:80
Motherboard Assembly Number : 73-16649-06
Motherboard Serial Number : FOC20237ZF2
Model Revision Number : J0
Motherboard Revision Number : A0
Model Number : WS-C3850-24XS
System Serial Number : FCW2025C0KA
Last reload reason : Reload Command
Switch 03
---------
Switch uptime : 15 hours, 15 minutes
Base Ethernet MAC Address : 00:56:2b:d9:71:80
Motherboard Assembly Number : 73-16649-06
Motherboard Serial Number : FOC20237ZG0
Model Revision Number : J0
Motherboard Revision Number : A0
Model Number : WS-C3850-24XS
System Serial Number : FCW2025C09R
Last reload reason : Reload Command
Switch 04
---------
Switch uptime : 15 hours, 15 minutes
Base Ethernet MAC Address : 00:56:2b:d8:cf:00
Motherboard Assembly Number : 73-16649-06
Motherboard Serial Number : FOC20237ZNA
Model Revision Number : J0
Motherboard Revision Number : A0
Model Number : WS-C3850-24XS
System Serial Number : FOC2024X19X
Last reload reason : Reload Command
Configuration register is 0x102A snapshot of CPU utilization:
CPU utilization for five seconds: 94%/18%; one minute: 94%; five minutes: 90%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
355 22207257 31706826 700 25.91% 22.36% 21.51% 0 SISF Switcher Th
100 6240314 300554 20762 20.47% 6.02% 6.38% 0 Crimson flush tr
250 9225912 11647469 792 9.67% 10.67% 12.53% 0 Spanning Tree
356 5010254 7654994 654 9.11% 5.43% 5.58% 0 SISF Main Thread
52 3076582 10132512 303 8.39% 3.35% 2.93% 0 ARP Snoop
126 3088462 29867557 103 3.03% 3.72% 3.61% 0 IOSXE-RP Punt Se
324 798535 10202444 78 1.43% 2.52% 2.16% 0 DAI Packet Proce
174 335234 539281 621 0.95% 4.32% 2.99% 0 MATM RP Shim Pro
80 525196 1199488 437 0.87% 2.43% 2.09% 0 IOSD ipc task
222 199145 223304 891 0.23% 0.23% 0.23% 0 CDP Protocol
539 136525 385569 354 0.15% 0.16% 0.16% 0 LLDP Protocol
305 287384 1185684 242 0.15% 3.31% 1.86% 0 IGMPSN
398 60295 1230894 48 0.15% 0.06% 0.06% 0 MMA DB TIMER
98 97747 555629 175 0.15% 0.58% 0.46% 0 cpf_process_tpQ
431 60906 1230847 49 0.15% 0.05% 0.06% 0 MMA DP TIMER
432 58013 2445117 23 0.15% 0.04% 0.05% 0 MMON MENG
15 41144 402680 102 0.07% 0.02% 0.03% 0 DB Lock Manager
538 30671 397301 77 0.07% 0.04% 0.02% 0 ONEP Network Ele
149 15627 32888 475 0.07% 0.02% 0.00% 0 SFF8472
204 60996 1230802 49 0.07% 0.04% 0.05% 0 VRRS Main thread
Any help would be highly appreciated.
