07-11-2022 11:58 AM
Hi, everyone!
Have two core catalyst 3750X switches in a stack with 15.2(1)E IOS version. When I add VLAN to configuration (doesn't matter what number it is) there is an increase in CPU utilization:
CPU utilization for five seconds: 99%/1%; one minute: 99%; five minutes: 99%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
247 2732742 130875 20880 59.29% 58.81% 58.65% 0 VMATM Callback
159 3503376928 3283516600 0 11.05% 11.62% 11.53% 0 Hulc LED Process
162 786990505 109715445 7173 5.60% 5.18% 5.20% 0 HL3U bkgrd proce
Also all network goes down like in broadcast-storm and on neighbour switches bpduguard is triggered in added VLAN:
%SPANTREE-CLUSTER_MEMBER_13-2-LOOPGUARD_BLOCK: Loop guard blocking port Port-channel1 on VLAN0210.
%SPANTREE-CLUSTER_MEMBER_13-2-LOOPGUARD_UNBLOCK: Loop guard unblocking port Port-channel1 on VLAN0210.
%SPANTREE-CLUSTER_MEMBER_13-2-LOOPGUARD_BLOCK: Loop guard blocking port Port-channel1 on VLAN0210.
But there are no loops on physical topology level.
Logs on 3750X contain this:
%HLFM-6-ADDR_SYNC: The system is retrying to sync the MAC addresses: reason 2
%HLFM-3-SEND_FAIL: Failed to send RPC message, req 3,
-Traceback= 67945Cz 26880C0z 2D61754z 2D61FECz 2D5EEF0z 2D5F044z 2D5771Cz 2D51C54z
%HLFM-3-SEND_FAIL: Failed to send RPC message, req 3,
-Traceback= 67945Cz 26880C0z 2D61754z 2D61FECz 2D5EEF0z 2D5F044z 2D5
%HLFM-3-SEND_FAIL: Failed to send RPC message, req 3,
-Traceback= 67945Cz 26880C0z 2D61754z 2D61FECz 2D5EEF0z 2D5F044z 2D5771Cz 2D51C54z
%HLFM-3-SEND_FAIL: Failed to send RPC message, req 3,
-Traceback= 67945Cz 26880C0z 2D61754z 2D61FECz 2D5EEF0z 2D5F044z 2D5771Cz 2D51C54z
%HLFM-6-ADDR_SYNC: The system is retrying to sync the MAC addresses: reason 2
%HLFM-3-SEND_FAIL: Failed to send RPC message, req 3,
-Traceback= 67945Cz 26880C0z 2D61754z 2D61Ftopoid 0
No entries in MAC-table in added VLAN.
Switches in a stack functioned in vtp client mode (don't ask why...). I tried to turn them in transparent mode, but that didn't help. Also I tried to replace one useless VLAN (there are 77 VLANs in total) on the new one, but unsuccessfully. Moreover when I returned deleted VLAN in configuration CPU rised to 99% and the same logs appeared in a log buffer.
I found similar problem: https://bst.cisco.com/bugsearch/bug/CSCto85486 and tried to capture BPDU frames with topology change flag. But there is no such traffic in captured dump. Perhaps the problem lies in VP instances. Unfortunately I have no strong knowledges about that and didn't find commands to disable it on 3750X.
Maybe someone have some ideas?
I think it is a software bug and don't know how to solve this problem (organization doesn't have service contract to download new IOS version or start a ticket in Cisco TAC). My last step is just reload the device. Maybe somebody know decision that doesn't require rebooting?
Solved! Go to Solution.
07-11-2022 02:15 PM - edited 07-11-2022 02:16 PM
Switch Uptime : 4 years, 13 weeks, 1 day, 10 hours, 49 minutes
The first step i will do is take the config backup out of the box, and take the maintenance window -reloading the stack will be advised here.
07-11-2022 12:02 PM
Forgot to mention that I had already added VLANs earlier and everything was fine
07-11-2022 12:31 PM
Are you config MST as stp mode?
07-11-2022 01:07 PM
Switch functions in rapid PVST mode
07-11-2022 12:37 PM
Post below information :
show version
show swtich
show vlan
show mac address (count)
show spann brief
Mostly you see this kind of problem, when you have stack switches or Long uptime of the device.
07-11-2022 01:16 PM
07-11-2022 02:15 PM - edited 07-11-2022 02:16 PM
Switch Uptime : 4 years, 13 weeks, 1 day, 10 hours, 49 minutes
The first step i will do is take the config backup out of the box, and take the maintenance window -reloading the stack will be advised here.
07-11-2022 11:31 PM
I thought about it first of all. But I'm afraid of reload the switch beacuse of so long uptime))) Maybe will try this week.
07-11-2022 02:01 PM
Hello,
have you tried to reload the switch(es) after adding the Vlan ?
07-11-2022 11:29 PM
No
07-11-2022 03:45 PM
Upgrade the firmware of the switch. 15.2(1)E is not a version I want to be caught with.
07-11-2022 11:28 PM
Organization doesn't have service contract and I can't download actual IOS version( Am I missing something?
07-12-2022 12:04 AM - edited 07-12-2022 01:27 AM
1. Read Cisco IOS and IOS XE Software Smart Install Denial of Service Vulnerability
2. Scroll down to the "Customers Without Service Contracts" section and read it very carefully:
Customers who purchase directly from Cisco but do not hold a Cisco service contract and customers who make purchases through third-party vendors but are unsuccessful in obtaining fixed software through their point of sale should obtain upgrades by contacting the Cisco TAC.
Customers should have the product serial number available and be prepared to provide the URL of this advisory as evidence of entitlement to a free upgrade.
3. Contact Cisco TAC via email (do not call TAC). Ask them to publish the IOS file: c3750e-universalk9-tar.152-4.E10.tar
VERY IMPORTANT:
Do not delay. 3750X is already past End-of-Support date and it will not be long before the IOS files will no longer be downloadable.
07-11-2022 10:48 PM
Hello
I woud suggest possible bug given the small amount of vlans you have running, curious though can you check on the total numnber of active logical stp instances
sh stp summary detail
07-12-2022 12:42 AM
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide