Catalyst 3750X VLAN adding issue

[viktar23]

Hi, everyone!

Have two core catalyst 3750X switches in a stack with 15.2(1)E IOS version. When I add VLAN to configuration (doesn't matter what number it is) there is an increase in CPU utilization:

CPU utilization for five seconds: 99%/1%; one minute: 99%; five minutes: 99%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
247 2732742 130875 20880 59.29% 58.81% 58.65% 0 VMATM Callback
159 3503376928 3283516600 0 11.05% 11.62% 11.53% 0 Hulc LED Process
162 786990505 109715445 7173 5.60% 5.18% 5.20% 0 HL3U bkgrd proce

Also all network goes down like in broadcast-storm and on neighbour switches bpduguard is triggered in added VLAN:

%SPANTREE-CLUSTER_MEMBER_13-2-LOOPGUARD_BLOCK: Loop guard blocking port Port-channel1 on VLAN0210.
%SPANTREE-CLUSTER_MEMBER_13-2-LOOPGUARD_UNBLOCK: Loop guard unblocking port Port-channel1 on VLAN0210.
%SPANTREE-CLUSTER_MEMBER_13-2-LOOPGUARD_BLOCK: Loop guard blocking port Port-channel1 on VLAN0210.

But there are no loops on physical topology level. 

Logs on 3750X contain this:

%HLFM-6-ADDR_SYNC: The system is retrying to sync the MAC addresses: reason 2
%HLFM-3-SEND_FAIL: Failed to send RPC message, req 3,
-Traceback= 67945Cz 26880C0z 2D61754z 2D61FECz 2D5EEF0z 2D5F044z 2D5771Cz 2D51C54z
%HLFM-3-SEND_FAIL: Failed to send RPC message, req 3,
-Traceback= 67945Cz 26880C0z 2D61754z 2D61FECz 2D5EEF0z 2D5F044z 2D5
%HLFM-3-SEND_FAIL: Failed to send RPC message, req 3,
-Traceback= 67945Cz 26880C0z 2D61754z 2D61FECz 2D5EEF0z 2D5F044z 2D5771Cz 2D51C54z
%HLFM-3-SEND_FAIL: Failed to send RPC message, req 3,
-Traceback= 67945Cz 26880C0z 2D61754z 2D61FECz 2D5EEF0z 2D5F044z 2D5771Cz 2D51C54z
%HLFM-6-ADDR_SYNC: The system is retrying to sync the MAC addresses: reason 2
%HLFM-3-SEND_FAIL: Failed to send RPC message, req 3,
-Traceback= 67945Cz 26880C0z 2D61754z 2D61Ftopoid 0

No entries in MAC-table in added VLAN.

Switches in a stack functioned in vtp client mode (don't ask why...). I tried to turn them in transparent mode, but that didn't help. Also I tried to replace one useless VLAN (there are 77 VLANs in total) on the new one, but unsuccessfully. Moreover when I returned deleted VLAN in configuration CPU rised to 99% and the same logs appeared in a log buffer.

I found similar problem: https://bst.cisco.com/bugsearch/bug/CSCto85486 and tried to capture BPDU frames with topology change flag. But there is no such traffic in captured dump. Perhaps the problem lies in VP instances. Unfortunately I have no strong knowledges about that and didn't find commands to disable it on 3750X.

Maybe someone have some ideas?

I think it is a software bug and don't know how to solve this problem (organization doesn't have service contract to download new IOS version or start a ticket in Cisco TAC). My last step is just reload the device. Maybe somebody know decision that doesn't require rebooting?

[viktar23]

Forgot to mention that I had already added VLANs earlier and everything was fine

[MHM Cisco World]

Are you config MST  as stp mode?

[viktar23]

Switch functions in rapid PVST mode

[balaji.bandi]

Post below information :

show version

show swtich

show vlan

show mac address  (count)

show spann brief

Mostly you see this kind of problem, when you have stack switches or Long uptime of the device.

[balaji.bandi · Accepted solution]

Switch Uptime : 4 years, 13 weeks, 1 day, 10 hours, 49 minutes

The first step i will do is  take the config backup out of the box, and take the maintenance window -reloading the stack will be advised here.

[viktar23]

I thought about it first of all. But I'm afraid of reload the switch beacuse of so long uptime))) Maybe will try this week.

[Georg Pauwen]

Hello,

have you tried to reload the switch(es) after adding the Vlan ?

[viktar23]

No

[Leo Laohoo]

Upgrade the firmware of the switch.  15.2(1)E is not a version I want to be caught with.

[viktar23]

Organization doesn't have service contract and I can't download actual IOS version( Am I missing something?

[Leo Laohoo]

1.  Read Cisco IOS and IOS XE Software Smart Install Denial of Service Vulnerability

2.  Scroll down to the "Customers Without Service Contracts" section and read it very carefully: 

Customers who purchase directly from Cisco but do not hold a Cisco service contract and customers who make purchases through third-party vendors but are unsuccessful in obtaining fixed software through their point of sale should obtain upgrades by contacting the Cisco TAC.

Customers should have the product serial number available and be prepared to provide the URL of this advisory as evidence of entitlement to a free upgrade.

---

3.  Contact Cisco TAC via email (do not call TAC).  Ask them to publish the IOS file:  c3750e-universalk9-tar.152-4.E10.tar

VERY IMPORTANT: 

Do not delay.  3750X is already past End-of-Support date and it will not be long before the IOS files will no longer be downloadable. 

[paul driver]

Hello
I woud suggest possible bug given the small amount of vlans you have running, curious though can you check on the total numnber of active logical stp instances
sh stp summary detail