Help Diagnosing Spanning Tree High CPU

[Andrew Bailey]

I am looking for some troubleshooting help for some Cisco blade switches that are running high CPU. I have two 3020 blade switches in an HP chassis that each have two 1G links port channeled a pair of Nexus 5548s. Spanning tree has been constantly running about 35% of CPU for the last couple of weeks causing management SVI latency and CLI lag. The Port channel is the root port and the switches have no other connections.

Here are the things I have tried in troubleshooting the issue.

-Remove links from port channel so that one is forwarding and one is blocking
-Removed the blocking link so that the switch only has one uplink.
-Converted from pvst to rstp
-Entered no spanning tree vlan <all vlans>  so when you do show spanning tree there are no instances of spanning tree
-Connected the single uplink to a different switch

Nothing has changed the continuous high spanning tree utilization of about 35%.

The 3020 switches server interfaces are configured as trunks for ESX running on the blades. It seems the only possible loop that could be causing this issue is on the ESX virtual switches, but I am not sure how that is possible. I say this because I have another pair of 3120s that have the exact same problem! However they were working fine (CPU normal) until the enclosure was populated and began switching traffic. After they began carrying a medium/heavy network switching load, the 3120s are running at a constant 56% spanning tree CPU utilization!

Any ideas?

[amohaida]

Hi Andrew,

could you please check the output "show spanning-tree details" , this output will give you an idea if there are topology changes and also the interface that receives those changes.

share the output with us if possible as well.

thanks,

Ahmad.

[Andrew Bailey]

I found the issue. I set up another stack of 3120s with the same configuration in an enclosure waiting to be provisioned. This enclosure has no blades in it and the spanning tree CPU utilization is about 12%. That is high for a switch doing nothing. I am guessing it would be higher if there was traffic on the switch. Cisco 3020s and 3120s only support 128 spanning tree instances. I have over 300 vlans. So the spanning tree process appears busy worrying about a possible loop on the vlans that do not have a spanning tree process, even though there is no loop because my switches have a single port channel. Maybe there is some other explanation. Anyway I found my problems after a couple days of troubleshooting!