Hello,
We have come across an interesting problem with a network loop and would appreciate any suggestions in working back to the root cause of the problem.
In summary we have a switch that is forwarding traffic on VLANs regardless of the VLANs applied to its trunk interfaces or the spanning tree port state applied to them. In order to demonstrate this, and hopefully verify that our logic is sound, I have outlined troubleshooting steps below.
We have resisted the temptation to reboot the switch stack as we think this is a software issue and do not want to resolve it with a reboot before being able to work out exactly what has failed and why.
Our questions surrounding this issue are:
- What process/operation of the switch controls application of VLANs allowed on trunk ports and spanning tree processes? How can we verify their operation?
- What are the best debugs to look at in order to confirm the switches operation?
- Has anyone else seen this issue or come across any bugs etc that could explain it?
Initially we suspected STP however after identifying the switch that is the source of the loop we disconnected one of the uplinks by shutting down its opposing end in our core. This broke the loop.
We found that we could remove all VLAN’s from the trunk interface on the core to one interface to the switch and re-enable it without the loop reoccurring.
We then re-enabled each of the individual VLANs from the trunk on the core one at a time. In each instance the loop returned however, the effect was lesser on a per-VLAN scale and gave us enough time to examine the STP port states. We verified that STP was reporting blocked ports within each VLAN’s topology. STP was blocking on the edge switch identified as the source of the loop.
We noted that some VLAN’s carry more traffic and the impact of the loop is more dramatic depending on the expanse and number of VLANs enabled.
We then tried removing all VLAN’s from the edge switch trunk port and found that regardless of which VLANs are allowed on this interface the loop returns. With all VLAN’s removed from the edge switch trunk we see the same impact , VLAN per VLAN, when we add individual VLANs from the core side of the link.
This looks like an unusual issue to us so we thought this would be an interesting one to put out to the community.
Key points concerning our topology are:
Collapsed core topology
Edge switch configuration: 4x switch stack with uplinks from separate members
Replacement fibre, SFP’s and modules tried in different switch members – problem persists
Alternate edge switch tried in core ports and no loop issue was found
3750x Edge switch using IOS version: 15.0.2-SE2
2x 6509e cores using IOS version: 12.2(17r)SX6
Thanks
Andy
