STP root bridge change/fluke

[inetupto9]

Hello,

First time posting, so please bear with me if I did not get all the information relating to our issue at first.

We have a pair (StackWise) of Cisco 3750G-24T-S switches that act as our "core" and then a spattering of around 30 "access" switches (ranging from 2950T, to 3560 & 3560G, to 2960S series). Setup has worked well, otherwise.  A few weeks a go, the decision was made to create a new VLAN for new project and when I went to create it on our core switches, I discovered it reported it was no longer the STP root and that this had now moved to one of our newer 2960S series rack switches over a year ago.  Yet no one can ever recall logging into the switches on that date to make any changes -- as confirmed by syslog.  It's not often new broadcast domains (VLANs) need to be created, and the last one was created in November 2013.  I have questions as the STP root bridge has unexpectedly moved to another switch -- at least for one VLAN -- and all the other switches in our layer-2 infrastructure all report this other switch as the root bridge.  I have concerns to create a new VLAN before resolving the issue and moving the STP root back to our 3750G cores.

I've attached a text document to illustrate the oddity with selective spanning-tree output on both the "core" and the aforementioned rack switch.  I am curious if anyone has encountered this before and what they did to resolve it.

Troubleshooting steps I've taken, with no change in topology or root bridge:

1) Lower the STP priority on the 3750G "core" switches

2) Increase the STP priority for "VLAN 444" on the "Rack-2A" switch to greater-than 32,768 (i.e. 40,960)

3) Reboot "core" switch stack.

4) Reboot the misbehaving rack switch.

Ideas to fix/resolve the issue permanently:

1) Delete "VLAN 444" on the misbehaving "Rack-2A" switch, then on the "core" switches.

2) Re-create (or create a different) VLAN 444 on "core"; the true STP root bridge (switches). Lower STP priority on "core" to 4096, then back to force topology change.

3) Unjoin "Rack-2A" switch from VTP domain, delete 'vlan.dat' from flash; re-join VTP domain and re-create switch configuration.

4) Migrate all other non-core switches to 'vtp mode client' to alleviate this headache from returning.

5) Other ideas???

Thanks.

-G

[Rolf Fischer]

Hello,

looks like Rack-2A does not receive (superior) BPDUs in VLAN 444 from the Core-Switch, so it declares itself to the root for this VLAN with the default priority 0x8000 (32.768) + VLAN-ID. Most likely there is a layer-2 connectivity problem between Rack-2A and the Core - they seem to be isolated/separate domains in VLAN 444.

Do you have a VLAN trunk allowed vlan-list configured on Gi1/0/25 (Root Port for all other VLANs) or anywhere else on the path to the Root (useful command: show interface trunk)? Does Gi1/0/25 learn any MAC-Addresses in VLAN 444 dynamically? Check the counters for send and received BPDUs along the path with the 'show spanning-tree vlan 444 detail' command, are they increasing like they should?

4) Migrate all other non-core switches to 'vtp mode client' to alleviate this headache from returning.

This is generally recommended but I don't think the problem is related to VTP.

HTH

Rolf

[inetupto9]

Thank you for your reply Rolf.

Do you have a VLAN trunk allowed vlan-list configured on Gi1/0/25 (Root Port for all other VLANs) or anywhere else on the path to the Root (useful command: show interface trunk)?

Yes, but only from our "core" 3750G switch stack.  "VLAN 444" is pruned on the core to the other switches as it is a pseudo-private network (VLAN) that is was solely intended for our routers and firewalls; of which they terminate only on the core switches and no where else.  That was the motivation: restrict VLAN 444 only to only those ports (and devices) that require it.  This VLAN is not specified on ports anywhere else on our "access" switches.  Perhaps the 'switchport trunk allowed vlan' statement also should be specified on all the access switches as well?

Does Gi1/0/25 learn any MAC-Addresses in VLAN 444 dynamically? Check the counters for send and received BPDUs along the path with the 'show spanning-tree vlan 444 detail' command, are they increasing like they should?

As far as I am aware they are.  The BPDU counters are increasing (see the new attachment); albeit only the "sent" counter for VLAN 444 on both our "core" and Rack-2A, who thinks it is root.  To contrast it with our other VLANs, the "received" counters are the vast majority on our "access" switches. Could there be a conflict with the difference in VLANs that VTP is advertising and what is being allowed via STP (i.e. from the "core")?

I see the conflict, although the proper way to correct is what is alluding me at present.

Thanks again.

-- G

[Rolf Fischer · Accepted solution]

Thanks for the additional information, I believe I understand now.

You created VLAN 444, VTP distributed the new vlan to all switches in the domain, but you don't want to have VLAN 444 traffic on your access switches, so you pruned it manually on the core's uplink trunk interfaces. Once the new VLAN is applied on the access-switches by VTP, a local Spanning-Tree instance will become active when at least one port in VLAN 444 is active (can be a trunk port too).

As far as I can see, everything works as expected. The pruning of VLAN 444 also applies to the SSTP BPDUs for this VLAN, so the access-switches are not aware of the (core) root bridge and have to declare themselves as the (isolated) root bridge.

So this is actually a cosmetic problem and there are ways to solve this:

• If you prune VLAN 444 on the trunks of the access-switches as well, the corresponding local spanning-tree instance will disappear because there is no longer an active port in VLAN 444.
• Alternatively, you could shutdown the VLAN on the access-switches (if they support the shutdown vlan command): c2960 Command Reference - shutdown vlan

HTH
Rolf

[inetupto9]

Rolf, I think that fixed it!

After adding the VLAN pruning ('switchport trunk allowed vlan 1-443,445-4094') on all our "access" switches' trunk ports, I can now confirm this brought the STP root back to our "core".  Also had to force a topology change (created a bogus VLAN and then deleted it), but it's back.  "Rack-2A" also shows that the core switches are back as the root.

3750-core# show vtp status
VTP Version capable             : 1 to 3
VTP version running             : Y
VTP Domain Name                 : XXXXXXXXXXX
VTP Pruning Mode                : Disabled
VTP Traps Generation            : Enabled
Device ID                       : xxxx.yyyy.6e00
Configuration last modified by 10.30.1.2 at 2-22-16 14:26:56
Local updater ID is 10.30.1.2 on interface Vl30 (lowest numbered VLAN interface found)

Feature VLAN:
--------------
VTP Operating Mode                : Server
Maximum VLANs supported locally   : 1005
Number of existing VLANs          : 20
Configuration Revision            : 88
MD5 digest                        : [RETRACTED]

My final step will be to set all our access switches to a VTP mode of "client" to ensure none of the other switches in our infrastructure can attempt a coup in the future (inherited this network).

Alternatively, you could shutdown the VLAN on the access-switches (if they support the shutdown vlan command): c2960 Command Reference - shutdown vlan

I can confirm that this command is also present on the 3560, 3560G, and 3750G series switches (assuming IOS >= 12.2(25)).

Thanks again!

-- G