Can someone explain what could possibly be causing the problem we've been seeing recently.
We have had to implement ip dhcp snooping and it's having a bizarre interaction with spanning tree when certain 3650 devices reboot.
Maybe it's because we're writing the dhcp binding table to the flash so it survives the rebooting and that is causing a weird interaction at boot. I don't now.
But what seems to be happening is that certain 3650 switches running 16.12.10 or higher that have ip dhcp snooping applied will randomly on a reboot start advertising the lowest interface's mac address as a new root for all allowed vlans, and this is often the either unused or completely disabled Gi 0/0 (RP management port).
Because we have loop and root guard applied everywhere this will start blocking various trunk ports. "%SPANTREE-2-LOOPGUARD_BLOCK: Loop guard blocking port GigabitEthernet1/1/1 on VLAN0xxx"
If the RP port (or lowest mac on the device) isn't disabled you can see this as the MAC address in one of the other switches that is showing the spanning tree root change, it shows up as "%SPANTREE-5-ROOTCHANGE: Root Changed for vlan xxx: New Root Port is GigabitEthernet1/1/1. New Root Mac Address is xxxx.xxxx.xxxx"
It doesn't matter that the port is unplugged and down/down, it still somehow is now advertising that it's the new root.
If it's admin shut (which most of ours are) the other switch doesn't show the mac address, the only mac you'll see is when the root changes back to what it was before it started blocking.
The only reason I know that it's DHCP snooping related is that it was the last change on the devices and when we remove it, the new root advertisement stops. It absolutely stops when the "ip dhcp snooping vlan xxx" is removed from the offending switch that is doing the root advertisement from it's RP port.
This is clearly a bug, and we had no issues for weeks after configuring dhcp snooping, but after it was running for a few weeks when we were doing switch IOS upgrades it started happening on some of them following reboots.
I can find no other case of this online, and I know it seems crazy to say they're connected but they are connected, somehow.
Here are the various errors someone will see when it's happening:
%SPANTREE-5-ROOTCHANGE: Root Changed for vlan xxx: New Root Port is GigabitEthernet1/1/1. New Root Mac Address is xxxx.xxxx.xxxx
%SPANTREE-2-LOOPGUARD_BLOCK: Loop guard blocking port GigabitEthernet1/1/1 on VLAN0xxx
%SPANTREE-2-LOOPGUARD_UNBLOCK: Loop guard unblocking port GigabitEthernet1/1/1 on VLAN0xxx.
%SPANTREE-5-TOPOTRAP: Topology Change Trap for vlan xxx
And it will happen on every allowed VLAN, and if the RP (or lowest mac address on the interfaces) port is not disabled you will see that as being advertised as the new root, but if it's admin shut then you won't see who is the new root, just that there is one and then the blocking. You will have to prune to find the offending device.
It did it on 16.12.10 and 16.12.11 on the 3650s, but the spanning tree changes impact every switch sharing the same VLANs.
I'm mostly asking this here so that the next unlucky network tech has a hope of knowing why their network is going crazy when it happens to them.
Has anyone seen this before?
