Tracking spanning tree TCN origin

[jedavis]

Does anyone have a good method for tracking down the origin of spanning tree topology changes in a network of primarily 3500XL switches? The core switches are 6000s and they identify the port that the last TCN was received on. The only TCN info I can find on 3500XLs is the number received. Even on 3560 switches I see only see the TCN count and how long ago it was received. Is there a debug command that will log this info? debug spanning events perhaps?

[andrew.prince]

What version of SPT are you running?

[jedavis]

I'm not sure what you mean "version". These are the XL series switches and they only support classic 802.1d. In other words, PVST not RPVST.

[andrew.prince]

Yep - cool, because 802.1d STP and PVST send a TCN when an access ports goes up/down.

So at the end of the day you could actually be chasing ghosts, of user ports flapping.

802.1w only sends a TCN when a RP/DP flaps and there actually is a change in the layer 2 topology.

[jedavis]

But PVST should NOT send a TCN when a port configured with portfast transitions state. All of the end-host access ports in this LAN should be configured with portfast. So I need to determine where the TCNs are originating.

I have a unicast flooding issue that is a result of asymmetric routing coupled with frequent TCNs. The mac-address aging on all switches is set to 4 hours (to match the arp timeout), and the unicast flooding would not be an issue if the switches actually retained the forwarding table that long. The TCNs are causing the fast-aging mechanism to flush the L2 forwarding table.

I realize that I need to address the asymmetric routing issue as I will always have TCNs from time to time. But we are getting way too many of them.

[glen.grant]

On newer switches show spanning tree detail might help but don't know if those old switches support that command or not .

[jedavis]

Right, and the only piece of info the "detail" seems to add is the length of time since the last TCN was received. And no, the old switches do not support it. But "debug span event" does the trick.

[andrew.prince]

Well I must admit my PVST is rusty and it's been a while, but I do know that RPVST+/PVRST+ do not use TCN's on access ports.

But the above is neither here nor there. The only thing I can think you do is log into the root switch and monitor the TCN status - and also log into the other switches and monitor there TCN status and when 1 increments - you will know what is happening - not very high tech. You could debug spantree on the 3500XL, BUT the last time I did that on a prodcution environment - the switch tanked.

[jedavis]

I went through all switches and baited them with "debug span event" last night. This morning I checked my lines and I had hooked a TCN!

The debug info logs the port that the TCN was received on, so I was able to track it to an access port that did not have portfast enabled. Log from the offending switch is below. It would be great if I could enable this type of logging without having to enable a debug.

Apr 23 07:19:05: %LINK-CLUSTER_MEMBER_8-3-UPDOWN: Interface FastEthernet0/30, changed state to down

Apr 23 07:19:05: ST: sent Topology Change Notice on GigabitEthernet0/2 vlan 1

Apr 23 07:19:05: ST: FastEthernet0/30 vlan 1 -> blocking

Apr 23 07:19:05: ST: FastEthernet0/18 vlan 1 -> blocking

Apr 23 07:19:06: %LINEPROTO-CLUSTER_MEMBER_8-5-UPDOWN: Line protocol on Interface FastEthernet0/30, changed state to down

Apr 23 07:19:06: %LINEPROTO-CLUSTER_MEMBER_8-5-UPDOWN: Line protocol on Interface FastEthernet0/18, changed state to down

Apr 23 07:19:07: %LINK-CLUSTER_MEMBER_8-3-UPDOWN: Interface FastEthernet 0/30, changed state to up

Apr 23 07:19:07: ST: FastEthernet0/30 vlan 1 -> listening

Apr 23 07:19:08: %LINK-CLUSTER_MEMBER_8-3-UPDOWN: Interface FastEthernet 0/18, changed state to up

Apr 23 07:19:08: ST: FastEthernet0/18 vlan 1 ->jump to forwarding from blocking

Apr 23 07:19:08: %LINEPROTO-CLUSTER_MEMBER_8-5-UPDOWN: Line protocol on Interface FastEthernet0/30, changed state to up

Apr 23 07:19:09: %LINEPROTO-CLUSTER_MEMBER_8-5-UPDOWN: Line protocol on Interface FastEthernet0/18, changed state to up

Apr 23 07:19:11: %LINK-CLUSTER_MEMBER_8-3-UPDOWN: Interface FastEthernet 0/30, changed state to down

Apr 23 07:19:11: ST: FastEthernet0/30 vlan 1 -> blocking

Apr 23 07:19:12: %LINEPROTO-CLUSTER_MEMBER_8-5-UPDOWN: Line protocol on Interface FastEthernet0/30, changed state to down

Apr 23 07:19:12: %LINK-CLUSTER_MEMBER_8-3-UPDOWN: Interface FastEthernet 0/30, changed state to up

Apr 23 07:19:12: ST: FastEthernet0/30 vlan 1 -> listening

Apr 23 07:19:13: %LINEPROTO-CLUSTER_MEMBER_8-5-UPDOWN: Line protocol on Interface FastEthernet0/30, changed state to up

Apr 23 07:19:22: %LINK-CLUSTER_MEMBER_8-3-UPDOWN: Interface FastEthernet 0/30, changed state to down

Apr 23 07:19:22: ST: FastEthernet0/30 vlan 1 -> blocking

Apr 23 07:19:23: %LINEPROTO-CLUSTER_MEMBER_8-5-UPDOWN: Line protocol on Interface FastEthernet0/30, changed state to down

Apr 23 07:19:24: %LINK-CLUSTER_MEMBER_8-3-UPDOWN: Interface FastEthernet 0/30, changed state to up

Apr 23 07:19:24: ST: FastEthernet0/30 vlan 1 -> listening

Apr 23 07:19:25: %LINEPROTO-CLUSTER_MEMBER_8-5-UPDOWN: Line protocol on Interface FastEthernet0/30, changed state to up

Apr 23 07:19:37: ST: FastEthernet0/30 vlan 1 -> learning

Apr 23 07:19:50: ST: sent Topology Change Notice on GigabitEthernet0/2 vlan 1

Apr 23 07:19:50: ST: FastEthernet0/30 vlan 1 -> forwarding

Apr 23 07:45:10: ST: stp_set_port_bandwidth_for_api

Apr 23 07:45:10: ST: Adjust topology

Apr 23 07:45:10: ST: stp_set_pathcost_for_api

Apr 23 07:45:10: ST: Adjust topology