Interesting Spanning-tree problems

Difan Zhao · ‎06-30-2011

Hi experts,

We have a client and we provide VOD services to them. They have a complex switching network. We only have read-only access to three of them. Anyway so they are experiencing intermittent blackout issue on their STBs. We don't think there is anything wrong with our system. We suspected that they may have networking issue. However we have to prove that. Right now in the log I see a lot MAC-address flapping errors:

Jun 24 04:55:01.273: %SW_MATM-4-MACFLAP_NOTIF: Host 0015.17c0.5b36 in vlan 2138 is flapping between port Te1/0/2 and port Gi1/0/12

Jun 24 04:55:11.036: %SW_MATM-4-MACFLAP_NOTIF: Host 0015.17c0.5b36 in vlan 2138 is flapping between port Gi1/0/12 and port Te1/0/2

Jun 24 06:13:57.091: %SW_MATM-4-MACFLAP_NOTIF: Host 0015.17c0.5b36 in vlan 2138 is flapping between port Te1/0/2 and port Gi1/0/12

Jun 24 06:18:40.911: %SW_MATM-4-MACFLAP_NOTIF: Host 0015.17c0.5b36 in vlan 2138 is flapping between port Te1/0/2 and port Gi1/0/12

Jun 24 08:41:36.165: %SW_MATM-4-MACFLAP_NOTIF: Host 0015.17c0.5b36 in vlan 2138 is flapping between port Te1/0/2 and port Gi1/0/12

Jun 25 15:29:48.493: %SW_MATM-4-MACFLAP_NOTIF: Host 0015.17c0.5b36 in vlan 2138 is flapping between port Te1/0/2 and port Gi1/0/12

Jun 25 15:32:59.522: %SW_MATM-4-MACFLAP_NOTIF: Host 0015.17c0.5b36 in vlan 2138 is flapping between port Te1/0/2 and port Gi1/0/12

Jun 25 15:33:35.825: %SW_MATM-4-MACFLAP_NOTIF: Host 0015.17c0.5b36 in vlan 2138 is flapping between port Te1/0/2 and port Gi1/0/12

Jun 26 14:58:18.260: %SW_MATM-4-MACFLAP_NOTIF: Host 0015.17c0.5b36 in vlan 2138 is flapping between port Te1/0/2 and port Gi1/0/12

Jun 28 12:01:01.710: %SW_MATM-4-MACFLAP_NOTIF: Host 0015.17c0.5b36 in vlan 2138 is flapping between port Te1/0/2 and port Gi1/0/12

Jun 28 12:01:06.089: %SW_MATM-4-MACFLAP_NOTIF: Host 0015.17c0.5b36 in vlan 2138 is flapping between port Te1/0/2 and port Gi1/0/12

Jun 29 09:18:06.739: %SW_MATM-4-MACFLAP_NOTIF: Host 0015.17c0.5b36 in vlan 2138 is flapping between port Te1/0/2 and port Gi1/0/12

Jun 29 09:19:27.339: %SW_MATM-4-MACFLAP_NOTIF: Host 0015.17c0.5b36 in vlan 2138 is flapping between port Te1/0/2 and port Gi1/0/12

Jun 30 06:11:33.545: %SW_MATM-4-MACFLAP_NOTIF: Host 0015.17c0.5b36 in vlan 2138 is flapping between port Te1/0/2 and port Gi1/0/12

The MAC address is the VOD server's MAC address. It's directly connected on the port Gi1/0/12. It's weird that it's flapping between the directly connected port and the uplink port. I will attach the network diagram. The VOD vlan is just one VLAN 2138. The VOD traffic includes multicast.

My questions are:

1. Does the log imply a network issue (spanning-tree issue)?

2. Is there a way to enable logging of the SPT convergence events?

3. Why the MAC is flapping between the directly connected port and the uplink port? How is that possible?

4. On the switch "VOD", the cost on the Te1/0/1 and Te1/0/2 are 3002!! However the running config doesn't set it specifically. Why the cost is like this:

#sh spanning-tree vlan 2136 interface te1/0/2 detail

...

Port path cost 3002, Port priority 128, Port Identifier 128.54.

...

interface TenGigabitEthernet1/0/2

description xxx

switchport trunk encapsulation dot1q

switchport trunk native vlan 2104

switchport trunk allowed vlan 2100-2116,2136-2140

switchport mode trunk

switchport priority extend trust

load-interval 30

storm-control broadcast level 0.10

storm-control action trap

no ip igmp snooping tcn flood

Any ideas??

Thanks!

Difan

milan.kulik · ‎07-01-2011

Hi,

IMHO, this might be symptoms of a loop in the customer LAN.

Following scenario could be happening:

The VOD server is sending a broadcast frame.

Your switch is receiving it on port Gi1/0/12, puts the source MAC address 0015.17c0.5b36 to the forwarding table present on port Gi1/0/12 and floods the broadcast frame out of all other ports (including the uplink port Te1/0/2).

Now imagine there's a switch with a STP loop in the customer LAN.

It receives the broadcast frame from the core and due to the loop sends it back to the core!

And finally, your switch is receiving the broadcast frame incoming on the uplink port Te1/0/2.

As the frame source MAC address is 0015.17c0.5b36, your switch enters an updated MAC address entry to his forwarding table and issues a syslog error message, as it noticed the same MAC source address on two ports!

This might be a problem of slow STP convergence in the customer LAN in theory.

Or some other STP problem.

Not easy to prove without access to their network.

You might be able to capture frames received on the VOD server by Wireshark, e.g., and ask the customer "How is it possible the server is receiving fames with his own MAC as a source?".

But not an easy discussion, I'm afraid.

HTH,

Milan

in vlan 2138 is flapping between port Te1/0/2 and port Gi1/0/12

davy.timmermans · ‎07-01-2011

Often it's related to the software of the NIC of a server-

Can you check maybe the drivers for the VOD NIC?

Difan Zhao · ‎07-01-2011

Hi Davy,

I don't think so... We have the newest REHL 5.5 build on it. It's not happening just to one server. All other devices on the switch have their MAC addresses flapping as well.

Thanks,

Difan

Difan Zhao · ‎07-01-2011

Hi Milan,

Thanks for the reply! Your explanation makes perfect sense. I also found some other logs which suggests a possible network loop:

Jun 28 02:38:56 PDT: %SPANTREE-2-LOOPGUARD_BLOCK: Loop guard blocking port TenGigabitEthernet1/1 on VLAN2138.

Jun 28 02:38:56 PDT: %SPANTREE-2-LOOPGUARD_UNBLOCK: Loop guard unblocking port TenGigabitEthernet1/1 on VLAN2138.

Does it imply that the Te1/1 may have a cabling issue which causes unidirectional link? This is the log from another Dist switch. I don't know how this switch is connected with the network in my diagram. The client just sends me the log.

Is there a command to enable the logging of the SPT events such as root change, BDPU tsn packets received, ...etc?

Thanks!

Difan

milan.kulik · ‎07-11-2011

Hi,

without knowing topology details it's difficult to say where exactlythe problem could be.

Yes, your customer could start spanning tree debugging, e.g., and with Buffer logging level debugging configured on a switch he should be able to see STP events in the Syslog buffer.

But as any debug, it's worth to be careful, as it could overload the CPU in a case of high number of changes.

BR,

Milan