mac relearning

[joeblough]

Good Day,

I apologize in advance for being able to offer only scant info regarding this network. I manage a very small piece of a large university network and due in part to design and in part to bizarre history, I don't have access to many pieces of equipment.

I work in a large residence hall with seven subnets serving residents and one for the administrative staff. The residential subnets connect to a Cisco 2950 and are trunked to a 2900XL.

The admin network is connected through three Foundry switches. Two are on a lower level with a fiber run to the third upstairs. This third is connected to the 2900XL and all vlan/subnets are trunked across the road to another 2950 where they are "de-trunked" and each vlan is connected to a physical port on a Cisco 7000.

I do not have access to the Foundry switches (history) or the router (design).

We rent out conference spaces and offer internet access for a fee. Our admin network uses static IPs, the residential networks utilize DHCP with a proprietary MAC authentication system. To simplify renting internet access, yesterday I configured a broadband router with an authorized MAC address and connected it to the 2950. Tested it fromour conference rooms and everything is fine.

I mention this broadband router because it is the only change between the good times and the bad. Though I have eliminated it and the bad persists.

So, the problem...

Every 60 seconds, precisely, the 2900XL port attached to the Foundry admin switch is forced to relearn mac addresses causing a slowdown that can allow a ping but timesout with web pages or e-mail downloads.

I had an extra 2900XL sitting around so I put that between the admin network and the other 2900XL. Then, the port connecting the admin network on the second 2900 exhibited the same behavior.

Knowing that the problem is not with 2900XL #1, I reverted to the original setup and re-connected the admin network to 2900XL #1.

There are six ports assigned to the admin vlan (#520) on 2900XL #1. I shutdown all but the active port, configured a static mac address for my workstation on the active port, and cleared all dynamic macs for the switch.

After relearning, the mac address for my workstation was learned on all 6 ports with switchport access vlan 520.

All switches are running SPT.

Any ideas greatly appreciated.

Thanks

[milan.kulik]

Hi,

1) You topology is complex and not clear from your message. Could you upload a simple scheme using Add Attachments NetPro feature?

2) How many VLANs are configured in your network?

Cat2900XLs do have a 64 STP instances limit with similar symptoms: when adding 65th VLAN to the VLAN database, one random (not necessarily the 65th) VLAN stops running STP and a network loop can occure.

3) Do you see any repeating STP TCNs or similar messages in your syslog?

Regrads,

Milan

[gamblin.er]

It sounds as if there may be a combination of problems here.

One or more switches is generating Topology Change Notifications. On receipt of each of these, a switch must flush its MAC addr table. Until it's relearned the relationship between MAC addresses and ports, it floods all traffic out all ports. So the switches behave like hubs - accounting for the degraded performance you describe.

To verify and quantify this:

- Some switches can display TCN count from the CLI or menu, or

- If you have SNMP access to the switches, look in the Bridge MIB.dot1dStp.TimeSinceTopologyChange and TopChanges. If the former doesn't get larger than a few minutes, you have a serious problem.

The next part is harder, because it requires you to gain an appreciation of how the overall network including the Foundry switches is configured. And if necessary influence changes.

Are there STP/TCN bugs associated with any of the switches/firmware in your network?

Is all switch firmware current?

Are all switches running IEEE spanning-tree? Any running Rapid STP? Especially, are there any that could cause loops if they are not?

In many networks I've seen the spanning-tree setup is pretty haphazard, with no effort made to control which switch is elected root. Consequently, root can be a small edge switch somewhere. To work reliably, one of the higher-powered switches has to be root.

Set the STP Priority of the main core switches so that one is primary root (lowest value of priority) and another is secondary (next lowest, but still lower than the access switches).

Finally, a sometimes-overlooked aspect is the MAC address aging time. Most switches default this to 5 minutes, but in a large network 60+ minutes may be a better choice. Increasing this won't fix the TCN problem, but it will reduce flooded traffic afterwards.

Hope this helps find and fix your problems.