Layer 2 port-channel flood

[Jamie Hancock]

Hi all,

I was wondering if you may be able to help me, hopefully I will provide enough information.

The background to my problem is this. Every Tuesday morning around 10AM we were experiencing network slowdowns, after many weeks fault finding I have narrowed it down to two backup jobs and two vlans. These are quite substantial backup jobs each 100GB in size.

The servers that are being backed-up are behind an ISA server which is controlling the routing for the subnets that these servers use. The ISA servers are load balanced using Microsoft NLB. It is thought by our security expert that this is the best way to secure these servers.

On our core switch (6513) we have a static route pointing to these subnets and the vlans are defined. Here is the basic config of one of the vlans:-

interface Vlan121
 description DB vlan
 no ip address
 ip flow ingress
 ip flow egress
end

ip route 192.168.221.0 255.255.255.0 192.168.219.10

I have managed to stop the flooding going to the user switches by denying the 121 vlan on the port-channels. The issue is still apparent however on our top of rack switches (server switches). The reason for this is, there are servers that require vlan 121 on nearly every top of rack switch.

If anyone can recommend a solution to this problem other than limiting vlans, I would greatly appreciate it.

If you require any further information, please let me know.

Kind regards,

Jamie.

[Aaron Harrison]

You may have a scenario much like the asymmetric scenario shown here, but with the HSRP routers replaced with your ISA servers: http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6000-series-switches/23563-143.html

I don't really know enough about your switch layout and where your ISA servers are on that layout to comment. It's possible that the route outbound on some occasions is via ISA-1, and back via ISA-2, and that asymettric routing path is generating the flooding scenario. 

If it is that sort of scenario, you may be able to alleviate it by following the guidance here:

http://www.cisco.com/c/en/us/support/docs/ip/hot-standby-router-protocol-hsrp/10583-62.html#t8

That is, setting each VLAN to 14400 seconds CAM timer:

mac-address-table aging-time seconds [vlan vlan_id]   

[Jamie Hancock]

Thanks for the reply Aaron.

I should mention that the NLB IP's are contained in their own VLAN. Would this make any difference to the information you have supplied?

Jamie.

[Aaron Harrison]

Hi Jamie

Perhaps a diagram would help?

Aaron

[Jamie Hancock]

Sorry it's taken me a while to upload a diagram. If you need clarification, let me know.

J.

[Jamie Hancock]

Bump^

[Jon Marshall]

Jamie

Your diagram shows vlan 124 but you are referring to vlan 121 in your description. Are you experiencing problems with both vlans ?

What vlan is the backup server on and where is this vlan in relation to the servers being backed up ?

Is the traffic actually being flooded to all ports or are you simply seeing a lot of traffic in vlan 121 because of the backups.

100Mbps is not a big backup. What time is it scheduled to run ?

If it was scheduled to run over night then if it is till running at 10 in the morning I agree there is an issue somewhere.

Edit - my mistake - 100Gbps not Mbps which is quite different :-). So please ignore last few points but address the others.

Jon

[Jamie Hancock]

Hi Jon, let me see if I can answer your questions.

There are four VLANs that reside behind the ISA VLAN -121, 122, 123 and 124. Any traffic that requires access to these VLANs have to pass through the ISA because the ISA dcontrols all the routing for the subnets associated with these VLANs.

The server that we are backing up lives on a VLAN 124 and the actual backup server lives on a VLAN outside of the protected VLANs. For the sake of argument lets say that the backup server is shown as the PC on VLAN 156. 

When the 90GB backup is initiated, the traffic propagates to all trunk ports throughout the network. I see traffic running at 500Mbps and it can last for up to 20 hours.

I have managed to stop the flood to all the user switches by denying the VLANs stated above. The problem is, I can't deny those VLANs to other top of rack switches because there is at lease one server in each switch that requires one of the VLANs.

In a nutshell, when large amounts of traffic pass through VLAN 666 (ISA) we see it flood to all trunk ports. We think this could be due to the nature of MLB forming a virtual MAC address. The core doesn't know about the MAC address so it sends a unicast flood to find out where it is.

If you have any ideas please let me know.

J.

[Jon Marshall]

Jamie

There are most definitely issues with NLB and unicast flooding an quick search will show you a number of threads related to the issue and possible workarounds.

That said I may be being a bit thick but isn't most of the traffic travelling away from the ISA servers ie. to the backup server ?

In which case I wouldn't necessarily expect to see it being flooded because the 6500 should know the mac address of the backup server as this is in a standard vlan (standard meaning not behind the ISAs).

Do you have just one 6500 and are all the vlans routed on there (except for the ISA vlans of course) ?

Jon

[Jamie Hancock]

Hi Jon,

Thank you for replying again. I have conducted many searches over the last few weeks all of which I have either tried such as the VLAN denials or I have examined the option for changing the NLB from unicast to multicast. This last option shouldn't be needed because the NLB address are contained within VLAN 666.

You are most definitely not being thick. It's me not explaining myself properly. The PC in the diagram is pulling the backup traffic from a server that is connected behind the ISA, does that make sense?

I have two 6500's that are connected via VSS. These manage all VLANs and everything is routed on there apart from the subnets that are routed by the ISA.

I hope I'm explaining myself properly...

J.

[Jon Marshall]

Jamie

You are most definitely not being thick.

I may surprise you yet :-)

Even if the PC is "pulling" data the major traffic flow is surely from the server behind the ISAs to the backup PC/server.

To be honest I've not done much with backups so I may be talking rubbish.

Basically if most of the traffic flow was to the ISAs then yes I can see how that might cause a flooding issue if the 6500 somehow had the wrong mac address entries.

But as most of the traffic is going the other way (I may be wrong here) then the ISAs are largely irrelevant ie. the ISAs simply forward to their default gateway on the 6500 and the 6500 should know where the backup server is ie. which port it's mac address is on.

If there is as much traffic going backwards as well as forwards then the ISAs could definitely be the culprit.

Have you looked at the mac address tables on the 6500 and the arp tables and what are they showing ?

Sorry for all the questions but still struggling to see where the main issue is.

Jon

[Jamie Hancock]

You have me thinking about the flow of data. I'm not that failure with backups. I will try a few file transfers from both directions and let you know the outcome.

J.

[Jamie Hancock]

Sorry for the late reply.

The flood happens when the traffic flows both ways.

We recently took off the bridge groups from the VLANS, could this be attributing to the issue?

J.