Re: VLAN leaking and ARP flood - Page 2

VictorAKur · ‎04-01-2009

Hi

I have a network with two 6509s, 720-3BXL in the core of it. The 6509s are connected to each other over a trunk with all VLANs allowed over it. Down the line there are top of the rack switches, each connected to both 6509s over trunks. Only VLANs configured on the top of the rack switches are allowed down this trunks, plus VLAN 101 (IP range 172.21.1.0/24), which is the management VLAN for all the networking equipment and some of the infrastructure servers (monitors, syslog ones etc). I have noticed recently that ARP requests destined to other VLANs get sent down VLAN 101. Here is a part of output from one of the switches:

Apr 1 10:16:21.428 gmt: IP ARP req filtered src 80.68.48.2 001b.0dec.59c0, dst 80.68.48.12 0000.0000.0000 wrong cable, interface Vlan101

Apr 1 10:16:21.436 gmt: IP ARP req filtered src 80.68.48.2 001b.0dec.59c0, dst 80.68.48.12 0000.0000.0000 wrong cable, interface Vlan101

Apr 1 10:16:25.328 gmt: IP ARP req filtered src 80.68.48.3 001b.0dec.5ac0, dst 80.68.48.25 0000.0000.0000 wrong cable, interface Vlan101

Where 80.68.43.2 and 80.68.43.3 are IPs of the core 6509s in VLAN 303.

There are also HSRP groups (with different IPs) for each VLAN configured on the 6509s.

I will be happy to provide more information, including diagrams, if needed.

Please help! :) as my brain is going into melt down already.

Regards

VictorAKur · ‎04-07-2009

Giuseppe

and any one who have had a look at it so far,

an answer to this question may or may not cast some light at the problem.

Why a 6509 with Sup7203BXL and IOS 12.2(33)SXH would ARP broadcast regulary, particulary on two VLANs? The broadcast looks like a typical port scan - as in every IP address in the range gets ARPed. Even though there are only 6 active IPs in this /24 network, every other IP gets ARPed very often. I have set mac-address-table aging to 14400 globaly.

But it did not help.

Any idea?

Giuseppe Larosa · ‎04-07-2009

Hello Victor,

the ARP timeout is a different timer, you have increased the timer of the L2 CAM table.

to check your ARP timer settings you can use

sh ip int vlan XX

and look for the ARP timeout line

default value is 4 hours.

You have told you have isolated the ACE appliances so there shouldn't be any paranoid device making controls ( I don't want to mean that ACE should do this)

using a sniffer on a pc on one of these vlans you can check the source MAC address of these frequent ARP requests.

For example you can use wireshark.

(probably you have already done this)

Hope to help

Giuseppe

VictorAKur · ‎04-07-2009

Yep

done wireshark on all VLANs.

It seems to be the two core switches that generate all this traffic.

I should probably say again that it is the management VLAN that gets hit by traffic destined to other VLANs, not any other VLAns.

We have VM Ware system where interfaces on the ESX are connected to both management VLAN and the data VLAN. And we have a number of Linux based systems, that use VLAN tagging. I have got to the point where I am considering to disconnect racks from the core in order to isolate the issue.

VictorAKur · ‎04-07-2009

Well I am assuming the ARP timeout is set to 4 hours as there is nothing about ARP timout in the sh ip int VLAN blah blah I can see.

sh ip int vlan 303

Vlan303 is up, line protocol is up

Internet address is x.x.x.x/24

Broadcast address is 255.255.255.255

Address determined by setup command

MTU is 1500 bytes

Helper address is not set

Directed broadcast forwarding is disabled

Multicast reserved groups joined: 224.0.0.2 224.0.0.5 224.0.0.6

Outgoing access list is not set

Inbound access list is not set

Proxy ARP is disabled

Local Proxy ARP is disabled

Security level is default

Split horizon is enabled

ICMP redirects are always sent

ICMP unreachables are always sent

ICMP mask replies are never sent

IP fast switching is enabled

IP Flow switching is disabled

IP CEF switching is enabled

IP CEF switching turbo vector

IP Null turbo vector

IP multicast fast switching is enabled

IP multicast distributed fast switching is disabled

IP route-cache flags are Fast, CEF

Router Discovery is disabled

IP output packet accounting is disabled

IP access violation accounting is disabled

TCP/IP header compression is disabled

RTP/IP header compression is disabled

Probe proxy name replies are disabled

Policy routing is disabled

Network address translation is disabled

BGP Policy Mapping is disabled

Output features: IP Post Routing Processing, HW Shortcut Installation

Post encapsulation features: MTU Processing, IP Protocol Output Counter, IP Sendself Check, HW Shortcut Installation

Sampled Netflow is disabled

IP Routed Flow creation is disabled in netflow table

IP Bridged Flow creation is disabled in netflow table

WCCP Redirect outbound is disabled

WCCP Redirect inbound is disabled

WCCP Redirect exclude is disabled

IP multicast multilayer switching is disabled

Giuseppe Larosa · ‎04-07-2009

Hello Victor,

sorry I was out of office the command to see the ARP timeout settings is:

sh int vlan xx

see

sh int vlan100 | inc ARP

Encapsulation ARPA, loopback not set

ARP type: ARPA, ARP Timeout 04:00:00

About the presence of Vmware ESX systems and linux boxes able to process vlan tagged frames they could play a role here.

One suggestion is to isolate one core switch to see if the problem is caused by the fact of being both on the same vlans.

If the abnormal ARP activity stops the problem is between the two core switches.

if the remaining switch still sends out a lot of ARP requests the problem can be in some Vmware or linux boxes

Hope to help

Giuseppe

VictorAKur · ‎04-08-2009

Giuseppe

it is 4 hours everywhere.

Still what can cause a 65009 to do very regular ARP broadcasts?

I am going to shut one of the boxes down to try what you suggested. I will have to do it next week though, so may not have any updates till then.

Thank you for your help.

VictorAKur · ‎04-09-2009

Another question

Could anyone explain please where the leak may be happening if the core switch sends traffic out of one VLAN and I see it hitting an access switch on another VLAN. There is one cable between the core switch and the access switch and only this two VLANs are allowed across the trunk on both ends.

Clutching straws here....

VictorAKur · ‎04-09-2009

We have found a DRAC port that was causing a lot (every 20 - 30 seconds) STP resets. As we are running RSTP it was causing immediate ARP flush and resulting unicast floods. That is the theory anyway.

In practice it did not resolve the original problem, but did reduce the number of VLAN leaks per a time slot.

I still get this on the core switches though, even though both ARP and CAM timeouts are the same and set to 14400. Any idea what can cause it?

Apr 9 15:21:01.225 gmt: IP ARP: rcvd req src 80.68.43.2 001b.0dec.60c0, dst 80.68.43.4 Vlan303

Apr 9 15:21:01.229 gmt: IP ARP: rcvd req src 80.68.43.2 001b.0dec.60c0, dst 80.68.43.6 Vlan303

Apr 9 15:21:01.229 gmt: IP ARP: rcvd req src 80.68.43.2 001b.0dec.60c0, dst 80.68.43.7 Vlan303

Apr 9 15:21:01.233 gmt: IP ARP: rcvd req src 80.68.43.2 001b.0dec.60c0, dst 80.68.43.9 Vlan303

Apr 9 15:21:01.237 gmt: IP ARP: rcvd req src 80.68.43.2 001b.0dec.60c0, dst 80.68.43.13 Vlan303

Apr 9 15:21:01.257 gmt: IP ARP: rcvd req src 80.68.62.2 001b.0dec.59c0, dst 80.68.62.19 Vlan201

Apr 9 15:21:01.269 gmt: IP ARP: rcvd req src 80.68.43.2 001b.0dec.60c0, dst 80.68.43.22 Vlan303

Apr 9 15:21:01.269 gmt: IP ARP: rcvd req src 80.68.43.2 001b.0dec.60c0, dst 80.68.43.23 Vlan303

Apr 9 15:21:01.273 gmt: IP ARP: rcvd req src 80.68.43.2 001b.0dec.60c0, dst 80.68.43.24 Vlan303

Apr 9 15:21:01.277 gmt: IP ARP: rcvd req src 80.68.43.2 001b.0dec.60c0, dst 80.68.43.25 Vlan303

Apr 9 15:21:01.277 gmt: IP ARP: rcvd req src 80.68.43.2 001b.0dec.60c0, dst 80.68.43.28 Vlan303

Apr 9 15:21:01.277 gmt: IP ARP: rcvd req src 80.68.43.2 001b.0dec.60c0, dst 80.68.43.26 Vlan303

Apr 9 15:21:01.281 gmt: IP ARP: rcvd req src 80.68.43.2 001b.0dec.60c0, dst 80.68.43.30 Vlan303

Apr 9 15:21:01.281 gmt: IP ARP: rcvd req src 80.68.43.2 001b.0dec.60c0, dst 80.68.43.29 Vlan303

sw2.ahf#

Apr 9 15:21:01.281 gmt: IP ARP: rcvd req src 80.68.43.2 001b.0dec.60c0, dst 80.68.43.27 Vlan303

Apr 9 15:21:01.321 gmt: IP ARP: rcvd req src 80.68.43.2 001b.0dec.60c0, dst 80.68.43.31 Vlan303

Apr 9 15:21:01.325 gmt: IP ARP: rcvd req src 80.68.43.2 001b.0dec.60c0, dst 80.68.43.32 Vlan303

Apr 9 15:21:01.329 gmt: IP ARP: rcvd req src 80.68.43.2 001b.0dec.60c0, dst 80.68.43.33 Vlan303

Apr 9 15:21:02.029 gmt: IP ARP: sent req src 80.68.43.3 001b.0dec.5ac0,

dst 80.68.43.81 0000.0000.0000 Vlan303

Apr 9 15:21:02.217 gmt: IP ARP: rcvd req src 80.68.43.2 001b.0dec.60c0, dst 80.68.43.34 Vlan303

Apr 9 15:21:02.229 gmt: IP ARP: rcvd req src 80.68.43.2 001b.0dec.60c0, dst 80.68.43.35 Vlan303

Apr 9 15:21:02.237 gmt: IP ARP: rcvd req src 80.68.43.2 001b.0dec.60c0, dst 80.68.43.36 Vlan303

Apr 9 15:21:02.237 gmt: IP ARP: rcvd req src 80.68.43.2 001b.0dec.60c0, dst 80.68.43.37 Vlan303

Apr 9 15:21:02.241 gmt: IP ARP: rcvd req src 80.68.43.2 001b.0dec.60c0, dst 80.68.43.38 Vlan303

Apr 9 15:21:02.245 gmt: IP ARP: rcvd req src 80.68.43.2 001b.0dec.60c0, dst 80.68.43.39 Vlan303

it seems it is always the same VLAN that gets hit.

ROBERT THOMSON · ‎07-18-2009

I have experienced the same problem of broadcast traffic and some unicast traffic leaking across vlans on 3750 switches when the ports are configured with a voice vlan for voip.