4500-X routing packets out of a L2 VLAN in error

[TG14]

Hello,

I am experiencing an odd problem and am assuming that it is related to the 4500-X switch (even though this is causing me wireless issues).

Problem:

I have a L2 VLAN trunked into a WLC for a guest network, with a single router on another interface in the same VLAN (607).  Sometimes packets are occasionally routed out of that VLAN even though the switch does not have a L3 SVI configured.  At the same time that packet will also be sent in the VLAN.

Example:

Client DHCP request leaves WLC trunk (inter 1/3 - VLAN 607) and is sent both to router (inter 1/9 - VLAN 607) in L2 and also some how is also seen in a packet capture on the core router link (inter 1/12 - VLAN 808).  There is a route in the routing table for the destination IP (10.1.x.x/16) for the DHCP request but the expectation was that as this was a L2 VLAN it would not matter and traffic should not leave the L2 VLAN.

[Flavio Miranda]

Hi

  Your design description is confused. However, mind that DHCP is a broadcast and if you have ip helper address on the L3 device, it will send out DHCP packet.

 How does your WLC is acting on this case? Is it a DHCP proxy or clints should ask DHCP locally?

   Also, why DHCP request could represent a problem ?  Are they being responded for some DHCP server or not?

[TG14]

Hi @Flavio Miranda 

Thanks for replying.  I know it sounds odd but it is probably pretty simple.

To try and explain a little simpler.

WLAN configured with a L3 interface 10.10.10.5 (VLAN607)

WLC with trunk connection on 4500-X interface 1/3 allowing VLAN607

4500-X with L2 VLAN607 configured (no SVI)

Guest Service Router configured with 10.10.10.1  with access port connection on 4500-X interface 1/9 (VLAN607)

This is all L2 on the switch at this point.

We also have a connection from the 4500-X switch to another switch at L3 (transit /29 network between switches, not related to the guest network).  These are purely routed ports with no ip helpers etc.

The switch is using dynamic routing so there is a routing table in use.

The problem is we can see the occasional packet of data from the guest network clients being sent to both the Guest router and out of the L3 interface between switches.  There is no reply to the one sent via the L3 interface as it ends up blocked at a firewall but the expected response is seen from the guest router.

The reason this is an issue is that the L2 nature of this VLAN was seen as secure previously but now we see packets "leaking" for want of a better description.  This should not happen as far as I know.

The thing which I am seeing which may be causing this, but I do not understand why, is that up until recently the network being access by the guest client for DHCP/DNS services with the guest provider was something like 10.50.50.0/24.  We are now using that range in our own corporate network and we now also have a route on the switch for 10.50.50.0/24 going via the core link indicated above.

Is there a reason why a switch would decide to route a packet if the VLAN the packet arrived in does not have a L3 SVI?  This is also only happening for a tiny amount of packets, we are talking 1-2 packets out of thousands if not tens of thousands.

In answer to your questions:

No, the WLC is not acting as a DHCP server, it is relaying DHCP requests

It is not actually a DHCP issue I am highlighting, this has also been some DNS requests.  DHCP was more of an example.  Assume this is DNS packets from now on if easier to imagine.

Thanks in advance.

[Georg Pauwen]

Hello,

I guess it would be interesting to find out what packets these are that are being routed out (what destination address these packets have, and what UDP/TCP ports are being used).

You could use a 'debug ip packet' with an access list, maybe it displays something useful:

access-list 101 permit ip 10.10.10.0 0.0.0.255 any
debug ip packet 101

[Elliot Dierksen]

The 4500 is going to route things via the SVI when it is handling L3 traffic. It acts at L2 and L3. As an example, it could have an SVI with 192.168.1.1/24 assigned to it. If there are L2 ports and there are hosts 192.168.1.2/24 and 192.168.1.3/24 talking to one another, that traffic will only be L2 from the perspective of the 4500. They are just L2 Ethernet frames. The only exception would be broadcasts.

[TG14]

Hi @Elliot Dierksen 

Thanks, in case I misunderstood you are saying this;

Device A - VLAN 100 192.168.100.10

Device B - VLAN 100 192.168.100.20

Device C - VLAN 200 192.168.100.10

Device D - VLAN 200 192.168.100.20

These devices should be able to talk to each other on the same VLAN at L2 even though they have the same IP networks but not between VLANs.  A -> B = OK and C -> D = OK

Layer 3:

The above would not work if an SVI was created for VLAN 100 and an SVI for VLAN 200 as they would end up with a clash of overlapping subnets.  This is how I understand it from a L2 and L3 perspective.  L2 packets should not be seen in different VLANs due to the nature of VLANs and broadcast domains.

So, the issue I am facing is that a L2 network which has a gateway configured in a router, not on the switch, should be classed as a L2 network to the switch (no packets should be able to get out of that L2 network other than via the gateway router).

I have attached a diagram with what I have as an example.  In the example VLAN 10 is a L2 network.  VLAN 99 is a L3 network linking 2 switches together.  So, a packet originates from a client on the wireless network destined for 192.168.100.50.  If this is a L2 only network the packet should pass to the gateway of VLAN 10 as a packet destined for a different subnet outside of the one the client is in.  As the gateway is a router connected to the switch the packet should end up with the destination MAC of the gateway router interface in VLAN 10.  This I believe is standard L2/L3 behaviour.  What I am seeing is the switch also send a packet (the same packet) but with a different destination MAC address of the VLAN 99 interface of the CORE switch.  This means the packet was routed also.

How can a packet be routed if the VLAN the packet arrived in does not have a L3 SVI on that switch?

[Elliot Dierksen]

You are listing the same IP address on 2 different VLAN's which isn't going to work.

2 devices on the same IP network talking to one another would be L2 from the perspective of the 4500, NOT L3. Same with any other access switches. It is just switching ethernet frames at the point.

[MHM Cisco World]

WLC send unicast to DHCP server you config on it.
Now the DHCP send by the WLC management interface toward the DHCP server, 
since they are in different Subnet WLC will send broadcast to find the Default GW, 
the management send with VLAN tag "which can be native VLAN".
from here we must look what is the default GW config the SW "even if it L3 it can have default GW".
so I think the traffic go to L3 SVI then return to guest router because the WLC select the L3 SVI as default GW.

[MHM Cisco World]

Depend on wlc config what is vlan for managment ip and dhcp server vlan ?

[TG14]

I may have confused things a little with trying to explain what I am seeing.

What I am seeing is that;

• ~99.9% of packets from a wireless client which are supposed to stay in a L2 VLAN between the controller and the router are fine.
• Occasionally a packet will be duplicated by something (suspect 4500x switch) and is then sent via another interface.
• We see 2 packets on the switch.  Same source MAC/IP etc. but different DST MAC (SAME IP).  One DST MAC is the guest router, the other is an interface on an upstream L3 switch.
• We believe that this "new" packet is routed due to us adding a new route into the switch which just happened to cover a DST IP used by the guest WIFI service (which should stay in its L2 network and be "invisible" to the rest of the switch.

Why would a switch generate another packet after receiving it and change it's DST MAC whilst still sending the original packet on?

[MHM Cisco World]

99.9 traffic use the dynamic IP of vlan you config in wlc for each wlan.

0.1 % traffic is any traffic that wlc is send from it management ip instead of user in wlan, this traffic like dhcp dns ....etc. what ever server wlc config.

Here dynamic IP of vlan for wlan have defualt gateway,

Management IP dont have defualt getaway or it have one that lead to router not SVI in SW, this is why you see traffic go and back to router "dhcp server".

[TG14]

Hello @MHM Cisco World 

I am not sure this is the case here.

Example:

1000 DNS request packets - source 192.168.10.10/aa:bb:cc:00:11:22 ---> destination 10.1.5.100/dd:dd:dd:33:44:55 (guest router MAC which is also the default gateway for 192.168.10.0/24 subnet)

Out of those 1000 DNS request packets 1 is duplicated somewhere and above becomes;

source 192.168.10.10/aa:bb:cc:00:11:22 ---> destination 10.1.5.100/bb:bb:bb:22:22:22 (MAC of a different L3 switch connected to the same switch as the guest router).  The reason this seems to happen is that the switch now how an ip route of 10.1.5.0/24 via interface with bb:bb:bb:22:22:22 MAC.

Why would a packet be duplicated somewhere and a different DST MAC be inserted?

[MHM Cisco World]

I assume this typo, default  gw must be in same subnet as host, here default gw is 10.... and host subnet is 192......

This host must be pass through any l3 then go to 10.....

[TG14]

No, that is not a typo.  The L3 gateway is the guest router for the guest network (L2 in the switch).

WLC ---------- ------- SWITCH --------- GUEST ROUTER

192.168.10.10---------  VLAN10 --------- 192.168.10.1 (MAC bb:bb:bb:22:22:22)

The packet will have the destination MAC of the router but the IP of the end destination (in this case a DNS server somewhere past the GUEST ROUTER).

My problem is I see 2 packets on the SWITCH, one sent in VLAN 10 with the DNS server destination IP and GUEST ROUTER MAC as it should be and another packet the same but the destination MAC is another ROUTER which isn't in VLAN 610.

[MHM Cisco World]

Hi Friend, Sorry for late reply,
there is CentralSW CentralAuth 
            localSW     centralAuth
this two mode push from WLC to AP connect to it, check the both VLAN use for Central and LocalSW.