Wireless LAN Controller looping/flooding DHCP packets

[tjoliveira]

Hi all,

We are facing a strange problem at a client's wireless solution.

Solution with 4x WLCs 5508, running v7.3.112.0, spread across 2x DCs. All 4x are active (no HA) with two mobility groups. The SSIDs are associated with a group interface (vlan select). All configurations are identical, being the IP addresses and hostname more or less the only configurations that change between them. The LAN is constructed pretty much the same way on both DCs and there are no STP loops (all interfaces forwarding).

Recently we activated two external DHCP servers. On 2x of the WLCs we have an issue: for each DHCP request packet from the clients (DHCP DISCOVER) thousands of packets arrive at the dhcp server. And I really mean thousands

The captures from the dhcp servers show that the “hops” field in the dhcp header  is increasing every dozen or so packets which means that the packet is relayed more than once but the relay agent IP is always the same (VLAN interface @ WLC). Every time the DHCP hops is increased, there is the addition of one option 82 field so the packets are getting larger and larger until they achieve and impressive 1500 bytes.

The WLC / DHCP don’t cope with this rate and at a certain point the interface is marked as "dirty" because no answers are received from the dhcp server and this stops the “loop”.

A thorough search on bug tool and the web didn’t retrieve any relevant information.

Rings any bell?

Regards,

Telmo Oliveira

[Rasika Nayanajith]

DHCP proxy enabled or disabled ?

Have you configured primary & secondary DHCP server information in all dynamic interfaces of your WLCs ?

Also what's the reason using DHCP option 82 ? Is it something can you disable on WLC & test.

Also to what hardware (Nexus ? ) these WLC connnect to ? 

On a side node your running 7.3.x software is defferred. Better to move onto 7.4.121.0 code.

HTH

Rasika

*** Pls rate all useful responses ***

[tjoliveira]

Hi,

Thank you for the quick reply.

Answering to your questions: DHCP proxy is enabled, pri and sec dhcp configured on all dyn interfaces, wlcs connect to c3750G-12S (soon to be replaced) and 7.3 will be upgraded soon.

It is my pleasure to announce that the root cause was found and a workaround was implemented.

Root cause

For some unknown reason, the switch where both WLCs are connected (c3750G-12S) was timing out the entries in the CAM relating to the DHCP servers, even when receiving frames/packets from them.

Behavior description

When the wlcs generate dhcp packets to the dhcp server, they send it to the switch and the switch, without a known mac on the cam, was flooding the frame to all interfaces. The frames flooded to the uplink would reach the dhcp server eventually but some frames were injected back to the wlc, strangely enough. When receiving these frames, the wlc would process (adding another option 82 field and incrementing the “hops” field on the dhcp header) and relay them to the dhcp server, back to the switch which would feedback the whole process.

Workaround

Static mac entries were configured for both dhcp servers’ macs pointing to the uplink

Follow ups

• This switch will be replaced soon for a new c4503-E so no more investment on this issue
• Any change on the dhcp servers’ mac addresses should be reflected on the switch configuration