WAP551 stops passing DHCP

[James Walton]

Hello

I have 4 WAP551s at one of our sites, and I have discovered a problem whereby they stop distributing/passing through DHCP addresses to clients. This happens roughly every 2 days, at which point I simply reboot them all and magically the clients get addresses again and everything is OK for another couple of days.

I don't think it's the DHCP server (Win 2012 R2) since it still gives out addresses on wired clients when this problem occurs.

I have upgraded them all to the latest firmware and tried running them in a cluster (single point setup mode) and separately with the same SSID. Problem occurs in both modes.

If I give my wireless interface a static IP, I get full network access, so I don't think there's any routing/switch config issues.

Any ideas please? They're under warranty but I have no Cisco technical support contract. I could try get them all replaced but it seems unlikely to be a hardware fault with all 4.

[James Walton]

 SR 633302507

[James Walton]

This issue is still occurring, and I do have an open ticket with Cisco, but while I wait for them to reply I thought I would post some updates on here in the hope that someone can recommend a solution.

I've observed that the device doesn't need to be rebooted for DHCP to start working again; yesterday the problem was occurring and I simply left it for a few hours and it started working normally.

I took a packet capture while trying to get an IP to see what could be going wrong, and I noticed that none of the DHCP transactions seem to complete. I see a lot of discover and offer packets, but no follow up requests or acks? I don't think its a client issue i.e. the client not sending requests, since if I go to another of the APs it gets an IP address right away.

It seems as though the request packet isn't getting through the WAP sometimes?

[Michal Bruncko]

Hello James,

just some questions:

- is DHCP server located on same subnet like wireless clients are?

- if so, did you see DHCP discovers coming into DHCP server with packet capture from that server

- if DHCP server is located in different subnet, are you able to get packet capture from router device which is in the middle? If not, are you at least to see/check logs from DHCP relay daemon running on gateway/router device? Did you see logs with packets there?

Do you have any switch in the middle in the data path with enabled DHCP snooping feature? If so, did you tried to disable it and monitor situation?

Have you tried to separate WAP management traffic from User/Production traffic? It could be done simply with creating another VLAN/Subnet between WAP and routing switch/router and putting SSID to that new VLAN.

Simply I don't wanted to believe that WAP somehow manipulating/dropping DHCP requests. It is not capable for that, it is just accesspoint without even any routing capabilities. Moreover all other traffic is crossing your WLAN correctly during DHCP issue (at least you have not complained about that). Lastly nobody else complained and I hope that almost in all scenarios DHCP service is used for clients behind WAP's (but of course such bug can exists for some very specific conditions).

just my two cents..

[James Walton]

Hi Michal,

Yes the DHCP server is on the same subnet, and I can see the same discover - offer with same transaction ID on the DHCP server packet capture.

I have 2 3560 switches in the network, neither have DHCP snooping enabled.

I haven't tried separating the management traffic; do you really think that will make a difference in this situation?

I'm tempted to factory reset the device, rejoin it to the cluster, and change the interface it's using on the switch, then monitor the situation.

[Michal Bruncko]

> Yes the DHCP server is on the same subnet, and I can see the same discover - offer with same transaction ID on the DHCP server packet capture.

Let me try to describe this more:

- you can see DHCP Discover and corresponding DHCP Offer in packet capture from DHCP server during this issue

- but you can't see that DHCP Offer from DHCP server in packet capture on Client during issue

is this correct?

[James Walton]

I can see the discover and corresponding offer on the server capture and the WAP capture.

It's difficult for me to assess the client since I'm not on that site today, but I'll try take a look when I'm there next, hopefully later this week.

[Michal Bruncko]

> I haven't tried separating the management traffic; do you really think that will make a difference in this situation?

Honestly no idea, but it could potentially help as you separate users-related traffic from WAP management traffic. But this is just an assumption. I would go for this option as it is relatively quick to implement and also good recommended practice to have both kind of communication separated.

[josh000014]

I'm curious to see the configs for the two switches.

[James Walton]

I've moved the WAP to a different switch port and factory reset it, then joined it back to the cluster. No problems in last 24 hours since doing this. If problems occur again I will post the switch configs.

[josh000014]

The is good news! Hopefully that was the trick.

[James Walton]

I thought this issue was resolved but sadly it is still occurring.

I have tested with 3 different laptops; one gets an IP address from the DHCP server - the other two don't. These tests were performed within minutes of each other.

I set up wireshark on the DHCP server and the 2 faulty clients - I can see the DHCP discover going out from the clients, hitting the server, the server sends back out the offer, but the offer does not reach either client! It's as though it just gets lost in the network somehow. The trace on the clients simply shows several discovers one after the other, with the server showing discover-offer, repeating.

This is intermittent and seems to affect random laptops. For example, my laptop has exactly the same NIC and driver version, and is connecting through the same WAP as one of the laptops that does not receive an IP address / DHCP offer.

I'm completely stumped at this time! What could be intercepting/preventing the offer packet from reaching the clients?

Edit: I've also done the above test again and with traces running on the 4 WAPs eth0 interface. The DHCP discover and offer both appear on the traces, so this suggests to me that the offer is vanishing in to thin air between the WAP and wireless client!

[Eric Moyers]

I see that your previous case was closed longer than 14 days ago. Please call in and open a new one. give them your old case number and then they can link the new and old together for reference. That you have already gathered some wireshark is great. We can help you go through them and possibly even take some other from different interface within the WAP to see what is happening to the DHCP. Let me know the new case number and I will have someone check in on it. I have moved to a different group, but can still help advise people to check in on some cases.

Eric Moyers
.:|:.:|:. CISCO | Cisco Presales Technical Support | Wireless Subject Matter Expert

Please rate helpful Posts and Let others know when your Question has been answered.

[James Walton]

Hi Eric,

I have a new case open now - 634062045. I've provided the technician with the packet captures and topology details of the network. Fingers crossed for a resolution as this has been driving me mad! :)

Cheers

[Eric Moyers]

Thank you for the info. I have already pulled it up and book marked to keep tabs on.

Eric Moyers

[James Walton]

For everyone's information: this issue was resolved by upgrading all devices to firmware version 1.1.2.3