Mobility Express Cluster- Client IP reported as Gateway IP address

[Devinder Sharma]

Hello All,

I have a strange situation wherein sometimes one of the client will show IP address to be the same as Ip address of gateway learnt from the external DHCP address. If I block that mac address via CLI, then after few minutes I see another client exhibiting the same issues. And if I block that second client, then I find a third client assumes the same address.

First I though that someone is playing tricks by assigning a static IP to impersonating the IP of the gateway, but now I find that this issue is only found in the Apple clients (it could be a MacBook pro, a mac book air). 

When this happens, 3 to 4% of the client devices will lose Internet access as they cannot get to their gateway and probably this client becomes a rogue gateway to pollute the mac cache on few clients. Of course these all devices remain connected to Wireless (and have solid signal and SNR and most devices are using 5GHz).

The cluster has 65 APs and few APs are 2802i and remaining are 1832i. It is a student dorm and hence devices are not in our control. Running 8.7.106.0 code on the Controller.

Anyone experienced such issues with this code or with any other code, please advise.

Thanks

[Ric Beeching]

Sounds like a fun bug. TAC Recommended ME versions are 8.5.140.0 or 8.8.111.0 so I would go to 8.5.140.0 unless there's a specific feature you need such as AP Groups which is only available in the higher codes.

https://www.cisco.com/c/en/us/support/docs/wireless/wireless-lan-controller-software/200046-tac-recommended-aireos.html#anc17

Ric

[Devinder Sharma]

Thanks Ric. Yes all signs of a bug. I had looked at TAC recommended AIrOS list for ME. So I wanted to downgrade to 8.5.140, but I also opened a TAC case and they have recommended to go to 8.8.111. So I will try upgrading to 8.8.111.0 in the morning. 

Very best,

[Ric Beeching]

Good luck, let us know how it goes!

[patoberli]

If it doesn't help, did you make sure that your DHCP is not actually providing the gateway IP address for lease? You need to take out the gateway (and WLC interfaces) address from the DHCP pool. Not all DHCP servers do that by default!

[Devinder Sharma]

We did upgrade and then even downgraded, both times advised by TAC. Now running 8.5.140.0 and I see up to 4 clients showing address as same as gateway. Of course gateway address plus another nine in the range are excluded from the dhcp scope.

Looking into dhcp server leases, each client have their distinct address, but it is the ME-WLC that is culprit here. It somehow starts thinking that the mac address of gateway is on the wireless, starts labeling them as the  and is one or several of the clients and thus causes ARP corruption and hence blackholing of traffic. We even tried using a different type of appliance as a dhcp server and this still happens.

All devices that are chosen to show their address as same as gateway address are apple mac variations (mac mini, mac air, macbook etc.).

I have this issue going on for 6 weeks and I am sure I am not the only one but TAC is not able to help. I have now requested them to raise the severity level to at least 2 else, response is so slow and it takes days to hear back on providing any inputs in response to their questions.

Thanks

[patoberli]

Now that I read that again, it remains me of an issue I had with a single 2802i in ME configuration running 8.8.111.0. I had a Netgear Bridge with several wired clients attached to the bridge. As soon as a second client on the bridge came online, my whole wireless was offline, for around a minute. For whatever reason, the ARP entries were wrong of my wired clients once that happened. After around 1 minute it recovered. I then replaced it with an Ubiquiti AC Wave2 model (the Cisco was just temporary) and guess what, with the Ubiquiti I had the exactly same issue, only that the Ubi never recovered until I rebooted it. Then I installed the latest beta firmware on it and it started to work correctly.

Anyway, did you also try 8.8.120.0? Maybe that fixes the issue.

[Devinder Sharma]

Thanks. I had asked support engineer for 8.8.120.0 and he said I should stick with TAC approved 8.5.140.0. While another TAC engineer had advised to use 8.8.110.0, which is also TAC approved, but after upgrading, I was then asked later to downgrade to 8.5.140.0. I will love to test 8.8.120.0, if TAC will approve that for my case.

[patoberli]

8.8.120.0 is still very new and that's the reason why it's not yet TAC approved. I guess in 1-2 weeks, unless some serious bugs are found, it will be TAC approved and 8.8.110.0 removed.

[Devinder Sharma]

Thanks. I will try again to ask them to let me try 120.0.

I ruled out any rogue dhcp server giving out addresses in the same subnet. I have dhcp snooping on all cisco switches and trusted are uplink. Unlike WLC, I cannot specify the dhcp server / required, so that it is more like a wireless dhcp snooping.