9800 Controller Giving "IP Address Theft" and Excluding Clients

[ferriterj1]

Hello!

I work in a school district that has three main high school buildings, one alternative placement building, and the administration office for a total of five separate buildings. 

Very recently, I've had one of our school buildings send reports of users whose MacBooks connect to the district WiFi and then immediately drop. When checking our 9800-40 WLC, I can see that the client has been excluded for the reason of "IP address theft". I had never seen this error before, and no other buildings were reporting it, so I thought it might have been a one-off error and didn't think much of it. Then a handful more reports came in from that one building and all were "IP address theft". 

I checked the building's Windows DHCP server, and nothing looked out of the ordinary. The IP addresses that were listed in the WLC for the excluded client matched the entry in DHCP. I've restarted the DHCP service just to be safe, but that did not help. I deleted the DHCP entry for any affected clients, and that helped one connect but did nothing for the others.

We have MAC address randomization disabled for these clients. We utilize Cisco ISE 3.2 on Patch 6 and these clients connect to the network via certificate auth. All certificates are current, valid, and trusted by the client devices. Our 9800 WLC is on 17.12.4 will all necessary additional files.

Now the reports have died down at the original building, and I'm starting to get reports at another building. This building has the same set up as the other, just a different IP subnet. So I'm at a loss! We never experienced this issue in our 4+ years of having the 9800, so I'm unsure what to do. 

I have a TAC case open, but wanted to get some input from the community. Thank you for any helpful tips you can provide!

[Scott Fella]

I would identify a few of the mac address and validate to make sure those are managed by your team.  These devices don't switch to another SSID, but are managed to only connect to one of your SSID's? What does your ISE logs look like, you should be able to catch the device getting authorized and then some failures? The detailed logs can help you and TAC, but you really need to be able to replicate the issue if possible.

[ferriterj1]

Thank you for your input!

The Cisco ISE live logs show nothing but successful authentications for one device that is a district-issued MacBook that is authenticating via cert auth. These devices can only connect to our staff SSID and can't migrate to another. Those settings are pushed down via a configuration profile from our Apple MDM. 

Some of the other excluded devices are cell phones, which we don't manage and could have MAC randomization turned on, so I'm more concerned about the district-issued devices for sure!

[Scott Fella]

Well I can tell you have I have seen when I was testing a few months ago.  I was seeing apple devices connect fine, then getting errors on ISE and the controller about a device failing.  The apple devices after connecting was actually also sending another auth with a different mac address.  This is what I saw on ISE with a test node so that I don't see all the logs from other devices.  Windows and android devices didn't show this issue, just Apple.  I thought I had a TAC case open on it, but can't find it. I might of worked this out with my Cisco ISE SE. 

[ferriterj1]

Thanks, Scott! That would make sense if you were getting an "IP address theft" exclusion reason if they were sending multiple authentication requests at the same time. Unfortunately (or fortunately?) I don't believe that is the case here. One auth request from their one MAC address, passing ISE via certificate authentication, but then dropping at the WLC because it thinks that the IP address given to that MAC address actually belongs to a different MAC address. 

I appreciate the help!

[Scott Fella]

I understand.  TAC should be able to dig something out especially is there is a known issue that might not be public.

[marce1000]

 - You  should try to debug an affected client using https://logadvisor.cisco.com/logadvisor/wireless/9800/9800ClientConnectivity
    these can be further examine using : Wireless Debug Analyzer
    Also checkout commands from https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/217738-monitor-catalyst-9800-kpis-key-performa.html#toc-hId-866973845 to get additional insights

    + Always have a validation of the 9800-40 controller configuration using the CLI command show tech wireless and
       feed the output from that into Wireless Config Analyzer
       Although initially and currently that is not directly related , yet it is strongly advised

 M.
       

[ferriterj1]

Thank you! I'll make sure to take a look at this with logs from this client!

[ferriterj1]

Hi @marce1000 

I ran the RA trace that I had taken for TAC through the debug analyzer, and took a look at the log entries related to "IP address theft". Namely, the ones that say "SISF theft event IP_THEFT_ATTEMPT IP <IP address> <vlan #> ifhdl: 0x91000070 owned by MAC: <other mac address>". However, when I dig into the "other mac address", I can see that it doesn't own the IP in our Windows DHCP server. Also, when searching ISE for that other mac address, it does not appear.

Are there any known bugs related to stale clients for the 9800 and could that be what is happening here? Basically, according to the logs, I don't believe the IP address belongs to the MACs that the 9800 believe that they do in these error messages. 

This was very helpful, thank you!

[marce1000]

  - You may find this thread interesting (too) : https://community.cisco.com/t5/wireless/client-excluded-for-quot-ip-address-theft-quot-on-cisco-9800-80/td-p/4960593

  M.