MacBooks and iPhones on 17.12.4 not getting IP

[AHaugen]

Hi,

Not sure if this is a really weird coincidence, or not.
We upgraded from 17.9.3 to 17.12.4 and alot of iphone and Mac Book users started having issue with our ssid that uses dot1x with PEAP-MSCHAPv2. WPA2/AES.
Controller is c9800-40, aps are 9115i, 9120i and 9120e.
11ax is enabled.
We don't have FT activated because we had issues with apple devices before.

There's not all devices, but apparently alot, mostly from Sonoma and Sequoia I believe.

Wanted to check if someone else has experienced issues with these versions and what you did to fix it, if anything.
Trying to get our customer support to clear the profile for the ssid on the apple devices and also delete the certificate, to see if that helps.

I did some radioactive traces but they show alot of different issues, some seem to be credential or authentication issues.
But some others are like CO_CLIENT_DELETE_REASON_MN_IDLE_TIMEOUT, O_CLIENT_DELETE_REASON_L2AUTH_CONNECT_TIMEOUT and also CO_CLIENT_DELETE_REASON_CLIENT_CREDENTIAL_FAILURE
Alot of the times, I can see that S_CO_IP_LEARN_IN_PROGRESS seems to be successful, but clients dont get an IP.

I've checked the dhcp and we have alot of available IPs in the pool.

I also lowered the EAP timeout to 2 seconds and 10 retires, from 30 seconds and 2 retries that was default.
Not sure if it helped, clients are still having issues, not sure if more or less, but still far from all.

We've tried turning off "private IP" or whats it called, the thing that randomizes mac addresses.
At first it seemed to work, but then we had a bunch of clients it didn't work on.
Also noticed when I was gonna monitor a device directly on an AP that it switched mac address even though the option was turned to off (sequoia).
IP MAC Binding is however enabled, to not exclude the clients that cant turn it off.

[balaji.bandi]

is only apple products issue, other Laptop working as expected.

Suggest to debug from controller using one of the device MAC and see what is wrong

also verify anything changed after upgrade - if you have taken the config back up and compare anything added and missing.

https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/config-guide/b_wl_16_10_cg/dhcp-for-wlans.html

is this authentication using ISE as radius ? check the Logs what is the reason its failing ?

[eglinsky2012]

If you check number of clients on 2.4 GHz vs. 5 GHz, what is the distribution on the APs near the clients? Typically, the vast majority if not all of them will be on 5 GHz (assuming relatively modern and capable clients and dual band SSID), so if most/all are on 2.4 GHz, you may be having the issue described in this post: https://community.cisco.com/t5/wireless/how-monitor-cisco-ap-load/m-p/5210173/highlight/true#M276657

If so, try rebooting the APs.

[marce1000]

  - The basic  c9800-40 configuration check is always useful , also after upgrades :
     Use the CLI command show tech wireless , not simple 'show tech' and feed the output from that into 
     Wireless Config Analyzer
     You will get lots of stuff and parameters to look into which could be related;==> This is so good

      Use https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/217738-monitor-catalyst-9800-kpis-key-performa.html#anc5
      to follow up on overall client behavior , obtaining stats , and possibly getting further insights

      - Best DHCP setup is described in https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/technical-reference/c9800-best-practices.html#DHCPbridgingandDHCPrelay

  M.

[Leo Laohoo]

Disable 802.11k, v r.  

If this works, then it could be CSCwm00078.

[AHaugen]

Thank you guys for all the replies and suggestions.

Yes, it's only apple devices, latest update from custom support is that it seems to be Sonoma and Sequoia on Mac Books, aswell as late versions of iOS on iPhones.
But it's also far from all, it do work for the majority, I can see in the controller that we have a bunch of MacBooks from 2020-2023 connected, aswell as iPhones and "Apple Devices", I also noticed some that connected as "Unknown device".

No, we're not using ISE as radius server, I'm not in charge of the radius server, but ye if that's the reason I will have him look if he can see anything in the logs.

Learned the hard way you have to use show tech wirelesss, I do have that output from after the upgrade, but all show tech I got from before is without wireless sadly.

We're routing the vlan for this SSID on a router, so ip helper is set over there, hasn't been an issue before, maybe this should be moved to the wlc?

Will definietly look into those monitor commands, thanks!

Starting with trying to disable k and v, r has always been off since we tried adaptive ft and haven't really had the time to troubleshoot why that isn't working since then, also apple devices having issues that time.

One thing I'm trying first is disabling OKC, I really can't recall if this was enabled in 17.9.3.
But if I understand that correctly it caches keys for encryption, maybe there is some issue here with WNCD roaming or something?
Our Site-Tag design has needed some work since the start, but lack of time has sadly pushed that to the future, so 2/3 of our APs are in the same Site-Tag, I did see that in 17.12.3(?) they changed how aps are deployed amongst WNCDs.
Just mentioning this because I saw in a radioactive trace a user had a bunch of client delets because of wncd raoming, probably intended, but since this has changed from our earlier version, it might be a bug there or something.

[Rich R]

> We're routing the vlan for this SSID on a router, so ip helper is set over there, hasn't been an issue before, maybe this should be moved to the wlc?
no - keeping that on the router is the recommended option (see Best Practice link below)
And there's also a nasty bug in the helper address code for SVI on 9800 if you have more than 1 server to relay to which we have opened. CSCwm73020

Have you installed all the SMUs and latest APSP?

[AHaugen]

No, those things are kinda new to me.
Should I always install them or when TAC asks me to do so?

Also a update, I got verified that authentication seem to be working. Atleast this client that I looked at yesterday.
He got authenticated.
I could see in the client list in the WLC that he got an IP.
But there were not requests on either of the DHCP servers.
The client did not show any IP.
We then tried to put in the IP as a static IP, and that worked.

My thinking is, the client has received an IP before from the DHCP, and somehow the WLC remembers that IP?
I tried to add dhcp helper and dhcp required for this WLAN/Policy.
Waiting to see if that helped.

[AHaugen]

Im not sure if my last change was what made things work or if Apple came with a bug fix or something.
Since the last change I did was friday afternoon, and now it's been a week without any reported issues.
Most of the ones that customer support had talked to got it working now.

The last thing I did was to put dhcp required and ip helper for the vlan and ssid.
It is routed on a router, not on the WLC, so I felt like it shouldn't matter, but if someone else runs into the issue with devices not getting an IP, I guess it could be something tot try.

[marce1000]

 - Good work  use https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/217738-monitor-catalyst-9800-kpis-key-performa.html#anc5
   to follow up on client behavior and to check if everything remains OK from a client perspective ,

M.