Client devices reporting 'Incorrect Wi-Fi password' on PEAP network

[t3rebello]

Hi folks,

Running a Catalyst 9800-40 WLC. 1463 WAP's connected, predominantly C9136I WAP's. 8,817 active clients with 4,796 clients on our SSID using PEAP authentication via ISE on the back end.  

We are observing clients will frequently disconnect from the network despite having strong RSSI and SNR levels. The behavior manifests as 'Incorrect Wi-Fi Password', even though the user has been previously connected with no issues and their wireless profile is saved on their phone. I have seen this exclusively on iPhone devices at this time. The DNAC log timestamps seem to lineup with the log 'Client has requested it be deleted' 

To remediate, users can simply hit cancel and wait, and they will reconnect to the network. I am working a TAC case parallel to this community post. Just wanted to throw this out there in case other people were seeing this in their environment. 

[listcsbgnetsecurity]

Hi Rich,

I have upgraded to version 17.12.4 because I don't have anything to rely on as Cisco TAC has abandoned the case asking me to raise on Apple. I am not sure whether those bugs are the ones related to the issue, but let's keep up checking users after upgrade.

[t3rebello]

I actually haven't realized until i saw your update this morning but...after updating to 17.12.4 I don't think I've seen the issue on my campus.

How about you? 

[listcsbgnetsecurity]

Good to hear that. How long have you updated your devices?

[Rich R]

We upgraded to 17.12.4 + APSP1 (has to be requested from TAC) on Tuesday.  We saw some AP crashes within minutes of upgrade due to "Beacon Stuck Reset Radio" even though it was middle of the night with no users and a few more since then so that seems to be a new regression - looks like CSCwm58430 - will open a TAC case for them!  Apart from that no user problems reported so far and seems to be pretty stable.
ps: just had a closer look - also a few AP crashes due to "kernel panic" which are obviously not resolved by APSP1.  Looks like they're using a lot of Meraki code now so I suspect they've imported bugs from the Meraki code which were not there before (note purely my opinion based on how much Meraki gets mentioned in the stack trace of the crash files)

FYI: 17.12.4 APSP1 fixes:
CSCwj72985 multiple wcpd crash during longevity test with ap in flex-LA/LS mode
CSCwj77042 Kernel Panic at "pc : splitmac_api_add_client+0x68/0x498[umac]"SF#07186679

Note that 17.12.4 APSP2 has been published now.  It mentions an extra fix in APSP1 (not mentioned in APSP1 release note) which is:
CSCwj39057 9130: Traffic loss and delays due to perceived channel utilization and interference
APSP2 fixes:
CSCwj66264 Half Duplex Mismatch messages seen on mGig port of 9300, 9400 switches

FYI: 17.12.4 resolves all the PSIRT advisories which affect 9800 which were announced the other day.

Update 10-10-2024: TAC believe the "Beacon Stuck Reset Radio" crashes are caused by CSCwm58430 - devs are still scratching their heads.  The suggested workaround is to disable individual channels on specific APs which I have no intention of doing at this point without further explanation from DE but apparently that stabilised the APs for another customer!  So they're saying on 17.12.4 certain APs won't work reliably on certain channels!

[eglinsky2012]

@Rich R- Have you had a chance to install APSP2 yet? How's that going? Any updates on the AP crashes? I'm gearing up to upgrade to 17.12.4/APSP2 and am wondering if there's anything I should watch out for. (I mean, there will inevitably be something to watch out for; the question is what.)

In the lab, I was unable to join a PSK test SSID on the 5 GHz radio of an 1815W on 17.12.4 APSP2. That was yesterday, and the WLC and APs were up for exactly 1 week following the upgrade/APSP at that point. Rebooting the AP resolved that. I'm going to do daily tests of that radio to make sure it keeps working; could have been a fluke or hardware issue.

[Rich R]

Hi @eglinsky2012 - we're not using any mgig on the 17.12.4 WLCs at the moment so no need for the SP2 fix.  So we'll stick with APSP1 for now.

[eglinsky2012]

---

@Rich R wrote:

Hi @eglinsky2012 - we're not using any mgig on the 17.12.4 WLCs at the moment so no need for the SP2 fix.  So we'll stick with APSP1 for now.

---

APSP3 is out for the following bugs:

CSCwk33521 Multiple 913x/916x AP Kernel Crashes (SF 07238396)
CSCwk58876  Multiple 9166 AP Kernel Crashes (SF 07238396)
CSCwm13005  Router Advertisement packets from clients result in ipv6 gateway change on the Access Point

 Since we upgraded to 17.12.4/APSP2 on Monday, we have had no crash logs generated to the WLC crash for any 9100 series APs. Only a handful of older APs. And we don't use IPv6. So, for now I'll skip APSP3. Just thought I'd mention it for others' benefit.

[Rich R]

Thanks for the update @eglinsky2012 

Naturally they didn't update the release notes! Feedback submitted.

[eglinsky2012]

---

@Rich R wrote:

Thanks for the update @eglinsky2012 

Naturally they didn't update the release notes! Feedback submitted.

---

Yeah, it's not in the "Release Notes for 17.12.4" link, but it is in the "README" page if you hover over the file name on the downloads site.

[listcsbgnetsecurity]

To be honest with you I have never updated an APSP into WLC. Every time I face any issue into wireless platform I search for a new version that correct bugs and issues. Is it common Cisco releasing APSP for all versions? Are you used to update them as soon as they are released?

Other thing is that this version and APSP2 doen't mention any fix on the original issue that we have initially discussed on this community topic that is regarding Apple devices being disconnected asking for password.

[Rich R]

@listcsbgnetsecurity  APSP come out in between major releases and sometimes back-port fixes from later releases to earlier releases too.
They should be assessed and tested just like any new software release.  The difference is they only change AP software not WLC software so you can concentrate the testing on the APs.

The APSP is supplemental to the base release it's applied to so you need to refer to the 17.12.4 release notes for fixes in 17.12.4 and then add the fixes that are in the APSP for the APs.

[Scott Fella]

It would never happen at home because they would not use PEAP as an example, so never compare that.  It's different if one has a lab at home and uses EAP-TLS, PEAP or even iPSK as an example.  I have seen 'Incorrect Wi-Fi password' when using ISE and iPSK only on iPhones, not other devices.  It also only happens if my device connects to a PSK SSID and then connects to a iPSK SSID.  The iPhone sends a different mac address and I was able to open up a TAC case because I had a test controller tied to a test ISE node and we were able to see the iPhone send two mac addresses.  One of course passed, the iPhones mac, but the other mac address showed up in ISE as a failure.  Again, this only happened with iPhones no matter if it was a 13, 14 or 15.  Androids, Linux, Windows and Mac's worked fine.  So you are heading the right direction. Also note, I don't see this with PEAP in our environment.

[t3rebello]

When I mentioned that they say that it doesn't happen at home, I know it's not an Apples to Apples (get it??? heh) comparison but I am just giving the perspective of the disgruntled user/customer. 

[Scott Fella]

I understand that because too hear that every time a user has issues.  Not like us engineers say that also, especially when your home network works better than when you are at work. These statements from users just shows you how they feel about the network and I really don't blame them.  That is why we always try to ensure what we do doesn't affect the user experience. 

If you have a lab, you should try to replicate the issue so you have more data to provide TAC.  It would be interesting to see if the iPhone is sending another auth with a bogus mac address.  Like what Rich mentioned, maybe upgrade to a different code version in hopes that it would help.  

[abdulrauf01251]

Thats Great, helpful post