Client devices reporting 'Incorrect Wi-Fi password' on PEAP network

[t3rebello]

Hi folks,

Running a Catalyst 9800-40 WLC. 1463 WAP's connected, predominantly C9136I WAP's. 8,817 active clients with 4,796 clients on our SSID using PEAP authentication via ISE on the back end.  

We are observing clients will frequently disconnect from the network despite having strong RSSI and SNR levels. The behavior manifests as 'Incorrect Wi-Fi Password', even though the user has been previously connected with no issues and their wireless profile is saved on their phone. I have seen this exclusively on iPhone devices at this time. The DNAC log timestamps seem to lineup with the log 'Client has requested it be deleted' 

To remediate, users can simply hit cancel and wait, and they will reconnect to the network. I am working a TAC case parallel to this community post. Just wanted to throw this out there in case other people were seeing this in their environment. 

[listcsbgnetsecurity]

Hi Rich,

I have upgraded to version 17.12.4 because I don't have anything to rely on as Cisco TAC has abandoned the case asking me to raise on Apple. I am not sure whether those bugs are the ones related to the issue, but let's keep up checking users after upgrade.

[t3rebello]

I actually haven't realized until i saw your update this morning but...after updating to 17.12.4 I don't think I've seen the issue on my campus.

How about you? 

[listcsbgnetsecurity]

Good to hear that. How long have you updated your devices?

[Rich R]

We upgraded to 17.12.4 + APSP1 (has to be requested from TAC) on Tuesday.  We saw some AP crashes within minutes of upgrade due to "Beacon Stuck Reset Radio" even though it was middle of the night with no users and a few more since then so that seems to be a new regression - looks like CSCwm58430 - will open a TAC case for them!  Apart from that no user problems reported so far and seems to be pretty stable.
ps: just had a closer look - also a few AP crashes due to "kernel panic" which are obviously not resolved by APSP1.  Looks like they're using a lot of Meraki code now so I suspect they've imported bugs from the Meraki code which were not there before (note purely my opinion based on how much Meraki gets mentioned in the stack trace of the crash files)

FYI: 17.12.4 APSP1 fixes:
CSCwj72985 multiple wcpd crash during longevity test with ap in flex-LA/LS mode
CSCwj77042 Kernel Panic at "pc : splitmac_api_add_client+0x68/0x498[umac]"SF#07186679

Note that 17.12.4 APSP2 has been published now.  It mentions an extra fix in APSP1 (not mentioned in APSP1 release note) which is:
CSCwj39057 9130: Traffic loss and delays due to perceived channel utilization and interference
APSP2 fixes:
CSCwj66264 Half Duplex Mismatch messages seen on mGig port of 9300, 9400 switches

FYI: 17.12.4 resolves all the PSIRT advisories which affect 9800 which were announced the other day.

Update 10-10-2024: TAC believe the "Beacon Stuck Reset Radio" crashes are caused by CSCwm58430 - devs are still scratching their heads.  The suggested workaround is to disable individual channels on specific APs which I have no intention of doing at this point without further explanation from DE but apparently that stabilised the APs for another customer!  So they're saying on 17.12.4 certain APs won't work reliably on certain channels!

[eglinsky2012]

@Rich R- Have you had a chance to install APSP2 yet? How's that going? Any updates on the AP crashes? I'm gearing up to upgrade to 17.12.4/APSP2 and am wondering if there's anything I should watch out for. (I mean, there will inevitably be something to watch out for; the question is what.)

In the lab, I was unable to join a PSK test SSID on the 5 GHz radio of an 1815W on 17.12.4 APSP2. That was yesterday, and the WLC and APs were up for exactly 1 week following the upgrade/APSP at that point. Rebooting the AP resolved that. I'm going to do daily tests of that radio to make sure it keeps working; could have been a fluke or hardware issue.

[Rich R]

Hi @eglinsky2012 - we're not using any mgig on the 17.12.4 WLCs at the moment so no need for the SP2 fix.  So we'll stick with APSP1 for now.

[eglinsky2012]

---

@Rich R wrote:

Hi @eglinsky2012 - we're not using any mgig on the 17.12.4 WLCs at the moment so no need for the SP2 fix.  So we'll stick with APSP1 for now.

---

APSP3 is out for the following bugs:

CSCwk33521 Multiple 913x/916x AP Kernel Crashes (SF 07238396)
CSCwk58876  Multiple 9166 AP Kernel Crashes (SF 07238396)
CSCwm13005  Router Advertisement packets from clients result in ipv6 gateway change on the Access Point

 Since we upgraded to 17.12.4/APSP2 on Monday, we have had no crash logs generated to the WLC crash for any 9100 series APs. Only a handful of older APs. And we don't use IPv6. So, for now I'll skip APSP3. Just thought I'd mention it for others' benefit.

[Rich R]

Thanks for the update @eglinsky2012 

Naturally they didn't update the release notes! Feedback submitted.

[eglinsky2012]

---

@Rich R wrote:

Thanks for the update @eglinsky2012 

Naturally they didn't update the release notes! Feedback submitted.

---

Yeah, it's not in the "Release Notes for 17.12.4" link, but it is in the "README" page if you hover over the file name on the downloads site.

[listcsbgnetsecurity]

After I have updated WLC to 17.12.4 it seems that the issue of Apple devices giving the message "Incorrect password" and asking for authentication all the time has stopped. Now we have been facing an issue where WLC suddenlly crashes and switchover to standby, the standby crashes in the same way switchover to the previous active and so on, keep flapping and meanwhile APs won't JOIN the WLC an Wifi environment remains out.

Cisco TAC has advised me update to 17.14.1. Has anyone updated for that version?

[marce1000]

    - @listcsbgnetsecurity  >... Now we have been facing an issue where WLC suddenly crashes 
                                              When that happens issue the command : show version | inc reload
                               

 M.

[listcsbgnetsecurity]

In order to recover the environment, I have manually rebooted. When occur again I will take this command, and I will post here.

[Rich R]

I'm not aware of anybody using 17.14.1 and I would not generally recommend using a limited support release.
Did TAC give a specific reason for recommending 17.14.1 - for example did it include a specific bug fix which they think is relevant to the problem?
If not then I would be very suspicious that they are just hoping it will help and it won't make any difference.
If they do point to a specific bug fix then ask them to provide that fix as a SMU for 17.12.4 rather than forcing you to upgrade to 17.14.1.

[listcsbgnetsecurity]

To be honest with you I have never updated an APSP into WLC. Every time I face any issue into wireless platform I search for a new version that correct bugs and issues. Is it common Cisco releasing APSP for all versions? Are you used to update them as soon as they are released?

Other thing is that this version and APSP2 doen't mention any fix on the original issue that we have initially discussed on this community topic that is regarding Apple devices being disconnected asking for password.

[Rich R]

@listcsbgnetsecurity  APSP come out in between major releases and sometimes back-port fixes from later releases to earlier releases too.
They should be assessed and tested just like any new software release.  The difference is they only change AP software not WLC software so you can concentrate the testing on the APs.

The APSP is supplemental to the base release it's applied to so you need to refer to the 17.12.4 release notes for fixes in 17.12.4 and then add the fixes that are in the APSP for the APs.