topic Issue with RPI CM4 on PEAP-MSCHAPv2 network MIC validation error in Wireless

Issue with RPI CM4 on PEAP-MSCHAPv2 network MIC validation error

overclockedap — Wed, 07 Aug 2024 21:11:08 GMT

Hi,

I am having a lot of trouble connecting our product to a customers WiFi network. Their network is entirely Cisco hardware with Cisco ISE I think its called. The network uses PEAP-MSCHAPv2 and the authentication device is a Cisco 9800 WLC.

Our product uses a Raspberry Pi CM4 running our own Yocto Linux build. The OS has standard networking and wpa_supplicant installed along with utilities for doing DHCP. We currently only want to connect with a static IP.

I have the WiFi working perfectly on every network I tested it on except for the customers network. I don't understand why its not working. I build a PEAP-MSCHAPv2 network at home to validate this and tested it on our in office network using the same with both static and DHCP IP's.

The packet capture from our device shows that we successfully complete the PEAP handshake and the Cisco network sends a final "success" message. That matches the packet captures I took on the other networks exactly.

The problem is that exactly 6 seconds later the Cisco network sends a message to our device asking it to reidentifying/reauthenticate which leads to it going through the PEAP handshake again and once again receiving a "success" message. This goes on in a loop forever.

On our device the WiFi connection indicator turns on, then off, then on forever.

A laptop or a phone can connect to the network using the same credentials/settings.

I have spent over a month on this and its driving me nuts.

The customer created a TAC case and the Cisco tech said nothing looked to be wrong. I was able to get in touch with some of the customers IT personal and so far a "radio active" capture revealed that the Cisco network is in fact deleting our device as a client and causing this to happen. It is a MIC validation error see below for full error text.

Hopefully next week I will have a more in depth error message since I believe there is a more detailed error message on the Cisco ISE portal/site something like that but the person who can access that is out of the office.

The only thing I noticed that was really different about the network was that with an "iw dev wlan0 scan" command I found that the two PEAP-MSCHAPV2 networks that worked had

authentication suite = IEEE8021X

Whereas the Cisco network shows up as:

authentication suite = IEEE8021X IEEE8021X/SHA-256

Here is an example wpa_supplicant.conf file that we have been using:

ctrl_interface=/var/run/wpa_supplicant ctrl_interface=DIR=/var/run/wpa_supplicant ctrl_interface_group=0 p2p_disabled=1 update_config=1 network={ ssid="exampleNetwork" priority=1 proto=RSN key_mgmt=WPA-EAP pairwise=CCMP eap=PEAP identity="exampleUsername" password="examplePassword" phase2="auth=MSCHAPV2" }

The specific error is a MIC validation error. The credentials are correct though so what is happening here???

L2 Authentication Key Exchange Start. Resolved VLAN: 2709, Audit Session id: 1234567890 Keymgmt: Failed to validate eapol mic. MIC mismatch. Keymgmt: Failed to validate eapol key m2. MIC validation failed Keymgmt: Failed to validate eapol mic. MIC mismatch. Keymgmt: Failed to validate eapol key m2. MIC validation failed Keymgmt: Failed to validate eapol mic. MIC mismatch. Keymgmt: Failed to validate eapol key m2. MIC validation failed Keymgmt: Failed to eapol key m1 retransmit failure. Max retries for M1 over Client delete initiated. Reason: CO_CLIENT_DELETE_REASON_KEY_MGMT_MIC_VALIDATION, details: , fsm-state

Re: Issue with RPI CM4 on PEAP-MSCHAPv2 network MIC validation error

overclockedap — Wed, 07 Aug 2024 21:29:19 GMT

Also one detail I forgot to mention not sure if its relevant:

The two networks that works are using:

802.1X-2001 and 802.1X-2004

The customer network that isn't working without device is using:

802.1X-2010

And it was also mentioned that we are clearly getting through layer 2. But that layer 3 does not seem to be working?

What would I need to change on my wpa_supplicant.conf file or my network interfaces file to make this work.

Could having no DNS servers set potentially cause this? Someone told me DNS servers may be needed for the layer 3 to work?

Its possible that our DNS servers were not set correctly on our device but I have fixed that now. And even though it was tested with hard coded DNS servers I wanted to test it again with this fix just to be extra sure that isn't the problem.

Re: Issue with RPI CM4 on PEAP-MSCHAPv2 network MIC validation error

Mark Elsen — Thu, 08 Aug 2024 06:12:28 GMT

- What software version is the 9800 WLC running , I would advise to go for 17.12.3 if not yet there.
The problem seems similar too https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwa93009
hence my advise to use the latest advisory software version for the controller and check if that can help ,

Re: Issue with RPI CM4 on PEAP-MSCHAPv2 network MIC validation error

Rich R — Mon, 12 Aug 2024 10:56:19 GMT

As Marce says the key question is: what version of software is the 9800 running?

There have been a number of bugs around MIC validation which have been fixed recently so moving to the latest TAC recommended code version (as per below link) might just resolve the problem. Currently that recommended version is 17.12.3, likely to become 17.12.4 in the next few weeks. You could also try turning off SHA-256 in case that is causing problems for your client but if that solves it then that probably means your client/driver/supplicant needs updating.

If you have a Cisco AP then you could do some testing and debugging yourself using EWC on AP or with 9800-CL WLC.

Re: Issue with RPI CM4 on PEAP-MSCHAPv2 network MIC validation error

overclockedap — Mon, 12 Aug 2024 14:37:13 GMT

The version of Cisco ISE is 3.1.0.518 and the 9800 WLV version is 17.09.04a, still waiting to see if we can get approved to upgrade it.

I went and built USB drivers for a very capable USB WiFi adapter with a Realtek chipset so that is working now. My thought was that by using this I could completely eliminate the Raspberry Pi WiFi chipset/driver as being the cause. If the exact same thing happens after switching then I believe it would strongly point toward a Cisco Network issue or wpa_supplicant issue.

Re: Issue with RPI CM4 on PEAP-MSCHAPv2 network MIC validation error

overclockedap — Mon, 12 Aug 2024 14:40:08 GMT

The 9800 WLC was describe to me as "Cisco IOS XE Software, v17.09.04a" not sure if the IOS XE makes any difference just wanted to add that detail.

Re: Issue with RPI CM4 on PEAP-MSCHAPv2 network MIC validation error

Rich R — Mon, 12 Aug 2024 23:05:37 GMT

IOS-XE is the type of IOS which the 9800 series WLCs run (versus IOS or IOS-XR which are other variants) - most of the newer Cisco platforms run on IOS-XE now. Some older platforms/architectures still run on IOS. IOS-XR is used by some of the larger service provider and data centre platforms.

Re: Issue with RPI CM4 on PEAP-MSCHAPv2 network MIC validation error

overclockedap — Mon, 19 Aug 2024 20:21:19 GMT

Ok so after a lot of debugging the issue was isolated to be the "Fast Transitioning" setting in the Cisco WiFi AP/network configuration. This is all done through ISE/9800 WLC with Cisco AP point hardware.

Utilizing that supplicant file above we tried connecting to a network with "Fast Transition = Adaptive".

The WiFi network was duplicated exactly with the one change being "Fast Transition = Disabled".

I was then able to connect to this new network that has fast transition fast transition disabled using the original wpa_supplicant.conf file I posted above. Everything works I was able to reach the local network / internet and the DNS servers were set properly.

So what is going on here exactly? It would seem that the Cisco network is improperly seeing our device as supporting fast transition and then we don't send up something it needs leading to the resetting issue.

Ideas?

Re: Issue with RPI CM4 on PEAP-MSCHAPv2 network MIC validation error

Rich R — Mon, 19 Aug 2024 23:29:39 GMT

It is not recommended to use Adaptive FT anymore for exactly this reason.

This is clearly documented in https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/technical-reference/c9800-best-practices.html#enablefasttransition
"The reality is that in a mixed-client network, some non-FT clients may experience issues in connecting to a WLAN with Adaptive FT, so the recommendation from Cisco is to configure a single WLAN with “802.11r mixed mode”, to allow for compatibility between 802.11r and non-802.11r clients: Set Fast Transition to enabled and select both FT and non-FT Authentication and Key Management (AKM) modes. This is called “802.11r mixed mode” as it allows clients to choose the AKM with or without 802.11r depending on their capability. Below is a configuration example for WPA/WPA2 security and 802.1x AKM:"

The best practice guide is recommended reading if you are working with 9800 WLCs.
You should also use the Config Analyzer (link below) to check your WLC config. That checks and highlights many of the Best Practice items for you.