802.1x & FlexConnect (H-REAP) Connectivity Issue

[aaroncoffman]

I'm seeking some input or ideas on some difficulties I'm running in to using RADIUS and FlexConnect APs.

The issue is intermittently users will lose connectivity as if they're de-authenticated. Their clients still believe they're associated to wireless but they have no network connectivity. On Windows 7 we receive the exclamation point over the signal strength indicator.

There doesn’t seem to be any rhyme or reason as to what is causing this. It doesn’t seem to happen at any particular time intervals or anything else I can identify. Sometimes users will go entire days without experiencing connectivity issues sometimes they can’t go five minutes.

When the clients are experiencing the issues they cannot even ping their default gateway.

The setup was initially the following:

Site A: 1142N APs and RADIUS server (server 2003) users are authenticating to.

Site B: Flex 7510 running code 7.0.116

Between site A and site B there is a site to site VPN with no restrictions.

After some time of working with TAC and not getting anywhere I setup the following:

Site A: 4402 WLC running code 7.0.116 connected the same 1142N APs HREAP mode. 

I had the same issue with connectivity with that setup.

Today I changed over to local mode and as of yet I haven’t had reports of connectivity issues.

When running ‘debug client MAC’ I see no indication of connectivity issues. I also have an NCS and I don’t see anything indicating what the issue is there as well.

I’ve tried with both enabling and disabling ‘H-REAP Local Auth’.

We seem to experience the issue in H-REAP standalone mode as well.

If anyone has any ideas or suggestions I’d be grateful.

Thanks,

Aaron

[Stephen Rodriguez]

do you have a HREAP Group configured for the AP at each site?

Steve

[aaroncoffman]

Steve,

I do indeed have H-REAP groups setup. I'm fairly confident that the H-REAP groups are configured properly as when in stand alone mode I am able to authenticate new clients to the RADIUS server and roam without issue.

Aaron

[dougcogley]

we had this same issue. There was a change to our proxy server to authenticate users sessions. When this times out windows 7 is restricted to the Internet to get to the page that validates window 7 can reach the internet even though the ! Is there internal resources worked fine. We added a by pass rule in the proxy and this fixed the issue. No changes to wireless controllers or clients was required.

Sent from Cisco Technical Support iPhone App

[aaroncoffman]

dougcogley,

Were your devices able to continue pinging their default gateway?

If they were not able to continue pinging what was the purpose of your proxy server, what was it doing? I'm not certain we have a smiliar setup in that regards.

Thanks,

Aaron

[dougcogley]

Yes they could get to all internal resources even though Windows 7 shows !.  That includes the default gateway.  Access to the inetrnet was being blocked creating the ! on the client. 

[aaroncoffman]

dougcogley,

I appreciate the input.

Our machines are not able to ping the default gateway, nor are they able to access any internal resources let alone external resources. This is an intermittent issue.

Regards,

Aaron

[aaroncoffman]

In addition to what's currently been mentioned I've updated the WLC software to 7.0.230. Still the issue continues.

Any thoughts?

[Dave Row]

Hi Aaron,

I am experiencing the same issue when using a WISM2 module in a 6500 chassis and WLC software 7.0.116.0.  I have recently (today) upgraded a redundant controller in the failover pair to 7.2.110.0 and reassociated the relevant AP's to this upgraded controller.

The APs (2 x AIR-CAP3502I-E-K9) have upgraded their software to the new version and are now in the same FlexConnect config as previously seen with H-REAP in 7.0.116.0. The intermittent connectivity continues with the a lack of connectivity at various times with no discernable triggers or fix.

These are the only two APs we run in FlexConnect and provide WiFi internet access to one of our clients (when working)

I'm hoping you managed to resolve this issue and are willing to share the solution!

Thanks

Dave

[aaroncoffman]

David,

I forgot all about this! My apologies.

At site A (where the APs and users are) I moved to a 5508 WLC running 7.0.116 and had the same issues in FlexConnect mode. I then upgraded the controller code to 7.2.103 and haven't had the issue since. It sounds like you're already at the latest code so unfortunately my fix may not be yours. However in my situation the WLC is on the same site/switch as the APs experiencing issues.

Regards,

Aaron

[Erik Boss]

Hi Aaron,

As I read your post, I think the latency between site A and B is too high.

That's why local mode would work fine but H-REAP isn't. Did you try to check if the AP's and WLC are on the same site?

Problem still exists?

I did some installations, also with H-REAP without any problems. Sometimes the line between both sites has a high latency, the AP's are moving from one controller to the other or couldn't join anymore.

Could you send us an traceroute with a good connection and with a bad connection?

Regard,  Erik

[scottchang]

I got same problem in WLC5508 with AP2602 (Flexconnect)

Looks like when session timeout is expired, AP put the client to de-authenticated.

But Client didn't know and the wireless status still show "connected"

Couldn't ping anywhere.

Below is the debug from Controller. Could anyone help this ? Appreicate

WLC is 7.3.101

(Cisco Controller) >*SNMPTask: Jul 02 17:02:06.974: 00:22:fb:9a:4b:16 Central Switch = FALSE 
*apfReceiveTask: Jul 02 17:03:40.010: 00:22:fb:9a:4b:16 0.0.0.0 DHCP_REQD (7) DHCP Policy timeout. Number of DHCP request 0 from client
*apfReceiveTask: Jul 02 17:03:40.010: 00:22:fb:9a:4b:16 0.0.0.0 DHCP_REQD (7) Pem timed out, Try to delete client in 10 secs.
*apfReceiveTask: Jul 02 17:03:40.010: 00:22:fb:9a:4b:16 Scheduling deletion of Mobile Station:  (callerId: 12) in 10 seconds
*spamApTask1: Jul 02 17:03:40.098: 00:22:fb:9a:4b:16 Received Idle-Timeout from AP 34:bd:c8:d9:79:70, slot 1 for STA 00:22:fb:9a:4b:16
*spamApTask1: Jul 02 17:03:40.098: 00:22:fb:9a:4b:16 apfMsDeleteByMscb Scheduling mobile for deletion with deleteReason 4, reasonCode 4

*spamApTask1: Jul 02 17:03:40.098: 00:22:fb:9a:4b:16 Scheduling deletion of Mobile Station:  (callerId: 30) in 1 seconds
*spamApTask1: Jul 02 17:03:40.098: 00:22:fb:9a:4b:16 CCKM: Sending cache delete
*spamApTask1: Jul 02 17:03:40.098: 00:22:fb:9a:4b:16 PMK: Sending cache delete
*spamApTask1: Jul 02 17:03:40.098: 00:22:fb:9a:4b:16 Removing PMK cache entry for station 00:22:fb:9a:4b:16
*osapiBsnTimer: Jul 02 17:03:41.010: 00:22:fb:9a:4b:16 apfMsExpireCallback (apf_ms.c:597) Expiring Mobile!
*apfReceiveTask: Jul 02 17:03:41.010: 00:22:fb:9a:4b:16 apfMsExpireMobileStation (apf_ms.c:5687) Changing state for mobile 00:22:fb:9a:4b:16 on AP 34:bd:c8:d9:79:70 from Associated to Disassociated

*apfReceiveTask: Jul 02 17:03:41.011: 00:22:fb:9a:4b:16 Sent Deauthenticate to mobile on BSSID 34:bd:c8:d9:79:70 slot 1(caller apf_ms.c:5781)
*apfReceiveTask: Jul 02 17:03:41.011: 00:22:fb:9a:4b:16 apfMsAssoStateDec
*apfReceiveTask: Jul 02 17:03:41.011: 00:22:fb:9a:4b:16 apfMsExpireMobileStation (apf_ms.c:5819) Changing state for mobile 00:22:fb:9a:4b:16 on AP 34:bd:c8:d9:79:70 from Disassociated to Idle

*apfReceiveTask: Jul 02 17:03:41.011: 00:22:fb:9a:4b:16 pemApfDeleteMobileStation2: APF_MS_PEM_WAIT_L2_AUTH_COMPLETE = 0.
*apfReceiveTask: Jul 02 17:03:41.011: 00:22:fb:9a:4b:16 0.0.0.0 DHCP_REQD (7) Deleted mobile LWAPP rule on AP [34:bd:c8:d9:79:70]
*apfReceiveTask: Jul 02 17:03:41.011: 00:22:fb:9a:4b:16 apfMs1xStateDec
*apfReceiveTask: Jul 02 17:03:41.011: 00:22:fb:9a:4b:16 Deleting mobile on AP 34:bd:c8:d9:79:70(1) 
*pemReceiveTask: Jul 02 17:03:41.012: 00:22:fb:9a:4b:16 0.0.0.0 Removed NPU entry.
*spamApTask1: Jul 02 17:03:41.105: 00:22:fb:9a:4b:16 Received Idle-Timeout from AP 34:bd:c8:d9:79:70, slot 1 for STA 00:22:fb:9a:4b:16
*apfMsConnTask_1: Jul 02 17:03:41.321: 00:22:fb:9a:4b:16 Adding mobile on LWAPP AP 34:bd:c8:d9:79:70(1) 
*apfMsConnTask_1: Jul 02 17:03:41.321: 00:22:fb:9a:4b:16 Association received from mobile on AP 34:bd:c8:d9:79:70
*apfMsConnTask_1: Jul 02 17:03:41.321: 00:22:fb:9a:4b:16 Global 200 Clients are allowed to AP radio

*apfMsConnTask_1: Jul 02 17:03:41.321: 00:22:fb:9a:4b:16 Max Client Trap Threshold: 0  cur: 1

*apfMsConnTask_1: Jul 02 17:03:41.321: 00:22:fb:9a:4b:16 Applying Interface policy on Mobile, role Unassociated. Ms NAC State 0 Quarantine Vlan 0 Access Vlan 0

*apfMsConnTask_1: Jul 02 17:03:41.321: 00:22:fb:9a:4b:16 Re-applying interface policy for client

*apfMsConnTask_1: Jul 02 17:03:41.321: 00:22:fb:9a:4b:16 0.0.0.0 START (0) Changing IPv4 ACL 'none' (ACL ID 255) ===> 'none' (ACL ID 255) --- (caller apf_policy.c:1839)
*apfMsConnTask_1: Jul 02 17:03:41.321: 00:22:fb:9a:4b:16 0.0.0.0 START (0) Changing IPv6 ACL 'none' (ACL ID 255) ===> 'none' (ACL ID 255) --- (caller apf_policy.c:2006)
*apfMsConnTask_1: Jul 02 17:03:41.321: 00:22:fb:9a:4b:16 In processSsidIE:3937 setting Central switched to FALSE
*apfMsConnTask_1: Jul 02 17:03:41.321: 00:22:fb:9a:4b:16 Applying site-specific Local Bridging override for station 00:22:fb:9a:4b:16 - vapId 1, site 'SMT_AP', interface 'vlan-smt-wifi'
*apfMsConnTask_1: Jul 02 17:03:41.321: 00:22:fb:9a:4b:16 Applying Local Bridging Interface Policy for station 00:22:fb:9a:4b:16 - vlan 4, interface id 16, interface 'vlan-smt-wifi'
*apfMsConnTask_1: Jul 02 17:03:41.321: 00:22:fb:9a:4b:16 Applying site-specific override for station 00:22:fb:9a:4b:16 - vapId 1, site 'SMT_AP', interface 'vlan-smt-wifi'
*apfMsConnTask_1: Jul 02 17:03:41.321: 00:22:fb:9a:4b:16 Applying Interface policy on Mobile, role Unassociated. Ms NAC State 2 Quarantine Vlan 0 Access Vlan 508

*

[Abha Jha]

posted below ans :-

for below message could you please try enabling the SKC support :-

No valid PMKID found in the MSCB PMKID cache for mobile 00:22:fb:9a:4b:16
*apfMsConnTask_1: Jul 02 17:03:41.322: 00:22:fb:9a:4b:16 Trying to compute a PMKID from MSCB PMK cache for mobile 00:22:fb:9a:4b:16

link :-

http://www.cisco.com/c/en/us/td/docs/wireless/controller/7-4/configuration/guides/consolidated/b_cg74_CONSOLIDATED/b_cg74_CONSOLIDATED_chapter_01010010.html

[scottchang]

Hi Abhajha,

Thanks for the reply.

I upgraded firmware from 7.3 to 7.6 then it seems fixed the problem.

Also saw there is an known issue in 7.6 release but the issue is still open and not solved yet.

Glad it works for my WLC but it definitely some problem in firmware and caused the random de-authentication issue in Flexconnect APs.

Thanks again