Getting disconnected randomly (5508 controller, 3300 series LAPs)

[keithsauer507]

I am at one of our remote offices and I am noticing my laptop, despite excellent signal strength is periodically losing IP connectivity on the wireless network.  When it drops, all of my IP connectivity stops (pings fail, RDP sessions "await reconnection", etc...).  The lower right corner still shows I'm connected to the hidden WPA2 Enterprise SSID.  The only way to reconnect is to select disconnect on it, then click connect again.  Immediately everything IP based starts working.

There is a 5508 controller in the headquarters.  The site I am at has a 30mbps fiber point to point WAN to the headquarters.  This site has 2 x 3300 series LAPs which are very good coverage.  H-REAP mode is on so traffic terminates at the local office because it is more efficient than traversing the LAN twice for things like local file and print sharing, dhcp, proper active directory sites and services mapping, etc...

The 5508 has a 2008 R2 server running NPS to do radius authentication and it verifies a domain certificate.  To be on the wireless you have to be a member of the domain.

Seems to not give me any problems at the home office so any idea's?

On the 5508 I see this around the times I lose IP connectivity:

*Dot1x_NW_MsgTask_4: Jan 08 14:00:53.599: #DOT1X-3-INVALID_WPA_KEY_MSG_STATE: 1x_eapkey.c:861 Received invalid EAPOL-key M2 msg in START  state - invalid secure bit; KeyLen 24, Key type 1, client 88:53:2e:xx:xx:xx

*Dot1x_NW_MsgTask_4: Jan 08 14:00:52.551: #DOT1X-3-INVALID_WPA_KEY_MSG_STATE: 1x_eapkey.c:861 Received invalid EAPOL-key M2 msg in START  state - invalid secure bit; KeyLen 24, Key type 1, client 88:53:2e:xx:xx:xx

*Dot1x_NW_MsgTask_4: Jan 08 14:00:52.387: #DOT1X-3-ABORT_AUTH: 1x_bauth_sm.c:447  Authentication Aborted for client 88:53:2e:xx:xx:xx

I have 0 (unlimited) as the max for user login policies so not sure why Authentication Aborted message appears.

WLC Software version

7.4.100.0

On the NPS server (2008 R2) I just see my username granted access because it matches the network health policy.

Laptop Sony VAIO SE2

Intel Centrino Advanced-N 6230

Driver version 15.1.1.1 Date: 3/12/2012

[keithsauer507]

jmglass@grove.iup.edu wrote:
Little late but fyi, 
Open Caveats on 7.4.100.0, CSCuc78713 could be related to this;
CSCuc78713
Symptom: Wireless client cannot receive broadcast packets after broadcast key rotation.

Broadcast packets... hmm that could explain why the problem only exists at our remote offices.  They of course are on a different subnet than the WLC, so if the WLC is broadcasting, of course that doesn't traverse the router > WAN.

[jmglass]

HI,

> I changed it to 86400 during the day.  You think I should VPN at night and change it, and change it back?

No, leave it at 86400.

My attempt at a joke about the comment in the open Caveat Workaround:

" • Executes the config advanced eap bcast-key-interval 86400 in the middle of the night "

Only reason we changed this back on 5.x code was we saw some clients, (~5~10%), getting knocked off on the hour.Traced it to the broadcast key rotation, so did set it to 24 hours at 4:00AM when we have the least amount of clients online. Always upgrade/reboot WLC about this time also, timer reset so not knocking clients off when it rotates.

best!

jim

[Scott Fella]

Haha... I didn't read the bug.

We were told to do this by TAC a few years back because of a certain device having issues. Clients didn't like the fact that the key wouldn't change for 24hrs. Some things you have to balance and eventually you don't want to have the users not have a good experience on the wireless.

Sent from Cisco Technical Support iPhone App

[keithsauer507]

Well this fix seems to help so this morning I rebooted the controller at 6:45 AM which is before anyone would be in to use the Wireless. I double checked show advanced eap and its still set to 86400. So the chance that the key has to be rotated should happen before people are on.

Thanks to everyone for this insight!

[keithsauer507]

Ok I am at a remote office today and it seems the wireless is still disconnecting.  It seems to be more frequent than before.  Now its almost like every half hour the IP traffic just stops flowing.  Signal is good (5 bars), signal stats are great.  Physical layer never disconnects... its the IP layer that just completely stops.

I did show advanced eap  can you confirm these settings are good?

EAP-Identity-Request Timeout (seconds)........... 30

EAP-Identity-Request Max Retries................. 2

EAP Key-Index for Dynamic WEP.................... 0

EAP Max-Login Ignore Identity Response........... enable

EAP-Request Timeout (seconds).................... 30

EAP-Request Max Retries.......................... 2

EAPOL-Key Timeout (milliseconds)................. 1000

EAPOL-Key Max Retries............................ 4

EAP-Broadcast Key Interval....................... 86400

[keithsauer507]

Checking the logs, theres a TON of these, but my mac address shows up with this message twice when the IP stops routing.

*Dot1x_NW_MsgTask_4: Feb 05 13:06:52.771: #DOT1X-3-INVALID_WPA_KEY_MSG_STATE: 1x_eapkey.c:861 Received invalid EAPOL-key M2 msg in START  state - invalid secure bit; KeyLen 24, Key type 1, client 88:53:2e:8e:85:XX

*Dot1x_NW_MsgTask_4: Feb 05 13:06:51.718: #DOT1X-3-INVALID_WPA_KEY_MSG_STATE: 1x_eapkey.c:861 Received invalid EAPOL-key M2 msg in START  state - invalid secure bit; KeyLen 24, Key type 1, client 88:53:2e:8e:85:XX

[Brendan Marmont]

Did you ever get a resolution for this, we are having the same issue?

Thanks

Brendan