Getting disconnected randomly (5508 controller, 3300 series LAPs)

[keithsauer507]

I am at one of our remote offices and I am noticing my laptop, despite excellent signal strength is periodically losing IP connectivity on the wireless network.  When it drops, all of my IP connectivity stops (pings fail, RDP sessions "await reconnection", etc...).  The lower right corner still shows I'm connected to the hidden WPA2 Enterprise SSID.  The only way to reconnect is to select disconnect on it, then click connect again.  Immediately everything IP based starts working.

There is a 5508 controller in the headquarters.  The site I am at has a 30mbps fiber point to point WAN to the headquarters.  This site has 2 x 3300 series LAPs which are very good coverage.  H-REAP mode is on so traffic terminates at the local office because it is more efficient than traversing the LAN twice for things like local file and print sharing, dhcp, proper active directory sites and services mapping, etc...

The 5508 has a 2008 R2 server running NPS to do radius authentication and it verifies a domain certificate.  To be on the wireless you have to be a member of the domain.

Seems to not give me any problems at the home office so any idea's?

On the 5508 I see this around the times I lose IP connectivity:

*Dot1x_NW_MsgTask_4: Jan 08 14:00:53.599: #DOT1X-3-INVALID_WPA_KEY_MSG_STATE: 1x_eapkey.c:861 Received invalid EAPOL-key M2 msg in START  state - invalid secure bit; KeyLen 24, Key type 1, client 88:53:2e:xx:xx:xx

*Dot1x_NW_MsgTask_4: Jan 08 14:00:52.551: #DOT1X-3-INVALID_WPA_KEY_MSG_STATE: 1x_eapkey.c:861 Received invalid EAPOL-key M2 msg in START  state - invalid secure bit; KeyLen 24, Key type 1, client 88:53:2e:xx:xx:xx

*Dot1x_NW_MsgTask_4: Jan 08 14:00:52.387: #DOT1X-3-ABORT_AUTH: 1x_bauth_sm.c:447  Authentication Aborted for client 88:53:2e:xx:xx:xx

I have 0 (unlimited) as the max for user login policies so not sure why Authentication Aborted message appears.

WLC Software version

7.4.100.0

On the NPS server (2008 R2) I just see my username granted access because it matches the network health policy.

Laptop Sony VAIO SE2

Intel Centrino Advanced-N 6230

Driver version 15.1.1.1 Date: 3/12/2012

[keithsauer507]

I updated Intel centrino 6230 driver to version 15.4.0.11 driver date 12/6/2012.  It still randomly disconnects.

I changed it to prefer 5.2 GHz band.  Good signal, 144mbps connection rate, but after an hour it disconnected me.  Not physical link layer (signal and association remained intact).  IP layer.  System stops forwarding / receiving IP communication until I manually disconnect and reconnect the network.

[Leo Laohoo]

Are you the only one having problems?   Have you tried other clients like another laptop, smartphone, tablet?

[keithsauer507]

I was the only one at that office yesterday with a laptop.  I do have an iPhone but that's not a good test because it always disconnects when it goes to sleep anyway. 

Anything in here worth changing on the WLC?

(Cisco Controller) >show advanced eap

EAP-Identity-Request Timeout (seconds)........... 30
EAP-Identity-Request Max Retries................. 2
EAP Key-Index for Dynamic WEP.................... 0
EAP Max-Login Ignore Identity Response........... enable
EAP-Request Timeout (seconds).................... 30
EAP-Request Max Retries.......................... 2
EAPOL-Key Timeout (milliseconds)................. 1000
EAPOL-Key Max Retries............................ 4
EAP-Broadcast Key Interval....................... 3600

[Leo Laohoo]

Use a different client to test.  Ask other people around.

[keithsauer507]

I visited another site today that has the same LAP's.  I had the same exact problem.  On the hour it seems the wireless, although connected with full signal strength - stopped passing IP traffic.  My IT coworker has a Dell 1397 wireless card and it does the same exact thing.

The people that work here did not seem to indicate there was an issue.  They have Dell advanced N 1030 and one has an Advanced N 6300 series.

The only difference between the people that work here and me and my coworkers is that our computer account in Active Directory is in a different OU (An OU representing our home office, and an OU under that representing the IT Area).

It almost seems as if there is an authenitcation issue.  I can auth correctly when I disconnect and then reconnect, but in about an hour it will just stop passing IP traffic without warning.

Any idea's where to look?  This just started happening with the latest WLC and LAP firmware update we performed back in December.  But it could also coincide with decommisioning of a 2003 IAS server and moving the WLC to authenticate to a 2008 R2 server running NAP.

[Leo Laohoo]

Here's one way to test if you have an authentication issue or not.  Create a hidden SSID with OPEN authentication.

If the connection is maintained, then you can look further.

Another way is to troll through your AD Event Logs.

[keithsauer507]

Well I'm going to try to move the EAP-Broadcast Key Interval back.  It was set to 3600 which in seconds equals 1 hour.  Seems like not only my laptop but others now have been reporting that every hour on the dot, they stop passing IP traffic.  The regular users just reboot, while people "in the know" disconnect and reconnect, and they are good for an hour.

Towards the bottom of this thread here:

https://discussions.apple.com/thread/3753111?start=0&tstart=0

They suggested this.

I ran this

(Cisco Controller) >config advanced eap bcast-key-interval 86400

Now when I show advanced eap I get this:

EAP-Identity-Request Timeout (seconds)........... 30

EAP-Identity-Request Max Retries................. 2

EAP Key-Index for Dynamic WEP.................... 0

EAP Max-Login Ignore Identity Response........... enable

EAP-Request Timeout (seconds).................... 30

EAP-Request Max Retries.......................... 2

EAPOL-Key Timeout (milliseconds)................. 1000

EAPOL-Key Max Retries............................ 4

EAP-Broadcast Key Interval....................... 86400

I just will take note if this fixes the problem or  not.  If it does not maybe I will return it back to the default 3600.

Your idea of a hidden SSID in the clear to rule out auth issues is a good one, but security wise I don't want an open SSID - hidden or not for an hour in public places.  Sure I could throw it on a VLAN with just one server and run a continuous ping... but I'm at the home office now which never has this issue.

Seems like when it's time for the key to be renewed, I'm thinking the renewal handshake at remote sites is just not making it back to the controller.  However the initial key handshake when you first boot up or associate to the SSID goes over the WAN no issue.  I only say this because at the home office where the WLC is physically located, there is no issue.

The WANs are a minimum of 10mbps over a Ethernet Virtual Private Line which is a busness level service provided by Verizon.

Some WLC info:

Product Version.................................. 7.4.100.0

Bootloader Version............................... 1.0.16

Field Recovery Image Version..................... 6.0.182.0

Firmware Version................................. FPGA 1.7, Env 1.8, USB console 2.2

Build Type....................................... DATA + WPS

[Leo Laohoo]

Your idea of a hidden SSID in the clear to rule out auth issues is a good one, but security wise I don't want an open SSID - hidden or not for an hour in public places.

You can still do this without other outside knowing.  You just crank down the power the antenna to "8" (lowest power).   This brings down the radius to a few metres.

[jmglass · Accepted solution]

Little late but fyi,

Open Caveats on 7.4.100.0, CSCuc78713 could be related to this;

CSCuc78713

Symptom: Wireless client cannot receive broadcast packets after broadcast key rotation.

Conditions: Dynamic WEP; 7.0.235.0, 7.2.110.0, and 7.3.101.0 controller software releases.

Workaround:

• Executes the config advanced eap bcast-key-interval 86400 in the middle of the night

• Change security setting to WPA2, and so on.

We set ours to max value since the 5.x code days, some clients drivers seemed to have problems and would disconnect on each rotation Did set it in the middle of the night so less impact when it rotated on the 24 hour.

jim

[Scott Fella]

What authentication are you using?  I see this issue on mac's running WLC code 7.4 and I have to disable session timeout for the mac's to be stable.  I'm using or testing WPA2/AES 802.1x

Thanks,

Scott

Help out other by using the rating system and marking answered questions as "Answered"

[keithsauer507]

We use WPA2+Enterprise.

RADIUS is 2008 R2 running NPS.

I changed it to 86400 during the day.  You think I should VPN at night and change it, and change it back?

I haven't heard any complaints since the change.

[Scott Fella]

Your using wpa2/AES only right? I would just leave it since that is the workaround to keep your devices stable.

Sent from Cisco Technical Support iPhone App

[Steven Williams]

Having similar issues on 2504 with code 7.6.130. Anyone have ideas?

[audiusinfo]

We had the same issue with vWLC 7.4.121.

Disabling TKIP on WPA and WPA2 auth helped to solve this issue.