WLC Applying cached RADIUS Override values for mobile

[kurosknight]

Hello!

We have a WiSM2 (version 7.4.110.0) with approx 200 APs. We are doing RADIUS authentication via a PacketFence backend. Everything usually works fine, but we are having an intermittent issue...

The WiSM2 gets its VLAN assignment for a client from the PacketFence server and does AAA override. If a client has not registered their device, go on one VLAN. Once they register, PacketFence disconnects them via RADIUS to the WiSM2, and then they should get their new VLAN assignment. This works fine in the majority of cases, but occasionally, after registering, the client disconnects and reconnects but is still put back on registration VLAN.

debug client mac shows this in the logs:

Applying cached RADIUS Override values for mobile 00:25:56:3d:f6:7b (caller pem_api.c:2210)

And I do not see the WiSM2 asking the PacketFence server for a VLAN assignment in the PacketFence logs.

Eventually, if the client stays disconnected long enough (5+ minutes), they can reconnect and get the proper VLAN assignment. I had previously opened a TAC about this, and they suggested a WiSM2 software upgrade and setting the Session Timeout on the WLAN to 900 seconds, which I did. This issue then disappeared for several weeks, but it has started happening again today (we saw it happen to about 15 clients throughout the day).

Anyone have any ideas on why this is happening, and how to stop the caching? Any thoughts would be greatly appreciated.

Here is the output from a show wlan of one of our WLANs we have seen this on:

WLAN Identifier.................................. 2

Profile Name..................................... BlitzNet

Network Name (SSID).............................. BlitzNet

Status........................................... Enabled

MAC Filtering.................................... Enabled

Broadcast SSID................................... Enabled

AAA Policy Override.............................. Enabled

Network Admission Control

  Client Profiling Status ....................... Disabled

   DHCP ......................................... Disabled

   HTTP ......................................... Disabled

  Radius-NAC State............................... Disabled

  SNMP-NAC State................................. Disabled

  Quarantine VLAN................................ 0

Maximum number of Associated Clients............. 0

Maximum number of Clients per AP Radio........... 200

Number of Active Clients......................... 538

Exclusionlist Timeout............................ 60 seconds

Session Timeout.................................. 900 seconds

User Idle Timeout................................ 300 seconds

User Idle Threshold.............................. 0 Bytes

NAS-identifier................................... WISM2_SDC

CHD per WLAN..................................... Enabled

Webauth DHCP exclusion........................... Disabled

Interface........................................ blitznet

Multicast Interface.............................. Not Configured

WLAN IPv4 ACL.................................... unconfigured

WLAN IPv6 ACL.................................... unconfigured

mDNS Status...................................... Disabled

mDNS Profile Name................................ unconfigured

DHCP Server...................................... Default

DHCP Address Assignment Required................. Disabled

Static IP client tunneling....................... Disabled

PMIPv6 Mobility Type............................. none

Quality of Service............................... Silver

Per-SSID Rate Limits............................. Upstream          Downstream

Average Data Rate................................   0                      0

Average Realtime Data Rate.......................   0                      0

Burst Data Rate..................................   0                      0

Burst Realtime Data Rate.........................   0                      0

Per-Client Rate Limits........................... Upstream          Downstream

Average Data Rate................................   0                      0

Average Realtime Data Rate.......................   0                      0

Burst Data Rate..................................   0                      0

Burst Realtime Data Rate.........................   0                      0

Scan Defer Priority.............................. 4,5,6

Scan Defer Time.................................. 100 milliseconds

WMM.............................................. Allowed

WMM UAPSD Compliant Client Support............... Disabled

Media Stream Multicast-direct.................... Disabled

CCX - AironetIe Support.......................... Enabled

CCX - Gratuitous ProbeResponse (GPR)............. Disabled

CCX - Diagnostics Channel Capability............. Disabled

Dot11-Phone Mode (7920).......................... Disabled

Wired Protocol................................... None

Passive Client Feature........................... Disabled

Peer-to-Peer Blocking Action..................... Drop

Radio Policy..................................... All

DTIM period for 802.11a radio.................... 1

DTIM period for 802.11b radio.................... 1

Radius Servers

   Authentication................................ ipofradiusserver 1812

   Accounting.................................... Global Servers

      Interim Update............................. Disabled

   Dynamic Interface............................. Disabled

   Dynamic Interface Priority.................... wlan

Local EAP Authentication......................... Disabled

Security

   802.11 Authentication:........................ Open System

   FT Support.................................... Disabled

   Static WEP Keys............................... Disabled

   802.1X........................................ Disabled

   Wi-Fi Protected Access (WPA/WPA2)............. Disabled

   WAPI.......................................... Disabled

   Wi-Fi Direct policy configured................ Disabled

   EAP-Passthrough............................... Disabled

   CKIP ......................................... Disabled

   Web Based Authentication...................... Disabled

   Web-Passthrough............................... Disabled

   Conditional Web Redirect...................... Disabled

   Splash-Page Web Redirect...................... Disabled

   Auto Anchor................................... Disabled

   FlexConnect Local Switching................... Disabled

   flexconnect Central Dhcp Flag................. Disabled

   flexconnect nat-pat Flag...................... Disabled

   flexconnect Dns Override Flag................. Disabled

   FlexConnect Vlan based Central Switching ..... Disabled

   FlexConnect Local Authentication.............. Disabled

   FlexConnect Learn IP Address.................. Disabled

   Client MFP.................................... Optional but inactive (WPA2 not configured)

   PMF........................................... Disabled

   PMF Association Comeback Time................. 1

   PMF SA Query RetryTimeout..................... 200

   Tkip MIC Countermeasure Hold-down Timer....... 60

AVC Visibilty.................................... Disabled

AVC Profile Name................................. None

Flow Monitor Name................................ None

Call Snooping.................................... Disabled

Roamed Call Re-Anchor Policy..................... Disabled

SIP CAC Fail Send-486-Busy Policy................ Enabled

SIP CAC Fail Send Dis-Association Policy......... Disabled

KTS based CAC Policy............................. Disabled

Assisted Roaming Prediction Optimization......... Disabled

802.11k Neighbor List............................ Disabled

802.11k Neighbor List Dual Band.................. Disabled

Band Select...................................... Disabled

Load Balancing................................... Disabled

Multicast Buffer................................. Disabled

Mobility Anchor List

WLAN ID     IP Address            Status

802.11u........................................ Disabled

MSAP Services.................................. Disabled

[Scott Fella]

What do you see in the radius logs? You see the radius sending the correct vlan? Also look at the client on the monitor tab and see what vlan it is placed on.

Sent from Cisco Technical Support iPhone App

[kurosknight]

There is nothing in the RADIUS server logs. It is as if the WiSM2 does not talk to it for the 2nd request. The flow for a problem client is like this:

1. New client associates

2. WiSM asks RADIUS server for VLAN

3. RADIUS Server hasn't seen it, so it puts it on VLAN 84 (our registration VLAN)

4. Client goes through captive portal

5. RADIUS server sends disconnect client message to WiSM

6. Client disconnects, reconnects

7. WiSM2 puts it back on VLAN 84, when it should put it on a VLAN determined by the SSID. The WiSM2 never asks the RADIUS server for the VLAN again, until the client has stayed disconnected for 5+ minutes, and I see the message in the wism2 log that I wrote above.

In the vast majority of cases, step 7 works properly. That is, when the client reconnects, it asks the RADIUS server what VLAN to put it on (I see it in the RADIUS server logs). I see the second request come in, and the RADIUS server replies with appropriate VLAN for the SSID.
After they get their proper VLAN, this doesn't occur again. It is as if the RADIUS server caches the client's VLAN override attribute somewhere and uses that, rather than asking the RADIUS server.

[Scott Fella]

Well if you look at the client in the monitor tab, the reason it's still keeps the same vlan and IP is that the client is still in a run state. It's different of they register on SSID X and them they have to join SSID Y. Since your using the same ssid, your client has to be removed from the WLC. The session timer is a hard timer and when that expires (900 seconds is 15 minutes) then they are removed from the WLC db. The idle timer is when the client sits idle and when that expires then the client is removed from the WLC. Certain devices work different because some will send a disconnect to the AP and some don't. So the best thing to do to test is to use a device (laptop and iPad) and remove the info on the radius or captive portal so the device looks like a brand new unregistered device. Take a look at the client information in the monitor tab in the WLC and you will see the client in the RUN state and it will also tell you what vlan and IP address the client has. Then go through your registration and see what happens when the client is disconnected. If you still see the client in the WLC, then that's why.

Sent from Cisco Technical Support iPhone App

[kurosknight]

Hi Scott,

You are correct, after disconnection the client is remaining in the RUN state for both RADIUS NAC State and Policy Manager State.

Other than lowering the idle timeout to something like 30 seconds, is there a way for it to not cache RADIUS NAC state?