802.1x Session Re-authentication timeout and DHCP

[kanansimpson]

Hello,

Has anyone experienced an issue with wireless client IP renewal on a 802.1x enabled WLAN/SSID when the Re-authentication timeout occurs?

Here is the issue..

I have a dot1x enabled WLAN. I have some wireless clients (a mixture/not the same) that will lose it's IP address after the Re-authentication timeout occurs. When this occurs, the client remains connected to the ap but will eventually show an APIPA address. I have enabled client debug on the the WLC and see that the client reauth logs after the timeout occurs. I know the reauth is fine (Client remains connected to ap). I've done several pcaps and it indicates that the DHCP server is receiving the Discover packet and replying with the Offer. However, the last place I see the offer packet is at the WLC up link port. From there, its not getting to the client to complete the process.

By default, the Re-authentication timeout is configured for 30 mins (or 1800 secs). As a work around, I've increase the Re-authentication timeout value to 12 hours. A 30 minute disconnect is not acceptable.

Has anyone experienced this issue or know anything about it?

Thanks Kindly.

[George Stefanick]

What is handling your DHCP. Ive seen issues with specific devices and getting DHCP. Recently a friend was troubleshooting a Jet and as it gets close to the hanger it would connect and push a download of data. The Jet wouldn't get an IP address. All the other devices would but the Jet. DHCP was served from a FW and once DHCP was handed out from a different box it would get and take a DHCP. 

Thats why I ask.

[kanansimpson]

We use InfoBlox DDI servers. We have a HA cluster and one single. I have tried adjusting the DHCP scope to only use 1 of the servers and that didnt improve anything. Same results.

[George Stefanick]

Can you for giggles set up a quick windows dhcp server and point a test interface to it ? I just pulled a working DHCP flow they are very similar .. Except for your client is continuing to ask for DHCP.. time and time again.

DHCP received op BOOTREQUEST

[kanansimpson]

I will try. Ill ask our server team to build one. I have a feeling the reason why its continuing to request and IP is because it's not getting the Offer from the server. For my experience with this, the client will keep send out DHCP Discover packets after the re auth (confirmed by wireshark). The switch port that the WLC uplinks to and the DHCP server both show the DHCP Discover and DHCP Offer packets in the pcap (same transID). However, the pcap on the client never recieves the Offer packet. All DHCP Discovers. Never in these pcaps trasactions, do I see a DHCP Request except when I disconnect and reconnect (forcing a new session) to the WLAN.

I know. Weird.  

The same servers are serving the rest of the networks (wired and old wireless) and no issues. I even untagged a port on the switch and plugged my laptop into it and never experienced an issue. 

I have a ticket open with TAC and still can't figure it out. 

What does this line mean? Do I just need to disable the session timer?

spamApTask2: Nov 10 12:07:26.077: [PA] a0:63:91:87:4c:90 Sent dot1x auth initiate message for mobile a0:63:91:87:4c:90
*Dot1x_NW_MsgTask_0: Nov 10 12:07:26.077: [PA] a0:63:91:87:4c:90 reauth_sm state transition 0 ---> 1 for mobile a0:63:91:87:4c:90 at 1x_reauth_sm.c:47
*Dot1x_NW_MsgTask_0: Nov 10 12:07:26.077: [PA] a0:63:91:87:4c:90 EAP-PARAM Debug - eap-params for Wlan-Id :1 is disabled - applying Global eap timers and retries
*Dot1x_NW_MsgTask_0: Nov 10 12:07:26.077: [PA] a0:63:91:87:4c:90 Disable re-auth, use PMK lifetime.

Thanks,

[George Stefanick]

In your capture I see DHCP offers. I also compared it to my DHCP debug here. Looks very similar expect for your client asking over and over.

Did you try rebooting the WLC?

So we don't use the session time ourselves. Even when you disable it I believe it set to 24 hours. While you think its disabled if a client was on for 24 hours it would get the boot. I know thats how it was in earlier code but that could have changed. 

Talk a little bit about the clients that are acting like this and the ones that aren't. Anything strike you odd?

[kanansimpson]

That's the weird part. Not all clients have the issue and they aare not all the same clients. I've seen it on a desktop using a netgear usb card, iphone, windows phone and game consoles. With that said, i guess its not just on the dot1x wlan. Thats been my primary focus as its used the heaviest. 

We are using AAA overirde. I'm going to see if client or system is trying to bounce it to another vlan. Thats a long shot.

I'm still not 100% sure its not the DHCP proxy. Something tells me that it could be that. I'm not sure though. Why would I see DHCP Offer packets coming back to the WLC and then it not reach the client. I know the packets are getting back to the WLC. Its a mystery between the WLC uplink and the client.

The way I understand it is that the DHCP proxy  uses the management or virtual interface to send out the DHCP packets and not even the interface the traffic generated on. For example, the dhcp discover is forward by another interface even through is generated on the vlan1010 test interface. Is this correct?

Thanks,

[George Stefanick · Accepted solution]

You have a backup controller maybe try and reproduce it on that one if you can't reboot production. Years ago it would use the management address it now and for a long time it's been sourced from the interface you put in the wlc if proxy is on. 

[kanansimpson]

Im not sure, the other day, I was tshooting an IP issues on another WLAN and I noticed source coming back from the DHCP was the virtual IP of the wlc. BTW, I have a  n+1 configuration. 1 HA pair and the n+1 wlc. All AP are on a single WLC.

[George Stefanick]

1.1.1.1 isn't routable. What you might be seeing is the dhcp mention of 1.1.1.1. But the actual interface where it leaves from is the controller interface for that wired side interface. It's a unicast frame. 

[kanansimpson]

No. What I meant was that the virtual interface is the one that does the DHCP proxy.

[kanansimpson]

Yes, We have two DHCP servers.

Interface Name................................... test-wifi-1010
MAC Address...................................... 10:05:ca:bf:a1:ad
IP Address....................................... <_WLC_VLAN_1010_Interface>
IP Netmask....................................... <VLAN_1010_Gateway>
IP Gateway....................................... <_Client_VLAN_1010_Gateway>
External NAT IP State............................ Disabled
External NAT IP Address.......................... 0.0.0.0
VLAN............................................. 1010
Quarantine-vlan.................................. 0
NAS-Identifier................................... CiscoWLC
Active Physical Port............................. LAG (8)
Primary Physical Port............................ LAG (8)
Backup Physical Port............................. Unconfigured
DHCP Proxy Mode.................................. Disabled
Primary DHCP Server.............................. <DHCP_SVR_1>
Secondary DHCP Server............................ <DHCP_SVR_2>
DHCP Option 82................................... Disabled
DHCP Option 82 bridge mode insertion............. Disabled
IPv4 ACL......................................... Unconfigured
mDNS Profile Name................................ Unconfigured
AP Manager....................................... No
Guest Interface.................................. N/A
3G VLAN.......................................... Disabled
L2 Multicast..................................... Enabled

Thanks,

[George Stefanick]

Do a show run of the SVI interface for this VLAN, 1010. 

[kanansimpson]

interface Vlan1010
vrf member hallnet
no ip redirects
ip address <ip addr2>
no ipv6 redirects
ip router ospf 1 area 0.0.0.0
hsrp version 2
hsrp 1010
priority 255
ip <ip_addr1>
ip dhcp relay address <DHCP_SVR_1>
ip dhcp relay address <DHCP_SVR_2>
description WLCStudentNet
no shutdown

Both IPs are defined as relays/helpers.

[George Stefanick]

Whats handing out your DHCP .. ?

[kanansimpson]

Here is the output for the WLAN.

Thanks