ARP not resolved while roaming

[O_A_H]

We have Cisco 3500 WLC with 2800 APs (8.10.196.0). APs operate in Local mode, so, SSID traffic is central switched. We have a roaming problem. We used 802.1x-FT. With each roaming event, the client will do ARP to the gateway after it finishes. The problem happens is that most of the times, the first ARP packet is not seen in the AP debugs nor on the AP SPAN packet capture, so, the clients waits 300-500ms and try another ARP which will make it's way to the WLC via CAPWAP but the WLC drops this packet and doesn't forward it to the core (GW), then the client waits again for 300-500ms and send 3rd ARP packet which makes it's way till the core and gets a reply. The client will not pass traffic till the GW ARP is resolved, which accordingly in this situation causes video call freeze for about 0.5 - 1 second (This can be seen from the packet capture on the laptop where the client only receives video traffic and not sending till ARP is resolved). If we assume that the first ARP packet is lost in the air, i find that strange that only first ARP is lost. What could be the issue with the first ARP packet? Would it be a bug (what is the bug ID)? We tested on PSK SSID and the ARP behavior was the same.
To overcome the WLC dropping the second ARP, I have converted APs to flexconnect and made the SSID local switching so that ARP will be sent from client->AP->core (bypass WLC). This enhanced the video call experience but still some freezes happens due to the 1st ARP packet that is being lost. I have tested to add static ARP entry of the GW on the client (to avoid sending GW ARP after each roaming event) but that didn't seem to have an effect and the client still do GW ARP after each roaming event. In a separate floor where we have only 9120 APs (not 2800), the same ARP behavior was seen.

Another side of the story is that, after the client does successful FT roaming, it will send EAPOL START message to start full reauthentication. Then, full reauthentication happens followed by DHCP and ARP (ARP issue is as stated above). From the laptop packet capture, we can confirm that the full reauthentication behaviour doesn't interrupt 2-way video traffic forwarding during that... it's only ARP (that happens in the end) will interrupt till it's resolved. Why this EAPOL START behavior happens? What is noticed that EAPOL START behaviour happens on laptops that use client certificate. But on clients with machine certificate, they don't do EAPOL START not do full reauthentication (Just FT quick roaming). All laptops are managed by Intune.

At this stage, i'm quite stuck tshooting this issue. The next action point probably would be to create open SSID and do OTA with packet capture on laptop and AP port to see what happens with the first ARP packet... but still as mentioned, it doesn't make sense that most of the time it's lost on Air (RF issue) that only impacts the 1st ARP packet and nothing else from the rest of the traffic.

[Leo Laohoo]

Does rebooting the APs help in any matter?

[O_A_H]

No unfortunately.. Neither rebooting WLC. This was tested while upgrading to 8.10.196.0 (in the process of tshooting) and also APs rebooting when converting from local to flexconnect.

[Rich R]

Good luck with that <smile>
There were a lot of similar problems on those APs (and in common with 9120 they have Broadcom chipset).
The problems were mostly fixed in https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwa73245 but it wouldn't surprise me if they missed off corner cases.  Also see Leo's list of bugs affecting 2800/3800/4800/1560 APs

As AireOS is end of life now there is zero chance of getting that fixed now.  You can look through all those bugs and try some of the workarounds suggested to see if they make any difference.  If not then your only option is to try upgrading to 9800 WLC.  The AP code is largely similar (but with additional fixes) and the WLC code is mostly new (and yes new bugs too) but at least if you still find the same problem in the latest code then you can at least open a TAC case and hopefully get it fixed.

Also, and this might solve your problem with the 2nd lost ARP on 9800 regardless of other bugs, at least for central switching: ARP proxy:
https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/technical-reference/c9800-best-practices.html#AddressResolutionProtocolARPproxy

[O_A_H]

Thanks for the response and the help.
For now, we decided to stop going further, and do a migration to 9800 later this year. The 50% roaming improvement brought by the flexconnect local switching change is a bit acceptable for the time being.

[MHM Cisco World]

If roaming is good then client no need to ask IP.

I will check my points and update you

MHM

[O_A_H]

Yes i assume so, but i have seen in packet capture (in proper good roaming clients) that still there are 2 packets in DHCP are done to ack the IP. I didn't expect that but it seems it is client side behaviour, and can be different amongst different clients.

[MHM Cisco World]

Let me check some command usful for this case.

It seem to me client do l3 roaming not l2 roaming.

But let me check first my note 

MHM

[MHM Cisco World]

show client detail <mac of any clinet face issue> <<- share this 

MHM

[O_A_H]

Thanks @MHM Cisco World for your attention.. as i explained above that we decided to stop going further with this tshooting, and do a migration to 9800 later this year. The 50% roaming improvement brought by the flexconnect local switching change is a bit acceptable for the time being.

However, if you mean about the DHCP ack packets during roaming, it would be interesting to figure that out.. here you are:

(WLC) >show client detail <omitted>
Client MAC Address............................... <omitted>
Client Username ................................. <omitted>
Client Webauth Username ......................... N/A
Hostname: ....................................... <omitted>
Device Type: .................................... Microsoft-Workstation
AP MAC Address................................... <omitted>
AP Name.......................................... <omitted>
AP radio slot Id................................. 1
Client State..................................... Associated
User Authenticated by ........................... RADIUS Server
Client User Group................................ <omitted>
Client NAC OOB State............................. Access
Wireless LAN Id.................................. 1
Wireless LAN Network Name (SSID)................. <omitted>
Wireless LAN Profile Name........................ <omitted>
WLAN Profile check for roaming................... Disabled
Hotspot (802.11u)................................ Not Supported
Connected For ................................... 8 secs
BSSID............................................ <omitted>
Channel.......................................... 52
IP Address....................................... <omitted>
Gateway Address.................................. <omitted>
Netmask.......................................... <omitted>
IPv6 Address..................................... <omitted>
Association Id................................... 31
Authentication Algorithm......................... Open System
Reason Code...................................... 1
Client IPSK-TAG.................................. N/A
Status Code...................................... 0
Client CCX version............................... No CCX support
Re-Authentication Timeout........................ 28397
QoS Level........................................ Gold
Avg data Rate.................................... 0
Burst data Rate.................................. 0
Avg Real time data Rate.......................... 0
Burst Real Time data Rate........................ 0
Avg Uplink data Rate............................. 0
Burst Uplink data Rate........................... 0
Avg Uplink Real time data Rate................... 0
Burst Uplink Real Time data Rate................. 0
802.1P Priority Tag.............................. 4
Security Group Tag............................... Unknown(0)
KTS CAC Capability............................... No
Qos Map Capability............................... Yes
WMM Support...................................... Enabled
APSD ACs....................................... BK BE VI VO
Current Rate..................................... 12.0
Supported Rates.................................. 6.0,12.0,18.0,24.0,36.0,48.0,
............................................. 54.0
Mobility State................................... Local
Mobility Move Count.............................. 0
Security Policy Completed........................ Yes
Policy Manager State............................. RUN
Pre-auth IPv4 ACL Name........................... none
Pre-auth IPv4 ACL Applied Status................. Unavailable
Pre-auth IPv6 ACL Name........................... none
Pre-auth IPv6 ACL Applied Status................. Unavailable
Pre-auth Flex IPv4 ACL Name...................... none
Pre-auth Flex IPv4 ACL Applied Status............ Unavailable
Pre-auth Flex IPv6 ACL Name...................... none
Pre-auth Flex IPv6 ACL Applied Status............ Unavailable
Pre-auth redirect URL............................ none
Audit Session ID................................. <omitted>
AAA Role Type.................................... none
Acct Interim Interval............................ 0
Local Policy Applied............................. none
IPv4 ACL Name.................................... none
AAA FlexConnect ACL Applied Status............... Unavailable
IPv4 ACL Applied Status.......................... Unavailable
IPv6 ACL Name.................................... none
IPv6 ACL Applied Status.......................... Unavailable
Post-auth Flex IPv6 ACL Name..................... none
Post-auth Flex IPv6 ACL Applied Status........... Unavailable
Layer2 ACL Name.................................. none
Layer2 ACL Applied Status........................ Unavailable
URL ACL Name..................................... none
URL ACL Applied Status........................... Unavailable
Client Type...................................... SimpleIP
mDNS Status...................................... Disabled
mDNS Profile Name................................ none
No. of mDNS Services Advertised.................. 0
Policy Type...................................... WPA2
Authentication Key Management.................... FT-802.1x
Encryption Cipher................................ CCMP-128 (AES)
Protected Management Frame ...................... No
Management Frame Protection...................... No
EAP Type......................................... EAP-TLS
Interface........................................ clients
VLAN............................................. 300
Quarantine VLAN.................................. 0
Access VLAN...................................... 300
Local Bridging VLAN.............................. 300
Client Capabilities:
Radio Capability........................... 802.11ax
CF Pollable................................ Not implemented
CF Poll Request............................ Not implemented
Short Preamble............................. Not implemented
PBCC....................................... Not implemented
Channel Agility............................ Not implemented
Listen Interval............................ 250
Fast BSS Transition........................ Implemented
11v BSS Transition......................... Implemented
Non-Operable Channels............................ None
Non-Prefer Channels.............................. None
Client Wifi Direct Capabilities:
WFD capable................................ No
Manged WFD capable......................... No
Cross Connection Capable................... No
Support Concurrent Operation............... No
Fast BSS Transition Details:
Reassociation Timeout...................... 20
Total Preauth APs.......................... 1
Preauth AP................................. <omitted>
Total SuccessFull Roam..................... 1
DNS Server details:
DNS server IP ............................. <omitted>
DNS server IP ............................. <omitted>
Assisted Roaming Prediction List details:

Client Dhcp Required: False
Allowed (URL)IP Addresses
-------------------------

AVC Profile Name: ............................... none
OpenDns Profile Name: ........................... none
Fastlane Client: ................................ No
Max DSCP: ....................................... 34
Nas Identifier: ................................. <omitted>
Fabric Statistics
--------------------

Client Statistics:
Number of Bytes Received................... 8569395
Number of Bytes Sent....................... 15227408
Total Number of Bytes Sent................. 15227408
Total Number of Bytes Recv................. 8569395
Number of Bytes Sent (last 90s)............ 6820498
Number of Bytes Recv (last 90s)............ 2511409
Number of Packets Received................. 20085
Number of Packets Sent..................... 26837
Number of Interim-Update Sent.............. 0
Number of EAP Id Request Msg Timeouts...... 0
Number of EAP Id Request Msg Failures...... 0
Number of EAP Request Msg Timeouts......... 0
Number of EAP Request Msg Failures......... 0
Number of EAP Key Msg Timeouts............. 0
Number of EAP Key Msg Failures............. 0
Number of Data Retries..................... 2733
Number of RTS Retries...................... 0
Number of Duplicate Received Packets....... 0
Number of Decrypt Failed Packets........... 0
Number of Mic Failured Packets............. 0
Number of Mic Missing Packets.............. 0
Number of RA Packets Dropped............... 0
Number of Policy Errors.................... 0
Radio Signal Strength Indicator............ -49 dBm
Signal to Noise Ratio...................... 48 dB
Client Detected as Inactive................ No
Client RBACL Statistics:
Number of RBACL Allowed Packets............ 0
Number of RBACL Denied Packets............. 0
Client Rate Limiting Statistics:
Number of Data Packets Received............ 0
Number of Data Rx Packets Dropped.......... 0
Number of Data Bytes Received.............. 0
Number of Data Rx Bytes Dropped............ 0
Number of Realtime Packets Received........ 0
Number of Realtime Rx Packets Dropped...... 0
Number of Realtime Bytes Received.......... 0
Number of Realtime Rx Bytes Dropped........ 0
Number of Data Packets Sent................ 0
Number of Data Tx Packets Dropped.......... 0
Number of Data Bytes Sent.................. 0
Number of Data Tx Bytes Dropped............ 0
Number of Realtime Packets Sent............ 0
Number of Realtime Tx Packets Dropped...... 0
Number of Realtime Bytes Sent.............. 0
Number of Realtime Tx Bytes Dropped........ 0
Nearby AP Statistics:
AEBWAP010(slot 1)
antenna0: 8 secs ago..................... -90 dBm
antenna1: 8 secs ago..................... -90 dBm
AEBWAP205(slot 1)
antenna0: 8 secs ago..................... -78 dBm
antenna1: 8 secs ago..................... -78 dBm
AEBWAP213(slot 1)
antenna0: 8 secs ago..................... -65 dBm
antenna1: 8 secs ago..................... -65 dBm
AEBWAP206(slot 0)
antenna0: 8 secs ago..................... -63 dBm
antenna1: 8 secs ago..................... -63 dBm
AEBWAP206(slot 1)
antenna0: 8 secs ago..................... -66 dBm
antenna1: 8 secs ago..................... -66 dBm
AEBWAP104(slot 1)
antenna0: 8 secs ago..................... -89 dBm
antenna1: 8 secs ago..................... -89 dBm
AEBWAP105(slot 0)
antenna0: 8 secs ago..................... -73 dBm
antenna1: 8 secs ago..................... -73 dBm
AEBWAP021(slot 1)
antenna0: 8 secs ago..................... -91 dBm
antenna1: 8 secs ago..................... -91 dBm
AEBWAP108(slot 0)
antenna0: 8 secs ago..................... -89 dBm
antenna1: 8 secs ago..................... -89 dBm
AEBWAP108(slot 1)
antenna0: 8 secs ago..................... -85 dBm
antenna1: 8 secs ago..................... -85 dBm
AEBWAP214(slot 1)
antenna0: 8 secs ago..................... -48 dBm
antenna1: 8 secs ago..................... -48 dBm
AEBWAP020(slot 0)
antenna0: 8 secs ago..................... -84 dBm
antenna1: 8 secs ago..................... -84 dBm
AEBWAP020(slot 1)
antenna0: 8 secs ago..................... -79 dBm
antenna1: 8 secs ago..................... -79 dBm
AEBWAP313(slot 1)
antenna0: 8 secs ago..................... -83 dBm
antenna1: 8 secs ago..................... -83 dBm
AEBWAP208(slot 1)
antenna0: 8 secs ago..................... -68 dBm
antenna1: 8 secs ago..................... -68 dBm
AEBWAP102(slot 0)
antenna0: 8 secs ago..................... -83 dBm
antenna1: 8 secs ago..................... -83 dBm
AEBWAP102(slot 1)
antenna0: 8 secs ago..................... -83 dBm
antenna1: 8 secs ago..................... -83 dBm
AEBWAP101(slot 0)
antenna0: 8 secs ago..................... -65 dBm
antenna1: 8 secs ago..................... -65 dBm
AEBWAP101(slot 1)
antenna0: 8 secs ago..................... -59 dBm
antenna1: 8 secs ago..................... -59 dBm
AEBWAP210(slot 1)
antenna0: 8 secs ago..................... -88 dBm
antenna1: 8 secs ago..................... -88 dBm
AEBWAP111(slot 1)
antenna0: 8 secs ago..................... -84 dBm
antenna1: 8 secs ago..................... -84 dBm
AEBWAP209(slot 1)
antenna0: 8 secs ago..................... -54 dBm
antenna1: 8 secs ago..................... -54 dBm
AEBWAP207(slot 0)
antenna0: 7 secs ago..................... -82 dBm
antenna1: 7 secs ago..................... -82 dBm
AEBWAP207(slot 1)
antenna0: 7 secs ago..................... -87 dBm
antenna1: 7 secs ago..................... -87 dBm

DHCP Server IP Address: ....................... <omitted>
Discover-offer time: 0

Request-ack time: 0

DHCP Server IP Address: ....................... <omitted>
Discover-offer time: 0

Request-ack time: 2013873

[MHM Cisco World]

Thanks' are you sure this client roaming?

I dont see last AP and new AP ?

Check client with issue' see AP name is it connect to same wlc or not

MHM

[O_A_H]

Yes sure.. you will not see the last AP in the output of this command.. this command only shows the current status of the connected client.

Parsing the debug file shows this event sequence. As you see, after successful FT roaming event, the client does DHCP

[MHM Cisco World]

Ok' 

Now all device effect by this ?

old abd new AP use same vlan for this ssid?

MHM

[MHM Cisco World]

https://medium.com/@wirelesslab.io/cisco-9800-wlc-client-disconnections-6850276b7e38

Check this article about dhcp request after roaming abd device that not support FT and run slow roaming.

[O_A_H]

Yes. Yes.
Thanks for sharing the article, but in this case the client does FT properly and does fast roam, then followed by DHCP. I didn't see in the article a point that would ring a bell in this case.