topic Re: ARP not resolved while roaming in Wireless

ARP not resolved while roaming

O_A_H — Mon, 15 Sep 2025 13:56:07 GMT

We have Cisco 3500 WLC with 2800 APs (8.10.196.0). APs operate in Local mode, so, SSID traffic is central switched. We have a roaming problem. We used 802.1x-FT. With each roaming event, the client will do ARP to the gateway after it finishes. The problem happens is that most of the times, the first ARP packet is not seen in the AP debugs nor on the AP SPAN packet capture, so, the clients waits 300-500ms and try another ARP which will make it's way to the WLC via CAPWAP but the WLC drops this packet and doesn't forward it to the core (GW), then the client waits again for 300-500ms and send 3rd ARP packet which makes it's way till the core and gets a reply. The client will not pass traffic till the GW ARP is resolved, which accordingly in this situation causes video call freeze for about 0.5 - 1 second (This can be seen from the packet capture on the laptop where the client only receives video traffic and not sending till ARP is resolved). If we assume that the first ARP packet is lost in the air, i find that strange that only first ARP is lost. What could be the issue with the first ARP packet? Would it be a bug (what is the bug ID)? We tested on PSK SSID and the ARP behavior was the same.
To overcome the WLC dropping the second ARP, I have converted APs to flexconnect and made the SSID local switching so that ARP will be sent from client->AP->core (bypass WLC). This enhanced the video call experience but still some freezes happens due to the 1st ARP packet that is being lost. I have tested to add static ARP entry of the GW on the client (to avoid sending GW ARP after each roaming event) but that didn't seem to have an effect and the client still do GW ARP after each roaming event. In a separate floor where we have only 9120 APs (not 2800), the same ARP behavior was seen.

Another side of the story is that, after the client does successful FT roaming, it will send EAPOL START message to start full reauthentication. Then, full reauthentication happens followed by DHCP and ARP (ARP issue is as stated above). From the laptop packet capture, we can confirm that the full reauthentication behaviour doesn't interrupt 2-way video traffic forwarding during that... it's only ARP (that happens in the end) will interrupt till it's resolved. Why this EAPOL START behavior happens? What is noticed that EAPOL START behaviour happens on laptops that use client certificate. But on clients with machine certificate, they don't do EAPOL START not do full reauthentication (Just FT quick roaming). All laptops are managed by Intune.

At this stage, i'm quite stuck tshooting this issue. The next action point probably would be to create open SSID and do OTA with packet capture on laptop and AP port to see what happens with the first ARP packet... but still as mentioned, it doesn't make sense that most of the time it's lost on Air (RF issue) that only impacts the 1st ARP packet and nothing else from the rest of the traffic.

15-09-2025: Update

We have migrated the APs to 9800-CL and also in Flexconnect local switching. And the issue persists with 1st ARP being lost.

I could see the same behaviour with each roaming event.

1- After roaming, the client will do ARP to the GW, and at this stage, it keeps receiving traffic from the sender, but sending nothing.

2- The 1st ARP packet most of the times doesn't get a response (90%). Then the client waits about 500ms before sending another ARP to the GW. Then the GW responses to the 2nd ARP, and then the client starts sending traffic again.

Re: ARP not resolved while roaming

Leo Laohoo — Thu, 12 Jun 2025 09:42:19 GMT

Does rebooting the APs help in any matter?

Re: ARP not resolved while roaming

O_A_H — Thu, 12 Jun 2025 09:44:41 GMT

No unfortunately.. Neither rebooting WLC. This was tested while upgrading to 8.10.196.0 (in the process of tshooting) and also APs rebooting when converting from local to flexconnect.

Re: ARP not resolved while roaming

Rich R — Sun, 15 Jun 2025 12:28:51 GMT

Good luck with that <smile>
There were a lot of similar problems on those APs (and in common with 9120 they have Broadcom chipset).
The problems were mostly fixed in https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwa73245 but it wouldn't surprise me if they missed off corner cases. Also see Leo's list of bugs affecting 2800/3800/4800/1560 APs

As AireOS is end of life now there is zero chance of getting that fixed now. You can look through all those bugs and try some of the workarounds suggested to see if they make any difference. If not then your only option is to try upgrading to 9800 WLC. The AP code is largely similar (but with additional fixes) and the WLC code is mostly new (and yes new bugs too) but at least if you still find the same problem in the latest code then you can at least open a TAC case and hopefully get it fixed.

Also, and this might solve your problem with the 2nd lost ARP on 9800 regardless of other bugs, at least for central switching: ARP proxy:
https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/technical-reference/c9800-best-practices.html#AddressResolutionProtocolARPproxy

Re: ARP not resolved while roaming

MHM Cisco World — Sun, 15 Jun 2025 13:45:21 GMT

If roaming is good then client no need to ask IP.

I will check my points and update you

MHM

Re: ARP not resolved while roaming

O_A_H — Mon, 04 Aug 2025 14:42:49 GMT

Thanks for the response and the help.
For now, we decided to stop going further, and do a migration to 9800 later this year. The 50% roaming improvement brought by the flexconnect local switching change is a bit acceptable for the time being.

Re: ARP not resolved while roaming

O_A_H — Mon, 04 Aug 2025 14:44:20 GMT

Yes i assume so, but i have seen in packet capture (in proper good roaming clients) that still there are 2 packets in DHCP are done to ack the IP. I didn't expect that but it seems it is client side behaviour, and can be different amongst different clients.

Re: ARP not resolved while roaming

MHM Cisco World — Mon, 04 Aug 2025 14:54:40 GMT

Let me check some command usful for this case.

It seem to me client do l3 roaming not l2 roaming.

But let me check first my note

MHM

Re: ARP not resolved while roaming

MHM Cisco World — Mon, 04 Aug 2025 15:09:05 GMT

show client detail <mac of any clinet face issue> <<- share this

MHM

Re: ARP not resolved while roaming

O_A_H — Tue, 05 Aug 2025 09:44:24 GMT

Thanks @MHM Cisco World for your attention.. as i explained above that we decided to stop going further with this tshooting, and do a migration to 9800 later this year. The 50% roaming improvement brought by the flexconnect local switching change is a bit acceptable for the time being.

However, if you mean about the DHCP ack packets during roaming, it would be interesting to figure that out.. here you are:

(WLC) >show client detail <omitted>
Client MAC Address............................... <omitted>
Client Username ................................. <omitted>
Client Webauth Username ......................... N/A
Hostname: ....................................... <omitted>
Device Type: .................................... Microsoft-Workstation
AP MAC Address................................... <omitted>
AP Name.......................................... <omitted>
AP radio slot Id................................. 1
Client State..................................... Associated
User Authenticated by ........................... RADIUS Server
Client User Group................................ <omitted>
Client NAC OOB State............................. Access
Wireless LAN Id.................................. 1
Wireless LAN Network Name (SSID)................. <omitted>
Wireless LAN Profile Name........................ <omitted>
WLAN Profile check for roaming................... Disabled
Hotspot (802.11u)................................ Not Supported
Connected For ................................... 8 secs
BSSID............................................ <omitted>
Channel.......................................... 52
IP Address....................................... <omitted>
Gateway Address.................................. <omitted>
Netmask.......................................... <omitted>
IPv6 Address..................................... <omitted>
Association Id................................... 31
Authentication Algorithm......................... Open System
Reason Code...................................... 1
Client IPSK-TAG.................................. N/A
Status Code...................................... 0
Client CCX version............................... No CCX support
Re-Authentication Timeout........................ 28397
QoS Level........................................ Gold
Avg data Rate.................................... 0
Burst data Rate.................................. 0
Avg Real time data Rate.......................... 0
Burst Real Time data Rate........................ 0
Avg Uplink data Rate............................. 0
Burst Uplink data Rate........................... 0
Avg Uplink Real time data Rate................... 0
Burst Uplink Real Time data Rate................. 0
802.1P Priority Tag.............................. 4
Security Group Tag............................... Unknown(0)
KTS CAC Capability............................... No
Qos Map Capability............................... Yes
WMM Support...................................... Enabled
APSD ACs....................................... BK BE VI VO
Current Rate..................................... 12.0
Supported Rates.................................. 6.0,12.0,18.0,24.0,36.0,48.0,
............................................. 54.0
Mobility State................................... Local
Mobility Move Count.............................. 0
Security Policy Completed........................ Yes
Policy Manager State............................. RUN
Pre-auth IPv4 ACL Name........................... none
Pre-auth IPv4 ACL Applied Status................. Unavailable
Pre-auth IPv6 ACL Name........................... none
Pre-auth IPv6 ACL Applied Status................. Unavailable
Pre-auth Flex IPv4 ACL Name...................... none
Pre-auth Flex IPv4 ACL Applied Status............ Unavailable
Pre-auth Flex IPv6 ACL Name...................... none
Pre-auth Flex IPv6 ACL Applied Status............ Unavailable
Pre-auth redirect URL............................ none
Audit Session ID................................. <omitted>
AAA Role Type.................................... none
Acct Interim Interval............................ 0
Local Policy Applied............................. none
IPv4 ACL Name.................................... none
AAA FlexConnect ACL Applied Status............... Unavailable
IPv4 ACL Applied Status.......................... Unavailable
IPv6 ACL Name.................................... none
IPv6 ACL Applied Status.......................... Unavailable
Post-auth Flex IPv6 ACL Name..................... none
Post-auth Flex IPv6 ACL Applied Status........... Unavailable
Layer2 ACL Name.................................. none
Layer2 ACL Applied Status........................ Unavailable
URL ACL Name..................................... none
URL ACL Applied Status........................... Unavailable
Client Type...................................... SimpleIP
mDNS Status...................................... Disabled
mDNS Profile Name................................ none
No. of mDNS Services Advertised.................. 0
Policy Type...................................... WPA2
Authentication Key Management.................... FT-802.1x
Encryption Cipher................................ CCMP-128 (AES)
Protected Management Frame ...................... No
Management Frame Protection...................... No
EAP Type......................................... EAP-TLS
Interface........................................ clients
VLAN............................................. 300
Quarantine VLAN.................................. 0
Access VLAN...................................... 300
Local Bridging VLAN.............................. 300
Client Capabilities:
Radio Capability........................... 802.11ax
CF Pollable................................ Not implemented
CF Poll Request............................ Not implemented
Short Preamble............................. Not implemented
PBCC....................................... Not implemented
Channel Agility............................ Not implemented
Listen Interval............................ 250
Fast BSS Transition........................ Implemented
11v BSS Transition......................... Implemented
Non-Operable Channels............................ None
Non-Prefer Channels.............................. None
Client Wifi Direct Capabilities:
WFD capable................................ No
Manged WFD capable......................... No
Cross Connection Capable................... No
Support Concurrent Operation............... No
Fast BSS Transition Details:
Reassociation Timeout...................... 20
Total Preauth APs.......................... 1
Preauth AP................................. <omitted>
Total SuccessFull Roam..................... 1
DNS Server details:
DNS server IP ............................. <omitted>
DNS server IP ............................. <omitted>
Assisted Roaming Prediction List details:

Client Dhcp Required: False
Allowed (URL)IP Addresses
-------------------------

AVC Profile Name: ............................... none
OpenDns Profile Name: ........................... none
Fastlane Client: ................................ No
Max DSCP: ....................................... 34
Nas Identifier: ................................. <omitted>
Fabric Statistics
--------------------

Client Statistics:
Number of Bytes Received................... 8569395
Number of Bytes Sent....................... 15227408
Total Number of Bytes Sent................. 15227408
Total Number of Bytes Recv................. 8569395
Number of Bytes Sent (last 90s)............ 6820498
Number of Bytes Recv (last 90s)............ 2511409
Number of Packets Received................. 20085
Number of Packets Sent..................... 26837
Number of Interim-Update Sent.............. 0
Number of EAP Id Request Msg Timeouts...... 0
Number of EAP Id Request Msg Failures...... 0
Number of EAP Request Msg Timeouts......... 0
Number of EAP Request Msg Failures......... 0
Number of EAP Key Msg Timeouts............. 0
Number of EAP Key Msg Failures............. 0
Number of Data Retries..................... 2733
Number of RTS Retries...................... 0
Number of Duplicate Received Packets....... 0
Number of Decrypt Failed Packets........... 0
Number of Mic Failured Packets............. 0
Number of Mic Missing Packets.............. 0
Number of RA Packets Dropped............... 0
Number of Policy Errors.................... 0
Radio Signal Strength Indicator............ -49 dBm
Signal to Noise Ratio...................... 48 dB
Client Detected as Inactive................ No
Client RBACL Statistics:
Number of RBACL Allowed Packets............ 0
Number of RBACL Denied Packets............. 0
Client Rate Limiting Statistics:
Number of Data Packets Received............ 0
Number of Data Rx Packets Dropped.......... 0
Number of Data Bytes Received.............. 0
Number of Data Rx Bytes Dropped............ 0
Number of Realtime Packets Received........ 0
Number of Realtime Rx Packets Dropped...... 0
Number of Realtime Bytes Received.......... 0
Number of Realtime Rx Bytes Dropped........ 0
Number of Data Packets Sent................ 0
Number of Data Tx Packets Dropped.......... 0
Number of Data Bytes Sent.................. 0
Number of Data Tx Bytes Dropped............ 0
Number of Realtime Packets Sent............ 0
Number of Realtime Tx Packets Dropped...... 0
Number of Realtime Bytes Sent.............. 0
Number of Realtime Tx Bytes Dropped........ 0
Nearby AP Statistics:
AEBWAP010(slot 1)
antenna0: 8 secs ago..................... -90 dBm
antenna1: 8 secs ago..................... -90 dBm
AEBWAP205(slot 1)
antenna0: 8 secs ago..................... -78 dBm
antenna1: 8 secs ago..................... -78 dBm
AEBWAP213(slot 1)
antenna0: 8 secs ago..................... -65 dBm
antenna1: 8 secs ago..................... -65 dBm
AEBWAP206(slot 0)
antenna0: 8 secs ago..................... -63 dBm
antenna1: 8 secs ago..................... -63 dBm
AEBWAP206(slot 1)
antenna0: 8 secs ago..................... -66 dBm
antenna1: 8 secs ago..................... -66 dBm
AEBWAP104(slot 1)
antenna0: 8 secs ago..................... -89 dBm
antenna1: 8 secs ago..................... -89 dBm
AEBWAP105(slot 0)
antenna0: 8 secs ago..................... -73 dBm
antenna1: 8 secs ago..................... -73 dBm
AEBWAP021(slot 1)
antenna0: 8 secs ago..................... -91 dBm
antenna1: 8 secs ago..................... -91 dBm
AEBWAP108(slot 0)
antenna0: 8 secs ago..................... -89 dBm
antenna1: 8 secs ago..................... -89 dBm
AEBWAP108(slot 1)
antenna0: 8 secs ago..................... -85 dBm
antenna1: 8 secs ago..................... -85 dBm
AEBWAP214(slot 1)
antenna0: 8 secs ago..................... -48 dBm
antenna1: 8 secs ago..................... -48 dBm
AEBWAP020(slot 0)
antenna0: 8 secs ago..................... -84 dBm
antenna1: 8 secs ago..................... -84 dBm
AEBWAP020(slot 1)
antenna0: 8 secs ago..................... -79 dBm
antenna1: 8 secs ago..................... -79 dBm
AEBWAP313(slot 1)
antenna0: 8 secs ago..................... -83 dBm
antenna1: 8 secs ago..................... -83 dBm
AEBWAP208(slot 1)
antenna0: 8 secs ago..................... -68 dBm
antenna1: 8 secs ago..................... -68 dBm
AEBWAP102(slot 0)
antenna0: 8 secs ago..................... -83 dBm
antenna1: 8 secs ago..................... -83 dBm
AEBWAP102(slot 1)
antenna0: 8 secs ago..................... -83 dBm
antenna1: 8 secs ago..................... -83 dBm
AEBWAP101(slot 0)
antenna0: 8 secs ago..................... -65 dBm
antenna1: 8 secs ago..................... -65 dBm
AEBWAP101(slot 1)
antenna0: 8 secs ago..................... -59 dBm
antenna1: 8 secs ago..................... -59 dBm
AEBWAP210(slot 1)
antenna0: 8 secs ago..................... -88 dBm
antenna1: 8 secs ago..................... -88 dBm
AEBWAP111(slot 1)
antenna0: 8 secs ago..................... -84 dBm
antenna1: 8 secs ago..................... -84 dBm
AEBWAP209(slot 1)
antenna0: 8 secs ago..................... -54 dBm
antenna1: 8 secs ago..................... -54 dBm
AEBWAP207(slot 0)
antenna0: 7 secs ago..................... -82 dBm
antenna1: 7 secs ago..................... -82 dBm
AEBWAP207(slot 1)
antenna0: 7 secs ago..................... -87 dBm
antenna1: 7 secs ago..................... -87 dBm

DHCP Server IP Address: ....................... <omitted>
Discover-offer time: 0

Request-ack time: 0

DHCP Server IP Address: ....................... <omitted>
Discover-offer time: 0

Request-ack time: 2013873

Re: ARP not resolved while roaming

MHM Cisco World — Tue, 05 Aug 2025 09:41:51 GMT

Thanks' are you sure this client roaming?

I dont see last AP and new AP ?

Check client with issue' see AP name is it connect to same wlc or not

MHM

Re: ARP not resolved while roaming

O_A_H — Tue, 05 Aug 2025 09:54:27 GMT

Yes sure.. you will not see the last AP in the output of this command.. this command only shows the current status of the connected client.

Parsing the debug file shows this event sequence. As you see, after successful FT roaming event, the client does DHCP

Re: ARP not resolved while roaming

MHM Cisco World — Tue, 05 Aug 2025 11:05:12 GMT

Ok'

Now all device effect by this ?

old abd new AP use same vlan for this ssid?

MHM

Re: ARP not resolved while roaming

MHM Cisco World — Tue, 05 Aug 2025 11:29:06 GMT

https://medium.com/@wirelesslab.io/cisco-9800-wlc-client-disconnections-6850276b7e38

Check this article about dhcp request after roaming abd device that not support FT and run slow roaming.

Re: ARP not resolved while roaming

O_A_H — Tue, 05 Aug 2025 12:20:29 GMT

Yes. Yes.
Thanks for sharing the article, but in this case the client does FT properly and does fast roam, then followed by DHCP. I didn't see in the article a point that would ring a bell in this case.

Re: ARP not resolved while roaming

MHM Cisco World — Tue, 05 Aug 2025 12:26:01 GMT

Check point 9 dhcp require

MHM

Re: ARP not resolved while roaming

O_A_H — Tue, 05 Aug 2025 12:28:40 GMT

I already did.. that's not the case

Re: ARP not resolved while roaming

MHM Cisco World — Tue, 05 Aug 2025 12:37:07 GMT

There are two dhcp process

Full dhcp process which exchange 4 dhcp message

And other dhcp process in which client send only request to inform dhcp server it still use same IP

client do second dhcp process after roaming not full dhcp process

am I right

MHM

Re: ARP not resolved while roaming

MHM Cisco World — Wed, 06 Aug 2025 16:26:53 GMT

last update
1- DHCP I think this clear now
2- GARP <<- how you detect if WLC not send GARP to core SW or not ?

3- Authentication Key Management.................... FT-802.1x <<- from info you share it clear that client support FT
4- check bug below

https://quickview.cloudapps.cisco.com/quickview/bug/CSCvn05881

Re: ARP not resolved while roaming

O_A_H — Thu, 07 Aug 2025 07:15:33 GMT

Thanks for sharing your ideas and your collaboration!
2- Where GARP fits in the picture here? I didn't see that in the link you shared.

4- Thanks for sharing the bug. This is not relevant in my case. Let's split the conversation for clarity, as per my tshooting, the roaming issue i have is due to 1st ARP packet from the client is dropped and that causes 500ms delay. This part we decided to stop tshooting as we plan to migrate to 9800 this year as i stated earlier. The 2nd part when we talked about why clients do DHCP each roaming event, and that's a different point, and that's what we were trying to figure out with my conversation with you.