C9800-80 ARP Packet Processing

[pengus]

Hi everyone,

My user currently RUN state. I capture radioactive trace, user send 2 arp packets to default gateway. One is destination MAC address ffff.ffff.ffff but other one's mac address unicast cisco MAC address (probabbly gateway's mac address). When I capture packets on L3 devices port which is gateway for user, only see unicast mac address, im not see arp packets have destination mac address ffff.ffff.ffff .  My controller does not have layer 3 interface on user vlan. There is no entry in vlan. My question is how to controller or user know gateway's mac address? Why broadcast arp packets reach gateway device?

Radioactive trace logs:

2023/03/21 15:33:37.448926 {wncd_x_R0-1}{2}: [sisf-packet] [21022]: (debug): RX: ARP from interface capwap_xx on vlan 218 Source MAC: 9061.aec4.c160 Dest MAC: ffff.ffff.ffff ARP REQUEST, ARP sender MAC: 9061.aec4.c160 ARP target MAC: 0000.0000.0000 ARP sender IP: a.a.a.20, ARP target IP: a.a.a.254,
2023/03/21 15:33:37.448974 {wncd_x_R0-1}{2}: [dot11] [21022]: (ERR): MAC: 005d.7317.b12f DOT11 fsm is not allocated yet.
2023/03/21 15:33:37.449008 {wncd_x_R0-1}{2}: [sisf-packet] [21022]: (debug): TX: ARP from interface capwap_xxx on vlan 218 Source MAC: 9061.aec4.c160 Dest MAC: 005d.7317.b12f ARP REQUEST, ARP sender MAC: 9061.aec4.c160 ARP target MAC: 0000.0000.0000 ARP sender IP: a.a.a.20, ARP target IP: a.a.a.254,

And only unicast arp packets reach to gateway device (Nexus 7700 series). And  005d.7317.b12f  MAC address not in my network. Also Controller does not have IP address on user vlan this means that wlc does not have any arp entry for this vlan. When I create svi for user vlan on wlc and IP address, this interface can ping to hsrp ip andress which user not access 

I found 005d.7317.b12f mac address where in ap interface network command output. But I did not figure out how i see this mac address on controller tx operation.

*****#show interfaces network

...

apr1v0 Link encap:Ethernet HWaddr 00:5D:73:17:B1:2F
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:150253652 errors:0 dropped:0 overruns:0 frame:0
TX packets:565731641 errors:0 dropped:8840 overruns:0 carrier:8682
collisions:0 txqueuelen:1000
RX bytes:1852084627 (1.7 GiB) TX bytes:2815180143 (2.6 GiB)
Interrupt:102 Memory:f8400000-f8600000

...

[marce1000]

  - FYI : https://www.cisco.com/c/en/us/products/collateral/wireless/catalyst-9800-series-wireless-controllers/guide-c07-743627.html#AddressResolutionProtocolARPproxy

 M.

[pengus]

Hi,

When I search the Arp proxy in the policy profile, it ensures that the arp packets destined for the wireless users are answered by the controller before they reach the user clients. The problem I'm having is that arp packets are unicast when the user asks for their gateway (i.e. an SVI IP address outside the controller).

Thanks for reply. Have a nice day...

[Flavio Miranda]

Hi

 Users knows gateway by DHCP information. Default gateway is infomed during DHCP reply from DHCP server. 

 User  send an ARP request asking who has IP address X.x.x.x? And this request is a broadcast ffff.ffff.ffff.on the users vlan.

They WLC will forward this to all ports on the vlan incuding the L3 devices. the L3 device  will reply with its Mac address. 

If you type arp command in a Windows machine, your are going to see the arp table and the gateway IP and ARP will be there. 

[pengus]

Hi, 

yes, client learn gateway IP address from dhcp process and it sends arp request for learnin mac address of default gateway. İ capture this packet with wireshark and this packet!s destination mac address ffff.ffff.ffff.I also started radioactive trace via wlc and I see the following output.

(debug): RX: vlan 218 Source MAC: xxxx.xxxx.xxxx Dest MAC: ffff.ffff.ffff ARP REQUEST, ARP sender MAC: xxxx.xxxx.xxxx ARP target MAC: 0000.0000.0000 ARP sender IP: a.a.a.a, ARP target IP: a.a.a.254,
 (ERR): MAC: 8c60.4f03.1441 DOT11 fsm is not allocated yet.
(debug): TX: ARP from interface capwap_90400018 on vlan 218 Source MAC: xxxx.xxxx.xxxx Dest MAC: 8c60.4f03.1441 ARP REQUEST, ARP sender MAC: xxxx.xxxx.xxxx ARP target MAC: 0000.0000.0000 ARP sender IP: a.a.a.a, ARP target IP: a.a.a.254

this log showed me that wlc changed the arp package from ffff to unicast and this mac address and these mac addresses are not available on my network. 

[Flavio Miranda]

It can be because your Access Point is in central mode and you have capwap tunnel in between. Which capwap tunnel, you can see the user mac address only after the tunnel is open by the wlc and exchange the packet with its gateway. 

[pengus]

Hi,

Yes, My access point local mode (central swittching)and send all traffic to controller via CAPWAP. But this arp problem only this ip address. this IP address hsrp IP address and user send svi IP address ping or arp packet bu not access hsrp ip address(I m sure that hsrp configuration correct).  Also when reload wlc this problem fix but after time this problem occur again. Briefly, WLC has no IP in user vlan and no ip arp entry for vlan. How can wlc convert a broadcast arp package to unicast? how did he find out this mac address? 

[Rich R]

Sounds to me like one of the clients is using the same IP address as your gateway.  Are you sure the gateway address is excluded on your DHCP server and you have DHCP required for clients?

What version of software are you using?

Have you checked your WLC config with https://cway.cisco.com/wireless-config-analyzer/ using output of "show tech wireless"?

[pengus]

Yes, I am sure that gateway IP addess is out of pool(pool end 200, hsrp gateway 254, one svi is 253 other 252).  C9800-80 software version 17.3.4c; ROM: 16.12(5r). I check bug search tool bug related to this th's problem but i did not found any problem or bugs. I investigate  Wireless Config Analyzer Express output but there are not any missing or wrong configuration.  

what i don't understand, client send broadcast arp packet to learning mac address of gateway but wlc change destination mac address from ffff.ffff.ffff to 005d.7317.b12f. I search this mac address whole brodcast domain but i didn't found. I delete this vlan on my controller for clearing cache and create again but the problem continues. When I reboot controller, problem is temporarily solved/ But after time approximately 3-4 day same problem occurs again. 

[Rich R]

So:

For a start update to latest 17.3 with all AP service packs applied.  Just because you didn't find any bugs doesn't mean they don't exist.  Sometimes they're hidden from us (Cisco internal only) and sometimes the description might be quite different or misleading.

Better still update to 17.6.5 or 17.9.3 and then if you're still seeing the same problem and can't explain it then open a TAC case.

[pengus · Accepted solution]

Yes, observing the radio mac address of an AP on the network (as a destination mac in the arp request as well); Thought there might be a bug in the image of APs. 

And we solved problem without restarting 9800. but we will find out in time if this solution can fix the situation permanently.

• disabled source guard at WLAN->Advanced tab
• Give default gateway IP address's to client statickly (this will cause ip address conflict with your gateway device)  
• undo changes (activate source guard, change client's network settings to automaticly)
• ARP packets are performing as they should and now users can access their gateways

[0_Kiss]

Hi pengus,
We just had the exact same issue happen a few days after upgrading to 17.9.4 (AP Service Packs not yet installed) and your workaround has helped.
How did it go for you after the workaround? Did the issue reoccur or was it a permanent fix?
Did you reach out to TAC for an explanation?
Thanks and best regards,
Oli

[pengus]

Hello Oli,

I did not reach out to TAC. I still don't know why something like this happened. We have not encountered this problem since the last date ‎03-24-2023. We have updated the controller(to 17.3.7) in the meantime. I'm really wondering why this problem occurs. Is this a result of a bug or due to another reason?

Best regards.