WLC 9800 Web Authentication for guest users blocked ARP request

[117222400]

Hi Experts

Recently we encounter an interesting issue. The below is network design

End user --> AP -->   WLC 9800  (DHCP) --> Gateway (Cluster vip .1, active ip .2) --> Internet

                                               |

                                     Web authentication server(Cisco ISE)

The end device connected to AP, and got IP address from WLC, and then access Web Authentication Server(ISE) to login by username and password. After that it can access internet. 

The WLC,AP, gateway interface are in the same VLAN, but the Web authentication server is in another subnet. The end device needs to connect to the Web authentication server through the gateway. The WLC can access the web authentication server.

The phenomenon is :  no redirection prompts on the end device and it can't reach the guest portal.

We find that on the gateway cluster (active member), the arp entry for the end device is not created, and after we add it to the static arp entry, everything is working. The guest portal prompts and user can login. The issue is,  end device must access the web authentication server through gateway, but gateway doesn't have the end user's arp entry so it can't forward packets to the end device. 

After tcpdump on the gateway, and capture packets from WLC, and also on the end device.

1. On the gateway and WLC, we can see many ARP request from the gateway to end device IP, but no any response from the end device. The gateway(cluster) 's virtural IP is x.x.x.1, the active member is x.x.x.2. The traffic is "Who has the end device ip x.x.x.87, please tell x.x.x.2". ( not to .1, I am thinking will it trigger some ARP anti-proofing ?)

2. On the end device, we can't see any ARP requests from the gateway. It seems the ARP requests were dropped by WLC or AP.

    We can see serval ARP announcements sent by end device, but it seems all these announcements were dropped by WLC and can't reach gateway.

I think switches between the AP and WLC shouldn't block/drop any ARP things as they are in the same broadcast domain.

Is there any anti-ARP proofing related security policy on the WLC or AP?  On the guest WLAN settings, the MAC filtering was enabled and the authentication/authorization are pointed to the Web Authentication server (Cisco ISE).

GuestPolicy

WLAN security Layer2

On the other hand, we can see from ISE server, there is traffic from the end device hitted. It seems the authentication traffic from end device, has reached ISE server, but can't go back. There is no arp entry on the gateway, so on layer 2 it can't be forwarded to the end device.

Last year, we use the WLC built-in Web authentication, no any issue. Because the end device doesn't need to go through the gateway to access the third party Web authentication server.

Any idea about it? or what's the next step should I trace the traffic/packet?

Thanks and best regards

Liang

[marce1000]

               - What software version is the WLC 9800 running ?

 M.

[117222400]

17.3.7 

[marce1000 · Accepted solution]

  - For starters and because of 17.3.x going EOL , I would go for 17.9.4a and test again , 

 M.

[117222400]

Thanks very much for you advice.

But it is working good when using built-in Web Authentication service. I also suspect:

1. The WLC blocked/dropped ARP request when the end device didn't pass authentication. ( when the Web Authentication is on the WLC, end device doesn't need to get through the gateway. )

2. is it possible because of the "P2P Blocking Action" was set to Drop? As the gateway is set to x.x.x.1, actually the end device will need to send ARP to the active cluster member x.x.x.2 .

[marce1000]

               >... is it possible because of the "P2P Blocking Action" was set to Drop?
  You can test P2P options according to : https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/config-guide/b_wl_16_10_cg/peer-to-peer-client-support.html

 M.

[117222400]

I turned off this option, unfortunately no help

[117222400 · Accepted solution]

Finally, we upgraded to 17.9.4a and the issue seems resolved , thanks very much. ( Although the root cause is still a mystery...) 

[MHM Cisco World]

GW send arp to wifi user ??
that not correct 
the wifi user need to send ARP to GW 
and did you check the GW config in DHCP 
maybe the Client use wrong GW 
MHM

[117222400]

Gateway interface, wifi users are in the same VLAN, we can see the wifi user send ARP request: Who has x.x.x.1, tell x.x.x.87, and the gateway replied its MAC address, that's why the packets can reach gateway and been forwarded to ISE. 

The gateway in DHCP is x.x.x.1 ( cluster virtual IP) , and the active member is x.x.x.2 , absolutely correct in DHCP settings.

[117222400]

Is it possible to check the dropped layer 2 packets by WLC ?  if yes, then maybe we can see a lot of ARP request packets from the gateway to the end device were dropped. 

I also did a Radioactive Trace on the end device, I can see the same thing as above:

1. ARP Request:  from end device to gateway (x.x.x.1)    are there, and replied by gateway, and received by end device.

2. ARP Request: from gateway(x.x.x.1) to end device, missing(or dropped by WLC), the end device didn't receive, so it won't reply.

[Rich R]

1. I second @marce1000's original suggestion - there are numerous problems with 17.3 code so in my opinion you are simply wasting your time troubleshooting while still using 17.3 code.  Start with upgrading as per TAC recommended link below.

2.  Using the internal DHCP server on WLC: first note the Best Practice guide: "The best practice is to use an external DHCP server" https://www.cisco.com/c/en/us/products/collateral/wireless/catalyst-9800-series-wireless-controllers/guide-c07-743627.html#InternalDHCPserver and take note of the guidelines "When you want to use the internal DHCP server, ensure that you configure SVI for the client VLAN and set the IP address as the DHCP server’s IP address"

It might be a good time for you to read all of the Best Practice guidelines as well as upgrading your code version before trying to progress any further with your troubleshooting.  And check your config using the Config Analyzer (link below).

[117222400]

Thank you very much for your reply.

1. I need to find the root cause then submit a change request.

2. About the DHCP, it works well, and it is global, also the SVI was set, I can see that the SVI relay DHCP request to the WLC's management interface to get an IP address.

I did another testing: disable the SVI, and set DHCP relay on the gateway, pointing to WLC DHCP interface. The interesting is the client can't get an IP address.

I thought the gateway,WLC,client should be in the same broadcast domain as they are all in layer 2 network environment. But actually it isn't......  Just suspect all traffic/packets will be blocked/dropped if the guest user didn't pass the web authentication, including unicast (ARP request),broadcast(DHCP request) between the gateway and the users.

Should I disable Central DHCP, Central Switching.. these features....it is only a layer 2 broadcast domain........

[handrews4]

What is in your ACL on the WLC? The redirect ACL that ISE should be sending to the WLC. If this is not correct traffic wont work.

Also second the suggestions of upgrading the WLC.

Is the gateway VSS or HSRP?

[117222400]

The attached is ACL for Web Authentication.  x.x.7.33 is ISE, and x.x.01.33 is DNS, they are both different VLAN as the Guest wifi.  I think it should be layer 3 ACL, and shouldn't block layer 2 ARP request/DHCP request broadcast.

The gateway is HSRP, virtual IP is x.x.x.1, active member is x.x.x.2, standby member is x.x.x.3, SVI is x.x.x.5. They are all in the same VLAN and same subnet.