WLC 3504 dropping or failing to forward traffic from Juniper SRX320

[TacticalDonut164]

(disclaimer: homelab)

Hey guys,

I am having an issue with the interaction between the Juniper SRX320 and the Cisco WLC 3504. Wireless clients that have their gateway set and traffic handled on/by the SRX320, have their traffic either dropped or not  forwarded by the WLC 3504. 

From packet captures, I can see that traffic leaves the client, egresses through the WLC up to the SRX, from where sessions are created. The SRX then sends return traffic out to the WLC (take DHCP and ICMP traffic as an example), and it appears in packet captures conducted on the WLC:

192.168.1.253 192.168.1.254 DHCP 382 DHCP Request
192.168.1.254 192.168.1.253 DHCP 325 DHCP ACK
192.168.1.254 192.168.1.1 ICMP 74 Destination unreachable (Port unreachable)
[previous message repeats many times]
192.168.1.254 192.168.1.1 ICMP 78 Echo (ping) reply    id=0xc00c, seq=2/512, ttl=64
192.168.1.254 192.168.1.1 ICMP 78 Echo (ping) reply    id=0xc00c, seq=3/768, ttl=64
8.8.8.8 192.168.1.1 ICMP 78 Echo (ping) reply    id=0x2c26, seq=9/2304, ttl=117
8.8.8.8 192.168.1.1 ICMP 78 Echo (ping) reply    id=0x2c26, seq=10/2560, ttl=117
8.8.8.8 192.168.1.1 ICMP 78 Echo (ping) reply    id=0x2c26, seq=11/2816, ttl=117
8.8.8.8 192.168.1.1 ICMP 78 Echo (ping) reply    id=0x2c26, seq=12/3072, ttl=117

On the layer 2 side of this pcap, the source MAC is the SRX, the destination MAC is the dynamic interface for that subnet on the WLC. 

However, from here, this traffic does not go anywhere. Either it is never forwarded to the AP, or the AP never forwards it to the client. This results in clients having zero layer 3 or higher connectivity to the gateway = clients cannot leave their subnet. 

Layer 2 is perfectly fine. All devices in the chain have correct/valid ARP entries.

Of course, you might be thinking that this is an issue with the Juniper product, not the Cisco product. However considering this is not an issue with Palo Alto firewall, there is something odd about the way the WLC handles the SRX's traffic, and given that packets appear on the WLC, it seems to me more of an issue with the WLC's handling of traffic. (especially since SRX > WLC initiated traffic does not work past the dynamic interface). I have not tried eliminating the WLC from the chain via EWC as I am very unsure of the process and doing so would take down wireless. 

I also posted in the Juniper forum to get assistance with checking the Juniper side but would like to tackle this from the Cisco side. My next option would probably be to replace the Cisco infrastructure with an HPE Aruba AP-655. 

Thanks guys! Configurations are attached to this post.

• Firewall: 2x Juniper SRX320-SYS-JB 23.4R2-S4.9
• Core switch: 1x Juniper EX3400-48T 23.4R2-S3.9
• Access switch: 1x Juniper EX3400-24P 23.4R2-S3.9
• Controller: 1x Cisco AIR-CT3504-K9 8.10.196.0
• AP: 1x Cisco C9130AXI-B

[marce1000]

  - Difficult to tackle such a combined problem ; for starters check the software version on the 3504
     and according to https://www.cisco.com/c/en/us/support/docs/wireless/wireless-lan-controller-software/200046-tac-recommended-aireos.html
    you should go for  8.10.196.0
    This is important these days as these older airos controllers are being phased out. It kind of become
    imperative to use the last (recommended) release available.

   For the rest use client debugging according to https://www.cisco.com/c/en/us/support/docs/wireless/wireless-lan-controller-software/213258-collect-debugs-from-wireless-lan-control.html
   Client debugs can be analyzed with Wireless Debug Analyzer

   You could consider testing it on the 9800 platform where you can download and deploy the virtual VM for free
   and in an initial period use APs with it too.
   If it then works , then abandoning the aireos controller would become a more important thing to do ,

  M.

[TacticalDonut164]

After a bit of troubleshooting on the Juniper side we discovered the following:

1. Static IPs work perfectly fine. 
2. If everything lives on the SRX, except for the DHCP server which lives on the PA-220, then everything works fine. This is the current workaround. Therefore the issue is specifically with the interaction between the WLC and SRX-issued DHCP addresses. 
3. Despite clients getting an IP (show in ipconfig and dynamically received, binding present in the 'show dhcp server binding interface reth2.1681'), packet captures via a SPAN on the WLC uplink show that the client is repeatedly spamming DHCP Discover packets.
4. Despite clients having ARP entries for the gateway and vice versa, packet captures on the SRX reveal that the client is spamming ARP requests, with corresponding ARP replies. 

What would you make of this behavior? One suggestion from the Juniper side was to disable DHCP relay on the WLC, but I'm not seeing a way to do so. 

edit - and to answer your question. The WLC is on the latest recommended version released May 2024. 8.10.196.0. The client debugs, they did not show much before, but now that I know I should generally be concerned with DHCP and ARP packets, I may try this again. Although with everything working now, I am a bit less motivated to continue messing with stuff, to be honest. 

[Scott Fella · Accepted solution]

There is a way to disable dhcp relay.  SSH to the controller and issue the following: config dhcp proxy disable

From the GUI: Controller > Advanced > DHCP uncheck Enable DHCP Proxy

[TacticalDonut164]

This fixed it! Thank you very much!

[Scott Fella]

That was an issue way back when when a FW use to hand out dhcp:) I think that is burned in many or the old school engineers here on the forums.  Ran into that so many times with customers in the past.  Glad that fixed the issue.

[marce1000]

@TacticalDonut164 : The WLC is on the latest recommended version released May 2024. 8.10.196.0. 
                                 Yeah I earlier mentioned 8.5.x but that was wrong , the reply has been corrected,

  M.

[ammahend]

do a capture on AP wired port, to see if the traffic is delivered to AP, you can span the port to another port and do capture, to see inside capwap, check this box in wireshark under preferences>protocol to see original packet inside capwap

[TacticalDonut164]

Thank you, I will have to try and look into that.