Mobility path down between 9800-40 and 9800CL

[michael18]

I have a problem getting the data path up between C9800-40 (foreign) 9800CL (anchor). the foreign is behind my firewall, the anchor behind a 3rd party firewall.

both are on ver: Cisco IOS XE Software, Version 17.06.04

captures show connectivity on my firewall inside int:

1: 09:06:50.217624 172.18.60.4.16667 > 10.40.251.10.16667: udp 130
2: 09:06:50.217670 172.18.60.4.16666 > 10.40.251.10.16666: udp 115
3: 09:06:50.218433 10.40.251.10.16667 > 172.18.60.4.16667: udp 121
4: 09:06:50.218479 10.40.251.10.16667 > 172.18.60.4.16667: udp 130
5: 09:06:50.218723 10.40.251.10.16666 > 172.18.60.4.16666: udp 110

capture from my firewall dmz interface:

300: 09:12:30.250582 802.1Q vlan#800 P0 172.18.60.3 > 10.40.251.10 icmp: 172.18.60.4 udp port 16667 unreachable
301: 09:12:40.251390 802.1Q vlan#800 P0 172.18.60.3 > 10.40.251.10 icmp: 172.18.60.4 udp port 16667 unreachable
302: 09:12:40.251589 802.1Q vlan#800 P0 172.18.60.3 > 10.40.251.10 icmp: 172.18.60.4 udp port 16667 unreachable
303: 09:12:50.252077 802.1Q vlan#800 P0 172.18.60.3 > 10.40.251.10 icmp: 172.18.60.4 udp port 16667 unreachable
304: 09:12:50.252291 802.1Q vlan#800 P0 172.18.60.3 > 10.40.251.10 icmp: 172.18.60.4 udp port 16667 unreachable
305: 09:12:54.653362 802.1Q vlan#800 P0 10.40.251.10 > 172.18.60.4 icmp: echo request
306: 09:12:54.654095 802.1Q vlan#800 P0 172.18.60.4 > 10.40.251.10 icmp: echo reply
307: 09:12:54.654934 802.1Q vlan#800 P0 10.40.251.10 > 172.18.60.4 icmp: echo request
308: 09:12:54.655605 802.1Q vlan#800 P0 172.18.60.4 > 10.40.251.10 icmp: echo reply
309: 09:12:54.656490 802.1Q vlan#800 P0 10.40.251.10 > 172.18.60.4 icmp: echo request
310: 09:12:54.657146 802.1Q vlan#800 P0 172.18.60.4 > 10.40.251.10 icmp: echo reply
311: 09:12:54.658031 802.1Q vlan#800 P0 10.40.251.10 > 172.18.60.4 icmp: echo request
312: 09:12:54.658489 802.1Q vlan#800 P0 172.18.60.4 > 10.40.251.10 icmp: echo reply
313: 09:12:54.659389 802.1Q vlan#800 P0 10.40.251.10 > 172.18.60.4 icmp: echo request
314: 09:12:54.660000 802.1Q vlan#800 P0 172.18.60.4 > 10.40.251.10 icmp: echo reply
315: 09:13:00.253038 802.1Q vlan#800 P0 172.18.60.3 > 10.40.251.10 icmp: 172.18.60.4 udp port 16667 unreachable
316: 09:13:00.253221 802.1Q vlan#800 P0 172.18.60.3 > 10.40.251.10 icmp: 172.18.60.4 udp port 16667 unreachable
317: 09:13:10.254289 802.1Q vlan#800 P0 172.18.60.3 > 10.40.251.10 icmp: 172.18.60.4 udp port 16667 unreachable

Mobility summary

IP                    Public Ip          MAC Address     Group Name     Multicast IPv4     Multicast IPv6      Status             PMTU

10.40.251.10     N/A            44b6.bee8.fa6b     Internal               0.0.0.0                    ::                   N/A                  N/A

172.18.60.4  172.18.60.4    000c.29d6.680f     Internal               0.0.0.0                    ::             Data Path Down     1385

ping from foreign to anchor

WLC001#ping 172.18.60.4
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.18.60.4, timeout is 2 seconds:
!!!!!

the anchor is behind a NAT. the foreign targets 172.18.60.4

All documentation seems to point to catalyst to AireOS. 

Where do i look for data path issues.

Thanks

[marce1000]

 - Check if these commands can provide additional info's :
                   show wireless mobility summary
                    show wireless stats mobility
                  show wireless stats mobility messages 
                  show platform hardware chassis active qfp feature wireless punt  statistics 

 M.

[marce1000]

 - (adding) : on both controllers review the configuration with (CLI) show tech wireless ; have the output analyzed with 
                                     https://cway.cisco.com/wireless-config-analyzer/

 M.

[Rich R]

What is 172.18.60.3?

[michael18]

Its is the 3rd party firewall interface. 

ive just found this document: Cisco Catalyst 9800 Series Wireless Controller Software Configuration Guide, Cisco IOS XE Amsterdam 17.3.x - NAT Support on Mobility Groups [Cisco Catalyst 9800 Series Wireless Controllers] - Cisco

We may have the mobility group IP info incorrect. I need to check it

[Rich R]

And the 3rd party firewall is blocking the mobility traffic:
172.18.60.3 > 10.40.251.10 icmp: 172.18.60.4 udp port 16667 unreachable
That's the firewall telling you it's dropped those packets which were destined for 172.18.60.4.