Confusing documentation for MX HA

[NineLine]

Disclaimer: i'm still new to Meraki, so if you are going to flame me for not knowing something, remember that i'm upskilling at the moment.

Overview of the Topology for the issue listed below

We've had an outage (on our Guest wifi) when the inside interface for the firewall at Data center 1 wentdown, because we didn't have a secondary MX set for the APs at our branch network.

-We have 2 MXs, one at each Data center.
-Each Data center uses it's own DHCP server for Guest Wifi.
-The DHCP servers are on a Catalyst 9300 switch (we've excluded range of IPs).
-We've decided to enable the feature in the documentation below, however that turned out to be a disaster. (When we raised this to Meraki TAC, they don't even understand how this feature works, unsurprisingly).
Meraki TAC is still trying to figure this out and wanted to see if the community has anything to add.
-We enabled the HA for MX concentrators.
-DHCP requests has to go through the Inside interface (dedicated for Guest-wifi) to reach the DHCP SVI on the cat9300 switch.
-The UI is bugged that when you selected "Re-associate" clients and save the page, then the page reloads that it's still marked as "don't re-associate".
-Once you put an ip in the "tunnel health ip" field, try to delete it and save, the ip will show up again upon refresh of the page. (Clearly a bug).
-Both DHCP scopes are reachable for the Access points at the branch and we've confirmed that through my personal cell phone wiifi connection and what ip addresss it grabbed. so if DHCP probes are somehow blocked, my cell phone shouldn't be able to get an ip from Guest-wifi.
-Once we've enabled that feature, the Guest Wifi at a pilot site kept flapping back and forth between the 2 data centers.
Changed VPN connections (8):
• The teleworker VPN connection to XXXXXXXXXX's SSID Guest in network Branch XXXXXXXXX is now down.
• The teleworker VPN connection to XXXXXXXXXX's SSID Guest in network Branch XXXXXXXXX is now down.
• The teleworker VPN connection to XXXXXXXXXX's SSID Guest in network Branch XXXXXXXXX is now up.
• The teleworker VPN connection to XXXXXXXXXX's SSID Guest in network Branch XXXXXXXXX is now down.
• The teleworker VPN connection to XXXXXXXXXX's SSID Guest in network Branch XXXXXXXXX is now down.
• The teleworker VPN connection to XXXXXXXXXX's SSID Guest in network Branch XXXXXXXXX is now up.
• The teleworker VPN connection to XXXXXXXXXX's SSID Guest in network Branch XXXXXXXXX is now down.
• The teleworker VPN connection to XXXXXXXXXX's SSID Guest in network Branch XXXXXXXXX is now down.

-We've received overnight an alert twice a minute from the logs above.
-The documentation doesn't mention exactly what is the acceptable use case for this feature.
-It doesn't show an example of how those DHCP packets look like and what flags are expected that the DHCP server is going to acknowledge.
-We've pointed the tunnel health IP field to the SVI of the DHCP scope on the Catalyst 9300 switch where the primary MX is. (The documentation doesn't say which DHCP server that this need to be pointed at, again no clear use case is defined here).
And we changed that to the standby. (The issue persisted)
-We didn't see any logs on the firewall inside interface that it was blocking anything.
-I ran packet capture on Cat9300 switch (at both datacenters) but i don't see anything coming as a probe.
-The ip address that the Tunnel health ip probes are pointing at is excluded from the DHCP scope.
-The documentation doesn't even say what the source ip of those probes are going to be when you set an ip address in that field.

What are we doing wrong here? how does this feature work? what DHCP options are included in those DHCP probes?

-Documentation for the feature i'm talking about:
https://documentation.meraki.com/MR/Access_Control/Secondary_MX_Concentrator_for_MR_Teleworker_VPN

[Philip D'Ath]

Are the MXs at the DCs in VPN Concentrator mode?

Are the remote MRs connecting over a WAN (no NAT involved), or over the Internet to the DC MXs?

What firmware version are you using on the MXs and MRs?

[NineLine]

the APs tunnel back to the MXs and yes, they are set to concentrator.
No NAT involved.
MR44 -> "firmware": "wireless-31-1-5"
MX100 -> "firmware": "wired-18-1-07"

[Philip D'Ath]

The MX100 is up to 18.107.13. I would try that version. I started reading through the release notes - but there have been so many releases since the version you are on, it was taking so long.

The current stable release version for MR is 31.1.7.1. I would start by moving to a firmware marked as stable.

One thing that comes to mind is that the MR and MX running in VPN concentrator mode typically need to present themselves to the Internet using the same public IP address. When this happens, they connect using their private IP addresses; otherwise, they attempt to build a tunnel over the Internet using the public IP address of the MX concentrator.

https://documentation.meraki.com/MX/Site-to-site_VPN/Configuring_Site-to-site_VPN_over_MPLS#Cisco_Meraki_VPN_Registry

So if your two concentrators present to the Internet using different public IP addresses, then the connection to one MX will be over the WAN, while the connection to the second MX will be to its public IP address, which will be out the Internet circuit of the first DC, over the Internet to the second DC, and then into the MX located there.

[NineLine]

I will double check on the code releases as i fetched them from a monitoring tool (API call) and im not sure how to navigate the convoluted meraki dashboard to get something as easy as that info.

in regards to the connectivity, the SDWAN tunnel bridge the communication as if both the datacenter and branch are directly connected. so they don't change any ips during the transport.
SDWAN tunnels are supposed to prevent the need for NAT unless you want to NAT lan side traffic explicitly to go natted inside the tunnel or DIA access for local internet.

The internet connectivity overall is hauled back to the data center for the branches to reach it. so everything must go through the data center to get to the internet.
So i don't see why would this work.
Even if the 2 datacenters want to talk to each other they are also connected over the SDWAN tunnels.

[Philip D'Ath]

Do the two DCs connect to the Internet using different public IP addresses?

The branches - do they connect via a WAN or SD-WAN? More specifically, does each branch have its own Internet connection with its own public IP address?

[NineLine]

Im honestly blown away how lackluster the troubleshooting process is on Meraki. you don't have the same tools that IOS-XE has, where you can run Packet trace to see if there are any drops on the dataplane QFP cpu to see if traffic is going through without disruption, EPCs, etc.

This is a simple issue, how do people troubleshoot more complex issue. SMDH

[NineLine]

i've looked through the dashboard again and this is what i see:

MR 31.1.5.1
MX 18.211

[CMR]

As @Philip D'Ath said, they are all quite a few versions behind, so could do with updating.

You show SD-WAN linking the sites, but no MXs at the site with the APs, are there MXs there, or another SD-WAN solution?

[NineLine]

i can't just throw random testing by just upgrading and hoping for the best.
The question was, whether the documentation was trying to say one thing or the other. what is the correct use case of this feature.

Further more, the SDWAN solution is called Cisco SDWAN, i didn't say viptela because it wouldn't make sense to say that. this renaming was done way back in 2019.

[Philip D'Ath]

I think you might have a fundamental design issue. But not enough information has been supplied to answer whether that is the case or not.

[NineLine]

what information did i not supply?
What is the issue with the design if you say i didn't give the entire description?

2 contradicting statements.

Regardless whether the design is the wrong one or not, does not address the fact that the document is not stating what is the right or wrong use case for it. or how it should be used. or how the DHCP packets are crafted. or what DHCP options are in them.

Those are the important questions, not what our topology is at the moment.

[Raphael_L]

Is there a good reason that could explain why you choose to configure the guest SSID in "tunneled mode" ?

I know we are drifting away from your issues , but I'm trying to understand the use case

[NineLine]

Since there is a requirement to backhaul all traffic back to the data center, guest traffic has been requested to be segregated.

[martin-lystad]

But you are running catalyst SD-WAN no? Why not reduce the complexity of your guest network solution and drop it in a separate guest VPN instead of dubbel tunneling?

Anyhow, seems like the tunnel is up. Have you tried to capture traffic in the MX vpn consentrators? You should be able to capture the de-capsulated packets there.