topic Control path down - between Intranet <> DMZ in Wireless

Control path down - between Intranet <> DMZ

holzhirt1 — Mon, 05 Jul 2021 07:22:29 GMT

We are about the deploy 2 WLCs 5508 in the DMZ to ensure traffic on mobile devices will be going thru Internet directly.

Between the Intranet and the DMZ we have a firewall, we did a lot of tests and the following is occuring :

normal pings between Intranet <> DMZ are fine

epings between Intranet <> DMZ are fine

mpings are not working between Intranet <> DMZ.

We checked carefully that the firewall is having IP Protocol 97 and UDP 16666 open but even with that in place mpings are still failing.

We opened completely the firewall during a couple of minutes same problem.

We checked the logs and we had the impression that IP Protocol 97 was clearly visible on the logs but not UDP.

I verified that WLCs in Intranet and / or DMZ are not using ACLs and it is not the case.

A debug mobility keep-alive show the same, IP Protocol 97 is ok but not UDP 16666...

So my anchor stays at Control path Down on both ends.

We don't have also ACLs on the switches in between...

We are running out of ideas here and I would be very glad to have more information how to process further on that...

Thanks for your help.

Update 14.03.2014 : I finally found the solution, it seems that when the network route is defined to widely this could affect the management interfaces during the tunnel mounting. This is not normal and not properly documented, usually network should serve only service port as the default gateway option is not configurable. Cisco will try to add this in the TAC knowledge bade for helping other people facing the same to spare some time in troubleshooting 🙂

in my case i simply changed the route to be really specific and the tunnel mounted immediately.

but thanks all for the suggestions and support provided.

Control path down - between Intranet <> DMZ

Scott Fella — Fri, 07 Mar 2014 17:17:26 GMT

Delete the mobility group from both WLC's and add it back on. Sometimes that is the issue especially if the FW is wide open.

Thanks,

Scott

*****Help out other by using the rating system and marking answered questions as "Answered"*****

Control path down - between Intranet <> DMZ

holzhirt1 — Fri, 07 Mar 2014 17:57:03 GMT

Hi Scott,

Thanks for your fast reply,

We agree that the Default Mobility Domain Name should be different in Intranet WLCs and in DMZ WLCs for security reasons ?

It is just important to create the anchor by specifying the Domain name configured on the destination controller ?

Thanks

Control path down - between Intranet <> DMZ

Scott Fella — Fri, 07 Mar 2014 17:59:02 GMT

The mobility domain name should be different... best practice and you should specify the mobility domain name when creating the anchor.

Thanks,

Scott

*****Help out other by using the rating system and marking answered questions as "Answered"*****

Control path down - between Intranet <> DMZ

holzhirt1 — Fri, 07 Mar 2014 18:01:54 GMT

Ok I removed the anchor on both and recreated it, but it is the same 😞

Control path down

I try also to reboot the WLC's in the DMZ but same problem,

Would be something else to check in order to mount up the EoIP tunnel properly?

Thanks

Control path down - between Intranet <> DMZ

Scott Fella — Fri, 07 Mar 2014 18:10:39 GMT

The only way to test if the FW is dropping the traffic is to mount the WLC in the inside for testing and create your mobility anchor. If this works, then something is dropping UDP 16666/16667.

Post your show mobility summary from both

Thanks,

Scott

*****Help out other by using the rating system and marking answered questions as "Answered"*****

Control path down - between Intranet <> DMZ

holzhirt1 — Fri, 07 Mar 2014 18:17:34 GMT

Ok this was my thought maybe

I realized something else,

In the DMZ the WLCs are connected to switches.

We have a trunk (2 Vlans configured one for management and one for users traffic) and WLCs are in LAG mode. 4 interfaces on 8 are connected currently.

We use a subnet like 172.21.1.xxx for the management interfaces.

The default GW of the management interface is the firewall.

But since a couple of hours the mobility anchor is instead of up / up is Data Path Down,

As the subnet is the same they should not use the default GW between them, why now the Data Path is down...

Is there anything special to verify / configure on the switches ?

Here the output :

DMZ WLC 1 = 172.21.2.6 :

Mobility Architecture ........................... Flat
Mobility Protocol Port........................... 16666
Default Mobility Domain.......................... DMZ_SG_MOBILITY
Multicast Mode .................................. Disabled
Mobility Domain ID for 802.11r................... 0xe8e1
Mobility Keepalive Interval...................... 10
Mobility Keepalive Count......................... 3
Mobility Group Members Configured................ 3
Mobility Control Message DSCP Value.............. 0

Controllers configured in the Mobility Group
MAC Address        IP Address       Group Name                        Multicast IP     Status
70:81:05:1f:e4:40 10.136.10.36     SG_MOBILITY                       0.0.0.0          Control Path Down
78:da:6e:8a:ee:20 172.21.2.5       DMZ_SG_MOBILITY                   0.0.0.0          Data Path Down
78:da:6e:8b:14:60 172.21.2.6       DMZ_SG_MOBILITY                   0.0.0.0          Up

Intranet WLC 1 = 10.136.10.36 :

(Cisco Controller) >show mobility summary

Mobility Architecture ........................... Flat
Mobility Protocol Port........................... 16666
Default Mobility Domain.......................... SG_MOBILITY
Multicast Mode .................................. Disabled
Mobility Domain ID for 802.11r................... 0xd9f7
Mobility Keepalive Interval...................... 10
Mobility Keepalive Count......................... 3
Mobility Group Members Configured................ 5
Mobility Control Message DSCP Value.............. 0

Controllers configured in the Mobility Group
MAC Address IP Address Group Name Multicast IP Status

70:81:05:1f:e4:40 10.136.10.36     SG_MOBILITY                       0.0.0.0          Up
78:da:6e:8b:14:60 172.21.2.6       DMZ_SG_MOBILITY                   0.0.0.0          Control Path Down
cc:ef:48:0c:85:80 10.136.10.32     SG_MOBILITY                       0.0.0.0          Up

Control path down - between Intranet <> DMZ

Rasika Nayanajith — Fri, 07 Mar 2014 18:29:35 GMT

mping verify the control path between two WLC. It use the UDP 16666 & in your case it is not working & hence control path is down.

eping verify the data path between two WLCs & uses EoIP. Since it is working for you no issue with EoIP.

Pls post the output of "show mobility summary" & "show sysinfo" of both WLCs to see any config issues.

HTH

Rasika

**** Pls rate all useful responses ****

Control path down - between Intranet <> DMZ

holzhirt1 — Fri, 07 Mar 2014 18:34:41 GMT

Hello,

show mobility summary is posted above, here the show sysinfo :

Intranet WLC 1 10.136.10.36 :

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 7.4.100.0
Bootloader Version............................... 1.0.1
Field Recovery Image Version..................... 6.0.182.0
Firmware Version................................. FPGA 1.3, Env 1.6, USB console 1.27
Build Type....................................... DATA + WPS

System Name...................................... xxxxx

System Location.................................. xxxx

System Contact................................... xxxxx
System ObjectID.................................. 1.3.6.1.4.1.9.1.1069
Redundancy Mode.................................. Disabled
IP Address....................................... 10.136.10.36
Last Reset....................................... Software reset
System Up Time................................... 369 days 10 hrs 47 mins 44 secs
System Timezone Location......................... (GMT +1:00) Amsterdam, Berlin, Rome, Vienna
System Stats Realtime Interval................... 5
System Stats Normal Interval..................... 180

Configured Country............................... Multiple Countries:AU,BE,CH,CN,DE,FR,GB,HK,IT,J2,MX,NL,RU,SG,TH,TR,US,ZA

--More-- or (q)uit
Operating Environment............................ Commercial (0 to 40 C)
Internal Temp Alarm Limits....................... 0 to 65 C
Internal Temperature............................. +40 C
External Temperature............................. +20 C
Fan Status....................................... OK

State of 802.11b Network......................... Enabled
State of 802.11a Network......................... Enabled
Number of WLANs.................................. 5
Number of Active Clients......................... 128

Memory Current Usage............................. Unknown
Memory Average Usage............................. Unknown
CPU Current Usage................................ Unknown
CPU Average Usage................................ Unknown

Burned-in MAC Address............................ 70:81:05:1F:E4:40
Power Supply 1................................... Present, OK
Power Supply 2................................... Present, OK
Maximum number of APs supported.................. 500

DMZ WLC 1 172.21.2.6 :

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 7.4.110.0
Bootloader Version............................... 1.0.18
Field Recovery Image Version..................... 7.6.95.16
Firmware Version................................. FPGA 1.7, Env 1.8, USB console 2.2
Build Type....................................... DATA + WPS

System Name...................................... xxxx

System Location..................................
System Contact...................................
System ObjectID.................................. 1.3.6.1.4.1.9.1.1069
Redundancy Mode.................................. Disabled
IP Address....................................... 172.21.2.6
Last Reset....................................... Software reset
System Up Time................................... 0 days 1 hrs 54 mins 0 secs
System Timezone Location.........................
System Stats Realtime Interval................... 5
System Stats Normal Interval..................... 180

Configured Country............................... CH - Switzerland
Operating Environment............................ Commercial (0 to 40 C)

--More-- or (q)uit
Internal Temp Alarm Limits....................... 0 to 65 C
Internal Temperature............................. +45 C
External Temperature............................. +33 C
Fan Status....................................... OK

State of 802.11b Network......................... Disabled
State of 802.11a Network......................... Disabled
Number of WLANs.................................. 0
Number of Active Clients......................... 0

Burned-in MAC Address............................ 78:DA:6E:8B:14:60
Power Supply 1................................... Present, OK
Power Supply 2................................... Present, OK
Maximum number of APs supported.................. 12

Thanks for your support 🙂

Re: Control path down - between Intranet <> DMZ

Rasika Nayanajith — Fri, 07 Mar 2014 18:48:17 GMT

Thanks for the inputs, configs looks ok to me.

Does all these WLCs having same virtual interface IP ?

Though it may not related, I can see lots of conflicting regulatory domain country codes configured on your DMZ controller ? Does this something you purposely configured ?

On a side note the software version (7.4.100.0) you are running on intranet WLCs are too buggy & recommend you to upgrade them to 7.4.121.0. Upgrade FUS to 1.9.0.0 as well.

HTH

Rasika

Control path down - between Intranet <> DMZ

holzhirt1 — Fri, 07 Mar 2014 18:58:40 GMT

In fact they were having the same virtual IP but I changed it, I just reverted back for all.

Intranet WLCs is serving several countries, so yes it is on purpose that we have several regulatory domains 🙂

Thanks for the tip about the version, how do you upgrade FUS by the way ?

Thank you

Re: Control path down - between Intranet <> DMZ

Ashish Chandra — Fri, 07 Mar 2014 19:11:37 GMT

Hi Holzhirt1,

PFB link for FUS upgrade.

Expect a downtime of 30-40 mins.

http://www.cisco.com/c/en/us/td/docs/wireless/controller/release/notes/fus_rn_OL-31390-01.pdf

Thanks,

Ashish.

Re: Control path down - between Intranet <> DMZ

Rasika Nayanajith — Fri, 07 Mar 2014 19:12:10 GMT

In fact they were having the same virtual IP but I changed it, I just reverted back for all.

Is there any difference made by that ? You should have same virtual IP address in order to mobility to work properly. So make sure it is same everywhere.

Intranet WLCs is serving several countries, so yes it is on purpose that we have several regulatory domains 🙂

It is not ideal configuring conflicting regulatory domain country codes in single WLC (could impact channels, power levels for certain APs, sometime certain AP radio band won't come up). Best would be having unique controller to serve same regulatory domain country code APs.

Thanks for the tip about the version, how do you upgrade FUS by the way ?

Here is the link to FUS 1.9.0.0 upgrade. Keep note that this will take 30-40min of downtime to your wireless & get a sufficient outage window to do this.

http://www.cisco.com/c/en/us/td/docs/wireless/controller/release/notes/fus_rn_OL-31390-01.html

In the below thread I have posted CLI commands for this upgrade with respect to 2504. You can follow that with required image downloads for 5508

https://supportforums.cisco.com/thread/2270290

HTH

Rasika

**** Pls rate all useful responses ****

Control path down - between Intranet <> DMZ

holzhirt1 — Fri, 07 Mar 2014 19:25:03 GMT

Now I have same virtual IPs on all but it is the same,

mpings are not going thru....

I will try again to remove mobility groups and re-create it...

thanks for the links Rasika

Control path down - between Intranet <> DMZ

Scott Fella — Fri, 07 Mar 2014 19:28:47 GMT

If that still doesn't work, move the one to the inside and reconfigure the management for testing. Make sure that it works..

You also have two DMZ SLC's, so put them in the same mobility group to test and see if both the control and data comes up.

Thanks,

Scott

*****Help out other by using the rating system and marking answered questions as "Answered"*****

Re: Control path down - between Intranet <> DMZ

George Stefanick — Fri, 07 Mar 2014 19:33:48 GMT

By having a different mobility name saves on mobility exchanges meaning less packets .. Imagine all your controllers have the same name .. Your anchor would get mobility messages from all your anchors which isn't needed.

I would sniff the fw interface and the interface outside the WLC interface .. Do you see the control packet ..

Sent from Cisco Technical Support iPhone App

Re: Control path down - between Intranet <> DMZ

Ali Aqrabawi — Fri, 07 Mar 2014 22:11:45 GMT

did you checked the logs on the firewall , why it's dropping the mpings ?

Control path down - between Intranet <> DMZ

holzhirt1 — Sat, 08 Mar 2014 09:37:29 GMT

First of all thanks all for your tips and contributions, really appreciated.

Dear Ali odd enough only Ethernet IP 97 are coming into the FW, no UDP 16666 visible.

I realized something else,

In the DMZ the WLCs are connected to switches.

We have a trunk (2 Vlans configured one for management and one for users traffic) and WLCs are in LAG mode. 4 interfaces on 8 are connected currently.

We use a subnet like 172.21.1.xxx for the management interfaces.

The default GW of the management interface is the firewall.

But since a couple of hours the mobility anchor between DMZ WLCs is instead of up / up is Data Path Down,

Epings are not passing thru....

As the subnet is the same they should not use the default GW between them, why now the Data Path is down...

Is there anything special to verify / configure on the switches ?

As we will have at least 2 Vlans we need to have trunk on the switches, I have no native Vlan configured and pings between DMZ WLCs are working...

Thanks

Control path down - between Intranet <> DMZ

Ali Aqrabawi — Sat, 08 Mar 2014 11:17:56 GMT

one thing to check on the switchport , the portchannel load-balance should be src-dst-ip,

I changed the management IPs

holzhirt1 — Mon, 10 Mar 2014 15:46:30 GMT

I changed the management IPs on the DMZ WLCs and reacreated the anchor strangely I was able to mount the tunnel immediately, before it was not passing thru for epings (Data Path Down), now it is up / up. Even if in my opinion it was not passing thru a FW.

However the link with the Intranet is still showing the same issue "Control Path Down".

I will have a look again with my colleague about the Firewall and if we can see at least UDP 16666 packets coming from one end or the other.

If this is not leading to something, I will follow the recommendation to mount it internall and see how the tunnel is reacting...

I keep you posted,

Thanks