Control path down - between Intranet <> DMZ

holzhirt1 · ‎03-07-2014

We are about the deploy 2 WLCs 5508 in the DMZ to ensure traffic on mobile devices will be going thru Internet directly.

Between the Intranet and the DMZ we have a firewall, we did a lot of tests and the following is occuring :

normal pings between Intranet <> DMZ are fine

epings between Intranet <> DMZ are fine

mpings are not working between Intranet <> DMZ.

We checked carefully that the firewall is having IP Protocol 97 and UDP 16666 open but even with that in place mpings are still failing.

We opened completely the firewall during a couple of minutes same problem.

We checked the logs and we had the impression that IP Protocol 97 was clearly visible on the logs but not UDP.

I verified that WLCs in Intranet and / or DMZ are not using ACLs and it is not the case.

A debug mobility keep-alive show the same, IP Protocol 97 is ok but not UDP 16666...

So my anchor stays at Control path Down on both ends.

We don't have also ACLs on the switches in between...

We are running out of ideas here and I would be very glad to have more information how to process further on that...

Thanks for your help.

Update 14.03.2014 : I finally found the solution, it seems that when the network route is defined to widely this could affect the management interfaces during the tunnel mounting. This is not normal and not properly documented, usually network should serve only service port as the default gateway option is not configurable. Cisco will try to add this in the TAC knowledge bade for helping other people facing the same to spare some time in troubleshooting

in my case i simply changed the route to be really specific and the tunnel mounted immediately.

but thanks all for the suggestions and support provided.

Scott Fella · ‎03-07-2014

Delete the mobility group from both WLC's and add it back on. Sometimes that is the issue especially if the FW is wide open.

Thanks,

Scott

*****Help out other by using the rating system and marking answered questions as "Answered"*****

-Scott
*** Please rate helpful posts ***

holzhirt1 · ‎03-07-2014

Hi Scott,

Thanks for your fast reply,

We agree that the Default Mobility Domain Name should be different in Intranet WLCs and in DMZ WLCs for security reasons ?

It is just important to create the anchor by specifying the Domain name configured on the destination controller ?

Thanks

Scott Fella · ‎03-07-2014

The mobility domain name should be different... best practice and you should specify the mobility domain name when creating the anchor.

Thanks,

Scott

*****Help out other by using the rating system and marking answered questions as "Answered"*****

-Scott
*** Please rate helpful posts ***

holzhirt1 · ‎03-07-2014

Ok I removed the anchor on both and recreated it, but it is the same :-(

Control path down

I try also to reboot the WLC's in the DMZ but same problem,

Would be something else to check in order to mount up the EoIP tunnel properly?

Thanks

Scott Fella · ‎03-07-2014

The only way to test if the FW is dropping the traffic is to mount the WLC in the inside for testing and create your mobility anchor. If this works, then something is dropping UDP 16666/16667.

Post your show mobility summary from both

Thanks,

Scott

*****Help out other by using the rating system and marking answered questions as "Answered"*****

-Scott
*** Please rate helpful posts ***

holzhirt1 · ‎03-07-2014

Ok this was my thought maybe

I realized something else,

In the DMZ the WLCs are connected to switches.

We have a trunk (2 Vlans configured one for management and one for users traffic) and WLCs are in LAG mode. 4 interfaces on 8 are connected currently.

We use a subnet like 172.21.1.xxx for the management interfaces.

The default GW of the management interface is the firewall.

But since a couple of hours the mobility anchor is instead of up / up is Data Path Down,

As the subnet is the same they should not use the default GW between them, why now the Data Path is down...

Is there anything special to verify / configure on the switches ?

Here the output :

DMZ WLC 1 = 172.21.2.6 :

Mobility Architecture ........................... Flat
Mobility Protocol Port........................... 16666
Default Mobility Domain.......................... DMZ_SG_MOBILITY
Multicast Mode .................................. Disabled
Mobility Domain ID for 802.11r................... 0xe8e1
Mobility Keepalive Interval...................... 10
Mobility Keepalive Count......................... 3
Mobility Group Members Configured................ 3
Mobility Control Message DSCP Value.............. 0

Controllers configured in the Mobility Group
MAC Address        IP Address       Group Name                        Multicast IP     Status
70:81:05:1f:e4:40 10.136.10.36     SG_MOBILITY                       0.0.0.0          Control Path Down
78:da:6e:8a:ee:20 172.21.2.5       DMZ_SG_MOBILITY                   0.0.0.0          Data Path Down
78:da:6e:8b:14:60 172.21.2.6       DMZ_SG_MOBILITY                   0.0.0.0          Up

Intranet WLC 1 = 10.136.10.36 :

(Cisco Controller) >show mobility summary

Mobility Architecture ........................... Flat
Mobility Protocol Port........................... 16666
Default Mobility Domain.......................... SG_MOBILITY
Multicast Mode .................................. Disabled
Mobility Domain ID for 802.11r................... 0xd9f7
Mobility Keepalive Interval...................... 10
Mobility Keepalive Count......................... 3
Mobility Group Members Configured................ 5
Mobility Control Message DSCP Value.............. 0

Controllers configured in the Mobility Group
MAC Address IP Address Group Name Multicast IP Status

70:81:05:1f:e4:40 10.136.10.36     SG_MOBILITY                       0.0.0.0          Up
78:da:6e:8b:14:60 172.21.2.6       DMZ_SG_MOBILITY                   0.0.0.0          Control Path Down
cc:ef:48:0c:85:80 10.136.10.32     SG_MOBILITY                       0.0.0.0          Up

holzhirt1 · ‎03-11-2014

Dear Scott, I mounted one of the DMZ WLCs into the Intranet, changed the management IP as well. Normal pings were going thru, epings as well but again mpings were not able to pass thru, So we can conclude that the firewall was not in cause, should I upgrade the software ? I have no clues why this WLC is doing that, Someone has an idea to go further on that? This becomes urgent for us and I need to go further :-) Thanks

Rasika Nayanajith · ‎03-07-2014

mping verify the control path between two WLC. It use the UDP 16666 & in your case it is not working & hence control path is down.

eping verify the data path between two WLCs & uses EoIP. Since it is working for you no issue with EoIP.

Pls post the output of "show mobility summary" & "show sysinfo" of both WLCs to see any config issues.

HTH

Rasika

**** Pls rate all useful responses ****

holzhirt1 · ‎03-07-2014

Hello,

show mobility summary is posted above, here the show sysinfo :

Intranet WLC 1 10.136.10.36 :

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 7.4.100.0
Bootloader Version............................... 1.0.1
Field Recovery Image Version..................... 6.0.182.0
Firmware Version................................. FPGA 1.3, Env 1.6, USB console 1.27
Build Type....................................... DATA + WPS

System Name...................................... xxxxx

System Location.................................. xxxx

System Contact................................... xxxxx
System ObjectID.................................. 1.3.6.1.4.1.9.1.1069
Redundancy Mode.................................. Disabled
IP Address....................................... 10.136.10.36
Last Reset....................................... Software reset
System Up Time................................... 369 days 10 hrs 47 mins 44 secs
System Timezone Location......................... (GMT +1:00) Amsterdam, Berlin, Rome, Vienna
System Stats Realtime Interval................... 5
System Stats Normal Interval..................... 180

Configured Country............................... Multiple Countries:AU,BE,CH,CN,DE,FR,GB,HK,IT,J2,MX,NL,RU,SG,TH,TR,US,ZA

--More-- or (q)uit
Operating Environment............................ Commercial (0 to 40 C)
Internal Temp Alarm Limits....................... 0 to 65 C
Internal Temperature............................. +40 C
External Temperature............................. +20 C
Fan Status....................................... OK

State of 802.11b Network......................... Enabled
State of 802.11a Network......................... Enabled
Number of WLANs.................................. 5
Number of Active Clients......................... 128

Memory Current Usage............................. Unknown
Memory Average Usage............................. Unknown
CPU Current Usage................................ Unknown
CPU Average Usage................................ Unknown

Burned-in MAC Address............................ 70:81:05:1F:E4:40
Power Supply 1................................... Present, OK
Power Supply 2................................... Present, OK
Maximum number of APs supported.................. 500

DMZ WLC 1 172.21.2.6 :

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 7.4.110.0
Bootloader Version............................... 1.0.18
Field Recovery Image Version..................... 7.6.95.16
Firmware Version................................. FPGA 1.7, Env 1.8, USB console 2.2
Build Type....................................... DATA + WPS

System Name...................................... xxxx

System Location..................................
System Contact...................................
System ObjectID.................................. 1.3.6.1.4.1.9.1.1069
Redundancy Mode.................................. Disabled
IP Address....................................... 172.21.2.6
Last Reset....................................... Software reset
System Up Time................................... 0 days 1 hrs 54 mins 0 secs
System Timezone Location.........................
System Stats Realtime Interval................... 5
System Stats Normal Interval..................... 180

Configured Country............................... CH - Switzerland
Operating Environment............................ Commercial (0 to 40 C)

--More-- or (q)uit
Internal Temp Alarm Limits....................... 0 to 65 C
Internal Temperature............................. +45 C
External Temperature............................. +33 C
Fan Status....................................... OK

State of 802.11b Network......................... Disabled
State of 802.11a Network......................... Disabled
Number of WLANs.................................. 0
Number of Active Clients......................... 0

Memory Current Usage............................. Unknown
Memory Average Usage............................. Unknown
CPU Current Usage................................ Unknown
CPU Average Usage................................ Unknown

Burned-in MAC Address............................ 78:DA:6E:8B:14:60
Power Supply 1................................... Present, OK
Power Supply 2................................... Present, OK
Maximum number of APs supported.................. 12

Thanks for your support :-)

Rasika Nayanajith · ‎03-07-2014

Thanks for the inputs, configs looks ok to me.

Does all these WLCs having same virtual interface IP ?

Though it may not related, I can see lots of conflicting regulatory domain country codes configured on your DMZ controller ? Does this something you purposely configured ?

On a side note the software version (7.4.100.0) you are running on intranet WLCs are too buggy & recommend you to upgrade them to 7.4.121.0. Upgrade FUS to 1.9.0.0 as well.

HTH

Rasika

holzhirt1 · ‎03-07-2014

In fact they were having the same virtual IP but I changed it, I just reverted back for all.

Intranet WLCs is serving several countries, so yes it is on purpose that we have several regulatory domains :-)

Thanks for the tip about the version, how do you upgrade FUS by the way ?

Thank you

Ashish Chandra · ‎03-07-2014

Hi Holzhirt1,

PFB link for FUS upgrade.

Expect a downtime of 30-40 mins.

http://www.cisco.com/c/en/us/td/docs/wireless/controller/release/notes/fus_rn_OL-31390-01.pdf

Thanks,

Ashish.

Rasika Nayanajith · ‎03-07-2014

In fact they were having the same virtual IP but I changed it, I just reverted back for all.

Is there any difference made by that ? You should have same virtual IP address in order to mobility to work properly. So make sure it is same everywhere.

Intranet WLCs is serving several countries, so yes it is on purpose that we have several regulatory domains :-)

It is not ideal configuring conflicting regulatory domain country codes in single WLC (could impact channels, power levels for certain APs, sometime certain AP radio band won't come up). Best would be having unique controller to serve same regulatory domain country code APs.

Thanks for the tip about the version, how do you upgrade FUS by the way ?

Here is the link to FUS 1.9.0.0 upgrade. Keep note that this will take 30-40min of downtime to your wireless & get a sufficient outage window to do this.

http://www.cisco.com/c/en/us/td/docs/wireless/controller/release/notes/fus_rn_OL-31390-01.html

In the below thread I have posted CLI commands for this upgrade with respect to 2504. You can follow that with required image downloads for 5508

https://supportforums.cisco.com/thread/2270290

HTH

Rasika

**** Pls rate all useful responses ****

holzhirt1 · ‎03-07-2014

Now I have same virtual IPs on all but it is the same,

mpings are not going thru....

I will try again to remove mobility groups and re-create it...

thanks for the links Rasika