Control path down - between Intranet <> DMZ

[holzhirt1]

We are about the deploy 2 WLCs 5508 in the DMZ to ensure traffic on mobile devices will be going thru Internet directly.

Between the Intranet and the DMZ we have a firewall, we did a lot of tests and the following is occuring :

normal pings between Intranet <> DMZ are fine

epings between Intranet <> DMZ are fine

mpings are not working between Intranet <> DMZ.

We checked carefully that the firewall is having IP Protocol 97 and UDP 16666 open but even with that in place mpings are still failing.

We opened completely the firewall during a couple of minutes same problem.

We checked the logs and we had the impression that IP Protocol 97 was clearly visible on the logs but not UDP.

I verified that WLCs in Intranet and / or DMZ are not using ACLs and it is not the case.

A debug mobility keep-alive show the same, IP Protocol 97 is ok but not UDP 16666...

So my anchor stays at Control path Down on both ends.

We don't have also ACLs on the switches in between...

We are running out of ideas here and I would be very glad to have more information how to process further on that...

Thanks for your help.                   

Update 14.03.2014 : I finally found the solution, it seems that when the network route is defined to widely this could affect the management interfaces during the tunnel mounting. This is not normal and not properly documented, usually network should serve only service port as the default gateway option is not configurable. Cisco will try to add this in the TAC knowledge bade for helping other people facing the same to spare some time in troubleshooting

in my case i simply changed the route to be really specific and the tunnel mounted immediately.

but thanks all for the suggestions and support provided.

[Scott Fella]

If that still doesn't work, move the one to the inside and reconfigure the management for testing.  Make sure that it works..

You also have two DMZ SLC's, so put them in the same mobility group to test and see if both the control and data comes up.

Thanks,

Scott

*****Help out other by using the rating system and marking answered questions as "Answered"*****

[George Stefanick]

By having a different mobility name saves on mobility exchanges meaning less packets .. Imagine all your controllers have the same name .. Your anchor would get mobility messages from all your anchors which isn't needed.

I would sniff the fw interface and the interface outside the WLC interface .. Do you see the control packet ..

Sent from Cisco Technical Support iPhone App

[holzhirt1]

Dear George,

We did a sniff in the beginning and UDP 16666 was not visible I believe, we have seen some malformed packets, but I will give it a try again today and see how it looking...

[Ali Aqrabawi]

did you checked the logs on the firewall , why it's dropping the mpings ?

[holzhirt1]

First of all thanks all for your tips and contributions, really appreciated.

Dear Ali odd enough only Ethernet IP 97 are coming into the FW, no UDP 16666 visible.

I realized something else,

In the DMZ the WLCs are connected to switches.

We have a trunk (2 Vlans configured one for management and one for users traffic) and WLCs are in LAG mode. 4 interfaces on 8 are connected currently.

We use a subnet like 172.21.1.xxx for the management interfaces.

The default GW of the management interface is the firewall.

But since a couple of hours the mobility anchor between DMZ WLCs is instead of up / up is Data Path Down,

Epings are not passing thru....

As the subnet is the same they should not use the default GW between them, why now the Data Path is down...

Is there anything special to verify / configure on the switches ?

As we will have at least 2 Vlans we need to have trunk on the switches, I have no native Vlan configured and pings between DMZ WLCs are working...

Thanks

[Ali Aqrabawi]

one thing to check on the switchport , the portchannel load-balance should be src-dst-ip,

[holzhirt1]

I changed the management IPs on the DMZ WLCs and reacreated the anchor strangely I was able to mount the tunnel immediately, before it was not passing thru for epings (Data Path Down), now it is up / up. Even if in my opinion it was not passing thru a FW.

However the link with the Intranet is still showing the same issue "Control Path Down".

I will have a look again with my colleague about the Firewall and if we can see at least UDP 16666 packets coming from one end or the other.

If this is not leading to something, I will follow the recommendation to mount it internall and see how the tunnel is reacting...

I keep you posted,

Thanks

[holzhirt1]

I followed Rasika's advice and upgraded both DMZ WLCs to latest FUS 1.9.0.0 as well as 7.4.121.0 firmware.

Problem is still the same...Control path down...

Anyone has an idea how to go further on that?

I'm out of ideas here :(

Thanks

[holzhirt1]

After more time in this issue without results,I opened a case to TAC, the engineer spent more than 2 hours with me in a webex without finding something else, the debugs are under investigations.

Let's cross the fingers we can find out what's wrong here...

[Francesco Bagnoli]

Hi,

I really don't understand the problem resolution. Do you mean that, in order to implement Guest-Anchor solution we have to define specific routes to Anchors from the foreign management?

I'm asking because we are experiencing problems with anchor in dmz but we still not have a solution. (TAC doesn't help).

Thanks for your support

Francesco

[holzhirt1]

Dear Sir,

Under controller > network routes, you can define routes when the WLC's are in DMZ and you are using the service-port to reach them.

If the routes entered are too widely defined like a whole B or A Class this could affect the behavior of the mobility tunnels.

It is better then to precisely define the routes to match only the management stations.

I hope this explanation is clear enough, otherwise feel free to ask me again :)

[Francesco Bagnoli]

Hi,

thanks for your answer. In our case we don't use service port to reach DMZ. Maybe this workaround is also needed in Guest tunneled via Management interface?

Thanks

Francesco

[Scott Fella]

Guest is always tunneled using the management ip address... that is defined in the mobility group configuration which has the mac address and the ip of the WLC.  You don't define a static route for mobility to work, you just need the ports opened to allow mobility to come up.

[G Money]

Thanks holzhirt1! I had the same issue and removing the broad network route resolved it.