Data path down control path up issue

[Adam Watts]

have just set up a WLC 4402 as a Guest WLan controler on the DMZ of our network.

i have sucsessfully managed to get our internal controllers to connect to it, with the exception of 1. it says the control path is up but the data path is down. the other 14 controllers worked fine, and in testing the last one was ok but it is now not working properly. the 2 controllers can ping each other but just won't create the data tunnel. there is a firewall in the middle but that has been set up to allow traffic between the 2 groups of controllers to be unrestricted.

the internal controllers are 4404's and all controllers are running the same version of code. 5.1.151.0

any ideas would be great.

Adam

[Ryan Bachand]

Thanks Dave! Spent hours troubleshooting this issue before coming across your post.

[Bradley Peacock]

Thanks Dave. I know it's an old post but i've inherited a network and i've little wireless experience. Our ISP switched to new firewall and the data path failed to come up. After failing back to the original firewall, a Juniper for reference, it was still down so the common denominator was the firewall. We read this forum thread early on so the ISP rebooted the original firewall on day 1. This never worked so i've spent 4 days troubleshooting this. After pinpointing the firewall as the problem the ISP had another look and there was a hung session still on port 97. The reboot hadn't cleared it. Once it was cleared the data path came up immediately.

[jjheck]

Also check the MAC addresses of the guest and anchor controllers.  The tunnel is established by the lower of the two MAC addresses.  We had an issue where one of our internal controllers was lower than the anchor controller and we had to tweak our Palo Alto firewall to get the packets to pass and not get dropped by the FW. 

[George Stefanick]

+5 JJ ...

I did not know that ...

[nickeoannidis]

Hi All,

Im having the same issue i have 10 controllers and 1 anchor mix of 4400 series and 5508's. All running 7.0.116.0 and Anchor is on 7.0.220.0.

Randomly data path goes down for x controller. If i reboot the anchor controller - all controllers data and control paths come up.

Anchor sits behind ASA 5520 on 8.4, i have ip any rule from the addresses of the foreigns to the anchor controller. Return traffic is permitted. Can't see any issue with ACL logic as the control and data path does work, at least for a time for some controllers. Should i change this to permit UDP CAPWAP first then IP Protocol 97 in a second rule?

I tried using the clear conn to see if it would come back when the data path is down for a specific controller, no cigar.

[craiglebutt]

I'm having same issue kind of.

The WLC is  a 8510 in HA was on 8.0.121 to 8.0.140.

The mobility was up, after the upgrade, all mobility are down to external company's.  We have anchors to City and County Councils and other organisation as we a multi tenanted building.

The Data Path is up and the Control Path is down.

Wireshark capture from both ends show udp 16666 going to and from each WLC.

I've tried going back to 8.0.121 didn't fix, went to 8.0.150 did fix.  waiting on TAC to reply.

There firewalls between most of these, but 6 WLC are not behind a firewall and are not connecting.  

We still have 2 8500 in HA on 8.0.121 and these are fine.

any ideas?

[powys]

We've discovered that we're having the same issue here. Three 5508 controllers, 2 internal (WLC-01 and WLC-02) and 1 anchor in the DMZ (WLC-03). If an access point runs from WLC-01 it's fine, but if on WLC-02 the guest wifi fails.

WLC-01 can see WLC-02 and WLC-03

WLC-02 can see WLC-01, link to the anchor (WLC-03) is "Data path down"

WLC-03 can see WLC-01, link to WLC-02 is "Data path down"

mping works from any WLC to any other.

eping does not work between WLC-02 and WLC-03 (anchor) in either direction.

The firewall (a WatchGuard Firebox X750e, firmware 11.3.4) is set to allow any IP traffic between the three controllers.

According to jjheck's post above the mobility links will be set up as follows, I don't know if that's a clue as to why it's misbehaving:-

Primary WLC-01 (MAC starting 50:3d:e5:ae) will establish the link to WLC-02 and WLC-03)

Secondary WLC-02 (MAC starting 64:00:f1:f6) will not try to establish any links

Anchor WLC-03 (MAC starting 64:00:f1:f1) will establish the link to WLC-01

All three are running 7.0.240.0.

The only differences between the running configs on WLC-01 and WLC-02 (apart from the pile of APs on WLC-01) are:-

Maximum number of APs supported.................. 200 (on WLC-01, 175 on WLC-02)

Cisco AP Default Master..................... Enabled (on WLC-01, Disable on WLC-02)

The whole lot was rebooted the weekend before last as we needed to shut down the server room for electrical work, but that made no difference.

Does anyone have any suggestions? We're scratching our heads here!

[Scott Fella]

You need to be able to eping from the guest anchor to both internal.  That is Ethernet over ip 97 and it seems like the FW is blocking that.  I would open the FW rule to allow everything between the guest anchor and the two foreign WLC's for now.

Thanks,

Scott

Help out other by using the rating system and marking answered questions as "Answered"

[powys]

The firewall already has a rule set up that allows any IP traffic between the three WLCs. Basically all three WLCs appear in both the "from" and "to" columns.

We've also tried splitting the rule in two, with one allowing any IP from the anchor to the two foreign WLCs and the other allowing any IP from the foreign WLCs to the anchor. This made no difference.

[Scott Fella]

Try to delete all the mobility info and re-creating them again.  Sometime that helps.

Thanks,

Scott

Help out other by using the rating system and marking answered questions as "Answered"

[mmccarte]

Worked for me

[grabonlee]

As Scott mentioned if protocol 97 isn't allowed then the Data path will not come up. I will suggest that you explicitly allow protocol 97 and not just any IP. Also on your firewall you can do a source or destination search only using your WLC IP address to find out what traffic is dropped between the Anchor and Foreign controller. For example your search string could specify the source as the Foreign controller IP.

[Dave Lewis]

As others have mentioned, please try explicitly allowing protocol 97, I don't know about Watchguards but see this post which explains that, in Cisco's world, allowing ip any any doesn't also allow 97/47,50 etc...

https://supportforums.cisco.com/thread/1002131

And this watchguard kb...

http://www.watchguard.com/help/docs/edge/10/en-us/content/en-us/policies/custom_policy_add_manually_e.html

Dave

[powys]

We've tried that, exactly the same. We now have four firewall rules relating to it:-

From anchor to WCS and both internal WLCs: ports 16666 to 16667

From WCS and both internal WLCs to anchor: ports 16666 to 16667

From anchor to WCS and both internal WLCs: IP protocol 97

From WCS and both internal WLCs to anchor: IP protocol 97

The "Any IP" rule between the internal WLCs, WCS and anchor WLC has been deleted.

It's weird how one WLC can see the anchor but the other (which sits alongside it) cannot.

Everything works except the data path between controller 2 and the  anchor (which would be the only link established from the anchor side according to the lowest-MAC-address rule,  the others being established by controller 1 which has the lowest address of the three).

[grabonlee]

Hi Powys,

Firstly, UDP port 16667 is no longer supported and was deprecated since controller version 5 as the secure mode never worked. Not that enabling affects your control traffic any way. To resolve the problem of the second WLC, I would suggest that you look at the firewall logs to see if Protocol 97 traffic is actually passed between the Foreign and Anchor WLC.