Solved: Anchor Controller Control/Data path Issues

christopher.wood · ‎03-19-2013

Hi,

I've read several of these discussions already, but I can't seem to find a solution for my issues with WLAN controllers.

I have two WLCs 5508s sitting in my network, and a third 5508 sitting in the DMZ. The control and data path will not come up to the Anchor controller sitting in the DMZ. We've opened the firewall up to make sure all traffic is allowed between the three WLCs, but still the paths do not come up.

I can ping the anchor controller no problem, but it won't mping or eping.

I did notice in the logs, this morning, that yesterday when I rebooted the anchor controller in the DMZ, the control path came up to one of the WLCs in the network for roughly 10 seconds and then went down and won't come back up.

All three are in the same mobility group, and the mobility group is setup on all three. I've double checked the IP addresses and MAC addresses several times. I'm not sure what I could be missing in the config that won't allow the paths to come up.

Any thoughts?

Thanks.

Christopher.

christopher.wood · ‎03-22-2013

So, as it turns out. It was a problem with the NAT configurations. The ASA, dude had to add in some new ones, and then we got Mobility up to both devcies, and then he had to do a 'clear conn' command in order for the Data paths to come up. Everything seems to be working like a charm there now.

I am however having a problem getting the Web Auth working properly. The DHCP is local to the anchor controller, and guests are getting the proper IP address, but everyone is getting authorized without the getting redirected to the Web Auth pages on the Anchor Controller. I tried going to http://1.1.1.1/login.html, but I can't seem to get there.

EDIT: New info here. So, this looks like it may have bee na routing issue on my side, partically. I can now access https://1.1.1.1/login.html from my cell phone, couldn't before until I changed some routing. However, it doesn't automatically take me to the Login page for Web Auth. I was able to authenticate on the page, but then still couldn't get to any website after being authenticated.

ALSO, my work laptop has proxy server information in the internet browsers, could that be causing an issue? As I still can't access https://1.1.1.1/login.html from my laptop.

Thanks.

View solution in original post

Stephen Rodriguez · ‎03-19-2013

if you can't mping and eping it sounds like a FW issue to me honestly.

can you post the show mobility summary from all three WLC?

HTH,
Steve

------------------------------------------------------------------------------------------------
Please remember to rate useful posts, and mark questions as answered

christopher.wood · ‎03-19-2013

Here is the Anchor Controller:

Symmetric Mobility Tunneling (current) .......... Enabled
Symmetric Mobility Tunneling (after reboot) ..... Enabled
Mobility Protocol Port........................... 16666
Default Mobility Domain.......................... OLG
Multicast Mode .................................. Disabled
Mobility Domain ID for 802.11r................... 0xde71
Mobility Keepalive Interval...................... 10
Mobility Keepalive Count......................... 3
Mobility Group Members Configured................ 3
Mobility Control Message DSCP Value.............. 0

Controllers configured in the Mobility Group
MAC Address        IP Address       Group Name                        Multicast IP     Status
30:f7:0d:30:d0:a0 10.129.254.51    OLG                               0.0.0.0          Control and Data Path Down
a4:93:4c:fb:5e:c0 172.16.2.10      OLG                               0.0.0.0          Up
d4:8c:b5:a7:36:00 10.129.254.50    OLG                              0.0.0.0          Control and Data Path Down

Here is the WLC1 in the network:

Symmetric Mobility Tunneling (current) .......... Enabled
Symmetric Mobility Tunneling (after reboot) ..... Enabled
Mobility Protocol Port........................... 16666
Default Mobility Domain.......................... OLG
Multicast Mode .................................. Disabled
Mobility Domain ID for 802.11r................... 0xde71
Mobility Keepalive Interval...................... 10
Mobility Keepalive Count......................... 3
Mobility Group Members Configured................ 3
Mobility Control Message DSCP Value.............. 0

Controllers configured in the Mobility Group
MAC Address        IP Address       Group Name                        Multicast IP     Status
30:f7:0d:30:d0:a0 10.129.254.51    OLG                               0.0.0.0          Up
a4:93:4c:fb:5e:c0 172.16.2.10      OLG                               0.0.0.0          Control and Data Path Down
d4:8c:b5:a7:36:00 10.129.254.50    OLG                               0.0.0.0          Up

Here is the WLC2 in the network:

Symmetric Mobility Tunneling (current) .......... Enabled
Symmetric Mobility Tunneling (after reboot) ..... Enabled
Mobility Protocol Port........................... 16666
Default Mobility Domain.......................... OLG
Multicast Mode .................................. Disabled
Mobility Domain ID for 802.11r................... 0xde71
Mobility Keepalive Interval...................... 10
Mobility Keepalive Count......................... 3
Mobility Group Members Configured................ 3
Mobility Control Message DSCP Value.............. 0

Controllers configured in the Mobility Group
MAC Address        IP Address       Group Name                        Multicast IP     Status
30:f7:0d:30:d0:a0 10.129.254.51    OLG                               0.0.0.0          Up
a4:93:4c:fb:5e:c0 172.16.2.10      OLG                               0.0.0.0          Control and Data Path Down
d4:8c:b5:a7:36:00 10.129.254.50    OLG                               0.0.0.0          Up

Stephen Rodriguez · ‎03-19-2013

on your anchor controller take a look at the mac address of the management interface. make sure it is the same as what is showing in the mobility config.

HTH,
Steve

------------------------------------------------------------------------------------------------
Please remember to rate useful posts, and mark questions as answered

christopher.wood · ‎03-19-2013

Well, it's not the same. If I got to Interfaces, I have a4:93:4c:fb:5e:cf. The MAC that it shows there is the one it populated itself in the mobility group. Should I be changing that to the MAC of the management interface on all devices?

Stephen Rodriguez · ‎03-19-2013

yes sir. I've seen this a few times where the management uses a variant of the burned in.

HTH,
Steve

------------------------------------------------------------------------------------------------
Please remember to rate useful posts, and mark questions as answered

christopher.wood · ‎03-19-2013

Weird, we have the same setup going on in another location, but they don't have the problem I'm having. I just looked at their anchor controller, and it's doing the same thing. The management interface has a different MAC than the auto assigned one from the mboility group, but theirs is working. That's weird. I'm goin change around these MACs though and see if that fixes me up.

christopher.wood · ‎03-19-2013

It doesn't let me change the MAC on the Anchor controller itself. Just says it can't create the new interface. Is there a way of doing this, or do I just create a new entry with the same IP address?

EDIT: Doesn't allow for that either. Is there a way to change the MAC to the proper one on the Anchor Controller?

Also, realized it's using the MAC address of the virtual interface, and the IP of the management.

christopher.wood · ‎03-20-2013

So, this still wouldn't work. I ended up setting up a switch and plugging the DMZ anchor controller into the switch with the one of the internal WLCs, they came up to each other just fine.

I put the anchor controller back into the DMZ behind the firewall, and had my firewall guy start making changes, as he started making changes, the Anchor controller and the one of the two WLCs came up to each other in their mobility groups, but the second WLC only had it's control path come up and the data stayed down. After a bit of time, the second controller's control path went back down. The first WLC and the Anchor controller stayed up to each other though.

I then proceeded to reboot the secondary controller, and that didn't fix it, so I tried rebooting the Anchor controller, and now it won't come up to either controller again. The paths are down on both WLCs again.

I'm starting to think I have a problem with my ASA firewall.

Scott Fella · ‎03-20-2013

It usually is... if you eping from one WLC to another, that test the Ethernet over ip or IP port 97. The mping will test the mobility which is udp 16666. So the one test I would do is open the FW up to allow everything between the foreign and anchor WLC's and see if the mobility comes up. You already tested it by connecting the management to an internal network bypassing the FW. They should see the FW dropping packets though.

Thanks,

Scott

Help out other by using the rating system and marking answered questions as "Answered"

-Scott
*** Please rate helpful posts ***

christopher.wood · ‎03-20-2013

That's the thing, we opened up everything between the WLCs and the Anchor Controller. Still doesn't work, than the guy I have configuring the ASA started messing with the ASA rules, and suddenly the mobility groups came up between one of the WLCs and the Anchor controller, the other WLC was in a Control Up, Data Down state. Then eventually the second controller just went Control and Path Down. However, the first WLC and the Anchor Controller still had the Control and Data paths Up. Then I rebooted the second controller, and that did nothing, then rebooted the anchor controller, and lost Control and Data to both again.

I feel like the ASA was allowing the traffic, but then saw something it didn't like and shut it down??? Or maybe the ASA is just messed up??

Scott Fella · ‎03-20-2013

hahaha.... tell that to your FW guy:)

So you should see one side down, the other side should also be down. Now I had one time seen an issue similar and the only fix was to remove all the mobility group information and actually add it back through the CLI not the GUI. Then like magic, the control and data path came up. The WLC was on 7.0.235.0 if that helps, but delete it from the CLI and add it back from the CLI and see if that works.

Thanks,

Scott

Help out other by using the rating system and marking answered questions as "Answered"

-Scott
*** Please rate helpful posts ***

Stephen Rodriguez · ‎03-20-2013

have the ASA guy do a show xlate ( think that's the command)

it could be the ASA is pinning traffic to the wrong interface. your ASA guy can do a capture specific to mobility on the inside and dmz interface, to make sure traffic is flowing as it should

HTH,
Steve

------------------------------------------------------------------------------------------------
Please remember to rate useful posts, and mark questions as answered

christopher.wood · ‎03-20-2013

I did the show xlate command myself.

I did show xlate | i 'anchor controller ip address'

I found nothing in the xlate with the anchor controller IP address, but I did find my two WLCs going to two similiar addresses, but not the anchor controller.

Ran a ping test from one WLC to the Anchor, got the replies and checked the show xlate again, and same deal.

Not sure if I was doing it properly, or know 100% what I'm looking for...

marstoyanoff · ‎03-21-2013

Hi Christopher,

Sure, the "show xlate" will not give you any output. I would do "sh xlate" only to identify an existing NAT translation from a private to a well-know public IPv4 address for my firewall. Capturing the traffic is the way to go. Let me know if you need any assistance with creating and analyzing captures.

Cheers,

Marty