Mobility tunnel between WLC9800 - IOS-XE and WLC3504 - AireOS

[DayeshShetty44737]

Hi Team,

I am trying to form a mobility tunnel between l between WLC9800 - CATOS and WLC3504 - AireOS . However, unable to do so. 

I am following below steps to achieve the same :

https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/213913-building-mobility-tunnels-on-catalyst-98.html#anc7

Control and Data Path is down at the moment . Both the WLCs are in different zones.  FW rule is open as well between them . I can ping it which means they are reachable as well however unable to form the tunnel between them.

Happy to discuss this further.

Regards,
Dayesh

[Scott Fella]

Are you running an AireOS code that is supported? I know for sure that mobility works as long as the code is supported and that the ports are open (udp 16667 and udp 16667). There are tools that you can test ports being sent and received. I probably would do that first just to make sure.

[DayeshShetty44737]

Thanks Scott.

Yes AireOS is on the supported version compatible version i.e 8.5.164.0 and firewalls rules are open too for that port.

Tried running mping and eping from WLC3504 , correct FW rules are getting hits.

Is there a command that I am missing on WLC9800 ?

Regards

[Scott Fella]

Not really, you just need to add the proper MAC address and verify if DTLS is defined on AireOS or not because that makes a difference on the 9800 peer configuration.

https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/213913-building-mobility-tunnels-on-catalyst-98.html#anc12

[Rich R]

Just me being pedantic but just to point out that the 9800 runs IOS-XE not CATOS.
CATOS was a very old operating system used on old Catalyst switches.

[DayeshShetty44737]

Apologies . Yeah its running on( C9800_IOSXE-K9), Version 16.12.3 and AireOS on  8.5.164.0

Following are the debug logs :

*mmListenFsm: Jul 07 16:24:58.950: Client initiating connection on 172.21.160.13:16666 <-> 172.23.20.15:16666
*mmListenFsm: Jul 07 16:24:58.950: Sending packet to 172.23.20.15:16666
*osapiBsnTimer: Jul 07 16:25:05.705: Tearing down link 172.21.160.13:16666 <-> 172.23.20.15:16666 due to timeout
*osapiBsnTimer: Jul 07 16:25:05.705: Disconnecting link 172.21.160.13:16666 <-> 172.23.20.15:16666 (dtls2_timer_handler)
*osapiBsnTimer: Jul 07 16:25:05.705: Nothing to send on link 172.21.160.13:16666 <-> 172.23.20.15:16666
*osapiBsnTimer: Jul 07 16:25:05.705: Crypto keys removed from DTLS engine.
*osapiBsnTimer: Jul 07 16:25:05.705: mm_dtls2_callback: Connection with 172.23.20.15:16666 is terminated

[DayeshShetty44737]

Any further suggestions will be appreciated?

Packet capture also looks fine between the two WLCs...

[Scott Fella]

Did you enable secure mobility on the 3504? It is a requirement since the 9800 has control plane encryption enabled by default. If you followed the doc step by step and also reviewed other docs like this:

https://www.cisco.com/c/en/us/td/docs/wireless/controller/technotes/8-8/b_c9800_wireless_controller-aireos_ircm_dg.html

You will need to open a TAC case to figure out what is missing.

[Rich R]

I think @Scott Fella is right - you've forgotten to enable encryption like the guide says. Your debug shows that you're using unencrypted mobility on port 16666 instead of encrypted on 16667. And your packet capture which you say 'looks fine' can't look fine because you'll see them trying to talk on 2 different ports! And that's why it reports a timeout.

[DayeshShetty44737]

Hi Scott ,

I have followed the below step from the document as well and enabled secure mobility on AireOS.

=============

On the 9800 WLC, control plane encryption is always enabled, which means that you need to have secure mobility enabled on the AireOS side.However, data link encryption is optional. If you enable it on the 9800 side, you will need to enable it on AireOS with config mobility group member data-dtls <mac-add Cat9800> enable.

=============

Data link encryption has been disabled on both the ends.

[DayeshShetty44737]

Hi Rudling,

Encryption is disabled at both the end . Also , the port below in the mobility summary is 16666 therefore I think negotiation over 16666 is correct.

Note: Port 16667 indicates secure-mode (encryption). Port 16666 indicates non secure-mode (no encryption).

(Cisco Controller) >show mobility summary

Mobility Protocol Port........................... 16666
Default Mobility Domain.......................... DEF-Mobility
Multicast Mode .................................. Disabled
Mobility Domain ID for 802.11r................... 0x676d
Mobility Keepalive Interval...................... 10
Mobility Keepalive Count......................... 3
Mobility Group Members Configured................ 2
Mobility Control Message DSCP Value.............. 0

Controllers configured in the Mobility Group
MAC Address IP Address Group Name Multicast IP Status
4c:e1:76:c1:d1:ab 172.23.20.15 ABC-Mobility  0.0.0.0 Control Path Down
70:18:a7:c8:1c:d0 172.21.160.13 DEF-Mobility 0.0.0.0 Up

Please share your thoughts.

Thanks,

[DayeshShetty44737]

And I continue to see the same debug..

*mmListenFsm: Jul 08 01:09:29.925: Client initiating connection on 172.21.160.13:16666 <-> 172.23.20.15:16666
*mmListenFsm: Jul 08 01:09:29.926: Sending packet to 172.23.20.15:16666
*osapiBsnTimer: Jul 08 01:09:36.843: Tearing down link 172.21.160.13:16666 <-> 172.23.20.15:16666 due to timeout
*osapiBsnTimer: Jul 08 01:09:36.843: Disconnecting link 172.21.160.13:16666 <-> 172.23.20.15:16666 (dtls2_timer_handler)
*osapiBsnTimer: Jul 08 01:09:36.843: Nothing to send on link 172.21.160.13:16666 <-> 172.23.20.15:16666
*osapiBsnTimer: Jul 08 01:09:36.843: Crypto keys removed from DTLS engine.
*mmMobility: Jul 08 01:09:36.843: DTLS Action Result message received
*osapiBsnTimer: Jul 08 01:09:36.843: mm_dtls2_db_status_down:933 Connections status down for entry 172.23.20.15:16666
*osapiBsnTimer: Jul 08 01:09:36.843: mm_dtls2_callback: DTLS Connection with 172.23.20.15:16666 is terminated,Sending update msg to mobility HB

[Rich R]

Did you read what Scott said? "It is a requirement since the 9800 has control plane encryption enabled by default."
And in the docs he pointed you to it clearly shows port 166667 being used in the debugs.
It's not an option - you MUST enable encryption on the AireOS end otherwise it will not work.
Data link encryption is optional, control plane encryption is not.
CLI Configuration Details for Mobility Peer on AireOS
config mobility group domain ircm
config mobility group member add 00:0c:29:a8:d5:77 172.20.227.73 ircm encrypt enable <<<<<<<<<

Even if you don't believe us or the documents try it and see what happens ;)

[Scott Fella]

So did you ever get this to work?

[DayeshShetty44737]

Hi Scott,

I raised a TAC with Cisco . The command to enable the secure mobility didn't work on the local mobility group in AireOS. Also , deleting the local mobility group in AireOS isn't allowed using the delete keyword.

It kept giving following error

---

(Cisco Controller) >config mobility group member delete 70:18:b7:18:12:d0

Cannot delete system mac

(Cisco Controller) >config mobility group member add 70:18:b7:18:12:d0 172.21.160.13 ABCD-Mobility encrypt enable

Mobility group with management IP address is already configured.

---

I've asked Cisco to reproduce it in the Lab and they are also recommending version upgrade from 8.5.164 to 8.8.111 which is misleading from the Cisco documents.

Awaiting Cisco's feedback.

Thanks.