WLC 9800 Mobility peer going down once standby node is up.

[RoadRunner4k]

We are testing a new set of WLC 9800-L in HA.  running on version 17.3.4c. 2 Mobility peers is configured and in UP state (when HA standby node is down) Once the standby node is up one of the tunnels go down, and cipher changes from TLS_NUM_RSA_WITH_AES_128_CBC_SHA to TLS_NUM_NULL_WITH_NULL_NULL

If i reload slot 2, it will reestablish after  a few minutes, and once standby is up it goes down. 

Does anyone experience the same? did i hit a bug, or did i misconfigure somewhere? 

WLC01#sh wireless mobility summary
Mobility Summary

Wireless Management VLAN: 7
Wireless Management IP Address: 10.40.99.10
Wireless Management IPv6 Address:
Mobility Control Message DSCP Value: 48
Mobility Keepalive Interval/Count: 10/3
Mobility Group Name: MBK_Test
Mobility Multicast Ipv4 address: 0.0.0.0
Mobility Multicast Ipv6 address: ::
Mobility MAC Address: f4bd.9e56.a2eb
Mobility Domain Identifier: 0x390d

Controllers configured in the Mobility Domain:

IP Public Ip MAC Address Group Name Multicast IPv4 Multicast IPv6 Status PMTU
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
10.40.99.10 N/A f4bd.9e56.a2eb MBK_Test 0.0.0.0 :: N/A N/A
10.40.99.110 10.40.99.110 706d.1517.a66d test 0.0.0.0 :: Up 1385
10.47.26.245 10.47.26.245 a093.5153.38e3 Anchor_Group 0.0.0.0 :: Up 1385

WLC01#sh wireless mobility peer ip 10.40.99.110

Mobility Peer Info
===================
Ip Address : 10.40.99.110
Public Ip Address : 10.40.99.110
MAC Address : 706d.1517.a66d
Group Name : test
Total Number of Clients on Peer : 0
Local Clients Exported to Peer : 0
Locally Anchored Peer Clients : 0
Keepalive Data Link Status : UP
Keepalive Control Link Status : UP
DTLS Data Link Cipher : TLS_NUM_NULL_WITH_NULL_NULL
DTLS Data Link Status : Disabled
DTLS Control Link Cipher : TLS_NUM_RSA_WITH_AES_128_CBC_SHA
DTLS Control Link Status : Key Plumbed
PMTU : 1385
Tunnel Plumbed : Yes
Tunnel IFID : 0xA0000001
Number of Data Path Flaps : 3
Last Data Path Flap : 01/20/2022 10:26:42 Central
Number of Data Packet Failure : 1022
Non Acked Data ping requests : None
Number of Control Path Flaps : 3
Last Control Path Flap : 01/20/2022 10:27:22 Central
Number of Control Packet Failure : 347
Non Acked Control ping requests : None
Is Anchor : No
Mobility event statistics
Roam stats
L2 roam count : 0
L3 roam count : 0
Handoff Status Received
Success : 0
Group mismatch : 0
Client unknown : 0
Client blacklisted : 0
SSID mismatch : 0
Denied : 0
Handoff Status Sent
Success : 0
Group mismatch : 0
Client unknown : 0
Client blacklisted : 0
SSID mismatch : 0
Denied : 0

WLC01#sh wireless mobility peer ip 10.47.26.245

Mobility Peer Info
===================
Ip Address : 10.47.26.245
Public Ip Address : 10.47.26.245
MAC Address : a093.5153.38e3
Group Name : Anchor_Group
Total Number of Clients on Peer : 0
Local Clients Exported to Peer : 0
Locally Anchored Peer Clients : 0
Keepalive Data Link Status : UP
Keepalive Control Link Status : UP
DTLS Data Link Cipher : TLS_NUM_NULL_WITH_NULL_NULL
DTLS Data Link Status : Disabled
DTLS Control Link Cipher : TLS_NUM_RSA_WITH_AES_128_CBC_SHA
DTLS Control Link Status : Key Plumbed
PMTU : 1385
Tunnel Plumbed : Yes
Tunnel IFID : 0xA0000002
Number of Data Path Flaps : 2
Last Data Path Flap : 01/11/2022 10:32:56 Central
Number of Data Packet Failure : 223
Non Acked Data ping requests : None
Number of Control Path Flaps : 1
Last Control Path Flap : 01/11/2022 10:27:06 Central
Number of Control Packet Failure : 81
Non Acked Control ping requests : None
Is Anchor : Yes
Mobility event statistics
Roam stats
L2 roam count : 0
L3 roam count : 0
Handoff Status Received
Success : 0
Group mismatch : 0
Client unknown : 0
Client blacklisted : 0
SSID mismatch : 0
Denied : 0
Handoff Status Sent
Success : 0
Group mismatch : 0
Client unknown : 0
Client blacklisted : 0
SSID mismatch : 0
Denied : 0

Once the standby Node is in STANDBY_HOT the tunnel goes down. (only one of them) Tunnel to 10.40.99.110

WLC01# show wireless mobility peer ip 10.40.99.110

Mobility Peer Info
===================
Ip Address : 10.40.99.110
Public Ip Address : 10.40.99.110
MAC Address : 706d.1517.a66d
Group Name : test
Total Number of Clients on Peer : N/A
Local Clients Exported to Peer : 0
Locally Anchored Peer Clients : 0
Keepalive Data Link Status : DOWN
Keepalive Control Link Status : DOWN
DTLS Data Link Cipher : TLS_NUM_NULL_WITH_NULL_NULL
DTLS Data Link Status : Disabled
DTLS Control Link Cipher : TLS_NUM_NULL_WITH_NULL_NULL <--
DTLS Control Link Status : Disabled
PMTU : 1385
Tunnel Plumbed : Yes
Tunnel IFID : 0xA0000001
Number of Data Path Flaps : 5
Last Data Path Flap : 01/20/2022 13:08:31 Central
Number of Data Packet Failure : 683
Non Acked Data ping requests : None
Number of Control Path Flaps : 5
Last Control Path Flap : 01/20/2022 13:09:51 Central
Number of Control Packet Failure : 236
Non Acked Control ping requests : None
Is Anchor : No
Mobility event statistics
Roam stats
L2 roam count : 0
L3 roam count : 0
Handoff Status Received
Success : 0
Group mismatch : 0
Client unknown : 0
Client blacklisted : 0
SSID mismatch : 0
Denied : 0
Handoff Status Sent
Success : 0
Group mismatch : 0
Client unknown : 0
Client blacklisted : 0
SSID mismatch : 0
Denied : 0

[marce1000]

 - Not a direct reply but always useful is to have a configuration check , on the (primary) 9800 issue (CLI) : show tech wireless, have the output analyzed by https://cway.cisco.com/tools/WirelessAnalyzer/ ,            check the advisories. 

 M.

[MHM Cisco World]

follow

[Scott Fella]

Well... when you create a mobility peering with another controller, its either for anchoring or when you are using N+1.  There is no need to configure a mobility group between two same model controllers that are going to be in HA.

[RoadRunner4k]

Im am aware of that its only done to test and verify the mobility tunnel (we got two WLC9800 in HA 10.40.99.10) and on in the same subnet on 10.40.99.110 WLC5520 on 8.10.162 and its to here we are losing the tunnel once the standby node in the WLC9800 HA pair is in STANDBY_HOT.

[Scott Fella]

Okay... that was not clear to me initially.  That being said, make sure you are using the mac address on the HA that is being shared.  Your 5520 should only have one entry to the 9800 in HA.  If you have an issue still, then maybe reach out to TAC.

[Rich R]

Had never noticed it before but yes seeing the same between 9800-80 HA pair on 17.6.1.

That's 9800-80 (HA-SSO pair) <-> 9800-80 (HA-SSO pair).

How did you notice it - is anything logged when you see the state change?

[Arshad Safrulla]

In 9800 WLC's control path between mobility peers are encrypted by default and data path encryption is optional. As you can see in the logs when the standby becomes active control path encryption is getting disabled and thus it takes the tunnel down. 

my action plan would be;

1. wait until the HA is reached SSO state and delete the existing tunnel.

2. Create the tunnel again (make sure HA state is SSO)

3. Make sure you follow the correct steps when adding the mobility peer. I would suggest you add SSC Hash as well.

4. Additionally also check whether you have configured unique mobility MAC address per WLC.

5. In the 5520 side, under the mobilty tunnel configuration make sure that the secure mobilty is enabled and data tunnel encryption is disabled. 

6. I have not come across any issues where both 9800 and AireOS wlc is in the same management subnet and caused issues. If you doubt this could have an impact try to move 9800 management to a different subnet and test again.

7. Open a TAC case if none of the above suggestions work.

Also make sure that the WLC uplinks are correctly configured and wireless management interfaces are correctly defined. Wireless Management interface must be an SVI as well

[Scott Fella]

Great steps @Arshad Safrulla.... 

I have tested this out on my own lab.  I brought up two 9800-CL 17.7.1 put them in HA, then configured the mobility tunnels to various controllers with different code and models.  Once the tunnel was up, issue this command on the active HA and grab that mac address.

show wireless mobility summary

Make sure you follow step 5 and validate the tunnel comes up after a few minutes.  Then you can issue a force-failover and the tunnel should stay up.

[Rich R]

Thanks @Scott Fella and @Arshad Safrulla - will have a look at ours next week ...

[RoadRunner4k]

Found the issue to be a wrong configured port connecting to the Standby WLC. 

[Rich R]

Can you be more specific @RoadRunner4k - what was wrong with the config?

[Scott Fella]

@RoadRunner4k did you mean by wrong port, you had connected the service port and not the RP port?