Anchor WLC 9800CL and Foreigns WLC 9800 LF

[JesusSeijas]

Hello There,

I'm in a new deployment with an Anchor 9800CL, where the mobility tunnels aren't fully establishing. I'm detecting the following errors on the WLC foreign.

 038011: Dec 1 23:25:16.095: %MM_NODE_LOG-4-DTLS_HANDSHAKE_FAIL: Chassis 1 R0/0: mobilityd: Mobility DTLS Ctrl handshake failed for IP: 192.168.254.22 HB is down, need to re-initiate DTLS handshake
038012: Dec 1 23:25:28.095: %MM_NODE_LOG-4-DTLS_HANDSHAKE_FAIL: Chassis 1 R0/0: mobilityd: Mobility DTLS Ctrl handshake failed for IP: 192.168.254.22 HB is down, need to re-initiate DTLS handshake
038013: Dec 1 23:25:37.096: %DTLS_TRACE_MSG-3-X509_CERT_VERIFY_ERR: Chassis 1 R0/0: mobilityd: Cert verify Error, Session:192.168.254.22[16666], Certificate hash is invalid
038014: Dec 1 23:25:37.096: %DTLS_TRACE_MSG-3-WLC_DTLS_ERR: Chassis 1 R0/0: mobilityd: DTLS Error, session:192.168.254.22[16666], Certificate validation failed

Status of the tunnel from Foreign WLC

Mobility Peer Info
===================
Ip Address : 192.168.254.22
Public Ip Address : 192.168.254.22
MAC Address : 11e.bdee.afff
Group Name : ANCHOR-WLC
Total Number of Clients on Peer : 0
Local Clients Exported to Peer : 0
Locally Anchored Peer Clients : 0
Data Link Encryption Status : Disabled
Keepalive Data Link Status : UP
Keepalive Control Link Status : UP
DTLS Data Link Cipher : TLS_NUM_NULL_WITH_NULL_NULL
DTLS Data Link Status : Disabled
DTLS Control Link Cipher : TLS_NUM_NULL_WITH_NULL_NULL
DTLS Control Link Status : Disabled
PMTU : 1385
Tunnel Plumbed : Yes
Tunnel IFID : 0xA0000B98
Number of Data Path Flaps : 0
Last Data Path Flap : Never

Status of the tunnel from Anchor WLC

Mobility Peer Info
===================
Ip Address : 172.16.131.180
Public Ip Address : 172.16.131.180
MAC Address : 18c6.5021.ce2f
Group Name : FOREIGN-WLC
Total Number of Clients on Peer : 446
Local Clients Exported to Peer : 0
Locally Anchored Peer Clients : 0
Data Link Encryption Status : Disabled
Keepalive Data Link Status : UP
Keepalive Control Link Status : UP
DTLS Data Link Cipher : TLS_NUM_NULL_WITH_NULL_NULL
DTLS Data Link Status : Disabled
DTLS Control Link Cipher : TLS_NUM_NULL_WITH_NULL_NULL
DTLS Control Link Status : Init
PMTU : 1385
Tunnel Plumbed : Yes

WMI trustpoint Anchor WLC

Anchor-WLC#show wire mana trustpoint
Trustpoint Name : Anchor-WLC_WLC_TP
Certificate Info : Available
Certificate Type : SSC
Certificate Hash : d1354d53a3d70a541b7c5e3097356e802f2eaa5d
Private key Info : Available
FIPS suitability : Not Applicable

WMI trustpoint Foreign WLC

Foreign-WLC01-V#show wire mana trustpoint
Trustpoint Name : CISCO_IDEVID_CMCA3_SUDI
Certificate Info : Available
Certificate Type : MIC
Certificate Hash : a35bde1f47bcce757a9911007b3634e6ac75dca3
Private key Info : Available
FIPS suitability : Not Applicable

Any idea @Mark Elsen @Scott Fella ?

Thanks

[Mark Elsen]

  - @JesusSeijas      Use show wireless management trustpoint and show crypto pki trustpoints
                  commands to verify your certificate information  on the  WLC 9800CL
 
                               Validate the complete configuration of this controller using the CLI command
                               show tech wireless and feed the output from that into Wireless Config Analyzer
                                  Use the full command as outlined in green; it does not work with show tech-support

  M.
                                   

[JesusSeijas]

Hello,

Thanks @Mark Elsen 

WMI trustpoint Anchor WLC (9800 CL)

Anchor-WLC#show wire mana trustpoint
Trustpoint Name : Anchor-WLC_WLC_TP
Certificate Info : Available
Certificate Type : SSC
Certificate Hash : d1354d53a3d70a541b7c5e3097356e802f2eaa5d
Private key Info : Available
FIPS suitability : Not Applicable

Anchor-WLC# show crypto pki trustpoints
Trustpoint SLA-TrustPoint:
Subject Name:
cn=Cisco Licensing Root CA
o=Cisco
Serial Number (hex): 01
Certificate configured.

Trustpoint TP-self-signed-2627895496:
Subject Name:
hostname=Anchor-WLC.dcdomain.com
cn=IOS-Self-Signed-Certificate-2627895496
Serial Number (hex): 01
Persistent self-signed certificate trust point
Using key label TP-self-signed-2627895496

Trustpoint WLC_CA:
Subject Name:
o=Cisco Virtual Wireless LAN Controller
cn=CA-vWLC_Anchor-WLC
Serial Number (hex): 01
Certificate configured.

Trustpoint Anchor-WLC_WLC_TP:
Subject Name:
o=Cisco Virtual Wireless LAN Controller
cn=CA-vWLC_Anchor-WLC
Serial Number (hex): 01
Certificate configured.
SCEP URL: http://192.168.254.22:80/cgi-bin

I did some debugs in Foreign WLC (9800 LF), and detect the possible issue. Maybe is it necessary to upload the Anchor 9800CL certificate to the foreign 9800LF?

[mm-client] [16941]: (debug): MAC: 0000.0000.0000 Sending keepalive_ctrl_rsp of XID (132) to (ipv4: 192.168.254.22 )
[mm-keepalive] [16941]: (note): Peer IP: 192.168.254.22 Control link set state to UP (was DOWN)
[errmsg] [16941]: (note): %MM_NODE_LOG-5-KEEP_ALIVE: R0/0: mobilityd: Mobility Control tunnel to peer IP: 192.168.254.22 changed state to UP
ap-upgrade] [15872]: (note): Process mobility tunnel Up message : Upgrade rec not found. Report name:
[mm-dtls] [16941]: (note): Peer IP: 192.168.254.22 Port: 16666 DTLS_CLEAR_KEY: DTLS keys cleared from MNC and FMAN
[mm-dtls] [16941]: (note): Peer IP: 192.168.254.22 Port: 16666, Local IP: 172.16.131.180 Port: 16666 DTLS_CLOSE_CB: DTLS connection is closed
[ewlc-dtls-sess] [16941]: (info): release client sm resource
[ewlc-dtls-sess] [16941]: (note): Remote Host: 192.168.254.22[16666] DTLS session destroy
[mm-client] [16941]: (debug): MAC: 001e.bdee.afff Received keepalive_data, sub type: 0 of XID (0) from (ipv4: 192.168.254.22 )
[mm-dtls] [16941]: (debug): Peer IP: 192.168.254.22 Port: 16667 MM_KA_DTLS_START: DTLS not supported

Thanks

[Mark Elsen]

   @JesusSeijas      Validate the complete configuration both controllers using the CLI command
                               show tech wireless and feed the output from that into Wireless Config Analyzer
                              Use the full command as outlined in green; it does not work with show tech-support

                              + Make sure to configure a unique mobility mac address
                              +  Check output from         show wireless mobility summary

  M.

[JesusSeijas · Accepted solution]

Hello @Mark Elsen ,

Yes, I ran WLCCA for both WLC's. Detect the following situation in the Foreign (9800LF)

DTLS_TRACE_MSG-3-X509_CERT_VERIFY_ERR Times Seen 376
First Seen 039900: Dec 2 05:34:38.090:
Last Seen 042646: Dec 2 13:22:39.243:
Text Chassis 1 R0/0: mobilityd: Cert verify Error, Session:192.168.254.22[16666], Certificate hash is invalid

DTLS_TRACE_MSG-3-WLC_DTLS_ERR Times Seen 376
First Seen 039901: Dec 2 05:34:38.090:
Last Seen 042647: Dec 2 13:22:39.243:
Text Chassis 1 R0/0: mobilityd: DTLS Error, session:192.168.254.22[16666], Certificate validation failed

In the Anchor (9800CL), the state of the DTLS Control Link Status is Init, and not showing errors in the WLCCA.

I solved the situation usign the same SSC hash that generated by the 9800CL to stablish HASHING for both WLCs. Will this behavior be required when using mobility tunnels with a CLOUD-type anchor?

[Mark Elsen]

   - @JesusSeijas           Yes , I think so

  M.

[balaji.bandi]

is this NAT 1:1 static NAT ?

Make sure you configure external IP on the anchor configuration. and try packet capture.

https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/221707-configure-9800-wireless-lan-controller-m.html#toc-hId--1386946523

[JesusSeijas]

Hello @balaji.bandi 

Yes, it's the same than the internal, it's that to say we're not applying NAT into the WLC and later on, because this communication can be established using private ip's. 

I solved the situation usign the same SSC hash that generated by the 9800CL to stablish HASHING for both WLCs. Will this behavior be required when using mobility tunnels with a CLOUD-type anchor?

Thanks

[balaji.bandi]

Both sides always need the same HASH configuration if you're using that.

Could you check the packet capture to see what is wrong? Once the configuration is verified?

[balaji.bandi]

Are these both on the same campus? Do you have any FW in between?

What code is running on both WLCs?

Check the config and port requirement :

https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/213912-configure-mobility-anchor-on-catalyst-98.html

depends on version check bug :

https://bst.cisco.com/quickview/bug/CSCwe60294

[JesusSeijas]

Hello @balaji.bandi 

What code is running on both WLCs?

Anchor WLC (9800CL) --> 17.15.4b

Foreign WLC (9800LF) --> 17.9.6

There are in different locations. Yes, we've different firewalls in the middle in this communication, it's permitted, keepalives communications in both WLC are OK.

Anchor WLC (9800CL)

Keepalive Data Link Status : UP
Keepalive Control Link Status : UP
DTLS Data Link Cipher : TLS_NUM_NULL_WITH_NULL_NULL
DTLS Data Link Status : Disabled
DTLS Control Link Cipher : TLS_NUM_NULL_WITH_NULL_NULL
DTLS Control Link Status : Init
PMTU : 1385
Tunnel Plumbed : Yes

Foreign WLC (9800LF)

Keepalive Data Link Status : UP
Keepalive Control Link Status : UP
DTLS Data Link Cipher : TLS_NUM_NULL_WITH_NULL_NULL
DTLS Data Link Status : Disabled
DTLS Control Link Cipher : TLS_NUM_NULL_WITH_NULL_NULL
DTLS Control Link Status : Disabled
PMTU : 1385
Tunnel Plumbed : Yes

I did some debugs in Foreign WLC (9800 LF), and detect the possible issue. Maybe is it necessary to upload the Anchor 9800CL certificate to the foreign 9800LF?

[mm-client] [16941]: (debug): MAC: 0000.0000.0000 Sending keepalive_ctrl_rsp of XID (132) to (ipv4: 192.168.254.22 )
[mm-keepalive] [16941]: (note): Peer IP: 192.168.254.22 Control link set state to UP (was DOWN)
[errmsg] [16941]: (note): %MM_NODE_LOG-5-KEEP_ALIVE: R0/0: mobilityd: Mobility Control tunnel to peer IP: 192.168.254.22 changed state to UP
ap-upgrade] [15872]: (note): Process mobility tunnel Up message : Upgrade rec not found. Report name:
[mm-dtls] [16941]: (note): Peer IP: 192.168.254.22 Port: 16666 DTLS_CLEAR_KEY: DTLS keys cleared from MNC and FMAN
[mm-dtls] [16941]: (note): Peer IP: 192.168.254.22 Port: 16666, Local IP: 172.16.131.180 Port: 16666 DTLS_CLOSE_CB: DTLS connection is closed
[ewlc-dtls-sess] [16941]: (info): release client sm resource
[ewlc-dtls-sess] [16941]: (note): Remote Host: 192.168.254.22[16666] DTLS session destroy
[mm-client] [16941]: (debug): MAC: 001e.bdee.afff Received keepalive_data, sub type: 0 of XID (0) from (ipv4: 192.168.254.22 )
[mm-dtls] [16941]: (debug): Peer IP: 192.168.254.22 Port: 16667 MM_KA_DTLS_START: DTLS not supported

Thanks