Mobility link down after update 17.3.3 on 9800-CL

[Frank Benders]

After updating wlc 9800-CL (17.3.1) to 17.3.3 the mobility link with wlc Aireos 8.5.164.0 went down. Trying to rebuild it failed.

Any ideas?

[marce1000]

 - How does it fail (which error messages are observed) -> And or check the logs of both controllers, when trying to rebuild.

 M.

[Frank Benders]

Thanks for replying:

You're right: always check your log files...

After updating wlc 9800-CL (17.3.1) to 17.3.3 the mobility link with wlc Aireos 8.5.164.0 went down. Trying to rebuild it failed.

Errors repeatedly on a 5508 wlc:

2021-03-10T10:31:42.858177+01:00 err 5508wlc-01 wlc-01: *mobilityCapwapSocketTask: Mar 10 10:31:42.921: %DTLS2-3-HANDSHAKE_FAILURE: dtls2.c:1502 DTLS handshake failed for link xxx.xx.xxx.244:16666 <-> xxx.xx.xxx.250:16666
2021-03-10T10:31:42.646138+01:00 err 5508wlc-01 wlc-01: *mobilityCapwapSocketTask: Mar 10 10:31:42.707: %SSHPM-3-GENERIC_CERT_ERROR: sshpmPkiApi.c:2243 Certificate validation failed! Reason , Certificate type : MIC, Certificate issuer :Other
2021-03-10T10:31:42.646138+01:00 err 5508wlc-01 wlc-01: *mobilityCapwapSocketTask: Mar 10 10:31:42.707: %SSHPM-3-UNKNOWN_CERT_ISSUER: sshpmPkiApi.c:2022 Invalid AP certificate. Issuer unknown

Errors on the 9800-CL wlc:

Mar 11 09:39:49.370: %DTLS_TRACE_MSG-3-WLC_DTLS_ERR: Chassis 1 R0/0: mobilityd: DTLS Error, session:xxx.xx.xxx.244[16666], Certificate validation failed

Mar 11 09:39:49.370: %CERT_MGR_ERRMSG-3-CERT_VALIDATION_ERR: Chassis 1 R0/0: mobilityd: Certificate Validation Error, Cert validation status:pki_ssl_status@pki_ssl_status:PKI_SSL_ERROR

Mar 11 09:39:49.368: %PKI-3-CERTIFICATE_INVALID_EXPIRED: Certificate chain validation has failed. The certificate (SN: 2AEEACF9000000139ADE) has expired. Validity period ended on 2020-11-30T11:27:53Z

Mar 11 09:39:38.421: %MM_INFRA_LOG-3-RECV_FAILED: Chassis 1 R0/0: mobilityd: Unable to receive mobility message aplist_update from ipv4: xxx.xx.xxx.244 . reason: Peer link is down

The problem is in the Validity period of the certificate. Should be nice to have a workaround for this.

In november last year this command: config ap cert-expiry-ignore mic enabled
AP's are returning to 5508wlc-01.

Must be the security part of mobility path which, I believe, is mandatory on the 9800-series.

[marce1000 · Accepted solution]

 - Check if the resolving-reply from this thread can help :

            https://community.cisco.com/t5/wireless/inter-release-controller-mobility-ircm-with-5508-fail-control/td-p/4273720

 M.

[Frank Benders]

Thanks for this reply and your time, followed these steps:

9800-CL#conf t
Enter configuration commands, one per line. End with CNTL/Z.
9800-CL(config)#crypto pki certificate map map1 1
9800-CL(ca-certificate-map)#issuer-name co Cisco Manufacturing CA
9800-CL(ca-certificate-map)#exit
9800-CL(config)#crypto pki trustpool policy
9800-CL(ca-trustpool)#match certificate map1 allow expired-certificate
9800-CL(ca-trustpool)#end
9800-CL#

next try rebuilding mobility-path:

Errors repeatedly on a 5508 wlc:

2021-03-15T10:24:47.062173+01:00 err 5508wlc-01 wlc-01: *mobilityCapwapSocketTask: Mar 15 10:24:47.104: %DTLS2-3-HANDSHAKE_FAILURE: dtls2.c:1502 DTLS handshake failed for link xxx.xx.xxx.244:16666 <-> xxx.xx.xxx.250:16666
2021-03-15T10:24:47.062044+01:00 err 5508wlc-01 wlc-01: *mobilityCapwapSocketTask: Mar 15 10:24:47.103: %SSHPM-3-GENERIC_CERT_ERROR: sshpmPkiApi.c:2243 Certificate validation failed! Reason , Certificate type : MIC, Certificate issuer :Other
2021-03-15T10:24:47.061849+01:00 err 5508wlc-01 wlc-01: *mobilityCapwapSocketTask: Mar 15 10:24:47.103: %SSHPM-3-UNKNOWN_CERT_ISSUER: sshpmPkiApi.c:2022 Invalid AP certificate. Issuer unknown

Errors on the 9800-CL wlc:

Mar 15 09:21:24.716: %MM_INFRA_LOG-3-RECV_FAILED: Chassis 1 R0/0: mobilityd: Unable to receive mobility message aplist_update from ipv4: xxx.xx.xxx.244 . reason: Peer link is down
Mar 15 09:20:47.036: %MM_NODE_LOG-5-KEEP_ALIVE: Chassis 1 R0/0: mobilityd: Mobility Control tunnel to peer IP: xxx.xx.xxx.244 changed state to UP

Link still down. But no certificate message anymore.

Kind regards.

[marce1000]

 - Try to reboot the 5508 , check if that helps , if not try to upgrade to the latest 8.5.164.x version available for the 5508

 M.

[Frank Benders]

Thanks for your help. I rebuild the mobility path again and now it works. Didn't have to reboot the controller.

Used this manual:https://www.cisco.com/c/en/us/td/docs/wireless/controller/8-9/config-guide/b_cg89/encrypted_mobility_tunnel.html

[j.rambeau]

I had the same issue and solved the problem thanks to the documentation. Solution is: if you are running a 9800-CL version, don't forget to configure the 9800 SSC Hash on the AireOS controller:

config mobility group member hash peer-ip-addr 40-digit-ssc-hash-key

Note	SSC hash is needed on for peers that do not use a MIC certificate. For example: Cisco Catalyst 9800-CL Wireless Controllers.

[j.rambeau]

deleted message