Solved: Re: Issues moving AP onto new vWLC

chappers69 · ‎09-18-2019

Hi,

We are having to migrate all our APs from one vWLC to another vWLC (due to various issues including migrating from VMWare to Hyper-V). I'm using 1 AP as a test before moving the rest.

I've had various issues in doing this, so will describe where I'm up to now!

Old vWLC is an AIR-CTVM-K9 on 7.6.100.0

New vWLC is an AIR-CTVM-K9 on 8.5.151.0

I've upgraded the AP to ap3g2-k9w8-tar.153-3.JF10

AP has had "clear capwap private-config" carried out on it, followed by configuringit with info for the controller I want it to join:

capwap ap controller ip address 192.168.100.226

I've checked the time on both the AP and the controller, and they are within 1 second of each other.

It appears the AP is conversing with the WLC, but will not join...

Messages appearing on the AP console are (repeatedly):

*Sep 19 05:37:30.031: AP has SHA2 MIC certificate - Using SHA2 MIC certificate for DTLS.
*Sep 19 05:37:30.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 192.168.100.226 peer_port: 5246
*Sep 19 05:37:30.003: %CAPWAP-1-SSC_CERT_AUTH_FAILED: Failed to authorize controller, SSC certificate validation failed.
*Sep 19 05:37:30.239: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 192.168.100.226 peer_port: 5246
*Sep 19 05:37:30.239: %CAPWAP-5-SENDJOIN: sending Join Request to 192.168.100.226
*Sep 19 05:37:35.239: %CAPWAP-5-SENDJOIN: sending Join Request to 192.168.100.226
*Sep 19 05:38:29.999: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 192.168.100.226:5246
*Sep 19 05:38:30.015: %CAPWAP-5-AP_EASYADMIN_INFO: AP Easy Admin information - apEasyAdminConfig: there is no full image exists, continue anyway!
*Sep 19 05:38:30.015: %CAPWAP-5-AP_EASYADMIN_INFO: AP Easy Admin information - Easy Admin is not enabled, turn it off!
*Sep 19 05:38:30.027: %LWAPP-3-CLIENTERRORLOG: NumOfSlots Mismatch Reinit all Radios config rcb:0 Cfg:2

Debug on WLC are (repeatedly):

*spamApTask1: Sep 19 05:29:20.103: sshpmGetCID: called to evaluate <cscoSha2IdCert>
*spamApTask1: Sep 19 05:29:20.103: sshpmGetCID: failed to find matching cert name cscoSha2IdCert
*spamApTask1: Sep 19 05:29:20.103: GetIDCert: Using SHA2 Id cert on WLC
*spamApTask1: Sep 19 05:29:20.103: sshpmGetCID: Found matching ID cert cscoDefaultIdCert in row 2
*spamApTask1: Sep 19 05:29:20.103: Get Cert from CID: For CID 18be2cf6 certType 1
*spamApTask1: Sep 19 05:29:20.103: Get Cert from CID: Found match of ID Cert in row 2
*spamApTask1: Sep 19 05:29:20.103: sshpmGetCID: called to evaluate <cscoSha2IdCert>
*spamApTask1: Sep 19 05:29:20.103: sshpmGetCID: failed to find matching cert name cscoSha2IdCert
*spamApTask1: Sep 19 05:29:20.103: GetDERIDKey: Using SHA2 Id cert Private Keys on WLC
*spamApTask1: Sep 19 05:29:20.103: sshpmGetCID: called to evaluate <cscoDefaultIdCert>
*spamApTask1: Sep 19 05:29:20.103: sshpmGetCID: Found matching ID cert cscoDefaultIdCert in row 2
*spamApTask1: Sep 19 05:29:20.103: GetPrivateKey: called to get key for CID 18be2cf6
*spamApTask1: Sep 19 05:29:20.103: Private Key found row 2 KeyBufLen 4096 Keylen 1192 PrivateKeyPtr 0x7f3dae7c3120
*spamApTask1: Sep 19 05:29:20.340: OpenSSL Get Issuer Handles: locking ca cert table
*spamApTask1: Sep 19 05:29:20.340: OpenSSL Get Issuer Handles: x509 subject_name /C=US/ST=California/L=San Jose/O=Cisco Systems/CN=AP3G2-a46c2aed1b8a/emailAddress=support@cisco.com
*spamApTask1: Sep 19 05:29:20.340: OpenSSL Get Issuer Handles: issuer_name /O=Cisco/CN=Cisco Manufacturing CA SHA2
*spamApTask1: Sep 19 05:29:20.340: OpenSSL Get Issuer Handles: CN AP3G2-a46c2aed1b8a
*spamApTask1: Sep 19 05:29:20.340: OpenSSL Get Issuer Handles: issuerCertCN Cisco Manufacturing CA SHA2
*spamApTask1: Sep 19 05:29:20.340: GetMac: MAC: a46c.2aed.1b8a
*spamApTask1: Sep 19 05:29:20.340: OpenSSL Get Issuer Handles: openssl Mac Address in subject is a4:6c:2a:ed:1b:8a
*spamApTask1: Sep 19 05:29:20.340: OpenSSL Get Issuer Handles: Cert Name in subject is AP3G2-a46c2aed1b8a
*spamApTask1: Sep 19 05:29:20.340: OpenSSL Get Issuer Handles: Extracted cert issuer from subject name.
*spamApTask1: Sep 19 05:29:20.340: NMSP:: Algo name matched SHA256
*spamApTask1: Sep 19 05:29:20.340: OpenSSL Get Issuer Handles: Cert is issued by Cisco Systems.
*spamApTask1: Sep 19 05:29:20.340: Retrieving x509 cert for CertName cscoMfgSha2CaCert
*spamApTask1: Sep 19 05:29:20.340: sshpmGetCID: called to evaluate <cscoMfgSha2CaCert>
*spamApTask1: Sep 19 05:29:20.340: Found CID 24420324 for certname cscoMfgSha2CaCert
*spamApTask1: Sep 19 05:29:20.340: CACertTable: Found matching CID cscoMfgSha2CaCert in row 7 x509 0x7f3db1787458
*spamApTask1: Sep 19 05:29:20.340: Retrieving x509 cert for CertName cscoRootSha2CaCert
*spamApTask1: Sep 19 05:29:20.340: sshpmGetCID: called to evaluate <cscoRootSha2CaCert>
*spamApTask1: Sep 19 05:29:20.340: sshpmGetCID: Found matching CA cert cscoRootSha2CaCert in row 6
*spamApTask1: Sep 19 05:29:20.340: Found CID 268efc2d for certname cscoRootSha2CaCert
*spamApTask1: Sep 19 05:29:20.340: Verify User Certificate: X509 Cert Verification return code: 1
*spamApTask1: Sep 19 05:29:20.340: Verify User Certificate: X509 Cert Verification result text: ok
*spamApTask1: Sep 19 05:29:20.340: sshpmGetCID: called to evaluate <cscoMfgSha2CaCert>
*spamApTask1: Sep 19 05:29:20.340: sshpmGetCID: Found matching CA cert cscoMfgSha2CaCert in row 7
*spamApTask1: Sep 19 05:29:20.340: Verify User Certificate: OPENSSL X509_Verify: AP Cert Verfied Using >cscoMfgSha2CaCert<
*spamApTask1: Sep 19 05:29:20.340: OpenSSL Get Issuer Handles: Check cert validity times (allow expired NO)
*spamApTask1: Sep 19 05:29:20.340: sshpmGetCID: called to evaluate <cscoDefaultIdCert>
*spamApTask1: Sep 19 05:29:20.340: sshpmGetCID: Found matching ID cert cscoDefaultIdCert in row 2
*spamApTask1: Sep 19 05:29:20.340: sshpmFreePublicKeyHandle: called with 0x7f3da7c08838
*spamApTask1: Sep 19 05:29:20.340: sshpmFreePublicKeyHandle: freeing public key
*sshpmLscTask: Sep 19 05:29:29.856: sshpmLscTask: LSC Task received a message 4
*spamApTask1: Sep 19 05:30:30.128: sshpmGetCID: called to evaluate <cscoSha2IdCert>

Does anyone know why the AP won't join the WLC, what little hair I have left is gradually being removed!

Many thanks.

Rasika Nayanajith · ‎09-19-2019

Did you activate licences on your vWLC (Management -> Software Activation -> Licences -> Adder Licences (add ap count & click add)

"show license summary" output should verify it.

HTH

Rasika

*** Pls rate all useful responses ***

View solution in original post

Sandeep Choudhary · ‎09-18-2019

Hi,

Try to disable the hash by using the command: config certificate ssc hash validation disable

Regards

Dont forget to rate helpful posts

chappers69 · ‎09-18-2019

Thanks for reply...

Sorry, I forgot to mention, this is already disabled:

SSC Hash validation.............................. Disabled.

Thanks.

Sandeep Choudhary · ‎09-18-2019

Also try this method:

1. https://support.cloudmylab.com/portal/kb/articles/ap-not-registering-to-vwlc

if still not works then chekc this bug and its workaround:

2. https://bst.cloudapps.cisco.com/bugsearch/bug/CSCva69352/?rfs=iqvred

Regards

Dont forget to rate helpful posts

chappers69 · ‎09-19-2019

Thanks Sandeep, really appreciate the assistance, however it still won't work..

I even tried adding the new controller into a mobility group, which looked to be promising, but the status of the new controller is "Control and Data Path Down", even though the devices can ping each other.

The AP console messages are still showing:

%CAPWAP-1-SSC_CERT_AUTH_FAILED: Failed to authorize controller, SSC certificate validation failed.Peer Certificate verification failed FFFFFFFF

%CAPWAP-3-ERRORLOG: Certificate verification failed!

DTLS_CLIENT_ERROR: ../capwap/base_capwap/capwap/base_capwap_wtp_dtls.c:467 Certificate verified failed!

Is there something I can do on the AP to maybe import the cert from the new controller, as it seems to be this it doesn't like?

Best regards.

Jurgens L · ‎09-18-2019

I've had the same issue, did you try:

config ap cert-expiry-ignore {mic|ssc} enable

chappers69 · ‎09-19-2019

Thanks for the tip Jurgens, I've just tried this and unfortunately hasn't changed anything. I'm still getting the error messages I just mentioned in my reply to Sandeep.

The AP is quite happy to join the old controller still, but not the new.

Best regards.

Rasika Nayanajith · ‎09-19-2019

Did you activate licences on your vWLC (Management -> Software Activation -> Licences -> Adder Licences (add ap count & click add)

"show license summary" output should verify it.

HTH

Rasika

*** Pls rate all useful responses ***

chappers69 · ‎09-19-2019

Thanks Rasika, this appears to have been the final part of the puzzle. Thanks also Sandeep, I believe this was also in one of the article you sent a link through to.

Best regards.