AP registered to 9800 can't join a 5508 with expired certificate.

[Tob]

9800-L-F running 17.3.2a

5508 running 8.5.140

AP: AIR-AP2802I-E-K9

Hello,

To begin with I'll supply a bit of history:

Around a month ago we noticed that the certificate of the currently active controller (5508) had expired and as a result when AP:s rebooted they could not register to the controller. To fix this we followed the contents of this link: https://www.cisco.com/c/en/us/support/docs/field-notices/639/fn63942.html and disabled the certificate check.

Now to the problem at hand:

We are currently in the process of changing controller to a 9800 and are getting to the point where we feel comfortable to switch over our production AP:s, so we started testing with a 2800 AP which were registered to the 5508 and joined it to the 9800 which went perfectly fine. The problem occurred when we tried to simulate a rollback via configuring the primary controller under high availability to be the 5508. The 2800 wont register to the 5508, it wont even try and join.

The logs on the AP tell me that it does find the 5508, both through DNS and manual configuration, when I check the logs on the 5508 I do see the discovery requests from the AP, I do not however see any join request at all. After checking the logs from the AP a bit more carefully I think that the issue is that the AP ignores the 5508 because it has an expired certificate:
Apr 27 07:44:59 kernel: [*04/27/2021 07:44:59.0003] CAPWAP State: DTLS Setup
Apr 27 07:44:59 kernel: [*04/27/2021 07:44:59.0010] dtls_connectionDB_add_connection: Number of DTLS connections exceeded two
Apr 27 07:44:59 kernel: [*04/27/2021 07:44:59.4873] display_verify_cert_status: Verify Cert: FAILED at 0 depth: certificate has expired
Apr 27 07:44:59 kernel: [*04/27/2021 07:44:59.4995] dtls_verify_con_cert: Controller certificate verification error
Apr 27 07:44:59 kernel: [*04/27/2021 07:44:59.4996] dtls_process_packet: Controller certificate verification failed
Apr 27 07:44:59 kernel: [*04/27/2021 07:44:59.5001] sendPacketToDtls: DTLS: Closing connection 0x559ed200.
Apr 27 07:44:59 kernel: [*04/27/2021 07:44:59.5002] Restarting CAPWAP State Machine.

The reason why disabling the certificate check is no longer active is likely because the AP sits on the 17.3.2 code where the check is not disabled. Do note that the 8.5.140 image is still in the AP:s storage.

My initial thought was, ok I have to disable the certificate validation check on the 9800, but I'm not sure if that is possible. This leaves me in an awkward position as the rollback is in jeopardy. Is it possible as on aireos to disable the certificate validation check on the 9800 controller or do I have to figure out another method of doing the rollback, maybe it's possible to manually force the AP to use the 8.5.140 code via a CLI command from the 9800?

[patoberli]

I wonder if you hit a downgrading issue with the very old 8.5.140.0.... You could try to first upgrade that to the latest version (which also fixes some issues in conjunction with the 9800 WLC): 

https://www.cisco.com/c/en/us/td/docs/wireless/controller/release/notes/crn85mr7_ircm.html

[Tob · Accepted solution]

There was no need to disable the certificate check, I just had to use the "SWAP" button (Configuration > Wireless > Access Points > %AP-of-interest% > Advanced > AP Image Management > "SWAP") on the AP in the GUI of the 9800, change the primary WLC on the high availability tab to the older 5508 (also under AP configuration in the GUI) and reboot the AP to get it to rejoin the 5508. The SWAP button seems to do exactly what the AP CLI CMD "configure boot path 1/2" does. Of importance under the circumstances of the issue I had is that the older WLC image must match the backup image of the AP.