Cisco Wlc2504 failure

[alfred.vesely]

Hi,

after a power loss, I often experienced that APs will not rejoin the controller due to false date and time. However, this time the problem isnt been solved by setting the correct time and I‘m getting the following error messages with no AP joining the controller anymore:

*sntpMainTask: Dec 08 18:28:51.556: #OSAPI-3-MSGQ_RUNNING_HIGH: osapi_msgq.c:874 Message queue SNTP-Q is nearing full. Capacity 3 Messages 3.
*osapiBsnTimer: Dec 08 18:28:35.612: #DTLS-3-HANDSHAKE_FAILURE: openssl_dtls.c:2962 Failed to complete DTLS handshake with peer 192.168.2.6
*osapiBsnTimer: Dec 08 18:28:34.412: #DTLS-3-HANDSHAKE_FAILURE: openssl_dtls.c:2962 Failed to complete DTLS handshake with peer 192.168.2.7
*osapiBsnTimer: Dec 08 18:28:12.812: #DTLS-3-HANDSHAKE_FAILURE: openssl_dtls.c:2962 Failed to complete DTLS handshake with peer 192.168.2.4
*sntpMainTask: Dec 08 18:28:01.536: #OSAPI-3-MSGQ_RUNNING_HIGH: osapi_msgq.c:874 Message queue SNTP-Q is nearing full. Capacity 3 Messages 3.
*sntpMainTask: Dec 08 18:27:31.524: #OSAPI-3-MSGQ_RUNNING_HIGH: osapi_msgq.c:874 Message queue SNTP-Q is nearing full. Capacity 3 Messages 3.
*sntpMainTask: Dec 08 18:27:21.520: #OSAPI-3-MSGQ_RUNNING_HIGH: osapi_msgq.c:874 Message queue SNTP-Q is nearing full. Capacity 3 Messages 3.
*osapiBsnTimer: Dec 08 18:27:19.612: #DTLS-3-HANDSHAKE_FAILURE: openssl_dtls.c:2962 Failed to complete DTLS handshake with peer 192.168.2.5
*sntpMainTask: Dec 08 18:26:31.500: #OSAPI-3-MSGQ_RUNNING_HIGH: osapi_msgq.c:874 Message queue SNTP-Q is nearing full. Capacity 3 Messages 3.

Any ideas how to fix? Thank you in advance and best regards, Alfred

[Flavio Miranda]

@alfred.vesely 

 what is the WLC  version and AP model and version?

The error message " Failed to complete DTLS handshake with" can often be related to certificate problem and depending on the WLC and AP model/version could be fixing by ignoring certificate on the capwap tunnel establishment with the command

config ap cert-expiry-ignore mic enable

[alfred.vesely]

WLC is running on 8.0.100.0, APs are 3702 und 1702, running on the same versions I guess

Can that command be set via GUI as well?

[marce1000]

 - The SNTP-Q messages seem to be related to a bug : https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvf33449
    In that context , use the latest/last release for the 2504 controller : https://software.cisco.com/download/specialrelease/9a6a7cf84f9fdf04b95c76e2ac7820e7

  APs . no longer joining could be due to https://www.cisco.com/c/en/us/support/docs/field-notices/639/fn63942.html
  In this case the reverse may be applicable for a simple test disable NTP and do set the controller back in time , reboot it and check if the APs can join. If so , the Field Notice does apply ,

 M.

[alfred.vesely]

Thank you! Setting the WLC back in time helped! Unfortunately, I dont have a service contract, so I guess there will be no chance to get a working firmware?

[marce1000]

 @alfred.vesely wrote : I dont have a service contract, so I guess there will be no chance to get a working firmware?
                                   No unfortunately not ; if the controller is on 8.3 and above the workaround for expired certificates mentioned by @Flavio Miranda can be used :  config ap cert-expiry-ignore mic enable
                                 config ap cert-expiry-ignore ssc enable

(ignore the font and colors change; apparently due to some cut and pasting from the FN)
 But you won't have a fix for the bug.

 Since the aireos platforms are getting EOL , using last/latest release would have been better , because there is no support any more from cisco after that release.
 You may also start looking into the 9800 platform(s); you can download and deploy the virtual controller for free for testing and to get familiar with the platform. For use APs must be licensed (not the 9800 VM)

 M.

[alfred.vesely]

Sorry for that question, but do you think it would be a good option to use a 3802 in mobility express as a controller as alternative to the 2504?

[marce1000]

 - SInce it stays in the same older technology (mobility express) I am not in favor of it ; you must also be able to download software for it , if needed 
    Also always check compatibilities using ; https://www.cisco.com/c/en/us/td/docs/wireless/compatibility/matrix/compatibility-matrix.html  when switching platforms.

   For the time being I would stick with the 2504 (with time set backwards), the SNTP-Q error is probably not that important.
   Untill a migration to 9800 controller(s) becomes possible,

 M.

[alfred.vesely]

Somehow sad, I really liked the WLC familiy - changing to the 9800 family doesnt make sense for a 7AP enviroment for me, so sadly I will replace the system with another manufacturer

[Rich R · Accepted solution]

Find a recent security advisory that affects 8.5 code and find the section which says "Customers without Contracts" then contact TAC quoting the URL of the advisory, the paragraph just mentioned and the version and URL https://software.cisco.com/download/specialrelease/9a6a7cf84f9fdf04b95c76e2ac7820e7 for the software you want to download and serial number of your WLC.  You'll have to mention which platform you need it for (2504) because they have all WLC models there at that URL.  Then TAC should publish it to you directly.

This advisory should be suitable: Cisco Wireless LAN Controller AireOS Software FIPS Mode Denial of Service Vulnerability because CSCwa40778 : Bug Search Tool (cisco.com) is fixed in 8.5.182.12. (even though the advisory itself says upgrade to 8.10)

"Customers Without Service Contracts

Customers who purchase directly from Cisco but do not hold a Cisco service contract and customers who make purchases through third-party vendors but are unsuccessful in obtaining fixed software through their point of sale should obtain upgrades by contacting the Cisco TAC: https://www.cisco.com/c/en/us/support/web/tsd-cisco-worldwide-contacts.html

Customers should have the product serial number available and be prepared to provide the URL of this advisory as evidence of entitlement to a free upgrade."

[MHM Cisco World]

Do you face before issue of AP joining WLC?

I think you before use workaround for cert. Expire by change date to be few years older' and now since wlc is use correct date the wlc is detect cert. Expire and AP can not join wlc.

MHM

[MHM Cisco World]

https://www.wiresandwi.fi/blog/cisco-wlc-or-ap-device-certificate-expired-what-you-can-do

MHM