Cisco AP AIR-AP2802E-E-K9 Join Issues with WLC 8500

[Faresnani]

Dear Community,

We are experiencing an issue with over 50 of our APs (model: AIR-AP2802E-E-K9) suddenly stopping appearing in the WLC (model: 8500 Series). These APs previously functioned correctly within our environment but are no longer showing up in the controller.

Please note the following:

• We do not have access to the APs' CLI, as the credentials are unavailable so we are unable to get some logs. We would be grateful if anyone could provide the default credentials so we can try
• These APs are located approximately 340 kilometers away from our office, so accessing the console directly is not an option.
• In terms of reachability, the Wireless LAN Controller (WLC-8500) can reach the access points (APs), with DHCP functioning correctly, and both Option 43 and DNS configurations in place.

(Cisco Controller) >ping 10.128.91.1

Send count=3, Receive count=3 from 10.128.91.1

• WLC-8500 EOL and EOS we can not open case support with Cisco
• We attempted to add APs to our new WLC 9800 as a workaround, and while they are appearing successfully, we currently do not have sufficient licenses to support the number of APs.

We would greatly appreciate any assistance or guidance in resolving this issue. Below are some relevant logs and system information for reference:

Debug Logs from WLC (debug capwap error enable)

*spamApTask3: Oct 15 16:04:42.694: [PA] 00:5d:73:e6:db:e0 Could not find image version of bundled AP(apType: 55)!!!
*spamApTask3: Oct 15 16:04:42.694: [PA] 00:5d:73:e6:db:e0 Unable to get AP Bundled Version. Using Controller Version!!!

*spamApTask6: Oct 15 16:04:42.764: [PA] 28:ac:9e:73:58:40 DTLS connection was closed
*spamApTask7: Oct 15 16:04:42.783: [PA] 28:ac:9e:73:58:40 DTLS connection was closed
u*spamApTask6: Oct 15 16:04:42.828: [PA] 28:ac:9e:73:58:40 DTLS connection was closed
*spamApTask0: Oct 15 16:04:42.846: [PA] 00:5d:73:e6:56:40 ApModel: AIR-AP2802E-E-K9

*spamApTask0: Oct 15 16:04:42.846: [PA] 00:5d:73:e6:56:40 Could not find image version of bundled AP(apType: 55)!!!
*spamApTask0: Oct 15 16:04:42.846: [PA] 00:5d:73:e6:56:40 Unable to get AP Bundled Version. Using Controller Version!!!

----------------------------------------------------------------------------------------------------------------

WLC System Information (show sysinfo)

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 8.3.150.0
RTOS Version..................................... 8.3.150.0
Bootloader Version............................... 7.5.102.0
Emergency Image Version.......................... 7.5.102.0

OUI File Update Time............................. Sun Sep 07 10:44:07 IST 2014

Build Type....................................... DATA + WPS

System Name...................................... SAU-AKH-DC-WLC-1
System Location..................................
System Contact...................................
System ObjectID.................................. 1.3.6.1.4.1.9.1.1615
Redundancy Mode.................................. SSO
IP Address....................................... 10.98.216.10
IPv6 Address..................................... ::
System Up Time................................... 172 days 4 hrs 16 mins 1 secs
System Timezone Location.........................
System Stats Realtime Interval................... 5

--More-- or (q)uit
System Stats Normal Interval..................... 180

Configured Country............................... SA - Saudi Arabia
Operating Environment............................ Commercial (10 to 35 C)
Internal Temp Alarm Limits....................... 10 to 38 C
Internal Temperature............................. +17 C
Fan Status....................................... OK

RAID Volume Status
Drive 0.......................................... Good
Drive 1.......................................... Good

State of 802.11b Network......................... Enabled
State of 802.11a Network......................... Enabled
Number of WLANs.................................. 15
Number of Active Clients......................... 1803

OUI Classification Failure Count................. 108716591

Burned-in MAC Address............................ F8:72:EA:67:3C:80
Power Supply 1................................... Present, OK
Power Supply 2................................... Present, OK
Maximum number of APs supported.................. 6000

--More-- or (q)uit
System Nas-Id.................................... SAU-AKH-DC-WLC-1
WLC MIC Certificate Types........................ SHA1
Licensing Type................................... RTU

---------------------------------------------------------------------------------------------------------------

(Cisco Controller) >show license summary

Feature name: ap_count (base)
License type: Permanent
License state: Active, In-use
RTU License Count: 1000

Feature name: ap_count
License type: Evaluation
License Eula: Not Accepted
Evaluation total period: 12 weeks 6 days
License state: Inactive, Not-In-Use
RTU License Count: 6000

Feature name: ap_count (adder)
License type: Permanent
License state: Active, In-use
RTU License Count: 400

[marce1000]

           >....the command is already applied but no issue still persists 
  - Currently I would also advice a reload of the WLC first , and check the situation afterwards ,

 M.

[Faresnani]

thank you for the advice, and we concerned about rebooting the WLC, is there any other option,

Can you come online session with us to troubleshoot? I will appreciate that @marce1000 

[Faresnani]

we also notice on our monitoring system Zabbix this below log 

" WLC Device has been replaced (new serial number received)"

[marce1000]

           >...we also notice on our monitoring system Zabbix this below log  (new serial number received)
  - Could be indicative of resource exhaustion too and Zabbix receiving garbage ; also pointing to
             1) WLC reboot , 

 M.
   

[marce1000]

   >...thank you for the advice, and we concerned about rebooting the WLC, is there any other option,
   There are not much alternatives , including a (the) upgrade afterwards , if it does not help.
   I think you have an 8510  , software to be found in https://software.cisco.com/download/specialrelease/9a6a7cf84f9fdf04b95c76e2ac7820e7
    If an upgrade would be done (strongly advised) , then use the certificate ignoring commands again

               >Can you come online session with us to troubleshoot?  
     It's not something I do , in all honesty , but in this case it wouldn't'  help ; because you have already
     provided all the needed output (such as an AP boot)
     The next phase is 'daring steps' (which are needed) ; see beginning of my reply , 
                  1) WLC reboot
                  2) If a no go ==> WLC upgrade

    There is one remaining thing which could be done first  , is checking the configuration of  the WLC 
     using WirelessAnalyzer input (procedure) for AireOs controllers
    and feed  the output from that into Wireless Config Analyzer
           

 M.

[Faresnani · Accepted solution]

@jagan.chowdam @marce1000 @Flavio Miranda 

Thank you for your support 

The issue has been resolved

• We identify that certificate expiry issue on the WLC.
• We checked which certificate is used by WLC for establishing DTLS connection with APs and the cert was "Cisco SHA1 device cert"  The certificate on the WLC was valid till 13th Aug 2024 as below:

Certificate Name: Cisco SHA1 device cert

--More-- or (q)uit

Subject Name :
C=US, ST=California, L=San Jose, O=Cisco Systems, CN=AIR-CT8510-K9-f872ea673c80, emailAddress=support@cisco.com
Issuer Name :
O=Cisco Systems, CN=Cisco Manufacturing CA
Serial Number (Hex):
76AAC0ED000000053C77
Validity :
Start : Aug 13 15:41:13 2014 GMT
End : Aug 13 15:51:13 2024 GMT
Signature Algorithm :
sha1WithRSAEncryption
Hash key :
SHA1 Fingerprint : 2f:b2:1d:e1:97:d6:30:59:7e:8e:ed:30:aa:2c:c8:28:42:b2:2c:5a
SHA256 Fingerprint : 82:2f:d7:ba:10:3f:8b:b2:44:80:82:f4:e2:87:09:cc:c4:8c:0d:22:ee:bb:63:1d:ee:a2:88:4a:2c:28:d6:93

• We removed the NTP server and reverted the WLC time to Aug 1st, 2024.
• The APs started to join the WLC.
• We readded the NTP server. the APs were stable on the WLC, and clients connected.
• The behavior is documented for AP-COS APs in Cisco bug ID CSCvb93909 which has fix in AireOS 8.5 and later.

WLC with the issue persisting:

WLC After Resolving the issue:

[marce1000]

                         >...The issue has been resolved
   Good work! ; remember to push your management and higher to invest further in the 9800 based solutions
   and compatible APs! (and not wait to long....)

  M.

[Flavio Miranda]

@Faresnani 

 Problem with certificate between WLC and AP happens when the AP is really old, which is not the case of the 2800. I dont believe you need to worry about it.

As I said before, if the AP is able to join the 9800, which is a newer device, it should be able to join the 8500. In my opinion, the problem is on the 8500 side.

If you did not any change on the WLC that could explain the issue, the alternative here is first reload the WLC and if not fix, upgrade.

[jagan.chowdam]

Please check if this option- Accept Local Significant Certificate (LSC)
enabled under Security --> Certificates --> LSC. If yes, please disable it, reboot AP and test.

LSC certificates are solely used by APs that need to prove their identity to the WLC. They do not exist by default on neither the WLC nor the APs. The LSC certificates need to be signed by a CA and later installed on both the WLC and the APs to mutually validate each other.

What about the APs which are already registered with the controller? Can you reboot one of the, if you have console access - then log the AP boot process and share. 

Jagan Chowdam

/**Pls rate useful responses**/

[jagan.chowdam]

Can you also share "show ap join stats detailed <AP MAC>" command output. 

8.3.150.0 is older version. I would also upgrade the code to Cisco recommended AirOS version (make sure the code supports all Access Point models).

Refer this link to get the current recommended AirOS version.

https://www.cisco.com/c/en/us/support/docs/wireless/wireless-lan-controller-software/200046-tac-recommended-aireos.html#toc-hId-1746850508

AP-AisOS code compatibility matrix:

https://www.cisco.com/c/en/us/td/docs/wireless/compatibility/matrix/compatibility-matrix.html

The default login credentials for the AP are: Cisco/Cisco. However, you can use Telnet/SSH only if the AP has joined the WLC and is in the RUN state.

Jagan Chowdam

/**Pls rate useful responses**/

[Faresnani]

@jagan.chowdam  Thank you for your prompt response. We will consider an upgrade following a thorough risk assessment of the current situation. Additionally, your point regarding access to the AP via SSH/Telnet is well-noted

(Cisco Controller) >show ap join stats detailed 683b.78d5.3854
No join information found for AP: 68:3b:78:d5:38:54

[marce1000]

  - Actually take care with my suggested version  , because there is difference between which exact controller model
  you are using , when using the last supported release , look at :
                         https://www.cisco.com/c/en/us/support/docs/wireless/wireless-lan-controller-software/200046-tac-recommended-aireos.html

                        For correct information , ==>per controller model

 M

[Rich R]

You didn't actually mention the WLC model (8500 series is not the model) but I can see from your screenshots that it's 8510 (show inventory will show you that)
Therefore you should be upgrading the 8510 to the latest available code version 8.5.182.12 (link below).
And the certificates issue is fully documented on field notice FN63942 (link below).
To permanently resolve the issue you need to follow all the steps in the field notice after upgrading the code otherwise your APs will be stranded again when they reboot or lose connection to the WLC.