AIR-AP1572EAC-A-K9 access point stuck in downloading

[rakesh nair]

HI Team,

I have an issue with our network. We are using WLC8540 with version 8.5.161.0 and APs with model AIR-AP1572EAC-A-K9.

Two of the APs which were working earlier is now showing as Downloading and stuck there.

We did a hard reboot of APs , still no change.

The ips is reachable, but not able to SSH.

Can anyone help here.

[rakesh nair]

I managed to SSH to the AP and this is the status.

[Flavio Miranda]

Hi

  I would console to the AP and try to clear up capwap config. 

clear capwap ap all

[rakesh nair]

AP is placed on top of a pole and is difficult to remove. We have SSH access to the AP, but this command is not working in it

[Leo Laohoo]

FN - 72524 - During Software Upgrade/Downgrade, Cisco IOS APs Might Remain in Downloading State After December 4, 2022 Due to Certificate Expiration

(1572 is the same hardware as the 2700/3700/IW3700.)

[rakesh nair]

Hi leo,

Can you suggest how to disable and manually configure NTP in WLC

Workaround

For any WLC that has APs stuck in the downloading state:

• Disable Network Time Protocol (NTP) on the WLC and manually set the WLC date/time to a date before December 2, 2022. The Cisco IOS AP will then be able to download and validate the image, install the new image, and join the controller. Once the AP has joined the controller, NTP can be re-enabled on the controller to assume the correct date and time.

[Rich R]

Well go into your WLC GUI and change the NTP settings! Controller -> NTP -> Server, remove them, then Commands - Set Time
If you don't know how to do that then READ the documentation:
https://www.cisco.com/c/en/us/support/docs/wireless/4400-series-wireless-lan-controllers/113334-ntp-wlc-config-00.html
https://www.cisco.com/c/en/us/td/docs/wireless/controller/8-5/config-guide/b_cg85/network_time_protocol_setup.html

But really you need to upgrade to 8.5.182.7 or 8.10.185.0 (depending on what APs you're supporting) as per the links below to resolve the issue.

[rakesh nair]

I have applied the work around that you provided , but it's not helping, also AP is now showing not joined

[Scott Fella]

Look at the logs or provide the logs so that others can help.  The output from the ap will tell you why the ap is not joining.

[Leo Laohoo]

---

@rakesh nair wrote:
I have applied the work around that you provided , but it's not helping, also AP is now showing not joined

---

Tell us, exactly, what was done. 

[Rich R]

> I have applied the work around that you provided
As the others have said describe EXACTLY what you did?
And provide the full logs (attach as a text file not a screenshot) on the AP from power on which will show why it doesn't join?

What you should be doing at a minimum is upgrade to 8.5.182.7 - have you done that?
Have you read all the field notices linked below?

[rakesh nair]

The AP is placed on top of a pole in airport and console to AP is very difficult. We need to rent out a a lift to do that.

Also upgrade of WLC which is currently in production in a shot notice, that also for two APs is very difficult to explain to management and grab approval.

AP was able to join WLC, but was stuck in downloading was the issue. but now it is not able to join. In WLC i am seeing below error.

*osapiBsnTimer: Apr 09 09:41:09.033: %DTLS-3-HANDSHAKE_FAILURE: [PA]openssl_dtls.c:3231 Failed to complete DTLS handshake with peer 10.24.200.139

[Leo Laohoo]

Is it possible to SSH into the AP? 

I have 3700s in my network and my workaround does not follow the official Cisco FN:  

1. SSH into the AP
2. debug capwap console cli
3. delete /f /r flash:c1570*
4. archive tar /x tftp://<TFTP IP address>/c1570-rcvk9w8-tar.153-3.JPP.tar flash:
5. Once the upload is finish, reboot the AP.

When the AP reboots, it will boot an recovery IOS-XE version on 17.11.1.  The AP will then join the controller and download the correct firmware.

[rakesh nair]

What i did, first removed NTP server from WLC, then manually set time to Nov 2022, then i applied below command in WLC

config ap cert-expiry-ignore mic enable

config ap cert-expiry-ignore ssc enable

[Rich R]

Then you probably have not read ALL the field notices below!

Nov 2022 will *only* help you with the Dec 2022 image cert bug (CSCwd80290) not the MIC certificate expiry.  For that you need the commands above and WLC date set to before the AP and/or WLC MICs expired.  Read the field notice (FN-63942) carefully - twice if need be - then follow the instructions, in the right order, carefully.
The problem with that one is the cert-expiry-ignore config will not be applied to the AP until after it has been able to join the WLC and complete download if required.

If that doesn't resolve your issue then you'll have to get on the AP console somehow to understand the problem better.  Sometimes a packet capture of the CAPWAP (UDP 5246) between AP and WLC might show you what's happening.  Consider staging a replacement AP then swapping the problem one to avoid hassle of trying to troubleshoot at height.  Then fix that one at your desk and use it to replace the next one.

As to upgrade of WLC - well that's why it pays to keep on top of field notices and required configurations and keep your software up to date pro-actively.  For now you have to solve the problem whichever way works best for you in the situation.