topic 1530AP boot loop - recovery? in Wireless

1530AP boot loop - recovery?

jmorrison_bcp — Sat, 28 May 2022 18:50:01 GMT

1530AP is stuck in a boot loop after changing high availability settings to point to a new controller running 8.3.150.0

The old controller is still online (version 8.0.152.0). Did the same thing with APs including 1530s and they are all registered, upgraded and working.

The 1530AP shows up on the new controller with ip address 0.0.0.0 even though it has an ip address.

It won't upgrade and won't go back to the old controller. On the WLC I get AP Disassociated. Base Radio MAC x:x:x:x ApName - APblahblah

Packet captures show it doing discovery, and it seems to negotiate capwap/dtls with the new controller.

The AP syslogs show:

%CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: x.x.x peer_port: 5246
%CAPWAP-5-SENDJOIN: sending Join Request to x.x.x.x
%SYS-5-RELOAD: Reload requested by CAPWAP CLIENT. Reload Reason: Reload.

Edit: fixed it by putting on the same vlan as the old controller and adding an ACL on the AP switch port blocking access to the new controller. The AP then registered back on the old controller.

Re: 1530AP boot loop - recovery?

Flavio Miranda — Sat, 28 May 2022 19:08:28 GMT

1530 is outdoor AP. They need to be permited on the WLC. SECURITY > AP Policy

Add the AP to the authorization list.

Re: 1530AP boot loop - recovery?

jmorrison_bcp — Sat, 28 May 2022 22:03:08 GMT

The AP MIC is already set in AP policy on the new controller.

Re: 1530AP boot loop - recovery?

Rich R — Sun, 29 May 2022 13:40:05 GMT

Could be a corrupt flash problem preventing it from downloading the new IOS from the new WLC.

https://www.cisco.com/c/en/us/support/docs/wireless-mobility/wireless-lan-wlan/213317-understanding-various-ap-ios-flash-corru.html

https://www.cisco.com/c/en/us/support/docs/field-notices/703/fn70330.html

You can try upgrading the IOS yourself manually after checking the flash.

https://software.cisco.com/download/home/284620606/type/280775090/release/15.3.3-JD17

Re: 1530AP boot loop - recovery?

jmorrison_bcp — Sun, 29 May 2022 19:55:23 GMT

Looks like the new AP code is indeed corrupt.

1530 is registered to old WLC and I enabled SSH

From ap:

#archive upload-sw /version ap1g3-k9w8-mx.153-3.JD17 ftp://x.x.x.x/test.tar

With bash cli:

tar xf test.tar
cd ap1g3-k9w8-mx.153-3.JD17
$ sha512sum -c < file_hashes
CA2.bin: OK
CA5.bin: OK
CO2.bin: OK
CO5.bin: OK
RO2.bin: OK
RO5.bin: OK
ap1g3-k9w8-mx.153-3.JD17: FAILED
img_sign_rel.cert: OK
img_sign_rel_sha2.cert: OK
info: OK
sha512sum: WARNING: 1 computed checksum did NOT match

To recap, recovery required:
1. create an ACL on switch port so the AP could not talk to the new WLC. This broke the boot loop and let it revert to old WLC
2. When the AP finally reverted to the old WLC, I enabled SSH
3. log in to AP, rename corrupt file to ap1g3-k9w8-mx.153-3.JD17.bad to make sure re-downloading didn't hit the same bad spot in flash
4. copy good file, delete bad file
5. reconfigure for new WLC

Re: 1530AP boot loop - recovery?

Rich R — Sun, 29 May 2022 21:02:50 GMT

Did you fsck the flash? (read the links I shared)

If not it will just keep causing more problems for that AP until you do!

You're lucky you only had 1 AP with the problem. When upgrading from an affected version of code you should be checking every AP as per the field notice - simply prevents problems like this.