Aironet 1800 Series APs not joining following firmware update

[mark.bell]

I've got something strange going on with two Cisco Aironet 1852is. Were previously working just fine. Updated the other day to the latest available firmware and ever since then, both APs will not show up in the Web GUI for Mobility Express at the same time. Both of the APs are in Mobility Express mode. They are in the same VLAN and same subnet. Both appear to keep operating just fine and users are able to connect to the wireless SSIDs in all areas of our building still. Whichever AP I restart last will not show up in the Web GUI and repeatedly does discovery requests, which are denied by the AP stating it's unable to find the MAC address of the freshly-reloaded AP.

Also: 

JICL-AP2 keeps repeating this to my Syslog server:

CAPWAP State: DTLS Teardown
upgrade.sh: Script called with args:[ABORT]
do ABORT, part1 is active part
upgrade.sh Cleanup tmp files

Every now and then the APs will both show up briefly in the GUI, only for one to disappear. The names of the APs are JICL-AP1 (192.168.1.8) and JICL-AP2 (192.168.1.10). JICL-AP1 is showing set as preferred master and primary. For some reason, AP2 also shows up as a primary, but there is no way in the CLI when SSHing into JICL-AP2 to change it away from being a primary. I've tried putting JICL-AP2 into CAPWAP mode and then converting back to Mobility Express and that has no effect on the behavior.  

I've already tried rolling back the firmware to the previous version but no dice there. 

[Leo Laohoo]

Console into the AP and reboot.  Post the entire boot-up process so we can see what is going on.  

Just a word of caution:  Cisco 1800 are very "prone" to hardware failures.  Emphasis on the word "prone".  

[Rich R]

Sounds to me like the 2 APs are running different versions of software.

How did you upgrade them?
Did you upgrade 1 and let the other download from TFTP server (need to setup TFTP server with new software files unzipped and configure TFTP server details on the ME WLC which you upgraded.  Subordinate AP is instructed to download the update from TFTP server)
Or did you upgrade both of them individually?
Exactly what version did you upgrade (them) to?  Can you do a "show sysinfo" (or show ver at AP CLI) on both of them to confirm the software versions?
As Leo says a *complete* console log from power-on on both APs should tell us all we need to know.
Don't paste in here - save the logs to <filename>.txt and attach the files.

[mark.bell]

Appreciate the quick replies. I will get answers to questions when I return to work tomorrow. I do know the firmware I upgraded to was 8.10.185.0. I upgraded using TFTP method in the Mobility Express Web GUI. One download covered both APs. Thanks again.

[mark.bell]

Good morning! This morning I decided to reset both APs after converting both back to CAPWAP. I then set one up as ME Controller. I had the same behavior. The second AP would attempt to join the first AP, but then ultimately fail. I decided to downgrade the AP set as ME controller to a much earlier version (8.8.130.0). The second AP picked up the image and both were displaying correctly in the ME Web GUI again. I'm now attempting to upgrade them both to the latest recommended image, which is 8.10.151.0. Will report back once the upgrade is complete. 

[Scott Fella]

Good luck... keep us posted.

[mark.bell]

Hey guys. So the behavior gets triggered in anything past version 8.10.105.0. That's the most recent firmware I can set it to before it starts breaking things. So I can't even get the latest recommended release. Good thing is that this version at least allows for WPA3 enablement, which is good enough on my end. Here is my model number: AIR-AP1852I-B-K9 (These are 1852I models).
If you're interested in looking at logs still, I can pull those for you guys. 

The release table goes in this order, by the way:

8.8.130.0 - works for me
8.10.105.0 - works for me
8.10.130.0 - doesn't work for me
8.10.151.0 - doesn't work
8.10.183.0 - doesn't work
8.10.185.0 - doesn't work

[Rich R]

That's remarkable! I can't believe you're the only person in the world using ME with 8.10!  I think there must be something about your setup that triggers this!