APs not Joining Mobility Express Controller

[mumbles202]

A few years ago I did a deployment with 3 1832is (converted all 3 APs to Mobility Express at the time so any of them could be the controller in the event of an issue).  Coverage was fine and everything worked well for some time, but recently 2 of the 3 APs won't rejoin the controller.  I can see them in CDP on the connected switch, confirmed the firmware version matches that of the controller (as it should since they were all on the same system), but they boot, seem to be waiting for a response from at controller, then reboot and keep cycling like this.  All 3 APs are on the same subnet and dhcp is available ( I haven't tried setting option yet but I can try that as well).  

I was considering going onsite and pulling one of the APs and trying to get console access while it boots to see if there is anything that can be gathered from it.  Unfortunately it reboots too quickly so by the time I see it online and I'm able to ssh to it it's just about ready to reboot again.  

Anything else I can try prior to going onsite and pulling these down?  I should be able to get debugs from the controller during a reboot if that would be helpful.

[Mark Elsen]

 - You will need to have a console infrastructure connected to one of those that can keep logs too, to further get an overall picture on this syndrome,

 M.

[jagan.chowdam]

Check and verify:

• Time and Date
• Certificate expiration - https://community.cisco.com/t5/wireless-mobility-knowledge-base/lightweight-ap-fail-to-create-capwap-lwapp-connection-due-to/ta-p/3155111

Jagan Chowdam

/**Pls rate useful responses**/

[Leo Laohoo]

What is the firmware of the WLC?

[Rich R]

I'd say most likely that it's hardware failure.  1832 have a much higher failure rate than other (even older) AP models.  We RMA at least 20 of them each month (often more).

You will need to get on the console to confirm what's happening.  It's worth making sure the software version is up to date (refer to TAC recommended link below) but since 1 is still working it's probably not the software.  You can also do a factory default reset on the 2 problem APs but most likely the conclusion will be hardware replacement.

[mumbles202]

Thanks for the post.  Yes, I've seen a couple 1832s that have weird issues (either booting up fully and all the menus working fine but none of the radios working or not booting up to a running state at all).  I'm seeing the device show up in CDP on the upstream switch so it's at least booting up to a point. 

I'll try to get console access to the 2 failed units next week and see what I can find.  Anything in particular that should be gathered while on the console?

[Scott Fella]

From experience, sometimes the ap just needs to be reset using the mode button.  If that doesn't work, then like what the others have mentioned, you might need to RMA them.

Step 1: Disconnect power (the power jack for external power or the Ethernet cable for in-line power) from the access point.
Step 2: Press and hold the MODE button while you reconnect power to the access point.
Step 3: Hold the MODE button until the Status LED turns amber (approximately 10 to 12 seconds), until the LED changes to red and release the Button. All access point settings return to factory defaults.

[Rich R]

After resetting as Scott explained collect the full console log from power on.
Sometimes the cause will be obvious (messages like radio not found etc) and sometimes not.
Sometimes you can see on packet capture (which you should be able to do remotely) that the discovery packet the AP sends to the controller is missing the board data (invalid Message Element)  which tells you the radios have failed, and in this case the AP never progresses to sending a join request.

Working:

Broken:

[mumbles202]

Thanks for the replies.  I'll try to get access to the APs (or at least the upstream switch) this week and gather those.

[Scott Fella]

@mumbles202 sounds good, keep us posted.

[mumbles202]

Would all of the above still apply to the 1852i?  Should be able to get access to them today to troubleshoot. 

[Scott Fella]

I don't see why not. Give it a try so its something you can document for your team in the future.