AP's not joining WLC and rebooting

[Feds]

Hi everyone,

I have a couple of 3702 AP's which have started rebooting a few days ago and are not joining the WLC (8510 running 8.0.150.0). They were fine before.
MAX number AP supported is 6000, and 3402 are currently joined.
In the few seconds they stay up, I can SSH them and the last logs I see before they reload are:


*Jan  9 05:21:08.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.x.x.x peer_port: 5246
*Jan  9 05:21:08.215: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 10.x.x.x peer_port: 5246
*Jan  9 05:21:08.215: %CAPWAP-5-SENDJOIN: sending Join Request to 10.x.x.x


AP's and WLC are not in the same VLAN but routing is ok, no FW between them.
As stated above, the other 3402 AP's are fine and can re-join the WLC after a reboot or a port bounce.
I can get some join info from the WLC for one of these AP's, but not for the other.


(Cisco Controller) >show ap join stats detailed 64:12:25:4b:xx:xx

Sync phase statistics
- Time at sync request received............................ Not applicable
- Time at sync completed................................... Not applicable

Discovery phase statistics
- Discovery requests received.............................. 100
- Successful discovery responses sent...................... 0
- Unsuccessful discovery request processing................ 100
- Reason for last unsuccessful discovery attempt........... Missing AP Manager interface
- Time at last successful discovery attempt................ Not applicable
- Time at last unsuccessful discovery attempt.............. Jan 09 15:54:37.931

Join phase statistics
- Join requests received................................... 0
- Successful join responses sent........................... 0
- Unsuccessful join request processing..................... 0
- Reason for last unsuccessful join attempt................ Not applicable
- Time at last successful join attempt..................... Not applicable
- Time at last unsuccessful join attempt................... Not applicable

Configuration phase statistics
- Configuration requests received.......................... 0
- Successful configuration responses sent.................. 0
- Unsuccessful configuration request processing............ 0
- Reason for last unsuccessful configuration attempt....... Not applicable
- Time at last successful configuration attempt............ Not applicable
- Time at last unsuccessful configuration attempt.......... Not applicable

Last AP message decryption failure details
- Reason for last message decryption failure............... Not applicable

Last AP disconnect details
- Reason for last AP connection failure.................... Not applicable
- Last AP disconnect reason................................ Not applicable

Last join error summary
- Type of error that occurred last......................... Lwapp discovery request rejected
- Reason for error that occurred last...................... Missing AP Manager interface
- Time at which the last join error occurred............... Jan 09 15:54:37.931

AP disconnect details
- Reason for last AP connection failure.................... Not applicable
                                                                           Ethernet Mac : 00:00:00:00:00:00  Ip Address : 0.0.0.0

I searched for "Missing AP Manager interface" around but I couldn't find anything helpful.
The only thing I noticed is that both AP's come up with a different software, 15.3(3)JA10 and recovery 15.2(4)JB1) while others run 15.3(3)JA11.
AP running 15.3(3)JA10 is the one for which I can collect the above join info.

Do you have any idea on why they are not joining or booting the right image?
Is there a way to force booting a specific image?

These AP's are remote and I'd avoid any risky procedure unless strictly necessary.

THANKS!
fed

[Sandeep Choudhary]

Hi,

Paste the output of these commands:

from WLC: sh sysinfo

from AP: sh version

Also paste the comnplete bootup process logs from AP console.

Regards

Dont forget to rate helpful posts

[Feds]

Hi Sandeep,

Thanks for replying. See attached. AP's are remote and I'm unable to get the console output.

As soon as they are reachable via SSH though I can connect to them and see the logs, also attached.

Unfortunately AP 2 this morning is not coming up at all, I wonder if there's a reload limit after which the AP simply stops trying.. UPDATE: I left its switch interface shut for a while and then AP came back up but keeps reloading.

Cheers
Fed

[Leo Laohoo]

In the list of APs, are the affected APs displaying the following characteristics: 
1. AP name has reverted back to default of APmacaddress; 
2. Primary/Secondary/Tertiary field are empty; 
3. AP Group is back to default-group; 
4. IP address is displayed as 0.0.0.0; 
5. In the list of APs, the AP in question is displaying a firmware version of 0.0.0.0 and constantly downloading; and/or
6. It doesn't "power up". The output to the command "sh power inline" only shows as "Ieee".

[Feds]

Hi Leo,

Thanks for replying.

1. AP name has reverted back to default of APmacaddress;

>>> No, they have both the correct hostname.

2. Primary/Secondary/Tertiary field are empty;

>>> Output of "sh capwap client config" shows Primary for both AP's, as expected. Sec/Tert are not populated.

3. AP Group is back to default-group;

>>> Output of "sh capwap client config" shows correct group for both AP's.

4. IP address is displayed as 0.0.0.0; 
>>> Not sure where I should see this.

5. In the list of APs, the AP in question is displaying a firmware version of 0.0.0.0 and constantly downloading;

>>> Not sure where I should see this.

and/or
6. It doesn't "power up". The output to the command "sh power inline" only shows as "Ieee".

>>> Ieee PD is displayed when AP reloads but as soon as CDP is received then AP model is shown. This happens for both AP's.

Cheers

Fed

[Leo Laohoo]

---

@Feds wrote:

4. IP address is displayed as 0.0.0.0; 
>>> Not sure where I should see this.

5. In the list of APs, the AP in question is displaying a firmware version of 0.0.0.0 and constantly downloading;

>>> Not sure where I should see this.

---

Controller GUI. 

[Feds]

Because both AP's are not joined to WLC I can't find them there.

Only thing I can see on WLC for AP 1 only (running 15.3(3)JA10) is what I pasted on the first message (output of show ap join stats detailed <AP 1 MAC addr>).

[Leo Laohoo]

So you can remote into the AP, right? Delete the IOS folder they're currently running.

debup capwap console cli
del /f /r flash:/ap3g2-k9w8-mx.153-3.JA10

And then reboot the AP.
See if the AP joins a controller or not.

[Feds]

AP 1 stays reachable for 28 seconds so I should have enough time to do it. However is there another way to force it booting a specific image? What if once JA10 is removed it tries to boot the recovery image 15.2xxxx instead of the correct 15.3(3)JA11?

[Leo Laohoo]

Won't make any difference. I just want to see if forcing the AP to boot a different IOS can make a difference.

[Feds · Accepted solution]

Hi Leo,

In the end I opened a TAC case and solution for both of them was:

- Connect to AP via SSH.

- Run:

debug capwap console cli
debug capwap client no-reload
format flash:

Once format completed both AP's re-joined the WLC, downloaded the right image and then rebooted.

At the end I had to reconfigure them but at least I didn't have to have them removed from the ceiling, etc..

Thanks for your help anyway.

Cheers
Fed