01-08-2018 10:35 PM - edited 07-05-2021 08:03 AM
Hi everyone,
I have a couple of 3702 AP's which have started rebooting a few days ago and are not joining the WLC (8510 running 8.0.150.0). They were fine before.
MAX number AP supported is 6000, and 3402 are currently joined.
In the few seconds they stay up, I can SSH them and the last logs I see before they reload are:
*Jan 9 05:21:08.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.x.x.x peer_port: 5246
*Jan 9 05:21:08.215: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 10.x.x.x peer_port: 5246
*Jan 9 05:21:08.215: %CAPWAP-5-SENDJOIN: sending Join Request to 10.x.x.x
AP's and WLC are not in the same VLAN but routing is ok, no FW between them.
As stated above, the other 3402 AP's are fine and can re-join the WLC after a reboot or a port bounce.
I can get some join info from the WLC for one of these AP's, but not for the other.
(Cisco Controller) >show ap join stats detailed 64:12:25:4b:xx:xx
Sync phase statistics
- Time at sync request received............................ Not applicable
- Time at sync completed................................... Not applicable
Discovery phase statistics
- Discovery requests received.............................. 100
- Successful discovery responses sent...................... 0
- Unsuccessful discovery request processing................ 100
- Reason for last unsuccessful discovery attempt........... Missing AP Manager interface
- Time at last successful discovery attempt................ Not applicable
- Time at last unsuccessful discovery attempt.............. Jan 09 15:54:37.931
Join phase statistics
- Join requests received................................... 0
- Successful join responses sent........................... 0
- Unsuccessful join request processing..................... 0
- Reason for last unsuccessful join attempt................ Not applicable
- Time at last successful join attempt..................... Not applicable
- Time at last unsuccessful join attempt................... Not applicable
Configuration phase statistics
- Configuration requests received.......................... 0
- Successful configuration responses sent.................. 0
- Unsuccessful configuration request processing............ 0
- Reason for last unsuccessful configuration attempt....... Not applicable
- Time at last successful configuration attempt............ Not applicable
- Time at last unsuccessful configuration attempt.......... Not applicable
Last AP message decryption failure details
- Reason for last message decryption failure............... Not applicable
Last AP disconnect details
- Reason for last AP connection failure.................... Not applicable
- Last AP disconnect reason................................ Not applicable
Last join error summary
- Type of error that occurred last......................... Lwapp discovery request rejected
- Reason for error that occurred last...................... Missing AP Manager interface
- Time at which the last join error occurred............... Jan 09 15:54:37.931
AP disconnect details
- Reason for last AP connection failure.................... Not applicable
Ethernet Mac : 00:00:00:00:00:00 Ip Address : 0.0.0.0
I searched for "Missing AP Manager interface" around but I couldn't find anything helpful.
The only thing I noticed is that both AP's come up with a different software, 15.3(3)JA10 and recovery 15.2(4)JB1) while others run 15.3(3)JA11.
AP running 15.3(3)JA10 is the one for which I can collect the above join info.
Do you have any idea on why they are not joining or booting the right image?
Is there a way to force booting a specific image?
These AP's are remote and I'd avoid any risky procedure unless strictly necessary.
THANKS!
fed
Solved! Go to Solution.
01-17-2018 04:41 PM
Hi Leo,
In the end I opened a TAC case and solution for both of them was:
- Connect to AP via SSH.
- Run:
debug capwap console cli debug capwap client no-reload format flash:
Once format completed both AP's re-joined the WLC, downloaded the right image and then rebooted.
At the end I had to reconfigure them but at least I didn't have to have them removed from the ceiling, etc..
Thanks for your help anyway.
Cheers
Fed
01-09-2018 12:25 AM
Hi,
Paste the output of these commands:
from WLC: sh sysinfo
from AP: sh version
Also paste the comnplete bootup process logs from AP console.
Regards
Dont forget to rate helpful posts
01-09-2018 03:19 PM - edited 01-09-2018 03:45 PM
Hi Sandeep,
Thanks for replying. See attached. AP's are remote and I'm unable to get the console output.
As soon as they are reachable via SSH though I can connect to them and see the logs, also attached.
Unfortunately AP 2 this morning is not coming up at all, I wonder if there's a reload limit after which the AP simply stops trying.. UPDATE: I left its switch interface shut for a while and then AP came back up but keeps reloading.
Cheers
Fed
01-09-2018 01:54 AM
In the list of APs, are the affected APs displaying the following characteristics:
1. AP name has reverted back to default of APmacaddress;
2. Primary/Secondary/Tertiary field are empty;
3. AP Group is back to default-group;
4. IP address is displayed as 0.0.0.0;
5. In the list of APs, the AP in question is displaying a firmware version of 0.0.0.0 and constantly downloading; and/or
6. It doesn't "power up". The output to the command "sh power inline" only shows as "Ieee".
01-09-2018 03:44 PM
Hi Leo,
Thanks for replying.
1. AP name has reverted back to default of APmacaddress;
>>> No, they have both the correct hostname.
2. Primary/Secondary/Tertiary field are empty;
>>> Output of "sh capwap client config" shows Primary for both AP's, as expected. Sec/Tert are not populated.
3. AP Group is back to default-group;
>>> Output of "sh capwap client config" shows correct group for both AP's.
4. IP address is displayed as 0.0.0.0;
>>> Not sure where I should see this.
5. In the list of APs, the AP in question is displaying a firmware version of 0.0.0.0 and constantly downloading;
>>> Not sure where I should see this.
and/or
6. It doesn't "power up". The output to the command "sh power inline" only shows as "Ieee".
>>> Ieee PD is displayed when AP reloads but as soon as CDP is received then AP model is shown. This happens for both AP's.
Cheers
Fed
01-09-2018 03:45 PM
@Feds wrote:
4. IP address is displayed as 0.0.0.0;
>>> Not sure where I should see this.
5. In the list of APs, the AP in question is displaying a firmware version of 0.0.0.0 and constantly downloading;>>> Not sure where I should see this.
Controller GUI.
01-09-2018 03:47 PM - edited 01-09-2018 03:52 PM
Because both AP's are not joined to WLC I can't find them there.
Only thing I can see on WLC for AP 1 only (running 15.3(3)JA10) is what I pasted on the first message (output of show ap join stats detailed <AP 1 MAC addr>).
01-09-2018 03:56 PM - edited 01-09-2018 03:58 PM
So you can remote into the AP, right? Delete the IOS folder they're currently running.
debup capwap console cli del /f /r flash:/ap3g2-k9w8-mx.153-3.JA10
And then reboot the AP.
See if the AP joins a controller or not.
01-09-2018 04:06 PM
AP 1 stays reachable for 28 seconds so I should have enough time to do it. However is there another way to force it booting a specific image? What if once JA10 is removed it tries to boot the recovery image 15.2xxxx instead of the correct 15.3(3)JA11?
01-09-2018 04:30 PM
01-17-2018 04:41 PM
Hi Leo,
In the end I opened a TAC case and solution for both of them was:
- Connect to AP via SSH.
- Run:
debug capwap console cli debug capwap client no-reload format flash:
Once format completed both AP's re-joined the WLC, downloaded the right image and then rebooted.
At the end I had to reconfigure them but at least I didn't have to have them removed from the ceiling, etc..
Thanks for your help anyway.
Cheers
Fed
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide