AP3802 Trouble Join to WLC

[fdharmawan]

Hi Guys,

Recently one of my AP3802 on the deployment suddenly went boot loop. When I console the AP, the following message constantly showing up.

[*09/14/2019 03:32:04.1960] Discovery Response from "WLC_IP"
[*09/14/2019 03:32:04.1960] Discovery response from MWAR ''running version 0.0.0.0 is rejected.
[*09/14/2019 03:32:04.1960] Failed to decode discovery response(status = 4).
[*09/14/2019 03:32:04.1961] CAPWAP SM handler: Failed to process message type 2 state 2.
[*09/14/2019 03:32:04.1961] Failed to handle capwap control message from controller - status 4
[*09/14/2019 03:32:04.1961] Failed to process unencrypted capwap packet 0x2115000 from "WLC_IP"
[*09/14/2019 03:32:04.1961] Failed to send message to CAPWAP state machine, msgId 0
[*09/14/2019 03:32:04.1962] Failed to send capwap message 0 to the state machine. Packet already freed.
[*09/14/2019 03:32:04.1962] IPv4 wtpProcessPacketFromSocket returned 4

The WLC was running with 8.5.151.0. Any of you guys experiencing this? Thank you.

[balaji.bandi]

I believe you are hitting with bug - worth checking if possible raise an TAC for remidiation.

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvf71074/?rfs=iqvred

[fdharmawan]

Hi Balaji,

Thanks for the reference. But unfortunately the bug page does not state the solution or workaround. Some of the known fixed release images are even below my current version, which is 8.5.151.0.

And for the TAC support, unfortunately I do not have that luxury right now.

[Leo Laohoo]

Look at the time and date of the AP logs.
Post the complete output to the following commands:
1. WLC: sh sysinfo;
2. WLC: sh time;
3. AP: sh version; and
4. AP: sh ip interface brief

[fdharmawan]

Hi Leo,

Below are the output you asked.

1. WLC: sh sysinfo;

(Cisco Controller) >show sysinfo

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 8.5.151.0
RTOS Version..................................... 8.5.151.0
Bootloader Version............................... 8.3.15.177
Emergency Image Version.......................... 8.2.166.0

OUI File Last Update Time........................ Sun Sep 07 10:44:07 IST 2014

Build Type....................................... DATA + WPS

System Name...................................... "Masked System Name"
System Location.................................. "Masked Location"
System Contact...................................
System ObjectID.................................. 1.3.6.1.4.1.9.1.2170
Redundancy Mode.................................. SSO
IP Address....................................... "Masked WLC IP"
IPv6 Address..................................... ::
System Up Time................................... 128 days 22 hrs 35 mins 37 secs
System Timezone Location......................... (GMT +7:00) Saigon, Hanoi, Bangkok, Jakatar

System Stats Realtime Interval................... 5
System Stats Normal Interval..................... 180

Configured Country............................... ID - Indonesia
Operating Environment............................ Commercial (10 to 35 C)
Internal Temp Alarm Limits....................... 10 to 38 C
Internal Temperature............................. +24 C
Fan Status....................................... OK

State of 802.11b Network......................... Enabled
State of 802.11a Network......................... Enabled
Number of WLANs.................................. 16
Number of Active Clients......................... 1657

OUI Classification Failure Count................. 630948

Memory Current Usage............................. 10
Memory Average Usage............................. 10
CPU Current Usage................................ 0
CPU Average Usage................................ 0

Flash Type....................................... Compact Flash Card
Flash Size....................................... 1073741824

Burned-in MAC Address............................ "Masked MAC address"
Power Supply 1................................... Present, OK
Power Supply 2................................... Present, OK
Maximum number of APs supported.................. 1500
System Nas-Id....................................
WLC MIC Certificate Types........................ SHA1/SHA2
Licensing Type................................... RTU

2. WLC: sh time;

(Cisco Controller) >show time

Time............................................. Tue Jan 21 09:03:58 2020

Timezone delta................................... 0:0
Timezone location................................ (GMT +7:00) Saigon, Hanoi, Bangkok, Jakatar

NTP Servers
NTP Version.................................. 3
NTP Polling Interval......................... 600

Index NTP Key Index NTP Server Status NTP Msg Auth Status
------- ---------------------------------------------------------------------

3. AP: sh version

>sh version
Restricted Rights Legend

Use, duplication, or disclosure by the Government is subject to
restrictions as set forth in subparagraph (c) of the Commercial
Computer Software - Restricted Rights clause at FAR sec. 52.227-19 and
subparagraph (c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

Cisco Systems, Inc.
170 West Tasman Drive
San Jose, California 95134-1706

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
If you require further assistance please contact us by sending email to
export@cisco.com.

This product contains some software licensed under the
"GNU General Public License, version 2" provided with
ABSOLUTELY NO WARRANTY under the terms of
"GNU General Public License, version 2", available here:
http://www.gnu.org/licenses/old-licenses/gpl-2.0.html

This product contains some software licensed under the
"GNU Library General Public License, version 2" provided
with ABSOLUTELY NO WARRANTY under the terms of "GNU Library
General Public License, version 2", available here:
http://www.gnu.org/licenses/old-licenses/lgpl-2.0.html

This product contains some software licensed under the
"GNU Lesser General Public License, version 2.1" provided
with ABSOLUTELY NO WARRANTY under the terms of "GNU Lesser
General Public License, version 2.1", available here:
http://www.gnu.org/licenses/old-licenses/lgpl-2.1.html

This product contains some software licensed under the
"GNU General Public License, version 3" provided with
ABSOLUTELY NO WARRANTY under the terms of
"GNU General Public License, Version 3", available here:
http://www.gnu.org/licenses/gpl.html.

This product contains some software licensed under the
"GNU Affero General Public License, version 3" provided
with ABSOLUTELY NO WARRANTY under the terms of
"GNU Affero General Public License, version 3", available here:
http://www.gnu.org/licenses/agpl-3.0.html.

Cisco AP Software, (ap3g3), C3802, RELEASE SOFTWARE
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Thu Jun 13 14:56:10 PDT 2019

ROM: Bootstrap program is U-Boot boot loader
BOOTLDR: U-Boot boot loader Version not available

"Masked AP name" uptime is 0 days, 0 hours, 1 minutes
Last reload time : Sat Sep 14 03:37:37 UTC 2019
Last reload reason :
cisco AIR-AP3802I-F-K9 ARMv7 Processor rev 1 (v7l) with 1028616/817892K bytes of memory.
Processor board ID "Masked board ID"
AP Running Image : 8.5.151.0
Primary Boot Image : 8.5.151.0
Backup Boot Image : 8.5.140.10
1 Multigigabit Ethernet interfaces
1 Gigabit Ethernet interfaces
1 802.11 Radios
Radio Driver version : 9.0.5.5-W8964
Radio FW version : 9.1.8.1
NSS FW version : N/A

Base ethernet MAC Address : "Masked MAC address"
Part Number : 73-018550-02
PCA Assembly Number : 000-00000-00
PCA Revision Number :
PCB Serial Number : "Masked Serial Number"
Top Assembly Part Number : 068-100531-02
Top Assembly Serial Number : "Masked Serial Number"
Top Revision Number : A0
Product/Model Number : AIR-AP3802I-F-K9

4. AP: sh ip interface brief

>sh ip int br
Interface IP-Address Method Status Protocol Speed Duplex
wired0 "Masked IP address" static up up 5000 full
wired1 unassigned unset down down n/a unknown
wifi0 n/a n/a administatively down down n/a n/a

[Leo Laohoo]

Controller is not using NTP.
I'd recommend configuring NTP.

[fdharmawan]

Hi Leo,

I already setup NTP on the WLC and still got the same error message.

[Leo Laohoo]

Console into the AP and reboot.
Post the entire boot-up process.

[fdharmawan]

Hi Leo,

Attached is the boot process of the AP. Thank you.

[Leo Laohoo]

Post the complete output to the following commands:
1. WLC: sh time;
2. WLC: sh ap join stats detail <AP NAME>;
3. AP: sh inventory; &

4. AP:  sh capw cli rcb
NOTE: Time and date is still incorrect, however, I'm also seeing the AP affected by CSCvf71074/CSCvh85082.

[fdharmawan]

Hi Leo,

1. WLC: sh time;

(Cisco Controller) >show time

Time............................................. Thu Jan 23 11:14:15 2020

Timezone delta................................... 0:0
Timezone location................................ (GMT +7:00) Saigon, Hanoi, Bangkok, Jakatar

NTP Servers
NTP Version.................................. 3
NTP Polling Interval......................... 600

Index NTP Key Index NTP Server Status NTP Msg Auth Status
------- ---------------------------------------------------------------------
1 0 10.152.16.123 In Sync AUTH DISABLED

2. WLC: sh ap join stats detail <AP NAME>;

(Cisco Controller) >show ap join stats detailed 78:72:5d:1c:34:dc
No join information found for AP: 78:72:5d:1c:34:dc

3. AP: sh inventory; &

Cisoc AP>sh inventory
NAME: AP3800, DESCR: Cisco Aironet 3800 Series (IEEE 802.11ac) Access Point
PID: AIR-AP3802I-F-K9 , VID: V02, SN: Cisco SN

4. AP:  sh capw cli rcb

ID-JKTAPP3601>sh capw cli rcb
[*09/14/2019 04:00:48.4395] waiting for WCP to initialize
[*09/14/2019 04:00:48.4395] waiting for WCP to initialize
[*09/14/2019 04:00:48.4395] waiting for WCP to initialize
AdminState : ADMIN_ENABLED
OperationState : DISCOVERY
Name : Cisco AP
SwVer : 8.5.151.0
HwVer : 1.0.0.0
MwarApMgrIp : ::
MwarName :
MwarHwVer : 0.0.0.0
Location : default location
ApMode : Local
ApSubMode : Not Configured
CAPWAP Path MTU : 576
CAPWAP UDP-Lite : Enabled
IP Prefer-mode : IPv4
AP Link DTLS Encryption : OFF
AP TCP MSS Adjust : Disabled
LinkAuditing : disabled
AP Group Name : Cisco AP Group
Cisco Trustsec Config
AP Inline Tagging Mode : Disabled
AP Sgacl Enforcement : Disabled
AP Override Status : Disabled

[Leo Laohoo]

Go HERE and enter the serial number of the AP. 

[fdharmawan]

Hi Leo,

The result was not affected.

I wonder whether you can factory reset the 3802 AP. Is it possible? I found this guide but it's for 3702AP. https://popravak.wordpress.com/2014/08/26/resetting-cisco-capwaplwap-access-point-to-factory-defaults/

[Jurgens L]

Had a similar problem with 2800 series AP's, if your time on the WLC is set up correctly you are most likely experiencing a bug. Log a TAC with Cisco, if they concur you are hitting the bug, then they will send you a replacement.

<<< Pls remember to rate all useful responses >>>

[fdharmawan]

Hi Jurgens,

Unfortunately I do not have TAC support right now.

Based on your experience with your 2800, did TAC replace the AP? Or something else can be done, like upgrade the image?

Thanks