Do you need that information?

ajc · ‎04-11-2014

Hi all,

In short, APs will join the controller immediately using the recovery image, but once it downloads 7.6.100 and reboots, it either cannot join the controller or will take up to 40 minutes.

The AP is in local mode. The AP 1040 eventually joined the controller, 40 minutes after we deleted all the files off the flash which is unacceptable.

*Apr 9 15:38:15.842: %CAPWAP-5-SENDJOIN: sending Join Request to 10.10.10.10

*Apr 9 15:38:15.851: %CAPWAP-3-ERRORLOG: Invalid event 10 & state 5 combination.

*Apr 9 15:38:15.851: %CAPWAP-3-ERRORLOG: CAPWAP SM handler: Failed to process message type 10 state 5.

*Apr 9 15:38:15.851: %CAPWAP-3-ERRORLOG: Failed to handle capwap control message from controller

*Apr 9 15:38:15.851: %CAPWAP-3-ERRORLOG: Failed to process encrypted capwap packet from 10.10.10.10

*Apr 9 15:38:16.304: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to down

*Apr 9 15:38:16.362: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset

*Apr 9 15:38:16.427: %CAPWAP-5-JOINEDCONTROLLER: AP has joined controller CONTROLLER-DC

I also had our technician console into a 2602 and we were seeing exact same errors.

SUMMARY OF THIS CASE:

--AP is 1042 and wlc is on 7.6.100.0

--using dhcp option 43 but the AP has static ip

--checked the private config on the AP and it was not sending the discovery request to the desired WLC

--cleared the private config

--primed the AP to desired WLC

--AP sending join request but receiving the following errors:

Apr 4 14:37:47.000: CAPWAP-3-ERRORLOG Go join a capwap controller

*Apr 4 14:36:47.000: CAPWAP-5-DTLSREQSEND DTLS connection request sent peer_ip: 10.10.10.11 peer_port: 5246

*Apr 4 14:36:47.169: DTLS_CLIENT_ERROR: ../capwap/base_capwap/dtls/base_capwap_dtls_handshake.c:90 First fragment for seq 2 is missing

*Apr 4 14:37:17.205: DTLS_CLIENT_ERROR: ../capwap/base_capwap/dtls/base_capwap_dtls_connection_db.c:2176 Max retransmission count reached!

--cleared the flash except rcv

--ap started downloading the image from the wlc but once it reboots the same errors appear again

--research made on the error message and apparently we are hitting a bug:

https://tools.cisco.com/bugsearch/bug/CSCul08933/?reffering_site=dumpcr

Leo Laohoo · ‎04-11-2014

May I ask if you can post the output to the following commands:

WLC: sh sysinfo;
WLC: sh time;
AP: sh version;
AP: sh ip interface brief; and
AP: sh inventory

ajc · ‎04-15-2014

Hi Leo,

Next the information, I omitted some serial number and descriptions due to confidentiality matters.

(Cisco Controller) >show sysinfo

Manufacturer's Name.............................. Cisco Systems Inc.

Product Name..................................... Cisco Controller Product Version.................................. 7.6.100.0 Bootloader Version............................... 1.0.1 Field Recovery Image Version..................... 6.0.182.0 Firmware Version................................. FPGA 1.3, Env 1.6, USB console 1.27 Build Type....................................... DATA + WPS

System Name...................................... ABCDEF System Location..................................

System Contact...................................

System ObjectID.................................. 1.3.6.1.4.1.9.1.1069 Redundancy Mode.................................. Disabled IP Address....................................... XXXXXXX Last Reset....................................... Software reset System Up Time................................... 32 days 14 hrs 1 mins 41 secs System Timezone Location......................... (GMT -5:00) Eastern Time (US and Canada) System Stats Realtime Interval................... 5 System Stats Normal Interval..................... 180

Configured Country............................... CA - Canada Operating Environment............................ Commercial (0 to 40 C)

Internal Temp Alarm Limits....................... 0 to 65 C Internal Temperature............................. +40 C External Temperature............................. +21 C Fan Status....................................... OK

State of 802.11b Network......................... Enabled State of 802.11a Network......................... Enabled Number of WLANs.................................. 5 Number of Active Clients......................... 2215

Burned-in MAC Address............................ F0:F7:55:2B:4A:80 Power Supply 1................................... Present, OK Power Supply 2................................... Absent Maximum number of APs supported.................. 500

(Cisco Controller) >show time

Time............................................. Mon Apr 14 08:40:08 2014

Timezone delta................................... 0:0 Timezone location................................ (GMT -5:00) Eastern Time (US and Canada)

NTP Servers

NTP Polling Interval......................... 36000

Index NTP Key Index NTP Server NTP Msg Auth Status

------- ----------------------------------------------------------------------------------

1 0 A.B.C.D AUTH DISABLED

AP>sh ver

Compiled Wed 18-Dec-13 21:36 by prod_rel_team

ROM: Bootstrap program is C3500 boot loader

BOOTLDR: C3500 Boot Loader (AP3G1-BOOT-M), Version 15.2 [chayan-apn-0411 116]

AP uptime is 2 days, 17 hours, 15 minutes System returned to ROM by reload System image file is "flash:/ap3g1-k9w8-mx.152-4.JB3/ap3g1-k9w8-xx.152-4.JB3"

Last reload reason:

cisco AIR-CAP3502I-A-K9 (PowerPC460exr) processor (revision A0) with 98294K/32768K bytes of memory.

Processor board ID XXXXXXXX

PowerPC460exr CPU at 666Mhz, revision number 0x18A8 Last reset from reload LWAPP image version 7.6.100.0

1 Gigabit Ethernet interface

2 802.11 Radios

32K bytes of flash-simulated non-volatile configuration memory.

Base ethernet MAC Address: 70:81:05:9E:F7:36

Part Number : 73-12175-05

PCA Assembly Number : 800-32268-05

PCA Revision Number : A0

PCB Serial Number : XXXXXXX

Top Assembly Part Number : 800-32891-01

Top Assembly Serial Number : FGL1534S7LP

Top Revision Number : A0

Product/Model Number : AIR-CAP3502I-A-K9

Configuration register is 0xF

AP>sh ip int br

Interface IP-Address OK? Method Status Protocol

BVI1 X.Y.W.Z YES TFTP up up

Dot11Radio0 unassigned NO unset up up

Dot11Radio1 unassigned NO unset up up

GigabitEthernet0 unassigned NO unset up up

GigabitEthernet0.1 unassigned YES unset up up

AP>sh inventory

NAME: "AP3500", DESCR: "Cisco Aironet 3500 Series (IEEE 802.11n) Access Point"

PID: AIR-CAP3502I-A-K9 , VID: V01, SN: XXXXXXX

Leo Laohoo · ‎04-15-2014

I omitted some serial number

You omitted the WRONG serial number.

ajc · ‎04-16-2014

Do you need that information?. If it so, what is the serial you want to check.

Let me take a look on the mesh ios you mentioned.

Leo Laohoo · ‎04-15-2014

I've got a suspicion your AP is running MESH IOS.

ajc · ‎04-16-2014

Hi Leo,

What is the implication of using MESH IOS with respect to this issue?

thanks

Leo Laohoo · ‎04-16-2014

If your AP is running mesh, then it means that you'll need to add the AP MAC address into the WLC database so the AP can join the controller.

Ali Aqrabawi · ‎04-20-2014

if Mesh the AP will not join at all ,

ajc · ‎06-04-2014

As a general information, the problem was a DTLS bug found on version 7.6.100 which was addressed on an special software 7.6.100.25 that we installed recently and tested.

regards

ajc · ‎04-17-2014

Looks like this is not the case. We tried what is indicated in the following link and still the same error/issue:

http://www.cisco.com/c/en/us/support/docs/wireless-mobility/wireless-mesh/70531-wlc-mesh-config.html

APs randomly will not join the controller