ap 3702i cant join controller

[snir_orlanczyk]

hi,

we have a 3702i that won't join our controller, (i have tried it on a wlc 5508 running 8.0.121 and on a 3860 runing ios-xe 3.6.3)

those are the log messeges that i got from the AP console while booting up:

*Jul 24 14:46:06.807: Cmd Timeout After 12s

*Jul 24 14:46:06.807: mvl_handle_timeout called cmdtype=[0x26]

*Jul 24 14:46:06.807: FW coredump command failed 0/0.
*Jul 24 14:46:06.807: No coredump info received from firmware
*Jul 24 14:46:06.819: Interface Dot11Radio0: not starting because stop-on-failure set
*Jul 24 14:46:13.075: %CDP_PD-4-POWER_OK: 15.4 W power - NEGOTIATED inline power source
*Jul 24 14:46:13.075: Interface Dot11Radio0: not starting because stop-on-failure set
*Jul 24 14:46:14.323: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up
*Jul 24 14:46:15.323: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to up
*Jul 24 14:47:38.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.100.104.253 peer_port: 5246
*Jul 24 14:47:39.483: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 10.100.104.253 peer_port: 5246
*Jul 24 14:47:39.483: %CAPWAP-5-SENDJOIN: sending Join Request to 10.100.104.253
*Jul 24 14:47:44.483: %CAPWAP-5-SENDJOIN: sending Join Request to 10.100.104.253
*Jul 24 14:48:32.879: %CDP_PD-4-POWER_OK: 15.4 W power - NEGOTIATED inline power source
*Jul 24 14:48:37.999: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.100.104.253:5246
*Jul 24 14:48:38.063: bsnInitRcbSlot: slot 0 has venus radio(UNSUPPORT)
*Jul 24 14:48:38.079: bsnUnlockDevice: not bring radio up: radio 0 is in admin disable state
*Jul 24 14:48:38.083: Interface Dot11Radio0: not starting because stop-on-failure set
*Jul 24 14:48:38.095: %LWAPP-4-CLIENTEVENTLOG: Not sending change state post as the radio admin is down, lrad state = 5
*Jul 24 14:48:38.099: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to administratively down
*Jul 24 14:48:38.115: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up
*Jul 24 14:48:39.099: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to down
*Jul 24 14:48:39.139: Interface Dot11Radio0: not starting because stop-on-failure set
*Jul 24 14:48:39.143: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to down
*Jul 24 14:48:39.151: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to reset
*Jul 24 14:48:40.183: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up
*Jul 24 14:48:41.183: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to up
*Jul 24 14:48:48.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.100.104.253 peer_port: 5246
*Jul 24 14:48:48.483: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 10.100.104.253 peer_port: 5246
*Jul 24 14:48:48.483: %CAPWAP-5-SENDJOIN: sending Join Request to 10.100.104.253
*Jul 24 14:48:53.483: %CAPWAP-5-SENDJOIN: sending Join Request to 10.100.104.253

anyone have an idea what can cause the AP DTLS to drop like that ? 

thank you,

Snir

[Scott Fella]

I would upgrade to 8.0.133.0 if you want to start on the v8.0. Also, place the AP on the same subnet as the WLC management. Do you have any other AP joined to the controller?  Can you post the console output when rebooting the AP so we can see the whole boot process including the code version of the AP. Attach that as a text file so the thread doesn't get so long.

-Scott 

*** Please rate helpful posts ***

[snir_orlanczyk]

i have attached the full console output from the the AP's boot process.

- both the ap and the controller are on the same subnet.

- yes i more aps connected to this controller .

- and i have tried connecting it to an ios-xe controller and had the exact same issue, but i'll try and and upgrade it today and let u know if it helps .

***** update : i've just updated my controller version to 8.0.133 and no change 

Snir

[alois.heilmaier]

I also had the same problem. Just hard powered off/on the AP.

It seems the AP tried to use LWAPP + CAPWAP. As I powered on the AP it worked via CAPWAPP.

Maybe you have a chance to install the AP behind L3 routing hop and use DNS with cisco-capwap-controller only. (make sure no DNS cisco-lwapp-controller entry exists).

[WiFi Trainers]

Hi Snir,

From the log messages: 

*Jul 24 14:47:38.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.100.104.253 peer_port: 5246
*Jul 24 14:47:39.483: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 10.100.104.253 peer_port: 5246
*Jul 24 14:47:39.483: %CAPWAP-5-SENDJOIN: sending Join Request to 10.100.104.253
*Jul 24 14:47:44.483: %CAPWAP-5-SENDJOIN: sending Join Request to 10.100.104.253

It looks like WLC is not responding to the join requests being sent. Can you please send the following outputs from the WLC:

debug capwap event enable

debug capwap error enable

Best Regards,

WiFi Trainers (www.wifitrainers.com)

Your one stop solution for all your wireless training needs!

******** Please rate if useful *********

[snir_orlanczyk]

i have attached the outputs you asked for, and i noticed this line :

*spamApTask4: Jul 25 10:51:48.781: 00:00:00:00:00:00 Not allocating an entry for the AP, received an AP with zero mac 00:00:00:00:00:00

seems odd to me, does anyone know what can cause such a behavior?

thank you,

Snir

[WiFi Trainers]

Hello,

Thanks for the logs. Here is what i see:

Join request recieved:

*spamApTask4: Jul 25 10:51:48.780: 00:00:00:00:00:00 Join Request from 10.100.104.3:31265

WLC evaluates the join message:

*spamApTask4: Jul 25 10:51:48.781: 00:00:00:00:00:00 Not allocating an entry for the AP, received an AP with zero mac 00:00:00:00:00:00
*spamApTask4: Jul 25 10:51:48.781: 00:00:00:00:00:00 Failed to allocate database entry for AP 10.100.104.3:31265

*spamApTask4: Jul 25 10:51:48.781: 00:00:00:00:00:00 Failed to add database entry for 10.100.104.3:31265

Join request denied:

*spamApTask4: Jul 25 10:51:48.782: 00:00:00:00:00:00 Join Request failed!

This is the important part where the WLC is trying to tell what type of message it is unable to decode:

*spamApTask4: Jul 25 10:51:48.782: 00:00:00:00:00:00 State machine handler: Failed to process  msg type = 3 state = 0 from 10.100.104.3:31265

We cover several such scenarios along with how to understand, debug and troubleshoot them in our 'Cisco Advanced Troubleshooting' courses. I have seen the issue happen primarily in two scenarios. One is easy to fix, while the other is harder :).

1. If the AP is in mesh mode (ordered in mesh mode or converted sometime) the WLC will fail to validate the AP authorization list since the AP entry is unavailable and will report a similar set of logs. If the AP has been ordered in mesh mode, do let me know and we can use a simple workaround to fix it.

2. The second scenario is that the AP is faulty and will need an RMA. I have personally seen this happen on one of my lab equipment before and it turned out to be faulty. So if this is a new AP and the only one facing this issue, it might just be the case. 

You can also try to

Boot the AP with the recovery image

or

Re-image the AP using the ROMMON method and check if it helps.

Best Regards,

WiFi Trainers (www.wifitrainers.com)

Your one stop solution for all your wireless training needs!

******** Please rate if useful *********

[snir_orlanczyk]

Hi'

im postive its not in mesh mode because i have deleted all of its configuration and as far as i know it was not ordered in mesh mode.

i have tried re-imaging ,and also booting up using the recovery image as you have suggested, saddly its a no go...

i want to make sure its a realy is falthy before i send it to RMA, so if you have anymore suggestion let me know.

thank you ,

Snir

[WiFi Trainers]

Hi Snir,

You are correct. Looking at the logs it is definitely not in mesh mode as the mesh state machine has not kicked in with the AP.

For me this looks to be an RMA unfortunately.

Best Regards,

WiFi Trainers (www.wifitrainers.com)

Your one stop solution for all your wireless training needs!

******** Please rate if useful *********

[Mitoca]

I am having a similar issue and mine is in Mesh Mode. I changed it to Local but it is not joining. What was the simple fix you mentioned? Thank you.