WLC is not recieving Join Requests from AP

[ERIK S]

I have ran these debug commands and below is the output the I recieved.

debug mac addr AP MAC ADDRESS
debug capwap event enable

debug capwap errors enable

debug capwap detail enable

debug PM PKI enable

debug DTLS Event Enable 

*spamApTask1: Apr 24 13:35:35.075: AP MAC ADDRESS Discovery Response sent to AP IP ADDRESS port 55037

*spamApTask1: Apr 24 13:35:35.075: AP MAC ADDRESS Discovery Response sent to AP IP ADDRESS:55037

*spamApTask1: Apr 24 13:35:35.075: AP MAC ADDRESS WTP already released
*spamApTask6: Apr 24 13:35:38.907: Received SPAM_ADD_MOBILE

*spamReceiveTask: Apr 24 13:35:39.358: Received SPAM_SEND_AP_PMK_CACHE_ENTRY

*spamApTask6: Apr 24 13:35:39.362: Received SPAM_ADD_MOBILE

*spamReceiveTask: Apr 24 13:35:42.160: Received SPAM_SEND_AP_PMK_CACHE_ENTRY

*spamApTask1: Apr 24 13:35:42.164: Received SPAM_ADD_MOBILE

*spamApTask1: Apr 24 13:35:45.011: AP MAC ADDRESS packet received of length 189 from AP IP ADDRESS:55037

*spamApTask1: Apr 24 13:35:45.011: AP MAC ADDRESS Msg Type = 1 Capwap state = 0

*spamApTask1: Apr 24 13:35:45.011: AP MAC ADDRESS Discovery Request from AP IP ADDRESS:55037

*spamApTask1: Apr 24 13:35:45.011: AP MAC ADDRESS msgEleLength = 1 msgEleType = 20

*spamApTask1: Apr 24 13:35:45.011: AP MAC ADDRESS Total msgEleLen = 160

*spamApTask1: Apr 24 13:35:45.011: AP MAC ADDRESS msgEleLength = 62 msgEleType = 38

*spamApTask1: Apr 24 13:35:45.011: AP MAC ADDRESS ApModel: AIR-CAP3502I-A-K9

*spamApTask1: Apr 24 13:35:45.011: AP MAC ADDRESS Total msgEleLen = 94

*spamApTask1: Apr 24 13:35:45.011: AP MAC ADDRESS msgEleLength = 40 msgEleType = 39

*spamApTask1: Apr 24 13:35:45.011: AP MAC ADDRESS Total msgEleLen = 50

*spamApTask1: Apr 24 13:35:45.011: AP MAC ADDRESS msgEleLength = 1 msgEleType = 41

*spamApTask1: Apr 24 13:35:45.011: AP MAC ADDRESS Total msgEleLen = 45

*spamApTask1: Apr 24 13:35:45.011: AP MAC ADDRESS msgEleLength = 1 msgEleType = 44

*spamApTask1: Apr 24 13:35:45.011: AP MAC ADDRESS Total msgEleLen = 40

*spamApTask1: Apr 24 13:35:45.011: AP MAC ADDRESS msgEleLength = 10 msgEleType = 37

*spamApTask1: Apr 24 13:35:45.011: AP MAC ADDRESS Vendor specific payload from AP AP MAC ADDRESS validated

*spamApTask1: Apr 24 13:35:45.011: AP MAC ADDRESS Total msgEleLen = 26

*spamApTask1: Apr 24 13:35:45.011: AP MAC ADDRESS msgEleLength = 22 msgEleType = 37

*spamApTask1: Apr 24 13:35:45.011: AP MAC ADDRESS Vendor specific payload from AP AP MAC ADDRESS validated

*spamApTask1: Apr 24 13:35:45.011: AP MAC ADDRESS Total msgEleLen = 0

*spamApTask1: Apr 24 13:35:45.011: AP MAC ADDRESS Join Priority Processing status = 0, Incoming Ap's Priority 1, MaxLrads = 50, joined Aps =36
*spamApTask1: Apr 24 13:35:45.012: AP MAC ADDRESS 1. 37 0

*spamApTask1: Apr 24 13:35:45.012: AP MAC ADDRESS 2. 232 3

*spamApTask1: Apr 24 13:35:45.012: AP MAC ADDRESS 3. 36 0

*spamApTask1: Apr 24 13:35:45.012: AP MAC ADDRESS 4. 50 0

*spamApTask1: Apr 24 13:35:45.012: AP MAC ADDRESS apType = 17 apModel: AIR-CAP3502I-A-K9

*spamApTask1: Apr 24 13:35:45.012: AP MAC ADDRESS apType: Ox11 bundleApImageVer: 8.0.152.0
*spamApTask1: Apr 24 13:35:45.012: AP MAC ADDRESS version:8 release:0 maint:152 build:0
*spamApTask1: Apr 24 13:35:45.012: AP MAC ADDRESS Discovery resp: AC Descriptor message element len = 40

*spamApTask1: Apr 24 13:35:45.012: AP MAC ADDRESS acName = SFD-WLC-5508-01

*spamApTask1: Apr 24 13:35:45.012: AP MAC ADDRESS Discovery resp:AC Name message element length = 59

*spamApTask1: Apr 24 13:35:45.012: AP MAC ADDRESS Discovery resp: WTP Radio Information msg length = 68

*spamApTask1: Apr 24 13:35:45.012: AP MAC ADDRESS Discovery resp: CAPWAP Control IPV4 Address len = 78

*spamApTask1: Apr 24 13:35:45.012: AP MAC ADDRESS Discovery resp: CAPWAP Control IPV6 Address len = 78

*spamApTask1: Apr 24 13:35:45.012: AP MAC ADDRESS Discovery resp: Mwar type payload len = 89

*spamApTask1: Apr 24 13:35:45.012: AP MAC ADDRESS Discovery resp: Expire MIC type payload len = 103

*spamApTask1: Apr 24 13:35:45.012: AP MAC ADDRESS Discovery resp: Time sync payload len = 118

*spamApTask1: Apr 24 13:35:45.012: AP MAC ADDRESS Discovery Response sent to AP IP ADDRESS port 55037

*spamApTask1: Apr 24 13:35:45.012: AP MAC ADDRESS Discovery Response sent to AP IP ADDRESS:55037

*spamApTask1: Apr 24 13:35:45.012: AP MAC ADDRESS WTP already released
*spamApTask1: Apr 24 13:35:45.012: AP MAC ADDRESS packet received of length 189 from AP IP ADDRESS:55037

*spamApTask1: Apr 24 13:35:45.012: AP MAC ADDRESS Msg Type = 1 Capwap state = 0

*spamApTask1: Apr 24 13:35:45.012: AP MAC ADDRESS Discovery Request from AP IP ADDRESS:55037

*spamApTask1: Apr 24 13:35:45.012: AP MAC ADDRESS Total msgEleLen = 160

*spamApTask1: Apr 24 13:35:45.012: AP MAC ADDRESS msgEleLength = 62 msgEleType = 38

*spamApTask1: Apr 24 13:35:45.013: AP MAC ADDRESS Total msgEleLen = 94

*spamApTask1: Apr 24 13:35:45.013: AP MAC ADDRESS msgEleLength = 40 msgEleType = 39

*spamApTask1: Apr 24 13:35:45.013: AP MAC ADDRESS Total msgEleLen = 50

*spamApTask1: Apr 24 13:35:45.013: AP MAC ADDRESS msgEleLength = 1 msgEleType = 41

*spamApTask1: Apr 24 13:35:45.013: AP MAC ADDRESS Total msgEleLen = 45

*spamApTask1: Apr 24 13:35:45.013: AP MAC ADDRESS msgEleLength = 1 msgEleType = 44

*spamApTask1: Apr 24 13:35:45.013: AP MAC ADDRESS Total msgEleLen = 40

*spamApTask1: Apr 24 13:35:45.013: AP MAC ADDRESS msgEleLength = 10 msgEleType = 37

*spamApTask1: Apr 24 13:35:45.013: AP MAC ADDRESS Vendor specific payload from AP AP MAC ADDRESS validated

*spamApTask1: Apr 24 13:35:45.013: AP MAC ADDRESS Total msgEleLen = 26

*spamApTask1: Apr 24 13:35:45.013: AP MAC ADDRESS msgEleLength = 22 msgEleType = 37

*spamApTask1: Apr 24 13:35:45.013: AP MAC ADDRESS Vendor specific payload from AP AP MAC ADDRESS validated

*spamApTask1: Apr 24 13:35:45.013: AP MAC ADDRESS Total msgEleLen = 0

*spamApTask1: Apr 24 13:35:45.013: AP MAC ADDRESS Join Priority Processing status = 0, Incoming Ap's Priority 1, MaxLrads = 50, joined Aps =36
*spamApTask1: Apr 24 13:35:45.013: AP MAC ADDRESS 1. 37 0

*spamApTask1: Apr 24 13:35:45.013: AP MAC ADDRESS 2. 232 3

*spamApTask1: Apr 24 13:35:45.013: AP MAC ADDRESS 3. 36 0

*spamApTask1: Apr 24 13:35:45.013: AP MAC ADDRESS 4. 50 0

*spamApTask1: Apr 24 13:35:45.013: AP MAC ADDRESS apType = 17 apModel: AIR-CAP3502I-A-K9

*spamApTask1: Apr 24 13:35:45.013: AP MAC ADDRESS apType: Ox11 bundleApImageVer: 8.0.152.0
*spamApTask1: Apr 24 13:35:45.013: AP MAC ADDRESS version:8 release:0 maint:152 build:0
*spamApTask1: Apr 24 13:35:45.013: AP MAC ADDRESS Discovery resp: AC Descriptor message element len = 40

Initially I thought that the DTLS Session port was the issue, however the port number is not being used in any other sessions.

I've also tried power cycling the AP by shutting and no shutting the port that it is connected to.

The AP and WLC are in different VLANs but the switch is doing Inter VLAN routing. I've verified that option 43 HEX is correct, I've also added a Host A record for the WLC IP Address so that it reflect Cisco-CAPWAP-Controller.localdomain.

Has any one encountered this in the past? 

Thanks in advance. 

[Leo Laohoo]

Post the complete output to the following commands:
1. WLC: sh sysinfo;
2. WLC: sh time;
3. AP: sh version; and
4. AP: sh ip interface brief

[ERIK S]

WLC sys INFO

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 8.0.152.0
Bootloader Version............................... 1.0.20
Field Recovery Image Version..................... 7.6.101.1
Firmware Version................................. FPGA 1.7, Env 1.8, USB console 2.2
Build Type....................................... DATA + WPS

System Name...................................... IT Company Name -WLC-5508-01
System Location.................................. US STATE EST TIME 
System Contact...................................
System ObjectID.................................. 1.3.6.1.4.1.9.1.1069
Redundancy Mode.................................. Disabled
IP Address....................................... 172.16.1.101
IPv6 Address..................................... ::
Last Reset....................................... Software reset
System Up Time................................... 544 days 12 hrs 4 mins 9 secs
System Timezone Location.........................
System Stats Realtime Interval................... 5
System Stats Normal Interval..................... 180

Configured Country............................... US - United States

--More-- or (q)uit
Operating Environment............................ Commercial (0 to 40 C)
Internal Temp Alarm Limits....................... 0 to 65 C
Internal Temperature............................. +40 C
External Temperature............................. +19 C
Fan Status....................................... OK

State of 802.11b Network......................... Enabled
State of 802.11a Network......................... Enabled
Number of WLANs.................................. 6
Number of Active Clients......................... 31

Burned-in MAC Address............................ MAC ADDRESS
Power Supply 1................................... Present, OK
Power Supply 2................................... Present, OK
Maximum number of APs supported.................. 500
System Nas-Id....................................
WLC MIC Certificate Types........................ SHA1

Time Zone 

Time............................................. Thu Apr 25 12:55:12 2019

Timezone delta................................... 0:0
Timezone location................................

NTP Servers
NTP Polling Interval......................... 86400

Index NTP Key Index NTP Server NTP Msg Auth Status
------- ----------------------------------------------------------------------------------
1 0 131.130.251.107 AUTH DISABLED

AP Version

Version :
Cisco IOS Software, C3500 Software (AP3G1-K9W8-M), Version 15.3(3)JA8, RELEASE SOFTWARE (fc1)

AP interface

Switch mode access 
Switch Acc VLAN (VLAN ID)

Spanning-tree portfast


I should note that I statically configured the AP with an IP/ Default Gateway & the controller's IP yesterday, both devices can ping each other even though they are on different VLANs since the SVI's reside on the same switch. 

However, after doing so the AP still won't join the WLC and the WLC isn't seeing join requests. 

[Leo Laohoo]

I want to see the COMPLETE output to the following commands:
1. AP: sh version; and
2. AP: sh ip interface brief

[ERIK S]

As requested

AP HostName #sh ver
Cisco IOS Software, C3500 Software (AP3G1-K9W8-M), Version 15.3(3)JA8, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2016 by Cisco Systems, Inc.
Compiled Fri 22-Apr-16 05:28 by prod_rel_team

ROM: Bootstrap program is C3500 boot loader
BOOTLDR: C3500 Boot Loader (AP3G1-BOOT-M), Version 15.3 [vtoky-imagetype 106]

AP MAC ADDRESS uptime is 19 minutes
System returned to ROM by reload
System restarted at 12:17:28 UTC Thu Apr 25 2019
System image file is "flash:/ap3g1-k9w8-mx.153-3.JA8/ap3g1-k9w8-xx.153-3.JA8"
Last reload reason:

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco AIR-CAP3502I-A-K9 (PowerPC460exr) processor (revision A0) with 98294K/32768K bytes of memory.
Processor board ID ***************
PowerPC460exr CPU at 666Mhz, revision number 0x18A8
Last reset from reload
LWAPP image version 8.0.133.0
1 Gigabit Ethernet interface
2 802.11 Radios

32K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address: AP MAC ADDRESS
Part Number : *************************
PCA Assembly Number : *************
PCA Revision Number : A0
PCB Serial Number : ************
Top Assembly Part Number : *****************
Top Assembly Serial Number : ***************
Top Revision Number : A0
Product/Model Number : AIR-CAP3502I-A-K9

Configuration register is 0xF

Interface IP-Address OK? Method Status Protocol
BVI1 172.17.12.52 YES DHCP up up
Dot11Radio0 unassigned NO unset up up
Dot11Radio1 unassigned NO unset up up
GigabitEthernet0 unassigned NO unset up up

Translating "CISCO-CAPWAP-CONTROLLER.ITCompanyName.com"...domain server (172.16.X.X) [OK]

Apr 25 12:34:55.882: %CAPWAP-5-DHCP_OPTION_43: Controller address 172.16.1.101 obtained through DHCP
Apr 25 12:34:55.882: %CAPWAP-5-DHCP_OPTION_43: Controller address 172.16.1.102 obtained through DHCP
Not in Bound state.
Apr 25 12:35:51.397: %CAPWAP-3-DHCP_RENEW: Could not discover WLC. Either IP address is not assigned or assigned IP is wrong. Renewing DHCP IP.
Apr 25 12:35:54.420: %DHCP-6-ADDRESS_ASSIGN: Interface BVI1 assigned DHCP address 172.17.12.52, mask 255.255.255.0, hostname AP AP HostName

^
% Invalid input detected at '^' marker.

Apr 25 12:36:02.398: %CAPWAP-5-DHCP_OPTION_43: Controller address 172.16.1.101 obtained through DHCP
Apr 25 12:36:02.398: %CAPWAP-5-DHCP_OPTION_43: Controller address 172.16.1.102 obtained through DHCP

Not in Bound state.
Apr 25 12:36:47.907: %CAPWAP-3-DHCP_RENEW: Could not discover WLC. Either IP address is not assigned or assigned IP is wrong. Renewing DHCP IP.
Apr 25 12:36:50.927: %DHCP-6-ADDRESS_ASSIGN: Interface BVI1 assigned DHCP address 172.17.12.52, mask 255.255.255.0, hostname AP442b.0391.4904

Apr 25 12:36:57.907: %CAPWAP-5-DHCP_OPTION_43: Controller address 172.16.1.101 obtained through DHCP
Apr 25 12:36:57.907: %CAPWAP-5-DHCP_OPTION_43: Controller address 172.16.1.102 obtained through DHCP

AP HOST NAME #ping Cisco-capwap-controller

Translating "Cisco-capwap-controller"...domain server (172.16.X.X) (172.16.X.X)

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.1.101, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/3 ms

[patoberli]

I assume there is no access-list between the two SVI?

[Edit]

You have accepted and activated the License on the WLC?

[ERIK S]

That is correct, the SVIs reside on the same switch and there are no VACLs implemented. The weird thing is that there are two APs on the same VLAN that ARE joined with the WLC. 

[Alex Pfeil]

You could also try to replace one of the working wireless access points to validate that you do not have a cabling issue.

[ERIK S]

License Store: Primary License Storage
StoreIndex: 0 Feature: base Version: 1.0
License Type: Permanent
License State: Active, Not in Use
License Count: Non-Counted
License Priority: Medium
License Store: Primary License Storage
StoreIndex: 1 Feature: base-ap-count Version: 1.0
License Type: Permanent
License State: Active, In Use
License Count: 50 /50 (Active/In-use)
License Priority: Medium
License Store: Evaluation License Storage
StoreIndex: 0 Feature: base-ap-count Version: 1.0
License Type: Evaluation
License State: Inactive
Evaluation total period: 8 weeks 4 days
Evaluation period left: 8 weeks 4 days
License Count: 500 / 0 (Active/In-use)
License Priority: None

(Cisco Controller) >show license capacity

Licensed Feature Max Count Current Count Remaining Count
-----------------------------------------------------------------------
AP Count 50 36 14

[patoberli]

Can't see any fault. I would say, reset the AP to factory defaults with the Mode button and try again.

[ERIK S]

That's probably what I will have to do, I am trying to figure out if there are any compatibility issues between the WLC and the AP but I can't seem to find anything that points to that as the issue.

[patoberli]

No, the two versions should be fine.

There is a little chance your AP has a flash image corruption (which is why I suggest you upgrade to a fixed version in any case): https://www.cisco.com/c/en/us/support/docs/wireless-mobility/wireless-lan-wlan/213317-understanding-various-ap-ios-flash-corru.html

[ERIK S]

Several of the APs that are refusing to join the WLC have Cisco IOS Software, C3500 Software (AP3G1-K9W8-M), Version 15.3(3)JA12, RELEASE SOFTWARE (fc2). 

At this point I think the best bet would to have TAC look at this but I truly appreciate the thought.

[Leo Laohoo]

Go back to the AP that isn't working and issue the command "dir". Post the output. I've got another trick.