Cisco 3502I cannot connect to vWLC

[mohfarid20]

Hello

I am new to wireless and I got Cisco AIR-CAP3502I-A-K9 wireless access point and installed vWLC (version: 8.5.161.0) on ESXi.
vWLC: management interface is bridged to outside and it is reached by switch and AP.
AP: is POE from the switch, getting IP from DHCP pool on Switch (also tried to statically assign CAPWAP IP/DG/controller IP).

the problem is, AP cannot join the WLC, and i ran Wireshark and could not see any CAPWAP traffic from the AP to WLC. see attached screenshot (ip in the screen shot is maybe changed from what is in the running config).

this log message keeps on appearing on the AP console:
(Translating "CISCO-CAPWAP-CONTROLLER.MOH.LOCAL"...domain server (172.16.1.1))

and sometimes the AP keeps on renewing its IP from DHCP: below logs:
*Apr 4 18:00:33.459: %DHCP-6-ADDRESS_ASSIGN: Interface BVI1 assigned DHCP address 172.16.1.58, mask 255.255.255.0, hostname AP4055.398e.0872
Translating "CISCO-CAPWAP-CONTROLLER.MOH.LOCAL"...domain server (172.16.1.1)
*Apr 4 18:00:39.423: %CAPWAP-5-DHCP_OPTION_43: Controller address 172.16.1.101 obtained through DHCP

attached show commands from AP, WLC and switch. and AP debug output and boot process.

Thanks in advance for you support

[Scott Fella]

Clear the AP’s nvram following this thread:

https://community.cisco.com/t5/other-wireless-mobility-subjects/how-to-restore-the-factory-settings/td-p/1991301

Did you also make sure you followed the vNIC settings in the guide. One thing is you need promiscuous mode enabled.

[mohfarid20]

Thank you @Scott Fella ,

I have already seen your reply in this post, and did the steps to erase but still no luck.

for vNIC, i enabled promiscuse mode (it was disabled) but nothing changed.

Its strange while using wireshark with SPAN port, i don't see any traffic (what so ever) sourced from the AP IP address to any IP (except when doing Ping), isn't the AP should send CAPWAP packets to the WLC IP address? i dont see that in the packet capture. it seems that the CAPWAP packets never leave the AP.

[mohfarid20]

the current status of the AP is:

1- got an IP from DHCP with all options.

%CAPWAP-5-DHCP_OPTION_43: Controller address 172.16.1.101 obtained through DHCP

%CAPWAP-3-DHCP_RENEW: Could not discover WLC. Either IP address is not assigned or assigned IP is wrong. Renewing DHCP IP.

2- AP assigns it self a new IP

then repeat the above process.

is this normal??

[Scott Fella]

That is not normal. When the ap can’t discover the wlc, the ap will reboot and mark the up as bad and will want to get a different IP address from the dhcp server. You would see bad address if using a Windows dhcp server.
So back to your situation, you can try to download the recovery image and tftp the image to the ap and delete the image you currently have in flash. Other than that, I don’t know what else you can do. If the ap is on the same subnet as the controller management, that is the simplest way for the ap to discover the controller.
Things I would do:

* factory rest the ap
* erase nvram
* upload recovery image to flash
* delete all other image files in flash
* try a different cable and switchport
* place the ap on the same subnet as the management
* use option 43 on the dhcp to see if that helps

[mohfarid20]

I will check out the recovery image,

this is the output of #dir flash:

AP4055.398e.0872#dir
Directory of flash:/

2 -rwx 214 Apr 4 2020 18:01:57 +00:00------------------------- env_vars
3 -rwx 61676 Apr 4 2020 18:02:05 +00:00 ---------------------- event.log
6 -rwx 6168 Apr 4 2020 18:03:03 +00:00 ----------------------- private-multiple-fs
5 -rwx 296 Apr 4 2020 18:02:34 +00:00 ------------------------ capwap-saved-config
43 drwx 0 Mar 1 2002 00:14:37 +00:00 ------------------------- configs
4 -rwx 128305 Sep 16 2014 02:18:27 +00:00 ------------------- event.r0
45 -rwx 64 Apr 4 2020 18:01:57 +00:00 ------------------------ sensord_CSPRNG1
46 -rwx 64 Apr 4 2020 18:01:57 +00:00 ------------------------ sensord_CSPRNG0
52 drwx 1152 Mar 1 1993 00:24:39 +00:00 ---------------------- ap3g1-k9w8-mx.153-3.JF12
11 -rwx 85 Mar 1 1993 00:40:19 +00:00 ------------------------ mesh_port_cfg.txt
8 -rwx 127211 May 2 2015 11:38:33 +00:00 -------------------- event.r1
7 -rwx 0 Mar 1 1993 00:40:37 +00:00 -------------------------- config.txt
9 -rwx 296 Mar 1 1993 01:18:11 +00:00 ------------------------ capwap-saved-config-bak

31481856 bytes total (20784640 bytes free)

even after typing #erase command in your previous reply, i didnt see any of those files get deleted.

what is not needed so i can manually delete.

[Scott Fella]

Erase nvram to another way to factory reset. You need to delete the image file but not until you have the rcv (recovery image) uploaded to the AP. Here is a thread that also goes over best ways to upload an image to an ap:

https://community.cisco.com/t5/wireless-and-mobility/recovery-image-topology-3602i-thru-2504-controller-from-tftp/td-p/2499395

[mohfarid20]

i have downloaded the rcv image, but this method requires an external power, i only have POE source. :(

[Leo Laohoo]

---

@mohfarid20 wrote:
i have downloaded the rcv image, but this method requires an external power, i only have POE source. :(

---

No, you don't. Console into the AP and issue the following command: 

archive download-sw tftp://<TFTP_IP_ADDRESS>/filename.tar

And then reboot. 

---

@mohfarid20 wrote:
it seems the packets are never made it to WLC

---

Can the AP ping the WLC?
Check firewall.

[mohfarid20]

ping was successful from the beginning, but sure it was the firewall block CAPWAP port.

i disabled the firewall and started to see good things happening :D,

but still not yet registered with the WLC

CAPWAP messages are exchanged between AP and WLC (see packet captures)

and also see "#debug capwap packets" on both the AP and WLC.

thanks for your support and for following up with the post.

[Leo Laohoo]

(Cisco Controller) >show ap join stats detailed 40:55:39:8E:08:72

What is the output to the above command? 

[mohfarid20]

(Cisco Controller) >show ap join stats detailed 40:55:39:8E:08:72
No join information found for AP: 40:55:39:8e:08:72


(Cisco Controller) >

nothing,

could u check "debug capwap events" output (attached)

i also read somewhere, i should add the AP mac to:

Security > AP Policies > (enabled) Authorize MIC APs against auth-list or AAA

and added AP MAC to > AP Authorization List

[mohfarid20]

when i issue "sh ver" on AP, i got this;

Base ethernet MAC Address: 40:55:39:8E:08:72

but when i issued "sh ap join stats summary all" on WLC, this is what i got

it seems the MAC is different

(Cisco Controller) >show ap join stats summary all

Number of APs.............................................. 1 

Base Mac             AP EthernetMac       AP Name                 IP Address         Status
e8:40:40:df:64:60    N A                  AP4055.398e.0872        172.16.1.97        Not Joined

so here is the output of "show ap join stats detailed e8:40:40:df:64:60"

(Cisco Controller) >show ap join stats detailed e8:40:40:df:64:60

Sync phase statistics
- Time at sync request received............................ Not applicable
- Time at sync completed................................... Not applicable

Discovery phase statistics
- Discovery requests received.............................. 80
- Successful discovery responses sent...................... 80
- Unsuccessful discovery request processing................ 0
- Reason for last unsuccessful discovery attempt........... Not applicable
- Time at last successful discovery attempt................ Apr 05 17:08:53.094
- Time at last unsuccessful discovery attempt.............. Not applicable

Join phase statistics
- Join requests received................................... 140
- Successful join responses sent........................... 0
- Unsuccessful join request processing..................... 140
- Reason for last unsuccessful join attempt................ Failed to add database entry
- Time at last successful join attempt..................... Not applicable
- Time at last unsuccessful join attempt................... Apr 05 17:09:08.269

Configuration phase statistics

--More-- or (q)uit
- Configuration requests received.......................... 0
- Successful configuration responses sent.................. 0
- Unsuccessful configuration request processing............ 0
- Reason for last unsuccessful configuration attempt....... Not applicable
- Time at last successful configuration attempt............ Not applicable
- Time at last unsuccessful configuration attempt.......... Not applicable

Last AP message decryption failure details
- Reason for last message decryption failure............... Not applicable

Last AP disconnect details
- Reason for last AP connection failure.................... Not applicable
- Last AP disconnect reason................................ Not applicable

Last join error summary
- Type of error that occurred last......................... Unsuccessful Lwapp join response sent
- Reason for error that occurred last...................... Failed to add database entry
- Time at which the last join error occurred............... Apr 05 17:09:08.269

AP disconnect details
- Reason for last AP connection failure.................... Not applicable
 Ethernet Mac : 00:00:00:00:00:00  Ip Address : 172.16.1.97

if you got time please have a look at the wireshark capture it maybe helpful.

[Scott Fella]

Look at the output from the console of the ap now since you have disabled the FW.

[mohfarid20]

this is the output of the console

AP4055.398e.0872#
*Apr  5 15:04:30.001: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 172.16.1.102:5246
*Apr  5 15:04:30.007: %DOT11-5-EXPECTED_RADIO_RESET: Restarting Radio interface Dot11Radio0 due to the reason code 27
*Apr  5 15:04:30.007: %DOT11-5-EXPECTED_RADIO_RESET: Restarting Radio interface Dot11Radio1 due to the reason code 27
*Apr  5 15:04:30.067: %CAPWAP-5-AP_EASYADMIN_INFO: AP Easy Admin information - EASY_ADMIN is not set, turn off easy admin service!
*Apr  5 15:04:30.067: %CAPWAP-5-AP_EASYADMIN_INFO: AP Easy Admin information - Easy Admin is not enabled, turn it off!
*Apr  5 15:04:30.080: %DOT11-5-EXPECTED_RADIO_RESET: Restarting Radio interface Dot11Radio0 due to the reason code 39
*Apr  5 15:04:30.080: %DOT11-5-EXPECTED_RADIO_RESET: Restarting Radio interface Dot11Radio1 due to the reason code 39
*Apr  5 15:04:30.099: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to administratively down
*Apr  5 15:04:30.099: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to administratively down
*Apr  5 15:04:31.105: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down
*Apr  5 15:04:31.127: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to down
*Apr  5 15:04:31.134: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to reset
*Apr  5 15:04:32.121: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up
*Apr  5 15:04:32.128: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to down
*Apr  5 15:04:32.150: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up
*Apr  5 15:04:32.156: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to down
*Apr  5 15:04:32.162: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset
*Apr  5 15:04:33.150: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to up
*Apr  5 15:04:33.156: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down
*Apr  5 15:04:33.181: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up
*Apr  5 15:04:34.182: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up
*Apr  5 15:04:49.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 172.16.1.102 peer_port: 5246
*Apr  5 15:04:49.264: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 172.16.1.102 peer_port: 5246
*Apr  5 15:04:49.264: %CAPWAP-5-SENDJOIN: sending Join Request to 172.16.1.102
*Apr  5 15:04:54.265: %CAPWAP-5-SENDJOIN: sending Join Request to 172.16.1.102
AP4055.398e.0872#

it keeps on repeating like this all day.