cisco AP (3702i) @ remote location not able download image from controller over option 43

[kbodiwala]

cisco AP (3702i) @ remote location not able download image from controller over option 43.

AP is getting IP address locally from switch through IP DHCP pool.

Option 60 & option 43 configured.

Able to ping controller at remote location over site to site vpn tunnel.

Controller IP Address - 10.1.1.72

remote location AP IP address - 10.7.4.5

ip dhcp pool Wi-Fi-AP
 import all
 network 10.7.4.0 255.255.255.248
 default-router 10.7.4.1
 dns-server 4.2.2.2
 option 60 ascii "Cisco AP c3700"
 option 43 hex f104.0a01.0148
 lease infinite

-------------------------
Device ID: AP002a.10a0.4440
Entry address(es):
  IP address: 10.7.4.5
Platform: cisco AIR-CAP3702I-B-K9,  Capabilities: Trans-Bridge Source-Route-Bridge
Interface: GigabitEthernet0/13,  Port ID (outgoing port): GigabitEthernet0
Holdtime : 174 sec

Version :
Cisco IOS Software, C3700 Software (AP3G2-RCVK9W8-M), Version 15.3(3)JC102, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2016 by Cisco Systems, Inc.
Compiled Fri 06-May-16 05:33 by prod_rel_team

advertisement version: 2
Duplex: full
Power drawn: 15.400 Watts
Power request id: 23615, Power management id: 1
Power request levels are:16800 15400 13000 0 0
Power Available TLV:

    Power request id: 0, Power management id: 0, Power available: 0, Power management level: 0
Management address(es):
  IP address: 10.7.4.5

[Lubos Zelinsky]

What's your WLC AirOs version?

[kbodiwala]

its 2504 WLC at remote location running software version 7.6.110.0.

[apindoria]

I had a similar issue with this bug when running that code on a Flex 7510.

Please check this https://quickview.cloudapps.cisco.com/quickview/bug/CSCuo35247

[Lubos Zelinsky]

Ok your software code does not support -B domain access point, which you're using AIR-CAP3702I-B-K9. Please see more details on the link below:

http://www.cisco.com/c/en/us/products/collateral/wireless/aironet-3700-series/bulletin-c25-737028.html

If you want support for your 3702 -B domain access point you have to upgrade at least to the software code 8.0.132.0 - B domain AP support introduced for 8.0 software code:

http://www.cisco.com/c/en/us/td/docs/wireless/controller/release/notes/crn80mr3.html#pgfId-1298773

Although I would rather pick 8.0.140.0 software as far as this is a Cisco suggested release for 8.0.

[kbodiwala]

I updated WLC software version to -8.0.140.0 , still getting same on controller. On the controller traps getting following message, it see AP then it get disassociated.

AP Disassociated. Base Radio MAC:00:2a:10:a0:44:40 ApName - AP002a.10a0.4440

[Lubos Zelinsky]

Can you gather outputs from the following commands?

debug capwap events enable

debug capwap errors enable

debug pm pki enable

show ap join stats detailed 00:2a:10:a0:44:40

[kbodiwala]

Cisco Controller) >debug capwap events enable

(Cisco Controller) >*spamApTask2: Mar 15 17:18:55.148: 54:a2:74:9b:91:c0 WTP Event Request from 10.1.100.126:14500

*spamApTask2: Mar 15 17:19:45.584: 00:2a:10:a0:44:40 ApModel: AIR-CAP3702I-B-K9

*spamApTask2: Mar 15 17:19:45.584: 00:2a:10:a0:44:40 Echo Timer Expiry: Missing Echo from AP00:2a:10:a0:44:40, Closing dtls Connection.
*spamApTask2: Mar 15 17:19:45.584: 00:2a:10:a0:44:40 Finding DTLS connection to delete for AP (10:7:4:5/1093)
*spamApTask2: Mar 15 17:19:45.584: 00:2a:10:a0:44:40 Disconnecting DTLS Capwap-Ctrl session 0x17e72498 for AP (10:7:4:5/1093)

*spamApTask2: Mar 15 17:19:45.584: 00:2a:10:a0:44:40 CAPWAP State: Dtls tear down

*spamApTask2: Mar 15 17:19:45.584: 00:2a:10:a0:44:40 acDtlsPlumbControlPlaneKeys: lrad:10.7.4.5(1093) mwar:10.1.1.72(5246)

*spamApTask2: Mar 15 17:19:45.584: 00:2a:10:a0:44:40 DTLS keys for Control Plane deleted successfully for AP 10.7.4.5

*spamApTask2: Mar 15 17:19:45.587: 00:2a:10:a0:44:40 apType = 46 apModel: AIR-CAP3702I-B-K9

*spamApTask2: Mar 15 17:19:45.587:
*spamApTask2: Mar 15 17:19:45.587: 00:2a:10:a0:44:40 Discovery Response sent to 10.7.4.5 port 1093

*spamApTask2: Mar 15 17:19:45.587: 00:2a:10:a0:44:40 Discovery Response sent to 10.7.4.5:1093

*spamApTask2: Mar 15 17:19:45.588: 00:2a:10:a0:44:40 DTLS connection closed event receivedserver (10.1.1.72/5246) client (10.7.4.5/1093)
*spamApTask2: Mar 15 17:19:45.588: 00:2a:10:a0:44:40 Entry exists for AP (10.7.4.5/1093)
*spamApTask2: Mar 15 17:19:45.588: 00:2a:10:a0:44:40 update ap status:00:2a:10:a0:44:40 ,index:11

*spamApTask2: Mar 15 17:19:45.588: 00:2a:10:a0:44:40 Unable to find deleted AP 00:2a:10:a0:44:40
*spamApTask2: Mar 15 17:19:45.588: 00:2a:10:a0:44:40 No AP entry exist in temporary database for 10.7.4.5:1093
**spamApTask2: Mar 15 17:19:55.575: 00:3a:7d:bd:6c:ff DTLS connection not found, creating new connection for 10:7:4:5 (1093) 10:1:1:72 (5246)

*spamApTask2: Mar 15 17:19:55.575: sshpmGetCID: called to evaluate <cscoSha2IdCert>

*spamApTask2: Mar 15 17:19:55.575: sshpmGetCID: comparing to row 4, ID cert >bsnSslWebauthCert<

*spamApTask2: Mar 15 17:19:55.575: sshpmGetCID: comparing to row 3, ID cert >bsnSslWebadminCert<

*spamApTask2: Mar 15 17:19:55.575: sshpmGetCID: comparing to row 2, ID cert >cscoDefaultIdCert<

*spamApTask2: Mar 15 17:19:55.575: sshpmGetCID: comparing to row 1, ID cert >bsnDefaultIdCert<

*spamApTask2: Mar 15 17:19:55.575: sshpmGetCID: comparing to row 0, ID cert >bsnOldDefaultIdCert<

*spamApTask2: Mar 15 17:19:55.575: sshpmGetCID: failed to find matching cert.

*spamApTask2: Mar 15 17:19:55.575: sshpmGetDERIDCert: Using SHA2 Id cert on WLC

*spamApTask2: Mar 15 17:19:55.575: sshpmGetCID: called to evaluate <cscoDefaultIdCert>

*spamApTask2: Mar 15 17:19:55.575: sshpmGetCID: comparing to row 4, ID cert >bsnSslWebauthCert<

*spamApTask2: Mar 15 17:19:55.575: sshpmGetCID: comparing to row 3, ID cert >bsnSslWebadminCert<

*spamApTask2: Mar 15 17:19:55.575: sshpmGetCID: comparing to row 2, ID cert >cscoDefaultIdCert<

*spamApTask2: Mar 15 17:19:55.575: sshpmGetCertFromCID: called to get cert for CID 1c681f38

*spamApTask2: Mar 15 17:19:55.575: sshpmGetCertFromCID: comparing to row 4, certname >bsnSslWebauthCert<

*spamApTask2: Mar 15 17:19:55.575: sshpmGetCertFromCID: comparing to row 3, certname >bsnSslWebadminCert<

*spamApTask2: Mar 15 17:19:55.575: sshpmGetCertFromCID: comparing to row 2, certname >cscoDefaultIdCert<

*spamApTask2: Mar 15 17:19:55.575: sshpmGetCID: called to evaluate <cscoSha2IdCert>

*spamApTask2: Mar 15 17:19:55.575: sshpmGetCID: comparing to row 4, ID cert >bsnSslWebauthCert<

*spamApTask2: Mar 15 17:19:55.575: sshpmGetCID: comparing to row 3, ID cert >bsnSslWebadminCert<

*spamApTask2: Mar 15 17:19:56.173: 00:3a:7d:bd:6c:ff acDtlsPlumbControlPlaneKeys: lrad:10.7.4.5(1093) mwar:10.1.1.72(5246)

*spamApTask2: Mar 15 17:19:56.174: 00:3a:7d:bd:6c:ff Allocated index from main list, Index: 10

*spamApTask2: Mar 15 17:19:56.174: 00:3a:7d:bd:6c:ff Using CipherSuite AES128-SHA
*spamApTask2: Mar 15 17:19:56.175: 00:3a:7d:bd:6c:ff DTLS keys for Control Plane are plumbed successfully for AP 10.7.4.5. Index 11

*spamApTask3: Mar 15 17:19:56.175: 00:3a:7d:bd:6c:ff DTLS Session established server (10.1.1.72:5246), client (10.7.4.5:1093)
*spamApTask3: Mar 15 17:19:56.175: 00:3a:7d:bd:6c:ff Starting wait join timer for AP: 10.7.4.5:1093
*spamApTask2: Mar 15 17:20:01.256: 00:2a:10:a0:44:40 Join Request from 10.7.4.5:1093

*spamApTask2: Mar 15 17:20:01.257: 00:3a:7d:bd:6c:ff Deleting AP entry 10.7.4.5:1093 from temporary database.
*spamApTask2: Mar 15 17:20:01.257: 00:2a:10:a0:44:40 AP with same name AP002a.10a0.4440 exist. Using default name AP002a.10a0.4440 for this AP. radId 10 dupRadId 10dupApMac:  00:2A:10:A0:44:40

*spamApTask2: Mar 15 17:20:01.257: 00:2a:10:a0:44:40 Join Version: = 134375780

*spamApTask2: Mar 15 17:20:01.257: 00:2a:10:a0:44:40 apType = 46 apModel: AIR-CAP3702I-B-K9

*spamApTask2: Mar 15 17:20:01.257:
*spamApTask2: Mar 15 17:20:01.257: 00:2a:10:a0:44:40 Join resp: CAPWAP Maximum Msg element len = 89

*spamApTask2: Mar 15 17:20:01.257: 00:2a:10:a0:44:40 Join Response sent to 10.7.4.5:1093

*spamApTask2: Mar 15 17:20:01.257: 00:2a:10:a0:44:40 CAPWAP State: Join

*spamApTask2: Mar 15 17:20:01.257: 00:2a:10:a0:44:40 capwap_ac_platform.c:1548 - Operation State 0 ===> 4

*apfMsConnTask_7: Mar 15 17:20:31.885: sending spamAddMobile aclName =

Sync phase statistics
- Time at sync request received............................ Not applicable
- Time at sync completed................................... Not applicable

Discovery phase statistics
- Discovery requests received.............................. 25
- Successful discovery responses sent...................... 12
- Unsuccessful discovery request processing................ 7
- Reason for last unsuccessful discovery attempt........... Discarding LWAPP Discovery Request from AP Since entry exists in CAPWAP
- Time at last successful discovery attempt................ Mar 15 17:19:45.587
- Time at last unsuccessful discovery attempt.............. Mar 15 17:19:45.584

Join phase statistics
- Join requests received................................... 13
- Successful join responses sent........................... 13
- Unsuccessful join request processing..................... 0
- Reason for last unsuccessful join attempt................ Not applicable
- Time at last successful join attempt..................... Mar 15 17:20:01.257
- Time at last unsuccessful join attempt................... Not applicable

Configuration phase statistics

--More-- or (q)uit
- Configuration requests received.......................... 0
- Successful configuration responses sent.................. 0
- Unsuccessful configuration request processing............ 0
- Reason for last unsuccessful configuration attempt....... Not applicable
- Time at last successful configuration attempt............ Not applicable
- Time at last unsuccessful configuration attempt.......... Not applicable

Last AP message decryption failure details
- Reason for last message decryption failure............... Not applicable

Last AP disconnect details
- Reason for last AP connection failure.................... Not applicable
- Last AP disconnect reason................................ Not applicable

Last join error summary
- Type of error that occurred last......................... Lwapp discovery request rejected
- Reason for error that occurred last...................... Discarding LWAPP Discovery Request from AP Since entry exists in CAPWAP
- Time at which the last join error occurred............... Mar 15 17:19:45.584

AP disconnect details
- Reason for last AP connection failure.................... Not applicable
                                                                           Ethernet Mac : 00:2a:10:a0:44:40  Ip Address : 10.7.4.5

*spamApTask2: Mar 15 17:21:01.347: 00:2a:10:a0:44:40 ApModel: AIR-CAP3702I-B-K9

[Lubos Zelinsky]

I can see your AP ends up in a join CAPWAP state on the WLC, however does not move to the next stage what's configuration - there are no configuration requests received and some of the discovery responses didn't reach the AP.

It might be something wrong with the connection between the AP and the WLC - what's your round trip latency on that link, is your CAPWAP control traffic prioritized over that WAN link?

If the link is stable and there is no problem with it, I would suggest to open a TAC case on this.

[apindoria]

Is the AP registering with the controller at all?

- Can you ping the controller IP when sourcing the ping from the gateway (L3 / SVI interface) of the AP

- Check local router/firewall to see if packet destined for the controller is even being sent

- Check routers/firewall in the path to see if the udp/5246 is being dropped

- As it is over a VPN tunnel ensure you have the MTU configured on the outside interfaces

[kbodiwala]

Yes able to ping controller IP via source vlan of AP's IP subnet.

verified UDP/5246 - yes its sending packet to remote WLC, WLC see that and  AP gets Disassociated.. and Also MTU configured for both inside/outside interface for 1500.

[Chandana Abeysinghe]

I've seen the some APs come with "Bridge" mode. Try adding the MAC address of the AP to MAC filtering( Security / AAA / TACACS+ / Mac Filtering). Once the the AP is joined, you may change it to the  mode you want (Flexconnect)

[kbodiwala]

I have applied MAC address of AP to MAC filtering on WLC, no change, still getting same

AP Disassociated. Base Radio MAC:00:2a:10:a0:44:40 ApName - AP002a.10a0.4440