AIR-CAP3602I-A-K9 cannot join WLC - packet loss on WAN

[jfgrenier]

Hi,

I have a WLC 2504 with 10 licences running 7.3.2.112.0.  I have 7 AP joined from different remote sites (WAN).  The other APs are 3602E.

I got a brand new 3602I that I want to join, from a remote site and I am getting these error messages and cannot join :

*Dec 21 14:35:18.859: %CAPWAP-5-DHCP_OPTION_43: Controller address x.x.x.x obtained through DHCP

*Dec 21 14:35:18.859: %CAPWAP-3-ERRORLOG: Did not get log server settings from DHCP.

*Dec 21 15:26:51.875: %CAPWAP-3-ERRORLOG: Go join a capwap controller

*May  9 14:48:39.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: x.x.x.x peer_port: 5246

*May  9 14:49:11.131: DTLS_CLIENT_ERROR: ../capwap/base_capwap/dtls/base_capwap_dtls_connection_db.c:2051 Max retransmission count reached!

*May  9 14:49:38.999: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to x.x.x.x:5246

As far as I can tell, the AP seems to be loosing packets that are coming/going away from its own subnet.

AP address : 192.168.1.1 (mask 255.255.255.0)

WLC address : 192.168.2.1 (mask 255.255.255.0)

If I ping the AP address from a computer in the *same subnet*, I get a perfect 0% packet loss.

If I ping the AP address from a computer in ANY *other subnets*, I get about 50% packet loss.

If I ping anything else *other than this single AP*, to and from any other subnets, I get a perfect 0% packet loss.

Only packets orginating or sent to the AP from other subnets are getting lost.

To me it looks like there is something wrong with the routing table of the AP, as if some packets were not properly returned ?

Any ideas to where I should look ?

Thanks a lot,

JFG

[Scott Fella]

I wouldn't worry too much about pings unless from one site your ping times are bad to the gateway of the other site in which the AP is connected.

You are using option 43, do you see the AP initially join? Are all your AP's in flex mode or local mode? Is the AP have the same country code as what is configured on the WLC? Is the other side where the AP is at behind a NAT?

Sent from Cisco Technical Support iPhone App

[Leo Laohoo]

Please post the following command outputs:

1.  WLC:  sh sysinfo;

2.  AP:  sh version; and

3.  AP:  sh inventory

[gwhite]

Hi Jean,

I had this issue on some brand new 3602i+e models that were received offsite.  We were getting 50% ping loss from the AP to the Controller and the AP couldn't complete a join.  Once we mirrored the AP's switchport to a sniffer we found that the AP was actually only sending 2 of the 5 echo requests and nothing was getting dropped, whatever was getting dropped never actually made it to the wire.

The fix we found was to put the AP on the same VLAN as the controller.  At that point the AP was able to connect and download updated code from the Controller.  Once the AP had reloaded with the new code we were able to put it back on it's original VLAN and it was able to connect via L3.

Controller code:

AIR-CT5500-K9-7-2-115-1.aes

Cisco needs to test the code being shipped on these APs.  Unfortunately I wasn't able to log a support case for this as the equipment was ordered by a private individual without support contracts and time was a major constraint.  I'm glad we haven't ordered these for our enterprise yet as this would have cost us huge $$$.  I'm lucky this was a small onsite visit with 30 APs.  If this happened at a remote site for our company, we would have had to RMA everything and incur further physical installation costs.

[Michel Pedersen]

Hi,

We are experiencing the same issue with a batch of new 3602I-E-K9 AP's where we get 50% packetloss regardless of where we try to ping them. They are also unable to connect to the central WLC to upgrade their firmware.

Do you know which firmware you had one yours when they weren't working? I notice from a CDP that it lists the software version as 12.4(25e)JAL1 so I'm curious if it's the same version you had when experiencing problems.

In any case; we'll try what you found to resolve your problem and connect them to the same network as the WLC and see if that works (just have to get them shipped from the other end of the country first...)

-Michel

[gwhite]

Hi Michel,

Sounds very similar to the issue I ran into.

Are you dropping packets when you ping directly from the default gateway address of the AP or a host on the same subnet?  You will have to use an extended ping command from the switch/router providing it's Cisco.    Sorry but I didn't write down the software version and I no longer have access to the site/equipment.  The software was definitely factory shipped with the APs though.

One idea I have instead of shipping all the AP's accross the country, is if you can get a TFTP server on the same subnet as the APs and try upgrading them via command line.  Have you logged a TAC case for this?  I couldn't due to the customer's lack of support contracts.

[Erwin Salazar]

Hi Guys,

This is actually a known issue related to the following caveat:

CSCue56163 - 12.4(25e)JAL1 AP recovery img does not work as expected.

The 12.4(25e)JAL1 recovery image installs two default routes in the AP's routing table: one to the default gateway, and the other to the interface. The latter route works only if proxy ARP is enabled on the gateway.  As a result, without proxy ARP, every other IP packet transmitted by the AP is dropped.  For more information on this issue, see the following CSC article:  New 3600 Series Access Points Cannot Join a WLC https://supportforums.cisco.com/docs/DOC-30836

Workaround:

1. Make sure that ip proxy-arp is configured (default setting for an IOS router), on the AP's subnet's default gateway. Also if ip  broadcast-address is defined on the vlan with something other than  255.255.255.255 the AP will not join. Either no this command or set it  to broadcast.  2. If console access is available on the AP, then disable IP routing -  then it should be able to join, and download the new IOS image:  ap#debug capwap console cli ap#configure terminal ap(config)#no ip routing  (wait for it to join)  This setting will not survive a reboot.  3. Install a different recovery (rcvk9w8) or lightweight IOS (k9w8)  image on the AP, such as 15.2(2)JA1

Cheers,
Erwin

______________________________________

How helpful was I? Don't forget to rate me when you have the chance!

[Michel Pedersen]

Hi Erwin,

thanks for the info. I just found out the same last night myself and have confirmed that disabling ip routing resolves the issue. After disabling IP routing through a console connection the AP was able to access the WLC and upgrade itself to a newer firmware version.

Just to reiterate;

Connecting a console cable to the AP and doing this resolves the issue:

ap#debug capwap console cli (Important as without this "configure terminal" will not be available on a lightweight AP)

ap#configure terminal

ap(config)#no ip routing

Waited 5 minutes and the AP was updated and rebooted itself.

I was not able to check the proxy arp solution to this as on this particular site I run Juniper routers.

regards

Michel

[gwhite]

Hi Erwin,

Thank you for the work arounds, this is great!  I'm sure this posting will help many others googling for answers.

5 Gold stars!