H-Reap APs do not joint WLC after a WAN outage

[marco_bartulihe]

I have a deployment with 16 H-Reap Access Points with static IP configured. There is a Windows DHCP server with option 43 configured a DNS for CISCO.

When I turn on the Access Points, they find and join the WLC normally. However, after a WAN outage (the Access Points lose connectivity with the WLC), the Access Points can´t find the WLC again. Here’s an output of a command debug lwapp client event from the LAP:

*Oct 10 17:09:17.733: %CAPWAP-5-CHANGED: CAPWAP changed state to DISCOVERY

*Oct 10 17:09:27.734: %CAPWAP-3-ERRORLOG: Selected MWAR 'EC-WLC1'(index 0).

*Oct 10 17:09:27.734: %CAPWAP-3-ERRORLOG: Go join a capwap controller

*Oct 10 17:09:28.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.16.0.50 peer_port: 5246

*Oct 10 17:09:28.000: %CAPWAP-5-CHANGED: CAPWAP changed state to 

*Oct 10 17:09:29.429: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 10.16.0.50 peer_port: 5246

*Oct 10 17:09:29.431: %CAPWAP-5-SENDJOIN: sending Join Request to 10.16.0.50

*Oct 10 17:09:29.431: %CAPWAP-5-CHANGED: CAPWAP changed state to JOIN

*Oct 10 17:09:29.618: %CAPWAP-5-CHANGED: CAPWAP changed state to CFG

., 1)10 17:09:32.616: %CAPWAP-3-ERRORLOG: Retransmission count for packet exceeded max(UNKNOWN_MESSAGE_TYPE (5)

*Oct 10 17:09:32.616: %CAPWAP-3-ERRORLOG: GOING BACK TO DISCOVER MODE

*Oct 10 17:09:32.616: %DTLS-5-SEND_ALERT: Send WARNING : Close notify Alert to 10.16.0.50:5246

*Oct 10 17:09:32.730: %CAPWAP-5-CHANGED: CAPWAP changed state to DISCOVERY

*Oct 10 17:09:42.732: %CAPWAP-3-ERRORLOG: Selected MWAR 'EC-WLC1'(index 0).

*Oct 10 17:09:42.732: %CAPWAP-3-ERRORLOG: Go join a capwap controller

*Oct 10 17:09:42.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.16.0.50 peer_port: 5246

*Oct 10 17:09:42.000: %CAPWAP-5-CHANGED: CAPWAP changed state to 

*Oct 10 17:09:43.432: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 10.16.0.50 peer_port: 5246

*Oct 10 17:09:43.433: %CAPWAP-5-SENDJOIN: sending Join Request to 10.16.0.50

*Oct 10 17:09:43.434: %CAPWAP-5-CHANGED: CAPWAP changed state to JOIN

*Oct 10 17:09:43.622: %CAPWAP-5-CHANGED: CAPWAP changed state to CFG

., 1)10 17:09:46.620: %CAPWAP-3-ERRORLOG: Retransmission count for packet exceeded max(UNKNOWN_MESSAGE_TYPE (5)

*Oct 10 17:09:46.620: %CAPWAP-3-ERRORLOG: GOING BACK TO DISCOVER MODE

I need to reboot the LAP to it find the WLC again.

Can someone help me, please?

Thank you very much.

Marco Bartulihe

[Nicolas Darchis]

Hi Marco,

you probably noticed tha the official complain of the ap is "max retransmission count exceeded". This combined with the fact that a reboot makes it work, make me come with several theories.

I think most of them would be confirmed by a sniffer trace taken at the AP port.

- If you have some specific firewall rules, maybe some part of the traffic is blocked. I've seen similar issues when the APs were behind a Natted network where the NAT wouldn't work when the APs were changing their source port (WLC port is always the capwap one while AP port is dynamic)

-Maybe it is an MTU negociation issue and it's renegotiated after a reboot.

In any case, the sniffer trace would show which is the packet that doesn't reach WLC because obviously the discovery is fine, it's the join where it seems some packets are dropped.

Nicolas

[Joseph Chambers]

Did you find out what is wrong and more importantly, how to fix?

I'm having same issue with 1131s and HREAP.

[marco_bartulihe]

No, we are still with this problem.

The WLC is configured to use LAG and is connected to two switches 6500 configured as VSS.

Thanks

[Joseph Chambers]

I've got a tac case open and they (TAC) are stumped.

[marco_bartulihe]

Hi guys,

For comparation purposes, let´s say the hardware used (APs and WLCs) and versions.

I am using the WLC 4402 (version 6.0.199.0) and APs 1242.

Today I found a BUG (CSCsy23704) that may explain this wrong behavior.

[Joseph Chambers]

I am using the WLC 2112 (version 6.0.199.0) and APs 1131.

[marco_bartulihe]

Hello Joe,

I am looking at Cisco site and the image 6.0.199.0 was deferred!

It was released in July and was deferred because the following problem:

The CAM table are not updated after a L2 roamming.

The image 6.0.199.0 was replaced by 6.0.199.4. The latest release is a 7.0 version, but it is too new to use in a production site.

I will try to update the WLC for the version .4 and see what happens. I will post the results.

If anyone update the WLC first, please post the results too.

Thanks

Marco Bartulihe.

[Joseph Chambers]

My bad...I'm on 6.0.199.4.  So that probably won't help you.

[oguarisco]

Hello all,

I confirm that Joe has reason, the same problem is happening to a WLC installed with Release 6.0.199.4...

I hope that Cisco will come out asap with a patched WLC-OS to avoid this annoying problem

Omar

[marco_bartulihe]

That´s a big problem, and the Cisco isn't helping a lot!

In the CSCsy23704 BUG details, the image 6.0.199.4 don´t appears in "Fixed-In".

The fixed-in images are:

6.0(121.0)

5.2(183.0)

12.4(21a)JA

6.0(182.0)

5.2(193.0)

As we can see in the attached file: Bug CSCsy23704.PNG.

We can try a downgrade to one of the fixed-in images or an upgrade to the newest 7.0.98.0(ED) image.

Have anyone tried the image 7.0.98.0?

Best regards,

Marco Bartulihe.

[oguarisco]

Hello Marco,

thanks for the useful information, I've the same dilemma but I think the downgrade will be the best solution after reading carefully the open bug of the Release chosen and the Resolved bug of the 6.0.199 ... hoping that there aren't others which will impact the business continuity

I'll keep you updated

Regards

Omar

[marco_bartulihe]

Hello Oguarrisco,

A downgrade shoud be the best option, but for what version?

I thought that a downgrade for 6.0.182.0 was a good ideia, but this image has several problems and should be avoided.

In the next day I will try the 7.0.98.0 and post the results.

If you have a solution with an image, please let us know.

Thank you very much,

Marco Bartulihe

[Joseph Chambers]

Did you try 7.0.98.0? Did it fix it?

[marco_bartulihe]

Hi guys,

I'm glad to say that we upgraded the WLC to version 7.0.98.0 and it worked out!

The issue was solved, so we saw that the bug CSCsy23704 affects the WLC model 4402 with version 6.0.199.4.

Regards,

Marco Bartulihe