Mesh LAP strange reload

esavorani · ‎09-02-2011

Hi, our costumers has implemented 2 AIR-WLC4402-50-K9 with Software Release 7.0.98.0, the wireless infrastructure consist in 2 Root-Mesh-LAP and 8 Mesh connect over-the-air to deploy outdoor coverage.

All the LAP are Aironet 1520 Series Mesh Access Points with equipped with 3 antennas for 2.4GHz and 1 antenna for 5GHz (backhaul).

For one year all seems to be ok, yesterday after a power outage of one Mesh-Root-LAP, 5 Mesh-Lap continues reload each 10-12 minutes, on the WLC Log you can see event like a reboot from AP Console, on the LAP console i can capture this event before the reload:

Log on LAP Mesh

%DOT11-6-GEN_ERROR: Error on Dot11Radio0 - Not Beaconing for too long - Current 0 Last 0

%SYS-5-RELOAD: Reload requested by Dot11 driver. Reload Reason:

Radio Not Beaconing for too long ....

LWAPP-5-CHANGED: CAPWAP changed state to DOWN

AP1780-Mesh uptime is 11 hours, 10 minutes

System returned to ROM by power spike

%DOT11-6-GEN_ERROR: Error on Dot11Radio0 - Not Beaconing for too long - Current 0 Last 0
%SYS-5-RELOAD: Reload requested by Dot11 driver. Reload Reason:
Radio Not Beaconing for too long ....
*Sep 1 16:05:43.399: %LWAPP-5-CHANGED: CAPWAP changed state to DOWN

What does it mean? That the beacon signal trasmitted from Root-Mesh-LAP cannot reach the Mesh-Lap and so the Mesh-LAP force a reload?

Where we should search the cause? In the power instability or in a interference on the 5GHz radio interface?

On one of mesh Lap I found a strange reason for a releoad:

AP1780-Mesh uptime is 11 hours, 10 minutes
System returned to ROM by power spike

Log on WLC

Log System Time Trap

0 Thu Sep 1 17:31:11 2011 AP Disassociated. Base Radio MAC:00:22:be:41:33:00

1 Thu Sep 1 17:31:11 2011 AP's Interface:1(802.11a) Operation State Down: Base Radio MAC:00:22:be:41:33:00 Cause=Heartbeat Timeout Status:NA

2 Thu Sep 1 17:31:11 2011 AP's Interface:0(802.11b) Operation State Down: Base Radio MAC:00:22:be:41:33:00 Cause=Heartbeat Timeout Status:NA

3 Thu Sep 1 17:31:06 2011 AAA Authentication Failure for UserName:c1520-0022BE40C000 User Type: WLAN USER

4 Thu Sep 1 17:29:59 2011 Mesh child node '00:22:be:40:c0:0f' is no longer associated with mesh node '00:1e:49:5b:60:00'.

5 Thu Sep 1 17:29:43 2011 AP Disassociated. Base Radio MAC:00:22:be:40:c0:00

6 Thu Sep 1 17:29:43 2011 AP's Interface:1(802.11a) Operation State Down: Base Radio MAC:00:22:be:40:c0:00 Cause=Heartbeat Timeout Status:NA

7 Thu Sep 1 17:29:43 2011 AP's Interface:0(802.11b) Operation State Down: Base Radio MAC:00:22:be:40:c0:00 Cause=Heartbeat Timeout Status:NA

8 Thu Sep 1 17:28:27 2011 AP's Interface:1(802.11a) Operation State Up: Base Radio MAC:00:22:be:40:d5:00 Cause=Radio reset due to Init. Status:NA

9 Thu Sep 1 17:28:27 2011 AP's Interface:0(802.11b) Operation State Up: Base Radio MAC:00:22:be:40:d5:00 Cause=Radio reset due to Init. Status:NA

10 Thu Sep 1 17:28:26 2011 AP 'AP1705-Mesh', MAC: 00:22:be:40:d5:00 disassociated previously due to AP Reset. Uptime: 0 days, 00 h 02 m 12 s . Last reset reason: reboot command from AP console.

11 Thu Sep 1 17:27:50 2011 AAA Authentication Failure for UserName:c1520-0022BE40D500 User Type: WLAN USER

12 Thu Sep 1 17:27:50 2011 Mesh child node '00:22:be:40:d5:0f' has changed its parent to mesh node '00:1e:49:5b:60:00' from mesh node '00:22:be:40:e3:0f'.

13 Thu Sep 1 17:27:11 2011 Mesh child node '00:22:be:40:d5:0f' is no longer associated with mesh node '00:22:be:40:e3:00'. Log System Time Trap
0 Thu Sep 1 17:31:11 2011 AP Disassociated. Base Radio MAC:00:22:be:41:33:00
1 Thu Sep 1 17:31:11 2011 AP's Interface:1(802.11a) Operation State Down: Base Radio MAC:00:22:be:41:33:00 Cause=Heartbeat Timeout Status:NA
2 Thu Sep 1 17:31:11 2011 AP's Interface:0(802.11b) Operation State Down: Base Radio MAC:00:22:be:41:33:00 Cause=Heartbeat Timeout Status:NA
3 Thu Sep 1 17:31:06 2011 AAA Authentication Failure for UserName:c1520-0022BE40C000 User Type: WLAN USER
4 Thu Sep 1 17:29:59 2011 Mesh child node '00:22:be:40:c0:0f' is no longer associated with mesh node '00:1e:49:5b:60:00'.
5 Thu Sep 1 17:29:43 2011 AP Disassociated. Base Radio MAC:00:22:be:40:c0:00
6 Thu Sep 1 17:29:43 2011 AP's Interface:1(802.11a) Operation State Down: Base Radio MAC:00:22:be:40:c0:00 Cause=Heartbeat Timeout Status:NA
7 Thu Sep 1 17:29:43 2011 AP's Interface:0(802.11b) Operation State Down: Base Radio MAC:00:22:be:40:c0:00 Cause=Heartbeat Timeout Status:NA
8 Thu Sep 1 17:28:27 2011 AP's Interface:1(802.11a) Operation State Up: Base Radio MAC:00:22:be:40:d5:00 Cause=Radio reset due to Init. Status:NA
9 Thu Sep 1 17:28:27 2011 AP's Interface:0(802.11b) Operation State Up: Base Radio MAC:00:22:be:40:d5:00 Cause=Radio reset due to Init. Status:NA
10 Thu Sep 1 17:28:26 2011 AP 'AP1705-Mesh', MAC: 00:22:be:40:d5:00 disassociated previously due to AP Reset. Uptime: 0 days, 00 h 02 m 12 s . Last reset reason: reboot command from AP console.
11 Thu Sep 1 17:27:50 2011 AAA Authentication Failure for UserName:c1520-0022BE40D500 User Type: WLAN USER
12 Thu Sep 1 17:27:50 2011 Mesh child node '00:22:be:40:d5:0f' has changed its parent to mesh node '00:1e:49:5b:60:00' from mesh node '00:22:be:40:e3:0f'.
13 Thu Sep 1 17:27:11 2011 Mesh child node '00:22:be:40:d5:0f' is no longer associated with mesh node '00:22:be:40:e3:00'.

Thanks a lot.

George Stefanick · ‎09-02-2011

I am going to venture to say this is a TAC case. I wonder during this blink in coverage your ap took a crap ..

"Satisfaction does not come from knowing the solution, it comes from knowing why." - Rosalind Franklin
___________________________________________________________

weterry · ‎09-03-2011

%DOT11-6-GEN_ERROR: Error on Dot11Radio0 - Not Beaconing for too long - Current 0 Last 0

%SYS-5-RELOAD: Reload requested by Dot11 driver. Reload Reason:

Radio Not Beaconing for too long ....

The AP reboot is a recovery mechanism when a radio is hung and hasn't transmitted any beacons in 10 minutes. You'd rather the AP reboot itself to try to start working again than forever be down (I think that is the logic at least).. No Beacons effectively results in no clients trying to connect (or stay connected)....

This is radio0, so that should be your 2.4Ghz radio and have nothing to do with Mesh itself.

Now, I'm aware of several pre-7.0.98.0 beacon loss/reboots, but nothing comes to mind for 7.0.98.0.

When in doubt, upgrade at least to get off 7.0.98.0 in general.....

George Stefanick · ‎09-03-2011

Keep the good info coming Wesley!

+5

I love details! "Book marking"

"Satisfaction does not come from knowing the solution, it comes from knowing why." - Rosalind Franklin
___________________________________________________________

esavorani · ‎09-05-2011

Thanks for your info! weterry. I haven't noticed that the radio in effect is radio0! In your opinion the reason of reload is that are no clients connected, but it's strange, Do you think that a good test can be tring to connected a client to this 3 LAP that continously reboot?

weterry · ‎09-05-2011

If the AP isn't beaconing.... I seriously doubt you're going to get any clients to connect.....

You say you're on 7.0.98.0? My suggestion would be to upgrade to 7.0.116.0 and then see what happens.. (but I dont have any particular bug as suspect)

esavorani · ‎09-08-2011

Today I have test a client connection on one of 3 LAP that continously reboot, The connection was succesfull without any strange reload or messages for 30 minutes.

At the same time I decide to set correctly the power level and channel for 2.4 GHz radio interface for All outdoor mesh LAP AIR-LAP1522AG-E-K9 (all the LAP have 2 8 dBi Omnidirection Antenna one for 2.4 and one for 5 GHz) the customers had set improperly to Power Level = 1 and static channel.

After the modification and the test on only one LAP, the 3 LAP stopped their reload with the strange message:

%DOT11-6-GEN_ERROR: Error on Dot11Radio0 - Not Beaconing for too long - Current 0 Last 0

%SYS-5-RELOAD: Reload requested by Dot11 driver. Reload Reason:

Radio Not Beaconing for too long ....

At the moment are UP from 8 Hours!! Seems to be solved but...

this afternoon i have decide to open a Cisco TAC, I will keep you informed

thanks a lot

esavorani · ‎10-04-2011

From: [mailto:xxxxx@cisco.com]
Sent: lunedì 12 settembre 2011 22:03

Cc: attach@cisco.com
Subject: RE: SR 618989113 - 09/09 - wlc4402: AP beacon stops on Ap1522

Hello,

Thanks for the info, from the information shown, I see you are running 7.0.98.0, this version has some serious issues with radio firmware that will cause the beacon stoppage as you have seen, it recommended that you move to either 7.0.98.218 or 7.0.116.0.

https://www.cisco.com/en/US/docs/wireless/controller/release/notes/crn7_0_116_0.html

https://www.cisco.com/en/US/docs/wireless/controller/release/notes/crn7_0_98_218.html

I have included URL’s to the release notes..