WLC 4402 - APs last reboot reason power loss

[lydia.walther]

Hi,

we have a Wireless LAN Controller WLC 4402, Software Version 4.1.192.22M (Mesh).

At the moment there are 6 Access Points in one house, one of them is the root AP. The others are only connected via electrical socket.

AP’s are: AIR-LAP1131AG-E-K9 / Boot Version 12.3.8.0 / IOS Version 12.4(3g)JMC1

I think the configuration worked fine over a period of 2 years.

Now the problem is that the number of access points connected to the controller is changing nearly every minute. At times there is only one AP connected to the controller. So it is impossible for the clients to build up a steady connection. They have to log in via the guest user web interface at short intervals or they can’t see the wireless lan.

First we thought it would be a problem of one or two “defect” access points. But we replaced 4 of the 6 access points and the problem is still there.

An example, yesterday:

13:17 o’clock – 2 AP’s

13:19 o’clock – 4 AP’s

13:21 o’clock – 5 AP’s

13:23 o’clock – 4 AP’s

13:25 o’clock – 5 AP’s

13:25 o’clock – 4 AP’s

13:26 o’clock – 5 AP’s

13:26 o’clock – 6 AP’s

13:28 o’clock – 4 AP’s

13:28 o’clock – 2 AP’s

…

Eye-catching is the following message of the log, which appears often: “52       Mon May 16 13:21:56 2011        AP 'AP001d.e557.6fd8', MAC: 00:1d:70:01:bc:20 disassociated previously due to AP Reset. Last reboot reason: power loss

”

Also eye-catching is the AP up time. The Root AP is up for over 80 days. But all the other access points show an up time of a few minutes until some hours…but not more.

Does anybody know what the problem could be?? I read some similar threads but no solution.

In the following an abstract of the log:

21        Mon May 16 13:25:43 2011        AP 'AP0021.d847.ffca', MAC: 00:23:5e:49:9d:e0 disassociated previously due to AP Reset. Last reboot reason: power loss

22        Mon May 16 13:25:30 2011        AP Disassociated. Base Radio MAC:00:1d:70:01:b5:a0

23        Mon May 16 13:25:30 2011        AP's Interface:1(802.11a) Operation State Down: Base Radio MAC:00:1d:70:01:b5:a0 Cause=Heartbeat Timeout

24        Mon May 16 13:25:30 2011        AP's Interface:0(802.11b) Operation State Down: Base Radio MAC:00:1d:70:01:b5:a0 Cause=Heartbeat Timeout

25        Mon May 16 13:25:05 2011        Mesh child node '00:23:5e:49:9d:ef' has changed its parent to mesh node '00:1f:ca:cc:b7:40' from mesh node '00:1d:70:01:bc:20'.

26        Mon May 16 13:24:10 2011        AP Disassociated. Base Radio MAC:00:23:5e:49:9d:e0

27        Mon May 16 13:24:10 2011        AP's Interface:1(802.11a) Operation State Down: Base Radio MAC:00:23:5e:49:9d:e0 Cause=New Discovery

28        Mon May 16 13:24:10 2011        AP's Interface:0(802.11b) Operation State Down: Base Radio MAC:00:23:5e:49:9d:e0 Cause=New Discovery

29        Mon May 16 13:23:59 2011        AP's Interface:0(802.11b) Operation State Up: Base Radio MAC:00:1d:71:e1:b2:20 Cause=Admin Configured

30        Mon May 16 13:23:59 2011        AP's Interface:0(802.11b) Operation State Down: Base Radio MAC:00:1d:71:e1:b2:20 Cause=Admin Configured

31        Mon May 16 13:23:59 2011        AP's Interface:1(802.11a) Operation State Up: Base Radio MAC:00:1d:71:e1:b2:20 Cause=Admin Configured

32        Mon May 16 13:23:59 2011        AP's Interface:0(802.11b) Operation State Up: Base Radio MAC:00:1d:71:e1:b2:20 Cause=Admin Configured

33        Mon May 16 13:23:58 2011        AP 'AP001d.45d8.4ea6', MAC: 00:1d:71:e1:b2:20 disassociated previously due to AP Reset. Last reboot reason: power loss

34        Mon May 16 13:23:31 2011        AP's Interface:0(802.11b) Operation State Up: Base Radio MAC:00:1d:70:01:b5:a0 Cause=Admin Configured

35        Mon May 16 13:23:31 2011        AP's Interface:0(802.11b) Operation State Down: Base Radio MAC:00:1d:70:01:b5:a0 Cause=Admin Configured

36        Mon May 16 13:23:31 2011        AP's Interface:1(802.11a) Operation State Up: Base Radio MAC:00:1d:70:01:b5:a0 Cause=Admin Configured

37        Mon May 16 13:23:31 2011        AP's Interface:0(802.11b) Operation State Up: Base Radio MAC:00:1d:70:01:b5:a0 Cause=Admin Configured

38        Mon May 16 13:23:30 2011        AP 'AP001d.e557.6f0c', MAC: 00:1d:70:01:b5:a0 disassociated previously due to AP Reset. Last reboot reason: power loss

39        Mon May 16 13:23:13 2011        Client Association: Client MAC:00:1d:71:e1:b2:2f Base Radio MAC :00:1d:70:01:bc:20 Slot: 1 User Name:c1130-001D45D84EA6

40        Mon May 16 13:23:13 2011        Mesh child node '00:1d:71:e1:b2:2f' has changed its parent to mesh node '00:1d:70:01:bc:20' from mesh node '00:1f:ca:cc:b7:40'.

41        Mon May 16 13:23:04 2011        Mesh child node '00:1d:70:01:b5:af' has changed its parent to mesh node '00:23:5e:49:9d:e0' from mesh node '00:1f:ca:cc:b7:40'.

42        Mon May 16 13:22:57 2011        AP 'AP0021.d847.ffca', MAC: 00:23:5e:49:9d:e0 disassociated previously due to AP Reset. Last reboot reason: power loss

43        Mon May 16 13:22:39 2011        AP Disassociated. Base Radio MAC:00:23:5e:49:9d:e0

44        Mon May 16 13:22:39 2011        AP's Interface:1(802.11a) Operation State Down: Base Radio MAC:00:23:5e:49:9d:e0 Cause=New Discovery

45        Mon May 16 13:22:39 2011        AP's Interface:0(802.11b) Operation State Down: Base Radio MAC:00:23:5e:49:9d:e0 Cause=New Discovery

46        Mon May 16 13:22:39 2011        AP 'AP0021.d847.ffca', MAC: 00:23:5e:49:9d:e0 disassociated previously due to AP Reset. Last reboot reason: power loss

47        Mon May 16 13:22:17 2011        Mesh child node '00:23:5e:49:9d:ef' has changed its parent to mesh node '00:1d:70:01:bc:20' from mesh node '00:1f:ca:cc:b7:40'.

48        Mon May 16 13:21:57 2011        AP's Interface:0(802.11b) Operation State Up: Base Radio MAC:00:1d:70:01:bc:20 Cause=Admin Configured

49        Mon May 16 13:21:57 2011        AP's Interface:0(802.11b) Operation State Down: Base Radio MAC:00:1d:70:01:bc:20 Cause=Admin Configured

50        Mon May 16 13:21:57 2011        AP's Interface:1(802.11a) Operation State Up: Base Radio MAC:00:1d:70:01:bc:20 Cause=Admin Configured

51        Mon May 16 13:21:57 2011        AP's Interface:0(802.11b) Operation State Up: Base Radio MAC:00:1d:70:01:bc:20 Cause=Admin Configured

52        Mon May 16 13:21:56 2011        AP 'AP001d.e557.6fd8', MAC: 00:1d:70:01:bc:20 disassociated previously due to AP Reset. Last reboot reason: power loss

53        Mon May 16 13:21:39 2011        Client Association: Client MAC:00:23:5e:49:9d:ef Base Radio MAC :00:1f:ca:cc:b7:40 Slot: 1 User Name:c1130-0021D847FFCA

54        Mon May 16 13:21:32 2011        Client Association: Client MAC:00:23:5e:49:9d:ef Base Radio MAC :00:1f:ca:cc:b7:40 Slot: 1 User Name:c1130-0021D847FFCA

55        Mon May 16 13:21:12 2011        Mesh child node '00:1d:71:e1:b2:2f' has changed its parent to mesh node '00:1f:ca:cc:b7:40' from mesh node '00:1d:70:01:b5:a0'.

56        Mon May 16 13:21:05 2011        Client Association: Client MAC:00:1d:70:01:bc:2f Base Radio MAC :00:1f:ca:cc:b7:40 Slot: 1 User Name:c1130-001DE5576FD8

57        Mon May 16 13:20:32 2011        Mesh child node '00:23:5e:49:9d:ef' has changed its parent to mesh node '00:1f:ca:cc:b7:40' from mesh node '00:1d:70:01:bc:20'.

58        Mon May 16 13:20:11 2011        Mesh child node '00:1d:70:01:bc:2f' is no longer associated with mesh node '00:1f:ca:cc:b7:40'.

59        Mon May 16 13:20:06 2011        Mesh child node '00:1d:70:01:b5:af' has changed its parent to mesh node '00:1f:ca:cc:b7:40' from mesh node '00:1d:70:01:bc:20'.

60        Mon May 16 13:19:57 2011        AP Disassociated. Base Radio MAC:00:1d:70:01:b5:a0

61        Mon May 16 13:19:57 2011        AP's Interface:1(802.11a) Operation State Down: Base Radio MAC:00:1d:70:01:b5:a0 Cause=Heartbeat Timeout

62        Mon May 16 13:19:57 2011        AP's Interface:0(802.11b) Operation State Down: Base Radio MAC:00:1d:70:01:b5:a0 Cause=Heartbeat Timeout

63        Mon May 16 13:19:49 2011        AP Disassociated. Base Radio MAC:00:1d:70:01:bc:20

64        Mon May 16 13:19:49 2011        AP's Interface:1(802.11a) Operation State Down: Base Radio MAC:00:1d:70:01:bc:20 Cause=Heartbeat Timeout

65        Mon May 16 13:19:49 2011        AP's Interface:0(802.11b) Operation State Down: Base Radio MAC:00:1d:70:01:bc:20 Cause=Heartbeat Timeout

66        Mon May 16 13:19:39 2011        AP Disassociated. Base Radio MAC:00:23:5e:49:9d:e0

67        Mon May 16 13:19:39 2011        AP's Interface:1(802.11a) Operation State Down: Base Radio MAC:00:23:5e:49:9d:e0 Cause=Heartbeat Timeout

68        Mon May 16 13:19:39 2011        AP's Interface:0(802.11b) Operation State Down: Base Radio MAC:00:23:5e:49:9d:e0 Cause=Heartbeat Timeout

69        Mon May 16 13:19:36 2011        AP Disassociated. Base Radio MAC:00:1d:71:e1:b2:20

70        Mon May 16 13:19:36 2011        AP's Interface:1(802.11a) Operation State Down: Base Radio MAC:00:1d:71:e1:b2:20 Cause=Heartbeat Timeout

71        Mon May 16 13:19:36 2011        AP's Interface:0(802.11b) Operation State Down: Base Radio MAC:00:1d:71:e1:b2:20 Cause=Heartbeat Timeout

Greetings Lydia

[Leo Laohoo]

How long has the WLC been up?

Have you tried out the new 7.0.98.0 or later firmware?

[lydia.walther]

It has been up for 27 days. A reboot was one of the first things we tried.

I think 7.0.98.0 firmware is only for controllers which are not running in mesh mode, isn't it? For mesh controllers the latest software is 4.2.207.54M and my controller has 4.1.192.22M

Maybe I should try the 4.2.207.54M.

[Leo Laohoo]

I think 7.0.98.0 firmware is only for controllers which are not running in mesh mode, isn't it? For mesh controllers the latest software is 4.2.207.54M and my controller has 4.1.192.22M

Starting from 6.X, mesh APs are supported.

Special Rules for Upgrading to Controller Software 7.0.116.0 in Mesh Networks

[lydia.walther]

Hi,

thank you for the link. Didn't know that.I will have a look.

I'm only wondering that there seems to be no Maintenance release of an Software version about 6.x

We made a few bad experiences with ED-Software on other wireless lan controllers.

[Leo Laohoo]

Didn't know that.

Me too until someone pointed my nose to the right direction.

I'm only wondering that there seems to be no Maintenance release of an Software version about 6.x

I'm recommending people to use the 7.0.116.0 firmware. 

[Leo Laohoo]

Didn't know that.

Me too until someone pointed my nose to the right direction.

I'm only wondering that there seems to be no Maintenance release of an Software version about 6.x

I'm recommending people to use the 7.0.116.0 firmware. 

[dipumj]

Hi All,

There is DOS in version below 6.0.196.0 for WLC 2100 series

       Summary   Status A vulnerability exists in several Cisco Wireless LAN Controller   products that could allow a remote, unauthenticated attacker to cause a   denial of service condition.               Fixed       The following products are affected:           Cisco 2100 Series Wireless LAN Controllers   Severity Cisco WLC526 Mobility Express Controller (AIR-WLC526-K9)           Cisco NME-AIR-WLC Modules for Integrated Services Routers (ISRs)   2   - severe Cisco NM-AIR-WLC Modules for Integrated Services Routers (ISRs)               

Note:   The Cisco NM-AIR-WLC have reached End-of-Life and End-of-Software   Maintenance. Please refer to the following document for more information:   http://www.cisco.com/en/US/prod/collateral/modules/ps2797/prod_end-of-life_notice0900aecd806aeb34....

            In   Last 3 Days                   Cisco has released free software updates that address this   vulnerability. There is no workaround for this vulnerability.   Product              This   advisory is posted at   http://www.cisco.com/warp/public/707/cisco-sa-20110427-wlc.shtml Cisco   4400 Series Wireless LAN Controllers                   PSIRT Evaluation:   Technology The Cisco PSIRT has assigned this bug the following CVSS version   2 score. The Base and Temporal CVSS scores as of the time of evaluation are   7.8/6.4:           https://intellishield.cisco.com/security/alertmanager/cvssCalculator.do?dispatch=1&version=2&vector=AV:N/AC:L/Au:N/C:N/I:N/A:C/E:F/RL:OF/RC:C                     CVE ID CVE-2011-1613 has been assigned to document this issue.   1st Found-In              Additional information on Cisco's security vulnerability policy   can be found at the following URL:   6.0(198.0)       http://www.cisco.com/en/US/products/products_security_vulnerability_policy.html  

    Fixed-In                  7.0(116.0)           7.0(98.216)           6.0(202.0)           6.0(200.0)           7.0(112.0)

with regards,

Dipu

[lydia.walther]

Hey,

last week I tried to upgrade the controller step-by-step.

Under the software 4.1.192.35M the problem was still there… access points were often disassociated from the controller.

There were problems with Software 4.2.207.54M too. The access points couldn’t reach the download-status. The log of the access point said that it could not open the tar-file.

So I went back to 4.1.192.35M. Of course the controller lost some of its configuration. I configured it new with the same settings as before.

After it the wireless connection seemed to be a little bit more robust. I asked the users to test the connection at the weekend and this is the answer:

“so we tested the WLAN during the weekend. The situation definitely improved with respect to before. Authetication is much faster and the connection (when active) is sensibly faster.

However the connection is still very unstable and it is necessary every five-ten minutes to reconnect (especially for intense network traffic like when watching a streaming content or using VOIP applications. Actually switching off and on again the WLAN card (Airport) often a new authentication is not required. However, without doing so the connection would not resume alone to a working state.

So what to say? Better than before (thanks) but far from being fixed. From last September to January we did not experience any problem, so it must be something that chronologically happened at the beginning of the year, it is not a systemic problem.

”

But there were no changes or anything else at the beginning of the year.

Do you mean it makes sense to resume upgrading? I’m a little bit afraid of more problems like under 4.2.207.54M

Greetings Lydia