AP541N - Degrading Service

[lucas4]

We have an AP541N that has been deployed to replace a Cisco 1200 AP (B/G radio).  The 1200 functioned perfectly in our environment.  The new AP541N on the other hand seems to work fine right after a reboot but immediately starts to degrade service.  Over a short period of time, the devices bandwidth degrades to the point were the wireless network is not usable.  This happens with just one device connected.  Eventually, the device stops accepting client connections.  We are unable to get any relevant logging out of the device to help diagnose the problem.

What should be our course of action?

Lucas Budman

Cognilytics

[sbaharaeen]

Jens,

I was having a problem sending you a private message, but I figured it out and sent it to you.

[opresterud]

Hi!

We was hit severly by this today, but have also seen it sporadicaly during testing. The system consists of 4 AP541 accesspoints with firmware 1.8.0 running in a cluster with 802.11bgn support, where 3 is connected directly to a SA540 gateway with cable, and one acts as repeater with WDS connection to one of the cabled AP's. All AP's broadcasts with the same SSID and on channel 9.

We have applied the recommended key refresh fix and removed WPA2 support. The AP's are now configures with WPA-PSK encryption with both TKIP and AES available. Todays usage scenario was about 20 laptop users roaming around for ~3 hours, and potentially moving from one AP to another. The internet connection is a shared 10Mbit access, where we typically measures a bandwith of 5-7Mbit/sec with speed tests at our location through a wireless AP that is connected with cable to the gateway. The speed degrade did occur on all AP's today, but some was hit more often than others (might seem to be related to number of clients connected). When the degrade happens, we could first sence it that the signal strength from the particular AP started to fluctuate more than normal and also drop for a periode of 1-2 seconds before reappearing (monitored using 'inSSIDer' WI-FI scanner), and if we then connected to the particular AP and ran a speed test through it, it had dropped down in the range 300-100Kbps, and after a short while it would hit ~60Kbps and stay there.

DHCP issues have not been seen here (DHCP assigned by the SA540), connecting to a degraded AP was always possible it seemed, but general all access through an AP that had the degrade state was extremely slow. To be able to access the web interface of such an AP, we typically had to ensure we had a connection to a different AP and then enter the web interface for the degraded AP through the normal functioning one.

To get it properly working for our users, we had to constantly monitor the four AP's, and do a reboot at first sign of degrade. Thankfully, since the AP's overlap each other, most of the connected clients was transferred to a different AP when we hit reboot, but without constantly personally monitoring this, the WLAN had practically gone down.

For us, this is a very serious issue, and really hope for a fix in a short time periode. We would gladly provide any kind of information requested about the issue, and also test other recommended fixes or workarounds for this if available.

- Ola

[ActiveInfoTech]

Is ther an update or time frame when this will be fixed.  Just deployed the AP yesterday to replace a Linksys and the Cisco AP541N is showing 1M connection where the Linksys is showing 11M connection.

[Steven DiStefano]

Tier 2 Support and the BU have communicated to us that the Level 1 SBSC agents were informed of this AP541N issue and another DHCP issue as being solved in the next FW release for this product.  When customers call SBSC, the support the agents will follow proper protocol and due diligence to determine the problem, and in the event it is considered to be resolved in the coming release, the case will be escalated to 2^nd level support.  At that time and with the Partners agreement, access to the beta firmware 1.9.0 in test right now can be given, once a beta agreement has been accepted.

Any Partner experiencing either DHCP issues or degrading service, please contact the SBSC for further resolution.

I do not have an ETA for the formal release of 1.9.0.

Steve DiStefano

Systems Engineer,

Partner Sales, U.S.

[farrshepherd]

Per your post below, here is our case # on this issue. We are a Cisco Select partner and have these Ap's at many locations are experiencing the problem at most of the locations.

Case # for Cisco Wireless Issue
614127237

[lucas4]

Cisco has replaced my original AP.  I have just deployed the new AP.  Unfortunately, I believe that it is experiencing the same degrading problem.  It is running firmware 1.8 (same as the original AP).  One interesting thing I did notice though is the following:

From the linux console the AP is reporting a lot of packet errors on the wireless link (this is with one client attached and uptime of one day):

wlan0     Link encap:Ethernet  HWaddr 00:21:29:00:7D:50 

          UP BROADCAST RUNNING  MTU:1500  Metric:1

          RX packets:162000 errors:650285 dropped:0 overruns:0 frame:15361294

          TX packets:421509 errors:3336 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:42518247 (40.5 MiB)  TX bytes:291051214 (277.5 MiB)

          Interrupt:44

Here are the radio interface stats for my Cisco 1200 in the same environment (Uptime on this AP is 4 weeks and has several clients connected):

Dot11Radio0 is up, line protocol is up

  Hardware is 802.11G Radio, address is 0012.01e5.2e90 (bia 0012.01e5.2e90)

  MTU 1500 bytes, BW 54000 Kbit, DLY 1000 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation 802.1Q Virtual LAN, Vlan ID  1., loopback not set

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 00:00:00, output 00:00:00, output hang never

  Last clearing of "show interface" counters never

  Input queue: 0/75/523/0 (size/max/drops/flushes); Total output drops: 130206

  Queueing strategy: fifo

  Output queue: 0/30 (size/max)

  5 minute input rate 5000 bits/sec, 9 packets/sec

  5 minute output rate 4000 bits/sec, 4 packets/sec

     29633972 packets input, 2302875179 bytes, 4 no buffer

     Received 235828 broadcasts, 0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 input packets with dribble condition detected

     26072809 packets output, 2208254017 bytes, 0 underruns

     21421 output errors, 0 collisions, 3 interface resets

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier

     0 output buffer failures, 0 output buffers swapped out

Lucas

[Steven DiStefano]

OK,  spoke to development team.

They suggested we may understand the 'degrading' issue and have a possible workaround, which may be related to "Key refresh".

Try to just disable 'Key Refresh' in each VAP.   Set the Broadcast Key Refresh Rate to 0 for each vap that does WPA(personal or enterprise).

[sbaharaeen]

I just heard back from the second tier support and they recommended disabling the Key Refresh as well.  I will try this tomorrow and let everyone know how it goes.  Is everyone who is experiencing this issue using WPA?

[jwalker]

I was having issues when I first bought the unit.  I switched over to N mode only and I'm only using WPA2 with AES and things have been running fine.

I will enable WPA and TKIP and attempt this workaround today.

Justin

[dmoore]

Has changing the Refresh Rate to 0 worked for anyone? I changed mine this morning but have not seen any difference. Pings to our core router that were 1ms on the LinkSys the AP541N replaced are still all over the place and are often in the 1000's on the 541.

[Steven DiStefano]

Dennis and all,

These workarounds are suggested until we can be sure of the root cause and fix it.

Based on what we think is going on, there are code changes under test right now.

But what would really be helpful, is if everyone can post their logs (attach to your discussion threads) so we can see clearly what is happening in your environments?

Steve

[sbaharaeen]

Came in this morning and the AP is not responding at all, even to the Windows 7 client that was working fine last week.  We are taking it offline and switching to our old setup.  I will try to run it in a test environment.  Attached are the logs from this morning.

[Steven DiStefano]

Thank you for the logs.   I passed them to development team.

Will advise whan I hear back.

[psmith]

My only user using this AP reported losing it 5 times yesterday.

He said it disappears, and then comes back about 5 minutes later.

I can't test it because it is on the other side of Canada from me.

But here is my log file, since the last reboot.

I will ask them not to reboot it again so I can capture more logs.

[Steven DiStefano]

Thanks for this.  Feeling pretty confident, based on your description of whats happening, that what we are fixing will address this, but more logs would be welcomed as well in the mean time since these logs are just showing disassociate deauth from an unknown reason.

Here are the reasons codes our team found in the log file.

1     /* Unspecified reason */

3     /* Deauthenticated because sending station is leaving (or has left) IBSS or ESS

4     /* Disassociated due to inactivity */

8     /* Disassociated because sending station is leaving (or has left) BSS

.

We will keep you posted and thank you for your help...