AP's disassociate during WLC upgrade

[Tony Davis]

I'm upgrading (6) WiSM 1's(12 controllers) from 7.0.98 to 7.0.116.  I've completed 3 controllers total so far, and on every one of them, the AP's disassociate during the upgrade process.  This happens when "Writing new APIB to flash disk"...  How am I supposed to predownload the image to the AP's if they keep disassociating? 

I started a continuous ping to the 3rd upgraded controller before starting the upgrade process.  Ping times were normal, <1ms, in the beginning... then spiked to 30-40ms in the final stages of the upgrade, which is when the AP's disassociated.  Coincidence or cause?

Thanks in advance!

Tony

[Scott Fella]

That doesn't seem right.... I haven't upgraded a WiSM in a while, but on the stand-alone WLC, you do not have the AP's disassociate from the WLC during the upload process.  Might want to see what others say or just open a TAC case.

[Tony Davis]

Actually, the pings rise starting with "Executing backup script", continue through RTOS to flash,

AP's started disassociating during the apib install

dropped more during the "fini script"

All are completely disassociated by the end of the upgrade

[Tony Davis]

Log from WLC GUI........

AP's Interface:1(802.11a) Operation State Down: Base Radio MAC:30:37:a6:c9:3b:00  Cause=Heartbeat Timeout Status:NA

AP's Interface:0(802.11b) Operation State Down: Base Radio MAC:30:37:a6:c9:3b:00  Cause=Heartbeat Timeout Status:NA

Have messages like this for every AP

[Scott Fella]

I would open a TAC case… maybe there is a bug id for that, I just don’t know.

[George Stefanick]

I can confirm this. I have a ton of WLCs I just updated and during the unpacking the aps would do the same thing on my 4400 and WISMs. Although I didnt notice it on the 5508s. I suspect the controller processor is under load and misses a few of the control plane packets.

How this became apparent for me is because of my WCS email alerts. I got blasted with ap dissociations.

[Stephen Rodriguez]

that a actually makes sense as that's the time the WLC is getting busy and pings could be delayed as its 'scavenger' traffic.

But they should stay associated until you reboot. Now there were some issues In earlier code where odd issues, AP rebooting, WLC pushing the code that was downloaded, before the WLC reboots.

Out of curiosity, are you pruning the non-wireless v,an from the WiSM port channel? Did you set a syslog for the AP? Excess broadcast traffic could cause an issue like this.

Steve

Sent from Cisco Technical Support iPad App

[Tony Davis]

We do not prune any VLANs on the port channels for the WiSM's...  It is apparent that the controller is "busy", but what is making the AP's heartbeat timer expire?

[George Stefanick]

Tony,

My issue was when the WLCs took the code. Once the code was on the WLCs the APs stay associated. Are you say when you push the image they are dropping ? Ive heard of this as well. A guy commented on twitter about this a few weeks ago. That I havent seen for myself.

[Tony Davis]

This is during the code download to the controller.  The AP's disassociate before I get a chance to push the image to the AP's...

[George Stefanick]

Ok, thats what I seen as well. But once the aps come back (like 2 seconds later) they stay on, or atleast they did with all of mine. After the WLC takes the new code and is done unpacking you then do the predownload.

I will mentioned I had an issue with 7.0.98.0 and predownloading to the aps. But it wasnt related to what you mentioned.

[Stephen Rodriguez]

because the wl. Is too busy processing other information and does not reply to the heartbeat from the AP. I bet of you looked at the secondary WLC you would see the ap moved when the heartbeat failed to its primary.

Can you try pruning the non-wireless vlan and setting a syslog for the ap so they aren't broadcasting, before you upgrade the next WLC and see if they remain stable on their primary?

Steve

Sent from Cisco Technical Support iPhone App

[George Stefanick]

Give the pruning a shot. Although my 40+ only allow the wireless vlans and we still had this issue. Also, we did notice some move to the secondary while most did not. They came right back up on the primary.

I suspect this is the norm and unless you pay extra close attention to the ap while upgrading you may not even notice this going on...

[Tony Davis]

These 6509 chassis are dedicated to wireless vlans.  The only VLAN that is allowed through those etherchannels that's NOT wireless is the management VLAN.  I seriously doubt that it is congesting anything. All other VLANs are wireless related.

[Tony Davis]

what if I disable the heartbeat timer during the upgrade?