topic Re: AP's disassociate during WLC upgrade in Wireless

AP's disassociate during WLC upgrade

Tony Davis — Sun, 04 Jul 2021 04:23:48 GMT

I'm upgrading (6) WiSM 1's(12 controllers) from 7.0.98 to 7.0.116. I've completed 3 controllers total so far, and on every one of them, the AP's disassociate during the upgrade process. This happens when "Writing new APIB to flash disk"... How am I supposed to predownload the image to the AP's if they keep disassociating?

I started a continuous ping to the 3rd upgraded controller before starting the upgrade process. Ping times were normal, <1ms, in the beginning... then spiked to 30-40ms in the final stages of the upgrade, which is when the AP's disassociated. Coincidence or cause?

Thanks in advance!

Tony

AP's disassociate during WLC upgrade

Scott Fella — Tue, 17 Jan 2012 13:56:20 GMT

That doesn't seem right.... I haven't upgraded a WiSM in a while, but on the stand-alone WLC, you do not have the AP's disassociate from the WLC during the upload process. Might want to see what others say or just open a TAC case.

AP's disassociate during WLC upgrade

Tony Davis — Tue, 17 Jan 2012 14:05:38 GMT

Actually, the pings rise starting with "Executing backup script", continue through RTOS to flash,

AP's started disassociating during the apib install

dropped more during the "fini script"

All are completely disassociated by the end of the upgrade

AP's disassociate during WLC upgrade

Tony Davis — Tue, 17 Jan 2012 14:08:33 GMT

Log from WLC GUI........

AP's Interface:1(802.11a) Operation State Down: Base Radio MAC:30:37:a6:c9:3b:00 Cause=Heartbeat Timeout Status:NA

AP's Interface:0(802.11b) Operation State Down: Base Radio MAC:30:37:a6:c9:3b:00 Cause=Heartbeat Timeout Status:NA

Have messages like this for every AP

Re: AP's disassociate during WLC upgrade

Scott Fella — Tue, 17 Jan 2012 14:09:36 GMT

I would open a TAC case… maybe there is a bug id for that, I just don’t know.

AP's disassociate during WLC upgrade

George Stefanick — Tue, 17 Jan 2012 14:11:29 GMT

I can confirm this. I have a ton of WLCs I just updated and during the unpacking the aps would do the same thing on my 4400 and WISMs. Although I didnt notice it on the 5508s. I suspect the controller processor is under load and misses a few of the control plane packets.

How this became apparent for me is because of my WCS email alerts. I got blasted with ap dissociations.

Re: AP's disassociate during WLC upgrade

Stephen Rodriguez — Tue, 17 Jan 2012 14:12:24 GMT

that a actually makes sense as that's the time the WLC is getting busy and pings could be delayed as its 'scavenger' traffic.

But they should stay associated until you reboot. Now there were some issues In earlier code where odd issues, AP rebooting, WLC pushing the code that was downloaded, before the WLC reboots.

Out of curiosity, are you pruning the non-wireless v,an from the WiSM port channel? Did you set a syslog for the AP? Excess broadcast traffic could cause an issue like this.

Steve

Sent from Cisco Technical Support iPad App

AP's disassociate during WLC upgrade

George Stefanick — Tue, 17 Jan 2012 14:23:12 GMT

Tony,

My issue was when the WLCs took the code. Once the code was on the WLCs the APs stay associated. Are you say when you push the image they are dropping ? Ive heard of this as well. A guy commented on twitter about this a few weeks ago. That I havent seen for myself.

Re: AP's disassociate during WLC upgrade

Tony Davis — Tue, 17 Jan 2012 14:31:52 GMT

This is during the code download to the controller. The AP's disassociate before I get a chance to push the image to the AP's...

Re: AP's disassociate during WLC upgrade

George Stefanick — Tue, 17 Jan 2012 14:41:37 GMT

Ok, thats what I seen as well. But once the aps come back (like 2 seconds later) they stay on, or atleast they did with all of mine. After the WLC takes the new code and is done unpacking you then do the predownload.

I will mentioned I had an issue with 7.0.98.0 and predownloading to the aps. But it wasnt related to what you mentioned.

Re: AP's disassociate during WLC upgrade

Tony Davis — Tue, 17 Jan 2012 14:43:11 GMT

We do not prune any VLANs on the port channels for the WiSM's... It is apparent that the controller is "busy", but what is making the AP's heartbeat timer expire?

Re: AP's disassociate during WLC upgrade

Stephen Rodriguez — Tue, 17 Jan 2012 14:54:07 GMT

because the wl. Is too busy processing other information and does not reply to the heartbeat from the AP. I bet of you looked at the secondary WLC you would see the ap moved when the heartbeat failed to its primary.

Can you try pruning the non-wireless vlan and setting a syslog for the ap so they aren't broadcasting, before you upgrade the next WLC and see if they remain stable on their primary?

Steve

Sent from Cisco Technical Support iPhone App

Re: AP's disassociate during WLC upgrade

George Stefanick — Tue, 17 Jan 2012 15:05:08 GMT

Give the pruning a shot. Although my 40+ only allow the wireless vlans and we still had this issue. Also, we did notice some move to the secondary while most did not. They came right back up on the primary.

I suspect this is the norm and unless you pay extra close attention to the ap while upgrading you may not even notice this going on...

Re: AP's disassociate during WLC upgrade

Tony Davis — Tue, 17 Jan 2012 15:19:24 GMT

These 6509 chassis are dedicated to wireless vlans. The only VLAN that is allowed through those etherchannels that's NOT wireless is the management VLAN. I seriously doubt that it is congesting anything. All other VLANs are wireless related.

Re: AP's disassociate during WLC upgrade

Tony Davis — Tue, 17 Jan 2012 15:27:22 GMT

what if I disable the heartbeat timer during the upgrade?

Re: AP's disassociate during WLC upgrade

Stephen Rodriguez — Tue, 17 Jan 2012 15:43:47 GMT

don't disable it but you could extend by maybe 5 seconds. Just remember to put it back when you're done upgrading.

Steve

Sent from Cisco Technical Support iPhone App

Re: AP's disassociate during WLC upgrade

George Stefanick — Tue, 17 Jan 2012 15:45:56 GMT

Tony

You mentioned this happens during the wlc upgrade. What does this have to do with the predownload.

Sent from Cisco Technical Support iPhone App

Re: AP's disassociate during WLC upgrade

Tony Davis — Tue, 17 Jan 2012 16:17:01 GMT

If the AP's keep moving to another controller while I'm upgrading the current controller, then I'll never have the opporunity to predownload the image to the AP, which is the whole point of this new feature. This results in a longer downtime because when the AP moves to a controller that has been upgraded, it has to download a new image and then reboot.

Consider the following scenario:

C1 has 40 AP's, running 7.0.98

C2 has 30 AP's, running 7.0.98

Upgrade software on C1 to 7.0.116, AP's move to C2 before having chance to predownload image to AP's

C1 has 0 AP's, running 7.0.116

C2 has 70 AP's, running 7.0.98

Upgrade C2 to 7.0.116, AP's disassociate and join C1, now has to download image(couldn't predownload) and reload

Is that more clear?

Re: AP's disassociate during WLC upgrade

Stephen Rodriguez — Tue, 17 Jan 2012 16:22:49 GMT

If you don't reboot the WLC until all are upgraded, it shouldn't be a problem. Even if the AP move, they hit a WLC with the code they are running and won't download by default. Then once all WLC are upgraded you can run the pre-download command, wait for that to finish, then do rolling reboots, or wait till 2am and reboot them all.

All depends on what you can do inside your organization WRT the rebooting of the individual WLC.

Steve

Re: AP's disassociate during WLC upgrade

George Stefanick — Tue, 17 Jan 2012 16:29:14 GMT

Tony,

Its very clear, but thats not the advisable process. You will have a mess on your hands for sure. I have 40 WLCs and the upgrade process is just that a process.

1. You upgrade the WLCs

2. After upgrade you then push the code to the AP

3. You then swap the code on the aps to primary

4. Then you reboot all controllers

Thats why you are having problems.

I blogged about this process. This may help you better understand the process.

http://www.my80211.com/cisco-wlc-cli-commands/2011/2/20/wlc-predownload-the-image-to-the-access-points-from-the-cont.html

I need my morning coffee