5508 Predownload not working

[Dan]

Hi

We have a 5508 on version 7.3.101.0. I have copied version 8.3.130.0 to it and it's ready to be rebooted. I tried to do a pre-download of the image to all my APs (all AIR-CAP2602I-E-K9) but it didn't go very well. About 200 out of 330 of them immediately dropped from the controller. When i SSH'd to it and watched one it was failing the image transfer and then the radio was turning off every 5 seconds for about 5-10 minutes, disconnecting users every time. Eventually they started working again but the predownload had failed. Only 1 actually predownloaded successfully. 3 have disappeared completely. I thought it was meant to happen in the background with no drops and to prevent downtime - not cause it.

When I consoled onto one i had next to me and attempted a predownload i saw that it was failing because of insufficient space but it didn't try to delete anything then did the same failure pattern before coming back again.

Freeing up enough space from the AP allows the predownload to complete - i deleted the recovery image so now it has a primary and a backup but no recovery - don't know if this is recommended. Obviously I would rather not do this individually for 330 APs... Is there a way I can do it through the controller on all of them?? Or is there anything else I can do? I don't really want the downtime of just rebooting the controller and letting them update that way, plus now I'm anxious that some of them are going to fail and disappear like the 3 from before.

Thanks

[Flavio Miranda]

Hi @Dan

   That´s a really bad situation. 2600 AP has only 32 MB memory space. Recovery image is 8MB, Primary Image is 14MB which leaves 10MB for the Secondary Image which is not enough.

 This AP is meant to be upgraded without Pre-Download apparently. 

The only solution for this is run a script deleting the recovery image as you already did. Or upgrade without pre-download.

  Run a script for this can be challenging but I attached a Python script that I used in the past for a similar situation. This script perform a test in a range of IP with ping. If an IP respond ping then the script connect to the IP and execute the command you want. You need to change it accordingly.

 This is useful if your APs use DHCP and you are not sure which IP they are using now. Hope that helps.

-If I helped you somehow, please, rate it as useful.-

[Dan]

Thanks for this. I'll give it a look when I'm back at work. Just realised I might be able to script something up using PSCP, assuming it can delete stuff from the device anyway. Never tested deleting on a switch, only pushing and pulling files.

Is it safe to delete the recovery image in this case? They will have the 7.3 as their backup still once they've been swapped but not sure what the purpose of the recovery is?

[Flavio Miranda]

If you are able to open a TAC I recommend. The recovery image is useful when something broke on AP like a corrupted image.

-If I helped you somehow, please, rate it as useful.-

[Dan]

We don't have TAC on our WLCs unfortunately.

If the primary image gets corrupt, wouldn't it boot to the backup image anyway and try to download the primary again?

[Scott Fella]

Dan,

The rcv image is for that purpose. If you delete that image and the primacy gets corrupted, the AP will not join the controller. 

[David Ritter]

Be brave, reboot after hours..  The AP's on a fast node with pull their fresh images quickly enough.  The slow ones with make your heart rate go up.

I usually have 5 or 6 scattered about that bark at a pre-download.  Oh a pre-download serves no purpose if one forgets  to command "swap" primary/secondary before reboot..  anyway

Next thing the wlc says it it can't reboot because some APs are still downloading..

>reset system forced

will get around that! 

[Dan]

Thanks all for your responses. I managed to convince them to let me reboot it but I'll be moving certain ones over to our backup (non-HA) controller to cover 24 hour sites during the down time. 

I'm more worried about APs failing their updates, getting stuck and disappearing from the controller - could this happen for any reason? We had 3 fail like this when I tried the predownload. When the original install was done before I worked here there wasn't anyway documentation created of the physical location of each AP and we have 60 sites so they won't be easy to find.

[David Ritter]

Ouch.  you have IP and MAC  data for each AP.  Hopefully the IP will point you to each building/switch.

preserve that up front.

I have Prime..  I just have to ask Prime who the AP's neighbor is and send the team to the field.

Don't clear any files..  let nature take its course.  The AP has no smarts..  While we might pre-download the new image, if the AP boots to the 'wrong' image, it will not know that the 'right' image is in the other slot.  It will just commence a fresh image download.

You might find one or two AP's that just seem to fail on a power-cycle.  Remember, the WLC will not know an AP is missing.

You have one WLC?  So the Monitor/Statistics/AP Join log should be current.. (unless its been cleared)  full of "Joined" entries..  should cover your entire family.  preserve that and compare it to the new list after wlc restart..  Without Prime that be the quickest way to know who is lost.

Good Luck

[Dan]

Can you do a 'reset system forced' using a schedule? like 'reset system forced at xx'?

Can't see that command using ? and dont want to put it in just in case it immediately reboots.