9800 // Upgrade to 17.12.4 // PreImage Download // Radios go Off

[MICHAEL SCHROEDER]

Hi All,

i worked to upgrade a 9800-40 in HA SSO from 17.6 to 17.12.4. After "install add" the Image i started the AP Preimage Download. During the Process, the APs stopped working, the Radios turned off and could not be restarted. After "install activate" all went back to normal. This was pain, but due to the Fact that i could reboot immediately there were not much trouble. But now i have to do it on a 4500 AP Installation, were going Offline AP by AP is not acceptable.

Does anyone face this Issue and have a hint to avoid this? I did predownloading very often in the Past, but never facing that before ...

TiA, Regards, Mike

[marce1000]

  - Probably shouldn't happen , when done a  next time , configure a syslog server in the AP Join Profile first , and examine logs
 (afterwards) => Look for insights , 

 M.

[Leo Laohoo]

Sounds like a 17.6.X bug.

[eglinsky2012]

Can you elaborate on: "the APs stopped working, the Radios turned off and could not be restarted."

Which GUI screen or CLI command revealed the radios were off? Were they in admin up / operational down state? Did they actually stop serving clients, or did it only appear that they were off?

[MICHAEL SCHROEDER]

Hi, good Question ... from my perspective the APs were admin up, but operational down. The Radios are shown as down in the WebUI, and the IT Admin in Charge complained no connectivity. Unfortunately the Customer lost some nerve, so i had no time for further investigations and could not really figure out what happen. Any Idea? I try to check the Syslogs, as Marce1000 advises. 

[Haydn Andrews]

Are you performing the pre download via the web interface vs CLI?

have had some wierd things happen via web interface with pre-download before.

[MICHAEL SCHROEDER]

Due to the Illusion of more control I did with CLI. I focused on output of sh ap image command. 

[Rich R]

I always use CLI - I agree that I feel more in control.
We did not see that upgrading from 17.9.4 to 17.12.4 but we preloaded the software to APs via TFTP to avoid the risk of corrupted CAPWAP image downloads.
I wonder if your APs might have been affected by corrupted images and whether that somehow affected them?

I still agree with the others that it should not happen but like Leo says it sounds like a bug in 17.6.

[MICHAEL SCHROEDER]

I agree, it sounds like a Bug. This makes it not very trustful to Upgrade the next Unit with more than 4500 APs in Operations.... Did you use WLAN Poller for Bulk Upload the Image? Can you short describe the process? Regards, Mike

[Leo Laohoo]

Just be warry of CSCwm07499.  

NOTE:  It may say "when upgrading WLC from 17.12.3 to 17.12.4", don't take every word at "face value".  Trust, but verify.  

[MICHAEL SCHROEDER]

Thx, Leo. Will check the Logs regarding this. Regards, Mike

[Rich R]

It was a very manual process!
We had multiple remote sites (therefore concern about image corruption) so we loaded the image to local site router and configured that as TFTP server.  IOS seems to allow a max of 10 clients to download at once so they had to be done in batches of 10 APs at a time.
We were intending to end up on 17.12.4 APSP1 so I extracted the AP images from the APSP1 file (C9800-universalk9_wlc.17.12.04.CSCwm37063.SPA.apsp.bin\C9800-mono-patch.17.12.04.CSCwm37063.SPA.apsp.bin\mnt\images\ap_images\) and loaded the required AP files onto the routers.  Then use the following to start the AP downloads from the WLC:
ap name <ap-name> remote command "archive download-sw /no-reload tftp://<server IP>/ap1g8-k9w8-tar.17.12.4.201.tar"
Note I renamed the original ap1g8 file to ap1g8-k9w8-tar.17.12.4.201.tar so it was absolutely clear for us what each file was.  At this point you would be using APSP3 so it will be version 17.12.4.203 from C9800-universalk9_wlc.17.12.04.CSCwm71871.SPA.apsp.bin\C9800-mono-patch.17.12.04.CSCwm71871.SPA.apsp.bin\mnt\images\ap_images\
I used 7-zip to open and extract the files.

You can use an Excel spreadsheet with your AP names to build the list of commands but doing 4500 could take a while if you're doing them in batches of 10 so you might want to use a proper TFTP server which can handle more than 10 at a time <smile>

Although the APs should automatically use the newly installed image we found that about half of ours re-downloaded anyway <sigh>!  That might have been related to a problem we had during upgrade which meant we didn't tell the APs to switch image before they tried to join the new code version.

If you have HA-SSO WLCs then beware of the corrupt binary config issue which requires you to delete the binary config - see https://community.cisco.com/t5/wireless/9800-80-active-and-standby-configuration-out-of-sync-17-12-4/m-p/5192911#M275424  Might be safer to just delete the binary config anyway to prevent the possibility of that happening (which I did for the second upgrade after encountering it on the first upgrade).  The WLC can boot fine without the binary config - it just logs a warning and can take slightly longer to boot.