Cisco WLC Upgrade SSO-Upload the new version when peer was down, so now image is not transferred to standby

[mbay]

Hi ,

we have 2 SSO pair 5508 controllers. we were planning to upgrade from 8.3 version to 8.5.161.0.

we had a weird situation last week,

while standby peer was down, we uploaded the new image to the primary one, and image uploaded successfully to the primary one, but we haven't restarted it yet, and so peer couldn't get the image that time.

So now peer is up again, now we want to complete both of them upgrade at the same time without any version mismatch.

but primary one has the new image, but standby doesn't.  

normally we should upload the new image while both was up, but we couldn't do that.

Does anybody have any idea how we should continue this upgrade? 

ıf we re-upload the same new image again, will it work? my suspicion is because primary has the new image, it may not accept it?

Does anybody have any experience about these kind of situation?

what can you suggest?

[Leo Laohoo]

Upload the firmware again and see if the primary will push to the standby. 

[Scott Fella]

When this happens, you can use the service port to upload the image to the standby.  Make sure the ports on the switch is set to shut or the port channel. This way the standby doesn't go into maintenance mode.  Once you upgrade the standby, it will reboot and then you can issue a no shut on the ports.  The standby will negotiate SSO with the primary and join as the standby,

[Grendizer]

you can use the standby RMI IP address if you don't have console access to the standby  then upload the new image using TFTP or FTP (this will work as long as you didn't reboot the active WLC) if you already rebooted the active WLC then the only way this could work if you don't have APs joining the active WLC or else this has to be going thru reforming the HA SSO config from scratch.

Important note: if you did reboot the Active WLC you may see problems, some APs will report old Standby image and you will see weird behavior so if you didn't reboot the active WLC fix the standby using the above methods or shut the connected switch ports to the standby and fix it later using console connection.

[mbay]

Hi Everyone,

Thanks for the replies.

Here is my experience. this is a rare issue, it can help someone else in the future.

We tried to re-upload the same image (8.5.161.0), but it gave http error, and we saw backup image error on config boot. Probably because there is the same image on primary ,it acted like that, and it didn't work

we decided to do 2 step upgrade,

when we tried to upload another image(8.5.151.0) between 8.3.150.0 and 8.5.161.0. 

first we upload the image and all worked fine, and image transferred the standby, and we restart the system this image, all worked fine,

and then we upload the image(8.5.161.0) this time, and it again worked.

now both SSO pair upgraded.

[Scott Fella]

Whenever you upgrade an SSO cluster and you are not familiar with how to recover or bring your environment back to stable, you should always monitor the upgrade from the console and also perform the upgrade from the console.  You would be able to see if the image is pushed to the standby controller or not.  You will also see if the standby is rebooting or not.  If one doesn't reboot while the other has and is close to coming back online, your SSO will be in maintenance mode.  This will require you to take the unit offline or take the primary offline to upgrade your standby or vice versa.  Upgrading isn't 100% so its always better to monitor your upgrade as close as possible.