Wireless Controller redundancy in Unknown due to incompatible software version - Bulk Sync Status Pending

[John Mendonca]

Hi ,

While doing a usual reload on standby , we missed to check the primary , backup boot image that was set to 8.2.140/8.0 which triggered the below. Our goal was to reload standby first then active with the goal of max. up-time.

In a scenario where after reloading the standby it booted with the latest version 8.2.140 while the active where the command to the peer standby was 8.0 so hence it resulted in a incompatible version ..Rebooted active controller and then that as well is now on 8.2.140 , since both controllers are now on the same 8.2.140 version why is the redundancy not forming as per the output below.. Is it the bootloader/ field recovery image version mismatch?

Thank you for your support .

O/P below for your reference

-----------------------------------------------------------------------------------

Active Wireless Controller

(Cisco Controller) >show redundancy summary
            Redundancy Mode = SSO ENABLED
                Local State = ACTIVE
                 Peer State = UNKNOWN - Communication Down
                       Unit = Primary
                    Unit ID = X.X.X.X
           Redundancy State = Non Redundant
               Mobility MAC = X.X.X.X
            BulkSync Status = Pending

(Cisco Controller) >show sysinfo

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 8.2.141.0
Bootloader Version............................... 1.0.1
Field Recovery Image Version..................... 6.0.182.0
Firmware Version................................. FPGA 1.3, Env 1.6, USB console 1.27
Build Type....................................... DATA + WPS
-----------------------------------------------------------------------------------------------------------

Standby Wireless Controller -

(Cisco Controller) >show redundancy summary
            Redundancy Mode = SSO ENABLED
                Local State = MAINTENANCE
                 Peer State = UNKNOWN - Communication Down
                       Unit = Secondary
                    Unit ID = X.X.X.X
           Redundancy State = Non Redundant
               Mobility MAC = X.X.X.X
Maintenance Mode = Enabled
Maintenance cause= Incompatible Software version

(Cisco Controller) >show sysinfo

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 8.2.141.0
Bootloader Version............................... 1.0.20
Field Recovery Image Version..................... 7.6.95.16
Firmware Version................................. FPGA 1.7, Env 1.8, USB console 2.2
Build Type....................................... DATA + WPS

Controller 5508 Models

[Leo Laohoo · Accepted solution]

The only thing I could see wrong is the FUS code (aka Bootloader) hasn't been upgraded.

[John Mendonca]

Thanks Leo, I have two questions :-

Since its a prod environment and observe that the primary WLC serving the Access Point clients is on a lower bootloader version than the secondary how can we do it without a downtime if the redundancy SSO is not formed in the cluster with the secondary standby controller. Can i downgrade the bootloader version on the standby Controller v 1.0.20 to match the active WLC 1.0.1 to prevent any outage ?

How does this bootloader mismatch happen ? Is it a must requirement check  in newer releases of wireless software that after 8.0 release that this field has to match as well . Did not see it on version 7.X and until 8.0.series Appreciate your support.

Rated 5 Stars. As always , this support community has by far been the best.

[Scott Fella]

With SSO, you can't reboot just one controller. Any mix match on the primary firmware will cause the controller that was rebooted to go into maintenance mode. Now what you have to do is bounce the standby and see if it syncs with the primary. Once in maintenance mode, you need to reboot.

-Scott 

*** Please rate helpful posts *** 

[John Mendonca]

Hi Scott,

Thanks for the response, By Bounce you mean reboot. Wondering how the bootcode changed as we did not change that . Could it be a recent check criteria by Cisco for the Boot loader version match as we did not see that in 7.4 until the 8.0 . Does it get updated with new code automatically with the new WLC 8.2 code ,  if so it should have done the same on both Active/Standby WLC. Since we are in production and all are AP are joined to the Primary WLC if i reboot the standby WLC will that be causing a downtime in anyway during the reboot ( i know it wont however from a  8.2 code outlook).

Appreciate

[Scott Fella]

Boot loader has nothing to do with this. You are not suppose to reboot one controller, but both.  This is what caused your one co trolled that you rebooted first to go into maintenance mode. The FUS is just something you need to update manually as the firmware itself doesn't upgrade this. 

With SSO, change in code will always have down time.

-Scott 

*** Please rate helpful posts *** 

[John Mendonca]

Thanks Scott. will reload the standby controller hopefully it syncs. appreciate all the inputs from Leo and yourself.

[Leo Laohoo]

If you're going to reboot both controllers you might as well upgrade the FUS of the other one to the latest standard.

[John Mendonca]

Thanks Leo and Scott. appreciate all the inputs .  Since we couldnt afford downtime I rebooted standby controller that was in maintenance mode and it was able to boot in sso redundant mode. We will definitely upgrade are fus code to latest version while upgrading our controllers to 8.2.151 in next two weeks and reboot both wireless controllers in a maintenance window. rated 5 Stars . Thanks

[Scott Fella]

Just make sure that you know, when you upgrade the FUS, it can take up to 45 minutes, so I would make your downtime aroun 1 1/2 hours to 2.  It's also best to have console connection on both if you have never upgraded the FUS so that you can control what gets upgraded. For example, if one part has the correct firmware, you can manually hit "N" to skip that from being upgraded again. This will shorten the time to upgrade. Since this is also in production, disable all the wlans during the upgrade since the AP's will be bouncing.  Also if your running a MS dhcp server, look at the AP dhcp scope and make sure that there are no "BAD ADDRESS" appearing or else you need to manually clear them. 

-Scott 

*** Please rate helpful posts ***