5520 WLC Redundancy Failure Event

[Moudar]

Hi,

I have two 5520 in SSO mode and i see a message on Prime:

"Redundancy Failure Event trap triggered by
Controller 'x.x.x.x' for the reason
'Error: Config Sync failed on Standby for
the usmdb:HA_send_usmDbSpamUSBModuleSet,
ReasonCode:5'"

When i check the 5520 CLI i see this:

(Cisco Controller) >show redundancy config-sync statistics

   Config Sync Counters........
   --------------------------------------------------------------
   Usmdb functions sent for sync...............................: 2394325
   Failed sync for usmdb sync..................................: 14

  UsmDbs failed:
  --------------------------------------------------------------
 1. UsmDb:HA_send_usmDbSpamUSBModuleSet, ErrType:Apply Config failed on Standby, Reason:5, TStamp:2023-Apr-06 09:49:39
 2. UsmDb:HA_send_usmDbSpamUSBModuleSet, ErrType:Apply Config failed on Standby, Reason:5, TStamp:2023-Apr-06 09:49:07
 3. UsmDb:HA_send_usmDbSpamUSBModuleOverrideSet, ErrType:Apply Config failed on Standby, Reason:5, TStamp:2023-Apr-06 09:49:07
 4. UsmDb:HA_send_usmDbSpamUSBModuleSet, ErrType:Apply Config failed on Standby, Reason:8, TStamp:2023-Mar-06 16:01:44
 5. UsmDb:HA_send_usmDbSpamUSBModuleOverrideSet, ErrType:Apply Config failed on Standby, Reason:2, TStamp:2023-Mar-06 16:01:44
 6. UsmDb:HA_send_usmDbSpamUSBModuleOverrideSet, ErrType:Apply Config failed on Standby, Reason:5, TStamp:2023-Jan-23 09:13:12
 7. UsmDb:HA_send_usmDbSpamUSBModuleSet, ErrType:Apply Config failed on Standby, Reason:5, TStamp:2023-Jan-23 08:29:07
 8. UsmDb:HA_send_usmDbSpamUSBModuleOverrideSet, ErrType:Apply Config failed on Standby, Reason:5, TStamp:2023-Jan-23 08:29:07
 9. UsmDb:HA_send_usmDbSpamUSBModuleSet, ErrType:Apply Config failed on Standby, Reason:8, TStamp:2023-Jan-09 09:26:41
 10. UsmDb:HA_send_usmDbSpamUSBModuleOverrideSet, ErrType:Apply Config failed on Standby, Reason:2, TStamp:2023-Jan-09 09:26:41
  --------------------------------------------------------------

We are running software version 8.10.181.3

Is there any bug in this version that causes this problem?

Does this problem impact the failover process if it would happen?

[marce1000]

                       - Could be a bug  , check redundancy status with  : show redundancy summary
          You may have a go with https://software.cisco.com/download/home/286284738/type/280926587/release/8.10.183.0  , and check if the problem remains in place or not ,

 M.

[Leo Laohoo]

Any reason why the controller is loaded with a "special release" firmware?

[Moudar]

We are using this version because it was the only working version at the time it was released. We had no problem since then.

But now i got this message!

[marce1000]

                                                  >...But now i got this message!
   - Inherent bugs may popup under the 'just circumstances'   (...) or for instance there could be a sort for resource exhaustion , then rebooting (both) could help ; both factors are an even more argument to go for latest or a  stable release : https://software.cisco.com/download/home/286284738/type/280926587/release/8.10.183.0

 M.

[Moudar]

So, if this is a bug, would it impact the failover process in case a failover happens?

If the answer is yes, what would the impact be?

[marce1000]

       - That is very difficult to determine,  as stated verify current health of the HA SSO setup with : show redundancy summary ,  avoiding such questions can be  achieved by upgrading , as I mentioned earlier.

  M.

[Moudar]

This is "show redundancy summary " from the management interface

(Cisco Controller) >show redundancy summary
            Redundancy Mode = SSO ENABLED
                Local State = ACTIVE
                 Peer State = STANDBY HOT
                       Unit = Secondary (Inherited AP License Count = 1500)
                    Unit ID = CC:46:D6:FC:A6:E9
           Redundancy State = SSO
               Mobility MAC = CC:46:D6:59:7B:4B
               Redundancy Port  = UP
            BulkSync Status = Complete
               Link Encryption = ENABLED
Average Redundancy Peer Reachability Latency = 245 Micro Seconds
Average Management Gateway Reachability Latency = 383 Micro Seconds

 Here is the active node:

show redundancy summary
            Redundancy Mode = SSO ENABLED
                Local State = ACTIVE
                 Peer State = STANDBY HOT
                       Unit = Secondary (Inherited AP License Count = 1500)
                    Unit ID = CC:46:D6:FC:A6:E9
           Redundancy State = SSO
               Mobility MAC = CC:46:D6:59:7B:4B
               Redundancy Port  = UP
            BulkSync Status = Complete
               Link Encryption = ENABLED
Average Redundancy Peer Reachability Latency = 233 Micro Seconds
Average Management Gateway Reachability Latency = 499 Micro Seconds

Here is the hot standby node:

(Cisco Controller-Standby) >show redundancy summary
            Redundancy Mode = SSO ENABLED
                Local State = STANDBY HOT
                 Peer State = ACTIVE
                       Unit = Primary
                    Unit ID = CC:46:D6:59:7B:4B
           Redundancy State = SSO
               Mobility MAC = CC:46:D6:59:7B:4B
               Redundancy Port  = UP
               Link Encryption = ENABLED
Average Redundancy Peer Reachability Latency = 249 Micro Seconds
Average Management Gateway Reachability Latency = 618 Micro Seconds

[marce1000]

           - Looks ok for the moment and or it seems that failovers can work correctly , 

 M.

[Rich R]

Agreed looks like failover will work but risk of some config loss due to sync errors.
That special version only had limited supported as an interim release so you should be upgrading to the TAC recommended release 8.10.185.0 (as per below) regardless to make sure you resolve known bugs and have a release which TAC will support.

If you still see the error after upgrading you can open a TAC case so they can analyse although a fix is unlikely at this point in the AireOS lifecycle.  At best they could offer a workaround.

[Moudar]

I did restart both (one at a time) and it went with no problems. But then they contiued to send that message:

 

So a service window to upgrade them should be done shortly!