cisco wlc 8540 HA SSO maintenance mode

girish_gavandi · ‎10-12-2016

Hi,

We have two Cisco WLC 8540 in HA pair. Connected to network with 4x10G links. running with image version 8.3.102.0. High availability SSO is setup and is working good.

We were testing a WLC HA Pair failure scenario wherein we removed the RP port connection from Active WLC. This made the standby WLC to go into maintenance mode (output on standby WLC as shown below). Standby could reach active wlc via RMI and also reach mgmt. default gateway.

(Cisco Controller) >show redundancy summary 
            Redundancy Mode = SSO ENABLED 
                Local State = MAINTENANCE 
                 Peer State = UNKNOWN - Communication Down 
                       Unit = Primary
                    Unit ID = 
           Redundancy State = Non Redundant
               Mobility MAC = 
Maintenance Mode = Enabled
Maintenance cause= Peer redundancy port is not reachable

At this time both the WLC have their fibre uplinks connected to network with only RP port down.

Next, we disconnected the fibre uplinks from active WLC. Standby WLC cannot reach active WLC over RMI and active WLC has lost reachability to standby over RMI and RP and also lost connectivity to mgmt default gateway (output on active WLC as shown below).

(Cisco Controller) >show redundancy summary 
            Redundancy Mode = SSO ENABLED 
                Local State = ACTIVE 
                 Peer State = UNKNOWN - Communication Down 
                       Unit = Secondary (Inherited AP License Count = 1500)
                    Unit ID = 
           Redundancy State = Non Redundant
               Mobility MAC = 
            BulkSync Status = Pending

Traffic loss was seen as the APs got stranded. Standby WLC was still connected with fibre uplinks which still had reachability to the gateway.

Question here is that in case of standby WLC which is in maintenance mode, after loosing reachability to active WLC via RMI, should reboot itself to come out of maintenance mode? If yes, after how much time?
If a WLC in HA pair is in maintenance mode, can come out of it by itself under which situation?
What exactly is a maintenance mode?

Requesting for expert comments.

Many Thanks,

Girish

Devaiah N K · ‎11-17-2016

Hi Girish,

Just came across this question, I have tried to answer this question in green with the below explanation:

Question here is that in case of standby WLC which is in maintenance mode, after loosing reachability to active WLC via RMI, should reboot itself to come out of maintenance mode? If yes, after how much time?

A: Standby WLC (Secondary Unit) will not come out of maintenance mode by itself. it will be in maintenance mode until you reboot the unit. When the device reboots, it will trigger a discovery process in the network to find its peer, if successful and finds its active pair, it negotiates the role and takes up the role of standby, if not it will go to maintenance mode.

If a WLC in HA pair is in maintenance mode, can come out of it by itself under which situation?

A:Never, it is a manual task to reboot the device. Please note that WLC can be accessible via console /service ports only.

What exactly is a maintenance mode?

A: There are many scenarios where the WLC may go into maintenance mode.

WLC goes to a maintenance mode to avoid split brain scenario where both the controllers assume it is an active unit (we can imagine the state of access points if this happens). To avoid this state, when the WLC is unable to reach its RMI/Peer goes to maintenance mode.

Below are some of the scenarios when a WLC can go into maintenance mode.

if the gateway is unreachable via Redundant Management Interface
Usually when out of box, when standby WLC with HA SKU which had never discovered peer during role negotiations.
When there is a scheduled reboot or when SSO is enabled for the first time, if there is a Software mismatch, the WLC which boots up first goes into active mode and the other WLC in Maintenance Mode.
When the Redundancy port is down.

There are other scenarios which can cause the maintenance mode, I have taken an excerpt from the HA guide (In the attachment).

Additional comments:

" Sometime back, for testing purpose, I removed the redundancy port of secondary unit in standby mode, this caused the primary unit in an active state to reboot, the standby WLC become the active controller, I connected the RP and that primary unit restarts and negotiated with its peer and became the standby controller.

If i wouldn't have plugged the RP cable of secondary unit, I assume primary unit would have gone to a maintenance mode.

Cheers,

Devaiah NK.

pwwiddicombe · ‎11-19-2016

Hmmm... What happens if you have a power outage (you have both down), and for some reason the primary or the switch primary is attached to doesn't come back up? WILL the secondary unit, on startup, then take over; or just go into maintenance mode ? Depending on the connectivity between the secondary and the (downed primary), the RP port may and may not be in an "UP" state.

girish_gavandi · ‎12-27-2016

Hi pwwiddicombe,

In this case the secondary unit in standby mode boots up, checks for any active WLC, finds that the active WLC is down and assumes the role of active WLC.

When the active WLC (which was down) comes up, it will check for any active WLC in the network and finds that there is one already UP. And then assumes the role of standby WLC.

Hope this helps.

-Girish

girish_gavandi · ‎12-27-2016

Hi Devaiah N K,

When you remove the RP port from secondary unit in standby mode, it immediately goes into maintenance mode. This is to prevent the active-active scenario.

When you re-plug the RP port, the standby WLC which is in maintenance mode reboots itself and comes up as standby WLC.

This is what I observed during our tests.

-Girish