Wireless controller 5520 stuck in maintenance mode.

[muzecisco]

I am having this weird behavior which i hope someone can provide good pointers.

My primary controller is stuck in maintenance mode. Switchover reason on the current active(Secondary unit) is Default gateway is not reachable. This happened on Thu Feb 20 15:40:37 2020. See reboot history logs below.

I have pinged gw and peer controller RP and RM ports and they are all reachable.

I have also reset system  but it did not come out of maintenance state.  Other than user initiated reset, its now giving Certificate transfer from Active to Standby failed ,(Version: 8.10.105.0) as reason for further resets before it gets stuck in Maintenance state.

Scott advised that i reset from console with no cable connected so i see if it becomes primary and active... i am still to do this.

Find below output from various show commands that may assist with further understanding.

(Cisco Controller) >show sysinfo

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 8.10.105.0
RTOS Version..................................... 8.10.105.0
Bootloader Version............................... 8.1.102.0
Emergency Image Version.......................... 8.1.102.0

(Cisco Controller) >show lag sum

LAG Enabled

(Cisco Controller) >show port summary

STP Admin Physical Physical Link Link
Pr Type Stat Mode Mode Status Status Trap POE
-- ------- ---- ------- ---------- ---------- ------ ------- ---------
1 Normal Forw Enable Auto 10000 Full Up Enable N/A
2 Normal Forw Enable Auto 10000 Full Up Enable N/A
RP Normal Forw Enable Auto Auto Up Enable N/A
SP Normal Disa Enable Auto Auto Down Enable N/A

Reset reason : active default gateway is not reachable ,(Version: 8.10.105.0)
Reset time : Thu Feb 20 15:40:38 2020
Trace back : 0x8e7b60 0x1c1a4ba 0x1cb1a34 0x1ca455a 0x1ca6a36 0x1c9c73e 0x1c31827 0x7dff 0xe898d
========================================
Reset reason : Certificate transfer from Active to Standby failed ,(Version: 8.10.105.0)
Reset time : Thu Feb 20 15:49:01 2020
Trace back : 0x8e7b60 0x1c1a4ba 0x1cb1a34 0x1ca7284 0x1ca6a36 0x1c9c9a3 0x1c31827 0x7dff 0xe898d
========================================
Reset reason : Certificate transfer from Active to Standby failed ,(Version: 8.10.105.0)
Reset time : Thu Feb 20 15:55:05 2020
Trace back : 0x8e7b60 0x1c1a4ba 0x1cb1a34 0x1ca7284 0x1ca6a36 0x1c9c9a3 0x1c31827 0x7dff 0xe898d
========================================
Reset reason : Certificate transfer from Active to Standby failed ,(Version: 8.10.105.0)
Reset time : Thu Feb 20 16:03:26 2020
Trace back : 0x8e7b60 0x1c1a4ba 0x1cb1a34 0x1ca7284 0x1ca6a36 0x1c9c9a3 0x1c31827 0x7dff 0xe898d
========================================
Reset reason : Certificate transfer from Active to Standby failed ,(Version: 8.10.105.0)
Reset time : Thu Feb 20 16:09:28 2020
Trace back : 0x8e7b60 0x1c1a4ba 0x1cb1a34 0x1ca7284 0x1ca6a36 0x1c9c9a3 0x1c31827 0x7dff 0xe898d
========================================
Reset reason : User Initiated reset ,(Version: 8.10.105.0)
Reset time : Fri Feb 21 23:03:13 2020
Trace back : 0x8e7b60 0x1c1a4ba 0x1c1b616 0x1fb4c81 0x23b0f09 0x1a50741 0x1a29e19 0x1a6b598 0x1a5af01 0x1a4c4a8 0x1a46d1a 0x1a70aaf 0x1c31827 0x7dff 0xe898d

--More-- or (q)uit
========================================
Reset reason : Certificate transfer from Active to Standby failed ,(Version: 8.10.105.0)
Reset time : Fri Feb 21 23:11:33 2020
Trace back : 0x8e7b60 0x1c1a4ba 0x1cb1a34 0x1ca7284 0x1ca6a36 0x1c9c9a3 0x1c31827 0x7dff 0xe898d
========================================
Reset reason : Certificate transfer from Active to Standby failed ,(Version: 8.10.105.0)
Reset time : Fri Feb 21 23:17:35 2020
Trace back : 0x8e7b60 0x1c1a4ba 0x1cb1a34 0x1ca7284 0x1ca6a36 0x1c9c9a3 0x1c31827 0x7dff 0xe898d
========================================
Reset reason : Certificate transfer from Active to Standby failed ,(Version: 8.10.105.0)
Reset time : Fri Feb 21 23:25:57 2020
Trace back : 0x8e7b60 0x1c1a4ba 0x1cb1a34 0x1ca7284 0x1ca6a36 0x1c9c9a3 0x1c31827 0x7dff 0xe898d
========================================
Reset reason : Certificate transfer from Active to Standby failed ,(Version: 8.10.105.0)
Reset time : Fri Feb 21 23:32:00 2020
Trace back : 0x8e7b60 0x1c1a4ba 0x1cb1a34 0x1ca7284 0x1ca6a36 0x1c9c9a3 0x1c31827 0x7dff 0xe898d
========================================
Reset reason : User Initiated reset ,(Version: 8.10.105.0)
Reset time : Sun Feb 23 00:56:04 2020
Trace back : 0x8e7b60 0x1c1a4ba 0x1c1b616 0x1fb4c81 0x23b0f09 0x1a50741 0x1a29e19 0x1a6b598 0x1a5af01 0x1a4c4a8 0x1a46d1a 0x1a70aaf 0x1c31827 0x7dff 0xe898d
========================================
Reset reason : Certificate transfer from Active to Standby failed ,(Version: 8.10.105.0)
Reset time : Sun Feb 23 01:04:47 2020

--More-- or (q)uit
Trace back : 0x8e7b60 0x1c1a4ba 0x1cb1a34 0x1ca7284 0x1ca6a36 0x1c9c9a3 0x1c31827 0x7dff 0xe898d
========================================
Reset reason : Certificate transfer from Active to Standby failed ,(Version: 8.10.105.0)
Reset time : Sun Feb 23 01:10:52 2020
Trace back : 0x8e7b60 0x1c1a4ba 0x1cb1a34 0x1ca7284 0x1ca6a36 0x1c9c9a3 0x1c31827 0x7dff 0xe898d
========================================
Reset reason : Certificate transfer from Active to Standby failed ,(Version: 8.10.105.0)
Reset time : Sun Feb 23 01:19:13 2020
Trace back : 0x8e7b60 0x1c1a4ba 0x1cb1a34 0x1ca7284 0x1ca6a36 0x1c9c9a3 0x1c31827 0x7dff 0xe898d
========================================
Reset reason : Certificate transfer from Active to Standby failed ,(Version: 8.10.105.0)
Reset time : Sun Feb 23 01:25:16 2020
Trace back : 0x8e7b60 0x1c1a4ba 0x1cb1a34 0x1ca7284 0x1ca6a36 0x1c9c9a3 0x1c31827 0x7dff 0xe898d

[Scott Fella · Accepted solution]

I would open a TAC case and see if this is related to this bug:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvq95397

There are other things you can try to do also but see if the primary comes up on its own. You can always shutdown the standby or force restart so that the primary comes back up and then see if the back syncs and you SSO is enabled. I would also then fail it over again so that your standby becomes primary and see if the primary works fine or has this issue. You can also remove the primary and factory default it and then while not connected to the network, follow the procedure to replace a primary controller in SSO and see if that fixes your issue. Whatever you do, test failover and make sure it works.

[muzecisco]

Thanks Scott

For my case, switching power OFF/ON did the magic. I am raising a TAC so we know if there is a known solution for this issue now.

You also suggested removing all cables and rebooting the WLC. I realized that  this option will correctly take the controller into Maintenance because the RP and RP ports will be down.

I am thinking one option that may work is to Disable and enable SSO. This option will however reboot the Active secondary unit.  I could not afford to have the secondary unit down so i didnt  try this.

Thumps up for the community!

[Scott Fella]

Removing the cables will not make it go into maintenance mode. I have done this before and this is also what the call “split brain” when both controllers think they should be the active. Disabling SSO will reboot your unit so I would not do that.

[muzecisco]

Well i also did remove the cables a reset from  CLI. It remained stuckc in maintenance. I guess its a bug.

I will feedback once cisco provides a conclusive investigation.

[Scott Fella]

Sounds good. Keep us posted!