Wireless controller 5520 stuck in maintenance mode.

[muzecisco]

Hi Team

I have my primary WLC 5520 (software 8.10.105.0) stuck in MAINTENANCE.  I have remotely tried reset system with out any success.

The software version is the same as standby HA. I can ping the RM  and RP ip address from both Primary to standby and vice versa.

On the standby WLC(currently active), reason for switchover is  Default gateway is not reachable. I tested pings and they are all OK.

May you kindly provide leads on what could be happening and how to solve this issue.

(Cisco Controller) >show redundancy sum
Redundancy Mode = SSO ENABLED
Local State = MAINTENANCE
Peer State = UNKNOWN - Communication Down
Unit = Primary
Unit ID = 08:96:
Redundancy State = Non Redundant
Mobility MAC = 08:96:
Redundancy Port = UP
Link Encryption = ENABLED
Maintenance Mode = Enabled
Maintenance cause= Too many reboots

I have also just checked the reboot history and i see the below message.

========================================
Reset reason : active default gateway is not reachable ,(Version: 8.10.105.0)
Reset time : Thu Feb 20 15:40:38 2020
Trace back : 0x8e7b60 0x1c1a4ba 0x1cb1a34 0x1ca455a 0x1ca6a36 0x1c9c73e 0x1c31827 0x7dff 0xe898d
========================================

--More-- or (q)uit
Reset reason : Certificate transfer from Active to Standby failed ,(Version: 8.10.105.0)
Reset time : Thu Feb 20 15:49:01 2020
Trace back : 0x8e7b60 0x1c1a4ba 0x1cb1a34 0x1ca7284 0x1ca6a36 0x1c9c9a3 0x1c31827 0x7dff 0xe898d
========================================
Reset reason : Certificate transfer from Active to Standby failed ,(Version: 8.10.105.0)
Reset time : Thu Feb 20 15:55:05 2020
Trace back : 0x8e7b60 0x1c1a4ba 0x1cb1a34 0x1ca7284 0x1ca6a36 0x1c9c9a3 0x1c31827 0x7dff 0xe898d
========================================
Reset reason : Certificate transfer from Active to Standby failed ,(Version: 8.10.105.0)
Reset time : Thu Feb 20 16:03:26 2020
Trace back : 0x8e7b60 0x1c1a4ba 0x1cb1a34 0x1ca7284 0x1ca6a36 0x1c9c9a3 0x1c31827 0x7dff 0xe898d
========================================
Reset reason : Certificate transfer from Active to Standby failed ,(Version: 8.10.105.0)
Reset time : Thu Feb 20 16:09:28 2020
Trace back : 0x8e7b60 0x1c1a4ba 0x1cb1a34 0x1ca7284 0x1ca6a36 0x1c9c9a3 0x1c31827 0x7dff 0xe898d
========================================
Reset reason : User Initiated reset ,(Version: 8.10.105.0)
Reset time : Fri Feb 21 23:03:13 2020
Trace back : 0x8e7b60 0x1c1a4ba 0x1c1b616 0x1fb4c81 0x23b0f09 0x1a50741 0x1a29e19 0x1a6b598 0x1a5af01 0x1a4c4a8 0x1a46d1a 0x1a70aaf 0x1c31827 0x7dff 0xe898d
========================================
Reset reason : Certificate transfer from Active to Standby failed ,(Version: 8.10.105.0)
Reset time : Fri Feb 21 23:11:33 2020
Trace back : 0x8e7b60 0x1c1a4ba 0x1cb1a34 0x1ca7284 0x1ca6a36 0x1c9c9a3 0x1c31827 0x7dff 0xe898d

Reset reason : Certificate transfer from Active to Standby failed ,(Version: 8.10.105.0)
Reset time : Fri Feb 21 23:17:35 2020
Trace back : 0x8e7b60 0x1c1a4ba 0x1cb1a34 0x1ca7284 0x1ca6a36 0x1c9c9a3 0x1c31827 0x7dff 0xe898d
========================================
Reset reason : Certificate transfer from Active to Standby failed ,(Version: 8.10.105.0)
Reset time : Fri Feb 21 23:25:57 2020
Trace back : 0x8e7b60 0x1c1a4ba 0x1cb1a34 0x1ca7284 0x1ca6a36 0x1c9c9a3 0x1c31827 0x7dff 0xe898d
========================================
Reset reason : Certificate transfer from Active to Standby failed ,(Version: 8.10.105.0)
Reset time : Fri Feb 21 23:32:00 2020
Trace back : 0x8e7b60 0x1c1a4ba 0x1cb1a34 0x1ca7284 0x1ca6a36 0x1c9c9a3 0x1c31827 0x7dff 0xe898d

[Scott Fella]

I’m going to assume that the code versions are the same? What you can do is disconnect the RP port and the connection to the network on the primary, since it’s down. Console in and reload the unit and see if it comes up as primary. Don’t connect the unit to the network, you just want to see if it detects the other unit down and comes up.

[Leo Laohoo]

---

@muzecisco wrote:

Peer State = UNKNOWN - Communication Down

---

RP is connected using copper cable.  Try replacing that first.

[muzecisco]

Thanks i will try this tomorrow.

From reboot history  see below some  informative logs i got.

========================================
Reset reason : active default gateway is not reachable ,(Version: 8.10.105.0)
Reset time : Thu Feb 20 15:40:38 2020
Trace back : 0x8e7b60 0x1c1a4ba 0x1cb1a34 0x1ca455a 0x1ca6a36 0x1c9c73e 0x1c31827 0x7dff 0xe898d
========================================

--More-- or (q)uit
Reset reason : Certificate transfer from Active to Standby failed ,(Version: 8.10.105.0)
Reset time : Thu Feb 20 15:49:01 2020
Trace back : 0x8e7b60 0x1c1a4ba 0x1cb1a34 0x1ca7284 0x1ca6a36 0x1c9c9a3 0x1c31827 0x7dff 0xe898d
========================================
Reset reason : Certificate transfer from Active to Standby failed ,(Version: 8.10.105.0)
Reset time : Thu Feb 20 15:55:05 2020
Trace back : 0x8e7b60 0x1c1a4ba 0x1cb1a34 0x1ca7284 0x1ca6a36 0x1c9c9a3 0x1c31827 0x7dff 0xe898d
========================================
Reset reason : Certificate transfer from Active to Standby failed ,(Version: 8.10.105.0)
Reset time : Thu Feb 20 16:03:26 2020
Trace back : 0x8e7b60 0x1c1a4ba 0x1cb1a34 0x1ca7284 0x1ca6a36 0x1c9c9a3 0x1c31827 0x7dff 0xe898d
========================================
Reset reason : Certificate transfer from Active to Standby failed ,(Version: 8.10.105.0)
Reset time : Thu Feb 20 16:09:28 2020
Trace back : 0x8e7b60 0x1c1a4ba 0x1cb1a34 0x1ca7284 0x1ca6a36 0x1c9c9a3 0x1c31827 0x7dff 0xe898d
========================================
Reset reason : User Initiated reset ,(Version: 8.10.105.0)
Reset time : Fri Feb 21 23:03:13 2020
Trace back : 0x8e7b60 0x1c1a4ba 0x1c1b616 0x1fb4c81 0x23b0f09 0x1a50741 0x1a29e19 0x1a6b598 0x1a5af01 0x1a4c4a8 0x1a46d1a 0x1a70aaf 0x1c31827 0x7dff 0xe898d
========================================
Reset reason : Certificate transfer from Active to Standby failed ,(Version: 8.10.105.0)
Reset time : Fri Feb 21 23:11:33 2020
Trace back : 0x8e7b60 0x1c1a4ba 0x1cb1a34 0x1ca7284 0x1ca6a36 0x1c9c9a3 0x1c31827 0x7dff 0xe898d

--More-- or (q)uit
========================================
Reset reason : Certificate transfer from Active to Standby failed ,(Version: 8.10.105.0)
Reset time : Fri Feb 21 23:17:35 2020
Trace back : 0x8e7b60 0x1c1a4ba 0x1cb1a34 0x1ca7284 0x1ca6a36 0x1c9c9a3 0x1c31827 0x7dff 0xe898d
========================================
Reset reason : Certificate transfer from Active to Standby failed ,(Version: 8.10.105.0)
Reset time : Fri Feb 21 23:25:57 2020
Trace back : 0x8e7b60 0x1c1a4ba 0x1cb1a34 0x1ca7284 0x1ca6a36 0x1c9c9a3 0x1c31827 0x7dff 0xe898d
========================================
Reset reason : Certificate transfer from Active to Standby failed ,(Version: 8.10.105.0)
Reset time : Fri Feb 21 23:32:00 2020
Trace back : 0x8e7b60 0x1c1a4ba 0x1cb1a34 0x1ca7284 0x1ca6a36 0x1c9c9a3 0x1c31827 0x7dff 0xe898d

[HSSNetworks]

@muzecisco 
So what was the outcome..? managed to resolve it?

We have same issue here with Software Version 8.5.182.7

[Rich R]

@muzecisco 8.10.105.0 is very old with LOTS of bugs.

You urgently need to plan an upgrade to at least 8.10.190.0 as per TAC recommended link below.
Note that 8.10.196.0 has just been released but will take at least 4 weeks before it becomes the recommended version.

@HSSNetworks - there's no single answer - you need to troubleshoot as per the advice above.  Start with power cycling the failed unit.  Also note that 8.5.182.7 was replaced by 8.5.182.11 (link below) so consider upgrading once you have the issue resolved.