HA standby WLC keeps reloading - 8.5.135.0

[omz]

Hi, 

The standby WLC in HA keeps reloading. What does IPC timeout mean? Is the standby unable to reach the primary?

(W854_01-Standby) >show logging last-reset

!!!Message and Trap Logs from Previous Reset

--------------------------------------------- Last Reboot MsgLog & Traplog ---------------------------------------------
Sys Name: W854_01
Model: AIR-CT8540-K9
Version: 8.5.135.0
Primary Boot Image: 8.5.135.0 (default) (active)
Backup Boot Image: 8.5.131.0
LastReset Reason: Planned Reset
Timestamp: Wed Aug 29 11:07:04 2018
SystemUpTime: 0 days 7 hrs 37 mins 1 secs

-------------------------------------------------------MsgLog Dump ------------------------------------------------------------

*rmgrMain: Aug 29 11:07:03.869: %RMGR-0-RED_HA_RELOAD: [SS]rmgr_utils.c:263 System is rebooting, reason: IPC timeout has occurred multiple times, category:Peer reload req
*rmgrMain: Aug 29 11:07:03.869: %RMGR-3-RED_HA_KA_STATS: [SS]rmgr_main.c:711 Keep-alive stats: peer RP KA loss count 0, peer RMI KA received count 26620
*rmgrMain: Aug 29 11:07:03.869: %RMGR-3-RED_HA_GW_STATS: [SS]rmgr_main.c:710 Default gateway stats: ping loss count 0, ping received count 1

Thanks for any help!

[Sandeep Choudhary]

I think you are hitting a bug related to IPC timeout...

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvi25532/?rfs=iqvred

Recommandation: dwongrade the software to 8.5(131.0) and try again.

Regards

Dont forget to rate helpful posts

[omz]

Hi Sandeep

Thanks for the response.

The WLC is running 8.5.135.0.

CSCvi25532 show up in Resolved Caveats for 8.5.131.0

CSCvk20484 show up in Resolved Caveats for 8.5.135.0 - IPC timeout and tracebacks reported on Cisco 8540 HA pair running 8.5.131.0 (8.5MR3)

You suggest downgrading to 8.5.131.0? 

Thanks 

[Sandeep Choudhary]

My suggestion is to downgrade to 8.5.131.0 and give a try.

if even after the downgrade you have the same issue then contact to Cisco TAC.

Regards

Dont forget to arte helpful posts 

[phil_carter]

Hello,

We have been seeing this very problem with a 8540 HA pair, running 8.5.135.0. Cisco are struggling to find a resolution or link to any known bugs - any resolution detail would be much appreciated.

Thanks

Phil

[Haydn Andrews]

This sounds like a bug, its very similar to CSCvi25532 and CSCvi25532, as already mentioned

Even tho they say they are fixed in 8.5.131.0. I have seen many times where its fixed in one release and then back in the next.

My recommendation if you can would be to raise a TAC case and get confirmation of the issue, and confirm what version to go to.

The TAC current 8.5 recommended version is 8.5.140.0 if you want to try upgrading and seeing how it goes if you don't have a support agreement

For future reference if your looking at software upgrades bookmark the TAC AIROS recommended software.

https://www.cisco.com/c/en/us/support/docs/wireless/wireless-lan-controller-software/200046-tac-recommended-aireos.html#anc12

[omz · Accepted solution]

• IPC - communication between the two boxes before IOSd boots up, transport of HA control messages (such as role selection, keep-alives and so on) and provide the transport for Interprocess Communication (IPC) between the two boxes.
• Upgrade to 8.5.140.0 did not stop the IPC timeouts.
• Packet captures on WLC RP ports showed missing packets.
• Intermediate Nexus switch was affected by CSCvj28133 and had Incrementing Rcv-Err and IntMacRx-Er Errors on uplinks to WLCs.
• Issue was resolved by upgrading the Nexus switches.