WLC 9800 losing console access when disconnecting fiber cable

[ciscofromit]

I'm getting a weird behaviour when two WLC 9800 are in HA configuration and I disconnect the fiber cable from TenGigabitEthernet 0/1/0 on the primary controller.

If I wait one minute, the controller naturally reboots and I can reconnect the fiber cable. After some minutes the HA syncs and everything gets back working.

If I reconnect the cable too fast, before the controller starts the reboot procedure (about 1 minute), I can't access the serial console anymore and the controller gets stuck in an unrecoverable state of unresponsiveness. The only way to get it back working is to power it off and on again.

I suspect this could be a firmware issue. The current version is 17.06.04.  Anyone who can help?

[balaji.bandi]

First, what is the goal here for this testing ?

In the real world, we don't expect this to occur, if that happens you are in a bigger situation of problem, are you testing this in Live environment.

Look at you steps you posted - when the controller lost connection (not sure what is Tengig 0/1/0 you using for ?)  both controllers may become the split-brain and become active and active.- when you try to connect back (not sure what status WLC that time, so you may see some crash here due to unexpected behavior)

You lost the console connection - this time are you able to ping the device? (both ?)

[ciscofromit]

Hello, thanks for replying! TenGigabitEthernet 0/1/0 is the interface to the access points and clients  vlans (trunk).

When the cable is disconnected, the primary controller becomes "removed" and the secondary becomes the primary. When I lose console access I can still ping both devices but serial console and access through service port are both gone until I power off and on the device.

This test could simulate somebody who erroneously disconnects a wrong cable an quickly reconnects it after realizing the mistake.

[balaji.bandi]

Do you have a high-level network diagram of how these are connected?

realizing the mistake  - engineer to pay high value if one does not understand the network connection and remove the mistake.

[d.palermo76]

Hi balaji.bandi,

I'm working on that customer too. The network diagram is very simple:2 wlc connected back to back with the RP port and having the TE0/1/0 connected to the core switch. HA mode is RMI+RP so disconnecting the TE0/1/0 interface from a WLC (or both) will not cause split brain. Virtual WLC are not showing the same behaviour under the same conditions.

[balaji.bandi]

this required some logs and freezing conditions never observed myself before. if the connection between WLC is intact.

[Rich R]

What is the redundancy state when that happens?
You said firmware but referred to 17.6.4 - that would be the IOS-XE software (semantics) but good to check the firmware too (ROMMON).  You didn't mention the actual WLC model but make sure you have the ROMMON and PHY up to date.
https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/214749-tac-recommended-ios-xe-builds-for-wirele.html#anc47

[d.palermo76]

Hi Rich,

the standy controller becomes active and the former active controller freezes.

We will check the package versions, PHY in particular (latest rommon is dated 15 nov. 2019 and the controllers are pretty new so I assume they will already have the latest rommon). I'll get back to you as soon as I have the data, thanks.

bye, Dario

[Arshad Safrulla]

Refer to the below document, section System and Network Fault Handling

High Availability SSO Deployment Guide for Cisco Catalyst 9800 Series Wireless Controllers, Cisco IOS XE Amsterdam 17.1

If it's not the documented behavior I would suggest reaching out to TAC. Also mention how the upstream switchports connecting to the WLC is configured.

I would suggest enabling spanning tree portfast trunk under the switchport connecting to the WLC and also using switchport allowed VLAN to limit VLAN propogation. More importantly do not configure any NATIVE vlan in the switchports connecting to the WLC.