Solved: N+1 HA and splitting AP's across WLC's

Charlie Jones · ‎08-12-2024

We are in the process of migrating our N+1 5520 envirnment to 9800 N+1 environment. Both WLC's are in the same datacenter. With the 5520's we used the N+1 design so that we could test OS upgrades before deploying, and it gave us another WLC if we needed to troubleshoot/test.

We decided to keep the N+1 design for our 9800's. Something I am considering is breaking up our AP's in half so that one half would have WLC 1 as its primary and WLC 2 as its secondary. For the other half, I would flip this. Does anyone see any problems in doing this? I don't know if this will introduce issues with future upgrades. In our 5520 environment, we upgraded AP's by upgrading the unoccupied WLC, and then migrate AP's to that controller. On our 5520's, we had all AP's in our AP group. We are taking advantage of tags to separate access points into geographical groups. My thought is since we have them tagged correctly, if we had to test something we could isolate the AP's in a new tag.

eglinsky2012 · ‎08-12-2024

That's a perfectly valid way of going about it. We effectively do this with 3 pairs of HA WLCs -- so we configure the APs for primary, secondary, and tertiary controller pairs. If I'm testing modifications to a site tag for example, I'll test the change on a WLC where the target APs are not joined, and move some to the WLC with the updated config to test. If you're testing code, you can move all the APs off one, upgrade it, and move some back to the upgraded one to test, or just let whatever APs are normally joined to that WLC upgrade with it. You can put both WLCs in a mobility group together to help devices with roaming between APs that are on different WLCs (that is, if there is any coverage overlap between them) -- this is especially helpful if both WLCs use different VLANs for client connections (assuming central switching).

One thing to look out for is code and config consistency. Yes, you can take advantage of the separate WLCs by doing code upgrades on one first, but know that if the WLC with all the APs on it dies, the APs will have to download the new code upon joining the secondary if it's running different code and reboot before they will be available for use again, which usually takes around 10 minutes (or more if going over a WAN or using Wave 1 APs).

Similarly, while you can test configuration changes on one, make sure eventually that you synchronize the changes across both controllers. This can be done manually, or what I do is download the startup config on the test WLC before and after the change and use a file compare tool such as Compare It to highlight the config lines that changed (or were removed) and copy those config lines to (or delete them from) the other controller as needed.

View solution in original post

eglinsky2012 · ‎08-12-2024

That's a perfectly valid way of going about it. We effectively do this with 3 pairs of HA WLCs -- so we configure the APs for primary, secondary, and tertiary controller pairs. If I'm testing modifications to a site tag for example, I'll test the change on a WLC where the target APs are not joined, and move some to the WLC with the updated config to test. If you're testing code, you can move all the APs off one, upgrade it, and move some back to the upgraded one to test, or just let whatever APs are normally joined to that WLC upgrade with it. You can put both WLCs in a mobility group together to help devices with roaming between APs that are on different WLCs (that is, if there is any coverage overlap between them) -- this is especially helpful if both WLCs use different VLANs for client connections (assuming central switching).

One thing to look out for is code and config consistency. Yes, you can take advantage of the separate WLCs by doing code upgrades on one first, but know that if the WLC with all the APs on it dies, the APs will have to download the new code upon joining the secondary if it's running different code and reboot before they will be available for use again, which usually takes around 10 minutes (or more if going over a WAN or using Wave 1 APs).

Similarly, while you can test configuration changes on one, make sure eventually that you synchronize the changes across both controllers. This can be done manually, or what I do is download the startup config on the test WLC before and after the change and use a file compare tool such as Compare It to highlight the config lines that changed (or were removed) and copy those config lines to (or delete them from) the other controller as needed.

Leo Laohoo · ‎08-12-2024

We are the opposite to @eglinsky2012.

We started with two pairs of 9800-80 HA SSO and we've converted them to N + 1 all because of "stability" with the firmware. We have seen alarming behaviours if our HA SSO reach a certain "threshold" and over a few weeks, for instance, the traffic would randomly stop.

In the other forums, someone has responded about the 03 May 2024 revision of the Cisco Catalyst 9800 Series Configuration Best Practices, particularly the "80%" rule.

The issue revolves around the 9800-40/-80/H1/H2 and the "load balancer" process, WNCD. (9800-L only has one WNCD "queue", hence, the 80%-rule does not apply.) When the load of the big controllers get to certain level, it will start to suffer. From what we've seen, 50% AP load is enough to push the controller to the wall (see picture below):

9800-80: IOS v17.12.3, 3080 APs, <10k daily client count, inter-controller roaming, 12 weeks uptime

For 9800-40 (and larger), please be mindful of the following:

<50% AP count
<50% client count
No inter-controller roaming (AireOS to IOS-XE &/or IOS-XE to IOS-XE)
No Web Authentication
NO to HA SSO
PSK or OPEN SSID and and nothing else

Charlie Jones · ‎08-14-2024

Thank you to both of you for your feedback. This was very helpful to hear what I was considering is a valid design.