WLC 9800 - EWC Failover Configuration Sync

[Richard Merrett]

I have recently been testing EWC capability and failover in various scenarios running on 9130AXI APs and have a query regarding configuration files remaining up-to-date across multiple devices.

EWC seems to work fairly well with dynamically allocating a Primary and Secondary controller when the system comes up. When that happens, the Primary controller syncs its config to the Secondary controller and everything is fine. I can also access/view/update both Primary and Secondary controller config using the CLI (updating nvram:startup-config or stby-nvram:startup-config). If the Primary controller goes offline, the Secondary controller becomes Primary and then a new Secondary is allocated and again the config is synced over. If the Secondary controller goes offline, then a new Secondary is allocated and again the config is synced. So far so good!

The issue comes in two specific scenarios;

1) If BOTH Primary and Secondary controllers go offline at the same time, the system allocates new Primary and Secondary controllers from the remaining EWC-capable APs. After a few minutes the system is back up and running, but this time the controller configuration is taken from the nvram:startup-config of the newly active Primary controller, which could be an older version of the configuration if that AP had not been a Primary or Secondary controller since the last saved configuration change took place.

2) If the entire system goes down (for example power outage or reboot of the switches to which all EWC-APs are connected then when the system comes back up and the controller selection process takes place, if the Primary controller function is allocated to an AP which was not a Primary or Secondary controller since the last saved configuration change took place, then the whole system comes up with a potentially older configuration, and that older configuration is then also synced to the Secondary controller too making the problem worse.

So, generally speaking everything is OK as long as you can guarantee that i) both Primary and Secondary controllers do not go offline together, and ii) at system startup you know that the Primary controller will be selected from one of the two APs which were Primary/Secondary controllers at the point the system went down.

I'm actually OK with there being a temporary outage if both Primary and Secondary controllers go offline at the same time, that's completely reasonable given EWC is only intended for a small installation with a lightweight level of high availability. It's the unpredictability of which configuration will be used when another EWC-capable AP takes control that is the issue for me. Which leads me to my question;

Whenever I make a configuration change on the Primary controller, this is automatically synced to the Secondary controller. However, is there any way to force that update to all other EWC-capable APs in the installation automatically? i.e. making sure the nvram:startup-config of the EWC controller partition on all EWC-capable APs is kept up to date whenever configuration changes? Or even a way to manually copy the new/updated configuration to other (non-controller) EWC-capable APs?

So far, the only 'workaround' I have found to this is to manually keep taking the Secondary controller AP offline forcing a new standby controller to be assigned, which causes the config to be synced. If I do this continually until all remaining APs in the installation have become Secondary (and therefore all been synced up) then everything is OK again, and I can then bring all the APs back online. However, this is annoying and only really sustainable in a very small installation (let's say around 5 APs) and would be major pain in an installation of 20+ APs.

I have so far found no way to access/copy data to the nvram:startup-config of the controller partition on any EWC-capable AP which is not currently the Primary or Secondary controller. If I could do that, then this may solve my issue!

Has anybody else come across this issue and/or provide suggestions for resolution? Note: I appreciate there are a lot of comments on other threads about recommending only two EWC-capable APs per installation and the rest of the APs being in CAPWAP only mode, but that's more of a way to avoid this issue than a resolution, and if that was mandatory then I presume Cisco wouldn't have allowed more than two EWC-capable APs per site in the first place; the whole point of multiple EWC-capable APs is so they can take over control if there are issues with the Primary and/or Secondary controllers, which does generally work fine as long as their startup config is up to date!

[Cisco_Virtual_Engineer]

Based on your question, it appears that you're asking if there's a way to ensure that all EWC-capable APs' startup-config are updated whenever configuration changes occur on the primary controller. After investigating, there doesn't appear to be an automatic method provided by Cisco to sync configuration changes to all EWC-capable APs.

One possible solution that I found in Cisco documentation for handling AP configurations involves converting a CAPWAP AP to an EWC AP. This process involves downloading and unzipping the EWC image, ensuring a DHCP server is in place to assign IP addresses to APs, convert the AP to EWC using specific commands and finally, configuring the AP from the wireless configuration wizard.

However, this process doesn't explicitly mention updating all EWC-capable APs' startup-config whenever configuration changes are made. It's possible that further steps or different procedures might need to be taken to achieve the specific configuration syncing you're describing.

For more detailed or specific support, I recommend reaching out to Cisco support directly or posting your question in Cisco's community forum for more targeted assistance.

Please provide more details about the configuration changes you want to sync if the above information doesn't meet your specific requirements, and I can provide additional assistance.

[JPavonM]

Why do you want all APs to be eWLC-capable?

Best practice would be to have only 2, or 3 at most as this is the number of maximum HA controllers you can configure for all APs.

[Richard Merrett]

Hi @JPavonM,

I don’t necessarily need all the APs to be EWC-capable, but it seems reasonable to have more than 2 that are EWC-capable because otherwise if all the other APs apart from the Primary/Secondary are only running CAPWAP images, then if either of the controllers go down I immediately lose HA because only those two APs could ever be used as controllers. Similarly, if maintenance is being performed on part of the network (for example rebooting a switch to which one or both of the current controllers are connected) then the system goes down completely, whereas if I have other APs running EWC images then they will take over and the system remains available also with HA.

The fact that i) initial election of the Primary controller in an EWC network decides between multiple APs which ones will be Primary/Secondary, and ii) if either Primary/Secondary controllers go down then the system automatically tries to find another EWC-capable AP to take over as the new Secondary; this tells me the EWC system was actually DESIGNED to have more than 2 EWC-capable APs in the network. You’ve even said yourself there that having 3 EWC-capable APs makes sense.

So, given the fact it is clearly intended that a network can contain more than 2 EWC-capable APs, my question is simply;

How do I copy the current controller configuration to other EWC-capable APs which are not currently controllers?

I appreciate this may not be something which can happen automatically and I appreciate not every single AP in the network should be EWC-capable, but even if i have just 3 EWC APs, how do I copy the controller configuration to the third AP? Amending the configurtion updates the Primary controller and syncs this to the Secondard controller AP, but I am not sure how to copy that updated config to the third (or fourth etc.) AP?

[balaji.bandi]

There is only 2 deployment SSO and N+1

If you like to sync all the configuration SSO (build the 2 Pair for more HA you looking to achieve)

N+1 you need to have Script or API to manage the config Manually.

Look some guide Lines :

https://www.cisco.com/c/en/us/products/collateral/wireless/catalyst-9800-series-wireless-controllers/guide-c07-744299.html

https://www.cisco.com/c/en/us/products/collateral/wireless/embedded-wireless-controller-catalyst-access-points/white-paper-c11-743398.html#EWCHAactiveandstandby

[Richard Merrett]

Hi @balaji.bandi 

Thanks for the reply and links. I’ve actually already read those documents multiple times but I still don’t see anywhere that tells me how to copy the controller config to a non-controller EWC AP. Are you able to point me to a specific command that shows how to copy this manually?

Note that EWC only supports SSO, not N+1. This question is very specific to EWC, because I cannot work out how to access the controller partition of the non-controller EWC-capable APs to copy the config over. This is different to a normal 9800 WLC because there you can access all controllers even those that are not currently part of the HA.

i’m looking specifically for a command to copy the current controller config to the nvram:startup-config of the non-controller EWC-capable APs in the network. Is this possible?

[balaji.bandi]

i’m looking specifically for a command to copy the current controller config to the nvram:startup-config of the non-controller EWC-capable APs in the network. Is this possible?

Short answer NO - the AP should be EWC for that to accept the Config.

I have read  your requirement you already have pair of AP running Active and standby, you looking additional AP should be standby in case both fails.

If you looking more High availability, if i were you i build 2 Pair of HA and keep split the network and maintain the config consistency.

or setup a alerts when one of the AP fails you can you build another EWC AP and replace it.

[Richard Merrett]

Hi @balaji.bandi 

I think you're missing the point here.

"Short answer NO - the AP should be EWC for that to accept the Config."

The AP's are all EWC-capable running the EWC image. I just need a way to copy the config to those EWC APs, because I cannot see a command to allow the config to be copied over?

"If you looking more High availability, if i were you i build 2 Pair of HA and keep split the network and maintain the config consistency."

This is not possible as you cannot run two separate EWC controllers pairs on the same network, this is not supported. In any case, I am not looking for additional HA I am happy with a dual-controller HA setup when the system is operational.

All I am asking is that since the EWC controllers are designed to be able to fail over to a third EWC-capable AP, I would like to keep the config updated on that third AP. That's all. I understand this may not be possible automatically, but I'm struggling to find a way to do this manually.

If the config on all EWC-capable APs in the network is not in sync, then when the system first boots up there is uncertainty because you can't control which AP will be selected as Primary controller, and so it's important that all EWC-capable APs are in sync otherwise the system could come up with the wrong (or an older) configuration.

As I mentioned in my original post, I'm not looking for advice on how to set up the network nor looking to change the HA characteristics, and I don't want to install physical 9800 WLCs as that's overkill for my use case. I am simply looking for a way to copy the controller config to other EWC APs in the network manually. It feels odd that there is not an easy way to do this, given that Cisco have specifically designed the system to automatically perform controller election at system startup from all EWC APs on the network.

If there is no way to do this then that's a bit rubbish, but I'm just reaching out to the community to see if anyone also has been able to solve this for their EWC network. Again, I'm not looking to change my setup or fundamentally redesign then network, just hoping that copying the configuration is possible.

[JPavonM]

Then your only way to do it is by using automation and some kind of scripting to get the running-config from the active eWLC, get to know the secondary eWLC, and copy the config to the third, fourth, fifth, ... eWLCs.

Not only Cisco, but any vendor support automatic sync of configs between HA systems for more than 2-peers.

If your deployment is too wide, I would recommend you to split it into areas and different subnets for eWLCs not to discover between them, so to have different controllers into every area. This way you can control which ones are the primary and secondary ones, although again you will have to sync configs between them, but this time you know which ones are the primary ones.

[Richard Merrett]

Hi @JPavonM 

Thanks for the reply. I think the main issue here is actually scenario (2) in my original post. Whilst the first scenario does also cause the same issue, the problem is most prevalent when the system first powers up. If I have a network running where there are more than two EWC-capable APs, then I cannot easily control ensuring that new APs with an older config do not become the primary controller.

As an example, let's say I have a network with four APs, all EWC-capable. AP1 and AP2 are Primary and Secondary, with AP3 and AP4 being EWC-capable but just members. Everything is running fine in dual SSO HA. Now, if AP4 has a hardware fault and I replace AP4 with AP5 (also running an EWC image) then AP5 becomes a member. However, if AP5 has the lowest MAC address of all the APs, then if I reboot the system (or there is a power failure) then AP5 will come back up as the Primary controller but because AP5 was only previously a member (not a controller) it does not have the latest config and the system comes back up with the wrong configuration.

That's just one example of how this can occur, hence my desire to be able to copy the config from the active controller over to other EWC-capable APs also.

I'm interested in your comment about scripting, but what I am missing is how do I access the nvram:startup-config of (in my above example) AP5 when this is just a member? As I understand it, I need to update the nvram:startup-config of the WLC partition on that AP device, but if the script is running remotely (either on the controller AP or on a remote server somewhere) then how does the script access that AP5 nvram: location? That's what I am struggling with, how do I access that location (from anywhere)? Even if the script is running on the AP itself, the script is running on the AP partition not the EWC partition, so how do I save the config down to the EWC-capable AP?

Any ideas how I can connect to the nvram: of member APs in the network, but specifically the WLC partition of those APs? Unless I have missed something in terms of how EWC's are set up?

[JPavonM]

But that faulty scenario you describe happen to all Cisco deployments. In the unlikely scenario were 2 APs in eWLC mode were down, and assuming you have configured all APs to be in Flexconnect mode, everything should work fine as far as you configure local AAA in the Flex profile for redundancy. Then after the outage the eWLC will bring back up and that's it.

But if you separate the eWLCs to be connected in different siwtches or even IT rooms, then you will add more resiliency to the deployment.

If this is not enough for you, then it seems to me you have not selected the proper technology for your deployment if you want everything to work independent of any WLC.

[Richard Merrett]

I don't want everything to work independently of the WLC, I'm completely happy that if the whole network goes down then I will lose the controllers and the APs will stop working. Even in FlexConnect mode, if both the controllers disappear then new clients cannot join, and I'm fine with that. I don't have any issues with the level of redundancy for HA when the system is up and running.

The issue is when the system is rebooted. Given that EWC deployments are intended to be small deployments (usually 20 APs or less, and often 5 APs or less) it's quite normal for everything to be connected to a single switch, or even if multiple switches are used then both of them may be connected to a single power source. So it's really that system reboot scenario (where all APs reboot) that is the main issue.

When the system reboots, it's unpredictable as to which EWC-AP will become the primary controller. This means that it's unpredictable as to which version of configuration will come back up when the system restarts, as it depends which EWC-AP is chosen as part of the initial election process. When election takes place, there is nothing checking which EWC-AP has the latest config stored, and so the election takes place and then the config stored on that specific AP is then used for the primary WLC.

All I am really asking is, to avoid the uncertainty on system restarts, to be able to somehow manually copy the latest config over to all EWC-APs. I'm happy doing this manually in my deployment as I only have a small number of APs, but the problem I have is I cannot find a command that I can use to perform the copy. Scripting has been mentioned, but even if I run a script I don't know how to copy to the EWC-AP controller partition.

Cisco actually designed EWC to be able to handle >2 EWC-capable APs on the same deployment (since the initial election process and also the failover process utilises more than 2 EWC APs), so it seems reasonable for me to ask how to copy the config, but I haven't yet seen a reply that tells me how to do that. I appreciate that if I need more HA or I want N+1 then my technology choice is wrong, and I agree that if my deployment was bigger then I'd either split it into different networks/VLANs or deploy physical controllers instead of EWC. However, I'm perfectly happy with only dual HA and the system becoming unavailable if both controllers go down. What I am not happy with is when it then restarts it comes up with an older/incorrect configuration in an unpredictable way.

Is there any command/script I can use to perform the copy? I know you mentioned scripting, but I'm really after a specific command because I cannot see how to access the nvram: of the EWC at all, even when logged directly in to the member APs, as the controller is running on a different partition to the AP, as the EWC-image partitions the AP to keep them separate?

[JPavonM]

Why don't you set 2+2 APs as eWLC and sync the config between the pairs using a python script?

Why do you need to set ALL APs as eWLCs? Is your environment such critical? If so, why do you not deploy a couple of C9800-CL to manage the APs? If you prefer physicals, why not a couple of C800-L?

You are asking for a feature that this soultion is not developed to, unless this would be a cloud managed solution where none of the APs manage the rest, this is the only way you can get this working like you want.

[Richard Merrett]

My environment is not critical enough to justify two physical controllers (or even two cloud 9800-CL), EWC works fine for the most part. I'm generally happy with the setup, and as I've said a few times in this thread I am not looking to increase the HA nor am I looking to have any automated failover or prevent the system going offline.

It's simply about ensuring that when the EWC controller comes up (for example from a full system reboot in my small deployment) it doesn't accidentally pick up an older configuration by electing an EWC AP at the outset which has an older version of config stored on it.

In my view, the fact that the election process itself (as per Cisco design and implementation of EWC) selects Primary and Secondary controllers from all available EWC APs (and does not restrict the election to just two, nor does it restrict the election to only those APs which were last acting controllers) then it seems to be expected by Cisco that there could be more than two EWC APs in the network. That being the case, it seems odd not to be able to manually copy over the latest config to all of them, even manually.

If the config was in sync then it would not matter which EWC APs were selected during election, as they'd all have the same config. It's a massive pain to reboot the switch/router only to find that when five APs come back online they are using a completely different config and none of the clients can connect anymore just because the 'wrong' AP was elected as Primary controller.

Given all the replies in this thread so far, I think you are saying there is no way at all to copy a configuration file from the current Primary controller to another EWC AP in the same network. Is that correct? If so, this feels very odd and a fairly basic omission to an otherwise solid implementation.