topic Re: Large Deployment and Flex connect in Wireless

Large Deployment and Flex connect

billydailey — Mon, 20 Nov 2023 15:50:09 GMT

Hello,

We are starting to ramp up a large deployment of 9166 APs with the 9800 WLC. While in the lab, we came up with the attached document for provisioning an AP for flex connect. It is very time consuming obviously. Is there another process for deploying large numbers of WAPs with the 9800s to be more efficient?

Also, is it normal for it to take the 9166s a long time to join the controller? We recently connected 17 all at once and it took almost 16 hours for them all to finally join the controller.

Re: Large Deployment and Flex connect

balaji.bandi — Mon, 20 Nov 2023 16:01:34 GMT

That is bizzard - each AP take maximum of 10-15 for AP to download image and reboot and join WLC.

not why this is taking long time back to consoled to AP checked why this is taking long time ?

how far AP to WLC ? all in same Enterprise lan

If you like to save time, i would upgrade the AP using some Test WLC offline and join Live WLC.

Re: Large Deployment and Flex connect

Mark Elsen — Mon, 20 Nov 2023 16:12:06 GMT

- Why are you using flexconnect ?

Re: Large Deployment and Flex connect

billydailey — Mon, 20 Nov 2023 16:15:30 GMT

Because of the amount of bandwidth being used, we do not want all the traffic flowing through the controller and also if the controller fails, you do not want client data to stop flowing correct?

Re: Large Deployment and Flex connect

billydailey — Mon, 20 Nov 2023 16:17:54 GMT

That is exactly what we did in the lab environment to save time, even then it seems like when we have more than 2 connected, it takes hours. Once we had all 17 updated and provisioned, we deployed over a 1g link and it still took them that long to join. This is all in a closed environment.

Re: Large Deployment and Flex connect

Mark Elsen — Mon, 20 Nov 2023 16:49:14 GMT

>...Because of the amount of bandwidth being used, we do not want all the traffic flowing through the controller and also if the controller fails, you do not want client data to stop flowing correct?
- If APs and controller are confined to a local intranet then best is to use capwap only (no flexconnect) ; as far as bandwidth and controller usage is concerned this will all work fine provided the controller is properly sized and or the particular model was bought correctly (concerning usage parameters = primarily number of APs )

> if the controller fails, you do not want client data to stop flowing correct?
- For that HA deployments are used , as stated on local intranets , do not use flexconnect

- For the long joining issue , start by checking controller software version , use latest advisory and or check if that can help :
currently 17.9.4a

Re: Large Deployment and Flex connect

balaji.bandi — Mon, 20 Nov 2023 16:51:56 GMT

i am going to look your configuration guide you posted.

by the way what is the IOS XE running Cat 9800 WLC ?

what kind of bandwidth you have where the WLC to remote AP where you doing flexconnect.

Re: Large Deployment and Flex connect

billydailey — Mon, 20 Nov 2023 17:43:06 GMT

We have a 9800 L C and the APs will be deployed at several remote locations, 25 to be exact at the moment, so flex connect would be the way to go? We have the license to deploy 400 APs.

Back to flex connect, is there a better solution to deploy them more efficiently than in the document i included in the original post?

> if the controller fails, you do not want client data to stop flowing correct?
- For that HA deployments are used , as stated on local intranets , do not use flexconnect

- For the long joining issue , start by checking controller software version , use latest advisory and or check if that can help :
currently 17.9.4a

I just looked and we are on 17.9.3 I do need to upgrade the controller.

Re: Large Deployment and Flex connect

Mark Elsen — Mon, 20 Nov 2023 17:56:48 GMT

>...We have a 9800 L C and the APs will be deployed at several remote locations, 25 to be exact at the moment, so flex connect would be the way to go?
Yes , for the APs in remote locations flexconnect is the best solution

>... is there a better solution to deploy them more efficiently than in the document i included in the original post?
Glancing at the document , most parts are one time only configurations on the controller (profiles , tags, ...) ; for the remote APs you may try to set the trunk immediately (with the needed vlans) and have them join the controller , to skip some steps. Check if that can work ,

For the long time joining issue also review these parameters :
https://www.cisco.com/c/en/us/td/docs/wireless/controller/technotes/8-8/b_flex_connect_catalyst_wirelss_branch_controller_dg.html#id_93580

Re: Large Deployment and Flex connect

JPavonM — Tue, 21 Nov 2023 09:38:51 GMT

Your deployment guide have no impact on the time to upgrade code, but only the WAN link, WAN utilization and WLC link utilzation.

If you deploy all APs at the same timefrom scratch, without any previous config, they all will download code from WLC so taking ages. But if you move the WLC to the same code the APs have after unboxing them, all APs will join quickly. After that you can configure them under the same Flex Profile, enalbe pre-download option, and perform the ISSU upgrade with pre-download so only 1-AP model per location will download code fomr WLC, and then it will push the code to neighbour APs.

Re: Large Deployment and Flex connect

Rich R — Thu, 23 Nov 2023 13:46:36 GMT

Right - so those times are ridiculous as others have said. The biggest limiting factor will be circuit bandwidth and round trip time (because the APs use TFTP over CAPWAP) for AP downloads but other than that 15 minutes at most.
Hard to say without seeing your complete WLC config but you might have made the same mistake we made when we first started using 9800. On AireOS Efficient Image Upgrade was disabled by default, on 9800 it's enabled by default. So if you use the same flex profile for different sites you'll have APs at 1 site trying to download the software from an AP at another site instead of the WLC. So rule number 1 is each site must have its own flex profile for those APs and you might want to try with and without Efficient Image Upgrade. In flex profile it's very misleadingly named as "predownload/no predownload" where predownload means use Efficient Image Upgrade and no predownload means download from WLC over CAPWAP. Obviously the person writing the IOS-XE code didn't actually understand the feature correctly!
https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/17-9/config-guide/b_wl_17_9_cg/m_eff_image_upgrade_ewlc.html
Also see https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/17-9/config-guide/b_wl_17_9_cg/m_predwnld_image_ap_ewlc.html#Cisco_Task.dita_e5d9f125-c65c-4b6a-9772-99217eb0acde
And there's a new feature from 17.11 which allows APs to download from WLC over https which is considerably faster than TFTP over CAPWAP but is done outside of the CAPWAP tunnel so you need the routing and connectivity in place for that and obviously you'd need to upgrade to 17.12.2:
https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/17-11/release-notes/rn-17-11-9800.html#whats-new-17111
https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/17-12/config-guide/b_wl_17_12_cg/m_eff_image_upgrade_ewlc.html#oob-ap-image-dld

> Is there another process for deploying large numbers of WAPs with the 9800s to be more efficient?
Hell yes!
1. Use CLI not GUI.
2. Use regex filters to apply the tags and profiles then
3. When an AP joins the WLC you rename it and then do a CAPWAP reset and then the regex filter does the rest for you when the AP re-joins.
ap name AP001122334455 name NEWAPNAME
(yes you need a naming convention which you can easily match with regex so something along the lines of SITENAME-APnn)
ap name NEWAPNAME reset capwap
All your tags and profiles should be predefined and since most of the sites will likely be using similar configs you can re-use in many cases. Always use "show ap tag summ" to make sure all the APs are correctly configured with the right tags.
See https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/17-9/config-guide/b_wl_17_9_cg/m_config_model.html#task_rnz_kyn_kz "Introduction to AP Filter"
Remember the filter only becomes active after you also create a filter priority (otherwise you might wonder why it's not working)
Probably a good time to remind you to read the entire best practice document because this is covered in that:
https://www.cisco.com/c/en/us/products/collateral/wireless/catalyst-9800-series-wireless-controllers/guide-c07-743627.html#Assigningtags
The one thing that is not covered by the tags and profiles is setting the AP WLC HA pri/sec/tertiary.
In the ap profile you can set "capwap backup primary" and "capwap backup secondary" but again I don't think the developers understood what they were doing because that doesn't actually configure the AP HA settings persistently. There's a new feature which can do that added from 17.9.2 which they've called "AP Fallback to Controllers Using AP Priming Profile" which also works off regex filters. It also uses up your limited number of regex filters.
https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/17-9/release-notes/rn-17-9-9800.html#Cisco_Concept.dita_97e48d4d-09ea-4a81-ace5-f6e5a68bd4d8

Re: Large Deployment and Flex connect

Kasper Roholt — Fri, 24 May 2024 06:34:51 GMT

Hmmm, Rich. Your post scare me a little bit as i was convinced that you don't need Flex Profile for every Site Tag to use efficient upgrade (This is just what i thought and not from any documentation). I ask our local Cisco dude and he find these TAC-eng. answers to similar:

Q: One of my customers has a 9800 flex deployment with separate site tags (<100 APs per site tag) for each site (physical location). They have the same flex profile across all sites. They have the WLC in the DC and multiple remote sites. Now, when they perform an Efficient image upgrade by triggering predownload, will 1 AP per model become the master AP at each site tag, so other APs download from a local AP? Wanting to make sure APs won’t be downloading image from other remote sites.

A: Yes in 9800 Efficient image upgrade selects one AP per Site tag that is not local (has a Flex Profile attached) and one AP per model per site tag downloads the image to later share it with the rest of the APs.

A from another TAC eng.: I was discussing this with Luis this morning, and the answer is that the maximum number of site tags is equal to the number of APs joined. In general, for FlexConnect deployments, each FlexConnect site needs to be its own site tag. (Otherwise, the "Efficient Image Upgrade" feature becomes the "Horribly Inefficient Image Upgrade" feature, as APs at spoke site X try to download from spoke site Y, and misery ensues.)

In my mind this indicate that it's not nescarry with multiple Flex Profiles (but Sitetag do the job).. At least that what i get out of it 🙂

Would be nice with some doc to clear this out ... Now just confused on a higher level 😉

Kasper

Re: Large Deployment and Flex connect

Rich R — Fri, 24 May 2024 09:15:33 GMT

Regardless of Efficient Image download you should have a different flex profile for each site because it's used for key distribution and you have the 300 APs and 3000 clients limit per flex profile.
https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/technical-reference/c9800-best-practices.html#FlexConnectsitetag

Re: Large Deployment and Flex connect

Kasper Roholt — Fri, 24 May 2024 09:57:28 GMT

Well, i must admit that i read this, as having site-tags sharing same Flex-Profile will give you key-distribution within site-tag

For FlexConnect deployments, site tag identifies the fast-roaming domain as client key caching and key distribution only happens within a single Flex site-tag.

Kasper

Re: Large Deployment and Flex connect

Leo Laohoo — Fri, 24 May 2024 10:28:46 GMT

@Kasper Roholt wrote:
Would be nice with some doc to clear this out ... Now just confused on a higher level

Documentation is the least of my worry right now.

Quality control (or the lack of) or quality assurance (or the lack of) of the software that gets regularly churned is currently my biggest thing that wakes me up at night screaming.

Allow me to share some bugs that keep me up all night:

CSCwi83124 (`nuff said.)
CSCwj08744 ("show running-config full | format" crashes the router. What can I say?)
CSCwi41337, CSCwj01239 (Tell me again why a licensing process is capable of crashing my router, switch, WLC?)
CSCwk09139 (Tell me again how it is easily possible for a router to crash just bouncing the port?)
CSCwj33979 (Let me get this straight, if I have more than 2800 APs joined to a 9800-80, it will take more than 6 minutes to completely generate the output of the simple command of "sh ap summary", however, if I ran the same command on an older 8540 I do not have the same problem.)
CSCwj48209 (A Cisco router may crash if CDP is enabled. Da fuq?)
CSCuq46932 (A Cisco router may crash if it's acting as a DHCP server. Wut?)
CSCwi46941 (A switch may lose the VLAN database after two consecutive "reloads". F*ck!)

Re: Large Deployment and Flex connect

Rich R — Fri, 24 May 2024 12:12:35 GMT

Agreed but there should be a one to one mapping of site and flex tag as I read it:
"All the settings for the AP in a Flex site tag are done at the Flex profile level, which is then assigned to the site tag."