AP3800 Mobility Express - agonisingly slow boot time

[Matthew Millman]

I've recently replaced a number of Aironet 1250's (which ran in Autonomous mode) with 3800's now running Mobility express

In this setup the access points aren't frequently used and are only powered on when needed (by PoE injector).

The first thing I've noticed is these seem to take significantly longer to boot than what they are replacing.

On the console I see this:

[*05/11/2019 06:47:14.1845] DOT11_CFG[0] Radio Mode is changed from FlexConnect to FlexConnect
[*05/11/2019 06:47:14.1849] DOT11_CFG[1] Radio Mode is changed from FlexConnect to FlexConnect
[*05/11/2019 06:47:14.2059] AP IPv4 Address updated from 0.0.0.0 to 192.168.0.159

several minutes delay - then eventually the WLC starts up and it comes to life

[*05/11/2019 06:49:55.6689] chatter: tohost_virtual :: ToHost: device 'virtual' went down
[*05/11/2019 06:49:55.7089] chatter: tohost_vlan0 :: ToHost: device 'vlan0' went down
[*05/11/2019 06:49:55.7697] chatter: tohost_vlan1 :: ToHost: device 'vlan1' went down

From what I've read this is the "FlexConnect" feature, which first tries to associate the AP to a controller, times out, then starts the local WLC

If this is the case, is this mandatory? If not, how can I cut this apparently unnecessary delay from the boot time?

[Leo Laohoo]

Is the AP servicing any other AP?
If not, then disable Flex.

[Matthew Millman]

It is not servicing any other APs. How do I disable flexconnect ?

I tried the below (didn't work)

(Cisco Controller) >config ap mode local

[patoberli]

Noticed the same with my 3802i and ME 8.8.110.0. It takes nearly 10 minutes for it to fully boot once it's powered. I never really invested a lot of time to troubleshoot it, because mine is running 24x7 anyway.
I'm not sure if you can actually disable Flexconnect with ME: https://www.cisco.com/c/en/us/support/docs/wireless/mobility-express/212484-configure-flexconnect-vlan-mappings-on-m.html

[pieterh]

did you by any chance in the past have multiple AP's active at the same time?

My suggestion is:

maybe some AP';s has registered at the "controller" on another AP that acts as the master.

and when booting checks if this is available, 

this has to timeout first before, the AP will start itself as "backup controller" ?

only after the controller has started, the AP can register itself to this controller.

Mind you, you'll find posts that if you want each AP to behave individually, you'll need a separate (v)lan for each!

=> reset the configuration an start fresh configuration using a separate (v)lan, see if this improves behavior

second do not use DHCP for assigning IP address to the "controller"

[Matthew Millman]

> did you by any chance in the past have multiple AP's active at the same time?

no

> second do not use DHCP for assigning IP address to the "controller"

I am not, however, it appears that it still requests an address via DHCP so it can look for a controller. The statically configured address doesn't kick in until after 6-8 minutes or so, i.e. when the WLC service starts

So far as I can see the boot process is this:

1) Boot AP

2) Get address via DHCP

3) Spend 3-4 minutes trying to find a controller (which doesn't and will never exist)

4) Finally boot and use the built-in controller

5) Apply static IP (if configured)

All I am trying to do here is cut out step 3. Apparently there is a way of doing it because I have log output from others suggesting it is possible.

[Some_Guy]

@Matthew Millman wrote:

The first thing I've noticed is these seem to take significantly longer to boot than what they are replacing.

From what I've read this is the "FlexConnect" feature, which first tries to associate the AP to a controller, times out, then starts the local WLC

If this is the case, is this mandatory? If not, how can I cut this apparently unnecessary delay from the boot time?

No, that's not FlexConnect. FlexConnect mode is the opposite of Local mode, which is when APs tunnel all their traffic through the controller. FlexConnect mode means the APs do not tunnel all their traffic through the controller, but are still controlled by it. It has nothing whatsoever to do with Mobility Express electing an AP to become the ME controller. Only standalone hardware WLCs can operate APs in Local mode; Virtual WLCs and ME controllers require APs to be in FlexConnect mode. So yes, FlexConnect is mandatory in your case, but no, it is not the cause of your problem.

Have you tried setting the AP to be the Preferred Master? It is sitting there waiting in order to give additional APs time to boot, so that they can elect which one should become the ME controller. If you set one as Preferred Master then perhaps it would skip the election and go straight to becoming controller. Just guessing.

I have noticed that some of my ME controllers boot up right away, and some do the waiting thing. Never cared to look into why, but I do know that some of my sites have a Preferred Master configured and some do not. So maybe that's it.

Probably the most popular way to avoid the long boot time is to cut down on unnecessary reboots. ;-) If you insist on doing things oddly, then "agony" is often the price.

[Matthew Millman]

Thanks for the insight.

I just tried your suggestion. Setting an AP as its own preferred master does not remove the bootup delay.

I am beginning to get the impression that these APs are not suitable for my application...

The boot time of the 1250's was a bearable, but this is ridiculous.

[Some_Guy]

@Matthew Millman wrote:

The boot time of the 1250's was a bearable, but this is ridiculous.

No, I have to disagree here. The ridiculous thing is powering the controller on and off for... some reason. Why do you feel the need to power it off? There are probably less ridiculous ways of accomplishing whatever it is you're trying to accomplish, if you cared to share what that is. Waiting an extra 2.5 minutes at boot is not ridiculous for the 99% of people who leave their critical network infrastructure powered on like it was meant to be.

[Matthew Millman]

I'm not going to disagree with your judgement of the application here. It is odd. I had a set of autonomous IOS APs installed in locations where they were only occasionally used, and were powered on by power injector when someone arrives (as previously stated). Because of the infrequent use it is hard to justify leaving them on 24/7, and also when there is a good mechanism to switch them.

Because I can't guarantee that any one in particular will be on (and they're separated by WAN links) having a master doesn't make sense. 

The problem now is that if you happen to need wireless immediately upon arrival, you've got to make a cup of tea.

It looks like 3800's were a poor choice for this application - either that or I've got to splash out on a dedicated controller, or change something else.

Interestingly 3700's appear to still have autonomous mode, so perhaps I can return these and see if I can get some of those.

[Some_Guy]

@Matthew Millman wrote:

I had a set of autonomous IOS APs installed in locations where they were only occasionally used, and were powered on by power injector when someone arrives (as previously stated). Because of the infrequent use it is hard to justify leaving them on 24/7, and also when there is a good mechanism to switch them.

Well if it was me, I'd argue that "takes forever to boot" is actually a very easy justification for leaving them on 24/7. :-)

Is your desire to power APs off when unused due to energy consumption or for infosec reasons? Shutting down the radio interfaces will stop a pretty hefty chunk of the energy consumption and essentially provides equivalent security to powering off. Maybe that could be an okay compromise? Maybe on a schedule?

Or put the injector on a motion sensor meant for lighting? So at least the boot-up process can get a head start as soon as somebody walks in to the place. Would be a hack for sure. (Since you apparently have the budget for Cisco I'm guessing you probably want a less hackish solution.)

Interestingly 3700's appear to still have autonomous mode, so perhaps I can return these and see if I can get some of those.

I'd be extremely wary of regularly powering off a 3700 due to the likelihood of flash corruption. The last models of IOS APs are notorious for flash corruption. I hardly have any 3700s but I do run into the issue a lot with 2600s at sites that have frequent power outages. You might want to read up on the flash corruption issues:

Field Notice: FN - 70330 - Cisco IOS Access Point Stranded Due to Flash Corruption Issue

Understanding Various AP-IOS Flash Corruption Issues

Also, even in autonomous mode I wouldn't call the boot-up time "fast" by any means. I guess it will be faster than booting ME, but in general, everything from Cisco takes a pretty decent chunk of time in order to POST and verify the checksum of the OS image. If fast boot time is a specific requirement you have, then a lot of enterprise-grade gear is probably a poor fit for you, unfortunately.

[Matthew Millman]

For security reasons I won't go into any more details of the scenario - 

I'd agree the APs alone may not burn much power, but if they were left on, they'd be doing nothing because switches are powered off at the same time too (along with a whole bunch of other stuff).

I guess I got lucky with the AP1250's (and perhaps I shouldn't even be upgrading them) - obscure requirements which they happened to meet. They've been going up and down for near a decade now without a hitch.

Either that or we'll have to get used to waiting an extra 5 minutes for wireless access to come up.

[patoberli]

Just to make sure, can you check under
Wireless Settings - Access Points, Edit icon on the master, General - Operating mode field - Make me controller is selected?

[Matthew Millman]

That option is disabled. But it doesn't look like it'd be any use anyway. (From the documentation):

For a master AP, the Operating Mode field shows AP & Controller. For other associated APs, this field shows AP Only. The Make Me Controller button is available only for subordinate APs that are capable of participating in the master election process.

[patoberli]

Found that in some other documentation that didn't mention that part. I had the hope to make it the always master that way.
Honestly I suggest to file a TAC, maybe there is some (badly) documented CLI command to reduce the time for the election process, or to enforce it to be master.