Large CUWN conversions - any war stories

[CHRISTOPHER KANE]

We are in the early stages of a proposed conversion from Autonomous to CUWN. We have experience with CUWN but have not run through such a large amount of conversions. We have roughly 600 APs (300 sites w/2 APs per site). We have tested several conversions in lab environments but we do not know the implications of trying to roll this out in an aggressive time frame.

I was hoping someone could share their experiences of such a large rollout. Any war stories or lessons learned would be appreciated.

Thanks for the time,

-chris

[Scott Fella]

In my opinion, I guess it depends on the ap that you will be converting. With the 1131 and 1242's, I have never ran into any issues, but on some 1100's and 1200's, I have. I would suggest testing more and getting a good game plan to converting.. which way works the best. Decide if you are going to upgrade from a local tftp or a tftp across the wan. I would pilot a couple stores in which someone can be out there just in case, before you start converting with no one onsite, if that is the plan.

My war stories is it bombing and I have to get in a lift to reset the AP.

[CHRISTOPHER KANE]

Right. The rate (percentage) of bombing is what I was wondering about. Knowing the limit of the conversion utility within WCS, I'm wondering how fast we can run.

-chris

[Scott Fella]

To be honest... I had better luck using the conversion tool on my laptop than WCS. Again... I think it comes down to what model ap.

[CFayNTAdmin83]

I have experience converting autonomous AP's to lightweight remotely. I recommend the following for 1220's and 1242's.

1. I use the LWAPP upgrade utility to get the lightweight code on the AP's. I do not upgrade more than 4 at a time. Usually I just do two to make sure; especially if it is an upgrade through a WAN connection.

2. If you have a computer at the remote site, install the utility on that and copy the IOS image to it. The upgrades can go much faster if your TFTP server is local to where the AP's are. You'll need RDP or PC Anywhere to get to the PC though.

3. I use the "cisco-lwapp-controller" DNS entry to get the AP's on the correct controller during an upgrade. There is also the DHCP option you can use as well if you have the same model of AP's. DHCP may prove to be better for your setup, as DNS takes a little while for the record to update. Although, you could just point the AP's to one DNS server (specified in the utility) and keep updating the host A record when you need to...

How many WLC's will you be using for the AP's?

Since you have 2 AP's per store I would recommend upgrading one store at a time. This will take longer, but it is safe.

Doing it 2AP's at a time at roughly 6 minutes for two AP's to get converted, I estimate that it would take you roughly 30 hours to get all the AP's converted. This doesn't include configuration of the AP's, so I'd estimate longer. I'm thinking one week of upgrading / configuring AP's like a mad man all day would get it done. This of course is assuming you have the controllers in place and vlans setup on your switches, etc. I hope this helps!

[CHRISTOPHER KANE]

Thanks for the post. Yes, that does help. We'll be limited to the window available for the upgrade. I wish I could do it all day long.

We already have CUWN installed and servicing other clients. So it really is just a matter of migrating more stuff to it. We'll be configuring the DHCP (will be hosted by local routers) and doing the conversion.

We also use the cisco-lwapp-controller DNS entry for our APs to find the controllers. For our other implementations we have had consistent success with that option whereas the DHCP Option 43 seems slightly less reliable.

I'm curious about why you've limited your conversions to 4. Since the WCS (WLC) documentation for the conversion utility sites 10 as supportable. Did you run into issues with doing more than 4 at a time?

[CFayNTAdmin83]

I only hijacked an AP twice out of the hundred or so that I have upgraded using this method. I know the earlier Utilities (2.1 I think) had an issue with upgrading a large number of AP's. I use the 2.5 version utility. My most recent upgrades were done through a WAN on my laptop. I had limited remote support at the time so I had to be very careful not to mess up the AP's. All 20 or so upgraded without issues doing 2 at a time, with all 20 in the IP file. I've upgraded more before, but no more than 4 at a time. I just wanted to share what works for me in terms of reliability. If you check your connections to the AP's (no dropped packets, routing issues) then you could probably get away with doing 4 at a time, just don't overdo it because the AP's will try to join the controller at roughly the same time. Since you have a large number of AP's it may be better to upgrade 2 to 4 AP's per controller at a time. This will significantly speed up the upgrades. You'll just need another PC to run another instance of the utility though...

[mark.cronin]

As mentioned before, if possible have a TFTP server at the remote site. I have used a Cisco router at the remote site and uploaded the LWAP image and configured the router as a TFTP server. Once the upgrade is complete disable tftp and remove the image from the router.

[CHRISTOPHER KANE]

Cool. Thanks for the advice. We'll try that.

Right now out=r game plan is to have two guys perform the upgrades with each doing 4 APs at a time. We'll see how that goes and possibly ramp up. We have plenty of controllers so we may experiment with where we force them to land. I'd like to find how many we can successfully perform in the allotted maintenance window.

Once we get rolling I'll post our progress.

Thanks,

-chris

[CHRISTOPHER KANE]

Initial deployment results:

After having completed various lab testing scenarios, we started rolling out in a slow first wave.

We are using a remote TFTP server and using the local router as the DHCP server, handing the APs the WLC (WiSMs) IP address for initial landing.

Several folks have suggested limiting the wave to 4 APs at a time. We were able to deploy at that rate. We then ramped up to 6 APs at a time and didn't experience any problems.

Thanks to everyone for your input. I'll provide more updates as we ramp up.

-chris

[Leo Laohoo]

Hi Chris,

Please continue the ongoing updates: Good as well as the bad news.

[CHRISTOPHER KANE]

Update:

We have two flavors of connectivity to the remote sites where the APs reside. DSL and MPLS (512k).

We have rolled the CUWN to several of the DSL sites. We are trying to work within a small maintenance window. So far, within a 3 hour block of time we have migrated in batches of between 14 and 18 APs at a time.

We have two guys doing the conversions. Each of them uses a different TFTP server. In both cases the TFTP servers are in a central location, not local to the site getting upgraded. We have plenty of WiSMs but for the time being we are forcing all of these particular APs to land on just two. We use the local router as the DHCP server and hand out the WLC IP addresses via the DHCP option.

Because we are doing the DSL connected sites first, we tend to see the APs coming into the WLCs in waves. We have not overloaded any one WLC with too many APs attempting to land at the same time.

We are curious to see how high we can ramp up once we start migrating the remote sites that are connected via the MPLS. We expect higher reliability with the MPLS connectivity than we experience with the various flavors of DSL speeds.

More updates to follow.

-chris

[Leo Laohoo]

Hi Chris,

What's the status? Did everything go according to plan?

[CHRISTOPHER KANE]

Things are going well.

We have been able to ramp up to getting 50 APs converted per maintenance window. It takes about 5 hours to get through the entire process. So far the majority of the time is finding which controller an AP landed on and moving it to where we want it. We are toying with the idea of decreasing the amount of controllers we have in our Mobility Group so that it becomes easier to find where an AP landed after it gets converted to LWAPP.

We have found some buggy stuff with applying the WLAN Override, the VLAN and the H-REAP portions of the configuration. Inconsistently the GUI (whether via WCS or directly on the Controller) will bark saying that it can't apply a setting without another setting first being applied. So even though all specifics are included within the template, we have to manually set each option. It appears to be a simple 'order of operation' type thing with the template and is inconsistent. It doesn't happen with every AP.

As a side note, we had a cool error pop up on us the other day. We were importing the AP list for the ones we were going to convert and we saw an error in WCS for one of the APs. The error was "Error: Autonomous AP Failed to add device to WCS Reason: SNMP operation to Device failed: Too few VarBinds in response." The problem turned out to be that the AP's b/g radios were non-functioning. By logging into the AP and looking at the available interfaces we saw all of the sub-interfaces of the radio were missing. A show version reflected that there was no 802.11 interface. The neat part was that WCS detected via SNMP that there wasn't any 802.11 being serviced from that AP.

I'll continue to post as we continue to refine the process. Our biggest lesson learned to date is that we need to maintain a checklist of items. We run through the checklist to make sure all items were covered (applying templates, creating maps, etc...).

Take care,

-chris

[Leo Laohoo]

Hi Chris,

Thanks for the updates.