AirOS to 9800 IOS Migration with 0% or with minimum Downtime

[CiscoWiFiGuy]

Hello Experts,

I am looking for your suggestions / Recommendations for below scenario and appreciate if you can share some information based on your experience.

We have a Client who is using 5508 as internal / Foreign  WLC in HA and another set of 5508 which they are using as anchor. Current access point model is not supporting 9800WLC platform. Staff SSID is using 802.1x and Guest is using auth portal via Cisco ISE.

For an upgrade ,They have got 2x9800-40 WLC which will be used as internal/foreign WLC in HA and 2x9800-L WLC which will be used as anchor WLC. New ap model is 9130 series. 

Initially We are planning to configure new Ap mgmt dhcp pool , new staff and guest subnet on new 9800 wlc and change the switch port vlan to new ap mgmt vlan while doing migration for one by one access points. But so far client has not confirmed this new change in their network and its yet to be confirmed. but i was going through some 9800 WLC-AireOS IRCM Deployment Guide and it seems we can do this migration with 0 or minimum downtime. So please share your views. 

https://www.cisco.com/c/en/us/td/docs/wireless/controller/technotes/8-8/b_c9800_wireless_controller-aireos_ircm_dg.html 

What i understand from this is new WLC and old WLC will be a part of same mobility group but staff/guest subnet will be different on both wlc but still there will be roaming etc.

So looking here for your suggestions and approach on doing this kind of migration.

Cheers

Have a Nice Day!

@leo @ras @scot @vinay 

[Leo Laohoo]

---

@CiscoWiFiGuy wrote:
it seems we can do this migration with 0 or minimum downtime.

---

Not in any way possible. 

Going from AireOS to IOS-XE means the AP will have to download the IOS-XE firmware and reboot.  Depending on the model of the AP, downloading the firmware will take about 90 seconds for Cheetah OS APs and about 15 minutes for older 2700/3700.  During that time, the AP will not be serving any wireless clients. 

Reboot is another thing and that's another 5 minutes.  

Several months ago, I moved about >2k x 2800/4800/1560 and 9130s (Healthcare sites) from AireOS to IOS-XE.  And I had a downtime of 4 minutes and 45 seconds.  That's all.  And the entire migration started from 12:00 (noon) on a normal Thursday workday and finished one hour later.  

[CiscoWiFiGuy]

Thanks Leo for your reply.

i missed to mention that we will keep access point ready with latest firmware during access point staging process where it will download the IOS and configs from wlc like tags , profiles etc and on day of migration , we will only change switch port vlan to new ap mgmt vlan and then it will be a matter of just connecting and bringing it online which definitely will have minimum outage which cant be omitted from whole process and its justified.

Could you please also add some of your experience with IRCM stuff ? will it help us to maintain client connectivity across both old and new wlc's once we have ap online in new wlc and nearby access points are on old wlc.

i had a look into their existing heatmaps and it looks they have good coverage overlap so i assume once we remove lets say ap1 , clients will be forced to connect to nearby ap and meanwhile during this time new ap will come online on new wlc and then if both old and new wlc's are in same mobility group , client can have roaming from old to new wlc and vice versa and keeping in mind roaming decisions are decided by client only but we can fine tune / optimize the roaming approach from wlc end .

So appreciate if you can please share your thoughts and we will be discussing with client our final approach towards this migration.

Thanks

[Leo Laohoo · Accepted solution]

---

@CiscoWiFiGuy wrote:
it looks they have good coverage overlap so i assume once we remove lets say ap1 , clients will be forced to connect to nearby ap

---

That depends if the wireless clients have "experiences" roaming.  My experience (above) was done in a healthcare environment and their wireless clients are accustomed to roaming.  We also provided ample warning (one- and two week notice) of WiFi interruptions.  

The most important of it all is our IOS-XE config was in a put into a "testing" phase for about 4 months.  We cut over several buildings earlier and those occupants went through the motion of testing each VLAN/VLAN Group to make sure everything was configured as identical as the AireOS.  

---

@CiscoWiFiGuy wrote:
i missed to mention that we will keep access point ready with latest firmware during access point staging process where it will download the IOS 

---

I doubt it.  

I am not bragging but the process to manually upload the firmware to the APs is not well known.  Not even to Cisco TAC.  There are only a few of us (in this forum) who knows how to do this.  I have seen AireOS-to-IOS-XE migration presentation(s) at Cisco Live and all of them talk about an outage time of >5 minutes.  

[Leo Laohoo]

---

@CiscoWiFiGuy wrote:
So appreciate if you can please share your thoughts and we will be discussing with client our final approach towards this migration.

---

First, you will need to start with this:  Supported Access Points in Cisco Catalyst 9800 Embedded Wireless Controller

The important part of this table is the relationship between the IOS-XE version and the filename of the AP firmware.  For obvious reasons, they need to match.  Let us say the "destination" version is an IOS-XE controller running 17.9.3.  If I look at the table it will say that for 17.9.3.50, the AP firmware will be "15.3(3)JPN2".  Take note of the suffix of the filename, i.  e.  JPN2, because this is extremely important.  

Next, be aware of the AP "family":  2800/3800/4800/1560 is one "family".  9120/9130 is another "family".  Each family shares the same firmware file.  That's right.  The filename "ap3g3-k9w8-tar.153-3.JPN2.tar" can be used by a 2800, 3800, 4800, 1560.  The filename "ap1g6a-k9w8-tar.153-3.JPN2.tar" can be used 9120 and 9130.  The most important part is the "prefix" which I have highlighted in RED. This means I do not need to download a firmware for the 2800, a firmware for the 3800, a firmware for the 4800, etc because the firmware for the 2800 can be used by same family.  

Put the firmware files into a central TFTP or HTTP server.  Make sure the APs can reach it and FW rules are allowed to transfer from the TFTP server to the AP subnet.  

To manually upload the firmware to the AP requires the AP to be joined to the AireOS. 

IMPORTANT:  During the manual upload of the firmware to the AP, WiFi services will not be disrupted.  

Cheetah OS:  

From AireOS WLC, the commands are: 

debug ap enable <AP NAME>
debug ap command "archive download-sw /no-reboot tftp://<TFTP IP ADDRESS>/ap3g3-k9w8-tar.153-3.JPN2.tar" <AP NAME>
!	Wait for 2 minutes
config ap secondary-base " " <AP NAME> 0.0.0.0
config ap primary-base <NEW_WLC_NAME> <AP NAME> <NEW_WLC_IP>
config ap reset <AP NAME>
y

IMPORTANT:  

The last command, rebooting the AP, is the most crucial and must be entered as the new Primary Controller details are entered.  If everything is entered in the right order, the AP will reboot and load the new firmware version.  It will then join the new IOS-XE controller.  Once it joins, the IOS-XE controller will apply the tags.  This entire process will take 4 minutes and 45 seconds. 

If the Primary Controller details are entered but the AP is not rebooted, the AP will join the new IOS-XE controller and download the firmware, reboot and re-joins the controller.  I can guarantee that this process will take about 9 minutes and in this 9 minutes, the AP will not be providing any WiFi services.  

Classic/Legacy IOS AP (2700/3700):  

NOTE:  Due to FN-72524, migrating Classic/Legacy IOS AP will use a different approach.  

Same as Cheetah OS, the firmware file with a prefix of "ap3g2" can be used by the 2600/3600 and 2700/3700/1572/IW3700 family of APs.  And, same as Cheetah OS, the suffix of the file is crucial so refer to the document "Supported Access Points in Cisco Catalyst 9800 Embedded Wireless Controller" carefully.  

If I want to move my Classic/Legacy IOS AP (2700/3700) to an IOS-XE controller on 17.9.3: 

debug ap enable <AP NAME>
debug ap command "debug capwap console cli" <AP NAME>
debug ap command "delete /f /r flash:ap3g2*" <AP NAME>
debug ap command "archive tar /x tftp://<TFTP IP ADDRESS>/ap3g2-k9w8-tar.153-3.JPN2.tar flash:" <AP NAME>
!	Wait for 30 minutes
config ap secondary-base " " <AP NAME> 0.0.0.0
config ap primary-base <NEW_WLC_NAME> <AP NAME> <NEW_WLC_IP>
config ap reset <AP NAME>
y

IMPORTANT:  
I strongly recommend breaking this into two "stages".  The first stage are the first four lines, namely:  

debug ap enable <AP NAME>
debug ap command "debug capwap console cli" <AP NAME>
debug ap command "delete /f /r flash:ap3g2*" <AP NAME>
debug ap command "archive tar /x tftp://<TFTP IP ADDRESS>/ap3g2-k9w8-tar.153-3.JPN2.tar flash:" <AP NAME>

And the next stage are the last four lines, namely:  

config ap secondary-base " " <AP NAME> 0.0.0.0
config ap primary-base <NEW_WLC_NAME> <AP NAME> <NEW_WLC_IP>
config ap reset <AP NAME>
y

The logic is because the Classic/Legacy IOS AP are slower.  It takes about 30 minutes for each AP to finish the manual upload of their respective firmware.  

The same as the Cheetah APs, the last command is crucial and it should not be omitted.  If the last line is omitted, the IOS-XE controller will push the firmware, again, to the AP and extending the outage time further.  

That's it.  Hope this helps.

[marce1000]

           >...They have got 2x9800-40 WLC which will be used as internal/foreign WLC in HA and 2x9800-L WLC 
 - Whilst you are in this migration project it is also very advisable to have a checkup review of the 9800 controller (master) configuration with the CLI command show tech wireless  : have the output analyzed with https://cway.cisco.com/wireless-config-analyzer/
                                                                  Checkout all advisories !

 M.

[Rich R · Accepted solution]

Leo focused on migrating the APs but since you say you are using all new 9130 APs I agree you can have them all pre-staged and ready to rock.

I would only use IRCM if you absolutely have to - it will be cleaner and easier if you can avoid that - maybe do one floor at a time rather than 1 AP at a time?  IRCM will just add extra complication that you can do without. 8.5 AireOS is end of software maintenance now so any bugs you find in 8.5 IRCM will not be fixed.  If you decide to use IRCM it will need to be the special 8.5.182.105 escalation image mentioned below (not in the standard software downloads).

But like Leo says the main thing will be to test really thoroughly with all user devices and use cases before starting the main migration.  You don't want unexpected surprises when you start migrating en masse.

[docjb0221]

Just a couple of things to add:

1. Migrate from AireOS WLC to Catalyst 9800 Using Automation with WLANPoller for AP Pre-download - Cisco has a method for-predownloading the code.

2. We migrated during an outage window so we weren't worried about downtime. However, the things that took the most time were the reboot to make the new code active and switching from flexconnect to local mode (we moved from a regional controller to local controllers).