Best Practice: IP Phone Firmwareupgrade in larger environments

[Lexxy]

Dear Community,

I am looking for a "best practice" how to do firmware upgrades in a company with >40.000 phones. I have the feeling that I can optimize the way how I am doing it right now.

Case:

I want to make all phones ready for CUCM 12.5+ and have to upgrade the firmware for all phones around the world.
34 different Device Types from "Cisco 6901" to "Cisco 9971". Over 40.000 Devices. 

As a 2nd point: I want to monitor this process.

What I am already doing:

Upload Firmware to PCD (Prime Collaboration Deplayment) --> automatic transfer to TFTP --> working with "Device Defaults" and "Phone Load Name".

Option 1 - Phone Load Name

Change Device Defaults back to old entry after Firmwareupload. Update Phone Load Name step by step, device type per device type, country by country and using a Bulk-Job for writing Phone Load Name and restart the phones. 

   Pro: less bandwith usage, TFTPs can handle it

   Con: It takes a looooong time and more than 100 bulk-jobs over weeks

Option 2 - Device Defaults

New Firmware information in the device defaults and also a bulk job for restart the phones in the different countries (county by county, device by device ...)

  Pro: no need to export and upload filtered lists

  Con: If a country has an network outage and is online again, it is possible that 6000 phones at the same time contacting the TFTP to download the new firmware --> takes too long --> time out --> big problems.

My question:

How can I do it in a better, more secure, more convienient and faster way?

Maybe with a external tool? 

Many thanks for feedback

Lexxy

[Chris Deren]

Easiest way is simply install latest device pack for your CUCM version or individual firmwares if preferred and it will automatically update the device defaults.  After restarting the phones they will upgrade.  If you want more controlled update, i.e. apply to subset of phones you can still follow the same procedure, but before you do take a BAT export of your device defaults so that after you perform the firmware install you can change them back to what they were (you can use BAT import) and then just apply the new firmware version by directly referencing the new version under the load filed of the devices.  You may also want to disable TFTP service while you are doing this to avoid premature phone upgrade.

[Mouse]

Lexxy,

I've done a firmware upgrade in a multi-location environment about 2 years ago .
Added problem was that some on the WAN links had such a packet loss and delay, it killed the TFTP downloads. (google a bit on TFTP and packet loss)
There are 2 features in the CUCM that kan be very usefull; Peer Firmware Sharing and Alternative Load Server.
Peer Firmware Sharing allows phone in the same subnet download distribute firmware files among eachother.
When multiple phone restart to do a firmware upgrade, they start to broadcast/communicate among eachother to establish a sort of pyramide structure when they startup after the reboot.
The phone at the "top" of this logical pyramide, downloads a firmware file and uses it for itself. But it also distributes it to 2 other phones below it. And those phones distribute it to 2 other, etc,etc.
This results in a drastic reduction in downloads of the firmware files from the central TFTP server.
I've done a firmware update for 2500 phones being done in 10 minutes.
You can enable the feature on the devices itself (if they support it), There is no control on how the distribution is done among the phones itself (it works automagically ;-))
Keep in mind that the phones only try to find other upgrade peers during the startup of the phone.
Once the phone has upgraded and it is operational, it will not respond to other phone trying the upgrade. You have to reboot a group (say; location) at once.

Next helpfull feature is Alternative Load Server (I'm not sure it is actually called like that)
Check the configuration of a phone. Try to find a field called "Load Server"
You can enter a IP address or hostname.
This can be the address of another TFTP server which the phone can use to download it's firmware files (and only those, not it's configuration files)
What I have done is create a local TFTP server on each location (can be your router) which the phones can reach. Then I pointed the phones to this Load Server (don't forget to check the tickbox behind the field to override the default system configuration)
To keep controle which phones upgrade to the new firmware, I also set the field "Phone Load Name".
This sets the firmware version for that specific phone and overrides the Default Firmware version of the system. I left the Default Firmware version for the system to the old version untill I had upgrade all the phones of that type.

Now, when you use both features the following happens;
After you have set the Load Server and Phone Load Name (can be done via Bulk Administration), you schedule a reset of the Phones per location on the time that the location is closed (again; bulk administration).
The phones restart at the moment and during startup start to do their Peer to Peer filesharing thing. The phone at the top of the pyramide will try to download the firmware files from the local Load Server and distribute it further to it's peers.
So if the local TFTP server responses, the whole firmware update is handled locally.

Using all this I could do firmware upgrades on locations during their night time. And the next day I check if the upgrade succeded.
Tip; when setting up a local TFTP server, also included the firmware files of your current version.
This way you can also use the local TFTP to do a rollback (if needed)