This time its a question to our customers:
If you insisted on keeping SMUs, SPs, Install rollback and the things you have gotten used to today, how would you change the install process to make it simpler, but still provide what we do now?
Or asked differently what do struggle with today when it comes to Installing/Upgrading Software and what would you like to see improved.
We want to hear your feedback. You can send a message too, please don't hold back..
Solved! Go to Solution.
>Turboboot is the last thing we want you guys to do, its slow and painful, turboboot should rarely be used.
Then you really have to do some education within Cisco. During the presales period as well as after buying the ASR9K, we've always been told to do turboboot from one major release to the other. The explanation is that it would clear all the previously installed SMU that could wrongly interact with the new install (a bit like reinstalling windows from scratch in the old days instead of trying a risky upgrade).
>We don't have a TCP stack in ROMMON and we don't plan to support it. We will support things like ONIE and IPXE in the future.
Then do you plan to support larger block size for TFTP? The CLI accepts it but gives you an error when you start the transfer. The idea is really to have good transfer rate to minimize downtime.
Our main issue with ASR9K is the very slow process to upgrade a router. As a service provider, we want our maintenance to have as little impact as possible to our customer by being the shortest possible.
For example we were running 4.3.4 on our ASR9K and faced few bugs. Most of them were fixed in SP3. While upgrading a router from 4.3.4 to SP3, we ran into a bug that completely crashed our router. We had to turboboot everything. We raised a ticket with TAC and they told us we had to install asr9k-px-4.3.4.CSCul58246.tar before upgrading to SP3. Unfortunately, this fix needs asr9k-px-4.3.4.CSCug75299.tar and asr9k-px-4.3.4.CSCui94441.tar. This means to upgrade a router from 4.3.4 to SP3, you actually need to reboot it 3 times!!! By optimizing as much as we can this process, this means a maintenance of ~50min which is a lot of downtime for our customers!
Why not releasing a 4.3.4-SP image each time your release a SP? At least providers could turboboot it already patched?
Concerning turboboot, the transfert speed thru an integrated management port is catastrophic. We can't specify a large block size to speed up things so even if we have a TFTP directly connected to the port, transfers are way too slow. A good way would be to be able to transfer files with HTTP / FTP during a turboboot.
Then if transfers are done in a fast & efficient way, we could save time by directly sending an uncompressed image over the network instead of waiting for the router to decompress the archive.
The key idea in all this is to be able to upgrade our routers with the lowest downtime possible. For chassis with dual RSP, even if we don't use ISSU, we should be able to upgrade one RSP, switch mastership to it (with a reset of all the protocols) and then upgrade the other one. At least this would be a few minutes downtime instead of a big downtime due to a full reboot of the router.
TAC did not give you the proper information. You can actually install all of the SMU together, then only reboot one (1) time.
RP/0/RSP1/CPU0:rack0(admin)# install add tar usb:/SMU-ASR9006/asr9k-px-4.3.4.CSCul93777.tar usb:/SMU-ASR9006/asr9k-px-4.3.4.CSCum70202.tar usb:/SMU-ASR9006/asr9k-px-4.3.4.CSCum43188.tar usb:/SMU-ASR9006/asr9k-px-4.3.4.CSCum51429.tar usb:/SMU-ASR9006/asr9k-px-4.3.4.CSCum03261.tar usb:/SMU-ASR9006/asr9k-px-4.3.4.CSCui94441.tar activate synchronous
It doesn't matter if the source is usb or disk0:/ , etc. You list all of the files together.
Also, you would benefit from issuing the "show hw-module fpd location all" and looking to see if any Field Programmable Devices need to be upgraded.
How do you do that for the SMU + SP at the same time? I'm able to install asr9k-px-4.3.4.CSCug75299.tar and asr9k-px-4.3.4.CSCui94441.tar at the same time but asr9k-px-4.3.4.CSCul58246.tar wants those two to be installed before.
Of course we need an other reboot if we have to upgrade the fpd. I didn't mention that on my post but that was the case.
You're correct, 94441 and 75299 are required first (reload), then 58246 (reload), then SP3 itself (reload).
Sorry I misread your post about the three (3) reloads.
I actually also ran into an issue where the asr9k-video-px-4.3.4 wasn't on the box, so then needed to figure out how to find that, ftp it over to the ASR, install and activate, then move on to the Service Pack.
Mathieu & Jas,
Did you know that you can install CSCul58246 + CSCui94441 + CSCug75299 + SP3 Into a single Install operation = 1 Reload? Even if you did not install CSCul58246 + CSCui94441 + CSCug75299 so long you didn't mind loading additional optional Pies, your device would still be ok. You would get the fixes for CSCul58246 + CSCui94441 + CSCug75299 inherently, when you install an SP, following the installation of any other SP no anomalies will be found. Install these SMUs if you want to avoid the anomalies in Releases which shipped before SP was envisioned, they are for backyard compatibility support only.
A device should ONLY be reloaded ONCE, any SMU+SMU even if there was a pre-requiste/Dependency can be tar'd or installed together. Point taken though, we will do more education in this area.
Not only I didn't know you can do that (if you have a doc / how to that would be great) but I think the guys at TAC that dealt with our case didn't know that either as they asked us to do the procedure I detailed in a previous post.
Point taken, we will educate.. and we will update content (not create more content). The ability to club SMUs, optional PIES, SPs, FP, MINI.Pie together during upgrade or during installation has been there from day one in XR, it gets overlooked..
Indeed I second. As an SP having 50 or so mins downtime especially due to some smu or other issue during upgrade is quite difficult to work with.
unfortunately although I like xr in terms of syntax and usability, upgrading the main code has always been somewhat painful with unexpected issues cropping up and needing intervention from tac
a patched image to turboboot or somehow upgrading one rsp and then perhaps committing to the other is an idea.
Currently I have a lab box where I simulate the upgrade as I'm not comfortable otherwise, but it won't be a lab box forever as it's an expensive solution
Mark. Should rarely have to Turboboot in the field, if you're asked to turboboot then you've been mislead. The only time to turboboot is:
A) Password recovery,
B) Disk corruption,
C) Not enough diskspace on RP/RSP to hold Two IOS XR images
All upgrades are simple and should be performed with an install operation, we may have over complicated the system and out document ion. We will work on that. Please keep providing feedback.
thanks for the feedback. Indeed I believe that more accurate documentation and guides would be useful as the issues we had faced were not mentioned in the upgrade guides, hence the need for tac.
if you know in advance that you need more steps than just install activate <new version> It can be planned better in advance esp for critical box in SP environment.
Installs from 4.3.2 onwards are just that, they are that simple, you can achieve this is one command install add activate <path/pies> we have gone to a great length of sharing to much detail in the upgrade instructions and that's in itself created fear of complexity. In fact it isn't. Upgrades are really simple, just a single command.. All the options we have provided are a cause for confusion. We will be working on simplifying the documentation.
In fact to be fair it was moving away from 4.1.2 that gave us most pain. I'm due to upgrade a 4.3 box to 5.1.3 in the next couple of weeks. Will let you know the result, thanks again
apologies for delay in getting back. upgrade from 4.3.4 to 5.1.3 went smoothly (ie no unexpected items), so thats great. It still took about 2-3 hours to do the entire upgrade including SMUs, not sure if that can be improved. The main concern here is the length of the maintenance window, especially if rollback is required for any reason.
Im avoiding service packs for now for the same reason that many others are doing so, namely that it is not possible to install individual SMUs if required (with some exceptions, granted).
separately, on 5.1.3 this popped up and cant find much except that router-id is no longer supported in terms of configured values.
So is there no deterministic way to have the box use a 'hard-wired' router id ?