Re: Prime Software Deployment Issue with IOS-XE 16.12.09

aharsh · ‎03-28-2023

So yesterday I noticed that IOS-XE 16.12.09 dropped for 3650/3850 switches. I went to go deploy the code to my switches and I ran into a few issues.

Note: Prime Infrastructure is on 3.10.2 with patch 3.10.3.

First issue I noticed was that when going through the workflow for deploying 16.12.9 is that the checkbox to "Activate" the code was missing, see below. Not sure why it's missing. If I go through the workflow for ISR 17.6.5 code, the checkbox is present.

This then leads me to my other two issues.

Issue #2, for switches that I left with "Insert boot command" checked, Activate to Sequential, and Device Running mode to "convert to install mode" Prime pushed the 16.12.9.bin file, ran the install command, but failed to reboot the switch. The error it said was MD5 hash does not match.

Issue #3, for switches that I unchecked "Insert boot command" (there were some switches I didn't want to reboot), prime pushed the 16.12.9.bin code, but failed to run the install command and then reloaded the switches. Why did Prime reload my switches when I told it to NOT insert the boot command? I specifically did NOT want my switches to reload. The error they had is shown below:

Anyone else have this issue with Prime? This is the first time Prime has severely messed me over while doing updates, it's been pretty rock solid for years, I have no clue why 16.12.9 is causing an issue for Prime.

A few side notes: 1) I noticed when I was uploading the code into Prime that it took much longer to complete the upload job. 2) This issue was present on 3x different prime servers (yes, I have 3x networks I manage). 3) The 16.12.9.bin uploaded into Prime was a good file, I checked the file hash from CCO and it matches. I even manual performed an install and reboot of one of the switches that prime pushed the BIN file too and switch is totally fine, so I know my BIN file is good.

Leo Laohoo · ‎03-28-2023

Try to upgrade the firmware manually: Cisco 3850: IOS-XE/Firmware Upgrade

aharsh · ‎03-28-2023

Yeah I've already manually done the upgrades, just need reload my switches. That's not what I was after with this post, I was wondering if anyone else was having this issue with Prime specifically. Thanks for sharing though.

newberntac · ‎04-10-2023

Any update on this? I have the same version of Prime and mine just did the same thing with 16.12.8 distribution to one of my 3850s - Activate was UNCHECKED and it still reloaded the switch, then failed to activate the new version! Meaning I'll have to reload it AGAIN at another time. Happened to be OK as the users left earlier than I anticipated, but could have been bad! But this can't continue and surely @aharsh and I aren't the ONLY ones with this issue.....

@aharsh , have you put in a TAC case yet? I'm about to open one.

aharsh · ‎04-10-2023

I avoid opening TAC cases at all possible, it's usually fairly difficult for me to work with them with how my situation is (on top of TAC just generally being hard to deal with).

I wonder if it was the 3.10.3 patch that has caused this issue. I have an eve-ng box, I might be able to lab up a 3.10.2 version and see if deployment works as expected then upgrade to 3.10.3 and see if it breaks.

newberntac · ‎04-11-2023

I agree! TAC has become very hard to deal with, but we're a small shop here (I do network, phones, and other, non-Cisco areas) and I don't have time to lab stuff and can only DREAM of a lab environment I opened TAC case 695360709 last night. Hopefully SOMEWHERE in this we can get an answer. I really doubt we are the ONLY ones with this issue and I have more switches I need to deploy code to.

newberntac · ‎04-11-2023

Well, my assigned engineer just pointed me to this bug - https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwe24843 - created on Feb 2, 2023, no work around, status is "fixed", but he said it will be resolved in 3.10.4..... Which won't be available until end of April, start of May at the earliest......

stefan.mathys · ‎06-18-2023

Hi, has anyone news about the "fix" or is it possible to get a previous version of 3.10.4 ? we have more than 1000 switches to upgrade so there is no chance to do this manually.

btw...we upgrade c9300 with 17.06.05.. it doesn't matter which IOS you would distribute. every switch reboots after distributing.

newberntac · ‎06-19-2023

I see where you can still download 3.10.0..... I was also thinking - could you roll back patches and device packs one at a time to see if one specific one clears it up?

Villa Fung · ‎06-21-2023

I have same issue on this PI version 3.10.2 and I did the same things with distribution to my 3650, then it is also reboot even I unchecked "Insert boot command" and I left with "Smart Flash Delete Before Distribution" checked.

stefan.mathys · ‎06-23-2023

open a TAC and you will get two .js files to fix this issue.

I tested it today and IOS Distuibution works now without Device reboot.

I'm on PRIME V 3.10.2 with 3.10.3 MR Patch

Michael-NW · ‎10-23-2023

Hi Stefan,
we just ran into the same issue, and since I can no longer select (EOL) the Cisco Prime in the TAC Case Manager, and we have already upgraded to 3.10.4 (and the bug is marked as resolved in the release notes - https://www.cisco.com/c/en/us/td/docs/net_mgmt/prime/infrastructure/3-10-4/release_notes/Cisco_Prime_infrastructure_3_10_4_Release_Notes/bk_cisco_prime_infrastructure_3_10_4_release_notes.html), what exactly did TAC Support assist you with, to resolve the issue?