3850 XOS upgrade fails

[a.otto]

Catalyst 3850

Hello,

I'm trying to upgrade a stack of two 3850's from 03.07.04E to 16.06.05 using the command:
software install file usbflash0:cat3k_caa-universalk9.16.06.05.SPA.bin switch 1-2 new force

Unfortunately the upgrade process stops after copying the file to all switches and the error message is not very meaningful:

LAB-3850-1.Gustavsburg#$-universalk9.16.06.05.SPA.bin switch 1-2 new force
Preparing install operation ...
[1]: Copying software from active switch 1 to switch 2
[1]: Finished copying software to switch 2
[1 2]: Starting install operation
[1]: % Operation install cannot be performed: install in progress. Operation aborted.
[1]: % An internal error was encountered. Operation aborted.
[2]: % An internal error was encountered. Operation aborted.

I tried to reload the whole stack but no change. Google search was not helpful at all. Does anyone have an idea?

[balaji.bandi]

Try below :

request platform software package install switch all file flash:XX.newimage.bin 

let us know how it goes, since it is lab,

[a.otto]

Thanks for your quick reply, unfortunately the "request" command seems to be
unavailable in 3.X.X version so I receive an error message when I tried it



Axel

[Leo Laohoo]

---

@a.otto wrote:

software install file usbflash0:cat3k_caa-universalk9.16.06.05.SPA.bin switch 1-2 new force

---

Can you please try this command:  

software install file flash-1:cat3k_caa-universalk9.16.06.05.SPA.bin new force verb on-reboot

The difference between the above command and the Cisco recommended on is the presence of the "verbose" option.  This will provide (limited) information why it is not successful.  The "reason" will appear in the "sh log".  

WARNING:  3.X.X do not "like" if the BIN file is located in the USB flash.  Highly recommended if the BIN file is copied into the flash before invoking the "software install" command.  

The other attractive thing is also the presence of the "buggy" option "on-reboot".  This means after the unpacking is complete, the stack will NOT reboot.  

NOTE:  This option is a "bug" because it is meant to be "no-reboot" (CSCve94966).  

Finally, I suspect the command to unpack was not successful because the flash was not "cleaned" before unpacking.  

[a.otto]

Hi Leo,

thanks a lot for taking care and your input! I tried what was recommended, unfortunately the result is very poor but maybe you have an idea what to do with the extended information....
I always do a clean up of the flash before I start a new try:

LAB-3850-1.Gustavsburg#software clean switch 1-2
Preparing clean operation ...
...

packages.conf
[1]: Files that will be deleted:
cat3k_caa-universalk9.16.06.05.SPA.bin
[2]: No unnecessary package or provisioning files found. Nothing to clean

As you have recommended I now started the upgrade process from flash instead of USB using the two additional parameter:

LAB-3850-1.Gustavsburg#$versalk9.16.06.05.SPA.bin new force verbose on-reboot
Preparing install operation ...
[1]: Copying software from active switch 1 to switch 2
[1]: Finished copying software to switch 2
[1 2]: Starting install operation
[1]: % Operation install cannot be performed: install in progress. Operation aborted.
[1]: % An internal error was encountered. Operation aborted.
[2]: % An internal error was encountered. Operation aborted.

So now we have some "debug" information in the log:
000162: 000098: Mar 12 11:07:26.372 CET: %IOSXE-7-PLATFORM: STANDBY:2 process xlogger: [packtool]:(debug): command line: [-P 0 --operation expand --filename /flash/ca (LAB-3850-1.Gustavsburg-2)
000163: 000099: Mar 12 11:07:26.373 CET: %IOSXE-7-PLATFORM: STANDBY:2 process xlogger: t3k_caa-universalk9.16.06.05.SPA.bin --destination /tmp/issu/provision/super_pk (LAB-3850-1.Gustavsburg-2)
000164: 000100: Mar 12 11:07:26.373 CET: %IOSXE-7-PLATFORM: STANDBY:2 process xlogger: gs -d] (LAB-3850-1.Gustavsburg-2)
000165: 000101: Mar 12 11:07:26.373 CET: %IOSXE-7-PLATFORM: STANDBY:2 process xlogger: file name = [/flash/cat3k_caa-universalk9.16.06.05.SPA.bin] (LAB-3850-1.Gustavsburg-2)
000102: Mar 12 11:0
000166: 000103: Mar 12 11:07:26.373 CET: %IOSXE-7-PLATFORM: STANDBY:2 process xlogger: pattern = [INVALID] (LAB-3850-1.Gustavsburg-2)
000167: 000104: Mar 12 11:07:26.373 CET: %IOSXE-7-PLATFORM: STANDBY:2 process xlogger: destination = [/tmp/issu/provision/super_pkgs] (LAB-3850-1.Gustavsburg-2)
000168: 000105: Mar 12 11:07:26.373 CET: %IOSXE-7-PLATFORM: STANDBY:2 process xlogger: operation = [expand] (LAB-3850-1.Gustavsburg-2)
000169: 000106: Mar 12 11:07:26.373 CET: %IOSXE-7-PLATFORM: STANDBY:2 process xlogger: debug = [4] (LAB-3850-1.Gustavsburg-2)
000170: 000107: Mar 12 11:07:26.373 CET: %IOSXE-7-PLATFORM: STANDBY:2 process xlogger: detail = [0] (LAB
000171: 000108: Mar 12 11:07:26.373 CET: %IOSXE-7-PLATFORM: STANDBY:2 process xlogger: force = [0] (LAB-3850-1.Gustavsburg-2)
000172: 000109: Mar 12 11:07:26.373 CET: %IOSXE-7-PLATFORM: STANDBY:2 process xlogger: wipe = [0] (LAB-3850-1.Gustavsburg-2)
000173: 000110: Mar 12 11:07:26.377 CET: %IOSXE-7-PLATFORM: STANDBY:2 process xlogger: xml = [0] (LAB-3850-1.Gustavsburg-2)
000174: 000111: Mar 12 11:07:26.377 CET: %IOSXE-7-PLATFORM: STANDBY:2 process xlogger: skip_prompt = [0] (LAB-3850-1.Gustavsburg-2)
000175: 000112: Mar 12 11:07:27.502 CET: %IOSXE-7-PLATFORM: STANDBY:2 process xlogger: Verifying parameters (LAB-3850-1.Gustavsburg-2)
000176: 000113: Mar 12 11:07:29.607 CET: %IOSXE-7-PLATFORM: STANDBY:2 process xlogger: ... parameters verified (LAB-3850-1.Gustavsburg-2)
000177: 000114: Mar 12 11:07:29.607 CET: %IOSXE-7-PLATFORM: STANDBY:2 process xlogger: Validating package type (LAB-3850-1.Gustavsburg-2)
000178: 000115: Mar 12 11:08:22.856 CET: %IOSXE-7-PLATFORM: STANDBY:2 process xlogger: bundle expanded (LAB-3850-1.Gustavsburg-2)
000179: 000116: Mar 12 11:08:22.867 CET: %IOSXE-7-PLATFORM: STANDBY:2 process xlogger: ... package type validated (LAB-3850-1.Gustavsburg-2)
000180: 000117: Mar 12 11:08:22.867 CET: %IOSXE-7-PLATFORM: STANDBY:2 process xlogger: Verifying package signatures (LAB-3850-1.Gustavsburg-2)
000181: 000118: Mar 12 11:08:23.736 CET: %IOSXE-7-PLATFORM: STANDBY:2 process xlogger: Signature verified on package cat3k_caa-guestshell.16.06.05.SPA.pkg (LAB-3850-1.Gustavsburg-2)
000182: 000119: Mar 12 11:08:38.168 CET: %IOSXE-7-PLATFORM: STANDBY:2 process xlogger: Signature verified on package cat3k_caa-rpcore.16.06.05.SPA.pkg (LAB-3850-1.Gustavsburg-2)
000183: 000120: Mar 12 11:08:38.870 CET: %IOSXE-7-PLATFORM: STANDBY:2 process xlogger: Signature verified on package cat3k_caa-webui.16.06.05.SPA.pkg (LAB-3850-1.Gustavsburg-2)
000184: 000121: Mar 12 11:08:39.104 CET: %IOSXE-7-PLATFORM: STANDBY:2 process xlogger: Signature verified on package cat3k_caa-srdriver.16.06.05.SPA.pkg (LAB-3850-1.Gustavsburg-2)
000185: 000122: Mar 12 11:08:40.298 CET: %IOSXE-7-PLATFORM: STANDBY:2 process xlogger: Signature verified on package cat3k_caa-rpbase.16.06.05.SPA.pkg (LAB-3850-1.Gustavsburg-2)
000186: 000123: Mar 12 11:08:40.299 CET: %IOSXE-7-PLATFORM: STANDBY:2 process xlogger: ... Package signatures verified (LAB-3850-1.Gustavsburg-2)
000187: 000124: Mar 12 11:08:40.299 CET: %IOSXE-7-PLATFORM: STANDBY:2 process xlogger: Copying package files (LAB-3850-1.Gustavsburg-2)
000188: 000125: Mar 12 11:08:42.469 CET: %IOSXE-7-PLATFORM: STANDBY:2 process xlogger: cat3k_caa-guestshell.16.06.05.SPA.pkg (LAB-3850-1.Gustavsburg-2)
000189: 000126: Mar 12 11:08:42.470 CET: %IOSXE-7-PLATFORM: STANDBY:2 process xlogger: cat3k_caa-rpbase.16.06.05.SPA.pkg (LAB-3850-1.Gustavsburg-2)
000190: 000127: Mar 12 11:08:42.470 CET: %IOSXE-7-PLATFORM: STANDBY:2 process xlogger: cat3k_caa-rpcore.16.06.05.SPA.pkg (LAB-3850-1.Gustavsburg-2)
000191: 000128: Mar 12 11:08:42.471 CET: %IOSXE-7-PLATFORM: STANDBY:2 process xlogger: cat3k_caa-srdriver.16.06.05.SPA.pkg (LAB-3850-1.Gustavsburg-2)
000192: 000129: Mar 12 11:08:42.471 CET: %IOSXE-7-PLATFORM: STANDBY:2 process xlogger: cat3k_caa-webui.16.06.0
000193: 000131: Mar 12 11:08:42.588 CET: %IOSXE-7-PLATFORM: STANDBY:2 process xlogger: ... Package files copied (LAB-3850-1.Gustavsburg-2)
000194: 000132: Mar 12 11:08:44.600 CET: %IOSXE-7-PLATFORM: STANDBY:2 process xlogger: [packtool]:(debug): Ensuring that cached content is written to media (LAB-3850-1.Gustavsburg-2)
000195: 000133: Mar 12 11:08:48.057 CET: %IOSXE-7-PLATFORM: STANDBY:2 process xlogger: [packtool]:(debug): cached content flushed to media (LAB-3850-1.Gustavsburg-2)
000196: 000134: Mar 12 11:08:48.074 CET: %IOSXE-7-PLATFORM: STANDBY:2 process xlogger: SUCCESS: Finished expanding all-in-one software package. (LAB-3850-1.Gustavsburg-2)
000197: 000135: Mar 12 11:08:50.105 CET: %IOSXE-7-PLATFORM: STANDBY:2 process xlogger: [packtool]:(debug): Received script exit status 0 (LAB-3850-1.Gustavsburg-2)

Unfortunately I was not able to find any helpful information for "Received script exit status 0"

Axel

[Leo Laohoo]

---

@a.otto wrote:

SUCCESS: Finished expanding all-in-one software package. (LAB-3850-1.Gustavsburg-2)

---

Can you post the complete output to the command "more flash-1:packages.conf | b for NOVA".

[a.otto]

Hi Leo,

No output at all:

LAB-3850-1.Gustavsburg#more flash-1:packages.conf | b for NOVA

LAB-3850-1.Gustavsburg#more flash-2:packages.conf | b for NOVA

LAB-3850-1.Gustavsburg#



Axel

[Leo Laohoo]

Ok, last trick:

1. Download 16.6.7;
2. Copy the BIN file into switch 1 and switch 2;
3. Change the boot variable string so the switch will boot the new firmware in Bundle Mode: 

boot system flash bootflash:cat3k_caa-universalk9.16.06.07.SPA.bin

4. Save the config and reboot the switch;
5. After the switch reboots, unpack the BIN file: 

request platform software package install switch all file flash-1:cat3k_caa-universalk9.16.06.07.SPA.bin new auto-copy verbose

6. If the unpacking is successful, check the packages.conf file: 

more flash-1:packages.conf | b for NOVA

7. Change the boot variable string to boot the packages.conf file:

no boot system flash bootflash:cat3k_caa-universalk9.16.06.07.SPA.bin
boot system flash bootflash:packages.conf

8. Save the config and reboot.

Hope this works.

[a.otto]

Everything went well until line #5. Microcontroller were updated and the
switch booted twice. A "sh ver" shows 16.06.07 but the unpacking fails:

LAB-3850-1.Gustavsburg#$rsalk9.16.06.07.SPA.bin new auto-copy verbose

--- Starting install local lock acquisition on switch 1 ---

Finished install local lock acquisition on switch 1



FAILED: switch local is not booted with provisioning file

Ensuring that cached content is written to media

... cached content flushed to media



Axel

[Leo Laohoo]

Please post the output to the command "sh version" and "dir flash-1:".

[a.otto]

Hi Leo

Here we go:

LAB-3850-1.Gustavsburg#sh ver
Cisco IOS XE Software, Version 16.06.07
Cisco IOS Software [Everest], Catalyst L3 Switch Software
(CAT3K_CAA-UNIVERSALK9-M), Version 16.6.7, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2019 by Cisco Systems, Inc.
Compiled Mon 23-Sep-19 12:59 by mcpre

Cisco IOS-XE software, Copyright (c) 2005-2019 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: IOS-XE ROMMON
BOOTLDR: CAT3K_CAA Boot Loader (CAT3K_CAA-HBOOT-M) Version 3.58, RELEASE
SOFTWARE (P)

LAB-3850-1.Gustavsburg uptime is 8 hours, 56 minutes
Uptime for this control processor is 9 hours, 0 minutes
System returned to ROM by Reload command at 13:25:57 CET Thu Mar 12 2020
System restarted at 13:42:42 CET Thu Mar 12 2020
System image file is "flash:cat3k_caa-universalk9.16.06.07.SPA.bin"
Last reload reason: Reload command

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found
at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
If you require further assistance please contact us by sending email to
export@cisco.com.

Technology Package License Information:

-----------------------------------------------------------------

Technology-package Technology-package

Current Type Next reboot

------------------------------------------------------------------

ipbasek9 Permanent ipbasek9

cisco WS-C3850-24P (MIPS) processor (revision AC0) with 852878K/6147K bytes
of memory.
Processor board ID FCW2045F0TJ
3 Virtual Ethernet interfaces
56 Gigabit Ethernet interfaces
8 Ten Gigabit Ethernet interfaces
2048K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
257008K bytes of Crash Files at crashinfo:.
257008K bytes of Crash Files at crashinfo-2:.
1550272K bytes of Flash at flash:.
1550272K bytes of Flash at flash-2:.
0K bytes of WebUI ODM Files at webui:.

Base Ethernet MAC Address : 28:52:61:a3:bd:80
Motherboard Assembly Number : 73-15805-05
Motherboard Serial Number : FOC20453AZK
Model Revision Number : AC0
Motherboard Revision Number : A0
Model Number : WS-C3850-24P
System Serial Number : FCW2045F0TJ

Switch Ports Model SW Version SW Image Mode


------ ----- ----- ---------- ---------- ----


* 1 32 WS-C3850-24P 16.6.7 CAT3K_CAA-UNIVERSALK9
BUNDLE
2 32 WS-C3850-24P 16.6.7 CAT3K_CAA-UNIVERSALK9
BUNDLE

Switch 02

---------

Switch uptime : 9 hours, 0 minutes

Base Ethernet MAC Address : 28:52:61:a6:fe:00
Motherboard Assembly Number : 73-15805-05
Motherboard Serial Number : FOC20452KLV
Model Revision Number : AC0
Motherboard Revision Number : A0
Model Number : WS-C3850-24P
System Serial Number : FOC2045X0SD

Configuration register is 0x102



And:

LAB-3850-1.Gustavsburg#dir flash-1:

Directory of flash:/

8082 -rw- 2097152 Mar 12 2020 17:05:57 +01:00 nvram_config
8092 -rw- 81992100 Oct 18 2018 03:40:07 +02:00
cat3k_caa-base.SPA.03.07.04E.pkg
8083 drwx 4096 Oct 18 2018 03:52:39 +02:00 virtual-instance
56562 -rw- 5667516 Oct 18 2018 03:40:04 +02:00
cat3k_caa-drivers.SPA.03.07.04E.pkg
8086 -rw- 36656136 Oct 18 2018 03:40:05 +02:00
cat3k_caa-infra.SPA.03.07.04E.pkg
8085 -rw- 46349632 Oct 18 2018 03:40:05 +02:00
cat3k_caa-iosd-universalk9.SPA.152-3.E4.pkg
24245 drwx 4096 Oct 18 2018 03:52:39 +02:00 vman_fdb
24247 drwx 4096 Oct 18 2018 03:54:13 +02:00 wnweb_store
8106 -rw- 1236 Oct 18 2018 03:41:35 +02:00 packages.conf
24241 drwx 4096 Mar 12 2020 13:39:53 +01:00 dc_profile_dir
8090 -rw- 1716 Jan 31 2019 07:40:55 +01:00 vlan.dat
8091 -rw- 128103 Mar 12 2020 13:40:13 +01:00 memleak.tcl
48481 drwx 4096 Mar 12 2020 13:39:36 +01:00 .installer
8084 drwx 4096 Mar 12 2020 13:43:27 +01:00 syslog
8087 -rw- 1861378 Mar 12 2020 21:42:04 +01:00 logging.text
8103 drwx 4096 May 6 2018 08:50:57 +02:00 core
8112 -rw- 404622964 Mar 12 2020 11:02:02 +01:00
cat3k_caa-universalk9.16.06.05.SPA.bin
8113 -rw- 407449018 Mar 12 2020 13:17:08 +01:00
cat3k_caa-universalk9.16.06.07.SPA.bin
8102 -rw- 111 Mar 12 2020 13:39:36 +01:00
bootloader_evt_handle.log
16161 drwx 4096 May 6 2018 08:43:21 +02:00 .prst_sync
32322 drwx 4096 May 6 2018 08:43:22 +02:00 .rollback_timer
40402 drwx 4096 May 6 2018 08:43:30 +02:00 gs_script
8104 -rw- 2097152 Mar 12 2020 17:05:58 +01:00 nvram_config_bkup
24244 -rw- 28904332 Oct 18 2018 03:40:05 +02:00
cat3k_caa-platform.SPA.03.07.04E.pkg
56569 -rw- 128582464 Oct 18 2018 03:40:05 +02:00
cat3k_caa-wcm.SPA.10.3.141.0.pkg

1562509312 bytes total (332558336 bytes free)

[Leo Laohoo]

There are still files for 3.7.4.  

Clean this up using the command 

request platform software package clean switch all 

Next, unpack 16.3.7 again:

req plat soft pack inst switch all file flash-1:cat3k_caa-universalk9.16.06.07.SPA.bin new auto-copy

[a.otto]

Hi Leo,

Unfortunately the switch insists that there is nothing to clean and so the
unpack action is not working. Is there something I can do manually?



LAB-3850-1.Gustavsburg#request platform software package clean switch all
This operation may take several minutes...
Running command on switch 1
Cleaning up unnecessary package files
No path specified, will use booted path consolidated:packages.conf
Cleaning sw/isos
Scanning boot directory for packages ... done.
Preparing packages list to delete ...
cat3k_caa-guestshell.16.06.07.SPA.pkg
File is in use, will not delete.
cat3k_caa-rpbase.16.06.07.SPA.pkg
File is in use, will not delete.
cat3k_caa-rpcore.16.06.07.SPA.pkg
File is in use, will not delete.
cat3k_caa-srdriver.16.06.07.SPA.pkg
File is in use, will not delete.
cat3k_caa-webui.16.06.07.SPA.pkg
File is in use, will not delete.
packages.conf
File is in use, will not delete.
done.

SUCCESS: No extra package or provisioning files found on media. Nothing to
clean.

Running command on switch 2
Cleaning up unnecessary package files
No path specified, will use booted path consolidated:packages.conf
Cleaning sw/isos
Scanning boot directory for packages ... done.
Preparing packages list to delete ...
cat3k_caa-guestshell.16.06.07.SPA.pkg
File is in use, will not delete.
cat3k_caa-rpbase.16.06.07.SPA.pkg
File is in use, will not delete.
cat3k_caa-rpcore.16.06.07.SPA.pkg
File is in use, will not delete.
cat3k_caa-srdriver.16.06.07.SPA.pkg
File is in use, will not delete.
cat3k_caa-webui.16.06.07.SPA.pkg
File is in use, will not delete.
packages.conf
File is in use, will not delete.
done.

SUCCESS: No extra package or provisioning files found on media. Nothing to
clean.


LAB-3850-1.Gustavsburg#$3k_caa-universalk9.16.06.07.SPA.bin new auto-copy


--- Starting install local lock acquisition on switch 1 ---
Finished install local lock acquisition on switch 1

FAILED: switch local is not booted with provisioning file

LAB-3850-1.Gustavsburg#

[Leo Laohoo]

del /f /r flash-1:*03.07.04E*.pkg
del /f /r flash-2:*03.07.04E*.pkg
del /f /r flash-1:cat3k_caa-universalk9.16.06.05.SPA.bin
del /f /r flash-2:cat3k_caa-universalk9.16.06.05.SPA.bin

Try the following (above).  

When done, re-try the unpacking again.  

[a.otto]

I really appreciate your help, unfortunately no luck again...



LAB-3850-1.Gustavsburg#del /f /r flash-1:*03.07.04E*.pkg
LAB-3850-1.Gustavsburg#del /f /r flash-2:*03.07.04E*.pkg
LAB-3850-1.Gustavsburg#$lash-1:cat3k_caa-universalk9.16.06.05.SPA.bin

LAB-3850-1.Gustavsburg#$lash-2:cat3k_caa-universalk9.16.06.05.SPA.bin

LAB-3850-1.Gustavsburg#$3k_caa-universalk9.16.06.07.SPA.bin new auto-copy
--- Starting install local lock acquisition on switch 1 ---
Finished install local lock acquisition on switch 1

FAILED: switch local is not booted with provisioning file
LAB-3850-1.Gustavsburg#