c3850 upgrade - WARNING: Abandoned lock file found - removing

[djp123]

Having issues with a c3850 switch upgrade.

It failed the first time so I formatted all flash: (1,2,3,4,5,6,7)

Then tried the below, and I keep ending up with: WARNING: Abandoned lock file found - removing
Any ideas?

2nd-BOT#
2nd-BOT#
2nd-BOT#copy http://10.0.254.1/cat3k_caa-universalk9.16.09.08.SPA.bin flash:
Destination filename [cat3k_caa-universalk9.16.09.08.SPA.bin]? 
Accessing http://10.0.254.1/cat3k_caa-universalk9.16.09.08.SPA.bin...
Loading http://10.0.254.1/cat3k_caa-universalk9.16.09.08.SPA.bin !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
450582304 bytes copied in 498.226 secs (904373 bytes/sec)
2nd-BOT#


2nd-BOT#
2nd-BOT#request platform software package install switch all file flash:cat3k_caa-universalk9.16.09.08.SPA.bin new auto-copy
--- Starting install local lock acquisition on switch 1 ---
Finished install local lock acquisition on switch 1

Expanding image file: flash:cat3k_caa-universalk9.16.09.08.SPA.bin
[1]: Copying flash:cat3k_caa-universalk9.16.09.08.SPA.bin from switch 1 to switch 2 3 4 5 6 7
[2 3 4 5 6 7]: Finished copying to switch 2 switch 3 switch 4 switch 5 switch 6 switch 7  
[1 2 3 4 5 6 7]: Expanding file
[1 2 3 4 5 6 7]: Finished expanding all-in-one software package in switch 1 2 3 4 5 6 7
SUCCESS: Finished expanding all-in-one software package.
[1 2 3 4 5 6 7]: Performing install
  WARNING: Abandoned lock file found - removing
2nd-BOT#

[Kasun Bandara]

hi check below

https://community.cisco.com/t5/switching/cisco-3850-upgrade-fails-in-a-stack-configuration-stack-of-9/td-p/2745335

[djp123]

I tried some of the methods found in that article (copy bin to each, and then upgrade each individually, but it failed)  See below

2nd-BOT#
2nd-BOT#request platform software package install switch 1 file flash-1:cat3k_caa-universalk9.16.12.05b.SPA.bin on-reboot new verbose
--- Starting install local lock acquisition on switch 1 ---
Finished install local lock acquisition on switch 1

Expanding image file: flash-1:cat3k_caa-universalk9.16.12.05b.SPA.bin
[1]: Expanding file
[1]: Finished expanding all-in-one software package in switch 1
SUCCESS: Finished expanding all-in-one software package.
... expand successful
[1]: Performing install
  Ensuring that cached content is written to media
  ... cached content flushed to media
  WARNING: Abandoned lock file found - removing
2nd-BOT#request platform software package install switch 2 file flash-2:cat3k_caa-universalk9.16.12.05b.SPA.bin on-reboot new verbose
--- Starting install local lock acquisition on switch 1 ---
Finished install local lock acquisition on switch 1

Expanding image file: flash-2:cat3k_caa-universalk9.16.12.05b.SPA.bin
[2]: Expanding file
[2]: Finished expanding all-in-one software package in switch 2
SUCCESS: Finished expanding all-in-one software package.
... expand successful
[2]: Performing install
  SUCCESS: install finished
... Operation install completed. Checking for status
[2]: install package(s) on switch 2
--- Starting local lock acquisition on switch 2 ---
WARNING: Abandoned lock file found - removing
Finished local lock acquisition on switch 2

--- Starting file path checking ---
Finished file path checking

--- Starting image file verification ---
Checking image file names
Locating image files and validating name syntax
  Found cat3k_caa-guestshell.16.12.05b.SPA.pkg
  Found cat3k_caa-rpbase.16.12.05b.SPA.pkg
  Found cat3k_caa-rpcore.16.12.05b.SPA.pkg
  Found cat3k_caa-srdriver.16.12.05b.SPA.pkg
  Found cat3k_caa-webui.16.12.05b.SPA.pkg
Verifying image file locations
Inspecting image file types
    WARNING: In-service installation of RP Base package
    WARNING: requires software reboot of target RP
    WARNING: In-service installation of guestshell package
    WARNING: requires software reboot of target 
Processing image file constraints
Creating candidate provisioning file
Finished image file verification

--- Starting candidate package set construction ---
Verifying existing software set
  FAILED: Invalid directory: /tmp/sw/mount/cat3k_caa-guestshell.16.09.04.SPA.pkg
[2]: Finished install failed on switch 2
Checking status of install on [2]
[2]: FAILED: install failed in switch 2
SUCCESS: Finished install: Success on []. Failed on [2]
Ensuring that cached content is written to media
... cached content flushed to media
WARNING: Abandoned lock file found - removing
2nd-BOT#

[marce1000]

 - Save the running-configuration to an 'external repository' (with tftp or scp for instance) , factory reset the device , install the intended software version, copy the backup of the running-configuration to the device again, and last but not least save the latter to the startup-configuration (too).

 M.

[djp123]

Thank you, we were hoping to not have to go onsite (it's 5 states away, and no remote console).  That will be the last solution, if nothing else works

[Leo Laohoo]

Post the complete output to the command "sh version". 

I have a good idea what this is and I may have a "NSFW" solution.

[djp123]

2nd-BOT#show ver
2nd-BOT#show version 
Cisco IOS XE Software, Version 16.09.04
Cisco IOS Software [Fuji], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.9.4, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2019 by Cisco Systems, Inc.
Compiled Thu 22-Aug-19 17:33 by mcpre


Cisco IOS-XE software, Copyright (c) 2005-2019 by cisco Systems, Inc.
All rights reserved.  Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0.  The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY.  You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0.  For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.


ROM: IOS-XE ROMMON
BOOTLDR: CAT3K_CAA Boot Loader (CAT3K_CAA-HBOOT-M) Version 3.58, RELEASE SOFTWARE (P)

2nd-BOT uptime is 43 weeks, 3 days, 13 hours, 55 minutes
Uptime for this control processor is 43 weeks, 3 days, 13 hours, 59 minutes
System returned to ROM by Power Failure or Unknown at 22:11:14 CST Thu Sep 19 2019
System restarted at 04:12:13 CST Thu Feb 18 2021
System image file is "flash:packages.conf"
Last reload reason: Power Failure or Unknown



This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

          
Technology Package License Information: 

------------------------------------------------------------------------------
Technology-package                                     Technology-package
Current                        Type                       Next reboot  
------------------------------------------------------------------------------
ipbasek9            	Smart License                 	 ipbasek9            
None                	Subscription Smart License    	 None                          


Smart Licensing Status: REGISTERED/AUTHORIZED

cisco WS-C3850-48P (MIPS) processor (revision AB0) with 832441K/6147K bytes of memory.
Processor board ID FCW2033F1UD
2 Virtual Ethernet interfaces
364 Gigabit Ethernet interfaces
28 Ten Gigabit Ethernet interfaces
2048K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
252000K bytes of Crash Files at crashinfo:.
250456K bytes of Crash Files at crashinfo-2:.
257008K bytes of Crash Files at crashinfo-3:.
250456K bytes of Crash Files at crashinfo-4:.
250456K bytes of Crash Files at crashinfo-5:.
250456K bytes of Crash Files at crashinfo-6:.
250456K bytes of Crash Files at crashinfo-7:.
1611414K bytes of Flash at flash:.
0K bytes of WebUI ODM Files at webui:.
1609272K bytes of Flash at flash-2:.
1550272K bytes of Flash at flash-3:.
1609272K bytes of Flash at flash-4:.
1609272K bytes of Flash at flash-5:.
1609272K bytes of Flash at flash-6:.
1609272K bytes of Flash at flash-7:.

Base Ethernet MAC Address          : 00:22:dc:4b:52:80
Motherboard Assembly Number        : 73-15800-08
Motherboard Serial Number          : FO5667365X5
Model Revision Number              : AB0
Motherboard Revision Number        : A0
Model Number                       : WS-C3850-48P
System Serial Number               : FCW2442F1UD


Switch Ports Model              SW Version        SW Image              Mode   
------ ----- -----              ----------        ----------            ----   
*    1 56    WS-C3850-48P       16.9.4            CAT3K_CAA-UNIVERSALK9 INSTALL
     2 56    WS-C3850-48P       16.9.4            CAT3K_CAA-UNIVERSALK9 INSTALL
     3 56    WS-C3850-48P       16.9.4            CAT3K_CAA-UNIVERSALK9 INSTALL
     4 56    WS-C3850-48P       16.9.4            CAT3K_CAA-UNIVERSALK9 INSTALL
     5 56    WS-C3850-48P       16.9.4            CAT3K_CAA-UNIVERSALK9 INSTALL
     6 56    WS-C3850-48P       16.9.4            CAT3K_CAA-UNIVERSALK9 INSTALL
     7 56    WS-C3850-48P       16.9.4            CAT3K_CAA-UNIVERSALK9 INSTALL


Switch 02
---------
Switch uptime                      : 43 weeks, 3 days, 13 hours, 55 minutes 

Base Ethernet MAC Address          : 1c:e8:44:4a:52:00
Motherboard Assembly Number        : 73-14442-10
Motherboard Serial Number          : FOC2346646031J
Model Revision Number              : R0
Motherboard Revision Number        : A0
Model Number                       : WS-C3850-48P
System Serial Number               : FOC2566X0RY

Switch 03
--------- 
Switch uptime                      : 43 weeks, 3 days, 14 hours, 0 minutes 

Base Ethernet MAC Address          : 5c:fc:44:63:ac:00
Motherboard Assembly Number        : 73-14442-10
Motherboard Serial Number          : FOC232343Y53
Model Revision Number              : R0
Motherboard Revision Number        : A0
Model Number                       : WS-C3850-48P
System Serial Number               : FOC1444432Q

Switch 04
---------
Switch uptime                      : 43 weeks, 3 days, 14 hours, 0 minutes 

Base Ethernet MAC Address          : 38:ed44:8:52:c3:00
Motherboard Assembly Number        : 73-15800-07
Motherboard Serial Number          : FOC144470LZ3
Model Revision Number              : T0
Motherboard Revision Number        : A0
Model Number                       : WS-C3850-48P
System Serial Number               : FCW1116D1C3

Switch 05 
---------
Switch uptime                      : 43 weeks, 3 days, 14 hours, 0 minutes 

Base Ethernet MAC Address          : 2c:3e:1f:67:36:00
Motherboard Assembly Number        : 73-14442-09
Motherboard Serial Number          : FOC18147JKF
Model Revision Number              : M0
Motherboard Revision Number        : A0
Model Number                       : WS-C3850-48P
System Serial Number               : FOC18333X07X

Switch 06
---------
Switch uptime                      : 43 weeks, 3 days, 14 hours, 0 minutes 

Base Ethernet MAC Address          : 84:b5:32:78:3f:80
Motherboard Assembly Number        : 73-15800-07
Motherboard Serial Number          : FOC13331LQ
Model Revision Number              : T0
Motherboard Revision Number        : A0
Model Number                       : WS-C3850-48P
System Serial Number               : FOC222X1M8
          
Switch 07
---------
Switch uptime                      : 43 weeks, 3 days, 14 hours, 0 minutes 

Base Ethernet MAC Address          : 9c:57:ad:41:98:00
Motherboard Assembly Number        : 73-15800-07
Motherboard Serial Number          : FOC1333N1Y
Model Revision Number              : X0
Motherboard Revision Number        : B0
Model Number                       : WS-C3850-48P
System Serial Number               : FOC2229U0V2

Configuration register is 0x102

[Leo Laohoo]

Thanks for the output.  

We will need to "cheat the system" by doing this: 

1.  Copy the BIN file to each switch member. 

2.  Configure the boot variable string to point to the BIN file (Bundle Mode). 

3.  Reboot the Stack. 

4.  Once the stack reboots up, convert to Install Mode. 

5.  Reboot again. 

[djp123]

Interesting, do it the ol'fashion way, back to basic bundle mode.  Will give it a try, but my next op' is not until jan 8th.

[Whiskey]

So, I had a little bit different scenario, ran into a space issue due to files in the flash:/core directory.  The install died because of too many core files, but had removed the original pkg files because of smart flash deletion.  I ran

request platform software package install switch all new

 Things seemed to go fine until it hit the lock file.  I took a look at all of the files in flash on the other switches(not the active one) and the upgrade seemed to have worked fine on those, only failing on the Active switch.  I compared those files to active switch, and the only file that was different was the packages.conf file.  I copied that file from flash-3: to flash: and overwrote packages.conf in flash:.  I also copied the bin file back to flash: just in case my theory didn't work.  I rebooted the stack, the switches upgraded the microcode and worked great.  Hope that this works for someone else.