Cisco AnyConnect Upgrade Failure

[ConfigMgr]

Hello,

Description of problem

Trying to upgrade my organization to version 4.9.01095 of the Cisco AnyConnect client. I am currently facing a 50% failure rate during all piloting. The failure occurs when the previous version is removed, it errors out. Then the  "C:\ProgramData\Cisco\Cisco AnyConnect Secure Mobility Client\Help" folder has all it's permissions removed and the new version fails due to lacking permissions. A restart seems to fix the issue but not ideal for over 10,000 machines.

Our version expires January 13th and was hoping to have piloting finished this week. Have a very difficult time with it and hopefully someone can shed some light on why I am experiencing so many problems.

Details

Current Version: 4.7.0.1076

Components installed: NAM, DART, Mobility Client, Umbrella on laptops

New version being deployed: 4.9.01095 with the same components

TargetOS: Windows 10 1709 & 1909

Device types: Physical workstations and laptops

Current deployment method (as detailed from Cisco's site)

• Remove the "C:\ProgramData\Cisco\Cisco AnyConnect Secure Mobility Client\Help" folder
  • This only works half the time, other times it is "in use"
  • EDIT - I now realize this may not matter. It appears the "...\Help\" folder is created during install, maybe? 
• Remove the following applications in order (if exist)
  • Cisco AnyConnect Umbrella Roaming Security Module
  • Cisco AnyConnect Network Access Manager
  • Cisco AnyConnect Secure Mobility Client
  • Cisco AnyConnect Diagnostics and Reporting Tool
• Install the following applications in order
  • Cisco AnyConnect Secure Mobility Client
  • Cisco AnyConnect Diagnostics and Reporting Tool
  • Cisco AnyConnect Network Access Manager
  • Cisco AnyConnect Umbrella Roaming Security Module

Prior deployment method (which still failed at 50%)

• Install the following applications in order
  • Cisco AnyConnect Secure Mobility Client
  • Cisco AnyConnect Diagnostics and Reporting Tool
  • Cisco AnyConnect Network Access Manager
  • Cisco AnyConnect Umbrella Roaming Security Module

Errors

• When removing the Secure Mobility Client
  • WIN64DUALFOLDERS: Substitution in 'C:\Program Files (x86)\Cisco\Cisco AnyConnect Secure Mobility Client\api-ms-win-core-errorhandling-l1-1-0.dll' folder had been blocked by the 1 mask argument (the folder pair's iSwapAttrib member = 0).
  • Executing op: FileRemove(,FileName=error.ico,,ComponentId={1E1D3448-C3E3-43C3-880D-106578810E76})
  • CustomAction RemoveFolder_Plugins returned actual error code 1603 but will be translated to success due to continue marking
  • Windows Installer removed the product. Product Name: Cisco AnyConnect Secure Mobility Client. Product Version: 4.7.01076. Product Language: 1033. Manufacturer: Cisco Systems, Inc.. Removal success or error status: 0.
  • RESTART MANAGER: Failed while restarting applications. Error: 5
• When installing the new version
  • Executing op: FolderCreate(Folder=C:\ProgramData\Cisco\Cisco AnyConnect Secure Mobility Client\Help\,Foreign=0,,)
  • Note: 1: 1303 2: C:\ProgramData\Cisco\Cisco AnyConnect Secure Mobility Client\Help
  • Product: Cisco AnyConnect Secure Mobility Client -- Error 1303. The installer has insufficient privileges to access this directory: C:\ProgramData\Cisco\Cisco AnyConnect Secure Mobility Client\Help. The installation cannot continue. Log on as administrator or contact your system administrator.
  • Error 1303. The installer has insufficient privileges to access this directory: C:\ProgramData\Cisco\Cisco AnyConnect Secure Mobility Client\Help. The installation cannot continue. Log on as administrator or contact your system administrator

What I am/am not doing

• Am not attempting to reboot after uninstallation of current version
  • This is not even an option as the uninstall fails before I could do this
• Am waiting 5 seconds between each uninstall, 30 seconds before each install

[ConfigMgr]

I now believe the "...\Help\" folder is a "Ghost folder" in the sense that once a reboot is performed, the folder is deleted. I have added an additional 25 seconds of sleep between the uninstall methods for a total of 30 seconds. I am hoping this is enough time to let the uninstaller finish before the next step starts.

I currently do not know of a way to remove this Help folder. All permissions are gone and I cannot take ownership again, though I have only had the opportunity to test on one device. I am sure there is a way to script that portion, but my total install code is approaching 200 lines of code now...

[stsargen]

I have seen this issue before. Window is marking the folder for deletion but the removal is not happening until a reboot is done. As a workaround I have seen the following work.
Create a help document as described in the Admin guide. https://www.cisco.com/c/en/us/td/docs/security/vpn_client/anyconnect/anyconnect40/administration/guide/b_AnyConnect_Administrator_Guide_4-0/customize-localize-anyconnect.html#ID-1408-00000291
Use your uninstall script or other means to make sure this file is in place before performing your upgrade.
Hope this Helps.

[ConfigMgr]

Thanks for the reply, this looks promising.

To reiterate:

• Create a "help_AnyConnect.html" file
  • I am assuming this can be a blank file with the appropriate extension (.html)
• Remove AnyConnect components
• Copy the "help_AnyConnect.html" to "C:\ProgramData\Cisco\Cisco AnyConnect Secure Mobility Client\Help"
• Install AnyConnect components

I can 100% confirm the moment the device reboots the "Help" folder is removed. I just want to verify if I need the html file in there before uninstalling or just after.

Thanks!

[stsargen · Accepted solution]

I tested and the file can be blank. The file only needs to be named "help_AnyConnect.xxx" if you are deploying from the ASA. If you are deploying out of band with SCCM, GPO, or manually then the filename is just AnyConnect.html. you may need to kill the vpnui.exe process and relaunch for the icon to show up in the GUI.
This should be the order of events.

1. Copy the "AnyConnect.html" to "C:\ProgramData\Cisco\Cisco AnyConnect Secure Mobility Client\Help"
2. You may need kill the vpnui.exe and restart it....
3. Remove AnyConnect components
4. Install AnyConnect components
Please let me know the results.

[ConfigMgr]

Thank you for the clarification. I will be testing another pilot this evening to ~100 devices. If successful I will post a code snippet of my entire method as I am installing via ConfigMgr. May be helpful to others who find this post as well.

[ConfigMgr · Accepted solution]

Another test this evening to ~140 devices. Only half online but I can report much, much higher success. Currently 50 success and two failures. Both failures are a very similar error, but reference the "Plugins" folder instead of the "Help" folder:

Product: Cisco AnyConnect Secure Mobility Client -- Error 1303. The installer has insufficient privileges to access this directory: C:\Program Files (x86)\Cisco\Cisco AnyConnect Secure Mobility Client\Plugins.

1/10/20 Update: the solution provided has taken me from a 50% success rate to 97%. This final pilot has went well enough to roll into production. It appears any device that fails with the error in this post or the one in my OP resolve after a reboot. Here is a brief summary of the steps I took for a successful deployment. Hopefully this helps others who may find this as I had a lot of difficulties with deployment. Also to note, I really do not know much about AnyConnect - we do not use the VPN portion of the client and may be using the solution differently than others.

Deployment

• I use the PSADT to package my applications and deploy via ConfigMgr
  • The PSADT has a few custom functions built in. You will see in my scripts some unfamiliar cmdlets and that is why
• I broke my script into three sections
  • Pre-Install
    
    • Backup user configuration files. If you upgrade with the new version of the installer it preserves these files but doing a clean remove/install gets rid of them. After the upgrade users will lose their saved home network connections and would be an inconvenience for them. Unfortunately using the new media to upgrade was a disaster for me, had to clean remove/install
    • Stop the VPNUI process if running, copy the "AnyConnect.html" over, restart the VPNUI process, continue (solution provided)
    • Remove each prior version if exists with a 30 second sleep between (the sleep was determined to be absolutely critical)
  • Install
    • Install each component in order
      • I check for battery status and if greater than 0 install Umbrella. I wanted this to be an all inclusive app package and deploy this out during imaging, to desktops, and to laptops - so both new instances and ugprades
  • Post-Install
    • Restore NAM profiles
    • Start the VPNUI process if not running

Code Snippets
Pre-Install\NAM BACKUP

		## Backup NAM profiles
		If (Test-Path -Path "C:\ProgramData\Cisco\Cisco AnyConnect Secure Mobility Client\Network Access Manager\system") {
			New-Folder -Path "C:\Windows\Temp\ACBackup"
			$Destination = "C:\Windows\Temp\ACBackup"
			$Source = "C:\ProgramData\Cisco\Cisco AnyConnect Secure Mobility Client\Network Access Manager\system"
				
			Get-ChildItem $Source -Recurse | ForEach {
				Copy-File -Path $_.FullName -Destination $Destination
			}
		}
		Else {
			# No backup files exist
		}

 
Pre-Install\Copy AnyConnect.html

		## Add the "AnyConnect.html" file to the help folder. I do not understand what this means at all. A Cisco employee replied to my post on their forums
		# and provided me with this solution
		# Post: https://community.cisco.com/t5/vpn/cisco-anyconnect-upgrade-failure/td-p/4268747
		# Cisco documentation: https://www.cisco.com/c/en/us/td/docs/security/vpn_client/anyconnect/anyconnect40/administration/guide/b_AnyConnect_Administrator_Guide_4-0/customize-localize-anyconnect.html#ID-1408-00000291
		# Copy AnyConnect.html only if it does not exist and process can be restarted
		$ExistingInstall = Test-Path "C:\ProgramData\Cisco\Cisco AnyConnect Secure Mobility Client\Network Access Manager\system"
		If ($ExistingInstall) {

			$HtmlFile = Test-Path "C:\ProgramData\Cisco\Cisco AnyConnect Secure Mobility Client\Help\AnyConnect.html" -ErrorAction SilentlyContinue
			If (!($HtmlFile)) {
			
				While (Get-Process -Name "VPNUI" -ErrorAction SilentlyContinue) {
					If ($CheckProcessRunning -le '10') {
						$CheckProcessRunning++
						Start-Sleep -Seconds 5
						Stop-Process -Name "VPNUI" -Force -ErrorAction SilentlyContinue
			
					}
					Else {
					Exit-Script -ExitCode 6175
					}
			
				}
			
				Copy-Item -Path "$dirSupportFiles\AnyConnect.html" -Destination "C:\ProgramData\Cisco\Cisco AnyConnect Secure Mobility Client\Help"
			
				While (!(Get-Process -Name "VPNUI" -ErrorAction SilentlyContinue)) {
					If ($CheckProcessStopped -le '10') {
						$CheckProcessStopped++
						Start-Sleep -Seconds 5
						Start-Process "${ENV:ProgramFiles(x86)}\Cisco\Cisco AnyConnect Secure Mobility Client\vpnui.exe"
			
					}
					Else {
					Exit-Script -ExitCode 6176
					}
			
				}
			}
				Else {
				# Html file already exists
				}
		}
			Else {
				# New Install
			}
		# Start a sleep to allow VPNUI to register the html file
		Start-Sleep -Seconds 30

Pre-Install\Remove prior versions

		## Removal of prior instances of each module, if exists
		# During install multiple errors were encountered. Loss of rights to ProgramData\Cisco, installers completing before they're finished, etc. 
		# https://www.cisco.com/c/en/us/td/docs/security/vpn_client/anyconnect/anyconnect49/administration/guide/b_AnyConnect_Administrator_Guide_4-9/deploy-anyconnect.html
		# Remove prior versions, in this specific order
		Remove-MSIApplications -Exact "Cisco AnyConnect Umbrella Roaming Security Module"
		Start-Sleep -Seconds 30

		Remove-MSIApplications -Exact "Cisco AnyConnect Network Access Manager"
		Start-Sleep -Seconds 30

		Remove-MSIApplications -Exact "Cisco AnyConnect Secure Mobility Client"
		Start-Sleep -Seconds 30

		Remove-MSIApplications -Exact "Cisco AnyConnect Diagnostics and Reporting Tool"
		Start-Sleep -Seconds 30

Installation

		## Cisco AnyConnect Installers
		# https://www.cisco.com/c/en/us/td/docs/security/vpn_client/anyconnect/anyconnect49/administration/guide/b_AnyConnect_Administrator_Guide_4-9/deploy-anyconnect.html
		# These installers must be done in this specific order

		# AnyConnect Core Client with VPN disabled
		Execute-MSI -Action Install -Path "$dirFiles\anyconnect-win-4.9.01095-core-vpn-predeploy-k9.msi" -Parameters "/QN /NoRestart PRE_DEPLOY_DISABLE_VPN=1"
		Start-Sleep -Seconds 30

		# AnyConnect Diagnostics and Reporting Tool
		Execute-MSI -Action Install -Path "$dirFiles\anyconnect-win-4.9.01095-dart-predeploy-k9.msi" -Parameters "/QN /NoRestart"
		Start-Sleep -Seconds 30

		# AnyConnect Network Access Manager client which will not work without a config file
		Execute-MSI -Action Install -Path "$dirFiles\anyconnect-win-4.9.01095-nam-predeploy-k9.msi" -Parameters "/QN /NoRestart"
		Start-Sleep -Seconds 30
		# Required config file
		Copy-File -Path "$dirSupportFiles\configuration.xml" -Destination "$ENV:ProgramData\Cisco\Cisco AnyConnect Secure Mobility Client\Network Access Manager\newConfigFiles"

		## Cisco Umbrella Installer
		# This should only install on laptops but does not appear to cause issues if installed on VDI/Desktop
		$Battery = (Get-WmiObject -Class Win32_Battery)
		If ($Battery.BatteryStatus -gt 0) {
			# Laptops 
			# This location should exist already, will create if not
			New-Folder -Path "$ENV:ProgramData\Cisco\Cisco AnyConnect Secure Mobility Client\Umbrella"
			# Umbrella requires a .json file in order to function
			Copy-File -Path "$dirSupportFiles\OrgInfo.json" -Destination "$ENV:ProgramData\Cisco\Cisco AnyConnect Secure Mobility Client\Umbrella"
			# AnyConnect Umbrella client
			Execute-MSI -Action Install -Path "$dirFiles\anyconnect-win-4.9.01095-umbrella-predeploy-k9.msi" -Parameters "/QN /NoRestart"
			# Umbrella apparently requires a reboot. Stopping and starting the Umbrella service
			# mitigated the need for reboot. This step was missing during initial testing
			Get-Service "acumbrellaagent" | Stop-Service -ErrorAction SilentlyContinue
			Start-Sleep -Seconds 5
			Get-Service "acumbrellaagent" | Start-Service -ErrorAction SilentlyContinue
			}

            Else {
				# Desktop, do not install
            }

Post-Install/Restoring backed up files

		## Restore NAM profiles
		If (Test-Path -Path "C:\Windows\Temp\ACBackup") {
			$Destination = "C:\ProgramData\Cisco\Cisco AnyConnect Secure Mobility Client\Network Access Manager\newConfigFiles"
			$Source = "C:\Windows\Temp\ACBackup"
				
			Get-ChildItem $Source -Recurse | ForEach {
				Copy-File -Path $_.FullName -Destination $Destination
			}
			Remove-Folder -Path "C:\Windows\Temp\ACBackup"
		}
		Else {
			# No backup files exist
		}