Symptom:VMware Tools upgrade fails due to various Selinux denials under various scenarios. VI-Client indicates tools status as Not running, Not Installed.
The following selinux denial is seen in System Logs (messages) when VMtools update attempt fails either via VI-client initiated automatic update or Automatic Update that takes place during boot up as long as VM Setting "Check and upgrade VMware Tools before each power on" is checked.
Feb 25 20:20:18 cucm-pub user 3 setroubleshoot: SELinux is preventing /usr/bin/perl from create access on the directory /var/lib/. For complete SELinux messages. run sealert -l 84003ecc-5de4-4e59-9ab8-1e7a28225c18
The following selinux denials is seen in System Logs (messages) when Vmtools update to 10.0 version or above is successful after putting System OS Security to Permissive mode followed by Update of Tools and then putting System OS Security back to Enforcing mode.
Feb 22 16:34:23 cucm-pub user 3 setroubleshoot: SELinux is preventing /usr/lib/vmware-caf/pme/bin/ManagementAgentHost from read access on the directory requests. For complete SELinux messages. run sealert -l 76069c58-d7be-482f-8391-4eb94d51ecd9
Feb 22 16:34:23 cucm-pub user 3 setroubleshoot: SELinux is preventing /usr/lib/vmware-caf/pme/bin/ManagementAgentHost from read access on the directory requests. For complete SELinux messages. run sealert -l 76069c58-d7be-482f-8391-4eb94d51ecd9
Feb 22 16:34:24 cucm-pub user 3 setroubleshoot: SELinux is preventing CThreadUtils::s from write access on the directory output. For complete SELinux messages. run sealert -l 9e71ec6f-cd83-43a5-8564-14f66e77e4ff
Feb 22 16:34:24 cucm-pub user 3 setroubleshoot: SELinux is preventing /usr/lib/vmware-caf/pme/bin/ManagementAgentHost from read access on the directory providerReg. For complete SELinux messages. run sealert -l 76069c58-d7be-482f-8391-4eb94d51ecd9
Feb 22 16:34:25 cucm-pub user 3 setroubleshoot: SELinux is preventing CThreadUtils::s from write access on the directory output. For complete SELinux messages. run sealert -l 9e71ec6f-cd83-43a5-8564-14f66e77e4ff
Under these conditions where VMtools 10.0 is running with CUCM 10.X or 11.X, Putting OS Security mode back to enforcing will inevitably lead to:
1. All available virtual memory is consumed by settroubleshootd because of continuous selinux denials
2. vmware-caf logs consume 100% of the active partition due to selinux denying log rotation to log file (/usr/lib/vmware-caf/pme/bin/ma-log4cpp_rolling.log)
Conditions:Problem is seen after Upgrading to latest builds of ESXi 5.5 or 6.0 builds greater than 3248547 which bundles 10240+ (10.0.0+) version of VMware Tools and brings in a new vmware-caf functionality.
The same condition will occur where selinux denials are logged preventing vmware-caf operations after a Fresh install of CUCM 10.X or 11.0 on top of ESXi 5.5 or 6.0 builds that bundles 10.X version of VMware Tools.
When CUCM 10.X or 11.X OVA is used the VM Setting named VMware Tools => "Check and update Tools during power cycling" will be enabled by default. This setting being enabled during a Fresh Installation operation allows VMware Tools upgrade to 10.0+ however post installation when selinux is put back in to enforcing mode the denials will start and you will face same running out of virtual memory as well as vmware-caf logs consuming 100% of the active root partition.
Workaround:Resolution to this issue is also available via a standalone COP file
ciscocm.VMwareTools2016a.cop.sgn posted to CCO Software Downloads in the Unified Communications Manager / CallManager / Cisco Unity Connection Utilities sections for CUCM 10.5 & 11.0
For ESXi Update / Patch Scenarios Apply this COP file prior to restarting CUCM applications where automatic update could take place during power up or prior to initiating automatic vmware tools update from vi-clients.
This cop file will update the Selinux Policy files so that VMWare tools upgrade completes without failing under various tools update scenarios as well as allow the VMware tools version 10.0+ to operate without causing excessive memory utilization and filling up the active partition with logs.
If you are recovering from a running out of active disk partition situation; Additionally, the COP file enables the CLI command "utils vmtools caf-logs delete" which can be used to delete CAF
logs which take up excessive disk space.
Fresh Install WorkaroundIf you must Fresh Install CUCM/CUC/IM&P/CER/UCCX 10.X or 11.X version that does NOT have this fix present on top of ESXi 5.5 or 6.0 which already has VMware Tools 10.X version bundled, as a workaround you may follow these steps:
- After deploying the VM using the respective OVA configuration
- Edit VM configuration
- Uncheck the option for VMware Tools => "Check and upgrade Tools during power cycling" <== This is checked by default on CUCM 10+ OVA, Unchecked on CUCM 8.X, 9.X OVAs
- Proceed with power on the VM and Fresh installation
- Post Fresh Install the COP File
- Edit VM configuration
- Re-Enable the option for VMware Tools => "Check and upgrade Tools during power cycling"
- Reboot the VM
Further Problem Description:Put OS Security back to enforcing mode only if you are absolutely sure that you are Updating VMware Tools to a version below 10.0. For reference look at this VMware tools version mapping doc to correlate your ESXi Host builds to bundled vmtools versions.
https://packages.vmware.com/tools/versions
