AnyConnect Posture Scan stuck at 10% for a long time

[jerryblack143]

Hello Everyone,

We are running into an issue whereby AnyConnect Posture/system scan gets stuck at 10% for select users. We have been working this with TAC for almost 4 months now with now success and the number of affected users seems to be growing each day.

Here are some details:

- Devices connecting through VPN are profiled for compliance before being allowed access.

- We do registry checks, applications check and processes check

- When we implemented posturing about 6 months ago, only one user was having this issue but about a two months ago, more and more users began having the issue.

- Sometimes, the scan takes up to an hour and during this time the user could not access the network.

- One thing that is common among impacted machines is that they are all running Windows Firewall version 10.0.19xxxx. Machines running Windows Firewall version 10.0.18xxxx dont seem to have this issue.

- There are some situations that if Windows firewall is temporarily disabled, the scan is quick. Also, in some situations, if the user switches to a different network (eg. hotspot) the issue goes away.

AnyConnect version 4.90xxxx

Compliance Module 4.3.xxxxx

Please any help will be appreciated as this is creating productivity issue.

Thanks

[Mike.Cifelli]

What has TAC suggested thus far if they have been engaged for 4 months? Can you share a DART bundle with us? What specific checks do you perform?  There could potentially be other options to possibly decrease wait time.  

- When we implemented posturing about 6 months ago, only one user was having this issue but about a two months ago, more and more users began having the issue.

Has there been any other client or network changes that you can think of that may have created additional issues?

- There are some situations that if Windows firewall is temporarily disabled, the scan is quick. Also, in some situations, if the user switches to a different network (eg. hotspot) the issue goes away.

Any bandwidth limitations between let's say a wired client versus a wireless client?

- Sometimes, the scan takes up to an hour and during this time the user could not access the network.

Probably not advised or best option, but have you considered allowing certain access via dacl when a client is pending posture assessment (status unknown)?

[jerryblack143]

Hi Mike,

Thanks for the response.

TAC is leaning towards an AM/AV on the systems being the culprit. However, we are not seeing any detection on the EDR for those endpoints impacted. Plus if its the EDR causing this issue, we should have been seeing it across the board but that is not the case.

We did whitelist the Cisco folders for select endpoints as advised by TAC but that did not help.

As i mentioned earlier, the only significant difference between impacted endpoints and non-impacted endpoints is the Windows Firewall versions.

There is one of the users that I asked her when the issue began and the date that she mentioned corresponds to the date Windows Firewall update occurred.

Regarding Bandwidth limitations, there is none that i am aware of and some of the users are connecting through LAN while others are connecting through WLAN.

The only change in our environment that I can think of is is Windows update.

Thanks

[Marcelo Morais]

Hi @jerryblack143 ,

 could you please share the Cisco AnyConnect ISE Posture Module - Event Viewer?

Best regards

[MHM Cisco World]

Hi

i am new in ISE 

but 

you mention that there is profiler in ISE ? 

If yes which probe you use ? 

I think the probe make slow issue 

[jerryblack143]

Hi MHM,

I am not sure that i understand your comment correctly. Can you clarify?

Thanks

[MHM Cisco World]

OK,
ISE profiler use probes like:-
DHCP
DNS 

NMAP

to detect :-
IP 
MAC 
OS 
FW version 

OS and FW version detect via NMAP, so can you start to check this point.

[Panos Bouras]

Hi @jerryblack143 

Have you tried to perform a packet capture from the affected PC in order to see if traffic to ISE is being send from PC during that time?

Also, and apologies for the obvious, have you tried disabling Windows firewall just for testing as to verify your theory? Or at least create an allow rule towards ISE IP range? Does your client responds timely then?

Also, can you alter your posture policy performing one check at a time, as to use a process of elimination on what check causes your delay?

[jerryblack143 · Accepted solution]

Just to update on this one.

TAC engaged Opswat team who combed through many logs that we collected and found where the delay was occurring. This delay was occurring when executing the Opswat dll’s (libwalocal.dll and libwaremoval.dll). These DLLs are opswat executable files and Opswat uses WMI queries/API calls on OS front to get the information about all the installed products on endpoint.

With this information provided by TAC/Opswat we spent time troubleshooting more on our end and narrowed this delay down to Windows as the culprit. It seems like Windows recently changed some file explorer behavior which causes mapped network drives to attempt to access the network drive once the user logs into Windows. This attempt by Windows to connect to the network drive holds up the resources that the Opswat DLL needs to complete gathering the device information. File explorer eventually releases these resources but it takes 10+ mins and sometimes up to 40 Mins.

What we did as a quick fix was to add a registry key that forces file explorer to wait for a full network connection before attempting to connect to network locations or rather wait for the user to manually attempt to connect to the drive before making the connection attempt. One drawback with this fix is that the network drives will show up with the red X beside them as if they are not reachable even after the device is fully connected to the network. This red X goes away when the user is fully connected and then clicks on the folder.

Below is the registry key that we added which seems like a fix for now but we are still monitoring to make sure the delay does not come back.

Thanks

Reg Key:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\NetworkProvider\ Set or create a DWORD RestoreConnection = 0

[MHM Cisco World]

I so happy for you solve issue, 
and as I suggest the issue from the ISE identify information collection. 
Good Job man

[Damien Miller]

Happy to hear you sorted it out. I have a customer with similar slow posture issues too, but this doesn't really seem like a great fix as far as user experience goes. Past projects have taught me that users will see this red X and end up calling the helpdesk. 

[jerryblack143]

Hi Damien,

I agree with your observation which we did consider but this was a better option verses having users wait for 20/30 mins and sometimes up to 45 mins.

We worked closely with TAC including some guys from the AnyConnect development team and Opswart during which we shared as much information as we could, so I am hoping that they will look into this to see if they have a way to address this issue from the AnyConnect perspective.

We did reach out to Microsoft as well, but as expected, didn't get much traction with that.

We are also looking internally to see if there is any kind of post action that we can trigger after posture checks are complete to reconnect the drives without user action.

Thanks