Cisco AnyConnect Umbrella Roaming Security Module

[dreaded190]

Hi,

So was reading up a lot on the difference between the Cisco Anyconnect Roaming Module vs the Umbrella Roaming,

Its clear that best practice is the Cisco anyconnect option as UR is EOL in a few months, so we recently deployed the new client company wide and that resolved some VPN resolving inconsistencies we had.

The problem, we have two computers that are giving me mixed signals and maybe you guys can help me understand the behind the scenes better.

So the AnyConnect client is supposed to handle DNS handover to the software at a Kernel level right, so according to the documentation the DNS is not changed at a system level, like if you do ipconfig /all you wont see 127.0.0.1 by DNS right?

We have two computers that i have been troubleshooting, they both get the loopback assigned at the system level but only to the wireless interface, the Ruckus wireless is on the same network/DNS Suffix as the ethernet. 

Investigating with a Netstat i can see 127.0.0.1:53 and a process ID pointing to dnscrypt that exists under the Anyconnect install folder??

So i thought its not supposed to use system DNS? Keep in mind i have uninstalled, deleted all the files associated confirmed and reinstalled, works for a while and then one day boom system DNS on Wireless interface 127.0.0.1??

This is a interesting one, anyone got some ideas or can explain what im missing?

Thanks

[kc-1]

Hi @dreaded190 

Did you ever find a solution to this issue? We just moved from Umbrella Roaming Client to Cisco Secure Client with Umbrella and are experiencing the exact same issue that you described. In all cases, the wireless nic randomly reverts to using 127.0.0.1 as the preferred DNS server. We can't figure out what's causing this to happen.

[MHM Cisco World]

Make new post 

It is better 

MHM

[Pulkit Mittal]

The computer is configured to send all DNS traffic through the Roaming Client on 127.0.0.1:53.  However, the Roaming Client remembers the list of DNS servers that you assign to the network interface and uses them for handling DNS traffic:

• If a DNS query is for an Internal Domain it is sent to the saved DNS servers from the network interface
•  All other DNS queries are sent directly to Umbrella DNS resolvers in the cloud

Here are links with further explanation.

How to Configure AnyConnect (with Umbrella) to Work with Twingate | Docs | Twingate 

Change Static IP/DNS settings with the Umbrella Roaming Client installed – Cisco Umbrella

If you find this useful, please mark it helpful and accept the solution.

[kc-1]

Hi @Pulkit Mittal Thanks for the response. However, we're having issues with Cisco Secure Client (AnyConnect) with the Umbrella module, not Umbrella Roaming Client which is EOL soon. From what I read, the Umbrella module with Cisco Secure Client doesn't need to set the loopback address on the NICs anymore. Instead, it handles all of that at the kernel level. I'm trying to figure out why the wireless NIC is randomly reverting to 127.0.0.1 as the preferred DNS server.

[Pulkit Mittal]

That is correct. I suggest reaching out to umbrella support to check if legacy temp data config of the roaming client is affecting the secure client umbrella module.

 What are the architectural differences between the clients?

Umbrella Roaming Client approach

The Umbrella Roaming Client uses a loopback adapter in order to inspect all requests sent to the  DNS servers specified in a computer's network adapters DNS settings. This requires that the roaming client resets the DNS server on all adapters to use 127.0.0.1, the localhost loopback address. 

The disadvantage of this approach is that some VPN clients will conflict with this configuration--either by mandating that the configuration matches what has been set by the administrator or by preventing a DNS resolver from running on 127.0.0.1. 

Alternatively, some VPN clients also overwrite an adapter's DNS settings with VPN values, overwriting the roaming client's DNS server address of 127.0.0.1 instead of the original values. This can cause the Umbrella Roaming Client and the conflicting software package to not function as designed, or cause a full DNS fail scenario where the configured DNS settings are lost at connect or disconnect.

Cisco Secure Client approach

With Cisco Secure Client, Umbrella is a module that can be installed. This module is able to control the adapter without changing the DNS settings on the interface, avoiding DNS change conflicts. Cisco Secure Client uses a kernel driver, which intercepts the DNS requests at a much lower level in the operating system. This more sophisticated mechanism has the advantage of not requiring that traffic for all adapters go through the loopback address, so the original DNS settings are maintained. This architectural difference means that the Umbrella module can retain much higher compatibility with other software when compared to the Umbrella Roaming Client.

If you find this useful, please mark it helpful and accept the solution.

[Copy_Pasta]

We have also rolled out Cisco Secure Client Umbrella Module to replace the EOL Umbrella Roaming Client, across our org this month, and witnessing this same problem.

Randomly the Wi-Fi adaptors on our estate of Windows 10 (22H2) PC's are being set to a Static value of 127.0.0.1 - very much the opposite of the explained behaviour in the blog (kernel driver). The old roaming client looks to be cleanly uninstalled / removed, so we can only conclude this behaviour is from the new Secure Client.

For us, when this occurs, all DNS is broken - the 'loopback' is not passing traffic, so we end up with a faulty endpoint. Only resolution is for IT to set the adaptor back to 'Automatic', which will restore access ... for a time ... We are seeing repeat incidents , ranging from a couple of hours to days, but it is worryingly ongoing and very disruptive.

Glad we found this post (going mad like the OP), but concerned there is no update or fix...? Anyone found a workaround or version that does not exhibit this behaviour? Has anyone had any offical response from Cisco support?

[kc-1]

Hi @Copy_Pasta I feel your pain! I do have a TAC case open with Cisco Support and we continue to troubleshoot the issue with them, but no solution has been identified. Out of curiosity, what version of Umbrella Roaming Client were your machines running when you moved to Cisco Secure Client? Our machines were running version 3.0.466 on Windows 10 22H2.

[Copy_Pasta]

@kc-1 : We were also on 3.0.466 on Win10 22H2 prior to upgrade to Secure Client version 5.1.2.42. We upgraded approximately 14,000 Windows PC's - and we now see (via an Intune proactive-remediation 'detection' check) this problem surface on approximately 150 machines Wi-Fi adaptors at any given time. This is transient and which machines 'repeat' this behaviour, and how quick is really random.

[kc-1]

@Copy_Pasta I can vouch for the randomness as well. If you can, I would open a TAC case and push to have it escalated. Go ahead and reference this thread as well. The more people we have report this issue, the better. I'll report back if there's any solid progress with my TAC case.

[aggelos-kanarelis]

Would it be possible to share the scripts you used? That could prove very useful to us, since we are facing the same issues and unfortunately my powershell is limited to one liners.

thanks

[Copy_Pasta]

My team fixed this via PowerShell, deployed via MS Intune (our Windows MDM) using the Proactive-Remediation feature. For those not familiar with this, its 2 scripts to 'detect' for the presence of a problem, and where found 'remediate' the issue (then re-check/detect).

As a result, this is a 2 scripter - attached herein as TXT files. You should be able to re-use this in some form. Its somewhat crude, but basically we took 2 approaches to resetting the 'bad' DNS values - via both the netsh command and iterating the registry values. We then restart the service, and between all this it does now finally 'stick' and the issue is now SOLVED. Hope this helps people.

To grumble: A very irritating issue all-in-all, and less than impressed to find the retrospective advice (added to select KB pages belatedly) that there is a bug in the migration process, that requires you (the customer) to manually stop the legacy service prior to the MSI uninstalls/installs (which any self-respecting installer will normally do). Why was/is this not fixed in the installers by Cisco? Why hasnt a fix been put out for a major and very-disruptive issue (clients no connectivity!) to clean this up in a patch/hotfix? Why wasnt migration advice made clearer on all the documentation pages? Very unimpressed... 

[aggelos-kanarelis]

thank you copy_pasta. When we were investigating the migration the KBs were lacking this info!!! Will have a look at the scripts and ammend them to fit our purpose. Meanwhile I've been working on a script of my own and if it works I'll upload to help others. Thanks again.

[aggelos-kanarelis]

As promised here's the detection script we created. Currently working on the fix script.

[mcdanelmatthew@lcsnet.com]

We are also experiencing this issue.
I have random computers that are reverting back to the 127.0.0.1 DNS IP Address on the Wireless Interfaces. 
We run the following commands to determine if the 127.0.0.1 address is applied to any of the interfaces:
netsh ip interface show dnsservers

Then we run the following command:
netsh interface ip set dns "Wi-Fi" dhcp