A study on how Cisco Umbrella Roaming Client works

Andre Camillo · ‎01-29-2019

A study on how Cisco Umbrella Roaming Client works

./intro/

There’s no better teacher than necessity, I’ve taken the time to research how Cisco’s Umbrella Roaming Client works.

The findings and information below is what I have found through reading and verification shown below, no confidential information is shared.

./intro/subject

If you do not know that already, Cisco Umbrella is pretty awesome!

The 100% Cloud-based DNS resolver from Cisco leverages:

Machine learning,

Cloud dashboard,

Custom integrations with third party solutions,

Advanced Malware Protection (Cisco AMP) integration for real-time file analysis, and even

Cloud-based Proxy for dodgy SSL-based websites,

Both on-network and off-network solutions for multiple Operating Systems.

All of this to make sure your DNS responds to the policies (and Safe Domains) you want it to.

Not to mention the many other additions and enhancements it has had the past year.

./intro/motivation

But often when I’m in the spotlight, presenting about the wonders of Cisco Umbrella and its On/Off-network capabilities, I’m asked:

“How does the Roaming Client work?”

You see, a core component of Off-network security for Cisco Umbrella is lightweight connector deployed as either a dedicated connector (Called “Roaming Client”, also referred to as “RC” from now on) or as a module within Anyconnect (referred to as “Roaming security module”) — and this is where all the off-network protection and content security enforcement magic happens.

This brief article aims to respond this question.

./answer/methodology

In order to answer this one question, then, we need to understand:

What does the client do in the OS when working, and how it affects the user’s OS decision to DNS resolution
How the client operates while within a Umbrella Protected Network (On-network)
How the client operates while Off-network (meaning at a guest, unprotected by Umbrella network)

Figure 2: RC Off the VPN — Modified version from Source

./answer/RoamingClient

Let’s get into it then, I want to go through how RC works.

Once installed in its host (Mac/Win) the client, binds itself to all NICs by:

Replacing their old DNS values to 127.0.0.1;
Changes the device’s DNS settings to Localhost (127.0.0.1), and;
Stores old DNS configurations for each NIC in text files
Executable then listens to Localhost:53 at all times
Creates a PPP Adapter
Downloads the “Internal Domains” list from the cloud

I detail the reasons why each of these actions are executed below:

Binds itself to all NICs: This is crucial as to not be bypassed while off-network.

Replacing their old DNS values to 127.0.0.1, example from my windows host :

Stores old DNS configurations for each NIC in text files

This can be verified in RC’s installation path: “C:\ProgramData\OpenDNS\ERC”

For each working adapter, RC creates a different Reference file.

See below, for my Wi-fi adapter:

Figure 4: RC’s Archive of old DNS setting, before RC installation

Executable then listens to Localhost:53 at all times

With all that, the Localhost is now answering for DNS resolution, meaning all DNS queries will be handled by the OS.

Ultimately, this is what allows the RC’s process to intercept DNS queries and respond to them.

Once the Client is installed, its sole job is to actively listen to port 53 (DNS) of the TCP/IP stack on the host while allowing resolution of local domains through the Internal Domains feature.

We can verify that the RC process is doing so (in Windows) with the command : “netstat -a -b” that lists all processes listening to TCP/IP ports in the host. See the outputs below:

Figure 5: RC process listening to Localhost IPv4, port 53

Figure 6: RC process listening to Localhost IPv6, port 53

Creates a PPP Adapter

This can be verified when RC is operational (off-network), by issuing the command : “ipconfig”

The PPP adapter is responsible for sending encrypted DNS queries to Umbrella Servers for resolution.

Figure 7: PPP adapter that RC creates when working

Downloads the “Internal Domains” list from the cloud

Also verified in RC’s installation path: “C:\ProgramData\OpenDNS\ERC”

These are created with default values if nothing is modified in the Cloud Dashboard.

Figure 8 shows the potential whitelist files downloaded:

Figure 8: whitelists files for internal Domains

Let’s compare how a roaming device with Roaming client installed acts when resolving the similar website in 2 different scenarios:

How the client operates while within a Umbrella Protected Network (On-network)
How the client operates while Off-network (meaning at a guest, unprotected by Umbrella network)

./answer/scenario1 (IN-Net/VPN):

“Upon establishing a connection to a VPN server, the Umbrella roaming client detects a new network connection in the system and changes the connection’s DNS settings to point toward the Umbrella roaming client. The Umbrella roaming client relies on being able to perform DNS lookups to Umbrella’s AnyCast DNS IP addresses (208.67.222.222/208.67.220.220).

If you are connecting to a VPN, the firewall associated with the VPN should allow access to Umbrella.”

From this document

You must notice how my NICs have had their DNS settings changed to what they were when RC was not working (using the old DNS info stored in the text files, mentioned above).

Let’s first identify what active settings we have in the NICs, with an IPCONFIG /ALL command (highlighted in Bold relevant information):

Windows IP Configuration
Host Name . . . . . . . . . . . . : <<omitted>>
Primary Dns Suffix . . . . . . . : <<DNS Suffix X>>

Node Type . . . . . . . . . . . . : Hybrid
IP Routing Enabled. . . . . . . . : No
WINS Proxy Enabled. . . . . . . . : No
DNS Suffix Search List. . . . . . : <<DNS Suffix X>>
<<DNS Suffix Y>>
<<DNS Suffix>>

Ethernet adapter Local Area Connection* 1:
Connection-specific DNS Suffix . : <<DNS Suffix X>>
Description . . . . . . . . . . . : <<VPN Provider>>
Physical Address. . . . . . . . . : <<VPN Provider’s MAC>>
DHCP Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes
IPv4 Address. . . . . . . . . . . : <<IPv4>>(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.255.192
Lease Obtained. . . . . . . . . . : Friday, 25 January 2019 7:59:01 AM
Lease Expires . . . . . . . . . . : Friday, 1 February 2019 7:58:59 AM
Default Gateway . . . . . . . . . : 0.0.0.0
DHCP Server . . . . . . . . . . . : <<VPN provided Internal IPv4>>

DNS Servers . . . . . . . . . . . : <<VPN provided Internal IPv4 1>>

<<VPN provided Internal IPv4 2>>

NetBIOS over Tcpip. . . . . . . . : Enabled

Wireless LAN adapter Wi-Fi:
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Intel(R) Dual Band Wireless-AC 8265
Physical Address. . . . . . . . . : D4–6D-6D-3B-EC-E1
DHCP Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes
Link-local IPv6 Address . . . . . : fe80::4c82:1ee7:21b:dc1e%12(Preferred)
IPv4 Address. . . . . . . . . . . : 192.168.100.11(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Lease Obtained. . . . . . . . . . : Friday, 25 January 2019 7:35:55 AM
Lease Expires . . . . . . . . . . : Saturday, 26 January 2019 7:35:53 AM
Default Gateway . . . . . . . . . :
DHCP Server . . . . . . . . . . . : 192.168.100.1
DHCPv6 IAID . . . . . . . . . . . : 81030509
DHCPv6 Client DUID. . . . . . . . : 00–01–00–01–23–7B-B0-BD-8C-16–45–3E-DF-9F
DNS Servers . . . . . . . . . . . : 192.168.100.1
NetBIOS over Tcpip. . . . . . . . : Enabled

*Notice there’s no PPP Adapter with Umbrella Connection

Then, let’s confirm all interfaces we have operational while the device is connected to a VPN, issuing command NETSTAT -RN:

======================================== Interface List
2… <<VPN Provider’s MAC>> …… <<VPN Provider>>
7…8c 16 45 3e df 9f ……Intel(R) Ethernet Connection (4) I219-V
14…54 8c 58 ab 08 01 ……Check Point Virtual Network Adapter For Endpoint VPN Client
17…d4 6d 6d 3b ec e2 ……Microsoft Wi-Fi Direct Virtual Adapter
11…d6 6d 6d 3b ec e1 ……Microsoft Wi-Fi Direct Virtual Adapter #2
5…00 50 56 c0 00 01 ……VMware Virtual Ethernet Adapter for VMnet1
19…00 50 56 c0 00 08 ……VMware Virtual Ethernet Adapter for VMnet8
12…d4 6d 6d 3b ec e1 ……Intel(R) Dual Band Wireless-AC 8265
1………………………Software Loopback Interface 1
===========================================================================

IPv4 Route Table
======================================= Active Routes:
Network Destination Netmask Gateway Interface Metric
0.0.0.0 0.0.0.0 On-link <<VPN IPv4>> 1

//omitted

These are the NSLOOKUP w/ default DNS resolution results to the website from the Roaming Client’s Host, confirming we’re using the VPN’s DNS servers:

Results for: nslookup.exe uol.com.br.
stdout:
Server: UnKnown
Address: <<VPN Internal IPv4>>

Name: uol.com.br
Addresses: 2804:49c:3101:401:ffff:ffff:ffff:45
2804:49c:3102:401:ffff:ffff:ffff:36
200.221.2.45

stderr:
Non-authoritative answer:

./answer/scenario2 (OFF-Net):

When working Off-network, meaning not in a VPN tunnel and over a guest network without Umbrella’s protection, the client’s rules kick in and start working.

Let’s first identify what we have in the Network interface card, with an IPCONFIG /ALL command:

Windows IP Configuration

Host Name . . . . . . . . . . . . : <<omitted>>
Primary Dns Suffix . . . . . . . : <<DNS Suffix X>>

Node Type . . . . . . . . . . . . : Hybrid
IP Routing Enabled. . . . . . . . : No
WINS Proxy Enabled. . . . . . . . : No
DNS Suffix Search List. . . . . . : <<DNS Suffix X>>
<<DNS Suffix Y>>
<<DNS Suffix>>

Wireless LAN adapter Wi-Fi:
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Intel(R) Dual Band Wireless-AC 8265
Physical Address. . . . . . . . . : D4–6D-6D-3B-EC-E1
DHCP Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes
Link-local IPv6 Address . . . . . : fe80::4c82:1ee7:21b:dc1e%12(Preferred)
IPv4 Address. . . . . . . . . . . : 192.168.100.11(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Lease Obtained. . . . . . . . . . : Friday, 25 January 2019 7:35:56 AM
Lease Expires . . . . . . . . . . : Saturday, 26 January 2019 7:35:55 AM
Default Gateway . . . . . . . . . : 192.168.100.1
DHCP Server . . . . . . . . . . . : 192.168.100.1
DHCPv6 IAID . . . . . . . . . . . : 81030509
DHCPv6 Client DUID. . . . . . . . : 00–01–00–01–23–7B-B0-BD-8C-16–45–3E-DF-9F
DNS Servers . . . . . . . . . . . : 127.0.0.1
NetBIOS over Tcpip. . . . . . . . : Enabled

PPP adapter Umbrella:
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Umbrella
Physical Address. . . . . . . . . :
DHCP Enabled. . . . . . . . . . . : No
Autoconfiguration Enabled . . . . : Yes
IPv4 Address. . . . . . . . . . . : 100.122.137.89(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.255.255
Default Gateway . . . . . . . . . :
NetBIOS over Tcpip. . . . . . . . : Disabled

Running NETSTAT -RN:

======================================== Interface List
2… <<VPN Provider’s MAC>> …… <<VPN Provider>>
7…8c 16 45 3e df 9f ……Intel(R) Ethernet Connection (4) I219-V
14…54 8c 58 ab 08 01 ……Check Point Virtual Network Adapter For Endpoint VPN Client
17…d4 6d 6d 3b ec e2 ……Microsoft Wi-Fi Direct Virtual Adapter
11…d6 6d 6d 3b ec e1 ……Microsoft Wi-Fi Direct Virtual Adapter #2
5…00 50 56 c0 00 01 ……VMware Virtual Ethernet Adapter for VMnet1
19…00 50 56 c0 00 08 ……VMware Virtual Ethernet Adapter for VMnet8
12…d4 6d 6d 3b ec e1 ……Intel(R) Dual Band Wireless-AC 8265
1………………………Software Loopback Interface 1
39………………………Umbrella
======================================= IPv4 Route Table
======================================= Active Routes:
Network Destination Netmask Gateway Interface Metric
0.0.0.0 0.0.0.0 192.168.100.1 192.168.100.11 40

These are the NSLOOKUP with default DNS resolution results to the website from the Roaming Client’s Host:

Results for: nslookup.exe uol.com.br.
stdout:
Server: localhost
Address: 127.0.0.1
Name: uol.com.br
Addresses: 2804:49c:3101:401:ffff:ffff:ffff:45
2804:49c:3102:401:ffff:ffff:ffff:36
200.147.3.157

stderr:
Non-authoritative answer:

*Exceptions:

Using RC under a different network, that is also protected by Umbrella (visiting a corporate client, for example).

In most cases when an Umbrella roaming client visits another organization or a guest protected network, the roaming client is set to remain active and will honour the policies of its home organization. However there are caveats, if the roaming client is part of its home org default policy(with lower priority or policy number 0), the roaming client will then use the policy of the guest organization due to it being in a higher policy.

Taken from here

2. Using RC in a network with a VA: When connected to a network with a Virtual Appliance (VA) , roaming client will always be overriden by it, disabling itself while connected to such network (Locally or via VPN).

If VAs are deployed at one or multiple locations within your organization, the Umbrella roaming client will disable itself and DNS settings will revert to the VA while connected to the location physically or via VPN. For more information about Umbrella roaming client behavior with VAs, read: Appendix B — Virtual Appliances and the Umbrella Roaming Client.

If all of your organization’s locations are using VAs, configuring the Internal Domains list for appliances only (not appliances and devices) will suffice; any time the Umbrella roaming client is not connected to the network in question, it won’t use Internal Domains set to “Appliances Only”, and treat any DNS queries to your domain as a public query (encrypted).

From this article

Caveats and further behaviour:

There may be VPN client’s and Roaming Client incompatibilities, refer to documentation for details.
To work under a compatible VPN, the RC must be able to perform a DNS lookup to Umbrella’s anycast addresses (208.67.222.222 / 208.67.220220)

./conclusion

The study was found to be conclusive in the sense that it could be verified how the client works in a Windows host.

I like the idea of this (benign and approved) process taking over all NICs’ DNS settings and constantly listening to DNS port from the TCP/IP stack. This ensures every domain resolution is going to go through the client, without creating overhead and mishandling of the OS’s “hosts” file.

From an application perspective this approach raises concern on how often we’re reviewing our clients’ DNS settings, as a malicious actor could imitate a similar behaviour to hijack some DNS responses to potential malicous servers.

This is why we must always confirm what authority is signing our pages’ Digital certificates.

./sources

Alex Pfeil · ‎02-20-2019

Thanks for posting how Umbrella works. Very informative!

dreaded190 · ‎05-22-2023

Verry informative thank you,

The AnyConnect client apparently works more at a Kernel level and does not change the "system" DNS

Have an issue at one of my clients,

If you interested in a brain teaser

https://community.cisco.com/t5/web-security/cisco-anyconnect-umbrella-roaming-security-module/td-p/4838674

Verifying how Umbrella Roaming Client works

A study on how Cisco Umbrella Roaming Client works

./intro/

./intro/subject

./intro/motivation

./answer/methodology

./answer/RoamingClient

./answer/scenario1 (IN-Net/VPN):

./answer/scenario2 (OFF-Net):

./conclusion

./sources

AnyConnect Certificate Based Authentication.

Getting past intermittent/unexplained 802.1x problems on Windows 7

Insights About Multiple Vulnerabilities in Cisco Discovery Protocol Implementations (CDPwn)