Excessive reverse DNS queries with Anyconnect Umbrella client

[bjames]

Hi,

We have a client the is experiencing excessive reverse DNS queries when running the Cisco Secure Client with only the Umbrella client loaded when they are VPN'd to Citrix.

We tested the older roaming client and do not have the same issues, we have a ticket open with Cisco and they explained the difference between the two clients but this still doesn't help us isolate which application is causing these and only when the Umbrella/Anyconnet client is loaded.

The Laptops are Dell Latitudes and we have troubleshot thisfor a while so I thought I would ask if anyone else has this exact experience. The next step is to create a new Laptop with a base image, the citrix and anyconnect clients and start adding apps until we see the issue. Cisco believes it is an app issue.

This is preventing a large scale roll-out of the client so any help would be appreciated.

Thanks

[ccieexpert]

i have not observed it... what do you mean excessive ? what percentage ? did you take packet capture (wireshark/sniffer) to see the dns traffic and see what are the excessive reverse dns.. hopefully you can figure what  it is and it might tell what application could be generating it...based on the query.

your testing with app by app is good.. i would also isolate using wireshark to see what are the requests... i not sure why adding umbrella client would make a difference, as Dns requests are made by a application. That is why your testing would be great and also packet captures.

What was the cisco response ? please paste it here so i can look at it...

[bjames]

Excessive as in 250MB worth in an hour or two. With the older client it would use less than that in two weeks. Lot's of Wireshark traces but that does not tell you which app it came from. Cisco is suggesting the same app isolation as they believe it is always doing it, but when the client is loaded it takes these requests over.

Cisco:

So the old client "intercepted" DNS queries by manipulating the DNS settings on any active NICs. What it would do is change the DNS servers assigned by DHCP to the loopback adapter. With the new client, we use a kernel driver to detect and intercept DNS queries instead of manipulating NIC settings. We made this change because it's more reliable and far more compatible with 3rd party VPNs.

Now, my hypothesis is that there is some program on the computer that is creating some sort of reverse lookups that are expected to be handled locally by the machine and never sent over the network. Because they were never put "on the wire" by the application it's why the old client never saw them. However, since our new client sees them being created at the kernel level, we are intercepting them and then trying to resolve them as if they were any other DNS query.

[ccieexpert]

It is possible , but generally dns queries are send to a server based on the NIC settings...

look at this thread where it shows how to detect dns queries by application. i think something like this may reveal the app..

https://www.reddit.com/r/pihole/comments/gytfbm/guide_how_to_discover_which_app_is_making_dns/?rdt=56541

the other way is to get lucky... for example app1 does reverse dns query for server1. app2 does reverse dns query for server2, app3 - server3... generally each app is querying a totally different domain.. so perhaps if you look at the wirehsark and sort on heavy hitters domains and then see which domains they are that might give some clue.. good luck..keep us posted on what you find.

**Please rate as helpful if this was useful**

[bjames]

The largest rDNS queries are for Microsoft lookups.

I'm suspecting a combination of things as we do not have this issue internally or with other clients.

[ndoud]

We saw issues this morning as well~8-830am central time.

[bjames]

We think we have isolated it a little more; if we disable ALL startup apps on the PC and try again we do not see the massive traffic. We are trying each app, as well as looking at more detailed application analysis to prove which on it is.

Right now it's looking like OneDrive, but we need more testing to verify.

I'm assuming it's a unique combination of Citrix Client, an app, and Umbrella/Cisco Secure Client.

I will post when I have more info.

[ccieexpert]

great once you identity the app, then look at traffic with and without secure client/umbrella..  Umbrella DNS *should not* generate any DNS traffic on its own... It could be other reasons either failed lookups etc that could be causing more traffic.. that is why it is important you dive deeper and look at packet captures before and after..

[bjames]

Without the Secure client everything works fine, so we know the issue is the with the client probably in concert with the Citrix client and the app. There was an updated Secure client today so we are testing that now. Umbrella is not generating anything, now that it's kernel based it's picking up traffic that would not normally go over the network interfaces (as referenced when we test with the old Roaming client which works fine)

[bjames]

OK tested the new client but same results (excessive UDP packets), we also disabled all startup items but still have the issue. We are now going to setup a fresh build laptop and try with it, with only the standard apps on it.

[bjames]

Fresh PC with only M365 and the Citrix and Umbrella client exhibit the same issues. The client my go with the old Roaming Client SW in the mean time to get the project going knowing there won't be any updates to it and they will have to move to Anyconnect when fixed. Provided Cisco with the Netstat -anode as per their request.