AnyConnect Secure Mobility Client v 3.0 adding route to local DHCP server?

[prospero]

Has anyone run across this? We just rolled the Cisco AnyConnect Secure Mobility Client version 3.0.1047 and have noticed that when the VPN connection is established, a route is added to the local PC for the DHCP server itself, which uses the IP address of the default gateway as the next hop. Since the DHCP server is local (and is not running on the same device as the default gateway) this effectively renders the DHCP server inaccessible since the packets go from the client PC to the default gateway at which point it dies (in this case the default gateway is a Cisco ASA).

Here is an example from the route print output (Windows 7 x64):

192.168.13.101  255.255.255.255    192.168.13.1  192.168.13.119     11

So in this case, 192.168.13.101 is the local server running DHCP (and DNS and AD). 192.168.13.1 is the inside interface of the ASA (the default gateway of all the boxes on that network). 192.168.13.119 is the IP address of the client machine running the AnyConnect software. If the client machine has a static IP address, a route entry is not added. Similarly, as soon as the VPN is disconnected, the route entry is removed.

So while I think I know what is happening, I have no idea why it is happening or what to do about it. We have tried reinstalling the VPN client (didn't fix it), tried running the VPN client on other systems (same problem) and had folks at various locations repeat the tests. If the DHCP server and the default gateway are the same device, it's no big deal. The problem is when they aren't.

Anyone seen anything like this or have any idea what might be causing it?

[Nicolas Meessen]

I think that AC adds the route to make sure that the PC can renew its dhcp lease (we don't want that traffic tunneled), the DEs probably counted on the fact that the default gateway would u-turn the traffic if the destination was on the same subnet (which is what most routers would do). In your case the u-turning doesn't happen and that's why it breaks.

We can probably open a bug for this, suggesting that if the DHCP server is on the same subnet, then add a route saying /32 via instead of the default gateway. (can you open a TAC case for that? If you're ok to do it, we can even do it straight from this thread)

In the meantime, there may be a way to get the ASA to u-turn the traffic (same-security-traffic permit intra-interface), or we can play with local-lan access and full/split tunneling (may or may not work, we'd need to test).

Nicolas

[prospero]

My counterpart is opening a TAC case on it. I can post the ticket here for tracking. Thanks!

[Michal Breskovec]

Hello, any result from TAC? I am found this issue in our company, but regarding Cisco IPSec VPN and can't find any reasonable way how to solve it.

[Nicolas Meessen]

Hi Michal,

If you're in the exact same situation, I'd recommend to allow the traffic to be u-turned on the local default gateway (same-security-traffic command if it's an ASA).

If not, please share your topology and details of the issue so I can take a look.

I found the TAC case that was opened for the thread above but no further resolution took place beyond suggesting to make sure that the u-turning on the default gateway works as expected.

[Michal Breskovec]

Hello Nicolas, thanks for quick answer. Default gateway is the C3750X device. ASA which serving like VPN concetrator is in different network. Traffic to this ASA is routed from C3750X via ISP's MPLS network. Regarding DHCP server - it is in different subnet than user's workstation and ip helper-address is used on C3750X to reach it.

Is it possible apply your solution to this scenario as well?

[Nicolas Meessen]

I think it should:

What's the route that you see installed on the client by the VPN software?

What's the default gateway of the client on the LAN (when not connected to VPN)?

What's the IP address you cannot reach?

Do you need to reach that IP over the VPN or is it ok to talk to it outside of the tunnel?

It's much easier to recommend something if I know the topology and we work on a specific instance.

[Michal Breskovec]

Route installed by VPN on the client:

Network Destination  Netmask             Gateway          Interface        Metric

10.25.152.1              255.255.255.255  10.25.149.254  10.25.149.44  1

Default gateway without active VPN: 10.25.149.254

IP which can't be reached: 10.25.152.1 - it is DHCP and DNS server, and unfortunately it is serving like file server as well.

It must be reached via VPN, because Split tunneling is not allowed in our company due to security rules.

If it will be better I can create TAC case for this issue.

[Nicolas Meessen]

There are a few things to consider here:

- The IPSec VPN client is EoL, so even if we consider this as a bug, it wouldn't be fixed

- fixing the file server access would break the DHCP renew which means there is no completely clean way to fix this, at least not at the IP level since the client can't route to the same destination using 2 different paths.

Is there any chance we could do a static policy NAT for the DHCP traffic so it appears to come from another IP? It's twisted and it may not work (the client might use the DHCP server IP embedded inside the payload and not the source IP) but if it does, then we'd fix the overlap.

Could the server use another IP address for the DHCP service (much like using a loopback for a certain service on a router?)

A third solution would be to NAT the destination server IP on the ASA for traffic from the IP pool going to the server. We'd need DNS doctoring as well to resolve the server's name to the NATted IP. This way the server would appear from the VPN client as being at a different IP, thereby fixing the overlap.

All these potential solutions are quite involved... you may be better off wityh a simpler design: splitting of your server into 2 or using something else to do DHCP for the VPN clients.

[Michal Breskovec]

Thanks for your effort and advice Nicolas. I will be thinking about these solutions. Anyway, maybe another option can be set another IP to the server via which it can be accessed from VPN.

[Nicolas Meessen]

Yup, this should work as well.