ASA-PIX: DHCP relay through VPN tunnel

[Kureli Sankar]

Happy New Year to everyone!

I spent an entire day working on this issue that  I decided to discuss with everyone today.

(click image to enlarge)

As you can see in the above topology, clients hanging off the inside interface of Local ASA are trying
to acquire an IP address from the DHCP server that is located on the other side of the tunnel behind the
Remote ASA.

Let us quickly run through how DHCP relay works.

1. Client  starts the DHCP process by sending a DHCPDISCOVER message to destination address
    255.255.255.255  - UDP port 67

2. ASA sees the broadcast and based on the dhcprelay server config it forwards the DHCPDISCOVER
    message as a unicast packet to the server's IP sourcing from the interface IP close to the server.
    In this case the outside IP address.

3. Server sends back DHCPOFFER as a unicast packet to the ASA - UDP port 67

    Server will send it to the destination IP address of the inside interface (giaddr) which is the
    dhcprelay enabled interface.

4. This packet will arrive on the outside interface and will be broadcast out the inside interface - UDP port 68.

5. Client receives the DHCPOFFER, and sends a DHCPREQUEST message to the server, that it accepts the offer.

6. The server will send back a DHCPACK message to the client.

7. Client upon receiving the DHCPACK, it will start communicating on the network.

Here is the partial config on the Local ASA:

dhcprelay server 10.48.65.13 outside
dhcprelay enable inside

DHCP relay packets will be sent to the server with the source address  of the outside interface IP
address and we need to encrypt that.

crypto acl includes:

access-list L-VPN extended permit ip host 10.252.17.6 host 10.48.65.13

access-list L-VPN extended permit ip host 10.150.1.1 host 10.48.65.13

Here is the partial config on the Remote ASA:

static (inside,outside) 10.150.1.1 10.252.17.6 netmask 255.255.255.255 (this is the problematic static)

static (inside,outside) 10.150.1.0 10.150.1.0 netmask 255.255.255.0

The first static 1-1 is a requirement because the DHCP server will only talk to IP addresses in
the 10.150.1.0/24 network.

crypto acl includes:

access-list R-VPN extended permit ip host 10.48.65.13 host 10.252.17.6

access-list R-VPN extended permit ip host 10.48.65.13 host 10.150.1.1

Technically the offer packets from the server will be sent to the interface IP address that has
dhcprelay enabled which is the inside interface 10.150.1.1 but, since this Remote ASA has
static 1-1 NAT configured it changes the destination IP address to 10.252.17.6 upon receiving

these packets destined to 10.150.1.1

Everyone with me so far? Now comes the interesting part, the problem.

The clients off the Local ASA never get an IP address.

According to debugs on the Local ASA the DHCP server seems to be sending offers but,
these packets do not egress the inside interface of the Local ASA.

dhcpd_forward_request: request from 0024.7e03.e3bf forwarded to 10.48.65.13.
DHCPRA: Received a BOOTREPLY from interface 1
DHCPRA: relay binding found for client 0024.7e03.e3bf.
DHCPRA: Adding rule to allow client to respond using offered address 10.150.1.10
DHCPRA: forwarding reply to client 0024.7e03.e3bf.

Now, where could they be going? On the outside we cannot capture because the traffic is
encrypted. Debugs and syslogs don't indicate any problem. I started to wonder if the ASA
was silently dropping these packets. Well, that should show in the asp drop captures shouldn't it?
Nope.  They weren't there either.

It was a mystery. I decided to throw a capture on the outside interface for udp 67 and udp 68
packets and wasn't I surprised to see the ASA broadcasting the offer packets off the outside interface!
Oh boy!! Why would it do that? The clients are on the inside so, why would this ASA send these packets
off the outside interafce and also create arp entries on the outside?

sh arp | i outside
arp outside 10.151.1.19 0024.7e03.d45d
arp outside 10.151.1.10 0024.7e03.e3bf

This was extremly strange. MAC address belongs to clients on the inside. DHCP offered IP address is
mapped to the MAC address properly but the interface is wrong. No wonder the inside clients don't
get the IP address!

Well, now we need to investigate why the ASA is doing what it is doing.  Well, time to read the DHCP RFC:

According to section 4.1.2 BOOTREPLY Messages
...

The 'giaddr' field can be used to identify the logical interface from which the reply must be sent (i.e., the host
or router interface connected to the same network as the BOOTP client). If the content of the 'giaddr' field does
not match one of the relay agent's directly-connected logical interfaces, the BOOTREPLY messsage MUST
be silently discarded
....

The ASA is supposed to receive these offer packets destined to the inside interface IP on the outside interface
but, due to the static 1-1 on the Remote ASA this is not the case. The offer packet's destinated IP address gets
changed to the outside IP address. Upon receiving these packets, without looking at the "giaddr" in the packet the
ASA chooses to broadcast it out on the outside interface thinking the clients are off of the outside interface which
is clearly not enabled for dhcprelay.

I know what everyone is thinking. Time to file a bug. Right. But, we need to fix the probelm at hand.  What can
we do resolve the problem right now?Using static IP address on the client PCs is not an option. Somehow we
need to make sure that the destination IP address in the offer packets sent by the server does not get changed.
How can we accomplish this? If we can do that then, things will start to work and we can file a bug later.

We decided to change the static 1-1 on the Remote ASA as follows:
static (inside,outside) 10.150.1.6 10.252.17.6 netmask 255.255.255.255

We just picked this address 10.150.1.6 that was not used anywhere. Since these are just udp packets, when going to
the server these will have the source address of 10.150.1.6 but, when the server responds back it will respond back to the
"giaddr" which is 10.150.1.1, will take the identity static and reach the Local ASA.

Once we did the above, DHCP relay started to work as expected.
I have filed this defect CSCtj68732.(NOTE: you will need to login with your Cisco Connection Online "CCO" credentials to access)

I hope this was informative and thanks for reading.

View sample dhcp captures
http://wiki.wireshark.org/SampleCaptures?action=AttachFile&do=view&target=dhcp.pcap

-Kureli

[Federico Coto Fajardo]

Kureli!

Great document and very interesting, thanks for sharing!!

Federico.

[Kureli Sankar]

Federico,

Thanks.  Glad you enjoyed reading. I tried to make it interesting for our readers .

-Kureli

[Mirza Arsalan Baig]

Hi Kureli,

static (inside,outside) 10.150.1.6 10.252.17.6 netmask 255.255.255.255

Will it work for any broadcast to send inside to outside (remote user over ip secc VPN tunnel)

Regards

Arsalan Baig

[Maykol Rojas]

Hi Kureli,

Isn't this static on the remote ASA backwards?

static (inside,outside) 10.150.1.1 10.252.17.6 netmask 255.255.255.255

Cheers

Mike

[Kureli Sankar]

No. It is correct. Tunnel terminates on the inside of the Remote ASA.

-Kureli

[Kureli Sankar]

Arsalan,

IPsec VPN does not support broadcast or multicast.  It supports only unicast traffic.

DHCP is broadcast but, the ASA acts as a relay, taking in the broadcast and sending a unicast packet out towards the DHCP server.

-Kureli

[lcaruso]

Great discussion and diagram.

Does this discussion apply to a situation where the tunnel terminates on the outside interface of both ASAs? Isn't that the more typical scenario, or to ask it this way, why would you want to terminate a tunnel on the inside interface instead of the outside interface?

I have a problem open with TAC for dhcp relay using the outside-to-ouside tunnel configuration, so I'm wondering if your discussion will solve this case since they said my configuration was correct.

Thanks.

[Kureli Sankar]

Thank you. I put in a lot of effort. Seems like it has paid off. Yes, it does apply to tunnels terminated on the outside. In this dicussion the tunnel is terminating on the inside of the remote ASA and for some odd requirement they chose to translate the outside interface IP of the Local ASA to the dhcp relay enabled interface IP address on the Remote ASA which caused the whole problem.

Pls. send this link to the TAC engineer so, they can investigate if this applies in your case or may be reach out to me internally.

Well, the reason for terminating the tunnel on the inside in this case is that the network behind the Remote ASA is untrusted or customer network.

-Kureli

[smailbouabdallah]

Hi,

How we configure the DHCPrelay if we have only one ASA for IPSEC remote VPN? I tried to configure the DHCPrelay and I could not receive an IP address.

Thanks,

[Issa NDIAYE]

your document is very important! thanks you very most!

best regards,

[Stuart Patton]

This article is a great find because lots of other people on the internet have said rather confidently that you add the local firewall's inside and outside IP address into the cryptomap ACL to tunnel the DHCP relay traffic but have obviously never tested  that it works.

Has anyone implemented this with the remote ASA (per the diagram) using 8.3 NATs?  I have tried everything I can think of but end up getting "Deny inbound UDP" messages.  These are not ACL drop messages, but indicate there is something wrong with the NAT...just can't figure it out.

Thanks

[Kureli Sankar]

Thank you Stuart.  I am not sure your scenario is exactly the same as the one that I mentioned above. The tunnel is terminating on the inside (high sec interface) of the remote ASA. Whether 8.3 or pre 8.3 this should work. You are correct Deny inbound UDP messages are not the ones that are denied due to ACL.  The interface for some reason is rejecting the packets. asp-drop captures should tell you why.

Open a thread on our discussion forum

https://supportforums.cisco.com/community/netpro/security/firewall

and we can discuss this further and get you up and running. Another thought is to open a TAC case which would be better because of the complexity involved in the topology and all the data that needs to be gathered simultaneously and analyzed.

-Kureli

[vijayasankar]

Hi,

We have the same problem observed in one of the working setup which is exactly similar to the setup that you have explained.

The observations remains the same.All of the sudden, the dhcp relay feature is broken.

WE have changed the static nat to another free ip at the remote firewall, and still the problem persists.

Here's what i have found more on this.

When i SPAN the local firewall outside interface ip and took the sniffer trace, i could see that all the DHCP offer messages from remote DHCP server is getting brodcasted on the outside interface of the firewall.

The DHCP offer packet is supposed to be transmitted by the local firewall to its inside interface, strangely for some reasons it is now not doing it.

As we lost almost 1 day with Business escalation, we have removed the dhcp relay configurations from local firewall and turned on DHCP relay in one of a L3 switch behind the local firewall. All is working well now as expected.

Certainly Not sure, why this feature should suddenly break, in a working setup..Any ideas.....

[Kureli Sankar]

Vijayasankar,

I believe the offer packets were encrypted so, you couldn't have possibly seen the desination IP address for these unicast offer packets.  I believe the NAT change may not have taken effect. Do you recall issuing a "sh xlate debug | i o.o.o.o for the local ASA's outside IP on the remote ASA and make sure it was taking the newly configured static to the un-used address in the server's preferred subnet? Did you also issue "sh xlate debug | i i.i.i.i for the inside IP address of the remote ASA so, it looked itself and DID NOT get changed back to the outside IP o.o.o.o?  Anytime you change NAT you would also have to clear all xlates and connections by issuing "clear local o.o.o.o" "clear local i.i.i.i" on the remote ASA where o.o.o.o is the IP address of the outside interface and i.i.i.i is the inside IP of the local ASA. I did discuss this issue with your engineer today. This kind of translation is not recommended - I mean translating the outside IP to inside IP of one ASA on another ASA.  NOT a good idea at all.  Always use an un-used address. 

-Kureli

[vijayasankar]

Hi,

Thanks a lot for the update.

The NAT changed was indeed taking effect and we could verify that using the packet capture taken in the remote firewall. The offer packets were coming from DHCP to the new NATTed id.

We could see all the DHCP discovery and DHCP offer packets when we do the pacekt capture in remote firewall.

The anomaly is observed on the local firwall.

The local firewall is receiving the DHCP offer from remote firewall,

however instead  of transmitting the offer packets to inside interface, the local firewall is sending the offer packet to the outside interface LAN. When we clear the ARP table for the outside interface in the local firewall and observe for a while, we could see slow the ARP table starts showing more inside ip address learned in the outside interface.

Apparently changing the NAT ip address had not changed the problem behavior. Whatever IP address we use to NAT, the local firewall was consistenly transmitting the DHCP offer only to the outside and not the inside interface.

We have erased the configuration completely on the local firwall, reloaded it, reconfigured it the same way and still the problem persists with the same behavior. Changed the NAT once more and collected the packet capture and logs and the problem symptom was consistent.

In my  network, i dont have a similar setup. This is very unique requirement for this setup.

Im not sure if such setups are  supported in the ASA firewall. May be im trying to make something work which is not expected to work (or) not as per a standard implementation... Whats your comment on this...

I have plenty of more setup like this which works perfectly fine. The only difference is in those setup, IP SEC vpn is not involved. DHCP relay feature would be turned on in teh inside interface and the relay is configured to reach a DHCP server which is located on a routed network after the firewalls outside interface. These setups works perfectly fine with no glitch.

Is this behavior of ASA firewall very unique when it comes to using DHCP relay with a IP Sec VPN tunnels and DHCP server which is routed over a IpSec Tunnel?

thanks/ Regards

Vijaya Sankar.K