ASA-PIX: DHCP relay through VPN tunnel

[Kureli Sankar]

Happy New Year to everyone!

I spent an entire day working on this issue that  I decided to discuss with everyone today.

(click image to enlarge)

As you can see in the above topology, clients hanging off the inside interface of Local ASA are trying
to acquire an IP address from the DHCP server that is located on the other side of the tunnel behind the
Remote ASA.

Let us quickly run through how DHCP relay works.

1. Client  starts the DHCP process by sending a DHCPDISCOVER message to destination address
    255.255.255.255  - UDP port 67

2. ASA sees the broadcast and based on the dhcprelay server config it forwards the DHCPDISCOVER
    message as a unicast packet to the server's IP sourcing from the interface IP close to the server.
    In this case the outside IP address.

3. Server sends back DHCPOFFER as a unicast packet to the ASA - UDP port 67

    Server will send it to the destination IP address of the inside interface (giaddr) which is the
    dhcprelay enabled interface.

4. This packet will arrive on the outside interface and will be broadcast out the inside interface - UDP port 68.

5. Client receives the DHCPOFFER, and sends a DHCPREQUEST message to the server, that it accepts the offer.

6. The server will send back a DHCPACK message to the client.

7. Client upon receiving the DHCPACK, it will start communicating on the network.

Here is the partial config on the Local ASA:

dhcprelay server 10.48.65.13 outside
dhcprelay enable inside

DHCP relay packets will be sent to the server with the source address  of the outside interface IP
address and we need to encrypt that.

crypto acl includes:

access-list L-VPN extended permit ip host 10.252.17.6 host 10.48.65.13

access-list L-VPN extended permit ip host 10.150.1.1 host 10.48.65.13

Here is the partial config on the Remote ASA:

static (inside,outside) 10.150.1.1 10.252.17.6 netmask 255.255.255.255 (this is the problematic static)

static (inside,outside) 10.150.1.0 10.150.1.0 netmask 255.255.255.0

The first static 1-1 is a requirement because the DHCP server will only talk to IP addresses in
the 10.150.1.0/24 network.

crypto acl includes:

access-list R-VPN extended permit ip host 10.48.65.13 host 10.252.17.6

access-list R-VPN extended permit ip host 10.48.65.13 host 10.150.1.1

Technically the offer packets from the server will be sent to the interface IP address that has
dhcprelay enabled which is the inside interface 10.150.1.1 but, since this Remote ASA has
static 1-1 NAT configured it changes the destination IP address to 10.252.17.6 upon receiving

these packets destined to 10.150.1.1

Everyone with me so far? Now comes the interesting part, the problem.

The clients off the Local ASA never get an IP address.

According to debugs on the Local ASA the DHCP server seems to be sending offers but,
these packets do not egress the inside interface of the Local ASA.

dhcpd_forward_request: request from 0024.7e03.e3bf forwarded to 10.48.65.13.
DHCPRA: Received a BOOTREPLY from interface 1
DHCPRA: relay binding found for client 0024.7e03.e3bf.
DHCPRA: Adding rule to allow client to respond using offered address 10.150.1.10
DHCPRA: forwarding reply to client 0024.7e03.e3bf.

Now, where could they be going? On the outside we cannot capture because the traffic is
encrypted. Debugs and syslogs don't indicate any problem. I started to wonder if the ASA
was silently dropping these packets. Well, that should show in the asp drop captures shouldn't it?
Nope.  They weren't there either.

It was a mystery. I decided to throw a capture on the outside interface for udp 67 and udp 68
packets and wasn't I surprised to see the ASA broadcasting the offer packets off the outside interface!
Oh boy!! Why would it do that? The clients are on the inside so, why would this ASA send these packets
off the outside interafce and also create arp entries on the outside?

sh arp | i outside
arp outside 10.151.1.19 0024.7e03.d45d
arp outside 10.151.1.10 0024.7e03.e3bf

This was extremly strange. MAC address belongs to clients on the inside. DHCP offered IP address is
mapped to the MAC address properly but the interface is wrong. No wonder the inside clients don't
get the IP address!

Well, now we need to investigate why the ASA is doing what it is doing.  Well, time to read the DHCP RFC:

According to section 4.1.2 BOOTREPLY Messages
...

The 'giaddr' field can be used to identify the logical interface from which the reply must be sent (i.e., the host
or router interface connected to the same network as the BOOTP client). If the content of the 'giaddr' field does
not match one of the relay agent's directly-connected logical interfaces, the BOOTREPLY messsage MUST
be silently discarded
....

The ASA is supposed to receive these offer packets destined to the inside interface IP on the outside interface
but, due to the static 1-1 on the Remote ASA this is not the case. The offer packet's destinated IP address gets
changed to the outside IP address. Upon receiving these packets, without looking at the "giaddr" in the packet the
ASA chooses to broadcast it out on the outside interface thinking the clients are off of the outside interface which
is clearly not enabled for dhcprelay.

I know what everyone is thinking. Time to file a bug. Right. But, we need to fix the probelm at hand.  What can
we do resolve the problem right now?Using static IP address on the client PCs is not an option. Somehow we
need to make sure that the destination IP address in the offer packets sent by the server does not get changed.
How can we accomplish this? If we can do that then, things will start to work and we can file a bug later.

We decided to change the static 1-1 on the Remote ASA as follows:
static (inside,outside) 10.150.1.6 10.252.17.6 netmask 255.255.255.255

We just picked this address 10.150.1.6 that was not used anywhere. Since these are just udp packets, when going to
the server these will have the source address of 10.150.1.6 but, when the server responds back it will respond back to the
"giaddr" which is 10.150.1.1, will take the identity static and reach the Local ASA.

Once we did the above, DHCP relay started to work as expected.
I have filed this defect CSCtj68732.(NOTE: you will need to login with your Cisco Connection Online "CCO" credentials to access)

I hope this was informative and thanks for reading.

View sample dhcp captures
http://wiki.wireshark.org/SampleCaptures?action=AttachFile&do=view&target=dhcp.pcap

-Kureli

[Kureli Sankar]

Vijaya Sankar,

The NAT changed was indeed taking effect and we could verify that using  the packet capture taken in the remote firewall. The offer packets were  coming from DHCP to the new NATTed id.

That NAT change to an un-used address is only for relay packets (request) going to the DHCP server - is only for one way traffic. DHCP server should send offer packets to the GI-ADDR (which is the IP of the inside int address of the lcoal ASa) and NOT the newly NAT-ed address. Pls. clarify your above statement.

Added to that you can only gather clear captures on the remote ASA on the outside interface. On the inside interface you cannot because traffic will be encrypted.  The offer packet's destination IP address should be the inside interface IP address of the local ASA while egressing the inside interface towards the local ASA.  Did you validate that?  How?

This setup is supported. As you can read in the comments that another reader had a similar setup in 8.3+ NAT and we resolved the issue right here on our forum.

The KEY here is that L-3 destination IP of the offer packets from the DHCP server when arriving on the local ASA, should remain un-changed all the way from the DHCP server THROUGH the remote ASA TO the local ASA and remain as the local ASA's inside interface IP address. 

The outstanding issue on Cisco side is the defect CSCtj68732 that we filed.  It is still not resolved.  So long as the outside interface on the local received offer packets destined to the inside int. IP, the ASA will process the offers properly - as the defect states.

<B>Symptom:</B>
DHCP Relay functionality on the PIX/ASA may incorrectly make a broadcast decision based the 
Layer-3 Destination IP instead of the embedded GI-Addr address. This does not conform to 
BOOTP Relay RFC..

<B>Conditions:</B>
The behavior is the same on all ASA/PIX codes.

<B>Workaround:</B>
Ensure that the BOOTP replies arrive at the ASA with a Layer-3 Destination IP address of the 
interface facing the DHCP clients.

-Kureli

[sansarav720e]

Hi Vijay ,

          Could you please brief your problem clealry , I do have a same problem in my setup , below to my firewall i have L3 switch where i have L3 SVI under this i have defined IP helper address (remote dhcp server IP)  ,where my end machine on this SVI VLAN will direct DHCP request to this IP helper address . But i cant see end user machine is offered IP address from remote dhcp server .

              when i define IP helper address on interface VLAN ,it will be unicast dhcp message or broadcast message ?? how it will because i could not see any hits on my firewall inside interface .. I have done capture also ,even i have defined specific UDP rule for DHCP i could nt see any hits on ACL .

I guess the helper address on interface VLAN is once again UDP broadcast messgae , in my scenario how this DHCP service is achievable over VPN tunnel ??

[vijayasankar]

Hi Sarvanan,

My setup was quite to accomodate a client restriction and hence it had challenges for DHCP relay. The workaround that i have implemented for that scenario, was infact moving the DHCP relay feature from the Firewall to L3 switch in Inside segment. It will work perfectly fine.

You need to ensure that appropriate rules are available in the firewall to allow this traffic ( and the encryption ACLs).

The flow will be as follows

1) MAchines broadcast DHCP discover message.

2) Switch SVI will listen to DHCP broadcast and relay the same onbehalf of the desktop as a unicast request to the DHCP servers defined in the helper address.

3) This unicast packet will flow through the firewall and reach the DHCP server.

4) DHCP server will send a offer message to your L3 SVI unicast

You can enable packet capture in the Firewalls inside interface for the packets originates/destined to the SVI. You should be able to see the DHCP discover unicast and offer unicast here. If not, some thing is wrong in the way the firewall and relay is setup.

Regards

Vijay

[sansarav720e]

Hi Vijay ,

    Thank for your reply note ,  As part of security measure we have disabled service dhcp on our network switches , due to this dhcp relay function was not working . IP helper address under SVI L3 VLAN was not redirecting dhcp request as unicast . due to service dhcp deactivation .

       I have turned on service dhcp along with

ip forward-protocol udp bootpc

ip forward-protocol udp bootps , then also service havent started properly .Then have given hardboot to network switches with service dhcp turned on , after reloading service dhcp was performing relay functioning properly , where i can see dhcp request as a unicast to remote dhcp server . Then my end machine was able to recieve ip address seamlessly .

one strange thing about this IOS code is ,when i have enabled service dhcp , relay function was not turned on , after rebooting it was working automatically . i have given out 15 hours without rebooting by enabling service dhcp & no dhcp .At last i had no chance to troubleshoot then adopted for rebooted , luck turned on .. having the services offering.

Quite good one...

below is debug for dhcp events.. dhcp server ip  address (10.84.100.218)

end client ip address from dhcp : 10.4.110.21/24

Dec 18 23:24:21.768 IST: DHCP: SDiscover: sending 306 byte length DHCP packet

Dec 18 23:24:21.768 IST: DHCP: SDiscover 306 bytes

Dec 18 23:24:21.768 IST:             B'cast on Vlan280 interface from 0.0.0.0

Dec 18 23:24:22.548 IST: DHCP: Received a BOOTREP pkt

Dec 18 23:24:22.548 IST: DHCP: offer received from 10.84.100.218

Dec 18 23:24:22.548 IST: DHCP: SRequest attempt # 1 for entry:

Dec 18 23:24:22.548 IST: DHCP: SRequest- Server ID option: 10.84.100.218

Dec 18 23:24:22.548 IST: DHCP: SRequest- Requested IP addr option: 10.4.110.21

Dec 18 23:24:22.556 IST: DHCP: SRequest placed lease len option: 86400

Dec 18 23:24:22.556 IST: DHCP: SRequest: 324 bytes

Dec 18 23:24:22.556 IST: DHCP: SRequest: 324 bytes

Dec 18 23:24:22.556 IST:             B'cast on Vlan280 interface from 0.0.0.0

Dec 18 23:24:22.833 IST: DHCP: Received a BOOTREP pkt

Dec 18 23:24:25.836 IST: DHCP Client Pooling: ***Allocated IP address: 10.4.110.21

Dec 18 23:24:25.954 IST: Allocated IP address = 10.4.110.21 255.255.255.0

Thanks for your time .........

[vijayasankar]

Oops. That was a terrible problem, when the features dont work as expected. Good that it is resolved now. Thanks for the update.

[sansarav720e]

Thanks vijay & kureli .. had a great understanding on IPSec ...

[jurgendendas]

Hello I did everythting you said, but it is still not working.

What can I do more to let it work???

Kind regards.

[Kureli Sankar]

Sorry to hear that.  If you have smartnet, I'd suggest opening a case and refering this doc.  This gets really involved and we need to gather captures and debugs in multiple points to figure out what may be causing this.

Let me know the SR no. once opened.

Thanks,

Kureli

[jurgendendas]

I don't think we have that. We are an NGO and bought the devices trough techsoup.

[Kureli Sankar]

In that case, pls. post your question on our firewall community

https://supportforums.cisco.com/community/netpro/security/firewall

with all the output that I listed in this blog.

-Kureli

[sansarav720e]

Hi Jurgen ,

    Can you explain your problem along with your exsiting network diagram . Simialrly you need to make sure all basic settings are up running ..

[jurgendendas]

Hello

I've made an new treath for it on this link: https://supportforums.cisco.com/message/3554062. Maybe we can discuss it there?

Kind regards.

[Stuart Patton]

Incidentally, I just noticed that defect CSCtj68732 filed by Poohguzhali Sankar was fixed in 8.4.4(5)

[ju_mobile]

Kureli,

Please can I ask in what circumstances would you need to apply the NAT on the REMOTE ASA.

Here is the partial config on the Remote ASA:

static (inside,outside) 10.150.1.1 10.252.17.6 netmask 255.255.255.255 (this is the problematic static)

static (inside,outside) 10.150.1.0 10.150.1.0 netmask 255.255.255.0

is this a mistype and do you mean local

many thanks

Julian

[ju_mobile]

Stuart,

Do you have further details available ?

Best Regards

Julian