Happy New Year to everyone!
I spent an entire day working on this issue that I decided to discuss with everyone today.
(click image to enlarge)
As you can see in the above topology, clients hanging off the inside interface of Local ASA are trying
to acquire an IP address from the DHCP server that is located on the other side of the tunnel behind the
Remote ASA.
Let us quickly run through how DHCP relay works.
1. Client starts the DHCP process by sending a DHCPDISCOVER message to destination address
255.255.255.255 - UDP port 67
2. ASA sees the broadcast and based on the dhcprelay server config it forwards the DHCPDISCOVER
message as a unicast packet to the server's IP sourcing from the interface IP close to the server.
In this case the outside IP address.
3. Server sends back DHCPOFFER as a unicast packet to the ASA - UDP port 67
Server will send it to the destination IP address of the inside interface (giaddr) which is the
dhcprelay enabled interface.
4. This packet will arrive on the outside interface and will be broadcast out the inside interface - UDP port 68.
5. Client receives the DHCPOFFER, and sends a DHCPREQUEST message to the server, that it accepts the offer.
6. The server will send back a DHCPACK message to the client.
7. Client upon receiving the DHCPACK, it will start communicating on the network.
Here is the partial config on the Local ASA:
dhcprelay server 10.48.65.13 outside
dhcprelay enable inside
DHCP relay packets will be sent to the server with the source address of the outside interface IP
address and we need to encrypt that.
crypto acl includes:
access-list L-VPN extended permit ip host 10.252.17.6 host 10.48.65.13
access-list L-VPN extended permit ip host 10.150.1.1 host 10.48.65.13
Here is the partial config on the Remote ASA:
static (inside,outside) 10.150.1.1 10.252.17.6 netmask 255.255.255.255 (this is the problematic static)
static (inside,outside) 10.150.1.0 10.150.1.0 netmask 255.255.255.0
The first static 1-1 is a requirement because the DHCP server will only talk to IP addresses in
the 10.150.1.0/24 network.
crypto acl includes:
access-list R-VPN extended permit ip host 10.48.65.13 host 10.252.17.6
access-list R-VPN extended permit ip host 10.48.65.13 host 10.150.1.1
Technically the offer packets from the server will be sent to the interface IP address that has
dhcprelay enabled which is the inside interface 10.150.1.1 but, since this Remote ASA has
static 1-1 NAT configured it changes the destination IP address to 10.252.17.6 upon receiving
these packets destined to 10.150.1.1
Everyone with me so far? Now comes the interesting part, the problem.
The clients off the Local ASA never get an IP address.
According to debugs on the Local ASA the DHCP server seems to be sending offers but,
these packets do not egress the inside interface of the Local ASA.
dhcpd_forward_request: request from 0024.7e03.e3bf forwarded to 10.48.65.13.
DHCPRA: Received a BOOTREPLY from interface 1
DHCPRA: relay binding found for client 0024.7e03.e3bf.
DHCPRA: Adding rule to allow client to respond using offered address 10.150.1.10
DHCPRA: forwarding reply to client 0024.7e03.e3bf.
Now, where could they be going? On the outside we cannot capture because the traffic is
encrypted. Debugs and syslogs don't indicate any problem. I started to wonder if the ASA
was silently dropping these packets. Well, that should show in the asp drop captures shouldn't it?
Nope. They weren't there either.
It was a mystery. I decided to throw a capture on the outside interface for udp 67 and udp 68
packets and wasn't I surprised to see the ASA broadcasting the offer packets off the outside interface!
Oh boy!! Why would it do that? The clients are on the inside so, why would this ASA send these packets
off the outside interafce and also create arp entries on the outside?
sh arp | i outside
arp outside 10.151.1.19 0024.7e03.d45d
arp outside 10.151.1.10 0024.7e03.e3bf
This was extremly strange. MAC address belongs to clients on the inside. DHCP offered IP address is
mapped to the MAC address properly but the interface is wrong. No wonder the inside clients don't
get the IP address!
Well, now we need to investigate why the ASA is doing what it is doing. Well, time to read the DHCP RFC:
According to section 4.1.2 BOOTREPLY Messages
...
The 'giaddr' field can be used to identify the logical interface from which the reply must be sent (i.e., the host
or router interface connected to the same network as the BOOTP client). If the content of the 'giaddr' field does
not match one of the relay agent's directly-connected logical interfaces, the BOOTREPLY messsage MUST
be silently discarded
....
The ASA is supposed to receive these offer packets destined to the inside interface IP on the outside interface
but, due to the static 1-1 on the Remote ASA this is not the case. The offer packet's destinated IP address gets
changed to the outside IP address. Upon receiving these packets, without looking at the "giaddr" in the packet the
ASA chooses to broadcast it out on the outside interface thinking the clients are off of the outside interface which
is clearly not enabled for dhcprelay.
I know what everyone is thinking. Time to file a bug. Right. But, we need to fix the probelm at hand. What can
we do resolve the problem right now?Using static IP address on the client PCs is not an option. Somehow we
need to make sure that the destination IP address in the offer packets sent by the server does not get changed.
How can we accomplish this? If we can do that then, things will start to work and we can file a bug later.
We decided to change the static 1-1 on the Remote ASA as follows:
static (inside,outside) 10.150.1.6 10.252.17.6 netmask 255.255.255.255
We just picked this address 10.150.1.6 that was not used anywhere. Since these are just udp packets, when going to
the server these will have the source address of 10.150.1.6 but, when the server responds back it will respond back to the
"giaddr" which is 10.150.1.1, will take the identity static and reach the Local ASA.
Once we did the above, DHCP relay started to work as expected.
I have filed this defect CSCtj68732.(NOTE: you will need to login with your Cisco Connection Online "CCO" credentials to access)
I hope this was informative and thanks for reading.
View sample dhcp captures
http://wiki.wireshark.org/SampleCaptures?action=AttachFile&do=view&target=dhcp.pcap
-Kureli
47 Comments
