NXOS DHCP Relay: Option 54 vs Option 82.11 in relayed DHCPREQUEST - why different?

Marc Luethi · ‎01-09-2020

Hi all

We're trying to tackle a DHCP relay problem between a set of N9K-C9332PQ (NXOS 9.3.2) and dnsmasq (doesn't work) resp Windows 2016 DHCP service (works with some tweaking).

SITUATION:

We have a BGP EVPN VXLAN (L2VNI and L3VNI) with anycast gateway setup for a subnet spanning multiple leaves (each of which is a VPC pair of N9K-C9332PQ). We need to run DHCP relay service and the DHCP server is "somewhere upstream", but in the same VRF.

Therefore, we can't rely on the anycast GW address being used as the giaddr, since the DHCP server's response might quite literally go anywhere.

So we use a loopback interface (in the same VRF) to be used as source for DHCP relaying, and we have the relay insert some Option 82 information (Suboptions Circuit ID, Remote ID, Link Selection, Server ID Override). The loopback's address is reachable from the DHCP server.

Essentially, we followed:
https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/93x/vxlan/configuration/guide/b-cisco-nexus-9000-series-nx-os-vxlan-configuration-guide-93x/b-cisco-nexus-9000-series-nx-os-vxlan-configuration-guide-93x_appendix_010111.html#id_1... and we configured:

ip dhcp relay
ip dhcp relay information option
ip dhcp relay information option vpn

[...]

interface loopback32
  description DHCP Relay source address for VLAN3632
  vrf member MYVRF
  ip address 10.32.15.39/32

[...]

interface Vlan3632
  description SVI for VLAN 3632
  vrf member MYVRF
  no ip redirects
  ip address 10.32.96.1/20
  fabric forwarding mode anycast-gateway
  ip dhcp relay address 10.32.33.40 
  ip dhcp relay source-interface loopback32

PROBLEM:

dnsmasq silently drops the relayed DHCPREQUEST packets, allegedly because of a mismatch between Option 54 (Server ID) and Option 82.11 (Server ID Override). The client then keeps repeating the DHCPREQUEST (which is being relayed perfectly well) until it gives up.

Please see below for the packet details and comments.

dnsmasq's maintainer, in http://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2018q1/011899.html, responding in a discussion thread/problem report, where relayed packets show a similar discrepancy between Option 54 and 82.11, quotes RFC5107 (emphasis of the passage in RFC 5107 by me):

That's old code, and pre-dates detailed git logs, but it's fairly clear that the test is implementing this part of RFC 5107

When servicing a DHCPREQUEST message, the DHCP server would normally look at the Server Identifier option for verification that the address specified there is one of the addresses associated with the DHCP server, silently ignoring the DHCPREQUEST if it does not match a configured DHCP server interface address. If the DHCPREQUEST message contains a Server Identifier Override suboption, however, comparison should be made between the address in this suboption and the Server Identifier option. If both the Server Identifier Override suboption and the Server Identifier option specify the same address, then the server should accept the DHCPREQUEST message for processing, regardless of whether or not the Server Identifier option matches a DHCP server interface.

So the problem is that the client, on renewal, is sending the server-ID as 10.36.20.19, which is tripping the test specified above.

In a nutshell: dnsmasq is not responding to the relayed DHCPREQUEST because the request fails the criteria as per RFC 5107.

STEPS TAKEN

For the sake of comparison, we set up a Windows 2016 Server with DHCP service. At first, the DHCP sequence was identical (discover -> offer -> request), including the same "mistake" in Option 54 of the relayed DHCPREQUEST. Eventually, the sever would return a DHCPNAK (whereas dnsmasq silently dropped the relayed request)

With an additional "dummy" DHCP scope (as in: all addresses of the scope excluded from the scope) covering the address range of all DHCP relay source addresses, things started to work and we got DHCPACKs. This is hinted at here: https://docs.microsoft.com/en-us/windows-server/networking/technologies/dhcp/dhcp-subnet-options

All relay agent IP addresses (GIADDR) must be part of an active DHCP scope IP address range. Any GIADDR outside of the DHCP scope IP address ranges is considered a rogue relay and Windows DHCP Server will not acknowledge DHCP client requests from those relay agents.

A special scope can be created to "authorize" relay agents. Create a scope with the GIADDR (or multiple if the GIADDR's are sequential IP addresses), exclude the GIADDR address(es) from distribution, and then activate the scope. This will authorize the relay agents while preventing the GIADDR addresses from being assigned.

On dnsmasq, we tried the same trick (setting up a dummy range covering all relay source addresses, but it did not help).

QUESTIONS:

Given a DHCP transaction where Server ID Override is needed, in the relayed DHCPREQUEST, what should be the content of the Server ID (Option 54) field? Should it be the overriding ID (as RFC 5107 seems to imply) or the real DHCP server's address (as the N9K seems to set it)?
Does RFC5107 really demand that Options 54 and 82.11 are identical?
(ostensibly, dnsmasq implements a check, but Windows's DHCP service seems to ignore this difference)
Is there a configuration option in NXOS 9.3.2 (that we failed to find so far) that would change the relay's behavior?

Thank you for your suggestions, thoughts and answers.

best regards

Marc

The packet dissections:

The relayed DHCPDISCOVER as it leaves the Nexus towards the DHCP server:

GIADDR is set to 10.32.15.39 (loopback32 of the given switch)
Option 82 Suboptions (e.g. Link Selection and Server ID Override) are set correctly

Internet Protocol, Src: 10.32.15.39 (10.32.15.39), Dst: 10.32.33.40 (10.32.33.40)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
        0000 00.. = Differentiated Services Codepoint: Default (0x00)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
    Total Length: 389
    Identification: 0xdb6c (56172)
    Flags: 0x00
        0.. = Reserved bit: Not Set
        .0. = Don't fragment: Not Set
        ..0 = More fragments: Not Set
    Fragment offset: 0
    Time to live: 255
    Protocol: UDP (0x11)
    Header checksum: 0x9a6c [correct]
        [Good: True]
        [Bad : False]
    Source: 10.32.15.39 (10.32.15.39)
    Destination: 10.32.33.40 (10.32.33.40)
User Datagram Protocol, Src Port: bootps (67), Dst Port: bootps (67)
    Source port: bootps (67)
    Destination port: bootps (67)
    Length: 369
    Checksum: 0xf666 [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
Bootstrap Protocol
    Message type: Boot Request (1)
    Hardware type: Ethernet
    Hardware address length: 6
    Hops: 1
    Transaction ID: 0x00002384
    Seconds elapsed: 0
    Bootp flags: 0x8000 (Broadcast)
        1... .... .... .... = Broadcast flag: Broadcast
        .000 0000 0000 0000 = Reserved flags: 0x0000
    Client IP address: 0.0.0.0 (0.0.0.0)
    Your (client) IP address: 0.0.0.0 (0.0.0.0)
    Next server IP address: 0.0.0.0 (0.0.0.0)
    Relay agent IP address: 10.32.15.39 (10.32.15.39)
    Client MAC address: 70:79:b3:ff:51:42 (70:79:b3:ff:51:42)
    Client hardware address padding: 00000000000000000000
    Server host name not given
    Boot file name not given
    Magic cookie: (OK)
    Option: (t=53,l=1) DHCP Message Type = DHCP Discover
        Option: (53) DHCP Message Type
        Length: 1
        Value: 01
    Option: (t=57,l=2) Maximum DHCP Message Size = 1200
        Option: (57) Maximum DHCP Message Size
        Length: 2
        Value: 04B0
    Option: (t=61,l=25) Client identifier
        Option: (61) Client identifier
        Length: 25
        Value: 00636973636F2D373037392E623366662E353134322D4769...
    Option: (t=12,l=13) Host Name = "ExpDHCPClient"
        Option: (12) Host Name
        Length: 13
        Value: 45787044484350436C69656E74
    Option: (t=55,l=9) Parameter Request List
        Option: (55) Parameter Request List
        Length: 9
        Value: 01060F2C790321962B
        1 = Subnet Mask
        6 = Domain Name Server
        15 = Domain Name
        44 = NetBIOS over TCP/IP Name Server
        121 = Classless Static Route
        3 = Router
        33 = Static Route
        150 = TFTP server address
        43 = Vendor-Specific Information
    Option: (t=60,l=8) Vendor class identifier = "ciscopnp"
        Option: (60) Vendor class identifier
        Length: 8
        Value: 636973636F706E70
    Option: (t=82,l=48) Agent Information Option
        Option: (82) Agent Information Option
        Length: 48
        Value: 010409010E30020628AC9E65F84F9714004F5330312D5245...
        Agent Circuit ID: 09010E30
        Agent Remote ID: 28AC9E65F84F
        DHCPv4 Virtual Subnet Selection: 004F5330312D52454730312D444556312D5A3030
        Server Identifier Override: 0A206001
        Link selection: 10.32.96.0
    End Option

The DHCPOFFER as it comes back from the DHCP server to the Relay

we can see that Option 54 is set to 10.32.96.1, as it had been requested by Option 82.11 (Server ID Override) before. The server must've understood Option 82.11.
the offered address (10.32.96.104) and mask (255.255.240.0) matches subnet 10.32.96.0/20, so the DHCP server either derived it from 82.11 or directly understood 82.5 (Link Selection)

Internet Protocol, Src: 10.32.33.40 (10.32.33.40), Dst: 10.32.15.39 (10.32.15.39)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0xc0 (DSCP 0x30: Class Selector 6; ECN: 0x00)
        1100 00.. = Differentiated Services Codepoint: Class Selector 6 (0x30)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
    Total Length: 395
    Identification: 0x55ba (21946)
    Flags: 0x00
        0.. = Reserved bit: Not Set
        .0. = Don't fragment: Not Set
        ..0 = More fragments: Not Set
    Fragment offset: 0
    Time to live: 61
    Protocol: UDP (0x11)
    Header checksum: 0xe159 [correct]
        [Good: True]
        [Bad : False]
    Source: 10.32.33.40 (10.32.33.40)
    Destination: 10.32.15.39 (10.32.15.39)
User Datagram Protocol, Src Port: bootps (67), Dst Port: bootps (67)
    Source port: bootps (67)
    Destination port: bootps (67)
    Length: 375
    Checksum: 0xa7bb [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
Bootstrap Protocol
    Message type: Boot Reply (2)
    Hardware type: Ethernet
    Hardware address length: 6
    Hops: 1
    Transaction ID: 0x00002384
    Seconds elapsed: 0
    Bootp flags: 0x8000 (Broadcast)
        1... .... .... .... = Broadcast flag: Broadcast
        .000 0000 0000 0000 = Reserved flags: 0x0000
    Client IP address: 0.0.0.0 (0.0.0.0)
    Your (client) IP address: 10.32.96.104 (10.32.96.104)
    Next server IP address: 10.32.33.40 (10.32.33.40)
    Relay agent IP address: 10.32.15.39 (10.32.15.39)
    Client MAC address: 70:79:b3:ff:51:42 (70:79:b3:ff:51:42)
    Client hardware address padding: 00000000000000000000
    Server host name not given
    Boot file name: /undionly.kpxe
    Magic cookie: (OK)
    Option: (t=53,l=1) DHCP Message Type = DHCP Offer
        Option: (53) DHCP Message Type
        Length: 1
        Value: 02
    Option: (t=54,l=4) DHCP Server Identifier = 10.32.96.1
        Option: (54) DHCP Server Identifier
        Length: 4
        Value: 0A206001
    Option: (t=51,l=4) IP Address Lease Time = 1 hour
        Option: (51) IP Address Lease Time
        Length: 4
        Value: 00000E10
    Option: (t=118,l=4) Subnet Selection Option = 10.32.96.0
        Option: (118) Subnet Selection Option
        Length: 4
        Value: 0A206000
    Option: (t=58,l=4) Renewal Time Value = 30 minutes
        Option: (58) Renewal Time Value
        Length: 4
        Value: 00000708
    Option: (t=59,l=4) Rebinding Time Value = 52 minutes, 30 seconds
        Option: (59) Rebinding Time Value
        Length: 4
        Value: 00000C4E
    Option: (t=1,l=4) Subnet Mask = 255.255.240.0
        Option: (1) Subnet Mask
        Length: 4
        Value: FFFFF000
    Option: (t=28,l=4) Broadcast Address = 10.32.111.255
        Option: (28) Broadcast Address
        Length: 4
        Value: 0A206FFF
    Option: (t=15,l=13) Domain Name = "myvrf.internal"
        Option: (15) Domain Name
        Length: 13
        Value: 6465762E692E657763732E6368
    Option: (t=6,l=8) Domain Name Server
        Option: (6) Domain Name Server
        Length: 8
        Value: 0A08034A0A0803AE
        IP Address: 10.8.3.74
        IP Address: 10.8.3.174
    Option: (t=3,l=4) Router = 10.32.96.1
        Option: (3) Router
        Length: 4
        Value: 0A206001
    Option: (t=82,l=48) Agent Information Option
        Option: (82) Agent Information Option
        Length: 48
        Value: 010409010E30020628AC9E65F84F9714004F5330312D5245...
        Agent Circuit ID: 09010E30
        Agent Remote ID: 28AC9E65F84F
        DHCPv4 Virtual Subnet Selection: 004F5330312D52454730312D444556312D5A3030
        Server Identifier Override: 0A206001
        Link selection: 10.32.96.0
    End Option

Upon this offer, the client sends its DHCP request which gets relayed to the Server...

The relayed DHCPREQUEST

The Nexus' DHCP relay sets 10.33.33.40 as Option 54 (Server Identifier). <--- but why?

Internet Protocol, Src: 10.32.15.39 (10.32.15.39), Dst: 10.32.33.40 (10.32.33.40)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
        0000 00.. = Differentiated Services Codepoint: Default (0x00)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
    Total Length: 401
    Identification: 0xdc6c (56428)
    Flags: 0x00
        0.. = Reserved bit: Not Set
        .0. = Don't fragment: Not Set
        ..0 = More fragments: Not Set
    Fragment offset: 0
    Time to live: 255
    Protocol: UDP (0x11)
    Header checksum: 0x9960 [correct]
        [Good: True]
        [Bad : False]
    Source: 10.32.15.39 (10.32.15.39)
    Destination: 10.32.33.40 (10.32.33.40)
User Datagram Protocol, Src Port: bootps (67), Dst Port: bootps (67)
    Source port: bootps (67)
    Destination port: bootps (67)
    Length: 381
    Checksum: 0xf675 [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
Bootstrap Protocol
    Message type: Boot Request (1)
    Hardware type: Ethernet
    Hardware address length: 6
    Hops: 1
    Transaction ID: 0x00002384
    Seconds elapsed: 0
    Bootp flags: 0x8000 (Broadcast)
        1... .... .... .... = Broadcast flag: Broadcast
        .000 0000 0000 0000 = Reserved flags: 0x0000
    Client IP address: 0.0.0.0 (0.0.0.0)
    Your (client) IP address: 0.0.0.0 (0.0.0.0)
    Next server IP address: 0.0.0.0 (0.0.0.0)
    Relay agent IP address: 10.32.15.39 (10.32.15.39)
    Client MAC address: 70:79:b3:ff:51:42 (70:79:b3:ff:51:42)
    Client hardware address padding: 00000000000000000000
    Server host name not given
    Boot file name not given
    Magic cookie: (OK)
    Option: (t=53,l=1) DHCP Message Type = DHCP Request
        Option: (53) DHCP Message Type
        Length: 1
        Value: 03
    Option: (t=57,l=2) Maximum DHCP Message Size = 1200
        Option: (57) Maximum DHCP Message Size
        Length: 2
        Value: 04B0
    Option: (t=61,l=25) Client identifier
        Option: (61) Client identifier
        Length: 25
        Value: 00636973636F2D373037392E623366662E353134322D4769...
    Option: (t=54,l=4) DHCP Server Identifier = 10.32.33.40
        Option: (54) DHCP Server Identifier
        Length: 4
        Value: 0A202128
    Option: (t=50,l=4) Requested IP Address = 10.32.96.104
        Option: (50) Requested IP Address
        Length: 4
        Value: 0A206068
    Option: (t=12,l=13) Host Name = "ExpDHCPClient"
        Option: (12) Host Name
        Length: 13
        Value: 45787044484350436C69656E74
    Option: (t=55,l=9) Parameter Request List
        Option: (55) Parameter Request List
        Length: 9
        Value: 01060F2C790321962B
        1 = Subnet Mask
        6 = Domain Name Server
        15 = Domain Name
        44 = NetBIOS over TCP/IP Name Server
        121 = Classless Static Route
        3 = Router
        33 = Static Route
        150 = TFTP server address
        43 = Vendor-Specific Information
    Option: (t=60,l=8) Vendor class identifier = "ciscopnp"
        Option: (60) Vendor class identifier
        Length: 8
        Value: 636973636F706E70
    Option: (t=82,l=48) Agent Information Option
        Option: (82) Agent Information Option
        Length: 48
        Value: 010409010E30020628AC9E65F84F9714004F5330312D5245...
        Agent Circuit ID: 09010E30
        Agent Remote ID: 28AC9E65F84F
        DHCPv4 Virtual Subnet Selection: 004F5330312D52454730312D444556312D5A3030
        Server Identifier Override: 0A206001
        Link selection: 10.32.96.0
    End Option

... and there is no DHCPACK from dnsmasq's DHCP service :-(

Please note that there is/was also a thread about this at

https://networkengineering.stackexchange.com/questions/64185/nxos-dhcp-relay-overwriting-option-54-server-identifier-differently-in-dhcpd

Marc Luethi · ‎01-10-2020

Possibly related:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCve17058

Marc Luethi · ‎01-28-2020

... and another thread from dnsmasq's Mailing List:

https://dnsmasq-discuss.thekelleys.org.narkive.com/jSbR3kHf/dnsmasq-not-sending-ack#post9

Quote (by dnsmasq's maintainer):

It would be worth reading RFC 5107 to get more background on this.
The idea is that the server-identifier IP address in the DHCP exchange
should be the IP address of the _relay_ and not the IP address of the
DHCP server. This ensures that _all_ DHCP exchanges take place through
the relay. (Otherwise, the server-id is the address of the server, and
when the client comes to renew the lease, it will unicast direct to the
server, bypassing the relay)

Note that this is sufficently unusual case that I certainly can't rule
out dnsmasq bugs, but it's worth looking at what the value of the
server-id options are in all the packets, and what your relay is adding
as option 82 - suboption 11. These should be the same and be the address
of the relay.

And the symptoms described by the user are the same: In the relayed DHCPREQUEST, Options 54 and 82.11 are not identical, but RFC 5107 seems to mandate that they are.

With dnsmasq, if they're not identical, the relayed DHCPREQUEST will be dropped and no DHCPACK is ever given back.

best regards

Marc

Doug Goldstein · ‎05-16-2024

@Marc Luethi were you able to come up with a solution for this? My understanding is the same as yours that Option 54 and Option 82.11 should be identical values per RFC 5107 but they are not in any configuration I've crafted.

MHM Cisco World · ‎05-16-2024

@Doug Goldstein make new post it better

MHM