ACI DHCP Relay discards recieved DHCP Offers that have option 82 set correctly

[petar.forai1]

Hi,

We're hitting an issue with a DHCP relay and an shared L3outEPG config that uses an external Infoblox DHCP server. The DHCP server is correctly interpreting option 82 magic that ACI needs (like the agent suboption that contains the DHCP scope to select - there's only one document detailing DHCP config from Tomas De Leon that basically explains everything).

My EP sends out the DISCOVER, the server answers with the correct IP from the scope.

Checking for know bugs, we've discovered that 2.0(1o) has an issue with dropping DHCP packets that have broadcast flag set. I have tried with bootp option broadcast and unicast - but that doesnt seem to make any difference.  It seems that the agent swallows the DHCP offer and the client doesn't proceed.

I have attached the DHCP server's point of view detailing the option 82 handling. 

We're kinda stuck understanding the DHCP relay agent errors and traces. It's not clear what the explicit reason of the nonprocessing of the offer is.

We have a TAC open for this.

best,

P

[Tomas de Leon]

Petar,

Jessica will help you resolve this issue.  I updated the CDET  that you mentioned so there is no misunderstanding with that issue.  the issue has not been resolved of root cause determined yet but this is what "I" know at this time.

The issue with DHCP-Relay and the broadcast bit being set in the DHCP Request & Responses appears to be related to specific ACI Hardware (i.e. the sugarbowl leaf).

I have tested with other non sugar bowl leaf nodes without any issues.
Tests have included 9396 & 93128 models with a internal fabric topology and external L3 Out topology. In my test topology DHCP-Relay works as expected for DHCP Request & Responses with the UNICAST & BROADCAST bit set.

This issue appears to be related to specific hardware.

Tested in versions:

2.0(1p), 2.0(1.112a)

In regards to your setup, I looked at the trace and something does not look right in your topology based on what i see in the trace.  Can you please update your case with a picture of the topology and reference the client VRFs and the Server VRF.  Also, Please add the SERVER's IP Gateway (i.e.. the BD\SVI gateway on which the Server resides)

Thanks

T.

[petar.forai1]

Hi Tomas,

I've attached the drawing. Hope it's clear how things are tied together.

[Tomas de Leon]

The wireshark trace and the show ip dhcp internal event history traces that you provided in the Cisco Tac Case was mismatched.  What I mean is that the samples were not the same client request for each sample set.

Please provide the following output to the Cisco Tac Case:

# acidiag fnvread
(please denote which leaf node is the location of the Endpoint and the location of the shared L30ut)

From the Service Leaf on where the CLIENT Endpoint resides:
# show vrf
# show ip interface brief vrf NAME <NAME = VRF NAME where the client EPG resides>
# show ip interface vlan## <## = VLAN number of the client EPG>
# show ip dhcp relay
# show dhcp internal event-history traces
# vsh_lc
module-1# show system internal epmc endpoint vlan ##
module-1# show system internal epmc endpoint vlan ## | egrep "IP|MAC|Vlan|VRF|phy"
(where ## is the VLAN number of the DHCP Client EPG)

From the Border Leaf on where the L3Out for DHCP Server resides:
# show dhcp internal event-history traces
# show vrf
# show ip interface brief vrf NAME <NAME = VRF NAME where the L3Out resides>
# show ip interface vlan## <## = VLAN number of VRF interface for L3Out>

Thanks

T.

[petar.forai1]

Added the logs to the TAC case. I would like to share relevant info with the people reading this forum in case anybody else runs into this issue, so I omitted the fnvread output as this contains hw chassis serials.

best,

P

[Tomas de Leon]

Ok,  I may have an idea of what the issue may be. On aci-edge-gw-7 & aci-edge-gw-8 in "IMP-IMBA_Private:HPC-mgmt", How do you have the EPG configured for the DHCP Clients?

What I mean by this is do you have a static path or static binding configured?  If so, what mode is it configured as?

From the trace, it appears that eth 1/21 is configured as "802.1P access mode"? Is that correct?

127) 2016 Aug 11 16:43:58.388322 _snoop_handle_istack_packet: 1359 : L2 type = 1; Private VLAN = 0

My understanding is that some PXEBOOT nics do not like seeing and do not work with vlan0.  In some cases, the configuration needs to be configured as "untagged"

Please let us know how your connections are to the device of eth 1/21.

Thanks!

T.

[petar.forai1]

Hi Tomas!

I do not have static paths configured within that EPG - I have just the domain associated with the EPG. 

I do have 802.1P access mode configured. I booted the server and tried a manual DHCP client call with the same results as for PXE. 

Adding paths to the EPG with 802.1P access OR untagged config for the path doesnt change things. The IP and tagging for the box is correctly set as I have fully working IP connectivity to the DHCP server (but no DHCP protocol working as the agent is still dropping?) as soon as I manually configure the server's interface.

[petar.forai1]

Jessica was amazing! 

We didn't truly solve the problem yet, but have a workaround:

The border leaf switches didn't have the DHCP relay address showing in the output of show ip dhcp relay. What we did now is we deployed two static paths where nothing is connected to within the EPG that contains the server with immediate deployment policy. Running show ip dhcp relay after that did show the relay IP on the border leafs. We do have regular DHCP working now while the OS is booted but are still troubleshooting VIC1387 PXE DHCP. We confirmed that broadcast bootp option is not triggering the dropping on our hw platform.

[petar.forai1]

One thing I noticed is that the border leaf that doesn't have the sever attached, but has the L3extOut EPG (shared L3 out) doesnt have the relay set, the leaf switches were the servers are attached do have it configured and it shows in the output show ip dhcp relay.

Is that supposed to be like this? 

Looking at the routes within overlay-1 VRF I don't see the relay's IP address being present