PXE DHCP issues in ACI

[mmacdonald70]

We have been trying to troubleshoot an issue with some servers for some time.  Just wondering if anybody else came across anything similar.

We have a number of blades with Mellenox ConnectX-3 40G servers that connect directly into 9332 leaf nodes.  We have an EBP configured and these ports are connected to it in Access (802.1P) Mode.

When we attempt a PXEboot from these NICS, we see the DHCP discover and span the EPG, we see the DHCPDISCOVER and the DHCPOFFER but nothing else.  If we boot into Linux and run dhclient, we obtain a proper IP address and we see all the DHCP packets in the trace.

The PXEboot also works with a different brand of NIC

I am suspecting an issue with the NÌC card but I was just wondering if anybody has anything that I should look at or if ACI would be mangling packets in a way that the PXE client might have an issue with.

[mmacdonald70]

In case anybody else comes across this issue, 802.1P access mode seems to send packets with a Vlan tag of 0.  Some PXE clients (including this one) don`t like that too much.  When I changed the port to untagged, everything was perfect.

[renekuiper83]

we're about to start using pxe boot for some clients in our ACI fabric and we are facing the same issue.. as allmost all of our hosts are esx with all traffic tagged we're considering creating a dedicated pxe (untagged) vlan for initial setup..

[mmacdonald70]

That's what we had to do.  The original design had us extending an EPG through a L2 out to an existing PXEboot lan in our legacy network.  The issue that we came accross was that we were unable to use 802.1P since the PXE clients didn't like it, and if we selected untagged, the L2 out wouldn't work.  The fix was to create a new PXEboot lan in ACI and route it out a L3 out.

I did notice that one of the changes in 1.3 is that 802.1p now sends packets untagged