DHCP Requests over Nexus Issue

[Michael Morrison]

Hello all, this is my first post so please be gentle.  I am having an issue with DHCP requests on our new server network.

First some background:

The new network consists of 2x Nexus 5548UP runing NX-OS 6.0.  Behind the N5ks are multiple FEX types, but for the purpose of this issue I am using 2232TM-E FEXs.  Each FEX is setup in an active-active scenario, therefore each individual FEX port belongs to each N5k which are configured identically.  The N5k have a vPC peer-link setup between them and each FEX is also setup with vPC.  The N5ks are uplinked to a pair of Catalyst 4500X switches that function as our "core" switches.  From there, we have our old network which consists of a pair of Catalyst 4507R switches.  We use tansfer networks between the N5k and 4500X and between the 4500X and 4507R runing EIGRP.

I have "Server A" on VLAN 100 connected to FEX 2232TM-E... "DHCP Server B" is on VLAN 11 connected to 4507R.  I have "ip dhcp relay address x.x.x.x" configured on VLAN 100 to point to DHCP Server B on the old network.  VLAN 100 is configured with HSRP (see running-config output below).

Onto the issue:

Server A is not getting a DHCP addesses assigned to him.  The DHCP Server B gets the Discover packet from Server A and sends back the Offer... but the Offer either never makes it to Server A, or Server A Request never makes it back the DHCP Server B.  I am not sure which is the case exactly.

I know it may be hard to picture the whole scenario I described above, but any pointers would be greatly appreciated.  I'll be glad to dig up a good diagram of the network or provide more configs if necessary.

Additional info:

version 6.0(2)N1(2)

feature dhcp

service dhcp

ip dhcp relay

N5k #1:

interface Vlan100

   description server vlan

   no shutdown

   no ip redirects

   ip address 10.112.100.2/23

   hsrp 100

     authentication text hsrp-100

     preempt delay minimum 180

     priority 200

     ip 10.112.100.1

   ip dhcp relay address 172.26.20.40

N5k #2:

interface Vlan100

   description server vlan

   no shutdown

   no ip redirects

   ip address 10.112.100.3/23

   hsrp 100

     authentication text hsrp-100

     preempt delay minimum 180

     priority 150

     ip 10.112.100.1

   ip dhcp relay address 172.26.20.40

Thanks for any information!

[Richard Burts]

As a starting point can you verify that the N5K has a correct route to network 172.26.20? And can you verify that the server and its 4507R have a correct route to network 10.112.100?

HTH

Rick

[Michael Morrison]

Here are the results of a tracert from a server in VLAN 11 to the gateway for VLAN 100:

K:\>tracert -d 10.112.100.1

Tracing route to 10.112.100.1 over a maximum of 30 hops

   1    1 ms    1 ms    <1 ms  172.26.20.243

   2    <1 ms    7 ms    4 ms  10.112.96.105

   3    <1 ms    2 ms    2 ms  10.112.100.1

Trace complete.

Here is a tracert from a server in VLAN 100 (staticly addressed) to the gateway for VLAN 11:

K:\>tracert -d 172.26.20.1

Tracing route to 172.26.20.1 over a maximum of 30 hops

   1    <1 ms    2 ms    2 ms  10.112.100.3

   2    <1 ms    6 ms    7 ms  10.112.96.73

   3    1 ms    <1 ms    <1 ms  172.26.20.1

Trace complete.

Does this help?  Or do you need more specific switch config information?

Also, I've attached a basic drawing of the network for reference.  I've ommitted all the IP information, just a visual representation of the connectivity.

[Richard Burts]

Thanks for this information. It does show that there is IP connectivity between the networks.

My next question would be whether you can verify that a correct DHCP scope has been configured on the server for network 10.112.100?

HTH

Rick

[Michael Morrison]

The DHCP server is setup correctly to the best of our knowledge.  I had 2 others in my organization check and verify the DHCP scope, one of these persons is our global DHCP admin and that's all he does each day.

Here is some strange behavior I noticed that may confirm the DHCP scope works, as well as shed some more light on the problem...

I connected a laptop to the same Nexus switch as the server having the issue (for reference all FEXs exhibit this issue, it's not specific to one FEX).  The laptop has both LAN and wireless connectivity, although I had the wireless card disabled when I first connected the LAN port.  The laptop would not grab an IP address when connected only to the LAN port.  The status of the LAN NIC was showing bytes sent, but no bytes received... which indicates to me that the DHCP Offer is not getting back to the laptop from the DHCP server.  What's strange is that at the very moment I enabled the wireless card and got a wireless DHCP address, the LAN NIC suddenly lit up and grabbed an IP from the 10.112.100.0 network.  So the offer from the DHCP server was received via the wireless connection, but not through the LAN.

That indicates 2 things to me.  A)  the DHCP should be configured properly since the laptop did get a correct address.  B)  the DHCP offer from the server is not reaching the Nexus FEX port.

Please let me know what else I can provide, this issue has me positivly stumped!

Thanks so far for the feedback!!!

[Michael Morrison]

Just another small bit of information... I tried connecting the server directly to the N5k to make sure the issue was not with the FEX vPC link... still the same behavior when directly connected to the N5k.

[paul driver]

hello

Looks like you can ping the G/W address for the dhcp vlan but can you ping the dhcp server? - I assume you can given that you eventually are recivieng dhcp allocation?

On the host do you have any software FW enabled?
Do you have dhcp snooping enabled?

res

Paul

Please don't forget to rate any posts that have been helpful.

Thanks.

[Michael Morrison]

I am able to ping directly to the DHCP server on VLAN 11 from the statically addressed server in VLAN 100.  No problem there.

The host having the issue does not have an OS installed yet, it is actually trying to PXE boot, so no issues with firewalls or software interference.  To ensure the issue wasn't just PXE related, I also tried with my laptop and a spare laptop to get a DHCP address on VLAN 100 and both failed in the same way mentioned above.

DHCP snooping is not enabled on VLAN 100.  From what I read about DHCP snooping, it is a nice feature, but not necessary for DHCP allocation.  Regardless, I did try turning on DHCP snooping ad playing around with a few of it's options to no avail.  I have since removed the DHCP snooping config since it did not help.

I guess I should also mention that we have another VLAN 98 on the Nexus switches that we use for our server iLo connections.  This VLAN / DHCP scope is working perfectly!  No issues with iLos grabbing a DHCP address on VLAN 98.  I compared the DHCP scopes for network 10.112.98.0 and 10.112.100.0 and they are identical.  Every option is setup the same with the exception of the network addresses of course and the "003 Router" DHCP option which each point to thier corresponding gateway.

[Michael Morrison]

Ok,  DHCP from laptops seems to be working fine now.  I do not honestly know what I may have changed or tweaked that resolved that issue, but DHCP appears to be fine.  Now the issue is related solely to PXE booting.  Getting PXE-E51 error "No DCHP or proxyDHCP offers were recieved."

Any additional thoughts?  PXE / OSD works fine from the old server network so I know that environment is OK, it's just not PXE booting from the new server network.

[Richard Burts]

I am glad that you have managed to get DHCP working through the Nexus and the FEX. If DHCP is clearly working I am a bit puzzled that PXE would report no DHCP offer received. I wonder if the issue is with some of the options that are used with PXE and not with regular DHCP.

I found this discussion of PXE and Nexus. Perhaps one of the alternatives discussed might help you.

https://supportforums.cisco.com/thread/2112756

HTH

Rick

[Michael Morrison]

Thanks Richard, there was some good info in that discussion, but still no luck.

I added the SCCM server (the PXE server) to the DHCP relay addresses with no change to the behavior still.  I also tried a few other commands such as "ip dhcp relay subnet-broadcast" and "ip dhcp relay address 172.26.21.255" and finally "ip dhcp relay address 10.112.101.255".  Still no luck.

At least the issue is narrowed down to PXE and not DHCP... although I'm not sure if that makes things any easier to troubleshoot, haha.

I will be sure to keep this discussion updated with any new information and give credit when it's resolved of course.  Thanks for the help so far!

[Michael Morrison]

One more thing that may be important to note.  I ran ethanalyzer on the N5k switches to view the traffic during a sucessful DHCP request and then during an unsucessful PXE boot.

I noticed that the sucessful DHCP Discover process was using "broadcast" packets, while the unsucessful PXE Discover process was using "unicast" packets.  Could this be an indication why the PXE process is failing? 

[Richard Burts · Accepted solution]

That is certainly an interesting and possibly very helpful observation. When the server is in a remote subnet we need to be able to forward the requests to that subnet. When the request is sent as broadcast there is the function to forward them to addresses (or subnets) that are remote. When the request is unicast I do not know of a way to forward those. So what unicast address is being used in the PXE requests?

HTH

Rick

[Michael Morrison]

One correction, I had that backwards.  Normal DHCP request was Unicast is sucessful...  PXE boot DHCP request was Broadcast and was unsucessful.

The DHCP Discover packet that shows Unicast has the following info:

Bootp flags:  0x0000 (Unicast)

Client IP address:  0.0.0.0 (0.0.0.0)

Your (client) IP address:  0.0.0.0 (0.0.0.0)

Next server IP address:  0.0.0.0 (0.0.0.0)

Relay agent IP address:  10.112.100.3 (10.112.100.3)

The PXE DHCP Discover packet that shows Broadcast has the following info:

Bootp flags:  0x8000 (Broadcast)

Client IP address:  0.0.0.0 (0.0.0.0)

Your (client) IP address:  0.0.0.0 (0.0.0.0)

Next server IP address:  0.0.0.0 (0.0.0.0)

Relay agent IP address:  10.112.100.3 (10.112.100.3)

10.112.100.3 is the VLAN 100 address of one of the Nexus 5548 switches, 10.112.100.2 is the other Nexus switch and they have HSRP betweeen then with address 10.112.100.1.

[paul driver]

Hello

Is  proxy ARP disabled?

res

Paul

Please don't forget to rate any posts that have been helpful.

Thanks.