Solved: Re: DHCP relay issue on Nexus 3k when used with vrf

Andreas Schneider · ‎03-22-2020

Hello,

I have a strange behavior on a Nexus 3k (3064) running "NXOS: version 7.0(3)I7(6)".

I have 2960-X(for the clients) connected via port-channel (and trunk) to two N3K (vpc on the Nexus side) - just L2.

The svi of the "access-vlan 3080" is configured with hsrp on the N3K, and also the dhcp-relay agent. I had to put the svi in a vrf called "Printer".

I can ping the DHCP-Server from the svi, and svi from the DHCP server. The thing is, if the svi is in a vrf(no matter which one) the client didn't get an IP-Address. If I put the svi in the default vrf, everything works fine.

The debug on the N3K shows the DHCPOFFER from the DHCP server, but the OFFER doesn't make it to the client.

Here is the config from the svi:

interface Vlan3080
  description Printer_Access
  vrf member Printer  
  no shutdown
  ip address 172.16.231.2/24
  ip router ospf 3080 area 0.0.0.20
  hsrp version 2
  hsrp 3080
    preempt
    priority 105
    ip 172.16.231.1
  ip dhcp relay address 10.67.244.30
  ip dhcp relay address 10.67.248.30

Attached is the debug-output from the N3K.

Kind regards,

Andreas

Cristian Matei · ‎03-26-2020

Hi,

There is something different, which could make all the difference, exactly between working and non-working. It seems that the Nexus behaves differently when running the DHCP Relay functionality in default VRF vs. user-defined VRF. In the global VRF, it uses source port number of 67 (how the plain old DHCP RFC was stating), while in the user VRF it uses a random source port number of 24528 in this case (how DHCP enhancements via RFC's allow DHCP to function, like RFC 8537). I suspect that your printers reject the DHCP Offer just because of seeing a different source port number than 67. If this is the case, nobody is to blame, there is just an incompatibility, but mostly because that may be an old printer, which clearly behaves like an old printer, with old habits?

You said you were able to get an address for another non-printer device, both in the global GRT and the VRF, right? If yes, you'll just have to use different printers, or ask the owner to perform some upgrades, maybe it will fix it, to be less strict.

Regards,

Cristian Matei.

View solution in original post

marce1000 · ‎03-22-2020

- You may want this thread and or some advisories from it useful :

https://community.cisco.com/t5/switching/dhcp-relay-issue-with-nexus-3k/td-p/3375639

M.

-- ' 'Good body every evening' ' this sentence was once spotted on a logo at the entrance of a Weight Watchers Club !

Andreas Schneider · ‎03-22-2020

Hi marce1000,

I know this thread, but it has nothing to do with vrf, I think.

Kind regards,

Andreas

balaji.bandi · ‎03-22-2020

Like to see the 2960 config along with nexus side port-channel and interface config.

DHCP relay guide lines :

DHCP relay agent will always use primary VIP address to communicate with DHCP server. DHCP relay agent does not consider use of secondary VIP addresses as long as primary VIP is available
DHCP relay agent behavior in case inter-vrf is different and requires use of Option-82 information in DHCP packets. DHCP server and clients will be in the same VRF and use of VIP is not supported for inter-vrf relay.

quick test like to do to confirm :

if theL2 switch has dual uplink, shutdown one of the uplink and let us know is the same behaviour ?

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

Andreas Schneider · ‎03-23-2020

Hi,

this is the port-channel configuration from the 2960:

interface Port-channel3
 description L2-rz1/2-nx3k
 switchport trunk allowed vlan 330,2118,2174,3040,3080
 switchport mode trunk
 switchport nonegotiate
end
!
interface TenGigabitEthernet1/0/1
 description L2-rz1-nx3k-e1/23
 switchport trunk allowed vlan 330,2118,2174,3040,3080
 switchport mode trunk
 switchport nonegotiate
 channel-group 3 mode active
end
!
interface TenGigabitEthernet2/0/2
 description L2-rz2-nx3k-e1/23
 switchport trunk allowed vlan 330,2118,2174,3040,3080
 switchport mode trunk
 switchport nonegotiate
 channel-group 3 mode active
end

and this is the Nexus

RZ1:
interface port-channel7
  description 2960-X
  switchport mode trunk
  switchport trunk allowed vlan 330,2118,2174,3040,3080
  speed 10000
  vpc 7
!
interface Ethernet1/23
  description 2960-X-t1/0/1
  switchport mode trunk
  switchport trunk allowed vlan 330,2118,2174,3040,3080
  channel-group 7 mode active

RZ2:
interface port-channel7
  description 2960-X
  switchport mode trunk
  switchport trunk allowed vlan 330,2118,2174,3040,3080
  speed 10000
  vpc 7
!
interface Ethernet1/23
  description 2960-X-Ten2/0/2
  switchport mode trunk
  switchport trunk allowed vlan 330,2118,2174,3040,3080
  channel-group 7 mode active

I already tested to switch off on leg on the 2960-X (also without hsrp) - no difference.

Kind regards,

Andreas

balaji.bandi · ‎03-23-2020

i will review the config and let you know my findings, other test, what if the device conneced to Nexus access port configure access VLAN that port, is the DHCP offering correct ?

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

Andreas Schneider · ‎03-23-2020

Hi,

infortunately this is a remote site, no chance to connect a client directly to the Nexus, at this moment.

Kind regards,

Andreas

paul driver · ‎03-23-2020

Hello
Append the follwing to the helper addressing , test again

int x/x/
vrf memebr Printer
ip dhcp relay address x.x.x.x use-vrf Printer

Please rate and mark as an accepted solution if you have found any of the information provided useful.
This then could assist others on these forums to find a valuable answer and broadens the community’s global network.

Kind Regards
Paul

Andreas Schneider · ‎03-23-2020

Hello Paul,

I already added the "use-vrf Printer" command, but no success. When I configure with "ip dhcp relay address x.x.x.x use-vrf Printer" it is not shown in the running config - only "ip dhcp relay address x.x.x.x".

Only when using a different vrf in this command, it is shown in the running config.

Kind regards,

Andreas

Cristian Matei · ‎03-23-2020

Hi,

Is the DHCP server in the same VRF as the clients/DHCP Relay Agent. You may need to configure "ip DHCP relay information option vpn".

Regards,

Cristian Matei.

Andreas Schneider · ‎03-23-2020

Hi,

the DHCP server is on a different vrf. I don't do "inter vrf routing". All my vrf's are connected to a Firewall, for now the FW is configured with "any any"-rule for this vrf. So, for the DHCP server it should just be a further scope.

This all works fine if the N3K is not the L3 relay interface. I testet the "option vpn" already, but no luck. So I hoped I'm missing a piece of configuration on the N3K.

This is the L3 interface config on the N3K:

interface Vlan3080
  description Printer_Access
  no shutdown
  ip address 172.16.231.2/24
  ip router ospf 1 area 0.0.0.20
  hsrp version 2
  hsrp 3080
    preempt
    priority 105
    ip 172.16.231.1
  ip dhcp relay address 10.67.244.30
  ip dhcp relay address 10.67.248.30

Kind regards,

Andreas

Cristian Matei · ‎03-23-2020

Hi,

Is the posted "debug" output and the below configuration from the same device? Most probably the debug output is from he other device. From the DHCP process point go view, everything looks normal, Nexus receives the DHCP Discovery, builds the DHCP Relay, receives the DHCP Offer and presents it to the client. But remember, this is from the DHCP point of view, it doesn't necessarily mean the packet LEAVES the Nexus. Can you do a packet capture on the client side and post the output to see if it receives any DHCP info from the DHCP Relay Agent? You could also do a debug on the Nexus, to see if the offer being relayed top the client, actually leaves the Nexus:

https://www.cisco.com/c/en/us/support/docs/switches/nexus-7000-series-switches/116136-trouble-ethanalyzer-nexus7000-00.html#anc11

Discovery:

2020 Mar 22 15:34:16.295657 dhcp_snoop: (PKT) DHCPDISCOVER on Intf Vlan3080(6), phy port-channel12(148), vlan 3080, vni 3080, pvlan 3080, vdc 0, vrf Printer

2020 Mar 22 15:34:16.295671 dhcp_snoop: (PKT)Packet: Len 390, l2 HDR 14, DHCP data 348, hdr flag 0x36

2020 Mar 22 15:34:16.295690 dhcp_snoop: (PKT)L2: DMAC ff ff ff ff ff ff , SMAC 00 21 b7 f5 1b 84 , PROTO 0x8

2020 Mar 22 15:34:16.295708 dhcp_snoop: (PKT)IP: DIP 255.255.255.255, SIP 0.0.0.0, tl 376, protocol 17, ttl 64

Relay to First DHCP Server:

2020 Mar 22 15:34:16.296013 dhcp_snoop: (PKT)source-intf is Vlan3080 vrf Printer

2020 Mar 22 15:34:16.296084 dhcp_snoop: (PKT)Client and Server are in the same VRF

2020 Mar 22 15:34:16.296070 dhcp_snoop: (PKT)Got src intf address 172.16.231.3

2020 Mar 22 15:34:16.296113 dhcp_snoop: (PKT)RELAYING to server 10.67.244.30

2020 Mar 22 15:34:16.296127 dhcp_snoop: (PKT)IP: DIP 10.67.244.30, SIP 172.16.231.3, tl 30721, protocol 17, ttl 255

2020 Mar 22 15:34:16.296140 dhcp_snoop: (PKT)UDP: sport 68, dport 67, len 356

2020 Mar 22 15:34:16.296158 dhcp_snoop: (PKT)DHCP HDR:op 1, htype 1, hlen 6, hops 1, xid 0x311201c5, secs 0, flags 0x0, ciaddr 0.0.0.0, yiaddr 0.0.0.0, siaddr 0.0.0.0, giaddr 172.16.231.3, chaddr 00 21 b7 f5 1b 84 , sname , file

Relay to Second DHCP Server:

2020 Mar 22 15:34:16.296364 dhcp_snoop: (PKT)source-intf is Vlan3080 vrf Printer

2020 Mar 22 15:34:16.296412 dhcp_snoop: (PKT)Got src intf address 172.16.231.3

2020 Mar 22 15:34:16.296423 dhcp_snoop: (PKT)Client and Server are in the same VRF

2020 Mar 22 15:34:16.296447 dhcp_snoop: (PKT)RELAYING to server 10.67.248.30

2020 Mar 22 15:34:16.296460 dhcp_snoop: (PKT)IP: DIP 10.67.248.30, SIP 172.16.231.3, tl 30721, protocol 17, ttl 255

2020 Mar 22 15:34:16.296471 dhcp_snoop: (PKT)UDP: sport 68, dport 67, len 356

2020 Mar 22 15:34:16.296489 dhcp_snoop: (PKT)DHCP HDR:op 1, htype 1, hlen 6, hops 1, xid 0x311201c5, secs 0, flags 0x0, ciaddr 0.0.0.0, yiaddr 0.0.0.0, siaddr 0.0.0.0, giaddr 172.16.231.3, chaddr 00 21 b7 f5 1b 84 , sname , file

Received DHCP Offer from one server:

2020 Mar 22 15:34:16.342665 dhcp_snoop: (PKT)Received ip pkt 0x10dcb6a4 of len 399 in vrf Printer(7) on intf Vlan3080 (6) sw_bd:3080 vni:0

2020 Mar 22 15:34:16.342694 dhcp_snoop: (PKT)RECIEVED packet type DHCP RELAY

2020 Mar 22 15:34:16.342746 dhcp_snoop: (PKT)L2: DMAC 00 00 00 00 00 00 , SMAC 00 00 00 00 00 00 , PROTO 0x8

2020 Mar 22 15:34:16.342764 dhcp_snoop: (PKT)IP: DIP 172.16.231.3, SIP 10.67.244.30, tl 31489, protocol 17, ttl 61

2020 Mar 22 15:34:16.342777 dhcp_snoop: (PKT)UDP: sport 40426, dport 67, len 379

2020 Mar 22 15:34:16.342796 dhcp_snoop: (PKT)DHCP HDR:op 2, htype 1, hlen 6, hops 0, xid 0x311201c5, secs 0, flags 0x0, ciaddr 0.0.0.0, yiaddr 172.16.231.9, siaddr 10.67.244.30, giaddr 172.16.231.3, chaddr 00 21 b7 f5 1b 84 , sname , file

DHCP Offer relayed to the DHCP Client:

2020 Mar 22 15:34:16.346223 dhcp_snoop: (PKT)TRANSMITTED

2020 Mar 22 15:34:16.346264 dhcp_snoop: (PKT)Packet: Len 413, l2 HDR 14, DHCP data 371, hdr flag 0x36

2020 Mar 22 15:34:16.346281 dhcp_snoop: (PKT)L2: DMAC 00 21 b7 f5 1b 84 , SMAC 54 7f ee 5c 1c bc , PROTO 0x8

2020 Mar 22 15:34:16.346296 dhcp_snoop: (PKT)IP: DIP 172.16.231.9, SIP 172.16.231.3, tl 399, protocol 17, ttl 255

2020 Mar 22 15:34:16.346306 dhcp_snoop: (PKT)UDP: sport 40426, dport 68, len 379

2020 Mar 22 15:34:16.346325 dhcp_snoop: (PKT)DHCP HDR:op 2, htype 1, hlen 6, hops 1, xid 0x311201c5, secs 0, flags 0x0, ciaddr 0.0.0.0, yiaddr 172.16.231.9, siaddr 10.67.244.30, giaddr 172.16.231.3, chaddr 00 21 b7 f5 1b 84 , sname , file

Regards,

Cristian Matei.

Andreas Schneider · ‎03-23-2020

Hi,

the debug is from the Nexus. There is no debug from the 2960-X, it's just L2. All the dhcp relay things are done on the Nexus.

Unfortunately this is at a remote site. No chance to get a trace at the access-port of the 2960-X. But I did the ethanalyzer on the Nexus, but get a permission denied if want to copy it away from the Nexus. Only a "ethanalyzer loca read" is possible.

I think this is how it should look like:

2020-03-23 15:53:40.897838 172.16.231.2 -> 10.67.244.30 DHCP DHCP Request - Transaction ID 0x3acaef97
2020-03-23 15:53:40.897995 172.16.231.2 -> 10.67.248.30 DHCP DHCP Request - Transaction ID 0x3acaef97
2020-03-23 15:53:40.961758 172.16.231.2 -> 172.16.231.132 DHCP DHCP ACK - Transaction ID 0x3acaef97
2020-03-23 15:53:44.776327 172.16.231.2 -> 10.67.244.30 DHCP DHCP Request - Transaction ID 0x3acaef97
2020-03-23 15:53:44.776432 172.16.231.2 -> 10.67.248.30 DHCP DHCP Request - Transaction ID 0x3acaef97
2020-03-23 15:53:44.787666 172.16.231.2 -> 172.16.231.132 DHCP DHCP ACK - Transaction ID 0x3acaef97
2020-03-23 15:53:49.280718 172.16.231.2 -> 10.67.244.30 DHCP DHCP Request - Transaction ID 0x3acaef97
2020-03-23 15:53:49.280804 172.16.231.2 -> 10.67.248.30 DHCP DHCP Request - Transaction ID 0x3acaef97
2020-03-23 15:53:49.291892 172.16.231.2 -> 172.16.231.132 DHCP DHCP ACK - Transaction ID 0x3acaef97
2020-03-23 15:53:57.888807 172.16.231.2 -> 10.67.244.30 DHCP DHCP Request - Transaction ID 0x3acaef97

But it I get this:

2020-03-23 16:07:35.881132 172.16.231.2 -> 10.67.244.30 DHCP DHCP Discover - Transaction ID 0x5d0601d5
2020-03-23 16:07:35.881197 172.16.231.2 -> 10.67.248.30 DHCP DHCP Discover - Transaction ID 0x5d0601d5
2020-03-23 16:07:35.926288 172.16.231.2 -> 172.16.231.9 DHCP DHCP Offer - Transaction ID 0x5d0601d5
2020-03-23 16:07:39.941124 172.16.231.2 -> 10.67.244.30 DHCP DHCP Discover - Transaction ID 0x5d0601d5
2020-03-23 16:07:39.941278 172.16.231.2 -> 10.67.248.30 DHCP DHCP Discover - Transaction ID 0x5d0601d5
2020-03-23 16:07:39.952323 172.16.231.2 -> 172.16.231.9 DHCP DHCP Offer - Transaction ID 0x5d0601d5
2020-03-23 16:07:48.002121 172.16.231.2 -> 10.67.244.30 DHCP DHCP Discover - Transaction ID 0x5d0601d5
2020-03-23 16:07:48.002223 172.16.231.2 -> 10.67.248.30 DHCP DHCP Discover - Transaction ID 0x5d0601d5
2020-03-23 16:07:48.009290 172.16.231.2 -> 172.16.231.9 DHCP DHCP Offer - Transaction ID 0x5d0601d5
2020-03-23 16:07:48.028619 172.16.231.2 -> 172.16.231.132 DHCP DHCP Offer - Transaction ID 0x5d0601d5

The DHCP OFFER reaches the Nexus, but I don't see any ACK's ...

One strange thing. If I configure a svi on the 2960-X (int vlan 3080) with "ip address dhcp", it gets an IP without any problem.

Kind regards,

Andreas

Cristian Matei · ‎03-23-2020

Hi,

You said there are two Nexus devices with HSRP. You posted the config from one with IP address of 172.16.231.2, while in the DHCP debugs, DHCP traffic was destined/sourced from 172.16.231.3.

2020 Mar 22 15:34:16.342694 dhcp_snoop: (PKT)RECIEVED packet type DHCP RELAY

2020 Mar 22 15:34:16.342746 dhcp_snoop: (PKT)L2: DMAC 00 00 00 00 00 00 , SMAC 00 00 00 00 00 00 , PROTO 0x8

2020 Mar 22 15:34:16.342764 dhcp_snoop: (PKT)IP: DIP 172.16.231.3, SIP 10.67.244.30, tl 31489, protocol 17, ttl 61

2020 Mar 22 15:34:16.346296 dhcp_snoop: (PKT)IP: DIP 172.16.231.9, SIP 172.16.231.3, tl 399, protocol 17, ttl 255

The fact that the 2960-x downstream switch can get an IP address with Nexus being the DHCP Relay, confirms that Nexus actually sends the DHCP Offer received from the DHCP server, towards the client, but something happens afterwards:

- either it does not leave Nexus (but if this would be the case, 2960-x could not get an IP)

- there is a layer2 forwarding problem somewhere and the DHCP client actually does not get the DHCP Offer

- the DHCP client receives the DHCP Offer, but it doesn't like it, or its DHCP Request does not reach Nexus (perform a packet capture on the remote client).

Regards,

Cristian Matei.

Andreas Schneider · ‎03-24-2020

Hi,

yes, one Nexus has the ".2", the other has the ".3" and the hsrp address is the ".1".

I think the last one:

"- the DHCP client receives the DHCP Offer, but it doesn't like it, or its DHCP Request does not reach Nexus (perform a packet capture on the remote client)."

is the most likely. I guess that the client doesn't like the DHCP offer.

But running the same configuration in the default vrf, everything works fine. It shouldn't make any difference, right?

I'll try to take a packet capture at the remote site.

Kind regards,

Andreas