VPC with HSRP and DHCP Relay

[j-mccarthy]

Hi

I have a pair of 7ks acting as layer 3 gateway for voice and data clients. Multiple stacks of 3750s are dual homed to the pair of 7ks in a VPC.

DHCP relay is configured on each 7k client vlan interface, pointing to the same DHCP server in the datacentre. The links between the N7K VPC pair and the datacentre are routed (EIGRP).

The problem is if the DHCP request is received on N7K-2 and the reply is routed back via N7K-1, N7K-1 is dropping the packet and client never gets an address, because it won't forward data across the peer link.

Similarly if the DHCP request is received on N7K-1 and the reply is routed back via N7K-2, N7K-2 is dropping the reply.

Any ideas ?? !! 

[Amit Singh]

Hi,

Could you please let us know the NX-OS version that you are running on N7K's. That's a normal VPC feature that if a peer switch gets a traffic over peer-link, it doesnot forward the traffic to other ports. The newer releases have also addressed few DHCP relay issues on N7K. Please check the release notes and see if addresses your behaviour.

We need to find out why the N7K is forwarding the traffic over peer-link? Do you have any "ip redirects" enable under the SVI?

Cheers,

-amit singh

[j-mccarthy]

I'm running latest 5.1(2).

Problem is the edge subnets (/24s) are being advertised to the network by each N7K. I cannot control which N7K will receive and process the DHCP Relay request from the clients. I can control which N7K receives the replies from the datacentre by tuning the routing protocol to prefer n7k-1 as the return path for all client subnets, but this does not help if the dhcp relay request is created by n7k-2.

If the relay is created by n7K-2 and the reply goes back to n7k-1, the replies are being dropped by n7k-1. The DHCP offer packets are not destined for the client, they are destined for the relay agent on n7k-2 (the VPC peer switch), so forwarding this unicast traffic across the peer link is not sub-optimal.

I also have situation where I cannot ping any of the client vlan interfaces on switch 2 (the hsrp vlans) when the traffic is routed via switch 1. Looks like my N7Ks are dropping all unicast traffic destined for the peer switch vpc interfaces. n7k-1 can ping all the client vlan interfaces on n7k-2, but if he gets a packet from the wan link destined for n7k-2 vlan interface he is dropping it. Same thing happens on n7k-2, he drops any traffic he receives destined for n7k-1 vlan interfaces.

I've opened a TAC case now so I'll see how I go with that.

[Chad Peterson]

If you are running EIGRP across your VPC, this is going to be causing you problems.

Could you upload the topology that you have here with regards to the DHCP path that isn't working right?  Also check your CoPP to make sure your not dropping DHCP there.  In older code we had a policy regarding DHCP...so something to check out.

You can read more about running routing protocols over VCP here:

https://supportforums.cisco.com/thread/2047031

Brad Hedlund has a good blog post about different scenarios with VPC and routing protocols:

http://bradhedlund.com/2010/12/16/routing-over-nexus-7000-vpc-peer-link-yes-and-no/

Take a look at those, if your topology is setup in a way where traffic coming into your Nexus will cross the peer-link then go out a VPC, that is what you need to fix.

Chad

[j-mccarthy]

Hi

I've read Brad's article before and it informed this design. All links into the VPC pair are routed L3 routed interfaces on the network side and L2 VPC port channels on the edge side. I did have a dedicated /31 SVI running over the VPC peer link as a routing protocol peer adjacency between the VPC peer switches, which is ok as per Brad's best practice recommendations.

TAC did ask me to change my routing peer from an SVI on the peer-link to a pair of dedicated physical L3 interfaces which I did, but it did not change the behaviour.

The traffic thats being dropped is not destined for a client via vpc, its destined for the directly connected vlan interface which resides on the peer switch.

[Chad Peterson]

Ahhh ok, so design is good.

So the flow of the traffic that is being dropped would be, say DHCP replay comes down from device connected to the top switch and is address for the SVI of the switch on the right.  It is sent via L3 which has two equal cost paths, 1 to left switch, 1 to right.  Load balancing happens to choose to send it to the switch on the left.

From there the switch on the left will L2 switch it to the right switch as its dest to its SVI, which then crosses the peer link.

Let me know if that is correct.  Mabey I'm reading it wrong

If I'm right, at this point the packet should get punted to the cpu and a new packet should be rebuilt with the info and be broadcasted out to the vlan where the host lives.  This should be allowed as the origninal packet is allowed to go to the CPU even after crossing VPC peer-link, and the new packet is completely new...it never touched the VPC peer-link.  If this is the case, check CoPP and any hardware rate-limiters:

'show policy-map interface control'

'show hardware rate-limit' (I believe thats the command, not at my lab at the moment)

If this isn't the path let me know

Chad

Message was edited by: Chad Peterson

[j-mccarthy]

You have it exactly right. This is the problem bit.

"'From there the switch on the left will L2 switch it to the right switch as its dest to its SVI, which then crosses the peer link."

Left switch is not L2 switching the reply packet over the peer link, it's dropping it. So the right switch never sees the dhcp relay reply. Or the ping packet if its a ping to right switch vlan interface that comes in via left switch.

The TAC engineer did some funky module level dbus/rbus debugs on both Left and Right switches and determined that left switch is punting the packets destined for the peer to it's own CPU rather than switching them out the peer link. The peer switch never receives them.

I accepted the default CoPP policy when I initially configured the switches, so I'll try removing the CoPP on Monday and see if that fixes it.

Thanks for all the responses guys!

[j-mccarthy]

My problem is solved

TAC alerted me to this bug in NX-OS 5.1(2) relating to vpc peer-gateway.

CSCtl11424    Unicast Packets not Forwarded via Peer-Link with Peer-Gateway Enabled

I've removed peer-gateway from both switches and everything works properly again. I don't think I really need it on anyway, all the outbound client traffic should be addressed to the HSRP virtual mac rather than the physical.

I guess vpc peer-gateway and 5.1(2) is a no-no

[Chad Peterson · Accepted solution]

Ah. Yes. Peer gateway is a bit of a problem in many situations. It's there for specialized appliances that don't sent traffic to their default gateway but instead just sent traffic back to the Mac address they received the frame from. Usually 10gig filers do this to improve performance.

Regardless it's a feature you only want to enable when needed. Glad you found it

Chad

Sent from Cisco Technical Support iPhone App