Nexus 5596's VPC peering requires HSRP on the vlan interfaces

[LA-Engineer]

When setting up two Nexus 5596's for VPC peering, do you also need to set up HSRP on any VLAN interfaces you set up?

I have a weird issue.  When I set up a VLAN interface on one Nexus, I also have to set it up on the other Nexus and setup HSRP as well or else my hosts have strange connectivity issues.

Thank you in advanced.

[Kelvin Willacey]

As long as you require gateway redundancy then you will need to do that. VPC operates a bit differently than VSS or stacking, it needs HSRP or some FHRP to provide gateway redundancy.

[LA-Engineer]

Thank you for your reply.  I think I found the problem.  I need to verify it.

My VPC peer configuration is configured with 'vpc peer-gateway'.  As I understand it, when a VPC switch receives a packet with the MAC address of the VPC peer switch, it will "handle" it locally.  This works well if I have VLAN interfaces configured with HSRP on both Nexus switches.

The problem I was having was that I was setting up VLAN interfaces on only one VPC peer.  This was for an isolated VRF and I didn't feel that it needed redundant first hop gateway.  The problem was that the hosts on these vlans were probably load balancing to both the VPC peer switches.  The packets sent to the VPC peer that did not have the VLAN interfaces were then "handled" locally due to 'vpc peer gateway'.  Since that switch does not have the VRF nor the VLAN interfaces locally, it dropped the packets rather than send it accross the VPC peer link to the switch that does have the VLAN interfaces.

This is my theory.  Can anyone verify this?

[Jon Marshall]

What is the connectivity from the hosts to the Nexus switches ie. are they connected to a FEX which then has a vPC to the Nexus pair ?

Jon

[LA-Engineer]

Hi Jon, the src host is on another switch that has a vPC to the nexus pair.  The dst host is also on a switch that has a vPC to the nexus pair.

I believe that this is what is happening:

Keep in mind that NX-1 is the only one with the VRF and VLAN interfaces.

1. Src sends packet to NX-2.
2. NX-2 sees the destination mac is for NX-1's SVI interface.
   1. Due to 'vpc peer-gateway' it drops the packet because it does not have an SVI on that vlan.
   2. Or, it sends the packet across the peer-link to NX-1. (not sure how this behaves)
3. NX-1 does its routing lookup and determines that the destination is across a vPC. 
   1. It drops the packet because it assumes that the vPC peer ALSO HAS the ability to route and switch the packet to the vPC destination.
   2. I know that this is what it does to L2 switched packets, but I am not aware if it still does this for L3 routed packets.

What do you think?

[Jon Marshall]

I'm not an expert on Nexus by any means but i don't think it is the peer gateway because there is no SVI on the peer switch so it cannot handle it locally ie. the destination mac will be for NX-1.

I believe what is happening is -

1) src sends packet to NX-2 on vPC.

2) NX-2 sees destination mac is for SVI on NX-1. It cannot handle it locally because there is no SVI. So it sends it across the peer link

3) NX-1 receives packet but it cannot send it out to the destination because the destination is reachable via a vPC ie.

the basic rule of loop avoidance for vPCs is that if a packet is received on a vPC member port and is then forwarded across the peer link it cannot then be sent out on any other vPC member port.

So i think NX-1 is the switch that is dropping the packet rather than NX-2.

But like i say i could be wrong.

Jon

[Jon Marshall]

Sorry couldn't read point 3) until i checked my e-mail.

Basically i think with point 3) we are saying the same thing.

Like you say it is an interesting point about L2 vs L3 so i can't say for sure like you but i suspect that is what is happening.

Hopefully some Nexus expert will wander along and straighten us out

Jon

[LA-Engineer]

Thank you Jon.  I find confidence in my understanding of this issue now that I have someone of your caliber that is basically saying the same thing.

I will see if I can get some official information from Cisco regarding this behavior with L3 routed packets. 

If what I suspect is true, then it becomes necessary that if you were to configure intervlan routing on the Nexus pair, you must mirror that L3 routing configuration over to both vPC peers.  And then, that means you'll have to configure HSRP.  Otherwise, you will have connectivity issues when one vPC host attempts to communicate with another vPC host, despite the fact that one of the Nexus is performing L3 routing and L2 re-encapsulation. 

I previously thought that the re-encapsulation erases that vPC tagging and that the vPC loop prevention wouldn't come into play.  I guess that isn't true.