Cisco NX 9396 vPC pair - HSRP standby peer is forwarding traffic over Peer-link instead of routing it. What am I missing?

[cadillacjack]

I am migrating our L3 infrastructure from a single 6500 to a pair of 9396 Nexus vPC peers.  I configured HSRP and per the documentation I made the vPC primary the HSRP Active and let the vPC seocndary fall into the standby roll..   So far all is well.   Both of 9Ks are connected to the upstream L3 routers so both should be able to route L3 traffic that is destined to the HSRP vMAC address to other networks.   I see the "G" in the mac table on both 9Ks for HSRP vMAC.  The primary has the vMAC entry pointed to the Sup-Eth1 port and the secondary has a vMAC entry point at the vPC Peer-Link.  I see this is correct per the Cisco docs I have seen so far.

THE PROBLEM:  So I when I start sending test packets from a host to destination that is off net I am seeing what I believe is incorrect traffic flow.  The HSRP standby vPC peer is NOT (L3)routing the packets but instead L2 forwards the packet over the Peer-Link to the HSRP active peer and lets it (L3) route the packet.

The traffic from the host is getting delivered to both vPC peers via the vPC member links as you would expect.

But all L3 routing is happening on the HSRP Active peer... again the standby is L2 forwarding the packets over the Peer-Link.

I dont understand why this happening and am worried this will be the case in production.

PLEASE NOTE:  I am using VIRL to lab up and test the changes.  So I dont know if VIRL/NX-OSv is causing this or if I am missing something here.

And shedding of light would be welcome.   

Thanks

[Reza Sharifi]

Both of 9Ks are connected to the upstream L3 routers

Are the 9ks cross connected to both upstream L3 routers or one of the 9ks connect to one router and the other one connect to the other router?

In addition to vPC peer-link, do you have a layer-3 routed link between the 9ks.

BTW, VIRL has limitations when using Nexus-OS.

HTH

[cadillacjack]

Both 9Ks are connected to both upstream routers.  

In addition the peer-link I do have a L3/Routed interface connected between the 9Ks for Layer 3 EIGRP Peering/Routing.

THanks!

[Reza Sharifi]

Ok, for testing can you disable the cross connect from each 9k and run the same test again?

So, one 9k to one router and the other 9k to the router router.

HTH

[cadillacjack]

ok.  I disabled the Cross-connect interfaces to the upstream routers.  So 9K1 is connected to one upstream router and 9k2 the other.

The only path from 1 peer 9k to the other is the Layer3 Link and the Peer-link.  They only install routes via the L3 link for routes that cant learn from other EIGRP Neighbors.

So what I would expect to see is that each 9K would route packets over the link to the upstream router or the L3 link between the 9K peers.  The captures however show that exact same thing I saw before.

When packets arrive at the 9k with HSRP Standby role it forwards the packets across to Peer-link to the HSRP active peer.  Instead of L3 routing packet is L3 forwards it to the HSRP active peer who currently "owns" the vMAC which it sees via the peer-link... 

The MAC table on the standby shows the vMAC advertised via the peer-link, which according to the Cisco docs is correct... Its still shows the "G".  and the MAC table on the HSRP active shows still show the Sup-Eth1.  

This is odd... I wish I had actual hardware for this to test with... I dont know if this is a VIRL/NX-OSv thing.

Thanks for looking at this with me!

J