route trace list firewall twice (skipping default gw)

[magnuspettersson76]

Hi
We have 2 Nexus 9004 (vpc-paired)
We use VRF-lite and put our clients, servers etc in different vrf´s
We use HSRP on all SVI´s

If i do a trace from a client-laptop (vrf-client) to a server (vrf DC) I get different routes and sometimes it also skipps client´s default gateway as first hop.

Client is on 10.20.51.0/24 network (gateway 10.20.51.1 - is the HSRP VIP-address)

"sh ip route vrf Client" in N9k

0.0.0.0/0, ubest/mbest: 1/0
*via 10.10.10.20%TRANSIT, [1/0], 5d02h, static  -------------10.10.10.20 is the VIP address of our checkpoint firewalls
10.20.51.0/24, ubest/mbest: 1/0, attached
*via 10.20.51.2, Vlan2051, [0/0], 3w0d, direct
10.20.51.1/32, ubest/mbest: 1/0, attached
*via 10.20.51.1, Vlan2051, [0/0], 3w0d, hsrp
10.20.51.2/32, ubest/mbest: 1/0, attached
*via 10.20.51.2, Vlan2051, [0/0], 3w0d, local

If I trace 10.11.10.11 (a server in vrf DC) I get:

1 <1 ms       <1 ms         <1 ms               10.10.10.21 -------------this is one of our checkpoint firewalls

2 <1 ms       <1 ms         <1 ms               10.10.10.21

3 <1 ms       <1 ms         <1 ms               10.11.10.11

If i trace again I get:

1 <1 ms       <1 ms         <1 ms               10.20.51.2

2 <1 ms       <1 ms         <1 ms               10.10.10.21

3 <1 ms       <1 ms         <1 ms               10.11.10.11

Network 10.10.10.0/24 is in vrf Transit and knows about all other networks (we use bgp to route-leak all vrf´s to transit)

I am curious about "peer-gateway" command. We do not use it in our vpc domain.

Any ideas?

Best regards
Magnus

[mlund]

Hi

Peer-gateway is somewhat confusing, I will try to explain what nexus is doing when using vpc.

Without peer-gateway, both nexus will actually forward traffic if the packet has the hsrp mac address as destination.

This is the default behavior.

But there are some server vendors that when sending traffic, they are using the mac address from the nexus that last send a packet towards the server, instead of the hsrp address. If the server is connected via a vpc port-channel, this packet will be sent out on an interface in any of the interfaces(hashed) on the port-channel. If it is send out on that interface that is reaching the nexus with the correct mac address, then this nexus will forward the packet. However if the hash decides to send the packet to the other nexus, it sees that this mac address is not my, it belongs to the neighbor nexus. Then it will forward this packet out the pvc-link to the neighbor nexus. The neighbor nexus will look up where to send this packet, and if the destination is out on a vpc port-channel, the packet will be dropped due to the rule that you can not forward a packet out a vpc if it comes from the vpc peer-link.

To overcome this behavior, the peer-gateway is introduced. So in the previous example, the first nexus sees the neighbor mac address, it also looks in the ip destination field, if the packet is not intended to the neighbor, it will forward it out on the vpc towards the destination, and not to the vpc neighbor. 

/Mikael