What is peer gateway?
Peer gateway is a feature which was developed to support network devices which use non-standard layer 2 packet forwarding in a vPC environment. This was first discovered by NetApp using the fastpath feature but other devices have started using this method as well over the years.
How exactly does this non-standard layer 2 forwarding work?
Here’s how a typical ping works from host to host in a vPC environment where the hosts are in different vlans:
Topology
In this scenario when HostA pings HostB the packet flow is as follows:
HostA Echo Request (pre-routing)
Based upon port-channel load balancing it will hash to one N7k1 or N7k2. For the purposes of this we will assume it hashes to N7k1. N7k1 will then route the frame. Here’s the frame HostB will receive:
HostA Echo Request (post-routing)
HostB then responds with the following:
HostB Echo Reply (pre-routing)
Again this will be subject to hashing. For the purposes of this we will assume it hashes to N7k2. N7k2 will route the frame and the frame HostA receives will look like this:
HostB Echo Reply (post-routing)
This is how it works with a properly behaving set of hosts and works as expected.
In the scenario where we need peer gateway the scenario looks like this:
HostA Echo Request (pre-routing)
Again we will assume that the packet hashes to N7k1. N7k1 does the routing and HostB receives this:
HostA Echo Request (post-routing)
Now here’s where it gets different. HostB replies with the following frame:
HostB Echo Reply (pre-routing)
As you can see HostB has just flipped the source and destination MAC address. Again this frame is subject to port-channel hashing. If it hashes to N7k1 then everything is great and no issues. If it hashes to N7k2 then the packet has to cross the peer link, and a special bit is set and it can’t leave N7k1 on a any port that is a member of a vPC. This functionality is how loops are prevented on a VPC.
So how does Peer Gateway work?
What peer gateway does is allow the nexus switches to route frames which are destined to the mac address of their peer device. In this way it works the same as HSRP in a vPC environment where both nexus switches forward the frames destined to either nexus’s physical mac addresses. When enable it you will get output like this:
N7k-1# show mac address-table vlan 10
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
G 10 0000.0c07.ac0a static - F F sup-eth1(R)
G 10 0024.986f.bac1 static - F F sup-eth1(R)
G 10 0024.986f.bac2 static - F F vPC Peer-Link(R)
Note the G flag even though the mac address is learned on the peer link. This means it will be treated like the HSRP virtual addresses and this switch will forward packets destined to that mac. The only exception is if a packet is destined to both the physical mac of the peer and the physical ip address. Under that circumstance the packet will be tunneled across the peer link.
Should I enable peer gateway?
If you don’t have any devices which behave in this fashion then you should not enable peer gateway. It should also be noted that using peer gateway to get around the limitation of no peering routing protocols across vPC’s is not an officially supported design and can result in performance issues.
