Next generation data centers across the world are taking advantage of Cisco’s Virtual PortChannel. As of recent, I’ve moved our core to a pair of Nexus 7010s running vPCs to the user edge, server edge, and some legacy systems. Over the past couple weeks I’ve also learned of a couple of features and wanted to bring them to light for everyone. I’ve also thrown in some information on some features that may not be new, but are still good to have in your tool bag.
Peer Gateway
vPC Peer Gateway allows devices to act as a gateway for packets that are destined to their vPC peer(s) MAC address. Typically this is used with NAS devices, or load-balancers that do not comply with Ethernet RFC standards. Such behavior can cause packets to be sent across the peer-link and end up being dropped on the other side.
The use of peer-gateway does not circumvent the exclusion from running L3 routing protocols over a vPC, or vPC enabled VLAN.
Peer-Gateway Configuration Example
core01(config)# vpc domain 42
core01(config-vpc-domain)# peer-gateway
core01(config-vpc-domain)# sh run vpc
vpc domain 42
role priority 100
peer-keepalive destination 10.25.15.249 source 10.25.15.250 vrf vpc-keepalive
peer-gateway
-- Repeat exactly for core02
What Does “vpc peer-gateway” Do?
If we left everything alone, the story would be complete. Unfortunately, storage vendors thought it would be a good idea to optimize their handling of Ethernet frames. Some NetApp and EMC equipment ignores the ARP reply given by the HSRP primary and instead forwards Ethernet frames to whichever MAC address it receives frames from. This is nonstandard behavior.
Using the diagram above, let’s assume say that the User PC is now a EMC Celera storage device. The Server sends its packets (IP destination 10.1.1.100) to Nexus B, which routes them to the Ethernet LAN. All IP packets with source IP 10.5.5.5 will be encapsulated in Ethernet frames with a source MAC address of 0022.5579.F643. The EMC Celera will cache the source MAC address of these frames, and when it has IP packets to send to 10.5.5.5, it will encapsulate them in Ethernet frames with a destination MAC of 0022.5579.F643. It is choosing to ignore its default gateway for these outbound packets.
I suppose the theory behind this feature was to eliminate the extra hop within the LAN. When HSRP is enabled, it is necessary to disable ICMP redirects. This means that the routers will not inform hosts on the LAN that a better default-gateway is available for a particular destination IP address. This storage feature saves a LAN hop.
Unfortunately, this optimization does not work well with vPC. vPC relies on virtual MAC address sharing to reduce utilization across the vPC peer-link. If hosts insist on addressing their frames to a specific router, suboptimal packet forwarding can occur. According to Cisco, “Packets reaching a vPC device for the non-local router MAC address are sent across the peer-link and could be dropped by the built in vPC loop avoidance mechanism if the final destination is behind another vPC.” At the application level we saw very poor performance due to these dropped packets. Enough of the packets got through to allow access to the storage device, but file load times were measured in the tens of seconds, rather than milliseconds.
The “vpc peer-gateway” allows HSRP routers to accept frames destined for their vPC peers. This feature extends the virtual MAC address functionality to the paired router’s MAC address. By enabling this feature, NX-OS effectively disables the storage vendors’ optimization.
