peer-gateway TTL

[Jose Maroto Grande]

cisco says:

Packets that arrive at the peer-gateway vPC device have their Time to Live (TTL) decremented, so that packets carrying a TTL of 1 might get dropped in transit due to TTL expiration. You should take this situation into account when the peer-gateway feature is enabled and particular network protocols that source packets with a TTL of 1 operate on a vPC VLAN.

I don’t understand this peer-gateway feature, because it is a normal behavior in transit ip packect (ip environment). I understand the problem with network protocols that peer-gateway produce, but I don’t understand the impact peer-gateway produce when I will set it in a production network. Maybe the impact is for transit traffic bettwen two pair of nexus at layer 3 (out of L2 vpc)?

Thanks everyone

[Mark Malone]

We use peer-gateway because we have NetAPP and F5s , you really only need it of you have these certain types of devices as they use fastpath

This explains why

https://supportforums.cisco.com/document/98811/peer-gateway-feature-nexus-7000

vpc domain 100
  peer-switch
  role priority 200
  system-priority 150
  peer-keepalive destination x.x.x.x source x.x.x.x vrf heartbeat
  peer-gateway
  ip arp synchronize

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Environment

Inline edit trigger

Symptom

A performance impact or outage is experienced when accessing the storage system through the LAN after implementing Cisco's vPC on Nexus 7000 or 5500 series switches, when ip.fastpath is enabled.

The storage system will respond to the MAC address of an individual switch/router and not the virtual MAC address of the two vPC devices.

Inline edit trigger

Cause

This issue is due to the fact that Cisco's vPC feature was not initially designed with the Data ONTAP Fastpath feature in mind. The Nexus 7000 series is Layer 3 aware. With vPC deployed, it can result in an asymmetric routing condition, which is explicitly not supported in combination with NetApp's implementation of ip.fastpath.

Fastpath is a feature that is enabled by default in Data ONTAP systems. Essentially, Fastpath eliminates overhead used by the routing logic (that is, route table lookups) by sending frames out of the same interface that we received the traffic on. This is done via interface to MAC address caching. This feature has been in place and used by NetApp for years. Other vendors do this as well.

Due to fastpath caching incoming MAC address information, considerations should be made when HSRP is in use in combination with vPCs as well. From the perspective of the storage system, if a frame arrives on an ethernet interface on the storage system with the physical MAC address of the switch, then the load balancing algorithim of a vif or ifgrp could choose to return the traffic to the originating MAC address through an interface connected to the neighboring switch (that does not use that MAC address), causing the potential for performance degradation or packet loss due to traffic traversing the peer-link.

Inline edit trigger

Solution

This incompatibility was recognized by Cisco and it is part of the reason that the vPC Peer-Gateway feature was introduced in NX-OS 4.2.1 for the 7000 series switches and NX-OS Release 5.0(3)N1(1) for the 5500 series switches. Enable the vPC Peer-Gateway feature on the Nexus switches. This will allow the switches to route traffic on the local switch even if the destination MAC is that of the peer switch.

[Jose Maroto Grande]

thank you Mark,

I understand peer-gateway and i need to configure it, because i have and storage device with the problem you described it.

but My question is about TTL when cisco says:

Packets that arrive at the peer-gateway vPC device have their Time to Live (TTL) decremented, so that packets carrying a TTL of 1 might get dropped in transit due to TTL expiration. You should take this situation into account when the peer-gateway feature is enabled and particular network protocols that source packets with a TTL of 1 operate on a vPC VLAN.

I think without peer-gateway, the normal behavior in transit traffic  for ip routing environment is decrement TTL, for that i don’t understand what cisco says. 

In my implementation i have two nexus 9500 in vpc domain and i have a truk between them selfs for vlans routing (4 vlan) with eigrp and ospf. There are several neighbors and are neighbors them self. Maybe when i set the command peer-gateway, neighbor relationships between two nexus will go to state down, but will it be possible another IP transit traffic could be affected? maybe transit ip traffic that it has to cross two 9500 with destination another router, may it?

Thanks allot for your information, it is very clear.

[BenBen]

My analysis:
If the routing destination is outside of Nexus. TTL will be decreased even if peer-gateway is not used, that is the packet reaches to the virtual HSRP mac.

The concern here is that if the packet destination is for the vpc peer mac and also vpc peer IP. In this scenario, the packet will be tunneled to the VPC peer and also decrease the TTL by one. This decrease of the TTL could cause issue. For example, the OSPF packet TTL could be set to 1, if the packet reach the Nexus, then then the Nexus tunnel the packet to the peer. At this moment, TTL will be 0 and the packet will be discarded. As a result, host from VPC will not be able to form OSPF neighbor with the other VPC peer. As a result OSPF over VPC is NOT supported in earlier releases of NX-OS

This behavior changed in later releases so to support OSPF over VPC. Please note OSPF over VPC still has its own limitations.

[mrflorida]

What they are saying in Cisco doc is ttl will be decremented if you cross the peer link. Lets say we have VPC setup with Sw0 and Sw1. If a packet is sent to the mac of sw0 and rvcd on sw1 (due to PC hash) it will be forwarded over the peer-link to switch 0 because we have peer-gateway enabled. However the TTL will be decremented by 1 since it will need to traverse the peer link. This is why routing peers will not come up over peer link as the ttl is decremented in hello packets.