Overlay Transport Virtualization (OTV) is a feature available on the Nexus 7000 series switches that enables extension of VLANs across Layer 3 (L3) networks. This enables new options of data center scale and design that have not been available in the past. The two common use cases include data center migration and workload mobility. Interestingly, many jump to a multiple physical data center scenario and start to consider stretched clusters and worry about data sync issues and while OTV can provide value in those scenarios it also is a valid solution inside the data center where L3 interconnects may segment the network but the need for mobility is present.
OTV in detail
OTV is significant in its ability to provide this extension without the hassles and challenges associated with traditional Layer 2 extension, such as merging STP domains, MAC learning, and flooding. OTV is designed to drop STP BPDUs across the Overlay interface, which means STP domains on each side of the L3 network are not merged. This is significant in that it minimizes fate sharing where a STP event in one domain ripples to other domains. Additionally, OTV uses IS-IS at its control plane to advertise MAC addresses and provide capabilities such as loop avoidance and optimized traffic handling. Finally, OTV doesn't have state that needs maintained as is required with pseudo wire transports like EoMPLS and VPLS. OTV is an encapsulating technology, and as such, adds a 42 byte header to each frame transported across the Overlay.
Below is the frame format in more detail.
As mentioned earlier, OTV has the ability to minimize the amount of traffic that crosses the Overlay network in addition to blocking STP. One of the first techniques involves the way OTV advertises MAC addresses. In a traditional Ethernet network, MAC addresses are flooded throughout the network. This behavior isn’t desirable across a Data Center Interconnect (DCI) where bandwidth may be at a premium, so OTV doesn’t flood but rather advertises MAC addresses it learns and will forward traffic only to known addresses. This is particularly useful because, in some cases, Content Addressable Memory (CAM) and Address Resolution Protocol (ARP) tables can get out of sync. When this happens, switches will flood traffic out every port in the VLAN in the hopes of reaching the destination MAC. This is referred to as unicast flooding. OTV doesn’t participate in unicast flooding and, as such, saves precious DCI bandwidth.
Additionally, OTV has a feature referred to as ARP neighbor discovery where it will snoop ARP requests as they traverse the network. This enables the OTV device to then respond to ARP requests locally for devices it learns about across the overlay, which in turn preserves the DCI bandwidth.
OTV also has the ability to effectively multi-home and provide load balancing and redundancy in a simple manner. The way this is accomplished in OTV is by the use of the concept of a site VLAN. The site VLAN is a VLAN that's dedicated to OTV and NOT extended across the Overlay but is trunked between the two OTV edge devices. This VLAN doesn't need any IP addresses or SVIs created, it just needs to exist and be added to the OTV config as shown below.
otv site-vlan 99
With the simple addition of this command, the OTV edge devices will discover each other locally and then use an algorithm to determine a role each edge device will assume on a per VLAN basis. This role is called the Authoritative Edge Device (AED). The AED is responsible for forwarding all traffic for a given VLAN including broadcast and multicast traffic. Today the algorithm aligns with the VLAN ID with one edge device supporting the odd numbered VLANs and the other supporting the even numbered VLANs. This is illustrated in the topology below.
Finally we’ll want to optimize our egress routing for the stretched VLANs so that we can have local devices make routing decisions. This is referred to as First Hop Redundancy Protocol (FHRP) localization. These protocols are Host Standby Routing Protocol (HSRP v1 and v2) Virtual Router Redundancy Protocol (VRRP), and Gateway Load Balancing Protocol (GLBP). These protocols allow two network devices to share a common IP address to be used as the default gateway on a subnet and provide redundancy and load balancing to clients in that subnet.
Before we can discuss FHRP localization, let's review why this might be significant to our design. Typically with FHRPs the members of the group are local to each other both logically and physically. Depending on the FHRP, there is load balancing or redirection between the devices to the "active" member to handle traffic. This works well when considered locally and most of us use it without a second thought.
When we start to stretch or extend our VLANs across distances, latency is introduced. While a 1ms one-way latency may not sound significant, when accumulated over a complete flow or transaction, it can become quite detrimental to performance. This is exacerbated if the two devices are both in the same location, but have default gateways in another data center. Sub optimal switching and routing at its finest. This effect is illustrated below where device A needs to talk with device B and the default gateway resides across a stretched VLAN.
We address this with OTV by implementing filters to prevent the FHRP peers in each opposite data centers from seeing each other and therefore becoming localized.
The configuration for the IP ACL looks like this:
ip access-list HSRP_IP
10 permit udp any 18.104.22.168/32 eq 1985
20 permit udp any 22.214.171.124/32 eq 1985
This access list matches the multicast addresses for HSRPv1, and HSRPv2, though can be modified for VRRP and GLBP.
This access-list is then applied as a VACL to filter the FHRP hellos from entering the OTV through the internal interfaces. The VACL looks like below where we’ll filter HSRP on VLAN 31-33.
vlan access-map HSRP_Local 10
match ip address HSRP_IP
vlan access-map HSRP_Local 20
match ip address ALL
vlan filter HSRP_Local vlan-list 16,23
One last step we need is to create a MAC ACL to specifically filter the FHRP MAC addresses as being advertised as reachable via OTV.
Then we refer to the mac-list in a route-map as below:
route-map OTV_HSRP_filter permit 10
match mac-list OTV_HSRP_VMAC_deny
The final step is to add the route-map to the OTV process with the following commands.
redistribute filter route-map OTV_HSRP_filter
With these components configured we can begin to stretch VLANs between the OTV devices. This is useful for data center migrations, workload mobility or any other applications that requires layer-2 connectivity without the risks traditionally considered with LAN extension.
Hi there.I'm confused about licenses and requirements needed for implementing cisco ACI Multi-Site.in cisco build and price, while you choose Nexus 9500, It doesn't let you select any 'ACI Advantage Term N9500' due to subscription License is not required ...
Hi,Is it possible to create a L3 port-channel from each spine to each IPN device?In my case, two 6509 in VSS are the IPN devices and I was thinking to have a layer 3 port-channel from each spine where each link ends in a different VSS member. In thi...
Hi community,Has anyone successfully configured PBR in a contract with vzAny acting as both Provider and Consumer (any to any)?Apparently, the release notes of 3.2(1), White Paper and Cisco Live BRKSEC-2048 all briefly mentioned we could, without fur...
I am doing a fresh install of APIC on my lab Standalone Controller due to HDD Replacement.APIC-SERVER-L2BIOS Version: C220M126.96.36.199c getting installation error, I feel it is related to TPM. Anyone who has faced similar challenge?Attaching snapsh...
Hi,Cisco Nexus, Other Data Center SwitchesJust need help to confirm the difference between the 3 Part Numbers for Cisco Nexus 9500 Series Chassis. As per my further research, it shows the only difference between the 3 is in the bundling (Components) as sh...