VXLAN/EVPN Configuration Example (N9k / p2p)

[Lukas Krattiger]

VXLAN/EVPN has been release on Nexus 9000 series in early February 2015, followed by Nexus 7000/7700 (F3/M3 Linecard) in Summer and Nexus 5600 later in 2015. Other Cisco platforms like the ASR 9000 and ASR 1000 also support VXLAN with EVPN control-plane.

As there are many request in how to configure VXLAN/EVPN on a given Platform, this Blog post should help to get you get started with a Nexus 9300/9500 (including Nexus 9x00 EX/FX)

While this example focuses on numbered IP interfaces or the so called P2P (point-to-point) approach, there is also a "ip unnumbered" example available.

Generally we would expect a Topology as shown below.

For the sake of this example, we are using the following Topology example, which is a subset of the Topology above.

The configuration example does cover the configuration of the following software components

- Underlay with OSPF, PIM Sparse (ASM) and Anycast-RP

- IP numbered interfaces (p2p interfaces)

- VXLAN

- MP-BGP EVPN Control-Plane

- VPC

We will focus on the configuration of Spine "1", Leaf "V1" and Leaf "V2"

Spine "1" Configuration:

hostname SPINE1

nv overlay evpn

feature ospf

feature bgp

feature pim

feature nv overlay

ip pim anycast-rp 10.254.254.254 10.250.250.101

ip pim rp-address 10.254.254.254 group-list 239.239.239.0/24

interface Ethernet3/1

  description Link to Leaf "V2"

  mtu 9216

  ip address 10.1.1.6/30

  ip ospf network point-to-point

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

  no shutdown

interface Ethernet3/2

  description Link to Leaf "V1"

  mtu 9216

  ip address 10.1.1.2/30

  ip ospf network point-to-point

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

  no shutdown

interface loopback0

  ip address 10.250.250.101/32

  ip ospf network point-to-point # will change OSPF interface back to Loopback; required for VPC

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

interface loopback254

  ip address 10.254.254.254/32

  ip ospf network point-to-point

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

router ospf UNDERLAY

  router-id 10.250.250.101

  log-adjacency-changes detail

router bgp 65500

  router-id 10.250.250.101

  address-family ipv4 unicast

  neighbor 10.250.250.0/24 remote-as 65500

    update-source loopback0

    address-family ipv4 unicast # optional for "show ip bgp summary" support

    address-family l2vpn evpn

      send-community both

      route-reflector-client

Leaf "V1" Configuration:

hostname LeafV1

nv overlay evpn

feature ospf

feature bgp

feature pim

feature interface-vlan

feature vn-segment-vlan-based

feature nv overlay

feature vpc

fabric forwarding anycast-gateway-mac 2020.DEAD.BEEF

ip pim rp-address 10.254.254.254 group-list 239.239.239.0/24

vlan 1,99-101,2500,3000

vlan 99

  name L2onlyHostSegment

  vn-segment 30099

vlan 100

  name L2L3HostSegment

  vn-segment 30000

vlan 101

  name L2L3HostSegment

  vn-segment 30001

vlan 2500

  name FabricBD

  vn-segment 50000

vlan 3000

  name VPCL3Peering

route-map FABRIC-RMAP-REDIST-SUBNET permit 10

  match tag 21921

vrf context TENANT1

  vni 50000

  rd auto

  address-family ipv4 unicast

    route-target both auto

    route-target both auto evpn

  address-family ipv6 unicast

    route-target both auto

    route-target both auto evpn

vpc domain 1

  peer-switch

  peer-keepalive destination 10.2.8.1 source 10.2.8.2 vrf management

  peer-gateway

  ip arp synchronize

interface Vlan100

  no shutdown

  vrf member TENANT1

  ip address 192.168.100.1/24 tag 21921

  fabric forwarding mode anycast-gateway

interface Vlan101

  no shutdown

  vrf member TENANT1

  ip address 192.168.101.1/24 tag 21921

  fabric forwarding mode anycast-gateway

interface Vlan2500

  description FabricBD

  no shutdown

  mtu 9216

  vrf member TENANT1

  ip forward

interface Vlan3000

  description VPC Layer-3 Peering for VXLAN

  no shutdown

  ip address 10.3.1.1/30 # Requires to be individual IP per VPC member

  ip ospf network point-to-point

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

nve infra-vlans 3000

# required for Nexus 9300-EX/FX or Nexus 9200

interface port-channel1

  description VPC Peer-Link

  switchport mode trunk

  spanning-tree port type network

  lacp suspend-individual

  vpc peer-link

hardware access-list tcam region vacl 0

# example region to free up space for arp-ether region

hardware access-list tcam region arp-ether 256 double-wide

# required for ARP suppression, requires reboot

# double-wide is required starting 7.0(3)I3(1)

# not required for Nexus 9300-EX/FX or Nexus 9200

interface nve1

  mtu 9216

  no shutdown

  source-interface loopback1

  host-reachability protocol bgp

  member vni 30000

    suppress-arp

    mcast-group 239.239.239.100

  member vni 30001

    suppress-arp

    mcast-group 239.239.239.101

  member vni 30099

    mcast-group 239.239.239.99

  member vni 50000 associate-vrf

interface Ethernet1/1

  switchport mode trunk

  spanning-tree port type edge trunk

  spanning-tree bpduguard enable

interface Ethernet1/47

  description Link for VPC Peer-Link

  switchport mode trunk

  channel-group 1 mode active

interface Ethernet1/48

  description Link for VPC Peer-Link

  switchport mode trunk

  channel-group 1 mode active

interface Ethernet2/1

  description Link to Spine "1"

  no switchport

  mtu 9216

  ip address 10.1.1.1/30

  ip ospf network point-to-point

  ip router ospf UNDERLAY area 0.0.0.0

interface loopback0 # Loopback for Router ID, routing adjacency and peering

  ip address 10.250.250.102/32

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

interface loopback1 # Loopback for VTEP only

  ip address 10.254.254.102/32

  ip address 10.254.254.1/32 secondary

  ip ospf network point-to-point # will change OSPF interface back to Loopback; required for VPC

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

router ospf UNDERLAY

  router-id 10.250.250.102

  log-adjacency-changes detail

router bgp 65500

  router-id 10.250.250.102

  address-family ipv4 unicast

  neighbor 10.250.250.101 remote-as 65500

    update-source loopback0

    address-family ipv4 unicast # optional for "show ip bgp summary" support

    address-family l2vpn evpn

      send-community both

  vrf TENANT1

    address-family ipv4 unicast

      advertise l2vpn evpn

      redistribute direct route-map FABRIC-RMAP-REDIST-SUBNET

evpn

  vni 30000 l2

    rd auto

    route-target import auto

    route-target export auto

  vni 30001 l2

    rd auto

    route-target import auto

    route-target export auto

  vni 30099 l2

    rd auto

    route-target import auto

    route-target export auto

Leaf "V2" Configuration:

hostname LeafV2

nv overlay evpn

feature ospf

feature bgp

feature pim

feature interface-vlan

feature vn-segment-vlan-based

feature nv overlay

feature vpc

fabric forwarding anycast-gateway-mac 2020.DEAD.BEEF

ip pim rp-address 10.254.254.254 group-list 239.239.239.0/24

vlan 1,99-101,2500,3000

vlan 99

  name L2onlyHostSegment

  vn-segment 30099

vlan 100

  name L2L3HostSegment

  vn-segment 30000

vlan 101

  name L2L3HostSegment

  vn-segment 30001

vlan 2500

  name FabricBD

  vn-segment 50000

vlan 3000

  name VPCL3Peering

route-map FABRIC-RMAP-REDIST-SUBNET permit 10

  match tag 21921

vrf context TENANT1

  vni 50000

  rd auto

  address-family ipv4 unicast

    route-target both auto

    route-target both auto evpn

  address-family ipv6 unicast

    route-target both auto

    route-target both auto evpn

vpc domain 1

  peer-switch

  peer-keepalive destination 10.2.8.2 source 10.2.8.1 vrf management

  peer-gateway

  ip arp synchronize

interface Vlan100

  no shutdown

  vrf member TENANT1

  ip address 192.168.100.1/24 tag 21921

  fabric forwarding mode anycast-gateway

interface Vlan101

  no shutdown

  vrf member TENANT1

  ip address 192.168.101.1/24 tag 21921

  fabric forwarding mode anycast-gateway

interface Vlan2500

  description FabricBD

  no shutdown

  mtu 9216

  vrf member TENANT1

  ip forward

interface Vlan3000

  description VPC Layer-3 Peering for VXLAN

  no shutdown

  ip address 10.3.1.2/30 # Requires to be individual IP per VPC member

  ip ospf network point-to-point

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

nve infra-vlans 3000

# required for Nexus 9300-EX/FX or Nexus 9200

interface port-channel1

  description VPC Peer-Link

  switchport mode trunk

  spanning-tree port type network

  lacp suspend-individual

  vpc peer-link

hardware access-list tcam region vacl 0

# example region to free up space for arp-ether region

hardware access-list tcam region arp-ether 256 double-wide

# required for ARP suppression, requires reboot

# double-wide is required starting 7.0(3)I3(1)

# not required for Nexus 9300-EX/FX or Nexus 9200

interface nve1

  mtu 9216

  no shutdown

  source-interface loopback1

  host-reachability protocol bgp

  member vni 30000

    suppress-arp

    mcast-group 239.239.239.100

  member vni 30001

    suppress-arp

    mcast-group 239.239.239.101

  member vni 30099

    mcast-group 239.239.239.99

  member vni 50000 associate-vrf

interface Ethernet1/1

  switchport mode trunk

  spanning-tree port type edge trunk

  spanning-tree bpduguard enable

interface Ethernet1/47

  description Link for VPC Peer-Link

  switchport mode trunk

  channel-group 1 mode active

interface Ethernet1/48

  description Link for VPC Peer-Link

  switchport mode trunk

  channel-group 1 mode active

interface Ethernet2/1

  description Link to Spine "1"

  no switchport

  mtu 9216

  ip address 10.1.1.5/30

  ip ospf network point-to-point

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

  no shutdown

interface loopback0 # Loopback for Router ID, routing adjacency and peering

  ip address 10.250.250.103/32

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

interface loopback1 # Loopback for VTEP only

  ip address 10.254.254.103/32

  ip address 10.254.254.1/32 secondary

  ip ospf network point-to-point # will change OSPF interface back to Loopback; required for VPC

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

router ospf UNDERLAY

  router-id 10.250.250.103

  log-adjacency-changes detail

router bgp 65500

  router-id 10.250.250.103

  address-family ipv4 unicast

  neighbor 10.250.250.101 remote-as 65500

    update-source loopback0

    address-family ipv4 unicast # optional for "show ip bgp summary" support

    address-family l2vpn evpn

      send-community both

  vrf TENANT1

    address-family ipv4 unicast

      advertise l2vpn evpn

      redistribute direct route-map FABRIC-RMAP-REDIST-SUBNET

      maximum-paths ibgp 2

evpn

  vni 30000 l2

    rd auto

    route-target import auto

    route-target export auto

  vni 30001 l2

    rd auto

    route-target import auto

    route-target export auto

  vni 30099 l2

    rd auto

    route-target import auto

    route-target export auto

[Lukas Krattiger]

In the LEAF configuration, you have 2 loopback interfaces defined.

You use Loopback 0 as a source of OSPF and BGP. Loopback 1 as the source of VTEP.

Why do you separate them?

You never want your control-plane sessions to be brought down when for example the NVE goes down. There are failure situations or with the usage of GIR or VPC where forwarding will continue to work as long as the control-plane is there but will blackhole/packet drop if the control-plane is down together with the data-plane (NVE). Having two loopback will give you better convergence in these failure and maintenance scenarios up to zero packet loss.

If I would like to create a completely separated VRF without leaks with the associated 2 SVI (anycast-gateway mode) belonging to it, how to do it?

1. Create another VRF 2

2. Create 3 vlan's (in that + one for transit)

3. Create a second NVE 2

     (should I use the same loopback interface as for nve 1?)

4. Advertise VRF by BGP

You just create a VRF, map the L3VNI, create a VLAN/SVI for the VRF (fabric facing), add the VRF to BGP and attach the VRF to the NVE. Once done, create a VLAN, map the L2VNI and use the SVI for the anycast-gateway. Last but not least, attach the L2VNi to the NVE with a multicast group or ingress-replication.

You can repeat this up to 900 VRF and 2000 L2VNI with SVI. There is no need for a second NVE or for a "transit" VLAN.

[Rajagopal Sivaramakrishnan]

Great post.. have a quick design question.. we have 2 datacenters .. old and new.. new has the fabric spine leaf. old has legacy stuff. L2 extension from old to new is through OTV, and for DC migration, all L2 willl be spanned through OTV. but there is also another L2 link, which was extended for a specific purpose (because storage was running out in old,  they built new esx clusters in new...)

now, there are prod servers in this esx cluster, which goes through the L2 link (not otv). This is not a part of the overall spine leaf fabric. They want to connect this to spine leaf fabric, which has a potential issue... L2 vlans are extended both on L2 link and OTV.  I hope there should be no problem with loops, but how will the packet forwarding happen on a MAC layer ? btw, otv terminates in border leaf, and the old network was planned to be exteded to border leaf too...

Thanks in advance.

Raj

[Lukas Krattiger]

Raj,

I would go through this link Optimizing Layer 2 DCI with OTV between Multiple VXLAN EVPN Fabrics (Multifabric) White Paper - Cisco

If there is a Layer-2 classic ethernet link or OTV, the requirements and best practices would be the same. I'm not sure what your exact situation is but I would be careful with Layer-2 loops and as said, please have a review of the document in the link above

[ulasinski]

I'm sorry for answering so late,

Lukas thank you very much for the answer. I did as you wrote and everything works.

I have the latest Cisco Nexus 9300 with 100Gbit uplinks with QSFP-40/100-SRBD.

If someone would like to hear my opinion, especially about cooperation with CNFM (there is no cooperation because C9364C is not supported), or about a quite alternative approach to configuration in DCNM (not fan for me) and its broken stability, write to me.

Why write about it.

Because at every step, at each vendor in the case of VXLAN/EVPN fabric, it is recommended to leave the CLI and control fabric by Web Management Interface.

However, I have a question for Lukas about how to solve the above task.

In a specific VRF for a specific subnet, I have a L3 hop in nexus fabric.

Subnetwork Y <--> VRF Z (SVI-A on N9K (anycast gateway)) <--> VXLAN EVPN fabric <--> VRF Z (SVI-B on N9K (anycast gateway)) <--> vSphere LACP LAG (VM GATEWAY INTERFACE) <--> Internet

In the environment, each LEAF working in a vPC pair.

at N9K-s for this VRF I have a "default route" via GATEWAY (static routing)

This GATEWAY is PaloAlto VM that's why I used anycast mode from the gateway (theoretically gateway can change the place in the cluster).

I'm going to the point

Clients from subnet Y are sensitive but impoverished.

I would like to use a different gateway for PA Maintenance's time. However, this gateway is not PA. Both support OSPF.

Normally in 6509 with which I have the most experience I would change the default route based on IP SLA. Here, in that case of N9K SVI ANYCAST, the IP SLA mechanism will always be disappointing. This solution is eliminated. I think about OSPF with the default route.

One of my supporters recommends assigning a dedicated n9k pair as a bordear Gateway and running HRRP on the PA side and run OSPF.

How would you do it?

Do you need hrrp in vpc mode?

Do I need ospf on all Leafs (when I use non anysast Gateway mode on SVI from PA side)?

[ronwilkerson]

Hi Lukas,

Wondering why EIGRP isn't a recommended IGP for the underlay for VXLAN?

[ciscoworlds]

Hi.

Does VXLAN EVPN inter-vxlan routing work in data plane on Nexus 9000v? I followed steps documented, but the ping messages don't pass between vxlans. Intra-vxlan traffic has no issue.

[ronwilkerson]

The VTEPs use a dedicated VLAN/VNI for inter-VXLAN routing.  So, no, it does not use the data plane.

[ciscoworlds]

ron wilkerson wrote:

The VTEPs use a dedicated VLAN/VNI for inter-VXLAN routing.  So, no, it does not use the data plane.

Hi; I didn't mean that. My issue is that I'm not getting any type-5 route on VTEPs and therefor, the inter-vxlan routing isn't working. I've pasted the VTEPs config on the following link.

https://supportforums.cisco.com/t5/server-networking/nexus-9000v-vxlan-evpn/m-p/3352221/highlight/false#M12408

The only difference between mine and what is stated in this doc written by lkrattig is that, I just have two VLANs associated with two VXLANs and two intermediate L3 VXLANs, but here I see there is another L2 VLAN VXLAN (99) in addition to the original L2 vlan.

[ulasinski]

What the client's ospf configuration should look like for interface SVI in the anycast gateway mode. The client has a PA Firewall VM that can migrate between different Leaf pairs (vPC) and wants to distribute the default path for his endpoints.

[lganeva]

There is a lot of confusion in regards to BUM underlay due to a statement from Cisco document on Multicast in VXLAN fabric which says the Nexus7700/Nexus9000 supports only PIM ASM. In reality, you can configure a mixed-up fabric with bgw running PIM sparse-mode and SPINE and LEAFs running PIM Bidir. This is working fine as long as you have supress-arp on BGW NVE interface or on the LEAF switches in the other site(sites) as PIM Bidir and PIM sparse-mode only differ when it comes to receiver and sending node registration with the RP.

Still, i believe this is an inconsistent design and I am wondering why the Cisco documentation says that PIM Bidir is not supported between 7k and 9k. 

https://www.cisco.com/c/en/us/td/docs/dcn/nx-os/nexus9000/102x/configuration/vxlan/cisco-nexus-9000-series-nx-os-vxlan-configuration-guide-release-102x/m_configuring_the_underlay.html

Table is Multicast Routing in the VXLAN Underlay 

[Lukas Krattiger]

@lganeva while you are right when it comes to Multi-Site, that you can use IR or Multicast respectively, you are not entirely correct when it comes to different Multicast modes (e.g. PIM Bidir with PIM ASM). There are ways that some different version of Multicast work together but your recommendation is resulting in something else. The reason why it works when you enable ARP suppression is your end-point learning doesn't require BUM anymore. This said, ARP suppression doesn't solve any of your BUM needs, the config you use has non-working Broadcast, non-working Unknown Unicast and non-working Multicast.