cancel
Showing results for 
Search instead for 
Did you mean: 
cancel

VXLAN/EVPN Configuration Example (N9k / p2p)

30245
Views
42
Helpful
54
Comments
Cisco Employee

VXLAN/EVPN has been release on Nexus 9000 series in early February 2015, followed by Nexus 7000/7700 (F3/M3 Linecard) in Summer and Nexus 5600 later in 2015. Other Cisco platforms like the ASR 9000 and ASR 1000 also support VXLAN with EVPN control-plane.

As there are many request in how to configure VXLAN/EVPN on a given Platform, this Blog post should help to get you get started with a Nexus 9300/9500 (including Nexus 9x00 EX/FX)

While this example focuses on numbered IP interfaces or the so called P2P (point-to-point) approach, there is also a "ip unnumbered" example available.

Generally we would expect a Topology as shown below.

EVPN.jpg

For the sake of this example, we are using the following Topology example, which is a subset of the Topology above.

EVPN.png

The configuration example does cover the configuration of the following software components

- Underlay with OSPF, PIM Sparse (ASM) and Anycast-RP

- IP numbered interfaces (p2p interfaces)

- VXLAN

- MP-BGP EVPN Control-Plane

- VPC

We will focus on the configuration of Spine "1", Leaf "V1" and Leaf "V2"

Spine "1" Configuration:

hostname SPINE1

nv overlay evpn

feature ospf

feature bgp

feature pim

feature nv overlay

ip pim anycast-rp 10.254.254.254 10.250.250.101

ip pim rp-address 10.254.254.254 group-list 239.239.239.0/24

interface Ethernet3/1

  description Link to Leaf "V2"

  mtu 9216

  ip address 10.1.1.6/30

  ip ospf network point-to-point

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

  no shutdown

interface Ethernet3/2

  description Link to Leaf "V1"

  mtu 9216

  ip address 10.1.1.2/30

  ip ospf network point-to-point

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

  no shutdown


interface loopback0

  ip address 10.250.250.101/32

  ip ospf network point-to-point # will change OSPF interface back to Loopback; required for VPC

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

interface loopback254

  ip address 10.254.254.254/32

  ip ospf network point-to-point

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

router ospf UNDERLAY

  router-id 10.250.250.101

  log-adjacency-changes detail

router bgp 65500

  router-id 10.250.250.101

  address-family ipv4 unicast

  neighbor 10.250.250.0/24 remote-as 65500

    update-source loopback0

    address-family ipv4 unicast # optional for "show ip bgp summary" support

    address-family l2vpn evpn

      send-community both

      route-reflector-client



Leaf "V1" Configuration:

hostname LeafV1

nv overlay evpn

feature ospf

feature bgp

feature pim

feature interface-vlan

feature vn-segment-vlan-based

feature nv overlay

feature vpc

fabric forwarding anycast-gateway-mac 2020.DEAD.BEEF

ip pim rp-address 10.254.254.254 group-list 239.239.239.0/24

vlan 1,99-101,2500,3000

vlan 99

  name L2onlyHostSegment

  vn-segment 30099

vlan 100

  name L2L3HostSegment

  vn-segment 30000

vlan 101

  name L2L3HostSegment

  vn-segment 30001

vlan 2500

  name FabricBD

  vn-segment 50000

vlan 3000

  name VPCL3Peering

route-map FABRIC-RMAP-REDIST-SUBNET permit 10

  match tag 21921

vrf context TENANT1

  vni 50000

  rd auto

  address-family ipv4 unicast

    route-target both auto

    route-target both auto evpn

  address-family ipv6 unicast

    route-target both auto

    route-target both auto evpn

vpc domain 1

  peer-switch

  peer-keepalive destination 10.2.8.1 source 10.2.8.2 vrf management

  peer-gateway

  ip arp synchronize

interface Vlan100

  no shutdown

  vrf member TENANT1

  ip address 192.168.100.1/24 tag 21921

  fabric forwarding mode anycast-gateway

interface Vlan101

  no shutdown

  vrf member TENANT1

  ip address 192.168.101.1/24 tag 21921

  fabric forwarding mode anycast-gateway

interface Vlan2500

  description FabricBD

  no shutdown

  mtu 9216

  vrf member TENANT1

  ip forward

interface Vlan3000

  description VPC Layer-3 Peering for VXLAN

  no shutdown

  ip address 10.3.1.1/30 # Requires to be individual IP per VPC member

  ip ospf network point-to-point

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode


nve infra-vlans 3000

# required for Nexus 9300-EX/FX or Nexus 9200

interface port-channel1

  description VPC Peer-Link

  switchport mode trunk

  spanning-tree port type network

  lacp suspend-individual

  vpc peer-link

hardware access-list tcam region vacl 0

# example region to free up space for arp-ether region


hardware access-list tcam region arp-ether 256 double-wide

# required for ARP suppression, requires reboot

# double-wide is required starting 7.0(3)I3(1)

# not required for Nexus 9300-EX/FX or Nexus 9200

interface nve1

  mtu 9216

  no shutdown

  source-interface loopback1

  host-reachability protocol bgp

  member vni 30000

    suppress-arp

    mcast-group 239.239.239.100

  member vni 30001

    suppress-arp

    mcast-group 239.239.239.101

  member vni 30099

    mcast-group 239.239.239.99

  member vni 50000 associate-vrf

interface Ethernet1/1

  switchport mode trunk

  spanning-tree port type edge trunk

  spanning-tree bpduguard enable

interface Ethernet1/47

  description Link for VPC Peer-Link

  switchport mode trunk

  channel-group 1 mode active

interface Ethernet1/48

  description Link for VPC Peer-Link

  switchport mode trunk

  channel-group 1 mode active


interface Ethernet2/1

  description Link to Spine "1"

  no switchport

  mtu 9216

  ip address 10.1.1.1/30

  ip ospf network point-to-point

  ip router ospf UNDERLAY area 0.0.0.0

interface loopback0 # Loopback for Router ID, routing adjacency and peering

  ip address 10.250.250.102/32

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

interface loopback1 # Loopback for VTEP only

  ip address 10.254.254.102/32

  ip address 10.254.254.1/32 secondary

  ip ospf network point-to-point # will change OSPF interface back to Loopback; required for VPC

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

router ospf UNDERLAY

  router-id 10.250.250.102

  log-adjacency-changes detail

router bgp 65500

  router-id 10.250.250.102

  address-family ipv4 unicast

  neighbor 10.250.250.101 remote-as 65500

    update-source loopback0

    address-family ipv4 unicast # optional for "show ip bgp summary" support

    address-family l2vpn evpn

      send-community both

  vrf TENANT1

    address-family ipv4 unicast

      advertise l2vpn evpn

      redistribute direct route-map FABRIC-RMAP-REDIST-SUBNET

evpn

  vni 30000 l2

    rd auto

    route-target import auto

    route-target export auto

  vni 30001 l2

    rd auto

    route-target import auto

    route-target export auto

  vni 30099 l2

    rd auto

    route-target import auto

    route-target export auto

Leaf "V2" Configuration:

hostname LeafV2

nv overlay evpn

feature ospf

feature bgp

feature pim

feature interface-vlan

feature vn-segment-vlan-based

feature nv overlay

feature vpc

fabric forwarding anycast-gateway-mac 2020.DEAD.BEEF

ip pim rp-address 10.254.254.254 group-list 239.239.239.0/24

vlan 1,99-101,2500,3000

vlan 99

  name L2onlyHostSegment

  vn-segment 30099

vlan 100

  name L2L3HostSegment

  vn-segment 30000

vlan 101

  name L2L3HostSegment

  vn-segment 30001

vlan 2500

  name FabricBD

  vn-segment 50000

vlan 3000

  name VPCL3Peering

route-map FABRIC-RMAP-REDIST-SUBNET permit 10

  match tag 21921

vrf context TENANT1

  vni 50000

  rd auto

  address-family ipv4 unicast

    route-target both auto

    route-target both auto evpn

  address-family ipv6 unicast

    route-target both auto

    route-target both auto evpn

vpc domain 1

  peer-switch

  peer-keepalive destination 10.2.8.2 source 10.2.8.1 vrf management

  peer-gateway

  ip arp synchronize

interface Vlan100

  no shutdown

  vrf member TENANT1

  ip address 192.168.100.1/24 tag 21921

  fabric forwarding mode anycast-gateway

interface Vlan101

  no shutdown

  vrf member TENANT1

  ip address 192.168.101.1/24 tag 21921

  fabric forwarding mode anycast-gateway

interface Vlan2500

  description FabricBD

  no shutdown

  mtu 9216

  vrf member TENANT1

  ip forward

interface Vlan3000

  description VPC Layer-3 Peering for VXLAN

  no shutdown

  ip address 10.3.1.2/30 # Requires to be individual IP per VPC member

  ip ospf network point-to-point

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode


nve infra-vlans 3000

# required for Nexus 9300-EX/FX or Nexus 9200

interface port-channel1

  description VPC Peer-Link

  switchport mode trunk

  spanning-tree port type network

  lacp suspend-individual

  vpc peer-link


hardware access-list tcam region vacl 0

# example region to free up space for arp-ether region


hardware access-list tcam region arp-ether 256 double-wide

# required for ARP suppression, requires reboot

# double-wide is required starting 7.0(3)I3(1)

# not required for Nexus 9300-EX/FX or Nexus 9200

interface nve1

  mtu 9216

  no shutdown

  source-interface loopback1

  host-reachability protocol bgp

  member vni 30000

    suppress-arp

    mcast-group 239.239.239.100

  member vni 30001

    suppress-arp

    mcast-group 239.239.239.101

  member vni 30099

    mcast-group 239.239.239.99

  member vni 50000 associate-vrf

interface Ethernet1/1

  switchport mode trunk

  spanning-tree port type edge trunk

  spanning-tree bpduguard enable

interface Ethernet1/47

  description Link for VPC Peer-Link

  switchport mode trunk

  channel-group 1 mode active

interface Ethernet1/48

  description Link for VPC Peer-Link

  switchport mode trunk

  channel-group 1 mode active


interface Ethernet2/1

  description Link to Spine "1"

  no switchport

  mtu 9216

  ip address 10.1.1.5/30

  ip ospf network point-to-point

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

  no shutdown

interface loopback0 # Loopback for Router ID, routing adjacency and peering

  ip address 10.250.250.103/32

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

interface loopback1 # Loopback for VTEP only

  ip address 10.254.254.103/32

  ip address 10.254.254.1/32 secondary

  ip ospf network point-to-point # will change OSPF interface back to Loopback; required for VPC

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

router ospf UNDERLAY

  router-id 10.250.250.103

  log-adjacency-changes detail

router bgp 65500

  router-id 10.250.250.103

  address-family ipv4 unicast

  neighbor 10.250.250.101 remote-as 65500

    update-source loopback0

    address-family ipv4 unicast # optional for "show ip bgp summary" support

    address-family l2vpn evpn

      send-community both

  vrf TENANT1

    address-family ipv4 unicast

      advertise l2vpn evpn

      redistribute direct route-map FABRIC-RMAP-REDIST-SUBNET

      maximum-paths ibgp 2

evpn

  vni 30000 l2

    rd auto

    route-target import auto

    route-target export auto

  vni 30001 l2

    rd auto

    route-target import auto

    route-target export auto

  vni 30099 l2

    rd auto

    route-target import auto

    route-target export auto


54 Comments
Enthusiast

Thanks for the overview Lukas. Do you have an estimate on when EVPN will be supported on the N5600s other than late 2015?

Cisco Employee

Dear Matthew,

please reach out to your trusted Cisco contact or myself for Early Field Trial (EFT). As you already know, we are aiming to deliver VXLAN/EVPN on N5600 within the Calendar Year 2015

Beginner

This is fantastic thanks Lukas. How much would the config differ for a ring of 4 x 9300's (no spines)? I am lab'ing a design of 2 per DC as a collapsed core to routed-access 4500E-Sup8's and 3850 stacks. I'm also not sure whether vPC will be a part.

Cisco Employee

Hi Brendan,

a ring would eliminate the functionality of the Spine. Either you create full-mesh BGP or you use a Leaf as a RR.

In regards to building a Campus Fabric with VXLAN/EVPN, would you mind to provide some additional info via direct email.

Kind Regards

-Lukas

Beginner

Hi Lukas, thanks for the reply. Full-mesh BGP seems the easy way given our limited plans to scale. How do I send you an email? Can't find the address listed.

Beginner

Hi Brendan,

Hi Lucas,


I am trying also to implement DCI with 4 x 9300's (no spines). I was wondering are there any concerns (except platform performance) about having RR,eBGP and vPC on the pair of leaf switches in each DC?


Is it possible to share with me additional info regarding this design.


Thank you in advance!


BR,

Stanislav

Beginner

Hi Stanislav, for vPC I have used separate physical links to ensure only vPC transits a particular port-channel as per the vPC recommendations, but I did wonder about using SVI's on that trunk for the underlay instead. But given the number of QSFP ports available and the low-cost of twinax 40G cables I went with dedicated.

Also you don't need RR's given you have no spines.

Beginner

Hi Lukas,

Using you same previous configuration you have I added a redistribute statement to inject the routes from BGP back to EIGRP but I don't see any of those routes in the EIGRP topology table for that VRF, is there anything special I have to do.

Thanks.

Enthusiast

I have a question on using the 7702 with the F3 card as a VXLAN BGP EVPN leaf. It shows in the NEXUS 7000 config guide for NX-OS VXLAN that the "feature vn-segment"(to map VLAN to VNID) and the "fabric forwarding anycast-gateway-mac" are not there in the 7k code. in the release notes it says that

This feature set includes; Flood and Learn using outer multicast group for Broadcast, unknown unicast and multicast traffic, and L2/L3 VXLAN Gateway.

VXLAN with the MP-BGP/EVPN control plane is supported with the Cisco Nexus 7000 series switch acting as border-leaf with no L2 gateway functionality, vPC or ingress replication support.

SO the documentation is ahead of what the code supports? I followed your config as well as your docs for Cisco live and have a 7702 with F3 carved into VDC's for spine and leafs, and I have two 9300's for leafs. The 7k spines and 9300 leafs work but I don't think the support is there to make a 7k act the same way as a 9300 leaf with L2 and L3 VXLAN support with BGP EVPN. This is a training lab environment I am trying to build for internal engineers and customers so was trying it with VDC's to cut down on physical boxes as leafs. . Any insight would be appreciated

Beginner

Hi Lukas,

thank you for your great work promoting vxlan with evpn control plane. Its potential seems be huge moving forward.

I was wondering if you are aware of any available resources for hands on with this technology, like PEC or similar?

Thanks and regards

Gustavo Novais

Beginner

Hello Lukas,

 

Thanks for this blog post. I have a general question about using SVI’s on the vteps.  I’m using n9372tx. Have vxlan/evpn similar to above. I do not have anycast GW configured nor do I have VPC.  I just wanted an SVI that I could ping.  I am not able to ping SVI IPs between vtep1 and vtep2.  Both are in the same vlan and vni.  

 

I can; however,  do the following:

 

Legend:

   Vtep1: interface Vlan2: ipaddr 198.51.100.1

   Vtep2: interface Vlan2: ipaddr 198.51.100.2

   HostA IP = 198.51.100.11

   HostB IP = 198.51.100.12

 

From HostA ping 198.51.100.1 -> success

From HostB ping 198.51.100.2 -> success

From HostA ping HostB 198.51.100.12 -> success

From HostB ping HostA 198.51.100.11 -> success

From HostA ping 198.51.100.2 -> fails

From HostB ping 198.51.100.1 -> fails

From Vtep1 ping Vtep2:Vlan2 198.51.100.2 -> fails (using ping vrf evpn-tenant1)

From Vtep2 ping Vtep1:Vlan2 198.51.100.1 -> fails (using ping vrf evpn-tenant1)

 

Is this expected?  Before I post my configs, I wanted to verify that individual SVIs were supported and not just SVIs for anycast GWs.

- T.L.Bowers

Cisco Employee

Thanks Gustavo,

We are working on getting the lab into PEC (Gold) but it is still some work to do there.

Stay tuned

-Lukas

Cisco Employee

Hi Tracey,

Thanks for your post

We are focusing only on Anycast Gateway in EVPN. Also, as your traffic for the End-Points (Hosts) and SVI should be in a VRF, the ping from SVI (VRF A) to VTEP (default-VRF) could not work as in different VRFs.

I hope this makes sense

Kind Regards

-Lukas

Beginner

Hi

I have read your configuration with a great interest.

I have a question concerning your configuration

Is it compatible with an external access switch with spanning-tree enabled ?

I mean, if I have 2 leaves with no VPC and an access switch with dual attachement

For me I am obliged to define my IP Fabric as root ridge for all VLANs and

enable STP on trunk links.

Is this design possible/recommended ?

Does STP BPDU travel through my IP Fabric (I mean inside my VXLAN tunnels

using nve interfaces) ?

Thanks for your response

Regards

Cisco Employee

Dear Amadou,

Thanks for the nice feedback, very much appreciated!

a VXLAN Switch (VTEP) can be connected to a Classic Ethernet (CE) Layer-2 Switch with Spanning-Tree enabled. There is no BPDU Guard configured on the VTEPs ports. Nevertheless, VXLAN is not forwarding BPDU or interacts with Spanning-Tree blck/fwd semantics and thus can introduce a loop. In the case you describe, where two non-VPC VTEP are both connected to a CE Switch, you would create a backdoor link and respectively a loop as per described behavior above.

The design of using the VTEP as Spanning-Tree root and using root-guard will definitely help to improve the situation but still, there is a chance for creating a loop.

I would not recommend at all to dual-connect a CE Switch to a non-multihomed (VPC) pair of VTEPs.

Best practice is to use a VPC domain and connect your CE Switch or Spanning-tree domain to the VXLAN domain.

I hope this helps. Kind Regards

-Lukas

CreatePlease to create content
Content for Community-Ad
August's Community Spotlight Awards
This widget could not be displayed.