cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
47828
Views
42
Helpful
56
Comments
Lukas Krattiger
Cisco Employee
Cisco Employee

VXLAN/EVPN has been release on Nexus 9000 series in early February 2015, followed by Nexus 7000/7700 (F3/M3 Linecard) in Summer and Nexus 5600 later in 2015. Other Cisco platforms like the ASR 9000 and ASR 1000 also support VXLAN with EVPN control-plane.

As there are many request in how to configure VXLAN/EVPN on a given Platform, this Blog post should help to get you get started with a Nexus 9300/9500 (including Nexus 9x00 EX/FX)

While this example focuses on numbered IP interfaces or the so called P2P (point-to-point) approach, there is also a "ip unnumbered" example available.

Generally we would expect a Topology as shown below.

EVPN.jpg

For the sake of this example, we are using the following Topology example, which is a subset of the Topology above.

EVPN.png

The configuration example does cover the configuration of the following software components

- Underlay with OSPF, PIM Sparse (ASM) and Anycast-RP

- IP numbered interfaces (p2p interfaces)

- VXLAN

- MP-BGP EVPN Control-Plane

- VPC

We will focus on the configuration of Spine "1", Leaf "V1" and Leaf "V2"

Spine "1" Configuration:

hostname SPINE1

nv overlay evpn

feature ospf

feature bgp

feature pim

feature nv overlay

ip pim anycast-rp 10.254.254.254 10.250.250.101

ip pim rp-address 10.254.254.254 group-list 239.239.239.0/24

interface Ethernet3/1

  description Link to Leaf "V2"

  mtu 9216

  ip address 10.1.1.6/30

  ip ospf network point-to-point

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

  no shutdown

interface Ethernet3/2

  description Link to Leaf "V1"

  mtu 9216

  ip address 10.1.1.2/30

  ip ospf network point-to-point

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

  no shutdown


interface loopback0

  ip address 10.250.250.101/32

  ip ospf network point-to-point # will change OSPF interface back to Loopback; required for VPC

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

interface loopback254

  ip address 10.254.254.254/32

  ip ospf network point-to-point

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

router ospf UNDERLAY

  router-id 10.250.250.101

  log-adjacency-changes detail

router bgp 65500

  router-id 10.250.250.101

  address-family ipv4 unicast

  neighbor 10.250.250.0/24 remote-as 65500

    update-source loopback0

    address-family ipv4 unicast # optional for "show ip bgp summary" support

    address-family l2vpn evpn

      send-community both

      route-reflector-client



Leaf "V1" Configuration:

hostname LeafV1

nv overlay evpn

feature ospf

feature bgp

feature pim

feature interface-vlan

feature vn-segment-vlan-based

feature nv overlay

feature vpc

fabric forwarding anycast-gateway-mac 2020.DEAD.BEEF

ip pim rp-address 10.254.254.254 group-list 239.239.239.0/24

vlan 1,99-101,2500,3000

vlan 99

  name L2onlyHostSegment

  vn-segment 30099

vlan 100

  name L2L3HostSegment

  vn-segment 30000

vlan 101

  name L2L3HostSegment

  vn-segment 30001

vlan 2500

  name FabricBD

  vn-segment 50000

vlan 3000

  name VPCL3Peering

route-map FABRIC-RMAP-REDIST-SUBNET permit 10

  match tag 21921

vrf context TENANT1

  vni 50000

  rd auto

  address-family ipv4 unicast

    route-target both auto

    route-target both auto evpn

  address-family ipv6 unicast

    route-target both auto

    route-target both auto evpn

vpc domain 1

  peer-switch

  peer-keepalive destination 10.2.8.1 source 10.2.8.2 vrf management

  peer-gateway

  ip arp synchronize

interface Vlan100

  no shutdown

  vrf member TENANT1

  ip address 192.168.100.1/24 tag 21921

  fabric forwarding mode anycast-gateway

interface Vlan101

  no shutdown

  vrf member TENANT1

  ip address 192.168.101.1/24 tag 21921

  fabric forwarding mode anycast-gateway

interface Vlan2500

  description FabricBD

  no shutdown

  mtu 9216

  vrf member TENANT1

  ip forward

interface Vlan3000

  description VPC Layer-3 Peering for VXLAN

  no shutdown

  ip address 10.3.1.1/30 # Requires to be individual IP per VPC member

  ip ospf network point-to-point

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode


nve infra-vlans 3000

# required for Nexus 9300-EX/FX or Nexus 9200

interface port-channel1

  description VPC Peer-Link

  switchport mode trunk

  spanning-tree port type network

  lacp suspend-individual

  vpc peer-link

hardware access-list tcam region vacl 0

# example region to free up space for arp-ether region


hardware access-list tcam region arp-ether 256 double-wide

# required for ARP suppression, requires reboot

# double-wide is required starting 7.0(3)I3(1)

# not required for Nexus 9300-EX/FX or Nexus 9200

interface nve1

  mtu 9216

  no shutdown

  source-interface loopback1

  host-reachability protocol bgp

  member vni 30000

    suppress-arp

    mcast-group 239.239.239.100

  member vni 30001

    suppress-arp

    mcast-group 239.239.239.101

  member vni 30099

    mcast-group 239.239.239.99

  member vni 50000 associate-vrf

interface Ethernet1/1

  switchport mode trunk

  spanning-tree port type edge trunk

  spanning-tree bpduguard enable

interface Ethernet1/47

  description Link for VPC Peer-Link

  switchport mode trunk

  channel-group 1 mode active

interface Ethernet1/48

  description Link for VPC Peer-Link

  switchport mode trunk

  channel-group 1 mode active


interface Ethernet2/1

  description Link to Spine "1"

  no switchport

  mtu 9216

  ip address 10.1.1.1/30

  ip ospf network point-to-point

  ip router ospf UNDERLAY area 0.0.0.0

interface loopback0 # Loopback for Router ID, routing adjacency and peering

  ip address 10.250.250.102/32

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

interface loopback1 # Loopback for VTEP only

  ip address 10.254.254.102/32

  ip address 10.254.254.1/32 secondary

  ip ospf network point-to-point # will change OSPF interface back to Loopback; required for VPC

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

router ospf UNDERLAY

  router-id 10.250.250.102

  log-adjacency-changes detail

router bgp 65500

  router-id 10.250.250.102

  address-family ipv4 unicast

  neighbor 10.250.250.101 remote-as 65500

    update-source loopback0

    address-family ipv4 unicast # optional for "show ip bgp summary" support

    address-family l2vpn evpn

      send-community both

  vrf TENANT1

    address-family ipv4 unicast

      advertise l2vpn evpn

      redistribute direct route-map FABRIC-RMAP-REDIST-SUBNET

evpn

  vni 30000 l2

    rd auto

    route-target import auto

    route-target export auto

  vni 30001 l2

    rd auto

    route-target import auto

    route-target export auto

  vni 30099 l2

    rd auto

    route-target import auto

    route-target export auto

Leaf "V2" Configuration:

hostname LeafV2

nv overlay evpn

feature ospf

feature bgp

feature pim

feature interface-vlan

feature vn-segment-vlan-based

feature nv overlay

feature vpc

fabric forwarding anycast-gateway-mac 2020.DEAD.BEEF

ip pim rp-address 10.254.254.254 group-list 239.239.239.0/24

vlan 1,99-101,2500,3000

vlan 99

  name L2onlyHostSegment

  vn-segment 30099

vlan 100

  name L2L3HostSegment

  vn-segment 30000

vlan 101

  name L2L3HostSegment

  vn-segment 30001

vlan 2500

  name FabricBD

  vn-segment 50000

vlan 3000

  name VPCL3Peering

route-map FABRIC-RMAP-REDIST-SUBNET permit 10

  match tag 21921

vrf context TENANT1

  vni 50000

  rd auto

  address-family ipv4 unicast

    route-target both auto

    route-target both auto evpn

  address-family ipv6 unicast

    route-target both auto

    route-target both auto evpn

vpc domain 1

  peer-switch

  peer-keepalive destination 10.2.8.2 source 10.2.8.1 vrf management

  peer-gateway

  ip arp synchronize

interface Vlan100

  no shutdown

  vrf member TENANT1

  ip address 192.168.100.1/24 tag 21921

  fabric forwarding mode anycast-gateway

interface Vlan101

  no shutdown

  vrf member TENANT1

  ip address 192.168.101.1/24 tag 21921

  fabric forwarding mode anycast-gateway

interface Vlan2500

  description FabricBD

  no shutdown

  mtu 9216

  vrf member TENANT1

  ip forward

interface Vlan3000

  description VPC Layer-3 Peering for VXLAN

  no shutdown

  ip address 10.3.1.2/30 # Requires to be individual IP per VPC member

  ip ospf network point-to-point

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode


nve infra-vlans 3000

# required for Nexus 9300-EX/FX or Nexus 9200

interface port-channel1

  description VPC Peer-Link

  switchport mode trunk

  spanning-tree port type network

  lacp suspend-individual

  vpc peer-link


hardware access-list tcam region vacl 0

# example region to free up space for arp-ether region


hardware access-list tcam region arp-ether 256 double-wide

# required for ARP suppression, requires reboot

# double-wide is required starting 7.0(3)I3(1)

# not required for Nexus 9300-EX/FX or Nexus 9200

interface nve1

  mtu 9216

  no shutdown

  source-interface loopback1

  host-reachability protocol bgp

  member vni 30000

    suppress-arp

    mcast-group 239.239.239.100

  member vni 30001

    suppress-arp

    mcast-group 239.239.239.101

  member vni 30099

    mcast-group 239.239.239.99

  member vni 50000 associate-vrf

interface Ethernet1/1

  switchport mode trunk

  spanning-tree port type edge trunk

  spanning-tree bpduguard enable

interface Ethernet1/47

  description Link for VPC Peer-Link

  switchport mode trunk

  channel-group 1 mode active

interface Ethernet1/48

  description Link for VPC Peer-Link

  switchport mode trunk

  channel-group 1 mode active


interface Ethernet2/1

  description Link to Spine "1"

  no switchport

  mtu 9216

  ip address 10.1.1.5/30

  ip ospf network point-to-point

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

  no shutdown

interface loopback0 # Loopback for Router ID, routing adjacency and peering

  ip address 10.250.250.103/32

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

interface loopback1 # Loopback for VTEP only

  ip address 10.254.254.103/32

  ip address 10.254.254.1/32 secondary

  ip ospf network point-to-point # will change OSPF interface back to Loopback; required for VPC

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

router ospf UNDERLAY

  router-id 10.250.250.103

  log-adjacency-changes detail

router bgp 65500

  router-id 10.250.250.103

  address-family ipv4 unicast

  neighbor 10.250.250.101 remote-as 65500

    update-source loopback0

    address-family ipv4 unicast # optional for "show ip bgp summary" support

    address-family l2vpn evpn

      send-community both

  vrf TENANT1

    address-family ipv4 unicast

      advertise l2vpn evpn

      redistribute direct route-map FABRIC-RMAP-REDIST-SUBNET

      maximum-paths ibgp 2

evpn

  vni 30000 l2

    rd auto

    route-target import auto

    route-target export auto

  vni 30001 l2

    rd auto

    route-target import auto

    route-target export auto

  vni 30099 l2

    rd auto

    route-target import auto

    route-target export auto


56 Comments
wang.weibin
Level 1
Level 1

Great post Lukas!

If I want to replace underlay ospf/ibgp with ebgp only based on this famous draft draft-ietf-rtgwg-bgp-routing-large-dc-02 , what will be the Evpn vxlan vni config looks like ( auto rt population no longer works in ebgp).

Would you compare pros and cons between ospf/ibgp and ebgp underlay in a evpn vxlan overlay fabric design?

Thanks!

Lukas Krattiger
Cisco Employee
Cisco Employee

Weibin,

This is absolutely possible and upcoming posts will cover the use of eBGP for Underlay and Overlay.

As you already lined out, there will be additional work in configuration like Next-Hop Unchanged, retain Route-Target and eBGP peering configuration as well as consideration in design aspects (AS#, route-targets, etc).

The draft linked focuses mainly on a Layer-3 fabric and not particular for a Underlay hosting VXLAN Overlay. As said, it is possible to follow this draft to implement a Underlay this way but certain parts will simply not apply like section 5.2.3 others might introduce unnecessary complexity (what is learned first; Link IP Prefix, VTEP IP Prefix or End-Host Prefixes).

There are some significant advantage in decoupling the Underlay and Overlay Control-Plane with little additional config lines.

I did cover some of these considerations in BRKDCT-3378 back at Cisco Live Milan and we will repeat this session during Cisco Live in San Diego.

Kind Regards

-Lukas

Advanced - Building simplified, automated and scalable DataCenter network with Overlays (VXLAN/FabricPath) (2015 Milan)

Building Data Center Networks with Overlays (VXLAN/EVPN & FabricPath) (2015 San Diego)

wang.weibin
Level 1
Level 1

Great, thanks Lukas. Looking forward to your next post and attending BRKDCT-3378 session.

-Weibin

Robert Parten
Level 1
Level 1

In my VXLAN environment I am actually noticing the Border Leaf switches, and we're using MP-BPG eVPN with ingress-replication, are able to peer upstream with EIGRP; however, under the BGP tenant VRF, address-family ipv4 unicast, we're not seeing any of the EIGRP learned routes be distributed amongst the other leaves other than the 2 Border leaf switches. Also, yes, we do have advertise l2vpn evpn enabled as I followed the Nexus 9K guide to configure it but I remain stumped,. Any sugggstions?

Robert Parten
Level 1
Level 1

Also, just to mention, this is an iBGP setup between leaf and spine.

Lukas Krattiger
Cisco Employee
Cisco Employee

Hi Robert,

I did create a Border Leaf configuration based on the Leaf configuration above without VPC. The configuration works as expected and I can see the respective EIGRP learned route on a internal Leaf received via iBGP EVPN AFI.

Kind Regards

-Lukas

Border Leaf “BL1” Configuration (BGP and EIGRP):

router eigrp Core

  autonomous-system 65500

  vrf TENANT1

    autonomous-system 65500

    address-family ipv4 unicast

interface Ethernet1/2

  vrf member TENANT1

  ip address 192.168.1.1/30

  ip router eigrp Core

  ip summary-address eigrp Core 192.168.0.0.0/16

  no shutdown

route-map REDIST-EIGRP permit 10

  match source-protocol EIGRP Core

router bgp 65500

  router-id 10.254.254.104

  address-family ipv4 unicast

  neighbor 10.254.254.101 remote-as 65500

    update-source loopback0

    address-family ipv4 unicast # optional for "show ip bgp summary" support

    address-family l2vpn evpn

      send-community both

  vrf TENANT1

    address-family ipv4 unicast

      advertise l2vpn evpn

      redistribute direct route-map FABRIC-RMAP-REDIST-SUBNET

      redistribute eigrp Core route-map REDIST-EIGRP

      maximum-paths ibgp 2

Border Leaf “BL1” Verification (BGP and EIGRP):

BL1# show ip route EIGRP vrf TENANT1

IP Route Table for VRF “TENANT1”

'*' denotes best ucast next-hop

'**' denotes best mcast next-hop

'[x/y]' denotes [preference/metric]

'%<string>' in via output denotes VRF <string>

192.168.0.0/16, ubest/mbest: 1/0

    *via Null0, [5/2816], 06:20:52, eigrp-Core, discard

200.200.200.200/32, ubest/mbest: 1/0

    *via 192.168.1.2, Eth1/2, [90/130816], 06:20:42, eigrp-Core, internal

BL1#

BL1#

BL1# show bgp l2vpn evpn 200.200.200.200 vrf TENANT1

Route Distinguisher: 10.254.254.104:3    (L3VNI 50000)

BGP routing table entry for [5]:[0]:[0]:[32]:[200.200.200.200]:[0.0.0.0]/224, version 4

Paths: (1 available, best #1)

Flags: (0x00000a) on xmit-list, is not in l2rib/evpn

  Advertised path-id 1

  Path type: local, path is valid, is best path

  AS-Path: NONE, path locally originated

    10.254.254.104 (metric 0) from 0.0.0.0 (10.254.254.104)

      Origin incomplete, MED 130816, localpref 100, weight 32768

      Received label 50000

      Extcommunity:  RT:65500:50000

  Path-id 1 advertised to peers:

    10.254.254.101

BL1#

Leaf “V1” Verification (BGP):

V1# show bgp l2vpn evpn 200.200.200.200 vrf TENANT1

BGP routing table information for VRF default, address family L2VPN EVPN

Route Distinguisher: 10.254.254.104:3

BGP routing table entry for [5]:[0]:[0]:[32]:[200.200.200.200]:[0.0.0.0]/224, version 10

Paths: (1 available, best #1)

Flags: (0x000002) on xmit-list, is not in l2rib/evpn, is locked

  Advertised path-id 1

  Path type: internal, path is valid, is best path, no labeled nexthop

  AS-Path: NONE, path sourced internal to AS

    10.254.254.104 (metric 43) from 10.254.254.104 (10.254.254.104)

      Origin incomplete, MED 130816, localpref 100, weight 0

      Received label 50000

      Extcommunity:  RT:65500:50000 ENCAP:8 Router MAC:002a.6a64.a142

  Path-id 1 not advertised to any peer

Route Distinguisher: 10.254.254.102:3    (L3VNI 50000)

BGP routing table entry for [5]:[0]:[0]:[32]:[200.200.200.200]:[0.0.0.0]/224, version 13

Paths: (1 available, best #1)

Flags: (0x00001a) on xmit-list, is in l2rib/evpn

  Advertised path-id 1

  Path type: internal, path is valid, is best path, no labeled nexthop

             Imported from 10.254.254.104:3:[5]:[0]:[0]:[32]:[200.200.200.200]:[0.0.0.0]/120

  AS-Path: NONE, path sourced internal to AS

    10.254.254.104 (metric 43) from 10.254.254.104 (10.254.254.104)

      Origin incomplete, MED 130816, localpref 100, weight 0

      Received label 50000

      Extcommunity:  RT:65500:50000 ENCAP:8 Router MAC:002a.6a64.a142

  Path-id 1 not advertised to any peer

V1#

V1#

V1# show ip route 200.200.200.200 vrf TENANT1

IP Route Table for VRF “TENANT1”

'*' denotes best ucast next-hop

'**' denotes best mcast next-hop

'[x/y]' denotes [preference/metric]

'%<string>' in via output denotes VRF <string>

200.200.200.200/32, ubest/mbest: 1/0

    *via 10.254.254.104%default, [200/130816], 06:41:19, bgp-65500, internal, tag 65500 (evpn)segid: 50000 tunnelid: 0x64176432 encap: 1

V1#

Lukas Krattiger
Cisco Employee
Cisco Employee

The outputs posted are from a iBGP setup with OSPF as Underlay and EIGRP for external VRF-Lite routing.

Kind Regards

-Lukas

Robert Parten
Level 1
Level 1

Thanks for the response Lukas.

I actually discovered what I was doing wrong and fixed it and it does look exactly as you configured, it was my forgetting this is MP-BGP and I needed to define the correct address family, simple mistakes cause the worst headaches at 4:00am.

Thanks for helping out the community.

Robert Parten
Level 1
Level 1

Gotcha, I too utilize OSPF and iBGP internally with RRs.

I personally prefer just to have the default route injected into the tenant VRF beacause we only have a single point of ingress/egress externally so I can't see why it would benefit me to use additional cycles to find a longer prefix when the default gateway will always be the VTEP of the Border Leafs. What say you, Lukas?

Lukas Krattiger
Cisco Employee
Cisco Employee

I personally follow the approach of only the default route within a Tenant, this helps me to save resources and removes the requirement of redistribution or originate commands, this applies specifically when using eBGP.

It also works with IGPs but requires additional configuration to ensure the default-route to disappear once external connectivity is gone (redistribution; not required with eBGP).

For troubleshooting purpose, I prefer to have a specific route being seen end-to-end and thus the example above. Slowly going step-by-step until we find a potential problem and mitigate it :-)

-Lukas

Robert Parten
Level 1
Level 1

Peeling back the layers is always a good idea.

Now, what about Jumbo frames between hosts in a VXLAN environment? Seems like it won't work for me even though everything has 9216 set for the MTU but from one host to another, in the same VXLAN VLAN, pinging with MTU 800, nadda. :/

Lukas Krattiger
Cisco Employee
Cisco Employee

Hi Robert,

I would suggest to peal back the layers, as you suggested :-)

- Server NIC Driver

- Virtual Switch (if VM)

- Switchport where Server or Hypervisor is connected

- SVI for Distributed Anycast Gateway (if Routed Traffic)

- SVI for VRF (if Routed Traffic)

- Routed-Interfaces towards Core

If all these interfaces are set to the respective MTU, it will work.

Kind Regards

-Lukas

Robert Parten
Level 1
Level 1

All those are a check, check, check. But with Cisco UCS in the mix, do you know of any documentation for setting up Cisco UCS with Jumbo frames for VXLAN. I've experimented with a few options, but I am trying ot keep my options open with integrating UCS with VXLAN using Jumbo frames.

Lukas Krattiger
Cisco Employee
Cisco Employee

Robert,

UCS is not aware of VXLAN if you do Network based Overlay, the VXLAN encap happens only at the first Switch (TOR) level. For the approach we are discussing here, we are talking about vanilla Jumbo frame support on UCS.

For UCS B-Series, you require to config Jumbo MTU on the FI (QoS) and also under the NIC (VIC) for the respective Blade. I think the following Link will help UCS with VMware ESXi End-to-End Jumbo MTU Configuration Example - Cisco

Robert Parten
Level 1
Level 1

Hey Lukas,

My apologies for not being clear, I sometimes don't always explains things completely. In the scenarios I was in, we had one of the classes, SILVER class, with a COS marking set for Jumbo and according to the RFC the VTEP does not preserve the 802.1Q information so when a COS marked frame is encapsulated in VXLAN that marking is gone and the receiving end UCS sees no COS marking and it delivered to Best-effort, which did not have MTU 9216 set; thus, anything with the df-bit set failed. However, starting with the vDS, I just set a DSCP value that is carried through VXLAN and on the switches a class-map to match the DSCP value and then a policy-map to set those DSCP matched packets with a COS value at the port-channel leading to the UCS and everything works right there.

I was actually confusing myself with how another implementation of VXLAN copies both the original DSCP and COS markings; however, it appears the Cisco implementation of VXLAN follows the RFC a lot closer than some vendors implementation, which is a good thing.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: