VXLAN/EVPN Configuration Example (N9k / ip unnumbered)

[Lukas Krattiger]

After many question on "ip unnumbered" and how the configuration would look, I took a spin on the already released "ip numbered" or "p2p" approach that I posted back in May 2015.

For the sake of this example, we are using the following Topology example.

The configuration example does cover the configuration of the following software components

- Underlay with OSPF, PIM Sparse (ASM) and Anycast-RP

- IP unnumbered interfaces

- VXLAN

- MP-BGP EVPN Control-Plane

- VPC

We will focus on the configuration of Spine "1", Leaf "V1" and Leaf "V2"

Spine "1" Configuration:

hostname SPINE1

nv overlay evpn

feature ospf

feature bgp

feature pim

feature nv overlay

ip pim anycast-rp 10.254.254.254 10.250.250.101

ip pim rp-address 10.254.254.254 group-list 239.239.239.0/24

interface Ethernet3/1

  description Link to Leaf "V2"

  mtu 9216

  ip unnumbered loopback0

  ip ospf network point-to-point

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

  no shutdown

interface Ethernet3/2

  description Link to Leaf "V1"

  mtu 9216

  ip unnumbered loopback0

  ip ospf network point-to-point

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

  no shutdown

interface loopback0

  ip address 10.250.250.101/32

  ip ospf network point-to-point # will change OSPF interface back to Loopback; required for VPC

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

interface loopback254

  ip address 10.254.254.254/32

  ip ospf network point-to-point

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

router ospf UNDERLAY

  router-id 10.250.250.101

  log-adjacency-changes detail

router bgp 65500

  router-id 10.250.250.101

  address-family ipv4 unicast

  neighbor 10.250.250.0/24 remote-as 65500

    update-source loopback0

    address-family ipv4 unicast # optional for "show ip bgp summary" support

    address-family l2vpn evpn

      send-community both

      route-reflector-client

Leaf "V1" Configuration:

hostname LeafV1

nv overlay evpn

feature ospf

feature bgp

feature pim

feature interface-vlan

feature vn-segment-vlan-based

feature nv overlay

feature vpc

fabric forwarding anycast-gateway-mac 2020.DEAD.BEEF

ip pim rp-address 10.254.254.254 group-list 239.239.239.0/24

vlan 1,99-101,2500,3000

vlan 99

  name L2onlyHostSegment

  vn-segment 30099

vlan 100

  name L2L3HostSegment

  vn-segment 30000

vlan 101

  name L2L3HostSegment

  vn-segment 30001

vlan 2500

  name FabricBD

  vn-segment 50000

vlan 3000

  name VPCL3Peering

route-map FABRIC-RMAP-REDIST-SUBNET permit 10

  match tag 21921

vrf context TENANT1

  vni 50000

  rd auto

  address-family ipv4 unicast

    route-target both auto

    route-target both auto evpn

  address-family ipv6 unicast

    route-target both auto

    route-target both auto evpn

vpc domain 1

  peer-switch

  peer-keepalive destination 10.2.8.1 source 10.2.8.2 vrf management

  peer-gateway

  ip arp synchronize

interface Vlan100

  no shutdown

  vrf member TENANT1

  ip address 192.168.100.1/24 tag 21921

  fabric forwarding mode anycast-gateway

interface Vlan101

  no shutdown

  vrf member TENANT1

  ip address 192.168.101.1/24 tag 21921

  fabric forwarding mode anycast-gateway

interface Vlan2500

  description FabricBD

  no shutdown

  mtu 9216

  vrf member TENANT1

  ip forward

interface Vlan3000

  description VPC Layer-3 Peering for VXLAN

  no shutdown

  ip address 10.3.1.1/30 # Requires to be individual IP per VPC member

  ip ospf network point-to-point

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

nve infra-vlans 3000

# required for Nexus 9x00-EX/FX or Nexus 9200

interface port-channel1

  description VPC Peer-Link

  switchport mode trunk

  spanning-tree port type network

  lacp suspend-individual

  vpc peer-link

hardware access-list tcam region vacl 0

# example region to free up space for arp-ether region

hardware access-list tcam region arp-ether 256 double-wide

# required for ARP suppression, requires reboot

# double-wide is required starting 7.0(3)I3(1)

# not required for Nexus 9x00-EX/FX or Nexus 9200

interface nve1

  mtu 9216

  no shutdown

  source-interface loopback1

  host-reachability protocol bgp

  member vni 30000

    suppress-arp

    mcast-group 239.239.239.100

  member vni 30001

    suppress-arp

    mcast-group 239.239.239.101

  member vni 30099

    mcast-group 239.239.239.99

  member vni 50000 associate-vrf

interface Ethernet1/1

  switchport mode trunk

  spanning-tree port type edge trunk

  spanning-tree bpduguard enable

interface Ethernet1/47

  description Link for VPC Peer-Link

  switchport mode trunk

  channel-group 1 mode active

interface Ethernet1/48

  description Link for VPC Peer-Link

  switchport mode trunk

  channel-group 1 mode active

interface Ethernet2/1

  description Link to Spine "1"

  no switchport

  mtu 9216

  ip unnumbered loopback0

  ip ospf network point-to-point

  ip router ospf UNDERLAY area 0.0.0.0

interface loopback0 # Loopback for Router ID, routing adjacency and peering

  ip address 10.250.250.102/32

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

interface loopback1 # Loopback for VTEP only

  ip address 10.254.254.102/32

  ip address 10.254.254.1/32 secondary

  ip ospf network point-to-point # will change OSPF interface back to Loopback; required for VPC

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

router ospf UNDERLAY

  router-id 10.250.250.102

  log-adjacency-changes detail

router bgp 65500

  router-id 10.250.250.102

  address-family ipv4 unicast

  neighbor 10.250.250.101 remote-as 65500

    update-source loopback0

    address-family ipv4 unicast # optional for "show ip bgp summary" support

    address-family l2vpn evpn

      send-community both

  vrf TENANT1

    address-family ipv4 unicast

      advertise l2vpn evpn

      redistribute direct route-map FABRIC-RMAP-REDIST-SUBNET

evpn

  vni 30000 l2

    rd auto

    route-target import auto

    route-target export auto

  vni 30001 l2

    rd auto

    route-target import auto

    route-target export auto

  vni 30099 l2

    rd auto

    route-target import auto

    route-target export auto

Leaf "V2" Configuration:

hostname LeafV2

nv overlay evpn

feature ospf

feature bgp

feature pim

feature interface-vlan

feature vn-segment-vlan-based

feature nv overlay

feature vpc

fabric forwarding anycast-gateway-mac 2020.DEAD.BEEF

ip pim rp-address 10.254.254.254 group-list 239.239.239.0/24

vlan 1,99-101,2500,3000

vlan 99

  name L2onlyHostSegment

  vn-segment 30099

vlan 100

  name L2L3HostSegment

  vn-segment 30000

vlan 101

  name L2L3HostSegment

  vn-segment 30001

vlan 2500

  name FabricBD

  vn-segment 50000

vlan 3000

  name VPCL3Peering

route-map FABRIC-RMAP-REDIST-SUBNET permit 10

  match tag 21921

vrf context TENANT1

  vni 50000

  rd auto

  address-family ipv4 unicast

    route-target both auto

    route-target both auto evpn

  address-family ipv6 unicast

    route-target both auto

    route-target both auto evpn

vpc domain 1

  peer-switch

  peer-keepalive destination 10.2.8.2 source 10.2.8.1 vrf management

  peer-gateway

  ip arp synchronize

interface Vlan100

  no shutdown

  vrf member TENANT1

  ip address 192.168.100.1/24 tag 21921

  fabric forwarding mode anycast-gateway

interface Vlan101

  no shutdown

  vrf member TENANT1

  ip address 192.168.101.1/24 tag 21921

  fabric forwarding mode anycast-gateway

interface Vlan2500

  description FabricBD

  no shutdown

  mtu 9216

  vrf member TENANT1

  ip forward

interface Vlan3000

  description VPC Layer-3 Peering for VXLAN

  no shutdown

  ip address 10.3.1.2/30 # Requires to be individual IP per VPC member

  ip ospf network point-to-point

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

nve infra-vlans 3000

# required for Nexus 9300-EX/FX or Nexus 9200

interface port-channel1

  description VPC Peer-Link

  switchport mode trunk

  spanning-tree port type network

  lacp suspend-individual

  vpc peer-link

hardware access-list tcam region vacl 0

# example region to free up space for arp-ether region

hardware access-list tcam region arp-ether 256 double-wide

# required for ARP suppression, requires reboot

# double-wide is required starting 7.0(3)I3(1)

# not required for Nexus 9300-EX/FX or Nexus 9200

interface nve1

  mtu 9216

  no shutdown

  source-interface loopback1

  host-reachability protocol bgp

  member vni 30000

    suppress-arp

    mcast-group 239.239.239.100

  member vni 30001

    suppress-arp

    mcast-group 239.239.239.101

  member vni 30099

    mcast-group 239.239.239.99

  member vni 50000 associate-vrf

interface Ethernet1/1

  switchport mode trunk

  spanning-tree port type edge trunk

  spanning-tree bpduguard enable

interface Ethernet1/47

  description Link for VPC Peer-Link

  switchport mode trunk

  channel-group 1 mode active

interface Ethernet1/48

  description Link for VPC Peer-Link

  switchport mode trunk

  channel-group 1 mode active

interface Ethernet2/1

  description Link to Spine "1"

  no switchport

  mtu 9216

  ip unnumbered loopback0

  ip ospf network point-to-point

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

  no shutdown

interface loopback0 # Loopback for Router ID, routing adjacency and peering

  ip address 10.250.250.103/32

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

interface loopback1 # Loopback for VTEP only

  ip address 10.254.254.103/32

  ip address 10.254.254.1/32 secondary

  ip ospf network point-to-point # will change OSPF interface back to Loopback; required for VPC

  ip router ospf UNDERLAY area 0.0.0.0

  ip pim sparse-mode

router ospf UNDERLAY

  router-id 10.250.250.103

  log-adjacency-changes detail

router bgp 65500

  router-id 10.250.250.103

  address-family ipv4 unicast

  neighbor 10.250.250.101 remote-as 65500

    update-source loopback0

    address-family ipv4 unicast # optional for "show ip bgp summary" support

    address-family l2vpn evpn

      send-community both

  vrf TENANT1

    address-family ipv4 unicast

      advertise l2vpn evpn

      redistribute direct route-map FABRIC-RMAP-REDIST-SUBNET

      maximum-paths ibgp 2

evpn

  vni 30000 l2

    rd auto

    route-target import auto

    route-target export auto

  vni 30001 l2

    rd auto

    route-target import auto

    route-target export auto

  vni 30099 l2

    rd auto

    route-target import auto

    route-target export auto

[hamed1900]

That is one of the best configuration example.

Questions:

- Using ip unumbered is much easier, why would we use the numbered model then?

Thanks,

Hamed

[Lukas Krattiger]

Hamed,

we are providing peoples choice of deploying a L3 underlay network. Aspects could be around "not all devices support unnumbered" or "I wanna use a traditional approach".

-Lukas

[tommy182]

Hello Lukas!

Please tell me, is it really unnecessary to configure "mtu 9216" on special SVI 3000(L3 over peer-link)?

I often see in configuration examples for this special SVI that we need to configure particular OSPF cost, but there is no information about what particular cost we have to configure(is it less or bigger than regular cost)

And the last one , what about BFD? I use BFD in my environment, but I`ve faced with weird issue, when I push DOWN uplinks on one of the vPC leaf switches and then push back to UP i see that BFD session from this switch doesn`t came up, it flap UP/DOWN and OSPF adjacencies doesn`t came up, but after ~5 minutes some way it`s stabilizing.

I think there is OSPF issue with this SVI-Over-PeerLink(cause if I push down uplinks from other vPC peer then on first vPC peer BFD and OSPF came UP immediately)

UPDATE!! Regarding BFD issue.

It will arise only when in addition to run BFD to OSPF we run it for PIM.

If I turn off BFD for PIM all works great. Interesting

[Lukas Krattiger]

Artem,

the "special" SVI for interconnecting the VPC members is to have an alternate path if all uplinks to the Spine fails. This said, this "L3 over peer-link" requires the similar attributes in regards of routing, BUM and mtu as your uplinks to the Spine.

In regards of cost, there is no specific value that is more recommended. Under normal circumstances, all traffic will anyway use the Spine uplink path as going via VPC peer-link will be 1 more hop and worse cost.

I normally don't use BFD for Leaf to Spine uplinks as in the case of a link failure, it is a lights-out event, the previous egress interface for the next-hop is down and other ECMP path will take over right away. We tested OSPF via VPC peer-link by using a SVI and I can't see your observation. Maybe your CoPP policy is not applied on hence creates some collateral damage (just a guess). I would suggest to open a TAC SR for looking into this issue as BFD for PIM and OSPF should be supported.

Thanks for the interest

-Lukas

[zmarpatt]

Hi Lukas,

Thanks for creating this post.

I have 2 queries.

1.)

Noticed that you have created lo1 to give it as the source-interface for nve1.

Why can't we re-use lo0 as source-interface for nve1 ?

2.)

Can you explain bit more on why the line is striked out ? and regarding the comment "will change OSPF interface back to Loopback; required for VPC" ?

"

  ip ospf network point-to-point # will change OSPF interface back to Loopback; required for VPC

"

Regards,

Zubin