BGP EVPN VXLAN Layer 2 Network over GRE IPSec Tunnel

alex.f. · ‎12-03-2023

Here is the Deal.
Our Customer needs to migrate from DC_A to DC_B.
DC_A is based on a VXLAN Fabric.

My approach is to build a direct Connection between this two Data Center.
I used following Hardware:

2x Catalyst C9500 24Y4C
2x Cisco 2851 Router
1x MGMT WS-C3550
1x MGMT C3560 (between the two Router)
2x Raspberry PI for Testing

My Config is based on the Guide from 2020. I used the "Layer 2 VNI with Back to Back Ingress Replication".

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9500/software/release/17-3/configuration_guide/vxlan/b_173_bgp_evpn_vxlan_9500_cg/configuring_evpn_vxlan_layer_2_overlay_network.html

The first Step was to build the BGP EVPN VXLAN between the two C9500 to get into the technology flying.

===== Underlay Network ====
The WAN is simulated between the two Router on Vlan 88.

R1#
interface GigabitEthernet0/0.88
description Transfer
encapsulation dot1Q 88
ip address 10.0.88.1 255.255.255.0
no snmp trap link-status
end
R1#

R2#
interface GigabitEthernet0/0.88
description Transfer
encapsulation dot1Q 88
ip address 10.0.88.2 255.255.255.0
no snmp trap link-status
end
R2#

For the Underlay Network I used OSPF as IGP to connect the two Sides together.

R1#
router ospf 1
router-id 1.1.1.1
log-adjacency-changes
passive-interface GigabitEthernet0/0.88
network 12.12.12.1 0.0.0.0 area 0
network 172.16.1.0 0.0.0.255 area 0

interface GigabitEthernet0/1
description Leaf-01
ip address 172.16.1.2 255.255.255.0
ip ospf network point-to-point
ip ospf mtu-ignore
duplex auto
speed auto
end
R1#

R2#
router ospf 1
router-id 2.2.2.2
log-adjacency-changes
passive-interface GigabitEthernet0/0.88
network 12.12.12.2 0.0.0.0 area 0
network 172.16.2.0 0.0.0.255 area 0

interface GigabitEthernet0/1
description Leaf-02
ip address 172.16.2.2 255.255.255.0
ip ospf network point-to-point
ip ospf mtu-ignore
duplex auto
speed auto
end
R2#

The Boarder Switches for EVPN VXLAN are also particpating on the OSPF to get a connection.
(Loopback0 is used for our BGP Overlay)
(Loopback1 is used for our VXLAN Overlay)

Leaf-01#
interface TwentyFiveGigE1/0/24
no switchport
ip address 172.16.1.1 255.255.255.0
ip ospf network point-to-point
ip ospf 1 area 0

interface Loopback0
ip address 172.16.255.1 255.255.255.255
ip ospf 1 area 0
!
interface Loopback1
ip address 172.16.254.1 255.255.255.255
ip ospf 1 area 0
!
router ospf 1
router-id 172.16.255.1
!
Leaf-01#

Leaf-02#
interface TwentyFiveGigE1/0/24
no switchport
ip address 172.16.2.1 255.255.255.0
ip ospf network point-to-point
ip ospf 1 area 0
!
interface Loopback0
ip address 172.16.255.2 255.255.255.255
ip ospf 1 area 0
!
interface Loopback1
ip address 172.16.254.2 255.255.255.255
ip ospf 1 area 0
!
router ospf 1
router-id 172.16.255.2
!
Leaf-02#

===== Overlay Network ====

## This Part should be on both Sides Leaf-01 and Leaf-02 identical ##
!
vlan 120
name SERVER
!
l2vpn evpn
replication-type static
router-id Loopback1
!
l2vpn evpn instance 120 vlan-based
encapsulation vxlan
replication-type ingress
!
vlan configuration 120
member evpn-instance 120 vni 10101
!
interface TwentyFiveGigE1/0/1
switchport access vlan 120
switchport mode access
!
interface nve1
no ip address
source-interface Loopback1
host-reachability protocol bgp
member vni 10101 ingress-replication
!
## This Part should be on bouth Sides Leaf-01 and Leaf-02 ##

The BGP using Loopback0 as its neighbour.

Leaf-01#
interface Vlan120
ip address 172.18.120.254 255.255.255.0
!
router bgp 65001
bgp log-neighbor-changes
no bgp default ipv4-unicast
neighbor 172.16.255.2 remote-as 65001
neighbor 172.16.255.2 update-source Loopback0
!
address-family ipv4
exit-address-family
!
address-family l2vpn evpn
neighbor 172.16.255.2 activate
neighbor 172.16.255.2 send-community both
exit-address-family
!
Leaf-01#

Leaf-02#
interface Vlan120
ip address 172.18.120.253 255.255.255.0
!
!
router bgp 65001
bgp log-neighbor-changes
no bgp default ipv4-unicast
neighbor 172.16.255.1 remote-as 65001
neighbor 172.16.255.1 update-source Loopback0
!
address-family ipv4
exit-address-family
!
address-family l2vpn evpn
neighbor 172.16.255.1 activate
neighbor 172.16.255.1 send-community both
exit-address-family
!
Leaf-02#

######

Testing
######
Leaf-01#sh arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 172.16.1.1 - abxx.xxxx.1c7f ARPA TwentyFiveGigE1/0/24
Internet 172.16.1.2 134 cdxx.xxxx.6f01 ARPA TwentyFiveGigE1/0/24
Internet 172.18.120.1 3 b827.eb07.1fcf ARPA Vlan120
Internet 172.18.120.253 1 abyy.xxxx.1bdf ARPA Vlan120
Leaf-01#

Leaf-02#sh arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 172.16.2.1 - abxx.xxxx.1bdf ARPA TwentyFiveGigE1/0/24
Internet 172.16.2.2 149 cdyy.yyyy.a001 ARPA TwentyFiveGigE1/0/24
Internet 172.18.120.2 1 b827.ebfb.7bba ARPA Vlan120
Internet 172.18.120.254 3 abyy.xxxx.1c7f ARPA Vlan120
Leaf-02#

Leaf-01#ping 172.18.120.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.18.120.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
Leaf-01#
Leaf-01#ping 172.18.120.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.18.120.2, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 1/1/1 ms
Leaf-01#

Leaf-02#ping 172.18.120.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.18.120.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
Leaf-02#ping 172.18.120.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.18.120.1, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 1/1/2 ms
Leaf-02#

Leaf-01#sh arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 172.16.1.1 - abxx.xxxx.1c7f ARPA TwentyFiveGigE1/0/24
Internet 172.16.1.2 136 cdxx.xxxx.6f01 ARPA TwentyFiveGigE1/0/24
Internet 172.18.120.1 0 b827.eb07.1fcf ARPA Vlan120
Internet 172.18.120.2 0 b827.ebfb.7bba ARPA Vlan120
Internet 172.18.120.253 0 abyy.xxxx.1bdf ARPA Vlan120
Leaf-01#

Leaf-02#sh arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 172.16.2.1 - abxx.xxxx.1bdf ARPA TwentyFiveGigE1/0/24
Internet 172.16.2.2 151 cdyy.yyyy.a001 ARPA TwentyFiveGigE1/0/24
Internet 172.18.120.1 0 b827.eb07.1fcf ARPA Vlan120
Internet 172.18.120.2 0 b827.ebfb.7bba ARPA Vlan120
Internet 172.18.120.254 0 abyy.xxxx.1c7f ARPA Vlan120
Leaf-02#

M02@rt37 · ‎12-03-2023

Hello @alex.f.

Big deal !

The key elements, such as the underlay network with OSPF, overlay network with BGP EVPN VXLAN, and testing with ARP and pings, appear to be in place. Your successful pings and ARP entries indicate that the connectivity is established.

Best regards
.ı|ı.ı|ı. If This Helps, Please Rate .ı|ı.ı|ı.

MHM Cisco World · ‎12-03-2023

As I know you need OTV to interconnect two DC.

Vxlan only for intra DC not fot inter DC.

MHM

alex.f. · ‎12-11-2023

Hi,

what do you mean?

I have no deep understanding or experience with DC technology.

I guess my PoC is working because of the GRE Tunnel between the R1 and R2 to get the IGP working.

OTV also does some GRE magic to get Ethernet going.

What would be your idea to connect two RZs?