VXLAN BGP EVPN Routing and Default Gateway

[Alex Pfeil]

We have 2 93180FX switches running 9.3.3 code or latest. We configured the 2 switches for VXLAN BGP EVPN. Switches are both configured with same L3 VPN IP address which would be the default gateway. 

Example: SW1 - 192.168.10.1 and SW2 - 192.168.10.1

SW1 has client 192.168.10.3 and SW2 has client 192.168.10.4.

Client on SW1 (192.168.10.3) can ping 192.168.10.1 and 192.168.10.4.

Client on SW2 (192.168.10.4) can ping 192.168.10.1 and 192.168.10.3.

SW1 (192.168.10.1) can ping 192.168.10.1 and 192.168.10.3 but cannot ping 192.168.10.4

SW2 (192.168.10.1) can ping 192.168.10.1 and 192.168.10.4 but cannot ping 192.168.10.3.

My question is:

1. Should each switch be able to ping all clients on both sides?

2. I was also wondering how Layer 3 routing would take place in the VXLAN BGP EVPN configuration.

More info: 

I setup another vlan with IP address 192.168.50.1 in the same vrf as the 192.168.10.x setup. The 192.168.10.4 client cannot ping that IP address. Switch 2 is the only (192.168.10.1) IP that can ping 192.168.50.1.

[Francesco Molino]

Hi

In a vxlan environment, your svi are configured as anycast-gateway. From the switch itself you won't be and to ping your clients because return traffic can hit any members of the fabric.

For routing, what do you mean? Inside the fabric or with outside of the fabric?

For your other subnet, can you share your config please?

[Sergiu.Daniluk · Accepted solution]

Hi Alex,

1. Should each switch be able to ping all clients on both sides?

If you have configured distributed anycast gateway, then the answer is No. Basically, the idea of anycast gateway is to configure same IP and same MAC address on all VTEPs.

Let's take for example the following configuration and topology:

# VLAN to VNI mapping
vlan 201
  vn-segment 6000

# Anycast Gateway MAC, identically configured on all VTEPs
fabric forwarding anycast-gateway-mac 0002.0002.0002

# Distributed IP Anycast Gateway (SVI)
# Gateway IP address needs to be identically configured on all VTEPs
interface vlan 201
  no shutdown
  vrf member Green
  ip address 192.168.1.254/24
  fabric forwarding mode anycast-gateway

When Device 1, 2 and 3 will need to communicate with (or let's say simply ping) the gateway, they will use dIP 102.168.1.254 and dMAC 0002.0002.0002. Since all the VTEPs are owning this IP and MAC, only the first one will respond to the ICMP packet.

This is why when you ping from VTEP 20, the Device 1 (192.168.1.1) you will not see the responses - the ICMP reply will be consumed by VTEP 10.

When devices from the same subnet / same vn-segment want to comunicate between them, the packets will simply be forwarded (encapsulated) over the Layer2 VNI - in the above topology in VNI 6000

2. I was also wondering how Layer 3 routing would take place in the VXLAN BGP EVPN configuration.

The routing happens different VNIs is beeing done over something which is called Layer 3 VNI. As you see below, this vlan / L3 VNI is used only for routing in vxlan.

Here is an example of configuration (this is just a part of it, not the full config).

Vlan 200
  vn-segment 5000

interface Vlan200
  no shutdown
  mtu 9216
  vrf member Green
  ip forward

interface nve1
  member vni 5000 associate-vrf

Hope you will find this explanation helpful.

Regards,

Sergiu

[Alex Pfeil]

I have a couple of more questions:

1. To extend layer 3 out to routers on both sides, is it possible to have a trunk that is a VRF member and extend multiple VLANs?

2. In OTV, we have a default gateway on both sides. Is it possible to do similar setup with VXLAN?

Thanks,

Alex

[Sergiu.Daniluk]

Hi Alex,

---

1. To extend layer 3 out to routers on both sides, is it possible to have a trunk that is a VRF member and extend multiple VLANs?

If you wish to extend L3 from a VTEP, you have quite a couple of options, including a (static or dynamic) routing over a trunk with vlans allowed (not sure what you mean trunk being a VRF member). However, if you plan doing a per-VRF IP routing external connectivity, you should check this config guide: https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/93x/vxlan/configuration/guide/b-cisco-nexus-9000-series-nx-os-vxlan-configuration-guide-93x/b-cisco-nexus-9000-series-nx-os-vxlan-configuration-guide-93x_chapter_011111.html 

---

2. In OTV, we have a default gateway on both sides. Is it possible to do similar setup with VXLAN?

---

Through the use of distributed anycast gateway, this is exactly what is being achieved.

Going back to my diagram, both VTEP 10 and VTEP 20 are configured with SVI 201 (IP 192.168.1.254 / MAC 0002.0002.0002), and both will act as local default gateway for servers connected in vlan 201.

Regards,

Sergiu

[Sergiu.Daniluk · Accepted solution]

Hello again Alex,

I would strongly recommend reading the vxlan whitepapers. There you will find different design guides and recommended topologies:

• https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/guide-c07-734107.html?cachemode=refresh#_Toc444553379 
• https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/white-paper-c11-732453.html 
• https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/white-paper-c11-739942.html 

Regards,

Sergiu