Hybrid overlays with MPBGP - EVPN

[mgr]

Hi, 

In our data center network, there is a requirement for having hybrid overlays as there is a mix of vmware NSX-v and physical work loads. There is a scenario in which hosts in the same subnet are spread across the physical and virtual environments. 

We are planning to implement leaf-spine architecture with Nexus 9Ks and have MP BGP - EVPN. 

1. Is it possible for the physical server to reach its gateway which is residing inside the virtual environment over VXLAN ? 

2. How to make my physical and virtual VTEPs communicate? Is there any configuration example for having hybrid overlays?

3. How to make the servers inside virtual environment communicate with L4 & L7 devices?

I am attaching a sample diagram for my requirement. 

[bayupw]

1. Is it possible for the physical server to reach its gateway which is residing inside the virtual environment over VXLAN ? 

I assume 10.10.10.x is your overlay network.

Do you use NSX logical switch?

Is the gateway of the virtual environment on NSX DLR?

Is your requirement to have physical server 10.10.10.6 on the same subnet as the network on the virtual environment?

NSX VXLAN is its own network virtualisation so if you want to extend that to outside world i.e. physical VLAN, you would need to do NSX Layer 2 bridging

https://docs.vmware.com/en/VMware-NSX-for-vSphere/6.4/com.vmware.nsx.admin.doc/GUID-ECE2893A-A1A6-4D43-93DA-AE4A97ABBF44.html

There are two types of Layer 2 bridging

a. Software based layer 2 bridging, native in VMware NSX. The physical VLAN that you want to bridge must be accessible from NSX VXLAN/VTEP VDS and need to be layer 2 adjacent. The gateway of physical VLAN can be the NSX DLR

b. Hardware based layer 2 bridging using Hardware VTEP gateway, for Nexus 9K see this link: https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/white-paper-c11-740091.html

This is bridging NSX logical switch to a physical port, the actual physical port VLAN can be on any VLAN. Requires the hardware VTEP on different layer 3 network, gateway of NSX logical switch and physical server cannot be on the DLR

If I'm not mistaken, last time for hardware VTEP to work in Nexus 9K, the switch cannot run on MP-BGP EVPN - but I'm not quite sure better to double check this

I have a slide deck that cover hardware bridging here: https://www.slideshare.net/bayupw/vmware-nsx-and-arista-l2-hardware-vtep-gateway-integration

2. How to make my physical and virtual VTEPs communicate? Is there any configuration example for having hybrid overlays?

They don't really need to communicate unless you want to do hardware based layer 2 bridging to NSX

3. How to make the servers inside virtual environment communicate with L4 & L7 devices?

Is communicating over Layer 3 acceptable?

If the L4/L7 devices need to be on the same subnet then you need layer 2 bridging

[mgr]

Hi,

Thanks a lot for your response. I have responded to your queries. The NSX in my environment is getting restricted to Leaf1 & Leaf2 which i had shown in my diagram. My physical server is not connected to leaf2 and so it does not have layer2 adjacency. This means that i cannot use NSX native L2 adjacency -software Layer2 bridging. Please correct me if i'm wrong. 

In order to overcome this restriction, i planned to use MP-BGP - another overlay in the switches. Let's say i put my virtual server 10.10.10.5 in vxlan 5000 and software L2 bridging of this vxlan to vlan500. I can locally create VLAN500 if leaf1 & leaf2. My physical server connected to leaf3 would also be put in vlan 500. I would use the MP-BGP to create a VTEP and map it to another VXLAN for bridging. Will this scenario work and allow me to have the gateway of the servers in the DLR?

I have answered your queries. 

I assume 10.10.10.x is your overlay network. -> This is the subnet for servers. 

Do you use NSX logical switch? -> Yes, we are planning to use NSX logical switch.

Is the gateway of the virtual environment on NSX DLR? -> Yes

Is your requirement to have physical server 10.10.10.6 on the same subnet as the network on the virtual environment? -> Yes. 

I am attaching a rough diagram to explain the scenario.