Is it possible to use Cisco 9k’s and VXLAN to communicate with VMware NSX/servers and allow VTEP between sites over the wide area where one site is using Cisco 9k’s communicating with a VWARE NSX/servers while the other site is only using VMware NSX/servers with no Cisco gear at site?
What is your use case or what are you trying to achieve?
I'm not sure how do you setup your VXLAN network but NSX has its own VXLAN which controlled by NSX, can be used by NSX Logical Switches and the VTEPs will be a VMkernel in ESXi hosts
You can create VXLAN in Cisco 9K but that VXLAN will be used for physical network, not related with NSX and the VTEPs will be in Cisco Nexus 9Ks.
Not sure what is "inter-operating" in your terms but if you have a use case where you want a VM connected to NSX Logical Switch to be layer-2 adjacent with a physical server or a port in physical switch (in this case Nexus 9K) you can do VXLAN Layer-2 bridging as per this whitepaper Configuring Cisco Nexus 9000 Series Switches for VMware NSX OVSDB Integration White Paper - Cisco
Please note from VMware point of view, Nexus 9K is not listed in their HCL VMware Compatibility Guide - Hardware VXLAN Gateway, probably not certified yet
But if you don't have that layer-2 bridging use case then NSX runs on any physical switch, you don't need a VXLAN capable physical network switch and you can run different gears across sites. Physical network will see NSX VXLAN as a UDP packet communicating between ESXi hosts and that's it!
We are trying to extend IP subnets between sites so servers don't require an IP address change as they are migrating servers from site 1 to site 3. Two of the sites (1 and 3) can use NSX while site 2 has Cisco Nexus into a VMware server farm. We trying to understand if we can possibly extend layer 2 connectivity over the layer 3 WAN. Eventually, site 2 and site 3 will be the only sites left but they will still require layer 2 connectivity between one another over WAN. We are trying not to purchase new hardware and we require it up and running yesterday :>(
Site 1 Subnets (1,2,3) NSX <----WAN ----> Site 2 (Subnets 1,2,3,4) Nexus to VMware Farm <----WAN ----> Site 3 (Subnets 1,2,3,4) NSX
It depends on your NSX setup, do you connect the VMs to NSX VXLAN logical switch networks or to a VLAN-backed PortGroup provided by the physical switch?
If the VMs are connected to the NSX logical switch networks you can extend the logical switch to multiple sites but you would need cross-vCenter NSX setup - https://blogs.vmware.com/networkvirtualization/2016/03/cross-vc-nsx-multi-site-solutions.html/
Additional note, if you need to have layer-2 adj between logical switch and physical network which can be achieved through layer-2 bridging, this feature is not supported in cross-vCenter setup
If the current NSX setup is not setup for cross-vCenter, you would need to perform some changes and probably some virtual/logical network migration within NSX.
Please note NSX does not do any ingress routing optimization so even if you can do local egress optimization in NSX, you would need to handle the north-south ingress routing outside of NSX.
If the VMs are connected to VLAN-backed PortGroup then you would need to rely on physical network switches to extend the subnets such as using VXLAN, OTV, EVPN or something else which typically need a compatible set of physical switches.
Extending layer-2 between sites normally a complex setup esp on the ingress routing and when you have stateful services like LB or firewall. This blog post might give you more idea about it: http://blog.senasosa.com/2017/01/dc-ingress-traffic-with-stretched-layer.html
Based on all the responses, it seems that NSX end to end at each site is the best way to go as an overlay (NSX layer 2) to the underlay (Physical Cisco layer 3 transport)
Just trying to find the answer myself.. I came across this URL where Cisco's switches are not mentioned ..
VxLAN being a vendor independent technology, I don't see any reason why the Cisco 9K's can not be used as the hardware VTEP. .. need to ask if someone has already done that before successfully.
Thanks - Gaurav