Nexus 9k VXLAN (topology without Spine)

[polinaovsyannikova]

Hi Team!

I need help, with design and I think some example of configuration.

I do not have experience with VXLAN, but there is a task to configure the network  for VSAN stretched cluster.

I have  6 Nexus 9000 EX, with a license LAN_ENTERPRISE_SERVICES_PKG  ( 3 sites - 2 Nexus with KA and VPC peer link for each sites).

Nexuses are interconnected by dark optical fiber, the speed of each port is 10G, in total this is 1 VPC link  per 40 G (for understanding pls see picture in attach)

My task is to make routing between sites so that virtual machines on each side can ping each other on different subnets ( for example: VM with ip 10.10.10.100 at Site 1 can ping VM 10.10.20.100 at Site 2)

I read a lot of documentation about VXLAN EVPN BGP, for me it is clear that for VXLAN underlay it is necessary to use routing protocols (for example, when using spine and leaf  topology, L3 connectivity is necessary)

In my case - Nexus 9300  EX Nexus cant't  use the logical interface PO as L3 interface ( no switchport), if I will use a  dedicated interface  for L3 connectivity for VXLAN underlay- I will lose bandwidth because my SFP module  is only 10g

Unfortunately, I do not have other equipment, only 6 nexuses. If someone else had a similar case, I will be very grateful for any tips and configuration examples.

[Francesco Molino]

Hi

You will have spines acting as leaves and border gateway (all roles on 1 vpc pair). The design you will need to implement that fits better your architecture is multi-site.
Here an official documentation:
https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/whitepaper-c11-742114.html

Just for your information, dci-links (underlay for vxlan) can only use a physical interface (sub-interface and svi aren't supported).

Will you maintain all L2 links? As I said you need a physical interface for dci-link otherwise your vxlan fabric won't come up. This means you can keep all these links (for redundancy and full mesh dci cloud) and allow just 1 vlan for example (let's assume vlan 100).
Then you will need a loop locally on each switch (connect port 1 to port 2). Port 1 will be in access mode for vlan 100 and port 2 will have an IP address (subnet for your dci cloud). This interface port 2 will be used as your dci-link for your vxlan fabric.
Is that clear? I could make a quick drawing if needed.

Afterwards, follow the doc i gave and you'll be able to setup l3vni (L3 vlans across all sites) or l2vni (L2 vlans).

I can help if you need assistance in your configurations.

[polinaovsyannikova]

Hi, Francesco! 

I apologize for the long answer.

Thanks for such a detailed post.

But I have some doubts about using a VXLAN EVPN Multi-Site.

According to the cisco documentation, a N93-FAB1K9  license is required for this solution, which I don't have.

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-x/vxlan/configuration/guide/b_Cisco_Nexus_9000_Series_NX-OS_VXLAN_Configuration_Guide_7x/b_Cisco_Nexus_9000_Series_NX-OS_VXLAN_Configuration_Guide_7x_chapter_01100.html#id_62629

About the topology - I understood your idea, but if it’s not difficult for you, could you draw a diagram, how will it work? Thanks in advance.

[lovsharm]

N93-FAB1K9 is essential for BGW

[polinaovsyannikova]

Thanks for your answer.

If I understand correctly, I can't use VXLAN EVPN Multi-Site Using vPC Border Gateways architecture without N93-FAB1K9, right?

[Francesco Molino]

Yes for multi site this is a requirement.
You can go with multi pod. It will be less scalable but based on your requirements it will work correctly.
Here a post where I've done some designs:
https://community.cisco.com/t5/data-center-switches/how-to-route-traffic-via-firewall-in-evpn-vxlan-fabric/m-p/4067761#M5561

Let me know if you need anything.

[polinaovsyannikova]

Thanks for the answer, maybe you have a link to the documents, where can I see some examples of   file configuration?

[Francesco Molino]

i don't have any links right now but if you go on Cisco website you'll have plenty of them. Or even better, look at Cisco live presentation for vxlan and you'll find all you're searching for.

[polinaovsyannikova]

Thank you for your answer.

I set up VXLAN  topology - like  full mesh connectivity, OSPF as an underlay protocol.

But now I have one more task. Currently, inter VXLAN routing works well.

But it is necessary to make sure that for some  virtual machines the Palo Alto firewall is the default gateway, and for the rest, the Nexus. And need to organize communication between VM machines.

Maybe you have examples of configurations that would help me a lot.

Thanks in advance.

[Francesco Molino]

I'm not sure I got you.
What do you mean by "But it is necessary to make sure that for some virtual machines the Palo Alto firewall is the default gateway, and for the rest, the Nexus. And need to organize communication between VM machines."

Does that means, some vlan will be l2vni where IP will be hosted on PA and other vlans will be l3vni and so IP on Nexus? This works as usual. You will need to do interconnection (L3 routing) between your PA and Nexus.

Can you clarify what you meant please to be sure we're on the same page?

[lovsharm]

Correct! you need license to make BGW(vpc/non-vpc) work.