How to route traffic via Firewall in EVPN + VxLAN fabric

[satish.txt1]

This is what i am trying to design new network, this is just on paper nothing finalized and i am sure it has lots of issue like firewall should be on leaf not border leaf etc. so that is why i am here to clear all those doubts. I have following question related Spine-leaf design.

• If we use anycast gateway in this design in that case leaf would be my edge gateway for all hosts connected to that leaf so how does Host-A will send traffic to Host-B via firewall?
• Should i disable anycast gateway so each VLAN traffic route via firewall (my firewall is gateway for all VLANs)
• what do you think about this design or this is not something i should be using?

[Francesco Molino]

Hi

In quite all designs, to deal with firewalls, we put them outside of the fabric to avoid any issues, specially with multipod or multisites.
Anyways, if you want your fw to be the default gateway of all your hosts, then you will only have l2vni and no anycast-gateway. Is that what you want? Because on your attached design it's written default gw on ASAs.

If you want L3 to be hosted on your Nexus, then you'll need to configure multiple VRFs with an interco to your fw for each of them (like default route to asa for each vrf). However, if you want to do dynamic routing, you won't be able to use anycast-gateway, you'll have to create a dedicated svi without anycast-gateway to interconnect with your asa and mount a dynamic routing neighborship.

To help you more, what's the goal you want to achieve?

[satish.txt1]

Thank you for reply, 

You got it what i am trying to say, let me tell you little history. currently i am running large datacenter on 180 Cisco nexus switch with dual vPC + STP and i am worried about to scaling it, we already have 4000 servers. Soon in next 6 month we are going to build new datacenter so looking for some robust and advance design, In current design we have Cisco ASA firewall and FW is gateway of all VLAN and we have bunch of rules to protect each VLAN for security purpose (same model i want to replicate in EVPN+VxLAN topology) 

As you said i have two option:

Disable anycast gateway: 

Is it possible to disable Anycast gateway i thought it's part of EVPN+VxLAN, If that is the solution then i can do that, but in that case it's going to suppress BUM traffic (and easy to scale?)  hope i get good performancing disabling anycast. 

Create multiple VRF:

If i create multiple VRF for each VLAN and not create SVI in that case i need to add Cisco interface to each VRF to router traffic between VLAN right? 

What do you suggest if your are doing this design?  I am looking for scaling thing and trying to get out from STP world. 

[Francesco Molino]

Yes you can disable anycast gateways and have SVIs running with a dedicated IP on each Spines for example. However, I had multiple issues where SVIs were up, can ping between them across VXLAN fabric, can do a telnet on port 179 on 1 side but not the other way around and so BGP wasn't coming up.

Workaround solution was to do a local loop on the switch itself (switchport trunk on 1 side and IP address on the other side as sub interfaces. This is if you want to keep your FW in the Fabric. This is a working solution for a HA ASA FW.

However, if you want to scale this up and use ASA Cluster for example, it's not going to be the design ever.

I would suggest putting an edge service switches on which FW will sit and connect into your overlay using sub-interfaces. I am always going this way and never got any issues plus designs are very scalable this way.

Hope that helps.

[satish.txt1]

Yes you can disable anycast gateways and have SVIs running with a dedicated IP on each Spines for example. However, I had multiple issues where SVIs were up, can ping between them across VXLAN fabric, can do a telnet on port 179 on 1 side but not the other way around and so BGP wasn't coming up.

I didn't get this point SVI running on Spines (I believe we don't need SVI on spines right?) 

Workaround solution was to do a local loop on the switch itself (switchport trunk on 1 side and IP address on the other side as sub interfaces. This is if you want to keep your FW in the Fabric. This is a working solution for a HA ASA FW.

Do you have example or snippet to understand?

I would suggest putting an edge service switches on which FW will sit and connect into your overlay using sub-interfaces. I am always going this way and never got any issues plus designs are very scalable this way.

Are you trying to say put dedicated service leaf switch for firewall device?  I don't understand about sub-interface idea?

[Francesco Molino]

Let me reply to you tomorrow with a quick design.

For svi, i talked about spines but same occurs for leaves. Maybe with a quick sketch you'll understand.

[satish.txt1]

Thanks waiting for your reply, also i have couple of Cisco nexus 9396PX switches in my stock so thinking to build lab and try your configuration there? 4 x 9396PX switches would be enough right to build EVPN+VxLAN lab? 

[Francesco Molino · Accepted solution]

I took an example of design with Spines in VPC.

On Option1 design:

ASA Failover will done use L2VNI.

You use SVI without anycast-gateway in order to have different L3 IP address on each SVI on each box. Then you put ASA data interface to that same vlan. Let's say you want to do BGP from Primary ASA to all 4 spines to make sure to have the best and smooth redundancy when switchover to Secondary ASA.
With that design, BGP peering will work between ASA and SVIs on Datacenter A but not between ASA and Datacenter B.

On Option2 design:

We use the same design but instead of using SVIs, we will create a physical loop on each box. 1 cable will be connected from port Eth1/1 to Eth1/2 within the same box. 1 side of the cable is in switchport mode whereas the other end is in L3 mode.

This works fine.

Option 3 is the best way to handle ASA (firewalling vlans) across a Fabric and is much more scalable than the other options.

Is that more clear?

[satish.txt1]

Thank you so much! it's much clear now, love it.. 

In option 3 do you have Cisco ASA connected to Edge switch as a trunk to run multiple VLANs or its L3 connection and if it's L3 connection then how multiple vlan route traffic. 

If you see in my case its very basic design i have single datacenter and all VLANs in same datacenter but i want my all VLAN connected to ASA as a gateway. ASA has all kind of rules to protect VLANs from each other based on specific services. 

soon i am going to setup lab to implement all kind of design to find out how it goes.

One more question lets say i remove cisco ASA in my design and implement anycast gateway in that case can i configure ACL on leaf switches to protect my VLAN intercommunication traffic if i want? 

[Francesco Molino]

Yes you can apply some acl.
Here are limitations and supported ACL in VXLAN infrastructure: https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-x/vxlan/configuration/guide/b_Cisco_Nexus_9000_Series_NX-OS_VXLAN_Configuration_Guide_7x/b_Cisco_Nexus_9000_Series_NX-OS_VXLAN_Configuration_Guide_7x_chapter_011.html

[satish.txt1]

If i can use ACL on VTEP then i don't need Cisco ASA for security, i can use leaf for filtering traffic between VLANs. 

This is what i am going to use and hope it support, do you think following both model would be great for basic EVPN-VXLAN design?

Spine - Cisco nexus 9336C-FX2 switches

Leaf - Cisco nexus 9396PX switches 

[Francesco Molino]

The leaf model isn't sold anymore.

If you already have them they are ok. I would say it depends more on your needs and type of ports you need.
Acls are ok but i would prefer a firewall to have more flexibility and usability in terms of acls.

[satish.txt1]

I have couple of Cisco nexus 9396PX in stock so thinking to reuse them here, and i totally agreed with you that cisco ASA is more flexible but just thinking if i don't configure anycast gateway in that case my packet need to hop more switches to talk even both VLAN under same switch, but anyway with in datacenter i don't think it will be big deal. 

Should i use MP-BGP for control plan for flood & learning (what would be more effective + easy to troubleshoot) what more people using for mid size network where we have 20 switches around?  

[Francesco Molino]

You will have mpbgp because you'll have different address-families (ipv4, l2vpn) and for multiple vrf (vxlan underlay and your tenant(s))

[Francesco Molino]

Please don't forget to rate my answers if they answered your questions and mark it as correct answer.