VxLAN+EVPN design question

[satish.txt1]

We are planning to build small datacenter which will have around 2 spine + 20 VTEP (leaf switches) requirement so related that i have following questions. 

Hardware model:

Spine: Cisco nexus 9336-FX2

Leaf: Cisco nexus 9396PX 

1. Does OSFP in underlay would be fine with 20 VTEP?  (or eBGP would be good choice?)

3. Does OSPF route reflector has any CPU impact with 20 VTEP or adding more in future? 

4. What would be best choice here for simplicity Multicast Vs Ingress-replication? 

5. I heard HER can't scale like multicast does so is there any limitation on HER can handle number of VTEP? 

6. If i do vPC on 2 leaf on racks does that count single VTEP or 2 VTEP? 

[Sergiu.Daniluk · Accepted solution]

Hi @satish.txt1 

1. Does OSFP in underlay would be fine with 20 VTEP? (or eBGP would be good choice?)

I do not see a problem running OSPF

3. Does OSPF route reflector has any CPU impact with 20 VTEP or adding more in future?

There is no route reflector in OSPF. Moreever, you can use OSPF with point-to-point links between leafs and spines, and you avoid DR/BDR election process.

4. What would be best choice here for simplicity Multicast Vs Ingress-replication?

Purely from simplicity of the configuration POV, BGP EVPN ingress replication is definitely easier. You do not have to deal with mcast configuration in the underlay and group selection for VNIs.

5. I heard HER can't scale like multicast does so is there any limitation on HER can handle number of VTEP?

You can verify the scalability for EVPN ingress replication (or head-end replication): https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/92x/scalability/guide_923/b_Cisco_Nexus_9000_Series_NX-OS_Verified_Scalability_Guide_923.html  (Section  VXLAN BGP eVPN Ingress Replication)  -> max number of VTEPs = 254.

6. If i do vPC on 2 leaf on racks does that count single VTEP or 2 VTEP?

It counts as 2 VTEPs. Remember, you can still connect orphan ports to a vPC peer.

Hope it helps and stay safe,

Sergiu

[satish.txt1]

Thank you for reply, 

- You said OSPF should work and if OSPF is simple solution then why people using eBGP for underlay and why they use Multiple AS for each leaf? 

- Does HER use lots of bandwidth compare ro Multicast is there any concern or does it impact any application? Let's say if i use L2VNI only design and don't use L3VNI or arp suppression in that case does it make HER performance worst? 

[f00z]

Because eBGP is.. BGP.. It's easy to write policies and filters but unless you really have a use for it, it doesn't matter.  There is no performance or even convergence difference between ospf and eBGP in the underlay unless there are some really crazy scale numbers which you write policies everywhere to optimize it using BGP.

No performance impact from ingress replication vs multicast.  It's just the VTEP itself does all the replication instead of pushing the replication out to the others.  With multicast it would propagate out and get replicated as it passes through each device splitting up the replication process across a whole lot of devices. With ingress replication, everything the end host sends broadcast or multicast wise is replicated to EVERY destination from the single VTEP it is connected to. So you can imagine if you send 1g of multicast and it has to to go 50 other VTEPS, that's 50 gigs of multicast, which is why you shouldn't use it if you do any tenant multicast traffic. It's perfectly fine for ARP and ND and VRRP and things like that but not actual tenant routed multicast or streaming or whatever else tenants do.

So basically if your network only has a tiny bit of multicast traffic and you don't do routed multicast for any customers, using IR is super easy. 

[satish.txt1]

We have very basic application in datacenter all based on IPv4 (UDP/TCP), We don't have any multicast application. so you are saying IR/HER is not good if we run multicast application on our server farms right? otherwise for standard IPv4 ARP kind of BUM it can handle smoothly. Just trying to understand where its going to consume resource and bandwidth. 

[f00z]

Right, if the tenants don't have a lot of BUM traffic you are fine. If you have applications that use multicast at high bandwidth then switch to mcast replication or TRM.

ARP/ND barely uses anything, especially with suppression. 

[satish.txt1]

We don't have any tenants because it for our own use, We don't have multicast application so i think I like to go with IR/HER and not add more component to troubleshoot which is not my skill at this moment. 

Also we are using all L2VNI style design (Because of legacy reason  we can't use anycast gateway also so in that case does ARP suppression help or not)?

[Sergiu.Daniluk]

Hello,

The advantage of OSPF, being an link-state protocol, is that it considers all the links between Leafs and Spines , as well as the respective link speeds, to compute the single best path or multiple equal-cost best paths through the network.

On the other hand, if you have a very large number of leaf switches in your fabric. eBGP is known for it's scalability (see https://tools.ietf.org/html/rfc7938 ), low periodic updates overhead and is also great for enforcing policies in a network. Plus, you use only one eBGP process for both underlay and overlay.

Ingress replication definitely uses more bandwidth compared with multicast option, and it also consumes more resources on the replication VTEP.

is there any concern or does it impact any application? - it depends on the amount of BUM traffic. If your network is forwarding a high load of multicast traffic, you should consider TRM - a more scalable solution, compared with normal multicast deployment or HER. 

Cheers,

Sergiu

[f00z]

1. Does OSFP in underlay would be fine with 20 VTEP?  (or eBGP would be good choice?)

  OSPF is fine as IGP and is recommended to use from Cisco

3. Does OSPF route reflector has any CPU impact with 20 VTEP or adding more in future? 

   You use iBGP for route reflector and also is fine, number of vtep doesn't matter as much as number of routes

4. What would be best choice here for simplicity Multicast Vs Ingress-replication? 

  Ingress replication works beautifully and no reason not to use it unless you have a lot of tenant multicast/multicast routing

5. I heard HER can't scale like multicast does so is there any limitation on HER can handle number of VTEP? 

  Scale isn't the issue, bandwidth is the issue. If you scale it out and you use a lot of multicast, one vtep could be

  replicating many gigabits/s of multicast.  If you only use multicast for ARP/ND and maybe some application sync

  stuff, it's not a big deal.  If the tenants need multicast other than simple things, it's best to configure underlay for it.

6. If i do vPC on 2 leaf on racks does that count single VTEP or 2 VTEP? 

  It counts as three VTEPS (for the pair) and uses ECMP resources for the shared VTEP if you have PIP routes. 

  One is shared VTEP, other is normal (PIP) VTEP.  If you have no type 5 and no PIP it should only count as one (shared) VTEP.

  I'd plan it out as taking up 3 vtep slots for the pair just in case. 

[satish.txt1]

In HER case what could be the issue with bandwidth or is there any performance impact using HER vs Multicast?