OSPF etherchannel between core routers

[hellomo]

I have two core routers at Site 1 and two core routers at Site 2. They are in OSPF area 0. They are considered my campus cores with multiple distribution routers connected to them, each with their own area. For the cores at each site should I build a link between them (2x40G) as an L3 ether-channel or as separate links. Diagram below. The green represents OSPF full neighbors. Which Site is considered best practice.

[paul driver · Accepted solution]

Hello
Personally id go for ECMP rather than a LAG, it will have a faster failover time and much easier to control the traffic if the need arises to policy route or path select on certain traffic/link

[Richard Burts]

The OP asks "should I build a link between them (2x40G) as an L3 ether-channel or as separate links". I agree with Paul that separate links would be better than ether channel. Configuring ether channel between sites suggests that vlans/subnets are shared between sites. As I understand the OP and the drawing it looks like vlans/subnets are in one site or the other, but not in both sites. So the links between sites are for transit traffic and I believe that separate links accomplish this better.

[MHM Cisco World]

If you have ECMP then you can easly load balance via CEF 

That hard to done via PO.

MHM

[balaji.bandi]

Depends on requirement, if that is Dark Fiber, you can use ECMP or Trunk as Layer 2, and you can have mesh OSPF neighbors. or if no Longer  Layer 2 requirement or extension , suggest to have p2p Layer 3 and OSPF peering.

Since i see there may be some Layer 2 loops in the diagram, depends on the VLAN spanning across.

[Ramblin Tech]

You do not mention the router models, but with 40G interfaces, I will assume that they forward in hardware (NPU) rather than software (CPU). Why is this important? Because it can impact two decision criteria mentioned in this discussion: (1) speed of link failure detection and (2) load sharing of traffic.

1. The fastest detection mechanism for link failure will be via hardware drivers if there is an L1 loss of signal. If there is no LoS, then keepalives must be used to detect the failure, with BFD being the generally preferred mechanism. LACP will be slow to detect link failure in the case of a LAG bundle (etherchannel), but with "modern" NPUs there is no reason to rely on LACP where there is support for hardware offload (HWO) of BoB/MicroBFD to monitor individual links. That is, use BoB (BFD over Bundle) to monitor the links instead of LACP and LAG link failure detection will be every bit as fast as failure detection on individual ECMP links via BFD. My underlying assumption here is that your NPU supports HWO of BoB, plus BoB-BLB (BFD over Logical Bundle) co-existence; BoB reporting individual link failures to the bundle-manager and BLB reporting L3 bundle failures to client routing protocols.
2. NPUs tend to use the same circuitry for calculating hash indexes for both LAG and ECMP, so there is likely to be little difference in the overall distribution of load-shared traffic between LAG and ECMP.  That is, with random, non-crafted traffic you should not see a load-sharing advantage either way.

So if both failure detection speed and load-sharing are generally equal between LAG and ECMP with NPU forwarding, which should you use? Network engineering is always about trade-offs and the trade-off that occurs to me is the fine granularity of traffic steering over individual links with ECMP versus its additional L3 address management. With a LAG bundle, there is only one IP subnet required for all the links in the bundle, but with ECMP, each link requires its own subnet. In your particular use-case, the additional subnet management is trivial, but for large networks with hundreds or thousands of nodes, each with multiple links to neighbor nodes, the IP address design, consumption and management are not necessarily trivial. Some network operators have the default posture that all inter-nodal links are LAG bundles, but again, this does not necessarily apply to your own use-case. With two links between your routers, you are not really going to go wrong either way.