ACI Intersite L3Out Egress Path Steering

[Nik Noltenius]

Hi folks,
I have a problem giving me sleepless nights.
We have a Multi-Site environment with two fabrics. Both sites are connecting to perimeter firewalls via BGP, which are both active and provide the same prefixes. We deployed Intersite L3Out so that in case of any L3Out going down, EPGs in that site can still communicate with the outside network using the remote L3Out.
Now we have problems with asymmetric routing and our goal is to always steer traffic to the L3Out in site A as long as everything is fine. This means the L3Out in site B - although associated with all EPGs - should not be used unless the L3Out in site A goes down.

I am definitely no BGP expert - far from it actually - but I tried AS-Path-Prepending and setting the local Preference to make the prefixes learned in A better or those in B worse. Nothing seems to work though, all traffic is always taking the local L3Out for it seems to be the best option.
I also watched Max Ardica's great videos on configuring Multi-Site (https://www.youtube.com/watch?v=M3uXv8Pf2GM) but what I take away from it - use an active/standby cluster - is not an option.
Any ideas how I could prefer one L3Out over the other?
Thank you and kind regards,
Nik

[RedNectar]

Hi @Nik Noltenius ,

At a quick glance. it sounds to me that you need to think about using ACI InterSite - there's a guide on the Unofficial ACI Guide here. I know my friend @Sergiu.Daniluk has had more experience with MSO tha I have - he may come up wit a better answer!

I hope this helps

---

Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem

---

[Nik Noltenius]

Thank you Chris,

yes, Intersite is already configured. It was a prerequisite to be able to use the remote's L3Out to begin with. Unfortunately in this case it's rather part of the problem than of the solution. I read everything Multi-Site - and much more - related on the the unofficial ACI guide before, though. So your hint is definitely helpful for those who might stumble across this discussion later.

Both the https://unofficialaciguide.com and your https://rednectar.net/category/cisco/aci are prime resources for me, so thank you for the great work!

[RedNectar]

Hi @Nik Noltenius ,

I realised after reading Robert's reply that I had not actually read the TITLE of your question.

(insert shame icon here)

And it would seem that your option (C) - host routes - might be the simplest way to achieve active/active if you descide to persue that path.

And many thanks for your kind words.

[Robert Burns]

With Intersite L3out, the traffic for each site should ALWAYS prefer the local L3out first, then pending it becoming unavailable would the traffic transit the ISN to the remote L3out.   In a true active/active site design this is what most would prefer.  I believe the only way to accomplish what you want is to have an Active/Passive design.

I'd like to know why in an A/A design you'd want to send all local traffic across your ISN to one site.

Robert

[Nik Noltenius]

Thank you Robert,

I agree but I'm looking for alternatives still.
There is one explanation and one real reason for the desired behavior. The explanation is that it has been done this way with the legacy network for years. In the legacy environment they used local-preference to keep routing to one location. For troublehooting purposes this was supposed to be easier because you would always know on which firewall and on which l3-switch to look for traffic. That's the explanation, since "we've always done it like this" is no technical argument.
From a design perspective it also has benefits. Part of the traffic is east-west between the sites. It will leave one fabric in tenant A and return to the other fabric in tenant B after being inspected by a firewall. Now the firewalls also have a connection in the outside world which they will use from a routing metric point of view. However it the DCI used for the ISN has a higher bandwidth than the outside connection, it woul make sense to prefer the internal way.
That being said, I again agree that an Active / Passive scenario would probably work best. The question remains: Are there alternatives? I thought about a few:
A) set a local-preferene on the L3Out for the ISN in Tenant infra. I think this should work, but I'm not sure if the L3Out in infra is somehow special and should not be messed with.

B) set import route control on the standby site's L3 out, block all routes from being imported, and set a backup static default route which then will only be used if the main L3out fails. I think this should work, too. ACI doesn't need the routes, a default would be sufficient. We're using BGP only to let the external world know about new networks.

C) let both L3Outs be active and activate Host-Routes for stretched BDs to avoid asymmetric routing. I'm not sure when - in the sence of how many host-routes - this might become problematic for the external device receiving all the prefixes.

D) use an unmanaged service graph with PBR. I've honestly not thought this one through, yet and I believe there are a couple of dependencies but in theory this should work, too.

Unfortunately I have no Multi-Site lab and every test must be neatly planned, so if possible I'd like you to comment on these options.

Kind regards,
Nik

[Robert Burns · Accepted solution]

Nik,
Some comments on your proposed options.   Keep in mind whatever option you go with ensure you account for both ingress & egress steering.  

A)- No-go.  don't mess with the infra.  You can really mess up your interPod/Site traffic doing so.  

B) Possible

C) I don't think this would accomplish what you want.  HBR (Host Border Routes) wouldn't force Site-B endpoint traffic out an L3out in Site-A, which is what you stated you want to happen.  It would only make external access endpoints in each site direct from that site's respective L3out, but it has no impact on egress traffic.  Site-B hosts would still access Site-B L3out. Scale could be an issue depending on how many HBRs you're advertising as well.

D) Also possible, but a little more complex to setup.

Another options E) would be to apply an ingress Route map to the L3out in Site-A and tuning the MED values to learned routes giving them a hiring priority before they're injected into the control plane. 

This is detailed in Max's whitepaper: https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-739609.html

Robert

[Nik Noltenius]

Hello Robert,

thank you for your input.

Regarding option C) you're correct. In this case I would have accepted, that all EPGs take their nearest exit and the HBRs would have been used to steer ingress traffic accordingly. Not achieving the original goal, but it would have been an okay-ish workaround.

The MED tuning seems to be exactly what I was looking for. I've read the white paper a couple of times so I honestly don't know how I could miss it. That was the story about the wood and the trees...

Anyway we tried a different option now which is letting the external BGP peer on the "secondary" site use AS-Path prepend to make it's prefixes less useful. This seems to work thus far, problem solved.

Regards,
Nik