cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements
1353
Views
20
Helpful
13
Replies
OmkarPol52290
Beginner

ACI OSPF Best path selection

OSPF-L3out.png

We have configured OSPF between FTD firewall and ACI. Also, We are able to form neighborship with FTD. Now the issue is that when We form neighborship we see two best path in Routing table so now it should be load-balancing. Can we do some changes on ACI so that only one best path is selected keeping SVI and VPC topology. 

3 ACCEPTED SOLUTIONS

Accepted Solutions

Hi @OmkarPol52290 ,


Also, the Firewall is in Act/Std


I wish I'd know that earlier! That changes my advice completely!

To begin, you really don't want to run a dynamic routing protocol on a VPC that is configured with an Active/Standby neigbour. So remove the OSPF configuration. 

Instead:

  • On the ACI side,
    1. configure a secondary IP address on each leaf. In fact, you'll notice that when working your way through the L3Out wizard, the option to add a Secondary IP automatically shows up when adding an SVI. Make sure you use the same secondary IP on each Leaf
      image.png
    2. Once the L3Out wizard is complete, you'll need to re-visit the Node Profiles that were created and add appropriate static routes for EACH node
      image.png
  • One the firewall side, you'll add static route(s) to point to the SECONDARY IP address you configured on the ACI side

You'll find a better explanation on the Unofficial ACI Guide blog

Now, if you really, really, really want to use OSPF [I don't like this design because one adjacency (on the standby link) probably won't ever form until the primary goes down, and there would be the full adjacency exchange as OSPF re-establishes itself], stick with the point-to-point links, and the two L3Outs. You PROBABLY could use ONE L3Out with two node profiles, or even one node profile with two nodes, but I'd have to research that before I'd give a definite answer - and since I haven't researched it this whole paragraph may be utterly wrong! 

RedNectar aka Chris Welsh. Forum Tips: 1. Click the camera icon then paste pictures in the grey area. 2. After clicking Preformatted, click HTML before pasting to preserve spacing

View solution in original post

Hi @OmkarPol52290 ,


We need dynamic routing as it's enabled on all infra and doesn't make sense to keep static just for ACI.


This reminds me of arguments I heard in the 1990s when people insisted that their Solaris workstations had to run RIP "because it's enabled on the network and doesn't make sense to just configure a default gateway"

When ACI was designed, it was designed to more like an "edge appliance" than a "core router". (In fact, v1.0 didn't even support transit routing). So generally having a default static route configured in ACI is not a big deal if there is only one external router. I don't know if your design has other external routers apart from the firewall, so I can't comment.

But, I don't see why we can't get some kind of OSPF design to work for you.  The secondary IP address thing is only relevant for static routing, and as for...


Also this secondary IP design always rasises a fault in ACI about packet-dropped. Are you also experiencing it.


I haven't experienced it, but to be honest I've only configured it in a lab situation as a proof of concept for myself.  Let's add a call-out to @Sergiu.Daniluk and ask for his opinion - he's had more real-world experience than I have, and I won't be offended if he comes up with a simple and completely different answer to mine

But at the moment, I think your best bet if you want to stick with OSPF is the two point-to-point links.

RedNectar aka Chris Welsh. Forum Tips: 1. Click the camera icon then paste pictures in the grey area. 2. After clicking Preformatted, click HTML before pasting to preserve spacing

View solution in original post

Hi Guys,

 

Interesting topic. Thanks Chris for tagging me.

First I would like to answer the question of this topic: "Can we do some changes on ACI so that only one best path is selected keeping SVI and VPC topology?" answer is NO. VPC and best path selection is a no-go because of multiple reasons, some of which are already explained by Chris.

Second, I would like to touch this topic: "So the requirement is that by default ECMP will happen which can lead to suboptimal forwarding according to a few of our colleagues due to which we have to make this change".

Could you elaborate what you mean by suboptimal forwarding? By design, ACI Fabric is a CLOS network, meaning from any compute leaf to your FTD, you will have the same number of hops regardless if you have 1 "best" leaf, or 2 active/active leafs (even if they are or not configured in vPC). The advantage of VPC is fault tolerance in case of one link failure between FTD and ACI Leafs.

Only "advantage" in having one "best" leaf is predictability of traffic forwarding/routing. But if I would have to choose between  predictability on which interface traffic will be forwarded and fault tolerance, the second one will always win especially in a data center.

If is not clear enough already, my strong suggestion is to avoid path selection and use vPC as-is (if using static routes) or two point-to-point links (if using OSPF simply to avoid DR/BDRs).

If not convinced already to use active/active Leafs, and you still plan on using a "best path" leaf, then I would suggest to use one single L3Out with two logical node profiles, one for each leaf, and in each logical node profile you will have one logical interface profile, with two SVIs (same IP) but different interface - one to Active FW and one to Standby FW.

Last but not least, "Also this secondary IP design always rasises a fault in ACI about packet-dropped. Are you also experiencing it." - I never seen problems or faults with secondary IP designs, unless configured incorrectly. Could be that what you are seeing is not about the secondary IP, but rather about vPC - packet drops (control plane packets like LLDP) between vpc peers which is a cosmetic bug.

 

Stay safe,

Sergiu

View solution in original post

13 REPLIES 13
ashok_boin
Contributor

Just influence the decision through "ip ospf cost" command to have your best path.


With best regards...
Ashok

But how can we do it for SVI for vPC in ACI? They will have the same interface policy.

RedNectar
Advocate

@OmkarPol52290 

My first though is "Why do you want to change the way it is designed to work?"

BUT

To answer your question - you'll need to re-design the setup so there is NO VPC - which is not a bad design anyway. But will require an extra IP on the FW. 

RedNectar aka Chris Welsh. Forum Tips: 1. Click the camera icon then paste pictures in the grey area. 2. After clicking Preformatted, click HTML before pasting to preserve spacing

Hey @RedNectar ,

 

Hope you're doing ok.

 

So the requirement is that by default ECMP will happen which can lead to suboptimal forwarding according to a few of our colleagues due to which we have to make this change. So, I tried it in Traditional Setup, and I was able to achieve it via 

 

router OSPF 1

maximum-path 1

############################################################################

Router#
*Dec 10 12:27:25.983: %SYS-5-CONFIG_I: Configured from console by consoleconf t
Router#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
+ - replicated route, % - next hop override

Gateway of last resort is not set

192.168.10.0/24 is variably subnetted, 2 subnets, 2 masks
C 192.168.10.0/24 is directly connected, FastEthernet0/0
L 192.168.10.1/32 is directly connected, FastEthernet0/0
O 192.168.20.0/24 [110/2] via 192.168.10.102, 00:00:04, FastEthernet0/0
[110/2] via 192.168.10.101, 00:00:04, FastEthernet0/0
Router#
Router#
Router#
Router#conf t
Enter configuration commands, one per line. End with CNTL/Z.
Router(config)#router ospf 1
Router(config-router)#maximum-paths 1
Router(config-router)#end
Router#wr
Building configuration...
[OK]
Router#
Router#
Router#
Router#
*Dec 10 12:27:39.807: %SYS-5-CONFIG_I: Configured from console by consoleconf t
Router#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
+ - replicated route, % - next hop override

Gateway of last resort is not set

192.168.10.0/24 is variably subnetted, 2 subnets, 2 masks
C 192.168.10.0/24 is directly connected, FastEthernet0/0
L 192.168.10.1/32 is directly connected, FastEthernet0/0
O 192.168.20.0/24 [110/2] via 192.168.10.102, 00:00:05, FastEthernet0/0
Router#

#############################################################################

On northbound I have Firepower so is there a way I can achieve it.

Hi @OmkarPol52290 ,

I think I see your problem - you are thinking of ACI as "a router"

But it isn't.

EACH LEAF is a router

So, what you REALLY have in the above scenario is something that looks (as far as OSPF is concerned) more like this:

image.png

A multi-access OSPF network with three routers, all on the same subnet.  This is made possible because when you create a L3Out in ACI and use a VPC as the attachment, ACI creates an internal hidden Bridge Domain that allows the two leaves to see each other and form an OSPF adjacency - with DR and BDR elections.  When the external device sends OSPF Hellos, the Hellos are seen by both leaves and each leaf forms an adjacency.

The problem is, since you've configured this as a VPC, each of the interfaces of the VPC are configured exactly the same - you can'd change the OSPF cost on either interface.

Now, from your explanation

ECMP will happen which can lead to suboptimal forwarding according to a few of our colleagues

I'm guessing that there are severs on one or other of the leaves that you'd like to

  1. ALWAYS forward traffic directly from the leaf to which they are attached, and your colleagues are worried that that the traffic will sometimes be sent to the other leaf before being forwarded to the FTD-FW
  2. have the reverse traffic (from the firewall) always be sent to the same leaf

As for #1 above - don't worry, That will always happen unless the particular leaf looses it's physical connection.

As for #2, ensuring that the return traffic favours one particular leaf, I honestly think you should not care.  ACI was designed to be flexible and dynamic, so trying to do push traffic one way or another is not a good idea generally.

However, the customer is always right, so let's assume you really really really want to implement this BAD idea.

  • Then the simplest way would be to configure the FTD-FW with static host routes with IP SLAs for the servers you want to always go to a particular Leaf, rather than trying to manipulate ACI to work in a way that it was not designed to do.
  • The other (ACI centric) way would be to forget the VPC and configure two point-to-point links.  

Like this

image.png

Now you configure two L3Outs, one for Leaf101 and one for Leaf102.  If you want the external router to prefer say Leaf102, then increase the cost of the interface in Leaf101.

I hope this helps.


Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem.


 

RedNectar aka Chris Welsh. Forum Tips: 1. Click the camera icon then paste pictures in the grey area. 2. After clicking Preformatted, click HTML before pasting to preserve spacing

Hey @RedNectar,

 

Thanks as always so I'm confused by the Last statement " Now you configure two L3Outs, " can't we use the same L3 out and manipulate the Priority per Leaf as we'll be able to have separate Leaf profiles and Interface profiles.

 

Also, the Firewall is in Act/Std so we need to deploy it in SVI as Routed Interface won't work.

Also, we did try /30 between Leaf 101/102  to FTD with /30 each but OSPF neighborship was not coming up. We still are looking into that. I doubt maybe This is due to the same Bridge domain but I don't think that could be the issue here.

I'll be online on Monday I'll check and let you know.

 

Thanks as always.

Hi @OmkarPol52290 ,


Also, the Firewall is in Act/Std


I wish I'd know that earlier! That changes my advice completely!

To begin, you really don't want to run a dynamic routing protocol on a VPC that is configured with an Active/Standby neigbour. So remove the OSPF configuration. 

Instead:

  • On the ACI side,
    1. configure a secondary IP address on each leaf. In fact, you'll notice that when working your way through the L3Out wizard, the option to add a Secondary IP automatically shows up when adding an SVI. Make sure you use the same secondary IP on each Leaf
      image.png
    2. Once the L3Out wizard is complete, you'll need to re-visit the Node Profiles that were created and add appropriate static routes for EACH node
      image.png
  • One the firewall side, you'll add static route(s) to point to the SECONDARY IP address you configured on the ACI side

You'll find a better explanation on the Unofficial ACI Guide blog

Now, if you really, really, really want to use OSPF [I don't like this design because one adjacency (on the standby link) probably won't ever form until the primary goes down, and there would be the full adjacency exchange as OSPF re-establishes itself], stick with the point-to-point links, and the two L3Outs. You PROBABLY could use ONE L3Out with two node profiles, or even one node profile with two nodes, but I'd have to research that before I'd give a definite answer - and since I haven't researched it this whole paragraph may be utterly wrong! 

RedNectar aka Chris Welsh. Forum Tips: 1. Click the camera icon then paste pictures in the grey area. 2. After clicking Preformatted, click HTML before pasting to preserve spacing

Hey @RedNectar ,

First of all really sorry that I forgot to tell you and also add that part in diagram.

 

We need dynamic routing as it's enabled on all infra and doesn't make sense to keep static just for ACI.

and this secondary IP with vPC I have done this multiple times. But we cannot do this here.

 

I'll be joining on monday in next 24 hours I'll let you know my findings.

 

Also this secondary IP design always rasises a fault in ACI about packet-dropped. Are you also experiencing it.

 

Hi @OmkarPol52290 ,


We need dynamic routing as it's enabled on all infra and doesn't make sense to keep static just for ACI.


This reminds me of arguments I heard in the 1990s when people insisted that their Solaris workstations had to run RIP "because it's enabled on the network and doesn't make sense to just configure a default gateway"

When ACI was designed, it was designed to more like an "edge appliance" than a "core router". (In fact, v1.0 didn't even support transit routing). So generally having a default static route configured in ACI is not a big deal if there is only one external router. I don't know if your design has other external routers apart from the firewall, so I can't comment.

But, I don't see why we can't get some kind of OSPF design to work for you.  The secondary IP address thing is only relevant for static routing, and as for...


Also this secondary IP design always rasises a fault in ACI about packet-dropped. Are you also experiencing it.


I haven't experienced it, but to be honest I've only configured it in a lab situation as a proof of concept for myself.  Let's add a call-out to @Sergiu.Daniluk and ask for his opinion - he's had more real-world experience than I have, and I won't be offended if he comes up with a simple and completely different answer to mine

But at the moment, I think your best bet if you want to stick with OSPF is the two point-to-point links.

RedNectar aka Chris Welsh. Forum Tips: 1. Click the camera icon then paste pictures in the grey area. 2. After clicking Preformatted, click HTML before pasting to preserve spacing

Hi Guys,

 

Interesting topic. Thanks Chris for tagging me.

First I would like to answer the question of this topic: "Can we do some changes on ACI so that only one best path is selected keeping SVI and VPC topology?" answer is NO. VPC and best path selection is a no-go because of multiple reasons, some of which are already explained by Chris.

Second, I would like to touch this topic: "So the requirement is that by default ECMP will happen which can lead to suboptimal forwarding according to a few of our colleagues due to which we have to make this change".

Could you elaborate what you mean by suboptimal forwarding? By design, ACI Fabric is a CLOS network, meaning from any compute leaf to your FTD, you will have the same number of hops regardless if you have 1 "best" leaf, or 2 active/active leafs (even if they are or not configured in vPC). The advantage of VPC is fault tolerance in case of one link failure between FTD and ACI Leafs.

Only "advantage" in having one "best" leaf is predictability of traffic forwarding/routing. But if I would have to choose between  predictability on which interface traffic will be forwarded and fault tolerance, the second one will always win especially in a data center.

If is not clear enough already, my strong suggestion is to avoid path selection and use vPC as-is (if using static routes) or two point-to-point links (if using OSPF simply to avoid DR/BDRs).

If not convinced already to use active/active Leafs, and you still plan on using a "best path" leaf, then I would suggest to use one single L3Out with two logical node profiles, one for each leaf, and in each logical node profile you will have one logical interface profile, with two SVIs (same IP) but different interface - one to Active FW and one to Standby FW.

Last but not least, "Also this secondary IP design always rasises a fault in ACI about packet-dropped. Are you also experiencing it." - I never seen problems or faults with secondary IP designs, unless configured incorrectly. Could be that what you are seeing is not about the secondary IP, but rather about vPC - packet drops (control plane packets like LLDP) between vpc peers which is a cosmetic bug.

 

Stay safe,

Sergiu

Hey @Sergiu.Daniluk,

 

Thanks for your reply.

It's as you suspected - The only "advantage" in having one "best" leaf is the predictability of traffic forwarding/routing. it's not for the underlay part. it's more from the end of the FTD firewall as it'll have the two best paths to go towards ACI fabric. So sometimes traffic will go from Leaf 101 to Leaf 102 a few times. as FW will send a packet to Leaf101 and the Server might be connected on Leaf 102.

 

As you said we have already configured it but the neighborship is not coming up but that is something we have to check.

 

I completely agree with you and Chris both but We cannot argue with the application team, for some reason they have a problem with this.

 

no issues as always thanks, guys.

I'll update you once I get the results.

 

As for that packet dropped Yes, it's like you said the description says packet-dropped. F1545 and 1547 (Is this it).

Also, I know I'm asking a lot but we see traceroute as sometimes it shows both the IPs for Leaf 

as let's say we have Leaf 1 (.11) and Leaf 2 (.12) and VIP is (.10) then we are observing trace after the FW hop as (.11) and (.12).

Note - This setup has Secondary IP configured with vPC with FW pointing at Sec IP. 

 

It's expected to see both leaf in traceroute if the traffic arrives on Leaf1 but the destination EP is single homed on Leaf2.

Regarding the faults, have a look on this: https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuz99892?rfs=iqvred

Its cosmetic - just ignore/squelch the fault.

"We cannot argue with the application team, for some reason they have a problem with this." -> in my opinion, if the problem they have is not sustained with technical arguments, then it's invalid. In the end, the network is configured for applications, so it's network engineer to decide what is the best/optimal design.

 

Anyway, I hope all the details provided by Chris and I will help you and your team to take the best decision for the final implementation.

 

Stay safe,

Sergiu

 

Hey @RedNectar and @Sergiu.Daniluk 

Thanks for your help.

I'll update you guys on progress once I have the complete this.