cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
891
Views
10
Helpful
3
Replies

Strange forwarding behavior with L3Out

Nik Noltenius
Spotlight
Spotlight

Hello community,

 

we have the following situation (drawing attached for better understanding):
One tenant, one VRF, one L3Out.
Two Firewalls connect to the L3Out, one dedicated to a specific segment (FW) and a general purpose perimeter firewall (PFW).
We also have a firewall implemented via an unmanaged Service Graph (SGFW) that is sort of perimeter and east-west.
The L3Out has two External Networks configured, EN1 with 0.0.0.0/0 and EN2 with 1.1.1.0/24.
EN1 consumes a Service Graph contract SGCon, EN2 consumes a normal contract NormCon.
vzAny for the VRF provides SGCon, NormCon is only provided by a limited set of EPGs.

 

Our intention was that by default all traffic in and out of the fabric has to go across the SGFW via Service Graph. Inside the fabric we can use the two contracts to decide if the firewall is crossed or not. For traffic in "normal" networks that seems to work fine, however we see issues with the transfer networks configured under the L3Out (4.4.4.0 and 5.5.5.0).

If we have a traffic flow from our special segment to an address outside the fabric, let's say between 1.1.1.1 and 9.9.9.9 it is steered through the SG in both direction, as expected.
If we have a traffic flow from the transfer network (5.5.5.0) to the external address 9.9.9.9 it goes to the service graph only one way, i.e. 9.9.9.9 -> 5.5.5.5 goes to the SGFW, 5.5.5.5 -> 9.9.9.9 does not. This is kind of strange since both networks from an External Network point of view should be in EN1 and the only contract EN1 has is SGCon.
To make it even more mysterious if we add 5.5.5.0/24 to EN2, which is supposed to use the non-SG contract NormCon, then 5.5.5.5 -> 9.9.9.9 starts going to the Service Graph.

 

Can anyone make sense of the traffic flows and why they behave as they do?


I have two assumptions but I can't seem to get my head around them:
1. It has to do with vzAny also including the External Networks, so that we have contracts where we don't want / expect them
2. There is something special about directly connected subnets in the L3Out

 

Any help would be much appreciated.

Thank you and kind regards,
Nik

1 Accepted Solution

Accepted Solutions

joezersk
Cisco Employee
Cisco Employee

Hi Nik:

 

While I cannot say I fully understand the finer details of your setup, can I ask you to try one thing to see if it has any effect?

Under EN1, rather than having 0.0.0.0/0 can you make it two entries like this?

0.0.0.0/1

and

128.0.0.0/1

In ACI, the 0/0 route has special treatment in that it is always marked as PcTag 15.  You are really only supposed to have one of these anywhere in the fabric.  By breaking the 0/0 route into two parts (that add up to the same boundaries) you avoid this.  

If that does not change things, can you also update what version of ACI you run and what model leafs you are using here?

View solution in original post

3 Replies 3

joezersk
Cisco Employee
Cisco Employee

Hi Nik:

 

While I cannot say I fully understand the finer details of your setup, can I ask you to try one thing to see if it has any effect?

Under EN1, rather than having 0.0.0.0/0 can you make it two entries like this?

0.0.0.0/1

and

128.0.0.0/1

In ACI, the 0/0 route has special treatment in that it is always marked as PcTag 15.  You are really only supposed to have one of these anywhere in the fabric.  By breaking the 0/0 route into two parts (that add up to the same boundaries) you avoid this.  

If that does not change things, can you also update what version of ACI you run and what model leafs you are using here?

Hello joezersk,

 

thank you for the hint. After reproducing the issue in our lab environment I implemented the workaround you suggested. Indeed forwarding behavior changes and seems to follow the expected rules. However, I don't fully understand what actually changes.
I understand that we are running transit routing with a single L3Out. In the documentation it states that "an external default subnet (0.0.0.0/0) cannot be used to match both source and destination [...]" (https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/ACI_Best_Practices/b_ACI_Best_Practices/b_ACI_Best_Practices_chapter_010010.html), I think that's the problem that applies in our scenario. But then there is the exception that with vzAny I can use 0.0.0.0/0 which also matches our configuration so I'm a little confused.

Could you try to explain, what happens here?

I mean if I try to understand the documentation I would think 'okay, you cannot have source and destination swimming in the unspecific soup of 0.0.0.0/0'. But then I would expect no communication at all, however what we see is one direction goes to the contract (Service Graph in our case) and the other direction is allowed unconditionally. I would rather have expected the traffic to be dropped then due to the strict white-listing policy.

I'm sorry for all those questions, but I'd rather understand what is happening and why than simply accept the solution.

 

Thank you and kind regards,

Nik

Well, it has to do with the fact that in ACI, when you drop a raw 0.0.0.0/0 as an external network, it will ALWAYS get a pcTag of 15.  It does not matter what VRF or tenant you use, if you have more than one 0/0, they will all have pcTag 15.  This can create situations around contract enforcement as ACI won't know which pcTag 15 to use.  This is why you can have only one 0/0 as an external network anywhere in the fabric.  In the other cases mentioned in the link you provided they are either disabling contract enforcement altogether, or using aggregates and route-maps that provision something other than pcTag 15.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Save 25% on Day-2 Operations Add-On License