Multi-Pod PBR for EW traffic

[tuanquangnguyen]

How would you normally design for this to avoid as much tromboning as possible? For the use case where PBR nodes are two different, individual pairs of A/S firewalls?

Since location-aware is not supported for EW contract, I am thinking of deploying two different PBR policies which can have either Pod1 or Pod2 FW as the primary PBR node, and the other one as the backup. Then apply contract accordingly.

Are you guys doing the same, or do you have a different approach?

[Sergiu.Daniluk]

I think what you are looking for is symmetric PBR: https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-739571.html#F77

Here is a config example:

https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-739971.html#SymmetricPBRconfigurationexample

Basically, under the PBR policy - L3 destination, you configure both clusters VIPs

Stay safe,

Sergiu

[tuanquangnguyen]

Hi Sergiu,

I am aware of symmetric PBR, however it does not eliminate traffic tromboning in case E-W case (due to hashing can redirect the flow to either PBR node in Pod1, or Pod2) - Figure 77. This would actually risk our IPN being overutilised.

It only helps in assuring return and subsequent traffic is forwarded to the original PBR node afterwards.

Thanks,

Tuan

[Sergiu.Daniluk]

But even if you have let;s say FW cluster from Pod1 acting as the primary cluster, you will not get rid of EW traffic hair-pinning to Pod1 if the EPs are both in Pod2, right?

Maybe it's better first to analyze how the traffic patterns are (the percentage of EW communication between pods is higher or lower compared with the one inside a pod) and then go for the solution which reduces the hairpinning to minimum. I do not think there is a solution which can 100% eliminate it.

Stay safe,

Sergiu

[tuanquangnguyen]

Hi @Sergiu.Daniluk,

Well actually, for EPGs that are in the same Pod, I am thinking of using different PBR policy. So my PBR policies could be defined like this:

- NS-PBR: Includes both PBR nodes, with Resilient Hashing enabled, Location-aware enabled and Health Check

This applies to contract between L3Out EPGs and Application EPGs

- EW-Pod1-PBR: Includes PBR node in Pod1 as Active, Pod2 as Backup PBR node. Resilient Hashing enabled, Location-aware disabled and Health Check

This applies to contract between App EPGs that only reside in Pod1

- EW-Pod2-PBR: Includes PBR node in Pod2 as Active, Pod1 as Backup PBR node. Resilient Hashing enabled, Location-aware disabled and Health Check

This applies to contract between App EPGs that only reside in Pod2

It is mainly the PBR design for E-W traffic between Pods (either from EPG in Pod1 to EPG in Pod2, or EPG in Pod1 and EPG in a stretched BD with EP resides in Pod2) that I am currently confused. I think I might be looking for a "one size that fits all" solution here, but also want to discuss if someone has a use case that has been deployed in a production Multi-Pod environment and willing to share some insights

Thank you,

Tuan

[tuanquangnguyen]

I might be overthinking the situation at this point.

Like the previous reply, I would have 3 different PBR policies (1 for NS and 2 for EW within same Pod).

I think for EW between EPGs in different Pods, I would just apply one of the 2 EW PBR policies to their contract subject - the traffic has to cross IPN sooner or later. This way, it is also easier for troubleshooting since the cross-Pod traffic is destined for a PBR node in one Pod and not in the other Pod.

[Jose Maroto Grande]

Hello tuanquangnguyen,

In my opinion, you could reduce the traffic tromboning as much as possible if you configure symmetric PBR + location based PBR. I aggre with Sergio, you can not reduce to 100% if you have source ep in POD1 and destination ep in POD2. You have to consider that to apply location based PBR with E/W traffic, consumer and provider EPG have to be configured in diferent VRFs. Documantation says:

"The use of location-based PBR, which drastically reduces traffic hair-pinning in the north-south case, is not supported for some cases such as east-west traffic flows when the provider and consumer EPGs are part of the same VRF. The reason is that in some scenarios the PBR policy may be applied on different leaf nodes for incoming and return traffic, and this behavior may lead to loss of traffic symmetry. This situation occurs, for example, if other contracts (not using PBR) are in place between the same pair of EPGs, as this causes the provider leaf to learn endpoint information for the consumer EPG."

https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-739571.html#Option3RoutedfirewallwithL3OutpeeringandsymmetricPBR

[tuanquangnguyen]

Hi @Jose Maroto Grande,

The fact that they would need to be in different VRFs would complicate the design, since there would be further consideration to:

- Route leaking

- PBR node's BD would need to be in the same VRF as either provider, consumer or both EPGs

And there are also some cases where the EPGs are already in the same VRF. (Existing EPGs in the same VRF in Pod1, now one is extended to Pod2, and endpoint moved to Pod2)

Thanks,

Tuan

[tuanquangnguyen]

Hi folks, digging my own post because it was on the same topic.

With the following particular note (copied from the White Paper)

Does it essentially mean that it is impossible to use the same one-armed interface for both North-South and cross-pod East-West traffic (as currently used for single-pod)? How would you guys tackle this?