How to have symmetric traffic in BGP Multihomed network

[S. B]

Hi,

Is there any solution (except using PBR) for controlling the outgoing traffic to the service providers from our side?

We want each buildings send/receive traffic go through each provider, while all providers can act as backup for others.

For controlling the incoming traffic, its easy.

Just advertising the specific/24 of the building + the complete /22 address, to each provider.

How about the outgoing traffic?

As we want it symmetric, so each building will send their own traffic from their specific service provider

If no solution, maybe with putting each customer/building in a vrf and the 3 service providers in a shared vrf? and with import/export? (makes it complicated)

Or maybe sending the traffic from each building to the correct router, and then controlling the outgoing traffic with BGP attributes?(but seems still in some failover scenarios, requires PBR)

Thanks

[Richard Burts]

Your question is asking how to control forwarding of outbound traffic based on the source of the packet (which building). This is pretty much the definition of PBR. But your question seems to exclude PBR as a solution. Is there a reason for this that you can explain to us?

[S. B]

Dear Mr.Richard,

Thanks.

For 2 reasons:

1- Because of the CPU load that PBR usually puts on the routers (L3 switches) which are located at the edge(the routers that are doing BGP at the Edge)

2- It Gets complicated when there are 2 routers(L3 switches) at the edge , so IP SLA needs to be added to it, as there are several failure situations (ISP Failure, Switch Failure, Router Failure, Edge Firewall failure)

wondering if there could be any other solution to handle this.

Is there any way to handle this with vrf import export? putting all the 3 internet connections in shared vrf, and importing to each building vrf with adding  a parameter that will force the building to send its traffic to the correct router that its ISP is connected to it (maybe with 3 ISP's not possible, but with 2 routers and 2 ISP's possible?)

I wanted to add another question.

With using PBR on the edge routers(layer 3 switches): If the Edge device is a Stack Cisco 3850 switch, is there any experience of its CPU handling 1 Gbps upload and and download traffic on it with the PBR configured? i mean mostly the PBR will put CPU load on it.

And also, can a nexus 3064 (nexus 3K) CPU , handle this amount of traffic on it with PBR configured? (with knowing that Nexus 3064 doesn't have PBR TCAM entry assigned to it and requires to remove unused TCAM resource and add to TCAM PBR resource)

Thanks

[Giuseppe Larosa]

Hello @S. B ,

>> i mean mostly the PBR will put CPU load on it.

Multilayer switches are able to implement PBR in CEF in TCAM table there will be a pointer to the next-hop.

PBR will not impact on main CPU of L3 switches or the feature would be not allowed given the great difference in forwarding capabilities between ASIC chips and the main CPU of the switch.

>> and requires to remove unused TCAM resource and add to TCAM PBR resource

This is clearly the way to go and it is platform specific for Nexus 3064

Hope to help

Giuseppe

[Richard Burts · Accepted solution]

Thanks for explaining your reservations about PBR. While it is true that on some switches the PBR is done in software and so does drive up CPU. But there are now many switches for which PBR is done in hardware. See this link for an example of that

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst4500/12-2/25ew/configuration/guide/conf/pbroute.pdf

and this link to a discussion in the community about hardware processing of PBR

https://community.cisco.com/t5/switching/does-pbr-in-vrf-is-hardware-switched-on-cat6509-sup720-3b/td-p/1500388

It is probably true that multiple failure scenarios would complicate the implementation of PBR. I believe that those multiple failure scenarios would also complicate the vrf approach.

I am not sure but I believe the vrf approach would not accomplish what you want. I am not aware of other solutions that would do it but wonder if perhaps something like SDWAN might provide a solution.

I do not have performance information about the 3850 or the 3064 in processing PBR. Perhaps someone else in the community might have that information and if so I hope they jump into this discussion.

[Richard Burts]

I am glad that our explanations have been helpful.  Thank you for marking this question as solved. This will help other participants in the community to identify discussions which have helpful information. This community is an excellent place to ask questions and to learn about networking. I hope to see you continue to be active in the community.

[S. B]

Thanks for your information

@Richard Burts 

@Giuseppe Larosa 

Actually, i posted another question in the following link, regarding to this post (performance information about the 3850 or the 3064 in processing PBR. ) and also about the meaning of the TCAM table entry for the PBR functionality.

But still no answer from anyone yet. (Can you also check)

https://community.cisco.com/t5/routing/cisco-nexus-3064-cisco-catalyst-3850-tcam-table-for-pbr-question/td-p/4441639

I still have question about managing the outgoing traffic in these types of designs. Where we want the send/receive traffic be symmetric. 

1- PBR

• vey complicated when failure scenarios are documented
• performance issues on the Edge router
• no exact document found on performance on devices on PBR

2- Doing source routing from the down layer device ( following picture)

• with policy based routing 
• with HSRP
• with vrf and VDOM separation

3- SDN (on the edge layer router)

Is there any experience to be shared on doing symmetric traffic (send/receive) on with the cisco routers (SDN) a the edge layer, while having their BGP functionality?

Thanks

[Richard Burts]

I will look at your new post and see if I have something to contribute to that discussion. But first let me respond to your points in this post

1- PBR

- PBR may become complicated when multiple failure scenarios must be addressed. Pretty much any solution you look at for multiple failure scenarios (other then dynamic routing protocol replacement of a failed primary route when an inferior route is available) is going to be complicated.

- performance issues on edge router when using PBR. Deploying any new feature on a router is going to have some impact on performance. I suspect that you over estimate the impact of PBR on the edge router but do not have sufficient technical details to discuss this aspect.

2- Source routing from down layer device

- I believe that PBR is an appropriate solution.

- I do not see how HSRP is a factor in this. HSRP can help control which edge router a building sends its traffic to. But how does HSRP influence which ISP the edge router sends the traffic to?

- I am not clear how vrf VDOM would work.

3- SDN

I have not experience to share about this.

[S. B]

Hi Mr.Richard,

For the VDOM,my idea was to split each subnet(customer) that is behind the firewall, in a separate VDOM, and send the outgoing route to the specific router that is handling the service provider connection.  So, the route path selection is done from the firewall side, and les PBR will be required on the edge router.

About, HSRP, the same idea. HSRP+PBR on the lower layer devices,(before the edge router), to send the traffic to  the specific router that is handling the service provider connection.  So, the route path selection is done from the fiewall side, and les PBR will be required on the edge router.

BR