BGP Traffic Routing

[Rob Sciarrino]

We currently have 2 Service Providers connected to 2 ASR1004'S.  On ASR1004A, Provider #1 is connected with a 10GB, Provider #2 is connected with a 10GB.  Provider #2 is also connected with a 10GB on ASR1004B.

After some BGP changes last week, I was trying to send more customer traffic through Provider #2 on ASR-A and ASR-B.  However, what has happened is more traffic is going out ASR-B and almost zero traffic on Provider #2 on ASR-A.  Provider #1 on ASR-A still has a significant amount of traffic.  I confirmed with Provider #2 that both 10GB interfaces go to separate routers, but with the same AS.  I've attached BGP configs with public information redacted.  I'll also attach a network diagram. 

Thoughts?

ASR1004-A

router bgp 1111
bgp router-id 192.168.1.199
bgp always-compare-med
bgp cluster-id 1111
bgp log-neighbor-changes
bgp deterministic-med
bgp bestpath compare-routerid
redistribute connected
redistribute static
neighbor 172.16.0.101 remote-as 100
neighbor 172.16.0.101 next-hop-self
neighbor 172.16.0.101 send-community
neighbor 172.16.0.101 soft-reconfiguration inbound
neighbor 172.16.0.101 route-map SET-LOCAL-PREF-75 in
neighbor 10.0.200.9 remote-as 200
neighbor 10.0.200.9 version 4
neighbor 10.0.200.9 soft-reconfiguration inbound
neighbor 10.0.200.9 route-map SET-LOCAL-PREF-100 in
neighbor 10.0.200.9 route-map poison-route-200 out

route-map SET-LOCAL-PREF-100 permit 10
set local-preference 100
!
route-map SET-LOCAL-PREF-75 permit 10
set local-preference 75

route-map poison-route-200 permit 10
set as-path prepend 1111 1111

ASR1004B

!
router bgp 1111
bgp router-id 192.168.1.200
bgp always-compare-med
bgp cluster-id 1111
bgp log-neighbor-changes
bgp deterministic-med
bgp bestpath compare-routerid
redistribute connected
redistribute static
neighbor 10.0.100.41 remote-as 200
neighbor 10.0.100.41 version 4
neighbor 10.0.100.41 soft-reconfiguration inbound
neighbor 10.0.100.41 route-map SET-LOCAL-PREF-200 in
neighbor 10.0.100.41 route-map poison-route-200 out

route-map SET-LOCAL-PREF-200 permit 10
set local-preference 200

route-map poison-route-200 permit 10
set as-path prepend 1111
!

end

[Richard Burts]

We have very limited information to work with and there may be things in your environment that we do not know and which would affect our advice. But on the surface this appears to be a fairly simple issue. The partial config that you provide shows ASR1004A with 2 EBGP peers and ASR1004B with a single EBGP peer. Is there an IBGP session between the ASR1004s that is not shown? The behavior suggests that there is an IBGP between them. We do not know anything about routing logic between the ASR1004s and the other routers in your network and there might be something in that which would affect how the routers forward traffic to the ISPs.

On ASR1004A you set local preference to 100 while ASR1004B sets local preference to 200. If AS 200 is advertising the same prefixes to both of your ASR1004 then you have established policy that you prefer to use ASR1004B with ASR1004A as a backup.

[Giuseppe Larosa]

Hello @Rob Sciarrino ,

on router ASR 1000 A I see :

>> route-map poison-route-200 permit 10
set as-path prepend 1111 1111

on router ASR 1000 B I see

>> route-map poison-route-200 permit 10
set as-path prepend 1111
!

if you are prepending your own AS twice out of ASR 1000A and once out of ASR 1000B it is natural that ASR 1000B will be preferred path for return traffic.

>> However, what has happened is more traffic is going out ASR-B and almost zero traffic on Provider #2 on ASR-A

I apologize you are meaning in the output direction to the internet ?

There is an iBGP session between the two devices  ASR 1000A , ASR 1000B?

I agree with @Richard Burts we have not enough information to be of help .

Hope to help

Giuseppe

[Rob Sciarrino]

@Giuseppe Larosa @Richard Burts  - Yes, the output towards the internet (my customer's download).  The AS Prepend behavior is what I expected, but not the traffic towards the internet.   There is an iBGP session between ASR1004A AND ASR1004B. I may have redacted too much information, so I've added some more. 

Thanks in advance for your input

ASR1004A

!
interface Loopback0
ip address 192.168.1.199 255.255.255.255

router bgp 1111
bgp router-id 192.168.1.199
bgp always-compare-med
bgp cluster-id 1111
bgp log-neighbor-changes
bgp deterministic-med
bgp bestpath compare-routerid
redistribute connected
redistribute static
neighbor 192.168.1.200 remote-as 1111
neighbor 192.168.1.200 update-source Loopback0
neighbor 192.168.1.200 next-hop-self
neighbor 172.16.0.101 remote-as 100
neighbor 172.16.0.101 next-hop-self
neighbor 172.16.0.101 send-community
neighbor 172.16.0.101 soft-reconfiguration inbound
neighbor 172.16.0.101 route-map SET-LOCAL-PREF-75 in
neighbor 10.0.200.9 remote-as 200
neighbor 10.0.200.9 version 4
neighbor 10.0.200.9 soft-reconfiguration inbound
neighbor 10.0.200.9 route-map SET-LOCAL-PREF-100 in
neighbor 10.0.200.9 route-map poison-route-200 out

route-map SET-LOCAL-PREF-100 permit 10
set local-preference 100
!
route-map SET-LOCAL-PREF-75 permit 10
set local-preference 75

route-map poison-route-200 permit 10

ASR1004B 

interface Loopback0
ip address 192.168.1.200 255.255.255.255
!
!
router bgp 1111
bgp router-id 192.168.1.200
bgp always-compare-med
bgp cluster-id 1111
bgp log-neighbor-changes
bgp deterministic-med
bgp bestpath compare-routerid
redistribute connected
redistribute static
neighbor 192.168.1.199 remote-as 1111
neighbor 192.168.1.199 update-source Loopback0
neighbor 192.168.1.199 next-hop-self
neighbor 10.0.100.41 remote-as 200
neighbor 10.0.100.41 version 4
neighbor 10.0.100.41 soft-reconfiguration inbound
neighbor 10.0.100.41 route-map SET-LOCAL-PREF-200 in
neighbor 10.0.100.41 route-map poison-route-200 out

route-map SET-LOCAL-PREF-200 permit 10
set local-preference 200

route-map poison-route-200 permit 10
match ip address prefix-list AS1111
set as-path prepend 1111
!

[Giuseppe Larosa · Accepted solution]

Hello @Rob Sciarrino ,

I think that @Richard Burts  has explained the behaviour towards the internet :

the higher value of local preference set on ASR 1004 B 200 makes its routes received from Provider 2 more preferred then those received by ASR 1000 A from same provider.

the iBGP session makes the local preference attribute to be sent to the other border router.

To see this take a prefix that has the best path  via provider 2 you should see:

on ISR 1000A two paths one eBGP and one iBGP with iBGP selected as best for higher local preference

on ISR 1000B you should see only one eBGP path and selected as best.

This will happen for all prefixes advertised by Provider 2 that are better then those advertised by Provider 1.

Hope to help

Giuseppe

[Rob Sciarrino]

@Giuseppe Larosa @Richard Burts - One last question, if I set the local pref to the same on Provider 2 on both ASRs and keep Provider one a "lower" local pref", then the eBGP algorithm and rules will apply for best route selection on Provider 2? I'm essentially trying to generate more traffic over both Provider 2 links.  

Thanks for the input! 

Rob 

[Richard Burts · Accepted solution]

Rob

If what you want to accomplish is outbound traffic preferring provider2 and provider1 acting mostly as backup/failover, then the same (higher) local preference on both provider2 BGP sessions and a lower local preference for provider1 should work. 

I am a little concerned about what it would take to share traffic on both provider 2 links. We do not know much about that provider and what they advertise to you. But if they advertise the same prefixes with the same parameters on both connections (as seems likely that they would) then the BGP route choice algorithm tie breaker would likely prefer the same peer for all of the prefixes. So not much sharing.

I have read about (but not used) a strategy for load sharing in which you have 2 different local preferences for provider2 (and both of them higher than the preference for provider1). You use prefix lists (or some other selection mechanism) to separate the prefixes advertised to you into 2 groups (perhaps classA and classB in one group and classC in another group). In the BGP session for provider2 on router1 you assign better local preference for classA/B and the lesser local preference for classC and on router2 you assign the lesser local preference for classA/B and the better local preference  for classC.

I have a concern about the approach that I just suggested. It may very well produce asymmetric paths. Your logic may send a packet to a destination in classA out router1 but when the response comes back it might come through router2. Perhaps in your environment that would not be a problem. But for a lot of firewall (and other security policy implementations) that is a problem.

[Rob Sciarrino]

Rick,

That was my thoughts as well, what would happen if the advertisements came back the same.  I'll mess around with the local preference and see how the routes behave. 

Thanks!

Rob 

[Richard Burts]

I have been thinking about this and have a few additional questions and suggestions.

We do not know what each provider is advertising to you. Is it just a default route, or is it a default route and selected other prefixes, or is it the complete Internet BGP table? But if it is just a default route then there is an interesting possibility that you can consider. You might set up Policy Based Routing such that parts of your network traffic toward the Internet is forwarded to ASR1004A and other parts of your network traffic toward the Internet is forwarded to ASR1004B.If both routers have the same local preference for provider 2 then each router would prefer to use the default route that it learned. But if either router is not receiving the advertised default from provider 2 then the provider 2 route leaned by the other router would be used and traffic is still forwarded to provider 2. If neither of your routers are learning the default route from provider 2 then the default route learned from provider 1 on ASR1004A would be used.

Am I correct in assuming that the outside interfaces of your routers have IP addresses that are in Public address blocks owned by the respective provider? And am I correct in assuming that each of your routers is doing address translation for outbound traffic using the Public IP address of the appropriate outbound interface? If so then this would seem to help with the possible issue of asymmetric traffic. If the packet to the Internet goes out with source address of the interface on ASR1004A then the provider should know which router this is and forward response traffic back to the correct router.