BGP inbound: 2 ISPs 2 Routers: Which ISP in?

noemi.berry · ‎05-08-2015

We have a BGP setup with 2 ISPs and 2 routers, and are announcing one portable prefix (we own it).

Our existing BGP config is very bare-bones and does no inbound tuning -- so how do can you tell why one ISP is chosen to route inbound to our prefix? It's easy to tell *which* ISP with traceroute, but I don't know how to tell *why* it's going in via Zayo instead of XO.

--- We'll fix this with as-path prepend in outbound route-maps -- just asking how the inbound path is selected by our upstream without that? Am missing something?

Showing advertised routes, there's no difference (sanitized output, our "prefix" is 9.9.9.0):

ROUTER1-XO# show ip bgp neigh 1.1.1.1 advertised-routes

BGP table version is 435178080, local router ID is 1.1.1.2

Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,

r RIB-failure, S Stale, m multipath, b backup-path, x best-external, f RT-Filter

Origin codes: i - IGP, e - EGP, ? - incomplete

Network Next Hop Metric LocPrf Weight Path

*> 9.9.9.0 0.0.0.0 0 32768 i

ROUTER2-ZAYO# show ip bgp neigh 2.2.2.1 advertised-routes

BGP table version is 574543284, local router ID is 2.2.2.2

Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,

r RIB-failure, S Stale, m multipath, b backup-path, x best-external, f RT-Filter

Origin codes: i - IGP, e - EGP, ? - incomplete

Network Next Hop Metric LocPrf Weight Path

*> 9.9.9.0 0.0.0.0 0 32768 i

Showing our prefix in our BGP tables, there's a minor difference, but I believe this only affects outbound.

ROUTER1-XO#show ip bgp | in 9.9.9.0

Network Next Hop Metric LocPrf Weight Path

*> 9.9.9.0 0.0.0.0 0 32768 i

ROUTER2-ZAYO#show ip bgp | in 9.9.9.0

Network Next Hop Metric LocPrf Weight Path

* i9.9.9.0 10.255.255.1 0 100 0 i

Sanitized configs -- REALLY bare-bones, will be updating with route-maps and as-path prepend in a route-map out:

ROUTER1-XO BGP config:

router bgp 66666

bgp log-neighbor-changes

bgp deterministic-med

network 9.9.9.0

neighbor 10.255.255.2 remote-as 66666

neighbor 10.255.255.2 next-hop-self

neighbor 10.255.255.2 soft-reconfiguration inbound

neighbor 1.1.1.1 remote-as 1111

neighbor 1.1.1.1 send-community

neighbor 1.1.1.1 soft-reconfiguration inbound

neighbor 1.1.1.1 filter-list 1 out

!

ip as-path access-list 1 permit ^$

ip as-path access-list 1 deny .*

ROUTER2-ZAYO BGP config:

router bgp 66666

bgp log-neighbor-changes

bgp deterministic-med

network 9.9.9.0

neighbor 10.255.255.1 remote-as 66666

neighbor 10.255.255.1 next-hop-self

neighbor 10.255.255.1 soft-reconfiguration inbound

neighbor 2.2.2.1 remote-as 2222

neighbor 2.2.2.1 send-community

neighbor 2.2.2.1 soft-reconfiguration inbound

neighbor 2.2.2.1 filter-list 1 out

!

ip as-path access-list 1 permit ^$

ip as-path access-list 1 deny .*

So why does inbound traffic go to Zayo?

We'll fix it with this route-map, below. The route-map will refer to the existing ip as-path access-list 1 (to allow routes that originate from us) . Confirming: The as-path prepend will take place after that match, so won't affect it, right?

route-map out-to-as2222 permit 10

match ip address prefix-list bgp-outbound

match as-path 1 <----- this won't be messed up by....

set as-path prepend 66666 66666 <------- ....this...?

That route-map will be applied to the ZAYO peer to dis-favor it for inbound routing:

neighbor 2.2.2.1 route-map out-to-as2222 out

Answers, clues, comments?

thanks!

xthuijs · ‎05-09-2015

upstream traffic you can control easily. in your config, your zayo seems to have a iBGP path also, but since we always prefer external over internal we'd generally want to route prefixes over an eBGP link then iBGP. You can modify that behavior by selecting eibgp max paths to also allow iBGP paths to be part of mpath selection.

downstream traffic, aka how we pull in traffic for our prefix in this case 9.9.9.0/24 is largely dependent on the general upstream network design. if inbound traffic comes in primarily via ZAYO then this is only caused by the fact that its as-path is likely shorter and therefore more preferred for many destinations on the internet.

You could somewhat affect that by advertising a longer path towards one ISP vs another to make inbound traffic more preferred on the shorter path, but pre-pending can be stripped also by your provider.

Unless you make peerng agreements with either of them it is just up to internet routing where you have little control or basically comes down to BGP path selection which on inter-AS generally comes down to AS-length as the decision maker....

for upstream as mentioned you have a lot more control how you want to exit out and which SP to use to "get there". an iBGP peering is necessary between the two border routers and then we can use some tricks to prefer one SP over the other with some simple routing tricks, either by advertising a default into the IGP (based on the availability of internet routes for instances, aka conditional advertisement of the default route).

there is a lot more to it obviously, but hopefully this gets you started to some extent already.

cheers!

xander

----

Xander Thuijs CCIS #6775

Principal Engineer IOS-XR/XR-routing

noemi.berry · ‎05-13-2015

Thanks so much for that answer, that helps a lot. I'm realizing that the first problem is defining what we *want*, and how. I've been reading that communities are another way of addressing inbound traffic, depending on what the ISPs use.

This is very interesting though:

>>for upstream as mentioned you have a lot more control how you want to exit out and which SP to use to "get there". an iBGP peering is necessary between the two border routers and then we can use some tricks to prefer one SP over the other with some simple routing tricks, either by advertising a default into the IGP (based on the availability of internet routes for instances, aka conditional advertisement of the default route).

>>

We do have iBGP between two border routers, which are still carrying full tables (no default), and no criteria for outbound path selection. Like for inbound, we first need to decide what we want -- preferring one ISP over another might not be a good plan anyway.

Any references on how to do conditional advertisements of a default route, or what conditions are commonly used?

thanks again.

xthuijs · ‎05-13-2015

Hi Noemi,

The way to control a default advertisement into your IGP can be done with RPL/route policy language. Some detail here: https://supportforums.cisco.com/document/88676/asr9000xr-understanding-and-using-rpl-route-policy-language

You can either use a full BGP mesh to have the full tables everywhere, or alternatively you can use an iBGP peering between your border routers and let "them figure out" which the shortest path to a destination is.

Then if some traffic arrives at border router-1 from your network you could haul it over to border-router-2 for exit out the internet. This can be useful if one provider charges differently based on upstream traffic for instnace or you just want to get to the shortest (AS) path to the destination.

For inbound, yeah you may want to identify what it is that you want to achieve and based on that we can opt for some ideas there. Generally, me personally wouldn't care how it "gets to me" as long as it does :) It also depends on your subnets, how large they are and if you can or have them summarized and whether the prefix(es) you own are geographically subnetted.

For instance, you probably can't advertise shorter then /23. so if you only have ONE /23, you WILL have to announce that by both.

If you have a /22 you can split that in two /23's and announce each of them differently ASSUMING that each /23 is geo close to that border router (otherwise it serves little purpose, you'll be pulling it in one way and hauling over your network to the other side :).

So one and other depends on:

- subnets/prefixes you own

- the location of those subnets of the prefixes

- whether those subnets are geo dispersed

- your peering agreement with both SP's and whether one is more cost effective then the other.

cheers!

xander

noemi.berry · ‎05-20-2015

At the moment our setup is so simple it regrettably doesn't need this level of tuning -- someday though. We have 2 separate ISPs per location, and we announce our own portable /24 and one /24 assigned from one of the ISPs, to both ISPs. We have customers reaching into our data center on those public IPs, so we do need to announce for the inbound traffic.

For outbound traffic, we're actually carrying full tables and that's definitely overkill, but I'm not sure I want to go to a single default route (conditional or not). Rather, I'd like to reduce the routing table size with prefix filtering by length.

To that end, I've asked this question as well:

https://supportforums.cisco.com/discussion/12512756/using-route-map-bgp-inbound-prefix-filtering-length

(This link you sent is for route-policy on IOS-XR, interesting but we're running plain old IOS.)

thanks much!

xthuijs · ‎05-20-2015

ok for IOS this is as simple, you need to define an ip prefix-set a-la this:

NPE-G1(config)#ip prefix-list test permit 1.1.1.1/8 le ?
<1-32> Maximum prefix length

and pull that in your route-map and that route-map needs to be tied to the neighbor configuration under router bgp.

to be honest however, I think filtering on prefix length is somewhat dangerous, you probably are best off advertising two defaults in this case and let the border routers figure out how to route it further.

cheers

xander