Multi-homed BGP, two ISPs, two routers: Should routes be unique?

[noemi.berry]

Basic setup: Two ISPs, two routers, iBGP between the two routers, internal OSPF.  BGP config is pretty standard, no special tweaking.  Each router carries about 19,000 routes.

Most routes go via one ISP or the other.  Some routes, however, go to the ISP that the router is attached to -- so, if you're on Router1 you'll go via ISP1, or on Router2, then you'll go via ISP2.

For example, Router1's ISP is 1.1.1.1; Router2's ISP is 2.2.2.2.   iBGP peering between Router1 (10.0.0.1) and Router2 (10.0.0.2).  The routing table includes:

Router1:

1.0.128.0/17        BGP    255  0            10.0.0.2

1.0.128.0/18        BGP    255  0            10.0.0.2

1.0.160.0/19        BGP    255  0            1.1.1.1

Router2:

1.0.128.0/17        BGP    255  0            2.2.2.2

1.0.128.0/18        BGP    255  0            2.2.2.2

1.0.160.0/19        BGP    255  0            2.2.2.2

The first two routes point to Router2, but the last one points to Router1 on Router1, and Router2 on Router2. (I'm aware it's a subnet of the other two).   There are many routes like this, with two exit points from the network, but many point to just one exit point from the network.

The question is: This is normal and acceptable, right?  Could someone explain the BGP behind this?  Why do some routes show up with one exit point from the network and some point to two?

[Jon Marshall · Accepted solution]

It all depends on the attributes that are with the BGP route advertisement.

You need to do a "sh ip bgp <network>" command on each router to see what is going on in terms of routes received and why one is preferred over the other or why both are being used one for each router.

So with the first two networks router 1 has picked router 2 as the path so obviously something before EBGP vs IBGP routes in the BGP best path selection has been used.

With the last network it looks like EBGP has been favoured over IBGP on each router so that would mean all other attributes used in the path selection that come before EBGP vs IBGP were equal.

Jon

[noemi.berry]

OK, I think I see what's going on here, which is: Nothing.  The logic is the same for all routes.

Each router learns the prefix (e.g. 1.0.160.0/19) from its own ISP.

Each router announces the prefix to its iBGP peers.

Each router performs a path selection criteria on all its own entries for that prefix.

Each router inserts the best path into its IP routing table.

That "best path" might be my ISP, or it might be via my iBGP peer,.  If I have two paths to one prefix, then I do the famous BGP path selection process and make a local decision.  The fact that Router1 decided that ISP1 was the best path had nothing to do with the fact that Router2 decided that ISP2 was its own best path.

That was actually a really dumb question in retrospect, but they have to be asked! :)  Now, just need to figure out which attribute values led to those decisions.

For anyone well-versed in BGP path selection, here's what Router1 used to decide that its iBGP peer had the better path for prefix 1.0.128.0/17:

Router1:

 BGP routing table entry information of 1.0.128.0/17:

 From            : 1.1.1.1

 Relay Nexthop   : 0.0.0.0

 Original nexthop: 1.1.1.1

 Community       : <6461:5997>

 AS-path         : 100 6461 174 38040 9737

 Origin          : igp

 Attribute value : MED 0, pref-val 0, pre 255

 State           : valid, external,

 This route is an atomic-aggregated route

 Aggregator      : AS 9737, Aggregator ID: 203.113.12.254

 Not advertised to any peer yet

 BGP routing table entry information of 1.0.128.0/17:

 From            : 10.0.0.2

 Relay Nexthop   : 10.0.253.2

 Original nexthop: 10.0.0.2

 Community       : <3356:3>, <3356:22>, <3356:100>, <3356:123>, <3356:575>, <3356:2011>

 AS-path         : 100 3356 38040 9737

 Origin          : igp

 Attribute value : MED 0, localpref 100, pref-val 0, pre 255

 State           : valid, internal, best,

 This route is an atomic-aggregated route

 Aggregator      : AS 9737, Aggregator ID: 203.113.12.254

 Not advertised to any peer yet

Picked the 2nd path (via its iBGP peer), I'm guessing because the AS-Path was shorter.

thanks!

[Jon Marshall]

Yes, from the looks of it the AS path is shorter so it chose router 2.

If you did the same for the route where both routers use their own ISP then I expect it to show all things equal including AS path length and then it gets down to EBGP vs IBGP which is why they choose the route from their own ISP.

Not a dumb question at all, can't count the number of times I've looked at something and wondered what was going on :-)

Jon

[noemi.berry]

>If you did the same for the route where both routers use their own ISP then I expect it to show all things equal including AS path length and then it gets down to EBGP vs IBGP which is why they choose the route from their own ISP.

That in fact is exactly what shows .... but both entries say "Origin: igp"?

Router1:

 BGP routing table entry information of 1.0.160.0/19:

 From            : 1.1.1.1

 Relay Nexthop   : 0.0.0.0

 Original nexthop: 1.1.1.1

 Community       : <6461:5997>

 AS-path         : 100 6461 2914 38040 9737

 Origin          : igp

 Attribute value : MED 0, pref-val 0, pre 255

 State           : valid, external, best,               <------------- picked my ISP

 This route is an atomic-aggregated route

 Aggregator      : AS 9737, Aggregator ID: 203.113.12.254

 Advertised to such 1 peer(s):

    192.168.1.2

 BGP routing table entry information of 1.0.160.0/19:

 From            : 10.0.0.2

 Relay Nexthop   : 10.0.253.2

 Original nexthop: 10.0.0.2

 Community       : <2914:410>, <2914:1405>, <2914:2406>, <2914:3400>, <3356:2>, <3356:22>, <3356:86>, <3356:503>, <3356:666>, <3356:2067>

 AS-path         : 100 3356 2914 38040 9737

 Origin          : igp

 Attribute value : MED 0, localpref 100, pref-val 0, pre 255

 State           : valid, internal,

 This route is an atomic-aggregated route

 Aggregator      : AS 9737, Aggregator ID: 203.113.12.254

 Not advertised to any peer yet

[Jon Marshall]

The origin type is just another attribute and it comes after AS path but before EBGP vs IBGP.

It is basically where was the route was learnt from.

Is there a specific question about that you have ?

Jon

[noemi.berry]

No, just an inspecific one, like how to tell which entry came from EBGP v. IBGP....  OK that's implied by the peer (EBGP: different ASNs; IBGP: my ASN).  

Following on my doctored example:

bgp 10002

 peer 10.0.0.2 as-number 10002   <--- IBGP peer

 peer 1.1.1.1 as-number 1111     <--- EBGP peer

Prefixes learned from the EBGP peer are favored over the IBGP peer.  The "Origin: igp" is another attribute, further down the path selection logic (and it's easy to fooled by the similar letters 'i g p').

 All stuff you "know" -- until you see it!  Thanks again for the BGP-101 lesson.