BGP with two ISPs and private AS

[Dmitry Podtikhov]

Hi

Assume I have two sites, each have two routers and two ISPs providing MPLS VPN service to me. Site routers use OSPF two exchange routes. ISP offer BGP as PE-CE routing protocol with the use of private AS numbers.

So here are my thoughts about the whole setup:

Each ISP will be in its own private AS, I will bring up EBGP session from each of my router to the appropriate ISP.

I will announce site's prefixes with several prepends to the secondary ISP so that the route priority was chosen according to the AS-PATH attribute. There is no problem with the sites that have one router for two ISPs, but there is some questions about dual router setup.

The problem is that EBGP routes recieved from other site will be chosen as the best on each router because of the administrative distance of 20, regardless of the protocol between this two site routers.

But I want my traffic to be routed only via ISP1 as the main ISP. There is a possibility that packets from the site will pass through the router with the secondary ISP and will go to the other site via the route recieved from secondary ISP because of the administrative distance of 20.

My thought is to set different private AS per each site router so that distance was equal and routes were chosen according to the AS-PATH.

I've set up a lab in dynamips - all is working, but maybe there is more effective way to reach my goal (e.g. IBGP between site router with changed protocol distance or something else) ?

[Peter Paluch]

Hello,

I have some comments regarding your suggested configuration.

Each ISP will be in its own private AS

You may do that in your lab but in real life, the ISP would have a public official AS number, not a private ASN. I suggest assigning a public ASN to your ISP routers.

The problem is that EBGP routes recieved from other site will be 
chosen as the best on each router because of the administrative distance
 of 20, regardless of the protocol between this two site routers.
But I want my traffic to be routed only via ISP1 as the main ISP

Then modify the local preference attribute of the routes received from ISP1 on the R1 and R3 to be higher than the local attribute of routes received on R2 and R4 from ISP2. The local preference is the second tiebreaker in the Cisco BGP bestpath algorithm (the first is the proprietary local Weight attribute) and has preference over any other attributes carried within BGP-learned routes.

You are probably thinking that running iBGP between your site routers will result in all internally-learned BGP routes to have the AD of 200 while the eBGP routes learned from ISP will have the AD of 20, and thus they will be preferred. This is not a correct assumption. The AD is only the result of the BGP bestpath algorithm, not its predecessor or a part of it. If R2 knows about a network X both from R1 and ISP2, it has to select which one is better. Because of BGP bestpath algorithm, if the local preference of this network via R1 is higher than the local preference via ISP2, the R2 will use the path via R1. And because this is an iBGP-learned route, it will be assigned the AD of 200 when it is inserted into the routing table of R2.

My thought is to set different private AS per each site router so that 
distance was equal and routes were chosen according to the AS-PATH.

No, that would be a very incorrect thing to do. A customer is a single entity and thus has a single AS number, even if a private one. Your task can be easily solved with ordinary BGP means.

Best regards,

Peter

[Dmitry Podtikhov]

Thanks for your explanation, Peter, that was my misunderstanding how the protocol works.

I've only one question - my ISPs give us MPLS VPNs interconnecting several sites together. My sites are connected to appropriate VRF instances on the PE routers, are there any problems with setting BGP as the CE-PE routing protocol with the use of private ASN in this case?

[Peter Paluch]

Hello,

Running a BGP for PE-CE communication is fine by itself, and also using the private numbers alone does not present any problem. What can be a problem is the fact that if your offices interconnected by the MPLS VPN use the same AS number then they would reject each other's networks because they would see their own AS in the AS_PATH attribute coming from the PE router and consider the advertisement looped. Note that this would happen with any AS numbers, private or public.

Solving this problem is actually your VPN provider's responsibility. He can either remove the private AS numbers from the AS_PATH attribute whenever he advertises the networks to your CE routers (using the neighbor remove-private-as command), or he can rewrite them to his own AS number (using the neighbor as-override command). This configuration would be performed on the PE routers. Should your provider fail to take care of this, you can also solve it on your CE routers by configuring them with the neighbor allowas-in command but that should be used only as a last resort.

So, the bottom line is - there is no problem in using the BGP as the PE-CE protocol if it is deployed correctly, and it is quite easy to deploy it correctly in your case.

Best regards,

Peter

[Dmitry Podtikhov]

Thank you for your replies, Peter.

You've described the situation when all my sites are in one  private AS, with EBGP links to my ISPs. But I still want to use separate ASN per site, because each site is a separate large object with several routers behind those BGP speakers.  Are there any caveats with this multi AS setup? Why not use several private ASN, there are plenty of them, and no need to use allowed-in or ask ISP to do something. Another reason is I want to have an opportunity to filter routes with the use of this private ASN. Isn't it more flexible setup?

[Peter Paluch]

Hello,

If you intend to use different private AS numbers then you won't have the problems I have described earlier. Technically, the private AS numbers are as good as any other AS numbers and if you do not plan to advertise the networks onto public internet, they will work just as neatly as any other numbers. I do not see any problems in using the private AS numbers the way you intend to. You should actually ask your provider to not remove the private AS numbers when advertising them to your CE routers if you want to perform filtering using your private ASNs.

Best regards,

Peter

[Dmitry Podtikhov]

Hi, Peter

I've got one more little question just for better understanding.

Assume I have two IBGP peers on one site, each of them have eBGP connection to appropriate ISP. Updates recieved from the main ISP are set with local preference 200 and advertised to IBGP peer. Each ISP send me the same prefix, main ISP with no prepend, and secondary ISP with several prepends of other site AS number.

#sh ip bgp on the router with secondary ISP connection reveal the whole picture - i can see routes learned via iBGP and eBGP with appropriate local preference and prepends

#sh ip bgp on the router with main ISP connection reveal only routes learned via eBGP peer and no routes from iBGP peer (so I have only half of the whole picture).

Further investigation revealed that for a short period of time router with secondary ISP connection advertise the prefixes it recieved from secondary ISP via iBGP to the main router (and i can see the routes in #sh ip bgp) and then just stop advertising. The whole setup is working fine, traffic is correctly routed in the case of a failure, so I think that's normal behaviour but I still want to understand the mechanism.

[Mahesh Gohil]

Hello,

It is bgp behavior..It advt. only the best route to neighbor

Router never send a route back to the same BGP session upon which it is received (split-horizon)

On contrary, when it selects a neighbor as the best next-hop (From your backup router toward primary router ),

the router make sure that neighbor is not pointing back to the local (your backup router sends widraw message to primary router)

router...This is accomplished with withrawl of route (poisioning)

This happens every scan interval (60 seconds)...i am not sure here

Your Backup router install both route because best is coming from primary router and not the ISP one.

Hope this is helpful

Regards

Mahesh

[Peter Paluch]

Hello Mahesh,

Nice reply Can I discuss one particular issue with you? You wrote:

Router never send a route back to the same BGP session upon which it is received (split-horizon)

It does seem so but I could not find any support for this in RFC 4271. Actually, the entire RFC does not even use the term "split horizon" and also, when reading it, there is no general explicit statement similar to split horizon. In my opinion, using the term "split horizon" with respect to the BGP is not correct because it is unfounded.

What is true is that:

• A route learned via iBGP must not be readvertised via iBGP (with the exception of route reflectors), thus also not back to where it originally came from. This is effectively a split horizon, valid for iBGP sessions.
• A route learned via eBGP may be advertised back (the RFC does not prohibit it!) but it would be dropped at the receiver because the AS_PATH would already contain the peer's AS

I have made a quick debug on C2691 IOS and while it does indeed seem that an eBGP-learned route is not advertised back, it is not mandated by RFC but it seems to be more of an optimization of the basic BGP implemented in IOS.

What are your thoughts on this? I find myself a little curious about this and would love to hear other opinions.

Best regards,

Peter

[Mahesh Gohil]

Hello Peter,

Thanks for your explanation.

A route learned via iBGP must not be readvertised via iBGP (with the exception of route reflectors), thus also not back to where it originally came from. This is effectively a split horizon, valid for iBGP sessions. 
A route learned via eBGP may be advertised back (the RFC does not prohibit it!) but it would be dropped at the receiver because the AS_PATH would already contain the peer's AS

Your first statement matches with the way split horizon works (Little difference is ibgp will not advt. to all other ibgp peer irrespective of whether it is learned from that peer or not).

I know that RFC do not use this word but I read it from cisco book "Configuring BGP in Ciso Routers Volume 1 version 3.2"

where they use split horizon..it may be because they are trying to make readers more comfortable with other routing protocol.

I do not agree here with your second statement... it may be advertised back to originator if next-hop is changes ..I am not sure..I will perform some

test in lab and come up with concrete ststement.

Regards

Mahesh

[Peter Paluch]

Hello Mahesh,

Thanks you for your time. Let me know if you find anything interesting.

I do not agree here with your second statement... it may be advertised back to originator if next-hop is changes

I am afraid I do not entirely understand this. Can you elaborate a little further?

Best regards,

Peter

[milan.kulik]

Hi Mahesh,

maybe the Cisco BGP split-horizon term comes from RFC 1771:

"A BGP speaker must never advertise an address of a peer to that peer as a NEXT_HOP, for a route that the speaker is originating. A BGP speaker must never install a route with itself as the next hop."

I've discussed that already in another thread here, see

https://supportforums.cisco.com/message/3052977#3052977

BR,

Milan

[Peter Paluch]

Hello Milan,

Thank you for joining in!

Let's use the RFC 4271 as the most current source about BGP, shall we? It has a similar statement in the section 5.1.3:

   A route originated by a BGP speaker SHALL NOT be advertised to a peer
   using an address of that peer as NEXT_HOP.  A BGP speaker SHALL NOT
   install a route with itself as the next hop.

Frankly, after reading it more times over and over, I have a strong feeling this is not related to our debate about "BGP split-horizon". In my understanding, what this statement says is that you as a BGP speaker shall not inject (i.e. originate) a route into your BGP database, set its NEXT_HOP attribute to an IP address of some its neighbor and advertise such a route to that neighbor. This appears to me to be a call for certain sanity in originating BGP routes - it is not in most cases not appropriate to originate routes as if on behalf on some of our neighbors and then subsequently to "force" those routes upon that neighbor. Also, this statement does not appear to apply to routes that are not originated but only forwarded within a BGP peering.

Mahesh suggested that a route learned via an eBGP BGP peering will not be advertised back through that same peering, and using a simple lab experiment, it indeed seems so. However, the RFC does not appear to mandate such behavior. It appears to me that this is an optimization of the basic BGP principle. Certainly, it does not make sense to learn, say, 50 MB of routes through an eBGP peering and then advertise all of them back - they would be so or so dropped because of the AS_PATH check. I see a great deal of logic there but what I am saying is that this behavior is not mandated by the RFC, and the RFC does not contain any explicit references to a "split horizon". The iBGP behavior is actually a superset of split horizon (it is even stronger) but no such behavior for eBGP sessions seems to be imposed.

Any thoughts?

Best regards,

Peter

[milan.kulik]

Hi Peter,

well, as none of us has English as his native language, it's difficult to dispute what some English sentence means in deep :-)

But you are probably right, the RFC does not contain any "BGP split horizon".

I tried to google this term and all links were discussing the iBGP behaviour.

But another interesting fact:

I found on one of my productive routers it IS ADVERTISING some prefixes back to the neighbors from which they were received!!

I see most of them having Origin incomplete together with multipath, but some of them are even having Origin IGP.

So far I really don't know what's the rule.

BR,

Milan

[Peter Paluch]

Hello Milan,

Thanks for your reply. True, neither me nor you are native English speakers but I do not think we should distrust our current English skills. I think we both have our English good enough. The formulation in the RFC, while a little convoluted, is decipherable after all. It is also nice to know that some of your routers actually advertise learned routes back via eBGP where they came from. That would indeed corroborate my original idea here that any optimization of eBGP in this particular regard is actually above the scope of the BGP RFC.

In any case, we could ask Dr. Rekhter for his explanation on this I tend to believe he would answer

Best regards,

Peter