Basic BGP question - eBGP & iBGP

[louis0001]

Hi,

we have an MPLS network which uses BGP as the routing protocol. All of our external sites are on this network and we have 3 central sites with each one having a router which connects to the MPLS network.

The central sites have core switches which interconnect the central sites. So basically, from the external sites, there is one way into each central site and pc's route to each one depending on what they are trying to achieve eg file server is at site 1, web server at site 2 etc.

In our core central sites, we use static routing which works but doesn't really give us any resilience and it's becoming a pain to manage.
So we're looking to change to a dynamic routing protocol.

So do we change to OSPF, EIGRP or try to use BGP?

All in all, there would be 6 routers involved. Two at each central site ie the core switch (connecting central sites together) and the MPLS router (to connect to external sites)

So, would I be correct in thinking, if I enabled iBGP on the core switches and the routers that connect to the external, the internal routes (currently static) would be advertised to the external clients? So 6 routers would be using iBGP to route to each other with 3 external routers would be using eBGP to route to the external networks.

So if central site 1 lost it's connection to the MPLS network, clients would then be able to reach that site via another central site? eg router through central site 2 to get to central site 1?

Or maybe just stick to OSPF/EIGRP for the central sites?

[Peter Paluch]

Hi Louis,

Can you perhaps draw a quick schematic of your network including the placement of the MPLS network, the central sites, and the external sites, plus the existing BGP deployment, and post it here? It would help us understand your network better before we can suggest any particular solution.

In general, though, running iBGP without an IGP protocol (RIP, EIGRP, OSPF, IS-IS) is cumbersome. iBGP peerings must be established between all routers, not just between directly connected neighbors, and these peerings are usually based on loopback addresses of individual routers. This means that every router has to know the loopback of every other router in the network, and has to know the path toward it before an iBGP session can come up. The reachability of the loopbacks is therefore the responsibility of an IGP.

Proper choice of IGP is always a contentious issue. With a relatively small network, though, any protocol would do. If all your devices are Cisco devices, EIGRP would be the way to go - it is fast to setup and reasonably easy to maintain. If you have a mixed vendor environment, OSPF is preferable, as EIGRP - even though recently published as an RFC - is not yet widely supported by other vendors.

Best regards,
Peter

[louis0001]

HI,

I've attached a diagram. Not much to see really. MPLS is routed using BGP. Each central site has a router that connects to the MPLS network and advertises it's local subnets via BGP.

Behind that, it is all static routing.

What I'm trying to achieve is if one of the interfaces goes down on R1, R2 or R3, traffic from the MPLS network can re-route itself via the other two links and find itself to all servers behind

eg if R1 goes down (which is the normal route for file servers, clients will be able to re-route via R2 or R3 to get to the file servers

The links behind the core switches are only used for replication and all traffic generally routes out into the MPLS from one site to come back into another which means that if one of the MPLS routers goes down, we can't route to it even though we have another route to it.

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages wha2tsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

You write if R1 fails you want to reroute via R2 or R3 but you also write links behind core switches are only used for replication.  So, can the links between your L3 switches be used or not?

Assuming they can be, each L3 switch and each router needs to "know" all the back end topology (best done via an IGP), and that topology dynamically advertised via BGP too.

[louis0001]

Hi Joseph,

currently they are only used for replication but we want to use them for redundancy also. So yes, we can use them. We have a fair amount of redundancy at each site eg ASA failover, core switches are stacked etc but this is the one area where we suffer due to the single incoming links. We know that if that link goes down, clients will lose connectivity to the servers they are trying to get to. So we are proposing re-routing them if this happens.

As it's an all Cisco environment, I was thinking of maybe going down the EIGRP route for the IGP. After that, I'm not sure how you would advertise the routes into BGP so that a client who would normal route via R1 for file services, would then route via R2 or R3 and then via the core switches to get to the file servers.

So any pointers would be really appreciated with this one.

[Peter Paluch · Accepted solution]

Hi Louis,

May I ask you about one additional detail?

In your last post, you wrote: "After that, I'm not sure how you would advertise the routes into BGP so that a client who would normal route via R1 for file services, would then route via R2 or R3 and then via the core switches to get to the file servers.", and in your original post, you wrote: "So basically, from the external sites, there is one way into each central site and pc's route to each one depending on what they are trying to achieve eg file server is at site 1, web server at site 2 etc."

What confuses me is the client routing as you have described it. You are suggesting that clients are doing some kind of routing choice depending on the site they want to reach. But in my opinion, the clients simply have a default route pointing into the MPLS cloud, and depending on whether they talk to 10.1.1.0/24, 10.1.2.0/24, 10.1.3.0/24, the MPLS cloud routes them through the corresponding router R1, R2, or R3. Am I correct?

Best regards,
Peter

[louis0001]

Oooops. Pressed "correct answer" instead of "reply". Can't see how to take the correct answer off now.

Yes you are right. Clients will point to the file servers via dns (they are at 10.1.1.50 & 10.1.1.51). Equally, they point to the web servers on 10.1.2.51 and so on. They do have a default route via another router (BGP) on the remote site where they are located.The MPLS cloud routes them appropriately.

So at one of the remote sites, the clients are simply looking for the 10.1.1.0/24 subnet which is advertised into BGP via R1. If R1 goes down, 10.1.1.0/24 simply drops out of the BGP routing table and the clients are unable to connect.

I'd like them to still route to 10.1.1.0/24 but via R2 or R3 which means that R2 or R3 has to advertise the 10.1.1.0/24 subnet.

I would imagine the first step is probably to get  rid of the static routing at the backend and make all routers aware of each others routes. I'm thinking of using EIGRP for this (unless anybody can suggest a better alternative?)

Once that is done, it's how to get R2 or R3 to know about the routes via BGP as the clients on the outside only route via this.

At the moment, with the static routes, if R1 goes down and if we could re-route to R2, clients will probably get to the files servers because the L3 switches have static routing in them and are aware of each other.

So I suppose its a two part project.
1. Get rid of the static routes (which we want to do)
2. Advertise the backend routes into each router so that id R1, R2 or R3 go down, clients are re-routed.

The diagram is slightly incorrect also as the the core switches are connected to each other so the one of the left also has a direct connection to the one of the right. (each core switch has an independent connection to the other core switches)

[Peter Paluch]

Hi Louis,

I think that you'd need to contact the site administrator to un-mark an answer as correct.

I am preparing a possible solution scenario for you, but one more question, hopefully the last one at this point: As your R1, R2, R3 peer with the MPLS VPN service provider using BGP, what autonomous system numbers (ASNs) are you using? Specifically, what are the ASNs of R1, R2, R3, and what are the ASNs of their neighbors they peer to?

Best regards,
Peter

[louis0001]

Hi Peter,

the ASN's are:
R1 = 65001
R2 = 65002
R3 = 65003

And all our other remote sites increment like this. Speaking with BT, they actually told me the local ASN doesn't matter (even if we have duplicates)

They all peer to 2882 (BT MPLS network) and basically we redistribute connected and static

[Peter Paluch · Accepted solution]

Hi Louis,

So this is my first suggestion - obviously, it might need lots and lots of polishes but at least something to start with.

1. Your routers R1, R2, R3 and your core switches S1, S2, S3 will be running EIGRP, exchanging the internal networks on the three central sites.
2. Routers R1, R2, R3 will redistribute all internal EIGRP networks into BGP. This way, each router will advertise to the MPLS cloud that it is capable of providing connectivity to each network at the three central sites. It would be beneficial to consult BT what BGP attributes they recommend to use so that we can manipulate the routing policies so that R1 will be preferred to reach networks at the file serves site, R2 will be preferred to reach networks at the web servers site, and R3 will be preferred to reach networks at the e-mail servers site. Even without BT's cooperation, we can use the AS_PATH prepending trick in BGP, but if they have a set of designated BGP attributes (perhaps a set of communities), using them would be preferable.
3. Routers R1, R2, R3 will be configured to accept only those routes through BGP from BT that do not have AS numbers 65001, 65002, or 65003 in their AS_PATH. In other words, routers R1, R2, R3 will never use the MPLS cloud to reach the networks on the three central sites. This is a deliberate decision on my part: None of the central sites will use the MPLS VPN as a backup to reach any other central site. While this may remove some of the redunancy, usually this kind of redundancy is not really intended, and it will simplify the routing loop prevention.
4. As a result of Step 3, routers R1, R2, R3 will learn only networks from remote sites through BGP. These routes will be redistributed from BGP into EIGRP on R1, R2 and R3, allowing each central site to know about the remote sites and also see the backup paths should the closest router become unavailable. At this point, we can also consider route summarization or injecting a conditional default route to prevent EIGRP from carrying too many networks in BGP.

Why would this work?

EIGRP running between R1, R2, R3, S1, S2, S3 will be responsible for the "inter-central" connectivity, and will be providing the list of all networks on all three central sites. These networks will be EIGRP-internal, with an administrative distance of 90.

R1, R2 and R3 will take up these EIGRP-internal routes and redistribute them into BGP. This way, the MPLS cloud will be aware that to reach any network on any of the three central sites, any router (R1, R2, or R3) can be used. This is where path attribute manipulation should come in, to make sure that the MPLS VPN cloud actually prefers the router that is closest to the respective destination network.

Without further configuration, some of R1, R2 and R3 would learn about the "inter-central" networks both from EIGRP and from BGP, but because you are running external BGP (eBGP) with your MPLS VPN, the eBGP-learned routes would be preferred thanks to their administrative distance of 20, as opposed to EIGRP distance of 90. Some routers would therefore prefer the connectivity through MPLS VPN as opposed to the internal "inter-central" connectivity. This is why I have suggested that R1, R2 and R3 are prevented from learning about their own internal, "inter-central" networks through BGP in the first place.

So R1, R2 and R3 will know about the "inter-central" networks only through EIGRP, and about the remote sites only through BGP. These BGP-learned networks can be then redistributed into EIGRP, possibly summarized so that EIGRP is not stressed with advertising hundreds of networks.

Of course, a BGP network redistributed into EIGRP on R1 will make it through EIGRP to R2 and R3. However, remember that this is a remote site network. Both R2 and R3 know this network from BGP already, and because the BGP administrative distance is 20, while the redistributed EIGRP network will have an administrative distance of 170, R2 and R3 will prefer the BGP-learned network whenever possible. This is one protection against redistributing a route back where it came from, possibly creating a routing loop. In addition, the redistribution from EIGRP into BGP on R1, R2 and R3 will be configured so that only internal EIGRP routes are redistributed, not external, providing further protection against routing loops.

In this scenario, if a link from R1 to the cloud breaks, the BGP session will go down, meaning that R1 stops learning any remote site networks through BGP and advertising any "inter-central" networks to the MPLS VPN. As a result, R1 will no longer advertise the remote sites through EIGRP (because the BGP session is down and R1 does not know about the remote networks through BGP anymore), so the central sites will converge through R2 and R3 to reach the remote networks. Conversely, because R1 no longer advertises the "inter-central" networks through BGP, the MPLS VPN will converge to the "inter-central" sites through R2 and R3. R1 itself will learn about remote sites through EIGRP.

If a link from R1 to S1 breaks, R1 will become unreachable as far as EIGRP is concerned, and so it will stop being considered as a possible path to remote sites. The rest of switches and routers will converge to another router, R2 or R3. R1 itself will lose connectivity to the "inter-central" networks (remember that it is prevented from learning about "inter-central" networks through BGP), but no one really cares because that makes R1 simply an isolated router with no end hosts behind it, and so it does not bother anyone. Because R1 has no connectivity to "inter-central" networks, it does not redistribute them from EIGRP into BGP, and so the MPLS VPN cloud will converge through another router to reach the "inter-central" networks.

This may take some time to digest - please feel welcome to ask further!

Best regards,
Peter

[louis0001]

Wow. What a reply and many, many thanks for taking the time to explain it.

I think the first thing I need to do is to get EIGRP up and running and then get rid of my static routes once I'm assured EIGRP is advertising all relevant internal routes.

The next stage would be the EIGRP/BGP advertisement. This bit I've never done before which is the bit I'm unsure about.

But what a great reply. Thank you.

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages wha2tsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

It's not unusual that MPLS vendors only support BGP to their cloud.  So to route across it, you might BGP peer with them or you might run an IGP (of your choice) across tunnels across the provider's MPLS cloud.  (An issue with tunnels, they might not support the same max MTU as the rest of your network.)

If you BGP peer with your MPLS vendor, you can run an IGP of your choice at each site or static route to your BGP router.  If you run IGP tunnels across the MPLS network, you can use the same IGP at your sites too.

Feature packages vary, but you might find more advanced IGPs and/or BGP require feature license upgrades to use.  You also might find, some smaller L3 switched might not even support advanced IGPs and/or BGP.