Design Question - Redundancy between 2 Routers (CEs)

[4everlearning]

Hello Guys,

I have a question regarding the link redundancy between 2 CEs, let's say we have the following scenario 

 CE1 _ __ _ _ _ _ _ _ _ __ _ _ _ _ _ _ _ _ _  10G link  running EBGP _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _  PE1

   _                                                                                                                                                                                         _ 

  One 10G link L2 tunk, and Three 1G links L3 between CEs                                                                                        _

   _                                                                                                                                                                                          _   

  CE2 _ __ _ _ _ _ _ _ _ __ _ _ _ _ _ _ _ _ _  10G link  running EBGP_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _  PE2

2 CEs connected to 2 PEs with 10G for each PE-CE connectivity using EBGP (L3 connectivity). now between the CEs there is One 10G Link used as L2 trunk and Three 1G Links used as L3 Trunk. 

Now let's say the link between PE1-CE1 goes down, all the traffic will be shifted to the link between PE2-CE2, now when CE2 receives the traffic from EBGP, will it use the L2 trunk or L3 trunk to send the traffic to CE1 using the L3 links through iBGP ?

Just to give an example, let's say there is an X device and Y device behind the CEs:

The X device is connected to the CEs using L2 connectivity (Multiple VRRP groups with CE1 being the Active for 50% and CE2 being the activity for the other 50%). Also we have the Y device which is connected to the CEs using L3 connectivity (OSPF), and then the CEs connect to the PEs using EBGP.

so now let's assume the ebgp link between CE1-PE1 goes, down, the traffic will be shifted and sent through CE2-PE2, now when CE2 receives the traffic, would CE2 send the traffic, in which it is acting as the Master for, directly out to internal network and only the rest of the traffic (the 50% where CE1 is the master for) will be sent through the ibgp link ?

or would the ibgp link carry all the traffic to CE1 anyway, and then CE2 will send it's concerned traffic to internal network and CE1 will send it's concerned traffic to internal network. 

We are using Cisco asr 1006 as CEs and we are connected to the same provider 2 CEs to 2 PEs. 

I would really appreciate if someone gives be a better idea from design point of view on how traffic moves between the links between CEs in case there is a failure between one of the CE links to external devices (PEs, or X and Y devices in our example).

Many thanks for the help. 

[Jon Marshall]

In case CE1-PE1 fails, the traffic will be sent through CE2-PE2 link and then redistributed to OSPF on CE2, and CE2 should be able to send them to internal network or to CE1 through OSPF (not iBGP) since OSPF routes have better AD than internal iBGP routes. 

I think the above is talking about inbound traffic but we are concerned with outbound traffic.

Inbound as discussed CE2 should send traffic direct although the fact you are running OSPF between the CEs (which i didn't realise before) may influence the path taken depending on how the internal topology is setup. It all depends on how the internal L3 devices running OSPF are connected to the CEs.

In terms of outbound traffic there are quite a few variables so it may be better to cover those when you have access to the equipment rather than speculate.

But as a general point if the CEs are peering with each other using IBGP and OSPF then as long as the CEs redistribute the EBGP learned routes (from their respective PES) into OSPF i can't see at the moment what you would gain by also redistributing IBGP routes into OSPF.

Jon

[4everlearning]

Hi Guys,

I checked the OSPF configuration and it is basically for the internal communication so there is no redistribution between BGP and OSPF, The reason of using OSPF is to control how traffic float to internal network. 

Some of the internal network devices have different entities that configured with ip addresses and some entities are only connected to CE1 while others are only connected to CE2, even though those devices can route traffic using internal switches, the designer of the network preferred to run OSPF and let the CEs choose the through which CE each entity have a better route, and do the routing on a CE level. 

So some of the interfaces going to internal devices are configured directly with OSPF, while others are configures with static routes, and then the static routes are redistributed to OSPF (the static routes are made with higher AD so the OSPF will be preferred). The whole subnet for those static routes are also redistributed to PEs through EBGP.

So PEs will be able to reach those routes, while CEs will always prefer the OSPF over BGP.

Does this scenario makes sense and it is a good design ? 

Thanks,

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Hard to say whether it makes sense and/or is a good design.

Generally, interior devices, if doing L3, need to "know" where to go for routes not in the interior routing protocol.  Often, this is done using a default route.  That might be done statically, but when working with BGP, you can dynamically inject a default.  (So, if a CE loses it eBGP peer, it stop injecting its default.)

Likewise, your external network needs to "know" about your interior OSPF network.  As BGP can "pick up" routes already in the route table, you don't need to use a redistribution command.

When working with eBGP, we often don't want to advertize every internal prefix and we want to avoid needless insertion and removal of routes.  For single peering setup, often have a null route aggregate work well but when working with multiple peers, a dynamic aggregate is better.

You mention redistribution of statics and modifications of ADs between statics and OSPF; that's sounds unusual.  Unclear why a static and OSPF should "know" of the same route if you're redistributing statics into OSPF.

[4everlearning]

Thanks Joseph, 

and sorry for the confusion caused by my previous post, i went through the configuration of internal devices and they are not routers and don't support OSPF. Sorry again for the confusion as im going through the configuration to understand the topology since there is no diagram for the network. 

So the CEs connect to internal network using L2 (VRRP) or L3 point to point connectivity, now the CEs run OSPF only between the CEs and advertise all internal routes either by redistribution of static/connected routes or network command.

by running OSPF between CEs, they will be able to reach each internal entity through the right CE instead of having to configure static routes since as i mentioned before some entities are only connected to CE1 while others are connected to CE2. 

As for the static routes configured to reach some of the internal networks and then redistributed to OSPF, I believe the reason of preferring the ospf over the static AD is making the CEs always use OSPF to reach those destination since by default they will prefer the static route since they have better AD (P.S. even for the few connectivities that use static routes, some static routes on CE1 are not configured on CE2 and vice versa)

Please let me know your feedback. 

[Joseph W. Doherty]

Not this time. 

[4everlearning]

Hi Guys, 

Just one more question regarding this scenario: 

For outbound traffic, in case no tracking configured, If CE1 is the VRRP master for one device and the CE1 to PE1 link fails (CE1 will remain the Master since no tracking is enabled). Now would the traffic traverse the link to CE2 using L2 trunk or L3 links ? 

My confusion is that CE1 will receive the traffic as L2 traffic, but at the same time it will receive the advertisement on how to reach service provider as L3 and since CE1-PE1 link is down, the path will be through CE2. so would the traffic move from CE1 to CE2 on the L2 trunk and then be sent to PE2 as L3 traffic ? 

or would traffic be sent from CE1 to CE2 as L3 using the L3 links and going out to PE2.as L3 traffic ? 

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Once the traffic gets to CE1 it will look for the "best" L3 path to its destination; normally PE1.  But when down, it will look for the "best" L3 path to PE2, and from what you've described (if my scorecard is correct) would be the iBGP hop between CE1 and CE2, i.e. the p2p L3 link.

[Jon Marshall]

I agree with Joseph which contradicts what i said earlier when you asked what would happen if the CE1 to PE1 link failed (i've updated the other answer).

I was thinking that you were tracking VRRP but you aren't so yes CE1 will have to make a routing decision and since you are not redistributing BGP into OSPF the traffic would need to be sent across the L3 links.

I wasn't sure, apart from VRRP and OSPF peerings, what the L2 trunk was used for but having reread the whole thread it looks like the internal OSPF devices are generally connected to either one or other of the CEs. If so then the link would be used primarily for routing inbound traffic to the correct CE ie.

if each CE is advertising all the internal networks to their respective PEs then the inbound traffic may end up at the CE not connected to the internal OSPF device and would then have to be sent across the L2 trunk to the other CE.

That may well explain why the L2 trunk link is larger than the combined L3 links.

Edit - if you are not redistributing BGP into OSPF then this suggests you are originating a default route into OSPF on each CE ?

Jon

[4everlearning]

Thanks Jon, 

What i understood from the configuration is that there is no redistribution between OSPF and BGP. 

OSPF is using completely separate L3 links than the ones used for iBGP. 

and the CEs run OSPF only between themselves (CE1 and CE2) and advertise all internal routes either by redistribution of static/connected routes or network command.

by running OSPF between CEs, they will be able to reach each internal entity through the right CE instead of having to configure static routes on each CE since as i mentioned before some entities are only connected to CE1 while others are connected to CE2. 

so OSPF is only for internal routes to save the time of configuring static routes.