Teaser

[romccallum]

On flying through my studying this weekend i came across a very interesting (well i think it was interesting) and i thought i would share the scenario for everyone to try to see if they get the answer to it spot on. Scenario

PE A connects to CE A

PE B connects to CE B

CE A connects via LAN to CE B

BGP is the routing protocol for the vpn and CEA&B are in the same VPN. There is an IBGP connection between CEA & B also.

Configure this scenario to avoid routing loops. What would you do and why is my question here. The CE routers are in AS 87 and the SP environment is AS 65001.

Enjoy!!

[sean]

You wanting configs for PE and CE or just PE?

[romccallum]

hmmm not really after configs but if you wish to try them then fine. Im more after what approach you would use.

[olorunloba]

Really a teser. I do not see the reason why a scenario like this will be, more especially since the CEs are connected on the same LAN.

However, AS-Overide will definitely be needed for communication across the SP backbone. Weights would need to be configured to influence the traffic path and ensure a loop free network.

Robert, what do you think?

[romccallum]

well there are two ways of doing this If as you say you use as-override then you will have to use soo to stop the loop happening. Now if you do go down this scenario break the ibgp connection between the 2 CEs - Q: can CE1 ping CE2?? Answer is absolutely 100% NO. This is my dillema - i have always done as-override and soo in this scenario - however the only way to get this work under failure scenarios as well is to use allowas-in on the CE routers. Now im left with 5 years of experience being doubted 1 day before i go to sit the CCIE SP lab. Im well confused.

[olorunloba]

I'm a little lost in your scenario, so let me try and clarify things.

I will agree with you that if you use AS-override, you should use SOO. However, I was thinking that the 2 CEs are the only CEs in the vpn. If so, then the SOO is not useful. Once you use SOO, the 2 CEs cannot use the MPLS core to communicate, as routes from one CE are not readvertised back to the second CE. Therefore AS-Overide and SOO is not an option.

Like you mentioned, you could use allowas-in on the CEs with extra configuration on the PEs. But you could still have the possibility of routing loops.

I think to prevent your routing loops, weight needs to be configured. This way, you can manually set on each router the preferred path, and the alternate path.

In summary, I think you have two options. Either AS-Override on the PE, or Allowas-in on the CEs. But in either case, you will need to configure weights to influence the paths.

What do you think?

[romccallum]

hmmm. you seem to be thinking that as-override and soo is only used for traffic coming into a dual homed site (from what i read) - SOO is used to stop traffic from a dual homed site coming back into itself (if im not mistaken). Using weight will not help as this doesnt stop any loops (ever) its all down to the as-path issue that you strip when using as-override. Also using allowas-in doesnt require any more config on the PE this command is a CE only command in this scenario.

This to me is a really weird one as it kinda goes against good design practices. You would tend to say that "how would the ibgp connection go down in the first place" - but hey it could easuly go down if the customer thought "yeah we can upgrade this switch as the traffic will go through the SP vpn". WRONG!! I really fail to understand how weight in BGP stops a routing loop my friend, maybe im missing something - i hope i am!.

[michael_davis]

This is a really good question. Thanks for getting me to focus on it. So here's some thoughts that may or may not be useful.

I'm not sure that either approach is "best practice" from a technical standpoint. I think it all just depends.

Here's some pros/cons I can think of. Maybe others have some additional to throw in.

By using override/soo, the provider maintains some measure of control against customer misconfiguration and certainly eliminates loops. Intra-site traffic utilizes IGP routes rather than VPN/EGP.

The downside is the behavior you are seeing. So, the IBGP peering, loopback to loopback must be maintained. Customer must have some level of redundancy in their network to ensure that there is an alternate, non-vpn path.

Allow-as doesn't have the failover issue.

However, by using allow-as on these CE, you will prefer the VPN path over the LAN path since each CE is learning the other CE's local networks via eBGP. This may not be desired behavior either.

Also, AS prepend used elsewhere by the customer may break AS path limit configured for allow-as on CEs and thus allow loops.

Other thoughts, anyone?

Michael

[romccallum]

Good answer although - "However, by using allow-as on these CE, you will prefer the VPN path over the LAN path since each CE is learning the other CE's local networks via eBGP. This may not be desired behavior either" This isnt true im afraid as the BGP process will install the ibgp learned router i.e. the CE's loopback due to the shorter AS path and origin being ibgp.

The control manner is exactly correct - an SP would want to protect themselves so would use SOO. However the day will come when a cusomter will break that IBGP connection and then realise that they cant use the SP connection - oh man im glad i left the SP environment because i dont want to try to explain to a customer that we (SP) are intetionally breaking their network.

THanks all for responding - tho it still doesnt solve my problem - Tomorrow i MAY face an issue like this in the LAB. What to do? Dont Know!!

[mheusinger]

Hello,

well the answer is: this is how BGP is designed to work! If you split an SP network (f.e. by killing all route reflectors) you can not expect to route traffic across other ASs to get internal reachability. This is against the interior BGP design. You simply can only achieve, what has been designed into a protocol.

(Side note: you could also ask to filter one single network from an OSPF database within a single area - this is not what the design allows for)

Second: in case you have iBGP you (usually) also have an IGP, which should be responsible for internal routing (thus AD 200 for iBGP).

So the proper configs would be f.e. with AS100 in the SP and AS 65001 for the customer:

ip vrf CE11

rd 100:1

route-target export 100:101

route-target import 100:101

interface Serial1/1

ip vrf forwarding CE11

ip address 10.1.11.1 255.255.255.252

router bgp 100

!

address-family ipv4 vrf CE11

neighbor 10.1.11.2 remote-as 65001

neighbor 10.1.11.2 version 4

neighbor 10.1.11.2 activate

neighbor 10.1.11.2 as-override

neighbor 10.1.11.2 advertisement-interval 5

neighbor 10.1.11.2 filter-list 1 in

neighbor 10.1.11.2 route-map CE11sooIN in

maximum-path ibgp 2

no auto-summary

no synchronization

network 10.1.11.0 mask 255.255.255.252

exit-address-family

ip as-path-list 1 permit ^65001(_65001)*$

route-map CE11sooIN permit 10

set extcommunity soo 100:11

!-----------------Configuration of CE11 -------------

interface Loopback0

ip address 11.11.11.11 255.255.255.255

interface Serial0

description interface to PE1

ip address 10.1.11.2 255.255.255.252

interface Ethernet0

description interface to CE12

ip address 192.168.1.1 255.255.255.0

router ospf 10

network 11.11.11.11 0.0.0.0 area 0

network 192.168.1.1 0.0.0.0 area 0

router bgp 65001

neighbor 10.1.11.1 remote-as 100

neighbor 12.12.12.12 remote-as 65001

neighbor 12.12.12.12 update-source Loopback0

network 11.11.11.11 mask 255.255.255.255

network 192.168.1.0

no auto-summary

PE2 and CE12 would be configured accordingly with IP addresses adjusted.

The better approach from a BGP prespective is to have a separate AS for each customer location. Then you do not even need as-override. Just plain good old BGP design between multiple ASes.

My 2 cents

Regards, Martin

[michael_davis]

Right, but since we're not talking about separate customer locations, only separate CE routers with a lan connection between them, does the the multiple AS approach make sense? It sounds like you are suggesting a eBGP peering between the CEs rather than an iBGP - but I'm probably misunderstanding.

Also, I was working on the assumption that with the allow-as approach, the CE would prefer VPN/eBGP on Admin distance alone. I need to lab this up as I'm concerned that the iBGP peering Rob was seeing was over the VPN rather than the LAN.

Rob's concern with customers is a big one - at least from my perspective. I want to echo that concern. So often MPLS is marketed as a superior replacement for L2 WAN networks. And it is not at all uncommon for customers to want to use the VPN as an additional backup path between CEs. The issue is not a technical one, since you could always design around it. But rather a customer expectation issue. We can argue that Sales shouldn't market that way, but when you come in after the MPLS network is already purchased, or you don't work for the carrier, the pain is very real.

[romccallum]

A few things here folks -

1. Martin you are going down the exact same path as me - this is the way it is designed to work so there ya go. However a customer will NEVER see it this way as they expect to be able to have inter site communication over the mpls vpn afterall what are they paying for. Also i do not see any reason for your input filter in your confiuration - to me it makes no sense at all.

2. Michael you have hit the nail on the head - I labbed this up and thought oh ok BUT why isnt the EBGP link being preferred to reach CE1's loopback from CE2?? I then realised that despite EBGP being AD 20 that the AS-path and origin is better for the IBGP connection.

Now on a par here - i could argue both points - why would a customer have this stupid setup where they could have a failure in their LAN and cause several other lans connected to each CE to stop communicating with each other - the answer is simple - because the can and because they are the customer. Dumb as it may be but hell ive been there far too many times.

2. Now as an SP i would never ever allow the customer to control the routing BUT in this case you actually have got to allow them to do it with allowas-in.

not my cup of tea i must admit but hey - if i get hit with this scenario tomorrow i will promise you the proctors ear will be well used ;-) AND i will prolly go for the as-override and the SOO method as that is what Cisco say is the correct way to do it - albeit in my findings it is the total WRONG way to do it.

Thank you all fo rthis discussion. It is nice to see that it is not just me who thinks WTF?

[olorunloba]

Hello Rob again,

Firstly, that was a typo, I meant to type "Like you mentioned, you could use allowas-in on the CEs with NO extra configuration on the PEs".

Then to the question at hand.

From the customer perspective, he has two routers that need to communicate across the SP backbone.

From the providers perspective also, there are 2 CEs that he is given an MPLS VPN service to. And these 2 CEs have a backdoor connection. The LAN connection from the SP perspective can be seen as a backdoor link.

Looking at it from these perspective, I will not consider the scenario as a multihome site. Hence, to achieve the sets of objective above, SOO is not an option. Once you use SOO, the scenario is that of multihome site, in which the CEs do not need the SP core to communicate under any circumstances.

Therefore, I will ask the question, how is possible to use AS-Override, and still avoid routing loops. To answer these, we might want to see what happens with AS-override without SOO. I think the main problem here is that routes from the site can be readvertised back to the site. And this readvertised routes would be preferred over any other learnt route, as eBGP has the best AD (20).

I think another key point to note is that BGP advertises only the best route and active route in the routing table. I thought of weights, since it is locally significant, as a way of controlling BGP path decision. If weights for the varous routes and neighbors can be configured, in such a way to mirror primary and floating routes, then I think there might not be a routing loop. From the scenario, each route can have a maximum of two paths for a prefix. The weights should therefore be configured in such a way that the primary and secondary path are consistent on all the routers.

It does not sound optimal to me, and I have not tested it, so I do not guarantee that it will work. It is just an idea. If you can lab it up, it will be appreciated.

Sounds like you going for the lab tomorrow. If so, I wish you the best.

I think you should check the following thread, Paresh explained well the routing loop issues with the Allowas-in.

http://forums.cisco.com/eforum/servlet/NetProf?page=netprof&forum=Network%20Infrastructure&topic=WAN%2C%20Routing%20and%20Switching&CommCmd=MB%3Fcmd%3Ddisplay_location%26location%3D.1dda52c4

In case the link does not open, thread title is BGP redundancy, under WAN Routing and Switching, last update on Jan 27.

[michael_davis]

Maybe the lesson to take to the lab is that there limitations to either approach.

I'm thinking that if you get a requirement that you think should use AS pre-pending, you now know there's a dependency where allow-as is used if you go that route. And if there's a requirement for failover, you gotta use allow-as. Dunno, just hoping there's a silver lining here.

Anyway, good luck tomorrow! Of course, if you pass, I'll need to get off my backside and schedule my next attempt. ;-)

Michael

[mheusinger]

Hello Michael,

the eBGP suggestion is for different locations of the customer. ONE location should be setup with iBGP between CEs. My point was just, that a separate AS per customer location does remove the requirement for as-override and thus SOO in the SP network - hence simplifying the SP BGP.

Second AD does not play a role in BGP path selection - only after best path is selected (external or internal) to decide whether to insert the best path into RIB or not. Side note: this is the reason for RIB failre indication in the BGP table - best path was selected but could not be inserted into routing table.

I totally agree to the comment that wrong customer expectation might be an issue - but hey, did you ever implement QoS? WOW there are some rumors and misconceptions around QoS, which make the ones about MPLS VPN a piece of cake ;-)

(SP technician: "What should be prioritized?" Customer: "Oh, I thought you could tell me!")

@Rob: the AS filter just is to avoid eBGP multihop "testing" by customers across different sites. This could also lead to suboptimal routing ;-)

And a final note: bad design does create problems. If the requirement exists, that LAN connectivity in a site should be backupped by the MPLS VPN, then BGP is the wrong choice. We are talking about backdoor links and that might be an approach as well.

Regards, Martin