Solved: BGP - 3 isps, 1 router doesn't want to play nicely.. - Page 2

gskhanna · ‎09-29-2003

I have 3 local routers connecting to 3 diff providers. All 3 connected with ibgp just fine, sharing all the routes. Router 1 and 2 also connected to the isp with ebgp and get routes just fine.

Now when Router 3 connects to it's isp, it establishes and gets all the routes from it's ISP, but then the ibgp to that router goes all wierd.. All the ibgp routers send withdrawl updates, and removes all the routes, then resends them all again. Over and over, sends/withdraws.

However the bgp connection never goes down between it and the ibgps, and the connection between Router 3 and the isp peer doesn't go down and the routes stay.

Ideas?

I am using basic config of:

neighbor <ibgprouter> remote-as 1

neighbor <isprouter> remote-as 2

Please any suggestions?

ruwhite · ‎10-02-2003

Second Question:

Can you point me to a sample config on how to have routes going to a certain AS take the path out a certain router?

--

There aren't any that I know of, or that I could fine by poking around the CCO docs.... What you want to do is to use an as path access-list to match on each provider, and then set the local preference for a match. So, say your two providers are AS65000 and AS65001, and you want to push some of the traffic over to AS65000. You can make it so AS65000 is preferred for 3 hop paths, even if the connection through AS65001 is normally preferred:

ip as-path access-list 100 permit ^[0-9]*$

ip as-path access-list 100 permit ^[0-9]*_[0-9]*$

ip as-path access-list 100 permit ^[0-9]*_[0-9]*_[0-9]*$

!

route-map prefer65000 permit 10

match as-path 100

set local-preference 110

!

router bgp x

neighbor x.x.x.x remote-as 65000

neighbor x.x.x.x route-map prefer65000 in

(from memory, not on a router, so I could have something wrong here)

This should match any routes of three AS hops, and force them to go through AS65000, even if you have a two hop route going through AS65001. You can adjust the outbound traffic by adjusting the number of [0-9]*_'s included in the as path access list--two, and it should prefer any two hop as paths to go through AS65000, three for any three hop paths, etc. So, for instance, if you want to influence more distant networks, where there's less likelihood of suboptimal routing, then add statements with 3 and 4 [0-9]*_'s.

I hope this makes sense....

:-)

Russ.W

gskhanna · ‎10-02-2003

Makes perfect sense, and your syntax was perfect also!

Thanks to both of you I now have a good stable network that works v well.

But one thing, :) The XO router is still getting too much traffic compared to the other links. I have added 4 of the [0-9]*_ to it allready. I will add another 1 or 2 and see how it goes. XO obviously has a LOT better routing then Yipes (main Link).

I'm soo happy. Thank you again! :)

ipotts · ‎10-03-2003

These changes will only affect outgoing traffic, not traffic incoming from XO. To affect this you need AS path pre-pending, when the routes in your AS are advertised.

ruwhite · ‎10-03-2003

Cool! I'm glad we helped!

So, two things.... I assume you're leaving in the two hop filter when you add the three hop filter, etc.

^[0-9]*$

^[0-9]*_[09-]*$

Catches one and two hops.

^[0-9]*$

^[0-9]*_[09-]*$

^[0-9]*_[0-9]*_[0-9]*$

Catched one, two, and three hops.... You have to include all of them, since the three hop won't catch the two hop (because of the spaces in the regex). At least I don't think it does....

Second, is this inbound or outbound traffic? The filters above will only inpact outbound, not inbound, and you need to try and balance both, if you're trying to get your money's worth out of the links.... The only real option at this point is as path prepend for inbound traffic (well there's another one, which I'll talk about below). The syntax for this should be:

route-map prefer65000 permit 10

set as-path prepend x x x

With your AS number in place of each x. Try with just one first, and see what that does to the traffic flows. Increase the number of prepends 'til you get the balance right. You might need to prepend towards more than one upstream ISP to get things to work right.

Another option is to work with your ISP to set the local preferences within their AS to something that prefers the route through another of your upstreams. This is generally easy to do--you set some community on your side, then you ask them to translate that community into a local preference on their side. Most ISP's already have this sort of thing set up, you just need the "magical community list."

Note, however, this could cut your traffic entirely out from one of your upstreams, pushing it all over to the other one, so.... It might not work as well as as path prepend, as limited as that approach really is.

:-)

Russ.W

gskhanna · ‎10-05-2003

Actually for inbound the prepend would work nice, but I actually did something different.

We have a /19 for our network, and we are advertising the entire /19 on all 3 routers. Now, I am advertising /21 and /24 out of that /19 on one router or another. and inbound traffic is actually routed to the right place because outside networks see the route as being closer due to "longest match". on the router with the /21 or /24 on it.

I know that some isp's ignore anything smaller then /20 I believe, but I'd say this has so far worked with 99.9%. I have yet to see a place that this didn't work with.

I will switch it over to the proper prepend method later or work with the community strings. 2 out of our 3 isp's allready gave us that info.

One more question for outbound traffic. I used the 2/3hop as path filtering. and I added it as long as 9 hops now. and our smaller link which goes thru xo is still getting a lot more traffic then desirable.

What I would like to do is set a certain subnet to only have outbound through a specific router. Because along with our /19, we have a another /19 from our isp, that is only inbound on one router, but now it seems to be taking the "best path" out, whereas we only want it go out thru one isp.

-GK

ruwhite · ‎10-05-2003

Most ISP's will accept anything under a /24, so this is an excellent scheme, much better than prepend, as long as you can get the granularity you want in controlling inbound load.

For the second question, hmmm.... You can mostly control that by controlling which router the ountbound traffic hits. So, adjust your igp metrics for that specific route so the outbound router you want to be chosen is always chosen, then it will choose its local link out, most likely.

As for the 9 hops, hmm.... Is the local pref actually being set on a numebr of routes when you try this filtering? Can you look at the other router's bgp table, and see if it is?

Russ.W

gskhanna · ‎10-05-2003

Ya, the smallest we would need to advertise is a /24 so it works good.

The 9 hops. The local preference is set to 110. I noticed when I had it set to 4 hops, it showed about 50,000 routes from router #1, when I increased it to 8 it sent up to 85,000 routes. and 9 it's at 90,000 routes.

for instance:

* i12.1.96.0/24 66.90.64.50 0 100 0 16631 174 2914 19024 14359 23306 i

* 66.237.108.29 3 0 2828 3561 19024 14359 23306 i

*>i 66.90.64.49 0 110 0 6517 7911 2914 19024 14359 23306 i

the .50 is router #2, and shows it as ibgp path. The 2nd is .29 is the xo route. and the .49 choosen as best path is the router #1 that majority of the traffic to go thru.

So it is working, I just think the isp on router #1 (Yipes, also known as williams) is just a lot worse then XO so that might be why.

The static subnets that we receive from our isps, the default gateways for those networks are on the respective routers. So it is hitting it correctly, and then choosing best path out, instead of staying on that rouer :)

Can you give me syntax for keeping it on there?

ruwhite · ‎10-05-2003

Hmmm.... If the XO router is pointing towards the router that connects to Yipes as its next hop, it should be directing traffic out through Yipes, rather than through XO. Could you post a show ip bgp route for just one route you're learning through both ISP's that should be impacted by the route map?

Russ.W

gskhanna · ‎10-05-2003

Here is a show ip bgp from Xo router.

* 12.5.136.0/24 66.237.108.29 3 0 2828 209 16759 i

*>i 66.90.64.49 0 110 0 6517 7911 209 16759 i

As you can see, the 108.29 (xo isp bgp peer) is 3 as's away. while 64.49 (my yipes local router) shows it as 4 as hops away and it is alos the ibgp peer route, and it is also the > best route. because I have the route map set to a silly 14 AS hops. Yipes routing truly does suck it seems.

It works perfectly, as trace routing to that form the xo router i get:

1 gate.fdcservers.net (66.227.96.1) 4 msec 0 msec 4 msec

2 209.120.155.13 [AS 6517] 0 msec 0 msec 4 msec

3 chcgil1wcx1-gige15-0.wcg.net (64.200.247.237) [AS 7911] 0 msec 0 msec 4 msec

4 brvwil1wcx3-pos3-0.wcg.net (64.200.236.30) [AS 7911] [MPLS: Label 12330 Exp 0] 0 msec 0 msec 4 msec

5 chcgil9lch1-pos7-1.wcg.net (64.200.210.118) [AS 7911] 0 msec 0 msec 4 msec

6 chcgil9lcx1-pos9-0-oc48.wcg.net (64.200.103.109) [AS 7911] 0 msec 0 msec 4 msec

7 chcgil9lcx1-qwest-gige14-1.wcg.net (64.200.228.190) [AS 7911] 0 msec 0 msec 4 msec

8 chi-core-02.inet.qwest.net (205.171.220.57) [AS 209] 0 msec 0 msec 4 msec

9 cer-core-01.inet.qwest.net (205.171.205.34) [AS 209] [MPLS: Label 587110 Exp 0] 0 msec 0 msec 0 msec

10 cer-core-02.inet.qwest.net (205.171.139.2) [AS

etc.. (after this trace seems to be filtered) but it shows it took the route thru yipes which I wanted.

Now what I need to do, is set certain local subnets to go thru only certain routers.

ie, i have 66.227.96.0 /20 on my router Yipes router. I want it to statically only go out thru the Yipes router. I do not want it to choose the best path or anything, just go out that Router.

ruwhite · ‎10-06-2003

Okay, that's cool.... The easy way to add this is by using a second level of local preference. So, on the Yipes router, which has no route map right now (correct? the route map is on the AO router at the moment?), you would do this:

access-list 10 permit 66.227.96.0 0.0.0.255

!

route-map preferme permit 10

match ip address 10

set local-preference 120

!

router bgp x

neighbor y

neighbor y route-map preferme in

Now, when you received routes that are within access list 10, they have a local preference of 120, which will beat the default local preference (100), or the local preference of 110 set elsewhere. This is simpler than doing excludes from the other filter, or anything like that.

You can also do this on the XO router, where the filter for as path length is already running, by adding another clause on the route map. Suppose you have this:

route-map prefer65000 permit 10

set local-preference 110

You can now add this to the config:

access-list 10 permit x.x.x.0 0.0.0.255

!

route-map prefer65000 permit 5

match ip address 10

set local-preference 120

You can also repel traffic from a router, by setting the local preference lower than 100.

Russ.W

gskhanna · ‎10-06-2003

Yes, the Yipes router has no route map configured. Now when I want to set routes that are coming FROM my yipes (66.227.96.0 /20) I need to do

access-list 10 permit 66.227.96.0 0.0.0.255

!

route-map preferme permit 10

match ip address 10

set local-preference 120

!

router bgp x

neighbor y

neighbor y route-map preferme in

question, what do i put for the neighbor command? as I want the router I am putting this on, to have local preference of 120.

Oh doh. I think I misunderstood you. This routemap would go on the other two routers that I do not want it to go thru, and then use this list to match the route coming from the yipes router, and set all routes coming from that neighbor as a higher local preference. and therby thru ibgp, say yipes router (.49) has the best route (local preference 120) so go thru there.

Right?

ruwhite · ‎10-07-2003

Yes, the config you gave above would be fine.... The route-map in would match the set of destinations you definitely want to send through yipes, setting their local preferences to 120, thus causing the other two SP routers to prefer this router for those destinations.

The same route learned on the other two routers will have, at the most, a local preference of 110.

(You could also use this on the other two routers, seeting the local preference lower, to repel traffic from them, and towards yipes, but it's easier to pull traffic towards yipes ont he yipes router than it is to push traffic towards yipes from the other two routers.)

Russ.W

gskhanna · ‎10-08-2003

I didn't get a chance to try this yet. I will tonight. One more additional question.

Won't this just influence the path for "incoming" packets to the network to be sent to that router. instead of outbound packets sourced from 66.227.96.0 ?

Cause I have 66.227.96.0 on my Yipes router, and I want that traffic from there to only go out thru the yipes router.

-GK

gskhanna · ‎10-08-2003

Someone just mentioned to me, that BGP won't help do this for outbound. That I need to do "policy routing" instead?

-GK

ruwhite · ‎10-17-2003

No, you don't need policy routing for outbound traffic flow.... You can use that, but it's a complicated way to solve the problem. Policy routing is primarily designed to control the next hop of traffic based on the source address or TOS bits of the packet, rather than the destination address. If you can control the traffic using route maps based on the destination addresses, it's much easier, and much better, in the long run, to manage and to troubleshoot.

Russ.W