BGP Routing Question - 2 routers, 2 uplinks, HSRP

bean · ‎09-08-2002

Okay, I have two 7206VXR routers named border1 and border2. border1 has a UUNet DS-3 uplink. border2 has a Genuity DS-3 uplink. The two routers are connected via a 100Mbps Ethernet connection. We're running eBGP between our network and the uplinks and iBGP between the two routers. Each router also has a 100Mbps Ethernet interface used to connect to our LAN switches. We're running HSRP on the two FE LAN interfaces.

The problem:

The majority of the traffic going out of our network goes out of the router with the ACTIVE HSRP interface.

Example:

When the active HSRP interface is border1 (UUNet), it was sending about 1Mbps out. border2 (standby HSRP) only had about 200kbps going out. When I lowered the priority of border1 so that border2 would have the active HSRP FE interface, border2 began sending out about 1Mbps and border1 began sending out about 200kbps.

To see what was going on, I did a 'show ip bgp 64.236.16.84' on each router:

border1#show ip bgp 64.236.16.84

BGP routing table entry for 64.236.16.0/20, version 764876

Paths: (2 available, best #2, table Default-IP-Routing-Table)

Advertised to non peer-group peers:

10.1.1.2

1 1668 5662

4.24.243.5 (metric 65536) from 10.1.1.2 (10.1.1.2)

Origin IGP, metric 5865, localpref 100, valid, internal

701 1668 5662

65.208.15.213 from 65.208.15.213 (137.39.5.96)

Origin IGP, localpref 100, valid, external, best

border2#show ip bgp 64.236.16.84

BGP routing table entry for 64.236.16.0/20, version 8186823

Paths: (2 available, best #2, table Default-IP-Routing-Table)

Advertised to non peer-group peers:

10.1.1.1

701 1668 5662

65.208.15.213 (metric 65536) from 10.1.1.1 (10.1.1.1)

Origin IGP, localpref 100, valid, internal

1 1668 5662

4.24.243.5 from 4.24.243.5 (4.24.1.41)

Origin IGP, metric 5865, localpref 100, valid, external, best

It looks like the problem is with the metrics. How do I correct this so that the routers will take the best path out of our network and disregard whether or not the data needs to be sent to the other router before doing so? I would like the traffic to be a bit more balanced, but I want to ensure that the best path is used.

Help!

Jordan

bean@webcoretech.com

steve.barlow · ‎09-08-2002

First off, achieving 50/50 load balancing doesn't happen often. 60/40 or 70/30 is often considered successful. Bear that in mind.

Having said that you can improve your situation. I think the best bet would be to implement MHSRP (have 2 HSRP groups so half your clients/servers point to one 7206 and the other half point to the other 7206). This link will help.

http://www.cisco.com/univercd/cc/td/doc/cisintwk/ics/cs009.htm#xtocid122333

Then I would set the weight attribute on each router so that any route learned from your eBGP peer is prefered over the same route learned from your iBGP. This way your next hop will always be the eBGP peer (hot potato routing, get it out of your AS as soon as possible). This will override the eBGP peer from influencing your routing choices and will let your MHSRP do the load balancing for you.

eg. neighbor 1.1.1.1 route-map SETWEIGHTIN in

route-map SETWEIGHTIN permit 10

set weight 2000

If MHSRP isn't an option for you, look at local preference on each router so that the router prefers the eBGP peers' local routes and directly attached customers (eg ip as-path access-list 20 permit ^701_[0-9]*$) over those routes learned via iBGP.

See the following link discussing BGP path selection:

http://www.cisco.com/warp/public/459/25.shtml

Hope that helps.

Steve

bean · ‎09-08-2002

Steve,

Thanks for your reply. It looks like if we implement MHSRP with the current setup, that in itself will help balance the traffic (enough), so tweaking the BGP settings would not required. This is a good solution and we will implement this if the below is not doable.

My preferred approach to solving this problem would be to have both routers see the same preferred route when doing the 'show ip bgp x.x.x.x'. If this could be accomplished, I think the traffic would be balanced (enough) since both are Tier 1 ISP's. The FE connection between the two routers is obvioulsy affecting the routing decision and I don't think it should.

I think the behavior that I'm trying to accomplish is for the 2 routers to make routing decisions like a single router would (i.e. treat the connection between the two as having a metric of 0). My problem is that I don't know if this is doable.

Jordan

bean@webcoretech.com

steve.barlow · ‎09-08-2002

The FE link between them is not affecting the routing decision, it could be a 56Kb or gig link and it would make no difference. BGP cares not about link speed. It only cares that they are iBGP peers. Remember BGP is not like other routing protocols and does everything it can to not load balance. The command max-paths let's you load balance but I wouldn't use it in your case.

To make both routers see all routes the same is possible for one or a few routes but for the whole table is not practical.

The best bet if you insist is to accept only the default route from each and use max-paths on both.

Steve

bean · ‎09-08-2002

Okay, one last question. When I do a 'show ip bgp', I get a lot of entries similar to the following:

*> 12.4.96.0/24 65.208.15.213 0 701 1239 14452 i

* i 4.24.243.5 2200 100 0 1 1239 14452 i

*> 12.4.96.0/23 65.208.15.213 0 701 1239 14452 i

* i 4.24.243.5 2200 100 0 1 1239 14452 i

*> 12.4.97.0/24 65.208.15.213 0 701 1239 14452 i

* i 4.24.243.5 2200 100 0 1 1239 14452 i

*> 12.4.114.0/24 65.208.15.213 0 701 7018 14065 i

* i 4.24.243.5 1200 100 0 1 7018 14065 i

*> 12.4.119.0/24 65.208.15.213 0 701 7018 14065 i

* i 4.24.243.5 1200 100 0 1 7018 14065 i

*> 12.4.125.0/24 65.208.15.213 0 701 1239 i

* i 4.24.243.5 2200 100 0 1 1239 i

(I'm not sure how this is going to turn out)

I have noticed that all announcements received from Genuity have a metric set (the metric varies and is 1200 or 2200 in all of the above examples). NONE of the announcements received from UUNet have anything specified in the 'Metric' column. Would this affect the routing decision? Is there something wrong here?

Jordan

t.baranski · ‎09-08-2002

Couple of things:

1) In my opinon, the fact that MHSRP requires a configuration change (new gateway) on half of your backbone devices is reason enough to avoid it if it's possible to acheive the desired load-balancing via BGP on the routers themselves.

2) I don't believe metrics are coming into play here unless for some reason you have 'bgp always-compare med' configured. By default (unless this command is configured), metrics aren't compared when they come from different AS's, which is the case in your configuration.

I believe the cause of your issue is step 7 of the BGP decision process: Prefer external (eBGP) over internal (iBGP) paths. As you're not malipulating local preference, this will be what the router uses to decide whether to send traffic out the WAN link or to its IBGP neighbor when the AS-PATH length is the same, which it is for the example you posted above. As both of your providers are Tier-1, this is likely to be the case for a significant percentage of the routes that you're seeing. So, the active HSRP router will always send traffic out its WAN link to destinations for which the AS-PATH length is the same from both providers.

What seems to me to be the best solution to this offhand (and I'm fairly new at this so I could be missing something obvious) is to do the following:

- Have each provider announce only their customer routes to you via BGP. This will ensure that traffic towards these destinations goes out the optimal WAN link.

- To load balance traffic not destined to a customer of either ISP, configure two default routes on the active HSRP router -- one pointing to the WAN link, one pointing to the FE link to the IBGP neighbor. Then, use either fast switching or CEF to load balance these two routes on a per-destination basis (per-packet is a bad idea). The standby HSRP router would have only 1 default route pointing to its WAN interface.

I believe this would work, but again, I could be missing something.

bean · ‎09-08-2002

1)

Actually, if you're already running HSRP, moving to MHSRP is easy. Only changes on the routers are required. I went ahead and did this to better balance the BGP traffic. In my situation, I created 2 HSRP groups. I made HSRP group 1 active on border1 and HSRP group 2 active on border 2. I then put half of my gateway addresses in group 1 and the other have in group 2. No need to reconfigure clients.

2)

You are correct. After taking a second look, the routers were always preferring the eBGP route.

To keep my config simple while achieving my goal, I just implemened MHSRP. That seems to have done the trick.

Thanks!

Jordan

bean@webcoretech.com