BGP Multi-homing: Design and Troubleshooting - Questions and Answers from live Webcast

Mani Ganesan · ‎02-21-2011

Introduction
BGP Multi-homing
Related Information

Introduction

Manigandan Ganesan is a Customer Support Engineer at the Cisco Technical Assistance Center Routing team in India. He also holds the CCIE certification.

This document contains the answers provided for the questions asked during the live "Ask the Expert" Webcast session on the Topic - BGP Multi-homing: Design and Troubleshooting.

The following experts were helping Manigandan to answer few of the questions asked during the session: Shankar Prasath R. and Mohana Kumar Kaliappan. Both are Cisco Support Engineers and hold a CCIE.

The related Ask The Expert sessions is available here. The Complete Recording of this live Webcast can be accessed here.

BGP Multi-homing

Q. Why when we try to set up eBGP peers through interface loopback, BGP session does not come up even when I use command multi-hop 3 or 2?

A. If multi hop is enabled on both sides with 2 or 3, the configuration should work. Make sure TTL security is not configured under BGP.

Q. Can dampen flapping peer as opposed to the routes from a flapping peer?

A. We cannot dampen a flapping peer. We can shut down a peer temporarily, troubleshoot the problem and then un-shut it.

Q. Can you provide an explanation of recursive routing in tunnels?

A. When a router learns the tunnel destination through the tunnel itself the it leads to recursive routing. This is mostly a misconfiguration. check: http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a0080094690.shtml

Q. If there is router A as an eBGP peer with Router B, Router E is an iBGP peer with B, and router E has a loopback interface and ddvertised, all the peers are up, but I cannot ping from router A to loopback interface on router E. I found the reason that I didn't Advertised the Physical interface on router A. Is this correct?

A. When you ping from A to E, it would have gone through the physical interface of A, i.e., with the physical interface as the source address for the ping. Now, if E does not know this address, then it won't be able to reach it. Also, by default, the next-hop is not changed in updates between iBGP peers.

Q. Tagging is optional transitive attribute, why it should enabled to receive?

A. Yes communities are optional transitive attributes. By default all the Cisco routers understand the community attribute. Routers by default do not send the community attributes to the neighbor and it should be enabled by the send-community command. If a neighbor sends a community attribute, the neighbor will always receive it and process it.

Q. Does the BGP failovers the VPN connections running through the border router?

A. BGP will not take care of VPN failover by itself. You can configure something like IPSec stateful failover.

Q. How is BGP Best Path Algorithm work?

A. BGP Best Path Algorithm has a set of rules which it has to go through before it chooses the best path. By Default BGP will look for only one best path unless we configure it for load-balancing or load sharing .If it is an IGP, and there are multiple equal costs paths, we can chose all of them and install them in to routing table but BGP doesn't work like that. BGP has to go through all ten steps and come down to the last step until it figures out which one is the best path. If I remember them correctly, the first one it will look for is BGP weight. If we are receiving multiple advertisements from our peers, if you want to pick the best one, its gonna start from here BGP weight and local preference, whether its IBGP or EBGP routes or is it local routes or routes learned from other guys. so it has to walk through all ten steps. And the last one is the router ID which cannot be the same, that's gonna be the tie breaker if all 9 steps are exactly the same. But at the 10th step what its gonna look for is whether we have this command BGP Multipath Configured.If that is the case instead of picking up just one best path, if all ten steps are same, its gonna pick both the equal paths and install them in to routing table. So this is the only way in BGP to make sure that we load balance. For more information read document Best Path Selection Algorithm.

Q. How do we define load balance outbound traffic?

A. Load Balancing of outbound traffic can be done using two different ways.

The firstway is using weight, if its just one router and you are pairing with two different service providers what you have to do is, it depends on whether we have received just the default routes from the provider or we are receiving the complete Internet routing table. If it just the default routes, just assign this command or configure this command "bgp multi-path" and make sure everything exactly looks the same features like AS Path, Local Preference. If everything is exactly the same we gonna install both the default routes.
If we are learning the whole bunch of Internet routing table, what we can do is, since we want to load-balance traffic between the first and the second service provider,is to divide the internet routing table in to two halves The first 50% starting from 1.0.0.0 to 127.0.0.0/8 and the second half gonna be 128.0.0.0 to 254. Once we match these set of routes and then assign a higher value of local preference for the first half and assign a higher value of local preference for the second half on two different routers which would make sure that we send traffic out of the first link, the packet is destined to the first half of the routing table, if the packet is destined to the second link,we gonna prefer router B or second router. So this is how we can achieve load balancing for out bound traffic.

Q. Can you better explain upgrading to newer IOS version of the code as it relates to BGP Scanner?

A. Lately I have seen certain customer's scenario's where CP is really high because of BGP Scanner and the router is actually getting the full routing table (i.e. the full internet routing table.) The reason is that the router was running very old version of the IOS which was not running an event driven model. So every 60 seconds when BGP scanner process gets triggered, is actually going through the complete routing table and the complete BGP table to figure out if all the next hop IP addresses are active. So, the only solution for that scenario was to upgrade the software because we cannot do anything about the routes we learnt,and we need all those routes from the Internet. In that case, if we upgrade to the latest code, I think after IOS version 12.4, every router is running an event driven model and the command Nest Hop Tracking (NHT) is enabled by default. You just need to make sure that you don't run a very old version of the IOS sofware, otherwise its going to be very difficult to avoid the problem of high CPU because of BGP Scanner. Just to summarize, every recent IOS software version of the code after 12.4 is running the event driven model and Nest Hop Tracking is enabled by default.So we dont need to worry about it.

Q. How would the setup be if you would peer with two ISPs: one time with the first, and twice with the second? Are there any considerations?

A. It depends on how many routers we have. When it comes to peering, peering twice or peering once doesn't matter, peering is always going to be active depending on how we configure our routing policies. Let's say, if we have a two-routers set up, the first router is connected to first ISP and the second router is connected the same ISP but twice. I think this is what I understand from the question, we have 3 connections, the first one is connected to one ISP, the second one is connected to same ISP twice. In that case, we can actually divide this scenario in to two halves.The first one is just connected to one ISP there is no multi-homing and the second router connected to same ISP twice, in this case we have use Metric MED value. This is gonna become Case 1) Mode of A or B that I explained. So, we have to use metric to make sure we send and receive or we receive traffic on both the links of the second router when it comes as a whole we would be having three links Multi-home to three different ISP's.So this is how it will work. Two set up first one with just one connection then second one would become multihome to the same ISP.

Q. How do we do load balance for inbound traffic?

A. There are two different ways depending on whether we are peering to same ISP or different ISP. If we are peering to the same ISP, anf you want to load balance inbound traffic, you can assign same the metric value, or don't configure any metric value (i.e. leave it default metric) and then make sure that the ISP sends traffic through both links. If we want to load balance traffic for inbound but we are connected to two different service providers, instead of doing As-Path prepending, send both the advertisements. Let's say, if we have a /20 or /19 blocks for example (as we saw in the presentation,) divide in to the blocks in two halves. If you want to achieve percentage load sharing, say 60% to fist ISP and 40% to the second, and if we know for a fact that this part of my network would be receiving more traffic, in that case just match them using a prefix list or access list and then advertise this particular specific route from the first router and advertise rest of the routes from the second router. So any traffic coming to this specific block would come through the first ISP and any traffic belongs to the rest of the network would come through thesecond ISP. This is another way of achieving load sharing for inbound traffic.

Q. What is asymmetric routing and what are the effects of it?

A. That's a very good question! Asymmetric routing happens when we send out traffic on one link, say link 1, and when we get the return traffic we would receive traffic on the other link. This would mean that we haven't configured BGP the right way. That means we don't have control on how traffic goes out of the network and how traffic comes back to our network .If we have a firewall in between before the packets reach our core network what gonna happen is firewall would never permit half opened TCP sessions. Packets are going to get dropped at the firewall level. So will have to make sure that for the network resources that we host when packets go out they go out on the same link and when they come back they come back on the same link. Also how do we ensure that? Cofigure BGP accordingly: use local preference, As-Path prepending or metric according to the scenarios we discussed and make sure we send and receive traffic on the same link.

Q. Can load sharing between different AS domains be achieved by applying the always compare MED attribute?

A. Actually we can. Technically there's a way using a command BGP Deterministic MED and BGP Always Compare MED. These are two commands that are used to make sure that if we receive two different MED values one from Autonomous System A and another one from Autonomous System B. By default we don't compare. But if we use these commands, BGP Deterministic MED or BGP Always Compare MED, the receiving autonomous system gonna compare both the MED values and make sure that the lowest wins.

Q. How do you achieve inbound load balancing with redundancy, if we are connected to different ISPs?

A. When we talk about load sharing, redundancy should be there. When I talk about it, when we advertise /19 block, we talked about dividing this block in two /20 blocks, so that we advertise the first sub block from router A, and the second sub block from router B. The key here is, along with this /20 block we would be advertising the /19 block as well. So, when we look for most specific mask, the first one is gone down. For example the /19 would be down and the /20 would be down. Now from the other side along with the /20 we do have the /19 which would cover our entire network. Therefore, even though our primary aim here is to load share traffic that's why we have divided in two different class /20 blocks so that we are still advertising the /19 blocks. Therefore, failover would be automatically done when the primary has gone down.

Q. What are the different ways of influencing outbound traffic from our AS in BGP?

A. There are multiple ways I should said. If it's just one router connecting to multiple circuits, we can actually use weight, so that it is going to make sure that we send our outbound traffic with the path which has the higher weight. For example, if we get two advertisements from two different service providers and you want to prefer the first, just match that particular route and then assign a higher weight for that route. So when any traffic going out of this router would prefer the first path. But if there are two routers (i.e. two gateways) and you want to tweak the way outbound traffic is sent out, then we have to use local preference and run iBGP between them. Local Preference is used when there are multiple routers in the autonomous system and you want to make sure or you want to tweak the way packets are being sent out of your network. So there are two ways Weight and Local Preference attributes.

Q. How do we improve convergence time in BGP?

A. When we talk about convergence, there are multiple timers in BGP that come in to the picture. The first one is BGP Scan Timer. Let's say that we have a router connected to two different ISP's and the route to first one is down (i.e. mean the route through the first one is gone down), but the BGP session is still stable. But we have to start sending traffic to the second one and instead of the best path. The best path should be through the second one because the route through the first one is down. So, how much time does the router takes to realize that the route through the first link is down and how much time do we take to start sending traffic out of the second link? This can be defined as convergence time.

What we can do is we can actually change the BGP scan timer, if our router can accommodate the BGP scanner process less than 60 seconds. By default it is 60 seconds. If we want to run BGP scanner every 30 seconds or so if you are just getting a default route, that's perfectly fine. Make sure that BGP scanner runs very quickly and ensure that it realizes quickly first one is down and change over to the other one.

The second timer is BGP Advertisement Interval. The interesting part in BGP is that unlike any other routing protocol it doesn't sent advertisement through out. So its not a event driven model. It's going to wait for specific time interval and collect all the changes that happen within that interval and advertise all the changes out in one go. It happens may be once in 30 seconds for EBGP and once in five seconds for IBGP by default. If you want to reduce these time interval we can actually improve convergence time because anytime there's a routing change we gonna advertise this change really quickly to the ISP and the ISP will realize that the route is going down and we will have to start sending traffic to the other link. Word of caution before we start tweaking the scan timer or advertisement intreval make sure that our router can support our router CPU can accomadate these changes.

Q. When and how can BGP Communities to be used?

A. BGP Communities can be compared with the Tags we use in IGP. So why do we do tagging in IGP? If you want to tag a specific route and the receiving router would understand these tagged routes should be treated differently either they have to be filtered out or they have to be given higher preference. In BGP, instead of doing tagging we would do something called community. So we would assign a specific community value to a route that we advertise out so that the receiving router would understand whatever route is coming with this specific community has to be treated differently either assign higher local preference or assign higher weight. So, when this is used in practical scenarios, if our ISP's can support communities what we can do is we can advertise a specific route with a community value and the ISP router can chose this route with the community value and assign a higher weight.This is another way of achieving inbound load sharing.This would involve our service providers work. We generally don't prefer that because we will have to call service providers and make changes every time.

Q. How can we load balance outgoing traffic from a single subnet (Multiple applications, e.g. application 1 on 10.10.10.5 & application 2 on 10.10.10.6). I have 2 neighborships via 2 service provider with branch site from central site router. traffic from application 1 should go via link1 and application 2 should go via link 2?

A. If you are looking to load balance just the outgoing traffic, you can very well use PBR instead of tweaking BGP. Create policy-maps accordingly on the router and set the next hop accordingly and that should do it.

Q. when should we use as-path prepend for incoming traffic when multihoming to dual ISP?

A. AS-PATH prepend is the preferred way to influence the incoming traffic if multihomed to two different ISPs as other methods of influencing incoming traffic like MED do not work across two different ISPs.