How bad is asymmetric routing in dual-homed BGP? Please advise

[DANIEL WANG]

I have a crude internet loadbalance setup using two different ISPs with assigned address spaces and two edge routers doing NAT. The two edge routers connect to a 3rd router acting as loadbalancer using two equal default routes with CEF per destination. I run EIGRP between these three routers. The loadbalancer then connects to a set of redundant ASA5510s outside the corporate vlans. For inbound connections to our Internet facing servers, I use one DNS server for each ISP assigned space, dual internal IPs on the servers, PBR on the loadbalancer to route server resonses to the edge router where the original request came through.

This low-cost solution has worked fairly well except for the browns-outs further upstream.

To tackle the brownout issues and achieve better, dynamic outbound loadbalance, I purchased two 1811 routers (to replace my old 2610 and 2500s) and plan to implement OER with BGP. We do own a /24 address and both ISPs (ATT and Roadrunner) allow BGP for class C size.

My main concern is whether asymmetric routing will affect our Internet applications. Obviously, web browsing is not affected. But we use video streaming sites for training purposes. We also have VPN connections, Webex for web conferencing, and persistent HTTPS for Extranet applications.

Moreover, since OER dynamically adjusts load on each ISP link, it may move one session from an over-subscribed link to the other, thus creating asymmetric routing in the middle of a session. How will this impact some of the above applications?

I need to replacing my old routers with the two new 1811 routers by the end of the month but I'd like to improve my current Internet setup if I could.

Any input from you will be greatly appreciated!

Daniel

[paolo bevilacqua]

Hi,

In short, asymmetrical routing is kind of normal in multihomed BGP installations, and should not cause any problem.

Hope this helps,please rate post if it does!

[coredumpbp]

Hi,

Before decide to have ot not asymmetric routing you must be sure about all your MAN ISP?s peering?s.

Some of the ISP?s are preferring some specific peers for some specific traffic in metropolitan connections, witch can be very dangerous for asymmetric routing, and you can end up in a big problem.

If you don?t know the local peering within the MAN always use prepend.

Advice: try to avoid asymmetric routing.

Rgds,

Florin

[DANIEL WANG]

Florin,

I am quite ignorant regarding BGP. We use a T1-speed connection over Frame to ATT and a cable (to be fiber) connection to TimeWarner. So local peering in the MAN is more likely to happen on the Timewarner side?

How bad can this be? Extra long AS Path? Can you give a bit more details or sources for reading?

This is a big decision for me. Once I move to OER and BGP, there will be no return for me (I could scrap the whole thing if it doesn't work well, but ... you get the point)

Thank a lot for the tip.

[DANIEL WANG]

p.bevilacqua,

That's good to know. I have been googling it but all I found was asymmetric routing is bad but no study on it based on specific applications and scenarios.

I need to justify the move to BGP and OER from our simple but working dual NAT/ISP setup.

Can you elaborate or give me some pointers?

Thanks a lot!

[paolo bevilacqua]

Hi,

BGP admittedly is not the easiest thing but when you want a certain level of redundancy in a professional installation there are no other choices. As you will discover, there is no easy and guaranteed way to influence inboud traffic. Yes you can use prepend and specific prefixes but due to the way your ISP can filter these techniques, it is not said the result is the wanted one. So you have to take the first step, begin the peering and observe how the traffic becomes by default. Chances are that it won't be too bad at all.

Controlling the outbound is much easier. is not even said that OER is needed and if you are new to BGP I would leave that for a second step and possibly never.

Going forward you will find that asymmetry is a natural consequence of being multi-homed and should not cause any problem. Often, trying to contrast it is just a lost battle that can lead to worse results.

Thanks for the rating and good luck!

[DANIEL WANG]

That's a great idea! I can get BGP set up and check the actually advertised routes, while keeping my current dual NAT/ISP setup in production, right?

My biggest concerns are

1) how to loadshare inbound, since the majority of our traffic load is inbound. I have the impression that BGP doesn't do loadsharing. Besides, there is probably no way to make the two routes completely equal.

2) how to better utilize our future fiber ethernet connection from TimeWarner.

The fiber bandwidth is going to be the multiple of our T1 through ATT. Unless we use ATT T1 as a backup, there is not way we can reasonably load balance the two disparate connections without something like OER, barring third-party appliances. Right now, ATT is our primary ISP and we like the sla we get.

Thanks a lot!

[paolo bevilacqua]

Yes you can have BGP and NAT at the same time.

Be aware that making seamless, error-free transitions requires a lot of attention and experience so plan ahead and schedule "potential downtime" or you risk irate users.

You have no way of knowing what will happen to incoming once you connect new circuits. For outgoing there are many way to balance and optimize. Then the fun begins. But if nothing works then ask your ISPs and if they collaborate, usually you will be set after a little of fiddling.

[DANIEL WANG]

By BGP and NAT at the same time, I meant keeping our current setup in the production, i.e., NATting to ATT's assigned address and NATting to TimeWarner's assigned address, while asking them to test/advertise our own ARIN IP space and AS number. That way we can play with BGP without anyone actually using it (they won't know our own public IP space until I update our DNS entries)

Will this give us any clue what our incoming route will be at different locations on the Internet? No live incoming connections can use BGP routing though, until I am ready to change DNS.

Could you give a brief list of methods for outgoing balance and optimization?

Sorry for continuously sounding so redundant and verbose. BGP is new to me. Our users have very low tolerance for any Internet issues.

Thanks a lot!

[paolo bevilacqua]

No worries, you are about to take the steps that many other admins have taken before you and is understandable that you want to be cautious. Then again show me someone that has tolerance to internet outages and we will have someone that probably doesn't even use a computer at all.

There are many many papers about that floating around, I don't really suggest you get deep into them as it could make things even more confusing.

Now for your case, once you start talking BGP still keeping the current NAT method, yes you will be able to see from various looking glasses how your prefix(es) looks like, but that won't tell you much about the actual amount of traffic you would receive when you switch services to the new addresses.

A scientific approach would be that you start doing netflow analysis _before_ switching to BGP, to discover which AS's are sourcing the most traffic to you. That is not complicated per se, the router does that with an netflow collector or look at ntop.org that works with a span port.

However unfortunately it won't be enough yet, because you would then start inferring based on your ISP topology, which would be an excessively loaded link, if any. I know of no real world tool that can help you in that.

As you can see the theoretical approach is too complicated for practical use in a small enterprise, so you are left with the pragmatic one: make a sane basic default configuration, and if any single circuit is able to carry all the traffic you currently have, there is no risk.

Then again, if both your ISP are tier one, chances are traffic will balance nicely and that's it. Remember, in BGP like in other things of life, you can't achieve the perfect balance; stability and control are enough.

Good luck!

[DANIEL WANG]

Thanks again for the great advice and experience.

I just found out we actually own two class C spaces. So if in the worst case our inbound traffic is really lopsided, I could either make the T1 link as a backup or have one Class C space preferred via the T1 and the other via the fiber, using prepend, one specific route and one more general route for each ISP.

Thanks so much for bringing ntop.org up. I need to look at it again since I don't have netflow ability on our core L3 switch. And my current 2500 Internet routers probably don't have that feature, either.

Daniel

[paolo bevilacqua]

Hi,

You don't need netflow on a router or switch to use ntop. Just span the internet traffic to a port to server and it will analyze it.