Multiple ISPs, poor performance to specific hosts
At my company we have recently added a link to a different ISP for redundancy. At the same time, in order to control and propagate routes properly we started using eBGP between both ISPs (formerly our sole ISP advertised our routes via static routing for us).
Since we have gone to two ISPs and using BGP, performance to specific web sites has been very on a per session basis but only in the downstream direction. Going to a popular website shows downstream throughput to be around the 300 kbps range and upload speeds in the 12000 kbps range. Both circuits bandwidth utilization is working as expected, but individual users are experiencing very slow throughput to specific websites. For example, at work I tried downloading an 8MB file from cisco.com and it took 45 minutes. At home, I downloaded the same file in 7 seconds, then transferred to my PC at work in 3 minutes. The connection to ISP 1 is a 25mb DS3 and the connection to ISP 2 is a 20mb Ethernet.
My configuration is such:
ISP 1 is connected via serial to a Cisco 3845 running IOS 12.4(10a) with sho ver reporting this for memory:
Cisco 3845 (revision 1.0) with 484352K/39936K bytes of memory.
ISP 2 is connected via ethernet to a different Cisco 3845 running IOS 12.4(10a) with sho ver reporting this for memory:
Cisco 3845 (revision 1.0) with 484352K/39936K bytes of memory.
Both routers are receiving the full BGP routing table from their respective ISPs.
On the inside, each router is connected to an ethernet switch. Also plugged into that Ethernet switch is our PIX535.
At the current time, HSRP is running on the inside interfaces for redundancy, so all outbound traffic exits through ISP 1. Both routers are advertising the same local routes to the internet, so inbound traffic enters through both ISPs depending on best route from the source. (Soon I will implement iBGP internally to choose best route to exit the my network but want to get this weirdness fixed first).
The only pattern that appears is that the hosts that are extremely slow appear to come inbound via ISP 1, but it's a very popular ISP so that pattern may just be a coincidence.
For troubleshooting, I've moved all my traffic to ISP 1 (shut down ISP 2) and tested throughput and it's extraordinarily bad. Then I've moved all my traffic to ISP 2 (shut down ISP 1) and tested throughput and it's also extraordinarily bad. I actually get a little better, but still good throughput when both ISP connections are taking traffic. In fact, when operating on only one ISP, my download speeds through the popular speed test website show 300 kbps down and 12000 up. Then with both ISPs I was getting 2000 kbps down and 12000 kbps up for a while. Then a few hours later, without cause, I was getting download speeds of 10000 kbps and 12000 kbps up. Meanwhile, the poor performing websites (www.cisco.com, www.microsoft.com, www.sun.com) continue to perform very poorly (the speed test website inbound traffic comes from ISP 2).
I may be rambling in this post as I'm extremely baffled by this anomaly.
Has anyone else ever experienced an issue such as this? Please post if so. If you need more information, I'd be happy to provide it. Any help is greatly appreciated.
Thanks!
Jon
