Need Help with ISP and BGP - Certain Prefixes We Advertise Are Slow

[jonathanw84]

Hello,

We have a very standard setup: We have two ISPs for internet, each one connects to their own ASR1001X, and those ASRs are connected via iBGP for route sharing and also have an HSRP VIP between them. We advertise the same 5 prefixes out to each ISP. ISP1 everything works great, ISP2 is where we see extremely slow download speeds on 4 of the 5 prefixes. 1 of the prefixes works as expected. 

BGP configuration is the same to both ISPs but when the certain prefixes route over ISP2, we see the issue. We've tried every combination possible. Shutting down BGP on ISP1 to route on ISP2 only and the issue is there. If we shut down ISP2 and route only on ISP1, everything works as expected. We've also tried flipping HSRP between the routers to ensure there is not an issue with inter connectivity between the two and this does not change anything either way.

We've checked everything we can think of from a L2 and L3 perspective in our environment and cannot find any issue. The issue definitely seems to be with ISP2 but they say everything is fine.

Any suggestions out there?

Thanks!

[paul driver]

Hello

Without looking at your configuration its hard to understand the cause of your issue, However when you have dual ISP peering you need to make sure you are not becoming a transit path for each ISP, so just advertsie ONLY you local prefixes to each ISP.

Example:
ip as-path acess-list 1 permit ^$

rtr1
router bgp xxx
neigbour < isp 1> x.x.x.x filter-list 1 out

rtr2
router bgp xxx
neigbour < isp 2> x.x.x.x filter-list 1 out

[jonathanw84]

Thanks Paul. Yep, we are doing exactly that with a prefix-list and are positive that only our 5 prefixes are being advertised out to each ISP.

ISP2 came on site tonight and bypassed our firewall and directly into their NID and went prefix by prefix and everything looked clean (we've been doing iperf and other Linux based tests to show the issue). So now we're thinking it could be the ASR that ISP2 is connected to, but not sure why. All interfaces are clean, it's licensed for 20 gigs. Really odd because this is a standard configuration we run at our other data centers and it's very simple.

One thing to note is this only impacts download speeds. Upload is fine. And this trend followed us during iperf testing where a remote iperf server could push to our client (so going in) and speeds we're horrendous while on ISP2. But fine on ISP1.