Browsing really slow at Corp Site Internet Ok

[fhoban]

Hi All,

I have a known internet provider issue at a corporate site which is severly affecting internet access/browsing. There is a confirmed issue in the ISP backbone so certain addresses are not reachabe like 8.8.8.8, cisco.com might be available for a while, then IP changes and it's not etc.

Local IT guy has patched a laptop into the ISP router and experienced the same issues just so we could eliminate the local infrastructure. Network setup is as follows

-Failover to MPLS for all traffic and vice versa via IPSEC tunnels to nearest DC. BGP/OSPF routing

-MPLS Primary connection

-Internet Backup

-Collapsed Core/Distribution/with Cat9k VSS / access layer stacks and ASA FW

-Normal traffic flow is everything goes via MPLS with internet breakout for the site local.

-DNS Server on site with access to internet for external domains. Alternate DNS server at DC

-No Content filtering in place

-Bit Defender Anti Virus on Desktops

-Network infrastructure checked for speed/duplex, MTU, Link issues etc. and all clean. Site was working perfectly up until 2 days ago with no known network changes.

The main issue is as follows:

If circuit goes down/has issues the local internet traffic will fail over to MPLS and exit at the nearest Data Center.

When this failover is in place all now previously unreachable destinations via the local ISP are reachable via the DC internet. Traceroutes work fine etc. except browsing is still just as bad if not worse, symptoms very similar.

I've been told no policies have been pushed out to workstations/site etc.

Lots of things don't add up/make sense. The internet uses a different provider at the DC as well.

My first thoughts were Content Filter/AV or even symptomatic of MTU issues.

Phones/Tablets exhibit the same issues with no AV on them.

Users using Anyconnect VPN from the office into the DC have no issues.

I thought the DNS may have some stale entries but I've got the cache cleared locally/flushed dns on workstation etc.

Not sure what else it could be.Any insights appreciated.

Thanks

[Georg Pauwen]

Hello,

--> Local IT guy has patched a laptop into the ISP router and experienced the same issues just so we could eliminate the local infrastructure.

Is there a chance to get the configuration of the ISP router ? Actually, what does the ISP say about this issue ?

[fhoban]

Probably not unfortunately. It's in France and I'm relying on local support who have ticket open.

All the local ISP said is we have found issues in our backbone which I knew anyway yet they still proceeded to send a tech out to site yesterday to check the equipment !

I'm more concerned now why it doesn't work when routed via the DC because the local ISP should be out of the picture.

[Georg Pauwen]

Hello,

tough one. Can you post the configs that are in the path, and over which you do have control ?

[fhoban]

I have control over all local site except ISP router and MPLS which is managed. I have no control over DC, however I do know that the internet at DC is a completely different provider than local office ; originally I was suspicious it was the same ISP which might have made a bit of sense. If understand it's a tough one to pin down. If you have anything specific in mind other than what I've outlined I can provide more info.

[Georg Pauwen]

Hello,

I would start out with doing a traceroute to 8.8.8.8, and then check all the interfaces that are being traversed (sh interfaces x). Is there congestion/drops, etc...

[fhoban]

I've done that. Everything is running clean. It's better today as they're clearly doing work to rectify the issue.

Sites that are reachable/routable over the local site internet are working fine. I've done numerous tests streaming video etc.

When I shut down this link and traffic diverts across MPLS 'everything' appears reachable via ping/traceroute. So for example from a recent test www.facebook was not reachable out local internet link and you could not browse to it. Once you divert the traffic over MPLS it becomes reachabe then however once you try browse you get 'Page not displayed'. It's exiting a different ISP as well.

[Grrshell]

Would the MPLS provider be able to assist you with packet capture / troubleshooting?