Problem with throughput between two routers on a split local loop

[baskervi]

We have a scenario that has been tough to figure out. We are in the process of migrating our systems to another building, and in the interim our ISP has created a split local loop, so that we can use our public IP range at both buildings. This has been running OK for a month or so, but this week we started to have throughput issues. Traffic between the two sites becomes very slow after about 2 hours of working just fine, and when this occurs pings go from 20 msec to 500 msec when we ping from the 7206 at site 1 to the C3900e at site 2. However, both routers can ping their Internet gateway during this time and have normal response times. The interesting thing is that when we ping from the C3900e to the 7206, after a handful of pings the circuits speed back up, and traffic passes as normal. Unless we run a continuous ping from site 2 to site 1, we inevitably run into the throughput problem multiple times during the day.

We've disabled ip cef globally or ip route-cache cef on an interface, but this didn't help. We've also contacted the ISP providing the split local loop, but they are not seeing problems on their end. The processors on both routers are less than 5%, and there are no low memory issues on either router. The routers were rebooted 2 days ago, and neither router interface has thrown up a single error. 

The 7206 is running 12.3(3)g, and the 3900e is running 15.2(4)M4.

Does anyone have an idea what we might check out? Thanks

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Can't say what your issue is, but I will say, if I only had a nickle for every time a service provider said their stuff is fine, I would be already retired. 

[baskervi]

I've been doing this long enough to have already suspected this, but I was hoping someone might point out something I've overlooked. Thanks for the reply.