New WAN network rollout - performance issues

[Ableton34]

Hi all.

Wondering if you can help. Where I work we have just finished rolling out a new WAN network to replace the previous one with a new sole ISP, previously it was a mixed ISP network with different flavour circuits connecting around 140 sites into our core.

Since we have migrated all the sites, we are getting an increasing number of them complaining about performance of all applications.

To give you some background, we are using BGP to connect CE and PE routers using P2P serial addresses, the BGP advertises the WAN ranges into the ISP cloud, ISP then sends all traffic back to our core network again using BGP and BGP advertises our internal LAN server ranges back out into the cloud.

We have a core and a DR site which have 2 pairs of resilient ASR routers which load share the traffic from various WAN sites inbound and out.

We also have an issue with the ISP in that their standard MTU size allowed in their cloud is 1492. I know this isnt standard and we have raised the issue, they cannot change this so we have had to have the ip mtu command on our WAN edge routers in order that there are no issues with the server traffic. This seemed to solve the original problem of nothing working properly.

Anyways, the complaints are getting more and more all the time and we can see  no network issues at all to the affected sites. Testing with ping/traceroute to check routing issues, no packet loss or high response times seen, a very healthy over all core to end site link. We just cannot work it out and have been scratching our heads for weeks.

Server team and most others are blaming the new network, with good reason but the network engineers just cannot find anything wrong! Traces have been done and all TCP conversations take place normally, IP SLA probes do not show anything untoward and our network monitoring tools arent seeing anything.

Has anyone experienced anything like this before, any ideas anyone as we have run out of them!

Thanks

[Ableton34]

Thanks for this.

I have already asked him if the MSS adjustment should be added to the branch routers along with the IP MTU command and he says no. Only required at the WAN routers at HQ.

I didnt know enough about its usage and had read online that you need to configure both together.

[Ableton34]

Thanks Georg, however my peer says no to both ends only HQ. If I knew enough to safely disagree with him I would but I think he will be the authority on it so will go with him for now but if the performance issues dont improve I will suggest we configure it on the branch routers also.

[Georg Pauwen · Accepted solution]

Hello,

both ends is good practice, but as Joseph suggested, HQ is probably sufficient.

I am very curious if that adjustment makes a difference...

[Ableton34]

Thanks.

I'm sure it wont have any effect on our performance issues, just clutching at straws now really in lieu of any real answers to the problem.

[Joseph W. Doherty]

If your video solution doesn't use TCP, MSS adjust won't help.

[Joseph W. Doherty]

BTW, rather than tuning buffers manually, you might see if your platform supports the buffers tune automatic global command.

[Joseph W. Doherty]

The MSS adjustment is often a key component in dealing with reduced performance when dealing with an Ethernet MTU less the the standard 1500.

That said, it only helps with TCP traffic, other IP traffic can still be impacted by the smaller MTU.

[Ableton34]

Thanks.

The network architect has asked for it to be placed on the outside interface facing the PE/WAN cloud. Is this correct or should it be placed on the LAN switch facing interface?

I am intrigued that he didnt specify to use this command earlier when we applied the MTU size of 1492.

[Georg Pauwen]

Hello,

outside interfaces should be ok for the adjustment.