WAN Testing

[Navneet_singh]

Hi,

We are rolling out a new application on our Client network, and are currently planning to upgrade the BW on the WAN links from all the spoke sites. All sites connect to the Datacenter (where the app server would be placed) through MPLS links.

The APP team will take a sample site, and would be doing load testing to simulate traffic on the WAN links  for about 10 users.

I have a few queries in regards to WAN testing

1) Should testing ideally be carried out during production hours, as this might give you a fair bit idea on the actual load the WAN link experiences with the new app traffic? Or should it be carried out Non-business hours, and the total amount of BW consumed (For the new APP) can be used to later add onto the normal data which flows across everyday.

2) For how long ideally the testing should be carried?

3) We have a Netflow analyzer tool to check on the WAN statistics. Is there anything else we need to check in terms of the WAN Queuing strategies etc?

4) Is there any rule which can be used to identify how much BW upgrade is needed?

Would really appreciate your help on this.

Cheers
Navneet

[esomarriba · Accepted solution]

Hi Navneet,

Things I would do:

1.- Get a baseline of bandwidth use of the spoke sites. If the spoke sites have a T1, what’s the average use?

2.- What kind of APP are you running? Is this application WAN optimize?

3.- If you have budget constraints (like many companies in the US) J you should consider blocking access to some sites using some kind of content filtering.

4.- I recommend do carry out all testing on NON-Business hour, that will make your testing more accurate. Run the test for 7 days and compare day-to-day analysis.

5.- Make sure before you start any testing on the WAN MPLS Link to test the link first, make sure you are not losing packet, and latency is decent. If packet loss is present call the ISP and fix this first before any APP Team gets involve, for political reasons.

6.- If the APP is mission critical, I recommend to implement QoS end-to-end. You should have at least 4 classes of traffic.

7.- Once you have the baseline for both type of traffic, (Business Hours + APP Load testing) make sure you are using 75% of total available bandwidth.

8.- If you super exceed the T1 in this example, I will definitely recommend upgrading your BW.

9.- You do have option REMEBER, if the MPLS Link is very stable, have the APP use the MPLS and get some cheap ADSL, DSL, or CABLE connection to offload WEB, EMAIL, non-critical using PBR or route-maps.

[vmiller]

All good ideas, make sure you understand how your provider is going to handle marked traffic.

At least in the States, there are a number of service plans on the provider side, you want to make sure

that any QOS you implement will be handled as you expect by MPLS.

[esomarriba]

Thanks vmiller,

Some if not all ISP will charge you extra for QoS marking. So don't be surprise on this one.

HTH,

Elyinn.-

[Navneet_singh]

Thanks esomarriba and vmiller for your response.

In regards to your queries to the points mentioned,

2) The APP here is a software used to automate all the hospital related tasks - esp patient counselling etc.

4) We have been given a timeframe of only 1 hour to carry out the testing during Non-business hours. I hope we could collect enough data from this.

6) APP is quite critical, and we are under discussion with the MPLS SPs to provide us QoS services.

I also needed to understand how to exactly check the BW utilization from a router's WAN interface? How should the input and output rate be looked at?

Cheers

Navneet

[sean_evershed]

Hi,

If you are upgrading the WAN links to the spoke, are you going to upgrade the link to data centre as well?

Good luck with the testing but I think that doing the testing during non business hours will give you unrealistic expectations of the actual performance of the application. I think the key question to answer is how it performs when the bandwidth to the site is high.

Most of the organisations that I have worked for due to the expense only pay for a minimal amount of WAN bandwidth and so consumption is high on average.

I'd also be curous to know how the team plan to simulate all the traffic consumed by an average user during the test. This should include things like browsing the web, software updates, emails, access to file shares, domain authentication etc etc.

I'm not sure if you VOIP at these sites as well but I would also suggest that a number of test phone calls are made at the same time as the application is tested. If the application is bandwidth intensive and there are underlying problems with your QoS policy then the appliation will adversley impact the quality of VOIP calls.

Please remember to rate all posts that are helpful

[vmiller · Accepted solution]

To the question of checking bandwidth, The simplest is a show interface command for the interface to the MPLS network.

Keep in mind, once you are in the cloud, you have no idea how your data is being forwarded, Your B/W numbers are only for

the local loop between you and the provider. There are all sorts of free tools available to monitor and measure this stuff,

given the critical nature of the application I would make that a priority.

Get your service provider in for a visit ASAP. Gently demand a presentation of their offerings for QOS support. Its not a matter of

if, its when.

read the following: http://shlang.com/writing/tcp-perf.html#

If you aren't the workstation support person, pass it off to them. you may want to consider having two test workstaions, so you can

expirement with the tcp window settings. Trust me it makes a difference.

[Navneet_singh]

Thanks a lot for your inputs. I had one more query in regards to WAN utilization. As per Cisco, Optimum Network Utilization for a WAN link is 70%. While looking at the Network reports, how long (timeframe) should the utilization be above 70% to suggest for an upgrade. I see quite a few times when the Util on few sites goes to about 80% but only for a few seconds.

Also, a weekly average report would suffice or rather we should be looking at a monthly average to come to any conclusions?

I would come back to you guys for more guidance if needed. Appreciate all your help here.

Cheers

Navneet

[Navneet_singh]

Hi All,

To further add to my query above, Iam attaching 3 BW util reports for reference. These have been taken with a time interval of 1, 5 and 15 min average. While recommending upgrade, we should always be looking at the peak point in the graph as reference. In 1 min average, the max utilization is 96.98%, but the 95th percentile is only 1.48 %.

Please suggest as to how best derive my suggestions looking at these graphs.

Appreciate your help here.

Cheers

Navneet

[esomarriba]

Hi navneet,

Hope you are doing well my friend. I don't see any congestion in the graphs. Is this business hours or non-business hour traffic? If it's BH, you should not advice any BW upgrade, I would recommend implementing QoS to ensure a good % to this application.

A saturated WAN Link is when you have 95% in use on average. That's a busy link. (depends on best practices, we have multiples school of thought on this one... If you ask business/management/finance people, they would recommend 95% to request BW upgrade, if you ask network techs/arch they are going to tell you 70-75%.)

So if you are at the point where you don't have any congestion on your WAN Link and need to ensure IP Quality for this type of traffic, right? If so, make sure you understand the behavior of this application, work together with the application team, are they using UDP or TCP or both. Ensure that all UDP get's deliver first, and TCP later... Remember TCP can send the application layer a ACK.

Break down all the components and factors of the application and network layer and make sure everything is on a complete harmony. Install netflow, application sniffers, I think SolarWinds has very good tools and they are free to use, this way you understand 100% the application and your network behavior before this goes to production.

My recommendation is to get as much information possible that way you minimize your risk of future issues.

HTH,

Elyinn

Message was edited by: esomarriba

[Navneet_singh]

Hi Elyinn,

Thanks for your inputs. The graphs I have attached are from a sample site for reference. But I agree with you that QoS would be a better option on links that are not saturated.

As you said, 70% utilization should be a reference point. Does that utilization be 70% for a sustained period of time, or can we suggest upgrades even if there are spikes even for a minute or so? (provided we have all QoS in place). What is the best way to check utlization? Is it better to check with a time interval of 1, 10 or 15 min average? The reason iam asking is because as you saw from the graphs, the Max utilization varies a lot in these time intervals.

CheersNavneet

[esomarriba]

Hi Navneet,

Sorry for the delay, You should run a 7 day usage report on the site, if the site is using 70% of BW, you should upgrade the link. Of course if the budget permits.

Will this application use any kind of EF46 traffic?

HTH,

Elyinn.-

[esomarriba]

Quick question, do you have the baseline for this application to determine the

QoS requirements?

[Navneet_singh]

Hi Elyinn,

Thanks for your reply. There is no EF46 type of traffic here. The Application in question would be TCP based, and we would be using AF31 services probably, and are currently talking with the SP on the same.

Cheers

Navneet

[esomarriba]

so now, what you need to do is start categorize all type of traffic in this link. Maybe your ISP give you AF31, AF32, AF33 to play with. This information can be gather using netflow, this will give you an in-depth view of what's on your link.

HTH,

Elyinn.-