VDI slowness on WAN

[pdeming]

We have a Data Center DS3 serving 30 T1 Branches. Our Netflow Analyzer shows us that 99% of the Traffic is PCoIP UDP 4172 with a very small amount to TCP 5000 controller Traffic. I have run tests on both DS3 and T1 proving I can flood each (including the 44 meg DS3) with PCoIP (UDP 4170) traffic in both directions. Delay is 7-14ms. No errors on any Router or Switch Interfaces. We are running VMWare's Horizon View 5.3 on ESXi 5.1 Hosts. The Hosts are on Cisco UCS Blades connected to 6248 Fabric Interconnects serviced by our VSS 6509 Core over four 10 gig Fibers. The Core is connected Gig to Cisco 7206 entering the TPAC MPLS Cloud Serving our Branches. The Branches all have either Cisco 2801's or 2901's bringing in the TPAC MPLS T1's connected to Cisco 3560 PoE Switches. Extensive QoS has been implemented in both directions and detailed attention was paid to the fan out of the 44 meg QoS feeding 30 1.5 meg circuits. In that none of the circuits reach 1/2 of the Bandwidth, there is no contention, therefore QoS isn't in use other than small amounts of EF marked VoIP Traffic at 3 of the 30 Branches. The VDI Terminal Clients are WYSE/DELL model PxN that use Teradici's OS.

We also run it on the Local LAN and 2 Remote Branches connected over a Layer 2 50 meg Metropolitan LAN. Leaving the VSS 6509 Core are the 6509 Access Switches connected to WYSE/DELL model PxN that use Teradici's OS.

The user experience at the Data Center and Metropolitan LAN connected Clients is great. No video Scrolling Delay at all, Key Strokes and Mouse movements are snappy.

The user experience at remote T1 Branches is very choppy video and mouse. Almost painful. Many of the Branches only have four simultaneous Sessions and 5 Branches have as many as 9 simultaneous Sessions. The required Bandwidth per Session should be between 80 and 350k. Our Sessions use an average of 200k per Session. The Choppiness is the same, no worse, with 4 or 9 Sessions. The T1's report 250k to 350k in use of 1,536k possible at that time. On rare occasion we have seen 500k to 900k, but that was rare. Remember earlier, we Bandwidth tested each T1 and DS3 circuit to full capacity using PCoIP Protocol (UDP 4271)

How do we fix this? It works perfectly on the LAN at the Data Center. It is slow at the Branches and the Branch Circuits are not being asked to transport more PCoIP Traffic that the Circuit DOES have Bandwidth for.

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

The contributing causes for your VDI slowness might be just one or many.  Difficult to tell based on just what you've posted.

Although you've noted you don't believe you have bandwidth issues and you have "extensive" QoS, bandwidth issues for extreme time sensitive traffic can happen at the millisecond level, which is often difficult to analyze, and your "extensive" QoS might not be the optimal.  Again, insufficient information to say.

Also, if I understand what you've posted, there's a 3rd party MPLS WAN cloud you're running across, and those too can be difficult to analyze.

Is VDI branch performance ever always good, like early AM on a weekend?  Is it usually at its worse during prime business hours?  If so, then likely congestion, somewhere, is the cause of the problem.

If VDI branch performance is never good, maybe it's just the additional latency, from WAN distance and/or slow serialization.  Or, something, somewhere in the path just isn't working as it should.  (I once chased a tier 1 WAN vendor, for 3 months, that their cloud performance, for just one link, just wasn't quite what I thought it should be (about 2% less than I expected).  Or course, they said, their network was fine, but they eventually discovered a line card, with out-of-date firmware had a known problem with high burst loads.  They updated the card's firmware and then performance was as I expected.  The moral of the story, some performance issues can be very difficult to diagnose.)

Ideally it would be great if you could get one of your branches up on another T1 WAN link, and compare how VDI then performs.

[pdeming]

The Slowness doesn't matter based on time of day or even only one session from Branch.  Were you suggesting that my 7206 DS3 QoS complexity could cause latency that effects VDI?  On the DS3 end, I broken it into 30 Classes, one for each Branch and then applied 3 Queue's to each. VoIP, VDI, and class-default.

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

No, I'm not suggest QoS complexity, but the policy, itself, might not be ideal.  Again, that's only one possible issue.

What's the NPE in the 7206?  The IOS version and feature set?

What's the actual QoS policy?  (I assume the 30 branch classes are alike, so you don't need to post all 30.)

What's the IOS versions and feature sets on the branch routers?  Their QoS policy?

[pdeming]

Please do take a look.  The ISP only supports 3 Queues so I focused on VoIP and VDI and gave everything else class-default. They are honoring my EF and AF41 but will not necessarily pass af41 as a second priority. they will only guarantee the percentages. I know there is no congestion, so bandwidth percentages shouldn't be an issue but what about no second priority for VDI. Could that be the problem?  I thought that I ruled it out by classifing VDI traffic EF for a couple of days. The slowness continued. We only had one Branch on VoIp at the time and they did complain proving to me that VDI was being successfully marked EF.

=================================================================

The 7206 is running NPE-G1 and IOS 12.4(24)T3 (c7200-advsecurityk9-mz.124-24.T3)

QoS:

!-------------------- VoIP Application Class Map--------------------------

class-map match-any VOICE

match dscp ef

!-------------------- VDI Application Class Map--------------------------

class-map match-any VDI

match access-group name PCOIP-UDP-4172

match access-group name PCOIP-UDP-50002

match access-group name MMR

match access-group name USB

match access-group name ATM

match access-group name PCOIP-TCP

match access-group name PCOIP-UDP

match access-group name RDP

match access-group name RDP-HTTP-VIEW

! ------------------------ Branches--Class Map--------------------------

class-map match-all 3-elcerrito

match access-group 103

class-map match-all 5-rodeo

match access-group 105

! ------------------------ Applications--Policy Map---------------------

policy-map QoS-Appl-Prim

class VDI

set dscp af41

bandwidth percent 75

class VOICE

priority percent 20

class class-default

! ------------------------ Branches--Policy Map--------------------------

policy-map QoS-Branch-Primary

class 3-elcerito

bandwidth 1370

service-policy QoS-Appl-Prim

class 5-albany

bandwidth 1370

service-policy QoS-Appl-Prim

!-------------------- Interface--------------------------

interface Serial1/0

description MPLS to Los Angeles

bandwidth 44210

ip address 172.17.221.130 255.255.255.252

service-policy output QoS-Branch-Primary

! --------------PCoIP--for-- VDI----------------------------

ip access-list extended PCOIP-TCP

permit tcp any eq 50002 any

permit tcp any any eq 50002

permit tcp any eq 50000 any

permit tcp any any eq 50000

ip access-list extended PCOIP-UDP

permit udp any any eq 50002

ip access-list extended PCOIP-UDP-4172

permit udp any any eq 4172

ip access-list extended PCOIP-UDP-50002

permit udp any eq 50002 any

permit udp any any eq 50002

permit udp any any eq 50000

permit udp any eq 50000 any

ip access-list extended RDP

permit tcp any any eq 3389

ip access-list extended RDP-HTTP-VIEW

permit tcp any any eq www

ip access-list extended RDP-HTTPS-VIEW

permit tcp any any eq 443

ip access-list extended USB

permit tcp any any eq 32111

! ------------------------ Branches----------------------------

access-list 103 permit tcp any 172.24.3.0 0.0.0.127

access-list 103 permit udp any 172.24.3.0 0.0.0.127

access-list 103 permit tcp any 172.24.103.0 0.0.0.127

access-list 103 permit udp any 172.24.103.0 0.0.0.127

access-list 103 permit ip any 172.24.3.0 0.0.0.127

access-list 105 permit tcp any 172.24.5.0 0.0.0.127

access-list 105 permit udp any 172.24.5.0 0.0.0.127

access-list 105 permit tcp any 172.24.105.0 0.0.0.127

access-list 105 permit udp any 172.24.105.0 0.0.0.127

access-list 105 permit ip any 172.24.105.0 0.0.0.127

================================================================

Most of the Branches are 2801s with IOS 12.4(25f) (flash:c2801-advsecurityk9-mz.124-25f.bin)

QoS:

class-map match-any VOICE

match dscp ef

class-map match-any VDI

match access-group name PCOIP-UDP-4172

match access-group name PCOIP-UDP-50002

match access-group name MMR

match access-group name USB

match access-group name ATM

match access-group name PCOIP-TCP

match access-group name PCOIP-UDP

match access-group name RDP

match access-group name RDP-HTTP-VIEW

policy-map T1-EGRESS-TRAFFIC

class VOICE

priority percent 20

class VDI

set ip dscp af41

bandwidth percent 75

class class-default

interface Serial0/1/0

bandwidth 1536

ip address 172.17.221.38 255.255.255.252

service-policy output T1-EGRESS-TRAFFIC

ip access-list extended MMR

permit tcp any eq 9427 any

permit tcp any any eq 9427

ip access-list extended PCOIP-TCP

permit tcp any eq 50002 any

permit tcp any any eq 50002

permit tcp any eq 50000 any

permit tcp any any eq 50000

ip access-list extended PCOIP-UDP

permit udp any eq 50002 any

permit udp any any eq 50002

ip access-list extended PCOIP-UDP-4172

permit udp any any eq 4172

ip access-list extended PCOIP-UDP-50002

permit udp any eq 50002 any

permit udp any any eq 50002

ip access-list extended RDP

permit tcp any eq 3389 any

ip access-list extended RDP-HTTP-VIEW

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

For your queuing policy, you might try:

policy-map Sample1

class VOICE

priority percent 20

class VDI

set ip dscp af41

priority percent 20

class class-default

bandwidth remaining percent 100

fair-queue

or

policy-map Sample2

class VOICE

priority percent 20

class VDI

set ip dscp af41

bandwidth remaining percent 99

fair-queue !should be supported on your 7200 IOS version

class class-default

bandwidth remaining percent 1

fair-queue

in addition, on the 7200

policy-map QoS-Branch-Primary

class aBranch

shape average 1275000

service-policy QoS-Appl-Prim

[Seth Bjorn]

From my experiences, a single T1 line for 8 users is insufficient bandwidth with any video content beyond an office user type workload. I would check the pool settings as well as the teradici pcoip group policy settings for your pool in question. Maybe you have build-to-lossless turned off or something.

From the network side I would check MTU first from the branch side. Make sure you can ping your connection servers with the do not fragment bit enabled and be sure the MTU you are using is correct.

As far as the QOS is concerned, you really should just have UDP 4172 matched and not all the HTTP and HTTPS and other ports you have. You could be congesting your queue as you've cast a pretty wide net there. Can you post show policy-map interface ser1/0 on your 7206? Also do similar for the branch interfaces on their routers.

Here is some nice reading material from VMware's documentation (link)

[d_alva]

pdeming,

We are seeing the exact same issue with our T1 sites.  Did you ever resolve the issue?