Output drops - performance issue with PCoIP

[ross.wiggins]

• Multi-site network spanning various locations in UK
• Site A has 10Mbps connection to site B
• Site B has 1Gbps connection to data centre (site C)
• Site A is a small office of less than 5 users.  They use Thinclients and PCoIP to establish VDI sessions to server infrastructure hosted in data centre
• Site A has been in operation for long time with no issues
• Recently the end-users have been reporting their VDI sessions are freezing.  This happens to all users at the same time.  Frequency varies and there is no pattern (from time perspective)
• Utilisation across WAN connection between site A & site B is low.  <20% utilisation.
• Big percentage of companies user base use the same PCoIP sessions.  They are spread over various sites.  None of those users report the same issue.  It's specific to users in site A
• QoS policy in data centre used access-list to mark PCoIP traffic with DSCP AF31.  WAN interfaces between router in site A & site B originally contained QoS policy to classify traffic based on DSCP value and then allocate 'bandwidth' rate of 5000kbps.  This has been in operation for sometime, without issue.
• It would seem there is an upstream issue with this setup and the DSCP markings aren't being seen on the WAN router at site B.  Therefore I have changed QoS policy on WAN router in site B to classify based on the same IP access-list as used in the data centre.
• Originally (before the QoS policy change) a continuous ping to an end-users Thinclient and the WAN IP in site A, would show packet loss/timeouts at the exact moment the end-users experienced their VDI sessions freezing.
• WAN interface in site B shows 'output drops' (thousands, hundred of thousands)
• Policy-map for WAN interface in site B shows all drops occurring for the PCoIP policy
• WAN interface in site A is mainly clean, with very small amount of drops (ten's)
• Since the QoS change in site B, the user experience has been better but not 100% back to how it used to be.
• WAN interface in site B still shows pretty much the same level of 'output drops' but the same continuous pings are better (more consistent with less periods of packet loss/timeouts and they do not seem to correlate with the time the users experience the issue)

QoS policy on site A & site B:

Extended IP access list QoS-ACL-PCoIP
10 permit tcp any any eq 4172 (1058 matches)
20 permit udp any any eq 4172 (63046564 matches)
30 permit tcp any any eq 32111 (24296 matches)
40 permit tcp any eq 4172 any (121 matches)
50 permit udp any eq 4172 any
60 permit tcp any eq 32111 any (13 matches)

Class Map match-all PCoIP (id 3)
Match access-group name QoS-ACL-PCoIP

Policy Map WAN_QoS_10Mbps_CCT
Class PCoIP
bandwidth 5000 (kbps)

Current interface stats for WAN interface site B:

FastEthernet0/1/0 is up, line protocol is up
Hardware is FastEthernet, address is 30f7.0d68.341c (bia 30f7.0d68.341c)
Internet address is X.X.X.X/XX
MTU 1500 bytes, BW 10000 Kbit/sec, DLY 1000 usec,
reliability 255/255, txload 12/255, rxload 3/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:04, output 00:00:00, output hang never
Last clearing of "show interface" counters 23:30:16
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 28185
Queueing strategy: Class-based queueing
Output queue: 0/1000/0 (size/max total/drops)
5 minute input rate 130000 bits/sec, 126 packets/sec
5 minute output rate 472000 bits/sec, 125 packets/sec
4542987 packets input, 670052544 bytes
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog
0 input packets with dribble condition detected
5037823 packets output, 2555086997 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

Policy-map stats:

FastEthernet0/1/0

Service-policy output: WAN_QoS_10Mbps_CCT

Class-map: PCoIP (match-all)
3723791 packets, 2454651292 bytes
5 minute offered rate 362000 bps, drop rate 0 bps
Match: access-group name QoS-ACL-PCoIP
Queueing
queue limit 64 packets
(queue depth/total drops/no-buffer drops) 0/28185/0
(pkts output/bytes output) 3695606/2429642222
bandwidth 5000 kbps

Do you have any suggestions on potential cause?

Site A router = Cisco 2821

Site B router = Cisco 3845

[Georg Pauwen]

Hello,

try and change the default queue limit (which is 64) to e.g. 512:

Policy Map WAN_QoS_10Mbps_CCT
 Class PCoIP
  bandwidth 5000 (kbps)
  queue-limit 512
  random-detect

[ross.wiggins]

I had actually tried increasing the queue-limit and ran this all of yesterday (to 128) and the user's advised yesterday was the best day they'd had since the issue started.  I didn't use random-detect though.

I have removed the increased queue-limit today as I wanted to see if it made a difference or if yesterday just happened to be a better day anyway.

There was still output drops on the site B WAN interface (with the increased queue-limit).

In the meantime, can you advise if there are any commands etc that can be used to help determine that it is in fact the queue depth that is the problem?

Or by nature of the fact that there are output drops, does that naturally point to a queuing issue?

I'd just like to understand (technically) why this is happening and (even though it hasn't been tried with a 512 limit yet or with random-detect) why that will resolve it.....

[Georg Pauwen]

Hello,

output drops usually means that there is more traffic received at the input than can be processed at the output. So increasing the queue size can help, however, you have a 10MB link, which nowadays could be considered ultra low bandwidth. Also, it looks like you have set speed of the FastEthernet interface to 10MB manually, make sure that is configured at the other end as well, or better yet, change both ends to 100MB (or speed auto/duplex auto) if possible.

For a more in depth technical overview of output drops, refer to the document below:

http://www.cisco.com/c/en/us/support/docs/routers/7200-series-routers/110850-queue-limit-output-drops-ios.html

[Joseph W. Doherty]

Utilisation across WAN connection between site A & site B is low.  <20% utilisation.

Over what time period?

Queue drops happen in the millisecond time range and often utilization usage is measured across minutes.

Perhaps there's some short term burst of traffic either in the PCoIP class or other classes that cause transient congestion leading to your packet drops.

With PCoIP, you do not want to delay (i.e. queue) it or drop (RED) it, you want to insure it has the bandwidth and priority it needs.  (NB: I recall PCoIP documentation suggests treating it like VoIP bearer traffic.  Unfortunately, if you do, it will directly compete with such VoIP traffic.  So, I suggest treating it not quite as well as VoIP, but as close as possible.)

You might try increasing the bandwidth allocation to the PCoIP class.  You might also try, if using a HQF variant of CBWFQ, enabling fair-queue in the class.  (Doing the latter, may not decrease overall class drops, but if there's one or a couple of flows congesting the class, it will keep them being as adverse to other low bandwidth flows in the same class.)

You might also try decreasing the tx-ring-limit on the egress queue to avoid its FIFO queuing impact.