IP SLA high packet loss from monitoring, CLI shows high tail drops?

[pdub206]

We use SolarWinds to monitor our IP SLA operations.  I've created several UDP VOIP jitter operations similar to:

ip sla 40001
udp-jitter 10.10.10.10 32768 source-ip 10.11.11.11 codec g711ulaw
threshold 1000

These operations work great, and we have ip sla responder enabled on the opposite side.  Recently we noticed a high amount of packet loss in our SolarWinds graphs, but I'm not sure how to correlate that to a statistic on the CLI.  This is occurring on multiple paths leaving our voice gateway, so it does not appear to be a single link causing the problem.  A statistic output indicates:

VGW#show ip sla statistics 40014
IPSLAs Latest Operation Statistics

IPSLA operation id: 40014
Type of operation: udp-jitter
Latest RTT: 53 milliseconds
Latest operation start time: 12:44:30 GMT Wed Sep 14 2016
Latest operation return code: OK
RTT Values:
Number Of RTT: 676 RTT Min/Avg/Max: 51/53/70 milliseconds
Latency one-way time:
Number of Latency one-way Samples: 0
Source to Destination Latency one way Min/Avg/Max: 0/0/0 milliseconds
Destination to Source Latency one way Min/Avg/Max: 0/0/0 milliseconds
Jitter Time:
Number of SD Jitter Samples: 675
Number of DS Jitter Samples: 675
Source to Destination Jitter Min/Avg/Max: 0/1/6 milliseconds
Destination to Source Jitter Min/Avg/Max: 0/1/17 milliseconds
Over Threshold:
Number Of RTT Over Threshold: 0 (0%)
Packet Loss Values:
Loss Source to Destination: 0
Source to Destination Loss Periods Number: 0
Source to Destination Loss Period Length Min/Max: 0/0
Source to Destination Inter Loss Period Length Min/Max: 0/0
Loss Destination to Source: 0
Destination to Source Loss Periods Number: 0
Destination to Source Loss Period Length Min/Max: 0/0
Destination to Source Inter Loss Period Length Min/Max: 0/0
Out Of Sequence: 0 Tail Drop: 324
Packet Late Arrival: 0 Packet Skipped: 3182
Voice Score Values:
Calculated Planning Impairment Factor (ICPIF): 1
MOS score: 4.34
Number of successes: 37
Number of failures: 0
Operation time to live: Forever

Cisco notes: "Tail drops are an indication that the IP SLA device is stressed beyond its capacity. Reduce the number of operations, packets per second speed by increasing the packet size of the probes and monitor the results."

We have only a few SLA operations on this router (ten total running every 60 seconds), and cpu is holding steady around 10-15%, memory at 40%.

So my question is, would tail drops be translated into packet loss in this scenario? The packets skipped counter has not incremented since I began checking this particular operation.

[AFROJ AHMAD]

Hi Patrick,

would tail drops be translated into packet loss in this scenario? No , as far as I know

Note for packet skipped issues, the issue is almost certainly local to the source, the responder won't be involved

you can create 2 new operations or you can apply these suggestions:

1. One with reducing the number of packets.
2. The other one with increasing the time out and threshold .

If any packet cannot be sent due to an internal or unexpected error, or because the timer wheel slot containing the packet is missed, it is counted as Packet Skipped.

Thanks-

Afroz

***Ratings Encourages Contributors ****

[pdub206]

Solarwinds indicates that the packet loss is due to the packet skipped counter rather than the actual packet loss of those packets which were sent.  It seems strange to me that packet skipped would be considered as loss, as they weren't even sent in the first place.  Originally I had thought the polling of packet loss would be specific to the actual loss counters, but I guess that shows me that I should assume nothing :D

I tried your suggestion of creating new operations with higher time out and threshold.  Oddly enough as soon as I created those operations, the original operation began to see significantly less packet loss (on the order of 90% reduction).  I have a feeling that there may be some bugs in the Cisco IOS XE code we are running, 3.16.02S.  The new operations also saw loss, but it was also significantly less.

I did not try the operation with fewer packets being sent, as various IP SLA documentation on how to do so indicated that those variables should not be changed.  I'll give that a shot as well and see if there is any change.