Out of order packer investigation

[Tod Larson]

We have the following wan setup...
Site 1 - asr1006 with ds3 link to ISP
Layer 3 VPN from ISP
Site 2 - 3945 isr with ds3 link to isp

Users experience problems with streaming video between sites. For testing I connected a laptop directly to each site router and ran jperf between the two. In the jperf output I see lots of out of order packets. I think that must be causing the video problems. However, I can not recreate the out of order packets using IP SLA on the routers.

How might I test for out of order packets on the routers? Thank you.

[Edison Ortiz]

You can configure Netflow and export the captured information into a collector.

Within the collector, you can verify the packet timestamp.

Regards,

Edison

[Tod Larson]

I wasn't aware that netflow could monitor packet by packet statistics.  Can you explain further?

Here are the configs I have now...

--- sla config

ip sla 200

udp-jitter 1.1.1.1 16384 num-packets 1000

frequency 30

ip sla schedule 200 life forever start-time now

-- netflow configuration

ip flow-cache timeout active 1

ip flow-export version 9

ip flow-export destination 2.2.2.2 9999

Thank you.

[Edison Ortiz]

Sure,

NetFlow Analyzer accounts for the following details from the NetFlow Packets :

     Source and destination IP address

    Input and output interface number

    Source and destination port number

    Layer 4 Protocol

    Number of packets in the flow

    Total Bytes in the flow

    Time stamp in the flow

    Source and destination AS

    TCP_Flag & TOS

Your global network configuration looks fine but also you need to enable netflow in the interface.

Regards,

[Tod Larson]

I'm using what's up gold as my netflow collector. I see data rates by protocol but nothing that looks helpful to me.

I don't understand how netflow would help since it does not know transmittal time from the source. I feel like I'm missing something here.

[Edison Ortiz]

I guess it depends on the collector but time stamp is definitely collected.

[Pavel Bykov]

Hi Tod,

For out of order packets you'd need burst of traffic with very little spacing in between.

Something like ~10-20 ping echo requests with 0 timeout.

You won't see the reply in the CLI, because timeout is 0, but you could capture that with embedded packet capture.

That isn't really a good test though, because you've already seen that there is reordering with your test from the laptop. So the next step would be to go hop by hop and see where reordering is happening (i.e. per packet load sharing, combined with latency difference on two circuits). There is no per-packet load sharing, right?

Are you looking for a continuous test for the future, or are you looking to fix the problem?

[Tod Larson]

There is no load per-packet load sharing. (unless the ISP is doing it)

I'm looking to fix the current problem.

I'm unable to see how netflow can help since it doesn't know the packet transmit time or what order to expect the packets in (as far as I can tell).