Cisco Nexus 3548 span configuration unordered network packets

[rehadiri]

Hello,

We have the following setup, when we start dumpcap logs on Host1 interfaces everything seems normal. But when we move logging to SPAN ports on 1/1 and 1/3 some packets gets logged in wrong order. For example TCP acks logged before requests. Logged packets are the same but some packets are disordered on SPAN ports. What can be the reason of this situation. Thanks

[rehadiri]

I wanted to give more details;

There are other hosts on Cisco 3548 like Host1, communication with Host2 and Host3 . For this reason Capturing network traffic on Host2 and Host3 ports seems more feasible.

To understand if there is a lack of resource on the Cisco 3548 , we have stopped monitor session 2, which has more traffic but the result is the same . There are disordered network packets on SPAN. We can see this situation on Wireshark as labeled black lines. Wireshark shows as errors but actually not. 

[Georg Pauwen]

Hello,

the default span-treshold on the 3548 is very low, 'due to the default SPAN threshold limit being low, t cannot handle the burst traffic for SPAN.

Try and increase the value, and check if the problem persists...

hardware profile buffer span-threshold <xx>

[rehadiri]

Thank you for you suggestion, i did "hardware profile buffer span-threshold 10" but unfortunately the result is the same.  

The same packets are in order on the Host1 but are disordered on SPAN ports. The hardware of the logging server and Host1 is the same. 

[Georg Pauwen]

Hello,

10 is still very low. I am not sure what the highest value is, but try something like 200.

[rehadiri]

It can be 2-100 , max value is 100. 

I am also checking show monitor drops session 1 & 2

There is no packet drop

[Georg Pauwen]

Hello,

can you post the pcapng file ?

[rehadiri]

Hello,

Here are the zipped pcap files . Wireshark shows black labeled lines as error , but actually these are not errors , i think the only problem is that the order of logged packets are incorrect.

[rehadiri]

I think that the source of the problem may not be the Cisco 3548. It can be about dumpcap timestamping. Dumpcap is using host timestamping and it may not be as sensitive as required so when the timestamping of network packet is not correct the order of the packets  are wrong too. 

Do you have an idea about this ?

Thank for your help

[Georg Pauwen]

Hello,

that might very well be that case. The Nexus 3548 has extremely low latency (I think they call it Algo Boost), and whatever packet capture software you are using, it might not be able to keep up with that speed (<200 nanseconds latency).

What mode is the switch in, Normal, Warp, or Warp SPAN ?

[rehadiri]

Hello,

İts normal SPAN . We tried WARP SPAN , because of lack of resource we couldnt make it the WARP SPAN with another SPAN together. We could make the WARP SPAN without any other SPAN session.

I dont know may be after increasing buffer span-threshold to 10 it may work .

Thanks for your help

[rehadiri · Accepted solution]

Hello,

At the end we understand the problem. Its not related to Cisco switch . Its about packet timestamping of the server host. Dumpcap packet capture is using the server software timing for timestamping of the packets and its not stable. The result of this  imprecise timing creates unordered packets in the log. So we replaced the NIC with a network card which can do hardware timestamping and  we use dumpcap option " --time-stamp-type=adapter_unsynced " and every packet gets ordered as expected.

The NIC which can't do dumpcap hardware timestamping is intel XL710 , the nic which can do dumpcap hardware timestamping is Mellanox Connectx-3 . Its about the model of the card, nothing about to Intel. There other intel NIC's that can do hw timestamping such as i350 or X550 .

Thanks

[Georg Pauwen]

Hello,

extremely interesting ! Thanks for sharing these details.