ā08-10-2022 02:05 AM
Hello,
We have the following setup, when we start dumpcap logs on Host1 interfaces everything seems normal. But when we move logging to SPAN ports on 1/1 and 1/3 some packets gets logged in wrong order. For example TCP acks logged before requests. Logged packets are the same but some packets are disordered on SPAN ports. What can be the reason of this situation. Thanks
Solved! Go to Solution.
ā09-03-2022 09:12 AM
Hello,
At the end we understand the problem. Its not related to Cisco switch . Its about packet timestamping of the server host. Dumpcap packet capture is using the server software timing for timestamping of the packets and its not stable. The result of this imprecise timing creates unordered packets in the log. So we replaced the NIC with a network card which can do hardware timestamping and we use dumpcap option " --time-stamp-type=adapter_unsynced " and every packet gets ordered as expected.
The NIC which can't do dumpcap hardware timestamping is intel XL710 , the nic which can do dumpcap hardware timestamping is Mellanox Connectx-3 . Its about the model of the card, nothing about to Intel. There other intel NIC's that can do hw timestamping such as i350 or X550 .
Thanks
ā08-10-2022 02:18 AM
I wanted to give more details;
There are other hosts on Cisco 3548 like Host1, communication with Host2 and Host3 . For this reason Capturing network traffic on Host2 and Host3 ports seems more feasible.
To understand if there is a lack of resource on the Cisco 3548 , we have stopped monitor session 2, which has more traffic but the result is the same . There are disordered network packets on SPAN. We can see this situation on Wireshark as labeled black lines. Wireshark shows as errors but actually not.
ā08-10-2022 10:09 AM
Hello,
the default span-treshold on the 3548 is very low, 'due to the default SPAN threshold limit being low, t cannot handle the burst traffic for SPAN.
Try and increase the value, and check if the problem persists...
hardware profile buffer span-threshold <xx>
ā08-10-2022 11:55 PM
Thank you for you suggestion, i did "hardware profile buffer span-threshold 10" but unfortunately the result is the same.
The same packets are in order on the Host1 but are disordered on SPAN ports. The hardware of the logging server and Host1 is the same.
ā08-11-2022 12:03 AM
Hello,
10 is still very low. I am not sure what the highest value is, but try something like 200.
ā08-11-2022 01:05 AM
It can be 2-100 , max value is 100.
I am also checking show monitor drops session 1 & 2
There is no packet drop
ā08-11-2022 11:56 PM
Hello,
can you post the pcapng file ?
ā08-12-2022 04:45 AM
ā08-12-2022 12:22 AM
I think that the source of the problem may not be the Cisco 3548. It can be about dumpcap timestamping. Dumpcap is using host timestamping and it may not be as sensitive as required so when the timestamping of network packet is not correct the order of the packets are wrong too.
Do you have an idea about this ?
Thank for your help
ā08-12-2022 11:22 AM
Hello,
that might very well be that case. The Nexus 3548 has extremely low latency (I think they call it Algo Boost), and whatever packet capture software you are using, it might not be able to keep up with that speed (<200 nanseconds latency).
What mode is the switch in, Normal, Warp, or Warp SPAN ?
ā08-12-2022 12:13 PM
Hello,
Ä°ts normal SPAN . We tried WARP SPAN , because of lack of resource we couldnt make it the WARP SPAN with another SPAN together. We could make the WARP SPAN without any other SPAN session.
I dont know may be after increasing buffer span-threshold to 10 it may work .
Thanks for your help
ā09-03-2022 09:12 AM
Hello,
At the end we understand the problem. Its not related to Cisco switch . Its about packet timestamping of the server host. Dumpcap packet capture is using the server software timing for timestamping of the packets and its not stable. The result of this imprecise timing creates unordered packets in the log. So we replaced the NIC with a network card which can do hardware timestamping and we use dumpcap option " --time-stamp-type=adapter_unsynced " and every packet gets ordered as expected.
The NIC which can't do dumpcap hardware timestamping is intel XL710 , the nic which can do dumpcap hardware timestamping is Mellanox Connectx-3 . Its about the model of the card, nothing about to Intel. There other intel NIC's that can do hw timestamping such as i350 or X550 .
Thanks
ā09-04-2022 01:05 AM
Hello,
extremely interesting ! Thanks for sharing these details.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide