Re: ExanicX25 exanic/exact-capture missing application Tx packets

dahiya · ‎11-07-2022

Hello,

I am new here, please move post to the relevant Board if I have chosen wrong one.

My (tcp) code is using vanila posix socket api, i.e. no libexanic or any funcky stuff. I tried running my application with and without exasock, issue is same. I tried both exanic-capture and exact-capture, issue is still same. I see tcpdump is capturing Tx packets when exasock is not used but eventually I do want to use exasock.
Relevant port info from exanic-config
Port 1:
Interface: enp1s0d1
Port speed: 10000 Mbps
Port status: enabled, SFP present, signal detected, link active
MAC address: ~~XX:XX:XX:XX:XX:XX~~
IP address: 10.230.88.135 Mask: 255.255.255.128
RX packets: 541925 ignored: 2563 error: 0 dropped: 0
TX packets: 95994

My server IP in below logs is 10.230.88.135, destination ports I am interested in are 10810 and 10815. As you can see only Rx is there
exact-capture (exanic-capture) dump with and without exasock
11 2022-11-07 16:29:06.643193920 172.19.12.85 → 10.230.88.135 TCP 66 10815 → 53939 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=16384
12 2022-11-07 16:29:06.804439768 10.230.88.129 → 224.0.1.129 PTPv2 100 Delay_Resp Message
13 2022-11-07 16:29:07.143351792 172.19.12.85 → 10.230.88.135 TCP 64 10815 → 53939 [ACK] Seq=1 Ack=71 Win=8388608 Len=0
14 2022-11-07 16:29:07.143530580 172.19.12.85 → 10.230.88.135 TCP 156 10815 → 53939 [PSH, ACK] Seq=1 Ack=71 Win=8388608 Len=98
15 2022-11-07 16:29:07.143550128 172.19.12.85 → 10.230.88.135 TCP 64 10815 → 53939 [FIN, ACK] Seq=99 Ack=71 Win=8388608 Len=0
16 2022-11-07 16:29:07.243434324 172.19.12.85 → 10.230.88.135 TCP 64 10815 → 53939 [ACK] Seq=100 Ack=72 Win=8388608 Len=0
17 2022-11-07 16:29:07.244144184 172.19.12.43 → 10.230.88.135 TCP 66 10810 → 58953 [SYN, ACK] Seq=0 Ack=1 Win=42340 Len=0 MSS=1460 WS=2048
18 2022-11-07 16:29:07.244249676 172.19.12.43 → 10.230.88.135 TCP 64 10810 → 58953 [ACK] Seq=1 Ack=83 Win=43008 Len=0
19 2022-11-07 16:29:07.247823696 172.19.12.43 → 10.230.88.135 TCP 132 10810 → 58953 [PSH, ACK] Seq=1 Ack=83 Win=43008 Len=74
20 2022-11-07 16:29:07.429142648 Cisco_ce:49:7b → Spanning-tree-(for-bridges)_00 STP 64 RST. Root = 32768/956/e8:65:49:ce:49:bc Cost = 0 Port = 0x8094
21 2022-11-07 16:29:07.700143352 10.230.88.129 → 224.0.1.129 PTPv2 110 Announce Message
22 2022-11-07 16:29:07.700568140 10.230.88.129 → 224.0.1.129 PTPv2 90 Sync Message
23 2022-11-07 16:29:07.701097416 10.230.88.129 → 224.0.1.129 PTPv2 90 Follow_Up Message
24 2022-11-07 16:29:08.181836424 10.230.88.133 → 10.230.88.255 UDP 305 1534 → 1534 Len=259
25 2022-11-07 16:29:09.248572132 172.19.12.43 → 10.230.88.135 TCP 356 10810 → 58953 [PSH, ACK] Seq=75 Ack=381 Win=45056 Len=298
26 2022-11-07 16:29:09.282850260 172.19.12.43 → 10.230.88.135 TCP 174 10810 → 58953 [PSH, ACK] Seq=373 Ack=443 Win=45056 Len=116

tcpdump without exasock
1 2022-11-07 16:08:57.509329 10.230.88.135 → 172.19.12.85 TCP 74 50697 → 10815 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=2315629815 TSecr=0 WS=1
2 2022-11-07 16:08:57.509913 172.19.12.85 → 10.230.88.135 TCP 66 10815 → 50697 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 SACK_PERM=1 WS=16384
3 2022-11-07 16:08:57.509918 10.230.88.135 → 172.19.12.85 TCP 54 50697 → 10815 [ACK] Seq=1 Ack=1 Win=64240 Len=0
4 2022-11-07 16:08:58.010002 10.230.88.135 → 172.19.12.85 TCP 124 50697 → 10815 [PSH, ACK] Seq=1 Ack=1 Win=64240 Len=70
5 2022-11-07 16:08:58.010078 172.19.12.85 → 10.230.88.135 TCP 60 10815 → 50697 [ACK] Seq=1 Ack=71 Win=8388608 Len=0
6 2022-11-07 16:08:58.010321 172.19.12.85 → 10.230.88.135 TCP 152 10815 → 50697 [PSH, ACK] Seq=1 Ack=71 Win=8388608 Len=98
7 2022-11-07 16:08:58.010322 10.230.88.135 → 172.19.12.85 TCP 54 50697 → 10815 [ACK] Seq=71 Ack=99 Win=64142 Len=0
8 2022-11-07 16:08:58.010343 172.19.12.85 → 10.230.88.135 TCP 60 10815 → 50697 [FIN, ACK] Seq=99 Ack=71 Win=8388608 Len=0
9 2022-11-07 16:08:58.050378 10.230.88.135 → 172.19.12.85 TCP 54 50697 → 10815 [ACK] Seq=71 Ack=100 Win=64141 Len=0
10 2022-11-07 16:08:58.110059 10.230.88.135 → 172.19.12.85 TCP 54 50697 → 10815 [FIN, ACK] Seq=71 Ack=100 Win=64141 Len=0
11 2022-11-07 16:08:58.110071 10.230.88.135 → 172.19.12.43 TCP 74 47867 → 10810 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=206912455 TSecr=0 WS=1
12 2022-11-07 16:08:58.110125 172.19.12.85 → 10.230.88.135 TCP 60 10815 → 50697 [ACK] Seq=100 Ack=72 Win=8388608 Len=0
13 2022-11-07 16:08:58.110667 172.19.12.43 → 10.230.88.135 TCP 74 10810 → 47867 [SYN, ACK] Seq=0 Ack=1 Win=43440 Len=0 MSS=1460 SACK_PERM=1 TSval=1431219647 TSecr=206912455 WS=2048
14 2022-11-07 16:08:58.110670 10.230.88.135 → 172.19.12.43 TCP 66 47867 → 10810 [ACK] Seq=1 Ack=1 Win=64240 Len=0 TSval=206912456 TSecr=1431219647
15 2022-11-07 16:08:58.110709 10.230.88.135 → 172.19.12.43 TCP 148 47867 → 10810 [PSH, ACK] Seq=1 Ack=1 Win=64240 Len=82 TSval=206912456 TSecr=1431219647
16 2022-11-07 16:08:58.110831 172.19.12.43 → 10.230.88.135 TCP 66 10810 → 47867 [ACK] Seq=1 Ack=83 Win=45056 Len=0 TSval=1431219647 TSecr=206912456
17 2022-11-07 16:08:58.112539 172.19.12.43 → 10.230.88.135 TCP 140 10810 → 47867 [PSH, ACK] Seq=1 Ack=83 Win=45056 Len=74 TSval=1431219649 TSecr=206912456
18 2022-11-07 16:08:58.112540 10.230.88.135 → 172.19.12.43 TCP 66 47867 → 10810 [ACK] Seq=83 Ack=75 Win=64166 Len=0 TSval=206912458 TSecr=1431219649
19 2022-11-07 16:09:00.110765 10.230.88.135 → 172.19.12.43 TCP 364 47867 → 10810 [PSH, ACK] Seq=83 Ack=75 Win=64166 Len=298 TSval=206914456 TSecr=1431219649
20 2022-11-07 16:09:00.119258 172.19.12.43 → 10.230.88.135 TCP 364 10810 → 47867 [PSH, ACK] Seq=75 Ack=381 Win=45056 Len=298 TSval=1431221656 TSecr=206914456

Please advise how to debug this further or what could be the possible causes.

cyriljose22 · ‎04-03-2023

Traffic transmitted via utilities that use libexanic or exasock, bypass the kernel, hence it will not be shown in tcpdump.

dahiya · ‎04-03-2023

Please go through details in original post. Question is more about exanic capture utility and not tcpdump. So let's take tcpdump out of picture to avoid confusion if it's causing any, and then summary of the issue is "exanic capture utilities are not capturing TX traffic whether application uses exasock or not"

Kamal Gupta · ‎08-01-2023

I am also facing the same issue,
The exact-capture or exanic-capture do not capture the Tx traffic, It just captures the Rx traffic.
Can someone please tell if this is a limitation or if we have any ways to capture/timestamp the Tx traffic.

dahiya · ‎11-04-2023

After some careful reading, for kernel bypass traffic, it seems that both exanic-capture and exact-capture tool only capture Rx traffic. Way to capture Tx traffic is to turn on loopback mode https://exablaze.com/docs/exanic/user-guide/config/#local-loopback-mode. But then real Rx stops and application (or even some lower layer like TCP) might behave badly. So proper way to capture Tx is at switch level. If some expert can confirm above that would close the case here.