Trex ARP strange issue

satish.txt1 — Mon, 23 Nov 2020 18:55:46 GMT

This is my setup to benchmark my server https://ibb.co/0KWwT8Z

Trex version v2.88

Trex server MAC

ens2f0: f8:f2:1e:5c:bf:78
ens2f1: f8:f2:1e:5c:bf:79

DUT server MAC

eth0: fa:16:3e:1c:be:82
eth1: fa:16:3e:d5:37:b1

trex_cfg.yaml

# cat /etc/trex_cfg.yaml
### Config file generated by dpdk_setup_ports.py ###

- version: 2
  #interfaces: ['07:00.0', '07:00.1']
  interfaces: ['ens2f0', 'ens2f1']
  port_info:
      - ip: 10.66.0.100
        default_gw: 10.66.1.33
      - ip: 10.68.0.100
        default_gw: 10.68.3.217

  platform:
      master_thread_id: 0
      latency_thread_id: 8
      dual_if:
        - socket: 0
          threads: [1,2,3,4,5,6,7,16,17,18,19,20,21,22,23]

My scenario file

- duration : 5
  generator :
          distribution : "seq"
          clients_start : "16.0.0.1"
          clients_end   : "16.0.0.255"
          servers_start : "48.0.0.1"
          servers_end   : "48.0.255.255"
          clients_per_gb : 10
          min_clients    : 10
          dual_port_mask : "1.0.0.0"
          tcp_aging      : 0
          udp_aging      : 0
  cap_info :
     - name: cap2/rtp_160k.pcap
       cps : 10.0
       ipg : 10000
       rtt : 10000
       w   : 4
     - name: cap2/rtp_250k_rtp_only_1.pcap
       cps : 10.0
       ipg : 10000
       rtt : 10000
       w   : 4
     - name: cap2/rtp_250k_rtp_only_2.pcap
       cps : 10.0
       ipg : 10000
       rtt : 10000
       w   : 4

Run command

./t-rex-64 --cfg /etc/trex_cfg.yaml -f /tmp/sfr.yaml -d 10 -m 1 -c 1 -k 2 --nc

In Test i am seeing Trex sending UDP stream but RX end not receiving any data and result is 100% packet drops. when i run tcpdump on my DUT device i can see its sending UDP stream to Trex RX end using correct MAC address. fa:16:3e:d5:37:b1 > f8:f2:1e:5c:bf:79

DUT server tcpdump

    16.0.0.17.58005 > 48.0.0.17.1024: [no cksum] UDP, length 1231
15:38:34.237250 fa:16:3e:d5:37:b1 > f8:f2:1e:5c:bf:79, ethertype IPv4 (0x0800), length 378: (tos 0x0, ttl 125, id 11822, offset 0, flags [none], proto UDP (17), length 364)
    16.0.0.18.9873 > 48.0.0.18.1026: [no cksum] UDP, length 336
15:38:34.237260 fa:16:3e:d5:37:b1 > f8:f2:1e:5c:bf:79, ethertype IPv4 (0x0800), length 1492: (tos 0x0, ttl 125, id 11852, offset 0, flags [none], proto UDP (17), length 1478)
    16.0.0.15.23195 > 48.0.0.15.1026: [no cksum] UDP, length 1450
15:38:34.237266 fa:16:3e:d5:37:b1 > f8:f2:1e:5c:bf:79, ethertype IPv4 (0x0800), length 434: (tos 0x0, ttl 125, id 11832, offset 0, flags [none], proto UDP (17), length 420)

As soon as test start i am seeing correct ARP entry mac-ip but in 2 second it get incomplete even my test is running and keep sending data.

[root@foo-2 ~]# arp -a | grep 10.68.0.100
? (10.68.0.100) at f8:f2:1e:5c:bf:79 [ether] on eth1
[root@foo-2 ~]# arp -a | grep 10.68.0.100
? (10.68.0.100) at <incomplete> on eth1

Any idea why ARP getting incomplete in 2 second?

UPDATE:

When i run tcpdump on DUT server then i found Trex RX side interface ask for ARP and DUI replied back but when DUT ask for ask then Trex not replying back (10.68.3.217 is eth1 IP of DUT server)

who is 01:03:05:07:09:00 and 01:03:05:07:09:01 ?

[root@foo-2 ~]# tcpdump -i eth1 -nn arp -e -vv
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
16:03:26.527165 f8:f2:1e:5c:bf:79 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 10.68.0.100 (01:03:05:07:09:00) tell 10.68.0.100, length 46
16:03:26.527192 f8:f2:1e:5c:bf:79 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 10.68.3.217 (01:03:05:07:09:01) tell 10.68.0.100, length 46
16:03:26.527201 fa:16:3e:d5:37:b1 > f8:f2:1e:5c:bf:79, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 10.68.3.217 is-at fa:16:3e:d5:37:b1, length 28
16:03:32.107249 fa:16:3e:d5:37:b1 > f8:f2:1e:5c:bf:79, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.68.0.100 tell 10.68.3.217, length 28
16:03:33.131245 fa:16:3e:d5:37:b1 > f8:f2:1e:5c:bf:79, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.68.0.100 tell 10.68.3.217, length 28
16:03:34.155245 fa:16:3e:d5:37:b1 > f8:f2:1e:5c:bf:79, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.68.0.100 tell 10.68.3.217, length 28
16:03:35.183759 fa:16:3e:d5:37:b1 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.68.0.100 tell 10.68.3.217, length 28
16:03:36.203260 fa:16:3e:d5:37:b1 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.68.0.100 tell 10.68.3.217, length 28
16:03:37.227240 fa:16:3e:d5:37:b1 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), lengt

topic Trex ARP strange issue in Tools

Trex ARP strange issue