TRex startup issue

[con-odonnell]

Have a 256GB physical system with two Intel E810-C 100Gbps nics

It seems to fail every time i've tried a combination of no limit_memory ro using 1024, 2048 or 4096 and it fails

i have approx 212GB before i run Trex but it all gets eaten up before it fails a minute or so after starting.

with limit_memory 2048

[root@NWXHEAD v3.00]# ./t-rex-64 -i --astf -c 2 -v 10 --mbuf-factor 0.2
The ports are bound/configured.
Starting TRex v3.00 please wait ...
Using configuration file /etc/trex_cfg.yaml
port limit : not configured
port_bandwidth_gb : 10
port_speed : 0
port_mtu : 0
if_mask : None
is low-end : 0
stack type :
limit_memory : 2048
thread_per_dual_if : 1
if : b5:00.0, b5:00.1,
enable_zmq_pub : 1
zmq_pub_port : 4500
m_zmq_rpc_port : 4501
src : 40:a6:b7:c0:8c:08
dest : a0:88:c2:83:51:22
src : 40:a6:b7:c0:8c:09
dest : a0:88:c2:83:51:23
memory per 2x10G ports
MBUF_64 : 16380
MBUF_128 : 8190
MBUF_256 : 8190
MBUF_512 : 8190
MBUF_1024 : 8190
MBUF_2048 : 4095
MBUF_4096 : 128
MBUF_9K : 512
TRAFFIC_MBUF_64 : 65520
TRAFFIC_MBUF_128 : 32760
TRAFFIC_MBUF_256 : 8190
TRAFFIC_MBUF_512 : 8190
TRAFFIC_MBUF_1024 : 8190
TRAFFIC_MBUF_2048 : 32760
TRAFFIC_MBUF_4096 : 128
TRAFFIC_MBUF_9K : 512
MBUF_DP_FLOWS : 524288
MBUF_GLOBAL_FLOWS : 5120
master thread : 0
rx thread : 2
dual_if : 0
socket : 1
[ 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 ]
CTimerWheelYamlInfo does not exist
flags : 8020a00
write_file : 0
verbose : 2
realtime : 1
flip : 0
cores : 2
single core : 0
flow-flip : 0
no clean close : 0
zmq_publish : 1
vlan mode : 0
client_cfg : 0
mbuf_cache_disable : 0
cfg file :
mac file :
out file :
client cfg file :
duration : 0
factor : 1
mbuf_factor : 0
latency : 0 pkt/sec
zmq_port : 4500
telnet_port : 4501
expected_ports : 2
tw_bucket_usec : 20.000000 usec
tw_buckets : 1024 usec
tw_levels : 3 usec
port : 0 dst:a0:88:c2:83:51:22 src:40:a6:b7:c0:8c:08
port : 1 dst:a0:88:c2:83:51:23 src:40:a6:b7:c0:8c:09
port : 2 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 3 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 4 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 5 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 6 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 7 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 8 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 9 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 10 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 11 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 12 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 13 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 14 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 15 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 16 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 17 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 18 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 19 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 20 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 21 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 22 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 23 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 24 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 25 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 26 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 27 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 28 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 29 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 30 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 31 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
Total Memory :
MBUF_64 : 68796
MBUF_128 : 34398
MBUF_256 : 9828
MBUF_512 : 9828
MBUF_1024 : 9828
MBUF_2048 : 33579
MBUF_4096 : 2048
MBUF_DP_FLOWS : 104857
MBUF_GLOBAL_FLOWS : 1024
get_each_core_dp_flows : 52428
Total memory : 148.80 Mbytes
core_list : 0,2,1,3
sockets : 1
active sockets : 1
ports_sockets : 1
1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
phy | virt
1 1
3 2
DPDK args
xx -l 0,2,1,3 -n 4 --log-level 3 --main-lcore 0 -a 0000:b5:00.0 -a 0000:b5:00.1 --legacy-mem --socket-mem 0,2048
PANIC in eth_dev_shared_data_prepare():
Cannot allocate ethdev shared data
./t-rex-64: line 106: 48990 Aborted ./_$(basename $0) $INPUT_ARGS $EXTRA_INPUT_ARGS

and with no limit_memory the error is:

DPDK args
xx -l 0,2,1,3 -n 4 --log-level 3 --main-lcore 0 -a 0000:b5:00.0 -a 0000:b5:00.1 --legacy-mem
EAL: FATAL: Cannot init memory
You might need to run ./trex-cfg once
EAL: Error - exiting with code: 1
Cause: Invalid EAL arguments
Killing Scapy server... Scapy server is killed

[root@NWXHEAD v3.00]# cat /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
5000
[root@NWXHEAD v3.00]# cat /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages
5000

[root@NWXHEAD v3.00]# cat /sys/devices/system/node/node1/hugepages/hugepages-1048576kB/nr_hugepages
0
[root@NWXHEAD v3.00]# cat /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages
0

(ive tried to preset 1GB pages too no luck).

i just cannot get it to start !

[bigevilbeard]

Not an expert on this, but from past thread sort of same ish issues, try configuring huge pages more explicitly, in the cfg file try

- port_limit: 2
  port_bandwidth_gb: 100
  ports: [b5:00.0, b5:00.1]
  limit_memory: 4096
  promiscuous: true

[con-odonnell]

Hi,

Many thanks for the reply !
Tried those and still same i.e. it eats up all the available ram then dies with:
1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
phy | virt
1 1
3 2
DPDK args
xx -l 0,2,1,3 -n 4 --log-level 3 --main-lcore 0 -a 0000:b5:00.0 -a 0000:b5:00.1 --legacy-mem --socket-mem 0,4096
PANIC in eth_dev_shared_data_prepare():
Cannot allocate ethdev shared data
./t-rex-64-o: line 106: 708862 Aborted ./_$(basename $0) $INPUT_ARGS $EXTRA_INPUT_ARGS

BTW I have a colleague with similar setup that has it running in a 128GB system but I can't on a 256, he was on 3.00 and I was on 3.0.6 so I tried 3.00 and same thing.
I did have a dpdk rpm installed but I removed that.
I also updated the ice drivers to latest (I believe 1.15)
All the Best

[bigevilbeard]

Very odd, it does sounds like a funky interaction between DPDK, TRex, and your specific system memory configuration here. Sadly beyond my knowledge here. I would open this on the repo https://github.com/cisco-system-traffic-generator/trex-core/issues unless someone has other ideas on this community