Issue while running T-rex , session frozen

indrajitmullick — Mon, 02 Sep 2024 13:21:46 GMT

I am using Red Hat linux with MLNX_OFED_LINUX-24.07-0.6.1.0 driver and drv=mlx5_core

T-rex version -> v3.05

Now, Binding is done with NIC card, but getting the session frozen once starting T-rex.

Please check the log while running T-rex :

============================

sudo ./t-rex-64 -i -v 7

Warning: Mellanox NICs were tested only with RedHat/CentOS 7.9
Correct usage with other Linux distributions is not guaranteed.
Starting Scapy server.... Scapy server is started
The ports are bound/configured.
Starting TRex v3.04 please wait ...
Using configuration file /etc/trex_cfg.yaml
port limit : not configured
port_bandwidth_gb : 10
port_speed : 0
port_mtu : 0
if_mask : None
is low-end : 0
stack type :
thread_per_dual_if : 1
if : b072:00:02.0, dummy,
enable_zmq_pub : 1
zmq_pub_port : 4500
m_zmq_rpc_port : 4501
src : 00:0d:3a:7d:74:60
dest : aa:bb:cc:f2:6c:f6
src : 00:0d:3a:7d:74:60
dest : aa:bb:cc:f2:6c:f6
memory per 2x10G ports
MBUF_64 : 16380
MBUF_128 : 8190
MBUF_256 : 8190
MBUF_512 : 8190
MBUF_1024 : 8190
MBUF_2048 : 4095
MBUF_4096 : 128
MBUF_9K : 512
TRAFFIC_MBUF_64 : 65520
TRAFFIC_MBUF_128 : 32760
TRAFFIC_MBUF_256 : 8190
TRAFFIC_MBUF_512 : 8190
TRAFFIC_MBUF_1024 : 8190
TRAFFIC_MBUF_2048 : 32760
TRAFFIC_MBUF_4096 : 128
TRAFFIC_MBUF_9K : 512
MBUF_DP_FLOWS : 524288
MBUF_GLOBAL_FLOWS : 5120
master thread : 0
rx thread : 1
dual_if : 0
socket : 0
[ 2 3 ]
CTimerWheelYamlInfo does not exist
flags : 8010f00
write_file : 0
verbose : 7
realtime : 1
flip : 0
cores : 1
single core : 0
flow-flip : 0
no clean close : 0
zmq_publish : 1
vlan mode : 0
client_cfg : 0
mbuf_cache_disable : 0
cfg file :
mac file :
out file :
client cfg file :
duration : 0
factor : 1
mbuf_factor : 1
latency : 0 pkt/sec
zmq_port : 4500
telnet_port : 4501
expected_ports : 2
tw_bucket_usec : 20.000000 usec
tw_buckets : 1024 usec
tw_levels : 3 usec
port : 0 dst:aa:bb:cc:f2:6c:f6 src:00:0d:3a:7d:74:60
port : 1 dst:aa:bb:cc:f2:6c:f6 src:00:0d:3a:7d:74:60
port : 2 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 3 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 4 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 5 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 6 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 7 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 8 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 9 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 10 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 11 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 12 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 13 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 14 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 15 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 16 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 17 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 18 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 19 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 20 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 21 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 22 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 23 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 24 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 25 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 26 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 27 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 28 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 29 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 30 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 31 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
Total Memory :
MBUF_64 : 81900
MBUF_128 : 40950
MBUF_256 : 16380
MBUF_512 : 16380
MBUF_1024 : 16380
MBUF_2048 : 36855
MBUF_4096 : 1024
MBUF_DP_FLOWS : 524288
MBUF_GLOBAL_FLOWS : 5120
get_each_core_dp_flows : 524288
Total memory : 248.40 Mbytes
core_list : 0,1,2
sockets : 0
active sockets : 1
ports_sockets : 1
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
phy | virt
2 1
DPDK args
xx -d libmlx5-64.so -l 0,1,2 -n 4 --log-level 8 --main-lcore 0 -a b072:00:02.0 --legacy-mem
EAL: Detected CPU lcores: 8
EAL: Detected NUMA nodes: 1
EAL: Static memory layout is selected, amount of reserved memory can be adjusted with -m or --socket-mem
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: 2 hugepages of size 1073741824 reserved, but no mounted hugetlbfs found for that size
EAL: Probe PCI driver: net_mlx5 (15b3:1018) device: b072:00:02.0 (socket 0)
TELEMETRY: No legacy callbacks, legacy socket not created
input : [b072:00:02.0, dummy]
dpdk : [b072:00:02.0]
pci_scan : [b072:00:02.0]
map : [ 0, 255]
TRex port mapping
-----------------
TRex vport: 0 dpdk_rte_eth: 0
TRex vport: 1 dpdk_rte_eth: 255
set driver name net_mlx5
driver capability : TCP_UDP_OFFLOAD TSO LRO
set dpdk queues mode to DROP_QUE_FILTER
DPDK devices 1 : 1
-----
0 : vdev b072:00:02.0
-----
Number of ports found: 2 (dummy among them: 1)

if_index : 4
driver name : net_mlx5
min_rx_bufsize : 32
max_rx_pktlen : 65536
max_rx_queues : 1024
max_tx_queues : 1024
max_mac_addrs : 128
rx_offload_capa : 0x19621f
tx_offload_capa : 0x196af
rss reta_size : 256
flow_type_rss : 0xf00000000803afbc
tx_desc_max : 65535
tx_desc_min : 0
rx_desc_max : 65535
rx_desc_min : 0
zmq publisher at: tcp://*:4500
rx_data_q_num : 1
rx_drop_q_num : 1
rx_dp_q_num : 0
rx_que_total : 2
--
rx_desc_num_data_q : 4096
rx_desc_num_drop_q : 64
rx_desc_num_dp_q : 0
total_desc : 4160
--
tx_desc_num : 1024
port 0 desc: MT27800 Family [ConnectX-5 Virtual Function]
rx_qid: 0 (64)
rx_qid: 1 (4096)
wait 1 sec

==================

Immediately after this, the session is frozen, even I could not access the second session. Kindly suggest.

Re: Issue while running T-rex , session frozen

Tyler Langston — Tue, 03 Sep 2024 20:44:46 GMT

Hi @indrajitmullick - welcome to our support community, we're glad you're here!

In reviewing your specific question, I'm afraid that we won't be able to help as much as we'd like. T Rex is an open source product that we aren't able to provide specific support for on the ThousandEyes board.

However, we do have a place where others may be able to weigh in and help - I am moving your post to their board to get the eyes of folks who frequent there (link here).

Best of luck with your question!

Re: Issue while running T-rex , session frozen

bigevilbeard — Wed, 04 Sep 2024 08:59:34 GMT

No expert here, looking at your log messages, couple of things you can check, look at the DPDK configuration and ensure that the --socket-mem option is specified correctly as this might cause issues with memory allocation and check/configure the hugepages correctly or adjust the --socket-mem option to avoid using hugepages, your log output shows that the system has 2 hugepages of size 1073741824 reserved, but no mounted hugetlbfs is found for that size. Also try and disable the zmq_publish option in the T-Rex configuration to see if that changes anything.

If none of this work, suggest to open an issue on the repo https://github.com/cisco-system-traffic-generator/trex-core/issues

Hope this helps.

topic Issue while running T-rex , session frozen in Tools

Issue while running T-rex , session frozen

Re: Issue while running T-rex , session frozen

Re: Issue while running T-rex , session frozen