cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
338
Views
1
Helpful
2
Replies

Issue while running T-rex , session frozen

indrajitmullick
Level 1
Level 1

I am using Red Hat linux with MLNX_OFED_LINUX-24.07-0.6.1.0 driver and drv=mlx5_core

T-rex version -> v3.05

Now, Binding is done with NIC card, but getting the session frozen once starting T-rex.

Please check the log while running T-rex :

============================

sudo ./t-rex-64 -i -v 7

Warning: Mellanox NICs were tested only with RedHat/CentOS 7.9
Correct usage with other Linux distributions is not guaranteed.
Starting Scapy server.... Scapy server is started
The ports are bound/configured.
Starting TRex v3.04 please wait ...
Using configuration file /etc/trex_cfg.yaml
port limit : not configured
port_bandwidth_gb : 10
port_speed : 0
port_mtu : 0
if_mask : None
is low-end : 0
stack type :
thread_per_dual_if : 1
if : b072:00:02.0, dummy,
enable_zmq_pub : 1
zmq_pub_port : 4500
m_zmq_rpc_port : 4501
src : 00:0d:3a:7d:74:60
dest : aa:bb:cc:f2:6c:f6
src : 00:0d:3a:7d:74:60
dest : aa:bb:cc:f2:6c:f6
memory per 2x10G ports
MBUF_64 : 16380
MBUF_128 : 8190
MBUF_256 : 8190
MBUF_512 : 8190
MBUF_1024 : 8190
MBUF_2048 : 4095
MBUF_4096 : 128
MBUF_9K : 512
TRAFFIC_MBUF_64 : 65520
TRAFFIC_MBUF_128 : 32760
TRAFFIC_MBUF_256 : 8190
TRAFFIC_MBUF_512 : 8190
TRAFFIC_MBUF_1024 : 8190
TRAFFIC_MBUF_2048 : 32760
TRAFFIC_MBUF_4096 : 128
TRAFFIC_MBUF_9K : 512
MBUF_DP_FLOWS : 524288
MBUF_GLOBAL_FLOWS : 5120
master thread : 0
rx thread : 1
dual_if : 0
socket : 0
[ 2 3 ]
CTimerWheelYamlInfo does not exist
flags : 8010f00
write_file : 0
verbose : 7
realtime : 1
flip : 0
cores : 1
single core : 0
flow-flip : 0
no clean close : 0
zmq_publish : 1
vlan mode : 0
client_cfg : 0
mbuf_cache_disable : 0
cfg file :
mac file :
out file :
client cfg file :
duration : 0
factor : 1
mbuf_factor : 1
latency : 0 pkt/sec
zmq_port : 4500
telnet_port : 4501
expected_ports : 2
tw_bucket_usec : 20.000000 usec
tw_buckets : 1024 usec
tw_levels : 3 usec
port : 0 dst:aa:bb:cc:f2:6c:f6 src:00:0d:3a:7d:74:60
port : 1 dst:aa:bb:cc:f2:6c:f6 src:00:0d:3a:7d:74:60
port : 2 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 3 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 4 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 5 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 6 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 7 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 8 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 9 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 10 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 11 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 12 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 13 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 14 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 15 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 16 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 17 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 18 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 19 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 20 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 21 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 22 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 23 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 24 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 25 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 26 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 27 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 28 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 29 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 30 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
port : 31 dst:00:00:00:01:00:00 src:00:00:00:00:00:00
Total Memory :
MBUF_64 : 81900
MBUF_128 : 40950
MBUF_256 : 16380
MBUF_512 : 16380
MBUF_1024 : 16380
MBUF_2048 : 36855
MBUF_4096 : 1024
MBUF_DP_FLOWS : 524288
MBUF_GLOBAL_FLOWS : 5120
get_each_core_dp_flows : 524288
Total memory : 248.40 Mbytes
core_list : 0,1,2
sockets : 0
active sockets : 1
ports_sockets : 1
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
phy | virt
2 1
DPDK args
xx -d libmlx5-64.so -l 0,1,2 -n 4 --log-level 8 --main-lcore 0 -a b072:00:02.0 --legacy-mem
EAL: Detected CPU lcores: 8
EAL: Detected NUMA nodes: 1
EAL: Static memory layout is selected, amount of reserved memory can be adjusted with -m or --socket-mem
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: 2 hugepages of size 1073741824 reserved, but no mounted hugetlbfs found for that size
EAL: Probe PCI driver: net_mlx5 (15b3:1018) device: b072:00:02.0 (socket 0)
TELEMETRY: No legacy callbacks, legacy socket not created
input : [b072:00:02.0, dummy]
dpdk : [b072:00:02.0]
pci_scan : [b072:00:02.0]
map : [ 0, 255]
TRex port mapping
-----------------
TRex vport: 0 dpdk_rte_eth: 0
TRex vport: 1 dpdk_rte_eth: 255
set driver name net_mlx5
driver capability : TCP_UDP_OFFLOAD TSO LRO
set dpdk queues mode to DROP_QUE_FILTER
DPDK devices 1 : 1
-----
0 : vdev b072:00:02.0
-----
Number of ports found: 2 (dummy among them: 1)


if_index : 4
driver name : net_mlx5
min_rx_bufsize : 32
max_rx_pktlen : 65536
max_rx_queues : 1024
max_tx_queues : 1024
max_mac_addrs : 128
rx_offload_capa : 0x19621f
tx_offload_capa : 0x196af
rss reta_size : 256
flow_type_rss : 0xf00000000803afbc
tx_desc_max : 65535
tx_desc_min : 0
rx_desc_max : 65535
rx_desc_min : 0
zmq publisher at: tcp://*:4500
rx_data_q_num : 1
rx_drop_q_num : 1
rx_dp_q_num : 0
rx_que_total : 2
--
rx_desc_num_data_q : 4096
rx_desc_num_drop_q : 64
rx_desc_num_dp_q : 0
total_desc : 4160
--
tx_desc_num : 1024
port 0 desc: MT27800 Family [ConnectX-5 Virtual Function]
rx_qid: 0 (64)
rx_qid: 1 (4096)
wait 1 sec

 

==================

 

Immediately after this, the session is frozen, even I could not access the second session. Kindly suggest.

 

 

 

 

 

 

 

 

 

2 Replies 2

Tyler Langston
Cisco Employee
Cisco Employee

Hi @indrajitmullick - welcome to our support community, we're glad you're here!

In reviewing your specific question, I'm afraid that we won't be able to help as much as we'd like. T Rex is an open source product that we aren't able to provide specific support for on the ThousandEyes board. 

However, we do have a place where others may be able to weigh in and help - I am moving your post to their board to get the eyes of folks who frequent there (link here). 

Best of luck with your question!

No expert here, looking at your log messages, couple of things you can check, look at the DPDK configuration and ensure that the --socket-mem option is specified correctly as this might cause issues with memory allocation and check/configure the hugepages correctly or adjust the --socket-mem option to avoid using hugepages, your log output shows that the system has 2 hugepages of size 1073741824 reserved, but no mounted hugetlbfs is found for that size. Also try and disable the zmq_publish option in the T-Rex configuration to see if that changes anything.

If none of this work, suggest to open an issue on the repo https://github.com/cisco-system-traffic-generator/trex-core/issues 

 Hope this helps.

Please mark this as helpful or solution accepted to help others
Connect with me https://bigevilbeard.github.io