05-28-2017 05:16 AM
I want to start Trex on 8 ports and while starting the trex, i am observing below error:
Number of ports found: 8
wait 10 sec ..........
port : 0
------------
link : link : Link Up - speed 1000 Mbps - full-duplex
promiscuous : 0
port : 1
------------
link : link : Link Up - speed 10 Mbps - full-duplex
promiscuous : 0
port : 2
------------
link : link : Link Up - speed 1000 Mbps - full-duplex
promiscuous : 0
port : 3
------------
link : link : Link Up - speed 1000 Mbps - full-duplex
promiscuous : 0
port : 4
------------
link : link : Link Up - speed 1000 Mbps - full-duplex
promiscuous : 0
port : 5
------------
link : link : Link Up - speed 100 Mbps - full-duplex
promiscuous : 0
port : 6
------------
link : link : Link Up - speed 1000 Mbps - full-duplex
promiscuous : 0
port : 7
------------
link : link : Link Up - speed 100 Mbps - full-duplex
promiscuous : 0
-------------------------------
RX core uses TX queue number 1 on all ports
core, c-port, c-queue, s-port, s-queue, lat-queue
------------------------------------------
1 0 0 1 0 2
2 2 0 3 0 2
3 4 0 5 0 2
4 6 0 7 0 2
-------------------------------
number of ports : 8
max cores for 2 ports : 1
max queue per port : 3
RING: Cannot reserve memory
ERROR there is not enough huge-pages memory in your system
EAL: Error - exiting with code: 1
Cause: Cannot init nodes mbuf pool nodes-2
07-21-2017 08:04 AM
Difference in scenario 1 and scenario 2:
Scenario 1: User 1 uses 2 ports with default zmq ports, able to start the server in first instance
User 2 uses 6 ports with user-defined zmq ports, prefix(different username). Able to start the server in second instance
Scenario 2: User 1 uses 2 ports with default zmq ports, able to start the server in first instance
User 2 using more than 6 ports be it 8 or more ports with user-defined zmq ports, prefix(different username). Unable to start the server in second instance with options specified mbuf, no-scapy
[root@trex-server v2.27]# free utility
total used free shared buff/cache available
Mem: 65861408 9017368 56410488 32676 433552 56472612
Swap: 32964604 0 32964604
[root@trex-server v2.27]# free -h
total used free shared buff/cache available
Mem: 62G 8.6G 53G 31M 423M 53G
Swap: 31G 0B 31G
[root@trex-server v2.27]# /proc/meminfo
-bash: /proc/meminfo: Permission denied
[root@trex-server v2.27]# cat /proc/meminfo
MemTotal: 65861408 kB
MemFree: 56409692 kB
MemAvailable: 56472652 kB
Buffers: 33620 kB
Cached: 268444 kB
SwapCached: 0 kB
Active: 196152 kB
Inactive: 163880 kB
Active(anon): 94008 kB
Inactive(anon): 952 kB
Active(file): 102144 kB
Inactive(file): 162928 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 32964604 kB
SwapFree: 32964604 kB
Dirty: 372 kB
Writeback: 0 kB
AnonPages: 62316 kB
Mapped: 137768 kB
Shmem: 32676 kB
Slab: 132288 kB
SReclaimable: 46284 kB
SUnreclaim: 86004 kB
KernelStack: 8304 kB
PageTables: 4856 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 61701004 kB
Committed_AS: 368112 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 429904 kB
VmallocChunk: 34325686432 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
HugePages_Total: 4096
HugePages_Free: 3072
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 120364 kB
DirectMap2M: 6057984 kB
DirectMap1G: 62914560 kB
[root@trex-server v2.27]#
07-21-2017 08:50 AM
Understood now. Running two TRexs instances is not simple. Making it running in optimal way it even more complex..
Our script run good for single instance.
1) Please send the config file for user 1) and user 2)
Make sure they have different cores assigned.
2) try to reduce mbuf in both instances to 0.2 (using --mbuf-factor)
You should have at least 8GB (tlb) + 2GB (heap) in your system
Hanoh
Sent from my iPhone
07-22-2017 03:43 AM
Now, I'm able to run the server with two instances from user1 and user2 using mbuf argument set 0.2.
Could you please tell me how to specify the cores accordingly to ports being used in yaml file?
Config information of user1 and user2 yaml files,
User1:
[root@trex-server v2.27]# cat cfg/asp.yaml
- port_limit : 2
version : 2
interfaces : ["86:00.0","86:00.1"]
prefix : server_iot
port_info :
- src_mac : '00:00:00:71:00:01'
- src_mac : '00:00:00:71:00:02'
User2:
[root@trex-server v2.27]# cat cfg/Trex.yaml
- port_limit : 10
version : 2
interfaces : ["02:00.0","02:00.1","02:00.2","02:00.3","07:00.0","07:00.1","07:00.2","07:00.3","86:00.2","86:00.3"]
# interfaces : ["02:00.0","02:00.1","02:00.2","02:00.3","07:00.0","07:00.1","07:00.2","07:00.3"]
prefix : server_wipro
zmq_pub_port : 5001
zmq_rpc_port : 5002
port_info :
- default_gw : '2.2.2.1'
src_mac : '00:00:00:71:00:11'
ip : '2.2.2.2'
- default_gw : '2.2.2.2'
src_mac : '00:00:00:71:00:12'
ip : '2.2.2.1'
- default_gw : '3.2.2.1'
src_mac : '00:00:00:71:00:13'
ip : '3.2.2.2'
- default_gw : '3.2.2.2'
src_mac : '00:00:00:71:00:14'
ip : '3.2.2.1'
- default_gw : '4.2.2.1'
src_mac : '00:00:00:71:00:15'
ip : '4.2.2.2'
- default_gw : '4.2.2.2'
src_mac : '00:00:00:71:00:16'
ip : '4.2.2.1'
- default_gw : '5.2.2.2'
src_mac : '00:00:00:71:00:17'
ip : '5.2.2.1'
- default_gw : '5.2.2.2'
src_mac : '00:00:00:71:00:18'
ip : '5.2.2.1'
- default_gw : '6.2.2.2'
src_mac : '00:00:00:71:00:19'
ip : '6.2.2.1'
- default_gw : '6.2.2.2'
src_mac : '00:00:00:71:00:20'
ip : '6.2.2.1'
# - default_gw : '7.2.2.2'
# src_mac : '00:00:91:00:00:01'
# ip : '7.2.2.1'
# - default_gw : '7.2.2.2'
# src_mac : '00:00:91:00:00:02'
# ip : '7.2.2.1'
[root@trex-server v2.27]#
07-23-2017 01:20 AM
Hi,
Easiest way to do it is using script, look at the options here:
sudo ./dpdk_setup_ports.py --help
You can run it first time with usual options to create config similar to your first one.
You will see section with cores config.
Then, for second config file, you can specify cores to exclude ( --ce argument to the script ) which are going to be used in first instance of TRex.
05-30-2024 05:16 AM
Hi Hanoh,
I have tried this option but still I am facing this issue.
bbarot@trex06.iad0:~/v3.02$ cat /etc/trex_cfg_593vlan.yaml
### Config file generated by dpdk_setup_ports.py ###
- port_limit: 12
version: 2
interfaces: ['--vdev=net_bonding0,mode=2,slave=00:07.0,slave=00:08.0,mac=52:54:00:ee:ee:04', '--vdev=net_bonding1,mode=2,slave=00:09.0,slave=00:0a.0,mac=52:54:00:ee:ee:03' ,'--vdev=net_bonding0,mode=2,slave=00:0b.0,slave=00:0c.0,mac=52:54:00:ef:ee:04', '--vdev=net_bonding1,mode=2,slave=00:0d.0,slave=00:0e.0,mac=52:54:00:ef:ee:03']
port_info:
- ip: 10.111.127.158
default_gw: 10.111.127.181
vlan: 584
- ip: 10.136.192.6
default_gw: 10.136.192.1
vlan: 593
- ip: 10.111.72.49
default_gw: 10.111.72.39
vlan: 610
- ip: 10.136.86.250
default_gw: 10.136.86.1
vlan: 605
platform:
master_thread_id: 0
latency_thread_id: 15
dual_if:
- socket: 0
threads: [1,2,3,4,5,6,7]
- socket: 1
threads: [8,9,10,11,12,13,14]
memory:
mbuf_64: 81894
mbuf_128: 40950
mbuf_256: 5000
mbuf_512: 64380
mbuf_1024: 64380
mbuf_2048: 64000
mbuf_4096: 100
mbuf_9k: 100
bbarot@trex06.iad0:~/v3.02$ sudo ./_t-rex-64 -i -c 7 --astf --cfg /etc/trex_cfg_593vlan.yaml --mbuf-factor 0.2 -v 7
Starting TRex v3.02 please wait ...
Using configuration file /etc/trex_cfg_593vlan.yaml
port limit : not configured
port_bandwidth_gb : 10
port_speed : 0
port_mtu : 0
if_mask : None
prefix : trex6
is low-end : 0
stack type :
limit_memory : 2048
thread_per_dual_if : 1
if : --vdev=net_bonding0,mode=2,slave=00:07.0,slave=00:08.0,mac=52:54:00:ee:ee:04, --vdev=net_bonding1,mode=2,slave=00:09.0,slave=00:0a.0,mac=52:54:00:ee:ee:03, --vdev=net_bonding0,mode=2,slave=00:0b.0,slave=00:0c.0,mac=52:54:00:ef:ee:04, --vdev=net_bonding1,mode=2,slave=00:0d.0,slave=00:0e.0,mac=52:54:00:ef:ee:03,
enable_zmq_pub : 1
zmq_pub_port : 4500
m_zmq_rpc_port : 4501
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
src : 00:00:00:00:00:00
dest : 00:00:00:00:00:00
memory per 2x10G ports
MBUF_64 : 81894
MBUF_128 : 40950
MBUF_256 : 5000
MBUF_512 : 64380
MBUF_1024 : 64380
MBUF_2048 : 64000
MBUF_4096 : 100
MBUF_9K : 100
TRAFFIC_MBUF_64 : 65520
TRAFFIC_MBUF_128 : 32760
TRAFFIC_MBUF_256 : 8190
TRAFFIC_MBUF_512 : 8190
TRAFFIC_MBUF_1024 : 8190
TRAFFIC_MBUF_2048 : 32760
TRAFFIC_MBUF_4096 : 128
TRAFFIC_MBUF_9K : 512
MBUF_DP_FLOWS : 524288
MBUF_GLOBAL_FLOWS : 5120
master thread : 0
rx thread : 15
dual_if : 0
socket : 0
[ 1 2 3 4 5 6 7 ]
dual_if : 1
socket : 1
[ 8 9 10 11 12 13 14 ]
CTimerWheelYamlInfo does not exist
flags : 18070f00
write_file : 0
verbose : 7
realtime : 1
flip : 0
cores : 7
single core : 0
flow-flip : 0
no clean close : 0
zmq_publish : 1
vlan mode : 1
client_cfg : 0
mbuf_cache_disable : 0
cfg file :
mac file :
out file :
client cfg file :
duration : 0
factor : 1
mbuf_factor : 0
latency : 0 pkt/sec
zmq_port : 4500
telnet_port : 4501
expected_ports : 4
tw_bucket_usec : 20.000000 usec
tw_buckets : 1024 usec
tw_levels : 3 usec
port : 0 dst:00:00:00:00:00:00 src:00:00:00:00:00:00 vlan:584
port : 1 dst:00:00:00:00:00:00 src:00:00:00:00:00:00 vlan:593
port : 2 dst:00:00:00:00:00:00 src:00:00:00:00:00:00 vlan:610
port : 3 dst:00:00:00:00:00:00 src:00:00:00:00:00:00 vlan:605
port : 4 dst:00:00:00:01:00:00 src:00:00:00:00:00:00 vlan:0
port : 5 dst:00:00:00:01:00:00 src:00:00:00:00:00:00 vlan:0
port : 6 dst:00:00:00:01:00:00 src:00:00:00:00:00:00 vlan:0
port : 7 dst:00:00:00:01:00:00 src:00:00:00:00:00:00 vlan:0
port : 8 dst:00:00:00:01:00:00 src:00:00:00:00:00:00 vlan:0
port : 9 dst:00:00:00:01:00:00 src:00:00:00:00:00:00 vlan:0
port : 10 dst:00:00:00:01:00:00 src:00:00:00:00:00:00 vlan:0
port : 11 dst:00:00:00:01:00:00 src:00:00:00:00:00:00 vlan:0
port : 12 dst:00:00:00:01:00:00 src:00:00:00:00:00:00 vlan:0
port : 13 dst:00:00:00:01:00:00 src:00:00:00:00:00:00 vlan:0
port : 14 dst:00:00:00:01:00:00 src:00:00:00:00:00:00 vlan:0
port : 15 dst:00:00:00:01:00:00 src:00:00:00:00:00:00 vlan:0
port : 16 dst:00:00:00:01:00:00 src:00:00:00:00:00:00 vlan:0
port : 17 dst:00:00:00:01:00:00 src:00:00:00:00:00:00 vlan:0
port : 18 dst:00:00:00:01:00:00 src:00:00:00:00:00:00 vlan:0
port : 19 dst:00:00:00:01:00:00 src:00:00:00:00:00:00 vlan:0
port : 20 dst:00:00:00:01:00:00 src:00:00:00:00:00:00 vlan:0
port : 21 dst:00:00:00:01:00:00 src:00:00:00:00:00:00 vlan:0
port : 22 dst:00:00:00:01:00:00 src:00:00:00:00:00:00 vlan:0
port : 23 dst:00:00:00:01:00:00 src:00:00:00:00:00:00 vlan:0
port : 24 dst:00:00:00:01:00:00 src:00:00:00:00:00:00 vlan:0
port : 25 dst:00:00:00:01:00:00 src:00:00:00:00:00:00 vlan:0
port : 26 dst:00:00:00:01:00:00 src:00:00:00:00:00:00 vlan:0
port : 27 dst:00:00:00:01:00:00 src:00:00:00:00:00:00 vlan:0
port : 28 dst:00:00:00:01:00:00 src:00:00:00:00:00:00 vlan:0
port : 29 dst:00:00:00:01:00:00 src:00:00:00:00:00:00 vlan:0
port : 30 dst:00 port : 31 dst:0 MBUF_128 MBUF_4096 Total memory core_list : 0,10,0,1,1,0,0,0,0, 4 7
5 8 2
9 13 12
DPDK args
xx -l 0,15,1,2,3,EAL: Detected CPU lcores: 16
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/trex6/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Probe PCI driver: net_i40e_vf (8086:154c) device: 0000:00:07.0 (socket 0)
EAL: Probe PCI driver: net_i40e_vf (8086:154c) device: 0000:00:08.0 (socket 0)
EAL: Probe PCI driver: net_i40e_vf (8086:154c) device: 0000:00:09.0 (socket 0)
EAL: Probe PCI driver: net_i40e_vf (8086:154c) device: 0000:00:0a.0 (socket 0)
EAL: Probe PCI driver: net_i40e_vf (8086:154c) device: 0000:00:0b.0 (socket 0)
EAL: Probe PCI driver: net_i40e_vf (8086:154c) device: 0000:00:0c.0 (socket 0)
EAL: Probe PCI driver: net_i40e_vf (8086:154c) device: 0000:00:0d.0 (socket 0)
EAL: Probe PCI driver: net_i40e_vf (8086:154c) device: 0000:00:0e.0 (socket 0)
TELEMETRY: No legacy callbacks, legacy socket not created
size of interfaces_vdevs 4
===>>>found net_bonding0 8
===>>>found net_bonding1 9
===>>>found net_bonding0 8
===>>>found net_bonding1 9
__eth_bond_slave_add_lock_free(476) - Slave device is already a slave of a bonded device
bond_ethdev_configure(3762) - Failed to add port 0 as slave to bonded device net_bonding0
__eth_bond_slave_add_lock_free(476) - Slave device is already a slave of a bonded device
bond_ethdev_configure(3762) - Failed to add port 1 as slave to bonded device net_bonding0
__eth_bond_slave_add_lock_free(476) - Slave device is already a slave of a bonded device
bond_ethdev_configure(3762) - Failed to add port 2 as slave to bonded device net_bonding1
__eth_bond_slave_add_lock_free(476) - Slave device is already a slave of a bonded device
bond_ethdev_configure(3762) - Failed to add port 3 as slave to bonded device net_bonding1
set driver name net_bonding
driver capability : TCP_UDP_OFFLOAD TSO SLRO
set dpdk queues mode to MULTI_QUE
DPDK devices 10 : 10
-----
0 : vdev 00:07.0
1 : vdev 00:08.0
2 : vdev 00:09.0
3 : vdev 00:0a.0
4 : vdev 00:0b.0
5 : vdev 00:0c.0
6 : vdev 00:0d.0
7 : vdev 00:0e.0
8 : vdev net_bonding0
9 : vdev net_bonding1
-----
Number of ports found: 4
if_index : 0
driver name : net_bonding
min_rx_bufsize : 0
max_rx_pktlen : 9728
max_rx_queues : 16
max_tx_queues : 16
max_mac_addrs : 16
rx_offload_capa : 0x226f
tx_offload_capa : 0x9fbf
rss reta_size : 64
flow_type_rss : 0x7ef8
tx_desc_max : 4096
tx_desc_min : 0
rx_desc_max : 4096
rx_desc_min : 0
set slave driver = net_i40e_vf
zmq publisher at: tcp://*:4500
rx_data_q_num : 0
rx_drop_q_num : 0
rx_dp_q_num : 7
rx_que_total : 7
--
rx_desc_num_data_q : 512
rx_desc_num_drop_q : 4096
rx_desc_num_dp_q : 512
total_desc : 3584
--
tx_desc_num : 1024
ERROR there is not enough huge-pages memory in your system
EAL: Error - exiting with code: 1
Cause: Cannot init mbuf pool small-pkt-const
bbarot@trex06.iad0:~/v3.02$
08-17-2023 03:57 AM
It appears that you're encountering an issue while trying to start Trex with 8 ports. The error message indicates a memory reservation problem related to huge-pages. Here's a breakdown of what's happening and how you might address it:
When starting Trex, you've successfully detected 8 ports and their respective link status and configurations. However, when attempting to initialize the RX core and set up memory pools, a specific error arises:
RING: Cannot reserve memory
ERROR there is not enough huge-pages memory in your system
EAL: Error - exiting with code: 1
Cause: Cannot init nodes mbuf pool nodes-2
This error suggests that the system is unable to reserve the required memory for the huge-pages, leading to the initialization failure. Huge-pages are a mechanism that provides larger memory pages to improve memory management efficiency.
To address this issue:
Check System Configuration: Ensure that your system has sufficient memory resources, including available huge-page memory, to accommodate the needs of running Trex with 8 ports.
Adjust Huge-Page Settings: You may need to adjust the huge-page settings on your system to allocate more memory for Trex. This can usually be done through your operating system's configuration or settings.
Reduce Port Configuration: As a temporary measure, you could try starting Trex with fewer ports to see if the issue is related to resource constraints. If it starts successfully with fewer ports, gradually increase the number of ports while monitoring memory usage.
Resource Optimization: Review other resource-intensive processes running on your system that could be competing for memory. Temporarily stopping or optimizing these processes might free up memory for Trex.
Consult Documentation: Check the Trex documentation or community forums for any specific recommendations or solutions related to this memory allocation issue.
Remember, addressing memory-related issues often involves a combination of system-level adjustments, resource management, and potentially seeking guidance from relevant technical resources.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide