Memory relaged issue while starting Trex server

[kasai]

I want to start Trex on 8 ports and while starting the trex, i am observing below error:

Number of ports found: 8                                   

wait 10 sec ..........                                     

port : 0                                                    

------------                                                

link         :  link : Link Up - speed 1000 Mbps - full-duplex

promiscuous  : 0                                             

port : 1                                                     

------------                                                 

link         :  link : Link Up - speed 10 Mbps - full-duplex 

promiscuous  : 0                                             

port : 2                                                     

------------                                                 

link         :  link : Link Up - speed 1000 Mbps - full-duplex

promiscuous  : 0                                             

port : 3                                                     

------------

link         :  link : Link Up - speed 1000 Mbps - full-duplex

promiscuous  : 0

port : 4

------------

link         :  link : Link Up - speed 1000 Mbps - full-duplex

promiscuous  : 0

port : 5

------------

link         :  link : Link Up - speed 100 Mbps - full-duplex

promiscuous  : 0

port : 6

------------

link         :  link : Link Up - speed 1000 Mbps - full-duplex

promiscuous  : 0

port : 7

------------

link         :  link : Link Up - speed 100 Mbps - full-duplex

promiscuous  : 0

-------------------------------

RX core uses TX queue number 1 on all ports

core, c-port, c-queue, s-port, s-queue, lat-queue

------------------------------------------

1        0      0       1       0      2

2        2      0       3       0      2

3        4      0       5       0      2

4        6      0       7       0      2

-------------------------------

number of ports         : 8

max cores for 2 ports   : 1

max queue per port      : 3

RING: Cannot reserve memory

ERROR there is not enough huge-pages memory in your system

EAL: Error - exiting with code: 1

  Cause: Cannot init nodes mbuf pool nodes-2

[apoleboi@cisco.com]

Difference in scenario 1 and scenario 2:

Scenario 1: User 1 uses 2 ports with default zmq ports, able to start the server in first instance

                    User 2 uses 6 ports with user-defined zmq ports, prefix(different username). Able to start the server in second instance

Scenario 2: User 1 uses 2 ports with default zmq ports, able to start the server in first instance

                    User 2 using more than 6 ports be it 8 or more ports with user-defined zmq ports, prefix(different username). Unable to start the server in second instance with options specified mbuf, no-scapy

[root@trex-server v2.27]# free utility

              total        used        free      shared  buff/cache   available

Mem:       65861408     9017368    56410488       32676      433552    56472612

Swap:      32964604           0    32964604

[root@trex-server v2.27]# free -h

              total        used        free      shared  buff/cache   available

Mem:            62G        8.6G         53G         31M        423M         53G

Swap:           31G          0B         31G

[root@trex-server v2.27]# /proc/meminfo

-bash: /proc/meminfo: Permission denied

[root@trex-server v2.27]# cat /proc/meminfo

MemTotal:       65861408 kB

MemFree:        56409692 kB

MemAvailable:   56472652 kB

Buffers:           33620 kB

Cached:           268444 kB

SwapCached:            0 kB

Active:           196152 kB

Inactive:         163880 kB

Active(anon):      94008 kB

Inactive(anon):      952 kB

Active(file):     102144 kB

Inactive(file):   162928 kB

Unevictable:           0 kB

Mlocked:               0 kB

SwapTotal:      32964604 kB

SwapFree:       32964604 kB

Dirty:               372 kB

Writeback:             0 kB

AnonPages:         62316 kB

Mapped:           137768 kB

Shmem:             32676 kB

Slab:             132288 kB

SReclaimable:      46284 kB

SUnreclaim:        86004 kB

KernelStack:        8304 kB

PageTables:         4856 kB

NFS_Unstable:          0 kB

Bounce:                0 kB

WritebackTmp:          0 kB

CommitLimit:    61701004 kB

Committed_AS:     368112 kB

VmallocTotal:   34359738367 kB

VmallocUsed:      429904 kB

VmallocChunk:   34325686432 kB

HardwareCorrupted:     0 kB

AnonHugePages:         0 kB

HugePages_Total:    4096

HugePages_Free:     3072

HugePages_Rsvd:        0

HugePages_Surp:        0

Hugepagesize:       2048 kB

DirectMap4k:      120364 kB

DirectMap2M:     6057984 kB

DirectMap1G:    62914560 kB

[root@trex-server v2.27]#

[hhaim@cisco.com]

Understood now. Running two TRexs instances is not simple. Making it running in optimal way it even more complex..

Our script run good for single instance.

1) Please send the config file for user 1) and user 2)

Make sure they have different cores assigned.

2) try to reduce mbuf in both instances to 0.2 (using --mbuf-factor)

You should have at least 8GB (tlb) + 2GB (heap) in your system

Hanoh

Sent from my iPhone

[apoleboi@cisco.com]

Now, I'm able to run the server with two instances from user1 and user2 using mbuf argument set 0.2.

Could you please tell me how to specify the cores accordingly to ports being used in yaml file?

Config information of user1 and user2 yaml files,

User1:

[root@trex-server v2.27]# cat cfg/asp.yaml

- port_limit      : 2

  version         : 2

  interfaces      : ["86:00.0","86:00.1"]

  prefix          : server_iot

  port_info       :

           - src_mac         :   '00:00:00:71:00:01'

           - src_mac         :   '00:00:00:71:00:02'

User2:

[root@trex-server v2.27]# cat cfg/Trex.yaml

- port_limit      : 10

  version         : 2

  interfaces      : ["02:00.0","02:00.1","02:00.2","02:00.3","07:00.0","07:00.1","07:00.2","07:00.3","86:00.2","86:00.3"]

#  interfaces      : ["02:00.0","02:00.1","02:00.2","02:00.3","07:00.0","07:00.1","07:00.2","07:00.3"]

  prefix          : server_wipro

  zmq_pub_port    : 5001

  zmq_rpc_port    : 5002

  port_info       :

          - default_gw      :   '2.2.2.1'

            src_mac         :   '00:00:00:71:00:11'

            ip              :   '2.2.2.2'

          - default_gw      :   '2.2.2.2'

            src_mac         :   '00:00:00:71:00:12'

            ip              :   '2.2.2.1'

          - default_gw      :   '3.2.2.1'

            src_mac         :   '00:00:00:71:00:13'

            ip              :   '3.2.2.2'

          - default_gw      :   '3.2.2.2'

            src_mac         :   '00:00:00:71:00:14'

            ip              :   '3.2.2.1'

          - default_gw      :   '4.2.2.1'

            src_mac         :   '00:00:00:71:00:15'

            ip              :   '4.2.2.2'

          - default_gw      :   '4.2.2.2'

            src_mac         :   '00:00:00:71:00:16'

            ip              :   '4.2.2.1'

          - default_gw      :   '5.2.2.2'

            src_mac         :   '00:00:00:71:00:17'

            ip              :   '5.2.2.1'

          - default_gw      :   '5.2.2.2'

            src_mac         :   '00:00:00:71:00:18'

            ip              :   '5.2.2.1'

          - default_gw      :   '6.2.2.2'

            src_mac         :   '00:00:00:71:00:19'

            ip              :   '6.2.2.1'

          - default_gw      :   '6.2.2.2'

            src_mac         :   '00:00:00:71:00:20'

            ip              :   '6.2.2.1'

#          - default_gw      :   '7.2.2.2'

#            src_mac         :   '00:00:91:00:00:01'

#            ip              :   '7.2.2.1'

#          - default_gw      :   '7.2.2.2'

#            src_mac         :   '00:00:91:00:00:02'

#            ip              :   '7.2.2.1'

[root@trex-server v2.27]#

[ybrustin]

Hi,

Easiest way to do it is using script, look at the options here:

sudo ./dpdk_setup_ports.py --help

You can run it first time with usual options to create config similar to your first one.

You will see section with cores config.

Then, for second config file, you can specify cores to exclude ( --ce argument to the script ) which are going to be used in first instance of TRex.

[devoraharvey3]

It appears that you're encountering an issue while trying to start Trex with 8 ports. The error message indicates a memory reservation problem related to huge-pages. Here's a breakdown of what's happening and how you might address it:

When starting Trex, you've successfully detected 8 ports and their respective link status and configurations. However, when attempting to initialize the RX core and set up memory pools, a specific error arises:

RING: Cannot reserve memory
ERROR there is not enough huge-pages memory in your system
EAL: Error - exiting with code: 1
Cause: Cannot init nodes mbuf pool nodes-2

This error suggests that the system is unable to reserve the required memory for the huge-pages, leading to the initialization failure. Huge-pages are a mechanism that provides larger memory pages to improve memory management efficiency.

To address this issue:

1. Check System Configuration: Ensure that your system has sufficient memory resources, including available huge-page memory, to accommodate the needs of running Trex with 8 ports.

2. Adjust Huge-Page Settings: You may need to adjust the huge-page settings on your system to allocate more memory for Trex. This can usually be done through your operating system's configuration or settings.

3. Reduce Port Configuration: As a temporary measure, you could try starting Trex with fewer ports to see if the issue is related to resource constraints. If it starts successfully with fewer ports, gradually increase the number of ports while monitoring memory usage.

4. Resource Optimization: Review other resource-intensive processes running on your system that could be competing for memory. Temporarily stopping or optimizing these processes might free up memory for Trex.

5. Consult Documentation: Check the Trex documentation or community forums for any specific recommendations or solutions related to this memory allocation issue.

Remember, addressing memory-related issues often involves a combination of system-level adjustments, resource management, and potentially seeking guidance from relevant technical resources.