Running multi-thread t-rex on AMD with Broadcom P2200G NIC card

Siba- · ‎10-15-2025

Hello,

I am failing to run multi-threaded t-rex.

I have done following till now:

1) I had upgraded firmware from following location:

https://docs.broadcom.com/docs/BCM957608-P2200G_FW_235.1.160.0

https://docs.broadcom.com/docs/NXE_Linux_niccli_235.1.151.0

2) Ran multi-threaded dpdk testpmd at both side.

3) Compiled master branch (v3.07) of https://github.com/cisco-system-traffic-generator/trex-core.git for linux_dpdk.

4) When I run testpmd at one side and single threaded t-rex (t-rex-64 -i), I could push traffic to testpmd at other end and I can observe TX and forwarded RX traffic at trex-console/tui terminal.

5) When I run trex for multiple cores, I am getting following error.

Please help me in resolving this error.

If you need any other details, please let me know.

Error:

====

# ./t-rex-64 -i -c 2
Starting Scapy server.... Scapy server is started
Trying to bind to vfio-pci ...
/usr/bin/python3 dpdk_nic_bind.py --bind=vfio-pci 0000:41:00.0 0000:41:00.1
/opt/trex/trex-core/scripts/dpdk_nic_bind.py:40: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
from distutils.util import strtobool
The ports are bound/configured.
Starting TRex v3.07 please wait ...
set driver name net_bnxt
driver capability : TCP_UDP_OFFLOAD TSO LRO
set dpdk queues mode to MULTI_QUE
Number of ports found: 2
zmq publisher at: tcp://*:4500
BNXT: bnxt_stats_reset_op(): Device Initialization not complete!
BNXT: bnxt_stats_reset_op(): Device Initialization not complete!
BNXT: bnxt_alloc_hwrm_rx_ring(): ring0 bnxt_init_one_rx_ring failed!
BNXT: bnxt_alloc_hwrm_rx_ring(): Failed to allocate receive queue 0, rc -12.
BNXT: bnxt_start_nic(): HWRM ring alloc failure rc: fffffff4
EAL: Error disabling MSI-X interrupts for fd 31
BNXT: bnxt_hwrm_port_phy_cfg_v2(): Force Link Down
BNXT: bnxt_alloc_hwrm_rx_ring(): ring0 bnxt_init_one_rx_ring failed!
BNXT: bnxt_alloc_hwrm_rx_ring(): Failed to allocate receive queue 0, rc -12.
BNXT: bnxt_start_nic(): HWRM ring alloc failure rc: fffffff4
EAL: Error disabling MSI-X interrupts for fd 31
BNXT: bnxt_hwrm_port_phy_cfg_v2(): Force Link Down
BNXT: bnxt_alloc_hwrm_rx_ring(): ring0 bnxt_init_one_rx_ring failed!
BNXT: bnxt_alloc_hwrm_rx_ring(): Failed to allocate receive queue 0, rc -12.
BNXT: bnxt_start_nic(): HWRM ring alloc failure rc: fffffff4
EAL: Error disabling MSI-X interrupts for fd 31
BNXT: bnxt_hwrm_port_phy_cfg_v2(): Force Link Down
BNXT: bnxt_alloc_hwrm_rx_ring(): ring0 bnxt_init_one_rx_ring failed!
BNXT: bnxt_alloc_hwrm_rx_ring(): Failed to allocate receive queue 0, rc -12.
BNXT: bnxt_start_nic(): HWRM ring alloc failure rc: fffffff4
EAL: Error disabling MSI-X interrupts for fd 31
BNXT: bnxt_hwrm_port_phy_cfg_v2(): Force Link Down

System details (Running t-rex):

=======================

41:00.0 Ethernet controller: Broadcom Inc. and subsidiaries Device 1760 (rev 11)
41:00.1 Ethernet controller: Broadcom Inc. and subsidiaries Device 1760 (rev 11)

DPDK version: v25.07

# modinfo bnxt_en | head -n 30

filename: /lib/modules/6.8.0-85-generic/kernel/drivers/net/ethernet/broadcom/bnxt/bnxt_en.ko

description: Broadcom BCM573xx network driver

license: GPL

srcversion: 5B7F03F6B8CC897DAE109B9

# cat /etc/trex_cfg.yaml
### Config file generated by dpdk_setup_ports.py ###
- version: 2
interfaces : ["41:00.0","41:00.1"] # list of the interfaces to bind run ./dpdk_nic_bind.py --status to see the list

port_info:
- dest_mac:  <redacted>
src_mac:  <redacted>
- dest_mac:  <redacted>
src_mac: <redacted>

platform:
master_thread_id: 0
latency_thread_id: 1
rx_desc: 4096
dual_if:
- socket: 0
threads: [24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47]

# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.8.0-85-generic root=UUID=<uuid> ro quiet splash iommu=pt amd_iommu=on default_hugepagesz=1G hugepagesz=1G hugepages=64 isolcpus=24-47 nohz=on nohz_full=24-47 rcu_nocbs=24-47 numa_balancing=disable nmi_watchdog=1 audit=0 hpet=disable tsc=reliable nosoftlockup selinux=0 processor.max_cstate=0 vt.handoff=7

# /usr/local/bin/dpdk-hugepages.py --show
Node Pages Size Total
0 64 1Gb 64Gb

# cat /proc/meminfo | grep Huge
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
FileHugePages: 0 kB
HugePages_Total: 64
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 1048576 kB
Hugetlb: 67108864 kB

System details:

===========

Linux version: 6.14.0-33-generic

01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries BCM57608 10Gb/25Gb/50Gb/100Gb/200Gb/400Gb Ethernet (rev 11)

01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries BCM57608 10Gb/25Gb/50Gb/100Gb/200Gb/400Gb Ethernet (rev 11)

DPDK version: v25.07

# modinfo bnxt_en | head -n 10

filename: /lib/modules/6.14.0-33-generic/kernel/drivers/net/ethernet/broadcom/bnxt/bnxt_en.ko.zst

description: Broadcom NetXtreme network driver

license: GPL

srcversion: 03A31CDE5FD17557EFDED9B

bigevilbeard · ‎10-16-2025

@Siba- based on your output shared, it looks like your issue is the broadcom firmware/driver and the trex configuration, around how its handeling multi-queue/multi-core operations, i guess this is why there is the many errors repeated bnxt_alloc_hwrm_rx_ring() failures.

Some google searches suggest to update the /etc/trex_cfg.yaml to only list the cores you intend to use here. But my thinking is checking if these two versions are compatible for multi-queue operation on the adapter you are using, and maybe swapping to use the official DPDK version packaged with this trex.

If you have not done so already, i would suggest opening an issue on the repo for the maintainers, there is also support groups listed here too https://github.com/cisco-system-traffic-generator/trex-core/blob/master/CONTRIBUTING.md

Hope this helps.

Please mark this as helpful or solution accepted to help others
Connect with me https://bigevilbeard.github.io

Siba- · ‎10-16-2025

@bigevilbeard Thanks for quick response.

I am using dpdk package that comes with https://github.com/cisco-system-traffic-generator/trex-core.git by compile it from linux_dpdk folder.

I will see if changing driver and firmware help in resolving this issue.

Thanks