TRex crashes with pyats job run

[aabdu]

Hi, 

When I start trex manually, via:

cd /opt/trex/DevX-TRex-v23.05-v2.99-63

&

./t-rex-64 -i

it runs in a stable state. However once I run pyats run job, TRex crashes with:

./t-rex-64: line 106: 184608 Segmentation fault      (core dumped) ./_$(basename $0) $INPUT_ARGS $EXTRA_INPUT_ARGS

From the pyats env where I ran the job...Here, I was able to restart trex again after it crashed in time for the job to proceed:

33: 2024-04-01T10:02:40:  +..............................................................................+

34: 2024-04-01T10:02:40:  :           Starting STEP 1: Initializing the environment for Genie            :

35: 2024-04-01T10:02:40:  +..............................................................................+

36: 2024-04-01T10:02:40:  New l3 vni major version: 10.2

37: 2024-04-01T10:02:40:  New l3 vni minor version: 2.0

38: 2024-04-01T10:02:40:  Testsuite major version: 10.2

39: 2024-04-01T10:02:40:  Testsuite minor version: 2.0

40: 2024-04-01T10:02:40:  The result of STEP 1: Initializing the environment for Genie is => PASSED

41: 2024-04-01T10:02:40:  +..............................................................................+

42: 2024-04-01T10:02:40:  :                   Starting STEP 2: Connect to the testbed                    :

43: 2024-04-01T10:02:40:  +..............................................................................+

44: 2024-04-01T10:02:41:  Unable to load TRex Client v2.39: invalid syntax (trex_stl_client.py, line 616)

45: 2024-04-01T10:02:42:  TRex Client v2.60 loaded

46: 2024-04-01T10:02:44:  TRex Client v2.88 loaded

47: 2024-04-01T10:02:45:  Selected TRex HLTAPI implementation: Routem+IOL HLTAPI

48: 2024-04-01T10:02:45:  Recognized TRex v2.99

49: 2024-04-01T10:02:45:  Found compatible client for server v2.99: client v2.88

How can I find out what is causing the crash? Where can access the crash logs or where the core dumped? What causes segmentation faults?

Thanks,
Amal

[bigevilbeard]

Your error message mentions a "core dump." This file contains information about the program's state at the time of the crash. You could use a debugger like gdb to analyze the core dump and identify the line of code that caused the issue. The core dump might be saved in the current working directory (where you ran PyATS) or a designated location specified in your system configuration. Look for a file with a name like core.<pid>, where <pid> is the process ID of the crashed TRex instance.

I would also check your PyATS logs for any additional error messages or clues related to the TRex crash, these might provide more context about the issue you are seeing. 

I would also create an issue here https://github.com/cisco-system-traffic-generator on the repo relevant.