High CPU load at the FMC (7.0.4) caused by python3 processes

[swscco001]

Hello everybody,

our customer has a FMCv 7.0.4 and two Firepower1120 running 7.0.4.

This morning I got a ticket because high CPU load on the FMC:

Service: CPU Load (nwc_cisco_cpu-load) 
CRITICAL - load-15 is 15.81 (15 min Load Average too high (= 15.81)), cpu (total): 58.17%
user: 56.17%
system: 1.91%
nice: 0.00%
wait: 0.09%
kernel: 0.00%
interrupt: 0.00%
load-1 is 2.40
load-5 is 7.21

On the FMC the Health Monitor shows no issue (see attached).

When I went on the CLI of the FMC and enter the 'top' command I got this:

root@firepower:/Volume/home/swsadmin# top
top - 09:24:16 up 28 days, 15:57,  1 user,  load average: 2.38, 2.46, 2.44
Tasks: 231 total,   3 running, 228 sleeping,   0 stopped,   0 zombie
%Cpu(s): 54.2 us,  1.3 sy,  0.0 ni, 44.3 id,  0.0 wa,  0.0 hi,  0.2 si,  0.0 st
MiB Mem :  32173.9 total,  12219.3 free,  15749.5 used,   4205.1 buff/cache
MiB Swap:   6728.7 total,   6727.6 free,      1.0 used.  15934.0 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
24004 lamplig+  20   0  701744  22452   7692 R  99.0   0.1   1160:33 python3     <=======
24074 lamplig+  20   0   92192  21896   8608 R  98.7   0.1   1149:59 python3     <=======
 8138 casuser   20   0   30.1g   1.4g  23988 S   7.6   4.4   3265:29 dbsrv17
 9832 root      20   0 4841868 225512   7192 S   5.3   0.7   1067:57 beam.smp
13164 root      20   0 1233648 397876  19132 S   2.0   1.2  40:34.50 /var/sf/bin/sfd
 8092 root      20   0  705304  17720  13768 S   0.7   0.1 216:40.50 CloudAgent
 8095 mysql     20   0 3255048   1.9g  16216 S   0.7   6.2   1574:49 mariadbd
 9568 root      20   0  407412   8148   5984 S   0.7   0.0 210:12.23 sftunnel
11269 www       20   0  609924 116340  17100 S   0.7   0.4  94:12.78 SFTop10Cacher
15033 monetdb   20   0 1993584 413220 208008 S   0.7   1.3  96:59.41 mserver5
 7124 swsadmin  20   0   14320   4624   3460 S   0.3   0.0   0:00.02 sshd
 9592 root      20   0 2300532  45300  25404 S   0.3   0.1 161:09.87 adi
 9597 lamplig+  20   0  148748  53372  23612 S   0.3   0.2  56:37.12 mongod
11091 www       20   0   39236  35932   8516 S   0.3   0.1  98:37.68 SFTop10Cacher
11136 root      20   0  191576 189852  85268 S   0.3   0.6  27:39.31 vault
12682 root      20   0 4606052   1.2g  21376 S   0.3   3.7 380:28.84 java
13122 root      20   0 2641968 539140  41244 S   0.3   1.6 181:32.26 SFDataCorrelato
15199 root      20   0 3969812 195132  18988 S   0.3   0.6 128:26.33 java
15275 root      20   0  679500 186592  18092 S   0.3   0.6  34:08.94 perl
23687 lamplig+  20   0 4659340 140332  18776 S   0.3   0.4   7:51.71 java
23748 lamplig+  20   0 5896336 277468  19332 S   0.3   0.8  11:02.89 java
    1 root      20   0    2292   1528   1428 S   0.0   0.0   0:15.05 init

Every time I enter the 'top' command the both python3 processes cause
high CPU load.

What is the reason for this and how could we get rid of?

Thanks a lot for every hint!



Thanks a lot!

[Cisco_Virtual_Engineer]

Hello,

It appears that the two Python3 processes are consuming a significant amount of CPU resources. This could be due to various reasons, such as unoptimized scripts, resource-intensive tasks, or even a bug in the code. Here are some steps you can take to diagnose and resolve the high CPU issue:

1. Identify the Python scripts: First, let's identify which Python scripts are running and causing the high CPU load. You can use the 'ps' command to get more information about the processes:

```
ps -aux | grep python3
```

This command will show you the full command line for the Python3 processes, including the script names and any arguments passed to them.

2. Review the Python scripts: Once you have identified the scripts, review their code to see if there are any obvious issues, such as infinite loops, unoptimized algorithms, or unnecessary resource-intensive tasks.

3. Monitor the system logs: Check the system logs for any error messages or warnings related to the Python scripts. You can use the 'journalctl' command to view the logs:

```
journalctl -u (service_name)
```

Replace `(service_name)` with the name of the service running the Python scripts, if applicable.

4. Optimize the Python scripts: If you find any issues in the scripts, try to optimize them to reduce CPU usage. This may involve refactoring the code, using more efficient algorithms, or reducing the frequency of resource-intensive tasks.

5. Restart the services or processes: If you have made changes to the scripts, restart the services or processes running the Python scripts to see if the CPU usage has improved. You can use the 'systemctl' command to restart a service:

```
systemctl restart (service_name)
```

If the Python scripts are running as standalone processes, you can use the 'kill' command to terminate them and then manually start them again:

```
kill (PID)
python3 (script_name)
```

Replace `(PID)` with the process ID of the Python script, and `(script_name)` with the name of the script.

6. Monitor the CPU usage: Continue monitoring the CPU usage using the 'top' command to see if the high CPU issue has been resolved.

If the issue persists even after following the above steps, it could be due to a bug in the software or an underlying issue with the system. In such cases, it is recommended to contact Cisco TAC for further assistance.

I hope this helps you resolve the high CPU issue on your FMC. If you have any further questions, please don't hesitate to ask.

[Cisco_Virtual_Engineer]

It appears that the two Python3 processes with high CPU usage are causing the issue. To resolve this, we need to first identify the root cause of the high CPU usage. Here are a few steps to help you troubleshoot the issue:

1. Identify the Python scripts associated with the high CPU usage processes:

Run the following command to find out more information about the Python3 processes:

```
ps aux | grep python3
```

This will show you the full command line for each process, including the script names and any arguments passed to them.

2. Analyze the log files:

Check the log files associated with the identified Python scripts to see if there are any errors or issues that could be causing the high CPU usage. The log files may be located in /var/log or a custom directory depending on the application.

3. Check for resource-intensive tasks:

The Python scripts might be running resource-intensive tasks, such as processing large amounts of data or running complex algorithms. If this is the case, you may need to optimize the scripts or allocate more resources to the FMC.

4. Examine the system and application configurations:

Review the configuration files for the applications using the Python scripts to see if there are any settings that could be causing the high CPU usage. This could include settings related to process scheduling, resource allocation, or other performance-related settings.

5. Monitor the system over time:

Keep an eye on the CPU usage over time to see if the high CPU usage is consistent or if it only occurs during specific periods or under specific conditions. This can help you narrow down the cause of the issue.

6. Update the software:

Ensure that the FMC and its associated applications are up-to-date. This can help resolve any known issues or bugs that could be causing the high CPU usage.

If none of these steps resolve the issue, you may need to contact Cisco TAC for further assistance. They can help you analyze the issue in more detail and provide specific guidance for your environment.

[MHM Cisco World]

what is the python3 script you use ?