Cisco ASA 5545-X High CPU usage

Davion Stewart — Tue, 12 Mar 2019 09:12:22 GMT

Hi guys,

Today i had a big problem with 2 Cisco ASA 5545-X with fire power services in active/standby mode. These are the perimeter firewalls.

The active firewall had a high CPU usage of 99%.

After working with Cisco TAC, it was identified that there was a loop between the active firewall and our internal firewall (Cisco 5585-X). For some reason, the internal firewall had a connection in the connection table that was pointing to the perimeter firewall instead of the correct interface.

The question that i have is how was this able to happen? Was wondering if this happened to anyone else so that we would be able to stop it from happening again.

Essentially what was happening is that the Perimeter firewall would send the traffic to the internal firewall and the internal firewall would then send the traffic back to the Perimeter firewall and so on.

See below for technical description:

HARDWARE/SOFTWARE: ASA5545 // ASA 9.4(2)11

SYMPTOMS: ASA experiencing 99% of CPU usage

Checked CPU and the process with the highest consumption was DATAPATH-1988:

Perimeter01/act# sh cpu

CPU utilization for 5 seconds = 91%; 1 minute: 92%; 5 minutes: 96%

Perimeter01/act# show proc cpu-us

Perimeter01/act# show proc cpu-usage sorted non

PC Thread 5Sec 1Min 5Min Process

- - 91.9% 88.5% 91.3% DATAPATH-0-1988

0x0000000000832066 0x00007fffdb41d4a0 0.3% 0.6% 0.5% CP Processing

0x0000000000976bf3 0x00007fffdb417da0 0.1% 0.1% 0.1% fover_health_monitoring_thread

0x000000000098e7fa 0x00007fffdb418fc0 0.1% 0.1% 0.1% fover_ip

0x000000000148bcf0 0x00007fffdb317d60 0.0% 0.7% 0.6% Unicorn Admin Handler

0x000000000148bcf0 0x00007fffdb3f90a0 0.0% 0.7% 0.6% Unicorn Admin Handler

0x000000000174f4f4 0x00007fffdb413c60 0.0% 0.6% 0.6% qos_metric_daemon

[] Threat-detection was enabled so, we access the ASA through ASDM and found that a lot of ICMP and ESP traffic was passing through the ASA and the following IPs were involved:

67.210.X.X

10.X.X.X

[] We decided to shun these IP address, but CPU dropped only from 99% to 88%

[] Looked at interfaces' errors and found that overruns were increasing in the GigabitEthernet 0/6 & 0/7 which are being used for Portchannel 12 which had both outside interfaces

[] We took a show tech of the ASA and found the following issues:

+ High probability of traffic loop on an interface

+ CPU hogs due to SNMP polling

Took 2 "show conn" outputs with 1 minute apart to see which were the top talkers and found the following:

Total estimated byte count diff: 3,839,427,079.00 bytes

Top talker connections connections

176,124,544.00 bytes ICMP INTERNET/DMZ 10.X.X.X:0 INTERNET/DMZ 134.159.159.138:3016

134,772,672.00 bytes ICMP INTERNET/DMZ 10.X.X.X:0 INTERNET/DMZ 134.159.159.138:3017

129,737,776.00 bytes ICMP INTERNET/DMZ 10.X.X.X:0 INTERNET/DMZ 183.232.164.41:2156

97,058,360.00 bytes ICMP INTERNET/DMZ 10.X.X.X:0 INTERNET/DMZ 203.205.176.12:6966

92,321,728.00 bytes ICMP INTERNET/DMZ 10.X.X.X:0 INTERNET/DMZ 202.41.225.22:26592

We configured a packet capture and confirmed that these traffic flows were in a loop, so we checked the internal FW and found that these packets should be sent to another interface instead of sending them back to the affected ASA but this ASA had a connection that stated the opposite (for the three internal hosts):

INTERNET/DMZ: 10.X.X.X/0 INTERNET/DMZ: 134.159.159.138/3017,

, flags , idle 0s, uptime 20h5m, timeout 2s, bytes 937721152

ICMP INTERNET/DMZ: 10.X.X.X/0 INTERNET/DMZ: 134.159.159.138/3016,

, flags , idle 0s, uptime 20h5m, timeout 2s, bytes 3519536512

So these connections for the three internal hosts and after that we saw the CPU dropped to 37% in the external FW.

Hmm well will give feedback

Davion Stewart — Wed, 12 Apr 2017 04:20:08 GMT

Hmm well will give feedback once i finish working with Cisco TAC on the issue

Re: Hmm well will give feedback

John Eze — Thu, 14 Nov 2019 20:30:40 GMT

I am experiencing the same thing on ASA 5545-x

Cisco Adaptive Security Appliance Software Version 8.6(1)2
Device Manager Version 6.6(1)

Compiled on Fri 01-Jun-12 02:16 by builders
System image file is "disk0:/asa861-2-smp-k8.bin"
Config file at boot was "startup-config"

CSCSASA-1 up 210 days 0 hours

Hardware: ASA5545, 12288 MB RAM, CPU Lynnfield 2660 MHz, 1 CPU (8 cores)
ASA: 6144 MB RAM, 1 CPU (1 core)
Internal ATA Compact Flash, 8192MB

SCSASA-1# sh cpu usage
CPU utilization for 5 seconds = 99%; 1 minute: 99%; 5 minutes: 99%

SCSASA-1# sh processes cpu-usage sorted non-zero
PC Thread 5Sec 1Min 5Min Process
- - 95.6% 95.6% 95.6% DATAPATH-0-1417
0x00000000013382a4 0x00007ffebb728b58 2.5% 2.5% 2.5% Logger
0x00000000006c27c2 0x00007ffebb71d6f8 1.1% 1.1% 1.1% CP Processing

Re: Cisco ASA 5545-X High CPU usage

Sheraz.Salim — Thu, 14 Nov 2019 20:36:01 GMT

please do share finding once the issue is fixed. this issue seem very interesting and informative. apologies you do not want to hear this but you explain the problem very well.

topic Cisco ASA 5545-X High CPU usage in Network Security

Cisco ASA 5545-X High CPU usage

It will be a software bug. I

Hmm well will give feedback

Re: Hmm well will give feedback

Re: Cisco ASA 5545-X High CPU usage