Hi guys,
Today i had a big problem with 2 Cisco ASA 5545-X with fire power services in active/standby mode. These are the perimeter firewalls.
The active firewall had a high CPU usage of 99%.
After working with Cisco TAC, it was identified that there was a loop between the active firewall and our internal firewall (Cisco 5585-X). For some reason, the internal firewall had a connection in the connection table that was pointing to the perimeter firewall instead of the correct interface.
The question that i have is how was this able to happen? Was wondering if this happened to anyone else so that we would be able to stop it from happening again.
Essentially what was happening is that the Perimeter firewall would send the traffic to the internal firewall and the internal firewall would then send the traffic back to the Perimeter firewall and so on.
See below for technical description:
HARDWARE/SOFTWARE: ASA5545 // ASA 9.4(2)11
SYMPTOMS: ASA experiencing 99% of CPU usage
Checked CPU and the process with the highest consumption was DATAPATH-1988:
Perimeter01/act# sh cpu
CPU utilization for 5 seconds = 91%; 1 minute: 92%; 5 minutes: 96%
Perimeter01/act# show proc cpu-us
Perimeter01/act# show proc cpu-usage sorted non
PC Thread 5Sec 1Min 5Min Process
- - 91.9% 88.5% 91.3% DATAPATH-0-1988
0x0000000000832066 0x00007fffdb41d4a0 0.3% 0.6% 0.5% CP Processing
0x0000000000976bf3 0x00007fffdb417da0 0.1% 0.1% 0.1% fover_health_monitoring_thread
0x000000000098e7fa 0x00007fffdb418fc0 0.1% 0.1% 0.1% fover_ip
0x000000000148bcf0 0x00007fffdb317d60 0.0% 0.7% 0.6% Unicorn Admin Handler
0x000000000148bcf0 0x00007fffdb3f90a0 0.0% 0.7% 0.6% Unicorn Admin Handler
0x000000000174f4f4 0x00007fffdb413c60 0.0% 0.6% 0.6% qos_metric_daemon
[] Threat-detection was enabled so, we access the ASA through ASDM and found that a lot of ICMP and ESP traffic was passing through the ASA and the following IPs were involved:
67.210.X.X
67.210.X.X
10.X.X.X
[] We decided to shun these IP address, but CPU dropped only from 99% to 88%
[] Looked at interfaces' errors and found that overruns were increasing in the GigabitEthernet 0/6 & 0/7 which are being used for Portchannel 12 which had both outside interfaces
[] We took a show tech of the ASA and found the following issues:
+ High probability of traffic loop on an interface
+ CPU hogs due to SNMP polling
Took 2 "show conn" outputs with 1 minute apart to see which were the top talkers and found the following:
Total estimated byte count diff: 3,839,427,079.00 bytes
Top talker connections connections
176,124,544.00 bytes ICMP INTERNET/DMZ 10.X.X.X:0 INTERNET/DMZ 134.159.159.138:3016
134,772,672.00 bytes ICMP INTERNET/DMZ 10.X.X.X:0 INTERNET/DMZ 134.159.159.138:3017
129,737,776.00 bytes ICMP INTERNET/DMZ 10.X.X.X:0 INTERNET/DMZ 183.232.164.41:2156
97,058,360.00 bytes ICMP INTERNET/DMZ 10.X.X.X:0 INTERNET/DMZ 203.205.176.12:6966
92,321,728.00 bytes ICMP INTERNET/DMZ 10.X.X.X:0 INTERNET/DMZ 202.41.225.22:26592
We configured a packet capture and confirmed that these traffic flows were in a loop, so we checked the internal FW and found that these packets should be sent to another interface instead of sending them back to the affected ASA but this ASA had a connection that stated the opposite (for the three internal hosts):
INTERNET/DMZ: 10.X.X.X/0 INTERNET/DMZ: 134.159.159.138/3017,
, flags , idle 0s, uptime 20h5m, timeout 2s, bytes 937721152
ICMP INTERNET/DMZ: 10.X.X.X/0 INTERNET/DMZ: 134.159.159.138/3016,
, flags , idle 0s, uptime 20h5m, timeout 2s, bytes 3519536512
So these connections for the three internal hosts and after that we saw the CPU dropped to 37% in the external FW.
