topic Re: ASA to ASA firewall Transit link Latency Issues in Network Security

ASA to ASA firewall Transit link Latency Issues

Pyie Phyo Htay — Sat, 04 Jan 2025 06:01:41 GMT

Hello Everyone,

I am currently experiencing network latency issues on the transit link between the ASA Core Firewall and the Edge Firewall, which is using the IP range 172.12.1.0/30.
The connection from the Edge Firewall to the Internet is functioning without latency issues, and the inter-VLAN network on the Core Firewall is also performing well with no latency concerns.

Please find the testing results below. This issue has been occurring for the past four days. I kindly request assistance in investigating and resolving this issue.

Testing Edge Firewall to Internet is Fine
.............................................................
mmedgefw01# ping OUTSIDE 8.8.8.8
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 20/26/30 ms
mmedgefw01#

Testing Edge to Core Firewall is Too many Latency
..............................................................................
mmedgefw01# ping internal 172.12.1.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.1.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 370/378/380 ms
mmedgefw01#

Interface Output Details for Edge Firewall
................................................................
mmedgefw01# show int gigabitEthernet 1/8 detail
Interface GigabitEthernet1/8 "internal", is up, line protocol is up
Hardware is Accelerator rev01, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
Description: Connect From DC_ASA01 to DC_CoreASA0/5
MAC address 10b3.d50c.5e41, MTU 1500
IP address 172.12.1.1, subnet mask 255.255.255.252
5822815774095 packets input, 738732956040586 bytes, 0 no buffer
Received 6 broadcasts, 0 runts, 1 giants
2 input errors, 0 CRC, 1 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
5831673410633 packets output, 749966242957387 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 0 interface resets
0 late collisions, 0 deferred
0 input reset drops, 2815 output reset drops
input queue (blocks free curr/low): hardware (1890/1647)
output queue (blocks free curr/low): hardware (1957/863)
Traffic Statistics for "internal":
4564030233261 packets input, 462204650950260 bytes
5831673410633 packets output, 644984402563344 bytes
2027461058814 packets dropped
1 minute input rate 47471 pkts/sec, 4774615 bytes/sec
1 minute output rate 47525 pkts/sec, 4790944 bytes/sec
1 minute drop rate, 23694 pkts/sec
5 minute input rate 46094 pkts/sec, 4744695 bytes/sec
5 minute output rate 46280 pkts/sec, 5075050 bytes/sec
5 minute drop rate, 25807 pkts/sec
Control Point Interface States:
Interface number is 9
Interface config status is active
Interface state is active
mmedgefw01#

Testing Client Users to Internet is Too Many Latency
.................................................................................
C:\Users\sysadmin>ping 8.8.8.8

Pinging 8.8.8.8 with 32 bytes of data:
Reply from 8.8.8.8: bytes=32 time=463ms TTL=119
Reply from 8.8.8.8: bytes=32 time=447ms TTL=119
Reply from 8.8.8.8: bytes=32 time=448ms TTL=119
Reply from 8.8.8.8: bytes=32 time=449ms TTL=119

Ping statistics for 8.8.8.8:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 447ms, Maximum = 463ms, Average = 451ms

C:\Users\sysadmin>ping 1.1.1.1

Pinging 1.1.1.1 with 32 bytes of data:
Reply from 1.1.1.1: bytes=32 time=466ms TTL=59
Reply from 1.1.1.1: bytes=32 time=463ms TTL=59
Reply from 1.1.1.1: bytes=32 time=465ms TTL=59
Reply from 1.1.1.1: bytes=32 time=465ms TTL=59

Ping statistics for 1.1.1.1:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 463ms, Maximum = 466ms, Average = 464ms

Testing Core to Edge Firewall is Too many Latency
..............................................................................
mmcorefw01# ping outside 172.12.1.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 360/370/380 ms
mmcorefw01# ping outside 172.12.1.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.12.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 370/378/380 ms
mmcorefw01#

Testing Core Firewall Inter-vlan network is Fine
........................................................................
mmcorefw01# ping 10.30.4.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.30.4.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
mmcorefw01# ping 10.30.1.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.30.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
mmcorefw01# ping 10.30.2.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.30.2.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
mmcorefw01#

Interface Output Details for Core Firewall
................................................................

mmcorefw01# show int gigabitEthernet 1/6 detail
Interface GigabitEthernet1/6 "outside", is up, line protocol is up
Hardware is Accelerator rev01, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
Description: To_external_Firewall
MAC address 4c77.6ddb.0d17, MTU 1500
IP address 172.12.1.2, subnet mask 255.255.255.252
4470321914307 packets input, 557350686697772 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
4463793158460 packets output, 548767561908401 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 0 interface resets
0 late collisions, 0 deferred
0 input reset drops, 112 output reset drops
input queue (blocks free curr/low): hardware (1845/1785)
output queue (blocks free curr/low): hardware (1957/1543)
Traffic Statistics for "outside":
4470297536508 packets input, 476862988019735 bytes
4463793158462 packets output, 468407354516332 bytes
3891468 packets dropped
1 minute input rate 44920 pkts/sec, 4679338 bytes/sec
1 minute output rate 44912 pkts/sec, 4681721 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 46776 pkts/sec, 4777599 bytes/sec
5 minute output rate 46782 pkts/sec, 4771532 bytes/sec
5 minute drop rate, 0 pkts/sec
Control Point Interface States:
Interface number is 7
Interface config status is active
Interface state is active
mmcorefw01#

Many Thanks.

Re: ASA to ASA firewall Transit link Latency Issues

MHM Cisco World — Sat, 04 Jan 2025 09:52:36 GMT

This slow of asa' mean issue here is asa have high cpu utilize.

MHM

Re: ASA to ASA firewall Transit link Latency Issues

Pyie Phyo Htay — Sun, 05 Jan 2025 04:35:01 GMT

When I checked the Core and Edge Firewall CPU usage, it was below 64%.

Yesterday, I replaced it with a new UTP cable, and the latency returned to 1-5 ms. However, several hours ago, the latency increased significantly to 130-300 ms.

Re: ASA to ASA firewall Transit link Latency Issues

MHM Cisco World — Sun, 05 Jan 2025 09:28:18 GMT

can I see show interface detail for interface connect to inside SW

MHM

Re: ASA to ASA firewall Transit link Latency Issues

Pyie Phyo Htay — Mon, 06 Jan 2025 03:24:53 GMT

sure bro, please kindly see the output
Core Firewall to Core Switch are connected via gi 1/1 - 1/2 by using portchannel 1.

Edge Firewall CPU Usage
.......................................
mmedgefw01# show cpu usage
CPU utilization for 5 seconds = 71%; 1 minute: 71%; 5 minutes: 70%

Core Firewall CPU Usage
.......................................
mmcorefw01# show cpu usage
CPU utilization for 5 seconds = 52%; 1 minute: 53%; 5 minutes: 52%

Core Firewall to Core Switch Interfaces Details
.......................................................................
mmcorefw01# show int gigabitEthernet 1/1 detail
Interface GigabitEthernet1/1 "", is up, line protocol is up
Hardware is Accelerator rev01, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
Description: From_mmcoresw01-Gi1/0/45
Active member of Port-channel1
MAC address 4c77.6ddb.0d12, MTU not set
IP address unassigned
75939910780 packets input, 85920472514775 bytes, 0 no buffer
Received 89191516 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
27428808800 packets output, 18627299086259 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 0 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops
input queue (blocks free curr/low): hardware (1933/1820)
output queue (blocks free curr/low): hardware (2047/947)
Control Point Interface States:
Interface number is 2
Interface config status is active
Interface state is active
mmcorefw01# show int gigabitEthernet 1/2 de
mmcorefw01# show int gigabitEthernet 1/2 detail
Interface GigabitEthernet1/2 "", is up, line protocol is up
Hardware is Accelerator rev01, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
Description: From_mmcoresw01-Gi1/0/46
Active member of Port-channel1
MAC address 4c77.6ddb.0d13, MTU not set
IP address unassigned
14563444842 packets input, 8378972393927 bytes, 0 no buffer
Received 44973209 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
857264089413 packets output, 190654612789646 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 0 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops
input queue (blocks free curr/low): hardware (1978/1820)
output queue (blocks free curr/low): hardware (2047/1087)
Control Point Interface States:
Interface number is 3
Interface config status is active
Interface state is active
mmcorefw01#

Re: ASA to ASA firewall Transit link Latency Issues

Pyie Phyo Htay — Mon, 06 Jan 2025 03:27:17 GMT

Core Firewall to Core Switch Interfaces
..........................................................
interface GigabitEthernet1/1
description From_mmcoresw01-Gi1/0/45
channel-group 1 mode active
no nameif
no security-level
no ip address
!
interface GigabitEthernet1/2
description From_mmcoresw01-Gi1/0/46
channel-group 1 mode active
no nameif
no security-level
no ip address

interface Port-channel1
lacp max-bundle 8
no nameif
no security-level
no ip address
!
interface Port-channel1.10
description WEB_VLAN Interface
vlan 10
nameif WEB_VLAN
security-level 50
ip address 10.30.1.1 255.255.255.0
!
interface Port-channel1.20
description Application_VLAN Interface
vlan 20
nameif APP_VLAN
security-level 100
ip address 10.30.2.1 255.255.255.0
!
interface Port-channel1.30
description Database_VLAN Interface
vlan 30
nameif DB_VLAN
security-level 100
ip address 10.30.3.1 255.255.255.0
!
interface Port-channel1.40
description Management_VLAN Interface
vlan 40
nameif MGMT_VLAN
security-level 100
ip address 10.30.4.1 255.255.255.0
!
interface Port-channel1.60
description VPN_Vlan Interface
vlan 60
nameif VPN_Vlan
security-level 100
ip address 10.30.6.1 255.255.255.0
!
interface Port-channel1.70
description API_VLAN
vlan 70
nameif API_VLAN
security-level 20
ip address 10.30.7.1 255.255.255.0
!
interface Port-channel1.80
description Virtualization VLAN Interface
vlan 80
nameif Virtualization_VLAN
security-level 100
ip address 10.30.8.1 255.255.255.0
!
interface Port-channel1.110
description Inter_VLAN Interface
vlan 110
nameif Inter_VLAN
security-level 100
ip address 10.30.11.1 255.255.255.0
!
interface Port-channel1.237
description Inter_JUMP
vlan 237
nameif Inter_JUMP
security-level 0
ip address 10.30.237.1 255.255.255.0
!
interface Port-channel1.238
description P_WIFI3
vlan 238
nameif P_WIFI3
security-level 0
ip address 10.30.238.1 255.255.255.0
!
interface Port-channel1.239
description Inter_JUMP_Share_Acc
vlan 239
nameif Jump_SHARE_ACCESS
security-level 0
ip address 10.30.239.1 255.255.255.0

Re: ASA to ASA firewall Transit link Latency Issues

MHM Cisco World — Mon, 06 Jan 2025 06:34:43 GMT

asa# show processes cpu-usage sorted non-zero <<- share this the cpu is 71% this little high we need to know which process run in cpu

MHM

Re: ASA to ASA firewall Transit link Latency Issues

Pyie Phyo Htay — Mon, 06 Jan 2025 07:00:28 GMT

mmedgefw01# show cpu usage
CPU utilization for 5 seconds = 69%; 1 minute: 69%; 5 minutes: 70%
mmedgefw01# show processes cpu-usage sorted non-zero
Hardware: ASA5516
Cisco Adaptive Security Appliance Software Version 9.8(4)25
ASLR enabled, text region 7f8db8529000-7f8dbc8c7e5c
PC Thread 5Sec 1Min 5Min Process
- - 29.5% 29.5% 29.6% DATAPATH-0-2045
- - 29.3% 29.4% 29.4% DATAPATH-1-2046
0x00007f8dba9b46dc 0x00007f8da12ca000 7.0% 7.0% 7.2% Logger
0x00007f8db937beb9 0x00007f8da12b9b00 3.1% 3.0% 3.1% CP Processing
0x00007f8dbaa3b3ab 0x00007f8da12ad740 0.2% 0.1% 0.0% snmp
0x00007f8dba7cf35a 0x00007f8da12ad3a0 0.1% 0.1% 0.1% IP SLA Mon Event Processor
0x00007f8dbaa71718 0x00007f8da12adae0 0.0% 0.1% 0.1% Unicorn Proxy Thread
mmedgefw01# show processes cpu-usage sorted non-zero
Hardware: ASA5516
Cisco Adaptive Security Appliance Software Version 9.8(4)25
ASLR enabled, text region 7f8db8529000-7f8dbc8c7e5c
PC Thread 5Sec 1Min 5Min Process
- - 29.5% 29.5% 29.5% DATAPATH-0-2045
- - 29.4% 29.4% 29.4% DATAPATH-1-2046
0x00007f8dba9b46dc 0x00007f8da12ca000 7.2% 7.1% 7.2% Logger
0x00007f8db937beb9 0x00007f8da12b9b00 3.1% 3.1% 3.1% CP Processing
0x00007f8dba7cf35a 0x00007f8da12ad3a0 0.1% 0.1% 0.1% IP SLA Mon Event Processor
0x00007f8dbaa71718 0x00007f8da12adae0 0.0% 0.1% 0.0% Unicorn Proxy Thread
0x00007f8dbaa3b3ab 0x00007f8da12ad740 0.0% 0.1% 0.0% snmp

Re: ASA to ASA firewall Transit link Latency Issues

Pyie Phyo Htay — Thu, 09 Jan 2025 05:25:25 GMT

Finally, I found the solution: the issue was caused by spiking CPU usage in the logger and data path, as shown below:

PC Thread 5Sec 1Min 5Min Process
- - 29.5% 29.5% 29.5% DATAPATH-0-2045
- - 29.4% 29.4% 29.4% DATAPATH-1-2046
0x00007f8dba9b46dc 0x00007f8da12ca000 7.2% 7.1% 7.2% Logger

mmedgefw01# show logging | inc ASA-2
%ASA-2-106016: Deny IP .30.4.99 on interface internal
%ASA-2-106016: Deny IP spoof from (172.12.1.1) to 10.30.4.99 on interface internal
%ASA-2-106016: Deny IP spoof from (172.12.1.1) to 10.30.4.99 on interface internal
%ASA-2-106016: Deny IP spoof from (172.12.1.1) to 10.30.4.99 on interface internal
%ASA-2-106016: Deny IP spoof from (172.12.1.1) to 10.30.4.99 on interface internal

mmcorefw01# show logging | inc ASA-4
0:08:53: %ASA-4-106023: Deny tcp src T1_Immg_Client:172.17.2.158/60166 dst outside:72.52.178.23/443 by access-group "imm-client-acl" [0x0, 0x0]
Jan 07 2025 00:08:53: %ASA-4-106023: Deny tcp src T1_Immg_Client:172.17.2.162/49387 dst outside:172.202.163.200/443 by access-group "imm-client-acl" [0x0, 0x0]
Jan 07 2025 00:08:53: %ASA-4-106023: Deny tcp src T1_Immg_Client:172.17.2.162/49389 dst outside:172.202.163.200/443 by access-group "imm-client-acl" [0x0, 0x0]
Jan 07 2025 00:08:53: %ASA-4-106023: Deny tcp src T1_Immg_Client:172.17.2.144/51642 dst outside:23.202.180.199/443 by access-group "imm-client-acl" [0x0, 0x0]

I attempted to look up the logged level with a "Deny" status due to a misconfiguration in the syslog server configuration on the firewall. The client PCs were sending some traffic to the internet, and the associated ACL was corrected.
To analyze the issue further, I used the show asp drop command and observed the output as shown below.
Subsequently, I executed the clear asp drop command, which resolved the latency issues.

mmedgefw01# show asp drop

Frame drop:
NAT-T keepalive message (natt-keepalive) 4967060
IPSEC tunnel is down (ipsec-tun-down) 4090
SVC Module does not have a channel for reinjection (mp-svc-no-channel) 3132
SVC Module does not have a session (mp-svc-no-session) 2882
SVC Module is in flow control (mp-svc-flow-control) 1255855
SVC Module unable to fragment packet (mp-svc-no-fragment) 194
VPN reclassify failed (vpn-reclassify-failed) 883
Invalid TCP Length (invalid-tcp-hdr-length) 347
Invalid UDP Length (invalid-udp-length) 180
No valid adjacency (no-adjacency) 8451140
No route to host (no-route) 4614275
Reverse-path verify failed (rpf-violated) 156
Flow is denied by configured rule (acl-drop) 2035733454858
Invalid SPI (np-sp-invalid-spi) 23982
First TCP packet not SYN (tcp-not-syn) 2847608
Bad TCP flags (bad-tcp-flags) 4
TCP Dual open denied (tcp-dual-open) 1489
TCP data send after FIN (tcp-data-past-fin) 7
TCP failed 3 way handshake (tcp-3whs-failed) 9586214
TCP RST/FIN out of order (tcp-rstfin-ooo) 3869990
TCP SEQ in SYN/SYNACK invalid (tcp-seq-syn-diff) 20856
TCP SYNACK on established conn (tcp-synack-ooo) 16115
TCP packet SEQ past window (tcp-seq-past-win) 147766
TCP invalid ACK (tcp-invalid-ack) 3161
TCP Out-of-Order packet buffer full (tcp-buffer-full) 2
TCP Out-of-Order packet buffer timeout (tcp-buffer-timeout) 100
TCP RST/SYN in window (tcp-rst-syn-in-win) 35868
TCP dup of packet in Out-of-Order queue (tcp-dup-in-queue) 416
TCP packet failed PAWS test (tcp-paws-fail) 6960
SSL first record invalid (ssl-first-record-invalid) 88
CTM returned error (ctm-error) 222
Early security checks failed (security-failed) 9938
Slowpath security checks failed (sp-security-failed) 161465
IP option drop (invalid-ip-option) 9890
ICMP Inspect bad icmp code (inspect-icmp-bad-code) 89
ICMP Inspect seq num not matched (inspect-icmp-seq-num-not-matched) 1625477
ICMP Error Inspect no existing conn (inspect-icmp-error-no-existing-conn) 4
DNS Inspect id not matched (inspect-dns-id-not-matched) 1023
FP L2 rule drop (l2_acl) 86852019
Unable to obtain connection lock (connection-lock) 3
Interface is down (interface-down) 29602393
Dropped pending packets in a closed socket (np-socket-closed) 2903
Dispatch queue tail drops (dispatch-queue-limit) 1258781304057
NAT failed (nat-xlate-failed) 37
Fragment reassembly failed (fragment-reassembly-failed) 1208748

Last clearing: Never

Flow drop:
Tunnel being brought up or torn down (tunnel-pending) 1766
SVC replacement connection established (svc-replacement-conn) 1138
VPN overlap conflict (vpn-overlap-conflict) 4
VPN decryption missing (vpn-missing-decrypt) 1660838
NAT reverse path failed (nat-rpf-failed) 3782
Inspection failure (inspect-fail) 202652
SSL bad record detected (ssl-bad-record-detect) 8684
SSL handshake failed (ssl-handshake-failed) 169547
DTLS hello processed and closed (dtls-hello-close) 4762
SSL record decryption failed (ssl-record-decrypt-error) 9
SVC inner policy mismatch failure (svc-selector-failure) 992856

Last clearing: Never

mmedgefw01# clear asp drop
mmedgefw01# show asp drop

Frame drop:
Flow is denied by configured rule (acl-drop) 101130
TCP RST/FIN out of order (tcp-rstfin-ooo) 2
FP L2 rule drop (l2_acl) 3
Interface is down (interface-down) 1
Dispatch queue tail drops (dispatch-queue-limit) 191

Last clearing: 11:51:17 RGN Jan 7 2025 by enable_15

Flow drop:

Re: ASA to ASA firewall Transit link Latency Issues

MHM Cisco World — Thu, 09 Jan 2025 06:24:46 GMT

Sorry but I dont think so'

The issue come from datapth not from logger' logger is only 7% datapth around 55%

- - 29.5% 29.5% 29.5% DATAPATH-0-2045
- - 29.4% 29.4% 29.4% DATAPATH-1-2046
0x00007f8dba9b46dc 0x00007f8da12ca000 7.2% 7.1% 7.2% Logger

There was command to check stuck datapth but I forget it' I will search again to find it and update you' in same time if issue happened again open TAC.

Datapth stuck is hard to solve.

MHM

Re: ASA to ASA firewall Transit link Latency Issues

ccieexpert — Thu, 09 Jan 2025 06:36:33 GMT

Good job fixing it yourself 🙂

clear asp drop by itself does nothing other just clear the counters.. i presume you reduced the load by fixing some logging also ACL which was constantly blocking some legitimate.. also you have some ip spoof you may want to look at it and fix it to again reduce the load.. ..Is that right ?