Cisco ASA 5520 High CPU usage | nat-no-xlate-to-pat-pool

[Neji Jihed]

Hello,

Our Cisco ASA 5520 firewall is running with 99% CPU, Processes Dispatch Unit is using over 90 % of CPU, and capture is showing below drop reason :

Drop-reason: (nat-no-xlate-to-pat-pool) Connection to PAT address without pre-existing xlate

firewall(config)# show processes cpu-usage sorted non-zero
PC Thread 5Sec 1Min 5Min Process
0x082a430c 0x6edd4ee4 98.5% 98.5% 97.8% Dispatch Unit
0x0911063d 0x6edad768 0.2% 0.2% 0.4% ssh
0x082be9da 0x6edcb07c 0.1% 0.1% 0.1% Logger
0x08502b76 0x6edc0ff0 0.1% 0.1% 0.1% fover_health_monitoring_thread
firewall(config)#


Any thoughts ?
Thank you,

[Bogdan Nita]

I do not think the it's the same problem.

I usually see 'PAT address without pre-existing xlate' for missconfigured nat rules.

The dispatch unit is the central packet processing process and for high dispatch cpu you usually need to have a look at traffic.

Show traffic, show perfmon and sh asp drop can give you an idea where the problem is.

HTH

Bogdan

[Neji Jihed]

Thank you for your reply, here is the output of the commands :
firewall# show asp drop

Frame drop:
<snp_drop_first> (<snp_drop_first>) 156238
Invalid encapsulation (invalid-encap) 11421
Invalid TCP Length (invalid-tcp-hdr-length) 15
No valid adjacency (no-adjacency) 686
Flow is denied by configured rule (acl-drop) 699140
First TCP packet not SYN (tcp-not-syn) 16580
Bad TCP flags (bad-tcp-flags) 84
TCP data send after FIN (tcp-data-past-fin) 25
TCP failed 3 way handshake (tcp-3whs-failed) 31654
TCP RST/FIN out of order (tcp-rstfin-ooo) 102300
TCP SEQ in SYN/SYNACK invalid (tcp-seq-syn-diff) 10
TCP SYNACK on established conn (tcp-synack-ooo) 91
TCP packet SEQ past window (tcp-seq-past-win) 172
TCP invalid ACK (tcp-invalid-ack) 18683
TCP replicated flow pak drop (tcp-fo-drop) 646
TCP Out-of-Order packet buffer full (tcp-buffer-full) 8910
TCP Out-of-Order packet buffer timeout (tcp-buffer-timeout) 1762
TCP RST/SYN in window (tcp-rst-syn-in-win) 77771
TCP dup of packet in Out-of-Order queue (tcp-dup-in-queue) 687
Slowpath security checks failed (sp-security-failed) 293388
Expired flow (flow-expired) 1
DNS Inspect packet too long (inspect-dns-pak-too-long) 98526
DNS Inspect id not matched (inspect-dns-id-not-matched) 45
FP L2 rule drop (l2_acl) 981246
Interface is down (interface-down) 165
Dropped pending packets in a closed socket (np-socket-closed) 6
Connection to PAT address without pre-existing xlate (nat-no-xlate-to-pat-pool) 33903
Received a multicast packet in the non-active device (mcast-in-nonactive-device) 53592

Last clearing: Never

Flow drop:
Inspection failure (inspect-fail) 168
Failed to allocate inspection (no-inspect) 10882
Inspect scansafe server not reachable (inspect-scansafe-server-not-reachable) 12548

Last clearing: Never
firewall# show perfmon

PERFMON STATS: Current Average
Xlates 765/s 5/s
Connections 2419/s 29/s
TCP Conns 2393/s 17/s
UDP Conns 24/s 10/s
URL Access 0/s 0/s
URL Server Req 0/s 0/s
TCP Fixup 234/s 20/s
TCP Intercept Established Conns 0/s 0/s
TCP Intercept Attempts 0/s 0/s
TCP Embryonic Conns Timeout 32/s 0/s
HTTP Fixup 234/s 20/s
FTP Fixup 0/s 0/s
AAA Authen 0/s 0/s
AAA Author 0/s 0/s
AAA Account 0/s 0/s

VALID CONNS RATE in TCP INTERCEPT: Current Average
N/A 94.44%
firewall#
firewall# show traffic
outside:
received (in 145000.180 secs):
157445561 packets 87290108247 bytes
1026 pkts/sec 601022 bytes/sec
transmitted (in 145000.180 secs):
160008603 packets 51506826224 bytes
1014 pkts/sec 355011 bytes/sec
1 minute input rate 10087 pkts/sec, 6751982 bytes/sec
1 minute output rate 8947 pkts/sec, 2867252 bytes/sec
1 minute drop rate, 29 pkts/sec
5 minute input rate 10337 pkts/sec, 7120569 bytes/sec
5 minute output rate 9269 pkts/sec, 2889681 bytes/sec
5 minute drop rate, 36 pkts/sec
inside:
received (in 145000.140 secs):
245590275 packets 53353651093 bytes
1012 pkts/sec 367008 bytes/sec
transmitted (in 145000.140 secs):
247605323 packets 89356654158 bytes
1026 pkts/sec 616015 bytes/sec
1 minute input rate 12949 pkts/sec, 2883846 bytes/sec
1 minute output rate 13753 pkts/sec, 6539266 bytes/sec
1 minute drop rate, 3 pkts/sec
5 minute input rate 13019 pkts/sec, 2931288 bytes/sec
5 minute output rate 13480 pkts/sec, 6204603 bytes/sec
5 minute drop rate, 3 pkts/sec
dmz-front-in:
received (in 145048.060 secs):
1120220 packets 141630196 bytes
7 pkts/sec 28 bytes/sec
transmitted (in 145048.060 secs):
1008390 packets 318832727 bytes
6 pkts/sec 2020 bytes/sec
1 minute input rate 11 pkts/sec, 1618 bytes/sec
1 minute output rate 11 pkts/sec, 4643 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 11 pkts/sec, 1320 bytes/sec
5 minute output rate 10 pkts/sec, 4419 bytes/sec
5 minute drop rate, 0 pkts/sec
dmz-front-out:
received (in 145048.090 secs):
291645583 packets 68016082615 bytes
2010 pkts/sec 468002 bytes/sec
transmitted (in 145048.090 secs):
303334413 packets 64088215895 bytes
2002 pkts/sec 441012 bytes/sec
1 minute input rate 14567 pkts/sec, 3913761 bytes/sec
1 minute output rate 15518 pkts/sec, 3722192 bytes/sec
1 minute drop rate, 31 pkts/sec
5 minute input rate 14637 pkts/sec, 3944269 bytes/sec
5 minute output rate 15595 pkts/sec, 3762372 bytes/sec
5 minute drop rate, 29 pkts/sec
dmz-back:
received (in 145048.120 secs):
46612 packets 4903621 bytes
0 pkts/sec 4 bytes/sec
transmitted (in 145048.120 secs):
42585 packets 2940933 bytes
0 pkts/sec 20 bytes/sec
1 minute input rate 0 pkts/sec, 13 bytes/sec
1 minute output rate 0 pkts/sec, 21 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 0 pkts/sec, 37 bytes/sec
5 minute output rate 0 pkts/sec, 26 bytes/sec
5 minute drop rate, 0 pkts/sec
AP-Network:
received (in 145048.150 secs):
1190192 packets 334681204 bytes
8 pkts/sec 2011 bytes/sec
transmitted (in 145048.150 secs):
225612 packets 82728425 bytes
1 pkts/sec 7 bytes/sec
1 minute input rate 20 pkts/sec, 7265 bytes/sec
1 minute output rate 11 pkts/sec, 4077 bytes/sec
1 minute drop rate, 11 pkts/sec
5 minute input rate 20 pkts/sec, 6536 bytes/sec
5 minute output rate 13 pkts/sec, 7639 bytes/sec
5 minute drop rate, 10 pkts/sec
GUEST:
received (in 145048.190 secs):
8912092 packets 1782711057 bytes
2 pkts/sec 12023 bytes/sec
transmitted (in 145048.190 secs):
14924003 packets 17921670138 bytes
14 pkts/sec 123023 bytes/sec
1 minute input rate 615 pkts/sec, 135517 bytes/sec
1 minute output rate 1017 pkts/sec, 1140473 bytes/sec
1 minute drop rate, 18 pkts/sec
5 minute input rate 876 pkts/sec, 144216 bytes/sec
5 minute output rate 1527 pkts/sec, 1859723 bytes/sec
5 minute drop rate, 15 pkts/sec
FAILOVER-LINK:
received (in 144996.240 secs):
214410 packets 17186932 bytes
1 pkts/sec 0 bytes/sec
transmitted (in 144996.240 secs):
214784 packets 16716916 bytes
1 pkts/sec 26 bytes/sec
1 minute input rate 1 pkts/sec, 105 bytes/sec
1 minute output rate 1 pkts/sec, 123 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 1 pkts/sec, 107 bytes/sec
5 minute output rate 1 pkts/sec, 125 bytes/sec
5 minute drop rate, 0 pkts/sec
STATE-LINK:
received (in 145048.250 secs):
90770328 packets 108984956920 bytes
3 pkts/sec 751015 bytes/sec
transmitted (in 145048.250 secs):
35940252 packets 43712386930 bytes
10 pkts/sec 301009 bytes/sec
1 minute input rate 1 pkts/sec, 46 bytes/sec
1 minute output rate 1775 pkts/sec, 2171295 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 1 pkts/sec, 46 bytes/sec
5 minute output rate 1775 pkts/sec, 2170648 bytes/sec
5 minute drop rate, 0 pkts/sec

----------------------------------------
Aggregated Traffic on Physical Interface
----------------------------------------
GigabitEthernet0/0:
received (in 145048.290 secs):
157513370 packets 90448517054 bytes
1026 pkts/sec 623012 bytes/sec
transmitted (in 145048.290 secs):
160011233 packets 54892280096 bytes
1014 pkts/sec 378026 bytes/sec
1 minute input rate 10084 pkts/sec, 6945147 bytes/sec
1 minute output rate 8946 pkts/sec, 3055433 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 10337 pkts/sec, 7318134 bytes/sec
5 minute output rate 9268 pkts/sec, 3082815 bytes/sec
5 minute drop rate, 0 pkts/sec
GigabitEthernet0/1:
received (in 145048.320 secs):
245649397 packets 58616975884 bytes
1012 pkts/sec 404001 bytes/sec
transmitted (in 145048.320 secs):
247609055 packets 94322080391 bytes
1026 pkts/sec 650013 bytes/sec
1 minute input rate 12954 pkts/sec, 3163926 bytes/sec
1 minute output rate 13754 pkts/sec, 6808964 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 13019 pkts/sec, 3203949 bytes/sec
5 minute output rate 13480 pkts/sec, 6470336 bytes/sec
5 minute drop rate, 0 pkts/sec
GigabitEthernet0/2:
received (in 145048.360 secs):
292818281 packets 75392769089 bytes
2018 pkts/sec 519006 bytes/sec
transmitted (in 145048.360 secs):
304389224 packets 71394235471 bytes
2009 pkts/sec 492002 bytes/sec
1 minute input rate 14582 pkts/sec, 4277611 bytes/sec
1 minute output rate 15536 pkts/sec, 4091740 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 14648 pkts/sec, 4305440 bytes/sec
5 minute output rate 15606 pkts/sec, 4123903 bytes/sec
5 minute drop rate, 0 pkts/sec
GigabitEthernet0/3:
received (in 145048.390 secs):
10118109 packets 2354688844 bytes
10 pkts/sec 16026 bytes/sec
transmitted (in 145048.390 secs):
15150063 packets 18340611647 bytes
15 pkts/sec 126000 bytes/sec
1 minute input rate 636 pkts/sec, 157778 bytes/sec
1 minute output rate 1030 pkts/sec, 1170485 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 897 pkts/sec, 171564 bytes/sec
5 minute output rate 1541 pkts/sec, 1902028 bytes/sec
5 minute drop rate, 0 pkts/sec
Management0/0:
received (in 145048.420 secs):
90732754 packets 112448610200 bytes
3 pkts/sec 775011 bytes/sec
transmitted (in 145048.420 secs):
36068682 packets 44382019856 bytes
11 pkts/sec 305003 bytes/sec
1 minute input rate 2 pkts/sec, 213 bytes/sec
1 minute output rate 1776 pkts/sec, 2203227 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 2 pkts/sec, 213 bytes/sec
5 minute output rate 1776 pkts/sec, 2202748 bytes/sec
5 minute drop rate, 0 pkts/sec
firewall#

Regards,

[Mohammed al Baqari]

Can you get the output of show perfmon detail?

Clear asp drop then issue the command show asp drop 5 times and post the
output.

[Neji Jihed]

Thank you for your reply, i have cleared the asp drop and issued the "show asp drop" five times, here is the output :

firewall# show asp drop

Frame drop:
<snp_drop_first> (<snp_drop_first>) 156238
Invalid encapsulation (invalid-encap) 11421
Invalid TCP Length (invalid-tcp-hdr-length) 15
No valid adjacency (no-adjacency) 686
Flow is denied by configured rule (acl-drop) 699140
First TCP packet not SYN (tcp-not-syn) 16580
Bad TCP flags (bad-tcp-flags) 84
TCP data send after FIN (tcp-data-past-fin) 25
TCP failed 3 way handshake (tcp-3whs-failed) 31654
TCP RST/FIN out of order (tcp-rstfin-ooo) 102300
TCP SEQ in SYN/SYNACK invalid (tcp-seq-syn-diff) 10
TCP SYNACK on established conn (tcp-synack-ooo) 91
TCP packet SEQ past window (tcp-seq-past-win) 172
TCP invalid ACK (tcp-invalid-ack) 18683
TCP replicated flow pak drop (tcp-fo-drop) 646
TCP Out-of-Order packet buffer full (tcp-buffer-full) 8910
TCP Out-of-Order packet buffer timeout (tcp-buffer-timeout) 1762
TCP RST/SYN in window (tcp-rst-syn-in-win) 77771
TCP dup of packet in Out-of-Order queue (tcp-dup-in-queue) 687
Slowpath security checks failed (sp-security-failed) 293388
Expired flow (flow-expired) 1
DNS Inspect packet too long (inspect-dns-pak-too-long) 98526
DNS Inspect id not matched (inspect-dns-id-not-matched) 45
FP L2 rule drop (l2_acl) 981246
Interface is down (interface-down) 165
Dropped pending packets in a closed socket (np-socket-closed) 6
Connection to PAT address without pre-existing xlate (nat-no-xlate-to-pat-pool) 33903
Received a multicast packet in the non-active device (mcast-in-nonactive-device) 53592

Last clearing: Never

Flow drop:
Inspection failure (inspect-fail) 168
Failed to allocate inspection (no-inspect) 10882
Inspect scansafe server not reachable (inspect-scansafe-server-not-reachable) 12548

Last clearing: Never

firewall# clear asp drop
firewall# show asp drop

Frame drop:
<snp_drop_first> (<snp_drop_first>) 15
Invalid encapsulation (invalid-encap) 1
Flow is denied by configured rule (acl-drop) 19
First TCP packet not SYN (tcp-not-syn) 1
TCP failed 3 way handshake (tcp-3whs-failed) 1
TCP RST/FIN out of order (tcp-rstfin-ooo) 32
TCP RST/SYN in window (tcp-rst-syn-in-win) 43
Slowpath security checks failed (sp-security-failed) 4
DNS Inspect packet too long (inspect-dns-pak-too-long) 16
FP L2 rule drop (l2_acl) 22
Connection to PAT address without pre-existing xlate (nat-no-xlate-to-pat-pool) 3

Last clearing: 11:16:18 met_dst Mar 28 2018 by jihed.neji

Flow drop:

Last clearing: 11:16:18 met_dst Mar 28 2018 by jihed.neji
firewall# show asp drop

Frame drop:
<snp_drop_first> (<snp_drop_first>) 41
Invalid encapsulation (invalid-encap) 1
Flow is denied by configured rule (acl-drop) 42
First TCP packet not SYN (tcp-not-syn) 2
TCP failed 3 way handshake (tcp-3whs-failed) 15
TCP RST/FIN out of order (tcp-rstfin-ooo) 46
TCP RST/SYN in window (tcp-rst-syn-in-win) 88
Slowpath security checks failed (sp-security-failed) 5
DNS Inspect packet too long (inspect-dns-pak-too-long) 48
FP L2 rule drop (l2_acl) 38
Connection to PAT address without pre-existing xlate (nat-no-xlate-to-pat-pool) 6

Last clearing: 11:16:18 met_dst Mar 28 2018 by jihed.neji

Flow drop:

Last clearing: 11:16:18 met_dst Mar 28 2018 by jihed.neji
firewall# show asp drop

Frame drop:
<snp_drop_first> (<snp_drop_first>) 94
Invalid encapsulation (invalid-encap) 1
Flow is denied by configured rule (acl-drop) 69
First TCP packet not SYN (tcp-not-syn) 2
TCP failed 3 way handshake (tcp-3whs-failed) 30
TCP RST/FIN out of order (tcp-rstfin-ooo) 66
TCP invalid ACK (tcp-invalid-ack) 1
TCP RST/SYN in window (tcp-rst-syn-in-win) 148
TCP dup of packet in Out-of-Order queue (tcp-dup-in-queue) 4
Slowpath security checks failed (sp-security-failed) 11
DNS Inspect packet too long (inspect-dns-pak-too-long) 76
FP L2 rule drop (l2_acl) 70
Connection to PAT address without pre-existing xlate (nat-no-xlate-to-pat-pool) 11

Last clearing: 11:16:18 met_dst Mar 28 2018 by jihed.neji

Flow drop:
Failed to allocate inspection (no-inspect) 2

Last clearing: 11:16:18 met_dst Mar 28 2018 by jihed.neji
firewall# show asp drop

Frame drop:
<snp_drop_first> (<snp_drop_first>) 140
Invalid encapsulation (invalid-encap) 2
Flow is denied by configured rule (acl-drop) 114
First TCP packet not SYN (tcp-not-syn) 2
TCP failed 3 way handshake (tcp-3whs-failed) 35
TCP RST/FIN out of order (tcp-rstfin-ooo) 80
TCP invalid ACK (tcp-invalid-ack) 4
TCP RST/SYN in window (tcp-rst-syn-in-win) 193
TCP dup of packet in Out-of-Order queue (tcp-dup-in-queue) 5
Slowpath security checks failed (sp-security-failed) 16
DNS Inspect packet too long (inspect-dns-pak-too-long) 92
FP L2 rule drop (l2_acl) 91
Connection to PAT address without pre-existing xlate (nat-no-xlate-to-pat-pool) 14

Last clearing: 11:16:18 met_dst Mar 28 2018 by jihed.neji

Flow drop:
Failed to allocate inspection (no-inspect) 2

Last clearing: 11:16:18 met_dst Mar 28 2018 by jihed.neji
firewall# show asp drop

Frame drop:
<snp_drop_first> (<snp_drop_first>) 194
Invalid encapsulation (invalid-encap) 2
Flow is denied by configured rule (acl-drop) 169
First TCP packet not SYN (tcp-not-syn) 3
TCP failed 3 way handshake (tcp-3whs-failed) 39
TCP RST/FIN out of order (tcp-rstfin-ooo) 103
TCP invalid ACK (tcp-invalid-ack) 21
TCP RST/SYN in window (tcp-rst-syn-in-win) 251
TCP dup of packet in Out-of-Order queue (tcp-dup-in-queue) 5
Slowpath security checks failed (sp-security-failed) 26
DNS Inspect packet too long (inspect-dns-pak-too-long) 113
FP L2 rule drop (l2_acl) 110
Connection to PAT address without pre-existing xlate (nat-no-xlate-to-pat-pool) 21

Last clearing: 11:16:18 met_dst Mar 28 2018 by jihed.neji

Flow drop:
Failed to allocate inspection (no-inspect) 2

Last clearing: 11:16:18 met_dst Mar 28 2018 by jihed.neji
firewall# show asp drop

Frame drop:
<snp_drop_first> (<snp_drop_first>) 254
Invalid encapsulation (invalid-encap) 2
Flow is denied by configured rule (acl-drop) 213
First TCP packet not SYN (tcp-not-syn) 3
TCP failed 3 way handshake (tcp-3whs-failed) 51
TCP RST/FIN out of order (tcp-rstfin-ooo) 122
TCP invalid ACK (tcp-invalid-ack) 37
TCP RST/SYN in window (tcp-rst-syn-in-win) 289
TCP dup of packet in Out-of-Order queue (tcp-dup-in-queue) 5
Slowpath security checks failed (sp-security-failed) 32
DNS Inspect packet too long (inspect-dns-pak-too-long) 128
FP L2 rule drop (l2_acl) 141
Connection to PAT address without pre-existing xlate (nat-no-xlate-to-pat-pool) 25

Last clearing: 11:16:18 met_dst Mar 28 2018 by jihed.neji

Flow drop:
Failed to allocate inspection (no-inspect) 2

Last clearing: 11:16:18 met_dst Mar 28 2018 by jihed.neji
firewall#

Thank you

[Bogdan Nita]

Following counters seem to be incriminating pretty fast: TCP RST/FIN out of order , TCP RST/SYN in window , DNS Inspect packet too long.

You could configure a capture to see exactly who is doing the traffic and verify if it is legitimate or not and if it can be stopped.

cap CAP-RST-SYN type asp-drop tcp-rst-syn-in-win buffer 1000000 circular-buffer
cap CAP-RST-FIN type asp-drop tcp-rstfin-ooo buffer 1000000 circular-buffer
cap CAP-DNS type asp-drop inspect-dns-pak-too-long buffer 1000000 circular-buffer

[Neji Jihed]

Thanks, i managed to resolve the DNS maximum length by increasing DNS maximum allowed packet lenght on the global policy

=====================================================================

===============
CAP-RST-SYN
===============

8418: 13:00:29.182745 802.1Q vlan#200 P0 @PROXY1.40100 > @UPSTREAM-PROXY-SERVER-CONNECTOR.8080: SWE 3026483265:3026483265(0) win 8192 <mss 1380,nop,wscale 8,nop,nop,sackOK>
8419: 13:00:29.242709 802.1Q vlan#200 P0 @PROXY4.19775 > @UPSTREAM-PROXY-SERVER-CONNECTOR.8080: SWE 2933989033:2933989033(0) win 8192 <mss 1380,nop,wscale 8,nop,nop,sackOK>
8420: 13:00:29.256822 802.1Q vlan#200 P0 @PROXY4.19781 > @UPSTREAM-PROXY-SERVER-CONNECTOR.8080: SWE 493632163:493632163(0) win 8192 <mss 1380,nop,wscale 8,nop,nop,sackOK>
8421: 13:00:29.304763 802.1Q vlan#200 P0 @PROXY2.27091 > @UPSTREAM-PROXY-SERVER-CONNECTOR.8080: SWE 2077561914:2077561914(0) win 8192 <mss 1380,nop,wscale 8,nop,nop,sackOK>
8422: 13:00:29.401621 802.1Q vlan#200 P0 @PROXY1.40162 > @UPSTREAM-PROXY-SERVER-CONNECTOR.8080: SWE 1893318:1893318(0) win 8192 <mss 1380,nop,wscale 8,nop,nop,sackOK>
8423: 13:00:29.431038 802.1Q vlan#200 P0 @PROXY1.40173 > @UPSTREAM-PROXY-SERVER-CONNECTOR.8080: SWE 2389956696:2389956696(0) win 8192 <mss 1380,nop,wscale 8,nop,nop,sackOK>
8424: 13:00:29.463827 802.1Q vlan#200 P0 @PROXY4.19828 > @UPSTREAM-PROXY-SERVER-CONNECTOR.8080: SWE 479335951:479335951(0) win 8192 <mss 1380,nop,wscale 8,nop,nop,sackOK>
8425: 13:00:29.480367 802.1Q vlan#200 P0 @PROXY4.19831 > @UPSTREAM-PROXY-SERVER-CONNECTOR.8080: SWE 646829171:646829171(0) win 8192 <mss 1380,nop,wscale 8,nop,nop,sackOK>
8426: 13:00:29.494710 802.1Q vlan#200 P0 @PROXY1.40192 > @UPSTREAM-PROXY-SERVER-CONNECTOR.8080: SWE 706034794:706034794(0) win 8192 <mss 1380,nop,wscale 8,nop,nop,sackOK>
8427: 13:00:29.496892 802.1Q vlan#200 P0 @PROXY1.40194 > @UPSTREAM-PROXY-SERVER-CONNECTOR.8080: SWE 3571270395:3571270395(0) win 8192 <mss 1380,nop,wscale 8,nop,nop,sackOK>
8427 packets shown
firewall#
=====================================================================

===============
CAP-RST-FIN
===============

2572: 13:05:12.888717 802.1Q vlan#200 P0 @PROXY-HA.8080 > @PROXY-CLIENT.50713: R 3028373259:3028373259(0) win 0 Drop-reason: (tcp-rstfin-ooo) TCP RST/FIN out of order
2573: 13:05:12.889373 @PROXY-CLIENT.52951 > @PROXY-HA.8080: R 891534262:891534262(0) win 0 Drop-reason: (tcp-rstfin-ooo) TCP RST/FIN out of order
2574: 13:05:12.959697 802.1Q vlan#200 P0 @PROXY-HA.8080 > @PROXY-CLIENT.49165: R 4143752165:4143752165(0) win 0 Drop-reason: (tcp-rstfin-ooo) TCP RST/FIN out of order
2575: 13:05:13.444206 802.1Q vlan#200 P0 @PROXY-HA.8080 > @PROXY-CLIENT.53390: R 1844042984:1844042984(0) win 0 Drop-reason: (tcp-rstfin-ooo) TCP RST/FIN out of order
2576: 13:05:13.569337 802.1Q vlan#800 P0 @PUBLIC-IP-FOR-GUEST-NETWORK.39692 > @UPSTREAM-PROXY-SERVER-CONNECTOR.8080: F 1248479373:1248479373(0) ack 3675873403 win 1369 Drop-reason: (tcp-rstfin-ooo) TCP RST/FIN out of order
2577: 13:05:13.581375 802.1Q vlan#800 P0 @PUBLIC-IP-FOR-GUEST-NETWORK.34243 > @UPSTREAM-PROXY-SERVER-CONNECTOR.8080: F 272944704:272944704(0) ack 878609742 win 1369 Drop-reason: (tcp-rstfin-ooo) TCP RST/FIN out of order
2578: 13:05:14.257112 802.1Q vlan#800 P0 @guest-NETWORK-CLIENT.46753 > 23.57.89.142.443: R 2360057826:2360057826(0) win 0 Drop-reason: (tcp-rstfin-ooo) TCP RST/FIN out of order
2579: 13:05:14.621504 802.1Q vlan#800 P0 @PUBLIC-IP-FOR-GUEST-NETWORK.34243 > @UPSTREAM-PROXY-SERVER-CONNECTOR.8080: FP 272944672:272944704(32) ack 878609742 win 1369 Drop-reason: (tcp-rstfin-ooo) TCP RST/FIN out of order
2580: 13:05:14.753821 802.1Q vlan#200 P0 @PROXY-HA.8080 > @PROXY-CLIENT.49856: R 2014518163:2014518163(0) win 0 Drop-reason: (tcp-rstfin-ooo) TCP RST/FIN out of order
2581: 13:05:16.683222 @UPSTREAM-PROXY-SERVER-CONNECTOR.8080 > @PUBLIC-IP-FOR-GUEST-NETWORK.57977: R 2455327392:2455327392(0) ack 1952478637 win 131
2582: 13:05:16.852739 802.1Q vlan#200 P0 @PROXY-HA.8080 > @PROXY-CLIENT.50884: R 59380247:59380247(0) win 0 Drop-reason: (tcp-rstfin-ooo) TCP RST/FIN out of order
2583: 13:05:17.617842 802.1Q vlan#200 P0 @PROXY-HA.8080 > @PROXY-CLIENT.54416: R 1401204225:1401204225(0) win 0 Drop-reason: (tcp-rstfin-ooo) TCP RST/FIN out of order
2584: 13:05:17.817477 802.1Q vlan#200 P0 @PROXY-HA.8080 > @PROXY-CLIENT.62737: R 1993565555:1993565555(0) win 0 Drop-reason: (tcp-rstfin-ooo) TCP RST/FIN out of order
2585: 13:05:17.894195 802.1Q vlan#800 P0 @PUBLIC-IP-FOR-GUEST-NETWORK.42977 > @UPSTREAM-PROXY-SERVER-CONNECTOR.8080: F 2689946459:2689946459(0) ack 1628644842 win 685 Drop-reason: (tcp-rstfin-ooo) TCP RST/FIN out of order
2585 packets shown
=====================================================================

Regards,

[Bogdan Nita]

Based on the output it seems something is wrong with the proxy setup.

You could disable tcp inspection on the asa, but the problem is not on the asa.

TCP RST/SYN in window:
SYN flag should not be used in data transfer. Only the first packet sent from each end should have this flag set.

TCP RST/FIN out of order:
TCP RST or TCP FIN packet received after the tcp session was closed. It could be caused by a host trying to use a closed tcp session, the other host would normally respond to that with a RST.

[Neji Jihed]

I fact it's a windows server 2012, there are no proxy rules on the proxy, all it's doing is forwarding the traffic to the upstream server collector.
How can i check if something is wrong on the proxy server ?

[Bogdan Nita]

Windows is unfortunately not my area of expertise.

You could maybe have a look at the system logs.

[Mohammed al Baqari]

I think the problem is high tcp conn. You conn rate is 2300 per sec. Is
this normal. Is it a heavy environment.

Check show host command with sort option to see who is causing this

[Neji Jihed]

This happened a while ago and stopped when i thought i upgraded the firmware of the firewall, and then it appeared again, i can't find the command show host !
do you mean show local-host ? if yes how can i sort the output ?

Thank you,

[Neji Jihed]

I have tried system logs, but they are not showing anything unusual !

Regards,

[Neji Jihed]

I have sorted out the output of show local-host using Excel :

local host: <@PROXY-HA>,
TCP flow count/limit 1412 unlimited
TCP embryonic count to host 69
TCP intercept watermark unlimited
UDP flow count/limit 0 unlimited
local host: <@PROXY3>,
TCP flow count/limit 1741 unlimited
TCP embryonic count to host 1
TCP intercept watermark unlimited
UDP flow count/limit 0 unlimited
local host: <@PROXY2>,
TCP flow count/limit 2147 unlimited
TCP embryonic count to host 1
TCP intercept watermark unlimited
UDP flow count/limit 0 unlimited
local host: <@PROXY1>,
TCP flow count/limit 3102 unlimited
TCP embryonic count to host 12
TCP intercept watermark unlimited
UDP flow count/limit 0 unlimited
local host: <@PROXY4>,
TCP flow count/limit 2350 unlimited
TCP embryonic count to host 2
TCP intercept watermark unlimited
UDP flow count/limit 1 unlimited
local host: <@PROXY-COLLECTOR-PUBLIC-IP>,
TCP flow count/limit 7935 unlimited
TCP embryonic count to host 906
TCP intercept watermark unlimited
UDP flow count/limit 0 unlimited

Wanted to mention that we are using scansafe as webfilter.
Thank you,