ASA/FTD Firewall dropping RST from Client after Server's Challenge-ACK

[Rajat Sharma]

Understanding TCP Challenge-ACK: Enhancing TCP Security, Overview

The TCP Challenge-ACK mechanism is an important security feature in the Transmission Control Protocol (TCP) that helps mitigate certain types of network attacks, particularly those involving blind spoofing. Introduced as part of [RFC 5961](https://datatracker.ietf.org/doc/html/rfc5961\), the Challenge-ACK mechanism ensures the integrity of a TCP connection by validating the legitimacy of incoming packets that appear suspicious or unexpected. This article explains what TCP Challenge-ACK is, why it is necessary, how it works, and its significance in maintaining secure and reliable TCP connections.

What is TCP Challenge-ACK?

A Challenge-ACK is a special acknowledgment (ACK) packet sent by a TCP endpoint to verify whether the sender of an incoming TCP packet is legitimate and has knowledge of the connection's current state. It is essentially a way to "challenge" the sender to confirm their authenticity.

This mechanism is primarily used in situations where the TCP stack detects unusual or suspicious packets, such as:

1. Invalid or unexpected sequence numbers.
2. Spoofed TCP RST (Reset) packets attempting to terminate a connection.
3. Spurious ACK packets that could indicate an attack.

By introducing an extra validation step before taking any critical action (e.g., closing a connection), the Challenge-ACK mechanism strengthens the security of TCP.

Why is TCP Challenge-ACK Necessary?

TCP is a reliable, connection-oriented protocol, but it was not originally designed with strong security mechanisms. Over time, several vulnerabilities have been identified, including blind spoofing attacks, where an attacker sends forged packets without access to the actual connection state. These attacks exploit TCP's reliance on predictable sequence numbers and acknowledgment numbers.

Common Attacks Addressed by Challenge-ACK

1. RST Injection (Reset Attack):

   • Attackers send spoofed TCP RST packets with guessed sequence numbers to terminate an active connection.
   • If the sequence number is within the valid range (window), the connection will be reset.

2. Data Injection:

   • Attackers forge TCP packets with guessed sequence numbers to inject malicious data into an active connection.

3. ACK Flooding (Exploitation of Middleboxes):

   • Attackers flood a target with fake ACK packets to exhaust resources or trigger unintended behavior in firewalls and routers.

The Need for Validation

Without a validation mechanism, TCP is vulnerable to blindly accepting packets that meet certain criteria (e.g., sequence numbers within the window). The Challenge-ACK mechanism introduces a challenge-response process to ensure that the sender has knowledge of the connection's current state, mitigating the risk of spoofed packets.

How Does TCP Challenge-ACK Work?

The Challenge-ACK mechanism is triggered when the TCP stack detects an unexpected packet that does not align with the current state of the connection. Here’s a step-by-step explanation:

1. Receive a Suspicious Packet:

   • A packet arrives with an invalid sequence number, acknowledgment number, or other anomalies that make it suspicious.
   • Examples:
     • A TCP RST packet with a sequence number outside the acceptable range.
     • An ACK packet that doesn’t correspond to the expected acknowledgment number.

2. Generate a Challenge-ACK:

   • Instead of immediately taking action (e.g., resetting the connection), the TCP stack sends a Challenge-ACK packet to the sender. This ACK contains:
     • The current acknowledgment number (ACK) of the connection.
     • The current sequence number of the sender.

3. Wait for a Valid Response:

   • The sender is expected to respond with a valid packet that matches the connection state (e.g., the correct sequence and acknowledgment numbers).
   • If the sender is legitimate, they will have the correct information and can respond appropriately.

4. Validate the Response:

   • If the response from the sender is valid, the connection continues normally.
   • If no valid response is received (or if the sender fails to respond), the suspicious packet is ignored, and no further action is taken.

TCP Challenge-ACK in Action

Example 1: Mitigating a TCP RST Attack

• Scenario:
  • An attacker attempts to terminate an active TCP connection by sending a forged RST packet with a guessed sequence number.
• Challenge-ACK Response:
  • The recipient of the RST sends a Challenge-ACK to verify the legitimacy of the RST packet.
  • If the attacker cannot respond correctly (because they lack knowledge of the connection’s sequence numbers), the RST packet is ignored.

Example 2: Spurious ACK Detection

• Scenario:
  • A middlebox or attacker sends spurious ACK packets, potentially interfering with the connection or consuming resources.
• Challenge-ACK Response:
  • The TCP endpoint sends a Challenge-ACK to confirm the sender's legitimacy. If the ACK is invalid, it is ignored.

Challenge-ACK Rate Limiting

To prevent abuse of the Challenge-ACK mechanism itself, modern TCP implementations implement rate limiting for Challenge-ACK packets. Without rate limiting, an attacker could send a large number of forged packets to trigger excessive Challenge-ACKs, potentially causing a denial-of-service (DoS) attack.

Linux Implementation

In Linux systems, the rate limit for Challenge-ACKs is controlled by the tcp_challenge_ack_limit parameter:

sysctl net.ipv4.tcp_challenge_ack_limit

The default value is typically set to 100, meaning that a maximum of 100 Challenge-ACKs can be sent per second.

Why Rate Limiting Is Important

• It ensures that the Challenge-ACK mechanism does not become a vector for DoS attacks.
• It balances security with resource efficiency.

Benefits of TCP Challenge-ACK

1. Increased Resilience:

   • Reduces the impact of blind spoofing attacks by validating the legitimacy of packets.

2. Backward Compatibility:

   • Challenge-ACK is an extension to the TCP protocol and does not require changes to existing applications or higher-layer protocols.

3. Low Overhead:

   • Minimal computational cost as it leverages existing TCP mechanisms (e.g., ACK packets).

4. Improved Connection Integrity:

   • Ensures that critical actions, such as resetting a connection, occur only if the request is legitimate.

Limitations of Challenge-ACK

1. Not a Complete Security Solution:

   • Challenge-ACK mitigates specific types of attacks (e.g., RST injection) but does not address all TCP vulnerabilities (e.g., SYN flooding, application-layer attacks).

2. Rate Limiting Trade-Off:

   • While rate limiting prevents DoS attacks, it may also delay Challenge-ACK responses during high traffic volumes.

3. Limited to TCP:

   • Challenge-ACK is specific to TCP and does not apply to other transport protocols.

Symptoms of issue seen when traffic Challenge ACK traffic passes through ASA/FTD

Firewall dropping RST from Client after Server's "Challenge ACK" preventing client from establishing TCP connections to server.

Environment

1. Any client-server architecture where the Server is configured to mitigate "Blind Reset Attack Using the SYN Bit" and sends "Challenge-ACK"

2. As a response to client's SYN, the Server challenges by sending an ACK to confirm the loss of the previous connection and the request to start a new connection.

3. This challenge ACK has acknowledgment number from previous connection and upon seeing the unexpected ACK, client sends a RST; thus tearing down TCP connection on the server also.

4. RFC5961: https://tools.ietf.org/html/rfc5961#section-4

5. Sample packet capture explaining such a flow:

> First ACK from the server has acknowledgement number different than SYN's sequence number.
> Client responding with RST with sequence number derived from challenge ACK.
> Upon receiving the RST, Server tears down old TCP connection and relies on the SYN retransmission from the client end to re-establish the connection.

Cause

1. When ASA/FTD firewall is placed between such client and server, it doesn't understand such a flow by default.

2. Firewall forwards the ACK received from server to the client however the RST from client is dropped by firewall.

15:36:43.524234 10.117.142.19.42098 > 192.168.198.28.443: R 2338118548:2338118548(0) win 0 Drop-reason: (tcp-rstfin-ooo) TCP RST/FIN out of order, Drop-location: frame 0x000055870affc2df flow (NA)/NA

LAB Replay Test Results

When packets were replayed in lab following was seen:

Client IP address 10.117.142.19
Sever IP address 192.168.198.28

Ingress Interface --> Inside
Egress Interface --> DMZ

When packets were replayed in lab following was seen:

ciscoasa(config-pmap-c)# sh conn address 192.168.198.28 long

27 in use, 17252 most used

Flags: A - awaiting inside ACK to SYN, a - awaiting outside ACK to SYN,

B - initial SYN from outside, b - TCP state-bypass or nailed,

C - CTIQBE media, c - cluster centralized,

D - DNS / Umbrella, d - dump, E - outside back connection, e - semi-distributed,

F - outside FIN, f - inside FIN,

G - group, g - MGCP, H - H.323, h - H.225.0, I - inbound data,

i - incomplete, J - GTP, j - GTP data, K - GTP t3-response

k - Skinny media, L - LISP triggered flow owner mobility

l - local director/backup stub flow

M - SMTP data, m - SIP media, n - GUP

N - inspected by Snort

O - outbound data, o - offloaded,

P - inside back connection

Q - Diameter, q - SQL*Net data,

R - outside acknowledged FIN,

R - UDP SUNRPC, r - inside acknowledged FIN, S - awaiting inside SYN,

s - awaiting outside SYN, T - SIP, t - SIP transient, U - up, u - STUN,

V - VPN orphan, v - M3UA W - WAAS,

w - secondary domain backup,

X - inspected by service module,

x - per session, Y - director stub flow, y - backup stub flow,

Z - Scansafe redirection, z - forwarding stub flow

TCP inside: 10.117.142.19/42098 (10.117.142.19/42098) dmz: 192.168.198.28/443 (192.168.198.28/443), flags SaAB , idle 0s, uptime 7s, timeout 30s, bytes 0

Initiator: 10.117.142.19, Responder: 192.168.198.28

ciscoasa(config-pmap-c)# sh logging | i 10.117.142.19

May 18 2025 08:14:55: %ASA-7-815004: OGS: Packet TCP from inside:10.117.142.19/42098 to dmz:192.168.198.28/443 matched (1) source network objects and (0) destination network objects total search entries (0). Resultant key-set has (1) entries

May 18 2025 08:14:55: %ASA-6-302013: Built inbound TCP connection 207860548 for inside:10.117.142.19/42098 (10.117.142.19/42098) to dmz:192.168.198.28/443 (192.168.198.28/443)

May 18 2025 08:15:03: %ASA-6-302014: Teardown TCP connection 207860548 for inside:10.117.142.19/42098 to dmz:192.168.198.28/443 duration 0:00:08 bytes 0 TCP Reset-O from inside

May 18 2025 08:15:04: %ASA-7-815004: OGS: Packet TCP from inside:10.117.142.19/42098 to dmz:192.168.198.28/443 matched (1) source network objects and (0) destination network objects total search entries (0). Resultant key-set has (1) entries

May 18 2025 08:15:04: %ASA-6-302013: Built inbound TCP connection 207860570 for inside:10.117.142.19/42098 (10.117.142.19/42098) to dmz:192.168.198.28/443 (192.168.198.28/443)

May 18 2025 08:15:06: %ASA-6-302014: Teardown TCP connection 207860570 for inside:10.117.142.19/42098 to dmz:192.168.198.28/443 duration 0:00:02 bytes 0 TCP Reset-O from inside

ciscoasa(config-pmap-c)# sh capture capi

12 packets captured

1: 08:14:55.463812 10.117.142.19.42098 > 192.168.198.28.443: S 2514567835:2514567835(0) win 29200 <mss 1460,nop,nop,timestamp 3896645 0,nop,wscale 7>

2: 08:14:56.464178 192.168.198.28.443 > 10.117.142.19.42098: . ack 969796095 win 366 <nop,nop,timestamp 3507892714 2317016501>

3: 08:14:57.464087 10.117.142.19.42098 > 192.168.198.28.443: R 2338118548:2338118548(0) win 0

4: 08:14:58.464041 10.117.142.19.42098 > 192.168.198.28.443: S 2514567835:2514567835(0) win 29200 <mss 1460,nop,nop,timestamp 3897648 0,nop,wscale 7>

5: 08:14:59.463950 10.117.142.19.42098 > 192.168.198.28.443: S 2514567835:2514567835(0) win 29200 <mss 1460,nop,nop,timestamp 3899652 0,nop,wscale 7>

6: 08:15:00.463965 10.117.142.19.42098 > 192.168.198.28.443: S 2514567835:2514567835(0) win 29200 <mss 1460,nop,nop,timestamp 3903664 0,nop,wscale 7>

7: 08:15:01.464255 10.117.142.19.42098 > 192.168.198.28.443: S 2514567835:2514567835(0) win 29200 <mss 1460,nop,nop,timestamp 3911680 0,nop,wscale 7>

8: 08:15:02.464224 192.168.198.28.443 > 10.117.142.19.42098: . ack 1950703923 win 366 <nop,nop,timestamp 3507907749 2317016501>

9: 08:15:03.464117 10.117.142.19.42098 > 192.168.198.28.443: R 3319026376:3319026376(0) win 0

10: 08:15:04.463873 10.117.142.19.42098 > 192.168.198.28.443: S 2514567835:2514567835(0) win 29200 <mss 1460,nop,nop,timestamp 3927712 0,nop,wscale 7>

11: 08:15:05.464117 192.168.198.28.443 > 10.117.142.19.42098: . ack 2373587500 win 366 <nop,nop,timestamp 3507923781 2317016501>

12: 08:15:06.464331 10.117.142.19.42098 > 192.168.198.28.443: R 3380905555:3380905555(0) win 0

12 packets shown

ciscoasa(config-pmap-c)# sh capture capd

11 packets captured

1: 08:14:55.463934 10.117.142.19.42098 > 192.168.198.28.443: S 3882890288:3882890288(0) win 29200 <mss 1380,nop,nop,timestamp 3896645 0,nop,wscale 7>

2: 08:14:56.464148 192.168.198.28.443 > 10.117.142.19.42098: . ack 2338118548 win 366 <nop,nop,timestamp 3507892714 2317016501>

3: 08:14:58.464072 10.117.142.19.42098 > 192.168.198.28.443: S 3882890288:3882890288(0) win 29200 <mss 1380,nop,nop,timestamp 3897648 0,nop,wscale 7>

4: 08:14:59.463965 10.117.142.19.42098 > 192.168.198.28.443: S 3882890288:3882890288(0) win 29200 <mss 1380,nop,nop,timestamp 3899652 0,nop,wscale 7>

5: 08:15:00.464011 10.117.142.19.42098 > 192.168.198.28.443: S 3882890288:3882890288(0) win 29200 <mss 1380,nop,nop,timestamp 3903664 0,nop,wscale 7>

6: 08:15:01.464300 10.117.142.19.42098 > 192.168.198.28.443: S 3882890288:3882890288(0) win 29200 <mss 1380,nop,nop,timestamp 3911680 0,nop,wscale 7>

7: 08:15:02.464194 192.168.198.28.443 > 10.117.142.19.42098: . ack 3319026376 win 366 <nop,nop,timestamp 3507907749 2317016501>

8: 08:15:03.464148 10.117.142.19.42098 > 192.168.198.28.443: R 392381533:392381533(0) win 0

9: 08:15:04.464102 10.117.142.19.42098 > 192.168.198.28.443: S 3521885890:3521885890(0) win 29200 <mss 1380,nop,nop,timestamp 3927712 0,nop,wscale 7>

10: 08:15:05.464087 192.168.198.28.443 > 10.117.142.19.42098: . ack 3380905555 win 366 <nop,nop,timestamp 3507923781 2317016501>

11: 08:15:06.464361 10.117.142.19.42098 > 192.168.198.28.443: R 93256314:93256314(0) win 0

11 packets shown

ciscoasa(config-pmap-c)# sh capture asp

Target: OTHER

Hardware: ASAv

Cisco Adaptive Security Appliance Software Version 9.16(4)170

ASLR enabled, text region 5587084fb000-55870c1547e5

1 packet captured

1: 08:14:57.464102 10.117.142.19.42098 > 192.168.198.28.443: R 2338118548:2338118548(0) win 0 Drop-reason: (tcp-rstfin-ooo) TCP RST/FIN out of order, Drop-location: frame 0x000055870affc2df flow (NA)/NA

1 packet shown

ciscoasa(config-pmap-c)#

Notice that there are two connections which are created, first one is below

May 18 2025 08:14:55: %ASA-6-302013: Built inbound TCP connection 207860548 for inside:10.117.142.19/42098 (10.117.142.19/42098) to dmz:192.168.198.28/443 (192.168.198.28/443)

This connection is created by first packet ( notice the timestamp )
1: 08:14:55.463812 10.117.142.19.42098 > 192.168.198.28.443: S 2514567835:2514567835(0) win 29200 <mss 1460,nop,nop,timestamp 3896645 0,nop,wscale 7>

Reset packet sent by 10.117.142.19 in response to challenge ack is dropped by firewall:

3: 08:14:57.464087 10.117.142.19.42098 > 192.168.198.28.443: R 2338118548:2338118548(0) win 0

The reason this packet is dropped as the sequence number of packet is 2338118548, which is less than the initial sequence number of connection 2514567835, so the packet is drooped with code TCP RST/FIN out of order. This drop can cause delays as challenge/ack exchange is interrupted.

Connection is torn down at the following time:

May 18 2025 08:15:03: %ASA-6-302014: Teardown TCP connection 207860548 for inside:10.117.142.19/42098 to dmz:192.168.198.28/443 duration 0:00:08 bytes 0 TCP Reset-O from inside

teardown is due to following packet:

9: 08:15:03.464117 10.117.142.19.42098 > 192.168.198.28.443: R 3319026376:3319026376(0) win 0

Notice that this packet is also response to challenge ack packet

8: 08:15:02.464224 192.168.198.28.443 > 10.117.142.19.42098: . ack 1950703923 win 366 <nop,nop,timestamp 3507907749 2317016501>

Why this reset is not drooped is the Sequence number 3319026376 is valid for this connection, it is greater than initial sequence number 2514567835 of the connection.

The second connection is created:

May 18 2025 08:15:04: %ASA-6-302013: Built inbound TCP connection 207860570 for inside:10.117.142.19/42098 (10.117.142.19/42098) to dmz:192.168.198.28/443 (192.168.198.28/443)

Packet that creates the connection

10: 08:15:04.463873 10.117.142.19.42098 > 192.168.198.28.443: S 2514567835:2514567835(0) win 29200 <mss 1460,nop,nop,timestamp 3927712 0,nop,wscale 7>

Another challenge ack is sent by server

11: 08:15:05.464117 192.168.198.28.443 > 10.117.142.19.42098: . ack 2373587500 win 366 <nop,nop,timestamp 3507923781 2317016501>

reset is sent by client

12: 08:15:06.464331 10.117.142.19.42098 > 192.168.198.28.443: R 3380905555:3380905555(0) win 0

This reset the firewall connection

May 18 2025 08:15:06: %ASA-6-302014: Teardown TCP connection 207860570 for inside:10.117.142.19/42098 to dmz:192.168.198.28/443 duration 0:00:02 bytes 0 TCP Reset-O from inside

Again, the reason that reset packet is not dropped is as this is valid, Sequence number: 3380905555 of reset is greater than initial sequence number 2514567835 of connection.

Observations

1. While dropping the out of window RST is actually an intended behavior, it breaks the Challenge-ACK mechanism.

While dropping the out of window RST is actually an intended behavior, it breaks the Challenge-ACK mechanism. 

-Dropping of out of window RST will depend on the sequence number chosen by the server in Challenge-ACK packet. It may so happen that RST packet sequence number is more than initial sequence number sent by the client, then RST will not be dropped. This is expected from Firewall handling of TCP packets as normally sequence number should not be lower than initial sequence number for the connection. So in some instances Challenge-ACK mechanism will not be interrupted.

Following exchange as an example to illustrate:

Captures on input interface

ciscoasa(config)# sh capture capi3

4 packets captured

   1: 11:05:52.412377       192.168.1.38.10001 > 4.2.2.2.7890: S 2514567835:2514567835(0) win 512

   2: 11:05:57.136147       192.168.1.38.10001 > 4.2.2.2.7890: R 2338118548:2338118548(0) win 512>>> This is dropped

2514567835>2338118548
True

   3: 11:06:46.764166       192.168.1.38.10000 > 4.2.2.2.7890: S 3788896101:3788896101(0) win 512

   4: 11:06:52.676020       192.168.1.38.10000 > 4.2.2.2.7890: R 3962930203:3962930203(0) win 512>>> This is not dropped

3788896101>3962930203
False

4 packets shown

captures on output interface

ciscoasa(config)# sh capture capo3

3 packets captured

   1: 11:05:52.412515       10.48.26.235.10001 > 4.2.2.2.7890: S 4017062026:4017062026(0) win 512

   2: 11:06:46.764288       10.48.26.235.10000 > 4.2.2.2.7890: S 11135791:11135791(0) win 512

   3: 11:06:52.676066       10.48.26.235.10000 > 4.2.2.2.7890: R 185169893:185169893(0) win 512>>Not dropped packet

3 packets shown

asp drop capture

ciscoasa(config)# sh capture asp 

Target:     OTHER

Hardware:   ASAv

Cisco Adaptive Security Appliance Software Version 9.16(4)170

ASLR enabled, text region 563976101000-563979d5a7e5

1 packet captured

   1: 11:05:57.136177       192.168.1.38.10001 > 4.2.2.2.7890: R 2338118548:2338118548(0) win 512 Drop-reason: (tcp-rstfin-ooo) TCP RST/FIN out of order, Drop-location: frame 0x0000563978c022df flow (NA)/NA>>Dropped packet

[helen628young]

Hello,

Thanks for give knowledgeable information ! PrepaidGiftBalance

[Rajat Sharma]

thanks @helen628young