topic Re: Unexpected TCP timestamp option cleared in server's response in Network Security

Unexpected TCP timestamp option cleared in server's response

bwallander — Mon, 11 Mar 2019 19:07:26 GMT

Hello,

I have a question about the functionality of the ASA firewall in regards to TCP option handling which I've yet to find any relavant documentation or known bugs for.

Consider the following scenario:

ASA 5520 - 8.0(4)

An HTTP client (outside) is making a simple port 80/tcp connection to an HTTP server hosted behind the ASA (inside). The client is specifically configured NOT to include the TCP timestamp value in it's initial SYN request to the server. The server however has a unique requirement where the timestamp option MUST be populated and included in it's SYN-ACK TCP header response to all clients, regardless of whether or not the client intially sent one. An iptables rule on the linux server is enforcing this requirement.

My question is, in the above scenario, will the ASA allow the TCP timestamp option (8) to pass back to the client in the SYN-ACK response, containing it's TCP options? Or will the ASA clear the option since the client did not send a timpestamp in it's initial SYN?

Based on our tests the latter appears to be occuring. The ASA looks to be silently clearing this option in the header of the response from the server, possibly because of the RFC violation or some other similar reason. By default, the TCP normalization engine on the ASA should allow the timestamp option to pass ("tcp-options timestamp allow").

An actual tcpdump from the client's perspective:

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
00:00:26.137565 IP client-machine.local.42788 > www-server.www: Flags [S], seq 3692322182, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
00:00:26.392831 IP www-server.www > client-machine.local.42788: Flags [S.], seq 2550944951, ack 3692322183, win 5792, options [mss 1380,sackOK,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,wscale 7], length 0
00:00:26.392844 IP client-machine.local.42788 > www-server.www: Flags [.], ack 1, win 46, length 0
00:00:26.392902 IP client-machine.local.42788 > www-server.www: Flags [F.], seq 1, ack 1, win 46, length 0
00:00:26.646815 IP www-server.www > client-machine.local.42788: Flags [F.], seq 1, ack 2, win 46, options [nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop], length 0
00:00:26.646826 IP client-machine.local.42788 > www-server.www: Flags [.], ack 2, win 46, length 0

Re: Unexpected TCP timestamp option cleared in server's response

Panos Kampanakis — Wed, 10 Nov 2010 16:45:32 GMT

The ASA will by default clear the timestamps.

If you want to allow them you need to use a tcp-map in the MPF. http://www.cisco.com/en/US/docs/security/asa/asa82/configuration/guide/conns_tcpnorm.html explains it and an example is here http://www.cisco.com/en/US/docs/security/asa/asa82/configuration/guide/conns_tcpnorm.html#wp1088337

hostname(config)# tcp-map tmap

hostname(config-tcp-map)# tcp-options timestamp allow

hostname(config-tcp-map)# class-map urg-class

hostname(config-cmap)# match port tcp range ftp-data telnet

hostname(config-cmap)# policy-map pmap

hostname(config-pmap)# class urg-class

hostname(config-pmap-c)# set connection advanced-options tmap

hostname(config-pmap-c)# service-policy pmap global

I hope it helps.

Re: Unexpected TCP timestamp option cleared in server's response

bwallander — Wed, 10 Nov 2010 16:50:42 GMT

For comparrison, here's another client-side perspective tcpdump when the timestamp value is included in the client's initial connection:

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
11:18:55.927671 IP client-machine.local.46822 > www-server.www: S 4090293543:4090293543(0) win 5840
11:18:55.928360 IP www-server.www > client-machine.local.46822: S 1018691747:1018691747(0) ack 4090293544 win 5792
11:18:55.928410 IP client-machine.local.46822 > www-server.www: . ack 1 win 46
11:18:57.361352 IP client-machine.local.46822 > www-server.www: F 1:1(0) ack 1 win 46
11:18:57.362314 IP www-server.www > client-machine.local.46822: F 1:1(0) ack 2 win 1448
11:18:57.362346 IP client-machine.local.46822 > www-server.www: . ack 2 win 46

Re: Unexpected TCP timestamp option cleared in server's response

bwallander — Wed, 10 Nov 2010 16:54:30 GMT

Thanks for the response. Are you sure that the ASA by default will clear the timestamp value? Why is the default tcp-map value for 'tcp-options timestamp' set to 'allow'? Also how does this explain TCP timestamp values NOT being cleared when the client sends a timestamp value in it's inital request(as seen in my example)?

-Buck

Re: Unexpected TCP timestamp option cleared in server's response

Panos Kampanakis — Wed, 10 Nov 2010 17:10:16 GMT

I mis-spoke in the statement for the default behavior. I double checked and unless you set it in the tcp-map, the ASA should allow TCP timestamps.

So, if you don't have it configured, the ASA should not change the value.

You can capture the packets on the ASA inside and outside to verify if the ASA is really clearing the response TCP timestamps.

Re: Unexpected TCP timestamp option cleared in server's response

bwallander — Wed, 10 Nov 2010 18:33:30 GMT

Thanks for clarifying. The ASA is not configured to modify the timestamp value.

We've also tried an MPF policy to allow any other possible TCP options (6,7, 9-255) in case something else was being set, however this didn't help.

After looking at our captures from the client, ASA, and server we've determined that the ASA seems to be 'silently' clearing this option in the server's response. I say silently because I've not found a way to actually see the ASA reporting this occuring (through syslog, statistical counter, etc).

Would there be a specific command, such as with the 'show asp table' command, to see if the ASA might be clearing these (or any) options from TCP headers?

-Buck

Re: Unexpected TCP timestamp option cleared in server's response

Panos Kampanakis — Wed, 10 Nov 2010 18:45:41 GMT

It is too deep in the processing of the header to catch it. But it shouldn't be doing it though.

Are you sure it is the ASA? Did you capture the packets in and out of the ASA?

What version are your running?

Re: Unexpected TCP timestamp option cleared in server's response

bwallander — Wed, 10 Nov 2010 21:41:28 GMT

There's been several tests conducted, including one where two hosts are next to each other on the same network segment and the options show up just fine. It's only the single network hop that the ASA introduces that changes between a successful test and a failed one.

Here's a test of a telnet to port 80 from a client machine in the "inside" segment of the firewall connecting to the webserver (The web server actually resides in a DMZ segment). The tcpdump is taken from the client machine in a separate console at the time of performing the telnet connection:

00:35:24.120280 IP (tos 0x10, ttl 64, id 44021, offset 0, flags [DF], proto: TCP (6), length: 52) inside-host.49899 > dmz-www-server.http: S, cksum 0xcaec (correct), 1075522845:1075522845(0) win 5840

00:35:24.120896 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto: TCP (6), length: 60) dmz-www-server.http > inside-host.49899: S, cksum 0x39b3 (correct), 3973415110:3973415110(0) ack 1075522846 win 5792 nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,wscale 7>

00:35:24.120928 IP (tos 0x10, ttl 64, id 44022, offset 0, flags [DF], proto: TCP (6), length: 40) inside-host.49899 > dmz-www-server.http: ., cksum 0xb4b3 (correct), 1:1(0) ack 1 win 46

00:35:24.121341 IP (tos 0x10, ttl 64, id 44023, offset 0, flags [DF], proto: TCP (6), length: 40) inside-host.49899 > dmz-www-server.http: F, cksum 0xb4b2 (correct), 1:1(0) ack 1 win 46

00:35:24.121846 IP (tos 0x0, ttl 64, id 50779, offset 0, flags [DF], proto: TCP (6), length: 52) dmz-www-server.http > inside-host.49899: F, cksum 0x7e9f (correct), 1:1(0) ack 2 win 46 <nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop>

00:35:24.121869 IP (tos 0x10, ttl 64, id 44024, offset 0, flags [DF], proto: TCP (6), length: 40) inside-host.49899 > dmz-www-server.http: ., cksum 0xb4b1 (correct), 2:2(0) ack 2 win 46

Notice the "nop,nop,nop,nop," where something (ASA?) is clearing the TCP timestamp option. Also note that this looks like the same method of clearing an option that the ASA uses to clear other TCP options, such as TCP option 19 for MD5 authentication.

Here's the same test but from a neighboring device in the same DMZ segment as the same server (No firewall traversal):

00:32:38.761460 IP (tos 0x10, ttl 64, id 46277, offset 0, flags [DF], proto: TCP (6), length: 52) dmz-host.50167 > dmz-www-server.http: S, cksum 0x0861 (correct), 1518129205:1518129205(0) win 5840

00:32:38.761665 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto: TCP (6), length: 60) dmz-www-server.http > dmz-host.50167: S, cksum 0x0eef (correct), 2756096771:2756096771(0) ack 1518129206 win 5792 timestamp 21264880 1,nop,wscale 7>

00:32:38.761697 IP (tos 0x10, ttl 64, id 46278, offset 0, flags [DF], proto: TCP (6), length: 40) dmz-host.50167 > dmz-www-server.http: ., cksum 0x087a (correct), 1:1(0) ack 1 win 4600:32:38.761947 IP (tos 0x10, ttl 64, id 46279, offset 0, flags [DF], proto: TCP (6), length: 40) dmz-host.50167 > dmz-www-server.http: F, cksum 0x0879 (correct), 1:1(0) ack 1 win 46

00:32:38.762182 IP (tos 0x0, ttl 64, id 51956, offset 0, flags [DF], proto: TCP (6), length: 52) dmz-www-server.http > dmz-host.50167: F, cksum 0x542b (correct), 1:1(0) ack 2 win 46

00:32:38.762206 IP (tos 0x10, ttl 64, id 46280, offset 0, flags [DF], proto: TCP (6), length: 40) dmz-host.50167 > dmz-www-server.http: ., cksum 0x0878 (correct), 2:2(0) ack 2 win 46

Notice the timestamp header in the first ack packet.

Re: Unexpected TCP timestamp option cleared in server's response

Panos Kampanakis — Wed, 10 Nov 2010 21:49:00 GMT

I see. I would like to see the back before and after traversing the FW just to make sure but it looks your are right.

Please open a TAC case to have it looked at.

Re: Unexpected TCP timestamp option cleared in server's response

Kureli Sankar — Wed, 10 Nov 2010 23:50:41 GMT

Do you have any module (IPS) on this 5520?

-KS

Re: Unexpected TCP timestamp option cleared in server's response

bwallander — Thu, 11 Nov 2010 00:31:41 GMT

Nope, no modules.

Re: Unexpected TCP timestamp option cleared in server's response

Kureli Sankar — Thu, 11 Nov 2010 02:56:43 GMT

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCsh83148

CSCsh83148 Tcp Timestamp unexpectedly set to 0 for flows reordered by the firewall

It appears that you are running a code where the fix is present.

We really need to get ingress and egress captures on the firewall and look at them.

-KS

Re: Unexpected TCP timestamp option cleared in server's response

bwallander — Thu, 11 Nov 2010 16:03:09 GMT

Thanks KS. I actually came across that bug as well during my searches but it doesn't seem to match in this situation enough to be relevant. I am working on getting captures from the device and can provide them when and if they become available.

In the meantime we're wondering if the TCP state bypass feature introduced in 8.2(1) might help things since it disables TCP normalization for those selected flows. For all I know 8.2 might even contain a fix in the normalizer without even needing to use this feature.

http://www.cisco.com/en/US/docs/security/asa/asa82/configuration/guide/conns_tcpstatebypass.html

Do you have any comment on this?

Re: Unexpected TCP timestamp option cleared in server's response

Kureli Sankar — Thu, 11 Nov 2010 17:42:45 GMT

Tcp state bypass will treat tcp packets as udp packets.

You will not see this issue if this is indeed caused by the ASA.

Also, I agree with you. If infact there is some other defect that you are running into, the upgrade to 8.2 may resolve the issue as well.

I'd upgrade and try without tcp state bypass. Before doing that I'd get ingress and egress captures and make sure that it is this ASA that is clearing the timestamps ingress to egress.

cap capin int inside match tcp any ho x.x.x.x eq 80

cap capout int outside match tcp any ho y.y.y.y eq 80

save the captures https://ip_address/capture/capin/pcap and https://ip_address/capture/capout/pcap

-KS

Re: Unexpected TCP timestamp option cleared in server's response

Andrew Ossipov — Thu, 11 Nov 2010 22:02:08 GMT

Hello Buck,

This is actually expected behavior. The server in this case is violating RFC 1323 that specifically states that the responder can only use TCP Timestamp option if the initiator used it initially:

A TCP may send the Timestamps option (TSopt) in an initial segment (i.e., segment containing a SYN bit and no ACK bit), and may send a TSopt in other segments only if it received a TSopt in the initial segment for the connection.

The ASA is enforcing this compliance unconditionally even if the default TCP Normalizer action is to not clear Timestamp. Enabling TCP State Bypass for the flow should lift this restriction, but this may open you up to other TCP-based network attacks.

Andrew

Re: Unexpected TCP timestamp option cleared in server's response

Kureli Sankar — Thu, 11 Nov 2010 23:37:24 GMT

Good find Andrew ! Again !

-KS

Re: Unexpected TCP timestamp option cleared in server's response

bwallander — Fri, 12 Nov 2010 16:01:06 GMT

Perfect, thank you Andrew. This is exactly what I was looking for. We're planning for an upgrade to 8.2(2) however it might take a few weeks to get clearance approval. Will let you know however.

Thank you to EVERYONE for your help!

-Buck