Troubleshooting TCP connections Across PIX Firewall

[limtohsoon]

Hi Sir,

My problem description:

There's a PIX firewall. A client (IP: 10.1.11.100) located on the outside interface of the PIX wants to connect to a server (IP: 172.16.76.20) on the inside interface of the PIX, on TCP ports 8081 & 8443. In actual fact, the physical topology is as follows:

Client---Router---(E1 link)---NATRouter---PIX---EnterpriseNetwork---CheckPointFW---Server(172.16.76.20)

The client's outside global IP is NATed by the NATRouter to the outside local IP 10.1.11.100.

The client able to connect to TCP port 8081 but fail on port 8443. These two ports have been opened on all firewall devices along the path, including routers' ACLs.

I'm troubleshooting the PIX. Below is syslog when client connects to 8081 and 8443.

Client 10.1.11.100 -> Server 172.16.76.20 TCP 8081 (Successful)

---------------------------------------------------------------

2007-01-17 15:44:13 Local4.Info 10.20.150.1 Jan 16 2007 22:47:51: %PIX-6-302013: Built inbound TCP connection 262269560 for outside:10.1.11.100/1033 (10.1.11.100/1033) to inside:172.16.76.20/8081 (172.16.76.20/8081)

2007-01-17 15:44:37 Local4.Info 10.20.150.1 Jan 16 2007 22:48:15: %PIX-6-302014: Teardown TCP connection 262269560 for outside:10.1.11.100/1033 to inside:172.16.76.20/8081 duration 0:00:24 bytes 2357 TCP Reset-O

Client 10.1.11.100 -> Server 172.16.76.20 TCP 8443 (Failed)

-----------------------------------------------------------

2007-01-17 15:57:32 Local4.Info 10.20.150.1 Jan 16 2007 23:01:10: %PIX-6-302013: Built inbound TCP connection 262273622 for outside:10.1.11.100/1037 (10.1.11.100/1037) to inside:172.16.76.20/8443 (172.16.76.20/8443)

2007-01-17 16:15:18 Local4.Info 10.20.150.1 Jan 16 2007 23:18:56: %PIX-6-302014: Teardown TCP connection 262273622 for outside:10.1.11.100/1037 to inside:172.16.76.20/8443 duration 0:17:46 bytes 18503 FIN Timeout

Below is explanation of the FIN timeout message:

http://www.cisco.com/en/US/partner/products/sw/secursw/ps2120/products_system_message_guide_chapter09186a0080441d0f.html#wp1039843

"Force termination after 15 seconds await for last ACK"

From the above syslog messages, can anyone deduce whether it's a client-side issue or the PIX illegitimately tearing down connections?

Please help.

Thank you.

B.Rgds,

Lim TS

[vijayasankar]

Hi,

A quick look at the syslog message for the TCP connection termination reveals that the connection( from client to server on port 8443) was live for some time( 17 minutes) and a considerable amount of data was exchanged over this connection.( indicated by the following "bytes 18503 FIN Timeout", in the syslog message.)

Hence it appears that the connection was through and working for a while and getting terminated improperly.

May be you need to check from the application front on what is happening over the tcp transaction between the client and the server.

To investigate further, capture sniffer/ethereal traces from the client and the server and examine what is happening. Im sure you will get enough clue from the sniffer trace.

Hope this helps.

-VJ

[limtohsoon]

Hi VJ,

Thanks for your response.

My customer had captured sniffer trace on the client and server during the time the syslog was captured. Sending the traces to me.

Do you think the PIX could be the culprit for the FIN Timeout message?

For the TCP connection to port 8081, it was torn down by the PIX after a TCP Reset-O. This is normal, right?

Thank you.

B.Rgds,

Lim TS

[jmia]

Hi,

If your seeing: TCP Reset-O this means that the connection was reset from a client on the outside. If your seeing

TCP Reset-I this means that the connection was reset from a client on the inside.

Hopes the above helps.

[limtohsoon]

Hi VJ,

Attached is the Ethereal sniffer trace captured on the client machine. The client's actual IP address is 172.20.21.100, which the NAT router translates to the outside local IP 10.1.11.100.

Look at frames#26-28; they show the 3-way handshake for connection to port 8443. Frame#28 shows the client sends an ACK segment with the error message TCP CHECKSUM INCORRECT. What does that mean? Is it a client apps issue?

I also notice the client sending many TCP Retransmission for previously sent [PSH, ACK] segment. E.g. frames#29-34.

The PIX firewall is running version 6.2. Could it run into the following bug:

Cisco Security Notice: Response to Cisco PIX TCP Connection Prevention

http://www.cisco.com/en/US/products/sw/secursw/ps2120/products_security_notice09186a008059a411.html

Please help.

Thank you.

B.Rgds,

Lim TS

[vijayasankar]

Hi Lim,

Thanks for providing the captures.

You have earlier mentioned that from client o server on port 8081 is working fine and only on port 8443 problems are faced.

Check frames 5 to 10 which was initiated by the client to the server on port 8081.

Even this transaction is having problems. Through out the trace we could see that the problem is observed for both port 8081 and 8443. So my observation is that this problem is common to both ports, and not specific only to port 8443.

Looks like the packets from client to the server are not reaching the server or not properly handled by the server.

It could be possible that the network is congested and the return packets from the server are dropping some where in the network.

Ensure that there are no L1/L2 problems in the network path from the client to the server.

To investigate further, you can take the sniffer capture simultaneously on both the client and the server to see where the problem is.

In the simultaneous capture, it will be easy for you to check whether the server had actually replied or not.

If the server had actually replied , but the packet had not reached the client => indicates a potential problem in the network path.

If the server itself is not replying back, then it points out to be a problem with the server.

The Checksum error that you are seeing is not an alarming one. You can check this URL to understand the reason for the same, http://www.ethereal.com/faq.html#q11.1.

Have the simultaneous capture done at both the client and the server to rule out the exact component which is causing this issue.

Hope this helps.

-VJ

[limtohsoon]

Hi VJ,

I was just updated by my customer that removing the static nat on the NAT router actually solves the problem. Now, the server sees the client source IP as 172.20.21.100.

They suspect it's the adjacent PIX firewall causing the problem. Below is current fixup config of the PIX:

fixup protocol ftp 21

fixup protocol http 80

fixup protocol h323 h225 1720

fixup protocol h323 ras 1718-1719

fixup protocol ils 389

fixup protocol rsh 514

fixup protocol rtsp 554

fixup protocol smtp 25

fixup protocol sqlnet 1521

fixup protocol sip 5060

fixup protocol skinny 2000

Is it necessary to define a fixup for port 8443? If it's NAT causing the issue, why communication to port 8081 works earlier?

Please provide some insight.

Thank you.

B.Rgds,

Lim TS

[vijayasankar]

Hi Lim,

Thanks for your update.

As stated earlier, from this ethereal capture, we could clearly see that the problem is not specific to port 8443. We could also see that same problem for port 8081 is the capture file.

NATing before the firewall shouldn't cause any issues like this.I have seen setups like this earlier, which are all working fine.

Probably you need to check the Device that is doing the NAT to investigate the root cause of this problem.

The "Fixup" feature is not related to this issue.

-VJ