established connection can send data but data never comes back

[erotavlas]

I have a sending application that it is establishing a TCPIP socket connection to a vlient that has a receiving application on another server (completely separate networks)  The receiving end has a Cisco Router 1605R and has allowed my connection using this firewall rule

access-list 101 permit tcp host xx.xxx.xx.xx any eq 5600 log 

(where the x are actually numbers corresponding to the senders IP address)

I can establish an outbound connection to the receiver and the connection shows up on the machine.  After the connection is established  I can send data and it is received by the receiving application (I observed this using a socket test application, data actually gets through the firewall)  However I need to send an acknowledgment back on the same session to the sender.  This cannot be transmitted and shortly after I try sending the connection is closed with error 10060.

Does anyone know what would cause this?  Able to send data one way but not send anything back on the same session?

I have a feeling this might be related to some configuration of their router but i do not know what to look for

Thanks in advance for any advice on how to diagnose this.

[fadlouni]

Hi.

If you are sending data on the tcp connection, that means the tcp 3-way handshake is already established. if the tcp session is established, that means at least one packet (the tcp syn-ack to your tcp syn) has came back to your client.

so that might mean that only the data part is not making it through. this could be a result of a firewall dropping the data traffic in one direction, or a problem at the application level.

You'll have to check if the router is running a firewall (cbac or zone-based firewall), or if the PC is running it's own firewall, and troubleshoot those. otherwise install packet capturing tools (like wireshark) on the sender/receiver and see what is happening from a packet level.

Regards,

Fadi.

[erotavlas]

Fadi, thank you for your reply.

What is the difference between the packet (the tcp syn-ack to your tcp syn) that is able to make it back and the rest of the packets that are unable to return back to the sender?  (why does this one get through?)

[fadlouni]

Hi.

the tcpy syn,ack has no data in it so it's pure layer 4 packet. while your packets have data, this data might have been blocked by a layer 7 firewall for some reason.

All of this is just speculation right now, without actually seeing the packets on sender and receiver side, we can't really know what exactly why dropped and why.

Regards,

Fadi.

[erotavlas]

What layer does this router operate on?

Also if the access list does not have an entry for the sender's IP, will the tcpy syn,ack still get through, or will it be denied? (i'm guessing it would get through if the firewall only inspects packets at the application layer?)

Thanks again

[fadlouni]

Hi.

without seeing the full router's config we can't really say. the acl you mentioned is pure layer 4. but if it's used in conjunction with with CBAC or zone-based firewall, then you have more layers involved.

also acls on their own are unidrectional. without seeing how this acl was applied, it could only be applied from one side to the other, and is not relevant in the other direction. so in principal, if this traffic is applied in direction of a->b . the return traffic from b->a is not validated with this acl.

again, we can keep speculating all we want about possible reasons why the return data traffic doesn't make it through, the router itself might not even be the device causing the issue. we should not troubleshoot this way. let's get the config and packet captures and we'll have a better view.

Regards,

Fadi.

[erotavlas]

So far I managed to do a packet capture on the site that is sending the data.  After the Syn/Ack packet is receved everything following are TCP retransmissions Order of the packets is like this

[SYN]

[SYN,ACK]

[ACK]

[PSH,ACK]

[TCP Retransmission] [PSH,ACK]

[TCP Retransmission] [PSH,ACK]

[TCP Retransmission] [PSH,ACK]

[TCP Retransmission] [PSH,ACK]

[TCP Retransmission] [PSH,ACK]

Only the [SYN,ACK] is received fine.

What else can I look for.

Thanks for any suggestion,

[fadlouni]

If only syn,ack seems to be making it through and not the rest, this looks like a behaviour of a firewall or proxy is on the path, and the firewall/proxy is the one answering the syn,ack instead of the client (so it's proxying the tcp connection before forwarding the traffic back to the source), and when 3 way is complete, the firewall tries to send syn,ack to source along with the rest of the packets but since we see retransmissions, this means this isn't making it through.

like i mentioned, we can only confirm all of this, if we get simultaneous captures from both ends.

Regards,

Fadi.

[erotavlas]

Hi Fadi,

We determined that the Cisco Router 1605R is the problem.  We determined this by removing it from the network.  After doing so no Connection Timeout errors (10060) were observed and all sites that were initially having trouble transmitting data succesfully connected and transmitted their data.

Now my question would be, aside from the access control list, what other settings on the router should we look at that may explain the behaviour (i,e, with no changes to the ACL, sites have intermittent blocks on their connectivity - sometimes the timeout errors can last for days then clear up)

Thanks in advance