TCP Established and BGP conflict with two ISP's?

[kduckett]

I have an extended ACL applied inbound to my two external interfaces (two separate ISP's using BGP). Within the ACL, I have an ACE for allowing established TCP connections, i.e. "access-list 100 permit tcp any any established ".

Will this cause a problem if a SYN packet enters one interface of the border router, the SYN ACK packet leaves the other external router interface and the ACK packet returns on the original interface?

Thanks,

Keith

[milan.kulik]

Hi,

the ACL is watching just the particular incoming packet, not the session history.

According to the Command Reference Guide http://www.cisco.com/en/US/docs/ios/12_3/ipaddr/command/reference/ip1_a1g.html#wp1078593

" The established keyword is used only for the TCP protocol to indicate an established connection. A match occurs if the TCP datagram has the ACK or RST bits set, which indicates that the packet belongs to an existing connection."

If BGP connection between the peers is not permitted by your ACL explicitely, your router has always to initiate the BGP session (BGP router ID is used to decide which neighbor starts the session).

But as long as your router starts the session, it doesn't matter which interface is used for returning traffic when both are having established ACE permitted.

BR,

Milan

[kduckett]

Hi Milan,

Thanks for the reply. The connections being denied are being initiated by an external network and I see the connection attempt in my syslog. Their SYN packet is coming in via Serial1/0 and my return SYN ACK packet is leaving via Serial 1/1 therefore interface Serial 1/0 never sees the return packet. Serial 1/0 only sees the first and third packets. When the external network sends their ACK packet in response to my SYN/ACK packet, the interface ACL drops the packet because it never saw and recorded the 2nd packet.

I have ACE entries which allow the external ISP's to initiate BGP updates.

Hopefully I am identifying the problem correctly.

Keith

[milan.kulik]

Hi Keith,

would you show the complete ACL plus the Syslog error message?

I would not expect an established ACL to record the whole SYN/ACK sequence ...

BR,

Milan

[Jon Marshall]

kduckett wrote:
Hi Milan,
Thanks for the reply. The connections being denied are being initiated by an external network and I see the connection attempt in my syslog. Their SYN packet is coming in via Serial1/0 and my return SYN ACK packet is leaving via Serial 1/1 therefore interface Serial 1/0 never sees the return packet. Serial 1/0 only sees the first and third packets. When the external network sends their ACK packet in response to my SYN/ACK packet, the interface ACL drops the packet because it never saw and recorded the 2nd packet.
I have ACE entries which allow the external ISP's to initiate BGP updates.
Hopefully I am identifying the problem correctly.
Keith

Keith

It boils down to the question of whether the "established" keyword records the interface the connection was seen on. Like Milan i don't think it does.

If the SYN packet is addressed to s1/0 interface but the SYN/ACK has a source address of s1/1 then it is not seen as the same TCP connection and will not work. This is nothing to do with the "established" keyword though.

Have i misunderstood your topology ?

Jon

[kduckett]

Jon and Milan,

Here is more info on my topology and the situation I am experiencing.

I have several customers who cannot access any of my web servers when I have both ISP interfaces active. It works when I disable either of the ISP interfaces and only have one active. I have two serial interfaces, Serial1/0 and Serial1/1. Bother serial interface have the same extended ACL applied inbound. All of these affected customers come in via Serial1/0. When I perform a traceroute back to the customers host, the packets all leave via interface serial1/1. BGP is enabled.

Let's assume the  customer's IP address is 10.1.1.1 and my web servers address is 172.16.1.1. I have a firewall (ASA 5500) in between the router and the web server and its external IP address is 192.168.1.1. I turn on the packet capturing packet feature of the firewall and I see the SYN and SYN/ACK packets to/from 10.1.1.1 and 172.16.1.1. Since my traceroute sends the packet out interface Serial1/1, I am making an assumption that the SYN/ACK going back to the customer's host of 10.1.1.1 is using the same interface since BGP states that this is the best route. I never see the ACK from the customers IP address 10.1.1.1 being captured by my firewall sniffer. Again, I am making another assumption that the router ACL is blocking the packet because it sees the ACK without an established connection as this interface only saw the first and third packets.

If you have any other ideas and have ruled out "TCP established" as the culprit, I would love to hear them.

Milan, unfortunately I cannot share with you my ACL.

Thanks for your replies.

Keith

[Jon Marshall]

kduckett wrote:
Jon and Milan,
Here is more info on my topology and the situation I am experiencing.
I have several customers who cannot access any of my web servers when I have both ISP interfaces active. It works when I disable either of the ISP interfaces and only have one active. I have two serial interfaces, Serial1/0 and Serial1/1. Bother serial interface have the same extended ACL applied inbound. All of these affected customers come in via Serial1/0. When I perform a traceroute back to the customers host, the packets all leave via interface serial1/1. BGP is enabled. 
Let's assume the  customer's IP address is 10.1.1.1 and my web servers address is 172.16.1.1. I have a firewall (ASA 5500) in between the router and the web server and its external IP address is 192.168.1.1. I turn on the packet capturing packet feature of the firewall and I see the SYN and SYN/ACK packets to/from 10.1.1.1 and 172.16.1.1. Since my traceroute sends the packet out interface Serial1/1, I am making an assumption that the SYN/ACK going back to the customer's host of 10.1.1.1 is using the same interface since BGP states that this is the best route. I never see the ACK from the customers IP address 10.1.1.1 being captured by my firewall sniffer. Again, I am making another assumption that the router ACL is blocking the packet because it sees the ACK without an established connection as this interface only saw the first and third packets. 
If you have any other ideas and have ruled out "TCP established" as the culprit, I would love to hear them.
Milan, unfortunately I cannot share with you my ACL.
Thanks for your replies. 
Keith

Keith

There is no natting going on on the router is there ?

I assumed that "established" does not record the interface but it may be worth testing if i get the time tomorrow.

Also, is it just acls with the established keyword on your router ie. nothing else could be blocking it ?

Jon

[kduckett]

Hi Jon,

Thanks again for looking ito this for me. It is greatly appreciated.

I am not using NAT on the router, just on the firewall. The keyword "established" only appears within the ACL's on the border router. The inbound ACL's are identical and permit HTTP/HTTPS traffic to the web servers. The firewall ACL also permits HTTP/HTTPS traffic to the web servers static NAT addresses.

Keith

[Giuseppe Larosa]

Hello Keith,

just a doubt are you using two lines in your ACL one with BGP keyword after destination address and one after source address?

We used to write ACLs in this way because the well known port is taken by one side of the eBGP session, just to be sure to be able to identify the BGP session whatever side uses the well known port.

Hope to help

Giuseppe

[kduckett]

Hi Giuseppe,

Yes, I have two entries in my ACL for BGP for each ISP. Here are the entries with a slight modification to obscure the IP addresses:

access-list 100 permit tcp host xx.xx.xx.41 eq 179 host xx.xx.xx.42 log
access-list 100 permit tcp host xx.xx.xx.41 host xx.xx.xx.xx.42 eq 179 log
access-list 100 permit tcp host xx.xx.xx.221 host xx.xx.xx.222 eq 179 log
access-list 100 permit tcp host xx.xx.xx.221 eq 179 host xx.xx.xx.222 log

Thanks,

Keith

[milan.kulik]

Hi Keith,

let me clarify if I understand your problem correctly:

a) There's no problem to establish your eBGP sessions, you are peering with both IPS concurrently with no problem.

b) But the users accessing your web servers are in trouble when they initiate a HTTP/HTTPS session through one serial line and the reply packets sent by your servers are sent back through the second serial line.

Correct?

My question then is:

Why do you apply established ACE inbound?

IMHO, it should be applied outbound on the serial lines, as you have to permit HTTP/HTTPS access to your servers in the beginning of your ACL anyway.

And then your servers are replying and you need to permit this traffic somehow (if an outbound ACL applied to your serial lines).

But it's still weird why your problem occurs only when two serial lines are active.

Don't you have reverse-path forwarding activated, e.g. (see http://www.cisco.com/web/about/security/intelligence/unicast-rpf.html )?

BR,

Milan

[kduckett]

Milan,

Sorry for the late reply but it has been a hectic day. Yes, we are able to establish the eBGP sessions with both ISPs with no problem. You also have the traffic pattern correct.

We started using "TCP established" when our HQ security management directed us to block specific TCP and UDP ports on the border router. Let's use TCP 2001, as an example, to be denied and reverse the direction of the data traffic. Since the router does not recognize connection states, a SYN packet initiated "internally" containing source port TCP 2001 and destination TCP 80 would cause a SYN/ACK packet where the source port was TCP 80 and the destination port being TCP 2001. The inbound ACL on the router's external interface would cause the packet to be denied. We are blocking several hundred ports even though we have an explicit deny statement at the end of our ACL. This was causing us sporadic problems when the ports matched the ACL paramaters even if it was legitimate traffic. TCP established seems to have resolved this issue. My preference is to permit only the ports that are needed inbound (80, 443, 25, 53, etc.) and deny everything else with the deny any any statement.

We have reverse-path forwarding enabled.

Thanks,

Keith

[milan.kulik · Accepted solution]

Hi Keith,

IMHO, reverse path checking is causing your trouble.

As you mentioned, the packets sent by remote users are coming through one serial interface, while the servers are sending packets back which are sent from your router through the second serial interface.

In that case, when reverse path checking checks the incoming packet, it finds the source IP address coming from a wromg direction (route to the source address points to the second interface) and drops it.

When only one serial interface is active, the problem "disapears", as reverse path checking result is OK.

It's strange you are saying you see the SYN packets on your FW, I would expect them not to pass at all.

So I'd suggest to disable reverse path checking and observe if that helps.

BR,

Milan

[kduckett]

Milan,

You were correct in your suspicion that the "ip verify unicast reverse-path" configuration on the serial interface was causing the problem. You are the man!!!

I apologize for the delay, but I had to wait until an optimal time to make the configuration change and test the affected applications.

Thanks again for your support and time!

Keith