Asymetric Routing With ASA

[Alex Pfeil]

I just wanted to share a configuration I have been working on.  There are two internet circuits to two different Internet service providers (ISPs).  Say we want to do BGP between the two ISPs and place a firewall between our boundary routers and our inside routers.

 R3 is an inside router with a default route to the firewall. The firewall has a default route to R1, this is for testing purposes. R1 has a route to a loopback interface configured on R2.  R2 has the loopback interface configured and a route back directly to the firewall.

Here is a packet coming in on Gi1/0 from R1.

*Oct 14 18:02:15.751: IP: s=192.168.3.3 (GigabitEthernet1/0), d=192.168.4.1, len 100, input feature 

*Oct 14 18:02:15.759: IP: s=192.168.3.3 (GigabitEthernet1/0), d=192.168.4.1, len 100, rcvd 2

Here is the reply going directly to the firewall from R2.

*Oct 14 18:02:15.767: ICMP: echo reply sent, src 192.168.4.1, dst 192.168.3.3, topology BASE, dscp 0 topoid 0
*Oct 14 18:02:15.771: FIBipv4-packet-proc: route packet from (local) src 192.168.4.1 dst 192.168.3.3
*Oct 14 18:02:15.775: FIBfwd-proc: packet routed by adj to GigabitEthernet0/0 192.168.1.1

This confirms the packet takes one route out to the internet and returns on a different path, the firewall allows the traffic through.

If we placle a default route to both ISPs on the firewall, we should be able equal cost load balance traffic to either outside boundary router, let BGP do its thing, and it should not matter which side the return traffic comes back in on.  I hope this article sparks some conversation and I look forward to comments.

Please mark this as helpful if you find it helpful.

[tonypearce1]

What's the question? 

From the diagram, looks like you have a single outside interface to both upstream routers. So I dont think that's asymmetric routing on the ASA.

[Alex Pfeil]

This is not a question. I just labbed this up to verify that if packets went out router one and returned on router 2, the ASA did not have any issues. The asymmetric routing takes place out on the internet. In the past, I have always had a subnet connected to the ASA which only had one internet connection so I was testing. I just wanted to share this as it might help someone.

[tonypearce1]

Yea, in that case it's not asymmetric routing on the ASA. You'll have a different experience (by default) with the ASA if you have asymmetry; where the ASA receives a TCP ACK on a different interface to where it sent the TCP SYN for example. 

Let me describe a scenario that I set up previously. A customer of mine had a London office and a NY office. They had a 100M fibre pipe coming in to their office in London. This 100M was divided up so that they had 80M internet and a 20M private circuit to their NY office. The link to NY was taken care of via a router attached to the LAN. They realised that because of the higher internet bandwidth, things tended to work better between the offices if they set up a site to site VPN over the 80M internet link; but they wanted everything to fail over to the private circuit if the local internet went down. So I used a routing protocol to take care of the auto failover which worked good as expected, however initially there was one problem. 

Test 1: was to pull the internet link upstream (so the ASA did not have an interface-down scenario). As expected, traffic between the offices continued to operate on the 20M link. 

Test 2: was to plug back in the upstream cable and allow everything to fail back to the ASA VPN. Which it did, HOWEVER here is the problem - Because the flows had been established already via the router and 20M link, when the traffic began routing back through the ASA, the ASA was denying the traffic. You see the ASA is a security appliance and so by default is very secure. Compared to a router, where by default it's primary concern is routing (not security). So, because the ASA was seeing this new traffic but did not see the TCP establishment phase (handshakes) it was denying it. This is asymmetric routing. 

Now we could also have another scenario where the internet is a bit flappy (due to an issue with a cable for example), so routing is changing often. The traffic would fail over fine to the router but failing back would cause the ASA to deny the traffic for a while, until the end devices realise their packets are not being acknowledged and so they reset and start the TCP conversations again. 

A workaround for these scenarios is to tell the ASA to pass traffic between private subnets and don't check for their state in the ASA state table. Ie. the traffic was being denied because the traffic was not a new connection set up (which it should have been) and did not already exist in the ASA's table. This is called TCP state bypass and allows the TCP traffic to flow (lowering the security somewhat to be like a router). 

Out on the internet,  I find that asymmetric routing happens often. In fact for me with an office in Australia and an office in Sri Lanka, this plagues me a lot. But my ASA's are unaffected by it, because to them they simply send out a packet on the outside interface and eventually they receive a response to the same interface. There's no asymmetry from the ASA's perspective and so no security checks are acknowledged there. Now the asymmetry on the internet affects me because of voice and video traffic. Normally the internet ISPs peering links are saturated and so experience packet loss in one direction. It's very annoying :) Normally I complain and ask them to use the same forward and reverse paths which go around the issue. 

Hope this helps :)

[Alex Pfeil]

Very good explanation. I was doing the test since we have two outside routers. We knew the slasymmetric routing would be taking place out to the internet, not specifically on the ASA. The main reason for doing the test was to make sure that the ASA would not complain if it sent out a packet to router 1 and received a packet from router 2.

Thank you for sharing! Great conversation!

[cdusio]

There's n o issue with this setup. You have a default route on the asa pointing to r1 that's why it's going out that way and you have a static route to the outside of the asa from R2 that's exactly what should be happening. ASA only cares if it sees traffic come in one interface and does not see the return traffic. So example outside>inside, the traffic comes in, goes out through another firewall somewhere else, then the return traffic tries to come in through the original ASA outside interface. That will  break it. You can't control the Internet side totally anyways so that stuff will depend on a lot of factors but that's not what you're asking.

[tonypearce1]

I see where you're coming from. Think about the packet as received by the ASA from R2, specifically about the source mac, source IP and destination mac, destination IP of the packet. When it's received by the ASA it will have a destination IP of the ASA or a NAT on the ASA that the ASA is responsible for.. The source IP will be some host on the internet (not R2). The destination mac will be that of the ASA, so that will be good. ASA doesnt care about the source mac. So the ASA sees the packets as valid. Once the ASA receives the packet destined for its mac address, the packet goes to layer 3. The ASA acts at layer 3 and above, so for example you cant filter by mac address on a routed ASA. 

For example, you could get a situation where you use HSRP on the upstream routers and HSRP fails over. The ASA would have arp'd for the mac of the upstream gateway which is now the dead router. So the ASA would be sending packets with a destination mac of the dead router which the active router wouldnt receive / pick up.