Network design - Azure BGP peering with ASAs

[Nenad Stojanovic]

Hello,

I have two ExpressRoute connections to Azure. They are on the ASAs. (Please see network diagram). Firewalls are running as standalone - no failiover pair. Core switch is running OSPF with firewalls. Firewalls are peering with AZURE using BGP.

ASA1 has a default route out to the internet. Azure is advertising specific prefixes to me and there is no public internet access.

Issue I have is that Azure ExpressRoutes are designed to be Active/Active and when I send traffic through primary firewall (ASA1), traffic returns to both primary and secondary link (performed captures). I tried to influence return path using AS prepend but Azure does not support that with Private ASN. Since firewalls are doing statefull inspection return traffic on the standby firewall is dropped and there is intermittent packet loss.

I tried to implement tcp bypass for specific traffic to Azure but that did not solve the problem. Not sure why?

Would active/active forwarding be possible using firewalls? Do they need to be in active/active failover in order to do that?

Is there any way to use active/standby forwarding with the current design - use one link to Azure as primary and the other one as a backup? 

Core switch stack:

Data subnet 192.168.15.0/24

Lo0 - Public ip address advertised to Azure - 209.x.x.24/32

router ospf 1
redistribute connected subnets
network 10.0.0.0 0.0.0.3 area 0
network 10.0.0.4 0.0.0.3 area 0

ASA1:

Natting all inside interesting traffic to 209.x.x.24 (which is the loopback0 on the switch) That is the only thing that I advertise to Azure

object network azure-net
nat (INSIDE,AZURE-PRI) dynamic 209.x.x.24

router ospf 1
network 10.0.0.0 255.255.255.252 area 0
network 172.16.0.0 255.255.255.252 area 0
log-adj-changes
redistribute connected metric 10 subnets
default-information originate
!

prefix-list LOCAL seq 5 permit 209.x.x.24/32

route-map TO-AZURE permit 10
match ip address prefix-list LOCAL
set ip next-hop 209.x.x.17

router bgp 655xx
bgp log-neighbor-changes
address-family ipv4 unicast
neighbor 172.16.0.2 remote-as 655xx
neighbor 172.16.0.2 activate
neighbor 209.x.x.18 remote-as 120xx
neighbor 209.x.x.18 activate
neighbor 209.x.x.18 route-map TO-AZURE out
network 209.x.x.24 mask 255.255.255.255 route-map TO-AZURE
no auto-summary
no synchronization
exit-address-family

route OUTSIDE 0.0.0.0 0.0.0.0 64.x.x.221 1 track 1

ASA2:

Natting all inside interesting traffic to 209.x.x.24 (which is the loopback0 on the switch)

object network azure-net
nat (INSIDE,AZURE-STBY) dynamic 209.x.x.24

router ospf 1
network 10.0.0.4 255.255.255.252 area 0
network 172.16.0.0 255.255.255.252 area 0
log-adj-changes
redistribute connected metric 10 subnets
default-information originate
!

prefix-list LOCAL seq 5 permit 209.x.x.24/32

route-map TO-AZURE permit 10
match ip address prefix-list LOCAL
set ip next-hop 209.x.x.21
set as-path prepend 655xx 655xx

router bgp 655xx
bgp log-neighbor-changes
address-family ipv4 unicast
neighbor 172.16.0.1 remote-as 655xx
neighbor 172.16.0.1 activate
neighbor 209.x.x.22 remote-as 120xx
neighbor 209.x.x.22 activate
neighbor 209.x.x.22 route-map TO-AZURE out
network 209.x.x.24 mask 255.255.255.255 route-map TO-AZURE
no auto-summary
no synchronization
exit-address-family
!

Thank you in advance!

[Marius Gunnerud]

I am assuming you were trying AS prepend on the ASA side?

I am not familiar with Azure, but does it support local preference?  You might be able to manipulate traffic on the Azure side with that.

Active/Active would not solve this issue as this would require two contexts and the contexts do not share state table information.

TCP bypass would be a solution here, but perhaps there was a traffic flow that you did not account for in your configuration?

If you have an Active/Standby setup, this would solve the issue as only one ASA will be active at any given time.

another solution you could look into is clustering the ASAs.  Then you will have two active ASAs forwarding traffic at the same time.

[Nenad Stojanovic]

We can't do anything on the Azure side routers. Only ASAs. Prepending was done on the ASA2. 

Azure side does Active/Active and as far as I see the only way to influence that would be to use AS prepend (on the ASA side) with Public ASN.

"TCP bypass would be a solution here, but perhaps there was a traffic flow that you did not account for in your configuration?"

My tcp bypass looked like this. Not sure what I am missing? I did it only for the 2 ip addresses that I was testing. 

access-list tcp_bypass extended permit tcp host 192.168.15.2 host 52.170.x.x 
access-list tcp_bypass extended permit tcp host 209.133.x.x host 52.170.x.x

class-map tcp_bypass
match access-list tcp_bypass
!

policy-map tcp_bypass_policy
class tcp_bypass
set connection random-sequence-number disable
set connection advanced-options tcp-state-bypass

service-policy tcp_bypass_policy interface AZURE (interface that goes to Azure). I also tried to implement service-policy on the inside interface at the same time.

"another solution you could look into is clustering the ASAs.  Then you will have two active ASAs forwarding traffic at the same time."

Thanks! That is what I am thinking would be the best solution but was not sure.