BGP failover testing

[Kyujin Choi]

Web service user (4.4.4.100) usually connect web server 64.47.x.x through ISP1.

When ISP 1 is down at VA, through BGP it traverses through ISP 2 (Red line), then MPLS -> Inside interface of ASA -> DMZ interface of ASA -> Web server.

Problem is asymmetric route when it comes back to 4.4.4.100 (Blue line), ASA has a connected interface of 64.47.x.x so it sends packets to outside interface.

I first tried to use IP SLA and static route, however connected interface is always preffered. ASA doesn't support PBR. 

Is there any way the packet can't go to connected interface when ISP 1 is down?

Thanks.

           

[rais]

This issue should be resolved using routing. When ISP1 is down, the default should be going out via internal network/MPLS rather than still following ISP1 firewall. When ISP1 is down, traffic came in through FW2, so it should leave FW2 as well.

HTH.

[Peter Koltl]

I first tried to use IP SLA and static route, however connected interface is always preferred.

The reply packet destination is 4.4.4.100, not 64.47.x.x so connected interface has no relevance. You should be able to manipulate the default route by object tracking.

[Kyujin Choi]

Peter, you are almost same page where I am.

You are absolutely right. This is not connected interface issue. This is a NAT issue. But, I couldn't solve the issue.

When ISP 1 is down at VA, user (4.4.4.100) is able to access internal server (10.0.0.100 <-> 64.47.x.200) by using NAT rule at ASA1 at MD.

So, packet goes 4.4.4.100 -> 64.47.x.200 -> 10.0.0.100 (NAT) inside

But, when user tries to access web server 4.4.4.100 -> 64.47.x.100  -> 10.0.4.100 (NAT) inside -> DMZ.  I have an issue.

As you can see I made a NAT exempt rule between DMZ and Inside. so internal user (10.1.1.100) is able to access 10.0.4.100.

However, when outside of user (4.4.4.100) tries to acces web server through ISP 2, ASA 1 translates 64.47.x.100 to 10.0.4.100.

So, I made NAT exempt rule like below. Any incoming traffic (source from DMZ 10.0.4.0/22) coming to DMZ or inside destined 10.0.4.0/22 (DMZ network) are exempted.

DMZ (security level 50) -> Inside (100) translated outbound

DMZ (security levle 50) <- inside (100) translated inbound.

But it didnt' work.

The reason that I said this is a NAT issue is when I tested witn Internal Server it was working fine. Also, Internal User (10.1.1.100) is able to access both DMZ and Inside.

I have to find out what NAT rule I need to have. I think problem maker is below because 10.0.4.100 translates outside with 64.47x.100 always outside. But I am not what destination I need to put here.

[Peter Koltl]

OK, I need ASA version (it seems quite old) and CLI config. (I'm not very familiar with older GUI)

[Kyujin Choi]

I can see NAT Asymmetric error log at ASA  (Left / VA side). In other words, 10.0.4.100 (source) -> 10.0.4.100(destination) exempt rule is needed. so I put another exempt rule like below, but it didn't work.

(Update) I also can see "Portmap translation creation failed for icmp src insdie: 4.4.4.100 dst DMZ 10.0.4.100.

[Kyujin Choi]

Log message...  Do I need PAT?

%ASA-3-305006: {outbound static|identity|portmap|regular) translation

creation failed for protocol src interface_name:source_address/source_port dst

interface_name:dest_address/dest_port

A protocol (UDP, TCP, or ICMP) failed to create a translation through the adaptive security appliance. The adaptive security appliance does not allow packets through that are destined for network or broadcast addresses. The adaptive security appliance provides this checking for addresses that are explicitly identified with static commands. For inbound traffic, the adaptive security appliance denies translations for an IP address identified as a network or broadcast address.

The adaptive security appliance does not apply PAT to all ICMP message types; it only applies PAT ICMP echo and echo-reply packets (types 8 and 0). Specifically, only ICMP echo or echo-reply packets create a PAT translation. As a result, when the other ICMP messages types are dropped, this message is generated.

The adaptive security appliance uses the global IP address and mask from configured static commands to differentiate regular IP addresses from network or broadcast IP addresses. If the global IP address is a valid network address with a matching network mask, then the adaptive security appliance does not create a translation for network or broadcast IP addresses with inbound packets.

For example:

static (inside,outside) 10.2.2.128 10.1.1.128 netmask 255.255.255.128

The adaptive security appliance responds to global address 10.2.2.128 as a network address and to 10.2.2.255 as the broadcast address. Without an existing translation, the adaptive security appliance denies inbound packets destined for 10.2.2.128 or 10.2.2.255, and logs this message.

When the suspected IP address is a host IP address, configure a separate static command with a host mask in front of the subnet static command (the first match rule for static commands). The following static commands cause the adaptive security appliance to respond to 10.2.2.128 as a host address:

static (inside,outside) 10.2.2.128 10.2.2.128 netmask 255.255.255.255

static (inside,outside) 10.2.2.128 10.2.2.128 netmask 255.255.255.128

The translation may be created by traffic started from the inside host with the IP address in question. Because the adaptive security appliance views a network or broadcast IP address as a host IP address with an overlapped subnet static configuration, the network address translation for both static commands must be the same.

[Kyujin Choi]

Well, I am trying double NAT rule.  65.160.x.100 -> 172.16.0.1 at ASA 1 (right)

172.16.0.1 -> 10.0.4.100 at ASA 2 (left)

I made a static route at ASA 1 (172.16.0.1/32  gateway outside) and redistributed through IGP (EIGRP)

I made a static NAT rule like below. So there are two NAT rules for 10.0.4.100 based on destination and interface.

but no luck

[Peter Koltl]

(Update) I also can see "Portmap translation creation failed for icmp src insdie: 4.4.4.100 dst DMZ 10.0.4.100.

That's because you don't have a matching line in

access-list inside_nat0_outbound   (i. e. the Exempt list)

Try to add

src any         dst  10.0.4.0 /22

To better understand, please describe the Layer3 hops

• from 4.4.4.x to DMZ
  
• from 10.1.1.100 to DMZ
  
• from 10.1.1.100 to 10.0.0.100
  

(I'd like to see whether all traffic is routed via VA-ASA)

[Kyujin Choi]

I can't add exempt rule because it tranlates to public IP when ISP 1 is ok.

In other words, normal situation, it will tranlate to public IP at ISP 1 side, which is fine now. So 10.0.4.0 tranlates to public IP as normal situation through ISP 1.

I am focusing on failover situlation of ISP 1.

• from 4.4.4.x to DMZ (this is a problem. from 4.4.4.x to DMZ might be ok,  but reverse packet is issue)
• from 10.1.1.100 to DMZ (ok ping is reachable because of NAT exempt rule)
• from 10.1.1.100 to 10.0.0.100  (ok ping is reachable because of NAT exempt rule)

As I understand, if User at HQ is able to access server in both DMZ and Inside at VA, NAT exempt rule is fine.

The issue is that tranlated internal IP is 10.0.4.0 which needs to translate again (or exempt) in ASA 2 (left)

[Peter Koltl · Accepted solution]

 I can't add exempt rule because it tranlates to public IP when ISP 1 is ok. 

Sorry, you're wrong. This exempt rule you need on inside interface

     src any         dst  10.0.4.0 /22

will translate exempt packets with DST address of 10.0.4.0 /22 from NAT and only if the packet enters on ASA-VA inside (inside-to-DMZ NAT 0). It wouldn't affect inside-to-outside (or DMZ-to-outside) NAT.

What I wanted to inquire is the Layer3 hops through these paths:

• from 4.4.4.x to DMZ                       ASA-MD - SW1-MD - SW2-VA - ASA-VA - SW2-VA   or similar?
• from 10.1.1.100 to DMZ                   SW1-MD - SW2-VA - ASA-VA - SW2-VA   or similar?
• from 10.1.1.100 to 10.0.0.100         ASA-MD - SW1-MD - SW2-VA -    or similar?

I want to find out if ASA-VA is always a Layer3 (routing) hop or there is a shortcut on VA switches when traffic arrives via MPLS.

Apply that NAT0 line and check the traffic with

show conn det long | i 4.4.4

[Kyujin Choi]

Peter, You are right. you pointed right one. I have questions.

1. I made exempt rule in inside interface like below before, it did not work. Only difference is source address from DMZ (10.0.4.0/22) to any. When I put any it worked.

   Prevoius config (didn't work)

  Your config (worked)

Question 1.

Can you explain why you said "any" because 64.49.69.100 ->10.0.4.0 tranlated at ASA 1 (right). So source address 10.0.4.0/22 should work, too if I understand correctly. If I miss something, please let me know.

Question 2.

You made an exempt rule at inside. As I understand, exempt rule are applying to both original and translated address bidirectional, in other words, each host (remote or local) can initiate session (in this case, both directions are applied exempt rule) Also, exempt rule can handle both directions. So I don't need to make any exempt rule at DMZ because one NAT exempt rule handle both direction. Then can I put exempt rule at DMZ side instead of Inside? If yes, can you suggest what rule I need to put in DMZ?  I am just curious.I tried like below, it didnt work. In other words, I guess applied interface is very important.

Thanks for your help.

[Peter Koltl]

A1

ASA-MD has a static NAT, it translates incoming packet's dst address

(and untranslates reply outgoing packets' src address).

On ASA-VA you need to exempt translating the src and dst address to anything by dynamic NAT.

A2

Exempt (NAT 0) rules are unidirectional. (Apart from the fact that reply packets are untranslated.)

Connection establisment direction determines which NAT rule is triggered.