topic Re: Inbound NAT issue with PBR on ASA in Network Security

Inbound NAT issue with PBR on ASA

tato386 — Fri, 21 Feb 2020 14:33:01 GMT

I have an ASA (9.6.3) with two interfaces connected to the Internet. The ASA default route is pointing to ISP A and I have PAT and NAT using ISP A working fine. I have a route-map using PBR that sets default next hop for certain clients to ISP B. For the clients using ISP B I also have PAT and NAT setup. PAT works fine and NAT works fine for _outbound_ traffic but I cannot get any inbound services to work.

Test show that it is not a problem with rules or NAT because if I add a static route on the ASA that uses ISP B for a particular Internet IP the inbound works. So I guess I need to add something else for NAT/PBR to work but I am not sure what. Any ideas?

Thanks
Diego

Re: Inbound NAT issue with PBR on ASA

Flavio Miranda — Sun, 22 Oct 2017 18:56:24 GMT

Hello @tato386

Really looks like routing problem, probably asymmetric routing. Probably a capture will give you the answer.

If possible, share you config here so that we can take a look.

-If I helped you somehow, please, rate it as useful.-

Re: Inbound NAT issue with PBR on ASA

Spooster IT Services — Mon, 23 Oct 2017 14:27:26 GMT

Hi Diego,

Can you please send me the configuration related to PBR that you have done on ASA?

Re: Inbound NAT issue with PBR on ASA

tato386 — Tue, 24 Oct 2017 22:25:48 GMT

sanitized config:

ASA Version 9.6(3)1
!
interface GigabitEthernet0/0
nameif inf_Data
security-level 100
ip address 10.1.1.254 255.255.255.0
policy-route route-map ALT-GATEWAY
!
interface GigabitEthernet0/1
desc /30 with /29 routeable block
nameif inf_ISPB
security-level 0
ip address 2.2.2.2 255.255.255.252
!
interface GigabitEthernet0/5
nameif inf_ISPA
security-level 0
ip address 1.1.1.2 255.255.255.248
!
!
object network host1
host 10.1.1.20
object network net_ISPB-PublicBlock
subnet 3.3.3.0 255.255.255.248
object network ip_ISPB-NAT
host 3.3.3.1

access-list acl_Firewall-ISPA extended permit icmp any any

access-list acl_Firewall-ISPB extended permit icmp any any
access-list acl_Firewall-ISPB extended permit tcp any object host1 eq telnet

access-list acl_ISPB-PBR extended permit ip object host1 any4
access-list acl_ISPB-PBR extended deny ip any4 any4
!
!
object network host1
nat (inf_Data,any) static ip_ISPB-NAT
!
access-group acl_Firewall-ISPB in interface inf_ISPB
access-group acl_Firewall-ISPA in interface inf_ISPA
!
route-map ALT-GATEWAY permit 10
match ip address acl_ISPB-PBR
set ip default next-hop 2.2.2.1
!
route inf_ISPA 0.0.0.0 0.0.0.0 1.1.1.1 1

Re: Inbound NAT issue with PBR on ASA

Spooster IT Services — Thu, 26 Oct 2017 11:11:24 GMT

Hi Diego,

Can you please run packet tracer as mentioned below and share the output with us?

packet tracer input int_ISPB tcp 8.8.8.8 12121 3.3.3.1 23 detailed

Re: Inbound NAT issue with PBR on ASA

tato386 — Thu, 26 Oct 2017 12:48:26 GMT

The packet trace looks as it should. The problem is that the ASA is trying to reply out of the wrong interface. If I add a static route to 8.8.8.8 using inf_ISPB it works. So it seems that PBR is respected when the inside host initiates a flow to the outside but it is not used for packets initiated from outside to inside hosts.

asa#packet input inf_ISPB tcp 8.8.8.8 1212 3.3.3.1 23 detailed

Phase: 1
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
object network host1
nat (inf_Data,any) static ip_Test
Additional Information:
NAT divert to egress interface inf_Data
Untranslate 3.3.3.1/23 to 10.1.1.20/23

Phase: 2
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group inf_ISPB_access_in in interface inf_ISPB
access-list inf_ISPB_access_in extended permit tcp any object host1 eq telnet
Additional Information:
Forward Flow based lookup yields rule:
in id=0x2aaac3165830, priority=13, domain=permit, deny=false
        hits=948, user_data=0x2aaab97918c0, cs_id=0x0, use_real_addr, flags=0x0, protocol=6
        src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
        dst ip/id=10.1.1.20, mask=255.255.255.255, port=23, tag=any, dscp=0x0
        input_ifc=inf_ISPB, output_ifc=any

Phase: 3
Type: CONN-SETTINGS
Subtype:
Result: ALLOW
Config:
class-map class-default
match any
policy-map global_policy
class class-default
set connection conn-max 0 embryonic-conn-max 0 random-sequence-number disable
service-policy global_policy global
Additional Information:
Forward Flow based lookup yields rule:
in id=0x2aaac3076560, priority=7, domain=conn-set, deny=false
        hits=3658, user_data=0x2aaac3073670, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
        src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
        input_ifc=inf_ISPB, output_ifc=any

Phase: 4
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x2aaac099fcb0, priority=0, domain=nat-per-session, deny=false
        hits=1074533, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=6
        src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
        input_ifc=any, output_ifc=any

Phase: 5
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x2aaac137e530, priority=0, domain=inspect-ip-options, deny=true
        hits=3977, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
        src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
        input_ifc=inf_ISPB, output_ifc=any

Phase: 6
Type: NAT
Subtype: rpf-check
Result: ALLOW
Config:
object network host1
nat (inf_Data,any) static ip_Test
Additional Information:
Forward Flow based lookup yields rule:
out id=0x2aaac40c3f00, priority=6, domain=nat-reverse, deny=false
        hits=972, user_data=0x2aaac40c5180, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
        src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
        dst ip/id=10.1.1.20, mask=255.255.255.255, port=0, tag=any, dscp=0x0
        input_ifc=any, output_ifc=inf_Data

Phase: 7
Type: USER-STATISTICS
Subtype: user-statistics
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
out id=0x2aaac306c690, priority=0, domain=user-statistics, deny=false
        hits=1023068, user_data=0x2aaac2ffd2c0, cs_id=0x0, reverse, flags=0x0, protocol=0
        src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
        input_ifc=any, output_ifc=inf_Data

Phase: 8
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0x2aaac099fcb0, priority=0, domain=nat-per-session, deny=false
        hits=1074535, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=6
        src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
        input_ifc=any, output_ifc=any

Phase: 9
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0x2aaac1317820, priority=0, domain=inspect-ip-options, deny=true
        hits=789889, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
        src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
        input_ifc=inf_Data, output_ifc=any

Phase: 10
Type: USER-STATISTICS
Subtype: user-statistics
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
out id=0x2aaac306d630, priority=0, domain=user-statistics, deny=false
        hits=3259, user_data=0x2aaac2ffd2c0, cs_id=0x0, reverse, flags=0x0, protocol=0
        src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
        input_ifc=any, output_ifc=inf_ISPB

Phase: 11
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 1019957, packet dispatched to next module
Module information for forward flow ...
snp_fp_tracer_drop
snp_fp_inspect_ip_options
snp_fp_tcp_normalizer
snp_fp_translate
snp_fp_adjacency
snp_fp_fragment
snp_ifc_stat

Module information for reverse flow ...
snp_fp_tracer_drop
snp_fp_inspect_ip_options
snp_fp_translate
snp_fp_tcp_normalizer
snp_fp_adjacency
snp_fp_fragment
snp_ifc_stat

Result:
input-interface: inf_ISPB
input-status: up
input-line-status: up
output-interface: inf_Data
output-status: up
output-line-status: up
Action: allow

Re: Inbound NAT issue with PBR on ASA

Spooster IT Services — Fri, 27 Oct 2017 11:27:11 GMT

Hi Diego,

Can you please make the following changes on the route map and test it?

route-map ALT-GATEWAY permit 10
match ip address acl_ISPB-PBR
no set ip default next-hop 2.2.2.1
set ip next-hop 2.2.2.1

If this still not working, then please take the captures of the traffic to find out the issue.
access-list test extended permit tcp any4 host 10.1.1.20 23
access-list test extended permit tcp host 10.1.1.20 23 any4
!
capture capi interface inf_Data access-list test
!

Re: Inbound NAT issue with PBR on ASA

tato386 — Sat, 28 Oct 2017 15:39:17 GMT

I adjusted the route-map as you suggested and it didn't make a difference. I also played around with moving the NAT to "before object NAT" and that didn't make a difference. I have attached the packet capture and it seems OK. It doesn't show the translated public IP but I am sure that it working because I have tested it using sites like ipchicken.com.

I appreciate your help very much but I am starting to think this is a bug.

Re: Inbound NAT issue with PBR on ASA

tato386 — Mon, 13 Nov 2017 13:45:37 GMT

According to TAC this is something that has worked in older versions but no longer available in newer ASA versions. I am pretty sure I have done this in the past so it does not sound totally off base. Not the answer I wanted to hear and very disappointing to have a useful feature removed.

Thanks to all who tried to help.

Diego

Re: Inbound NAT issue with PBR on ASA

dbogdan — Sat, 23 Mar 2019 13:31:45 GMT

Did you ever get this to work? I face the same issue when attempting to use a route-map. I have to add the route for the route-map to receive traffic from the outside, which kinda defeats the purpose. May as well just define a pile of routes instead.

Any advice would be appreciated!!

Re: Inbound NAT issue with PBR on ASA

tato386 — Mon, 25 Mar 2019 21:36:53 GMT

Sorry I was never able to get this to work but there have been several software updates to ASA since I was messing around with this. Have you tried using a recent build? Maybe they changed the behavior back?

Re: Inbound NAT issue with PBR on ASA

chris-goulder — Tue, 29 Oct 2019 09:04:38 GMT

The 'old way' of making this type of setup work was to include a floating static route for the second internet path

Referring to your config above include: -

route inf_ISPB 0.0.0.0 0.0.0.0 2.2.2.2 100

This adds internet route to the table, that not used for normal traffic due to the higher metric but completes the picture for PBR / NAT inbound traffic flows

Re: Inbound NAT issue with PBR on ASA

tato386 — Tue, 29 Oct 2019 21:48:40 GMT

At this time I don't have a setup where I can test this but I surely appreciate the info. It might come in handy at some point.

Thank you!

Re: Inbound NAT issue with PBR on ASA

Nishanna Gunasekera — Thu, 02 Apr 2020 02:54:33 GMT

I can verify that this works. Thank you Chris!

Re: Inbound NAT issue with PBR on ASA

dbogdan — Thu, 02 Apr 2020 16:59:35 GMT

Thanks for sorting this out. I will test at some future point.

Re: Inbound NAT issue with PBR on ASA

tato386 — Tue, 19 May 2020 21:33:09 GMT

I just ha the opportunity to try this on an ASA5515 running 9.12.3.9 and it worked like a charm.

Thank you CG!

Diego

Re: Inbound NAT issue with PBR on ASA

dbogdan — Wed, 20 May 2020 14:21:43 GMT

good to hear. As you know it worked outbound, but not inbound for me. Wht did you do differently?

Re: Inbound NAT issue with PBR on ASA

tato386 — Wed, 20 May 2020 15:37:22 GMT

Not exactly my case.

My outbound was always working via PBR. My issue was inbound and the floating static fixed it.

Re: Inbound NAT issue with PBR on ASA

dbogdan — Wed, 20 May 2020 16:20:21 GMT

understood. Yes it works outbound and if you initiate from inside. Outside-in didn't work for me at initiation. The packet just dropped. thanks!

Re: Inbound NAT issue with PBR on ASA

chris-goulder — Fri, 22 May 2020 07:13:25 GMT

The reason for this lays with the ASA connection table - once a tcp connection tuple is in the connection table that is used for applicable flows until the flow ends or the entry times out.

PBR is used to establish a flow using different routing to that in the general routing table, but once the flow is established and the entry in the connection table created, subsequent packets are handled just by the path identified in the connection table. PBR is not used once there is an entry in the connection table for a traffic flow

Also it is worth noting that typically when configuring PBR you are defining route policy for inside initiated traffic flows heading outside and that not have any bearing on outside initiated inbound traffic flows trying to head inside

So what is likely happening in your 'outside initiated traffic did not get through' case is along the lines of: -

1. tcp syn packet arrives at an outside i/f (but not the one holding the default route)

2. this allowed by the acl in place

3. this initiates the creation of an entry in the connection table for the tcp src|dst flow

4. syn packet forwarded to the inside host

5. inside host sends a syn/ack packet back to the source via the ASA

6. ASA checks its connection table and finds an entry for the tcp flow (entry added at 3. above)

7. As a connection table entry exists it is used and any PBR config does not come into play

8. ASA tries to forward the syn/ack reply packet out of the outside interface the flow came in on (connection table information)

9. this fails as there no applicable route in the ASA route table that would handle the flow on that i/f and the packet gets dropped

What you are doing by putting a floating static default route on your 'other' outside i/f is fixing the problem at step 9. in the above list. The ASA will see the floating default route on the interface and thus will be able to L3 forward it out of the outside interface the flow came in on

This behaviour can also crop up in asymmetric routing cases, whether PBR involved or not. The classic symptom being you can ping something but you cant get a tcp connection to establish. Depending on the scenario involved sometimes a floating static can help, other times you have to use a tcp state bypass policy (kludge) to fix it - but overall, unless of course you have no other option, the best solution is to try and fix/avoid the asymmetric routing in the first place as even if you find some config that will work it will likely break again later when something innocuous on the far side of your network changes

-Chris