NAT Translation Issue on ASA5580

[montgomerywr]

ASA5580 running 8.3(1)

I think I have a NAT translation issue that I've been banging my head against for a couple of days now.  I have an internal server that needs to pass through our ASA firewall and then get's routed to a B2B partner's routers (local in our data center).  At that point the partner has a VPN established to send the traffic to it's final destination.

I have the following configuration: 

object network server
   host 192.168.165.162
object network server_nat
   host 199.67.6.170

nat (inside,b2b_dmz) source static server server_nat

access-list inside_in extended permit tcp host 192.168.165.162 gt 1023 host 9.9.9.9 range 1414 1416 (hitcnt=123)

When the traffic is initiated from the internal server, it hits the firewall rule and is allowed.  From my understanding, at that time, the internal server IP should be NAT'd to the external IP.  However, if I do a "show nat" command, I do not see any translate_hits.

1 (inside) to (b2b_dmz) source static server server_nat
    translate_hits = 0, untranslate_hits = 0

Any help would be greatly appreciated.

[joseoroz]

please provide the output of the following commands:

sh xlate | i 192.168.165.162

packet in inside tcp 192.168.165.162 1025 9.9.9.9 1414

[montgomerywr]

The output requested is below.  As you can see in the portion I've shown in red, the step hits the NAT translation rule, but then does not translate.  I have injected traffic from the b2b_dmz into the inside and that IS successfully translated.

show xlate | i 192.168.165.162
NAT from inside:192.168.165.162 to b2b_dmz:199.67.6.170

packet-tracer in inside tcp 192.168.165.162 5000 9.9.9.9 $

Phase: 1
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (any,any) source static any any
Additional Information:
NAT divert to egress interface b2b_dmz
Untranslate 9.9.9.9/1414 to 9.9.9.9/1414

Phase: 2
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group inside_in in interface inside
access-list inside_in extended permit tcp object-group plexuat gt 1023 object-group chase_uat range 1414 1416
object-group network plexuat
 network-object object PlexT1
 network-object object ST1A
 network-object object ST1B
object-group network chase_uat
 network-object host 9.9.9.9
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x3f05ac88, priority=13, domain=permit, deny=false
        hits=548, user_data=0x36d58f00, cs_id=0x0, use_real_addr, flags=0x0, protocol=6
        src ip/id=192.168.165.162, mask=255.255.255.255, port=0
        dst ip/id=9.9.9.9, mask=255.255.255.255, port=0,
        sport range<0> : 1024-65535 dscp=0x0
        input_ifc=inside, output_ifc=any

Phase: 3
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x3f43fee8, priority=0, domain=inspect-ip-options, deny=true
        hits=113255489, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
        src ip/id=0.0.0.0, mask=0.0.0.0, port=0
        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
        input_ifc=inside, output_ifc=any

Phase: 4
Type: FOVER
Subtype: standby-update
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x3f57eb88, priority=20, domain=lu, deny=false
        hits=34248324, user_data=0x0, cs_id=0x0, flags=0x0, protocol=6
        src ip/id=0.0.0.0, mask=0.0.0.0, port=0
        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
        input_ifc=inside, output_ifc=any

Phase: 5
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (inside,b2b_dmz) source static ST1A ST1A_nat
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x3f5f0f08, priority=6, domain=nat, deny=false
        hits=21, user_data=0x3f04f4b8, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
        src ip/id=192.168.165.162, mask=255.255.255.255, port=0
        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
        input_ifc=inside, output_ifc=b2b_dmz

Phase: 6
Type: NAT
Subtype: rpf-check
Result: ALLOW
Config:
nat (any,any) source static any any
Additional Information:
 Forward Flow based lookup yields rule:
 out id=0x3fe91f10, priority=6, domain=nat-reverse, deny=false
        hits=163205836, user_data=0x3fe91d20, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
        src ip/id=0.0.0.0, mask=0.0.0.0, port=0
        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
        input_ifc=any, output_ifc=any

Phase: 7
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
 Reverse Flow based lookup yields rule:
 in  id=0x3f3aa0e0, priority=0, domain=inspect-ip-options, deny=true
        hits=235905, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
        src ip/id=0.0.0.0, mask=0.0.0.0, port=0
        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
        input_ifc=b2b_dmz, output_ifc=any

Phase: 8
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 163340236, packet dispatched to next module
Module information for forward flow ...
snp_fp_tracer_drop
snp_fp_inspect_ip_options
snp_fp_tcp_normalizer
snp_fp_translate
snp_fp_adjacency
snp_fp_fragment
snp_ifc_stat

Module information for reverse flow ...
snp_fp_tracer_drop
snp_fp_inspect_ip_options
snp_fp_translate
snp_fp_tcp_normalizer
snp_fp_adjacency
snp_fp_fragment
snp_ifc_stat

Result:
input-interface: inside
input-status: up
input-line-status: up
output-interface: b2b_dmz
output-status: up
output-line-status: up
Action: allow

[WILLIAM STEGMAN]

Do you have any rules with a higher order # than

nat (inside,b2b_dmz) source static server server_nat? 

sh run nat will list your rules, look for a rule that is on top of the one you created for this traffic.  If so, try adding

nat (inside,b2b_dmz) 1 source static server server_nat

[montgomerywr]

Yes, when I originally entered the NAT statement it was after an any,any statement, and I would see the any/any NAT statement hit in the packet-tracer.  The NAT statement is now the first statement (using the command as you've shown) and it's still not working.

As you can see in the following step of the packet-tracer, it is hitting the correct NAT statement, but the NAT is not taking place:

Phase: 5
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (inside,b2b_dmz) source static server server_nat
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x3f5f0f08, priority=6, domain=nat, deny=false
        hits=21, user_data=0x3f04f4b8, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
        src ip/id=192.168.165.162, mask=255.255.255.255, port=0
        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
        input_ifc=inside, output_ifc=b2b_dmz

[WILLIAM STEGMAN]

How is it you're determining the NAT isn't taking place?  Based on your 2nd packet tracer, the NAT translation looks good.  Your initial packet tracker showed it getting NATed from ST1A to ST1A_nat. This recent packet tracer show server to server_nat, which based on your object config will nat 192.168.165.162 to  199.67.6.170 when that flow goes from the inside across your b2b_dmz interface.  If this is failing, it might be somewhere other than the NAT portion.  Have you run a packet capture downstream of your firewall to see if traffic from 199.67.6.170 is appearing?  Also, based on your packet tracer, I didn't see a route other than the default route being used.  You didn't include the routing section of the packet tracer.  If you're routing to public IPs across your b2b_dmz interface, make sure you add routes so that traffic doesn't use the default route out your outside interface.

[montgomerywr]

I'm determining that the NAT isn't happening from a multitude of sources:

First is the firewall log:

10 2014/04/29 09:22:35.057 CDT 172.17.65.20 Apr 29 2014 09:22:35 hqpublicdmzfw : %ASA-6-302013: Built outbound TCP connection 163956188 for b2b_dmz:169.111.118.79/1414 (169.111.118.79/1414) to inside:192.168.165.163/1829 (192.168.165.163/1829)

From my experience, when an address is NAT'd it appears in the firewall log where I've highlighted.

Second is the fact that translate_hits are not incrementing when I use the show nat command:

hqpublicdmzfw# show nat
Manual NAT Policies (Section 1)
1 (inside) to (b2b_dmz) source static server server_nat
    translate_hits = 0, untranslate_hits = 0

Lastly, I put an access-list & logged the results on a switch further down the path and have seen the traffic come from the un-NAT'd address.

The routing statement in the firewall is below & points to the L3 of the B2B distribution switch & is correctly routed with static routes from there to the partner's routers.

route b2b_dmz 169.111.118.79 255.255.255.255 172.17.65.59 1

Thank you for all your help.  It's really driving me crazy.  An interesting point to make is if I initiate the traffic from the b2b_dmz to inside, the public IP is translated to the private IP & the untranslate_hits increments.

[jumora]

Can you do me a favor an try to establish a connection from b2b_dmz to inside and print out the logs. You can do it with a packet tracer and check the real time log viewer on ASDM filtering the source address that you are using or do it via CLI if you know.

[Marius Gunnerud]

Have you tried configuring the nat under the auto-nat and not under manual nat?  I have experience issues in the past of a similar nature when trying to do something similar with manual nat.

object network server_nat
   host 199.67.6.170

object network server
   host 192.168.165.162
   nat (inside,b2b_dmz) static server_nat

--

Please remember to select a correct answer and rate

[montgomerywr]

Our firewall administrator has an any,any static NAT at the end of the NAT list.  My understanding is that those manual entries are Section 1 & the auto-nat statements are Section 2.  Since any translation would match that any/any in Section 1, the auto-NAT would never be considered in Section 2.

Is that correct?

[Marius Gunnerud]

That is correct.

As per the output of your show xlate command traffic is not being NATed correctly and it would seem that the traffic is hitting a different NAT statement.

Would you be able to post all your NAT statements so we help you troubleshoot further?

--

Please remember to select a correct answer and rate