ASA 8.4(3) - applying NAT breaks my site to site tunnel - "routing failed"

[Paul Cobley]

So I am preconfiguring a couple of 5510's before shipping to site. I have my Site to Site VPN tunnel up fine and can ping from internal subnets between sites. However as soon as I configure NAT on my outside interface my pings die. I checked out a very comprehensive config guide posted by TAC and I think the answer is to configure twice-NAT which I believe I have done. Still I get no packets down the tunnel.

One clue I have found is that I get this message logged when the NAT is applied & affecting the routing "ASA-6-110003: Routing failed to locate next hop for ICMP from Outside:10.56.8.4/512 to Internal:172.16.60.253/0"

Output of sh run object / sh run object-group / sh run nat / show nat from the two ASA's :-

SITE 1

========= sh run object
object network obj_any
subnet 0.0.0.0 0.0.0.0
object network BH-Asterisk
host x.x.x.x
description BG Hill Asterisk
object network BH-Exchange
host x.x.x.x
description BG Hill Exchange Server
object network DH-AV
subnet 10.56.20.0 255.255.255.0
description DH AV
object network DH-Asterisk
host x.x.x.x
description DH Asterisk
object network DH-Exchange
host 10.56.1.253
description DH Exchange
object network DH-Guests
subnet 10.56.8.0 255.255.255.0
description DH guests
object network DH-MOI
subnet 10.56.24.0 255.255.255.0
description DH MOI
object network DH-Phones
subnet 10.56.16.0 255.255.255.0
description DH Phones
object network DH-Security
subnet 10.56.32.0 255.255.255.0
description DH security
object network DH-Internal
subnet 10.56.1.0 255.255.255.0
description DH internal
object network BH-Internal
subnet 10.60.1.0 255.255.255.0
description BH internal
object network BH-Phones
subnet 10.60.16.0 255.255.255.0
description BH Phones
object network BH-Security
subnet 10.60.32.0 255.255.255.0
description BH Security
object network BH-AV
subnet 10.60.20.0 255.255.255.0
description BH AV
object network BH-Guests
subnet 10.60.8.0 255.255.255.0
description BH Guests
object network BH-ASA
host 1.1.1.1
object network DH-ASA
host 1.1.1.2
object network BH-RAS
subnet 10.60.99.0 255.255.255.0
object network DH-RAS
subnet 10.56.99.0 255.255.255.0
object network NETWORK_OBJ_10.56.99.0_26
subnet 10.56.99.0 255.255.255.192
object network BH-UC560
host 172.16.60.253
object network DH-UC560
host 172.16.56.253

========RJ5510-DOHA# sh run object-group
object-group network BGHill
description Subnets in BGHill
network-object object BH-Internal
network-object object BH-Phones
network-object object BH-AV
network-object object BH-Security
network-object object BH-Guests
network-object object BH-RAS
network-object object BH-UC560
object-group network DH
description Subnets in DH
network-object object DH-AV
network-object object DH-Guests
network-object object DH-MOI
network-object object DH-Phones
network-object object DH-Security
network-object object DH-Internal
network-object object DH-RAS
network-object object DH-UC560

=======RJ5510-DH# sh run nat
nat (AV,Outside) source static DH DH destination static BGHill BGHill
nat (Guest,Outside) source static DH DH destination static BGHill BGHill
nat (Internal,Outside) source static DH DH destination static BGHill BGHill
nat (Phones,Outside) source static DH DH destination static BGHill BGHill
nat (Security,Outside) source static DH DH destination static BGHill BGHill
nat (MOI,Outside) source static DH DH destination static BGHill BGHill
!
object network DH-AV
nat (AV,Outside) dynamic interface
object network DH-Exchange
nat (Internal,Outside) static x.x.x.x
object network DH-Guests
nat (Guest,Outside) dynamic interface
object network DH-MOI
nat (MOI,Outside) dynamic interface
object network DH-Phones
nat (Phones,Outside) dynamic interface
object network DH-Security
nat (Security,Outside) dynamic interface
object network DH-Internal
nat (Internal,Outside) dynamic interface

========RJ5510-DH# show nat
Manual NAT Policies (Section 1)
1 (AV) to (Outside) source static DH DH   destination static BGHill BGHill
    translate_hits = 0, untranslate_hits = 386
2 (Guest) to (Outside) source static DH DH   destination static BGHill BGHill
    translate_hits = 180, untranslate_hits = 0
3 (Internal) to (Outside) source static DH DH   destination static BGHill BGHill
    translate_hits = 0, untranslate_hits = 0
4 (Phones) to (Outside) source static DH DH   destination static BGHill BGHill
    translate_hits = 0, untranslate_hits = 0
5 (Security) to (Outside) source static DH DH   destination static BGHill BGHill
    translate_hits = 0, untranslate_hits = 0
6 (MOI) to (Outside) source static DH DH   destination static BGHill BGHill
    translate_hits = 0, untranslate_hits = 0

Auto NAT Policies (Section 2)
1 (Internal) to (Outside) source static DH-Exchange x.x.x.x
    translate_hits = 0, untranslate_hits = 0
2 (Internal) to (Outside) source dynamic DH-Internal interface 
    translate_hits = 0, untranslate_hits = 0
3 (Guest) to (Outside) source dynamic DH-Guests interface 
    translate_hits = 2, untranslate_hits = 0
4 (Phones) to (Outside) source dynamic DH-Phones interface 
    translate_hits = 0, untranslate_hits = 0
5 (AV) to (Outside) source dynamic DH-AV interface 
    translate_hits = 0, untranslate_hits = 0
6 (MOI) to (Outside) source dynamic DH-MOI interface 
    translate_hits = 0, untranslate_hits = 0
7 (Security) to (Outside) source dynamic DH-Security interface 
    translate_hits = 0, untranslate_hits = 0

SITE 2 :-

================# sh run object
object network obj_any
subnet 0.0.0.0 0.0.0.0
object network BH-Asterisk
host x.x.x.x
description BH Hill Asterisk
object network BH-Exchange
host 10.60.1.253
description BH Hill Exchange Server
object network DH-AV
subnet 10.56.20.0 255.255.255.0
description DH AV
object network DH-Asterisk
host x.x.x.x
description DH Asterisk
object network DH-Exchange
host x.x.x.x
description DH Exchange
object network DH-Guests
subnet 10.56.8.0 255.255.255.0
description DH guests
object network DH-MOI
subnet 10.56.24.0 255.255.255.0
description DH MOI
object network DH-Phones
subnet 10.56.16.0 255.255.255.0
description DH Phones
object network DH-Security
subnet 10.56.32.0 255.255.255.0
description DH security
object network DH-Internal
subnet 10.56.1.0 255.255.255.0
description DH internal
object network BH-Internal
subnet 10.60.1.0 255.255.255.0
description BH internal
object network BH-Phones
subnet 10.60.16.0 255.255.255.0
description BH Phones
object network BH-Security
subnet 10.60.32.0 255.255.255.0
description BH Security
object network BH-AV
subnet 10.60.20.0 255.255.255.0
description BH AV
object network BH-Guests
subnet 10.60.8.0 255.255.255.0
description BH Guests
object network BH-ASA
host 1.1.1.1
object network DH-ASA
host 1.1.1.2
object network NETWORK_OBJ_10.60.99.0_26
subnet 10.60.99.0 255.255.255.192
object network BH-RAS
subnet 10.60.99.0 255.255.255.0
object network DH-RAS
subnet 10.56.99.0 255.255.255.0
object network BH-UC560
host 172.16.60.253
object network DH-UC560
host 172.16.56.253

================# sh run object-group
object-group network BHHill
description Subnets in BH Hill
network-object object BH-Internal
network-object object BH-Phones
network-object object BH-AV
network-object object BH-Security
network-object object BH-Guests
network-object object BH-RAS
network-object object BH-UC560
object-group network DH
description Subnets in DH
network-object object DH-AV
network-object object DH-Guests
network-object object DH-MOI
network-object object DH-Phones
network-object object DH-Security
network-object object DH-Internal
network-object object DH-RAS
network-object object DH-UC560

================# sh run nat
nat (Internal,Outside) source static BHHill BHHill destination static DH DH
nat (AV,Outside) source static BHHill BHHill destination static DH DH
nat (Guest,Outside) source static BHHill BHHill destination static DH DH
nat (Phones,Outside) source static BHHill BHHill destination static DH DH
nat (Security,Outside) source static BHHill BHHill destination static DH DH
!
object network BH-Exchange
nat (Internal,Outside) static x.x.x.x
object network BH-Internal
nat (Internal,Outside) dynamic interface
object network BH-Phones
nat (Phones,Outside) dynamic interface
object network BH-Security
nat (Security,Outside) dynamic interface
object network BH-AV
nat (AV,Outside) dynamic interface
object network BH-Guests
nat (Guest,Outside) dynamic interface

================# sh nat
Manual NAT Policies (Section 1)
1 (Internal) to (Outside) source static BHHill BHHill   destination static DH DH
    translate_hits = 421, untranslate_hits = 178
2 (AV) to (Outside) source static BHHill BHHill   destination static DH DH
    translate_hits = 0, untranslate_hits = 0
3 (Guest) to (Outside) source static BHHill BHHill   destination static DH DH
    translate_hits = 0, untranslate_hits = 0
4 (Phones) to (Outside) source static BHHill BHHill   destination static DH DH
    translate_hits = 0, untranslate_hits = 0
5 (Security) to (Outside) source static BHHill BHHill   destination static DH DH
    translate_hits = 0, untranslate_hits = 0

Auto NAT Policies (Section 2)
1 (Internal) to (Outside) source static BH-Exchange x.x.x.x 
    translate_hits = 0, untranslate_hits = 0
2 (Internal) to (Outside) source dynamic BH-Internal interface 
    translate_hits = 0, untranslate_hits = 0
3 (Guest) to (Outside) source dynamic BH-Guests interface 
    translate_hits = 0, untranslate_hits = 0
4 (Phones) to (Outside) source dynamic BH-Phones interface 
    translate_hits = 0, untranslate_hits = 0
5 (AV) to (Outside) source dynamic BH-AV interface 
    translate_hits = 0, untranslate_hits = 0
6 (Security) to (Outside) source dynamic BH-Security interface 
    translate_hits = 0, untranslate_hits = 0
RJ5510-BH#

I must admit I'm scoobied with this one but hopefully someone will spot the catch ?

Thanks

[Julio Carvajal]

Hello Paul,

Can you be more specific on the objects.. The problem is that you have several object networks (that belong to different interfaces so when the packets comes back to the ASA the ASA will not know where to send the packets)

Do you understand what I mean?

Please change that and let me know!

Regards,

Julio

DO rate all the helpful posts

[Paul Cobley]

Thanks for the prompt response Julio.

I'm not sure I can reduce the number of network objects because each of these represents a subnet on the main inside network. So The ASA is actually the default gateway for about 6 x VLANs in each site and the inside interface is a DOT1Q trunk. So each network object relates to a subnet in each site.

How can I change the network objects under these circumstances ?

I only have a single default route on the ASA outside interface which routes all traffic to the ISP.

[Julio Carvajal]

Hello Paul

You will Need to create a object group for the networks behind the inside interface of the ASA

Then create one for the subnets behind another interface of the ASA

and Keep going like that untill you cover all the subnets and their respective interfaces.

Then create the Twice NAT need it for that.. It would be the same thing but in this case the ASA wil have a beter knowledge of all your network.

Regards,

Julio

DO rate all the helpful posts

[Paul Cobley]

Hi Julio

Ok happy to this although I'm a little unclear of exactly how to acheive this. Is there an example of what you mean on CCO ?

I appreciate your assistance - I did a fair bit of research of this error but never saw anything that suggested the network objects could be the problem.

Many thanks

Paul.

[Julio Carvajal · Accepted solution]

In fact the problem is with the NAT due to the fact you are using the same object on different NAT statements that are attached to different interfaces.

The ASA might get crazy with that..

I got to leave right now.

As soon as I get back I will explain this a little further.

Regards,

Julio

Do rate all the helpful posts

[Paul Cobley]

Hi Julio,

No need - you have been very helpful and i think I see what you mean. When I get in the lab later today I will change the nat statements on both sites to be more specific. I already have network objects which specify the internal subnets indidvidually so I will reuse these existing network objects but just change the NAT statements.

So, where I currently have :-

nat (AV,Outside) source static BHHill BHHill destination static DH DH

I will change this to be morer specific :-

nat (AV,Outside) 1 source static BH-AV BH-AV destination static DH DH where the source network object only identifes the single subnet that is behind the AV subinterface.

I will do this & post the results - success or failure.

Many thanks for your assistance so far.

[Julio Carvajal]

Hello Paul,

Be mi guest,

Regards,

Julio

[eoghan_murtagh]

Hi Paul and Julio,

I've experienced a similar problem to this and Julio's answer partly pointed me in the correct direction.

Julio, you are 100% correct in saying that

"

In fact the problem is with the NAT due to the fact you are using the same object on different NAT statements that are attached to different interfaces.

The ASA might get crazy with that.."

you definitely need to separate out the traffic but there is also a "bug" introduced in ASA 8.4.2 and a new requirement for the "route-lookup" keyword at the end of a double NAT statement see below

CSCtr16184 Bug Details

To-the-box traffic fails from hosts over vpn after upgrade to 8.4.2.

Symptom: After upgrading the ASA to 8.4.2, all management traffic to-the-box(including icmp/telnet/ssh/ASDM) from hosts over the VPN (L2L or Remote ACcess VPN) may fail when destined to the management-access interface IP address.Conditions: 1. Issue is observed if ASA is on 8.4.2. Not observed on 8.4.1. 2. Users directly connected to the internal interfaces face no issues with icmp/telnet/ssh/asdm to their respective interfaces.Workaround: The problem can be traced to a Manual NAT statement that overlaps with the management-access interface IP address. The NAT statement must have both the source and destination fields. Adding the "route-lookup" keyword at the end of the NAT statement resolves the issue.Ex: ASA's Management-Access Interface IP address is 192.168.1.1.! Overlapping NAT statement: nat (inside,outside) source static obj-192.168.1.0 obj-192.168.1.0 destination static obj-vpn obj-vpn! New Statement: nat (inside,outside) source static obj-192.168.1.0 obj-192.168.1.0 destination static obj-vpn obj-vpn route-lookup

So in short from your orginal config in relation to the AV interface for example I would do the following:

no nat (AV,Outside) source static BHHill BHHill destination static DH DH

nat (AV,Outside) source static BH-AV BH-AV destination static DH DH route-lookup

Apply similar for each nat statement/interface.

[Julio Carvajal]

Hello Sr,

Thanks for the information

Regards,