Solved: cannot ping due to nat config

lcaruso · ‎03-10-2011

Hi,

I've been tasked with cleaning up some old client configs. One site has an issue that it cannot ping anything outside even though all of the necessary acls are in place. Packet Tracer gave me this, but I don't know how to interpret what is wrong. Any help is appreciated.

ciscoasa# packet-tracer input outside icmp 4.2.2.2 0 8 192.168.3.3 detail

Phase: 1
Type: ACCESS-LIST
Subtype: 
Result: ALLOW
Config:
Implicit Rule
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0xd7dfbc80, priority=1, domain=permit, deny=false
        hits=23089, user_data=0x0, cs_id=0x0, l3_type=0x8
        src mac=0000.0000.0000, mask=0000.0000.0000
        dst mac=0000.0000.0000, mask=0100.0000.0000
Phase: 2
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in   192.168.3.0     255.255.255.0   inside
Phase: 3
Type: ACCESS-LIST
Subtype: log
Result: ALLOW 
Config:
access-group outside_access_in in interface outside
access-list outside_access_in extended permit icmp any any 
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0xd8a0d220, priority=12, domain=permit, deny=false
        hits=2, user_data=0xd613bd50, cs_id=0x0, flags=0x0, protocol=1
        src ip=0.0.0.0, mask=0.0.0.0, port=0
        dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
Phase: 4
Type: IP-OPTIONS
Subtype: 
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0xd7dfe3b8, priority=0, domain=inspect-ip-options, deny=true
        hits=1148, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
        src ip=0.0.0.0, mask=0.0.0.0, port=0
        dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
Phase: 5
Type: INSPECT 
Subtype: np-inspect
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0xd7dfe030, priority=66, domain=inspect-icmp-error, deny=false
        hits=78, user_data=0xd7dfdf18, cs_id=0x0, use_real_addr, flags=0x0, protocol=1
        src ip=0.0.0.0, mask=0.0.0.0, port=0
        dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
Phase: 6
Type: VPN
Subtype: ipsec-tunnel-flow
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0xd86ee000, priority=12, domain=ipsec-tunnel-flow, deny=true
        hits=84, user_data=0x0, cs_id=0x0, flags=0x0, protocol=0
        src ip=0.0.0.0, mask=0.0.0.0, port=0
        dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
Phase: 7
Type: HOST-LIMIT
Subtype: 
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0xd7daf808, priority=0, domain=host-limit, deny=false
        hits=182, user_data=0x0, cs_id=0x0, flags=0x0, protocol=0
        src ip=0.0.0.0, mask=0.0.0.0, port=0
        dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
Phase: 8
Type: NAT
Subtype: rpf-check
Result: DROP
Config:
nat (inside) 1 0.0.0.0 0.0.0.0
  match ip inside any outside any
    dynamic translation to pool 1 (12.x.y.138 [Interface PAT])
    translate_hits = 30, untranslate_hits = 1
Additional Information:
 Forward Flow based lookup yields rule:
 out id=0xd7e28440, priority=1, domain=nat-reverse, deny=false
        hits=182, user_data=0xd7e282e8, cs_id=0x0, flags=0x0, protocol=0
        src ip=0.0.0.0, mask=0.0.0.0, port=0
        dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
Result:
input-interface: outside
input-status: up
input-line-status: up
output-interface: inside
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule

mirober2 · ‎03-11-2011

Hi Larry,

The packet tracer that you posted in your first message is not quite correct. The packet will not ingress the outside interface with a destination IP of 192.168.3.3, so the results you see there are not valid. Also, there is no ICMP packet with a type of 0 and a code of 8, so you'll see some confusing packet tracer output sometimes.

You are trying to get the hosts on the inside to send a ping to something on the Internet, correct? If so, please post this packet-tracer output instead:

packet-tracer in inside icmp 192.168.3.3 8 0 4.2.2.2

Also, it would be good to setup a quick capture on the inside and outside interfaces to see what part of the ping is failing (i.e. echo request or echo reply).

-Mike

View solution in original post

mirober2 · ‎03-11-2011

Hi Larry,

Based on that output it looks like something upstream might be filtering the ICMP packets. Are there any other hops that these pings go through that you have access to so you can check for ACLs? If not, does your ISP allow ICMP to pass?

The captures you setup should confirm this for you as well. I'm guessing you'll see the echo requests go out the outside interface, but no response ever comes back. It would be good to get pings working from the ASA itself before you worry about getting traffic from your internal hosts out through the ASA.

-Mike

View solution in original post

mirober2 · ‎03-11-2011

Hi Larry,

Based on those captures, something between your ASA and 4.2.2.2 is blocking ICMP packets. You'll notice nothing ever comes back to your outside interface, which means either the echo request is not reaching the server (most likely), or something is blocking the echo reply packet as it is heading back toward your ASA.

If you don't manage any of the next hop devices in the path, I would suggest contacting your ISP and asking if this is expected behavior for their network.

Hope that helps.

-Mike

View solution in original post

Jennifer Halim · ‎03-10-2011

Can you please share your translation statements that is configured on the firewall?

sh run nat

sh run global

sh run static

Also if there is any reference to ACL from the above output, please also share. Thanks.

lcaruso · ‎03-10-2011


ciscoasa# sh run nat
nat (inside) 0 access-list inside_nat0_outbound
nat (inside) 1 0.0.0.0 0.0.0.0
ciscoasa# sh run static
static (inside,outside) 12.x.y.139 192.168.3.4 netmask 255.255.255.255 
ciscoasa# sh run global
global (outside) 1 interface

lcaruso · ‎03-10-2011

think I'll open a TAC case on this one

mirober2 · ‎03-11-2011

Hi Larry,

The packet tracer that you posted in your first message is not quite correct. The packet will not ingress the outside interface with a destination IP of 192.168.3.3, so the results you see there are not valid. Also, there is no ICMP packet with a type of 0 and a code of 8, so you'll see some confusing packet tracer output sometimes.

You are trying to get the hosts on the inside to send a ping to something on the Internet, correct? If so, please post this packet-tracer output instead:

packet-tracer in inside icmp 192.168.3.3 8 0 4.2.2.2

Also, it would be good to setup a quick capture on the inside and outside interfaces to see what part of the ping is failing (i.e. echo request or echo reply).

-Mike

lcaruso · ‎03-11-2011

Hi Mike,

Nice Catch!

I'll setup the captures and post. The packet tracer works, but this doesn't

ciscoasa# ping 4.2.2.2

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 4.2.2.2, timeout is 2 seconds:

?????

Success rate is 0 percent (0/5)

ciscoasa# ping inside 4.2.2.2

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 4.2.2.2, timeout is 2 seconds:

?????

Success rate is 0 percent (0/5)

ciscoasa(config)# packet-tracer in inside icmp 192.168.3.3 8 0 4.2.2.2

Phase: 1

Type: ACCESS-LIST

Subtype:

Result: ALLOW

Config:

Implicit Rule

Additional Information:

MAC Access list

Phase: 2

Type: ROUTE-LOOKUP

Subtype: input

Result: ALLOW

Config:

Additional Information:

in 0.0.0.0 0.0.0.0 outside

Phase: 3

Type: ROUTE-LOOKUP

Subtype: input

Result: ALLOW

Config:

Additional Information:

in 192.168.3.0 255.255.255.0 inside

Phase: 4

Type: ACCESS-LIST

Subtype: log

Result: ALLOW

Config:

access-group inside_in in interface inside

access-list inside_in extended permit ip any any log

Additional Information:

Phase: 5

Type: IP-OPTIONS

Subtype:

Result: ALLOW

Config:

Additional Information:

Phase: 6

Type: INSPECT

Subtype: np-inspect

Result: ALLOW

Config:

Additional Information:

Phase: 7

Type: NAT

Subtype:

Result: ALLOW

Config:

nat (inside) 1 0.0.0.0 0.0.0.0

match ip inside any outside any

dynamic translation to pool 1 (12.x.y.138 [Interface PAT])

translate_hits = 17993, untranslate_hits = 950

Additional Information:

Dynamic translate 192.168.3.3/0 to 12.x.y.138/61849 using netmask 255.255.255.255

Phase: 8

Type: NAT

Subtype: host-limits

Result: ALLOW

Config:

nat (inside) 1 0.0.0.0 0.0.0.0

match ip inside any inside any

dynamic translation to pool 1 (No matching global)

translate_hits = 0, untranslate_hits = 0

Additional Information:

Phase: 9

Type: HOST-LIMIT

Subtype:

Result: ALLOW

Config:

Additional Information:

Phase: 10

Type: FLOW-CREATION

Subtype:

Result: ALLOW

Config:

Additional Information:

New flow created with id 47720, packet dispatched to next module

Result:

input-interface: inside

input-status: up

input-line-status: up

output-interface: outside

output-status: up

output-line-status: up

Action: allow

mirober2 · ‎03-11-2011

Hi Larry,

Based on that output it looks like something upstream might be filtering the ICMP packets. Are there any other hops that these pings go through that you have access to so you can check for ACLs? If not, does your ISP allow ICMP to pass?

The captures you setup should confirm this for you as well. I'm guessing you'll see the echo requests go out the outside interface, but no response ever comes back. It would be good to get pings working from the ASA itself before you worry about getting traffic from your internal hosts out through the ASA.

-Mike

lcaruso · ‎03-11-2011

Here are the captures. Right now I'm restricted to pinging from the ASA

ciscoasa# ping inside 4.2.2.2

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 4.2.2.2, timeout is 2 seconds:

?????

Success rate is 0 percent (0/5)

ciscoasa# sh cap capout

5 packets captured

1: 02:59:11.223682 802.1Q vlan#2 P0 192.168.3.253 > 4.2.2.2: icmp: echo request

2: 02:59:13.219684 802.1Q vlan#2 P0 192.168.3.253 > 4.2.2.2: icmp: echo request

3: 02:59:15.219669 802.1Q vlan#2 P0 192.168.3.253 > 4.2.2.2: icmp: echo request

4: 02:59:17.219654 802.1Q vlan#2 P0 192.168.3.253 > 4.2.2.2: icmp: echo request

5: 02:59:19.219623 802.1Q vlan#2 P0 192.168.3.253 > 4.2.2.2: icmp: echo request

5 packets shown

ciscoasa# sh cap capin

0 packet captured

0 packet shown

ciscoasa# sh access-list capin-acl

access-list capin-acl; 2 elements; name hash: 0x3fc459e9

access-list capin-acl line 1 extended permit ip host 4.2.2.2 any log debugging interval 300 (hitcnt=0) 0xc451331e

access-list capin-acl line 2 extended permit ip any host 4.2.2.2 log debugging interval 300 (hitcnt=0) 0x2464646b

ciscoasa# sh access-list capout-acl

access-list capout-acl; 2 elements; name hash: 0x1862133e

access-list capout-acl line 1 extended permit ip host 4.2.2.2 any log debugging interval 300 (hitcnt=0) 0x22af6efe

access-list capout-acl line 2 extended permit ip any host 4.2.2.2 log debugging interval 300 (hitcnt=5) 0x1a23f569

ciscoasa#

ciscoasa# sh cap

capture asp type asp-drop all [Buffer Full - 524155 bytes]

capture capin type raw-data access-list capin-acl interface inside [Capturing - 0 bytes]

capture capout type raw-data access-list capout-acl interface outside [Capturing - 670 bytes]

mirober2 · ‎03-11-2011

Hi Larry,

Based on those captures, something between your ASA and 4.2.2.2 is blocking ICMP packets. You'll notice nothing ever comes back to your outside interface, which means either the echo request is not reaching the server (most likely), or something is blocking the echo reply packet as it is heading back toward your ASA.

If you don't manage any of the next hop devices in the path, I would suggest contacting your ISP and asking if this is expected behavior for their network.

Hope that helps.

-Mike

lcaruso · ‎03-11-2011

Hi Mike,

Many thanks for your input--it truly helped !

Our client will be calling his ISP to resolve this issue.

Best Regards,

Larry

lcaruso · ‎03-12-2011

Well, it turns out it is not the ISP...

If you have any icmp permit statements on your ASA, that stops the ability to ping out on the Internet from the ASA

you can't even ping the directly connected isp router if there is an icmp permit statement present

it seems icmp permit mucks with the traces as well

i proved this time and again simply by taking those statements out and pinging successfully and putting them back in and having pings fail

my thanks go out to TAC Costa Rica for uncovering the true cause of this issue.