Unable to ping outside interfaces (from outside)

lucas.jereska · ‎11-13-2018

I'm in the process of setting up AnyConnect VPN to our ASA (5525) and in the process have realized that I'm unable to ping either of my outside interfaces from the internet. A basic rundown of my setup:

Two outside interfaces - outside and outside_centurylink (each routes to a separate internet circuit). I'm using policy based routing to utilize both internet circuits simultaneously. This works without any issues.

I have the following icmp settings configured:

icmp permit any outside
icmp permit any unreachable outside
icmp permit any time-exceeded outside
icmp permit any echo-reply outside
icmp permit any outside_centurylink
icmp permit any unreachable outside_centurylink
icmp permit any time-exceeded outside_centurylink
icmp permit any echo-reply outside_centurylink

Packet-tracer indicates that the icmp traffic should be allowed:

Phase: 1
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop 68.xxx.xxx.xxx using egress ifc  identity

Phase: 2
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x2aaaca0ba6b0, priority=121, domain=permit, deny=false
        hits=1019, user_data=0x0, cs_id=0x0, flags=0x0, protocol=1
        src ip/id=0.0.0.0, mask=0.0.0.0, icmp-type=0, tag=any
        dst ip/id=0.0.0.0, mask=0.0.0.0, icmp-code=0, tag=any, dscp=0x0
        input_ifc=outside_centurylink, output_ifc=identity

Phase: 3
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x2aaac71cc1a0, priority=0, domain=nat-per-session, deny=true
        hits=11134872, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=0
        src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
        input_ifc=any, output_ifc=any

Phase: 4
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x2aaac7cbab50, priority=0, domain=inspect-ip-options, deny=true
        hits=1941717, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
        src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
        input_ifc=outside_centurylink, output_ifc=any

Phase: 5
Type: CLUSTER-REDIRECT
Subtype: cluster-redirect
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x2aaac7cb32d0, priority=208, domain=cluster-redirect, deny=false
        hits=3890, user_data=0x0, cs_id=0x0, flags=0x0, protocol=0
        src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
        input_ifc=outside_centurylink, output_ifc=identity

Phase: 6
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x2aaac7cb90e0, priority=66, domain=inspect-icmp, deny=false
        hits=3864, user_data=0x2aaac7cb8650, cs_id=0x0, use_real_addr, flags=0x0, protocol=1
        src ip/id=0.0.0.0, mask=0.0.0.0, icmp-type=0, tag=any
        dst ip/id=0.0.0.0, mask=0.0.0.0, icmp-code=0, tag=any, dscp=0x0
        input_ifc=outside_centurylink, output_ifc=identity

Phase: 7
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x2aaac7cba360, priority=66, domain=inspect-icmp-error, deny=false
        hits=3864, user_data=0x2aaac7cb98d0, cs_id=0x0, use_real_addr, flags=0x0, protocol=1
        src ip/id=0.0.0.0, mask=0.0.0.0, icmp-type=0, tag=any
        dst ip/id=0.0.0.0, mask=0.0.0.0, icmp-code=0, tag=any, dscp=0x0
        input_ifc=outside_centurylink, output_ifc=any

Phase: 8
Type: VPN
Subtype: ipsec-tunnel-flow
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x2aaaca0c2f90, priority=13, domain=ipsec-tunnel-flow, deny=true
        hits=1079, user_data=0x0, cs_id=0x0, flags=0x0, protocol=0
        src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
        dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
        input_ifc=outside_centurylink, output_ifc=any

Phase: 9
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 28363008, packet dispatched to next module
Module information for forward flow ...
snp_fp_tracer_drop
snp_fp_inspect_ip_options
snp_fp_inspect_icmp
snp_fp_adjacency
snp_fp_fragment
snp_ifc_stat

Module information for reverse flow ...

Result:
input-interface: outside_centurylink
input-status: up
input-line-status: up
output-interface: NP Identity Ifc
Action: allow

The only odd thing I can spot is from ASDM's packet tracer:

Anyone have any thoughts on this? I'm pulling my hair out here and feel like there's got to be something obvious that I'm just overlooking.

AndreaTornaghi · ‎11-13-2018

For AnyConnect session are you using outside interface or outside_centurylink interface?

Did you try to open a session on port 443 on both interface from internet?

Could you post your config about AnyConnect?

lucas.jereska · ‎11-13-2018

My AnyConnect config is not yet completed - once I realized the interfaces were unreachable from outside I stopped to tackle that.

outside_centurylink is where AnyConnect will be enabled initially, which is why I was running the packet trace on that specific interface.

AndreaTornaghi · ‎11-13-2018

So both interfaces are unreachable by ping from outside.

Do you have configured a default route on outside interface?

lucas.jereska · ‎11-13-2018

This prompted to go back and recheck the routing and I've learned a couple of things:

1. Ping to the outside interface is working. It's only outside_centurylink that's not.

2. The route that I set up for outside_centurylink is not in the routing table.

The gateway of last resort is the gateway for the outside interface - I'm guessing that traffic coming in from outside_centurylink is falling back to the gateway of last resort when trying to respond and failing since it's in a different subnet. A cursory google search is pointing me to ECMP as a possible solution, so I may need to dig deeper there.

AndreaTornaghi · ‎11-13-2018

Ping is failing because ASA is answering with the wrong interface.

You should track default route on outside and use the second outside interface as backup.