Re: Packet tracer output- what ACL is dropping this L2L VPN traffic?

jmaxwellUSAF · ‎01-25-2023

Hello. Please view below packet tracer output...

ASA1120# packet-tracer input inside icmp (!! ip of my inside_in interface !!) 8 0 (!! ip of remote inside interface through L2L tunnel !!) $

Phase: 1
Type: UN-NAT
Subtype: static
Result: ALLOW
Elapsed time: 37888 ns
Config:
nat (inside,outside) source static MY-LAN-NETWORK-1 MY-LAN-NETWORK-1 destination static VENDOR1-LAN-NETWORK-1 VENDOR1-LAN-NETWORK-1
Additional Information:
NAT divert to egress interface outside
Untranslate (!! ip of my inside_in interface !!)/0 to (!! ip of remote inside interface through L2L tunnel !!)/0

Phase: 2
Type: ACCESS-LIST
Subtype:
Result: DROP
Elapsed time: 9344 ns
Config:
Implicit Rule
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f3e3daff3c0, priority=501, domain=permit, deny=true
hits=3, user_data=0x7, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=(!! ip of my inside_in interface !!), mask=255.255.255.255, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0, nsg_id=none
input_ifc=inside, output_ifc=any

Result:
input-interface: inside
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: drop
Time Taken: 47232 ns
Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x0000562a86cf3817 flow (NA)/NA

Below is the inside_in ACL, verified attached to the inside interface, AFTER packet tracer execution (hitcnt=0)...

access-list inside_in line 1 remark THIS ALLOWS TRAFFIC FROM SUBNET TO VENDOR1 SUBNET VIA L2L TUNNEL
access-list inside_in line 2 extended permit ip 172.16.1.0 255.255.255.0 172.16.8.0 255.255.255.0 (hitcnt=0) 0xedf8ec93

Possibly relevant (?)-- snippet from VPN config...
==========
8. Configure the ACL for VENDOR1_VPN_FILTER_ACL-1.
#access-list VENDOR1_VPN_FILTER_ACL-1 extended
#10 permit ip 172.16.8.0 255.255.255.0 172.16.1.0 255.255.255.0 (!! This seems to imply remote to local traffic. !!)

10. Configure Internal Group Policy & attributes.
#group-policy 2.2.2.2 internal
#group-policy 2.2.2.2 attributes
#vpn-filter VENDOR1_VPN_FILTER_ACL-1
#vpn-tunnel-protocol ikev2
#pfs enable
==========

Very relevant: Below data suggests/confirms that only 2 access lists are applied on this ASA,-- ACL inside_in on the inside interface, and ACL outside_in on the relevant outside1 interface.

ASA1120# sh run | inc access-group

access-group inside_in in interface inside
access-group outside_in in interface outside1

It seems the traffic is being dropped because it is first referencing a different ACL? What ACL would this be?

Thank you.

MHM Cisco World · ‎01-25-2023

Implicit Rule <<- meaning the default ACL which I think in your case deny ip any any in end of ACL of IN interface
OR
it can the default behave of ASA which not allow traffic by default toward the ASA intefrace.

for second point you must not use IP of any interface in our packet-tracer, use other IP and check again.

jmaxwellUSAF · ‎01-25-2023

1. "nat (inside,outside) source static MY-LAN-NETWORK-1 MY-LAN-NETWORK-1 destination static VENDOR1-LAN-NETWORK-1 VENDOR1-LAN-NETWORK-1"

...in above code, does "outside" refer to nameif "outside"?

2. "Implicit Rule <<- meaning the default ACL which I think in your case deny ip any any in end of ACL of IN interface
OR it can the default behave of ASA which not allow traffic by default toward the ASA interface." Which ACL is "the default ACL"?

3. "for second point you must not use IP of any interface in our packet-tracer, use other IP and check again." What is "second point" you refer to?

Thank you.

MHM Cisco World · ‎01-25-2023

2- it can the default behave of ASA which not allow traffic by default toward the ASA intefrace.

for second point you must not use IP of any interface in our packet-tracer, use other IP and check again.

jmaxwellUSAF · ‎01-25-2023

Progress...
1. I needed to use "outside1" instead of "outside.
2. with packet tracer I needed to use NETWORK address of inside interface, NOT INTERFACE address.
Behold new data...

ASA1120# packet-tracer input inside icmp 172.16.1.0 172.16.8.0 detailed

Phase: 1
Type: UN-NAT
Subtype: static
Result: ALLOW
Elapsed time: 35328 ns
Config:
nat (inside,outside1) source static MY-LAN-NETWORK-1 MY-LAN-NETWORK-1 destination static VENDOR1-LAN-NETWORK-1 VENDOR1-LAN-NETWORK-1
Additional Information:
NAT divert to egress interface outside1
Untranslate 172.16.8.0/0 to 172.16.8.0/0

Result:
input-interface: inside
input-status: up
input-line-status: up
output-interface: outside1
output-status: up
output-line-status: up
Action: drop
Time Taken: 35328 ns
Drop-reason: (sp-security-failed) Slowpath security checks failed, Drop-location: frame 0x0000562a86cfd95d flow (NA)/NA

======
ASA1120# clear asp drop
ASA1120# show asp drop

Frame drop:
Flow is denied by configured rule (acl-drop) 25
Flow denied due to resource limitation (unable-to-create-flow) 6

Flow drop:
Tunnel being brought up or torn down (tunnel-pending) 40
Need to start IKE negotiation (need-ike) 4

=====

I have no idea what ACL this refers to "Flow is denied by configured rule (acl-drop) 25" Please assist?

Thank you.

Rob Ingram · ‎01-25-2023

@jmaxwellUSAF I've amended your packet-tracer, please run the following and provide the full output.

packet-tracer input inside icmp 172.16.1.5 8 0 172.16.8.5 detailed

jmaxwellUSAF · ‎01-25-2023

ASA1120# packet-tracer input inside icmp 172.16.1.5 8 0 172.16.8.5 detailed

Phase: 1
Type: UN-NAT
Subtype: static
Result: ALLOW
Elapsed time: 33280 ns
Config:
nat (inside,outside1) source static MY-LAN-NETWORK-1 MY-LAN-NETWORK-1 destination static VENDOR1-LAN-NETWORK-1 VENDOR1-LAN-NETWORK-1
Additional Information:
NAT divert to egress interface outside1
Untranslate 172.16.8.5/0 to 172.16.8.5/0

Phase: 2
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Elapsed time: 7424 ns
Config:
access-group inside_in in interface inside
access-list inside_in remark THIS ALLOWS TRAFFIC FROM LOCAL SUBNET TO VENDOR1 SUBNET VIA L2L TUNNEL
access-list inside_in extended permit ip 172.16.1.0 255.255.255.0 172.16.8.0 255.255.255.0
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f3e940699e0, priority=13, domain=permit, deny=false
hits=0, user_data=0x7f3e1fe20f40, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=172.16.1.0, mask=255.255.255.0, port=0, tag=any
dst ip/id=172.16.8.0, mask=255.255.255.0, port=0, tag=any, dscp=0x0, nsg_id=none
input_ifc=inside, output_ifc=any

Phase: 3
Type: NAT
Subtype:
Result: ALLOW
Elapsed time: 7424 ns
Config:
nat (inside,outside1) source static MY-LAN-NETWORK-1 MY-LAN-NETWORK-1 destination static VENDOR1-LAN-NETWORK-1 VENDOR1-LAN-NETWORK-1
Additional Information:
Static translate 172.16.1.5/0 to 172.16.1.5/0
Forward Flow based lookup yields rule:
in id=0x7f3e3de30cd0, priority=6, domain=nat, deny=false
hits=0, user_data=0x7f3e3deb22c0, cs_id=0x0, flags=0x0, protocol=0
src ip/id=172.16.1.0, mask=255.255.255.0, port=0, tag=any
dst ip/id=172.16.8.0, mask=255.255.255.0, port=0, tag=any, dscp=0x0, nsg_id=none
input_ifc=inside, output_ifc=outside1

Phase: 4
Type: NAT
Subtype: per-session
Result: ALLOW
Elapsed time: 7424 ns
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f3e3b16ba0, priority=0, domain=nat-per-session, deny=true
hits=88360, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0, nsg_id=none
input_ifc=any, output_ifc=any

Phase: 5
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Elapsed time: 7424 ns
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f3e3daf5e40, priority=0, domain=inspect-ip-options, deny=true
hits=159901, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0, nsg_id=none
input_ifc=inside, output_ifc=any

Phase: 6
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Elapsed time: 24064 ns
Config:
class-map inspection_default
match default-inspection-traffic
policy-map global_policy
class inspection_default
inspect icmp
service-policy global_policy global
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f3e3fd3a8a0, priority=70, domain=inspect-icmp, deny=false
hits=3, user_data=0x7f3e3fd8120, cs_id=0x0, use_real_addr, flags=0x0, protocol=1
src ip/id=0.0.0.0, mask=0.0.0.0, icmp-type=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, icmp-code=0, tag=any, dscp=0x0, nsg_id=none
input_ifc=inside, output_ifc=any

Phase: 7
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Elapsed time: 3584 ns
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f3e3daf5650, priority=66, domain=inspect-icmp-error, deny=false
hits=16833, user_data=0x7f3e3daf52e0, cs_id=0x0, use_real_addr, flags=0x0, protocol=1
src ip/id=0.0.0.0, mask=0.0.0.0, icmp-type=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, icmp-code=0, tag=any, dscp=0x0, nsg_id=none
input_ifc=inside, output_ifc=any

Phase: 8
Type: VPN
Subtype: encrypt
Result: DROP
Elapsed time: 13312 ns
Config:
Additional Information:
Forward Flow based lookup yields rule:
out id=0x7f3e3f60c8a0, priority=70, domain=encrypt, deny=false
hits=6586, user_data=0x0, cs_id=0x7f3e3f61ba80, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0, nsg_id=none
input_ifc=any, output_ifc=outside1

Result:
input-interface: inside
input-status: up
input-line-status: up
output-interface: outside1
output-status: up
output-line-status: up
Action: drop
Time Taken: 10836 ns
Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x0000562a86cf8d95 flow (NA)/NA

Rob Ingram · ‎01-25-2023

@jmaxwellUSAF sorry as it's a VPN you need to run packet-tracer twice, the second output would have the correct result.

If that is the same refer to this link which describes the drop reason seems to indicates a problem with the crypto ACL.

jmaxwellUSAF · ‎01-25-2023

(!! command second time execution below !!)

ASA1120# packet-tracer input inside icmp 172.16.1.5 8 0 172.16.8.5 detail

Phase: 1
Type: UN-NAT
Subtype: static
Result: ALLOW
Elapsed time: 35328 ns
Config:
nat (inside,outside1) source static MY-LAN-NETWORK-1 MY-LAN-NETWORK-1 destination static VENDOR1-LAN-NETWORK-1 VENDOR1-LAN-NETWORK-1
Additional Information:
NAT divert to egress interface outside1
Untranslate 172.16.8.5/0 to 172.16.8.5/0

Phase: 2
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Elapsed time: 8192 ns
Config:
access-group inside_in in interface inside
access-list inside_in remark THIS ALLOWS TRAFFIC FROM LOCAL SUBNET TO VENDOR1 SUBNET VIA L2L TUNNEL
access-list inside_in extended permit ip 172.16.1.0 255.255.255.0 172.16.8.0 255.255.255.0
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f3e940699e0, priority=13, domain=permit, deny=false
hits=1, user_data=0x7f3e1fe20f40, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=172.16.1.0, mask=255.255.255.0, port=0, tag=any
dst ip/id=172.16.8.0, mask=255.255.255.0, port=0, tag=any, dscp=0x0, nsg_id=none
input_ifc=inside, output_ifc=any

Phase: 3
Type: NAT
Subtype:
Result: ALLOW
Elapsed time: 8192 ns
Config:
nat (inside,outside1) source static MY-LAN-NETWORK-1 MY-LAN-NETWORK-1 destination static VENDOR1-LAN-NETWORK-1 VENDOR1-LAN-NETWORK-1
Additional Information:
Static translate 172.16.1.5/0 to 172.16.1.5/0
Forward Flow based lookup yields rule:
in id=0x7f3e3de30cd0, priority=6, domain=nat, deny=false
hits=1, user_data=0x7f3e3deb22c0, cs_id=0x0, flags=0x0, protocol=0
src ip/id=172.16.1.0, mask=255.255.255.0, port=0, tag=any
dst ip/id=172.16.8.0, mask=255.255.255.0, port=0, tag=any, dscp=0x0, nsg_id=none
input_ifc=inside, output_ifc=outside1

Phase: 4
Type: NAT
Subtype: per-session
Result: ALLOW
Elapsed time: 8192 ns
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f3e3b16ba0, priority=0, domain=nat-per-session, deny=true
hits=840362, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0, nsg_id=none
input_ifc=any, output_ifc=any

Phase: 5
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Elapsed time: 8192 ns
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f3e3daf5e40, priority=0, domain=inspect-ip-options, deny=true
hits=159956, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0, nsg_id=none
input_ifc=inside, output_ifc=any

Phase: 6
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Elapsed time: 32256 ns
Config:
class-map inspection_default
match default-inspection-traffic
policy-map global_policy
class inspection_default
inspect icmp
service-policy global_policy global
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f3e3fd3a8a0, priority=70, domain=inspect-icmp, deny=false
hits=4, user_data=0x7f3e3fd8120, cs_id=0x0, use_real_addr, flags=0x0, protocol=1
src ip/id=0.0.0.0, mask=0.0.0.0, icmp-type=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, icmp-code=0, tag=any, dscp=0x0, nsg_id=none
input_ifc=inside, output_ifc=any

Phase: 7
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Elapsed time: 3584 ns
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f3e3daf5650, priority=66, domain=inspect-icmp-error, deny=false
hits=16848, user_data=0x7f3e3daf52e0, cs_id=0x0, use_real_addr, flags=0x0, protocol=1
src ip/id=0.0.0.0, mask=0.0.0.0, icmp-type=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, icmp-code=0, tag=any, dscp=0x0, nsg_id=none
input_ifc=inside, output_ifc=any

Phase: 8
Type: VPN
Subtype: encrypt
Result: DROP
Elapsed time: 9728 ns
Config:
Additional Information:
Forward Flow based lookup yields rule:
out id=0x7f3e3f60c8a0, priority=70, domain=encrypt, deny=false
hits=65486, user_data=0x0, cs_id=0x7f3e3f61ba80, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0, nsg_id=none
input_ifc=any, output_ifc=outside1

Result:
input-interface: inside
input-status: up
input-line-status: up
output-interface: outside1
output-status: up
output-line-status: up
Action: drop
Time Taken: 113664 ns
Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x0000562a86cf8d95 flow (NA)/NA

=======

FULL (obfuscated) LOCAL TUNNEL CONFIG BELOW...

crypto ikev2 enable outside1
crypto ikev2 policy 100
encryption aes-256
integrity sha-256
group 14
prf sha-256
lifetime seconds 86400
tunnel-group 2.2.2.2 type ipsec-l2l
tunnel group 2.2.2.2 ipsec-attributes
ikev2 local-authentication pre-shared-key !@#$%^&*(
ikev2 remote-authentication pre-shared-key !@#$%^&*(
crypto ipsec ikev2 ipsec-proposal VENDOR1-PROPOSAL-1
protocol esp encryption aes-256
protocol esp integrity sha-512
access-list outside_in line 1 remark >>ACL REFERENCED BY CRYPTO-MAP<<
access-list outside_in line 5 extended permit ip 172.16.1.0 255.255.255.0 172.16.8.0 255.255.255.0
crypto map VENDOR1-cryptomap-1 interface outside1
crypto map VENDOR1-cryptomap-1 1 match address outside_in
crypto map VENDOR1-cryptomap-1 1 set peer 2.2.2.2
crypto map VENDOR1-cryptomap-1 1 set ikev2 ipsec-proposal VENDOR1-PROPOSAL-1
object network MY-LAN-NETWORK-1
subnet 172.16.1.0 255.255.255.0
object network VENDOR1-LAN-NETWORK-1
subnet 172.16.8.0 255.255.255.0
nat (inside,outside1) source static MY-LAN-NETWORK-1 MY-LAN-NETWORK-1 destination static VENDOR1-LAN-NETWORK-1 VENDOR1-LAN-NETWORK-1
access-list VENDOR1_VPN_FILTER_ACL-1 extended permit ip 172.16.8.0 255.255.255.0 172.16.1.0 255.255.255.0
group-policy 2.2.2.2 internal
group-policy 2.2.2.2 attributes
vpn-filter value VENDOR1_VPN_FILTER_ACL-1
vpn-tunnel-protocol ikev2
pfs enable

jmaxwellUSAF · ‎01-26-2023

I think I'm figuring out the root cause + solution.

Please tell me...

#crypto map VENDOR1-cryptomap-1 interface outside1
#crypto map VENDOR1-cryptomap-1 1 match address outside_in
#crypto map VENDOR1-cryptomap-1 1 set peer 2.2.2.2
#crypto map VENDOR1-cryptomap-1 1 set ikev2 ipsec-proposal VENDOR1-PROPOSAL-1

Should ACL "outside_in" be attached to interface "outside1" in, or should ACL "outside_in" be attached to nothing ?

MHM Cisco World · ‎01-26-2023

I run lab and config VPN-filter to deny traffic between R2 and R3
then run Packet-tracer and this what I get,
it sure that VPN-filter appear in packet-tracer.

the phase VPN encrypt is Drop!
can I see the ACL you use for L2L VPN ??

jmaxwellUSAF · ‎01-26-2023

I was just now investigating this same thing...

>>>Below ACL is NOT being hit...
access-list VENDOR1_VPN_FILTER_ACL-1; 1 elements; name hash: 0x76180f95
access-list VENDOR1_VPN_FILTER_ACL-1 line 1 extended permit ip 172.16.8.0 255.255.255.0 172.16.1.0 255.255.255.0 (hitcnt=0) 0xf192d1f4
---

!! Below config is relevant to above data !!

!! 8. Configure the ACL for VENDOR1_VPN_FILTER_ACL-1. !!
#access-list VENDOR1_VPN_FILTER_ACL-1 extended
#10 permit ip 172.16.8.0 255.255.255.0 172.16.1.0 255.255.255.0

!! 9. Configure Internal Group Policy & attributes. !!
#group-policy 2.2.2.2 internal
#group-policy 2.2.2.2 attributes
#vpn-filter VENDOR1_VPN_FILTER_ACL-1
#vpn-tunnel-protocol ikev2
#pfs enable

---

!! below is full (obfuscated) VPN config. !!

crypto ikev2 enable outside1
crypto ikev2 policy 100
encryption aes-256
integrity sha-256
group 14
prf sha-256
lifetime seconds 86400
tunnel-group 2.2.2.2 type ipsec-l2l
tunnel group 2.2.2.2 ipsec-attributes
ikev2 local-authentication pre-shared-key 12345
ikev2 remote-authentication pre-shared-key 12345
crypto ipsec ikev2 ipsec-proposal VENDOR1-PROPOSAL-1
protocol esp encryption aes-256
protocol esp integrity sha-512
access-list outside_in line 1 remark >>ACL REFERENCED BY CRYPTO-MAP<<
access-list outside_in line 5 extended permit ip 172.16.1.0 255.255.255.0 172.16.8.0 255.255.255.0
crypto map VENDOR1-cryptomap-1 interface outside1
crypto map VENDOR1-cryptomap-1 1 match address outside_in
crypto map VENDOR1-cryptomap-1 1 set peer 2.2.2.2
crypto map VENDOR1-cryptomap-1 1 set ikev2 ipsec-proposal VENDOR1-PROPOSAL-1
object network MY-LAN-NETWORK-1
subnet 172.16.1.0 255.255.255.0
object network VENDOR1-LAN-NETWORK-1
subnet 172.16.8.0 255.255.255.0
nat (inside,outside1) source static MY-LAN-NETWORK-1 MY-LAN-NETWORK-1 destination static VENDOR1-LAN-NETWORK-1 VENDOR1-LAN-NETWORK-1
access-list VENDOR1_VPN_FILTER_ACL-1 extended permit ip 172.16.8.0 255.255.255.0 172.16.1.0 255.255.255.0
group-policy 2.2.2.2 internal
group-policy 2.2.2.2 attributes
vpn-filter value VENDOR1_VPN_FILTER_ACL-1
vpn-tunnel-protocol ikev2
pfs enable

MHM Cisco World · ‎01-26-2023

ASA1120# packet-tracer input inside icmp 172.16.8.5 8 0 172.16.1.5 detail <<- I think you mistake the subnet, you need to flapping subnet

jmaxwellUSAF · ‎01-26-2023

1. I updated/edited my last reply. Please view update.

2. Other message board helper suggested I use this ASA1120# packet-tracer input inside icmp 172.16.8.5 8 0 172.16.1.5 detail

Actually when i changed x.x.x.0 to x.x.x.5 , the command seemed to advance from previous errors.

I don't understand what you mean by "flapping subnet".

MHM Cisco World · ‎01-26-2023

can I see the advance packet-tracer ?