Site to Site VPN FMC FTD (acl-drop) flow (tunnel-pending)

[olivares]

Hello everyone,

I have a drop flow issue in a Site to Site VPN between a Firepower Threat Defense 2130 (Version 6.6.4), managed via FMC, and a third-party VPN peer; and I need your help.

Please see attached topology for reference.

The VPN is established and working as expected as long as the traffic comes from my LAN and directs to the remote server with IP address 172.16.32.54 (this is not the actual IP address, all addresses used in these case are public IP addresses so, I'm not posting them here). The LAN is using the 172.16.0.0/16 prefix.

However, when I try to reach the server from another device outside of my LAN, from the subnet 172.16.79.0/24 (please note that it belongs to the same space 172.16.0.0/16, but as I wrote before, I'm using actual public IP addresses), the FTD 2130 drops the flow.

It's worth to note that the devices from the 172.16.79.0/24 can reach the router, the FTD 2130 and the LAN.

The reason of the drop, according to the Packet Tracer and a Capture w/Trace from the FMC is:

Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x000000aab2464694 flow (tunnel-pending)/snp_sp_action_cb:1748

When the packets come from the Inside interface of the FTD, they're allowed to enter into the tunnel but when they come from the Outside interface of the FTD, they're dropped. This is, packets coming from outside and directed to the tunnel are dropped.

The ACP explicitly allows this flow, from outside to the IP address of the remote server. As a matter of fact, the Packet Tracer doesn't show that there's any ACL blocking the traffic, the DROP comes in the phase type VPN, subtype IPSec-tunnel-flow.

Please see the output from the Packet Tracer:

FTD2130# packet-tracer input outside tcp 172.16.79.34 10001 172.16.32.54 10001 detailed

Phase: 1
Type: CAPTURE
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0xffa0dd89e0, priority=13, domain=capture, deny=false
hits=595526137, user_data=0xff8e926e80, cs_id=0x0, l3_type=0x0
src mac=0000.0000.0000, mask=0000.0000.0000
dst mac=0000.0000.0000, mask=0000.0000.0000
input_ifc=outside, output_ifc=any

Phase: 2
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
Forward Flow based lookup yields rule:
in id=0xff96644440, priority=1, domain=permit, deny=false
hits=31210449885, user_data=0x0, cs_id=0x0, l3_type=0x8
src mac=0000.0000.0000, mask=0000.0000.0000
dst mac=0000.0000.0000, mask=0100.0000.0000
input_ifc=outside, output_ifc=any

Phase: 3
Type: ROUTE-LOOKUP
Subtype: No ECMP load balancing
Result: ALLOW
Config:
Additional Information:
Destination is locally connected. No ECMP load balancing.
Found next-hop 172.16.32.54 using egress ifc outside(vrfid:0)

Phase: 4
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (any,any) source static NO_172.16.79.34_32 NO_172.16.79.34_32 destination static NO_172.16.32.54_32 NO_172.16.32.54_32
Additional Information:
NAT divert to egress interface outside(vrfid:0)
Untranslate 172.16.32.54/10001 to 172.16.32.54/10001

Phase: 5
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group CSM_FW_ACL_ global
access-list CSM_FW_ACL_ advanced permit tcp ifc outside object-group FMC_INLINE_src_rule_268443679 ifc outside object-group FMC_INLINE_dst_rule_268443679 eq 10001 rule-id 268443679
access-list CSM_FW_ACL_ remark rule-id 268443679: ACCESS POLICY: COMPANY-VPN - Default
access-list CSM_FW_ACL_ remark rule-id 268443679: L7 RULE: COMPANY_VPN_Project 2 City
object-group network FMC_INLINE_src_rule_268443679
description: Auto Generated by FMC from src of UnifiedNGFWRule# 2 (COMPANY-VPN/default)
network-object object NO_172.16.79.34_32
network-object object NO_172.16.32.54_32
object-group network FMC_INLINE_dst_rule_268443679
description: Auto Generated by FMC from dst of UnifiedNGFWRule# 2 (COMPANY-VPN/default)
network-object object NO_172.16.32.54_32
network-object object NO_172.16.79.34_32
Additional Information:
This packet will be sent to snort for additional processing where a verdict will be reached
Forward Flow based lookup yields rule:
in id=0xffbb707170, priority=12, domain=permit, deny=false
hits=6, user_data=0xff7bb94600, cs_id=0x0, use_real_addr, flags=0x0, protocol=6
src ip/id=172.16.79.34, mask=255.255.255.255, port=0, tag=any, ifc=outside(vrfid:0)
dst ip/id=172.16.32.54, mask=255.255.255.255, port=10001, tag=any, ifc=outside(vrfid:0), vlan=0, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 6
Type: CONN-SETTINGS
Subtype:
Result: ALLOW
Config:
class-map class-default
match any
policy-map global_policy
class class-default
set connection advanced-options UM_STATIC_TCP_MAP
service-policy global_policy global
Additional Information:
Forward Flow based lookup yields rule:
in id=0xff9136d9e0, priority=7, domain=conn-set, deny=false
hits=88019193, user_data=0xff9136b5c0, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=outside(vrfid:0), output_ifc=any

Phase: 7
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (any,any) source static NO_172.16.79.34_32 NO_172.16.79.34_32 destination static NO_172.16.32.54_32 NO_172.16.32.54_32
Additional Information:
Static translate 172.16.79.34/10001 to 172.16.79.34/10001
Forward Flow based lookup yields rule:
in id=0xff9b83f9c0, priority=6, domain=nat, deny=false
hits=62, user_data=0xff9b836b20, cs_id=0x0, flags=0x0, protocol=0
src ip/id=172.16.79.34, mask=255.255.255.255, port=0, tag=any
dst ip/id=172.16.32.54, mask=255.255.255.255, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 8
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0xffbab0c850, priority=0, domain=nat-per-session, deny=false
hits=70747137, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 9
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0xff96649ab0, priority=0, domain=inspect-ip-options, deny=true
hits=257342404, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=outside(vrfid:0), output_ifc=any

Phase: 10
Type: VPN
Subtype: ipsec-tunnel-flow
Result: DROP
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0xff90260cf0, priority=69, domain=ipsec-tunnel-flow, deny=false
hits=2381, user_data=0x0, cs_id=0xffa099c210, reverse, flags=0x0, protocol=0
src ip/id=172.16.0.0, mask=255.255.0.0, port=0, tag=any
dst ip/id=172.16.32.54, mask=255.255.255.255, port=0, tag=any, dscp=0x0
input_ifc=outside(vrfid:0), output_ifc=any

Result:
input-interface: outside(vrfid:0)
input-status: up
input-line-status: up
output-interface: outside(vrfid:0)
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x000000aab2464694 flow (tunnel-pending)/snp_sp_action_cb:1748

Interestingly, when I add the option bypass-checks to the Packet Tracer shown, the packets are not dropped, which suggests that there's actually a security check that is impeding the flow.

To my surprise, if in the Packet Tracer I change the incoming interface to Inside, the packets are allowed:

FTD2130# packet-tracer input inside tcp 172.16.79.43 10001 172.16.32.54 10001 detailed

Phase: 1
Type: ROUTE-LOOKUP
Subtype: No ECMP load balancing
Result: ALLOW
Config:
Additional Information:
Destination is locally connected. No ECMP load balancing.
Found next-hop 172.16.32.54 using egress ifc outside(vrfid:0)

Phase: 2
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (any,any) source static NO_172.16.79.43_32 NO_172.16.79.43_32 destination static NO_172.16.32.54_32 NO_172.16.32.54_32
Additional Information:
NAT divert to egress interface outside(vrfid:0)
Untranslate 172.16.32.54/10001 to 172.16.32.54/10001

Phase: 3
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group CSM_FW_ACL_ global
access-list CSM_FW_ACL_ advanced permit ip object-group FMC_INLINE_src_rule_268443679 object-group FMC_INLINE_dst_rule_268443679 rule-id 268443679
access-list CSM_FW_ACL_ remark rule-id 268443679: ACCESS POLICY: COMPANY-VPN - Default
access-list CSM_FW_ACL_ remark rule-id 268443679: L7 RULE: COMPANY_VPN_Project 2 City
object-group network FMC_INLINE_src_rule_268443679
description: Auto Generated by FMC from src of UnifiedNGFWRule# 2 (COMPANY-VPN/default)
network-object object NO_172.16.79.43_32
network-object object NO_172.16.32.54_32
object-group network FMC_INLINE_dst_rule_268443679
description: Auto Generated by FMC from dst of UnifiedNGFWRule# 2 (COMPANY-VPN/default)
network-object object NO_172.16.32.54_32
network-object object NO_172.16.79.43_32
Additional Information:
This packet will be sent to snort for additional processing where a verdict will be reached
Forward Flow based lookup yields rule:
in id=0xffa4e51bb0, priority=12, domain=permit, deny=false
hits=0, user_data=0xff7bfb1800, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=172.16.79.43, mask=255.255.255.255, port=0, tag=any, ifc=any
dst ip/id=172.16.32.54, mask=255.255.255.255, port=0, tag=any, ifc=any, vlan=0, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 4
Type: CONN-SETTINGS
Subtype:
Result: ALLOW
Config:
class-map class-default
match any
policy-map global_policy
class class-default
set connection advanced-options UM_STATIC_TCP_MAP
service-policy global_policy global
Additional Information:
Forward Flow based lookup yields rule:
in id=0xff9136ec70, priority=7, domain=conn-set, deny=false
hits=237941, user_data=0xff9136b5c0, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=inside(vrfid:0), output_ifc=any

Phase: 5
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (any,any) source static NO_172.16.79.43_32 NO_172.16.79.43_32 destination static NO_172.16.32.54_32 NO_172.16.32.54_32
Additional Information:
Static translate 172.16.79.43/10001 to 172.16.79.43/10001
Forward Flow based lookup yields rule:
in id=0xff9b83f9c0, priority=6, domain=nat, deny=false
hits=64, user_data=0xff9b836b20, cs_id=0x0, flags=0x0, protocol=0
src ip/id=172.16.79.43, mask=255.255.255.255, port=0, tag=any
dst ip/id=172.16.32.54, mask=255.255.255.255, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 6
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0xffbab0c850, priority=0, domain=nat-per-session, deny=false
hits=70748205, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 7
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0xffa46ae560, priority=0, domain=inspect-ip-options, deny=true
hits=35197803, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=inside(vrfid:0), output_ifc=any

Phase: 8
Type: VPN
Subtype: encrypt
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
out id=0xffb5f1f060, priority=70, domain=encrypt, deny=false
hits=558, user_data=0x4f817bcc, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=172.16.0.0, mask=255.255.0.0, port=0, tag=any
dst ip/id=172.16.32.54, mask=255.255.255.255, port=0, tag=any, dscp=0x0
input_ifc=any(vrfid:65535), output_ifc=outside

Phase: 9
Type: NAT
Subtype: rpf-check
Result: ALLOW
Config:
nat (any,any) source static NO_172.16.79.43_32 NO_172.16.79.43_32 destination static NO_172.16.32.54_32 NO_172.16.32.54_32
Additional Information:
Forward Flow based lookup yields rule:
out id=0xff5c8546e0, priority=6, domain=nat-reverse, deny=false
hits=4, user_data=0xff5c895b00, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=172.16.79.43, mask=255.255.255.255, port=0, tag=any
dst ip/id=172.16.32.54, mask=255.255.255.255, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 10
Type: VPN
Subtype: ipsec-tunnel-flow
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0xffb5060100, priority=69, domain=ipsec-tunnel-flow, deny=false
hits=558, user_data=0x4f81850c, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=172.16.32.54, mask=255.255.255.255, port=0, tag=any
dst ip/id=172.16.0.0, mask=255.255.0.0, port=0, tag=any, dscp=0x0
input_ifc=outside(vrfid:0), output_ifc=any

Phase: 11
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0xffbab0c850, priority=0, domain=nat-per-session, deny=false
hits=70748207, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 12
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0xff96649ab0, priority=0, domain=inspect-ip-options, deny=true
hits=257353880, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=outside(vrfid:0), output_ifc=any

Phase: 13
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 207672882, packet dispatched to next module
Module information for forward flow ...
snp_fp_inspect_ip_options
snp_fp_tcp_normalizer
snp_fp_snort
snp_fp_translate
snp_fp_tcp_normalizer
snp_fp_adjacency
snp_fp_encrypt
snp_fp_fragment
snp_fp_tracer_drop
snp_ifc_stat

Module information for reverse flow ...
snp_fp_inspect_ip_options
snp_fp_ipsec_tunnel_flow
snp_fp_tcp_normalizer
snp_fp_translate
snp_fp_snort
snp_fp_tcp_normalizer
snp_fp_adjacency
snp_fp_fragment
snp_fp_tracer_drop
snp_ifc_stat

Phase: 14
Type: EXTERNAL-INSPECT
Subtype:
Result: ALLOW
Config:
Additional Information:
Application: 'SNORT Inspect'

Phase: 15
Type: SNORT
Subtype:
Result: ALLOW
Config:
Additional Information:
Snort Trace:
Packet: TCP, SYN, seq 1790816069
Session: new snort session
AppID: service unknown (0), application unknown (0)
Firewall: allow rule, id 268443679, allow
Snort id 2, NAP id 1, IPS id 0, Verdict PASS
Snort Verdict: (pass-packet) allow this packet

Result:
input-interface: inside(vrfid:0)
input-status: up
input-line-status: up
output-interface: outside(vrfid:0)
output-status: up
output-line-status: up
Action: allow

It seems that there's indeed a problem when this specific traffic comes from the Outside interface.

Any ideas or suggestions?

Any help will be highly appreciated.

Thanks!

Oscar

[robaker2]

Oscar,

Packet-tracers from the WAN interface (source interface of the crypto map) will show this drop in that way since the packet-tracer used cannot generate an encrypted packet. Sourcing from the LAN side interface(s) as you did in the other one will be the best option and will show if any NAT or ACP (access control policy) rules might be causing issues with the traffic. Since the inside packet-tracer shows this traffic allowed into the VPN flow there could be an issue on the other side with allowing this traffic though that peer VPN device when started from that side. I would suggest having them setup packet captures to check that side when the traffic is started or if they can run something like the packet-tracer we have to test the traffic in the sending direction as you have.

[olivares]

Thank you, robaker2.

Regarding to your comment "...since the packet-tracer used cannot generate an encrypted packet...", it's my understanding that the packet being generated is not an encrypted one, all packets coming from 172.16.79.0/24 are not encrypted. In that way, they're the same as the ones arriving to the "inside" interface, but I can be wrong and in the end, it would be only a problem in the packet tracer utility, am I right?

The thing is that I've also captured live packets in the FMC and the results are the same, they're dropped with the same exact message that the packet tracer shows:

Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x000000aab2464694 flow (tunnel-pending)/snp_sp_action_cb:1748

This is what makes me think that something is wrong in the FTD, packets coming from the network 172.16.79.0/24 make their way to the Outside interface, they pass access-lists, route-lookup, and NAT but when they reach the VPN phase, they're dropped (and the tunnel is working, I have a constant ping from the LAN and the remote devices reply me, I also see those packets in captures, I verified the tunnel with the show crypto ikev2 sa and was even debugging the ikev2 sessions in order to se if there were any flap in the tunnel... everything is ok...).

It seems to me that it's not an ACP problem either, given that the packets pass the Phase 5 and the FTD explicitly shows the rules that allow the packets so, I don't get why the reason of the drop is listed as "acl-drop". However, the drop reason also points to "flow (tunnel-pending) as the drop location and I really don't know how to interpret this information.

I'm trying to get captures from the other side of the VPN as you kindly suggest, but is a very limited device and I reviewed the configuration, its traffic of interest coming from the tunnel is the network 172.16.0.0/16 and as I said before, packets coming from different networks find their way to the tunnel, their common characteristic is that they enter to the FTD from the Inside interface.

I'm also going to try to manually re-route the flow of packets coming from the remote location in a way that the FTD receives them from the inside interface and I'll post the results. My hunch is that once I have re-routed the packets, the FTD will allow them to enter the tunnel, as shown in the packet tracer's simulation and in the very same way it allows the other packets to enter the tunnel.

Thanks again!

[olivares]

I just wanted to post an update on this issue.

I haven't been able to actually solve the problem, packets coming to the Outside interface that should be redirected to the Site-to-Site VPN endpoint are simply dropped. I haven't found any documentation that could suggest why the FTD is dropping this traffic. Maybe it's something that the platform does and there's no way to change it or maybe it is a bug.

However, the idea of re-routing the packets through other devices on the network in a way that the FTD receives the flow on its Inside interface, turned out to be a fix for this situation (as it was expected, given that the Packet Tracer was predicting precisely, such behavior).

Now the traffic to and from the VPN is working ok with the network 172.16.79.0/24, the only difference is that the FTD sees such flow from its Inside interface, rather than the Outside one.

I'll schedule a maintenance window in order to upgrade the device to the most recent version of its OS and see if any of the updates fix the problem. If I find the origin of the problem or its solution, I'll post it here.

Thanks!

[Pavan Gundu]

nat (any,any) source static NO_172.16.79.43_32 NO_172.16.79.43_32 destination static NO_172.16.32.54_32 NO_172.16.32.54_32

From the no NAT rule you have configured, the first thing I would do is add the options route-lookup and no-proxy-arp