Dear team,
I'm experiencing some odd behavior on my FTD devices.
I currently have a couple VPNs setup with different partners and all for the same purpose: getting the traffic to get to a VIP on my F5 device and, futhermore, get to the app servers.
All the partners until today were working just fine and the most recent VPN partner is showing some odd behavior.
The VPN establishes phase 1 and 2 (we're using IKEv1) and, whenever he sends traffic, I do see packets incrementing on the decaps but never get replied (encaps).
After some capture and troubleshooting, it seems that the firewall is dropping the packet due to an ACL statement (per the logs). The problem is that the ACL do exist to allow the traffic to pass through.
Any thoughts on what might be causing that kind of behavior?
Just as a traffic flow overview:
Source: 10.6.20.4
Destination: 45.233.232.230
Service: 8721
Obs.: The packet-tracer output displays an IPSec flow drop.
Here are a couple logs:
> show capture
capture capasp type asp-drop all buffer 1000000 circular-buffer [Stopped - 20660 bytes]
match ip host 45.233.232.230 host 10.6.20.4
> show capture capasp detail
Capture manually stopped
225 packets captured
....
224: 10:20:53.157569 7070.8bae.e0e0 4c77.6ddb.084e 0x8100 Length: 78
802.1Q vlan#610 P0 10.6.20.4.32027 > 45.233.232.230.8721: S [tcp sum ok] 1783849477:1783849477(0) win 5840 <mss 1398,sackOK,timestamp 768178106 0,nop,wscale 7> (DF) [tos 0x10] (ttl 64, id 48068) Drop-reason: (acl-drop) Flow is denied by configured rule
225: 10:21:17.159156 7070.8bae.e0e0 4c77.6ddb.084e 0x8100 Length: 78
802.1Q vlan#610 P0 10.6.20.4.32027 > 45.233.232.230.8721: S [tcp sum ok] 1783849477:1783849477(0) win 5840 <mss 1398,sackOK,timestamp 768202106 0,nop,wscale 7> (DF) [tos 0x10] (ttl 64, id 48069) Drop-reason: (acl-drop) Flow is denied by configured rule
> packet-tracer input inet tcp 10.6.20.4 23145 45.233.232.230 8721 detailed
Phase: 1
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop 45.233.232.230 using egress ifc p-inet
Phase: 2
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group CSM_FW_ACL_ global
access-list CSM_FW_ACL_ advanced permit tcp ifc inet object-group G_EXT-Partners-Auth_All ifc p-inet object I_ADQ_TSP-Switch-Auth-VIP_45.233.232.230_32 object-group DXC_Auth rule-id 268438680
access-list CSM_FW_ACL_ remark rule-id 268438680: ACCESS POLICY: FTD-INET-SP - Mandatory
access-list CSM_FW_ACL_ remark rule-id 268438680: L7 RULE: Partners to DXC Auth (VIP)
object-group network G_EXT-Partners-Auth_All
.....
network-object object H_CHK_UOL-Server-Gateway-01_10.6.20.4_32
object-group service DXC_Auth tcp
group-object DXC_Auth_Crypt
group-object DXC_Auth_NCrypt
Additional Information:
This packet will be sent to snort for additional processing where a verdict will be reached
Forward Flow based lookup yields rule:
in id=0x7f89bf8a04e0, priority=12, domain=permit, deny=false
hits=1207, user_data=0x2aaad0db0180, cs_id=0x0, use_real_addr, flags=0x0, protocol=6
src ip/id=10.6.20.4, mask=255.255.255.255, port=0, tag=any, ifc=inet
dst ip/id=45.233.232.230, mask=255.255.255.255, port=8721, tag=any, ifc=p-inet, vlan=0, dscp=0x0
input_ifc=any, output_ifc=any
Phase: 3
Type: CONN-SETTINGS
Subtype:
Result: ALLOW
Config:
class-map class-default
match any
policy-map global_policy
class class-default
set connection advanced-options UM_STATIC_TCP_MAP
service-policy global_policy global
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f8a105b91a0, priority=7, domain=conn-set, deny=false
hits=5553166, user_data=0x7f8a105af570, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=inet, output_ifc=any
Phase: 4
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f8a1817b9f0, priority=0, domain=nat-per-session, deny=false
hits=24319946, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=any
Phase: 5
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f8a18b58130, priority=0, domain=inspect-ip-options, deny=true
hits=23037415, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=inet, output_ifc=any
Phase: 6
Type: VPN
Subtype: ipsec-tunnel-flow
Result: DROP
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f8a1a95b2d0, priority=70, domain=ipsec-tunnel-flow, deny=false
hits=1297, user_data=0x0, cs_id=0x7f89cfea3c30, reverse, flags=0x0, protocol=0
src ip/id=10.6.20.4, mask=255.255.255.255, port=0, tag=any
dst ip/id=45.233.232.230, mask=255.255.255.255, port=0, tag=any, dscp=0x0
input_ifc=inet, output_ifc=any
Result:
input-interface: inet
input-status: up
input-line-status: up
output-interface: p-inet
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule
> show crypto ipsec sa peer x.x.x.113
peer address: x.x.x.113
Crypto map tag: CSM_inet_map, seq num: 25, local addr: y.y.y.251
access-list CSM_IPSEC_ACL_18 extended permit ip host 45.233.232.230 host 10.6.20.4
local ident (addr/mask/prot/port): (45.233.232.230/255.255.255.255/0/0)
remote ident (addr/mask/prot/port): (10.6.20.4/255.255.255.255/0/0)
current_peer: x.x.x.113
#pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
#pkts decaps: 56, #pkts decrypt: 56, #pkts verify: 56
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts comp failed: 0, #pkts decomp failed: 0
#pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
#PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
#TFC rcvd: 0, #TFC sent: 0
#Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
#send errors: 0, #recv errors: 0
local crypto endpt.: y.y.y.251/0, remote crypto endpt.: x.x.x.113/0
path mtu 1500, ipsec overhead 74(44), media mtu 1500
PMTU time remaining (sec): 0, DF policy: copy-df
ICMP error validation: disabled, TFC packets: disabled
current outbound spi: BBF068CB
current inbound spi : CCC4102D
inbound esp sas:
spi: 0xCCC4102D (3435401261)
transform: esp-aes-256 esp-sha-hmac no compression
in use settings ={L2L, Tunnel, PFS Group 5, IKEv1, }
slot: 0, conn_id: 258781184, crypto-map: CSM_inet_map
sa timing: remaining key lifetime (kB/sec): (3914996/26879)
IV size: 16 bytes
replay detection support: Y
Anti replay bitmap:
0x01FFFFFF 0xFFFFFFFF
outbound esp sas:
spi: 0xBBF068CB (3153094859)
transform: esp-aes-256 esp-sha-hmac no compression
in use settings ={L2L, Tunnel, PFS Group 5, IKEv1, }
slot: 0, conn_id: 258781184, crypto-map: CSM_inet_map
sa timing: remaining key lifetime (kB/sec): (3915000/26879)
IV size: 16 bytes
replay detection support: Y
Anti replay bitmap:
0x00000000 0x00000001
