Packet being dropped by Snort in FTD despite hitting valid policy

[Shomi]

Hi all,

I'm fairly new to Cisco FTD so I'm wondering if anyone here can help me with an issue I'm currently having on my network. I can't seem to be able to reach a server via port 80/HTTP and I can see the traffic hitting my firewall rule "test-acp-rule" but Snort is dropping it for some reason (see packet tracer result below). I run a debug and issued the packet tracer again but can't seem to figure out what the issue is from the output of the debug (see the debug output further down). At this stage I don't want to bypass Snort so I'm wondering if anyone can help. Many thanks in advance.

packet-tracer input cloud-1 tcp 172.16.10.1 1234 192.168.100.10 80 detailed

Phase: 1
Type: UN-NAT
Subtype: static
Result: ALLOW
Elapsed time: 30726 ns
Config:
nat (inside-1,cloud-1) source static local-range-grp local-range-grp destination static CLOUD_PCS-GRP CLOUD_PCS-GRP description NONAT FOR VPNS to Outside
Additional Information:
NAT divert to egress interface inside-1(vrfid:0)
Untranslate 192.168.100.10/80 to 192.168.100.10/80

Phase: 2
Type: OBJECT_GROUP_SEARCH
Subtype:
Result: ALLOW
Elapsed time: 0 ns
Config:
Additional Information:
Source Object Group Match Count: 27
Destination Object Group Match Count: 10
Object Group Search: 270

Phase: 3
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Elapsed time: 341 ns
Config:
access-group LAB_FW_ACL_ global
access-list LAB_FW_ACL_ advanced permit tcp ifc cloud-1 object-group FMC_INLINE_src_rule_111111111 ifc inside-1 object-group FMC_INLINE_dst_rule_111111111 object-group HTTP rule-id 111111111
access-list LAB_FW_ACL_ remark rule-id 111111111: ACCESS POLICY: LAB-TEST-ACP - Default
access-list LAB_FW_ACL_ remark rule-id 111111111: L7 RULE: test-acp-rule
object-group network FMC_INLINE_src_rule_111111111(hitcnt=0)
network-object object lab-pc-1(hitcnt=0)
network-object object lab-pc-2(hitcnt=0)
network-object object lab-pc-3(hitcnt=0)
object-group network FMC_INLINE_dst_rule_111111111(hitcnt=175)
network-object object cloud-pc-1(hitcnt=95)
network-object object cloud-pc-2(hitcnt=80)
object-group service HTTP tcp
port-object eq www
Additional Information:
This packet will be sent to snort for additional processing where a verdict will be reached
Forward Flow based lookup yields rule:

Phase: 4
Type: CONN-SETTINGS
Subtype:
Result: ALLOW
Elapsed time: 341 ns
Config:
class-map class-default
match any
policy-map global_policy
class class-default
set connection advanced-options UM_STATIC_TCP_MAP
service-policy global_policy global
Additional Information:
Forward Flow based lookup yields rule:

Phase: 5
Type: NAT
Subtype:
Result: ALLOW
Elapsed time: 341 ns
Config:
nat (inside-1,cloud-1) source static local-range-grp local-range-grp destination static CLOUD_PCS-GRP CLOUD_PCS-GRP description NONAT FOR VPNS to Outside
Additional Information:
Static translate 172.16.10.1/1234 to 172.16.10.1/1234
Forward Flow based lookup yields rule:

Phase: 6
Type: NAT
Subtype: per-session
Result: ALLOW
Elapsed time: 341 ns
Config:
Additional Information:
Forward Flow based lookup yields rule:

Phase: 7
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Elapsed time: 341 ns
Config:
Additional Information:
Forward Flow based lookup yields rule:

Phase: 8
Type: FOVER
Subtype: standby-update
Result: ALLOW
Elapsed time: 29019 ns
Config:
Additional Information:
Forward Flow based lookup yields rule:

Phase: 9
Type: FLOW-EXPORT
Subtype:
Result: ALLOW
Elapsed time: 2845 ns
Config:
Additional Information:

Phase: 10
Type: NAT
Subtype: rpf-check
Result: ALLOW
Elapsed time: 11380 ns
Config:
nat (inside-1,cloud-1) source static local-range-grp local-range-grp destination static CLOUD_PCS-GRP CLOUD_PCS-GRP description NONAT FOR VPNS to Outside
Additional Information:
Forward Flow based lookup yields rule:

Phase: 11
Type: NAT
Subtype: per-session
Result: ALLOW
Elapsed time: 41537 ns
Config:
Additional Information:

Phase: 12
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Elapsed time: 1707 ns
Config:
Additional Information:
Reverse Flow based lookup yields rule:

Phase: 13
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Elapsed time: 48934 ns
Config:
Additional Information:
New flow created with id 442007830, packet dispatched to next module
Module information for forward flow ...
snp_fp_inspect_ip_options
snp_fp_tcp_normalizer
snp_fp_snort
snp_fp_translate
snp_fp_tcp_normalizer
snp_fp_adjacency
snp_fp_fragment
snp_fp_tracer_drop
snp_ifc_stat

Module information for reverse flow ...
snp_fp_inspect_ip_options
snp_fp_tcp_normalizer
snp_fp_translate
snp_fp_snort
snp_fp_tcp_normalizer
snp_fp_adjacency
snp_fp_fragment
snp_fp_tracer_drop
snp_ifc_stat

Phase: 14
Type: EXTERNAL-INSPECT
Subtype:
Result: ALLOW
Elapsed time: 13087 ns
Config:
Additional Information:
Application: 'SNORT Inspect'

Phase: 15
Type: SNORT
Subtype:
Result: DROP
Elapsed time: 1097601 ns
Config:
Additional Information:
Snort Trace:
Packet: TCP, SYN, seq 33333333
Session: deleting snort session, reason: stale and not cleaned
Session: deleted snort session using 0 bytes; protocol id:(5) : LWstate 0x0 LWFlags 0x0
Session: new snort session
AppID: service unknown (0), application unknown (0)
Firewall: block rule, id 222222222, drop
Snort: processed decoder alerts or actions queue, drop
Snort id 9, NAP id 2, IPS id 0, Verdict BLOCKFLOW, Blocked by Firewall
Snort Verdict: (black-list) black list this flow

Result:
input-interface: cloud-1(vrfid:0)
input-status: up
input-line-status: up
output-interface: inside-1(vrfid:0)
output-status: up
output-line-status: up
Action: drop
Time Taken: 1278541 ns
Drop-reason: (firewall) Blocked or blacklisted by the firewall preprocessor, Drop-location: frame 0x000000aaae0c1ff0 flow (NA)/NA

system support trace
Enable firewall-engine-debug too? [n]: y
Please specify an IP protocol: tcp
Please specify a client IP address: 172.16.10.1
Please specify a client port:
Please specify a server IP address: 192.168.100.10
Please specify a server port: 80
WARNING: Running trace with generic filters might cause CPU spike
Monitoring packet tracer and firewall debug messages

172.16.10.1-1234 - 192.168.100.10-80 6 AS 1-1 CID 0 Packet: TCP, SYN, seq 33333333
172.16.10.1-1234 - 192.168.100.10-80 6 AS 1-1 CID 0 Session: deleting snort session, reason: stale and not cleaned
172.16.10.1-1234 - 192.168.100.10-80 6 AS 1-1 CID 0 Session: deleted snort session using 0 bytes; protocol id:(5) : LWstate 0x0 LWFlags 0x0
172.16.10.1-1234 - 192.168.100.10-80 6 AS 1-1 CID 0 Session: new snort session
172.16.10.1-1234 - 192.168.100.10-80 6 AS 1-1 CID 0 AppID: service unknown (0), application unknown (0)
172.16.10.1-1234 > 192.168.100.10-80 6 AS 1-1 I 9 new firewall session
172.16.10.1-1234 > 192.168.100.10-80 6 AS 1-1 I 9 Starting AC with minimum 1670, 'test-acp-rule', and SrcZone first with zones 24 -> 28, geo 0 -> 0, vlan 0, source sgt type: 0, source sgt tag: 0, ISE sgt id: 0, dest sgt type: 0, ISE dest sgt tag: 0, svc 0, payload 0, client 0, misc 0, user 9999997, icmpType 0, icmpCode 0
172.16.10.1-1234 > 192.168.100.10-80 6 AS 1-1 I 9 no match rule order 1670, 'test-acp-rule', src network, FQDN and GEO
172.16.10.1-1234 > 192.168.100.10-80 6 AS 1-1 I 9 no match rule order 1671, 'inside_access_out_#123', IPProto
172.16.10.1-1234 > 192.168.100.10-80 6 AS 1-1 I 9 no match rule order 1673, 'Security_Servers_access_~#37-2', DstPort
172.16.10.1-1234 > 192.168.100.10-80 6 AS 1-1 I 9 no match rule order 1674, 'Any_Block>IS_TestBox', DstZone
172.16.10.1-1234 > 192.168.100.10-80 6 AS 1-1 I 9 no match rule order 1676, 'Any_Block>'Security_Servers', DstZone
172.16.10.1-1234 > 192.168.100.10-80 6 AS 1-1 I 9 no match rule order 1710, 'dmz_zone_out_#3-1003 #06', DstZone
172.16.10.1-1234 > 192.168.100.10-80 6 AS 1-1 I 9 no match rule order 1748, 'Deny_Any>DMZ1', DstZone
172.16.10.1-1234 > 192.168.100.10-80 6 AS 1-1 I 9 no match rule order 1750, 'Deny_Any>DMZ2', DstZone
172.16.10.1-1234 > 192.168.100.10-80 6 AS 1-1 I 9 no match rule order 1752, 'Deny_Any>Outside', DstZone
172.16.10.1-1234 > 192.168.100.10-80 6 AS 1-1 I 9 match rule order 1753, 'Default Action', action Block
172.16.10.1-1234 > 192.168.100.10-80 6 AS 1-1 I 9 Logging SOF with rule_id = 222222222 ruleAction = 4 ruleReason = 0
172.16.10.1-1234 > 192.168.100.10-80 6 AS 1-1 I 9 MidRecovery data sent for rule id: 222222222,rule_action:4, rev id:674019941, rule_match flag:0x2
172.16.10.1-1234 > 192.168.100.10-80 6 AS 1-1 I 9 HitCount data sent for rule id: 222222222,
172.16.10.1-1234 > 192.168.100.10-80 6 AS 1-1 I 9 deny action
172.16.10.1-1234 - 192.168.100.10-80 6 AS 1-1 CID 0 Firewall: block rule, 'Default Action', drop
172.16.10.1-1234 - 192.168.100.10-80 6 AS 1-1 CID 0 Snort: processed decoder alerts or actions queue, drop
172.16.10.1-1234 > 192.168.100.10-80 6 AS 1-1 I 9 Deleting session
172.16.10.1-1234 > 192.168.100.10-80 6 AS 1-1 I 9 deleting firewall session flags = 0x28003, fwFlags = 0x80, session->logFlags = 8002e7e008c0
172.16.10.1-1234 - 192.168.100.10-80 6 AS 1-1 CID 0 Snort id 9, NAP id 2, IPS id 0, Verdict BLOCKFLOW
172.16.10.1-1234 - 192.168.100.10-80 6 AS 1-1 CID 0 ===> Blocked by Firewall
Trace buffer and verdict reason are sent to DAQ

[MHM Cisco World]

As I get you have server in Inside and you want to access from cloud interface.

Points net to clarify 

1- un-NAT is remarking as VPN

2- what IP you use in packet tracer' 

packet-tracer input cloud-1 tcp 172.16.10.1 <this IP reachable via cloud > 1234 192.168.100.10 <this real or mapped IP> 80 detailed

MHM

[Shomi]

Hi @MHM Cisco World 
1. This NONAT rule is not required and will be removed because it's not a VPN traffic
2. packet-tracer input cloud-1 tcp 172.16.10.1 1234 192.168.100.10 80 detailed

Many thanks

Shomi

[tvotna]

What's going on guys? This is a second post with exactly same problem and symptoms? @atsukane and @Shomi which versions do you use?

We need to correlate the two outputs by rule-id to understand why Snort drops down to default ACP action instead of matching correct Snort rule. What was the real rule-id in the packet-tracer output? Also, can you find the rule shown by packet-tracer by rule-id in the /var/sf/detection_engines/<UUID>/ngfw.rules file?

access-group LAB_FW_ACL_ global
access-list LAB_FW_ACL_ advanced permit tcp ifc cloud-1 object-group FMC_INLINE_src_rule_111111111 ifc inside-1 object-group FMC_INLINE_dst_rule_111111111 object-group HTTP rule-id 111111111
access-list LAB_FW_ACL_ remark rule-id 111111111: ACCESS POLICY: LAB-TEST-ACP - Default
access-list LAB_FW_ACL_ remark rule-id 111111111: L7 RULE: test-acp-rule
object-group network FMC_INLINE_src_rule_111111111(hitcnt=0)
network-object object lab-pc-1(hitcnt=0)
network-object object lab-pc-2(hitcnt=0)
network-object object lab-pc-3(hitcnt=0)
object-group network FMC_INLINE_dst_rule_111111111(hitcnt=175)
network-object object cloud-pc-1(hitcnt=95)
network-object object cloud-pc-2(hitcnt=80)
object-group service HTTP tcp
port-object eq www

172.16.10.1-1234 > 192.168.100.10-80 6 AS 1-1 I 9 Starting AC with minimum 1670, 'test-acp-rule', and SrcZone first with zones 24 -> 28, geo 0 -> 0, vlan 0, source sgt type: 0, source sgt tag: 0, ISE sgt id: 0, dest sgt type: 0, ISE dest sgt tag: 0, svc 0, payload 0, client 0, misc 0, user 9999997, icmpType 0, icmpCode 0
172.16.10.1-1234 > 192.168.100.10-80 6 AS 1-1 I 9 no match rule order 1670, 'test-acp-rule', src network, FQDN and GEO
172.16.10.1-1234 > 192.168.100.10-80 6 AS 1-1 I 9 no match rule order 1671, 'inside_access_out_#123', IPProto
172.16.10.1-1234 > 192.168.100.10-80 6 AS 1-1 I 9 no match rule order 1673, 'Security_Servers_access_~#37-2', DstPort
172.16.10.1-1234 > 192.168.100.10-80 6 AS 1-1 I 9 no match rule order 1674, 'Any_Block>IS_TestBox', DstZone
172.16.10.1-1234 > 192.168.100.10-80 6 AS 1-1 I 9 no match rule order 1676, 'Any_Block>'Security_Servers', DstZone
172.16.10.1-1234 > 192.168.100.10-80 6 AS 1-1 I 9 no match rule order 1710, 'dmz_zone_out_#3-1003 #06', DstZone
172.16.10.1-1234 > 192.168.100.10-80 6 AS 1-1 I 9 no match rule order 1748, 'Deny_Any>DMZ1', DstZone
172.16.10.1-1234 > 192.168.100.10-80 6 AS 1-1 I 9 no match rule order 1750, 'Deny_Any>DMZ2', DstZone
172.16.10.1-1234 > 192.168.100.10-80 6 AS 1-1 I 9 no match rule order 1752, 'Deny_Any>Outside', DstZone
172.16.10.1-1234 > 192.168.100.10-80 6 AS 1-1 I 9 match rule order 1753, 'Default Action', action Block

[tvotna · Accepted solution]

I now see that SYN packet generated by packet-tracer hits "test-acp-rule", but this rule is not hit in Snort. FQDN is used by both of you, @atsukane and @Shomi , not sure about GEO. Get rid of FQDN for test?

172.16.10.1-1234 > 192.168.100.10-80 6 AS 1-1 I 9 Starting AC with minimum 1670, 'test-acp-rule', and SrcZone first with zones 24 -> 28, geo 0 -> 0, vlan 0, source sgt type: 0, source sgt tag: 0, ISE sgt id: 0, dest sgt type: 0, ISE dest sgt tag: 0, svc 0, payload 0, client 0, misc 0, user 9999997, icmpType 0, icmpCode 0
172.16.10.1-1234 > 192.168.100.10-80 6 AS 1-1 I 9 no match rule order 1670, 'test-acp-rule', src network, FQDN and GEO

[Shomi]

Hi @tvotna thanks for your input. This is a bit of a weird one. We changed the FQDN in the policy to it's real IP address and that works but the problem is we need to keep the FQDN because the IP addresses are potentially changeable. Any thoughts why the IP works but fqdn fails? show dns host shows the correct IP address for the hostname.

Many thanks

Shomi

[tvotna · Accepted solution]

Perhaps a bug. Do conditions of this one match your ACP rule?

FTD: Firewall is not matching ACP rules with multiple FQDN objects configured
CSCwh64784

Symptom: Traffic does not match an ACP rule which has more than one FQDN object specified as source or destination networks. Instead, another rule below will be matched.

Conditions: 1) An ACP rule is configured with more than one FQDN object as a matching condition. 2) There are no IP-based objects in source or destination networks.

Workaround: For FQDN-based rules specify only one FQDN object. If needed, create a separate rule for every FQDN that should be matched.

Further Problem Description: When only FQDN objects are used as a network condition, the rule will be ignored by the firewall, even if a DNS server is reachable and FQDNs are successfully resolved.

[Shomi]

Hi @tvotna thank you for your input. We changed the FQDNs to their corresponding IPs and the service is working so it appears it's a bug as detailed in CSCwh64784 which you have shared. Thank you for your help.

[Marius Gunnerud]

Are you sure that the Firewall is able to resolve the FQDN?  Are the DNS servers that you configured during setup reachable from the FTD management interface?  And, are the DNS servers you configured on the FTD the ones that are able to resolve these FQDNs?

These, and probably a couple more, are the questions you need to ask yourself if defining the IP works but the FQDN does not.

[Shomi]

Hi @Marius Gunnerud. Thank you for your input but it's not a firewall DNS or FQDN issue in this case, rather it appears it's a bug as detailed in the article shared above by tvotna.