FTD2120 - asp drop - no-adjacency

[AigarsK]

Hi All,

I am seeking advice as I am seeing issue with traffic traversing over S2S VPN. I have attached topology of the remote site.

Remote site is running FTD2120 in Active/Standby version 7.3.1.1 and HQ running FTD2120 in Active/Standby version 7.0.6.

Site to Site VPN is up between sites and I see encaps and decaps of traffic traversing the VPN and I have implemented NAT and ACL's to allow the traffic between sites on specific zone etc.

However I am not able to ping from HQ to the Remote site device further down in SDA network and I see packets incrementing for asp drop for "No valid adjacency (no-adjacency)" I did packet capture for this asp drop reason and indeed I am able to see my ping messages there.

to give you an example, from HQ, I am pinging with source 10.70.0.101 to destination 10.90.65.199.

On Remote Site's FTD, I have security group XYZ_AV which has 10.90.2.33(34)/29 configured on firewall and has 10.90.2.35/29 on Fusion Switch 1 and 10.90.2.36/29 on Fusion Switch 2. There is static route on FTD in place with SLA for subnet 10.90.64.0/20 with next hop set 10.90.2.35:

S 10.90.64.0 255.255.240.0 [1/0] via 10.90.2.35, xyz_av

If I check route from FTD for 10.90.65.199 is as followed:

> show route 10.90.65.199

Routing entry for 10.90.64.0 255.255.240.0
Known via "static", distance 1, metric 0
Routing Descriptor Blocks:
* 10.90.2.35, via xyz_av
Route metric is 0, traffic share count is 1

If I was to check arp table, there is entry for 10.90.2.35, I am also able to Ping this 10.90.2.35 from firewall itself.

I have also run "system support firewall-engine-debug" command and can see that there is match for allow rule for connection coming from HQ to this site and vice versa.

Any advise on how to fix this issue would be much appreciated.

[MHM Cisco World]

share 
show arp 
show arp stati

capture of asp drop 
and packet-tracer you use 
MHM

[AigarsK]

Here you go:

> show arp
xyz_net_mgmt 10.90.2.3 242a.040d.48de 14028
xyz_net_mgmt 10.90.2.5 0000.0c07.ac01 14028
xyz_net_mgmt 10.90.2.4 748f.c28d.695e 14028
failover-link 172.21.255.2 cc79.d7c3.c5bc 526
xyz_prod 10.90.2.20 748f.c28d.6975 11356
xyz_prod 10.90.2.19 242a.040d.48f5 12107
xyz_iot 10.90.2.28 748f.c28d.6941 12467
xyz_iot 10.90.2.29 0000.0c07.ac04 14028
xyz_av 10.90.2.35 242a.040d.48f0 14028
xyz_av 10.90.2.36 748f.c28d.6970 14088
xyz_sec 10.90.2.11 242a.040d.48de 11897
xyz_sec 10.90.2.12 748f.c28d.695e 11927
dmz_mgmt 172.21.0.10 cc79.d7d0.1ef1 14028
dmz_mgmt 172.21.0.11 6c4e.f661.53f1 14028
xyz_guest_wifi 172.21.1.5 0000.0c07.ac06 12047
xyz_guest_wifi 172.21.1.4 748f.c28d.6941 12377
outside xxx.xxx.xx.xx 8c1e.804d.e700 7

> show arp statistics
Number of ARP entries in ASA: 19

Dropped blocks in ARP: 158
Maximum Queued blocks: 16
Queued blocks: 0
Interface collision ARPs Received: 0
ARP-defense Gratuitous ARPS sent: 0
Total ARP retries: 138
Unresolved hosts: 0
Maximum Unresolved hosts: 8

capture asp type asp-drop

5: 11:19:39.074001 10.70.0.101 > 10.90.65.199 icmp: echo request Drop-reason: (no-adjacency) No valid adjacency, Drop-location: frame 0x000000aaad30c07c flow (NA)/NA

system support firewall-engine-debug

10.70.0.101 8 -> 10.90.65.199 0 1 AS=0 ID=1 GR=1-1 app event with client no change, service changed, payload no change, referred no change, misc no change, url no change, tls host no change, bits 0x5

10.70.0.101 8 -> 10.90.65.199 0 1 AS=0 ID=1 GR=1-1 New firewall session
10.70.0.101 8 -> 10.90.65.199 0 1 AS=0 ID=1 GR=1-1 using HW or preset rule order 3, 'SITE-to-SITE VPN Prod Traffic', action Trust and prefilter rule 0
10.70.0.101 8 -> 10.90.65.199 0 1 AS=0 ID=1 GR=1-1 fastpath action
10.70.0.101 8 -> 10.90.65.199 0 1 AS=0 ID=1 GR=1-1 Got end of flow event from hardware with flags 00002001
10.70.0.101 8 -> 10.90.65.199 0 1 AS=0 ID=1 GR=1-1 Rule Match Data: rule_id 0, rule_action 0 rev_id 0, rule_flags 3
10.70.0.101 8 -> 10.90.65.199 0 1 AS=0 ID=1 GR=1-1 Generating an EOF event with rule_id = 268435457 ruleAction = 3 ruleReason = 0
10.70.0.101 8 -> 10.90.65.199 0 1 AS=0 ID=1 GR=1-1 Received EOF, deleting the snort session
10.70.0.101 8 -> 10.90.65.199 0 1 AS=0 ID=1 GR=1-1 Deleting Firewall session flags=0x6e0040, logFlags=0x0
10.70.0.101 8 -> 10.90.65.199 0 1 AS=0 ID=1 GR=1-1 app event with client no change, service changed, payload no change, referred no change, misc no change, url no change, tls host no change, bits 0x5

From Remote site:

> packet-tracer input xyz_av icmp 10.90.65.199 1 2 10.70.0.101 detailed

Phase: 1
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Elapsed time: 36679 ns
Config:
Implicit Rule
Additional Information:
Forward Flow based lookup yields rule:
in id=0xffb2a60760, priority=1, domain=permit, deny=false
hits=4453179, user_data=0x0, cs_id=0x0, l3_type=0x8
src mac=0000.0000.0000, mask=0000.0000.0000
dst mac=0000.0000.0000, mask=0100.0000.0000
input_ifc=xyz_av, output_ifc=any

Phase: 2
Type: UN-NAT
Subtype: static
Result: ALLOW
Elapsed time: 23031 ns
Config:
nat (xyz_av,outside) source static VRF-xyz_av-10.90.64.0-20 VRF-xyz_av-10.90.64.0-20 destination static REMOTE-VPN-NET REMOTE-VPN-NET
Additional Information:
NAT divert to egress interface outside(vrfid:0)
Untranslate 10.70.0.101/0 to 10.70.0.101/0

Phase: 3
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Elapsed time: 13008 ns
Config:
access-group NGFW_ONBOX_ACL global
access-list NGFW_ONBOX_ACL advanced permit object-group |acSvcg-268435457 ifc xyz_av object-group |acSrcNwg-268435457 ifc outside object-group |acDestNwg-268435457 rule-id 268435457 event-log flow-end
access-list NGFW_ONBOX_ACL remark rule-id 268435457: ACCESS POLICY: NGFW_Access_Policy
access-list NGFW_ONBOX_ACL remark rule-id 268435457: L5 RULE: SITE-to-SITE VPN Prod Traffic
object-group service |acSvcg-268435457
service-object ip
object-group network |acSrcNwg-268435457
group-object LOCAL-VPN-NET
group-object REMOTE-VPN-NET
object-group network |acDestNwg-268435457
group-object LOCAL-VPN-NET
group-object REMOTE-VPN-NET
Additional Information:
This packet will be sent to snort for additional processing where a verdict will be reached
Forward Flow based lookup yields rule:
in id=0xff9c3052f0, priority=12, domain=permit, deny=false
hits=20195, user_data=0xffe3846b80, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=10.90.0.0, mask=255.255.0.0, port=0, tag=any, ifc=xyz_av(vrfid:0)
dst ip/id=10.70.0.0, mask=255.255.0.0, port=0, tag=any, ifc=outside(vrfid:0), vlan=0,
src nsg_id=none, dst nsg_id=none
dscp=0x0, input_ifc=any, output_ifc=any

Phase: 4
Type: NAT
Subtype:
Result: ALLOW
Elapsed time: 13008 ns
Config:
nat (xyz_av,outside) source static VRF-xyz_av-10.90.64.0-20 VRF-xyz_av-10.90.64.0-20 destination static REMOTE-VPN-NET REMOTE-VPN-NET
Additional Information:
Static translate 10.90.65.199/0 to 10.90.65.199/0
Forward Flow based lookup yields rule:
in id=0xffa03b1b30, priority=6, domain=nat, deny=false
hits=20677, user_data=0xffb5a658a0, cs_id=0x0, flags=0x0, protocol=0
src ip/id=10.90.64.0, mask=255.255.240.0, port=0, tag=any
dst ip/id=10.70.0.0, mask=255.255.0.0, port=0, tag=any
src nsg_id=none, dst nsg_id=none
dscp=0x0, input_ifc=xyz_av(vrfid:0), output_ifc=outside(vrfid:0)

Phase: 5
Type: NAT
Subtype: per-session
Result: ALLOW
Elapsed time: 13008 ns
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0xffb2023600, priority=0, domain=nat-per-session, deny=true
hits=293889, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any,
src nsg_id=none, dst nsg_id=none
dscp=0x0, input_ifc=any, output_ifc=any

Phase: 6
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Elapsed time: 13008 ns
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0xffb29a9830, priority=0, domain=inspect-ip-options, deny=true
hits=84906, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
src nsg_id=none, dst nsg_id=none
dscp=0x0, input_ifc=xyz_av(vrfid:0), output_ifc=any

Phase: 7
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Elapsed time: 46915 ns
Config:
class-map inspection_default
match default-inspection-traffic
policy-map global_policy
class inspection_default
inspect icmp
service-policy global_policy global
Additional Information:
Forward Flow based lookup yields rule:
in id=0xffc04c5440, priority=70, domain=inspect-icmp, deny=false
hits=4313, user_data=0xffc04c10e0, cs_id=0x0, use_real_addr, flags=0x0, protocol=1
src ip/id=0.0.0.0, mask=0.0.0.0, icmp-type=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, icmp-code=0, tag=any,
src nsg_id=none, dst nsg_id=none
dscp=0x0, input_ifc=xyz_av(vrfid:0), output_ifc=any

Phase: 8
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Elapsed time: 8530 ns
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0xffc04d3400, priority=70, domain=inspect-icmp-error, deny=false
hits=4313, user_data=0xffc04cf0a0, cs_id=0x0, use_real_addr, flags=0x0, protocol=1
src ip/id=0.0.0.0, mask=0.0.0.0, icmp-type=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, icmp-code=0, tag=any,
src nsg_id=none, dst nsg_id=none
dscp=0x0, input_ifc=xyz_av(vrfid:0), output_ifc=any

Phase: 9
Type: VPN
Subtype: encrypt
Result: ALLOW
Elapsed time: 17060 ns
Config:
Additional Information:
Forward Flow based lookup yields rule:
out id=0xffc03ba9e0, priority=70, domain=encrypt, deny=false
hits=27535, user_data=0x28024e4, cs_id=0xffc0130100, reverse, flags=0x0, protocol=0
src ip/id=10.90.0.0, mask=255.255.0.0, port=0, tag=any
dst ip/id=10.70.0.0, mask=255.255.0.0, port=0, tag=any
src nsg_id=none, dst nsg_id=none
dscp=0x0, input_ifc=any(vrfid:65535), output_ifc=outside

Phase: 10
Type: NAT
Subtype: rpf-check
Result: ALLOW
Elapsed time: 5971 ns
Config:
nat (xyz_av,outside) source static VRF-xyz_av-10.90.64.0-20 VRF-xyz_av-10.90.64.0-20 destination static REMOTE-VPN-NET REMOTE-VPN-NET
Additional Information:
Forward Flow based lookup yields rule:
out id=0xffb59d4810, priority=6, domain=nat-reverse, deny=false
hits=20674, user_data=0xffb446f920, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=10.90.64.0, mask=255.255.240.0, port=0, tag=any
dst ip/id=10.70.0.0, mask=255.255.0.0, port=0, tag=any
src nsg_id=none, dst nsg_id=none
dscp=0x0, input_ifc=xyz_av(vrfid:0), output_ifc=outside(vrfid:0)

Phase: 11
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Elapsed time: 107478 ns
Config:
Additional Information:
New flow created with id 458182, packet dispatched to next module
Module information for forward flow ...
snp_fp_inspect_ip_options
snp_fp_snort
snp_fp_inspect_icmp
snp_fp_translate
snp_fp_adjacency
snp_fp_encrypt
snp_fp_fragment
snp_fp_tracer_drop
snp_ifc_stat

Module information for reverse flow ...

Phase: 12
Type: EXTERNAL-INSPECT
Subtype:
Result: ALLOW
Elapsed time: 14501 ns
Config:
Additional Information:
Application: 'SNORT Inspect'

Phase: 13
Type: SNORT
Subtype: appid
Result: ALLOW
Elapsed time: 11272 ns
Config:
Additional Information:
service: ICMP(3501), client: (0), payload: (0), misc: ICMP(3501)

Phase: 14
Type: SNORT
Subtype: firewall
Result: ALLOW
Elapsed time: 125196 ns
Config:
Network 0, Inspection 0, Detection 2, Rule ID 268435457
Additional Information:
Starting rule matching, zone 8 -> 2, geo 0 -> 0, vlan 0, src sgt: 0, src sgt type: unknown, dst sgt: 0, dst sgt type: unknown, user 9999997, no url or host, no xff
Matched rule ids 268435457 - Trust

Result:
input-interface: xyz_av(vrfid:0)
input-status: up
input-line-status: up
output-interface: outside(vrfid:0)
output-status: up
output-line-status: up
Action: allow
Time Taken: 448665 ns

>

[MHM Cisco World]

you use different IP in packet-tracer not 10.57.65.199 which show in ASP ARP drop ?

MHM

[AigarsK]

My apologies, I did replace them for sharing purposes and that one skipped through my checks.

[MHM Cisco World · Accepted solution]

> packet-tracer input outside icmp 10.70.0.101 1 2 10.90.65.199  detailed <<- share this also,

10.90.65.x <<- this subnet is direct connect to FTD interface ? if not do you have route for this subnet ?

MHM

[AigarsK]

Thank you MHM,

I found an error in my configuration, I had run the command you asked, but I only paid attention to last entry stating "Drop-reason: (ipsec-spoof) IPSEC Spoof detected" completely omitting the fact that I had made error in NAT.

When this firewall was in its initial setup, I had one NAT entry for /16 which would have covered all IP addresses used in downstream in their according security zones.

What was happening here is as followed, traffic which was initiated from Remote Site would originate from correct security zone/interface which would created session state and NAT entry and be sent back across the VPN, as it it expects data in return it would use this session and NAT entry to return traffic from HQ to client. (confirmation bias as clients in downstream would get IP addresses from DHCP in HQ and DNS responses).

But any traffic generated by HQ, would traverse the S2S VPN and due to NAT, it got egressed on wrong interface and subsequently dropped.

Thanks for leading me onto this issue!