IPSec VPN Tunnel Lan-to-Lan decrypt count zero

[El Rondo]

Hi, 

I have one issue with IPSec tunnel Lan-to-Lan between ASA 5525x (v9.8) and ASA FPR 2110 (v9.16). My Tunnel is up but ping between each client was not successful. Both peer status sh cry isakmp sa in "MM_ACTIVE".
I ran packet-tracer icmp between peer and result shows ALLOW for every phase 1 and 2. Debug command sh cry ipsec sa shows packets encrypt is non zero but decrypt is zero for both peer.

Spoiler

#pkts encaps: 80, #pkts encrypt: 80, #pkts digest: 80
#pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0

I suspected NAT was the issue and until now I haven't found the root cause. Hopefully someone who had the solution could help me to fix the issue.

[MHM Cisco World]

Ok' 

In outside you use acl' 

Add to this acl line 

Permit ip <remote lan><local lan>

Sure it will work.

MHM

[El Rondo]

HQ
I apply this ACL outside in interface
# access-list OUTSIDE_ACCESS_IN extended permit ip object PT_SERVER object SERVER 
access-group OUTSIDE_ACCESS_IN in interface WAN
and it doesnt capture any hitcount. below is the results.
access-list OUTSIDE_ACCESS_IN line 1 extended permit ip object PT_SERVER object SERVER (hitcnt=0) 0x6265b166
access-list OUTSIDE_ACCESS_IN line 1 extended permit ip 172.31.1.0 255.255.255.0 172.16.4.0 255.255.255.0 (hitcnt=0) 0x6265b166

PT
# access-list OUTSIDE_ACCESS_IN extended permit ip object HQ_SERVER object SERVER 
access-group OUTSIDE_ACCESS_IN in interface WAN

it doesnt capture any hitcount
access-list OUTSIDE_ACCESS_IN line 1 extended permit ip object HQ_SERVER object SERVER (hitcnt=0) 0x66f5970d
access-list OUTSIDE_ACCESS_IN line 1 extended permit ip 172.16.4.0 255.255.255.0 172.31.1.0 255.255.255.0 (hitcnt=0) 0x66f5970d

[MHM Cisco World]

Any update?

Happy news maybe

MHM

[El Rondo]

Sorry for the delay due to another project go live.

I,ve verify that group policy should be ok for both HQ and PT

HQ
group-policy GroupPolicy_10.151.21.3 internal
group-policy GroupPolicy_10.151.21.3 attributes
vpn-tunnel-protocol ikev1
tunnel-group 10.151.21.3 type ipsec-l2l
tunnel-group 10.151.21.3 general-attributes
default-group-policy GroupPolicy_10.151.21.3
tunnel-group 10.151.21.3 ipsec-attributes
ikev1 pre-shared-key *****

PT
group-policy GroupPolicy_10.152.25.34 internal
group-policy GroupPolicy_10.152.25.34 attributes
vpn-tunnel-protocol ikev1
tunnel-group 10.152.25.34 type ipsec-l2l
tunnel-group 10.152.25.34 general-attributes
default-group-policy GroupPolicy_10.152.25.34
tunnel-group 10.152.25.34 ipsec-attributes
ikev1 pre-shared-key *****

Its just one question when issued command at  FW FPR 2110 located at PT # sh NAT detail it shows

1 (nlp_int_tap) to (WAN) source dynamic nlp_client_0_0.0.0.0_17proto53_intf2 interface destination static nlp_client_0_ipv4_2 nlp_client_0_ipv4_2 service nlp_client_0_17svc53_1 nlp_client_0_17svc53_1
translate_hits = 89915, untranslate_hits = 89918
Source - Origin: 169.254.1.3/32, Translated: 10.151.21.3/29
Destination - Origin: 0.0.0.0/0, Translated: 0.0.0.0/0
Service - Origin: udp destination eq domain , Translated: udp destination eq domain
2 (nlp_int_tap) to (WAN) source dynamic nlp_client_0_ipv6_::_17proto53_intf2 interface ipv6 destination static nlp_client_0_ipv6_4 nlp_client_0_ipv6_4 service nlp_client_0_17svc53_3 nlp_client_0_17svc53_3
translate_hits = 0, untranslate_hits = 0
Source - Origin: fd00:0:0:1::3/128, Translated:
Destination - Origin: ::/0, Translated: ::/0
Service - Origin: udp destination eq domain , Translated: udp destination eq domain

Manual NAT Policies (Section 1)
1 (LAN) to (WAN) source static SERVER SERVER destination static HQ_SERVER HQ_SERVER
translate_hits = 337865, untranslate_hits = 337865
Source - Origin: 172.31.1.0/24, Translated: 172.31.1.0/24
Destination - Origin: 172.16.4.0/24, Translated: 172.16.4.0/24

** I have no idea what is nlp_int_tap and does it have contribute to this issue?

[MHM Cisco World]

Bad news for me
anyway life not easy 

Manual NAT Policies (Section 1) <<- since you have NO-NAT in Section 1 any other NAT will not effect your traffic 

can you do troubleshooting one more time
this time I add more trouble point, before each trouble do ping from LAN to LAN 

NOTE:- show vpn-seesiondb l2l <<- there is bytes count in this command so please do it twice before ping LAN to LAN and after ping, check if the counter is increase or not 

thanks a lot for your time

MHM

[El Rondo]

Hi, here I tested 4 trouble as per your suggestion. Trouble 5,6,7 and 9. However test 8 and 10 unable to run due to unrecognised command.

Trouble 5: (PT)

# show vpn-sessiondb l2l

Session Type: LAN-to-LAN

Connection : 10.152.25.34
Index : 82 IP Addr : 10.152.25.34
Protocol : IKEv1 IPsec
Encryption : IKEv1: (1)AES256 IPsec: (1)AES256
Hashing : IKEv1: (1)SHA1 IPsec: (1)SHA1
Bytes Tx : 96060 Bytes Rx : 0
Login Time : 22:46:14 UTC Mon May 6 2024
Duration : 2h:14m:53s

** counter Tx is increase but not Rx (0)

Trouble 6: (HQ)

# sh vpn-sessiondb l2l

Session Type: LAN-to-LAN

Connection : 10.151.21.3
Index : 57240 IP Addr : 10.151.21.3
Protocol : IKEv1 IPsec
Encryption : IKEv1: (1)AES256 IPsec: (1)AES256
Hashing : IKEv1: (1)SHA1 IPsec: (1)SHA1
Bytes Tx : 50968 Bytes Rx : 0
Login Time : 15:44:16 UTC Mon May 6 2024
Duration : 2h:17m:18s

** counter Tx is increase but not Rx (0)

Trouble 7: (HQ)

# capture asp-drop type asp-drop acl-drop
# clear capture /all
# sh capture
capture asp-drop type asp-drop acl-drop [Capturing - 144565 bytes]
# sh capture asp-drop

12 packets captured

1: 23:25:16.040830 802.1Q vlan#500 P1 18.161.180.116.443 > 180.72.143.77.57422: FP 2335604001:2335604147(146) ack 1747894912 win 131 <nop,nop,timestamp 1607499032 1923148257> Drop-reason: (acl-drop) Flow is denied by configured rule
2: 23:25:16.459784 802.1Q vlan#500 P1 79.127.235.6.443 > 180.72.143.77.63192: R 1641191309:1641191309(0) win 0 Drop-reason: (acl-drop) Flow is denied by configured rule
3: 23:25:16.464926 201.162.74.201 > 10.151.25.5: icmp: 201.162.74.201 udp port 43864 unreachable Drop-reason: (acl-drop) Flow is denied by configured rule
4: 23:25:16.474676 802.1Q vlan#500 P1 137.184.200.122.80 > 180.72.143.77.5601: S 976303494:976303494(0) win 65535 <mss 1460> Drop-reason: (acl-drop) Flow is denied by configured rule
5: 23:25:16.476980 802.1Q vlan#500 P1 18.161.180.116.443 > 180.72.143.77.57422: FP 2335604001:2335604147(146) ack 1747894912 win 131 <nop,nop,timestamp 1607499468 1923148257> Drop-reason: (acl-drop) Flow is denied by configured rule
6: 23:25:17.106928 201.162.78.177 > 10.151.25.5: icmp: 201.162.78.177 udp port 40683 unreachable Drop-reason: (acl-drop) Flow is denied by configured rule
7: 23:25:17.341016 802.1Q vlan#500 P1 18.161.180.116.443 > 180.72.143.77.57422: FP 2335604001:2335604147(146) ack 1747894912 win 131 <nop,nop,timestamp 1607500332 1923148257> Drop-reason: (acl-drop) Flow is denied by configured rule
8: 23:25:17.607009 201.162.69.63 > 10.151.25.5: icmp: 201.162.69.63 udp port 58026 unreachable Drop-reason: (acl-drop) Flow is denied by configured rule
9: 23:25:17.894393 802.1Q vlan#500 P1 79.127.235.11.443 > 180.72.143.77.52905: R 600701383:600701383(0) win 0 Drop-reason: (acl-drop) Flow is denied by configured rule
10: 23:25:19.027571 802.1Q vlan#500 P1 17.248.224.3.443 > 180.72.143.77.65319: R 69559393:69559393(0) win 0 Drop-reason: (acl-drop) Flow is denied by configured rule
11: 23:25:19.030683 802.1Q vlan#500 P1 17.248.224.3.443 > 180.72.143.77.65319: R 69559393:69559393(0) win 0 Drop-reason: (acl-drop) Flow is denied by configured rule
12: 23:25:19.068798 802.1Q vlan#500 P1 18.161.180.116.443 > 180.72.143.77.57422: FP 2335604001:2335604147(146) ack 1747894912 win 131 <nop,nop,timestamp 1607502060 1923148257> Drop-reason: (acl-drop) Flow is denied by configured rule
12 packets shown

# sh asp drop

Frame drop:
Flow is denied by configured rule (acl-drop) 103
First TCP packet not SYN (tcp-not-syn) 122
TCP RST/FIN out of order (tcp-rstfin-ooo) 52
TCP packet SEQ past window (tcp-seq-past-win) 1
TCP RST/SYN in window (tcp-rst-syn-in-win) 2
Connection to PAT address without pre-existing xlate (nat-no-xlate-to-pat-pool) 18

Last clearing: 23:45:55 UTC May 6 2024 by enable_15

Flow drop:

Last clearing: 23:45:55 UTC May 6 2024 by enable_15

** count is increasing

Trouble 9: (PT)

# capture asp-drop type asp-drop acl-drop
# clear capture /all
# sh capture
capture asp-drop type asp-drop acl-drop [Capturing - 144500 bytes]

# sh capture asp-drop
Target: MIPS
Hardware: FPR-2110
Cisco Adaptive Security Appliance Software Version 9.16(3)23
ASLR enabled, text region aaab1a5000-aaafd89bcc

28 packets captured

1: 06:44:29.663204 802.1Q vlan#500 P0 17.188.169.98.3483 > 180.72.143.78.53317: udp 16 Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x000000aaad135fb0 flow (NA)/NA

2: 06:44:29.832522 172.31.11.109.55451 > 8.8.4.4.53: udp 39 Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x000000aaad135fb0 flow (NA)/NA

3: 06:44:30.026808 802.1Q vlan#500 P0 1.9.87.98.443 > 180.72.143.78.44494: R 3739394844:3739394844(0) win 0 Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x000000aaad135fb0 flow (NA)/NA

4: 06:44:30.391993 802.1Q vlan#500 P0 175.136.17.189.50269 > 180.72.143.78.54704: udp 52 Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x000000aaad135fb0 flow (NA)/NA

5: 06:44:30.497914 802.1Q vlan#500 P0 110.159.120.171.57180 > 180.72.143.78.7680: S 3562103374:3562103374(0) win 64240 <mss 1452,nop,wscale 8,nop,nop,sackOK> Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x000000aaad135fb0 flow (NA)/NA

6: 06:44:31.043393 802.1Q vlan#500 P0 60.54.187.222.65085 > 180.72.143.78.7680: S 1863545524:1863545524(0) win 64240 <mss 1452,nop,wscale 8,nop,nop,sackOK> Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x000000aaad135fb0 flow (NA)/NA

7: 06:44:31.789860 802.1Q vlan#500 P0 163.181.90.98.443 > 180.72.143.78.41702: R 1331858764:1331858764(0) win 0 Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x000000aaad135fb0 flow (NA)/NA

8: 06:44:32.216709 802.1Q vlan#500 P0 17.188.169.98.3483 > 180.72.143.78.53317: udp 16 Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x000000aaad135fb0 flow (NA)/NA

9: 06:44:32.224613 802.1Q vlan#500 P0 17.188.169.98.3483 > 180.72.143.78.53317: udp 16 Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x000000aaad135fb0 flow (NA)/NA

10: 06:44:32.379847 802.1Q vlan#500 P0 27.125.240.20 > 180.72.143.78 icmp: 27.125.240.20 udp port 7154 unreachable Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x000000aaad135fb0 flow (NA)/NA

11: 06:44:32.384547 802.1Q vlan#500 P0 175.136.17.189.50269 > 180.72.143.78.54704: udp 52 Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x000000aaad135fb0 flow (NA)/NA

12: 06:44:32.386759 172.31.11.108.48286 > 8.8.4.4.53: udp 39 Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x000000aaad135fb0 flow (NA)/NA

13: 06:44:32.722252 802.1Q vlan#500 P0 52.216.33.8.443 > 180.72.143.78.61804: F 2694094271:2694094271(0) ack 1430004393 win 251 Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x000000aaad135fb0 flow (NA)/NA

14: 06:44:32.726418 802.1Q vlan#500 P0 89.110.23.11.11348 > 180.72.143.78.14063: udp 103 Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x000000aaad135fb0 flow (NA)/NA

15: 06:44:33.124993 802.1Q vlan#500 P0 52.182.143.210.443 > 180.72.143.78.51626: S 2368795233:2368795233(0) ack 2486179318 win 65535 <mss 1440,nop,wscale 8,nop,nop,sackOK> Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x000000aaad135fb0 flow (NA)/NA

16: 06:44:34.040235 802.1Q vlan#500 P0 27.125.240.20 > 180.72.143.78 icmp: 27.125.240.20 udp port 7154 unreachable Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x000000aaad135fb0 flow (NA)/NA

17: 06:44:34.146659 802.1Q vlan#500 P0 180.75.235.166.16502 > 180.72.143.78.49989: udp 35 Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x000000aaad135fb0 flow (NA)/NA

18: 06:44:34.156806 802.1Q vlan#500 P0 71.18.253.182.27471 > 180.72.143.78.49989: udp 89 Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x000000aaad135fb0 flow (NA)/NA

19: 06:44:34.269013 802.1Q vlan#500 P0 172.217.25.195.443 > 180.72.143.78.14669: R 1560519085:1560519085(0) win 0 Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x000000aaad135fb0 flow (NA)/NA

20: 06:44:34.340665 802.1Q vlan#500 P0 180.75.235.166.16502 > 180.72.143.78.49989: udp 35 Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x000000aaad135fb0 flow (NA)/NA

21: 06:44:34.349789 802.1Q vlan#500 P0 71.18.253.182.27471 > 180.72.143.78.49989: udp 89 Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x000000aaad135fb0 flow (NA)/NA

22: 06:44:34.541201 802.1Q vlan#500 P0 180.75.235.166.16502 > 180.72.143.78.49989: udp 35 Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x000000aaad135fb0 flow (NA)/NA

23: 06:44:34.550310 802.1Q vlan#500 P0 71.18.253.182.27471 > 180.72.143.78.49989: udp 89 Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x000000aaad135fb0 flow (NA)/NA

24: 06:44:34.741111 802.1Q vlan#500 P0 180.75.235.166.16502 > 180.72.143.78.49989: udp 35 Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x000000aaad135fb0 flow (NA)/NA

25: 06:44:34.750281 802.1Q vlan#500 P0 71.18.253.182.27471 > 180.72.143.78.49989: udp 89 Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x000000aaad135fb0 flow (NA)/NA

26: 06:44:34.837298 172.31.11.109.40924 > 8.8.8.8.53: udp 39 Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x000000aaad135fb0 flow (NA)/NA

27: 06:44:35.131859 169.254.1.3.123 > 47.254.196.78.123: udp 48 Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x000000aaad135fb0 flow (NA)/NA

28: 06:44:35.243365 802.1Q vlan#500 P0 157.240.236.18.443 > 180.72.143.78.20932: R 1328992193:1328992193(0) win 0 Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x000000aaad135fb0 flow (NA)/NA

28 packets shown

# sh asp drop

Frame drop:
No route to host (no-route) 20
Flow is denied by configured rule (acl-drop) 295
First TCP packet not SYN (tcp-not-syn) 680
TCP failed 3 way handshake (tcp-3whs-failed) 3
TCP RST/FIN out of order (tcp-rstfin-ooo) 164
TCP packet SEQ past window (tcp-seq-past-win) 220

Last clearing: 06:46:26 UTC May 7 2024 by enable_15

Flow drop:
Inspection failure (inspect-fail) 4

Last clearing: 06:46:26 UTC May 7 2024 by enable_15

** count is increasing

[MHM Cisco World]

capture asp-drop type asp-drop No route to host

The drop acl dont show tunnel IP nor the LAN to LAN so asl-drop not issue here

But since we from beginning suspect of PBR do above capture abd ping

MHM