Cisco ASA 9.16 Ikev1 site to site -> PFSense

[linuxman]

I am trying to setup a L2L IPSec VPN between a Cisco ASA and an PfSense software firewall. The VPN tunnel comes up but the issue is that something in my ASA will not let the local traffic go through the tunnel.

When I ping from the PfSense side, I see the traffic is going through the tunnel and hits the ASA, but the ASA is unable to respond.

This is how I configured my ASA (relevant portions)

crypto ipsec ikev1 transform-set TFS esp-aes esp-sha-hmac 
crypto ipsec ikev1 transform-set DALLAS esp-aes esp-sha-hmac 
crypto ipsec security-association pmtu-aging infinite
crypto dynamic-map DYNMAP 1 set ikev1 transform-set TFS
crypto map CMAP 1 ipsec-isakmp dynamic DYNMAP
crypto map CMAP 3 match address DALLAS
crypto map CMAP 3 set peer 123.123.123.123
crypto map CMAP 3 set ikev1 transform-set DALLAS
crypto map CMAP interface WAN
crypto ca trustpool policy
crypto ikev1 enable WAN
crypto ikev1 policy 1
 authentication pre-share
 encryption aes
 hash sha
 group 5
 lifetime none
crypto ikev1 policy 2
 authentication pre-share
 encryption aes
 hash sha
 group 14
 lifetime none
crypto ikev1 policy 3
 authentication pre-share
 encryption aes-256
 hash sha
 group 14
 lifetime none
---
---
access-list SPLITTUNNEL standard permit 192.168.0.0 255.255.255.0 
access-list DALLAS extended permit ip object LAN object DALLAS_IP 
access-list DALLAS extended permit ip object DALLAS_IP object LAN
---
---
tunnel-group 123.123.123.123 type ipsec-l2l
tunnel-group 123.123.123.123 ipsec-attributes
 ikev1 pre-shared-key *****
---
---
nat (GI5,WAN) source static LAN LAN destination static VPN VPN no-proxy-arp route-lookup
nat (GI4,WAN) source static LAN LAN destination static VPN VPN no-proxy-arp route-lookup
nat (GI6,WAN) source dynamic LAN interface
nat (GI2,WAN) source dynamic LAN interface
nat (GI3,WAN) source dynamic LAN interface
nat (GI4,WAN) source dynamic LAN interface
nat (GI7,WAN) source dynamic LAN interface
nat (GI8,WAN) source dynamic LAN interface
nat (any,WAN) source static LAN LAN destination static DALLAS_IP DALLAS_IP
!
object network GI5
 nat (GI5,WAN) dynamic interface

What I have tried

1. Adding static route route WAN 192.168.100.0 255.255.255.0 123.123.123.123

2. Added different NAT rules to nat the LAN to the DALLAS_IP remote subnet.

Now this is what I get when I try running a packet trace.

ASA(config)# packet-tracer input GI5 icmp 192.168.0.25 1 1 192.168.100.99

Phase: 1
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (any,WAN) source static LAN LAN destination static DALLAS_IP DALLAS_IP
Additional Information:
NAT divert to egress interface WAN
Untranslate 192.168.100.99/0 to 192.168.100.99/0

Phase: 2
Type: NAT
Subtype: 
Result: ALLOW
Config:
nat (any,WAN) source static LAN LAN destination static DALLAS_IP DALLAS_IP
Additional Information:
Static translate 192.168.0.25/0 to 192.168.0.25/0

Phase: 3
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 4
Type: IP-OPTIONS
Subtype: 
Result: ALLOW
Config:
Additional Information:

Phase: 5
Type: NAT
Subtype: 
Result: ALLOW
Config:
nat (any,WAN) source static LAN LAN destination static DALLAS_IP DALLAS_IP
Additional Information:
Static translate 192.168.0.25/0 to 192.168.0.25/0

Phase: 6
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
              
Phase: 7
Type: IP-OPTIONS
Subtype: 
Result: ALLOW
Config:
Additional Information:

Phase: 8
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
class-map inspection_default
 match default-inspection-traffic
policy-map global_policy
 class inspection_default
  inspect icmp 
service-policy global_policy global
Additional Information:

Phase: 9
Type: INSPECT
Subtype: np-inspect
Result: ALLOW 
Config:
Additional Information:

Phase: 10
Type: VPN
Subtype: encrypt
Result: DROP
Config:
Additional Information:

Result:
input-interface: GI5
input-status: up
input-line-status: up
output-interface: WAN
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x0000556b4d48c9f5 flow (NA)/NA

Now with an error like that claiming it is an ACL denying traffic is confusing when the PfSense firewall has the correct ACL's configured as well as the ASA. I'm basically mentally defeated at this point by a black metal box and have no idea what could possibly be the issue... Any ideas?

[linuxman]

i'm on my Mac pinging 192.168.100.99 which is a Windows VM with the firewall off. No response.

[MHM Cisco World]

do you config ICMP inspection in global_policy ?

[linuxman]

Yes, that is added. I have a printer in my home network as well, browsing to the IP on the pfSense side would be another way of checking but that also doesn't seem to work.

[MHM Cisco World]

I try lab to find in which case there is ONLY one way traffic.
I find that NAT if not config right can effect traffic and you will get one way traffic through IPsec. 
can you share 

show nat 
show run nat

also share 
show interface ip bri
show route 

[linuxman]

SHOW NAT

Spoiler

Manual NAT Policies Implicit (Section 0)
1 (nlp_int_tap) to (LAN) source dynamic nlp_client_0_0.0.0.0_17proto53_intf11 interface  destination static nlp_client_0_ipv4_6 nlp_client_0_ipv4_6 service nlp_client_0_17svc53_5 nlp_client_0_17svc53_5
    translate_hits = 0, untranslate_hits = 0
2 (nlp_int_tap) to (WAN) source dynamic nlp_client_0_0.0.0.0_17proto53_intf2 interface  destination static nlp_client_0_ipv4_2 nlp_client_0_ipv4_2 service nlp_client_0_17svc53_1 nlp_client_0_17svc53_1
    translate_hits = 0, untranslate_hits = 0
3 (nlp_int_tap) to (LAN) source dynamic nlp_client_0_ipv6_::_17proto53_intf11 interface ipv6  destination static nlp_client_0_ipv6_8 nlp_client_0_ipv6_8 service nlp_client_0_17svc53_7 nlp_client_0_17svc53_7
    translate_hits = 0, untranslate_hits = 0
4 (nlp_int_tap) to (WAN) source dynamic nlp_client_0_ipv6_::_17proto53_intf2 interface ipv6  destination static nlp_client_0_ipv6_4 nlp_client_0_ipv6_4 service nlp_client_0_17svc53_3 nlp_client_0_17svc53_3
    translate_hits = 0, untranslate_hits = 0

Manual NAT Policies (Section 1)
1 (any) to (WAN) source static LAN LAN  destination static DALLAS_IP DALLAS_IP
    translate_hits = 6615, untranslate_hits = 6615
2 (GI5) to (WAN) source static LAN LAN  destination static VPN VPN no-proxy-arp route-lookup
    translate_hits = 10, untranslate_hits = 10
3 (GI4) to (WAN) source static LAN LAN  destination static VPN VPN no-proxy-arp route-lookup
    translate_hits = 0, untranslate_hits = 0

Auto NAT Policies (Section 2)
1 (GI5) to (WAN) source static FORWARD_HTTP interface  service tcp www www 
    translate_hits = 0, untranslate_hits = 1527
2 (GI5) to (WAN) source static FORWARD_HTTPS interface  service tcp https https 
    translate_hits = 0, untranslate_hits = 3186
3 (GI5) to (WAN) source static FORWARD_PLEX interface  service tcp 32400 32400 
    translate_hits = 0, untranslate_hits = 76
4 (GI5) to (WAN) source static FORWARD_SSH interface  service tcp ssh ssh 
    translate_hits = 0, untranslate_hits = 314
5 (GI2) to (WAN) source dynamic GI2 interface 
    translate_hits = 0, untranslate_hits = 0
6 (GI3) to (WAN) source dynamic GI3 interface 
    translate_hits = 6614, untranslate_hits = 4528
7 (GI4) to (WAN) source dynamic GI4 interface 
    translate_hits = 31189, untranslate_hits = 46720
8 (GI5) to (WAN) source dynamic GI5 interface 
    translate_hits = 356470, untranslate_hits = 74659
9 (GI6) to (WAN) source dynamic GI6 interface 
    translate_hits = 25825, untranslate_hits = 134
10 (GI7) to (WAN) source dynamic GI7 interface 
    translate_hits = 0, untranslate_hits = 0
11 (GI8) to (WAN) source dynamic GI8 interface 
    translate_hits = 0, untranslate_hits = 0

SHOW RUN NAT

Spoiler

nat (any,WAN) source static LAN LAN destination static DALLAS_IP DALLAS_IP
nat (GI5,WAN) source static LAN LAN destination static VPN VPN no-proxy-arp route-lookup
nat (GI4,WAN) source static LAN LAN destination static VPN VPN no-proxy-arp route-lookup
!
object network FORWARD_HTTP
 nat (GI5,WAN) static interface service tcp www www 
object network FORWARD_HTTPS
 nat (GI5,WAN) static interface service tcp https https 
object network FORWARD_PLEX
 nat (GI5,WAN) static interface service tcp 32400 32400 
object network FORWARD_SSH
 nat (GI5,WAN) static interface service tcp ssh ssh 
object network GI5
 nat (GI5,WAN) dynamic interface
object network GI4
 nat (GI4,WAN) dynamic interface
object network GI6
 nat (GI6,WAN) dynamic interface
object network GI2
 nat (GI2,WAN) dynamic interface
object network GI3
 nat (GI3,WAN) dynamic interface
object network GI7
 nat (GI7,WAN) dynamic interface
object network GI8
 nat (GI8,WAN) dynamic interface

SHOW INTERFACE IP BRIEF

Spoiler

Interface                  IP-Address      OK? Method Status                Protocol
Virtual0                   127.1.0.1       YES unset  up                    up  
GigabitEthernet1/1         111.111.111.111  YES DHCP   up                    up  
GigabitEthernet1/2         192.168.0.1     YES unset  up                    up  
GigabitEthernet1/3         192.168.0.1     YES unset  up                    up  
GigabitEthernet1/4         192.168.0.1     YES unset  up                    up  
GigabitEthernet1/5         192.168.0.1     YES unset  up                    up  
GigabitEthernet1/6         192.168.0.1     YES unset  up                    up  
GigabitEthernet1/7         192.168.0.1     YES unset  up                    up  
GigabitEthernet1/8         192.168.0.1     YES unset  down                  down
Internal-Control1/1        unassigned      YES unset  up                    down
Internal-Data1/1           unassigned      YES unset  up                    down
Internal-Data1/2           unassigned      YES unset  down                  down
Internal-Data1/3           unassigned      YES unset  up                    up  
Internal-Data1/4           169.254.1.1     YES unset  up                    up  
Management1/1              10.10.10.1      YES CONFIG down                  down
BVI1                       192.168.0.1     YES CONFIG up                    up

SHOW ROUTE

Spoiler

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2, V - VPN
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, + - replicated route
       SI - Static InterVRF
Gateway of last resort is <GW> to network 0.0.0.0

S*       0.0.0.0 0.0.0.0 [1/0] via <GW>, WAN
C       <GW NET> 255.255.224.0 is directly connected, WAN
L        <HOME IP> 255.255.255.255 is directly connected, WAN
C        192.168.0.0 255.255.255.0 is directly connected, LAN
L        192.168.0.1 255.255.255.255 is directly connected, LAN

[MHM Cisco World]

192.168.100.99 <<- this your PC 

this you show route 

C        192.168.0.0 255.255.255.0 is directly connected, LAN
L        192.168.0.1 255.255.255.255 is directly connected, LAN

I do lab and lab finally 
I see that the route-lookup is missing always in your case ?
why ? finally I found that IP you use for PC is different than you use for your LAN.

correct it and check again.

[linuxman]

No, 192.168.100.0/24 is the pfSense subnet, remote subnet to my ASA.

ASA subnet is 192.168.0.0/24, local subnet.

And I also don't have any manual routes configured. That is what I was trying to do before and it didn't help anyways.

[MHM Cisco World]

You use DHCP to assign IP to OUT of ASA and I think you use setroute 
now are other peer use IKEv1 set peer using IP same as IP your local ASA get from DHCP?

[linuxman]

A bit of a word salad, but I think I understood what you meant.

Just to clarify, the remote network is behind a pfSense firewall not another ASA.

But yes, the tunnel is actually IKEv2 and the pfSense firewall is set to use the peer IP and not a domain name.

I did try IKEv1 at one point but I got the same issue where I could not ping across...

[MHM Cisco World]

clear crypto ipsec sa inactive <<- can you try this command in ASA 

[linuxman]

Hello, late reply - busy week... I did try that and it seemed to have cleared nothing. Still same issue... I really am not sure why on earth this VPN isn't working, idk what my ASA is doing but I feel like just booting up my old Fortigate since it works flawless with Fortigate...

[MHM Cisco World]

OK let me check last time all config may be I can figure out the issue. 
I will update you tonight.

[MHM Cisco World]

use capture to see if the traffic is enter to ASA IN interface 
 NO capture VPN interface IN << use NO to disable capture 

matching the SPI inbound and outbound with ASP table 
we do 
show crypto ipsec sa
show asp table vpn-context detail 

check that the SPI is match the SPI in vpn-context, you will see many VPN-context use peer IP to select the one toward L2L remote peer.

[linuxman]

I'll have to come back to this in the next week or so. Been over 20 days since I've ran backups on my servers which was the whole purpose of this VPN. So I'll try this again but on a test network - meanwhile I'll just put my Fortigate back in place while I figure out the ASA.