Solved: ASA5516 Site-to-Site VPN not communicating one way

ringpowersolutions · ‎01-11-2024

We have site-to-site VPNs between 3 sites using ASA Firewalls. We can ping devices from the first site (Head Office) to the second site (Remote Office) without issues, and the Remote Office can also ping the Head Office. The Head Office and the Remote Office can ping the third site (DataCenter) without issue. We are also able to access data across the VPN in both Offices to the DataCenter. The DataCenter is unable to ping either the Head Office or Remote Office, or initiate any data traffic to the sites.

Previously, we had only the Head Office and DataCenter and they were able to ping and transfer data without issue. For business requirements, we needed to add the Remote Office and the issues have started since then and are unable to remove the VPN. The Remote Office is now live as users are able to work, but automated processes triggered from the DataCenter to the Head Office have stopped working.

We have run packet-tracer input [INT] icmp [INT_IP] 8 1 [EXT_IP] detail on all sites and the response is that the traffic is allowed over the VPN on all sites.

We have run clear isakmp sa on the DataCenter side and that has not resolved the issue.

The command show crypto isakmp sa detail run on the DataCenter shows two active tunnels.

The command show crypto ipsec sa shows the correct local address, correct current peer, correct local and remote idents and the ACL that was configured for the traffic. It does not show any errors with the packets across the VPN.

For troubleshooting purposes, we temporarily set all ACLs to ip permit any any and icmp permit any any on our firewalls to no success.

We verified the policy-map global_policy settings included icmp and that had resolved issues we had with the Remote Office not being able to ping the DataCenter, but did not resolve the issue we have with the DataCenter sending communications through the VPN to the other sites.

The DataCenter currently has a single Internet connection and we only have a static route pointing all unknown traffic to the next hop IP Address for all external traffic. This was the setup before the Remote Office was added and was working until recently.

Any idea what could be causing this problem, or any commands to run to resolve the issue?

ringpowersolutions · ‎02-05-2024

We have found the issue. Problem was the Offices had two NAT statements that were conflicting. Below is the commands used to troubleshoot this.

capture asp type asp-drop all
show capture asp | include 10.169.2.69
   8: 13:24:43.496235       802.1Q vlan#13 P0 10.169.2.69.52411 > 10.168.8.27.161:  udp 78 Drop-reason: <b>(no-adjacency) No valid adjacency</b>

The "(no-adjacency) No valid adjacency" from the capture of asp-drop traffic seems to be a routing issue from what we found online. We checked the Routing table and everything looked fine. We checked the NAT entries and found the following statements:

nat (if_mgmt,any) source static OBJ-10.168.0.0-16 OBJ-10.168.0.0-16 destination static OBJ-10.169.0.0-16 OBJ-10.169.0.0-16 no-proxy-arp
nat (if_swr,any) source static OBJ-10.168.0.0-16 OBJ-10.168.0.0-16 destination static OBJ-10.169.0.0-16 OBJ-10.169.0.0-16 no-proxy-arp

We replaced the NAT statement for if_mgmt as shown below. This then resolved the issue immediately.

object-group network OBJ-10.168.1.32-20
network-object 10.168.1.32 255.255.240.0

no nat (if_mgmt,any) source static OBJ-10.168.0.0-16 OBJ-10.168.0.0-16 destination static OBJ-10.169.0.0-16 OBJ-10.169.0.0-16 no-proxy-arp
nat (if_mgmt,any) source static OBJ-10.168.1.32-20 OBJ-10.168.1.32-20 destination static OBJ-10.169.0.0-16 OBJ-10.169.0.0-16 no-proxy-arp

Thank you @MHM Cisco World and @Rob Ingram for your help with this issue.

View solution in original post

Rob Ingram · ‎01-11-2024

@ringpowersolutions when you run "show crypto ipsec sa" do the encap|decap counters increase on the DC firewall? The usual issue is a NAT issue, you may need a NAT exemption rule to ensure traffic is not unintentially translated behind the outside interface. The packet-tracer output should provide information on what NAT rule is matched.

Please provide the output of "show crypto ipsec sa" and the packet-tracer output.

ringpowersolutions · ‎01-11-2024

DC# sh crypto ipsec sa
interface: inet
Crypto map tag: cm_VPN_Hub, seq num: 12, local addr: #.#.#.#

access-list cracl_VPN_HeadOff extended permit ip 10.169.0.0 255.255.0.0 #.#.#.# 255.255.255.0
local ident (addr/mask/prot/port): (10.169.0.0/255.255.0.0/0/0)
remote ident (addr/mask/prot/port): (#.#.#.#/255.255.255.0/0/0)
current_peer: #.#.#.#

#pkts encaps: 93301, #pkts encrypt: 93301, #pkts digest: 93301
#pkts decaps: 61049, #pkts decrypt: 61049, #pkts verify: 61049
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 93301, #pkts comp failed: 0, #pkts decomp failed: 0
#pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
#PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
#TFC rcvd: 0, #TFC sent: 0
#Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
#send errors: 0, #recv errors: 0

local crypto endpt.: #.#.#.#/0, remote crypto endpt.: #.#.#.#/0
path mtu 1500, ipsec overhead 74(44), media mtu 1500
PMTU time remaining (sec): 0, DF policy: copy-df
ICMP error validation: disabled, TFC packets: disabled
current outbound spi: 55677904
current inbound spi : 7139BA90

inbound esp sas:
spi: 0x7139BA90 (1899608720)
SA State: active
transform: esp-aes-256 esp-sha-hmac no compression
in use settings ={L2L, Tunnel, IKEv1, }
slot: 0, conn_id: 238034944, crypto-map: cm_VPN_Hub
sa timing: remaining key lifetime (kB/sec): (4358528/9091)
IV size: 16 bytes
replay detection support: Y
Anti replay bitmap:
0xFFFFFFFF 0xFFFFFFFF
outbound esp sas:
spi: 0x55677904 (1432844548)
SA State: active
transform: esp-aes-256 esp-sha-hmac no compression
in use settings ={L2L, Tunnel, IKEv1, }
slot: 0, conn_id: 238034944, crypto-map: cm_VPN_Hub
sa timing: remaining key lifetime (kB/sec): (4279867/9091)
IV size: 16 bytes
replay detection support: Y
Anti replay bitmap:
0x00000000 0x00000001

Crypto map tag: cm_VPN_Hub, seq num: 12, local addr: #.#.#.#

access-list cracl_VPN_HeadOff extended permit ip 10.169.0.0 255.255.0.0 #.#.#.# 255.255.224.0
local ident (addr/mask/prot/port): (10.169.0.0/255.255.0.0/0/0)
remote ident (addr/mask/prot/port): (#.#.#.#/255.255.224.0/0/0)
current_peer: #.#.#.#

#pkts encaps: 772633, #pkts encrypt: 772633, #pkts digest: 772633
#pkts decaps: 578899, #pkts decrypt: 578899, #pkts verify: 578899
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 772633, #pkts comp failed: 0, #pkts decomp failed: 0
#pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
#PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 4
#TFC rcvd: 0, #TFC sent: 0
#Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
#send errors: 0, #recv errors: 0

local crypto endpt.: #.#.#.#/0, remote crypto endpt.: #.#.#.#/0
path mtu 1500, ipsec overhead 74(44), media mtu 1500
PMTU time remaining (sec): 0, DF policy: copy-df
ICMP error validation: disabled, TFC packets: disabled
current outbound spi: A7C64889
current inbound spi : 8C90A26C

inbound esp sas:
spi: 0x8C90A26C (2358289004)
SA State: active
transform: esp-aes-256 esp-sha-hmac no compression
in use settings ={L2L, Tunnel, IKEv1, }
slot: 0, conn_id: 238034944, crypto-map: cm_VPN_Hub
sa timing: remaining key lifetime (kB/sec): (4363566/27929)
IV size: 16 bytes
replay detection support: Y
Anti replay bitmap:
0xFFFFFFFF 0xFFFFFFFF
outbound esp sas:
spi: 0xA7C64889 (2814789769)
SA State: active
transform: esp-aes-256 esp-sha-hmac no compression
in use settings ={L2L, Tunnel, IKEv1, }
slot: 0, conn_id: 238034944, crypto-map: cm_VPN_Hub
sa timing: remaining key lifetime (kB/sec): (4347145/27929)
IV size: 16 bytes
replay detection support: Y
Anti replay bitmap:
0x00000000 0x00000001

Crypto map tag: cm_VPN_Hub, seq num: 10, local addr: #.#.#.#

access-list cracl_VPN_RemOff extended permit ip 10.169.0.0 255.255.0.0 #.#.#.# 255.255.224.0
local ident (addr/mask/prot/port): (10.169.0.0/255.255.0.0/0/0)
remote ident (addr/mask/prot/port): (#.#.#.#/255.255.224.0/0/0)
current_peer: #.#.#.#

#pkts encaps: 817123, #pkts encrypt: 817123, #pkts digest: 817123
#pkts decaps: 842034, #pkts decrypt: 842034, #pkts verify: 842034
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 817123, #pkts comp failed: 0, #pkts decomp failed: 0
#pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
#PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
#TFC rcvd: 0, #TFC sent: 0
#Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
#send errors: 0, #recv errors: 0

local crypto endpt.: #.#.#.#/0, remote crypto endpt.: #.#.#.#/0
path mtu 1500, ipsec overhead 74(44), media mtu 1500
PMTU time remaining (sec): 0, DF policy: copy-df
ICMP error validation: disabled, TFC packets: disabled
current outbound spi: F42499E2
current inbound spi : 4DAA6CB9

inbound esp sas:
spi: 0x4DAA6CB9 (1303014585)
SA State: active
transform: esp-aes-256 esp-sha-hmac no compression
in use settings ={L2L, Tunnel, IKEv1, }
slot: 0, conn_id: 194744320, crypto-map: cm_VPN_Hub
sa timing: remaining key lifetime (kB/sec): (3913686/27928)
IV size: 16 bytes
replay detection support: Y
Anti replay bitmap:
0xFFFFFFFF 0xFFFFFFFF
outbound esp sas:
spi: 0xF42499E2 (4096039394)
SA State: active
transform: esp-aes-256 esp-sha-hmac no compression
in use settings ={L2L, Tunnel, IKEv1, }
slot: 0, conn_id: 194744320, crypto-map: cm_VPN_Hub
sa timing: remaining key lifetime (kB/sec): (3910673/27927)
IV size: 16 bytes
replay detection support: Y
Anti replay bitmap:
0x00000000 0x00000001

DC# packet-tracer input lan2 icmp #.#.#.# 8 1 #.#.#.# detail

Phase: 1
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (lan2,inet) source static og_Lan2_Int og_Lan2_Int destination static og_RO_LAN og_RO_LAN no-proxy-arp description NONAT
Additional Information:
NAT divert to egress interface inet
Untranslate #.#.#.#/0 to #.#.#.#/0

Phase: 2
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop #.#.#.# using egress ifc lan2

Phase: 3
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (lan2,inet) source static og_Lan2_Int og_Lan2_Int destination static og_RO_LAN og_RO_LAN no-proxy-arp description NONAT
Additional Information:
Static translate #.#.#.#/0 to #.#.#.#/0
Forward Flow based lookup yields rule:
in id=0x7f0bafb16c30, priority=6, domain=nat, deny=false
hits=17721082, user_data=0x7f0bb034df90, cs_id=0x0, flags=0x0, protocol=0
src ip/id=#.#.#.#, mask=255.255.255.0, port=0, tag=any
dst ip/id=#.#.#.#, mask=255.255.0.0, port=0, tag=any, dscp=0x0
input_ifc=lan2, output_ifc=inet

Phase: 4
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f0baed2e0e0, priority=0, domain=nat-per-session, deny=true
hits=170785688, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 5
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f0bafe5c230, priority=0, domain=inspect-ip-options, deny=true
hits=88327819, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=lan2, output_ifc=any

Phase: 6
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
class-map inspection_default
match default-inspection-traffic
policy-map global_policy
class inspection_default
inspect icmp
service-policy global_policy global
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f0b941bf550, priority=70, domain=inspect-icmp, deny=false
hits=1411965, user_data=0x7f0b94177200, cs_id=0x0, use_real_addr, flags=0x0, protocol=1
src ip/id=0.0.0.0, mask=0.0.0.0, icmp-type=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, icmp-code=0, tag=any, dscp=0x0
input_ifc=lan2, output_ifc=any

Phase: 7
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f0b94177c80, priority=70, domain=inspect-icmp-error, deny=false
hits=1411963, user_data=0x7f0baff144a0, cs_id=0x0, use_real_addr, flags=0x0, protocol=1
src ip/id=0.0.0.0, mask=0.0.0.0, icmp-type=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, icmp-code=0, tag=any, dscp=0x0
input_ifc=lan2, output_ifc=any

Phase: 8
Type: VPN
Subtype: encrypt
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
out id=0x7f0b4d373490, priority=70, domain=encrypt, deny=false
hits=27666, user_data=0x2b7c264, cs_id=0x7f0bafc380d0, reverse, flags=0x0, protocol=0
src ip/id=#.#.#.#, mask=255.255.0.0, port=0, tag=any
dst ip/id=#.#.#.#, mask=255.255.224.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=inet

Phase: 9
Type: ACCESS-LIST
Subtype: filter-aaa
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
out id=0x7f0b6ba1faa0, priority=13, domain=filter-aaa, deny=false
hits=2019, user_data=0x7f0ba35adb00, filter_id=0xb(acl_FW_VPN_Office), protocol=1
src ip=0.0.0.0, mask=0.0.0.0, icmp-type=0
dst ip=0.0.0.0, mask=0.0.0.0, icmp-code=0

Phase: 10
Type: NAT
Subtype: rpf-check
Result: ALLOW
Config:
nat (lan2,inet) source static og_Lan2_Int og_Lan2_Int destination static og_RO_LAN og_RO_LAN no-proxy-arp description NONAT
Additional Information:
Forward Flow based lookup yields rule:
out id=0x7f0bad821e10, priority=6, domain=nat-reverse, deny=false
hits=16923586, user_data=0x7f0baf21bb10, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=#.#.#.#, mask=255.255.255.0, port=0, tag=any
dst ip/id=#.#.#.#, mask=255.255.0.0, port=0, tag=any, dscp=0x0
input_ifc=lan2, output_ifc=inet

Phase: 11
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 167795664, packet dispatched to next module
Module information for forward flow ...
snp_fp_tracer_drop
snp_fp_inspect_ip_options
snp_fp_inspect_icmp
snp_fp_translate
snp_fp_adjacency
snp_fp_encrypt
snp_fp_fragment
snp_ifc_stat

Module information for reverse flow ...

Result:
input-interface: lan2
input-status: up
input-line-status: up
output-interface: inet
output-status: up
output-line-status: up
Action: allow

The encap|decap counters increase when running the command multiple time. Sorry if this is a double-post, as I thought I posted it earlier and went back to the topic and couldn't see my previous post.

MHM Cisco World · ‎01-11-2024

share the
show crypto ipsec sa
packet-tracer you test
show ip route

MHM

ringpowersolutions · ‎01-11-2024

pr-fw-1# packet-tracer input lan2 tcp #.#.#.# https #.#.#.# https detail

Phase: 1
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (lan2,inet) source static og_Lan2_Int og_Lan2_Int destination static og_HO_LAN og_HO_LAN no-proxy-arp description NONAT
Additional Information:
NAT divert to egress interface inet
Untranslate #.#.#.#/443 to #.#.#.#/443

Phase: 2
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop #.#.#.# using egress ifc lan2

Phase: 3
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (lan2,inet) source static og_Lan2_Int og_Lan2_Int destination static og_HO_LAN og_HO_LAN no-proxy-arp description NONAT
Additional Information:
Static translate #.#.#.#/443 to #.#.#.#/443
Forward Flow based lookup yields rule:
in id=0x7f0bafb16c30, priority=6, domain=nat, deny=false
hits=17729209, user_data=0x7f0bb034df90, cs_id=0x0, flags=0x0, protocol=0
src ip/id=#.#.#.#, mask=255.255.255.0, port=0, tag=any
dst ip/id=#.#.#.#, mask=255.255.0.0, port=0, tag=any, dscp=0x0
input_ifc=lan2, output_ifc=inet

Phase: 4
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f0bb09c5950, priority=1, domain=nat-per-session, deny=true
hits=109737307, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 5
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f0bafe5c230, priority=0, domain=inspect-ip-options, deny=true
hits=88383100, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=lan2, output_ifc=any

Phase: 6
Type: FOVER
Subtype: standby-update
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f0bafae30f0, priority=20, domain=lu, deny=false
hits=21553247, user_data=0x0, cs_id=0x0, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=lan2, output_ifc=any

Phase: 7
Type: VPN
Subtype: encrypt
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
out id=0x7f0b4d77d5f0, priority=70, domain=encrypt, deny=false
hits=14500, user_data=0x2b7c264, cs_id=0x7f0bafc380d0, reverse, flags=0x0, protocol=0
src ip/id=#.#.#.#, mask=255.255.0.0, port=0, tag=any
dst ip/id=#.#.#.#, mask=255.255.224.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=inet

Phase: 8
Type: ACCESS-LIST
Subtype: filter-aaa
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
out id=0x7f0b4d382ee0, priority=13, domain=filter-aaa, deny=false
hits=187, user_data=0x7f0ba35a8700, filter_id=0xb(acl_FW_VPN_RemOff), protocol=0
src ip=#.#.#.#, mask=255.255.255.0, port=0
dst ip=#.#.#.#, mask=255.255.255.0, port=0

Phase: 9
Type: NAT
Subtype: rpf-check
Result: ALLOW
Config:
nat (lan2,inet) source static og_Lan2_Int og_Lan2_Int destination static og_HO_LAN og_HO_LAN no-proxy-arp description NONAT
Additional Information:
Forward Flow based lookup yields rule:
out id=0x7f0bad821e10, priority=6, domain=nat-reverse, deny=false
hits=16931479, user_data=0x7f0baf21bb10, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=#.#.#.#, mask=255.255.255.0, port=0, tag=any
dst ip/id=#.#.#.#, mask=255.255.0.0, port=0, tag=any, dscp=0x0
input_ifc=lan2, output_ifc=inet

Phase: 10
Type: VPN
Subtype: ipsec-tunnel-flow
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0x7f0b6b7e8e00, priority=70, domain=ipsec-tunnel-flow, deny=false
hits=14492, user_data=0x22709fc, cs_id=0x7f0bafc380d0, reverse, flags=0x0, protocol=0
src ip/id=#.#.#.#, mask=255.255.224.0, port=0, tag=any
dst ip/id=#.#.#.#, mask=255.255.0.0, port=0, tag=any, dscp=0x0
input_ifc=inet, output_ifc=any

Phase: 11
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0x7f0bb09c5950, priority=1, domain=nat-per-session, deny=true
hits=109737309, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 12
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0x7f0bafaadb10, priority=0, domain=inspect-ip-options, deny=true
hits=170421360, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=inet, output_ifc=any

Phase: 13
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 167869945, packet dispatched to next module
Module information for forward flow ...
snp_fp_tracer_drop
snp_fp_inspect_ip_options
snp_fp_tcp_normalizer
snp_fp_translate
snp_fp_adjacency
snp_fp_encrypt
snp_fp_fragment
snp_ifc_stat

Module information for reverse flow ...
snp_fp_tracer_drop
snp_fp_inspect_ip_options
snp_fp_ipsec_tunnel_flow
snp_fp_translate
snp_fp_tcp_normalizer
snp_fp_adjacency
snp_fp_fragment
snp_ifc_stat

Result:
input-interface: lan2
input-status: up
input-line-status: up
output-interface: inet
output-status: up
output-line-status: up
Action: allow

pr-fw-1# show crypto ipsec sa
interface: inet
Crypto map tag: cm_VPN_Hub, seq num: 12, local addr: #.#.#.#

access-list cracl_VPN_HeadOff extended permit ip #.#.#.# 255.255.0.0 #.#.#.# 255.255.255.0
local ident (addr/mask/prot/port): (#.#.#.#/255.255.0.0/0/0)
remote ident (addr/mask/prot/port): (#.#.#.#/255.255.255.0/0/0)
current_peer: #.#.#.#

#pkts encaps: 118605, #pkts encrypt: 118605, #pkts digest: 118605
#pkts decaps: 77493, #pkts decrypt: 77493, #pkts verify: 77493
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 118605, #pkts comp failed: 0, #pkts decomp failed: 0
#pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
#PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
#TFC rcvd: 0, #TFC sent: 0
#Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
#send errors: 0, #recv errors: 0

local crypto endpt.: #.#.#.#/0, remote crypto endpt.: #.#.#.#/0
path mtu 1500, ipsec overhead 74(44), media mtu 1500
PMTU time remaining (sec): 0, DF policy: copy-df
ICMP error validation: disabled, TFC packets: disabled
current outbound spi: 55677904
current inbound spi : 7139BA90

inbound esp sas:
spi: 0x7139BA90 (1899608720)
SA State: active
transform: esp-aes-256 esp-sha-hmac no compression
in use settings ={L2L, Tunnel, IKEv1, }
slot: 0, conn_id: 238034944, crypto-map: cm_VPN_Hub
sa timing: remaining key lifetime (kB/sec): (4354148/4392)
IV size: 16 bytes
replay detection support: Y
Anti replay bitmap:
0xFFFFFFFF 0xFFFFFFFF
outbound esp sas:
spi: 0x55677904 (1432844548)
SA State: active
transform: esp-aes-256 esp-sha-hmac no compression
in use settings ={L2L, Tunnel, IKEv1, }
slot: 0, conn_id: 238034944, crypto-map: cm_VPN_Hub
sa timing: remaining key lifetime (kB/sec): (4252863/4392)
IV size: 16 bytes
replay detection support: Y
Anti replay bitmap:
0x00000000 0x00000001

Crypto map tag: cm_VPN_Hub, seq num: 12, local addr: #.#.#.#

access-list cracl_VPN_HeadOff extended permit ip #.#.#.# 255.255.0.0 #.#.#.# 255.255.224.0
local ident (addr/mask/prot/port): (#.#.#.#/255.255.0.0/0/0)
remote ident (addr/mask/prot/port): (#.#.#.#/255.255.224.0/0/0)
current_peer: #.#.#.#

#pkts encaps: 908256, #pkts encrypt: 908256, #pkts digest: 908256
#pkts decaps: 699382, #pkts decrypt: 699382, #pkts verify: 699382
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 908256, #pkts comp failed: 0, #pkts decomp failed: 0
#pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
#PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 4
#TFC rcvd: 0, #TFC sent: 0
#Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
#send errors: 0, #recv errors: 0

local crypto endpt.: #.#.#.#/0, remote crypto endpt.: #.#.#.#/0
path mtu 1500, ipsec overhead 74(44), media mtu 1500
PMTU time remaining (sec): 0, DF policy: copy-df
ICMP error validation: disabled, TFC packets: disabled
current outbound spi: A7C64889
current inbound spi : 8C90A26C

inbound esp sas:
spi: 0x8C90A26C (2358289004)
SA State: active
transform: esp-aes-256 esp-sha-hmac no compression
in use settings ={L2L, Tunnel, IKEv1, }
slot: 0, conn_id: 238034944, crypto-map: cm_VPN_Hub
sa timing: remaining key lifetime (kB/sec): (4340615/23232)
IV size: 16 bytes
replay detection support: Y
Anti replay bitmap:
0xFFFFFFFF 0xFFFFFFFF
outbound esp sas:
spi: 0xA7C64889 (2814789769)
SA State: active
transform: esp-aes-256 esp-sha-hmac no compression
in use settings ={L2L, Tunnel, IKEv1, }
slot: 0, conn_id: 238034944, crypto-map: cm_VPN_Hub
sa timing: remaining key lifetime (kB/sec): (4312974/23232)
IV size: 16 bytes
replay detection support: Y
Anti replay bitmap:
0x00000000 0x00000001

Crypto map tag: cm_VPN_Hub, seq num: 10, local addr: #.#.#.#

access-list cracl_VPN_RemOff extended permit ip #.#.#.# 255.255.0.0 10.175.0.0 255.255.224.0
local ident (addr/mask/prot/port): (#.#.#.#/255.255.0.0/0/0)
remote ident (addr/mask/prot/port): (10.175.0.0/255.255.224.0/0/0)
current_peer: #.#.#.#

#pkts encaps: 927623, #pkts encrypt: 927623, #pkts digest: 927623
#pkts decaps: 951430, #pkts decrypt: 951430, #pkts verify: 951430
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 927623, #pkts comp failed: 0, #pkts decomp failed: 0
#pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
#PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
#TFC rcvd: 0, #TFC sent: 0
#Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
#send errors: 0, #recv errors: 0

local crypto endpt.: #.#.#.#/0, remote crypto endpt.: #.#.#.#/0
path mtu 1500, ipsec overhead 74(44), media mtu 1500
PMTU time remaining (sec): 0, DF policy: copy-df
ICMP error validation: disabled, TFC packets: disabled
current outbound spi: F42499E2
current inbound spi : 4DAA6CB9

inbound esp sas:
spi: 0x4DAA6CB9 (1303014585)
SA State: active
transform: esp-aes-256 esp-sha-hmac no compression
in use settings ={L2L, Tunnel, IKEv1, }
slot: 0, conn_id: 194744320, crypto-map: cm_VPN_Hub
sa timing: remaining key lifetime (kB/sec): (3905335/23230)
IV size: 16 bytes
replay detection support: Y
Anti replay bitmap:
0xFFFFFFFF 0xFFFFFFFF
outbound esp sas:
spi: 0xF42499E2 (4096039394)
SA State: active
transform: esp-aes-256 esp-sha-hmac no compression
in use settings ={L2L, Tunnel, IKEv1, }
slot: 0, conn_id: 194744320, crypto-map: cm_VPN_Hub
sa timing: remaining key lifetime (kB/sec): (3881866/23229)
IV size: 16 bytes
replay detection support: Y
Anti replay bitmap:
0x00000000 0x00000001

pr-fw-1# sh route

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, V - VPN
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, + - replicated route
Gateway of last resort is #.#.#.# to network 0.0.0.0

S* 0.0.0.0 0.0.0.0 [1/0] via #.#.#.#, inet
*** OUTPUT OMITTED ***

All other connections in the "show route" command are L (Local) or C (Connected) connections. We only have a default static route, as all traffic goes out of this port:

route inet 0.0.0.0 0.0.0.0 #.#.#.# 1

Rob Ingram · ‎01-11-2024

@ringpowersolutions your replies keep dispearing because you are pasting configuration into the body of the email (they may turn up again later if not permanently blocked).

Your output seems ok, you have a NAT exemption rule in place and the counters are increasing. From the DC ping through to the remote site and take a packet capture on the remote side, is the traffic received? Could there be a local firewall on the devices you are pinging that restricts ping from the DC networks?

MHM Cisco World · ‎01-11-2024

let check routing later
one more point the SPI
user_data=0x2b7c264 <<- this SPI of encryption

nbound esp sas:
spi: 0x4DAA6CB9
outbound esp sas:
spi: 0xF42499E2

so check for which SPI is this, it can there is conflict in ACL of IPsec VPN run in HQ
MHM

MHM Cisco World · ‎01-11-2024

Phase: 2
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop #.#.#.# using egress ifc lan2 <<- next-hop is same as interface the traffic come from ??
it routing issue
first add no route-lookup in NAT
and check the routing table

MHM

ringpowersolutions · ‎01-11-2024

@Rob Ingram packet captures to the remote site appears to be coming through. I can see the packets increasing in both directions when run on the internal interfaces of the remote site, and I ran the command as both capture NAME interface INTERNAL match ip DC_IP RS_IP and capture NAME interface INTERNAL match ip RS_IP DC_IP to see the results and both reporting traffic travelling through.

The only other security past these ASAs on the remote sites would be 9200L switch ACLs. These ACLs are on VLANs specifically and the VLAN I am currently testing has an explicit permit any any before any deny rules, specifically to rule the ACLs out of being the cause of this issue.

ringpowersolutions · ‎01-11-2024

Sorry, correction, traffic is not being received. The information was skewed due to me running tests to a server that I had other processes communicating with it at the same time. I have redone the tests to a server in the DC that there is no communications from the test machine and am seeing no packets at the remote site.

Rob Ingram · ‎01-11-2024

@ringpowersolutions and if you capture on the inside interface of the ASA closest to the source, is the traffic received by that local firewall? What is the routing table of the firewall closest to the source?

packet-tracer input lan2 icmp #.#.#.# 8 1 #.#.#.# detail

Your packet-tracer is incorrect, using 8 1 would result in a bad code error (on my ASA code at least) use 8 0 - provide the output. Your IP addresses are masked, what are these IP addresses you are using? Devices behind the firewalls?

Regardless I assume you've tried with real traffic and there is still a problem?

ringpowersolutions · ‎01-11-2024

@Rob Ingram No idea why I was using 1 for that test instead of 0. Test result is almost the same, showing a couple of extra steps from above and the result being that it is allowed to continue via VPN. Results attached.

I am testing with DC IP 10.169.2.69 and Head Office IP 10.168.12.105. I have tried ping tests from 10.169.2.69 to 10.168.12.105 and that is failing. Ping tests from 10.168.12.105 to 10.169.2.69 work without issue. These devices sit behind their respective site's ASA devices.

MHM Cisco World · ‎01-11-2024

user_data=0x2b7c264 <<- same SPI appear in all your packet tracer test

show crypto ipsec sa
check for which tunnel this SPI
then check it ACL with your issue VPN acl
it can sometime we use subnet mask wrong and VPN is conflict
MHM

ringpowersolutions · ‎01-11-2024

@MHmh sorry for not responding to your previous message about that, but it did not load when I was posting earlier responses.

Thank you for this information, as it appears that none of the active IPSec tunnels at either the DC or Head Office has this SPI that you have highlighted. I assume that if I ran clear ipsec sa on the DC, it should resolve this issue? We have a few hours left before I can run the command, but will run it when people are not working and report back tomorrow.

MHM Cisco World · ‎01-11-2024

clear crypto ipsec sa inactive <<- use this instead
and check this to see for which this hex is

ASA# SHOW ASP TABLE VPN-CONTEXT DETAIL

MHM