10-13-2021 05:58 AM
Hello everybody,
our customer has a S2S VPN-Tunnel with his Firepower 1120 (7.0.0.1) and a remote Sophos XG.
They can communicate through the tunnel but causes error message in the IKEv1 debug output
and the IPSec counters stays 0. The counters of other tunnels on this Firepower 1120
count the encrypted and decrypted packets normally.
Own IP address is 80.148.37.11 and peer IP address is 93.104.193.53.
debug crypto ipsec 255:
... IPSEC ERROR: Asynchronous operation failed, SPI: 0xBFEE3C18, user: 93.104.193.53, peer: 93.104.193.53, error: 0x01, ctm_ipsec_sa_async_processing:78 IPSEC ERROR: Failed to free SPI 0xBFEE3C18 IPSEC ERROR: Failed to complete the DELETE SA command from IKE IPSEC ERROR: Failed to send the message to IKE IPSEC ERROR: Failed to send the message to IKE IPSEC ERROR: Failed to send the message to IKE IPSEC ERROR: Failed to send the message to IKE IPSEC ERROR: Failed to send the message to IKE IPSEC ERROR: Failed to send the message to IKE IPSEC ERROR: Failed to send the message to IKE IPSEC ERROR: Failed to send the message to IKE IPSEC ERROR: Failed to send the message to IKE IPSEC ERROR: Failed to send the message to IKE IPSEC ERROR: Failed to send the message to IKE IPSEC DEBUG: outbound SA (SPI 0xC0F44FD3, Handle 0x91C445C5) state change from active to dead IPSEC INFO: Setting an IPSec timer of type SA Purge Timer for 30 seconds with a jitter value of 0 IPSEC INFO: IPSec SA PURGE timer started SPI 0x91C445C5 IPSEC DEBUG: inbound SA (SPI 0x3C537DF4, Handle 0x87FB4DFF) state change from active to dead IPSEC INFO: Setting an IPSec timer of type SA Purge Timer for 30 seconds with a jitter value of 0 IPSEC INFO: IPSec SA PURGE timer started SPI 0x87FB4DFF IPSEC: Received a PFKey message from IKE IPSEC DEBUG: Received a DELETE PFKey message from IKE for an inbound SA (SPI 0x3C537DF4) IPSEC DEBUG: Outbound SA (SPI 0xC0F44FD3) destroy started, state dead IPSEC: Destroy current outbound SPI: 0xC0F44FD3 IPSEC DEBUG: Outbound SA (SPI 0xC0F44FD3) free started, state dead IPSEC DEBUG: Outbound SA (SPI 0xC0F44FD3, Handle 0x91C445C5) state change from dead to dead IPSEC INFO: Destroying an IPSec timer of type SA Purge Timer IPSEC INFO: Setting an IPSec timer of type SA Purge Timer for 30 seconds with a jitter value of 0 IPSEC INFO: IPSec SA PURGE timer started SPI 0x91C445C5 IPSEC DEBUG: Deleting the outbound encrypt rule for SPI 0xC0F44FD3 IPSEC: Increment SA NP ref counter for outbound SPI 0xC0F44FD3, old value: 0, new value: 1, (ctm_ipsec_delete_acl_entry:6841) IPSEC: Deleted outbound encrypt rule, SPI 0xC0F44FD3 Rule ID: 0x0000150a764a2b90 ...
firepower# sh cry ipsec sa peer 93.104.193.53
peer address: 93.104.193.53 Crypto map tag: CSM_outside_map, seq num: 6, local addr: 80.148.37.11 access-list CSM_IPSEC_ACL_1 extended permit ip 192.168.18.0 255.255.255.0 172.17.0.0 255.255.255.0 local ident (addr/mask/prot/port): (192.168.18.0/255.255.255.0/0/0) remote ident (addr/mask/prot/port): (172.17.0.0/255.255.255.0/0/0) current_peer: 93.104.193.53 #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0 #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0 #pkts compressed: 0, #pkts decompressed: 0 #pkts not compressed: 0, #pkts comp failed: 0, #pkts decomp failed: 0 #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0 #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0 #TFC rcvd: 0, #TFC sent: 0 #Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0 #send errors: 0, #recv errors: 0 local crypto endpt.: 80.148.37.11/0, remote crypto endpt.: 93.104.193.53/0 path mtu 1500, ipsec overhead 74(44), media mtu 1500 PMTU time remaining (sec): 0, DF policy: copy-df ICMP error validation: disabled, TFC packets: disabled current outbound spi: C7E0EFC1 current inbound spi : 9B3B0B6C
In the IKEv1 debug I don't see any problem.
I also generated a VPN Troubleshooting file in FMC.
I searched in the WWW but the outcome was not useful.
All outputs are attached.
How can I remove the error messages in the IPSec debug and let the counters increase?
Thanks a lot for every hint!
Bye
R.
Solved! Go to Solution.
10-14-2021 03:07 AM
Hi @swscco001 Your debug level was set to the very highest level, so that's probably background chatter. I don't see any errors in that output, so I expect its not causing an issue.
10-13-2021 06:05 AM - edited 10-13-2021 06:35 AM
You appear to have inbound and outbound esp SAs, so the VPN appears to be established. Do you have a NAT exemption rule to ensure VPN traffic is not unintentially translated? This is usually why you've not encrypted any outbound traffic, as after translation it does not match the crypto ACL that defines the interesting traffic.
Run packet-tracer from the CLI and provide the output for review.
10-13-2021 07:09 AM
Hi Rob,
thanks for your fast reaction!
There are no NAT entries in the NAT screen of the FMC and the CLI.
I entered the following packet-tracer command:
firepower# packet-tracer input inside icmp 192.168.100.24 8 0 172.17.0.33 Phase: 1 Type: ROUTE-LOOKUP Subtype: No ECMP load balancing Result: ALLOW Config: Additional Information: Destination is locally connected. No ECMP load balancing. Found next-hop 172.17.0.33 using egress ifc outside(vrfid:0) Phase: 2 Type: ACCESS-LIST Subtype: log Result: ALLOW Config: access-group CSM_FW_ACL_ global access-list CSM_FW_ACL_ advanced permit ip ifc inside object-group VPN-Mutterhaus-LOCAL ifc outside object-group VPN-Mutterhaus-REMOTE rule-id 268442624 access-list CSM_FW_ACL_ remark rule-id 268442624: ACCESS POLICY: NWB-VPN-FW - Default access-list CSM_FW_ACL_ remark rule-id 268442624: L7 RULE: NWB -> Mutterhaus object-group network VPN-Mutterhaus-LOCAL network-object object nw-192.168.110.0-24_NWB-CTX-WORKER network-object object nw-192.168.18.0-24_NWB-MANAGEMENT network-object object nw-192.168.100.0-24_NWB-SERVER network-object object nw-192.168.55.0-24 network-object object nw-192.168.105.0-24_NWB-dmz-mail object-group network VPN-Mutterhaus-REMOTE network-object object n-172.21.1.0_24 network-object object n-172.17.8.0_24 network-object object n-172.17.16.0_24 network-object object n-172.18.16.0_24 network-object object n-172.20.0.0_24 network-object object n-172.20.16.0_24 network-object object n-172.21.3.0_24 network-object object n-172.19.16.0_24 network-object object n-172.18.0.0_24 network-object object n-172.21.2.0_24 network-object object n-172.19.0.0_24 network-object object n-172.17.0.0_24 network-object object n-172.17.98.0_24 network-object object n-172.21.8.0_24 network-object object n-172.21.9.0_24 network-object object n-172.21.4.0_24 network-object object n-172.20.8.0_24 network-object object n-172.19.8.0_24 Additional Information: This packet will be sent to snort for additional processing where a verdict will be reached Phase: 3 Type: CONN-SETTINGS Subtype: Result: ALLOW Config: class-map class-default match any policy-map global_policy class class-default set connection advanced-options UM_STATIC_TCP_MAP service-policy global_policy global Additional Information: Phase: 4 Type: NAT Subtype: per-session Result: ALLOW Config: Additional Information: Phase: 5 Type: IP-OPTIONS Subtype: Result: ALLOW Config: Additional Information: Phase: 6 Type: ACCESS-LIST Subtype: log Result: ALLOW Config: access-group CSM_FW_ACL_ global access-list CSM_FW_ACL_ advanced permit ip ifc inside object-group VPN-Mutterhaus-LOCAL ifc outside object-group VPN-Mutterhaus-REMOTE rule-id 268442624 access-list CSM_FW_ACL_ remark rule-id 268442624: ACCESS POLICY: NWB-VPN-FW - Default access-list CSM_FW_ACL_ remark rule-id 268442624: L7 RULE: NWB -> Mutterhaus object-group network VPN-Mutterhaus-LOCAL network-object object nw-192.168.110.0-24_NWB-CTX-WORKER network-object object nw-192.168.18.0-24_NWB-MANAGEMENT network-object object nw-192.168.100.0-24_NWB-SERVER network-object object nw-192.168.55.0-24 network-object object nw-192.168.105.0-24_NWB-dmz-mail object-group network VPN-Mutterhaus-REMOTE network-object object n-172.21.1.0_24 network-object object n-172.17.8.0_24 network-object object n-172.17.16.0_24 network-object object n-172.18.16.0_24 network-object object n-172.20.0.0_24 network-object object n-172.20.16.0_24 network-object object n-172.21.3.0_24 network-object object n-172.19.16.0_24 network-object object n-172.18.0.0_24 network-object object n-172.21.2.0_24 network-object object n-172.19.0.0_24 network-object object n-172.17.0.0_24 network-object object n-172.17.98.0_24 network-object object n-172.21.8.0_24 network-object object n-172.21.9.0_24 network-object object n-172.21.4.0_24 network-object object n-172.20.8.0_24 network-object object n-172.19.8.0_24 Additional Information: This packet will be sent to snort for additional processing where a verdict will be reached Phase: 7 Type: CONN-SETTINGS Subtype: Result: ALLOW Config: class-map class-default match any policy-map global_policy class class-default set connection advanced-options UM_STATIC_TCP_MAP service-policy global_policy global Additional Information: Phase: 8 Type: NAT Subtype: per-session Result: ALLOW Config: Additional Information: Phase: 9 Type: IP-OPTIONS Subtype: Result: ALLOW Config: Additional Information: Phase: 10 Type: ACCESS-LIST Subtype: log Result: ALLOW Config: access-group CSM_FW_ACL_ global access-list CSM_FW_ACL_ advanced permit ip ifc inside object-group VPN-Mutterhaus-LOCAL ifc outside object-group VPN-Mutterhaus-REMOTE rule-id 268442624 access-list CSM_FW_ACL_ remark rule-id 268442624: ACCESS POLICY: NWB-VPN-FW - Default access-list CSM_FW_ACL_ remark rule-id 268442624: L7 RULE: NWB -> Mutterhaus object-group network VPN-Mutterhaus-LOCAL network-object object nw-192.168.110.0-24_NWB-CTX-WORKER network-object object nw-192.168.18.0-24_NWB-MANAGEMENT network-object object nw-192.168.100.0-24_NWB-SERVER network-object object nw-192.168.55.0-24 network-object object nw-192.168.105.0-24_NWB-dmz-mail object-group network VPN-Mutterhaus-REMOTE network-object object n-172.21.1.0_24 network-object object n-172.17.8.0_24 network-object object n-172.17.16.0_24 network-object object n-172.18.16.0_24 network-object object n-172.20.0.0_24 network-object object n-172.20.16.0_24 network-object object n-172.21.3.0_24 network-object object n-172.19.16.0_24 network-object object n-172.18.0.0_24 network-object object n-172.21.2.0_24 network-object object n-172.19.0.0_24 network-object object n-172.17.0.0_24 network-object object n-172.17.98.0_24 network-object object n-172.21.8.0_24 network-object object n-172.21.9.0_24 network-object object n-172.21.4.0_24 network-object object n-172.20.8.0_24 network-object object n-172.19.8.0_24 Additional Information: This packet will be sent to snort for additional processing where a verdict will be reached Phase: 11 Type: CONN-SETTINGS Subtype: Result: ALLOW Config: class-map class-default match any policy-map global_policy class class-default set connection advanced-options UM_STATIC_TCP_MAP service-policy global_policy global Additional Information: Phase: 12 Type: NAT Subtype: per-session Result: ALLOW Config: Additional Information: Phase: 13 Type: IP-OPTIONS Subtype: Result: ALLOW Config: Additional Information: Phase: 14 Type: ACCESS-LIST Subtype: log Result: ALLOW Config: access-group CSM_FW_ACL_ global access-list CSM_FW_ACL_ advanced permit ip ifc inside object-group VPN-Mutterhaus-LOCAL ifc outside object-group VPN-Mutterhaus-REMOTE rule-id 268442624 access-list CSM_FW_ACL_ remark rule-id 268442624: ACCESS POLICY: NWB-VPN-FW - Default access-list CSM_FW_ACL_ remark rule-id 268442624: L7 RULE: NWB -> Mutterhaus object-group network VPN-Mutterhaus-LOCAL network-object object nw-192.168.110.0-24_NWB-CTX-WORKER network-object object nw-192.168.18.0-24_NWB-MANAGEMENT network-object object nw-192.168.100.0-24_NWB-SERVER network-object object nw-192.168.55.0-24 network-object object nw-192.168.105.0-24_NWB-dmz-mail object-group network VPN-Mutterhaus-REMOTE network-object object n-172.21.1.0_24 network-object object n-172.17.8.0_24 network-object object n-172.17.16.0_24 network-object object n-172.18.16.0_24 network-object object n-172.20.0.0_24 network-object object n-172.20.16.0_24 network-object object n-172.21.3.0_24 network-object object n-172.19.16.0_24 network-object object n-172.18.0.0_24 network-object object n-172.21.2.0_24 network-object object n-172.19.0.0_24 network-object object n-172.17.0.0_24 network-object object n-172.17.98.0_24 network-object object n-172.21.8.0_24 network-object object n-172.21.9.0_24 network-object object n-172.21.4.0_24 network-object object n-172.20.8.0_24 network-object object n-172.19.8.0_24 Additional Information: This packet will be sent to snort for additional processing where a verdict will be reached Phase: 15 Type: CONN-SETTINGS Subtype: Result: ALLOW Config: class-map class-default match any policy-map global_policy class class-default set connection advanced-options UM_STATIC_TCP_MAP service-policy global_policy global Additional Information: Phase: 16 Type: NAT Subtype: per-session Result: ALLOW Config: Additional Information: Phase: 17 Type: IP-OPTIONS Subtype: Result: ALLOW Config: Additional Information: Phase: 18 Type: INSPECT Subtype: np-inspect Result: ALLOW Config: class-map inspection_default match default-inspection-traffic policy-map global_policy class inspection_default inspect icmp service-policy global_policy global Additional Information: Phase: 19 Type: INSPECT Subtype: np-inspect Result: ALLOW Config: Additional Information: Phase: 20 Type: DEBUG-ICMP Subtype: Result: ALLOW Config: Additional Information: Phase: 21 Type: VPN Subtype: encrypt Result: ALLOW Config: Additional Information: Phase: 22 Type: VPN Subtype: ipsec-tunnel-flow Result: ALLOW Config: Additional Information: Phase: 23 Type: DEBUG-ICMP Subtype: Result: ALLOW Config: Additional Information: Phase: 24 Type: NAT Subtype: per-session Result: ALLOW Config: Additional Information: Phase: 25 Type: IP-OPTIONS Subtype: Result: ALLOW Config: Additional Information: Phase: 26 Type: FLOW-CREATION Subtype: Result: ALLOW Config: Additional Information: New flow created with id 183476054, packet dispatched to next module Phase: 27 Type: EXTERNAL-INSPECT Subtype: Result: ALLOW Config: Additional Information: Application: 'SNORT Inspect' Phase: 28 Type: SNORT Subtype: Result: ALLOW Config: Additional Information: Snort Trace: Packet: ICMP Session: new snort session Firewall: allow rule, id 268442624, allow Snort id 1, NAP id 4, IPS id 2, Verdict PASS Snort Verdict: (pass-packet) allow this packet Result: input-interface: inside(vrfid:0) input-status: up input-line-status: up Action: allow
It seems to be ok and I don't wonder because the users on the remote side can
access their targets on local side and other S2S tunnels work normally.
The question is why we get these IPSec error messages and the counters stay 0?
Do you have still any other idea or need further information to say more?
Thanks a lot!
10-13-2021 07:17 AM
@swscco001 is traffic even routed to this firewall? Run a packet capture on the inside interface to confirm if the FTD even receives the traffic
10-14-2021 01:51 AM
Hi Rob,
I asked the customer to run a permanent ping from 192.168.100.25 to 172.17.0.33 and captured (attached) it
on the inside interface and all is looking fine. So packets are arriving and were replied.
But the counters stay 0:
firepower# sh cry ipsec sa peer 93.104.193.53 peer address: 93.104.193.53 Crypto map tag: CSM_outside_map, seq num: 6, local addr: 80.148.37.11 access-list CSM_IPSEC_ACL_1 extended permit ip 192.168.18.0 255.255.255.0 172.20.16.0 255.255.255.0 local ident (addr/mask/prot/port): (192.168.18.0/255.255.255.0/0/0) remote ident (addr/mask/prot/port): (172.20.16.0/255.255.255.0/0/0) current_peer: 93.104.193.53 #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0 #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0 #pkts compressed: 0, #pkts decompressed: 0 #pkts not compressed: 0, #pkts comp failed: 0, #pkts decomp failed: 0 #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0 #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0 #TFC rcvd: 0, #TFC sent: 0 #Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0 #send errors: 0, #recv errors: 0 local crypto endpt.: 80.148.37.11/0, remote crypto endpt.: 93.104.193.53/0 path mtu 1500, ipsec overhead 74(44), media mtu 1500 PMTU time remaining (sec): 0, DF policy: copy-df ICMP error validation: disabled, TFC packets: disabled current outbound spi: C7CAA570 current inbound spi : 481FEF21 ...
And the debug output is again:
firepower# debug crypto ipsec 255 firepower# IPSEC ERROR: Failed to send the message to IKE IPSEC INFO: IPSec SA Purge timer expired SPI 0xB58BF5C5 IPSEC INFO: Destroying an IPSec timer of type SA Purge Timer IPSEC: Received a PFKey message from IKE IPSEC DEBUG: Inbound SA (SPI 0xBFEE3C18) destroy started, state dead IPSEC: Destroy current inbound SPI: 0xBFEE3C18 IPSEC DEBUG: Inbound SA (SPI 0xBFEE3C18) free started, state dead IPSEC DEBUG: Inbound SA (SPI 0xBFEE3C18, Handle 0xB58BF5C5) state change from dead to dead IPSEC INFO: Setting an IPSec timer of type SA Purge Timer for 30 seconds with a jitter value of 0 IPSEC INFO: IPSec SA PURGE timer started SPI 0xB58BF5C5 IPSEC ERROR: Asynchronous operation failed, SPI: 0xBFEE3C18, user: 93.104.193.53, peer: 93.104.193.53, error: 0x01, ctm_ipsec_sa_async_processing:78 IPSEC ERROR: Failed to free SPI 0xBFEE3C18 IPSEC ERROR: Failed to complete the DELETE SA command from IKE IPSEC ERROR: Failed to send the message to IKE IPSEC ERROR: Failed to send the message to IKE IPSEC ERROR: Failed to send the message to IKE IPSEC ERROR: Failed to send the message to IKE IPSEC ERROR: Failed to send the message to IKE IPSEC ERROR: Failed to send the message to IKE IPSEC ERROR: Failed to send the message to IKE IPSEC ERROR: Failed to send the message to IKE IPSEC ERROR: Failed to send the message to IKE IPSEC INFO: IPSec SA Purge timer expired SPI 0xB58BF5C5 IPSEC INFO: Destroying an IPSec timer of type SA Purge Timer IPSEC: Received a PFKey message from IKE IPSEC DEBUG: Inbound SA (SPI 0xBFEE3C18) destroy started, state dead IPSEC: Destroy current inbound SPI: 0xBFEE3C18 IPSEC DEBUG: Inbound SA (SPI 0xBFEE3C18) free started, state dead IPSEC DEBUG: Inbound SA (SPI 0xBFEE3C18, Handle 0xB58BF5C5) state change from dead to dead IPSEC INFO: Setting an IPSec timer of type SA Purge Timer for 30 seconds with a jitter value of 0 IPSEC INFO: IPSec SA PURGE timer started SPI 0xB58BF5C5 IPSEC ERROR: Asynchronous operation failed, SPI: 0xBFEE3C18, user: 93.104.193.53, peer: 93.104.193.53, error: 0x01, ctm_ipsec_sa_async_processing:78 IPSEC ERROR: Failed to free SPI 0xBFEE3C18 IPSEC ERROR: Failed to complete the DELETE SA command from IKE IPSEC ERROR: Failed to send the message to IKE IPSEC ERROR: Failed to send the message to IKE IPSEC ERROR: Failed to send the message to IKE IPSEC ERROR: Failed to send the message to IKE IPSEC ERROR: Failed to send the message to IKE IPSEC ERROR: Failed to send the message to IKE
It's hard to understand what the reason could be. I assume there is a misconfiguration in the tunnel parameters that is not hard enough to make the tunnel impossible but have those effects.
Do you still have any idea?
Thanks a lot!
Bye
R.
10-14-2021 01:59 AM
@swscco001 you've provided the output of the IPSec SA of 192.168.18.0 to 172.20.16.0
local ident (addr/mask/prot/port): (192.168.18.0/255.255.255.0/0/0) remote ident (addr/mask/prot/port): (172.20.16.0/255.255.255.0/0/0)
...but the ping was run between 192.168.100.25 to 172.17.0.33, so there would be another IPSec SA for that communication, which would hopefully have counters increasing.
10-14-2021 02:59 AM
Hi Rob,
you are right - i looked at the wrong place in a very long output.
This is the ribt place:
firepower# sh cry ipsec sa peer 93.104.193.53
...
Crypto map tag: CSM_outside_map, seq num: 6, local addr: 80.148.37.11 access-list CSM_IPSEC_ACL_1 extended permit ip 192.168.100.0 255.255.255.0 172.17.0.0 255.255.255.0 local ident (addr/mask/prot/port): (192.168.100.0/255.255.255.0/0/0) remote ident (addr/mask/prot/port): (172.17.0.0/255.255.255.0/0/0) current_peer: 93.104.193.53 #pkts encaps: 13892379, #pkts encrypt: 13892379, #pkts digest: 13892379 #pkts decaps: 7179274, #pkts decrypt: 7179274, #pkts verify: 7179274 #pkts compressed: 0, #pkts decompressed: 0 #pkts not compressed: 13892379, #pkts comp failed: 0, #pkts decomp failed: 0 #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0 #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0 #TFC rcvd: 0, #TFC sent: 0 #Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0 #send errors: 0, #recv errors: 0
...
But are the debug error messages regarding IPSec to this peer IP normal and not serious?:
...IPSEC ERROR: Asynchronous operation failed, SPI: 0xBFEE3C18, user: 93.104.193.53, peer: 93.104.193.53, error: 0x01, ctm_ipsec_sa_async_processing:78 IPSEC ERROR: Failed to free SPI 0xBFEE3C18 IPSEC ERROR: Failed to complete the DELETE SA command from IKE IPSEC ERROR: Failed to send the message to IKE IPSEC ERROR: Failed to send the message to IKE IPSEC ERROR: Failed to send the message to IKE ...
I have to explain this to the customer ...
Thanks a lot!
Bye
R.
10-14-2021 03:07 AM
Hi @swscco001 Your debug level was set to the very highest level, so that's probably background chatter. I don't see any errors in that output, so I expect its not causing an issue.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide