Strange traffic coming from far end VPN peer

[asidd]

We enabled Solarwinds NetPath to monitor a 443 service over a site-to-site VPN between our ASA 5516-X and client's Checkpoint 80.30. As soon as we enable the probe we noticed a very strange traffic coming from the far-end and then the tunnel would flap at random times causing outages to production. It was very hard to find why this was happening but after some painstaking investigations we found that it was definitely the Probe causing it and we replicated it by shutting it down and restarting it to make sure this was the root cause.

The site-to-site tunnel is working fine otherwise and we have no other problems and it happens only after we start the probe. Any ideas on why this would happen? I have put the logs below and masked the IP addresses only. There is no IPv6 enabled on either their end or ours.

Nov 30 2020 13:56:44: %ASA-4-402116: IPSEC: Received an ESP packet (SPI= 0x07CA7F97, sequence number= 0x19017) from X.X.X.X (user= X.X.X.X) to X.X.X.X.  The decapsulated inner packet doesn't match the negotiated policy in the SA.  The packet specifies its destination as 3bf1:c7ed:152:bf0f:661d:f65a:20a6:95d, its source as cc07:1911:88a5:a1b0:b694:3023:60ea:ab7d, and its protocol as 255.  The SA specifies its local proxy as X.X.X.X/255.255.255.0/ip/0 and its remote_proxy as X.X.X.X/255.255.255.255/ip/0.

Nov 30 2020 13:56:45: %ASA-4-402116: IPSEC: Received an ESP packet (SPI= 0x07CA7F97, sequence number= 0x194AB) from X.X.X.X (user= X.X.X.X) to X.X.X.X.  The decapsulated inner packet doesn't match the negotiated policy in the SA.  The packet specifies its destination as e609:b541:26fb:396a:8ae2:1694:7fed:e54, its source as 8314:3e4b:3f68:157a:44d2:c65c:4207:fc2d, and its protocol as 255.  The SA specifies its local proxy as X.X.X.X/255.255.255.0/ip/0 and its remote_proxy as X.X.X.X/255.255.255.255/ip/0.

Nov 30 2020 13:56:49: %ASA-6-302016: Teardown UDP connection 514110655 for INSIDE01:X.X.X.X/4500 to identity:X.X.X.X/4500 duration 99:43:42 bytes 23197935182

Nov 30 2020 13:56:49: %ASA-5-750007: Local:X.X.X.X:4500 Remote:203.171.207.4:4500 Username:X.X.X.X IKEv2 SA DOWN. Reason: unknown

[Rob Ingram]

@asidd 

What is the source and destination IP addresses that you've filtered?

What is the local and remote network(s) in the crypto ACL that defines the interesting traffic?

Is the source and destination IP addresses defined in the crypto ACL?

Do you have a NAT exemption rule to ensure the traffic is not unintentially natted?

[asidd]

Hi Rob,

Yes we have local and remote interesting traffic defined in CMAP ACL. 

ACE Entry 1: Local: 172.16.102.0/24

             Remote: IP1, IP2, IP3, IP4 (different network to the first and defined by individual IPs instead of a network)

ACE Entry 2: Local: 172.16.102.0/24

             Remote: 10.246.180.0/24

NAT exempt is not configured on the VPN configuration because we have an internet firewall and an internal firewall (both ASA 5516-X) and we use a public IP on the former with a static configuration. Internet firewall (IP) NAT to OUTSIDE interface of internal firewall.

The probe is really like any other traffic from L2L. Thinking why would it matter if it is a Probe traffic or actual production traffic? We have production working just fine until we enabled a probe from 172.16.102.29.

Regards

Atif

[Rob Ingram]

Whatever these different remote host networks you are using are, is an IPSec SA established? Run "show crypto ipsec sa" and determine whether SA has been estalbished and if encaps|decaps are increasing.

If NAT configured on the ASA, traffic could be unintentially being natted - you'll need to confirm. Run packet-tracer and provide the output.