Hi,
We're trying to debug some weird TCP behaviour using a site-to-site IPSec tunnel.
Our end: Cisco ASA 5506-X 9.6(4)29
Their end: unknown Checkpoint
We've been happily running this IPSec tunnel for 4+ years and suddenly, after the other end did some maintenance, we started seeing this behaviour:
* Starting with the VPN tunnel established after an ASA reload at our end, the TCP traffic is flowing back and forth no problems through the tunnel.
* After an IPSec rekey, with the tunnel still established, all connections initiated from our end are timing out with ASA log entries like:
%ASA-6-302013: Built outbound TCP connection 30004 for ge6outside:remoteip/rport (remoteip/rport) to ge5inside:localip/lport (localip/lport)
%ASA-6-302014: Teardown TCP connection 30004 for ge6outside:remoteip/rport (remoteip/rport) to ge5inside:localip/lport duration 0:00:30 bytes 0 SYN Timeout
When in this state, running a traffic capture shows ISAKMP traffic both ways on port 500 between our ASA and the other end every 10s, give or take. However, we only see outbound traffic on the tunnel from us to them, nothing coming back from the other end. Sanitized capture sample:
ISAKMP traffic
10:35:01.283277 IP asa.pub.lic.ip.500 > chekpoint.pub.lic.ip.500: isakmp: child_sa inf2[I]
10:35:01.297762 IP chekpoint.pub.lic.ip.500 > asa.pub.lic.ip.500: isakmp: child_sa inf2[R]
10:35:11.282454 IP asa.pub.lic.ip.500 > chekpoint.pub.lic.ip.500: isakmp: child_sa inf2[I]
10:35:11.297320 IP chekpoint.pub.lic.ip.500 > asa.pub.lic.ip.500: isakmp: child_sa inf2[R]
10:35:21.281507 IP asa.pub.lic.ip.500 > chekpoint.pub.lic.ip.500: isakmp: child_sa inf2[I]
10:35:21.296057 IP chekpoint.pub.lic.ip.500 > asa.pub.lic.ip.500: isakmp: child_sa inf2[R]
10:35:31.280534 IP asa.pub.lic.ip.500 > chekpoint.pub.lic.ip.500: isakmp: child_sa inf2[I]
10:35:31.294999 IP chekpoint.pub.lic.ip.500 > asa.pub.lic.ip.500: isakmp: child_sa inf2[R]
IPSec traffic:
10:35:06.528577 IP asa.pub.lic.ip > chekpoint.pub.lic.ip: ESP(spi=0x937cc142,seq=0x8f), length 104
10:35:14.848437 IP asa.pub.lic.ip > chekpoint.pub.lic.ip: ESP(spi=0x937cc142,seq=0x90), length 104
10:35:30.401018 IP asa.pub.lic.ip > chekpoint.pub.lic.ip: ESP(spi=0x937cc142,seq=0x91), length 104
10:35:38.592527 IP asa.pub.lic.ip > chekpoint.pub.lic.ip: ESP(spi=0x937cc142,seq=0x92), length 104
10:35:46.913631 IP asa.pub.lic.ip > chekpoint.pub.lic.ip: ESP(spi=0x937cc142,seq=0x93), length 104
10:36:27.294984 IP asa.pub.lic.ip > chekpoint.pub.lic.ip: ESP(spi=0x937cc142,seq=0x94), length 104
10:36:28.292302 IP asa.pub.lic.ip > chekpoint.pub.lic.ip: ESP(spi=0x937cc142,seq=0x95), length 104
* The TCP connection between localip (ours) and remoteip (theirs) starts to work again after the tunnel gets recreated.
Notes:
- Not every rekey causes the issue. Sometimes we see rekey events logged by ASA and the TCP traffic still works fine. But more often than not a rekey will cause the problem.
- When the TCP connections through the tunnel stop working our ASA router shows:
ASA# show crypto ipsec sa
interface: ge6outside
Crypto map tag: ikev2-map-MYVPN1, seq num: 2, local addr: asa.pub.lic.ip
access-list ikev2-list-MYVPN1 extended permit ip localnet 255.255.255.0 remotenet 255.255.255.0
local ident (addr/mask/prot/port): (localnet/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (remotenet/255.255.255.0/0/0)
current_peer: chekpoint.pub.lic.ip
#pkts encaps: 4, #pkts encrypt: 4, #pkts digest: 4
#pkts decaps: 3, #pkts decrypt: 3, #pkts verify: 3
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 4, #pkts comp failed: 0, #pkts decomp failed: 0
#pre-frag successes: 0, #pre-frag failures: 0, #fragments created:
0
#PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
#TFC rcvd: 0, #TFC sent: 0
#Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
#send errors: 0, #recv errors: 0
local crypto endpt.: asa.pub.lic.ip/500, remote crypto endpt.: chekpoint.pub.lic.ip/500
path mtu 1500, ipsec overhead 78(44), media mtu 1500
PMTU time remaining (sec): 0, DF policy: copy-df
ICMP error validation: disabled, TFC packets: disabled
current outbound spi: F71A95B8
current inbound spi : 0B2E4BB0
ASA# show crypto ipsec stats | inc ytes|crypt
Bytes: 502410
Decompressed bytes: 502410
Decryptions: 6143
Decryption failures: 0
Bytes: 1311463
Uncompressed bytes: 1311463
Encryptions: 20461
Encryption failures: 0
I don't know at the moment what can cause the issue and we can't rely on complete and accurate communicate with the team at other end,
so we're trying to gather as much information as we can from our end.
I'm not familiar with how Checkpoint devices work, so this question might look funny, but: is there something in the Checkpoint device
configuration that can cause IP routing / NAT changes upon a tunnel rekey? If feels like some event at the other end, caused or following a
rekey, leads to all our SYN packets to be dropped and there could be multiple reasons for this. VPN acceleration could be one, what other
reasons we could / should consider?
Wondering what your thoughts are. Thanks.
