Site-to-site VPN only one SA passes traffic at a time

[noticketnomas]

I have a site-to-site VPN tunnel between two sites both running Cisco ASA on 9.8(4)10 code.  The tunnel has about 5 prefixes on A side and 3 from Z side.  I can see phase 1 and 2 are established with no error.  However, it appears I can only pass traffic in a single SA at a time, despite all the other SAs showing up.  I've checked and rechecked config on both sides and the ACLs match.  I also checked NAT-T and NAT exempts and they're configured properly.  I also checked for any possible IP overlaps and there are none.

Here's a summary of the symptoms:

• When the tunnel is first established, I cannot ping anything into the remote network for a short while, but something eventually responds.
• While I'm pinging a subnet at the remote side, the other two do not respond.
• I checked the malfunctioning SA and verified that the traffic is getting encap'ed at the local side, but when I checked the remote SA, there's no encap and decap at all.  It almost seems like the traffic is getting dropped midway.
• I can get another SA to work by stopping pings to the working subnet and run pings to another subnet.  The effect is not immediately.  It seems like I have to wait for something to time out and then the desired SA starts working.

Has anyone run into a similar issue or know how to fix it?  Thanks!

[Rahul Govindan]

Good job getting detailed symptoms. From what you have mentioned, it definitely looks like there might be drops in the path. IF we assume that local ASA sends encrypted traffic outbound successfully, then one of the differences between each of the SA's would be the SPI value. If there is an ISP device that keeps session status based on SPI, it might be dropping when it sees traffic with the same source/destination containg new SPI's. 

I would also apply a capture on the outside interface of both ASA's to make sure that you see outbound encrypted traffic on the local ASA. This will also help you see any differences in traffic from different SA's.

[noticketnomas]

Thanks for the response.  For the subnet that is not working, I captured ESP traffic at both ends and cross check the SPIs in the result.  The local side is definitely sending out the encrypted traffic but the SPI value is missing at the remote receiving side.  I will reach out to the ISP.