Last Thursday we started to experience some odd behaviour with a L2L IPSEC tunnel we have with one of our customers. This tunnel was created a few months back and had been working without issue up until the end of last week. I manage our side of the tunnel and have not made any changes. The other admin for the other side swears they made no changes either.
Anyways here are the details.
We have the following local addresses as part of the interesting traffic: 192.168.2.0/24, 192.168.7.0/24 and 192.168.21.0/24.
We have the following remote addresses as part of the interesting traffic: 10.50.0.0/16, 10.75.0.0/16 and 10.76.0.0/16
* there are actually more address ranges but I don't believe it is important for this discussion *
We got a report from one of our users that they were unable to access an RDP machine from 192.168.21.56 to 10.50.0.233. I logged onto the firewall and noted that there was no current SA for 192.168.21.0/24 to 10.50.0.0/16. There were other active IPSEC SAs, for instance 192.168.21.0/24 to 10.75.0.0/16.
I tried to access the 10.50.0.233 endpoint, while monitoring our ASA, and figured to see messages relating to a failed phase 2. But I didn't see any messages. I enabled debug crypto ?? options on the ASA and saw that the 5-tuple was correclty matching the correct crypto map for that Peer IP address. But after that, nothing. It was like the firewall wasn't even attempting to initiate the tunnel.
My initial thought was that there was some mis-match between the local/remote addresses on both sides of the tunnel. So I contact the admin on the other end and had him send me what he had for the local/remote addresses. They ended up matching what I had. Here is where it gets really strange. In an attempt to narrow down the address range that may be causing the issue I removed all but 192.168.2.0/24 from my end and tore down the tunnel. What happened next, the tunnel came up and an SA was created for 192.168.21.0/24 to 10.75.0.0/16, even though 192.168.21.0/24 wasn't in the local address list for the interesting traffic. By that point I was very confused.
I ended up putting all the address ranges back to normal and all of a sudden we were able to bring up a connection between 192.168.21.56 to 10.50.0.233. Then 10 minutes later that stopped working. On top of that other SAs that were active ended up going away and wouldn't come back.
After pulling out my hair for a bit I figured a reboot of the firewall was inorder. We rebooted and everything sort of went back to normal. Everything was working but except for the initial issue of getting to 10.50.0.233 from 192.168.21.56. Left it for the weekend and came back on Monday. At the start of the day we were able to access 10.50.0.233 and then about 30 minutes later, no connectivity again.
Running through possibilities I thought maybe a license issue, we were hitting a limit on the number of SAs we can bring up. Our license says Base license and allows 250 Total VPN Peers and 250 Other VPN peers. We are not anywhere close to that number.
One more piece of information. When we initially built the tunnel we tried to build with IKEv2 and had some initial issues. So we enabled IKEv1 as well to see if we could get that up and then corrected our IKEv2 issues. But we left both IKEv1 and IKEv2 enabled on that tunnel group. What happens now is that when the tunnel comes up, some SA are IKEv1 and others are IKEv2 for the same Peer IP. Seems to switch back and forth for varying local and remote addresses.
The version of ASA we are running is 9.1 (3). We were on 9.1 and during our reboot on Friday decided to update to the latest thinking maybe we had encountered a bug.
Also opened a SMARTnet case with Cisco on this.
Any suggestions on additional debugging I can do would be greatly appreciated. As I stated I have enabled various debug options (debug crypto ikev2 ..).
