We actually resolved this issue on our own after a couple days, but I wanted to do a quick writeup here since I couldn't find any other posts about it.
We migrated from older ASAs running 9.12 to newer Cisco Secure Firewalls running ASA code 9.20. We have about 30 tunnels to various partners running VPN solutions from at least 6 different vendors. Immediately after cutover, all but 2 tunnels came up fine.
For the 2 tunnels that we couldn't establish, we noticed we were sending the first IKEv2 phase 1 packet and getting no response at all. We discovered if the remote partners initiated from their side, the tunnels established successfully. We found out that both of these partners were running older ASAs on the last patch of 9.12.
After many hours of troubleshooting and guessing, we determined if we removed DH group 31 from every phase 1 proposal (a global config to all tunnels in ASA), we could finally initiate the tunnels. We then tested having a single proposal with group 31 at the top of our priority list and at the bottom of our priority list. Starting at the top of our phase 1 proposal priority, it looks like ASA 9.12 will succeed if there's a match before seeing anything with group 31, or fail if it sees group 31 before a match. Even if the proposal has groups 31 and 14 in a single proposal, it will fail. I assume it's the same with groups 15 and 16.
This was a very stressful issue that we didn't see with any other firewall vendors. I wanted to document it here in case anyone else comes across this. It's unfortunate that the global phase 1 proposal design of ASA forces us to prioritize weaker proposals whenever a partner device (which happens to also be ASA) can't interpret the stronger ones. I know ASA 9.12 is out of support, but we don't have the power to force partners to upgrade. It would be nice if ASA could support phase 1 proposals like they do for phase 2: specific proposals for each VPN peer.
