Core issue
Self-signed certificates work only with a Secure Sockets Layer (SSL) connection and fail when IPSec is used.
Resolution
IPSec LAN-to-LAN tunnels do not work with self-signed certificates on routers.
Once both routers have signed their own certificates (acting as a Certificate Authority (CA) for their own certificates), they do not trust each other because the certificate signing authority is not the same. Self-signed certificates work for SSL connections, but they do not work with the Internet Security Association and Key Management Protocol (ISAKMP) or IPSec Rivest, Shamir, and Adelman (RSA) signature implementation because the CA is required to sign or authenticate the certificates.
For more details, refer to Router-to-Router IPSec (RSA Keys) on GRE Tunnel with RIP Configuration Example.
Note: A CA is recommended. Otherwise, certificates must be transported to each router manually. This is similar to authentication using RSA encryption, where public keys must be transferred to each router.
