Every Router connected to the Internet should be protected with an Access-Control-List (ACL) that filters the traffic that is sent to the router. This document shows which Access-List-Entries (ACEs) are needed to allow IPSec-Traffic into the router.
Note1: This applies to IOS-Routers with IOS 12.4+. With older IOS-releases there is more configuration needed. Leave a comment if you are interested in that.
Note2: This does not apply to the ASA where these ACEs are not needed. On the ASA, the interface-ACL by default only filters traffic that is sent through the ASA, but not traffic that is sent to the ASA.
The shown configuration is based on the following topology:

Prerequisite:
The Router should run a basic staefull firewall that allows return-traffic to enter the router-interface without the need for ACEs. That is part of the baseline-security and simplifies the config. Here we use the legacy CBAC because it's much easier to understand and to implement then the more powerful Zone-Based-Firewall:
ip inspect name FW ftp
ip inspect name FW tcp router-traffic
ip inspect name FW udp router-traffic
ip inspect name FW icmp router-traffic
!
interface GigabitEthernet0/0
ip access-group SITE-A-INTERNET-IN in
ip inspect FW out
!
ip access-list extended SITE-A-INTERNET-IN
deny ip any any
Based on that config we extend the ACL with the needed ACEs.
Scenario 1: Only Site-to-Site VPNs with static Peers
Here we assume that all IPSec-Peers have static IPs that are not NATted anywhere. We need to allow the IPSec Data-traffic which is IP-Protocol 50 (ESP) and UDP/500 which is used for ISAKMP. Many examples on the internet show ACEs for IP/51 (AH). But that is normally not used for VPNs.
We start with the creation of an object-group with all VPN-Peers (for object-groups we need at least IOS 12.4(20)T+):
object-group network IPSEC-PEERS
host 198.51.100.1
host 203.0.113.1
Then we change the ACL to allow ESP- and ISAKMP-traffic from the peers to reach the router-interface:
ip access-list extended SITE-A-INTERNET-IN
permit esp object-group IPSEC-PEERS host 192.0.2.1
permit udp object-group IPSEC-PEERS host 192.0.2.1 eq isakmp
permit icmp object-group IPSEC-PEERS host 192.0.2.1 echo
The last line in the ACL is not needed for the VPN-functionality. But beeing able to ping between the IPsec-peers makes troubleshooting much easier.
Scenario 2: Peers are behind a NAT-device with fixed IPs
If some IPsec-peers are hidden behind a NAT-device, then IPsec uses a technology named NAT-Traversal which starts to communicate with UDP/500 but changes later to UDP/4500.
In this case the ACL needs the following entries:
ip access-list extended SITE-A-INTERNET-IN
permit esp object-group IPSEC-PEERS host 192.0.2.1
permit udp object-group IPSEC-PEERS host 192.0.2.1 eq isakmp non500-isakmp
permit icmp object-group IPSEC-PEERS host 192.0.2.1 echo
The entries in the object-group are still the global public IPs of the peer-router, not the internal (possibly private) IPs of the peer VPN-device.
If you know that all peers are behind a NAT-device, then you don't even need the ACE allowing IP/50, ESP. All user-traffic is sent on port udp/4500 in that case.
Scenario 3: IPsec-peers with dynamic IP-addresses
This is the scenario with Remote-Access-VPNs or Site-to-Site VPNs where some spokes don't have fixed IP-addresses. For Remote-Access-VPNs you should always assume that the client can be behind a NAT-device. The resulting ACL is the following:
ip access-list extended SITE-A-INTERNET-IN
permit esp any host 192.0.2.1
permit udp any host 192.0.2.1 eq isakmp non500-isakmp
! generally allow ping from the internet if your security-policy allows that:
permit icmp any host 192.0.2.1 echo
Here we don't need the object-group with the IPsec-peers any more as we don't know their IP-addresses anyway.
Have fun protecting your VPNs!