DMVPN Backup Tunnel Issues - invalid local address & IPSec policy invalidated proposal with error 8

[KMSystemsAdmin]

I've been cleaning and redoing our companies DMVPN setup after we installed new fiber.  I broke the work down into chunks so I could test on the weekends so I have a backup of the config before each update was made. 

Once I got so far into the configs I was able to get our system running on a primary and backup tunnel.  On 12/24 I was doing some testing and ended up rebooting the hub router (Cisco 2901).  After the reboot tunnel2 will not allow any remote sites to connect.  I have tried restoring known working configs and all of them repeat the same problem despite having worked before.  I also verified with a diff that the crypto configs did not change. 

The error I'm seeing in the log is:

Dec 29 07:52:00.282: IPSEC(ipsec_process_proposal): invalid local address 216.201.26.121
Dec 29 07:52:00.282: ISAKMP-ERROR: (1392):IPSec policy invalidated proposal with error 8
Dec 29 07:52:00.282: ISAKMP-ERROR: (1392):phase 2 SA policy not acceptable! (local 216.201.26.121 remote 64.83.210.14)

I've been searching online and keep finding different scenarios that point to the crypto map as the problem.  I have tried creating a new crypto map and ended up with the same result.  Does anyone have any idea of where to start troubleshooting? My other tunnel is working great.  It's just the backup interface tunnel that won't bring up the remote sites after a restart of the router.  

The full log message from debug dmvpn all all is:

Dec 29 07:51:59.834: NHRP RIB_RWATCH: Debugging is ON
Dec 29 07:52:00.190: ISAKMP: (1397):purging node -1456123904
Dec 29 07:52:00.278: ISAKMP-PAK: (1392):received packet from 64.83.210.14 dport 500 sport 500 Global (R) QM_IDLE
Dec 29 07:52:00.278: ISAKMP: (1392):set new node 1027212289 to QM_IDLE
Dec 29 07:52:00.278: ISAKMP: (1392):processing HASH payload. message ID = 1027212289
Dec 29 07:52:00.278: ISAKMP: (1392):processing SA payload. message ID = 1027212289
Dec 29 07:52:00.278: ISAKMP: (1392):Checking IPSec proposal 1
Dec 29 07:52:00.278: ISAKMP: (1392):transform 1, ESP_AES
Dec 29 07:52:00.278: ISAKMP: (1392): attributes in transform:
Dec 29 07:52:00.278: ISAKMP: (1392): encaps is 1 (Tunnel)
Dec 29 07:52:00.278: ISAKMP: (1392): SA life type in seconds
Dec 29 07:52:00.278: ISAKMP: SA life duration (VPI) of 0x0 0x1 0x51 0x80
Dec 29 07:52:00.278: ISAKMP: (1392): SA life type in kilobytes
Dec 29 07:52:00.278: ISAKMP: SA life duration (VPI) of 0x0 0x46 0x50 0x0
Dec 29 07:52:00.278: ISAKMP: (1392): authenticator is HMAC-SHA
Dec 29 07:52:00.278: ISAKMP: (1392): key length is 256
Dec 29 07:52:00.278: ISAKMP: (1392):atts are acceptable.
Dec 29 07:52:00.278: IPSEC(validate_proposal_request): proposal part #1
Dec 29 07:52:00.278: IPSEC(validate_proposal_request): proposal part #1,
(key eng. msg.) INBOUND local= 216.201.26.121:0, remote= 64.83.210.14:0,
local_proxy= 216.201.26.121/255.255.255.255/47/0,
remote_proxy= 64.83.210.14/255.255.255.255/47/0,
protocol= ESP, transform= esp-aes 256 esp-sha-hmac (Tunnel),
lifedur= 0s and 0kb,
spi= 0x0(0), conn_id= 0, keysize= 256, flags= 0x0
Dec 29 07:52:00.282: IPSEC(ipsec_process_proposal): invalid local address 216.201.26.121
Dec 29 07:52:00.282: ISAKMP-ERROR: (1392):IPSec policy invalidated proposal with error 8
Dec 29 07:52:00.282: ISAKMP-ERROR: (1392):phase 2 SA policy not acceptable! (local 216.201.26.121 remote 64.83.210.14)
Dec 29 07:52:00.282: ISAKMP: (1392):set new node 1765664467 to QM_IDLE
Dec 29 07:52:00.282: ISAKMP: (1392):Sending NOTIFY PROPOSAL_NOT_CHOSEN protocol 3
spi 574547216, message ID = 1765664467
Dec 29 07:52:00.282: ISAKMP-PAK: (1392):sending packet to 64.83.210.14 my_port 500 peer_port 500 (R) QM_IDLE
Dec 29 07:52:00.282: ISAKMP: (1392):Sending an IKE IPv4 Packet.
Dec 29 07:52:00.282: ISAKMP: (1392):purging node 1765664467
Dec 29 07:52:00.282: ISAKMP-ERROR: (1392):deleting node 1027212289 error TRUE reason "QM rejected"

My Configs:

crypto isakmp policy 1
encr aes 256
authentication pre-share
!
crypto isakmp policy 2
encr 3des
authentication pre-share
group 2
!
crypto isakmp policy 3
encr 3des
authentication pre-share
group 2
lifetime 28800
crypto isakmp key $key address 0.0.0.0
crypto isakmp keepalive 500
crypto isakmp aggressive-mode disable
!
interface Tunnel2
bandwidth 3072
ip address 192.168.253.1 255.255.255.0
ip access-group 150 in
no ip redirects
ip mtu 1400
ip nbar protocol-discovery
ip nhrp authentication $key
ip nhrp map multicast dynamic
ip nhrp network-id 3
ip nhrp holdtime 600
ip nhrp server-only
ip tcp adjust-mss 1360
delay 2000
tunnel source GigabitEthernet0/0
tunnel mode gre multipoint
tunnel key $key
tunnel protection ipsec profile protect-gre-2 shared

crypto ipsec profile protect-gre-2
set security-association lifetime seconds 86400
set transform-set AES-Set-2

crypto ipsec transform-set AES-Set-2 esp-aes 256 esp-sha-hmac
mode tunnel

[balaji.bandi]

Dec 29 07:52:00.282: IPSEC(ipsec_process_proposal): invalid local address 216.201.26.121
Dec 29 07:52:00.282: ISAKMP-ERROR: (1392):IPSec policy invalidated proposal with error 8

based on the information, something has changed, what is the gig0/0 Interface IP address ?

216.201.26.121  - what interface this was configured ? (it is good to post full configuraiton to look)

there is good document to diagnosis same error :

https://community.cisco.com/t5/security-documents/l2l-vpn-troubleshooting-quot-ipsec-policy-invalidated-proposal/ta-p/3115660

[KMSystemsAdmin]

I've tried that IP in 2 different configurations because I wasn't sure how I should handle this. Long story short this router is one of 3 that make up our head end and it used to have a much different setup with a 4G modem directly attached. The fiber is now connected to our internet router.

I've tried both of these configurations:

1. Sharing 2 IP's on a common interface that connects to our Internet Router.

interface GigabitEthernet0/0
description Outside WAN Interface
bandwidth 80000
ip address 216.201.26.121 255.255.240.0 secondary
ip address 70.63.17.71 255.255.255.240
ip access-group 198 in
no ip redirects
no ip unreachables
no ip proxy-arp
ip nbar protocol-discovery
ip nat outside
ip virtual-reassembly in
duplex full
speed 100
no mop enabled
crypto map SDM_CMAP_1
end

2. Setting the IP up on a seperate interface.  This is how I had it working originally but had some Policy Based Routing issues so I moved it to the same interface as the other network.  I can do either interface but I was hoping to use a single interface to clean up the object tracking. 

interface GigabitEthernet0/0/0
ip address 216.201.26.121 255.255.240.0
no ip redirects
no ip unreachables
no ip proxy-arp
ip nbar protocol-discovery
ip flow ingress
ip nat outside
ip virtual-reassembly in
shutdown
duplex auto
speed auto
media-type rj45
end

I have attached a cleaned up version of the running config with this message.  I removed personally identifying info, keys, and some of the ACL stuff we use to manage branch traffic.  If you need anything else please let me know.  Thank you for taking the time to look at this.

[Georg Pauwen]

Hello,

the fastest way to solve this is to post the full running configurations (sh run) of both the hub and the spoke, so we can lab this up.

[MHM Cisco World]

two tunnel use same IPSec profile, use different tunnel key.

try different tunnel key and see result.

[KMSystemsAdmin]

I apologize but when I removed the keys from the tunnel configs I didn't specify they are different.  Each of the 4 tunnels uses a separate key with the command "tunnel key $key".  If you're referring to another key can you please point it out to me? I also see a "ip nhrp authentication" config on the tunnel interfaces.  Those passwords are the same on each tunnel but I didn't think this was part of the IPSEC configs.  

I am attaching a copy of one of our branches.  Thank you for taking a look at this problem.

[MHM Cisco World]

OK, 
shared ipsec profile using with tunnel use same tunnel source.
But 
as I see the same Hub have two different tunnel with same spoke 
and also you use shared in hub tunnel.
as I know you can use shared only in Spoke and hub use different ipsec profile.
check this point.

[KMSystemsAdmin]

I'll look at updating some of these configs for the tunnels.  You make a good point with the way they're sharing info.  But I do have an explanation for that and I got everything working for now.

I came in early this morning to do some testing and I decided to take a few steps back and pull some of my new configs.  I then consoled to the router and rebooted it looking for errors.  I didn't find anything wrong during boot.  I then put my configs back in and placed my backup internet IP on its own interface (gi0/0/0).  Everything came up and started working.  I can now see my spokes over EIGRP and DMVPN commands using both tunnels.

Our configuration is confusing and I inherited this setup so I'm still testing and learning it myself. Every location in our business has a primary internet and a backup internet.  So there are 4 tunnels but only two of them can be active at a time.  The backups are 4G modems and they are rate limited so we leave the interface off until the primary goes down.  This guarantees a spoke can only talk to 2 tunnels at a time because the backup is off until the primary goes down.  The layout is:

Tun0: Spoke Primary Internet <-> Hub Primary Internet

Tun1: Spoke Backup Internet <-> Hub Primary Internet

Tun2: Spoke Primary Internet <-> Hub Backup Internet

Tun3: Spoke Backup Internet <-> Hub Backup Internet

Thank you for taking the time to look at this.  The help is greatly appreciated.