Re: Site to Site VPN/Tunnel changes in new ISRs

wwishart1 · ‎06-29-2020

Good morning,

We recently replaced our 1841 and 881 routers, with one 4331 Router.

I am having difficulties in getting the VPNs and tunnels working on the new router.

While I am having a few issues, the big pressing issue I have is that the tunnels, and tunnel protection are not working the same way. I know we needed to do a license for the 881 in order to setup tunnels, and I'm fairly certain when we purchased this router, it was one with the higher license (we had two options, and I think it was about a $600ish difference)

Basically with the tunnel protection. If I build the tunnel, it comes up, but if I add the tunnel protection line into the mix, it dies and keeps throwing a %CRYPTO-6-IKMP_MODE_FAILURE: Processing of Informational mode failed with peer at xxx.xxx.xxx.xxx. We are using the same ISAKMP Policies and crypto maps from the old router.

Rob Ingram · ‎06-29-2020

Hi,
Please can you provide the full output of the debug and your configuration of the routers.

wwishart1 · ‎06-30-2020

Here is the result of debug crypto ipsec and debug crypto isakmp error, I'm not sure if you needed something else. There are a number of endpoints that are trying to connect that I dropped the crypto map for. The tunnel is my high priority at the moment. Attached are the configs of the routers. 881 and 1841 were the original routers (There was a tunnel between the two of them as well, that is no longer there) which was replaced by the 4331. Oxmead router is at a different location. I hope I supplied enough intelligent information.

Crypto IPSEC debugging is on

Jun 30 11:52:42.095: IPSEC(validate_proposal_request): proposal part #1
Jun 30 11:52:42.096: IPSEC(validate_proposal_request): proposal part #1,
(key eng. msg.) INBOUND local= 50.195.108.145:0, remote= 64.19.142.250:0,
local_proxy= 172.16.120.0/255.255.248.0/256/0,
remote_proxy= 192.168.147.0/255.255.255.0/256/0,
protocol= ESP, transform= esp-aes 256 esp-sha-hmac (Tunnel),
lifedur= 0s and 0kb,
spi= 0x0(0), conn_id= 0, keysize= 256, flags= 0x0
Jun 30 11:52:42.096: map_db_find_best did not find matching map
Jun 30 11:52:42.096: IPSEC(ipsec_process_proposal): transform proposal not supported for identity:
{esp-aes 256 esp-sha-hmac }isakmp ?

Crypto ISAKMP Error debugging is on
Jun 30 11:52:58.004: IPSEC:(SESSION ID = 27) (key_engine) request timer fired: count = 2,
(identity) local= 50.195.108.145:0, remote= 98.110.23.72:0,
local_proxy= 50.195.108.145/255.255.255.255/47/0,
remote_proxy= 98.110.23.72/255.255.255.255/47/0
Jun 30 11:52:58.007: ISAKMP-ERROR: (0):deleting SA reason "Death by retransmission P1" state (I) MM_NO_STATE (peer 98.110.23.72)
Jun 30 11:52:58.007: ISAKMP-ERROR: (0):deleting SA reason "Death by retransmission P1" state (I) MM_NO_STATE (peer 98.110.23.72)
Jun 30 11:52:58.008: IPSEC(key_engine): got a queue event with 1 KMI message(s)
Jun 30 11:52:58.067: IPSEC(sa_request): ,
(key eng. msg.) OUTBOUND local= 50.195.108.145:500, remote= 98.110.23.72:500,
local_proxy= 50.195.108.145/255.255.255.255/47/0,
remote_proxy= 98.110.23.72/255.255.255.255/47/0,
protocol= ESP, transform= esp-aes 256 esp-sha512-hmac (Transport),
lifedur= 86400s and 4608000kb,
spi= 0x0(0), conn_id= 0, keysize= 256, flags= 0x0
Jun 30 11:52:58.091: ISAKMP-ERROR: (0):Notify has no hash. Rejected.
Jun 30 11:52:58.092: ISAKMP-ERROR: (0):(0): Unknown Input IKE_MESG_FROM_PEER, IKE_INFO_NOTIFY: state = IKE_I_MM1
Jun 30 11:52:58.092: %CRYPTO-6-IKMP_MODE_FAILURE: Processing of Informational mode failed with peer at 98.110.23.72
ESG-Router#
Jun 30 11:53:08.145: IPSEC(key_engine): got a queue event with 1 KMI message(s)
Jun 30 11:53:08.146: IPSEC(key_engine): got a queue event with 1 KMI message(s)
Jun 30 11:53:10.188: ISAKMP-ERROR: (0):No pre-shared key with 173.71.75.7!
Jun 30 11:53:10.188: ISAKMP-ERROR: (0):Preshared authentication offered but does not match policy!
Jun 30 11:53:10.188: ISAKMP-ERROR: (0):atts are not acceptable. Next payload is 0
Jun 30 11:53:10.188: ISAKMP-ERROR: (0):Hash algorithm offered does not match policy!
Jun 30 11:53:10.189: ISAKMP-ERROR: (0):atts are not acceptable. Next payload is 0
Jun 30 11:53:10.189: ISAKMP-ERROR: (0):Preshared authentication offered but does not match policy!
Jun 30 11:53:10.189: ISAKMP-ERROR: (0):atts are not acceptable. Next payload is 0
Jun 30 11:53:10.189: ISAKMP-ERROR: (0):no offers accepted!
Jun 30 11:53:10.189: ISAKMP-ERROR: (0):phase 1 SA policy not acceptable! (local 50.195.108.145 remote 173.71.75.7)
Jun 30 11:53:10.189: ISAKMP-ERROR: (0):deleting SA reason "Phase1 SA policy proposal not accepted" state (R) MM_NO_STATE (peer 173.71.75.7)
Jun 30 11:53:10.190: ISAKMP-ERROR: (0):(0): FSM action returned error: 2
Jun 30 11:53:10.190: ISAKMP-ERROR: (0):deleting SA reason "Phase1 SA policy proposal not accepted" state (R) MM_NO_STATE (peer 173.71.75.7)
Jun 30 11:53:10.191: IPSEC(key_engine): got a queue event with 1 KMI message(s)
Jun 30 11:53:11.904: ISAKMP-ERROR: (0):Diffie-Hellman group offered does not match policy!
Jun 30 11:53:11.904: ISAKMP-ERROR: (0):atts are not acceptable. Next payload is 0
Jun 30 11:53:11.904: ISAKMP-ERROR: (0):Hash algorithm offered does not match policy!
Jun 30 11:53:11.905: ISAKMP-ERROR: (0):atts are not acceptable. Next payload is 0
Jun 30 11:53:12.112: IPSEC(validate_proposal_request): proposal part #1
Jun 30 11:53:12.112: IPSEC(validate_proposal_request): proposal part #1,
(key eng. msg.) INBOUND local= 50.195.108.145:0, remote= 64.19.142.250:0,
local_proxy= 172.16.120.0/255.255.248.0/256/0,
remote_proxy= 192.168.147.0/255.255.255.0/256/0,
protocol= ESP, transform= esp-aes 256 esp-sha-hmac (Tunnel),
lifedur= 0s and 0kb,
spi= 0x0(0), conn_id= 0, keysize= 256, flags= 0x0
Jun 30 11:53:12.112: map_db_find_best did not find matching map
Jun 30 11:53:12.113: IPSEC(ipsec_process_proposal): transform proposal not supported for identity:
{esp-aes 256 esp-sha-hmac }
Jun 30 11:53:12.113: ISAKMP-ERROR: (1773):IPSec policy invalidated proposal with error 256
Jun 30 11:53:12.114: ISAKMP-ERROR: (1773):phase 2 SA policy not acceptable! (local 50.195.108.145 remote 64.19.142.250)
Jun 30 11:53:12.114: ISAKMP-ERROR: (1773):deleting node 795059527 error TRUE reason "QM rejected"
Jun 30 11:53:28.067: IPSEC:(SESSION ID = 27) (key_engine) request timer fired: count = 1,
(identity) local= 50.195.108.145:0, remote= 98.110.23.72:0,
local_proxy= 50.195.108.145/255.255.255.255/47/0,
remote_proxy= 98.110.23.72/255.255.255.255/47/0
Jun 30 11:53:28.067: IPSEC(sa_request): ,
(key eng. msg.) OUTBOUND local= 50.195.108.145:500, remote= 98.110.23.72:500,
local_proxy= 50.195.108.145/255.255.255.255/47/0,
remote_proxy= 98.110.23.72/255.255.255.255/47/0,
protocol= ESP, transform= esp-aes 256 esp-sha512-hmac (Transport),
lifedur= 86400s and 4608000kb,
spi= 0x0(0), conn_id= 0, keysize= 256, flags= 0x0
Jun 30 11:53:28.068: ISAKMP-ERROR: (0):SA is still budding. Attached new ipsec request to it. (local 50.195.108.145, remote 98.110.23.72)
Jun 30 11:53:28.068: ISAKMP-ERROR: (0):Error while processing SA request: Failed to initialize SA
Jun 30 11:53:28.068: ISAKMP-ERROR: (0):Error while processing KMI message 0, error 2.
Jun 30 11:53:58.067: IPSEC:(SESSION ID = 27) (key_engine) request timer fired: count = 2,
(identity) local= 50.195.108.145:0, remote= 98.110.23.72:0,
local_proxy= 50.195.108.145/255.255.255.255/47/0,
remote_proxy= 98.110.23.72/255.255.255.255/47/0
Jun 30 11:53:58.069: ISAKMP-ERROR: (0):deleting SA reason "Death by retransmission P1" state (I) MM_NO_STATE (peer 98.110.23.72)
Jun 30 11:53:58.069: ISAKMP-ERROR: (0):deleting SA reason "Death by retransmission P1" state (I) MM_NO_STATE (peer 98.110.23.72)
Jun 30 11:53:58.070: IPSEC(key_engine): got a queue event with 1 KMI message(s)
Jun 30 11:53:58.495: IPSEC(sa_request): ,
(key eng. msg.) OUTBOUND local= 50.195.108.145:500, remote= 98.110.23.72:500,
local_proxy= 50.195.108.145/255.255.255.255/47/0,
remote_proxy= 98.110.23.72/255.255.255.255/47/0,
protocol= ESP, transform= esp-aes 256 esp-sha512-hmac (Transport),
lifedur= 86400s and 4608000kb,
spi= 0x0(0), conn_id= 0, keysize= 256, flags= 0x0
Jun 30 11:53:58.530: ISAKMP-ERROR: (0):Notify has no hash. Rejected.
Jun 30 11:53:58.531: ISAKMP-ERROR: (0):(0): Unknown Input IKE_MESG_FROM_PEER, IKE_INFO_NOTIFY: state = IKE_I_MM1
Jun 30 11:53:58.531: %CRYPTO-6-IKMP_MODE_FAILURE: Processing of Informational mode failed with peer at 98.110.23.72
Jun 30 11:54:12.279: ISAKMP-ERROR: (0):No pre-shared key with 173.71.75.7!
Jun 30 11:54:12.280: ISAKMP-ERROR: (0):Preshared authentication offered but does not match policy!
Jun 30 11:54:12.280: ISAKMP-ERROR: (0):atts are not acceptable. Next payload is 0
Jun 30 11:54:12.280: ISAKMP-ERROR: (0):Hash algorithm offered does not match policy!
Jun 30 11:54:12.280: ISAKMP-ERROR: (0):atts are not acceptable. Next payload is 0
Jun 30 11:54:12.280: ISAKMP-ERROR: (0):Preshared authentication offered but does not match policy!
Jun 30 11:54:12.280: ISAKMP-ERROR: (0):atts are not acceptable. Next payload is 0
Jun 30 11:54:12.280: ISAKMP-ERROR: (0):no offers accepted!
Jun 30 11:54:12.281: ISAKMP-ERROR: (0):phase 1 SA policy not acceptable! (local 50.195.108.145 remote 173.71.75.7)
Jun 30 11:54:12.281: ISAKMP-ERROR: (0):deleting SA reason "Phase1 SA policy proposal not accepted" state (R) MM_NO_STATE (peer 173.71.75.7)
Jun 30 11:54:12.281: ISAKMP-ERROR: (0):(0): FSM action returned error: 2
Jun 30 11:54:12.282: ISAKMP-ERROR: (0):deleting SA reason "Phase1 SA policy proposal not accepted" state (R) MM_NO_STATE (peer 173.71.75.7)
Jun 30 11:54:12.282: IPSEC(key_engine): got a queue event with 1 KMI message(s)no debug crypto ipsec

Rob Ingram · ‎06-30-2020

Sorry but I am confused, are you attempting to use the virtual tunnel interface or crypto map? The 4331 only has the crypto map, the Oxmead router has a crypto map and tunnel interfaces.

Which router is this debug from? There are numerous errors but they could be from multiple sources attempting to establish a tunnel. Errors such as missing PSK, incorrect Phase 1 and 2 attributes etc
Can you clarify which routers you are attempting to establish a tunnel with and ideally provide the debugs from both.

wwishart1 · ‎06-30-2020

I unfortunately can't debug the other side of the tunnel.

To the best that I recall... the crypto map that is shown should only be for GRE protocol to one of the endpoints. 4331 <-> Oxmead Router is the current tunnel I am trying to get protected. Let me double check the file I sent, I am thinking the txt file of the 4331 config may not have been the one I was intending to grab.

wwishart1 · ‎06-30-2020

Here is the current config. Sorry for the confusion. Tunnel1 is the tunnel in question.

Rob Ingram · ‎06-30-2020

Ok, so to confirm you have the issue between 4331 and Oxmead using Tunnel1? In which case on the Oxmead router you've specified the destination as 50.xxx.xxx.148 but the 4331 ip address is 50.xxx.xxx.145, amend this, unless this is just a typo when you modified the config to upload?

Oxmead

interface Tunnel1
 tunnel source 98.xxx.xxx.72
 tunnel destination 50.xxx.xxx.148

4331

interface Tunnel1
 tunnel source 50.xxx.xxx.145
 tunnel destination 98.xxx.xxx.72

HTH

wwishart1 · ‎07-01-2020

Sorry, it is a typo. If I drop the tunnel protection clause from both sides, the tunnel comes up,

Jul 1 12:34:00.523: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel1, changed state to up
Jul 1 12:34:02.777: %DUAL-5-NBRCHANGE: EIGRP-IPv4 10: Neighbor 192.168.100.2 (Tunnel1) is up: new adjacency

But when I try to add the protection statement, it drops.

Currently I have it up with no protection so I can access the other side of the tunnel to make programming changes. Whenever I do testing, I through a reload in 30 statement first so I can regain control if what I tried didn't work.

Rob Ingram · ‎07-01-2020

Is traffic being NATTED along the path?
Do you have any ACLs or Firewalls in place that could be blocking ESP, UDP/500 or UDP/4500 (if using NAT)?

wwishart1 · ‎07-01-2020

Nothing that I have setup. Unless the 4331 does things a little differently. We have yet to start doing anything with the zone based firewall features. I had added them through the WebUI recently and then removed it when I realized the initial configuration that WebUI does wasn't going to work for us and I had to get traffic back up quickly. But I was having this issue prior to that I believe. I did strip out any of the inbound NAT INSIDE STATIC rules, but there aren't any that should (or at least none that previously affected it)

wwishart1 · ‎07-01-2020

Could it be a licensing issue? I know I had to upgrade the security license on the 881s originally.

My current licenses/features are:

Router#sh lic
Index 1 Feature: appxk9
Period left: Not Activated
Period Used: 0 minute 0 second
License Type: EvalRightToUse
License State: Active, Not in Use, EULA not accepted
License Count: Non-Counted
License Priority: None
Index 2 Feature: uck9
Period left: Not Activated
Period Used: 0 minute 0 second
License Type: EvalRightToUse
License State: Active, Not in Use, EULA not accepted
License Count: Non-Counted
License Priority: None
Index 3 Feature: securityk9
Period left: Life time
License Type: Permanent
License State: Active, In Use
License Count: Non-Counted
License Priority: Medium
Index 4 Feature: ipbasek9
Period left: Life time
License Type: Permanent
License State: Active, In Use
License Count: Non-Counted
License Priority: Medium
Index 5 Feature: FoundationSuiteK9
Period left: Not Activated
Period Used: 0 minute 0 second
License Type: EvalRightToUse
License State: Active, Not in Use, EULA not accepted
License Count: Non-Counted
License Priority: None
Index 6 Feature: AdvUCSuiteK9
Period left: Not Activated
Period Used: 0 minute 0 second
License Type: EvalRightToUse
License State: Active, Not in Use, EULA not accepted
License Count: Non-Counted
License Priority: None
Index 7 Feature: cme-srst
Period left: Not Activated
Period Used: 0 minute 0 second
License Type: EvalRightToUse
License State: Active, Not in Use, EULA not accepted
License Count: 0/0 (In-use/Violation)
License Priority: None
Index 8 Feature: hseck9
Index 9 Feature: throughput
Period left: Not Activated
Period Used: 0 minute 0 second
License Type: EvalRightToUse
License State: Active, Not in Use, EULA not accepted
License Count: Non-Counted
License Priority: None
Index 10 Feature: internal_service

Rob Ingram · ‎07-01-2020

No, I don't think so, it says the Security license is active and in use.

On the 4331, you've got 2 default routes and using IP SLA, have you confirmed that the SLA is definately UP and you are using the correct default route? Can you ping the remote router?

wwishart1 · ‎07-15-2020

I'm sorry, I have been out on vacation. Yes, IP SLAs tracked routes are working correctly. Default route, would be over separate ISP, however I would expect if that were the case, I would not be able to get the tunnel to come up without protection. I would also imagine tunnel source would target the correct interface. I can ping the remote router. (Again, If I take protection off the tunnel, the tunnel comes up and routes traffic as would be expected. My concern being that the traffic is unprotected by encryption.

wwishart1 · ‎07-15-2020

Sorry, misspoke, default route is actually the ISP that the tunnel source is using. That's our broadband connection.

wwishart1 · ‎08-16-2020

Anyone? Hello?