IPSec Tunnel and RMA ISR 4331

paul1202 · ‎10-21-2018

Hi,

I have had to replace a ISR 4331 which has a field notice replacement against it.

The router basically provides an IPSec Tunnel over a WAN circuit to another site while routing OSPF on top.

I installed the same version of software on the replacement router and copied and paste the original configuration (hostname changed to align with new company standards) and enabled the RTU licenses.

When connecting the new WAN circuit, the IPSec does not negotiate phase 1 and hence the tunnel does not come up. Both ends of the circuit and Tunnel are pingable. If I disable IPSec on the tunnel, the tunnel comes up.

#sh crypto isakmp sa
IPv4 Crypto ISAKMP SA
dst src state conn-id status
186.17.230.1 186.17.230.2 MM_KEY_EXCH 1010 ACTIVE
186.17.230.1 186.17.230.2 MM_NO_STATE 1009 ACTIVE (deleted)

IPv6 Crypto ISAKMP SA

I thought that maybe the preshared key was possibly copied incorrectly, but it looks fine and performing a debug of IPSec we see that the preshared key looks OK.

*Oct 20 10:30:12.335 GMT: %SYS-5-CONFIG_I: Configured from console by console
*Oct 20 10:30:13.096 GMT: ISAKMP: (0):SA request profile is (NULL)
*Oct 20 10:30:13.096 GMT: ISAKMP: (0):Created a peer struct for 186.17.230.1, peer port 500
*Oct 20 10:30:13.096 GMT: ISAKMP: (0):New peer created peer = 0x7F07FF9E15B0 peer_handle = 0x8000000D
*Oct 20 10:30:13.096 GMT: ISAKMP: (0):Locking peer struct 0x7F07FF9E15B0, refcount 1 for isakmp_initiator
*Oct 20 10:30:13.096 GMT: ISAKMP: (0):local port 500, remote port 500
*Oct 20 10:30:13.096 GMT: ISAKMP: (0):set new node 0 to QM_IDLE
*Oct 20 10:30:13.096 GMT: ISAKMP: (0):Find a dup sa in the avl tree during calling isadb_insert sa = 7F07F23EB648
*Oct 20 10:30:13.096 GMT: ISAKMP: (0):Can not start Aggressive mode, trying Main mode.
*Oct 20 10:30:13.096 GMT: ISAKMP: (0):found peer pre-shared key matching 186.17.230.1
*Oct 20 10:30:13.096 GMT: ISAKMP: (0):constructed NAT-T vendor-rfc3947 ID
*Oct 20 10:30:13.096 GMT: ISAKMP: (0):constructed NAT-T vendor-07 ID
*Oct 20 10:30:13.096 GMT: ISAKMP: (0):constructed NAT-T vendor-03 ID
*Oct 20 10:30:13.096 GMT: ISAKMP: (0):constructed NAT-T vendor-02 ID
*Oct 20 10:30:13.096 GMT: ISAKMP: (0):Input = IKE_MESG_FROM_IPSEC, IKE_SA_REQ_MM
*Oct 20 10:30:13.096 GMT: ISAKMP: (0):Old State = IKE_READY New State = IKE_I_MM1
*Oct 20 10:30:13.096 GMT: ISAKMP: (0):beginning Main Mode exchange
*Oct 20 10:30:13.096 GMT: ISAKMP-PAK: (0):sending packet to 186.17.230.1 my_port 500 peer_port 500 (I) MM_NO_STATE30:13.096 GMT: ISAKMP: (0):Sending an IKE IPv4 Packet.
*Oct 20 10:30:13.103 GMT: ISAKMP-PAK: (0):received packet from 186.17.230.1 dport 500 sport 500 Global (I) MM_NO_STATE
*Oct 20 10:30:13.103 GMT: ISAKMP: (0):Input = IKE_MESG_FROM_PEER, IKE_MM_EXCH
*Oct 20 10:30:13.103 GMT: ISAKMP: (0):Old State = IKE_I_MM1 New State = IKE_I_MM2
*Oct 20 10:30:13.103 GMT: ISAKMP: (0):processing SA payload. message ID = 0
*Oct 20 10:30:13.103 GMT: ISAKMP: (0):processing vendor id payload ISAKMP: (0):vendor ID seems Unity/DPD but major 69 mismatch
*Oct 20 10:30:13.103 GMT: ISAKMP: (0):vendor ID is NAT-T RFC 3947
*Oct 20 10:30:13.103 GMT: ISAKMP: (0):found peer pre-shared key matching 186.17.230.1
*Oct 20 10:30:13.103 GMT: ISAKMP: (0):local preshared key found
*Oct 20 10:30:13.103 GMT: ISAKMP: (0):Scanning profiles for xauth ...
*Oct 20 10:30:13.103 GMT: ISAKMP: (0):Checking ISAKMP transform 1 against priority 10 policy
CCBI_UKVAIDC1_RTP_RT_3#T: ISAKMP: (0): encryption 3DES-CBC
*Oct 20 10:30:13.103 GMT: ISAKMP: (0): hash SHA
*Oct 20 10:30:13.103 GMT: ISAKMP: (0): default group 1
*Oct 20 10:30:13.103 GMT: ISAKMP: (0): auth pre-share
*Oct 20 10:30:13.103 GMT: ISAKMP: (0): life type in seconds
*Oct 20 10:30:13.103 GMT: ISAKMP: life duration (VPI) of 0x0 0x1 0x51 0x80
*Oct 20 10:30:13.103 GMT: ISAKMP: (0):atts are acceptable. Next payload is 0
*Oct 20 10:30:13.103 GMT: ISAKMP: (0):Acceptable atts:actual life: 0
*Oct 20 10:30:13.103 GMT: ISAKMP: (0) Acceptable atts:life: 0
*Oct 20 10:30:13.103 GMT: ISAKMP: (0):Fill atts in sa vpi_length:4
*Oct 20 10:30:13.103 GMT: ISAKMP: (0):Fill atts in sa life_in_seconds:86400
*Oct 20 10:30:13.103 GMT: ISAKMP: (0):Returning Actual lifetime: 86400
*Oct 20 10:30:13.103 GMT: ISAKMP: (0):Started lifetime timer: 86400.
*Oct 20 10:30:13.103 GMT: ISAKMP: (0):processing vendor id payload
*Oct 20 10:30:13.103 GMT: ISAKMP: (0):vendor ID seems Unity/DPD but major 69 mismatch
*Oct 20 10:30:13.103 GMT: ISAKMP: (0):vendor ID is NAT-T RFC 3947
*Oct 20 10:30:13.103 GMT: ISAKMP: (0):Input = IKE_MESG_INTERNAL, IKE_PROCESS_MAIN_MODE
*Oct 20 10:30:13.103 GMT: ISAKMP: (0):Old State = IKE_I_MM2 New State = IKE_I_MM2
*Oct 20 10:30:13.104 GMT: ISAKMP-PAK: (0):sending packet to 186.17.230.1 my_port 500 peer_port 500 (I) MM_SA_SETUP
*Oct 20 10:30:13.104 GMT: ISAKMP: (0):Sending an IKE IPv4 Packet.
*Oct 20 10:30:13.104 GMT: ISAKMP: (0):Input = IKE_MESG_INTERNAL, IKE_PROCESS_COMPLETE
*Oct 20 10:30:13.104 GMT: ISAKMP: (0):Old State = IKE_I_MM2 New State = IKE_I_MM3
*Oct 20 10:30:13.108 GMT: ISAKMP-PAK: (0):received packet from 186.17.230.1 dport 500 sport 500 Global (I) MM_SA_SETUP
*Oct 20 10:30:13.108 GMT: ISAKMP: (0):Input = IKE_MESG_FROM_PEER, IKE_MM_EXCH
*Oct 20 10:30:13.108 GMT: ISAKMP: (0):Old State = IKE_I_MM3 New State = IKE_I_MM4
*Oct 20 10:30:13.108 GMT: ISAKMP: (0):processing KE payload. message ID = 0
*Oct 20 10:30:13.110 GMT: ISAKMP: (0):processing NONCE payload. message ID = 0 ISAKMP: (0):found peer pre-shared key matching 186.17.230.1
*Oct 20 10:30:13.111 GMT: ISAKMP: (1007):processing vendor id payload
*Oct 20 10:30:13.111 GMT: ISAKMP: (1007):vendor ID is Unity
*Oct 20 10:30:13.111 GMT: ISAKMP: (1007):processing vendor id payload
*Oct 20 10:30:13.111 GMT: ISAKMP: (1007):vendor ID is DPD
*Oct 20 10:30:13.111 GMT: ISAKMP: (1007):processing vendor id payload
*Oct 20 10:30:13.111 GMT: ISAKMP: (1007):speaking to another IOS box!
*Oct 20 10:30:13.111 GMT: ISAKMP: (1007):#received payload type 20
*Oct 20 10:30:13.111 GMT: ISAKMP: (1007):His hash no match - this node outside NAT
*Oct 20 10:30:13.111 GMT: ISAKMP: (1007):received payload type 20
*Oct 20 10:30:13.111 GMT: ISAKMP: (1007):No NAT Found for self or peer
*Oct 20 10:30:13.111 GMT: ISAKMP: (1007):Input = IKE_MESG_INTERNAL, IKE_PROCESS_MAIN_MODE
*Oct 20 10:30:13.111 GMT: ISAKMP: (1007):Old State = IKE_I_MM4 New State = IKE_I_MM4
*Oct 20 10:30:13.111 GMT: ISAKMP: (1007):Send initial contact
*Oct 20 10:30:13.111 GMT: ISAKMP: (1007):SA is doing
*Oct 20 10:30:13.111 GMT: ISAKMP: (1007):pre-shared key authentication using id type ID_IPV4_ADDR
*Oct 20 10:30:13.111 GMT: ISAKMP: (1007):ID payload
next-payload : 8
type : 1
*Oct 20 10:30:13.111 GMT: ISAKMP: (1007): address : 186.17.230.2
*Oct 20 10:30:13.111 GMT: ISAKMP: (1007): protocol : 17
port : 500
length : 12
*Oct 20 10:30:13.111 GMT: ISAKMP: (1007):Total payload length: 12
*Oct 20 10:30:13.111 GMT: ISAKMP-PAK: (1007):sending packet to 186.17.230.1 my_port 500 peer_port 500 (I) MM_KEY_EXCH
*Oct 20 10:30:13.111 GMT: ISAKMP: (1007):Sending an IKE IPv4 Packet.
*Oct 20 10:30:13.111 GMT: ISAKMP: (1007):Input = IKE_MESG_INTERNAL, IKE_PROCESS_COMPLETE
*Oct 20 10:30:13.111 GMT: ISAKMP: (1007):Old State = IKE_I_MM4 New State = IKE_I_MM5
*Oct 20 10:30:14.142 GMT: ISAKMP-PAK: (1007):received packet from 186.17.230.1 dport 500 sport 500 Global (I) MM_KEY_EXCH
*Oct 20 10:30:14.143 GMT: ISAKMP: (1007):phase 1 packet is a duplicate of a previous packet.
*Oct 20 10:30:14.143 GMT: ISAKMP: (1007):retransmitting due to retransmit phase 1
*Oct 20 10:30:14.642 GMT: ISAKMP: (1007):retransmitting phase 1 MM_KEY_EXCH...
*Oct 20 10:30:14.642 GMT: ISAKMP: (1007):: incrementing error counter on sa, attempt 1 of 5: retransmit phase 1
*Oct 20 10:30:14.642 GMT: ISAKMP: (1007):retransmitting phase 1 MM_KEY_EXCH
*Oct 20 10:30:14.642 GMT: ISAKMP-PAK: (1007):sending packet to 186.17.230.1 my_port 500 peer_port 500 (I) MM_KEY_EXCH
*Oct 20 10:30:14.642 GMT: ISAKMP: (1007):Sending an IKE IPv4 Packet.
*Oct 20 10:30:24.642 GMT: ISAKMP: (1007):retransmitting phase 1 MM_KEY_EXCH...
*Oct 20 10:30:24.642 GMT: ISAKMP: (1007):: incrementing error counter on sa, attempt 2 of 5: retransmit phase 1
*Oct 20 10:30:24.642 GMT: ISAKMP: (1007):retransmitting phase 1 MM_KEY_EXCH
*Oct 20 10:30:24.642 GMT: ISAKMP-PAK: (1007):sending packet to 186.17.230.1 my_port 500 peer_port 500 (I) MM_KEY_EXCH
*Oct 20 10:30:24.642 GMT: ISAKMP: (1007):Sending an IKE IPv4 Packet.
*Oct 20 10:30:25.143 GMT: ISAKMP-PAK: (1007):received packet from 186.17.230.1 dport 500 sport 500 Global (I) MM_KEY_EXCH
*Oct 20 10:30:25.143 GMT: ISAKMP: (1007):phase 1 packet is a duplicate of a previous packet.
*Oct 20 10:30:25.143 GMT: ISAKMP: (1007):retransmission skipped for phase 1 (time since last transmission 501)
*Oct 20 10:30:25.891 GMT: ISAKMP-ERROR: (1007):ignoring request to send delete notify (sa not authenticated) src 186.17.230.2 dst 186.17.230.1
*Oct 20 10:30:34.643 GMT: ISAKMP: (1007):retransmitting phase 1 MM_KEY_EXCH...
*Oct 20 10:30:34.643 GMT: ISAKMP: (1007):: incrementing error counter on sa, attempt 3 of 5: retransmit phase 1
*Oct 20 10:30:34.643 GMT: ISAKMP: (1007):retransmitting phase 1 MM_KEY_EXCH
*Oct 20 10:30:34.643 GMT: ISAKMP-PAK: (1007):sending packet to 186.17.230.1 my_port 500 peer_port 500 (I) MM_KEY_EXCH
*Oct 20 10:30:34.643 GMT: ISAKMP: (1007):Sending an IKE IPv4 Packet.
*Oct 20 10:30:35.144 GMT: ISAKMP-PAK: (1007):received packet from 186.17.230.1 dport 500 sport 500 Global (I) MM_KEY_EXCH
*Oct 20 10:30:35.145 GMT: ISAKMP: (1007):phase 1 packet is a duplicate of a previous packet.
*Oct 20 10:30:35.145 GMT: ISAKMP: (1007):retransmission skipped for phase 1 (time since last transmission 502)
*Oct 20 10:30:35.404 GMT: ISAKMP-ERROR: (1007):ignoring request to send delete notify (sa not authenticated) src 186.17.230.2 dst 186.17.230.1
*Oct 20 10:30:43.096 GMT: ISAKMP: (1007):set new node 0 to QM_IDLE
*Oct 20 10:30:43.096 GMT: ISAKMP-ERROR: (1007):SA is still budding. Attached new ipsec request to it. (local 186.17.230.2, remote 186.17.230.1)
*Oct 20 10:30:43.099 GMT: ISAKMP-ERROR: (0):Error while processing SA request: Failed to initialize SA
*Oct 20 10:30:43.099 GMT: ISAKMP-ERROR: (0):Error while processing KMI message 0, error 2.

The IPSec config is below;

crypto isakmp policy 10
encr 3des
authentication pre-share
crypto isakmp key 6 TSWTc^DU\IXfcNhVUX\V^NdWfZEXT]\\dPIeJ_Z address 186.17.230.1
!
crypto ipsec security-association replay window-size 1024
!
crypto ipsec transform-set SET-TO-OB4F esp-3des esp-md5-hmac
mode tunnel
!
crypto ipsec profile VPN-TO-OB4F
set transform-set SET-TO-OB4F
set pfs group20
!

I've read other posts where folks have suggested changing the Keys to something completely new as they have become "corrupt"!

Any help appreciated.

Paul

rmfalconer · ‎10-26-2018

What's the output of 'show license'?

Richard Burts · ‎10-27-2018

Asking about the license is an interesting thought. But if it were an issue with the license I would expect problems in attempting to configure the feature, and if it were an issue with license I would certainly expect that problem to prevent the beginning of isakmp negotiation. The fact that we are seeing the negotiation makes me believe that the license is ok.

Paul

The negotiation begins and goes through several phases MM1 to MM2 to MM3 to MM4 to MM5. If there were problems with the key I would not expect the negotiation to get to those phases. But it might be interesting to change both sides to use a new (and perhaps simpler) key and see if it makes any difference.

Can you check and verify that the other side matches the pfs group, and matches specifying tunnel mode?

Am I correct in thinking that this is a VTI tunnel for ipsec? Can you show us that part of the config? Your comment that if you remove the ipsec from the tunnel that it comes up correctly emphasizes that the problem is with the negotiation for crypto. And a VTI tunnel will not come up if the crypto negotiation fails.

The debug is clear that there is a problem, but not clear about what the problem is. Could you get debug output from the peer? Perhaps that would have better identification of the issue?

HTH

Rick

HTH

Rick

paul1202 · ‎10-29-2018

Thanks for your reply Rick.

Speaking with the customer and doing further checking, it appears there is a master key which is used to encrypt any exisiting keys in the router configuration using AES. and which cannot be seen in or obtained in the configuration.

password encryption aes

key config-key passowrd-encrypt

I am hoping to try the above in the next day or so to confirm the fix.

Paul

Richard Burts · ‎11-02-2018

Paul

Were your attempts with the master key successful? If not can you answer the questions that I asked?

HTH

Rick

HTH

Rick

paul1202 · ‎11-03-2018

Hi Rick,

So for the late reply.

Yes, changing the master key resolved the issue. The customer wasn't sure what they were so I simply removed the keys and created new ones on both sides.

The Tunnel came straight up and all traffic encrypted across it.

Thanks again for your interest.

Regards,

Paul

Richard Burts · ‎11-03-2018

Paul

Glad to know that the problem is resolved. +5 for the update. Good to know that the problem was about the master keys. If the problem was about keys I am surprised that the isakmp negotiation got through several phases. But apparently it did.

HTH

Rick

HTH

Rick