05-30-2019 12:14 AM - edited 06-18-2019 09:30 PM
Site-to-site vpn tunnel drops off suddenly (connection is flapping).
I issued the 'debug crypto isakmp' command on this router and realized my connection was flapping.
The problematic router (C897VA) is a new and configuration on it is the same with other already working routers (spoke routers) on the network. This router is going to be a new spoke on the network. There is only one vpn tunnel established on it. Configuring DMVPN using GRE over IPSec.
The next hop device Is ISP. The problematic router performs the NAT.
Public IP is pingable during both situation: when the VPN is down, and when the VPN is up
Software version is quite new.
crypto isakmp policy 1
encr aes
authentication pre-share
group 2
crypto isakmp key <removed> address 0.0.0.0 no-xauth
crypto isakmp keepalive 20 3
Here is a debug output:
ISAKMP-ERROR: (0):deleting SA reason "Death by retransmission P1" state (I) MM_NO_STATE
ISAKMP: (0):Deleting peer node by peer_reap for X.X.X.X: 2866AA4
ISAKMP: (0):deleting node 1802653311 error FALSE reason "IKE deleted"
ISAKMP: (0):deleting node -506512794 error FALSE reason "IKE deleted"
ISAKMP: (0):Input = IKE_MESG_INTERNAL, IKE_PHASE1_DEL
ISAKMP: (0):Old State = IKE_I_MM1 New State = IKE_DEST_SA
Typically, ISAKMP uses UDP as its transport protocol. ISAKMP traffic normally goes over UDP port 500, unless NAT-T is used in which case UDP port 4500 is used. When Phase 1 ISAKMP fails I noticed that the debug output shows ISAKMP traffic going over UDP port 500, not over UDP port 4500:
ISAKMP: (0):beginning Main Mode exchange
ISAKMP-PAK: (0):sending packet to X.X.X.X my_port 500 peer_port 500 (I) MM_NO_STATE
------------------------------------
When everything is ok with connection output of the debug shows this line below:
ISAKMP-PAK: (2001):sending packet to X.X.X.X my_port 4500 peer_port 4500 (I) MM_KEY_EXCH (… and so on)
When it’s not ok then it starts from this line:
ISAKMP-PAK: (0):sending packet to X.X.X.X my_port 500 peer_port 500 (I) MM_NO_STATE (… and so on)
------------------------------------------
For more info open the attached files! All public IP-addresses given in the files are replaced with: X.X.X.X, Y.Y.Y.Y, Z.Z.Z.Z
05-30-2019 01:05 AM
Hi,
What interface is this spoke router using? The debugs indicate the Cellular0 modem several times,
11:33:49: %CISCO800-2-MODEM_UP: Cellular0 modem is now UP
11:33:53: Call failed....
11:33:57: %LINEPROTO-5-UPDOWN: Line protocol on Interface Cellular0, changed state to up
is that the WAN interface and is it flapping?....therefore causing the tunnel to drop
You have IP SLA configured:-
11:32:42: %TRACK-6-STATE: 2 ip sla 2 reachability Down -> Up
What is this tracking? Is it related to the Tunnel/WAN interface?
You will still see ISAKMP UDP/500 traffic until ISAKMP Phase 1 is complete, at which point the encrypted traffic will encapsulated inside UDP/4500 packets.
05-31-2019 04:06 AM
06-04-2019 11:09 PM
06-05-2019 01:17 AM
Hello incognito,
according to the following link
https://learningnetwork.cisco.com/thread/36740
The UDP port 4500 is used with NAT-T also to encapsulate ESP packets not only ISAKMP packets.
From the log files:
Your WAN interface is cell0, and it has getting a private IP address, that will be NATTED later in mobile service provider backbone.
To be noted the Cell0 gets the SAME private IP address in both cases when the VPN is setup and when ISAKMP phase I on UDP port 500 is stucked.
From the log file about VPN up some lines may be missing at the begining as the first lines include
>> 12:58:12: ISAKMP-PAK: (2001):sending packet to X.X.X.X my_port 4500 peer_port 4500 (I) MM_KEY_EXCH
What happened before? It is important to understand the initial packet exchange when VPN setup is successful.
Hope to help
Giuseppe
06-05-2019 10:26 PM
Thanks for the answer!
There are all debugs (when VPN setup is successful and not successful) are attached "when VPN is down_debug.txt" and "when VPN is up_debug.txt"
When everything is ok (when VPN is up) with the connection output of the debug starts showing this line below:
ISAKMP-PAK: (2001):sending packet to X.X.X.X my_port 4500 peer_port 4500 (I) MM_KEY_EXCH (… and so on)
When it’s not ok (when VPN is down) then it starts from this line:
ISAKMP-PAK: (0):sending packet to X.X.X.X my_port 500 peer_port 500 (I) MM_NO_STATE (… and so on)
--------------------------------------------------------------
For more info open a new attached file: "when VPN is down for a few seconds.txt" this file contains a debug when VPN was down for a few seconds.
---------------------------------------------------------
All public IP-addresses given in the files are replaced with: X.X.X.X, Y.Y.Y.Y, Z.Z.Z.Z
06-07-2019 02:26 PM
In the normal process of negotiating for ISAKMP it begins by using port 500. If the negotiation proceeds successfully it detects that the peer is associated with NAT and begins to use port 4500. Your statements are consistent with that. In the file where the VPN is up it uses 4500 and in the files where VPN is down it uses only 500. What the file shows is that in the file where VPN is down it is sending attempts to negotiate to the peer but is not receiving any messages. We need to figure out why the peer is not responding in that case.
HTH
Rick
06-18-2019 09:28 PM
Thank you Richard for the answer
I attached new word files and images with an explanations. Please take a look
The problem is related to IPSec (the problem is related to encryption).
In the network there is configured DMVPN Phase 1 (spoke-hub-spoke)
The problem is related with IPSec IKE Phase 1 (ISAKMP) or in other words - ISAKMP negotiation fail.
The "show crypto isakmp sa" command shows the ISAKMP SA to be in MM_NO_STATE, meaning the main-mode (IKE Phase 1) failed.
In the beginning the problem (IPSec IKE Phase 1 (ISAKMP)) was with "spoke-to-hub" connection. Now it's functioning, but this problem hasn't gone totatlly and
it is still working unstable (VPN tunnel flapping).
In addition to this we ran into "spoke-to-spoke" connection problem and the result of "Debug crypto isakmp" output is the same with "spoke-to-hub"s
debug output.
To be precise, some "spoke-to-spoke" connections are working fine while some of them aren't working.
More information is in attached files. Please pay attention to the attached files!
06-20-2019 07:31 AM
Your message says there are new word files. But I do not see any files.
HTH
Rick
06-21-2019 02:31 AM
06-22-2019 07:58 AM
Thank you for the new files. The output of the new file for when not working is similar to the earlier output. It shows very clearly that your router is attempting to initiate isakmp negotiation. It is transmitting, retransmitting, incrementing the error count, over and over and then deletes the peer, and starts over again. It is clear in this output that you are not receiving any isakmp packet from the peer. So the question now is why is the remote peer not sending any isakmp packets?
HTH
Rick
06-23-2019 09:18 PM
hello
"So the question now is why is the remote peer not sending any isakmp packets?" - Yes
Why some "spoke-to-spoke" connections are working fine while some of them aren't working. We have one hub and 17 spokes. Everything fine with 7 of them, but with 10 of them we have a problem...
06-24-2019 08:56 AM
Please help me understand the symptoms a bit better. There is 1 hub and 17 spokes. Am I understanding correctly that every spoke is able to successfully establish the encrypted tunnel to the hub? Am I understanding correctly that for the 7 spokes that work, they can establish encrypted tunnels among the 7? Can these 7 that work establish encrypted tunnels to any of the 10? For the 10 that do not work are they able to establish encrypted tunnels to any other hub?
Other than having unique IP addressing, are there things in the config of the 10 problem spokes that is different from the 7 successful spokes? Perhaps debug from one of the problem spokes might provide some insight.
HTH
Rick
06-18-2019 09:37 PM
Hello, Giuseppe
Could you please take a look to newly attached files?
I attached new word files and images with an explanations.
The problem is related to IPSec (the problem is related to encryption).
In the network there is configured DMVPN Phase 1 (spoke-hub-spoke)
The problem is related with IPSec IKE Phase 1 (ISAKMP) or in other words - ISAKMP negotiation fail.
The "show crypto isakmp sa" command shows the ISAKMP SA to be in MM_NO_STATE, meaning the main-mode
(IKE Phase 1) failed.
In the beginning the problem (IPSec IKE Phase 1 (ISAKMP)) was with "spoke-to-hub" connection. Now it's functioning, but this problem hasn't gone totatlly and it is still working unstable (VPN tunnel flapping).
In addition to this we ran into "spoke-to-spoke" connection problem and the result of "Debug crypto isakmp" output is the same with "spoke-to-hub"s debug output.
To be precise, some "spoke-to-spoke" connections are working fine while some of them aren't working.
More information is in attached files. Please pay attention to the attached files!
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide