cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1074
Views
0
Helpful
5
Replies

Most DMVPN sites down after ASA upgrade

Zahan Al-Rashid
Level 1
Level 1

Hi Guys,


My DMVPN sites were up and running fine however I recently upgraded my ASA 5520 from 8.31 to 8.47; this took all but 1 DMVPN site down. The ACL on FW is same for both but all spoke sites are stuck on MM no state for example:

10.151.62.254 10.215.23.130 MM_NO_STATE 0 ACTIVE

But I am not seeing this on hub. Only 1 QM_IDLE for the site that is up on hub.

The hub has only managed to Peer with 1 site even though the FW has the same ACL rules in place for all sites. I can ping Spoke from Hub and Visa versa. I have used packet tracer to check the site above is allowed in via outside interface and it is. However I see nothing on logs to see an attempt by site above to come in.

Traceroute from spoke sites show that both the working Spoke and non-working spoke site are hitting the ASA and I can see Teardown ICMP connection messages on ASA for tracerroutes and pings from non-working site (I am logging all IP traffic to Hub which is 10.151.62.254).

Any idea what could trigger DMVPN to fail for most sites in this way after upgrading from 8.3 - 8.4? I would like to try everything before reverting it back and it's odd that at least 1 site is up and running!

Many Thanks


Zee

5 Replies 5

pjain2
Cisco Employee
Cisco Employee

by DMVPN do you mean, dynamic to static ipsec tunnels from router to asa?

if after the upgrade you see all the tunnels down, then you need to collect debugs for one particular remote peer on the asa.

below are the debugs to run on the ASA:

debug crypto condition peer <peer ip>

debug crypto isakmp 127

debug crypto ipsec 127

Thanks for your assist on this. 

I have run debug command on spoke and it shows this:

029371: Aug 25 08:39:52.314: ISAKMP:(0):purging node 441711265
029372: Aug 25 08:39:52.314: ISAKMP:(0):purging node -492540660
029373: Aug 25 08:39:52.314: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE...
029374: Aug 25 08:39:52.314: ISAKMP (0): incrementing error counter on sa, attempt 5 of 5: retransmit phase 1
029375: Aug 25 08:39:52.314: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE
029376: Aug 25 08:39:52.314: ISAKMP:(0): sending packet to 10.151.62.254 my_port 500 peer_port 500 (I) MM_NO_STATE
029377: Aug 25 08:39:52.314: ISAKMP:(0):Sending an IKE IPv4 Packet.
029378: Aug 25 08:40:02.314: ISAKMP:(0):purging SA., sa=871E45D0, delme=871E45D0
029379: Aug 25 08:40:02.314: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE...
029380: Aug 25 08:40:02.314: ISAKMP:(0):peer does not do paranoid keepalives.

029381: Aug 25 08:40:02.314: ISAKMP:(0):deleting SA reason "Death by retransmission P1" state (I) MM_NO_STATE (peer 10.151.62.254)
029382: Aug 25 08:40:02.314: ISAKMP:(0):deleting SA reason "Death by retransmission P1" state (I) MM_NO_STATE (peer 10.151.62.254)
029383: Aug 25 08:40:02.314: ISAKMP: Unlocking peer struct 0x872102A8 for isadb_mark_sa_deleted(), count 0
029384: Aug 25 08:40:02.314: ISAKMP: Deleting peer node by peer_reap for 10.151.62.254: 872102A8
029385: Aug 25 08:40:02.314: ISAKMP:(0):deleting node -1046278856 error FALSE reason "IKE deleted"
029386: Aug 25 08:40:02.314: ISAKMP:(0):deleting node 1896685570 error FALSE reason "IKE deleted"
029387: Aug 25 08:40:02.314: ISAKMP:(0):Input = IKE_MESG_INTERNAL, IKE_PHASE1_DEL
029388: Aug 25 08:40:02.314: ISAKMP:(0):Old State = IKE_I_MM1 New State = IKE_DEST_SA

029389: Aug 25 08:40:02.314: ISAKMP:(0): SA request profile is (NULL)
029390: Aug 25 08:40:02.314: ISAKMP: Created a peer struct for 10.151.62.254, peer port 500
029391: Aug 25 08:40:02.314: ISAKMP: New peer created peer = 0x872102A8 peer_handle = 0x8000360D
029392: Aug 25 08:40:02.314: ISAKMP: Locking peer struct 0x872102A8, refcount 1 for isakmp_initiator
029393: Aug 25 08:40:02.314: ISAKMP: local port 500, remote port 500
029394: Aug 25 08:40:02.314: ISAKMP: set new node 0 to QM_IDLE
029395: Aug 25 08:40:02.314: ISAKMP: Find a dup sa in the avl tree during calling isadb_insert sa = 871E45D0
029396: Aug 25 08:40:02.314: ISAKMP:(0):Can not start Aggressive mode, trying Main mode.
029397: Aug 25 08:40:02.314: ISAKMP:(0):found peer pre-shared key matching 10.151.62.254
029398: Aug 25 08:40:02.314: ISAKMP:(0): constructed NAT-T vendor-rfc3947 ID
029399: Aug 25 08:40:02.314: ISAKMP:(0): constructed NAT-T vendor-07 ID
029400: Aug 25 08:40:02.314: ISAKMP:(0): constructed NAT-T vendor-03 ID
029401: Aug 25 08:40:02.314: ISAKMP:(0): constructed NAT-T vendor-02 ID
029402: Aug 25 08:40:02.314: ISAKMP:(0):Input = IKE_MESG_FROM_IPSEC, IKE_SA_REQ_MM
029403: Aug 25 08:40:02.314: ISAKMP:(0):Old State = IKE_READY New State = IKE_I_MM1

029404: Aug 25 08:40:02.314: ISAKMP:(0): beginning Main Mode exchange
029405: Aug 25 08:40:02.314: ISAKMP:(0): sending packet to 10.151.62.254 my_port 500 peer_port 500 (I) MM_NO_STATE
029406: Aug 25 08:40:02.314: ISAKMP:(0):Sending an IKE IPv4 Packet.

The problem is on the ASA the working site I can see ingress and egress for the HUB but for the site with DMVPN not working I can see Ingress traffic but no egress (from Inside interface)? Packet tracer shows both are allowed in from outside to inside but the ones that are not working showing nothing on Inside Egress (See Attachements). Can this be due to a bug?

in the attached packet capture,we can see that the udp 500 packet is going and no return traffic coming back. did you apply a similar capture on the remote end??

if the remote end is receiving the udp 500 and replying back, and it is not being received by your ASA, it could imply a udp 500 block with an intermediate devices such that of an ISP.

instead of doing packet tracer for udp 500 for public ip's, try to do a packet tracer for traffic in the crypto acl(which is the correct way to test if the vpn phase is being hit and the traffic will take the correct output interface). on doing the packet tracer, you will see "VPN encrypt drop" as the tunnel is not up but apart from that you can see what nat rules the traffic is going to hit.

I put a capture on the Inside interface but not for return traffic instead to see outbound traffic. The ASA is sending to traffic to the Hub for one site (2nd page on the file attached shows this) but for the site not working it shows nothing outbound (1st page on attachment). 

I am not sure what I can do to make my packet tracing better? I am not Natting as the ISP does the Natting for me. I can see the address coming into the ASA from both Spokes However only 1 of them is being allowed to go the Hub for and for some unknown reason the other one doesn't make it through from what I can see. 

The Hub is 10.151.62.254

Spokes are 10.215.23.130 and 10.194.80.2.  So far I get same results for both spokes on packet tracer so I am not sure why 1 works and the other doesn't. ISAKMP uses UDP 500 when it sets up this is why I was checking the trace for that port an protocol.

The ASA sits in-between the Hub and Spoke devices. 

I am a little confused on this, the title says DMVPN on the ASA, but ASA's do not support DMVPN's so are you talking about DMVPN passing through the ASA and terminating on a router on the inside or are these just separate site-to-site VPN's?

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: