S2S VPN (ASA <> Meraki) stopped working - need help identifying reason

[mike_t2]

Hi All

I have a S2S VPN from an ASA 5506 to a Meraki MX which was working fine but now has stopped.  The date when it stopped was roughly when the ISP made some changes to their router at the Meraki end and so that is where I suspect the issue lies, but I would like some help to identify what the issue may be as the ISP is saying their config is fine (!). 

This is the only S2S VPN in the network, so I can't test from another ASA, but I did test a Client VPN.  When I started investigating, I set up a client VPN to the Meraki which did not work.  The ISP then opened up ports 500 and 4500 and the clien VPN then worked.

For the S2S VPN, usng sho cry isa sa I can see that it gets to MM_ACTIVE, but sho cry ipsec sa, shows There are no ipsec sas.

Looking at the ASA with ASDM, I can see the VPN come up but Rx bytes and Tx bytes always stay at 0 (I assume because there is no ipsec sa?)

Also when the VPN is at MM_ACTIVE state and I do a packet trace, I get a (acl-drop) Flow is denied by configured rule,

Phase: 10
Type: VPN
Subtype: encrypt
Result: DROP

Another thing to mention is that the VPN is using NAT traversal as my firewalls have been given private addresses, and the ISP devices have the public addresses.

Can anyone help in guiding me to find what the issue might be, or help me collect information to put to the ISP so that they can fix the issue (if it is indeed an issue in their network)

Thanks in advance

Mike 

[Sheraz.Salim]

On cisco ASA NAT traversal is enable by default. as you can see the "show crypto isakmp sa" showing you MM_ACTIVE this means your phase1 come up. but in your second command "show crypto ipsec sa peer x.x.x.x" you do not see encap and decap. This could be an issue with Phase 2. you also mentioned you did a packet tracer. normally when you fired the packet tracer command once it will show you Phase 10 drop. you have to fire up this command twice.

from the above issue as long as your phase1 is comming up there seem to be no issue from the ISP side. as UDP port 500 is translated to 4500.

can you ask the remote end/client to sent constant ping to your crymap-map acl network ip. or you can sent some traffic to remote end pings etc.

also could you issues these command on the ASA cli

capture VPN type isakmp interface outside match ip host y.y.y.y host x.x.x.x
copy /pcap capture:VPN tftp

also please show us the packet tracer result end with detail.

plus give us the output of show crypto ikev1 sa detail and show crypto ipsec sa peer x.x.x.x detail. prior to issuing this command please sent some interesting traffic to remote end (pings etc)

[mike_t2]

Hi Sheraz

The files you wanted are attached. For the captures, I've also included a pcap file from the remote (Meraki) end.  And for the packet-tracer, previously I sent interesting traffic before running the tracer, but this time I ran the tracer twice as you requested.

To explain the addresses, at the ASA end the ASA outside interface is 192.168.30.233 and the ISP device has public address of 154.66.120.121, at the remote (Meraki) end, the internet interface is 192.168.31.200 with a public address of 41.216.228.196.

Thanks for your help

Mike

[Sheraz.Salim]

looking into your packet tracer rules I did see NAT exemption rule but it show as any any. normally it should be the source subnet and destin net. as example below.

object network Local-Subnet
   subnet 192.168.xx.0 255.255.255.0
!
object network Remote-Subnet
  subnet 192.168.xx.0 255.255.255.0
!
nat (inside,outside) source static Local-Subnet Local-Subnet dest static Remote-Subnet Remote-Subnet no proxy arp route-lookup
!
access-list cryptoacl extended permit ip object Local-Subnet object Remote-Subnet
!
crypto map VPN 20 set peer x.x.x.x
crypto map VPN 20 match address crytoacl

[MHM Cisco World]

friend the ACL for IPSec is missing ?
try do packet-tracer again "do it twice" as this recommend from cisco,
and please share the second packet-tracer here.

one more point, you say the ISP is change on other side, are you meaning the ISP change IP ? if that case then you need to change the "Set Peer <IP Address> from your side.

[mike_t2]

Hi - which acl do you mean?  do you mean the cryptomap?  in which case it is there.

For the packet trace - my 2nd post has the results of it being run twice

The ISP just changed their router which my firewall connects to - same IP addresses, so I would have thought I didn;t need to change anything

Thanks

[Sheraz.Salim]

could you please share the firewall configuration. you can take off the presharekeys and orignial IP addressess off.

Just I notice your isakmp show up as below

1   IKE Peer: 41.216.228.196
    Type    : L2L             Role    : initiator
    Rekey   : no              State   : MM_WAIT_MSG6

MM_WAIT_MSG6 mean you not matching the PSK key double check the key are same.

your Marki peer ip address is 154.66.120.121 (as looking in the wireshark) where as you have setup peer in ASA as 41.216.228.196. did the ISP change the Maraki public ip address? if the Maraki ip address changed did you change the peer ip address in your ASA setting?

[mike_t2]

The config is attached.

The ISP has not changed any IP addresses, as far as I can tell, they're doing destination NAT on their routers to do port forwarding, i.e

• from the ASA, the peer is the pubic address of the Meraki (41.216.228.196 which is physically on the ISP router), their router changes this to the physical IP address on the Meraki outside interface (192.168.31.200), so in the Meraki captures you see 154.66.120.121 <> 192.168.31.200
• from the Meraki, the peer is the public address of the ASA (154.66.120.121 which is physically on the ISP router), their router changes this to the physical IP address on the ASA outside interface (192.168.30.233), so in the ASA captures you see 41.216.228.196 <> 192.168.30.233

[Sheraz.Salim]

configuration look good. Now next step is to run the debug commands

debug crypto condition peer x.x.x.x
debug crypto ikev1 platform 255
debug crypto ikev1 protocol 255
debug crypto ipsec 255
ter monitor

. prior to run these command before if using putty. go to logging and log the session so you shall have all the output saved on the file.

once debugs are capture open an other putty session log in to firewall and issue the command undebug all  and ter no mon.

also run the packet tracer again.

in regards to the Public ip address that make sense as I noted in your previous wireshake capture the SPI values on the both captuere were same which make sense.

[mike_t2]

Hi Sheraz, putty log is attached.  I couldn't do the ikev1 platform and protocol commands, so I just did ikev1

thanks for looking 

[Sheraz.Salim]

Thank you for sharing the logs.

Jul 13 03:08:36 [IKEv1 DEBUG]Group = 41.216.228.196, IP = 41.216.228.196, IKE MM Initiator FSM error history (struct &0x00007f9e32aeccf0)  <state>, <event>:  MM_DONE, EV_ERROR-->MM_WAIT_MSG6, EV_PROB_AUTH_FAIL-->MM_WAIT_MSG6, EV_TIMEOUT-->MM_WAIT_MSG6, NullEvent-->MM_SND_MSG5, EV_SND_MSG-->MM_SND_MSG5, EV_START_TMR-->MM_SND_MSG5, EV_RESEND_MSG-->MM_WAIT_MSG6, EV_TIMEOUT

looking into the logs you are failing at MM_WAIT_MSG6.

This message indicates that the Pre-shared keys do not match between the peers. The initiator has sent message 5 to the Remote Peer and the Remote peer was not able to validate the Pre-shared key and doesn't respond. The best thing to do here is work with the Remote side to confirm the Pre-shared keys. When validating the Pre-shared keys, look at special characters or spaces as these can be problematic for some termination devices. or it would be due to NAT-Travesel I have see issue in the past.

could you make sure this command is enable at your end.

crypto isakmp nat-traversal

if PSK key are same on both side. than it  has to be the NAT-T

Since you are using NAT-T (ASA behind a NAT device) the communication will be as below

ASA <------> Maraki

MM1 -->

<-- MM2

MM3 -->

<-- MM4

** Communication will now shift to UDP 4500 from UDP 500 **

MM5 -->

<-- MM6 (Which is Not being Received)

So we have couple of possibilities here

1. Maraki doesn't like MM5 because of pre-shared key mismatch. Though as you said this has been verified.

2. Maraki does send MM6 back to ASA, but it never makes it (In which case you need to verify ISP etc)

Please verify this as below

a. On the ASA setup the capture as below

access-list capture permit ip host host
access-list capture permit ip  host host
capture outside access-list capture interface outside

b. Try to bring up the tunnel and then take a look at "show capture outside" Output. If you do not see any UDP 4500 packet coming back from Maraki then issue is Not on the ASA.

3. I would also verify on Maraki by doing a capture in front that Maraki 'is' sending MM6 back to ASA.

[MHM Cisco World]

Hi

interface GigabitEthernet1/1
 nameif outside
 security-level 0
 ip address dhcp setroute

so
ASA-EdgeRouter-ISP-Meraki 

EdgeRouter have NAT from is public IP to Outside of ASA with port 500 and 4500.
the issue is outside get IP from DHCP not static IP.

are this config is work before?? 

[mike_t2]

Yes, you are right, in full it is

ASA <> ISP edge router1 <> ISP network <> ISP edge router2 <> Meraki

I can see that DHCP might be an issue, but the addresses at each end have been stable for some time, and it was working like this for over a year before stopping

[MHM Cisco World]

may be the Outside get same IP for past and recently the Outside get new IP that the PAT in edge router is different.
so since you can not check IP before try config the IP same as edge router PAT and check IPSec.

[mike_t2]

I don't want to reconfigure the outside address as I'm working on this remotely and I don't want to lose my connection