ASA S2S VPN Issue

stanislav.pilat · ‎03-05-2019

ASAHello,

I'm facing a problem with S2S VPN between 2 ASAs 5506-X ver. 9.9(2)18.

The VPN was working fine 6-7 months but last week It suddenly went down.

I did a few troubleshooting steps (debug crypto ikev1 128) and packet capture on both sides.

I can see Initiator tries to send the first packet:

Feb 28 14:42:08 [IKEv1 DEBUG]Pitcher: received a key acquire message, spi 0x0
Feb 28 14:42:08 [IKEv1]IP = x.x.x.x, IKE Initiator: New Phase 1, Intf XXX, IKE Peer x.x.x.x local Proxy Address x.x.x.x, remote Proxy Address x.x.x.x, Crypto map (MAP)
Feb 28 14:42:08 [IKEv1 DEBUG]IP = x.x.x.x, constructing ISAKMP SA payload
Feb 28 14:42:08 [IKEv1 DEBUG]IP = x.x.x.x, constructing NAT-Traversal VID ver 02 payload
Feb 28 14:42:08 [IKEv1 DEBUG]IP = x.x.x.x, constructing NAT-Traversal VID ver 03 payload
Feb 28 14:42:08 [IKEv1 DEBUG]IP = x.x.x.x, constructing NAT-Traversal VID ver RFC payload
Feb 28 14:42:08 [IKEv1 DEBUG]IP = x.x.x.x, constructing Fragmentation VID + extended capabilities payload
Feb 28 14:42:08 [IKEv1]IP = x.x.x.x, IKE_DECODE SENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + VENDOR (13) + VENDOR (13) + VENDOR (13) + NONE (0) total length : 172
Feb 28 14:42:16 [IKEv1]IP = x.x.x.x, IKE_DECODE RESENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + VENDOR (13) + VENDOR (13) + VENDOR (13) + NONE (0) total length : 172
Feb 28 14:42:24 [IKEv1]IP = x.x.x.x, IKE_DECODE RESENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + VENDOR (13) + VENDOR (13) + VENDOR (13) + NONE (0) total length : 172
Feb 28 14:42:32 [IKEv1]IP = x.x.x.x, IKE_DECODE RESENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + VENDOR (13) + VENDOR (13) + VENDOR (13) + NONE (0) total length : 172
Feb 28 14:42:40 [IKEv1 DEBUG]IP = x.x.x.x, IKE MM Initiator FSM error history (struct &0x00002aaac3c6e190) <state>, <event>: MM_DONE, EV_ERROR-->MM_WAIT_MSG2, EV_RETRY-->MM_WAIT_MSG2, EV_TIMEOUT-->MM_WAIT_MSG2, NullEvent-->MM_SND_MSG1, EV_SND_MSG-->MM_SND_MSG1, EV_START_TMR-->MM_SND_MSG1, EV_RESEND_MSG-->MM_WAIT_MSG2, EV_RETRY
Feb 28 14:42:40 [IKEv1 DEBUG]IP = x.x.x.x, IKE SA MM:ad4019ef terminating: flags 0x01000022, refcnt 0, tuncnt 0
Feb 28 14:42:40 [IKEv1 DEBUG]IP = x.x.x.x, sending delete/delete with reason message

On the receiver's side, I can see nothing - same debug level enabled.

packet capture shows the message left the initiator but it didn't arrive at the responder's side.

ICMP is working during this period of unavailability.

After reload, everything works fine (the tunnel is UP), but in a few days, the issue returns.
I've changed the lifetime timer for phase 1 and the issue doesn't show up so often but it is still there.

Any ideas? :)

Thank You

Rahul Govindan · ‎03-05-2019

This is strange indeed. Any chance that your provider is limiting IKE and IPsec traffic after a certain limit- or sending through another path? This does not completely explain why the tunnel comes up after a reload. After the reload, did you check to see which side initiated the tunnel?

stanislav.pilat · ‎03-07-2019

Hi Rahul,

thanks for your reply.

I've contacted providers on both sides. They say nothing is blocked, but they will check some logs etc.. I don't think they will find something.

Yes, I've checked who is initiator - still the same ASA.. although there is one interesting find - when I try to build the tunnel from the opposite side I can see the state is MM_WAIT_MSG3. It's the same issue but only a little bit shifted = the packet from ASA A is lost in transit or the ASA A sends the packet only in its thoughts :)

I'm gonna upgrade both sides to 9.10.x. I don't think the upgrade will solve this issue but you never know, right? :)

SP.

Rahul Govindan · ‎03-07-2019

Ok, have you checked the working and non-working captures? What I would be interested to see is the destination MAC address of the outgoing packet. I am wondering if the ASA somehow builds the wrong arp table for your next hop (maybe a duplicate ip). That could also explain why a reload fixes the issue, but not how ping still works. Worth investigating this though.