topic Re: IPSEC tunnel Issues in Network Security

IPSEC tunnel Issues

CiscoBrownBelt — Fri, 21 Feb 2020 17:01:58 GMT

Have an ISPEC tunnel between an ASA and Router that will go down periodically and not be able to be brought back up and/or both sites can't reach each other unless the SAs are manually renegotiated on my end. Below is debug for platform/protocol 127 (changed IPs for security).

I am also looking for good docs in regards to reading IPSEC debugs or logs in general to be more familiar with the IPSEC commication process.

# IKEv2-PROTO-7: (27628): Timer expired, Sending DPD

IKEv2-PROTO-7: (27628): SM Trace-> SA: I_SPI=B537105995B57695 R_SPI=65C53893C7ED92A9 (I) MsgID = 00000001 CurState: READY Event: EV_SEND_DPD
IKEv2-PROTO-7: (27628): Action: Action_Null
IKEv2-PROTO-7: (27628): SM Trace-> SA: I_SPI=B537105995B57695 R_SPI=65C53893C7ED92A9 (I) MsgID = 00000001 CurState: INFO_I_BLD_INFO Event: EV_SEND_DPD
IKEv2-PROTO-4: (27628): Sending DPD/liveness query
IKEv2-PROTO-4: (27628): Building packet for encryption.
IKEv2-PROTO-7: (27628): SM Trace-> SA: I_SPI=B537105995B57695 R_SPI=65C53893C7ED92A9 (I) MsgID = 00000001 CurState: INFO_I_BLD_INFO Event: EV_ENCRYPT_MSG
IKEv2-PROTO-7: (27628): SM Trace-> SA: I_SPI=B537105995B57695 R_SPI=65C53893C7ED92A9 (I) MsgID = 00000001 CurState: INFO_I_BLD_INFO Event: EV_NO_EVENT
IKEv2-PLAT-4: (27628): Encrypt success status returned via ipc 1
IKEv2-PROTO-7: (27628): SM Trace-> SA: I_SPI=B537105995B57695 R_SPI=65C53893C7ED92A9 (I) MsgID = 00000001 CurState: INFO_I_BLD_INFO Event: EV_OK_ENCRYPT_RESP
IKEv2-PROTO-7: (27628): Action: Action_Null
IKEv2-PROTO-7: (27628): SM Trace-> SA: I_SPI=B537105995B57695 R_SPI=65C53893C7ED92A9 (I) MsgID = 00000001 CurState: INFO_I_BLD_INFO Event: EV_TRYSEND
IKEv2-PROTO-4: (27628): Checking if request will fit in peer window
(27628):
IKEv2-PROTO-4: (27628): Sending Packet [To 100.1.1.1:500/From 200.1.1.1:500/VRF i0:f0]
(27628): Initiator SPI : B537105995B57695 - Responder SPI : 65C53893C7ED92A9 Message id: 337
(27628): IKEv2 INFORMATIONAL Exchange REQUESTIKEv2-PROTO-5: (27628): Next payload: ENCR, version: 2.0 (27628): Exchange type: INFORMATIONAL, flags: INITIATOR (27628): Message id: 337, length: 88(27628):
Payload contents:
(27628): ENCR(27628): Next payload: NONE, reserved: 0x0, length: 60
(27628): Encrypted data: 56 bytes
(27628):
IKEv2-PLAT-5: (27628): SENT PKT [INFORMATIONAL] [200.1.1.1]:500->[100.1.1.1]:500 InitSPI=0xb537105995b57695 RespSPI=0x65c53893c7ed92a9 MID=00000151
IKEv2-PROTO-7: (27628): SM Trace-> SA: I_SPI=B537105995B57695 R_SPI=65C53893C7ED92A9 (I) MsgID = 00000151 CurState: INFO_I_BLD_INFO Event: EV_CHK_INFO_TYPE
IKEv2-PROTO-7: (27628): SM Trace-> SA: I_SPI=B537105995B57695 R_SPI=65C53893C7ED92A9 (I) MsgID = 00000151 CurState: INFO_I_WAIT Event: EV_NO_EVENT
(27628):
IKEv2-PROTO-4: (27628): Received Packet [From 100.1.1.1:500/To 200.1.1.1:500/VRF i0:f0]
(27628): Initiator SPI : B537105995B57695 - Responder SPI : 65C53893C7ED92A9 Message id: 337
(27628): IKEv2 INFORMATIONAL Exchange RESPONSEIKEv2-PROTO-5: (27628): Next payload: ENCR, version: 2.0 (27628): Exchange type: INFORMATIONAL, flags: RESPONDER MSG-RESPONSE (27628): Message id: 337, length: 88(27628):
Payload contents:
(27628):
(27628): Decrypted packet:(27628): Data: 88 bytes
IKEv2-PLAT-4: (27628): Decrypt success status returned via ipc 1
(27628): REAL Decrypted packet:(27628): Data: 0 bytes
IKEv2-PROTO-7: (27628): SM Trace-> SA: I_SPI=B537105995B57695 R_SPI=65C53893C7ED92A9 (I) MsgID = 00000151 CurState: INFO_I_WAIT Event: EV_RECV_INFO_ACK
IKEv2-PROTO-4: (27628): Processing ACK to informational exchange
IKEv2-PROTO-7: (27628): SM Trace-> SA: I_SPI=B537105995B57695 R_SPI=65C53893C7ED92A9 (I) MsgID = 00000151 CurState: INFO_I_WAIT Event: EV_CHK_INFO_TYPE
IKEv2-PROTO-7: (27628): SM Trace-> SA: I_SPI=B537105995B57695 R_SPI=65C53893C7ED92A9 (I) MsgID = 00000151 CurState: EXIT Event: EV_CHK_PENDING
IKEv2-PROTO-7: (27628): Processed response with message id 337, Requests can be sent from range 338 to 342
IKEv2-PROTO-7: (27628): SM Trace-> SA: I_SPI=B537105995B57695 R_SPI=65C53893C7ED92A9 (I) MsgID = 00000151 CurState: EXIT Event: EV_NO_EVENT
IKEv2-PROTO-7: (27628): SM Trace-> SA: I_SPI=B537105995B57695 R_SPI=65C53893C7ED92A9 (I) MsgID = 00000151 CurState: EXIT Event: EV_FREE_NEG
IKEv2-PROTO-7: (27628): Deleting negotiation context for my message ID: 0x151

Re: IPSEC tunnel Issues

Rob Ingram — Wed, 10 Apr 2019 15:39:51 GMT

Hi,

What are the lifetime timers configured on both the Router and ASA? They should be configured the same

I can see from the output above DPD is configured on one device, is it also configured on the other device?

Are you able to get the debugs from the other device at the sametime?

IPSec Troubleshooting and debug commands

IOS IKEv2 debug troubleshooting technote

ASA IKEv2 Debugs

Common ASA VPN troubleshooting

HTH

Re: IPSEC tunnel Issues

Richard Burts — Wed, 10 Apr 2019 17:20:16 GMT

I found the posted debug output difficult to interpret. The one thing that is clear is that this device sent an encrypted packet and successfully received and de-encrypted a packet from the peer.

The question about lifetime timers is a good one. If there is a period where there is not interesting traffic to send through the vpn then the timers may expire and the vpn comes down. It should come back up if there is interesting traffic to send through the vpn. Your comment that when the vpn is down that it must be brought back up from your end is interesting. It suggests that perhaps your end has a dynamically assigned IP address and the remote peer has a static IP. That is a valid configuration but one result of that is that the vpn must be initiated from the peer with the dynamic address and can not be initiated from the peer with the static IP. We can discuss why that is the case if your want. But for now I will not get into those details.

For purposes of troubleshooting I would suggest that you enable debug for isakmp. And as @Rob Ingram suggests it would be nice to see debug output from both sides.

HTH

Rick

Re: IPSEC tunnel Issues

CiscoBrownBelt — Wed, 10 Apr 2019 18:51:27 GMT

Awesome RIchard!

No all peer addresses are static. To clarify, I am the only one who brings the tunnel back up as I don't handle the remote end device, so getting debugs on that end is not easy at this point.

There are no manually configured dpd/keep-alives for the tunnel on my end so I assume it is just the default timers?? I do know the remote router has: dpd 10 2 on-demand - could that play a factor?

Yes I could run a debug for isakmp I will get back on that.

Re: IPSEC tunnel Issues

CiscoBrownBelt — Wed, 10 Apr 2019 18:54:21 GMT

Per my message to Richard, just to help you clarify all peer addresses are static. I am the only one who brings the tunnel back up as I don't handle the remote end device, so getting debugs on that end is not easy at this point.

There are no manually configured dpd/keep-alives for the tunnel on my end so I assume it is just the default timers?? I do know the remote router has: dpd 10 2 on-demand - could that play a factor?

Yes I could run a debug for isakmp I will get back on that.

Re: IPSEC tunnel Issues

Rob Ingram — Wed, 10 Apr 2019 19:17:30 GMT

The debug output on your end indicates it sent a DPD query, so I imagine your device is configured to send DPD.

Do you mean your device is literally the only peer that can bring the tunnel up? When the VPN was down did the 3rd party attempt to bring up the tunnel and fail? The remote device may have configured the VPN to respond/answer only, therefore your device can only initiate the establishment of the VPN tunnel. It's a long shot, but worth checking.

HTH

Re: IPSEC tunnel Issues

Richard Burts — Wed, 10 Apr 2019 21:19:08 GMT

DPD is enabled on ASA by default. So if there is no manual configuration then the default timers are used, which is idle for 10 seconds initiates the query and retries every 2 seconds. The threshold (how long idle) and retry values can be changed. For Cisco to Cisco site to site vpn both peers must enable DPD or both peers must disable DPD. It is possible to set it up so that a peer will respond to DPD query but will not send DPD query (set the threshold to infinite on the peer who will only respond).

See this link for documentation about DPD on ASA

https://www.cisco.com/c/en/us/td/docs/security/asa/asa94/config-guides/cli/vpn/asa-94-vpn-config/vpn-groups.html?bookSearch=true

I do not believe that DPD would cause problems. DPD was designed to identify and deal with a peer which has become unresponsive. If debug for ISAKMP shows that DPD is terminating sessions it is not the case that DPD caused the problem. What caused the problem was that the peer became unresponsive and DPD identified that situation.

So please do run the debug for ISAKMP and post the output.

HTH

Rick

Re: IPSEC tunnel Issues

Richard Burts — Wed, 10 Apr 2019 21:35:25 GMT

After posting my response I re-read the discussion and want to respond to this statement

the remote router has: dpd 10 2 on-demand - could that play a factor?

The router implementation is different from the ASA implementation, particularly in the option for on-demand (which is the default for IOS routers). When the router is operating on-demand it will respond to any DPD query it receives. But it does not send query on a periodic basis. Instead the router will only send the DPD query when it has something to send to the peer and has not received anything from the peer in a while.

I certainly do not believe that the router being configured dpd 10 2 on-demand is a factor in this issue.

HTH

Rick

Re: IPSEC tunnel Issues

Rob Ingram — Wed, 10 Apr 2019 21:39:05 GMT

Just to clarify, I was not implying DPD was the issue, merely attempting to confirm whether both devices had DPD configured as this would help resolve any issue.

Re: IPSEC tunnel Issues

Richard Burts — Wed, 10 Apr 2019 21:43:35 GMT

I appreciate the clarification. And I agree. I was responding more to the question from the original poster about whether dpd 10 2 on-demand might be a factor in the issue.

HTH

Rick

Re: IPSEC tunnel Issues

CiscoBrownBelt — Thu, 11 Apr 2019 02:17:26 GMT

Yes it sent it but perhaps I am overlooking but nothing shows up when doing show run | grep keep or similar type searches. I will double check.

Third party has never tried. Help desk typically calls me and I bring it up. I will have to try and check to see if it is configured for respond/answer only. Not sure about the config so I have to look it up I guess.

Re: IPSEC tunnel Issues

CiscoBrownBelt — Thu, 11 Apr 2019 02:18:53 GMT

Ok I will do that when possible and get back to you.

Re: IPSEC tunnel Issues

CiscoBrownBelt — Thu, 11 Apr 2019 02:20:07 GMT

Right gotcha. I was not sure.

Re: IPSEC tunnel Issues

Richard Burts — Thu, 11 Apr 2019 13:08:08 GMT

You mention that

perhaps I am overlooking but nothing shows up when doing show run | grep keep or similar type searches.

and I want to respond to that. On ASA dpd is enabled by default. So unless the config changes the default parameters it is expected that nothing would show up in show run.

At this point I believe that debug crypto isakmp will be the best tool to investigate this problem.

HTH

Rick

[edit] it occurs to me to ask how frequently does this happen? Is it daily? Weekly? or what?

Re: IPSEC tunnel Issues

CiscoBrownBelt — Thu, 11 Apr 2019 15:37:36 GMT

Thanks!
So I don't quite have the ISAKMP option - I was going to set a condition to show only debug for the trouble VPN. Shall I choose ikev2? See below:
ASA# debug crypto ?

ca Set PKI debug levels
condition Set IPSec/ISAKMP debug filters
engine Set crypto engine debug levels
goid Set crypto map GOID debug levels
ike-common Set IKE common debug levels
ikev1 Set IKEV1 debug levels
ikev2 Set IKEV2 debug levels
ipsec Set IPSec debug levels
ss-api Set Crypto Secure Socket API debug levels
vpnclient Set EasyVPN client debug levels

Re: IPSEC tunnel Issues

Rob Ingram — Thu, 11 Apr 2019 15:55:02 GMT

Your debugs indicate you are using IKEv2, so debug crypto ikev2

FYI, on newer firmware ISAKMP = IKEv1

HTH

Re: IPSEC tunnel Issues

Richard Burts — Thu, 11 Apr 2019 21:19:56 GMT

This is an interesting question whether to specify ikev1 or ikev2. My instinct was to suggest ikev1 since in my experience this is more widely used. But I looked at the debug in the original post. And as @Rob Ingram says, it is quite clear that this implementation is using ikev2. So +5 for that observation. I wonder if there are other helpful things we might learn if we saw a sanitized copy of the ASA config?

HTH

Rick

Re: IPSEC tunnel Issues

MSJ1 — Thu, 26 Jan 2023 02:04:35 GMT

What was the fix here ?