ikev2 VPN tunnel troubleshooting

andreycgipokorskiy · ‎01-31-2023

Hello Community!

Need expert advice on troubleshooting the ikev2 VPN tunnel. The tunnel is in "UP" state and the remote and local selectors are also in UP state. The role of the tunnel is "RESPONDER" on our side. As far as I understand, this means that the remote site must initiate a VPN connection. Problem: we have local traffic that should connect to the remote servers, but the firewall has no routes to those IP addresses through the VPN (V ) is connected in the routing table. We have a second firewall and it works fine, no problem. Is it possible if the VPN s-t-s tunnel is initiated from the remote site as our ikev2 "ANSWER" tunnel role? Do I have to call the remote site to confirm the tunnel and access control list settings?

Thank you!
Sincerely,

Andrey P.

Rob Ingram · ‎01-31-2023

@andreycgipokorskiy run "show crypto ipsec sa" and determine if the encap|decap counters are increasing to confirm whether there are actually IPSec SAs established. The role responder means only the initiator can initially establish the tunnel, once up either side can transmit data (assuming firewall rules permit this). Only if the tunnel is down can the initiator establish the tunnel again. If you wanted either side to establish a tunnel, you'd configure both peers to be bidirectional, meaning they can act as initiator and responder.

andreycgipokorskiy · ‎01-31-2023

Thank you Rob! "show crypto ipsec sa | i encap|decap" shows no increments. So my thought was right when I said the remote side should initiate the tunnel connection But "show crypto ikev2 sa" shows that the tunnel's Status:UP-ACTIVE
Tunnel-id Local Remote Status Role
xxxxxxxx xxxxxxxxxx/500 xxxxxxxxxx/500 READY RESPONDER

The next step to check the remote side VPN settings
Is it correct?

Rob Ingram · ‎01-31-2023

@andreycgipokorskiy well IKEv2 SAs are established, but are there actually "inbound esp sas" and "outbound esp sas" (IPSec SA) established? Provide the output of "show crypto ipsec sa" for clarity.

Are there any hits on these counters? Or are they both 0?

Are there hits in one direction only (inbound, but no outbound or vice versa)?

These may indicate different issues.

Has the VPN ever worked?

andreycgipokorskiy · ‎01-31-2023

#show crypto ipsec sa
interface: internet_
Crypto map tag: map_1, seq num: 1, local addr: xxxxxxxxxx

access-list acl_name extended permit ip host xxxxxxxxxx host xxxxxxxxxx
local ident (addr/mask/prot/port): (xxxxxxxxxx/255.255.255.255/0/0)
remote ident (addr/mask/prot/port): (xxxxxxxxxx/255.255.255.255/0/0)
current_peer: xxxxxxxxxx

#pkts encaps: 20121, #pkts encrypt: 20121, #pkts digest: 20121
#pkts decaps: 30641, #pkts decrypt: 30641, #pkts verify: 30641
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 20121, #pkts comp failed: 0, #pkts decomp failed: 0
#pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
#PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
#TFC rcvd: 0, #TFC sent: 0
#Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
#send errors: 0, #recv errors: 0

local crypto endpt.: xxxxxxxxxx/500, remote crypto endpt.: xxxxxxxxxx/500
path mtu 1500, ipsec overhead 78(44), media mtu 1500
PMTU time remaining (sec): 0, DF policy: copy-df
ICMP error validation: disabled, TFC packets: disabled
current outbound spi: 8F80FE56
current inbound spi : 69FC3148

inbound esp sas:
spi: 0x69FC3148 (1778135368)
transform: esp-aes-256 esp-sha-256-hmac no compression
in use settings ={L2L, Tunnel, IKEv2, }
slot: 0, conn_id: 3330048, crypto-map: CSM_internet_bnc_map
sa timing: remaining key lifetime (kB/sec): (4331517/2560)
IV size: 16 bytes
replay detection support: Y
Anti replay bitmap:
0x00FFFFFF 0xFFFFFFFF
outbound esp sas:
spi: 0x8F80FE56 (2407595606)
transform: esp-aes-256 esp-sha-256-hmac no compression
in use settings ={L2L, Tunnel, IKEv2, }
slot: 0, conn_id: 3330048, crypto-map: CSM_internet_bnc_map
sa timing: remaining key lifetime (kB/sec): (4285438/2560)
IV size: 16 bytes
replay detection support: Y
Anti replay bitmap:
0x00000000 0x00000001

Crypto map tag: map_2, seq num: 3, local addr: 1xxxxxxxxxx

access-list acl_name extended permit ip host xxxxxxxxxx host xxxxxxxxxx
local ident (addr/mask/prot/port): (1xxxxxxxxxx/255.255.255.255/0/0)
remote ident (addr/mask/prot/port): (xxxxxxxxxx/255.255.255.255/0/0)
current_peer: xxxxxxxxxx

#pkts encaps: 11136, #pkts encrypt: 11136, #pkts digest: 11136
#pkts decaps: 11151, #pkts decrypt: 11151, #pkts verify: 11151
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 11136, #pkts comp failed: 0, #pkts decomp failed: 0
#pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
#PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
#TFC rcvd: 0, #TFC sent: 0
#Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
#send errors: 0, #recv errors: 0

local crypto endpt.: xxxxxxxxxx/500, remote crypto endpt.: xxxxxxxxxx/500
path mtu 1500, ipsec overhead 78(44), media mtu 1500
PMTU time remaining (sec): 0, DF policy: copy-df
ICMP error validation: disabled, TFC packets: disabled
current outbound spi: E6929A88
current inbound spi : 18063AE0

inbound esp sas:
spi: 0x18063AE0 (403061472)
transform: esp-aes-256 esp-sha-256-hmac no compression
in use settings ={L2L, Tunnel, IKEv2, }
slot: 0, conn_id: 4345856, crypto-map: CSM_internet_bnc_map
sa timing: remaining key lifetime (kB/sec): (4193274/601)
IV size: 16 bytes
replay detection support: Y
Anti replay bitmap:
0xFFFFFFFF 0xFFFFFFFF
outbound esp sas:
spi: 0xE6929A88 (3868367496)
transform: esp-aes-256 esp-sha-256-hmac no compression
in use settings ={L2L, Tunnel, IKEv2, }
slot: 0, conn_id: 4345856, crypto-map: CSM_internet_bnc_map
sa timing: remaining key lifetime (kB/sec): (4147194/601)
IV size: 16 bytes
replay detection support: Y
Anti replay bitmap:
0x00000000 0x00000001

andreycgipokorskiy · ‎01-31-2023

The counters are not incremented
There is no hits in both directions

Yes, It worked before, but I didn't find any changes for last two years on firewall

Rob Ingram · ‎01-31-2023

@andreycgipokorskiy your other message with the output of "show crypto ipsec sa" seems to have disappeared whilst I was typing this response.

The output looks fine, you've got 2 pairs of IPSec SAs, only those hosts objects defined in the crypto ACL with already establish IPSec SA will be encrypted over the VPN.

Are the servers not communicating over the VPN one of the hosts with an active IPSec SA (check the local ident|remote ident in "show crypto ipsec sa")? If not then interesting traffic will need to be generated, if the remote peer is initiator only, then they will need to generate the traffic to establish another IPSec SA or reconfigure the tunnel to be bidirectional.

MHM Cisco World · ‎01-31-2023

Yes please re-share the
show crypto ipsec sa <<-

andreycgipokorskiy · ‎01-31-2023

Rob
I can't share the output ... system deleted it for second time now

Rob Ingram · ‎01-31-2023

@andreycgipokorskiy can you please provide the answers to the questions above?

MHM Cisco World · ‎01-31-2023

show crypto ikev2 sa detail <<- I need to see output of this

andreycgipokorskiy · ‎01-31-2023

show crypto ikev2 sa detail

IKEv2 SAs:

Session-id:13769, Status:UP-ACTIVE, IKE count:1, CHILD count:1

Tunnel-id Local Remote Status Role
1570393723 xxx.xxx.xxx.xxx/500 xxx.xxx.xxx.xxx/500 READY RESPONDER
Encr: AES-CBC, keysize: 256, Hash: SHA256, DH Grp:2, Auth sign: PSK, Auth verify: PSK
Life/Active Time: 28800/17546 sec
Session-id: 13769
Status Description: Negotiation done
Local spi: AB8130CC31F4A66E Remote spi: F19599252F028F7D
Local id: xxx.xxx.xxx.xxx
Remote id: xxx.xxx.xxx.xxx
Local req mess id: 1235 Remote req mess id: 705
Local next mess id: 1235 Remote next mess id: 705
Local req queued: 1235 Remote req queued: 705
Local window: 1 Remote window: 1
DPD configured for 10 seconds, retry 2
NAT-T is not detected
Child sa: local selector xxx.xxx.xxx.xxx/0 - xxx.xxx.xxx.xxx/65535
remote selector xxx.xxx.xxx.xxx/0 - xxx.xxx.xxx.xxx/65535
ESP spi in/out: 0xf96b6470/0xc5378576
AH spi in/out: 0x0/0x0
CPI in/out: 0x0/0x0
Encr: AES-CBC, keysize: 256, esp_hmac: SHA256
ah_hmac: None, comp: IPCOMP_NONE, mode tunnel

IKEv2 SAs:

Session-id:14006, Status:UP-ACTIVE, IKE count:1, CHILD count:1

Tunnel-id Local Remote Status Role
1687900421 xxx.xxx.xxx.xxx/500 xxx.xxx.xxx.xxx/500 READY RESPONDER
Encr: AES-CBC, keysize: 256, Hash: SHA256, DH Grp:2, Auth sign: PSK, Auth verify: PSK
Life/Active Time: 28800/13334 sec
Session-id: 14006
Status Description: Negotiation done
Local spi: 0D20A313BD3B5C75 Remote spi: E3AFB3BEE6232BB8
Local id: xxx.xxx.xxx.xxx
Remote id: xxx.xxx.xxx.xxx
Local req mess id: 910 Remote req mess id: 542
Local next mess id: 910 Remote next mess id: 542
Local req queued: 910 Remote req queued: 542
Local window: 1 Remote window: 1
DPD configured for 10 seconds, retry 2
NAT-T is not detected
Child sa: local selector xxx.xxx.xxx.xxx/0 - xxx.xxx.xxx.xxx/65535
remote selector 172.23.190.4/0 - 172.23.190.4/65535
ESP spi in/out: 0x7cc0e76f/0x296a6966
AH spi in/out: 0x0/0x0
CPI in/out: 0x0/0x0
Encr: AES-CBC, keysize: 256, esp_hmac: SHA256
ah_hmac: None, comp: IPCOMP_NONE, mode tunnel

andreycgipokorskiy · ‎01-31-2023

I hop you can see it now

MHM Cisco World · ‎01-31-2023

show crypto ikev2 sa detail <<- I see this
but as you mention
show crypto ipsec sa <<- disappear again
can you share it last time ?

MHM Cisco World · ‎01-31-2023

You hidden the IP which I think public IP.
that OK.
what I get is
show crypto ikev2 sa <<- show two peer each is UP-active
show crypto ipsec sa <<- this also give two peer

but there is mismatch
first peer is match
ESP spi in/out: 0xf96b6470/0xc5378576
current outbound spi: C5378576
current inbound spi : F96B6470

second peer is mismatch
ESP spi in/out: 0x7cc0e76f/0x296a6966
current outbound spi: D50D7FB4
current inbound spi : 76979EEA

so try clear ipsec sa and check again.