VPN established but packets are not reaching the other end

[talalmakki]

Good day,

I have setup VPN (IKEv2) between two sites. Once is a Cisco 1921K9 with a security license and version 15.0(1r)M16 IOS, while the other end is a pfSense (some sort of SW I guess).

The other end is managed by another party.

The tunnel comes up in both phase1 and 2 and the inbound and outbound ESP sas are both active.

The tunnel worked for some time, then out of the blue, packets would not pass through to the other end.

Resetting the tunnel, reloading the router, rebuilding the configuration from scratch would not allow data flow.

I can see counters on the crypto ACL increasing when sending interesting traffic.

IPS is also showing increased counters for encapsulated, encrypted, and digested packets (all identical numbers) with 0 error counters.

The other side is reporting the same behavior.

we tried changing the mode to IKEv1 and got the exact same behavior: tunnel is fully up but traffic would not show up on the other end.

I made a debug on my side and kept getting the following:

*Aug 19 09:38:40.419: IKEv2:(SESSION ID = 2,SA ID = 1):Received Packet [From 83.98.173.23:500/To 41.242.27.194:500/VRF i0:f0]

Initiator SPI : 6001D6E7766C3D73 - Responder SPI : EE2BB20899CBB687 Message id: 72

IKEv2 INFORMATIONAL Exchange REQUEST

Payload contents:

*Aug 19 09:38:40.419: IKEv2:(SESSION ID = 2,SA ID = 1):Received DPD/liveness query

*Aug 19 09:38:40.419: IKEv2:(SESSION ID = 2,SA ID = 1):Building packet for encryption.

*Aug 19 09:38:40.419: IKEv2:(SESSION ID = 2,SA ID = 1):Sending ACK to informational exchange

*Aug 19 09:38:40.423: IKEv2:(SESSION ID = 2,SA ID = 1):Sending Packet [To 83.98.173.23:500/From 41.242.27.194:500/VRF i0:f0]

MRAYRTR01#Initiator SPI : 6001D6E7766C3D73 - Responder SPI : EE2BB20899CBB687 Message id: 72

IKEv2 INFORMATIONAL Exchange RESPONSE

Payload contents:

ENCR

Any idea??

[Rob Ingram]

@talalmakki 

So the "encaps" counters are increasing? but no "decaps"?

If encaps are increasing on your device that would imply you are sending traffic, does the other end confirm they are decrypting the traffic?

Have you both confirm that traffic is not unintentially being natted? Normally you'd define a NAT exemption rule.

How are you testing communication?

Provide the output of "show crypto ipsec sa" and your configuration

[talalmakki]

Yes encaps is increasing while decaps is still 0

The other side report exactly the same behavior, where they have increased encaps but 0 decaps.

Traffic on my side is configured correctly as I configured an extended ACL for use in PAT that excludes interesting traffic.

Testing is done using pings.

PS. Did I mention I have an IKEv1 VPN tunnel with another customer that is fully functional?

#sh cry ike sa
IPv4 Crypto IKEv2 SA

Tunnel-id Local Remote fvrf/ivrf Status
1 My_peer/500 Remote_peer/500 none/none READY
Encr: AES-CBC, keysize: 256, PRF: SHA256, Hash: SHA256, DH Grp:14, Auth sign: PSK, Auth verify: PSK
Life/Active Time: 28800/12390 sec

IPv6 Crypto IKEv2 SA

#sh cry ips sa peer remote_peer

interface: GigabitEthernet0/0
Crypto map tag: TRVMRA, local addr my_peer

protected vrf: (none)
local ident (addr/mask/prot/port): (10.226.0.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (10.210.212.0/255.255.255.0/0/0)
current_peer remote_peer port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 6204, #pkts encrypt: 6204, #pkts digest: 6204
#pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0
#send errors 0, #recv errors 0

local crypto endpt.: 41.242.27.194, remote crypto endpt.: 83.98.173.23
plaintext mtu 1438, path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0/0
current outbound spi: 0xC560DC4A(3311459402)
PFS (Y/N): Y, DH group: group14

inbound esp sas:
spi: 0x2638EE9(40079081)
transform: esp-256-aes esp-sha256-hmac ,
in use settings ={Tunnel, }
conn id: 2033, flow_id: Onboard VPN:33, sibling_flags 80000040, crypto map: TRVMRA
sa timing: remaining key lifetime (sec): 1821
Kilobyte Volume Rekey has been disabled
IV size: 16 bytes
replay detection support: Y
Status: ACTIVE(ACTIVE)

inbound ah sas:

inbound pcp sas:

outbound esp sas:
spi: 0xC560DC4A(3311459402)
transform: esp-256-aes esp-sha256-hmac ,
in use settings ={Tunnel, }
conn id: 2034, flow_id: Onboard VPN:34, sibling_flags 80000040, crypto map: TRVMRA
sa timing: remaining key lifetime (sec): 1821
Kilobyte Volume Rekey has been disabled
IV size: 16 bytes
replay detection support: Y
Status: ACTIVE(ACTIVE)

outbound ah sas:

outbound pcp sas: