Silently dropping IPSec traffic due to miss-match in traffic selectors

Martin Varbanov · ‎07-18-2024

Hi there!

We've been running into the same issue a couple of times now and I do not seem to find any information on the ASR itself on when this is happening. Long story short:

- we have a ASR1006-X router (version 16.06.05) where we build different IPSec tunnels for different customers (in different VRFs).

- All of the tunnels are crypto-map based with every customer being a different entry.

- We've had a couple of cases now where IKEv2 (have not noticed IKEv1 to do this) phase-1 is built successfully, phase-2 SAs are established and everything seems to be running smoothly. However one of the SAs (not all) behaves weird, where no traffic is dropped, but no traffic is being encapsulated. After extensive troubleshooting in our network, we confirmed that the traffic is indeed being sent to the Cisco from the MPLS, but nothing is encapsulated into the IPSec.

No drops here:

R1#show cry ses ivrf customer_1 detail

Crypto session current status

Interface: GigabitEthernet0/1/1

Profile: customer_1

Uptime: 02:00:07

Session status: UP-ACTIVE

Peer: x.x.x.x port 500 fvrf: (none) ivrf: customer_1

Phase1_id: x.x.x.x

Desc: (none)

Session ID: 25096162

IKEv2 SA: local z.z.z.z/500 remote x.x.x.x/500 Active

Capabilities:(none) connid:97 lifetime:21:59:53

IPSEC FLOW: permit ip 10.133.80.0/255.255.240.0 10.30.208.0/255.255.240.0

Active SAs: 0, origin: crypto map

Inbound: #pkts dec'ed 999 drop 0 life (KB/Sec) 0/0

Outbound: #pkts enc'ed 0 drop 0 life (KB/Sec) 0/0

IPSEC FLOW: permit ip 10.133.80.0/255.255.240.0 10.30.240.0/255.255.240.0

Active SAs: 0, origin: crypto map

Inbound: #pkts dec'ed 123456 drop 0 life (KB/Sec) 0/0

Outbound: #pkts enc'ed 123456 drop 0 life (KB/Sec) 0/0

IPSEC FLOW: permit ip 10.133.80.0/255.255.240.0 10.153.0.0/255.255.0.0

Active SAs: 0, origin: crypto map

Inbound: #pkts dec'ed 7537 drop 0 life (KB/Sec) 0/0

Outbound: #pkts enc'ed 7537 drop 0 life (KB/Sec) 0/0

IPSEC FLOW: permit ip 10.133.80.0/255.255.240.0 10.155.0.0/255.255.0.0

Active SAs: 0, origin: crypto map

Inbound: #pkts dec'ed 4563456345 drop 0 life (KB/Sec) 0/0

Outbound: #pkts enc'ed 4563456345 drop 0 life (KB/Sec) 0/0

Nor here:

R1#sh crypto ipsec sa peer x.x.x.x detail

interface: GigabitEthernet0/1/1

Crypto map tag: my_crypto-map, local addr z.z.z.z

protected vrf: customer_1

local ident (addr/mask/prot/port): (10.133.80.0/255.255.240.0/0/0)

remote ident (addr/mask/prot/port): (10.30.208.0/255.255.240.0/0/0)

current_peer x.x.x.x port 500

PERMIT, flags={origin_is_acl,}

#pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0

#pkts decaps: 999, #pkts decrypt: 999, #pkts verify: 999

#pkts compressed: 0, #pkts decompressed: 0

#pkts not compressed: 0, #pkts compr. failed: 0

#pkts not decompressed: 0, #pkts decompress failed: 0

#pkts no sa (send) 0, #pkts invalid sa (rcv) 0

#pkts encaps failed (send) 0, #pkts decaps failed (rcv) 0

#pkts invalid prot (recv) 0, #pkts verify failed: 0

#pkts invalid identity (recv) 0, #pkts invalid len (rcv) 0

#pkts replay rollover (send): 0, #pkts replay rollover (rcv) 0

##pkts replay failed (rcv): 0

#pkts tagged (send): 0, #pkts untagged (rcv): 0

#pkts not tagged (send): 0, #pkts not untagged (rcv): 0

#pkts internal err (send): 0, #pkts internal err (recv) 0

local crypto endpt.: z.z.z.z, remote crypto endpt.: x.x.x.x

plaintext mtu 1500, path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0/1/1

current outbound spi: 0x0(0)

PFS (Y/N): N, DH group: none

inbound esp sas:

spi: 0xD360A3F0(3546326000)

transform: esp-gcm 256 ,

in use settings ={Tunnel, }

conn id: 42773, flow_id: HW:40773, sibling_flags FFFFFFFF80000048, crypto map: my_crypto-map

sa timing: remaining key lifetime (k/sec): (4607998/3247)

IV size: 8 bytes

replay detection support: Y replay window size: 512

Status: ACTIVE(ACTIVE)

inbound ah sas:

inbound pcp sas:

outbound esp sas:

spi: 0x664C02F0(1716257520)

transform: esp-gcm 256 ,

in use settings ={Tunnel, }

conn id: 42772, flow_id: HW:40772, sibling_flags FFFFFFFF80000048, crypto map: my_crypto-map

sa timing: remaining key lifetime (k/sec): (4607998/3247)

IV size: 8 bytes

replay detection support: Y replay window size: 512

Status: ACTIVE(ACTIVE)

outbound ah sas:

outbound pcp sas:

I've also tried to run monitor capture on the interface connected to the MPLS, to see if the traffic is even reaching the Cisco, but that was not successful. I've also ran every debug option related to "debug crypto", that I could find, but nothing in the logs showed packets to be dropped.

Running out of options, I started to look at the configuration from the customer side, for any hints on what's going on, when I noticed the difference in our and theirs ACLs. I then deleted the crypto-map entry, changed the ACLs on our end to match theirs, re-applied the crypto-map entry and everything worked.

So my question is - what did I miss in terms of troubleshooting (is there some place I could see these silently dropped packets) and why is the IPSec SA established in the first place, when the traffic selectors do not match. And the most interesting part was that even though they did not match, the customer was able to successfully forward traffic via the IPSec from their end.

The configuration on our end is very basic:

crypto ikev2 keyring customer_1

peer customer_1

address x.x.x.x

pre-shared-key My_KEY

!

crypto ikev2 profile customer_1

match identity remote address x.x.x.x 255.255.255.255

authentication remote pre-share

authentication local pre-share

keyring local customer_1

ivrf customer_1

!

crypto map my_crypto-map 2200 ipsec-isakmp

set peer x.x.x.x

set transform-set vpn-gcm-256

set pfs group20

set ikev2-profile customer_1

match address customer_1

reverse-route static

ip access-list extended customer_1

permit ip 10.133.80.0 0.0.15.255 10.30.208.0 0.0.15.255

permit ip 10.133.80.0 0.0.15.255 10.30.240.0 0.0.15.255

permit ip 10.133.80.0 0.0.15.255 10.153.0.0 0.0.255.255

permit ip 10.133.80.0 0.0.15.255 10.155.0.0 0.0.255.255

MHM Cisco World · ‎07-18-2024

Hi Martin
you use ivrf so there is vrf use in this router
can I see the interface config crypto map under it and interface that connect to traffic need to encrypt

thanks

MHM

Martin Varbanov · ‎07-18-2024

Hi MHM,

The interface configuration is as follows:

WAN interface, where the crypto-map is applied

Building configuration...

Current configuration : 449 bytes
!
interface GigabitEthernet0/1/1
ip address z.z.z.z 255.255.255.248
standby 10 ip z.z.z.1
standby 10 timers 1 4
standby 10 priority 110
standby 10 preempt delay minimum 60
standby 10 authentication password
standby 10 track 999 decrement 30
logging event subif-link-status
crypto map my_crypto-map redundancy my_crypto-map
end

MPLS link

Building configuration...

Current configuration : 474 bytes
!
interface Port-channel41.1809
encapsulation dot1Q 1809
ip address y.y.y.y 255.255.255.252
ip ospf authentication message-digest
ip ospf message-digest-key 1 md5 7 password
ip ospf network point-to-point
ip ospf dead-interval 4
ip ospf hello-interval 1
logging event subif-link-status
mpls ip
mpls ldp igp sync delay 5
bfd interval 750 min_rx 750 multiplier 3
end

MHM Cisco World · ‎07-20-2024

we see small pieces of photo
there is HSRP and there config MPLS
can you draw topolgy
did you use RRI with HSRP
it can the default route use one HSRP but the GW use other HSRP and this break the IPSec

MHM

MHM Cisco World · ‎07-20-2024

one other point,
the traffic selector for crypto map "policy based vpn" is same as protect prefix
the traffic selector for route based VPN is 0.0.0.0

so is ASR use policy based and other vendor use route-base then the traffic will drop
you need to match the VPN in both side

MHM

ccieexpert · ‎07-18-2024

Since this is an ASR there is a control plane and a data plane (hardware/TCAM). If the same IPSEC sa works sometimes and sometimes not, there is something either in the traffic selectors or the hardware programming.. you said there is a mismatch in traffic selectors ? how is it different ? have you compared working vs not working "show crypto ipsec sa " ..

you can run a packet-trace/FIA trace when you have the problem and that will confirm if the packet is coming into the ASR and then dropped by what feature ? or where it is going..

if you have TAC support, then open a case with them as they can help you better.

** Please mark as helpful if this was helpful :)**

Martin Varbanov · ‎07-18-2024

Hi,

Since this is an ASR there is a control plane and a data plane (hardware/TCAM). If the same IPSEC sa works sometimes and sometimes not, there is something either in the traffic selectors or the hardware programming.. you said there is a mismatch in traffic selectors ? how is it different ? have you compared working vs not working "show crypto ipsec sa "

So the 2 cases that come to mind are the following:

1st one we had "10.0.0.0/22 to 10.138.40.0/29" as a traffic selector on the customer end, while on our end we had "10.138.40.0/29 to 10.0.0.0/24", "10.138.40.0/29 to 10.0.1.0/24" and "10.138.40.0/29 to 10.0.2.0/24". On our end there were 3 IPSec SAs built and the issue was that only one (at random) was not encrypting the traffic back to the customer. The decrypted packet counters were incrementing, so the customer was forwarding traffic to us for all the networks via their single SA. We suspect that the rotation of the non-working subnet was related to the SA lifetime, as at some point the non-working subnet started encrypting traffic, while at the same time one of the other networks stopped working.

2nd case was on our end we had "10.69.42.0/28 to 0.0.0.0/0" as a traffic selector, while on the customer end we had like 20 entries for different "10.0.x.0/2x networks to 10.69.42.0/28". On the ASR the IPSec SA was built and we were forwarding/encrypting back traffic from some networks, but others were not working. It was tricky to capture this issue, as from our perspective the issue was related to different networks and we had 0.0.0.0/0, so we were not supposed to treat them differently.

you can run a packet-trace/FIA trace when you have the problem and that will confirm if the packet is coming into the ASR and then dropped by what feature ? or where it is going..

I will definitely try this the next time we run into this kind of issue.

if you have TAC support, then open a case with them as they can help you better.

No TAC support unfortunately.

ccieexpert · ‎07-19-2024

I forgot to attach the link for packet trace

https://www.cisco.com/c/en/us/support/docs/content-networking/adaptive-session-redundancy-asr/117858-technote-asr-00.html

but i suspect your problem may be to mismatch SAs.. do you have any overlapping ACLs on either side ? you may wan to check ipsec/ike debugs to see what is being send from your and what is being received from other side ? the responder can always accept a subset of proxy id ... so if you configure 10.0.0.0/8 - you as a responder can accept a smaller one like 10.1.1.0/24 ... so you want to verify there is no overlap and also match in terms of mask.. its possible that the proxy id are for smaller mask thus the encryption is not happening as it doest match the right mask... please verify that.. good luck...