07-18-2024 07:02 AM
Hi there!
We've been running into the same issue a couple of times now and I do not seem to find any information on the ASR itself on when this is happening. Long story short:
- we have a ASR1006-X router (version 16.06.05) where we build different IPSec tunnels for different customers (in different VRFs).
- All of the tunnels are crypto-map based with every customer being a different entry.
- We've had a couple of cases now where IKEv2 (have not noticed IKEv1 to do this) phase-1 is built successfully, phase-2 SAs are established and everything seems to be running smoothly. However one of the SAs (not all) behaves weird, where no traffic is dropped, but no traffic is being encapsulated. After extensive troubleshooting in our network, we confirmed that the traffic is indeed being sent to the Cisco from the MPLS, but nothing is encapsulated into the IPSec.
No drops here:
R1#show cry ses ivrf customer_1 detail
Crypto session current status
Interface: GigabitEthernet0/1/1
Profile: customer_1
Uptime: 02:00:07
Session status: UP-ACTIVE
Peer: x.x.x.x port 500 fvrf: (none) ivrf: customer_1
Phase1_id: x.x.x.x
Desc: (none)
Session ID: 25096162
IKEv2 SA: local z.z.z.z/500 remote x.x.x.x/500 Active
Capabilities:(none) connid:97 lifetime:21:59:53
IPSEC FLOW: permit ip 10.133.80.0/255.255.240.0 10.30.208.0/255.255.240.0
Active SAs: 0, origin: crypto map
Inbound: #pkts dec'ed 999 drop 0 life (KB/Sec) 0/0
Outbound: #pkts enc'ed 0 drop 0 life (KB/Sec) 0/0
IPSEC FLOW: permit ip 10.133.80.0/255.255.240.0 10.30.240.0/255.255.240.0
Active SAs: 0, origin: crypto map
Inbound: #pkts dec'ed 123456 drop 0 life (KB/Sec) 0/0
Outbound: #pkts enc'ed 123456 drop 0 life (KB/Sec) 0/0
IPSEC FLOW: permit ip 10.133.80.0/255.255.240.0 10.153.0.0/255.255.0.0
Active SAs: 0, origin: crypto map
Inbound: #pkts dec'ed 7537 drop 0 life (KB/Sec) 0/0
Outbound: #pkts enc'ed 7537 drop 0 life (KB/Sec) 0/0
IPSEC FLOW: permit ip 10.133.80.0/255.255.240.0 10.155.0.0/255.255.0.0
Active SAs: 0, origin: crypto map
Inbound: #pkts dec'ed 4563456345 drop 0 life (KB/Sec) 0/0
Outbound: #pkts enc'ed 4563456345 drop 0 life (KB/Sec) 0/0
Nor here:
R1#sh crypto ipsec sa peer x.x.x.x detail
interface: GigabitEthernet0/1/1
Crypto map tag: my_crypto-map, local addr z.z.z.z
protected vrf: customer_1
local ident (addr/mask/prot/port): (10.133.80.0/255.255.240.0/0/0)
remote ident (addr/mask/prot/port): (10.30.208.0/255.255.240.0/0/0)
current_peer x.x.x.x port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
#pkts decaps: 999, #pkts decrypt: 999, #pkts verify: 999
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0
#pkts no sa (send) 0, #pkts invalid sa (rcv) 0
#pkts encaps failed (send) 0, #pkts decaps failed (rcv) 0
#pkts invalid prot (recv) 0, #pkts verify failed: 0
#pkts invalid identity (recv) 0, #pkts invalid len (rcv) 0
#pkts replay rollover (send): 0, #pkts replay rollover (rcv) 0
##pkts replay failed (rcv): 0
#pkts tagged (send): 0, #pkts untagged (rcv): 0
#pkts not tagged (send): 0, #pkts not untagged (rcv): 0
#pkts internal err (send): 0, #pkts internal err (recv) 0
local crypto endpt.: z.z.z.z, remote crypto endpt.: x.x.x.x
plaintext mtu 1500, path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0/1/1
current outbound spi: 0x0(0)
PFS (Y/N): N, DH group: none
inbound esp sas:
spi: 0xD360A3F0(3546326000)
transform: esp-gcm 256 ,
in use settings ={Tunnel, }
conn id: 42773, flow_id: HW:40773, sibling_flags FFFFFFFF80000048, crypto map: my_crypto-map
sa timing: remaining key lifetime (k/sec): (4607998/3247)
IV size: 8 bytes
replay detection support: Y replay window size: 512
Status: ACTIVE(ACTIVE)
inbound ah sas:
inbound pcp sas:
outbound esp sas:
spi: 0x664C02F0(1716257520)
transform: esp-gcm 256 ,
in use settings ={Tunnel, }
conn id: 42772, flow_id: HW:40772, sibling_flags FFFFFFFF80000048, crypto map: my_crypto-map
sa timing: remaining key lifetime (k/sec): (4607998/3247)
IV size: 8 bytes
replay detection support: Y replay window size: 512
Status: ACTIVE(ACTIVE)
outbound ah sas:
outbound pcp sas:
I've also tried to run monitor capture on the interface connected to the MPLS, to see if the traffic is even reaching the Cisco, but that was not successful. I've also ran every debug option related to "debug crypto", that I could find, but nothing in the logs showed packets to be dropped.
Running out of options, I started to look at the configuration from the customer side, for any hints on what's going on, when I noticed the difference in our and theirs ACLs. I then deleted the crypto-map entry, changed the ACLs on our end to match theirs, re-applied the crypto-map entry and everything worked.
So my question is - what did I miss in terms of troubleshooting (is there some place I could see these silently dropped packets) and why is the IPSec SA established in the first place, when the traffic selectors do not match. And the most interesting part was that even though they did not match, the customer was able to successfully forward traffic via the IPSec from their end.
The configuration on our end is very basic:
crypto ikev2 keyring customer_1
peer customer_1
address x.x.x.x
pre-shared-key My_KEY
!
crypto ikev2 profile customer_1
match identity remote address x.x.x.x 255.255.255.255
authentication remote pre-share
authentication local pre-share
keyring local customer_1
ivrf customer_1
!
crypto map my_crypto-map 2200 ipsec-isakmp
set peer x.x.x.x
set transform-set vpn-gcm-256
set pfs group20
set ikev2-profile customer_1
match address customer_1
reverse-route static
ip access-list extended customer_1
permit ip 10.133.80.0 0.0.15.255 10.30.208.0 0.0.15.255
permit ip 10.133.80.0 0.0.15.255 10.30.240.0 0.0.15.255
permit ip 10.133.80.0 0.0.15.255 10.153.0.0 0.0.255.255
permit ip 10.133.80.0 0.0.15.255 10.155.0.0 0.0.255.255
07-18-2024 11:53 AM
Hi Martin
you use ivrf so there is vrf use in this router
can I see the interface config crypto map under it and interface that connect to traffic need to encrypt
thanks
MHM
07-18-2024 10:55 PM
Hi MHM,
The interface configuration is as follows:
WAN interface, where the crypto-map is applied
Building configuration...
Current configuration : 449 bytes
!
interface GigabitEthernet0/1/1
ip address z.z.z.z 255.255.255.248
standby 10 ip z.z.z.1
standby 10 timers 1 4
standby 10 priority 110
standby 10 preempt delay minimum 60
standby 10 authentication password
standby 10 track 999 decrement 30
logging event subif-link-status
crypto map my_crypto-map redundancy my_crypto-map
end
MPLS link
Building configuration...
Current configuration : 474 bytes
!
interface Port-channel41.1809
encapsulation dot1Q 1809
ip address y.y.y.y 255.255.255.252
ip ospf authentication message-digest
ip ospf message-digest-key 1 md5 7 password
ip ospf network point-to-point
ip ospf dead-interval 4
ip ospf hello-interval 1
logging event subif-link-status
mpls ip
mpls ldp igp sync delay 5
bfd interval 750 min_rx 750 multiplier 3
end
07-20-2024 05:00 AM
we see small pieces of photo
there is HSRP and there config MPLS
can you draw topolgy
did you use RRI with HSRP
it can the default route use one HSRP but the GW use other HSRP and this break the IPSec
MHM
07-20-2024 05:23 AM
one other point,
the traffic selector for crypto map "policy based vpn" is same as protect prefix
the traffic selector for route based VPN is 0.0.0.0
so is ASR use policy based and other vendor use route-base then the traffic will drop
you need to match the VPN in both side
MHM
07-18-2024 05:29 PM - edited 07-18-2024 05:30 PM
Since this is an ASR there is a control plane and a data plane (hardware/TCAM). If the same IPSEC sa works sometimes and sometimes not, there is something either in the traffic selectors or the hardware programming.. you said there is a mismatch in traffic selectors ? how is it different ? have you compared working vs not working "show crypto ipsec sa " ..
you can run a packet-trace/FIA trace when you have the problem and that will confirm if the packet is coming into the ASR and then dropped by what feature ? or where it is going..
if you have TAC support, then open a case with them as they can help you better.
** Please mark as helpful if this was helpful :)**
07-18-2024 10:57 PM
Hi,
Since this is an ASR there is a control plane and a data plane (hardware/TCAM). If the same IPSEC sa works sometimes and sometimes not, there is something either in the traffic selectors or the hardware programming.. you said there is a mismatch in traffic selectors ? how is it different ? have you compared working vs not working "show crypto ipsec sa "
So the 2 cases that come to mind are the following:
1st one we had "10.0.0.0/22 to 10.138.40.0/29" as a traffic selector on the customer end, while on our end we had "10.138.40.0/29 to 10.0.0.0/24", "10.138.40.0/29 to 10.0.1.0/24" and "10.138.40.0/29 to 10.0.2.0/24". On our end there were 3 IPSec SAs built and the issue was that only one (at random) was not encrypting the traffic back to the customer. The decrypted packet counters were incrementing, so the customer was forwarding traffic to us for all the networks via their single SA. We suspect that the rotation of the non-working subnet was related to the SA lifetime, as at some point the non-working subnet started encrypting traffic, while at the same time one of the other networks stopped working.
2nd case was on our end we had "10.69.42.0/28 to 0.0.0.0/0" as a traffic selector, while on the customer end we had like 20 entries for different "10.0.x.0/2x networks to 10.69.42.0/28". On the ASR the IPSec SA was built and we were forwarding/encrypting back traffic from some networks, but others were not working. It was tricky to capture this issue, as from our perspective the issue was related to different networks and we had 0.0.0.0/0, so we were not supposed to treat them differently.
you can run a packet-trace/FIA trace when you have the problem and that will confirm if the packet is coming into the ASR and then dropped by what feature ? or where it is going..
I will definitely try this the next time we run into this kind of issue.
if you have TAC support, then open a case with them as they can help you better.
No TAC support unfortunately.
07-19-2024 12:24 AM
I forgot to attach the link for packet trace
but i suspect your problem may be to mismatch SAs.. do you have any overlapping ACLs on either side ? you may wan to check ipsec/ike debugs to see what is being send from your and what is being received from other side ? the responder can always accept a subset of proxy id ... so if you configure 10.0.0.0/8 - you as a responder can accept a smaller one like 10.1.1.0/24 ... so you want to verify there is no overlap and also match in terms of mask.. its possible that the proxy id are for smaller mask thus the encryption is not happening as it doest match the right mask... please verify that.. good luck...
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide