Re: ASR1002 - %IPSEC-3-REPLAY_ERROR Y CPU 95%

lisandro · ‎12-29-2023

Dear, very good morning,
I have this problem that every 15 minutes I have the following log and all the tunnel interfaces of our ASR1002 that we have on the DC go down. This asr is our HUB and we have many branches like SPOKE.
When we see these logs, in addition to all interfaces in the tunnel being down, the CPU also reaches 95%.
Then they begin to lift until 15 more minutes have passed.

I add the record here

%IOSXE-3-PLATFORM: F0: cpp_cp: QFP:0.0 Thread:016 TS:00001222179961130001 %IPSEC-3-REPLAY_ERROR: IPSec SA receives anti-replay error, DP Handle 806, src_addr (IP DESTINATION BRANCH ROUTER) , dest_addr (ASR SOURCE IP ), SPI 0xbef52b76

058462: Dec 29 12:33:32.934 GTM: %DUAL-5-NBRCHANGE: EIGRP-IPv4 177: Neighbor 10.163.36.183 (Tunnel11) is down: holding time expired
058463: Dec 29 12:33:32.999 GTM: %DUAL-5-NBRCHANGE: EIGRP-IPv4 177: Neighbor 10.163.36.120 (Tunnel11) is down: holding time expired
058464: Dec 29 12:33:33.042 GTM: %DUAL-5-NBRCHANGE: EIGRP-IPv4 177: Neighbor 10.163.36.220 (Tunnel11) is down: holding time expired
058465: Dec 29 12:33:37.842 GTM: %DUAL-5-NBRCHANGE: EIGRP-IPv4 177: Neighbor 10.163.39.150 (Tunnel11) is down: holding time expired
058466: Dec 29 12:33:39.381 GTM: %DUAL-5-NBRCHANGE: EIGRP-IPv4 177: Neighbor 10.163.36.112 (Tunnel11) is down: holding time expired
058467: Dec 29 12:33:40.354 GTM: %DUAL-5-NBRCHANGE: EIGRP-IPv4 177: Neighbor 10.163.36.88 (Tunnel11) is down: holding time expired
058468: Dec 29 12:33:42.667 GTM: %DUAL-5-NBRCHANGE: EIGRP-IPv4 177: Neighbor 10.163.36.232 (Tunnel11) is down: holding time expired
058469: Dec 29 12:33:44.036 GTM: %DUAL-5-NBRCHANGE: EIGRP-IPv4 177: Neighbor 10.163.36.158 (Tunnel11) is down: holding time expired
058470: Dec 29 12:33:44.293 GTM: %DUAL-5-NBRCHANGE: EIGRP-IPv4 177: Neighbor 10.163.36.92 (Tunnel11) is down: holding time expired
058471: Dec 29 12:33:58.346 GTM: NHRP-ERROR: Rate limited: Too many packets
058472: Dec 29 12:33:58.738 GTM: NHRP-ERROR: Rate limited: Too many packets
058473: Dec 29 12:33:59.294 GTM: NHRP-ERROR: Rate limited: Too many packets
058474: Dec 29 12:33:59.364 GTM: NHRP-ERROR: Rate limited: Too many packets
058475: Dec 29 12:33:59.776 GTM: NHRP-ERROR: Rate limited: Too many packets
058476: Dec 29 12:33:59.812 GTM: NHRP-ERROR: Rate limited: Too many packets
058477: Dec 29 12:34:00.031 GTM: NHRP-ERROR: Rate limited: Too many packets
058478: Dec 29 12:34:00.341 GTM: %DUAL-5-NBRCHANGE: EIGRP-IPv4 177: Neighbor 10.163.36.220 (Tunnel11) is up: new adjacency
058479: Dec 29 12:34:00.352 GTM: %DUAL-5-NBRCHANGE: EIGRP-IPv4 177: Neighbor 10.163.36.183 (Tunnel11) is up: new adjacency
058480: Dec 29 12:34:00.593 GTM: NHRP-ERROR: Rate limited: Too many packets
058481: Dec 29 12:34:01.463 GTM: NHRP-ERROR: Rate limited: Too many packets
058482: Dec 29 12:34:01.817 GTM: NHRP-ERROR: Rate limited: Too many packets
058483: Dec 29 12:34:02.116 GTM: NHRP-ERROR: Rate limited: Too many packets
058484: Dec 29 12:34:03.609 GTM: %DUAL-5-NBRCHANGE: EIGRP-IPv4 177: Neighbor 10.163.36.158 (Tunnel11) is up: new adjacency
058485: Dec 29 12:34:03.615 GTM: %DUAL-5-NBRCHANGE: EIGRP-IPv4 177: Neighbor 10.163.36.120 (Tunnel11) is up: new adjacency
058486: Dec 29 12:34:03.616 GTM: %DUAL-5-NBRCHANGE: EIGRP-IPv4 177: Neighbor 10.163.36.92 (Tunnel11) is up: new adjacency
058487: Dec 29 12:34:03.622 GTM: %DUAL-5-NBRCHANGE: EIGRP-IPv4 177: Neighbor 10.163.36.232 (Tunnel11) is up: new adjacency
058488: Dec 29 12:34:07.488 GTM: %DUAL-5-NBRCHANGE: EIGRP-IPv4 177: Neighbor 10.163.36.88 (Tunnel11) is up: new adjacency
058489: Dec 29 12:34:07.547 GTM: %DUAL-5-NBRCHANGE: EIGRP-IPv4 177: Neighbor 10.163.36.112 (Tunnel11) is up: new adjacency
058490: Dec 29 12:34:07.571 GTM: %DUAL-5-NBRCHANGE: EIGRP-IPv4 177: Neighbor 10.163.39.150 (Tunnel11) is up: new adjacency

Ruben Cocheno · ‎12-29-2023

@lisandro

That seems to be related anti-replay droped packets and purpose of replay checks is to protect against malicious repetitions of packets. However, there are some scenarios where a failed replay check might not be due to a malicious reason:

The error might result from a sufficient packet that is reordered in the network path between the tunnel endpoints. This can likely occur if there are multiple network paths between the peers.

The error might be caused by unequal packet processing paths inside the Cisco IOS. For example, fragmented IPsec packets that require IP reassembly before decryption might be delayed enough, that they fall outside of the replay window by the time they are processed.
The error might be caused by the Quality of Service (QoS) enabled on the sending IPsec endpoint or within the network path. With the Cisco IOS implementation, IPsec encryption occurs before QoS in the egress direction. Certain QoS features, such as Low Latency Queueing (LLQ), could cause IPsec packet delivery to become out-of-order and dropped by the receiving endpoint due to a replay check failure.

A network configuration/operational issue can duplicate packets as they transit the network.

An attacker (man-in-the-middle) could potentially delay, drop, and duplicate the ESP traffic.

The Solution in such scenarios, it is possible to increase the size of the replay window or disable the replay check to ensure that such delays are considered acceptable and the legitimate packets are not discarded. By default, the replay window size is fairly small (window size of 64). If you increase the size, it does not greatly increase the risk of an attack. For information on how to configure an IPsec Anti-Replay Window, refer to the How to Configure IPsec Anti-Replay Window: Expanding and Disabling document.

Tag me to follow up.
Please mark it as Helpful and/or Solution Accepted if that is the case. Thanks for making Engineering easy again.
Connect with me for more on Linkedin https://www.linkedin.com/in/rubencocheno/

Leo Laohoo · ‎12-29-2023

What firmware is the router on?

Post the complete output to the command "sh platform resources".

lisandro · ‎01-02-2024

Hello, thanks for the response and sorry for the delay, I pass the command output

SEC-MPLS-01#sh platform resources
**State Acronym: H - Healthy, W - Warning, C - Critical
Resource Usage Max Warning Critical State
----------------------------------------------------------------------------------------------------
RP0 (ok, active) H
Control Processor 14.20% 100% 90% 95% H
DRAM 1760MB(46%) 3783MB 90% 95% H
ESP0(ok, active) H
Control Processor 22.16% 100% 90% 95% H
DRAM 626MB(66%) 946MB 90% 95% H
QFP H
TCAM 1372cells(2%) 65536cells 45% 55% H
DRAM 132273KB(50%) 262144KB 80% 90% H
IRAM 9329KB(7%) 131072KB 80% 90% H
SIP0 H
Control Processor 4.11% 100% 90% 95% H
DRAM 311MB(67%) 460MB 90% 95% H

Leo Laohoo · ‎01-02-2024

@lisandro wrote:

QFP H
 DRAM 132273KB(50%) 262144KB 80% 90% H
SIP0 H
 DRAM 311MB(67%) 460MB 90% 95% H

These are unusually high.

lisandro · ‎01-02-2024

What would that be? thank you

MHM Cisco World · ‎12-30-2023

can I see the config of Spoke and Hub tunnel
MHM

lisandro · ‎01-02-2024

Hello, good morning and sorry for the delay.

I pass the configuration of the HUB and one of the Spokes

----------------------------------------------------

HUB

interface Tunnel20
description DMVPN claro
bandwidth 100000
ip address 10.165.44.4 255.255.252.0
no ip redirects
ip mtu 1400
no ip next-hop-self eigrp 177
no ip split-horizon eigrp 177
ip nhrp map multicast dynamic
ip nhrp network-id 250
ip nhrp holdtime 300
ip tcp adjust-mss 1360
load-interval 60
cdp enable
tunnel source GigabitEthernet0/1/4
tunnel mode gre multipoint
tunnel vrf claro-mpls
tunnel protection ipsec profile claro-mpls

interface GigabitEthernet0/1/4
description MPLS claro
ip vrf forwarding claro-mpls
ip address 10.165.0.25 255.255.255.248
negotiation auto

---------------------------------------------------------

SPOKE

interface Tunnel20
description DMVPN claro
bandwidth 4000
ip address 10.165.44.116 255.255.252.0
no ip redirects
ip mtu 1400
ip flow ingress
ip flow egress
ip nat outside
ip nhrp map multicast 10.165.0.25
ip nhrp map 10.165.44.4 10.165.0.25
ip nhrp network-id 20
ip nhrp nhs 10.165.44.4
ip virtual-reassembly in
ip summary-address eigrp 177 10.131.160.0 255.255.248.0
ip tcp adjust-mss 1360
load-interval 30
delay 50000
qos pre-classify
cdp enable
tunnel source GigabitEthernet0/1
tunnel mode gre multipoint
tunnel vrf claro-mpls
tunnel protection ipsec profile claro-mpls

interface GigabitEthernet0/1
description MPLS claro
bandwidth 4096
ip vrf forwarding claro-mpls
ip address 10.165.3.162 255.255.255.248
load-interval 30
duplex auto
speed auto
service-policy output encolaSEC

LOG

004389: Jan 2 09:32:34.680 GTM: %IOSXE-3-PLATFORM: F0: cpp_cp: QFP:0.0 Thread:083 TS:00000236965258540210 %IPSEC-3-REPLAY_ERROR: IPSec SA receives anti-replay error, DP Handle 191, src_addr 10.165.3.162, dest_addr 10.165.0.25, SPI 0xcef02917

MHM Cisco World · ‎01-02-2024

ip nhrp holdtime 3600 <<- add this in both Hub and Spoke
tunnel key 20 <<- add this in both Hub and Spoke

do above and check

MHM

marce1000 · ‎12-30-2023

- Use latest and or latest advisory software version and check if that can help,

M.

-- Each morning when I wake up and look into the mirror I always say ' Why am I so brilliant ? '
When the mirror will then always repond to me with ' The only thing that exceeds your brilliance is your beauty! '