ipsec anti-replay errors on 1 gig VPN tunnel

[paul amaral]

Hi, I have two ASR 1001-x routers connected over a busy VPN tunnel. These routers are connected via Gig interface at 1000 mbs. This tunnel constantly goes over 800mbs on average. The issue is am seeing a lot of anti-replay errors on one side of the tunnel, the receiving router, let’s call it router B.

Dec 27 12:10:00.290: %IOSXE-3-PLATFORM: R0/0: cpp_cp: QFP:0.0 Thread:068 TS:00025270237225501860 %          IPSEC-3-REPLAY_ERROR: IPSec SA receives anti-replay error, DP Handle 37, src_addr 172.17.3.1, dest          _addr 172.17.3.2, SPI 0x24b2b94a

This is the receiving interface on router B, on avg this will go over 800mbs of constant legit traffic
Output queue: 0/40 (size/max)

  30 second input rate 717132000 bits/sec, 61034 packets/sec

On the sending side, router A I have QOS configured as such,

Policy Map Voice_and_Proto

    Class VOICE

      priority 25 (%)

    Class routing_proto_CM

      bandwidth 5 (%)

    Class voip-signal-cm

      bandwidth 10 (%)

    Class class-default

       packet-based wred, exponential weight 4
      queue limit 4166 packets

      dscp    min-threshold    max-threshold    mark-probablity

      ----------------------------------------------------------

      default (0)   -                -                1/10

      fair-queue
Fair-queue: per-flow queue limit 1041 packets

On both routers I have increased the replay window, crypto ipsec security-association replay window-size 1024. However, I’m still seeing a large amount if replay errors,

#pkts replay rollover (send): 0, #pkts replay rollover (rcv) 0

    ##pkts replay failed (rcv): 35901775

        replay detection support: Y  replay window size: 1024

I know the replay errors are related to the large amount of traffic but is there anything else I can do, especially with the QOS setup that will help alleviate some of these replay errors? I am seeing errors with or without congestions so are these errors common and expected on a 1 Gig ipsec link. Since I can’t further increase the replay window, I’m looking for ways to minimize the replay errors and make the link more efficient. I can disable the replay feature but I’m not sure if this is recommended. I also believe that my QOS is setup correctly and adjusting the queue limits will have no positive impact. My priority queue has at most about 1 meg of traffic and I’m not sure if this is causing a delay in other queues/packets leading to the anti-replay errors.

I did note on the sending router around 400mbs outgoing traffic is about 33K packets a second which is a lot more than the max replay size of 1024 on the receiving router and I’m assuming this is one of the reasons I’m seeing anti replay errors, there’s just a lot of encrypted traffic and at times packets can arrive out of order even though it’s a symmetrical link.

30 second output rate 368936000 bits/sec, 33345 packets/sec

I’m looking for some advice on what I can do here, and if disabling anti replay is my best bet or not.

TIA. Paul

[MHM Cisco World]

Increase ipsec window.

[paul amaral]

I have done that and increased the max window size to 1024 per original post  and still seeing it. Looking for further advise. 

thanks Paul

[MHM Cisco World]

https://www.cisco.com/c/en/us/support/docs/ip/internet-key-exchange-ike/116858-problem-replay-00.html#anc7

OK then check this doc. at it end there is link to pdf about the QoS for VoIP and Video in IPSec.
hope this help you.

[paul amaral · Accepted solution]

Looking more into my issue I found this was related to QOS combined with ipsec and the amount of high encrypted traffic generated, 1 Gig between routers. When you setup LLC queuing those packets will get de-queued first regardless of their anti replay sequence number and sometimes the sequence number on those packets can cause the anti replay window to slide causing lower priority queued packets to miss that window once de-queued. 

 Usually the solution is to increase the window size the max which is 1024, however this will not always work on a VPN tunnel that is constantly transmitting 1 gig of traffic and has QOS enabled. I already had the max window size configured and I still was seeing lots of dropped packets due to anti-replay failures. Note these failures can add up and will lead to retransmissions causing performance issues.

crypto ipsec security-association replay window-size 1024

What I ended up doing was to disable the anti replay on the side of the connection that was reporting all the anti replay errors and dropping packets. This was router B which is a real-time backup site that gets a constant real time data uploaded from router A. Router A has QOS setup on the outgoing interface that is used for ipsec source interface. The enabling of QOS with ipsec was the cause of my out of order packets that caused replay failures.  

crypto ipsec security-association replay disable

What I should of done initially was to setup multi sequence number with the following command,

crypto ipsec security-association multi-sn

This would cause the sending router and the receiving router to understand there's different QOS queues and packets from those queues will have their own sequence numbers. So packets in LLC will have their own sliding window as well as other packets in non priority queues. However this was a little late for me since this is part of DMVPN and once you enable the above command it must be enabled on all the routers or packets will be dropped.  More info here, https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/sec_conn_dplane/configuration/xe-16-8/sec-ipsec-data-plane-xe-16-8-book/sec-ipsec-antireplay.html

Again I ended up just disabling the anti-replay checks which for me is ok since this is private metro-e line and I'm not too worried about man in the middle attacks. If I was do to this from the beginning I would most likely enable multi-SN.  I hope this helps someone else.

thanks Paul