Hello Mohammed,

Mohammed Raza Ali · ‎12-17-2016

Dear Experts,

I have a problem with a Cisco 2811 - ISR Router, every day the Utilization of CPU is high, this has been observed from past 3 months, below are the few outputs:

show version

Cisco IOS Software, 2800 Software (C2800NM-ADVIPSERVICESK9-M), Version 12.4(15)XZ, RELEASE SOFTWARE (fc2) Technical Support: http://www.cisco.com/techsupport Copyright (c) 1986-2008 by Cisco Systems, Inc.

Compiled Fri 11-Apr-08 17:50 by prod_rel_team

ROM: System Bootstrap, Version 12.4(13r)T, RELEASE SOFTWARE (fc1)

Router uptime is 40 weeks, 1 day, 1 hour, 39 minutes System returned to ROM by power-on System restarted at 19:01:39 GST Wed Mar 9 2016 System image file is "flash:c2800nm-advipservicesk9-mz.124-15.XZ.bin"

This product contains cryptographic features and is subject to United States and local country laws governing import, export, transfer and use. Delivery of Cisco cryptographic products does not imply third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for compliance with U.S. and local country laws. By using this product you agree to comply with applicable laws and regulations. If you are unable to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to export@cisco.com.

Cisco 2811 (revision 53.51) with 249856K/12288K bytes of memory.

Processor board ID FCZ123273CD

2 FastEthernet interfaces

2 Virtual Private Network (VPN) Modules

DRAM configuration is 64 bits wide with parity enabled.

239K bytes of non-volatile configuration memory.

62720K bytes of ATA CompactFlash (Read/Write)

Configuration register is 0x2102

show process cpu sorted | ex 0.00

CPU utilization for five seconds: 26%/13%; one minute: 58%; five minutes: 60%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

117 2486832220 757054161 3284 11.71% 26.68% 26.33% 0 IP Input

183 122771641317795678 9 0.73% 0.55% 0.52% 0 HQF Shaper Backg

74 108583200 39296482 2763 0.49% 0.43% 0.43% 0 BGP I/O

313 28096760 25259035 1112 0.16% 0.10% 0.08% 0 BGP Router

3 872 337 2587 0.08% 0.24% 0.15% 514 SSH Process

116 3330516 736225440 4 0.08% 0.09% 0.08% 0 IP ARP Retry Age

184 1982604 239702639 8 0.08% 0.04% 0.06% 0 RBSCP Background

110 3346356 736225798 4 0.08% 0.07% 0.08% 0 ACCT Periodic Pr

Show log

001266: Dec 11 11:28:12.150 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=3563, sequence number=620948

001267: Dec 11 12:34:23.517 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=3571, sequence number=840741

001268: Dec 11 15:22:00.901 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=3595, sequence number=817406

001269: Dec 11 16:42:53.192 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=3611, sequence number=121177

001270: Dec 11 17:41:01.985 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=3619, sequence number=168948

001271: Dec 11 18:06:50.324 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=3619, sequence number=585921

001272: Dec 11 23:08:39.310 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=3659, sequence number=570476

001273: Dec 12 09:12:24.304 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=3747, sequence number=1350227

001274: Dec 12 10:24:05.597 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=3759, sequence number=723113

001276: Dec 12 11:05:08.874 GST: %PARSER-6-EXPOSEDLOCKACQUIRED: Exclusive configuration lock acquired by user 'USER' from terminal '514' -Process= "SSH Process", ipl= 0, pid= 3

001277: Dec 12 11:05:18.774 GST: %SYS-5-CONFIG_I: Configured from console by USER on vty0 (10.2.10.27)

001278: Dec 12 11:05:18.778 GST: %PARSER-6-EXPOSEDLOCKRELEASED: Exclusive configuration lock released from terminal '514' -Process= "SSH Process", ipl= 0, pid= 3

001279: Dec 12 11:10:31.572 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=3765, sequence number=1553543

001280: Dec 12 12:01:45.518 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=3771, sequence number=2490532

001281: Dec 12 13:35:46.700 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=3789, sequence number=659595

001282: Dec 12 13:51:00.991 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=3789, sequence number=1498685

001283: Dec 12 14:03:49.253 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=3789, sequence number=2109470

001284: Dec 12 15:27:30.222 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=3805, sequence number=1084270

001285: Dec 12 15:44:48.963 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=3805, sequence number=2223527

001286: Dec 12 17:03:57.042 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=3817, sequence number=2708864

001287: Dec 12 17:15:59.098 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=3823, sequence number=442718

001288: Dec 12 18:04:25.809 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=3829, sequence number=910650

001289: Dec 12 18:08:16.260 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=3829, sequence number=1133966

001290: Dec 12 18:47:48.061 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=3835, sequence number=667072

001291: Dec 12 19:23:19.240 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=3835, sequence number=1693584

001292: Dec 12 19:52:21.167 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=3843, sequence number=360203

001293: Dec 13 12:01:55.546 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=3981, sequence number=1806725

001294: Dec 13 12:26:21.422 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=3987, sequence number=992481

001295: Dec 13 15:35:47.794 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=13, sequence number=2091582

001296: Dec 13 16:53:31.182 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=25, sequence number=2032108

001297: Dec 13 16:55:09.209 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=25, sequence number=2123303

001298: Dec 13 19:17:13.622 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=43, sequence number=2317325

001299: Dec 13 21:07:08.595 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=59, sequence number=454916

001300: Dec 13 21:12:35.112 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=59, sequence number=755127

001301: Dec 13 21:42:07.769 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=67, sequence number=204757

001302: Dec 13 23:57:47.947 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=83, sequence number=420100

001303: Dec 14 09:36:46.604 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=163, sequence number=2969434

001304: Dec 14 09:40:25.174 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=169, sequence number=286185

001305: Dec 14 09:45:47.392 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=169, sequence number=747782

001306: Dec 14 10:25:09.141 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=177, sequence number=435770

001307: Dec 14 12:12:34.831 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=195, sequence number=394654

001308: Dec 14 14:07:52.184 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=205, sequence number=2384515

001309: Dec 14 14:10:15.344 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=205, sequence number=2566772

001310: Dec 14 14:39:52.633 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=213, sequence number=1632877

001311: Dec 14 16:42:58.064 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=231, sequence number=2204225

001312: Dec 14 17:48:57.727 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=243, sequence number=1180955

001313: Dec 14 18:17:22.145 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=249, sequence number=378495

001314: Dec 14 18:18:57.384 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=249, sequence number=489448

001315: Dec 14 18:50:44.377 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=249, sequence number=2485322

001316: Dec 14 20:24:21.128 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=265, sequence number=399448

001317: Dec 14 23:57:06.618 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=297, sequence number=40209

001318: Dec 15 01:27:48.569 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=305, sequence number=399464

001319: Dec 15 02:32:47.185 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=313, sequence number=345511

001320: Dec 15 08:46:22.828 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=369, sequence number=485465

001321: Dec 15 08:50:59.168 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=369, sequence number=708008

001322: Dec 15 09:28:25.736 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=369, sequence number=2817168

001323: Dec 15 13:30:42.110 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=407, sequence number=2896334

001324: Dec 15 14:16:47.507 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=421, sequence number=371237

001325: Dec 15 14:18:05.021 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=421, sequence number=483888

001326: Dec 15 14:37:30.334 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=421, sequence number=1472473

001327: Dec 15 15:22:16.235 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=429, sequence number=1761495

001328: Dec 15 16:11:11.667 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=433, sequence number=2629008

001329: Dec 15 16:15:18.446 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=433, sequence number=2980815

001330: Dec 15 16:19:08.916 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=439, sequence number=210708

001331: Dec 15 16:58:39.766 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=447, sequence number=148941

001332: Dec 15 17:26:15.937 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=447, sequence number=2410394

001333: Dec 15 18:04:04.492 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=451, sequence number=2344264

001334: Dec 15 18:23:43.449 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=457, sequence number=824403

001336: Dec 15 18:57:31.255 GST: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

connection id=465, sequence number=78886

Please let me know if you need more information from my side.

Georg Pauwen · ‎12-18-2016

Hello,

try and increase the IPSec Anti Replay Windows size (default is 64) to e.g. 256:

2811ISR(config)#crypto ipsec security-association replay window-size 256

The error message you see in your logs might also come from a replay attack. Check the below document:

http://www.cisco.com/c/en/us/support/docs/ip/internet-key-exchange-ike/116858-problem-replay-00.html

Mohammed Raza Ali · ‎12-19-2016

Hi buddy,

Thanks for your valuable input.

How can I check the current window-size..

Also, if I change to 256, does it needs to be changed on other end too, since the connectivity is via MPLS.

One last thing, if this doesn't work out, can I revert it back to original size(64).

Regards,

Mohammed.

Georg Pauwen · ‎12-20-2016

Hello,

you can check the current replay window size with the 'show crypto map tag X' as in the example below:

---------------------------------------------------------------------------

2811ISR# show crypto map tag VPN_IPSEC

Crypto Map "VPN_IPSEC" 10 ipsec-isakmp
WARNING: This crypto map is in an incomplete state!
(missing peer or access-list definitions)
No matching address list set.
Security association lifetime: 4608000 kilobytes/3600 seconds
PFS (Y/N): N
Transform sets={
}
Antireplay window size = 128

--------------------------------------------------------------------------

You don't need to change it at the other end. The maximum window size is 1024, so you might as well configure the maximum value, as there is no impact on security or performance.

You can always revert back to the originally configured value.

Mohammed Raza Ali · ‎12-26-2016

Dear Pauwen,

Sorry for responding it lately.

Can I revert it back to default size, if it doesn't work on 256.

Thanks in advance.

Regards,

Mohammed.

Georg Pauwen · ‎12-26-2016

Hello Mohammed,

yes, you can always revert back to the default size of 64. That said, what is the size now ?

2811ISR# show crypto map tag VPN_IPSEC

Crypto Map "VPN_IPSEC" 10 ipsec-isakmp
WARNING: This crypto map is in an incomplete state!
(missing peer or access-list definitions)
No matching address list set.
Security association lifetime: 4608000 kilobytes/3600 seconds
PFS (Y/N): N
Transform sets={
}
Antireplay window size = 128 <-- ?

High CPU Usage in 2811 Router