We are seeing an issue with one of our Cisco 3900 series routers (C3900-SPE100/K9 ) that is performing as a spoke DMVPN router.
The issue is an increase in the #recv errors as shown below:
sh cry ipsec sa peer 195.95.131.57
Load for five secs: 4%/3%; one minute: 3%; five minutes: 2% Time source is NTP, 11:48:09.722 BST Wed Aug 17 2016
interface: Tunnel10
Crypto map tag: Tunnel10-head-0, local addr 10.7.2.5
protected vrf: corp_vrf
local ident (addr/mask/prot/port): (10.7.2.5/255.255.255.255/47/0)
remote ident (addr/mask/prot/port): (X.X.X.X/255.255.255.255/47/0)
current_peer 195.95.131.57 port 4500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 506869133, #pkts encrypt: 506869133, #pkts digest: 506869133
#pkts decaps: 483537993, #pkts decrypt: 483537993, #pkts verify: 483537993
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0
#send errors 0, #recv errors 74543
local crypto endpt.: 10.7.2.5, remote crypto endpt.: X.X.X.X
plaintext mtu 1442, path mtu 1500, ip mtu 1500, ip mtu idb (none)
current outbound spi: 0x30F8C714(821610260)
PFS (Y/N): N, DH group: none
inbound esp sas:
spi: 0xE052CBF7(3763522551)
transform: esp-256-aes esp-sha-hmac ,
in use settings ={Transport UDP-Encaps, }
conn id: 5395, flow_id: Onboard VPN:5395, sibling_flags 80004000, crypto map: Tunnel10-head-0
sa timing: remaining key lifetime (k/sec): (4103204/379)
IV size: 16 bytes
replay detection support: Y
Status: ACTIVE(ACTIVE)
inbound ah sas:
inbound pcp sas:
outbound esp sas:
spi: 0x30F8C714(821610260)
transform: esp-256-aes esp-sha-hmac ,
in use settings ={Transport UDP-Encaps, }
conn id: 5396, flow_id: Onboard VPN:5396, sibling_flags 80004000, crypto map: Tunnel10-head-0
sa timing: remaining key lifetime (k/sec): (4067350/379)
IV size: 16 bytes
replay detection support: Y
Status: ACTIVE(ACTIVE)
outbound ah sas:
outbound pcp sas:
The sh version command on the spoke router also provides the following information:
sh ver
Load for five secs: 1%/1%; one minute: 2%; five minutes: 2% Time source is NTP, 11:54:09.759 BST Wed Aug 17 2016 Cisco IOS Software, C3900 Software (C3900-UNIVERSALK9-M), Version 15.4(3)M1, RELEASE SOFTWARE (fc1) Technical Support: http://www.cisco.com/techsupport Copyright (c) 1986-2014 by Cisco Systems, Inc.
Compiled Sat 25-Oct-14 07:15 by prod_rel_team
ROM: System Bootstrap, Version 15.0(1r)M16, RELEASE SOFTWARE (fc1)
sfo1nr02p uptime is 3 weeks, 18 hours, 21 minutes System returned to ROM by reload at 17:13:32 BST Tue Jul 26 2016 System restarted at 17:32:43 BST Tue Jul 26 2016 System image file is "flash0:c3900-universalk9-mz.SPA.154-3.M1.bin"
Last reload type: Normal Reload
Last reload reason: Reload Command
This product contains cryptographic features and is subject to United States and local country laws governing import, export, transfer and use. Delivery of Cisco cryptographic products does not imply third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for compliance with U.S. and local country laws. By using this product you agree to comply with applicable laws and regulations. If you are unable to comply with U.S. and local laws, return this product immediately.
A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
If you require further assistance please contact us by sending email to export@cisco.com.
Cisco CISCO3925-CHASSIS (revision 1.0) with C3900-SPE100/K9 with 999424K/49152K bytes of memory.
Processor board ID FCZ190260MT
3 Gigabit Ethernet interfaces
1 terminal line
1 Virtual Private Network (VPN) Module
DRAM configuration is 72 bits wide with parity enabled.
255K bytes of non-volatile configuration memory.
255488K bytes of ATA System CompactFlash 0 (Read/Write)
License Info:
License UDI:
-------------------------------------------------
Device# PID SN
-------------------------------------------------
*1 C3900-SPE100/K9 x.x.x.x
Technology Package License Information for Module:'c3900'
------------------------------------------------------------------------
Technology Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------------
ipbase ipbasek9 Permanent ipbasek9
security securityk9 Permanent securityk9
uc None None None
data None None None
NtwkEss None None None
CollabPro None None None
Configuration register is 0x2102
To give a little more information on how the DMVPN solution is configured, we have a cisco ASA sat between the internet and the spoke router doing NAT, and the same deployment in Manchester where the HUB router is deployed.
The DMVPN tunnels are up and DMVPN appears to work as intended, however, I need to understand what the recv errors are as we are having a latency/speed performance issue with a core business application, the application is based in Manchester, the spoke is based in the US. The users in the US are reporting the issue when connecting to the application. It is my opinion that it is an issue with the application, however the receive error count has been flagged and I could do with help providing an explanation.
Many Thanks
Gareth
