Re: 60Mb internet connection slow to 20 Mb (Down) speed on DMVPN connection.

jasonku · ‎12-15-2017

We have 60Mb (down)/6 MB (Up) internet service at remote site. We are getting close to 60MB (down)/6 Mb (up) without DMVPN tunnel but it dropped to 20 Mb down/6 Mb up if we enabled the DMVPN connection. The router model is CISCO2921/K9 at remote site and ASR 1002-X on hub sites. This is only site is using Cisco 2921 model and other remote sites have ISR4300 router.

Does anyone have an idea why we are getting only 20 Mb down speed on DMVPN connection?

Is this hardware limitation of Cisco 2921 model? Any help would be appreciated.

Remote Config:

crypto isakmp policy 100
encr aes
hash md5
authentication pre-share
group 2
crypto isakmp key xxxxxxx address xx.xx.xx.xx
!
!
crypto ipsec transform-set dmvpn2-trans esp-aes esp-md5-hmac
mode transport
!
crypto ipsec profile DMVPN2
set transform-set dmvpn2-trans
!
!

interface Tunnel2
description DMVPN to HUB
ip address 10.255.202.32 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp map multicast dynamic
ip nhrp map 10.255.202.1 xx.xx.xx.xx
ip nhrp map multicast xx.xx.xx.xx
ip nhrp network-id 2
ip nhrp holdtime 300
ip nhrp nhs 10.255.202.1
ip tcp adjust-mss 1360
keepalive 5 2
tunnel source GigabitEthernet0/1
tunnel mode gre multipoint
tunnel key xxxx
tunnel protection ipsec profile DMVPN2
end

interface GigabitEthernet0/1
ip address 47.xx.xx.xx 255.255.255.252
duplex full
speed 1000
end

Joseph W. Doherty · ‎12-15-2017

From Cisco's performance white paper, your 2921 should be able to handle 60/6 Mbps. The whitepaper documents the 2921 of handling 105 Mbps of encrypted traffic (at 75% CPU, IMIX).

What your CPU stats look like when you top out?

Georg Pauwen · ‎12-15-2017

Hello,

is your HSEC-K9 license activated on the 2921 ?

jasonku · ‎12-19-2017

yes we applied SEC-K-9 license on this router. I believe this license allowed upto 85 Mbps unidirectional.

-----------------------------------------------------------------
Technology Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------
appx None None None
uc None None None
security securityk9 RightToUse securityk9
ipbase ipbasek9 Permanent ipbasek9

Georg Pauwen · ‎12-19-2017

Hello,

just to be sure, can you post the output of 'show license feature' ?

jasonku · ‎12-21-2017

Here is the output of "show license feature"

MAN-MDF-F1-RT2#show license feature
Feature name Enforcement Evaluation Subscription Enabled RightToUse
ipbasek9 no no no yes no
securityk9 yes yes no yes yes
uck9 yes yes no no yes
datak9 yes yes no no yes
NtwkEssSuitek9 yes yes no no yes
CollabProSuitek9 yes yes no no yes
ios-ips-update yes yes yes no yes
SNASw yes yes no no yes
hseck9 yes no no no no
cme-srst yes yes no no yes
mgmt-plug-and-play yes no no no no
mgmt-lifecycle yes no no no no
mgmt-assurance yes no no no no
mgmt-onplus yes no no no no
mgmt-compliance yes no no no no

Georg Pauwen · ‎12-21-2017

Hello,

the license looks ok. Can you post the full configuration of the router ? We might be able to fine tune a few things. In theory, you should get 85MB up/down...

jasonku · ‎12-21-2017

I really appreciate your response. Here is the running config.

MAN-MDF-F1-RT2#terminal length 0
MAN-MDF-F1-RT2#sh run
Building configuration...

Current configuration : 7895 bytes
!
! Last configuration change at 14:47:34 CST Thu Dec 14 2017 by jku.admin
! NVRAM config last updated at 14:48:09 CST Thu Dec 14 2017 by jku.admin
!
version 15.4
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime msec localtime
service timestamps log datetime msec localtime
service password-encryption
!
hostname MAN-MDF-F1-RT2
!
boot-start-marker
boot-end-marker
!
!
card type t1 0 0
logging buffered 128000 informational
logging monitor informational
enable secret 5 xxxxxxx
!
aaa new-model
!
!
aaa group server tacacs+ TACACS-ACS
server 10.8.12.28
server 10.8.12.29
ip tacacs source-interface Loopback0
!
aaa authentication login default group TACACS-ACS local-case
aaa authentication login no_tacacs line
aaa authorization exec default group TACACS-ACS local if-authenticated
aaa authorization commands 1 default group TACACS-ACS local
aaa authorization commands 15 default group TACACS-ACS local
aaa accounting commands 1 default start-stop group TACACS-ACS
aaa accounting commands 15 default start-stop group TACACS-ACS
!
!
!
!
!
aaa session-id common
clock timezone CST -6 0
clock summer-time CST recurring
clock calendar-valid
no network-clock-participate wic 0
!
!
!
!
!
!
no ip source-route
!
!
!
!
!
!
!
!
no ip bootp server
ip domain name deltakedu.corp
ip cef
no ipv6 cef
!
multilink bundle-name authenticated
!
!
cts logging verbose
!
!
license udi pid CISCO2921/K9 sn xxxxx
license accept end user agreement
!
!
archive
log config
logging enable
notify syslog contenttype plaintext
hidekeys
path ftp://CiscoIOSBackup:105Backup!@10.8.16.52/$h
write-memory
username xxxx privilege 15 secret 5 xxxx
!
redundancy
!
!
!
!
!
controller T1 0/0/0
cablelength long 0db
channel-group 0 timeslots 1-24
!
controller T1 0/0/1
cablelength long 0db
channel-group 0 timeslots 1-24
!
!
!
crypto isakmp policy 100
encr aes
hash md5
authentication pre-share
group 2
crypto isakmp key xxxx@xxx address 208.78.xx.xx
!
!
crypto ipsec transform-set dmvpn2-trans esp-aes esp-md5-hmac
mode transport
!
crypto ipsec profile DMVPN2
set transform-set dmvpn2-trans
!
!
!
!
!
!
!
interface Loopback0
ip address 10.155.0.2 255.255.255.255
!
interface Tunnel2
description DMVPN to HUB
ip address 10.255.202.32 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp map multicast dynamic
ip nhrp map 10.255.202.1 xx.xx.168.14
ip nhrp map multicast xx.xx.168.14
ip nhrp network-id 2
ip nhrp holdtime 300
ip nhrp nhs 10.255.202.1
ip tcp adjust-mss 1360
keepalive 5 2
tunnel source GigabitEthernet0/1
tunnel mode gre multipoint
tunnel key xxxx
tunnel protection ipsec profile DMVPN2
!
interface Embedded-Service-Engine0/0
no ip address
shutdown
!
interface GigabitEthernet0/0
description CONNECTED MAN-MDF-F1-SW1
ip address 10.155.2.2 255.255.255.0
duplex auto
speed auto
!
interface GigabitEthernet0/1
ip address xx.xx.70.138 255.255.255.252
duplex full
speed 1000
!
interface GigabitEthernet0/2
no ip address
shutdown
duplex auto
speed auto
!
interface Serial0/0/0:0
no ip address
!
interface Serial0/0/1:0
no ip address
!
router ospf 1
router-id 10.155.0.2
auto-cost reference-bandwidth 10000
redistribute bgp 65155 subnets route-map BGP_INTO_OSPF
passive-interface default
no passive-interface GigabitEthernet0/0
network 10.155.0.2 0.0.0.0 area 0
network 10.155.2.0 0.0.0.255 area 0
default-information originate metric 200 metric-type 1
!
router bgp 65155
bgp log-neighbor-changes
redistribute ospf 1 route-map OSPF_INTO_BGP
neighbor 10.155.0.1 remote-as 65155
neighbor 10.155.0.1 update-source Loopback0
neighbor 10.155.0.1 next-hop-self
neighbor 10.155.0.1 soft-reconfiguration inbound
neighbor 10.255.202.1 remote-as 65152
neighbor 10.255.202.1 send-community
neighbor 10.255.202.1 soft-reconfiguration inbound
neighbor 10.255.202.1 route-map DMVPN-HUB-IN in
neighbor 10.255.202.1 route-map DMVPN-OUT out
!
ip forward-protocol nd
!
ip bgp-community new-format
ip as-path access-list 1 permit ^$
no ip http server
no ip http secure-server
!
ip tftp source-interface GigabitEthernet0/0
ip route xx.xx.168.14 255.255.255.255 xx.xx.70.137
ip tacacs source-interface Loopback0
ip ssh source-interface Loopback0
ip ssh version 2
!
ip access-list extended DMVPN-INBOUND-RULE
permit udp host xx.xx.168.14 eq isakmp host xx.xx.70.138 eq isakmp
permit udp host xx.xx.168.14 eq non500-isakmp host xx.xx.70.138 eq non500-isakmp
permit esp host xx.xx.168.14 host xx.xx.70.138
permit tcp host xx.xx.168.14 host xx.xx.70.138 eq 22
permit icmp host xx.xx.168.14 host xx.xx.70.138
!
!
ip prefix-list DMVPN-HUB-LOOPBACK seq 5 permit xx.xx.168.14/32
ip prefix-list DMVPN-HUB-LOOPBACK seq 10 permit 10.255.200.1/32
!
ip prefix-list DefaultRoute seq 5 permit 0.0.0.0/0
!
ip prefix-list Local_Management_Subnet seq 10 permit 10.155.0.1/32
ip prefix-list Local_Management_Subnet seq 20 permit 10.155.0.2/32
ip prefix-list Local_Management_Subnet seq 30 permit 10.155.0.3/32
ip prefix-list Local_Management_Subnet seq 40 permit 10.155.0.241/32
!
route-map DMVPN-HUB-IN deny 10
match ip address prefix-list DMVPN-HUB-LOOPBACK
!
route-map DMVPN-HUB-IN permit 20
match ip address prefix-list DefaultRoute
set local-preference 90
!
route-map DMVPN-OUT permit 5
description add 1:209 to local management subnet
match ip address prefix-list Local_Management_Subnet
set community 1:209
!
route-map DMVPN-OUT permit 10
description PERMIT LCAL AS ROUTES ONLY
match as-path 1
!
route-map BGP_INTO_OSPF permit 10
set metric 200
set metric-type type-1
!
route-map OSPF_INTO_BGP permit 10
match route-type internal
!
!

tacacs-server host 10.8.12.28 single-connection key 7 xxxx
tacacs-server host 10.8.12.29 single-connection key 7 xxxx
tacacs-server directed-request
access-list 55 permit 10.155.2.0 0.0.0.255

!
!
!
control-plane
!
!
vstack

line con 0
exec-timeout 0 0
password 7 xxxxx
logging synchronous
login authentication no_tacacs
stopbits 1
line aux 0
line 2
no activation-character
no exec
transport preferred none
transport output pad telnet rlogin lapb-ta mop udptn v120 ssh
stopbits 1
line vty 0 4
exec-timeout 5 0
logging synchronous
transport preferred none
transport input ssh
line vty 5 15
exec-timeout 5 0
logging synchronous
transport preferred none
transport input ssh
!
scheduler allocate 20000 1000
ntp server 172.16.7.3 prefer
ntp server 172.16.7.4
ntp server 172.16.7.5
ntp server 172.16.7.6
!
end

MAN-MDF-F1-RT2#terminal length 24
MAN-MDF-F1-RT2#exit

Georg Pauwen · ‎12-22-2017

Hello,

the only thing I could think of is to try different MTU (and tcp adjust-mss) values. Obviously, make sure that the current size you have configured is identical on both ends.

Send a ping with different packet sizes and the don't fragment (-f) bit set to the other side, and check at which size you get the response below:

C:\windows\system32>ping -f -l 1480 www.cisco.com

Pinging e2867.dsca.akamaiedge.net [95.100.136.187] with 1480 bytes of data:
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.

Ping statistics for 95.100.136.187:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

jasonku · ‎12-26-2017

It was pingable with 1400 MTU without any packet loss. It appears there is no issue with DF size. I don't know what else can be issue to make slower speed on DMVPN connection.

MAN-MDF-F1-RT2#ping

Protocol [ip]:
Target IP address: 10.255.202.1
Repeat count [5]: 100
Datagram size [100]: 1400
Timeout in seconds [2]:
Extended commands [n]: y
Source address or interface:
Type of service [0]:
Set DF bit in IP header? [no]: yes
Validate reply data? [no]:
Data pattern [0xABCD]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 100, 1400-byte ICMP Echos to 10.255.202.1, timeout is 2 seconds:
Packet sent with the DF bit set
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (100/100), round-trip min/avg/max = 36/37/80 ms

Georg Pauwen · ‎12-26-2017

Hello,

odd indeed. What is the output of:

show platform cerm-information

?

jasonku · ‎12-26-2017

yeah.. I agree, this is really odd..

MAN-MDF-F1-RT2#show platform cerm-information
Crypto Export Restrictions Manager(CERM) Information:
CERM functionality: ENABLED

----------------------------------------------------------------
Resource Maximum Limit Available
----------------------------------------------------------------
Tx Bandwidth(in kbps) 85000 85000
Rx Bandwidth(in kbps) 85000 85000
Number of tunnels 225 224
Number of TLS sessions 1000 1000

Resource reservation information:
D - Dynamic
-----------------------------------------------------------------------
Client Tx Bandwidth Rx Bandwidth Tunnels TLS Sessions
(in kbps) (in kbps)
-----------------------------------------------------------------------
VOICE 0 0 0 0
IPSEC D D 1 N/A
SSLVPN D D 0 N/A

Statistics information:
Failed tunnels : 0
Failed sessions : 0
Failed tx bandwidth: 0
Failed rx bandwidth: 0
Failed encrypt pkts: 0
Failed decrypt pkts: 0
Failed encrypt pkt bytes: 0
Failed decrypt pkt bytes: 0
Passed encrypt pkts: 121395
Passed decrypt pkts: 127363
Passed encrypt pkt bytes: 33579348
Passed decrypt pkt bytes: 60339412

Georg Pauwen · ‎12-26-2017

Hello,

what is the CPU utilization when the DMVPN is enabled (show proc cpu) ?