50Mb cable connection slows to 3Mb after dmvpn

[James Sears]

Hi all,

We have a 50Mb cable connection to the Internet at a remote site. We have set up a dmvpn back to a hub router at the hospital which is in another state. We can test straight from the cable modem and we are getting our 45-50Mb speed. The problem is that when we go through the dmvpn tunnel0 we are only getting 3Mb. I've been testing this with Iperf which is plugged into a switch right off the hub. The config for the hub and spoke are below. Does anyone have an idea why we are losing so much bandwidth?

Thanks,

HUB Config:

FMC-101-R204#sh run
Building configuration...

Current configuration : 5206 bytes
!
! Last configuration change at 15:58:43 EST Tue Oct 8 2013 by jsears
! NVRAM config last updated at 23:00:42 EST Tue Oct 8 2013
!
version 12.4
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname FMC-101-R204
!
boot-start-marker
boot-end-marker
!
enable secret 5 $1$h4vR$pc/1KZPI2DkvYpXXyAXed/
!
aaa new-model
!
!
aaa authentication login default group tacacs+ local
aaa authentication login CONSOLE line
aaa authorization exec default group tacacs+ local
aaa accounting exec default start-stop group tacacs+
aaa accounting commands 0 default start-stop group tacacs+
aaa accounting commands 1 default start-stop group tacacs+
aaa accounting commands 15 default start-stop group tacacs+
!
aaa session-id common
clock timezone EST -5
!
!
ip cef
!
!
ip vrf inet
rd 1:1
!
no ip domain lookup
ip domain name fhmi.org
ip name-server 10.101.4.12
ip name-server 10.101.4.13
ip auth-proxy max-nodata-conns 3
ip admission max-nodata-conns 3
!
!
voice-card 0
no dspfarm
!
chat-script dial ABORT ERROR "" "AT Z" OK "ATDT \97065093952" TIMEOUT 60 CONNECT \c ABORT invalid TIMEOUT 60 login: super Password: super \r
!
!
!
!
!
!
!
!
!
!
!
!
!
username P(ink)Floyd privilege 15 secret 5 $1$W8c6$fU6723EXWB/NIU.LGBZ44.
!
!
!
crypto keyring inet vrf inet
  pre-shared-key address 0.0.0.0 0.0.0.0 key XXXXXXX
!
crypto isakmp policy 10
encr 3des
authentication pre-share
group 2
crypto isakmp keepalive 20
crypto isakmp profile inet
   keyring inet
   match identity address 0.0.0.0 inet
!
!
crypto ipsec transform-set strong1 esp-3des esp-sha-hmac
crypto ipsec transform-set strong2 ah-sha-hmac esp-3des
crypto ipsec transform-set strong3 esp-aes 256 esp-sha-hmac
!
crypto ipsec profile dynvpn
set transform-set strong1 strong2 strong3
set isakmp-profile inet
!
!
!
!
!
interface Loopback0
ip address 192.168.255.1 255.255.255.255
!
interface Loopback1
no ip address
!
interface Loopback2
no ip address
!
interface Tunnel0
description ***Backup DSL VPN***
bandwidth 51200
ip address 10.101.255.1 255.255.255.0
no ip redirects
no ip next-hop-self eigrp 1
ip nhrp authentication cisco
ip nhrp map multicast dynamic
ip nhrp network-id 1
ip tcp adjust-mss 1436
no ip split-horizon eigrp 1
no ip mroute-cache
tunnel source FastEthernet0/0
tunnel mode gre multipoint
tunnel key XXXX
tunnel vrf inet
tunnel protection ipsec profile dynvpn
!
interface FastEthernet0/0
description Outside
ip vrf forwarding inet
ip address 216.207.248.4 255.255.255.0
ip access-group VPN in
duplex auto
speed auto
!
interface FastEthernet0/1
description Inside
ip address 10.101.1.204 255.255.255.0
duplex auto
speed auto
!
interface Async0/0/0
ip unnumbered FastEthernet0/1
encapsulation ppp
dialer in-band
async mode interactive
!        
interface Async0/0/1
no ip address
encapsulation slip
!
router eigrp 1
network 10.0.0.0
no auto-summary
!
ip forward-protocol nd
ip route vrf inet 0.0.0.0 0.0.0.0 216.207.248.1
!
!
no ip http server
no ip http secure-server
!
ip access-list extended VPN
permit esp any any
permit gre any any
permit udp any any eq isakmp
permit icmp any any
!
kron occurrence saveconfig at 23:00 recurring
policy-list saveconfig
!
kron policy-list saveconfig
cli write
!
logging trap notifications
logging source-interface FastEthernet0/1
logging 10.101.40.10
snmp-server community XXXXXX RO
snmp-server community XXXXXX RW
snmp-server ifindex persist
snmp-server location Site 101 - FMC
snmp-server host 10.101.40.10 XXXXXX
!
!
!
tacacs-server host 10.101.15.47
tacacs-server host 10.101.15.48
tacacs-server directed-request
tacacs-server key 7 0822455D0A160B1206
!
control-plane
!
!
!
!
!
!
!
!
!
banner motd ^C
       ,,,,,
      /'^ ^'\
     ((o)-(o))      **Floyd Information Technology**
--oOOO--(_)--OOOo-------------------------------------------------

You Are Attempting To Access a Private        |           |
Network.  Unauthorized Access is Strictly    :|:         :|:
Forbidden.  Violators Will be Prosecuted!   :|||:       :|||:
                        - Management     ..:|||||||:...:|||||||:..
  .oooO                                  _________________________
  (   )      Oooo.
---\ (-------(   )------------------------------------------------
    \_)       ) /              $(hostname)
             (_/
*Please login with your AAA Credentials all information is logged*
^C       
!
line con 0
exec-timeout 15 0
password 7 0620032E554A4D394D5F17181F
login authentication CONSOLE
line aux 0
modem InOut
transport input all
transport output all
line 0/0/0
script connection dial
modem InOut
modem autoconfigure discovery
rotary 1
transport input all
escape-character NONE
autoselect slip
stopbits 1
speed 115200
line 0/0/1
stopbits 1
speed 115200
flowcontrol hardware
line vty 0 4
access-class 9 in
exec-timeout 15 0
privilege level 15
password 7 14311E0415006E0B6C60362631
transport input telnet ssh
line vty 5 15
access-class 9 in
exec-timeout 15 0
privilege level 15
password 7 14311E0415006E0B6C60362631
transport input telnet ssh
!
scheduler allocate 20000 1000
ntp clock-period 17179365
ntp server 10.104.250.11 prefer
ntp server 10.104.250.12
!
end

Spoke Config:

CTR-122-R001#sh run
Building configuration...

Current configuration : 7266 bytes
!
! Last configuration change at 19:44:55 UTC Wed Oct 9 2013 by jsears
! NVRAM config last updated at 19:48:00 UTC Wed Oct 9 2013 by jsears
!
version 12.4
service timestamps debug datetime msec
service timestamps log datetime localtime
service password-encryption
!
hostname CTR-122-R001
!
boot-start-marker
boot-end-marker
!
card type t1 0 1
logging buffered 16384 notifications
enable secret 5 $1$6SQs$CD1yJYXrnO9.qngaPIT0P/
!
aaa new-model
!
!
aaa authentication login default group tacacs+ local
aaa authentication login CONSOLE line
aaa authorization exec default group tacacs+ local
aaa accounting exec default start-stop group tacacs+
aaa accounting commands 0 default start-stop group tacacs+
aaa accounting commands 1 default start-stop group tacacs+
aaa accounting commands 15 default start-stop group tacacs+
!
aaa session-id common
!
resource policy
!
no network-clock-participate wic 1
ip subnet-zero
ip tcp path-mtu-discovery
!
!
ip cef
!
!
ip flow-cache timeout active 1
ip domain name fhmi.org
ip name-server 10.101.4.12
ip name-server 10.101.4.13
ip sla monitor 40000
type http operation get url http://greenway190001/ source-ipaddr 10.122.255.2
timeout 180000
owner SW.IpSla.ORION.Solarwinds
frequency 300
ip sla monitor schedule 40000 life forever start-time now ageout 3600
!
!
voice-card 0
no dspfarm
!
!
!
!
!
!
!
!
!
!
!
!
!
!        
username P(ink)Floyd privilege 15 secret 5 $1$crN3$sAVupf.lLy9gKqjauQXQl1
!
!
controller T1 0/1/0
framing esf
linecode b8zs
channel-group 0 timeslots 1-24
!
controller T1 0/1/1
framing esf
linecode b8zs
channel-group 0 timeslots 1-24
!
class-map match-any Voice-Calls
  description Voice-Calls
match ip dscp ef
class-map match-any Voice-Call-Control
  description Call-Control
match  dscp af31
class-map match-any Telnet-Traffic
  description Telnet-SSH
match protocol telnet
match protocol ssh
class-map match-any MISSION-CRITICAL
  description MISSION-CRITICAL-TRAFFIC
match access-group name MISSION-CRITICAL
!
!
policy-map QOS
  description Traffic-QoS
class Voice-Calls
  priority 282
class Voice-Call-Control
  bandwidth 64
class MISSION-CRITICAL
  bandwidth 600
class Telnet-Traffic
  bandwidth 64
class class-default
  fair-queue
  random-detect
!
!
!
crypto isakmp policy 10
encr 3des
authentication pre-share
group 2
crypto isakmp key XXXXXX address 0.0.0.0 0.0.0.0 no-xauth
crypto isakmp keepalive 20
!
!
crypto ipsec transform-set strong1 esp-3des esp-sha-hmac
crypto ipsec transform-set strong2 ah-sha-hmac esp-3des
crypto ipsec transform-set strong3 esp-aes 256 esp-sha-hmac
!
crypto ipsec profile dynvpn
set transform-set strong1 strong2 strong3
!
!
!
!
!
interface Tunnel0
bandwidth 51200
ip address 10.101.255.122 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp authentication cisco
ip nhrp map 10.101.255.1 216.207.248.4
ip nhrp map multicast 216.207.248.4
ip nhrp network-id 1
ip nhrp holdtime 600
ip nhrp nhs 10.101.255.1
ip tcp adjust-mss 1360
tunnel source 24.197.22.162
tunnel mode gre multipoint
tunnel key XXXXX
tunnel protection ipsec profile dynvpn
!
interface Multilink122
description Centre, AL ML122
bandwidth 64
ip address 10.122.255.2 255.255.255.252
ip nbar protocol-discovery
ip route-cache flow
load-interval 30
ppp multilink
ppp multilink group 122
!
interface GigabitEthernet0/0
no ip address
ip route-cache flow
duplex full
speed 100
!
interface GigabitEthernet0/0.1
encapsulation dot1Q 1 native
ip address 10.122.1.1 255.255.255.0
ip accounting precedence input
no snmp trap link-status
!
interface GigabitEthernet0/0.2
description Wireless Managment
encapsulation dot1Q 2
ip address 10.122.2.1 255.255.255.0
ip helper-address 10.101.202.203
no snmp trap link-status
!
interface GigabitEthernet0/0.6
description Static IPs
encapsulation dot1Q 6
ip address 10.122.6.1 255.255.255.0
no snmp trap link-status
!
interface GigabitEthernet0/0.7
description IP Telephony
encapsulation dot1Q 7
ip address 10.122.7.1 255.255.255.0
no snmp trap link-status
!
interface GigabitEthernet0/0.8
description PACS
encapsulation dot1Q 8
ip address 10.122.8.1 255.255.255.0
no snmp trap link-status
!
interface GigabitEthernet0/0.59
description DCHP IPs
encapsulation dot1Q 59
ip address 10.122.59.1 255.255.255.0
ip helper-address 10.101.202.203
no snmp trap link-status
!
interface GigabitEthernet0/0.199
description Wireless
encapsulation dot1Q 199
ip address 10.122.199.1 255.255.255.0
ip helper-address 10.101.202.203
no snmp trap link-status
!
interface GigabitEthernet0/0.248
description fhmisecure
encapsulation dot1Q 248
ip address 10.122.248.1 255.255.255.0
ip helper-address 10.101.202.203
no snmp trap link-status
!
interface GigabitEthernet0/1
description Charter
ip address 24.197.22.162 255.255.255.248
ip access-group vpn in
duplex auto
speed auto
!
interface Serial0/1/0:0
description ML122 CID=A3HCGS413947SC
bandwidth 1536
no ip address
encapsulation ppp
no fair-queue
ppp multilink
ppp multilink group 122
!
interface Serial0/1/1:0
description ML122 CID=A3HCGS413987SC
bandwidth 1536
no ip address
encapsulation ppp
no fair-queue
ppp multilink
ppp multilink group 122
!
router eigrp 1
network 10.0.0.0
no auto-summary
eigrp stub connected
!
ip classless
ip route 216.207.248.4 255.255.255.255 24.197.22.161
!
!
ip http server
no ip http secure-server
!
ip access-list extended MISSION-CRITICAL
permit ip any host 10.104.10.31
permit ip any host 10.104.10.25
permit ip any host 10.104.10.34
permit ip any host 10.104.20.26
permit ip any host 10.101.15.116
ip access-list extended VPN
permit esp any any
permit gre any any
permit udp any any eq isakmp
permit icmp any any
!
logging trap notifications
logging source-interface GigabitEthernet0/0.1
snmp-server community XXXXXX RO
snmp-server community XXXXXX RW
snmp-server ifindex persist
snmp-server location Site 122 - Centre, AL
snmp-server host 10.101.40.10 XXXXXX
!
!
!
tacacs-server host 10.101.15.47
tacacs-server host 10.101.15.48
tacacs-server directed-request
tacacs-server key 7 0822455D0A160B1206
!
control-plane
!
!
!
!
!
!
!
!
!
banner motd ^C
       ,,,,,
      /'^ ^'\
     ((o)-(o))      **Floyd Information Technology**
--oOOO--(_)--OOOo-------------------------------------------------

You Are Attempting To Access a Private        |           |
Network.  Unauthorized Access is Strictly    :|:         :|:
Forbidden.  Violators Will be Prosecuted!   :|||:       :|||:
                        - Management     ..:|||||||:...:|||||||:..
  .oooO                                  _________________________
  (   )      Oooo.
---\ (-------(   )------------------------------------------------
    \_)       ) /              $(hostname)
             (_/
*Please login with your AAA Credentials all information is logged*
^C
!
line con 0
exec-timeout 15 0
password 7 0220085412024B0104060C0A16
login authentication CONSOLE
line aux 0
no exec
line vty 0 4
exec-timeout 15 0
password 7 052D0A0038480A29514D120118
transport input telnet ssh
line vty 5 15
exec-timeout 15 0
password 7 052D0A0038480A29514D120118
transport input telnet ssh
!
scheduler allocate 20000 1000
ntp clock-period 17180141
ntp server 10.104.250.11 prefer
ntp server 10.104.250.12
!
end

[Leo Laohoo]

You forget one important detail:  The exact model of your router.

.... Your HUB router has FastEthernet and your Spoke is GigabitEthernet?

Is your HUB a 2600/3700 or 1800/2800 router?  If this is the case, 2800 router can't handle 50mb traffic.

[James Sears]

It is a 2821 on the spoke and the hub is a 2811. Sorry I forgot to state the model. So the 2811 can't handle 50Mb's? Even though it can't handle this, is it limited to 3Mb's? It just seems like such an extreme decrease- 50-3. I'll look into using another router as an endpoint for this tunnel.  Thank you for your reply.

[Leo Laohoo]

James,

Look at this link.

This document shows the 2811 capable of up to 61.44 Mbps.  This value of "61.44" means the router can push, IN A SINGLE DIRECTION and WITHOUT ENCRYPTION, that much.

If you factor in a two-way traffic and will full encryption, then your 2811 can push between 15.36 Mbps (61.44 / 4 = 15.36) and 20 Mbps.

Next thing I want to know is if your 2811 came with built-in encryption module or not.  The best way to determine is the output to the command "sh crypto engine brief".

[James Sears]

Hi Leo,

Yes. It looks like it does have a vpn module. Below is the output of the command:

FMC-101-R204#sh crypto engine brief

        crypto engine name:  Virtual Private Network (VPN) Module

        crypto engine type:  hardware

                     State:  Enabled

                  Location:  onboard 0

              Product Name:  Onboard-VPN

        Middleware Version:  v1.2.0

          Firmware Version:  v2.2.0

              Time running:  4294967 seconds

               Compression:  Yes

                       DES:  Yes

                     3 DES:  Yes

                   AES CBC:  Yes (128,192,256)

                  AES CNTR:  No

     Maximum buffer length:  4096

          Maximum DH index:  0300

          Maximum SA index:  0300

        Maximum Flow index:  2400

      Maximum RSA key size:  2048

        crypto engine name:  Cisco VPN Software Implementation

        crypto engine type:  software

             serial number:  64B06E67

       crypto engine state:  installed

     crypto engine in slot:  N/A

[Leo Laohoo]

Ok, that rules it out.

Any line errors on the physical interface of your WAN (both sides)?

[James Sears]

I don't see any. Wait... there are input errors. Below is the show interface output:

Spoke interface:

CTR-122-R001#sh int gi0/1

GigabitEthernet0/1 is up, line protocol is up

  Hardware is MV96340 Ethernet, address is 0024.97fd.e8a1 (bia 0024.97fd.e8a1)

  Description: Charter

  Internet address is 24.197.22.162/29

  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 1000Mb/s, media type is T

  output flow-control is XON, input flow-control is XON

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 00:11:38, output 00:00:00, output hang never

  Last clearing of "show interface" counters never

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 0 bits/sec, 2 packets/sec

  5 minute output rate 148000 bits/sec, 54 packets/sec

     7005368 packets input, 1520398551 bytes, 0 no buffer

     Received 9669 broadcasts, 0 runts, 0 giants, 0 throttles

     12 input errors, 0 CRC, 12 frame, 0 overrun, 0 ignored

     0 watchdog, 0 multicast, 0 pause input

     0 input packets with dribble condition detected

     11961158 packets output, 3803800952 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 pause output

     0 output buffer failures, 0 output buffers swapped out

Hub interface:

FMC-101-R204#sh int fa0/0

FastEthernet0/0 is up, line protocol is up

  Hardware is MV96340 Ethernet, address is 001b.2ac1.8218 (bia 001b.2ac1.8218)

  Description: Outside

  Internet address is 216.207.248.4/24

  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 100Mb/s, 100BaseTX/FX

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 00:00:00, output 00:00:01, output hang never

  Last clearing of "show interface" counters never

  Input queue: 0/75/12238/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 142000 bits/sec, 51 packets/sec

  5 minute output rate 3000 bits/sec, 2 packets/sec

     421023305 packets input, 3907268548 bytes

     Received 77204717 broadcasts, 0 runts, 0 giants, 1768 throttles

     37174 input errors, 0 CRC, 0 frame, 0 overrun, 37174 ignored

     0 watchdog

     0 input packets with dribble condition detected

     530714409 packets output, 2526737141 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 unknown protocol drops

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier

     0 output buffer failures, 0 output buffers swapped out

[Leo Laohoo]

Your SPOKE, aka "remote end", only has 12 CRC.  It's nothing.

Your HUB, aka, "local end", has significant errors but compared to the incoming packets (37174 / 42123305 * 100 = 8%) is nothing.

reliability 255/255, txload 1/255, rxload 1/255

I am, however, interested in this.  What is the speed now?  Is it still sitting at 3 Mbps?  "txload" and "rxload" shows nothing.  So it's not a bandwidth issue and you are not over-utilizing the link. 

Input queue: 0/75/12238/0 (size/max/drops/flushes); Total output drops: 0

I have never seen this value to be that high before.

PS:  It's been awhile since I've done any DMVPN configuration so I am a bit rustic.  I'll need to find where I stashed those notes of mine.

[James Sears]

Leo it will be Monday before I'm onsite and can test the bandwidth again. The high amount of errors on the hub may be from other sites and has not been cleared in a long time. I'll reset those counters. Your help is much appreciated. Monday I have some work to do at the remote site and will test again. I can't thank you enough for your help. I'll post more Monday and if you can add your insight to the issue that would be great. Talk to you then.

[paolo bevilacqua]

Your HUB, aka, "local end", has significant errors but compared to the incoming packets (37174 / 42123305 * 100 = 8%) is nothing. 

I've not looked at this case details, but 8% is not "nothing". It is actually non acceptable..

[Leo Laohoo]

I've not looked at this case details, but 8% is not "nothing". It is actually non acceptable..

Paolo,

You saying this is a cause of concern?

[Vasilii Mikhailovskii]

Hi, actually

37174 / 42123305 * 100 = 0,09%

As a test I would suggest to clear counter on both sides; and test transfer rates back and forth (in both directions).

If it sustained about 3M, then try to capture traffic with WireShark.

Try to use different OS, because latest Microsoft OS (Win2008R2/W7/W8) are less sensetive to latency

Try to transfer traffic using several threads (tcp connections).

During the test try sh proc cpu sor 1min and sh mem.

Some notes about configuration:

• Tunnel0 configuration is not consistent about MTU; it should be ip mtu 1400, ip tcp adjust-mss 1360 on both ends;
• I see policy-map, but config does not mention any application of it (shaper could be a cause).

PS: check internal interface for duplex mismatch and any errors as well

[paolo bevilacqua · Accepted solution]

As Leo as indicated above, and old ISR router cannot handle 50 mbps circuit.

[James Sears]

I've got a lot of things to try now. Thanks you everyone. In the end I think this old router just can't handle it. I'll try some of the config changes also.

Thanks again.

[Joseph W. Doherty]

Disclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

A 2811 should be able to pass more than 3 Mbps, properly configured.  And on the subject of proper configuration, you might be fragmenting packets as your MTU, mss-adjust and PMTUD look to be suboptimal for GRE/IPSec.

You may find this helpful: http://www.cisco.com/en/US/tech/tk827/tk369/technologies_white_paper09186a00800d6979.shtml