cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
6396
Views
0
Helpful
26
Replies

Unbelievably slow DMVPN connection.

gmulvaney1
Level 1
Level 1

Hi, I've been working on this for 3 days and no resolution. Hopefully some smarter folks here will throw me a bone!

I have a hub and spoke phase 3 topology that works great except for speed- internet speeds (DIA) are as expected, spoke is 100M and hub is 1 gig link.

The hub is an old Cisco 2911 running  15.7(3)M5, the spoke is a small ISR (881W, wireless disabled) running 15.4(3)M8.

 

Actual bandwidth through the tunnel (iperf3) is only 2.2-3Mbits average- I'm pulling what little hair I have left out over that.

Configs are below, I've played w/ the tcp-mss settings and mtu, it didn't really make much difference. I also removed the tunnel protection as a test and the BW on both sets was actually slightly slower, but a negligible difference. 

Hub Config:

interface Tunnel1
bandwidth 1000000
ip vrf forwarding labnetwork
ip address 192.168.185.2 255.255.255.0
no ip redirects
ip mtu 1400
no ip next-hop-self eigrp XXXX
no ip split-horizon eigrp XXXX
ip nhrp authentication XXXX

ip nhrp network-id 69
ip nhrp holdtime 450
ip nhrp redirect
ip virtual-reassembly in max-reassemblies 1000
no ip split-horizon
ip tcp adjust-mss 1365
delay 1000
tunnel source GigabitEthernet0/0.1
tunnel mode gre multipoint
tunnel key XXX
tunnel path-mtu-discovery
tunnel bandwidth transmit 1000000
tunnel bandwidth receive 1000000
tunnel protection ipsec profile ROME
end

 

Spoke config:

interface Tunnel1
bandwidth 1000000
bandwidth receive 100000
ip address 192.168.185.1 255.255.255.0
no ip redirects
ip mtu 1400
ip nat inside
ip nhrp authentication XXXX
ip nhrp map 192.168.185.2 158.93.X.X
ip nhrp map multicast 158.93.X.X
ip nhrp network-id 69
ip nhrp holdtime 450
ip nhrp nhs 192.168.185.2
ip nhrp shortcut
ip nhrp redirect
ip virtual-reassembly in
ip tcp adjust-mss 1365
delay 1000
tunnel source FastEthernet4
tunnel mode gre multipoint
tunnel key XXX
tunnel path-mtu-discovery
tunnel bandwidth transmit 100000
tunnel bandwidth receive 100000
tunnel protection ipsec profile ROME

 

Anyone has a solution the beer is on me the next time you're in Augusta, Ga.

 

Thanks!

Jerry

26 Replies 26

balaji.bandi
Hall of Fame
Hall of Fame

Look at the thread and some Limitation :

 

https://community.cisco.com/t5/routing/50mb-cable-connection-slows-to-3mb-after-dmvpn/m-p/2317120

 

also attached router performance and support speeds

 

 

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

Thanks for the quick response. I fully expected a performance hit, but from 100 down to sub 3 Mb seems way out of line. In any case the speeds were virtually identical with the crypto removed.  It does appear to have hardware acceleration:

crypto engine name: Virtual Private Network (VPN) Module
crypto engine type: hardware
State: Enabled
Location: onboard 0
Product Name: Onboard-VPN
FW Version: 1
Time running: 114174 seconds
Compression: Yes
DES: Yes
3 DES: Yes
AES CBC: Yes (128,192,256)
AES CNTR: No
Maximum buffer length: 4096
Maximum DH index: 0000
Maximum SA index: 0000
Maximum Flow index: 1000
Maximum RSA key size: 0000

 

I'd expect at least 10-15M with crypto.

 

Thanks!

@Georg Pauwen is much ahead of me to reply to you, start with the basic config of DMVPN, not additional fancy stuff start with and see if the performance increases, he suggested more no commands i would agree on wit that too.

 

let us know the outcome.

 

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

Hello,

 

a few things in your configuration don't look right, make the changes/additions marked in bold. Also, can you post the full configs of both the hub and the spoke ? We might be able to spot something else...

 

HUB 

 

interface Tunnel1
--> no bandwidth 1000000
--> no ip vrf forwarding labnetwork
ip address 192.168.185.2 255.255.255.0
no ip redirects
ip mtu 1400
no ip next-hop-self eigrp XXXX
no ip split-horizon eigrp XXXX
ip nhrp authentication XXXX
--> ip nhrp map multicast dynamic
ip nhrp network-id 69
--> ip nhrp holdtime 300
ip nhrp redirect
ip virtual-reassembly in max-reassemblies 1000
no ip split-horizon
--> ip tcp adjust-mss 1360
--> no delay 1000
tunnel source GigabitEthernet0/0.1
tunnel mode gre multipoint
tunnel key XXX
--> no tunnel path-mtu-discovery
--> no tunnel bandwidth transmit 1000000
--> no tunnel bandwidth receive 1000000
tunnel protection ipsec profile ROME
end

 

Spoke

 

interface Tunnel1
--> no bandwidth 1000000
--> no bandwidth receive 100000
ip address 192.168.185.1 255.255.255.0
no ip redirects
ip mtu 1400
--> no ip nat inside
ip nhrp authentication XXXX
ip nhrp map 192.168.185.2 158.93.X.X
ip nhrp map multicast 158.93.X.X
ip nhrp network-id 69
--> ip nhrp holdtime 300
ip nhrp nhs 192.168.185.2
ip nhrp shortcut
--> no ip nhrp redirect
ip virtual-reassembly in
--> ip tcp adjust-mss 1360
--> no delay 1000
tunnel source FastEthernet4
tunnel mode gre multipoint
tunnel key XXX
--> no tunnel path-mtu-discovery
--> no tunnel bandwidth transmit 100000
--> no tunnel bandwidth receive 100000
tunnel protection ipsec profile ROME

Thank you- changes made, the only thing that I couldn't remove or lose connectivity from here was the vrf. Basically the same results, average was 2.54. Configs attached, minus passwords, etc. There's probably a few extraneous commands from old configs but there doesn't look like anything that would affect performance. I appreciate your looking at this.

BTW, the nhrp map multicast dynamic appears to be a default on this license, minus static mappings. When it's entered it doesn't show in the running config, but checking the mappings I see it's functioning:

AB-Lab-2911-Gateway#sh ip nhrp multicast
I/F NBMA address
Tunnel1 73.190.199.62 Flags: dynamic (Enabled)
Tunnel1 75.76.206.181 Flags: dynamic (Enabled)

 

Hello,

 

it is kind of hard to understand your topology. Your interface:

 

interface GigabitEthernet0/1
bandwidth 10000
ip vrf forwarding labnetwork
ip address 192.168.170.0 255.255.255.254
ip nat inside
ip virtual-reassembly in
duplex auto
speed auto

 

has a /31 mask, is that right ? Also, you have NAT enabled (both versions), what exactly are you trying to NAT ?  Can you provide a schematic drawing of your topology, showing what is connected to what, and what is being routed where ?

Hello,

 

also, from which IP address to which IP address are you testing the connection ?

Hi Georg,

 

Attached is a quick drawing (.jpg, it wouldn't let me attach the Visio file)  I put together. The lab changes quite a bit so I never have maintained one.

I think you're looking at the same thing that I've been thinking about, the VRF may be causing issues although it shouldn't and traffic/cpu/crypto/CEF all looks very clean.

Tentative plan is to move the hub from the main lab to the other public IP (here) and swap the 881 for a 2921 I pulled out of mothballs this morning. That way I can easily test the tunnel speed from the other spoke without impacting other lab testing.

It'll probably be Monday or Tues but I'll update on the results. I'd feel better if I could see some dropped packets or debug errors, but nothing- it's just slow. Maybe the VRF is causing issues. Thanks!

Thank for the making changes and reported back

 

you were mentioned, before you deploy a VPN, you able to get real bandwidth as per your expectation.

after deploying DMVPN you have a poor performance like 3mb max ? is this correct?

 

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

Hi,

 

You're correct. I know that with a lower end ISR as a spoke I can expect a performance hit, but 90+ % is a little extreme. I should , with this router and low end IPSEC, be getting at least 10-15M.

Thanks!

Joseph W. Doherty
Hall of Fame
Hall of Fame
Your gig hub and 100 Mbps spoke connection actually provide that much bandwidth (duplex) or that's just the physical hand-off bandwidths?

Since you hub has 10x the bandwidth of your spoke, one issue might be congestion on Internet egress to your spoke. You might derive some benefit shaping your hub traffic going to spoke(s) to correspond with the spoke's available bandwidth. (One the subject of available bandwidth, are the Internet connections used for other than tunnel traffic?)

In your later post, with full configs, why are you clearing the DF bit? Why are you using tunnel keys? Are you passing any UDP traffic across the tunnel? If so, much?

Hi Joseph,


Thanks for the reply.

Spoke bandwidth is less, it's commercial internet so runs around 75-80% of 100M but the traffic is pretty light, I've never seen it approach even 50 % of capacity. The hub is true bandwidth, it's connected to a 3gig connection provided by the state. No internet traffic traverses the tunnels, all are split tunneled w/ internet traffic going out locally- only 192.168 traffic goes across the tunnels. UDP traffic is minimal now, only when we do backups of equipment or something like that, although we do plan on testing some voice and video multicast later this spring.

I can probably get rid of the crypto ipsec df-bit clear, at one time I had a fragmentation problem but fixed that. I'll dump that when I start testing again. I probably don't need the tunnel keys w/ the ipsec but it's an organizational standard.

Thanks,

Jerry

Hello,

 

for the sake of testing, you might also want to try a different encryption (on both ends obviously):

 

crypto isakmp policy 10
encr aes
hash sha256
authentication pre-share
group 5

!

crypto ipsec transform-set LAB esp-aes 256 esp-sha256-hmac
mode tunnel

Although you not seen spoke traffic exceed 50%, but over what time measurement. If you're not familiar with "microbursts", I suggest you Google them and read up on them.

When I asked about sharing bandwidth, I didn't mean within the tunnel, but sharing the same physical internet link for both tunnel and non-tunnel ("raw") Internet traffic. If you're doing that, and if you have a transient congestion issues, solving with QoS is near impossible.

Further when you say commercial Internet runs 75-80%, in this case I'm not asking about (your) measured usage but what bandwidth your provider is supposed to be providing; i.e. full port bandwidth on some lessor committed rate.
Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Innovations in Cisco Full Stack Observability - A new webinar from Cisco