GRE over IPSEC performance Issue

[jpdeboer1]

Hi,

I have an issue with bandwidth performance, i have an GRE over IPSEC tunnel between two ISR4431 Routers. Both of these routers have IPSEC, HSEC and PERF licenses installed. At the remote location we have a 500Mbit connection, when we bypass the Tunnel we measure speeds of 500 - 600 Mbits. But when we go over the GRE IPSEC tunnel we only get max 170 Mbits.

See below Setup:

Internet<---Core Router<----ISR4431<----GRE Over IPSEC<--- Remote ISR4431<---Local LAN

GRE over IPSEC configuration:

crypto isakmp policy 10
authentication pre-share
crypto isakmp key "Pre-Share Key" address Peer IP Address

crypto ipsec transform-set IPSEC esp-aes esp-sha256-hmac
mode tunnel

crypto ipsec profile IPSEC
set transform-set IPSEC

interface Tunnel10
ip address 10.93.69.69 255.255.255.252
ip pim sparse-mode
ip tcp adjust-mss 1272
ip igmp version 3
load-interval 30
tunnel source IP Address
tunnel destination IP Address
tunnel path-mtu-discovery
tunnel bandwidth transmit 10000000
tunnel bandwidth receive 10000000
tunnel protection ipsec profile IPSEC

See below Licenses which are in use:

Index 3 Feature: securityk9
Period left: Life time
License Type: Permanent
License State: Active, In Use
License Count: Non-Counted
License Priority: Medium

Index 4 Feature: ipbasek9
Period left: Life time
License Type: Permanent
License State: Active, In Use
License Count: Non-Counted
License Priority: Medium

Index 6 Feature: hseck9
Period left: Life time
License Type: Permanent
License State: Active, In Use
License Count: Non-Counted
License Priority: Medium

Index 7 Feature: throughput
Period left: Life time
License Type: Permanent
License State: Active, In Use
License Count: Non-Counted
License Priority: Medium

Index 8 Feature: internal_service

Tested it with different MTU sizes, and different transform sets.

Did someone else encounter this issue ? Or does someone notice i miss a piece of config ?

Thanks in Advance!

[Philip D'Ath]

Is this a symmetric test?  As in, are you generating 170Mb/s in total, or 170 Mb/s in each direction?  Are you using large TCP packets for the test?

Can you show the top half dozen lines of "show process cpu" when you are doing a throughput test please.

Does anything interesting appear in the log?

What software version are you running on the router?

[jpdeboer1]

Hi,

This is a one way test, we are running the test with iperf, 1 linux server at the remote end and 1 linux server at the hub side.

I have attached 2 txt files which contain cpu information, one for TCP test and one for UDP test. I have tested it with large packets and small packets, but i cant get more that 170 / 180 Mbits through the tunnel.

Both of the routers use the same software, see below:

Cisco IOS XE Software, Version 03.13.04.S - Extended Support Release
Cisco IOS Software, ISR Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 15.4(3)S4, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Mon 05-Oct-15 11:24 by mcpre

Cisco IOS-XE software, Copyright (c) 2005-2015 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: IOS-XE ROMMON

C2H-AMS-AR1 uptime is 13 hours, 48 minutes
Uptime for this control processor is 13 hours, 50 minutes
System returned to ROM by reload
System image file is "bootflash:/isr4400-universalk9.03.13.04.S.154-3.S4-ext.SPA.bin"
Last reload reason: Reload Command

[Philip D'Ath]

Could you try upgrading to one of the gold star releases for your router and see if that has any impact.

https://software.cisco.com/download/release.html?mdfid=284358776&softwareid=282046477&release=3.13.7S&relind=AVAILABLE&rellifecycle=ED&reltype=latest

[jpdeboer1]

Thanks for the quick reply, I have updated both routers to the latest software you mentioned, but this did not resolve the issue.

We also tried the speedtest over a plain GRE tunnel withour encryption, but we still saw the same throughput.

We also made a speedtest going directly to the internet from each router so we did not have a tunnel inbetween, we than saw throughputs of 500 / 600 Mbits.

So only when we put the traffic over a GRE IPSEC / GRE tunnel we see degrated throughput.

[jpdeboer1]

I would like to let you know i found the problem, BGP routing was chaged due to an issue at one of our providers, due to that the GRE IPSEC tunnel was build over the wrong connection, a connection that only supports 200Mbit. After the tunnel was build over the correct connection we saw about 500Mbit throughput over the tunnel.

Thanks for your time to help me find the problem!