Throughput is noticeably degraded through GRE tunnel

[maximtretiakov]

Hello,

I need to troubleshoot the network where throughput is noticeably degraded through GRE tunnel.

MTU 1500 and Speed 100mb/s on all physical links along the path.

HOST1—L2SW—RTR2811—SW3560—RTR3725—FWLPIX—SW3750—HOST2

                                   |-------IPSec Tunnel-----|

                                   |----------------------------GRE-------------------------|

GRE tunnel is built between RTR2811 and SW3750 through IPSec tunnel between RTR2811 and RTR3725.

GRE tunnel is needed for specific purposes.

GRE 24 byte, and  IPSec 52byte overheads decrease the PMTU.

Problem1 – Performance is highly degraded:

Host1 can upload files to Host2 with the rate 300 kb/s

Host1 can download files from Host2 with the rate 370kb/s

High CPU usage on RTR2811 and SW3750 during files transmission (IP Input) up to~90%

Path W/O GRE through IPSec tunnel only: ~ 3mb/s, with CEF ~ 5mb/s

     What I have done is described below:

1.1     Set “ip mtu 1396" on both tunnel if’s (1400 did not improve the   performance, somehow fragments length were 1424 but should be no more   than 1420 to go through the GRE + IPSec bundle) 

Transfer rate improved: download H1 from H2 - 1100 kb/s, upload H1 to H2 ~ 400kb/s

1.2   Enabled CEF on RTR2811 and RTR3725. 

             the transfer rate improved: Host1 from Host2 - 2100 kb/s,  Host1 to Host2 900kb/s

CPU usage on RTR2811 reduced to normal (“IP Input” uses from 1 to 10%)

CPU usage on SW3750 is still high (“IP Input” uses  ~50% during peaks)

1.3   Set “ip tcp 1396” on RTR2811’s tunnel if (it looks like no improvements)

1.4   PMTUD seems not working. I tried “ip tcp path-mtu-discovery”  on SW3750 and RTR2811.

debug ip icmp keeps silence and there is no increment on acl counters (unreachable, time-exceeded)

I tried variants without DF cleaning on routers, and tried to rely on PMTUD, did not help.

(no route-map policy DF bit = 0, ip tcp path-mtu-discovery, tunnel path-mtu-discovery)

Problem2   – CEF was intentionally turned off on RTR2811 and RTR3725 by previous   network administrator. It must have been done with some special purpose   so I am curious why he did it. Could CEF introduce some disadvantages   for Security, HSRP(on RTR3725), or have other negative issues?

I have exhausted all my variants to solve this performance issues. Could anyone advise something, please?

Thanks very much in advance.

Some details from commands output are attached as a text file.

[rsimoni · Accepted solution]

Hi Maxim,

SW3750 stands for Catalyst 3750?

If yes this is the problem (or a big part which contributes to it).

GRE tunnels are NOT supported on hardware on Cat3750, that means that GRE traffic is handled by the CPU in software (this is why you see high CPU on that box).

Moreover performance of the CPU are far lower than hardware ASICs meaning that beside possible control plane instability which youmight see on the 3750 you will likely have lots of traffic loss, therefore performance degradation.

First step is to remove the GRE tunnel from the 3750.

About disabling CEF I have to admit that I really don't know why previous engineers disabled it. It is a bad idea as it causes high CPU utilization and lower performance. I am not aware of any security concerns arising for its presence but it might be me... in any case if there are security concerns they must be addressed by different means and not by disabling CEF.

regards,

Riccardo

[maximtretiakov]

Hi Riccardo,

Yes, saying SW3750 I meant Catalyst 3750.

Thank you very much for your answer, I will think how to acheive my goals without GRE on Catalyst 3750.

I did not find information about any disadvatages of using CEF in any way too.

Thank you again,

Maxim

[milan.kulik]

Hi,

IMHO, you might have tried

tunnel path-mtu-discovery

or

ip tcp adjust-mss

commands instead of ip tcp path-mtu-discovery.

See http://www.cisco.com/en/US/customer/tech/tk827/tk369/technologies_white_paper09186a00800d6979.shtml

or http://www.cisco.com/application/pdf/paws/25885/pmtud_ipfrag.pdf

for many details.

I also used ip mtu 1500 in my interface Tunnel xxx configuration which was fragmenting the packets and reassembling on the other tunnel side no matter of the D-F bit. But I'm not sure if this feature is supported on 3750s.

HTH,

Milan

[maximtretiakov]

Hi Milan,

Sorry, there is mistake in my previous post:

1.3   Set “ip tcp 1396” on RTR2811’s tunnel if (it looks like no improvements)

corrected:

1.3  Set “ip tcp adjust-mss 1396” on RTR2811’s tunnel if (it looks like no improvements)

So, I have already tried the options "ip  tcp adjust-mss 1396" and "tunnel path-mtu-discovery" with the standard  interface MTU or set "ip mtu 1396" on both tunnel interfaces. This  options helped to improve throughput from 300kb/s to 2000kb/s, but no  more.

I built the path through an additional VLAN without using  GRE tunnel and achieved up to 4mb/s and low CPU utilisation on SW  Catalyst 3750 (about 5% instead 60% during traffic peaks)

Thank you!

[rsimoni]

this is good news