Cisco 891F with ZBF and IKEv2 tunnel slow download speeds

[igor.hamzic81]

Hi all,

in one of our branch offices we have switched from a dedicated line between locations to a internet link(50 Mbps) with IKEv2 tunnel to the main office on our 891F router. Since the router is now connected to the internet we also implemented ZBF on it for basic security.

All the traffic from the branch office is sent over the VPN tunnel to the main office and then to wherever it needs to go.

Since the change the users are having problems with slow download speeds from the main office from whatever source (ie. downloads from the internet, from office servers, ...).
Strangely enough traffic upload to the main office servers or to the internet or web surfing are fast using the same path and going over the same equipment.

This problem has me scratching my head as I really can't pinpoint the problem as the internet link itself is stable, VPN tunnel is stable, ZBF configuration is quite simple, there are no drops on the interfaces, router itself has CPU usage of 20 - 30%, there is no NAT.

I tried removing ZBF between INSIDE and OUTSIDE zones with no effect, simplified the ZBF, added ooo paramater map as I read it could help but nothing produced any result.

My suspicion is that something on the 891F router is the problem but I just can't figure it out.
I'm attaching the relevant config and hope that someone can point to where to look and how to solve this issue.

[MHM Cisco World]

You dont use DIA' so the problem may be not in your router it can in HQ routers that slow download speed.

The traffic need to go to HQ and then through tunnel to your router.

Check HQ 

[igor.hamzic81]

Hi. Can you explain what is DIA and can you elaborate on your answer? All traffic, both upload and download, goes over the VPN tunnel but only download to the branch office experiences the problems.

On the HQ side we have ASA5525X as the VPN GW and after that we have a FTD 2130 which moves the traffic to the required destination.
The links on the HQ side between firewalls are 1Gbps, Internet links are 10Gbps. There are no drops on the interfaces and devices on the HQ side are not showing any problems.

[MHM Cisco World]

Direct internet access (DIA) meaning traffic go to internet no need to go to HQ' only traffic need to access server in HQ need to pass through tunnel.

But I think you dont ise DIA because of security' the asa is secure data to your branch router.

I will check if there is command to check up/download throughput in asa.

[igor.hamzic81]

Correct. We do not use direct Internet access due to security reasons and tunnel all traffic to the ASA which then forwards it to FTD which handles all the traffic moving the traffic then to the Internet, HQ servers or to whatever else is needed.

[Joseph W. Doherty]

On my phone, unable to look at your config, but two common issues are: 1) impact of fragmentation across a tunnel and 2) pushing data faster than path supports.

From what you describe, I suspect #2 might be the major issue as you note branch has 50 Mbps (port running at 100?) and HQ has 10g.

Does either side shape for the 50 Mbps?  If you do shape for 50 Mbps, using what specific parameters?

Other changes might also be made to improve throughput, but unless you're shaping, likely there's a high drop rate across the Internet not directly visible to you.

[igor.hamzic81]

I'm not doing any shaping in the moment on either side. I will try some traffic shaping to see if it helps.

[igor.hamzic81]

I managed to add some traffic shaping on both sides of the link but with the same results. The upload to HQ is super fast but the download to the branch is still slow and choppy(starts OK, slows down, stops, goes slow again, goes fast).

On the branch side 891F router I simply added a policy map to shape to 50 Mbps on the outside link of the router with this configuration.

policy-map Traffic-shape-pmap
class class-default
shape average 50000000

sh policy-map Traffic-shape-pmap
Policy Map Traffic-shape-pmap
Class class-default
Average Rate Traffic Shaping
cir 50000000 (bps)

show policy-map interface gi8
GigabitEthernet8

Service-policy output: Traffic-shape-pmap

Class-map: class-default (match-any)
463676 packets, 201109055 bytes
5 minute offered rate 773000 bps, drop rate 0000 bps
Match: any
Queueing
queue limit 64 packets
(queue depth/total drops/no-buffer drops) 0/510/0
(pkts output/bytes output) 463166/200452531
shape (average) cir 50000000, bc 200000, be 200000
target shape rate 50000000

On the HQ side I implemented traffic shaping on the FTD firewall(behind the VPN gateway and does all the "smart" stuff for the branch office traffic) through the use of a QoS policy that is limiting the traffic in both upload and download to 50 Mbps.

[Joseph W. Doherty]

Results not surprising.  Sorry if my mention of need for shaper implied such alone would solve issue.  I'll provide more information when I get to my PC.

[MHM Cisco World]

starts OK, slows down, stops, goes slow again, goes fast <<- this can be from the MTU and MSS you use, reduce both by 40 bytes and check the performance 

[igor.hamzic81]

Do I do this on both inside and outside interfaces on the router or only on the outside interface?

[MHM Cisco World]

Outside only

[igor.hamzic81]

I have dropped the MTU and MSS on the outside interface both to 1460 and then to 1400 but the results are the same.

[MHM Cisco World]

Ip mtu 1400

Ip tcp mss 1360

<- tcp mss must lower than mtu

[Joseph W. Doherty]

Additionally, those settings are placed on tunnel interface.

Technically, you want IP MTU on both tunnel interfaces but the

TCP

command only needs to be applied once, anywhere in the transit path for unencapsulated traffic, usually, though paired with IP MTU.