Cisco 891F with ZBF and IKEv2 tunnel slow download speeds

[igor.hamzic81]

Hi all,

in one of our branch offices we have switched from a dedicated line between locations to a internet link(50 Mbps) with IKEv2 tunnel to the main office on our 891F router. Since the router is now connected to the internet we also implemented ZBF on it for basic security.

All the traffic from the branch office is sent over the VPN tunnel to the main office and then to wherever it needs to go.

Since the change the users are having problems with slow download speeds from the main office from whatever source (ie. downloads from the internet, from office servers, ...).
Strangely enough traffic upload to the main office servers or to the internet or web surfing are fast using the same path and going over the same equipment.

This problem has me scratching my head as I really can't pinpoint the problem as the internet link itself is stable, VPN tunnel is stable, ZBF configuration is quite simple, there are no drops on the interfaces, router itself has CPU usage of 20 - 30%, there is no NAT.

I tried removing ZBF between INSIDE and OUTSIDE zones with no effect, simplified the ZBF, added ooo paramater map as I read it could help but nothing produced any result.

My suspicion is that something on the 891F router is the problem but I just can't figure it out.
I'm attaching the relevant config and hope that someone can point to where to look and how to solve this issue.

[MHM Cisco World]

correct, he mention INside and OUTside so I reply OUTside, 
OUTside is tunnel here.
just want to confirm 

[igor.hamzic81]

I have never configured tunnel interface like this. Any pointers how to apply changes to a specific tunnel interface?

[MHM Cisco World]

interface GigabitEthernet8
 description Softnet internet link
 ip address x.x.x.x 255.255.255.248
 zone-member security OUTSIDE
 duplex auto
 speed auto
 no cdp enable
 crypto map Styria-VPN

 @Joseph W. Doherty  he use IPSec under the WAN interface I see his attachment.
@igor.hamzic81 under above interface 

[igor.hamzic81]

Then I have been changing the MTU and MSS under the correct interface with no success I'm afraid.

[MHM Cisco World]

more reduce the MTU

ip mtu 1360 
ip tcp mss 1300

[igor.hamzic81]

I have tried the

mtu and

mss-adjust

commands on the outside interface and went to MTU 1360 and MSS 1320 but there has been no change. The situation remains unchanged with choppy download and fast upload from branch. Quite frustrating.

@Joseph W. Doherty
Does that mean I should apply these settings to the tunnel interface or should placing them on the outside interface be enough?

[Elliot Dierksen]

You should apply the

MTU

and

adjust-mss

to the tunnel interfaces, not the physical interfaces.

[Joseph W. Doherty]

Will need OP confirmation, or review of attached config (on my phone, cannot open attachments), but crypto might not being using any tunnel interfaces.  (If so, I haven't seen that usage in years).

[igor.hamzic81]

When I do

show ip int brief

I see the following:

Interface IP-Address OK? Method Status Protocol
Async3 unassigned YES unset administratively down down
BRI0 unassigned YES NVRAM administratively down down
BRI0:1 unassigned YES unset administratively down down
BRI0:2 unassigned YES unset administratively down down
FastEthernet0 unassigned YES NVRAM administratively down down
GigabitEthernet0 unassigned YES unset up up
GigabitEthernet1 unassigned YES unset administratively down down
GigabitEthernet2 unassigned YES unset administratively down down
GigabitEthernet3 unassigned YES unset administratively down down
GigabitEthernet4 unassigned YES unset administratively down down
GigabitEthernet5 unassigned YES unset administratively down down
GigabitEthernet6 unassigned YES unset administratively down down
GigabitEthernet7 unassigned YES unset administratively down down
GigabitEthernet8 x.x.x.x YES NVRAM up up
NVI0 x.x.x.x YES unset up up
Vlan1 unassigned YES unset administratively down down
Vlan15 10.71.3.1 YES NVRAM up up
Vlan16 10.71.1.1 YES NVRAM up up
Vlan20 10.71.2.1 YES NVRAM up up

Gi8 and NVI0 interfaces have the same public IP.

[Joseph W. Doherty]

I've just erased most of what this reply previously contained, because although I still suspect what's the likely cause of your performance issue, and further believe, it might be remediated, it's not a trivial undertaking.  Using shapers, would also likely be a key part of mitigation, as might HQ Internet access for raw/regular Internet traffic separate from VPN transit Internet traffic.

You best option might be to try to find a consultant to work with you.

BTW, the branch shaper stats show:

463676 packets, 201109055 bytes
(queue depth/total drops/no-buffer drops) 0/510/0
(pkts output/bytes output) 463166/200452531

463676 - 463166 = 510 packets dropped (.11%)
201109055 - 200452531 = 656,524 bytes dropped

We can now "see" what might (why might? because we also don't know how your provider enforces the 50 Mbps rate) have been happening within the Internet transit path.  (From these stats, branch to HQ, doesn't appear to be experiencing much of a problem.)

[Elliot Dierksen]

---

@Joseph W. Doherty wrote:

crypto might not being using any tunnel interfaces.

---

Good point. I was thinking DMVPN.

In that case, the

adjust-mss

should be on the inside interface of the devices on each side of the link. I would not change the MTU.

[Joseph W. Doherty]

"Good point. I was thinking DMVPN."

Yup or p2p GRE or VTI tunnels.

"In that case, the

adjust-mss

should be on the inside interface of the devices on each side of the link."

With the old crypto interface approach, there's no tunnel interface equivalent.  In the attachment, it should work if applied on all internal interfaces (in this case, the SVIs) that will use crypto.

If applied on the crypto interface, hmm I wonder - consider 

TCP adjust-mss

in light of:

Because

crypto map

is directly attached to physical interfaces, there is no clear feature separation in the underlay transport vs. overlay IPsec session. This adds complexity to the support features that deal with both pre- and post-encapsulated traffic; for example, QoS pre-classify and IP access-group. And there are features that work only with either pre- or post-encapsulated traffic but not with both; for example, embedded packet capture and NetFlow. There is lack of support for multicast traffic, non-IP traffic, and equal-cost multipath (ECMP) traffic.

Logically,

TCP adjust-mss

must be applied to the clear text flow.

Oh, BTW, also looking at the above, came across, the

crypto map approach

is on its way out.

"I would not change the MTU."

Change MTU, definitely not  Change IP MTU, usually recommended.  However, same issue with the above, shouldn't be applied to encrypted traffic.  If applied to crypto interface, is it applied to clear text, encrypted or both?

It too, might be used on SVIs.

Although often generally suggested, for avoiding fragmentation, in the real world, likely it  matters little when

TCP adjust-mss

is being used.  This because, I suspect most non-TCP traffic doesn't set DF bit.  And for TCP traffic,

TCP adjust-mss

doesn't depend on DF bit setting.

[MHM Cisco World · Accepted solution]

If there is other sites' move router to other sites and check same config' 

I See one time same issue and finally the issue was isp.

Dont waste your time 

Thanks 

MHM

[Joseph W. Doherty · Accepted solution]

BTW @igor.hamzic81, to be clear, I'm willing to further help you, as, again, I've had experience (successfully) doing what you're doing.  Again, though, doing this across a multipoint transit infrastructure, where you have no control over such an infrastructure, does create many potential issues, and to mitigate some of those, it's not "pretty", or as I wrote earlier, trivial.

To use the Internet, well, as a Enterprise class dedicated WAN takes a big commitment, in time and, sometimes, capital.  However, doing this also usually offers a great ROI.

Again, as doing such is not trivial, is why I recommend trying to find a consultant that can help you accomplish this.  As, the assistance you need, I believe cannot be easily rendered though this site.

BTW, if you want to "prototype" using the Internet as your WAN transit, add an inexpensive Internet connection, at HQ, and create a p2p link between HQ and your branch.  p2p is easy to optimize, and could be done at little cost in time or capital.

If you like how well such a prototype works, then you might consider going to a DMVPN like approach (like mentioned by @Elliot Dierksen), but although they are rather simple to setup, to get them to work well (i.e. avoiding performance issues), is the non-trivial part.

[igor.hamzic81]

Hi all,

sorry for the lack of response but other projects took my time. Anyway from the information I got it seems that the problem is in one router of our ISP which seems to be swamped with traffic. They are now in the process of replacing it and I hope that the situation will be better next week when I will be able to confirm it really is OK(I will close the topic then).

Thank you all for your help and advice. I really got some new info for the future and I really need to look towards changing the VPN tunnel from the

crypto map

to a tunnel interface in the future.