Cisco 2951/k9 vpn IWAN throughput issue

[Beau Clark]

So I have IWAN pushed out to multiple locations and have run into a pretty serious performance issue. The 2951 router caps out somewhere between 20 and 30 Mbps. The CPU on the router goes to 99%, seems to not be able to handle more than 30Mbps. If I remove IWAN and just run the transfer, I get 50Mbps of throughput which is the bandwidth of the circuit. 

Has anyone seen this issue. I have tested and found this to be an issue on 2 different 2951 routers. 

I am running on IOS version "c2951-universalk9-mz.SPA.155-3.M5.bin" Do I need a different IOS version? I would like to see the router give me at least 50 Mbps of IWAN throughput. The documentation says I should get up to 170Mbps of VPN throughput.

IWAN configuration pushed to router from APIC-EM version 1.5

Thanks in advance for taking a look at this.

[Francesco Molino]

Hi

First of all i would go with ios 15.5.3M7 which is the recommended software. Lot of bugs have been correct in M6 version.

Then when you're getting high cpu, what's the process spiking? Do you have any logs when this is occurring?

[Beau Clark]

Ok, I have upgraded to c2951-universalk9-mz.SPA.155-3.M7.bin

Nothing in the logs during a large file copy, just 3 messages about the traffic class path changing. IWAN Class of service messages as it chooses the alternate path for some traffic, they look like this:

DOMAIN-5-TC_PATH_CHG: Traffic class Path Changed.

I transferred 500 Megabytes of data and the speed never went above 22Mbps. If I remove IWAN, I get 47Mbps+

Nothing shown when I check the processor...

Router#sh proc cpu sort
CPU utilization for five seconds: 99%/94%; one minute: 94%; five minutes: 59%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
13 6644 8 830500 1.11% 0.13% 0.06% 0 Licensing Auto U
2 1732 76 22789 1.04% 0.72% 0.35% 0 Load Meter
274 4012 26 154307 0.96% 0.11% 0.03% 0 PA Export
481 46712 4473 10443 0.89% 1.33% 0.77% 0 CENT_PROBE
460 5416 68 79647 0.81% 0.15% 0.08% 0 PERF MON ASYNC E
188 32676 22274 1467 0.66% 0.77% 0.64% 0 IP Input
480 1620 35 46285 0.29% 0.08% 0.01% 0 CENT EXPORTER TH
95 7312 403 18143 0.07% 0.10% 0.08% 0 Per-Second Jobs
114 8940 1203 7431 0.07% 0.09% 0.09% 0 Netclock Backgro
325 1164 430 2706 0.07% 0.05% 0.02% 644 Virtual Exec
307 708 233 3038 0.07% 0.01% 0.00% 0 WCCP V2 Protocol
408 7136 360 19822 0.07% 0.16% 0.36% 0 CENT-BR-0
356 6408 959 6681 0.07% 0.04% 0.09% 0 Crypto IKEv2
476 336 194 1731 0.07% 0.00% 0.00% 0 EIGRP-IPv4
459 1552 1466 1058 0.07% 0.19% 0.21% 646 Virtual Exec
93 260 76 3421 0.07% 0.00% 0.00% 0 Compute load avg
17 2448 324 7555 0.00% 0.04% 0.01% 0 Environmental mo
16 0 1 0 0.00% 0.00% 0.00% 0 BGP I/O

I really appreaciate any information you might be able to provide. Thanks for taking a look.

Do you have any other ideas?

[Francesco Molino]

Not sure yet, but I guess you kinda have same bug I faced...

This is impacting all 15.5.3. Can you upgrade to a later ios? Like 15.6 or 15.7

There were 2 bugs I faced. 1 of them is : https://bst.cloudapps.cisco.com/bugsearch/bug/CSCux60466/?referring_site=bugquickviewredir

The other I've to do some research to recall the bug id

[Marek.Bedy]

I have the same issue on IOS 15.7(3)M1 .

[Francesco Molino]

You're still limited to 30 Mbps? Can you share your config please?
Also what's the CPU without iWAN and with iWAN? Can you share some outputs?

[Beau Clark]

30 Mbps? I wish... My 2951 routers cap out at about 22 Mbps, my 2921s cap out at 13 Mbps. 

When little to no load on the circuit, the CPU is around 20-40%, it does not show what process is using the processor, it just shows CENT_PROBE using the most processor at some where around 5-8%, everything else combined might show an additional 2%.

I have attached a redacted copy of my config, but it is unlikely the config will tell you anything. It is a basic config pushed out from APIC-EM brownfield deployment.

I am doing WCCP redirection of traffic for WAN acceleration to a 594 wan accelerator. This makes a big performance difference for the site, getting me more throughput, but not solving the problem. 

My internet IP address is a public address directly on the internet on a /29 (not through a firewall). I have a local ASA firewall connected to the same /29 where I am routing out local internet traffic so it does not go across the IWAN tunnels. 

If I kick off a large file transfer, just a simple CIFS transfer, the CPU will spike to 100% within 20 seconds and stay at 100% through the entire transfer. Within 10 seconds of the transfer finishing, the CPU drops back down to 10-40%, depending on the site.

Since most of my IWAN sites are on 29xx routers, this is consistent behavior at EVERY site with a 29xx router. My sites with ASR routers run smooth, my server guys replicate data at 100Mbps through the IWAN links connected ASR to ASR with no issue at all, none of the CPUs on the ASR spike at all. Users do not even realize that replication data is being sent to the site. So this may be the solution, go out and buy ASR routers for sites with over 15Mbps links. 

So, thinking it may be an issue with a version of something IWAN related. I have upgraded:

Prime from 3.2 to 3.4

APIC-EM from 1.5 to 1.6.1

APIC-EM IWAN app from 1.5 to 1.6.1

APIC-EM PnP to 1.6.2

IOS from 15.5.x to 15.7.3.M2

WAN accelerators to version 6.2.3d

LiveAction from 4.2 to 7

The IPSEC process does spike up a little bit during the transfer, maybe 5%, but what I think is causing the high CPU is the COS/QOS, since I do not see a process associated with the problem, and I have seen 28xx routers demonstrate similar phantom high CPU behavior with high throughput doing just COS. 

Thank you for taking a few minutes to take a look at this issues, I really appreciate any input you are able to provide. 

[Beau Clark]

Still experiencing the issue on 15.7.3.M2...

[Francesco Molino]

I thought you would go with the recommended ios version but you took a more recent one.
Can you open a TAC case?
My last iWAN deployed has few 2951 using 15.6.3M4 and no issue reported up to now (deployed 2 weeks ago).

[Marek.Bedy]

@Francesco Molino what is performance with 15.6.3M4 on yours 2951 ?

[Francesco Molino]

In this setup it was 10M policed link and we're getting more or less between 8 to 9Mb.
We did the test when there was no charge.

[Beau Clark]

So you have an IWAN site with a 2951 that only has 1 - 10Mbps link.

Is the WAN link private WAN or public Internet?

[Francesco Molino]

2 links 10Mbps (MPLS and Internet)

[Beau Clark]

15.7.3M is the Cisco recommended version for IWAN version 2.2.1, 15.6 is the recommended version for IWAN 2.2:

https://www.cisco.com/c/en/us/td/docs/solutions/Enterprise/Intelligent_WAN/release/notes/iwanrn-2-2-1.html

How much actual throughput can you get through your 2951 routers running 15.6? Is this a production deployment or a lab deployment? What does the processor look like when you are pushing a few gigabytes at a bandwidth level over 30Mbps? Can I see a copy of your working config?