bandwidth throughput limit on tunnel interface

[Latchum Naidu]

Hi All,

We have two sites using Cisco 1841 routers and they are on a DMVPN cloud.

Both the sites have a 10 Mb internet link through which all the traffic passes.

When I do a UDP bandwidth test between these two sites using the “IPERF” tool with even 7Mb as bandwidth, I see that the routers CPU goes up on both sides. Its about 95%, and we see packet loss.

Why is this happening, cant these routers handle even this much of traffic through the tunnel interface?

What is the amount of traffic that can be sent through the Tunnel (DMVPN) interface.

Regards,

Naidu.

[wzhang]

Hi,

There really is no hard throughput limit on the tunnel interface. The performance of the router depends on the platform selection (both chassis and hardware encryption accelerator) and software features enabled. For the 1841 using the onboard crypto module, you should be able to get up to 20M of encrypted throughput with an IMIX-like traffic profile. When you observed 95% CPU, was the cpu usage mostly under network interrupt or a particular process (show proc cpu)? What's the traffic profile you used for your test? Was the ip mtu on the tunnel interface properly tuned so that there is no reassembly on the receiver (check the reassembly counter in "show ip traffic") which is the typical cause for high cpu due to excessive IP Input activity?

Thanks,

Wen

[Latchum Naidu]

Hi Wen,

Thanks for your response.

Was the cpu usage mostly under network interrupt or a particular process (show proc cpu)?
We observed the high cpu usage when we are doing bandwidth test (with 7Mb only)

What's the traffic profile you used for your test?
I didn't understand it clearely. We are using "Ipref" to check available bandwidth

Was the ip mtu on the tunnel interface properly tuned so that there is no reassembly on the receiver (check the reassembly counter in "show ip traffic")
The MTU we tuned on the tunnel interface is "1400"

Which is the typical cause for high cpu due to excessive IP Input activity?
I didn't understand it, you mean when doing bandwidth test which process is causing high cpu is that right?

Please revert me if you need any infor..

Regards,

Naidu.

[wzhang]

Hi,

Could you provide the first 10 lines of the "show proc cpu sorted" output when you are running the test? Please also provide 2 captures of "show ip traffic" output, once before the test and once after. By "traffic profile", I meant what was your traffic mix that's going through the tunnel? Is it a constant stream with fixed packet size? or IMIX? What's the average packet size?

Thanks,

Wen

[Latchum Naidu]

Hi Wen,

Find the below factors for device hardware and IOS details :

Cisco IOS Software, 1841 Software (C1841-ADVIPSERVICESK9-M), Version 12.4(13b), RELEASE SOFTWARE (fc3)

Cisco 1841 (revision 6.0) with 233472K/28672K bytes of memory.
Processor board ID FCZ103511WH
2 FastEthernet interfaces
1 Virtual Private Network (VPN) Module
DRAM configuration is 64 bits wide with parity disabled.
191K bytes of NVRAM.
62720K bytes of ATA CompactFlash (Read/Write)

Find the below VPN tunnel configuration:

interface Tunnel10
bandwidth 10000
ip address 192.168.220.60 255.255.255.0
no ip redirects
ip mtu 1400
ip nbar protocol-discovery
ip nhrp authentication xxxxxxxxxxxx
ip nhrp map multicast dynamic
ip nhrp map multicast 193.xxx.xxx.xxx
ip nhrp map 192.168.220.1 193.xxx.xxx.xxx
ip nhrp network-id 1
ip nhrp nhs 192.168.220.1
ip nhrp cache non-authoritative
ip tcp adjust-mss 1360
no ip split-horizon eigrp 100
tunnel source FastEthernet0/1
tunnel mode gre multipoint
tunnel key xxxxxxxxxx
tunnel protection ipsec profile dmvpn-profile shared
end

When we saw hi CPU the process whicht was taking maximum cycles was IP Input.

We were doing a test with a too "IPERF", below is the command.

iperf -c xxx.xxx.xxx.xxx -b 7M -i 30 -t 600 -d

Regards,

Naidu.

[wzhang]

Hi,

If you are seeing high cpu in the IP Input process, then that implies packets are being process switched as opposed to being CEF switched. This is one of the most common performance issue on IOS routers. The next step is find out how much traffic is being process switched and why. You can start by using "show int stat" to find out which interface has the most processed traffic, and in which direction. Also make sure you rule out the obvious reasons such as CEF disabled in the configuration, access-list logging, or ip reassembly issues as outlined in my last response. If you still cannot figure out what's causing the packets to be punted out of the CEF path into process, the troubleshooting steps may be somewhat involved considering the version you are running does not have an easy way to tell you the punt reason. I would suggest you open a TAC case to have this investigate further.

Thanks,

Wen

[Latchum Naidu]

Hi Wen,

Thanks again for your reply.

What if we see packets processor swicted on tunnel interface in received? under "packets in"

How to resolve that?

Thanks in advance.

Regards,

Naidu.

[wzhang]

Hi,

There are many reasons why packets can be process switched. I assume you have ruled out the obvious ones mentioned in my last post, ie., ip reassembly, access-list logging, etc. One thing that you can do to further narrow this down is to remove all features from the tunnel interface and start with a bare-bone configuration. Test with transit traffic (through-the-router as opposed to to-the-router), and you should cef switch all traffic. Once you have this as a baseline, start adding features one at a time and retest. This will give you a pretty good idea on what feature is causing process switching. Again, troubleshooting switch path issues can be fairly involved, and that's what TAC is for .

Thanks,

Wen

[Latchum Naidu]

Hi Wen,

We will have to check about the features remove under tunnel config in the service window as it is production.

The another problem what i observed is on this router is "Huge buffer misses" which is droping the translation packets.

                                 This hour           Today

Huge Buffer Misses 10409 misses   82278 misses

Can you please suggest how i can control buffer misses.

Thanks in advance.

Regards,

Naidu.

[wzhang]

Hi, Naidu:

Huge buffers are 18K in size, and is not commonly used. The most typical use of huge buffer is ip reassembly. I know I've asked this a couple of times already, but you did check for reassembly and you don't see any, correct? Also, buffer "misses" is generally not a problem in and by itself, as it simply indicates the number of free buffers on the free list has dropped below the "min" threshold, packets shouldn't be dropped at this point. To fix buffer misses or failures, you'd first need to identify what's using those buffers, and then use buffer tuning to address that. See  http://www.cisco.com/en/US/partner/products/hw/routers/ps133/products_tech_note09186a00800a7b80.shtml.

Thanks,

Wen