Re: Unexplained CPU usage on a Cisco 2811

stefan.radovanovici · ‎07-25-2013

Hi All,

I have a Cisco 2811 which terminates some serial connections from a customers. There is NAT configured in order to provide internet access and in addition, two encrypted GRE Tunnels over a leased line providing access to some services. There is some minimal QoS configured to prioritize Voice but nothing complex. I am running 12.4(24)T.

The problem I am having is what I think is excessive CPU usage. The CPU is at a constant 50-60% usage with only about 1.5Mbps average going over the internet connection and about 1Mbps average over the encrypted GRE tunnel. There are 5 serial interface, the average bandwidth there is less than 512kbps. I find it hard to believe that all this would justify the 60% cpu usage, or higher when it peaks.

What is weird is that show proc cpu his shows high usage but show proc cpu sorted shows almost no CPU usage at all:

Router#sh proc cpu his

Router 10:35:52 AM Thursday Jul 25 2013 MEST

444444444555555555555555555555555555555555555555555555555555

333388888444445555577777333336666666666888886666655555111110

100

90

80

70

60 ********** *************************

50 ********************************************************

40 ************************************************************

30 ************************************************************

20 ************************************************************

10 ************************************************************

0....5....1....1....2....2....3....3....4....4....5....5....6

0 5 0 5 0 5 0 5 0 5 0

CPU% per second (last 60 seconds)

555455565866976559454545444555555565665555455544544444444444

810521116003477969636171978144365449004276760059361828535543

100 *

90 * *

80 * ** *

70 * #** *

60 * ****###**** ** #*** ** *

50 #***##*###########*#***#***###############*#****** * ** **

40 ############################################################

30 ############################################################

20 ############################################################

10 ############################################################

0....5....1....1....2....2....3....3....4....4....5....5....6

0 5 0 5 0 5 0 5 0 5 0

CPU% per minute (last 60 minutes)

* = maximum CPU% # = average CPU%

Router#show proc cpu sorted

CPU utilization for five seconds: 61%/56%; one minute: 57%; five minutes: 51%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

182 1371308 128454496 10 2.39% 2.34% 2.28% 0 HQF Shaper Backg

205 15336 3550 4320 0.87% 0.08% 0.06% 515 SSH Process

118 681500 400673 1700 0.63% 0.28% 0.27% 0 IP Input

246 202012 74019 2729 0.23% 0.10% 0.10% 0 Crypto IKMP

2 51596 104116 495 0.07% 0.08% 0.07% 0 Load Meter

321 29560 45467 650 0.07% 0.02% 0.01% 0 IP-EIGRP: PDM

317 227236 544031 417 0.07% 0.04% 0.05% 0 NAT MIB Helper

I am not seeing any fragmentation to speak of, traffic is fast switched ... I don't know what is causing the CPU load. I don't even know if it's real, considering that "show proc cpu sorted" shows nothing.

Any ideas ?

Best regards,

Stefan

Ivan Shirshin · ‎07-29-2013

Hi,

CPU cycles are taken in your case not by process but mostly by interrupts (2nd number in the following output).

CPU utilization for five seconds: 61%/56%;

It could be some unwanted traffic or traffic being sent to CPU for handling. Or drops on interfaces. Or simply performance issue. Follow this guide for interrupts troubleshooting:

http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af0.shtml

And check this link for similar problem:

https://supportforums.cisco.com/thread/2184745

Kind Regards,
Ivan Shirshin

**Please grade this post if you find it useful.

Kind Regards,
Ivan

Joseph W. Doherty · ‎07-29-2013

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

GRE/IPSec - is router configured to minimize the need to fragment? (http://www.cisco.com/en/US/tech/tk827/tk369/technologies_white_paper09186a00800d6979.shtml)

stefan.radovanovici · ‎07-31-2013

Hello Joseph,

Yes, as far as I can tell the GRE Tunnels are configured to avoid/minimize fragmentation. The MTU is set low enough to allow for all IPSEC headers, also for MPLS encapsulation (the tunnel destination is reachable over a MPLS cloud). tunnel path-mtu-discovery is also configured on the tunnel interfaces.

show ip traffic shows a little bit of fragmentation (almost nothing compared to the total amount of traffic going throgh the tunnel) but it does show something I don't understand:

Router#sh ip traffic | i rag

Frags: 4 reassembled, 0 timeouts, 0 couldn't reassemble

3922 fragmented, 7853 fragments, 144979 couldn't fragment

The could't fragment counter increases slowly and I can't find any info about what it means. Does it show big incoming packets with the DF bit set ? Or does it mean something else ?

Regards,

Stefan

Joseph W. Doherty · ‎07-31-2013

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

You're also using ip tcp mss-adjust?

Off the top-of-my-head, don't know exactly what that fragmentation stat is telling us.

BTW, in the past I've seen a 2811's CPU max out at about 20 Mbps (duplex) production traffic, so I also think your CPU usage appears abnormally high for just about 2 Mbps (duplex).

stefan.radovanovici · ‎08-02-2013

No, we don't have that command configured yet. Looking at the traffic, almost all packets are less than 1000bytes, and quite a lot of it is voice traffic. That means the ip tcp mss-adjust would not really help that much.

We noticed that removing encryption from the GRE tunnel drops the CPU usage by at least 10%. That seems excessive to me for only 2Mbps traffic on average considering the encryption is supposed to be in hardware. It goes back up when we re-enable encryption.

There are a lot of nat translation present on the router though, some of them coming via the tunnel. They all go to the internet. Maybe that in combination with the GRE/IPSEC tunnel causes the CPU load, who knows. If I am not mistaken NAT is process switched.

@Ivan: Thanks for the two links, I'll have a look.

Regards,

Stefan

Joseph W. Doherty · ‎08-02-2013

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Surprisingly little fragmentation can drive CPU hard, especially when doing encryption as it's possible a too large packet is fragmented, then encrypted, and the encryption creates a need to fragment again.

Small packets, too, will drive the CPU harder for their bandwidth. The 2811 is rated at 120 Kpps (for minimum packets sizes, I believe), which should allow up to about (in a perfect situation) 30 Mbps duplex.

Yes, the encryption, itself, is performed by on-board hardware, but there's also additional overhead when doing encryption, handled by the main CPU. A delta of 10%, I don't think, is totally out-of-bounds. (The on-board AIM also, I recall [?], doesn't offer quite the performance of the optional add-on encryption module.)

Your overall CPU still seems high for the total volume of traffic, again under 2 Mbps or so, correct? Unfortunately it can be very difficult to "see" into interrupt CPU consumption.

If you have maintenance on this 2811, it's something you might open with TAC.

You might also try a different IOS version (is yours the latest patch level in that specific release chain?).

stefan.radovanovici · ‎08-02-2013

Hello Joseph,

I was mistaken it seems, we do have ip tcp mss-adjust configured on the tunnel interface. We run 12.4(24)T, maybe we will try with another IOS which is not part of the T Train, see if that help.

Regards,

Stefan

Joseph W. Doherty · ‎08-02-2013

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

BTW, we use 12.4(24)T3 for our VPN G1 ISRs that don't have sufficient RAM/FLASH to run later. It seems to work fairly well for us.

jawad-mukhtar · ‎08-02-2013

is there any natting going on for UDP.

for more troubleshooting

enable netflow on router to monitor souce and destination traffic with ports.

Jawad