ISR 4451 High CPU Usage

[justclash4]

Hi there,

We have a ISR 4451 with Cisco IOS XE Software, Version 16.06.03.

We are facing with high cpu usage recently.

Average usage was between 1 and 3 percent. I configured bandwidth shaping on 3 tunnel interfaces and after 3 or 4 days cpu usage began to grow up.

I need to detect root cause and fix this issue before it goes about 90% or above

I appreciate your help and consideration

[Leo Laohoo]

Post the complete output to the following output: 

• sh version

• sh platform software resource

• sh platform software status con brief

NOTE:  Do not post screenshots.

[justclash4]

show version:

Cisco IOS XE Software, Version 16.06.03
Cisco IOS Software [Everest], ISR Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.6.3, RELEASE SOFTWARE (fc8)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2018 by Cisco Systems, Inc.
Compiled Wed 28-Feb-18 23:54 by mcpre

Cisco IOS-XE software, Copyright (c) 2005-2018 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: IOS-XE ROMMON

R-2 uptime is 2 years, 51 weeks, 2 days, 6 hours, 35 minutes
Uptime for this control processor is 2 years, 51 weeks, 2 days, 6 hours, 36 minutes
System returned to ROM by Reload Command at 18:26:24 TEH Wed Mar 13 2019
System restarted at 18:29:55 TEH Wed Mar 13 2019
System image file is "bootflash:isr4400-universalk9.16.06.03.SPA.bin"
Last reload reason: Reload Command

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Suite License Information for Module:'esg'

--------------------------------------------------------------------------------
Suite Suite Current Type Suite Next reboot
--------------------------------------------------------------------------------
FoundationSuiteK9 None None None
securityk9
appxk9

AdvUCSuiteK9 None None None
uck9
cme-srst
cube


Technology Package License Information:

-----------------------------------------------------------------
Technology Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------
appxk9 None None None
uck9 None None None
securityk9 securityk9 Permanent securityk9
ipbase ipbasek9 Permanent ipbasek9

cisco ISR4451-X/K9 (2RU) processor with 1796073K/6147K bytes of memory.
Processor board ID FGL214680QJ
4 Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
7057407K bytes of flash memory at bootflash:.
0K bytes of WebUI ODM Files at webui:.

Configuration register is 0x2102

---------------------------------------------------------------------------------------------------

show platform software resource not found

instead, show platform resource:




R-2# show platform software re?
replication

R-2# show platform res
R-2# show platform resources
**State Acronym: H - Healthy, W - Warning, C - Critical
Resource Usage Max Warning Critical State
----------------------------------------------------------------------------------------------------
RP0 (ok, active) H
Control Processor 13.90% 100% 90% 95% H
DRAM 2438MB(64%) 3804MB 90% 95% H
ESP0(ok, active) H
QFP H
DRAM 116622KB(22%) 524288KB 80% 90% H
IRAM 0KB(0%) 0KB 80% 90% H
CPU Utilization 20.00% 100% 90% 95% H

---------------------------------------------------------------------------------------------------

show platform software status con brief:

R-2#show platform software status control-processor brief
Load Average
Slot Status 1-Min 5-Min 15-Min
RP0 Healthy 1.10 1.14 1.24

Memory (kB)
Slot Status Total Used (Pct) Free (Pct) Committed (Pct)
RP0 Healthy 3896280 2495408 (64%) 1400872 (36%) 2323440 (60%)

CPU Utilization
Slot CPU User System Nice Idle IRQ SIRQ IOwait
RP0 0 11.68 5.39 0.00 81.71 0.00 1.19 0.00
1 5.10 4.30 0.00 89.90 0.00 0.70 0.00
2 13.60 5.80 0.00 80.60 0.00 0.00 0.00
3 7.00 5.70 0.00 86.88 0.00 0.40 0.00
4 10.28 7.09 0.00 82.51 0.00 0.09 0.00
5 9.00 5.00 0.00 85.98 0.00 0.00 0.00
6 12.10 6.10 0.00 81.80 0.00 0.00 0.00
7 6.09 6.79 0.00 87.01 0.00 0.09 0.00

---------------------------------------------------------------------------------------------------

[Leo Laohoo]

---

@justclash4 wrote:

DRAM 2438MB(64%)

---

Currently, the CPU is fine but memory, however, is not.  There is a slow memory leak.  Ideally, memory utilization should be <48%.

Can I see the "first page" of the following output:  sh proc memory platform sort location rp active 

[justclash4]

R-2#sho processes memory platform sorted location rp active
System memory: 3896280K total, 2496296K used, 1399984K free,
Lowest: 1399984K
Pid Text Data Stack Dynamic RSS Total Name
--------------------------------------------------------------------------------
11045 11937 249740 132 28140 249740 4158420 fman_fp_image
11023 89 207148 132 28180 207148 3678612 cpp_cp_svr
2599 251690 896428 136 124 896428 3431160 linux_iosd-imag
10089 68 133188 132 6764 133188 3035744 cpp_sp_svr
9821 363 105088 136 3832 105088 3009372 cpp_ha_top_leve
9337 151 101824 152 2788 101824 2999508 cpp_driver
8616 65 31996 132 1100 31996 2906268 cpp_cdm_svr
29720 7080 112020 136 2264 112020 1274548 fman_rp
24677 230 94816 136 5108 94816 971496 bsm
8283 266 43280 132 3772 43280 849036 cman_fp
27000 202 82080 136 7112 82080 807936 cli_agent
20171 472 60588 136 4236 60588 768284 iomd
2878 151 79412 136 268 79412 633492 smd
27090 97 83916 136 3484 83916 621492 dbm
1003 460 60164 136 25780 60164 490232 smand

[Leo Laohoo]

---

@justclash4 wrote:

R-2#sho processes memory platform sorted location rp active
System memory: 3896280K total, 2496296K used, 1399984K free,
Lowest: 1399984K
Pid Text Data Stack Dynamic RSS Total Name
--------------------------------------------------------------------------------
11045 11937 249740 132 28140 249740 4158420 fman_fp_image
11023 89 207148 132 28180 207148 3678612 cpp_cp_svr
2599 251690 896428 136 124 896428 3431160 linux_iosd-imag
10089 68 133188 132 6764 133188 3035744 cpp_sp_svr
9821 363 105088 136 3832 105088 3009372 cpp_ha_top_leve
9337 151 101824 152 2788 101824 2999508 cpp_driver
8616 65 31996 132 1100 31996 2906268 cpp_cdm_svr
29720 7080 112020 136 2264 112020 1274548 fman_rp
24677 230 94816 136 5108 94816 971496 bsm
8283 266 43280 132 3772 43280 849036 cman_fp
27000 202 82080 136 7112 82080 807936 cli_agent
20171 472 60588 136 4236 60588 768284 iomd
2878 151 79412 136 268 79412 633492 smd
27090 97 83916 136 3484 83916 621492 dbm
1003 460 60164 136 25780 60164 490232 smand

---

Thanks for the output.  I cannot see anything suspicious.  

Can you generate the entire output and then export it to an Excel spreadsheet. 

Sort "RSS" column from highest to lowest.  Whatever pops on top, that is the culprit. 

Next, firmware upgrade.  Upgrade to the latest 16.6.X and stay at 16.6.X.  

WARNING:  Do not upgrade to 16.12.X because the train is very unstable.  

Cisco ISR & ASR 1k Routers: IOS-XE/Firmware Upgrade (Install Mode)

[balaji.bandi]

 Version 16.06.03.  - this quite OLD code, if you get a chance worth uplift to new version of code possible. ( try 16.12.5 or 17.X)

also, router uptime more than 2 years.. it's worth rebooting and check also advised.

[Georg Pauwen]

Hello,

in addition to the other remarks, also post the full running configuration of your router, maybe we can spot something that causes high CPU utilization...

[Joseph W. Doherty]

Did the jump in CPU appear on the software task side or the interrupt side?  (In your posting, most of the CPU is being used on the software side.  Ideally, we like to see software side use a small percentage of the total CPU.)

Your top four software tasks seem, to me, abnormally high.  Might be a bug.  Might be something with how you've configured logging.

If you remove tunnel shaping, does CPU usage drop?

[justclash4]

Hi,

I removed the shaping configuration from tunnel interface but nothing changed in cpu utilization.

I am logging all of the NAT events from 5 or 6 month ago.

Is it possible that this high cpu utilization is cause of a bug? the router uptime is close to 3 years. If it was a bug it should be happened from the beginning of router liveliness.

[marce1000 · Accepted solution]

                         >.... If it was a bug it should be happened from the beginning of router liveliness.

 - Not at all , 3 years uptime is a very long period , then bugs may appear related to resource leaks or and management of resources (CPU , memory...)

 M.

[Joseph W. Doherty]

As @marce1000 notes, some bugs can take a while to show themselves.

One way to help identify if you're dealing with a time related bug, is reload/rebooting the system and "see" whether issues reappear, immediately, or not.  If the don't reappear immediately, and do reappear "later", more likely it's a time related bug (however, there are non-bug issues that can arise, over time, that are not "bugs" but are due to your traffic and the way IOS works - usually caused by not having enough RAM).

You're pretty certain there's been no major changes is traffic volume, traffic type, syslogging volume, etc., when you noticed the sudden CPU increase?

Like some of the others, if you can, I would suggest upgrading to a more recent version of the IOS.  Personally, if you don't need "newer" IOS features, and your software train is still being supported, I would just upgrade to the latest, or recommended, release of that train, as also recommended by @Leo Laohoo.  (Remember, later releases of the same software train are bug fixes.  In theory, an "improved" version of the software you're now running.)