Cisco 4431 routing constantly at 95% memory use

[Anthony O'Reilly]

I can see the router running at 95% memory , there’s a “ license auto update “ process using all of the cpu.

Router# sh proc mem sorted holding
Processor Pool Total: 1688360784 Used: 306333648 Free: 1382027136
lsmpi_io Pool Total: 6295128 Used: 6294296 Free: 832

PID TTY Allocated Freed Holding Getbufs Retbufs Process
0 0 410367160 131747168 257960896 0 0 *Init*
64 0 1198391792 1198391792 29864 0 0 Licensing Auto U

The router details are:

#sh ver
Cisco IOS XE Software, Version 03.16.04b.S - Extended Support Release
Cisco IOS Software, ISR Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 15.5(3)S4b, RELEASE SOFTWARE (fc1)

Suite License Information for Module:'esg'

--------------------------------------------------------------------------------
Suite Suite Current Type Suite Next reboot
--------------------------------------------------------------------------------
FoundationSuiteK9 None None None
securityk9
appxk9

AdvUCSuiteK9 None None None
uck9
cme-srst
cube

Technology Package License Information:

-----------------------------------------------------------------
Technology Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------
appxk9 None None None
uck9 None None None
securityk9 securityk9 Permanent securityk9
ipbase ipbasek9 Permanent ipbasek9

cisco ISR4431/K9 (1RU) processor with 1648789K/6147K bytes of memory.
Processor board ID FCZ22174018
1 Ethernet interface
4 Gigabit Ethernet interfaces
1 ATM interface
32768K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
6598655K bytes of flash memory at bootflash:.

The inventory on the router is as follows

#sh inv
NAME: "Chassis", DESCR: "Cisco ISR4431 Chassis"
PID: ISR4431/K9 , 

NAME: "Power Supply Module 0", DESCR: "250W AC Power Supply for Cisco ISR4430"
PID: PWR-4430-AC , 

NAME: "Power Supply Module 1", DESCR: "250W AC Power Supply for Cisco ISR4430"
PID: PWR-4430-AC , 

Any ideas to get this CPU memory

NAME: "Fan Tray", DESCR: "Cisco ISR4430 Fan Assembly"
PID: ACS-4430-FANASSY , VID: , SN:

NAME: "module 0", DESCR: "Cisco ISR4431 Built-In NIM controller"
PID: ISR4431/K9 , VID: , SN:

NAME: "NIM subslot 0/1", DESCR: "NIM-VAB-A - Multi-mode xDSL with VDSL bonding and ADSL1/2/2+ Annex A over POTS"
PID: NIM-VAB-A , 

NAME: "NIM subslot 0/0", DESCR: "Front Panel 4 ports Gigabitethernet Module"
PID: ISR4431-X-4x1GE , 

NAME: "module R0", DESCR: "Cisco ISR4431 Route Processor"
PID: ISR4431/K9 , 

NAME: "module F0", DESCR: "Cisco ISR4431 Forwarding Processor"
PID: ISR4431/K9 , 

Any ideas to get this CPU memory?

Thanks

[Leo Laohoo]

Wow. That's one old IOS.
Has anyone tried using a more recent version?

[valebaf88]

hello

I have an ISR 4331 with IOS XE Software, Version 16.06.05.

The issue was the high memory process not the cpu process.

------------------ show process memory sorted ------------------

Processor Pool Total: 1839083368 Used: 366550080 Free: 1472533288
lsmpi_io Pool Total: 6295128 Used: 6294296 Free: 832

PID TTY Allocated Freed Holding Getbufs Retbufs Process
79 0 280754735504 280536856576 10983088 0 0 IOSD ipc task
10 0 5999380672 5999421648 215464 5736394714 5736424442 Pool Manager

I only see these two processes increased "IOSD ipc task and Pool Manager"

In the forums of cisco web site is realted to this bugs.

- IOSD ipc task memory leaks when SPAs are inserted CSCsz21461

- Memory leak in Pool Manager process CSCtg23115

And i found aditional about the high memory process in the ISR4300

https://thwack.solarwinds.com/docs/DOC-186977

[Anthony O'Reilly]

Hi,

When we upgraded the router to the latest version solved the issue with CPU and memory resources.

Thanks

Anthony.

[Michal Stanczyk]

The IOS memory usage shouldn't be a concern as there's 1.4GB memory free in the Processor pool (on IOS side).

IOS is one of the many processes running on an IOS-XE platform.

I suspect you might be experiencing high memory usage on the kernel (platform) side.

We would need to see the platform memory usage to determine what's causing high memory usage:

# show platform resources
# show process memory platform sorted

HTH,

Michal

[Leo Laohoo]

Michal,
1. The OP is using a rebuild firmware. I've already recommended early on to upgrade the firmware.
2. The OP has upgraded the firmware the the problem went away.

[Michal Stanczyk]

OK, I'm not sure what OP means here but I was referring to the question from "valebaf88" posted 2 days ago which was related to ISR4k running 16.6.5 code :)

[Leo Laohoo]

Cool.

[Michal Stanczyk]

Anthony,

ISR4431 runs the IOS-XE operating system where IOS runs as a linux process among many other processes.

You have 4 GB of total memory and IOS gets only about 1.7 GB for its purposes. The remaining memory is available for other processes.

"sh proc mem sorted" refers to the IOS memory which is completely OK in your case (1.3 GB free).

Look at "show proc mem platform sorted" to determine the linux processes that might be consuming large amount of memory.

It's possible that with 4GB of DRAM you're reaching the limit of system resources for the type of deployment you have (how big is the BGP table?).

Thanks,

Michal

[Anthony O'Reilly · Accepted solution]

Hi,

I logged a case with Cisco TAC. From the sh tech logs, they were able to determined and recommemd the following:

Basically this bug CSCvd01631 is preventing the proper creation of kernel core files whenever something fails with it and forcing a timeout as you mentioned, which as a consequence is preventing us to know what is actually making the router to crash.

We need to upgrade the software of this router to 16.7(1) or greater, to have this code fix that will let us know what the kernel is doing to constantly crash the box.

So far the router has been up and stable for 6 days now.

Thanks for the replies and your help.