cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Bookmark
|
Subscribe
|
3615
Views
1
Helpful
10
Replies

ISR 4k Memory Behavior

shortcut144
Level 1
Level 1
Started this new position in an environment that is all ISR 4ks. I've not troubleshot memory in the ISR platform before and don't have a ton of baseline behavior.
 
First I want to be sure I understand the basics. Correct me if I am wrong.
 
After reading documentation, memory can be looked at in two areas. You IOSd which is the daemon that we typically interact with and then you have IOS-XE which is the underlying Linux OS that hosts the Daemon.
 
IOSd consumes half the memory of IOS-XE and reserves this for routing operations.
 
We've got 30+ of these things and I am seeing longs in quite a few like the ones below.
 
227431: Feb 10 08:01:32.630: %PLATFORM-4-ELEMENT_WARNING: SIP1: smand: RP/0: Used Memory value 89% exceeds warning level 88%
 
Ok.  RP/0 is the memory of IOS-XE.  But it looks like IOSd has good memory room.

ROUTER# show processes memory
Processor Pool Total: 1839083368 Used: 572480328 Free: 1266603040
lsmpi_io Pool Total: 6295128 Used: 6294296 Free: 832
 
This is a 4331, so pretty small.  Is consuming that amount of memory normal on these smaller routers?  Users aren't currently impacted but my logs have me concerned.
1 Accepted Solution
10 Replies 10

balaji.bandi
Hall of Fame
Hall of Fame

troubleshoot what is consuming to identify the issue :

 

https://www.cisco.com/c/en/us/td/docs/routers/access/4400/troubleshooting/memorytroubleshooting/isr4000_mem.html

 

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

Leo Laohoo
Hall of Fame
Hall of Fame

Please provide the complete output to the following commands: 

sh version
sh platform resources sh platform soft status con brief

Appreciate the look.

 

ROUTER#show version
Cisco IOS XE Software, Version 16.06.05
Cisco IOS Software [Everest], ISR Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.6.5, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2018 by Cisco Systems, Inc.
Compiled Mon 10-Dec-18 13:10 by mcpre


Cisco IOS-XE software, Copyright (c) 2005-2018 by cisco Systems, Inc.
All rights reserved.  Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0.  The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY.  You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0.  For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.


ROM: IOS-XE ROMMON

ROUTER uptime is 27 weeks, 4 days, 43 minutes
Uptime for this control processor is 27 weeks, 4 days, 46 minutes
System returned to ROM by PowerOn at 18:21:23 CST Thu Feb 28 2019
System restarted at 08:42:06 CDT Mon Aug 2 2021
System image file is "bootflash:isr4300-universalk9.16.06.05.SPA.bin"
Last reload reason: PowerOn



This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.



Suite License Information for Module:'esg'

--------------------------------------------------------------------------------
Suite                 Suite Current         Type           Suite Next reboot
--------------------------------------------------------------------------------
FoundationSuiteK9     None                  None           None
securityk9
appxk9

AdvUCSuiteK9          None                  None           None
uck9
cme-srst
cube


Technology Package License Information:

-----------------------------------------------------------------
Technology    Technology-package           Technology-package
              Current       Type           Next reboot
------------------------------------------------------------------
appxk9           None             None             None
uck9             uck9             Permanent        uck9
securityk9       securityk9       Permanent        securityk9
ipbase           ipbasek9         Permanent        ipbasek9

cisco ISR4331/K9 (1RU) processor with 1795979K/6147K bytes of memory.
Processor board ID FLM2032W1T5
3 Gigabit Ethernet interfaces
4 Voice FXO interfaces
32768K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
3223551K bytes of flash memory at bootflash:.
0K bytes of WebUI ODM Files at webui:.

Configuration register is 0x2102


ROUTER#show platform resources
**State Acronym: H - Healthy, W - Warning, C - Critical
Resource                 Usage                 Max             Warning         Critical        State
----------------------------------------------------------------------------------------------------
RP0 (ok, active)                                                                               H
 Control Processor       1.30%                 100%            90%             95%             H
  DRAM                   3391MB(89%)           3790MB          90%             95%             H
ESP0(ok, active)                                                                               H
 QFP                                                                                           H
  DRAM                   141485KB(53%)         262144KB        80%             90%             H
  IRAM                   426KB(20%)            2048KB          80%             90%             H
  CPU Utilization        1.00%                 100%            90%             95%             H



ROUTER#show platform software status control-processor brief
Load Average
 Slot  Status  1-Min  5-Min 15-Min
  RP0 Healthy   1.66   1.57   1.59

Memory (kB)
 Slot  Status    Total     Used (Pct)     Free (Pct) Committed (Pct)
  RP0 Warning  3881124  3473536 (89%)   407588 (11%)   3053328 (79%)

CPU Utilization
 Slot  CPU   User System   Nice   Idle    IRQ   SIRQ IOwait
  RP0    0   0.80   1.00   0.00  98.20   0.00   0.00   0.00
         1   2.40   1.20   0.00  96.40   0.00   0.00   0.00
         2   3.80   0.90   0.00  95.30   0.00   0.00   0.00
         3   1.60   0.80   0.00  97.60   0.00   0.00   0.00
         4   1.50   1.50   0.00  97.00   0.00   0.00   0.00
         5  56.45   5.10   0.00  38.43   0.00   0.00   0.00
         6  42.30  33.10   0.00  24.60   0.00   0.00   0.00
         7   0.00   0.00   0.00 100.00   0.00   0.00   0.00

ROUTER#


@shortcut144 wrote:
Memory (kB)
 Slot  Status    Total     Used (Pct)     Free (Pct) Committed (Pct)
  RP0 Warning  3881124  3473536 (89%)   407588 (11%)   3053328 (79%)

27 weeks uptime and the control plane is exhibiting a memory leak.  Not good. 

I need the "first" page of the output to the following command: 

sh process memory platform sorted location r0

Remember, do not provide the "entire" list.  I just need to see the first "page" of the output.  

 


@shortcut144 wrote:
         5  56.45   5.10   0.00  38.43   0.00   0.00   0.00
         6  42.30  33.10   0.00  24.60   0.00   0.00   0.00

Hmmmm ... Can you also monitor that the CPU is not "hot spinning"?  Re-run the same command to see if the two CPUs are not "stuck".

Code is pretty old but yea.  Thought it might be sort of odd to have such high memory.  2018 was modern at one time.

 

ROUTER#sh process memory platform sorted location r0
System memory: 3881124K total, 3473628K used, 407496K free,
Lowest: 407496K
   Pid    Text      Data   Stack   Dynamic       RSS     Total              Name
--------------------------------------------------------------------------------
  4179  251718   1159076     136       124   1159076   3427852   linux_iosd-imag
 12099   11894    254660     132     28984    254660   2254644     fman_fp_image
  8049      89    259124     132     23516    259124   1773924        cpp_cp_svr
 30273    7095    115144     136      2264    115144   1276588           fman_rp
  9863   18194    677640     136       428    677640   1163728    qfp-ucode-utah
  5660      68    127572     132      3568    127572   1134072        cpp_sp_svr
  3962     151    121616     132      2936    121616   1120916        cpp_driver
  4988     363    104820     136      3668    104820   1110732   cpp_ha_top_leve
 26284     230     93324     136      3548     93324   1037008               bsm
  2348     231     43192     132      3764     43192    849000           cman_fp
 18179     472     61368     136      4252     61368    768348              iomd
 19140     472     63236     136      4180     63236    768276              iomd
 27780     202     81828     136      7124     81828    734296         cli_agent
  6174     151     78724     136       268     78724    630776               smd
 28238      97     83356     136      3556     83356    619468               dbm
  4759     460     59324     136     24556     59324    489016             smand
 27640    1080     46244     136      4356     46244    405696             cmand
  9752     624     28968     136      1672     28968    347320              cmcc
 10933     624     27404     136      1416     27404    347064              cmcc
  8850     498     10292     132       132     10292    270960          libvirtd
  1201      36     19272     136      1424     19272    264968               psd
 29909     326     20796     136      2156     20796    251520              hman
 31245     113     18440     136      1664     18440    247500              lman
 11486     326     16300     136      1144     16300    240000              hman
 10417     326     16472     136      1144     16472    240000              hman
  7378     326     16604     132      1144     16604    239996              hman
   703      39     15032     136      2160     15032    199856             plogd
 14603      39     14464     136      1668     14464    199364             plogd
 14075      39     14468     136      1668     14468    199364             plogd
  8592      39     14716     132      1668     14716    199360             plogd
 18288      52     10580     136       596     10580    195772          ngiolite
 11218      26     10732     136      1464     10732    176396         tams_proc
 10449      72     10396     136       836     10396    171552         tamd_proc
  9264      16      9252     136       688      9252    164968   tam_svcs_esg_cf
 26215       9      3316     136       260      3316     63756             rotee
 32252       9      3300     132       260      3300     63752             rotee
 24965       9      3468     132       260      3468     63752             rotee
 24833       9      3312     132       260      3312     63752             rotee
  5504       9      3324     132       260      3324     63752             rotee
  4452       9      3344     132       260      3344     63752             rotee
 32506       9      3288     136       260      3288     63636             rotee
 32118       9      3312     136       260      3312     63636             rotee
 31558       9      3316     136       260      3316     63636             rotee
 30753       9      3284     136       260      3284     63636             rotee
 --More--

 

I did submit a TAC Case after posting yesterday and they pointed me to this suggesting an upgrade.

 

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuc40262

This guide is great!  Thank you very much for your help.

johnlloyd_13
Level 9
Level 9

hi,

it might be due to a bug.

if your 4331 has smartnet, best is to open a TAC case.

TAC should be able to check their database for the bug fix or recommend OS upgrade.

marce1000
Hall of Fame
Hall of Fame

 

 - FYI : https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvq15179   , check software version preferably use an advisory release , if applicable : https://software.cisco.com/download/home/285018115/type/282046477/release/Amsterdam-17.3.4a , other bugs could be involved too https://bst.cloudapps.cisco.com/bugsearch?pf=prdNm&kw=PLATFORM-4-ELEMENT_WARNING&bt=custV&sb=anfr

 M.



-- Each morning when I wake up and look into the mirror I always say ' Why am I so brilliant ? '
    When the mirror will then always repond to me with ' The only thing that exceeds your brilliance is your beauty! '