05-15-2022 12:46 AM
Hello everyone,
we have a bunch of ISR 4431 running universalk9.03.16.04b.S.155-3.S4b-ext with no load at all. We have a big problem with memory utilization (As always, CPU and Memory management is one the biggest problems with Cisco softwares).
Here is the output of different commands. That would be great if you could give us a hand:
Router#show processes memory sorted Processor Pool Total: 1688360784 Used: 264057808 Free: 1424302976 lsmpi_io Pool Total: 6295128 Used: 6294296 Free: 832 PID TTY Allocated Freed Holding Getbufs Retbufs Process 0 0 353660936 108857696 227620776 0 0 *Init* 79 0 533775128 302040 7594720 0 0 IOSD ipc task 389 0 3889216 5160 3926048 799092 0 EEM ED Syslog 0 0 847546560 844385288 3117360 8177103 0 *Dead* 409 0 1438984 26792 1442056 0 0 EEM Server 155 0 3218992 91360 1053512 0 0 CWAN OIR Handler 185 0 1036624 66816 1023520 0 0 IP ARP Adjacency 1 0 789776 0 819640 0 0 Chunk Manager 6 0 522608 21808 461360 0 0 RF Slave Main Th 390 0 376128 5160 412832 67468 0 EEM ED Generic 0 0 0 0 395592 0 0 *MallocLite* 162 0 2397163680 1496916312 381456 0 0 CDP Protocol 189 0 614472 339384 250592 0 0 mDNS 277 0 1808 0 247672 0 0 COPS 508 0 167720 448 221136 0 0 MRIB Process 10 0 19150047736 19150089016 215080 18311499785 18311529513 Pool Manager 15 0 20955311736 20955132000 195336 0 0 ARP Input 374 0 1349008 1199016 194168 0 0 Crypto CA 334 0 153800 448 183216 0 0 RUDPV1 Main Proc 271 0 130168 3216 180968 0 0 IP RIB Update
What is the problem with this devices? Holding memory? Or maybe Getbufs/ Retbufs?
Another command:
Router#show platform software status control-processor brief Load Average Slot Status 1-Min 5-Min 15-Min RP0 Healthy 0.98 0.58 0.25 Memory (kB) Slot Status Total Used (Pct) Free (Pct) Committed (Pct) RP0 Healthy 3965680 3678204 (93%) 287476 ( 7%) 1739072 (44%) CPU Utilization Slot CPU User System Nice Idle IRQ SIRQ IOwait RP0 0 1.70 0.80 0.00 97.49 0.00 0.00 0.00 1 2.00 3.00 0.00 95.00 0.00 0.00 0.00 2 0.70 9.20 0.00 90.10 0.00 0.00 0.00 3 1.10 0.50 0.00 98.40 0.00 0.00 0.00 4 0.10 0.50 0.00 99.40 0.00 0.00 0.00 5 0.30 0.20 0.00 99.50 0.00 0.00 0.00 6 1.20 0.40 0.00 98.40 0.00 0.00 0.00 7 0.30 0.30 0.00 99.39 0.00 0.00 0.00
These routers are truley important so we are not allowes to upgrade them to any tricky/buggy version. We are also on RTU license and switching to Cisco One is another challenge. So, if you recommend an upgrade please think about the consideration.
***If someone from Cisco read this topic:
PLEASE PLEASE PLEASE
simplify licensing and donwloading/ upgrading your products. Out troubles are getting bigger everyday and those centralized mindsets are not compatible with complex scenarios. We purchase a product, so then we need the upgrades as simple as possible. Why you need different kinds of accounts and contracts? It's just an obstacle
05-15-2022 01:54 AM
@salemmahara wrote:
Memory (kB) Slot Status Total Used (Pct) Free (Pct) Committed (Pct) RP0 Healthy 3965680 3678204 (93%) 287476 ( 7%) 1739072 (44%)
F*ck that is high.
Please post the complete output to the command "sh process memory platform sort location r0".
NOTE: I only want to see the "first page".
05-15-2022 02:51 AM
Hello @Leo Laohoo
Here is the output:
Router#sho processes memory platform sorted location r0 System memory: 3965680K total, 1739400K used, 2226280K free Lowest: 286320K Pid Text Data Stack Dynamic RSS Total Name -------------------------------------------------------------------------------- 3149 11416 176004 204 20716 8698952 10370144 fman_fp_image 2255 78 168268 132 19892 8687984 9912076 cpp_cp_svr 2897 62 113820 84 2272 8633536 9322480 cpp_sp_svr 2695 323 93876 84 3208 8613592 9294812 cpp_ha_top_leve 2421 144 91428 148 2520 8611148 9281584 cpp_driver 2000 59 23692 84 988 8543408 9202552 cpp_cdm_svr 26602 382300 742664 216 56 742908 2445128 linux_iosd-imag 25317 6416 81912 184 1688 81912 1058628 fman_rp 1602 101 19816 84 3276 19828 508856 cman_fp 23676 247 76312 84 4956 84764 371368 bsm 31325 749 16604 84 876 16604 179608 vman 29476 385 118456 100 107968 118464 127032 smand 7727 29 9888 84 280 75444 79808 ngiolite 7614 358 27676 84 3512 27688 77832 iomd 24598 956 32920 84 3680 32940 53272 cmand 26413 303 30984 84 1084 30984 41564 hman 5062 303 29200 84 844 29200 39224 hman 3311 303 29200 84 844 29200 39224 hman 4718 570 11312 84 948 11312 29648 cmcc 10526 101 1316 420 132 1316 22424 automount 24021 892 1056 84 0 1056 21832 libvirtd 27771 85 808 84 132 808 18224 pcscd 28529 29 6364 84 676 6364 17396 psd 27504 103 6152 84 1232 6160 16212 lman 28347 33 5596 84 1572 5604 14516 plogd 30884 18 4108 84 1160 4108 10152 tams_proc 30745 12 3960 84 676 3964 9548 tamd_proc 30228 11 3180 84 536 3180 9308 tam_svcs_esg_cf 28034 635 4712 84 3528 4712 5972 periodic.sh 30983 4 1656 84 132 1656 4812 rotee 30398 4 1656 84 132 1656 4812 rotee 30113 4 1656 84 132 1656 4812 rotee
05-15-2022 03:30 AM - edited 05-15-2022 03:33 AM
@salemmahara wrote:
Pid Text Data Stack Dynamic RSS Total Name -------------------------------------------------------------------------------- 3149 11416 176004 204 20716 8698952 10370144 fman_fp_image 2255 78 168268 132 19892 8687984 9912076 cpp_cp_svr 2897 62 113820 84 2272 8633536 9322480 cpp_sp_svr 2695 323 93876 84 3208 8613592 9294812 cpp_ha_top_leve 2421 144 91428 148 2520 8611148 9281584 cpp_driver 2000 59 23692 84 988 8543408 9202552 cpp_cdm_svr
The items in RED are the causes of the memory leak. You have hit multiple memory leak bugs and I cannot begin to enumerate what might cause them to be that high.
The most common causes of these are:
If you can upgrade the firmware of the router, upgrade to the latest 3.16.X train, 3.16.10.
05-19-2022 11:52 AM
Hello Leo,
these routers are running HSRP on both sides plus some static routing and 20-30 lines ACL. A set of simple NATs.
05-19-2022 12:01 PM
Is 3.16.10s safe to upgrade while I’m using my RTU license.
05-19-2022 03:54 PM
@salemmahara wrote:
Is 3.16.10s safe to upgrade while I’m using my RTU license.
3.X.X does not support Cisco Smart License.
05-15-2022 03:18 AM
Are this routet connect to isp via bgp?
05-19-2022 11:26 AM
Hello,
Not at all
05-19-2022 12:19 PM
05-20-2024 01:53 PM
Old thread i know.
But i'm having that problem. Connecting to an ISP caused the issue for me.
05-15-2022 04:09 AM
memory issue please find attachment cisco doc.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide