cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1925
Views
5
Helpful
11
Replies

c9400 memory utilization

jcombalicer
Level 1
Level 1

Hello,

 

Anyone had encountered this error message on 9410 with SUP-1XL-Y? switch is running with VSL mode.

%PLATFORM-3-ELEMENT_CRITICAL: Chassis 1 R0/0: smand: 2/RP/0: Used Memory value 97% exceeds critical

level 95%

 Thanks in advance.

11 Replies 11

Leo Laohoo
Hall of Fame
Hall of Fame

Post the complete output to the following commands: 

  1. sh version
  2. sh platform software status control-processor brief
  3. sh processes memory platform sorted location switch active r0

Hi Leo,

Below are the results

CORE#sh version
Cisco IOS XE Software, Version 17.06.01
Cisco IOS Software [Bengaluru], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 17.6.1, RELEASE SOFTWARE (fc6)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2021 by Cisco Systems, Inc.
Compiled Fri 30-Jul-21 15:19 by mcpre


Cisco IOS-XE software, Copyright (c) 2005-2021 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.


ROM: IOS-XE ROMMON
BOOTLDR: System Bootstrap, Version 16.12.2r, RELEASE SOFTWARE (P)

CORE uptime is 42 weeks, 3 days, 23 hours, 40 minutes
Uptime for this control processor is 42 weeks, 3 days, 23 hours, 48 minutes
System returned to ROM by Image Install at 16:20:37 Manila Thu Dec 2 2021
System restarted at 12:40:27 Manila Fri Dec 3 2021
System image file is "bootflash:packages.conf"
Last reload reason: Image Install

 

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.


Technology Package License Information:

------------------------------------------------------------------------------
Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------------------
network-advantage Smart License network-advantage
dna-advantage Subscription Smart License dna-advantage
AIR License Level: AIR DNA Advantage
Next reload AIR license Level: AIR DNA Advantage


Smart Licensing Status: Registration Not Applicable/Not Applicable

cisco C9410R (X86) processor (revision V01) with 1841914K/6147K bytes of memory.
Processor board ID FXS2510Q23N
92 Virtual Ethernet interfaces
208 Ten Gigabit Ethernet interfaces
4 TwentyFive Gigabit Ethernet interfaces
4 Forty Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
15995824K bytes of physical memory.
11161600K bytes of Bootflash at bootflash:.
1638400K bytes of Crash Files at crashinfo:.
11161600K bytes of Bootflash at bootflash-2-0:.
1638400K bytes of Crash Files at crashinfo-2-0:.

Base Ethernet MAC Address : 44:b6:be:1f:01:c0
Motherboard Assembly Number : 18BFB
Motherboard Serial Number : FXS250900P2
Model Revision Number : V02
Motherboard Revision Number : 5
Model Number : C9410R
System Serial Number : FXS2510Q23N

Switch 02
---------
Base Ethernet MAC Address : 44:b6:be:1f:03:80
Motherboard Assembly Number : 18BFB
Motherboard Serial Number : FXS250900NA
Model Revision Number : V02
Motherboard Revision Number : 5
Model Number : C9410R
System Serial Number : FXS2510Q23M

CORE#sh platform software status control-processor brief
Load Average
Slot Status 1-Min 5-Min 15-Min
1-RP0 Healthy 0.66 1.02 0.92
2-RP0 Healthy 0.65 0.55 0.55

Memory (kB)
Slot Status Total Used (Pct) Free (Pct) Committed (Pct)
1-RP0 Healthy 15995824 10423008 (65%) 5572816 (35%) 10505396 (66%)
2-RP0 Critical 15995824 15696172 (98%) 299652 ( 2%) 11831952 (74%)

CPU Utilization
Slot CPU User System Nice Idle IRQ SIRQ IOwait
1-RP0 0 5.60 1.50 0.00 92.80 0.00 0.10 0.00
1 5.30 1.00 0.00 93.70 0.00 0.00 0.00
2 5.29 1.29 0.00 93.40 0.00 0.00 0.00
3 5.09 0.99 0.00 93.90 0.00 0.00 0.00
4 5.30 1.30 0.00 93.30 0.00 0.10 0.00
5 6.00 1.40 0.00 92.50 0.00 0.10 0.00
6 4.99 1.19 0.00 93.70 0.00 0.09 0.00
7 4.09 0.99 0.00 94.80 0.00 0.09 0.00
2-RP0 0 3.50 1.00 0.00 95.50 0.00 0.00 0.00
1 4.50 0.90 0.00 94.59 0.00 0.00 0.00
2 3.40 1.00 0.00 95.59 0.00 0.00 0.00
3 3.50 1.00 0.00 95.50 0.00 0.00 0.00
4 5.90 1.30 0.00 92.79 0.00 0.00 0.00
5 5.40 1.00 0.00 93.60 0.00 0.00 0.00
6 3.90 1.10 0.00 94.89 0.00 0.10 0.00
7 3.50 0.90 0.00 95.59 0.00 0.00 0.00

CORE#sh processes memory platform sorted location switch active r0
System memory: 15995824K total, 10424816K used, 5571008K free,
Lowest: 5560756K
Pid Text Data Stack Dynamic RSS Name
----------------------------------------------------------------------
8055 262 4517352 136 4490784 4517352 nif_mgr
5854 256396 1321880 136 488 1321880 linux_iosd-imag
22216 200 702876 136 123840 702876 fed main event
655 7234 420260 136 4004 420260 fman_rp
13383 8149 236800 136 4688 236800 fman_fp_image
4272 448 234892 136 4880 234892 dbm
18991 1126 204304 668 50824 204304 iomd
19586 1126 204220 660 50824 204220 iomd
20298 1126 203468 660 50824 203468 iomd
21259 1126 202468 664 50824 202468 iomd
13686 87 192124 136 4028 192124 pubd
18517 788 180868 136 38516 180868 smand
16056 178 173460 136 12940 173460 sessmgrd
17169 1126 150876 660 2696 150876 iomd
23733 447 131484 136 2732 131484 cman_fp
6276 56 124264 136 108 124264 cli_agent
23185 382 106764 136 3396 106764 repm
8294 78 94336 136 132908 94336 plogd
6411 1956 54236 136 8584 54236 cmand
13234 8 45468 136 6808 45468 python3
25711 228 43324 136 23444 43324 tms
8835 916 42716 136 3804 42716 cmcc
5932 916 42692 136 3716 42692 cmcc
3279 916 42280 136 3712 42280 cmcc
32316 916 41956 136 3832 41956 cmcc
12832 916 38076 136 2504 38076 cmcc
20735 916 37280 136 2104 37280 cmcc
30179 916 37128 136 2084 37128 cmcc
5552 470 37120 136 2408 37120 stack_mgr
27842 916 37092 136 2076 37092 cmcc
24280 916 36980 136 2084 36980 cmcc
18510 827 36800 136 1808 36800 install_mgr
16618 916 36552 136 1972 36552 cmcc
13864 70 35780 136 1696 35780 psd
30821 477 29408 488 3460 29408 hman
6159 471 28084 132 1800 28084 libvirtd
7827 153 25436 136 2056 25436 lman
21068 47 24896 136 1552 24896 service_mgr
11276 47 24748 136 1376 24748 service_mgr
8373 209 24436 136 2068 24436 btman
23176 477 23476 136 2188 23476 hman
27211 477 22388 136 1752 22388 hman
15897 477 22384 136 1752 22384 hman
31023 209 22344 136 2468 22344 btman
12284 477 22320 136 1752 22320 hman
14313 209 22264 136 1808 22264 btman
22369 209 22232 136 1796 22232 btman
32098 477 22228 136 1752 22228 hman
5287 477 22228 136 1752 22228 hman
2801 477 22228 136 1752 22228 hman
29661 477 22168 136 1748 22168 hman
8122 477 22132 136 1752 22132 hman
20094 477 22000 136 1752 22000 hman
24308 209 21964 136 1712 21964 btman
23575 477 21948 136 1752 21948 hman
18238 209 21872 136 1780 21872 btman
28957 209 21708 136 1792 21708 btman
7863 209 21700 136 1816 21700 btman
10282 209 21696 136 1812 21696 btman
4791 209 21672 136 1812 21672 btman
1907 209 21640 136 1816 21640 btman
25991 209 21620 136 1784 21620 btman
7537 116 20056 136 1516 20056 keyman
11867 83 15220 136 1148 15220 epc_ws_liaison
6585 59 13652 136 1120 13652 tamd_proc
21521 1653 13420 136 2892 13420 nginx
21531 1653 13232 136 2892 13232 nginx
6867 62 13132 136 1656 13132 tams_proc
1 917 12388 1036 3720 12388 systemd
6081 84 12072 132 396 12072 virtlogd
17434 747 11828 376 8936 11828 ncd.sh
7239 51 11688 136 808 11688 tam_svcs_esg_cf
17974 747 11624 376 8672 11624 issu_stack.sh
30715 747 10684 376 8540 10684 issu_stack.sh
30673 747 10620 376 8540 10620 issu_stack.sh
13773 57 9908 136 396 9908 pttcd
19811 747 9024 376 6132 9024 periodic.sh
5858 747 8556 380 5344 8556 pvp.sh
30374 747 8452 372 5240 8452 pvp.sh
31401 747 8400 380 5344 8400 pvp.sh
32739 747 8396 372 5344 8396 pvp.sh
2794 747 8228 376 5344 8228 pvp.sh
8827 747 7584 372 4084 7584 reflector.sh
194 107 7552 132 4600 7552 systemd-journal
15366 747 7488 372 4056 7488 chasync.sh
4774 747 7476 380 4544 7476 pvp.sh
4559 747 7368 372 4488 7368 psvp.sh
22614 747 7124 372 4036 7124 pvp.sh
19384 747 7068 372 3888 7068 pvp.sh
6066 747 7068 372 3620 7068 droputil.sh
21116 747 6980 372 4012 6980 pvp.sh
219 470 6976 132 1320 6976 systemd-udevd
16099 747 6884 376 3888 6884 pvp.sh
9304 747 6872 372 4020 6872 pvp.sh
12857 747 6848 376 4020 6848 pvp.sh
21532 1653 6776 136 2892 6776 nginx
3987 747 5952 380 2824 5952 pman
7611 747 5948 372 2836 5948 pman
17083 747 5944 380 2836 5944 pman
20931 747 5908 380 2712 5908 pman
19306 747 5896 372 2840 5896 pman
15282 747 5892 380 2828 5892 pman
13066 747 5892 376 2712 5892 pman
21409 747 5888 380 2836 5888 pman
2238 747 5884 376 2828 5884 pman
12854 747 5876 380 2828 5876 pman
15888 747 5872 380 2848 5872 pman
20613 747 5868 380 2840 5868 pman
31581 747 5860 380 2712 5860 pman
30688 747 5860 380 2840 5860 pman
3696 747 5860 380 2840 5860 pman
11281 747 5856 376 2840 5856 pman
18691 747 5852 376 2824 5852 pman
29239 747 5848 380 2828 5848 pman
19144 747 5848 380 2840 5848 pman
5928 747 5844 380 2836 5844 pman
20063 747 5824 376 2824 5824 pman
8135 747 5824 372 2836 5824 pman
23033 747 5816 376 2828 5816 pman
19352 747 5816 376 2824 5816 pman
13936 747 5804 372 2836 5804 pman
8671 747 5784 376 2828 5784 pman
18050 747 5764 380 2840 5764 pman
6638 747 5764 376 2712 5764 pman
7109 747 5748 380 2836 5748 pman
23281 747 5740 380 2840 5740 pman
21366 747 5732 376 2828 5732 pman
1810 747 5732 380 2840 5732 pman
5077 747 5728 376 2836 5728 pman
32659 747 5724 380 2828 5724 pman
5659 747 5724 376 2836 5724 pman
27807 747 5720 380 2712 5720 pman
31818 747 5716 380 2828 5716 pman
13573 747 5716 376 2836 5716 pman
22149 747 5712 380 2824 5712 pman
4499 747 5712 376 2828 5712 pman
28380 747 5708 380 2840 5708 pman
18349 747 5700 380 2840 5700 pman
13404 747 5700 380 2836 5700 pman
10736 747 5700 380 2832 5700 pman
9187 747 5700 380 2828 5700 pman
14490 747 5696 376 2824 5696 pman
3611 747 5696 376 2824 5696 pman
1464 747 5692 376 2824 5692 pman
26700 747 5688 376 2828 5688 pman
18352 747 5684 376 2840 5684 pman
7867 747 5684 376 2836 5684 pman
30021 747 5680 376 2832 5680 pman
17599 747 5680 380 2836 5680 pman
6767 747 5680 380 2836 5680 pman
16815 747 5676 376 2824 5676 pman
16785 747 5676 380 2712 5676 pman
6852 747 5676 376 2824 5676 pman
22521 747 5672 376 2828 5672 pman
10934 747 5672 376 2824 5672 pman
7374 747 5672 376 2836 5672 pman
22092 747 5668 380 2828 5668 pman
19454 747 5668 372 2828 5668 pman
28594 747 5664 376 2824 5664 pman
24577 747 5664 380 2712 5664 pman
20797 747 5660 372 2824 5660 pman
5335 747 5656 376 2840 5656 pman
23241 747 5648 380 2828 5648 pman
11678 747 5640 376 2828 5640 pman
9097 747 5632 380 2712 5632 pman
18760 747 5628 376 2824 5628 pman
7413 747 5628 372 2828 5628 pman
6349 747 5616 372 2836 5616 pman
25790 747 5612 372 2824 5612 pman
13847 46 5584 136 188 5584 journalctl
15886 747 5484 372 3584 5484 chasync.sh
9354 747 5412 376 2080 5412 iptbl.sh
6123 103 4380 132 1588 4380 rpc.mountd
15377 42 4360 132 136 4360 flash-rec
18734 747 4076 136 1536 4076 flash_check.sh
813 168 3928 132 264 3928 dbus-daemon
22466 9 3864 136 268 3864 rotee
876 9 3840 132 268 3840 rotee
18182 9 3832 136 268 3832 rotee
15652 9 3828 132 268 3828 rotee
21452 9 3816 136 268 3816 rotee
6566 9 3816 136 268 3816 rotee
23444 9 3808 136 268 3808 rotee
18504 9 3808 136 268 3808 rotee
31692 9 3804 132 268 3804 rotee
31393 9 3804 136 268 3804 rotee
16821 9 3804 136 268 3804 rotee
13649 9 3804 136 268 3804 rotee
30303 9 3800 136 268 3800 rotee
23562 9 3800 136 268 3800 rotee
14508 9 3800 136 268 3800 rotee
7752 9 3800 136 268 3800 rotee
31992 9 3796 136 268 3796 rotee
28160 9 3796 136 268 3796 rotee
6978 9 3796 136 268 3796 rotee
6074 747 3796 132 788 3796 oom.sh
3380 9 3796 132 268 3796 rotee
6325 9 3792 132 268 3792 rotee
18847 9 3788 136 268 3788 rotee
8891 9 3784 132 268 3784 rotee
8279 9 3784 136 268 3784 rotee
4296 9 3784 136 268 3784 rotee
24986 9 3776 136 268 3776 rotee
29549 9 3768 136 268 3768 rotee
1960 9 3768 136 268 3768 rotee
24091 9 3764 136 268 3764 rotee
31733 9 3756 136 268 3756 rotee
20487 9 3756 132 268 3756 rotee
17189 9 3756 136 268 3756 rotee
17035 9 3752 132 268 3752 rotee
10106 9 3752 132 268 3752 rotee
20160 9 3744 136 268 3744 rotee
19247 9 3744 136 268 3744 rotee
29255 9 3740 136 268 3740 rotee
7179 9 3740 136 268 3740 rotee
5849 9 3740 136 268 3740 rotee
4985 9 3740 136 268 3740 rotee
4764 9 3740 132 268 3740 rotee
22877 9 3736 136 268 3736 rotee
17244 9 3736 136 268 3736 rotee
9472 9 3736 136 268 3736 rotee
20172 9 3732 136 268 3732 rotee
14961 9 3732 136 268 3732 rotee
17008 9 3728 136 268 3728 rotee
12984 9 3728 136 268 3728 rotee
9538 9 3724 136 268 3724 rotee
996 9 3724 136 268 3724 rotee
23639 9 3720 132 268 3720 rotee
22478 9 3720 136 268 3720 rotee
9646 9 3720 132 268 3720 rotee
8466 9 3720 132 268 3720 rotee
19450 9 3716 136 268 3716 rotee
13415 9 3716 136 268 3716 rotee
30483 9 3712 132 268 3712 rotee
13721 9 3712 132 268 3712 rotee
6630 9 3712 132 268 3712 rotee
22579 9 3708 136 268 3708 rotee
22389 9 3708 132 268 3708 rotee
17719 9 3708 136 268 3708 rotee
3887 9 3708 136 268 3708 rotee
20940 9 3704 136 268 3704 rotee
13510 9 3704 136 268 3704 rotee
11623 9 3704 136 268 3704 rotee
19199 9 3700 136 268 3700 rotee
19159 9 3700 136 268 3700 rotee
10133 9 3700 136 268 3700 rotee
5519 9 3700 136 268 3700 rotee
11385 9 3696 136 268 3696 rotee
8010 9 3696 136 268 3696 rotee
2644 9 3696 136 268 3696 rotee
21187 9 3692 132 268 3692 rotee
19424 9 3692 136 268 3692 rotee
4839 9 3692 132 268 3692 rotee
21346 9 3688 136 268 3688 rotee
7541 9 3688 136 268 3688 rotee
6969 9 3688 136 268 3688 rotee
15744 9 3684 136 268 3684 rotee
12191 9 3684 136 268 3684 rotee
27115 9 3680 136 268 3680 rotee
21661 9 3680 132 268 3680 rotee
11996 9 3680 136 268 3680 rotee
26319 9 3676 136 268 3676 rotee
21587 9 3676 136 268 3676 rotee
19927 9 3676 136 268 3676 rotee
18874 9 3676 136 268 3676 rotee
7853 9 3676 136 268 3676 rotee
9502 9 3664 132 268 3664 rotee
4702 9 3660 136 268 3660 rotee
7271 9 3656 136 268 3656 rotee
6052 9 3656 136 268 3656 rotee
29013 9 3652 136 268 3652 rotee
2621 9 3648 136 268 3648 rotee
5225 9 3644 136 268 3644 rotee
15368 747 3252 132 132 3252 flash_recovery.
843 45 3176 132 132 3176 nfsdcld
9801 747 3160 132 132 3160 mcp_wdttickle.s
766 89 3004 132 132 3004 auditd
848 38 2876 132 132 2876 rpc.idmapd
12307 747 2728 136 132 2728 stack_sntp.sh
19528 747 2680 136 132 2680 sort_files_by_i
15379 122 2560 132 260 2560 xinetd
15386 122 2532 132 132 2532 xinetd
6133 75 2472 132 132 2472 rpc.statd
6790 565 2308 132 252 2308 sshd
12233 122 2248 132 132 2248 xinetd
6125 122 2228 132 132 2228 xinetd
777 31 2156 132 132 2156 audispd
6082 43 2028 132 132 2028 rpcbind
12339 45 2004 136 132 2004 sntp
8913 21 1804 132 132 1804 inotifywait
30710 21 1784 136 132 1784 inotifywait
30739 21 1716 136 132 1716 inotifywait
30686 21 1712 136 132 1712 inotifywait
15918 21 1704 136 132 1704 inotifywait
14510 21 1704 132 132 1704 inotifywait
30542 21 1692 132 132 1692 inotifywait
30668 21 1676 136 132 1676 inotifywait
6096 747 1668 132 132 1668 boothelper_evt.
7356 21 1664 132 132 1664 inotifywait
17900 21 1656 132 132 1656 inotifywait
4769 21 1656 132 132 1656 inotifywait
15856 21 1644 132 132 1644 inotifywait
9707 21 1632 132 132 1632 inotifywait
1423 21 1632 132 132 1632 inotifywait
8515 21 1616 132 132 1616 inotifywait
24397 21 1564 132 132 1564 inotifywait
23484 30 1564 136 132 1564 sleep
3799 21 1564 132 132 1564 inotifywait
32035 21 1560 132 132 1560 inotifywait
10903 21 1552 132 132 1552 inotifywait
21292 21 1548 132 132 1548 inotifywait
21248 21 1544 132 132 1544 inotifywait
21071 30 1488 136 132 1488 sleep
6111 21 1488 132 132 1488 inotifywait
24112 30 1480 132 132 1480 sleep
24109 30 1424 132 132 1424 sleep
23848 30 1396 136 132 1396 sleep
6454 30 1392 132 132 1392 sleep

 

 


@jcombalicer wrote:
8055 262 4517352 136 4490784 4517352 nif_mgr

Memory leak due to nif_mgr process. 

Upgrade the firmware to the latest 17.6.X soon or else the secondary line card will crash.

Hi Leo,

I need supporting report about this memory leak so we plan the downtime. The switch is already in production for a year and we cannot do the upgrade and reload the switch without supporting report about the memory leak issue on this version of image.

Thanks


@jcombalicer wrote:
I need supporting report about this memory leak so we plan the downtime.
  1. I do not work for Cisco -- If management wants an official report from Cisco (about the cause of the memory leak), raise a TAC Case. 
  2. Do hurry.  The "sh platform software status control-processor brief" output shows the redundant line card is already 98% memory utilized and will crash shortly.  If management wants to avoid a downtime, time is running out soon.  

marce1000
Hall of Fame
Hall of Fame

 

 - Upgrade to latest advisory release : https://software.cisco.com/download/home/286320394/type/282046477/release/Bengaluru-17.6.3 , if applicable , check if that can help.

 M.



-- Each morning when I wake up and look into the mirror I always say ' Why am I so brilliant ? '
    When the mirror will then always repond to me with ' The only thing that exceeds your brilliance is your beauty! '

I cant do the upgrade we need to schedule the downtime.

nif_mgr <<- this process for stack cable 
are you see any log about connect disconnect or any other log about stack status ?

Hi MHM,

No there is no log related to  connect disconnect or any log about stack.

Thanks

jcombalicer
Level 1
Level 1

Hi,

 

Rebooting the affected supervisor module fixed the issue. I'm monitoring the memory utilization before we proceed the upgrade.

I would be very careful using a software version that ends with a "1" (17.6.1).