- Cisco Community
- Technology and Support
- Networking
- Switching
- Nexus 9K high memory utilization
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-06-2022 07:41 AM
Hello everybody,
on our network we are using a pair of Cisco Nexus 9K (9300) in vPC mode; the switches are configured with some SVIs, VRRP enabled, and make some static routing.
These switches have been running for more than six year without a reboot, and starting from two months ago I noticed that the memory utilization in increasing, ranging around 90% of the total available memory.
I'm planning to reboot both switches (once authorized), and would like to know if there is any check I can do before the reboot to gather some information about why the ram started to fill.
Apart from this, can someone confirm that I should reboot the "master" unit first and then the other one, as we do when installing a software upgrade?
Thank you in advance!
Solved! Go to Solution.
- Labels:
-
LAN Switching
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-06-2022 08:08 AM
Hi,
I'm planning to reboot both switches (once authorized), and would like to know if there is any check I can do before the reboot to gather some information about why the ram started to fill.
Before you reboot, gather all the logs for reference. Also, if you have a support contract on the switch, I recommend opening a ticket with TAC and having them investigate further before rebooting.
Apart from this, can someone confirm that I should reboot the "master" unit first and then the other one, as we do when installing a software upgrade?
If you are upgrading, I would reboot the master first, and then the secondary. This way, once you reboot the secondary, the old master will be the primary switch again.
HTH
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-06-2022 08:08 AM
Hi,
I'm planning to reboot both switches (once authorized), and would like to know if there is any check I can do before the reboot to gather some information about why the ram started to fill.
Before you reboot, gather all the logs for reference. Also, if you have a support contract on the switch, I recommend opening a ticket with TAC and having them investigate further before rebooting.
Apart from this, can someone confirm that I should reboot the "master" unit first and then the other one, as we do when installing a software upgrade?
If you are upgrading, I would reboot the master first, and then the secondary. This way, once you reboot the secondary, the old master will be the primary switch again.
HTH
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-06-2022 09:11 AM
>.... if there is any check I can do before the reboot to gather some information about why the ram started to fill.
Connect to device with https://cway.cisco.com/cli , at the top left run or press 'System Diagnostics'
M.
-- Each morning when I wake up and look into the mirror I always say ' Why am I so brilliant ? '
When the mirror will then always repond to me with ' The only thing that exceeds your brilliance is your beauty! '
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-06-2022 12:21 PM
can you share this
NSK# show processes memory
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-06-2022 11:27 PM
Hi, this is the output of the "show processes memory". Firmware version is 7.0(3)I4(2).
PID MemAlloc MemLimit MemUsed StackBase/Ptr Process
----- -------- ---------- ---------- ----------------- ----------------
1 176128 0 4325376 7c0b4530/7c0b3b58 init
2 0 0 0 0/0 kthreadd
3 0 0 0 0/0 ksoftirqd/0
6 0 0 0 0/0 migration/0
7 0 0 0 0/0 watchdog/0
8 0 0 0 0/0 migration/1
10 0 0 0 0/0 ksoftirqd/1
12 0 0 0 0/0 watchdog/1
13 0 0 0 0/0 migration/2
15 0 0 0 0/0 ksoftirqd/2
16 0 0 0 0/0 watchdog/2
17 0 0 0 0/0 migration/3
19 0 0 0 0/0 ksoftirqd/3
20 0 0 0 0/0 watchdog/3
21 0 0 0 0/0 cpuset
22 0 0 0 0/0 khelper
23 0 0 0 0/0 kdevtmpfs
24 0 0 0 0/0 netns
25 0 0 0 0/0 sync_supers
26 0 0 0 0/0 bdi-default
27 0 0 0 0/0 kblockd
28 0 0 0 0/0 ata_sff
29 0 0 0 0/0 khubd
30 0 0 0 0/0 rpciod
39 0 0 0 0/0 kworker/u:1
51 0 0 0 0/0 khungtaskd
52 0 0 0 0/0 kswapd0
53 0 0 0 0/0 ksmd
54 0 0 0 0/0 fsnotify_mark
55 0 0 0 0/0 unionfs_siod
56 0 0 0 0/0 nfsiod
57 0 0 0 0/0 crypto
72 0 0 0 0/0 scsi_eh_0
73 0 0 0 0/0 scsi_eh_1
74 0 0 0 0/0 scsi_eh_2
75 0 0 0 0/0 scsi_eh_3
76 0 0 0 0/0 scsi_eh_4
78 0 0 0 0/0 kworker/u:3
79 0 0 0 0/0 cnic_wq
80 0 0 0 0/0 bnx2x
81 0 0 0 0/0 edac-poller
82 0 0 0 0/0 deferwq
138 0 0 0 0/0 loop0
646 0 0 0 0/0 loop1
648 0 0 0 0/0 loop2
2094 0 0 0 0/0 kworker/0:2
2255 0 0 0 0/0 jbd2/sda4-8
2256 0 0 0 0/0 ext4-dio-unwrit
2597 0 0 0 0/0 jbd2/sda5-8
2598 0 0 0 0/0 ext4-dio-unwrit
2603 0 0 0 0/0 jbd2/sda6-8
2604 0 0 0 0/0 ext4-dio-unwrit
2682 0 0 0 0/0 flush-8:0
4932 0 0 0 0/0 loop3
4968 0 0 0 0/0 jbd2/sda3-8
4969 0 0 0 0/0 ext4-dio-unwrit
4978 0 0 0 0/0 jbd2/sda2-8
4979 0 0 0 0/0 ext4-dio-unwrit
4992 0 0 0 0/0 jbd2/sda7-8
4993 0 0 0 0/0 ext4-dio-unwrit
5016 184320 0 8638464 7d7611e0/7d761000 portmap
5029 0 0 0 0/0 lockd
5030 0 0 0 0/0 nfsd
5032 323584 0 15368192 875e35d0/875e3288 rpc.mountd
5034 208896 0 15093760 ddbacbb0/ddbac948 rpc.statd
5214 0 0 0 0/0 kworker/3:0
5332 0 0 0 0/0 ppm
5400 0 0 0 0/0 loop4
5401 0 0 0 0/0 ext4-dio-unwrit
5647 167936 0 2269184 ffb50c00/ffb50ab0 mcelog
5898 94208 0 3166208 ffb5c1d0/ffb5b958 sh
5899 14209024 0 446779392 ffdfabb0/ffdfa910 sysmgr
5900 13127680 0 420933632 ffcd6780/ffcd64e0 sysmgr
5914 14118912 0 20221952 ffdaa9b0/ffdaa3b0 libvirtd
5920 0 0 0 0/0 kworker/3:1
6118 0 0 0 0/0 ppm
6258 0 0 0 0/0 mping-thread
6259 0 0 0 0/0 mping-thread
6291 0 0 0 0/0 cctrl_kthread
6339 0 0 0 0/0 redun_kthread
6360 0 0 0 0/0 usd_mts_kthread
6365 0 0 0 0/0 ls-notify-mts-t
6582 389120 735279001 2994176 ff8ed900/ff8ed750 xinetd
6583 434176 317279001 2740224 ffe96130/ffe95f40 tftpd
6584 2306048 640189510 336601088 ffa99bf0/ffa98f20 sdwrapd
6586 503808 0 280035328 ffdd3710/ffdd333c dme_proxy
6587 51171328 0 496615424 ff9abaa0/ff9aa9c0 platform
6590 167215104 0 758538240 ffe00820/ffe0075c event_manager
6591 141025280 0 650620928 ffaac7f0/ffaac72c policyelem
6595 454656 0 280031232 ffee6240/ffee6070 sdwrapd
6597 2154496 0 285880320 ffdabdd0/ffdaada0 pfmclnt
6628 10780672 0 420995072 ffee1cd0/ffee1b40 dme_asap_backend
6629 20910080 812320115 455356416 ffc54430/ffc4ff0c syslogd
6631 12181504 972782131 445616128 ffab2250/ffab1b78 vshd
6632 7892992 942540684 345169920 ffdcfc70/ffdcf6b0 smm
6633 2031616 667833689 293507072 ff90c240/ff90bf10 psshelper
6634 53411840 924999680 394579968 ffa2dad0/ffa2cec0 pixm_vl
6635 53796864 1045957651 490364928 ff9d5880/ff9d4c70 pixm_gl
6636 1007616 0 10735616 ff87e390/ff87e0e0 nxapi
6637 200749056 0 707039232 fff9daf0/fff9d860 nginx
6638 3170304 837235520 400834560 ff940640/ff93da8c mmode
6639 286720 0 3432448 ff9a5540/ff9a41b0 lmgrd
6640 1388544 0 337625088 fff284d0/fff27c50 fs-daemon
6641 14647296 0 448626688 ffdbadd0/ffdb9c8c feature-mgr
6642 1249280 0 335589376 fff1bc90/fff1ba50 confcheck
6643 3166208 623743577 337465344 fffdaeb0/fffda2d0 capability
6644 5251072 665594899 342671360 ffe0fa40/ffe0e21c bloggerd
6645 1773568 667675993 291012608 ff814650/ff814320 psshelper_gsvc
6653 33542144 1436260531 558047232 ffe7a550/ffe799dc clis
6654 1589248 666127756 339013632 ffb0a540/ffb0a078 licmgr
6663 307200 0 3461120 ff8c0b50/ff8c0990 cisco
6675 991232 0 336314368 ffa27250/ffa27080 xmlma
6676 1916928 636699008 337272832 ffa5a410/ffa591d0 vmm
6677 4780032 700537516 352092160 ffdf3810/ffdf3308 vman
6678 13340672 866243276 445472768 ffb5e7b0/ffb5dc60 vdc_mgr
6679 9949184 0 289624064 ffcf76a0/ffcf546c usbhsd
6683 1835008 680295193 339578880 ffaf6cd0/ffaf65a0 ttyd
6684 946176 604999084 280567808 ff8f3d20/ff8f3888 sysinfo
6685 1921024 669840371 338915328 ffe857f0/ffe847b0 snmpmib_proc
6686 1228800 639148716 338038784 ffdfd2e0/ffdfcc40 sksd
6688 1720320 622414425 338542592 ffc50070/ffc4fb9c res_mgr
6689 1318912 427279001 332603392 ffbf5b30/ffbf4cec pyproxy
6690 3620864 1987316531 292134912 ffbc5fc0/ffbc4f8c plugin
6691 548864 700675008 318537728 ffc48d30/ffc48890 plog_sup
6692 892928 1484169420 280805376 ffd25860/ffd252a8 patch-installer
6693 10022912 4294967295 759083008 ffaa9750/ffaa91a0 nbproxy
6694 1683456 783802272 336756736 ffd6d810/ffd6c590 mvsh
6695 512000 0 280899584 ffa0b380/ffa0af30 mping_server
6696 2772992 685810048 343531520 ffa6cc10/ffa6b880 module
6697 13205504 805347065 355770368 fffb0e50/fffb08d0 kim
6698 1941504 626239680 337960960 ffc79710/ffc786e0 evms
6700 9408512 775625804 347131904 fffcad10/fffcaac8 epld_upgrade_stdby
6701 4415488 668934745 339902464 ff8288f0/ff827880 diagmgr
6702 4177920 1198766784 752046080 ff91f700/ff91db7c dhclient
6703 12541952 1054536665 351719424 ffd70df0/ffd70550 crdcfg_server
6704 1003520 0 337055744 ff83d1f0/ff83cb90 core-dmon
6705 119828480 0 659779584 ff92c800/ff92c73c confelem
6706 2007040 669069913 339845120 ff855390/ff854360 clk_mgr
6707 638976 460279001 287543296 ffa4f1a0/ffa4e6c8 bios_daemon
6708 12419072 1476094963 533876736 ff9d4fb0/ff9d43a8 ascii-cfg
6709 13058048 0 446046208 ff931de0/ff930b18 securityd
6710 2297856 0 342016000 ff91d3b0/ff91b5fc cert_enroll
6711 12890112 0 446472192 ffc9c440/ffc9afe0 aaa
6715 1396736 714083673 340463616 ffa180e0/ffa17798 obfl
6716 21348352 1645618585 462168064 ffb237f0/ffb22400 aclmgr
6722 18247680 1790424268 650776576 ff835410/ff835030 urib
6724 1859584 888500518 337047552 ff8b9f40/ff8b8f10 evmc
6727 5943296 669538496 342913024 fffd6320/fffd4fd0 diagclient
6737 5001216 0 340529152 fff8d700/fff8c6a0 xbar
6738 17399808 768682675 354586624 ffc82a40/ffc81fcc device_test
6742 1544192 715124467 340545536 ffb5f4b0/ffb5e5a0 ExceptionLog
6743 11509760 807881254 445349888 ffa1a240/ffa19af0 bootvar
6744 1114112 0 334548992 ffedf490/ffedf030 cardclient
6745 34770944 802365414 473804800 ffc172d0/ffc16290 ifmgr
6746 25300992 1076438105 469577728 ffdba390/ffdb9f60 l3vm
6765 13570048 685384691 353710080 ffe7a310/ffe79e70 statsclient
6800 266240 0 12836864 78f33750/78f2b2d0 incrond
6851 8413184 949506342 354549760 ffeb5360/ffeb4dd0 npacl
6870 33746944 1224169356 770908160 ffcf4e80/ffcf4930 adjmgr
6871 21569536 1205192588 749711360 ffb571f0/ffb570b0 u6rib
6879 75345920 1415698188 726204416 ffafc230/ffafc0e0 arp
6881 24104960 1301720563 884776960 ff84d040/ff84cad0 icmpv6
6882 24584192 978306137 416235520 ffc06b60/ffc06a30 pktmgr
6898 54153216 1574836096 944336896 ffcac710/ffcac150 netstack
6922 13422592 1176875974 723795968 fff21940/fff1c89c radius
6924 13369344 1154296089 729210880 fff1f3e0/fff1e20c cdp
6925 5947392 997301888 343101440 ff9efe00/ff9ee28c cfs
6926 507904 625317862 280035328 ffb71310/ffb710d8 ip_dummy
6927 507904 625317862 280035328 ff831940/ff831708 ipv6_dummy
6928 4218880 1198536998 751808512 ffede610/ffedca9c otm
6929 37040128 1154853939 786464768 ffe570c0/ffe51f0c snmpd
6930 507904 625317862 280035328 ff81fcf0/ff81fab8 tcpudp_dummy
6940 2019328 735279001 332345344 ffeacc00/ffea8e0c dcos-xinetd
6975 2150400 771340979 338505728 ffb5d590/ffb5d28c callhome
7498 0 0 0 0/0 plugin
7551 56455168 1624293580 470491136 ffbd6a40/ffbd4e5c port-profile
7561 27406336 1310221811 883138560 ff88b600/ff88b4e0 rpm
7563 4292608 708863865 353136640 ffc422e0/ffc4073c pltfm_config
7564 5931008 948154662 350060544 ffea0820/ffe9f800 plcmgr
7566 14667776 1061362368 448364544 ffd3e4b0/ffd3c92c pfstat
7568 13434880 1320674803 856924160 ffc82c30/ffc7e95c ntp
7569 4603904 1046202240 621010944 ffa90920/ffa8edac monitor
7570 14360576 1208111398 775880704 ff9154f0/ff9153d0 m6rib
7571 11730944 770674995 353128448 ff8d3c80/ff8d20dc lim
7572 36610048 985861324 405618688 ffb65fd0/ffb65a80 l2rib
7573 77631488 1117101568 435101696 ffd52320/ffd514bc ipfib
7574 40394752 1541774988 823087104 ffa94c80/ffa94740 igmp
7575 28921856 808713804 464568320 ff9f61e0/ff9f441c eth_port_channel
7576 3035136 897520588 347672576 ffd14120/ffd125ac adbm
7577 2297856 942482112 338509824 ffc180c0/ffc170a0 acllog
7592 15802368 712472851 355172352 ffbf9210/ffbf766c eltm
7596 21975040 854437619 467484672 ffb4e5a0/ffb4cd1c vlan_mgr
7610 1511424 0 333488128 ffd1bdb0/ffd17fdc ntpd
7612 1531904 697346214 336756736 ffb348f0/ffb34510 eth_dstats
7613 24256512 991083878 468684800 ffc1b2d0/ffc19c30 ipqosmgr
7614 15085568 1062831193 451735552 ffb810f0/ffb800c0 lacp
7622 35782656 906373318 476381184 ffa24b80/ffa22dac ethpm
7623 44072960 1388452384 500416512 ffe7c290/ffe7a4ec l2fm
7624 33292288 688687296 378732544 ff9f0400/ff9ee66c aclqos
7625 19730432 1340354329 461086720 ffea4830/ffea3d10 stp
7626 4366336 678792998 341712896 ff90f1f0/ff90e1a0 stripcl
7642 16130048 989745715 456417280 ffa38a60/ffa377b0 copp
7652 4702208 1264318899 634081280 ffeb1ef0/ffead4fc vpc
7653 2072576 678815526 338681856 ffe86f40/ffe85f10 u2
7654 4923392 1331122636 344227840 ffdf07b0/ffdeebbc spm
7655 2371584 1202677644 722636800 ffe58700/ffe56b8c sal
7656 18264064 1486729395 719826944 fff3ede0/fff3ecc0 mrib
7657 16396288 1075082201 365699072 ffbe9c60/ffbe809c mfdm
7658 2367488 680149184 342294528 ffafeb90/ffafdb10 mcm
7659 2572288 943009267 342224896 ffce61e0/ffce464c l2pt
7660 19718144 818388313 458969088 ffc73470/ffc7188c interface-vlan
7668 21983232 962635660 415424512 ffc7dc60/ffc7c0ec ufdm
7669 8130560 1078978700 345919488 ffb1a360/ffb196b0 m2rib
7672 11886592 1173634406 705236992 ffd32390/ffd322b0 mcastfwd
7697 0 0 0 0/0 wdpunch_thread
7742 0 0 0 0/0 bkncmd
7743 0 0 0 0/0 bknevt
7783 3575808 0 287432704 ffa940f0/ffa9341c bloggerd
7863 1732608 0 286425088 ffff64a0/ffff6170 psshelper
7864 1732608 0 286425088 ff840850/ff840520 psshelper
7865 71057408 0 469352448 ffef8c60/ffef87c0 plog_lc
7866 843776 0 280580096 fff4fc70/fff4f6e8 patch_installer
7867 466944 0 280535040 ff8ec980/ff8ec038 obfl_lc
7868 1183744 0 284327936 ffbe1af0/ffbe0870 mvsh
7869 1253376 0 284467200 ffe03660/ffe02630 evmc
7870 80203776 0 482349056 ff820ca0/ff82052c dt_helper
7871 3031040 0 348794880 ffca7d90/ffca6a40 diagclient
7872 17182720 0 360239104 ffca8350/ffca7ac0 crdcfg_server
7873 499712 0 280338432 ffa34260/ffa33ff8 capability
7884 76320768 0 478597120 ffda3ed0/ffda346c device_test
7919 811008 0 345006080 ffcafbf0/ffcaf710 crdclient
7921 200384512 0 634863616 ffd6ac20/ffd6a78c t2usd
7923 21573632 0 743190528 ffa62e00/ffa62160 nsusd
8131 72155136 0 474206208 ffd3fb90/ffd3f6d0 dc3_sensor
8141 91025408 0 499462144 ff9c1380/ff9c0e50 bfdc
8142 143904768 0 556765184 ffa193b0/ffa1833c iftmc
8143 115154944 0 525111296 ffcb4bc0/ffcb3fbc pixc
8144 78970880 0 482656256 ffba7d20/ffba7120 port_client
8145 83636224 0 489664512 ff8b66d0/ff8b6230 stats_client
8146 81498112 0 490598400 ffe64520/ffe634ec vntagc
8147 113397760 0 524271616 ff80d3f0/ff80c8cc mtm
8153 175144960 0 597024768 ff8f8160/ff8f72fc ipfib
8156 118140928 0 537980928 ffea77d0/ffea658c aclqos
8157 81772544 0 505847808 ffeb2440/ffeb140c ptplc
8162 82186240 0 492183552 ff96d1a0/ff96c16c monc
8163 92790784 0 496664576 ffb8c260/ffb8b1cc xbar_client
8235 0 0 0 0/0 ppm
8935 0 0 0 0/0 kworker/0:1
8963 184320 0 4325376 7cc43360/7cc43248 klogd
9835 0 0 0 0/0 ppm
9967 0 0 0 0/0 ppm
12079 0 0 0 0/0 ppm
12143 0 0 0 0/0 ppm
12160 5550080 1210387545 767406080 fff523b0/fff4d74c hsrp_engine
12539 0 0 0 0/0 kworker/2:2
13039 188416 0 2572288 ff8fdf30/ff8fb888 getty
13364 0 0 0 0/0 ppm
13381 0 0 0 0/0 ppm
18573 153915392 758766694 496168960 ffe79940/ffe78b30 diag_port_lb
22872 1777664 0 335474688 ff9988f0/ff9941ac dcos_sshd
22877 13152256 0 495513600 ffa5b610/ffa52f58 vsh
22930 184320 0 6586368 979b4a90/979b48c8 more
22931 13287424 0 495706112 ffa5b610/ffa52838 vsh
22932 0 0 0 f735960/f735658 ps
26436 0 0 0 0/0 kworker/2:0
27712 0 0 0 0/0 kworker/1:0
29466 0 0 0 0/0 kworker/1:2
31800 0 0 0 0/0 ppm
All processes: MemAlloc = 4278689792
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-27-2022 09:17 AM
As the Cisco TAC confirmed, there were some bugs causing memory leak. Rebooting the switches solved and now we are upgrading to a fixed version.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-27-2022 09:33 AM
Yeah, memory leaks are notorious for crashing the switch, and if you don't reboot in a planned maintenance window, it will get rebooted at a very bad time.
Good Luck with the upgrade!
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide