Nexus 9K high memory utilization

[Gianluca Renzi]

Hello everybody,

  on our network we are using a pair of Cisco Nexus 9K (9300) in vPC mode; the switches are configured with some SVIs, VRRP enabled, and make some static routing.

These switches have been running for more than six year without a reboot, and starting from two months ago I noticed that the memory utilization in increasing, ranging around 90% of the total available memory.

I'm planning to reboot both switches (once authorized), and would like to know if there is any check I can do before the reboot to gather some information about why the ram started to fill.

Apart from this, can someone confirm that I should reboot the "master" unit first and then the other one, as we do when installing a software upgrade?

Thank you in advance!

[Reza Sharifi · Accepted solution]

Hi,

I'm planning to reboot both switches (once authorized), and would like to know if there is any check I can do before the reboot to gather some information about why the ram started to fill.

Before you reboot, gather all the logs for reference. Also, if you have a support contract on the switch, I recommend opening a ticket with TAC and having them investigate further before rebooting.

Apart from this, can someone confirm that I should reboot the "master" unit first and then the other one, as we do when installing a software upgrade?

If you are upgrading, I would reboot the master first, and then the secondary. This way, once you reboot the secondary, the old master will be the primary switch again.

HTH

[marce1000]

       >.... if there is any check I can do before the reboot to gather some information about why the ram started to fill.

  Connect  to device with https://cway.cisco.com/cli , at the top left run or press 'System Diagnostics'

 M.

[MHM Cisco World]

can you share this 
NSK# show processes memory

[Gianluca Renzi]

Hi, this is the output of the "show processes memory". Firmware version is 7.0(3)I4(2).

PID    MemAlloc  MemLimit    MemUsed     StackBase/Ptr      Process
-----  --------  ----------  ----------  -----------------  ----------------
    1    176128  0           4325376     7c0b4530/7c0b3b58  init
    2         0  0           0                  0/0         kthreadd
    3         0  0           0                  0/0         ksoftirqd/0
    6         0  0           0                  0/0         migration/0
    7         0  0           0                  0/0         watchdog/0
    8         0  0           0                  0/0         migration/1
   10         0  0           0                  0/0         ksoftirqd/1
   12         0  0           0                  0/0         watchdog/1
   13         0  0           0                  0/0         migration/2
   15         0  0           0                  0/0         ksoftirqd/2
   16         0  0           0                  0/0         watchdog/2
   17         0  0           0                  0/0         migration/3
   19         0  0           0                  0/0         ksoftirqd/3
   20         0  0           0                  0/0         watchdog/3
   21         0  0           0                  0/0         cpuset
   22         0  0           0                  0/0         khelper
   23         0  0           0                  0/0         kdevtmpfs
   24         0  0           0                  0/0         netns
   25         0  0           0                  0/0         sync_supers
   26         0  0           0                  0/0         bdi-default
   27         0  0           0                  0/0         kblockd
   28         0  0           0                  0/0         ata_sff
   29         0  0           0                  0/0         khubd
   30         0  0           0                  0/0         rpciod
   39         0  0           0                  0/0         kworker/u:1
   51         0  0           0                  0/0         khungtaskd
   52         0  0           0                  0/0         kswapd0
   53         0  0           0                  0/0         ksmd
   54         0  0           0                  0/0         fsnotify_mark
   55         0  0           0                  0/0         unionfs_siod
   56         0  0           0                  0/0         nfsiod
   57         0  0           0                  0/0         crypto
   72         0  0           0                  0/0         scsi_eh_0
   73         0  0           0                  0/0         scsi_eh_1
   74         0  0           0                  0/0         scsi_eh_2
   75         0  0           0                  0/0         scsi_eh_3
   76         0  0           0                  0/0         scsi_eh_4
   78         0  0           0                  0/0         kworker/u:3
   79         0  0           0                  0/0         cnic_wq
   80         0  0           0                  0/0         bnx2x
   81         0  0           0                  0/0         edac-poller
   82         0  0           0                  0/0         deferwq
  138         0  0           0                  0/0         loop0
  646         0  0           0                  0/0         loop1
  648         0  0           0                  0/0         loop2
 2094         0  0           0                  0/0         kworker/0:2
 2255         0  0           0                  0/0         jbd2/sda4-8
 2256         0  0           0                  0/0         ext4-dio-unwrit
 2597         0  0           0                  0/0         jbd2/sda5-8
 2598         0  0           0                  0/0         ext4-dio-unwrit
 2603         0  0           0                  0/0         jbd2/sda6-8
 2604         0  0           0                  0/0         ext4-dio-unwrit
 2682         0  0           0                  0/0         flush-8:0
 4932         0  0           0                  0/0         loop3
 4968         0  0           0                  0/0         jbd2/sda3-8
 4969         0  0           0                  0/0         ext4-dio-unwrit
 4978         0  0           0                  0/0         jbd2/sda2-8
 4979         0  0           0                  0/0         ext4-dio-unwrit
 4992         0  0           0                  0/0         jbd2/sda7-8
 4993         0  0           0                  0/0         ext4-dio-unwrit
 5016    184320  0           8638464     7d7611e0/7d761000  portmap
 5029         0  0           0                  0/0         lockd
 5030         0  0           0                  0/0         nfsd
 5032    323584  0           15368192    875e35d0/875e3288  rpc.mountd
 5034    208896  0           15093760    ddbacbb0/ddbac948  rpc.statd
 5214         0  0           0                  0/0         kworker/3:0
 5332         0  0           0                  0/0         ppm
 5400         0  0           0                  0/0         loop4
 5401         0  0           0                  0/0         ext4-dio-unwrit
 5647    167936  0           2269184     ffb50c00/ffb50ab0  mcelog
 5898     94208  0           3166208     ffb5c1d0/ffb5b958  sh
 5899  14209024  0           446779392   ffdfabb0/ffdfa910  sysmgr
 5900  13127680  0           420933632   ffcd6780/ffcd64e0  sysmgr
 5914  14118912  0           20221952    ffdaa9b0/ffdaa3b0  libvirtd
 5920         0  0           0                  0/0         kworker/3:1
 6118         0  0           0                  0/0         ppm
 6258         0  0           0                  0/0         mping-thread
 6259         0  0           0                  0/0         mping-thread
 6291         0  0           0                  0/0         cctrl_kthread
 6339         0  0           0                  0/0         redun_kthread
 6360         0  0           0                  0/0         usd_mts_kthread
 6365         0  0           0                  0/0         ls-notify-mts-t
 6582    389120  735279001   2994176     ff8ed900/ff8ed750  xinetd
 6583    434176  317279001   2740224     ffe96130/ffe95f40  tftpd
 6584   2306048  640189510   336601088   ffa99bf0/ffa98f20  sdwrapd
 6586    503808  0           280035328   ffdd3710/ffdd333c  dme_proxy
 6587  51171328  0           496615424   ff9abaa0/ff9aa9c0  platform
 6590  167215104  0           758538240   ffe00820/ffe0075c  event_manager
 6591  141025280  0           650620928   ffaac7f0/ffaac72c  policyelem
 6595    454656  0           280031232   ffee6240/ffee6070  sdwrapd
 6597   2154496  0           285880320   ffdabdd0/ffdaada0  pfmclnt
 6628  10780672  0           420995072   ffee1cd0/ffee1b40  dme_asap_backend
 6629  20910080  812320115   455356416   ffc54430/ffc4ff0c  syslogd
 6631  12181504  972782131   445616128   ffab2250/ffab1b78  vshd
 6632   7892992  942540684   345169920   ffdcfc70/ffdcf6b0  smm
 6633   2031616  667833689   293507072   ff90c240/ff90bf10  psshelper
 6634  53411840  924999680   394579968   ffa2dad0/ffa2cec0  pixm_vl
 6635  53796864  1045957651  490364928   ff9d5880/ff9d4c70  pixm_gl
 6636   1007616  0           10735616    ff87e390/ff87e0e0  nxapi
 6637  200749056  0           707039232   fff9daf0/fff9d860  nginx
 6638   3170304  837235520   400834560   ff940640/ff93da8c  mmode
 6639    286720  0           3432448     ff9a5540/ff9a41b0  lmgrd
 6640   1388544  0           337625088   fff284d0/fff27c50  fs-daemon
 6641  14647296  0           448626688   ffdbadd0/ffdb9c8c  feature-mgr
 6642   1249280  0           335589376   fff1bc90/fff1ba50  confcheck
 6643   3166208  623743577   337465344   fffdaeb0/fffda2d0  capability
 6644   5251072  665594899   342671360   ffe0fa40/ffe0e21c  bloggerd
 6645   1773568  667675993   291012608   ff814650/ff814320  psshelper_gsvc
 6653  33542144  1436260531  558047232   ffe7a550/ffe799dc  clis
 6654   1589248  666127756   339013632   ffb0a540/ffb0a078  licmgr
 6663    307200  0           3461120     ff8c0b50/ff8c0990  cisco
 6675    991232  0           336314368   ffa27250/ffa27080  xmlma
 6676   1916928  636699008   337272832   ffa5a410/ffa591d0  vmm
 6677   4780032  700537516   352092160   ffdf3810/ffdf3308  vman
 6678  13340672  866243276   445472768   ffb5e7b0/ffb5dc60  vdc_mgr
 6679   9949184  0           289624064   ffcf76a0/ffcf546c  usbhsd
 6683   1835008  680295193   339578880   ffaf6cd0/ffaf65a0  ttyd
 6684    946176  604999084   280567808   ff8f3d20/ff8f3888  sysinfo
 6685   1921024  669840371   338915328   ffe857f0/ffe847b0  snmpmib_proc
 6686   1228800  639148716   338038784   ffdfd2e0/ffdfcc40  sksd
 6688   1720320  622414425   338542592   ffc50070/ffc4fb9c  res_mgr
 6689   1318912  427279001   332603392   ffbf5b30/ffbf4cec  pyproxy
 6690   3620864  1987316531  292134912   ffbc5fc0/ffbc4f8c  plugin
 6691    548864  700675008   318537728   ffc48d30/ffc48890  plog_sup
 6692    892928  1484169420  280805376   ffd25860/ffd252a8  patch-installer
 6693  10022912  4294967295  759083008   ffaa9750/ffaa91a0  nbproxy
 6694   1683456  783802272   336756736   ffd6d810/ffd6c590  mvsh
 6695    512000  0           280899584   ffa0b380/ffa0af30  mping_server
 6696   2772992  685810048   343531520   ffa6cc10/ffa6b880  module
 6697  13205504  805347065   355770368   fffb0e50/fffb08d0  kim
 6698   1941504  626239680   337960960   ffc79710/ffc786e0  evms
 6700   9408512  775625804   347131904   fffcad10/fffcaac8  epld_upgrade_stdby
 6701   4415488  668934745   339902464   ff8288f0/ff827880  diagmgr
 6702   4177920  1198766784  752046080   ff91f700/ff91db7c  dhclient
 6703  12541952  1054536665  351719424   ffd70df0/ffd70550  crdcfg_server
 6704   1003520  0           337055744   ff83d1f0/ff83cb90  core-dmon
 6705  119828480  0           659779584   ff92c800/ff92c73c  confelem
 6706   2007040  669069913   339845120   ff855390/ff854360  clk_mgr
 6707    638976  460279001   287543296   ffa4f1a0/ffa4e6c8  bios_daemon
 6708  12419072  1476094963  533876736   ff9d4fb0/ff9d43a8  ascii-cfg
 6709  13058048  0           446046208   ff931de0/ff930b18  securityd
 6710   2297856  0           342016000   ff91d3b0/ff91b5fc  cert_enroll
 6711  12890112  0           446472192   ffc9c440/ffc9afe0  aaa
 6715   1396736  714083673   340463616   ffa180e0/ffa17798  obfl
 6716  21348352  1645618585  462168064   ffb237f0/ffb22400  aclmgr
 6722  18247680  1790424268  650776576   ff835410/ff835030  urib
 6724   1859584  888500518   337047552   ff8b9f40/ff8b8f10  evmc
 6727   5943296  669538496   342913024   fffd6320/fffd4fd0  diagclient
 6737   5001216  0           340529152   fff8d700/fff8c6a0  xbar
 6738  17399808  768682675   354586624   ffc82a40/ffc81fcc  device_test
 6742   1544192  715124467   340545536   ffb5f4b0/ffb5e5a0  ExceptionLog
 6743  11509760  807881254   445349888   ffa1a240/ffa19af0  bootvar
 6744   1114112  0           334548992   ffedf490/ffedf030  cardclient
 6745  34770944  802365414   473804800   ffc172d0/ffc16290  ifmgr
 6746  25300992  1076438105  469577728   ffdba390/ffdb9f60  l3vm
 6765  13570048  685384691   353710080   ffe7a310/ffe79e70  statsclient
 6800    266240  0           12836864    78f33750/78f2b2d0  incrond
 6851   8413184  949506342   354549760   ffeb5360/ffeb4dd0  npacl
 6870  33746944  1224169356  770908160   ffcf4e80/ffcf4930  adjmgr
 6871  21569536  1205192588  749711360   ffb571f0/ffb570b0  u6rib
 6879  75345920  1415698188  726204416   ffafc230/ffafc0e0  arp
 6881  24104960  1301720563  884776960   ff84d040/ff84cad0  icmpv6
 6882  24584192  978306137   416235520   ffc06b60/ffc06a30  pktmgr
 6898  54153216  1574836096  944336896   ffcac710/ffcac150  netstack
 6922  13422592  1176875974  723795968   fff21940/fff1c89c  radius
 6924  13369344  1154296089  729210880   fff1f3e0/fff1e20c  cdp
 6925   5947392  997301888   343101440   ff9efe00/ff9ee28c  cfs
 6926    507904  625317862   280035328   ffb71310/ffb710d8  ip_dummy
 6927    507904  625317862   280035328   ff831940/ff831708  ipv6_dummy
 6928   4218880  1198536998  751808512   ffede610/ffedca9c  otm
 6929  37040128  1154853939  786464768   ffe570c0/ffe51f0c  snmpd
 6930    507904  625317862   280035328   ff81fcf0/ff81fab8  tcpudp_dummy
 6940   2019328  735279001   332345344   ffeacc00/ffea8e0c  dcos-xinetd
 6975   2150400  771340979   338505728   ffb5d590/ffb5d28c  callhome
 7498         0  0           0                  0/0         plugin
 7551  56455168  1624293580  470491136   ffbd6a40/ffbd4e5c  port-profile
 7561  27406336  1310221811  883138560   ff88b600/ff88b4e0  rpm
 7563   4292608  708863865   353136640   ffc422e0/ffc4073c  pltfm_config
 7564   5931008  948154662   350060544   ffea0820/ffe9f800  plcmgr
 7566  14667776  1061362368  448364544   ffd3e4b0/ffd3c92c  pfstat
 7568  13434880  1320674803  856924160   ffc82c30/ffc7e95c  ntp
 7569   4603904  1046202240  621010944   ffa90920/ffa8edac  monitor
 7570  14360576  1208111398  775880704   ff9154f0/ff9153d0  m6rib
 7571  11730944  770674995   353128448   ff8d3c80/ff8d20dc  lim
 7572  36610048  985861324   405618688   ffb65fd0/ffb65a80  l2rib
 7573  77631488  1117101568  435101696   ffd52320/ffd514bc  ipfib
 7574  40394752  1541774988  823087104   ffa94c80/ffa94740  igmp
 7575  28921856  808713804   464568320   ff9f61e0/ff9f441c  eth_port_channel
 7576   3035136  897520588   347672576   ffd14120/ffd125ac  adbm
 7577   2297856  942482112   338509824   ffc180c0/ffc170a0  acllog
 7592  15802368  712472851   355172352   ffbf9210/ffbf766c  eltm
 7596  21975040  854437619   467484672   ffb4e5a0/ffb4cd1c  vlan_mgr
 7610   1511424  0           333488128   ffd1bdb0/ffd17fdc  ntpd
 7612   1531904  697346214   336756736   ffb348f0/ffb34510  eth_dstats
 7613  24256512  991083878   468684800   ffc1b2d0/ffc19c30  ipqosmgr
 7614  15085568  1062831193  451735552   ffb810f0/ffb800c0  lacp
 7622  35782656  906373318   476381184   ffa24b80/ffa22dac  ethpm
 7623  44072960  1388452384  500416512   ffe7c290/ffe7a4ec  l2fm
 7624  33292288  688687296   378732544   ff9f0400/ff9ee66c  aclqos
 7625  19730432  1340354329  461086720   ffea4830/ffea3d10  stp
 7626   4366336  678792998   341712896   ff90f1f0/ff90e1a0  stripcl
 7642  16130048  989745715   456417280   ffa38a60/ffa377b0  copp
 7652   4702208  1264318899  634081280   ffeb1ef0/ffead4fc  vpc
 7653   2072576  678815526   338681856   ffe86f40/ffe85f10  u2
 7654   4923392  1331122636  344227840   ffdf07b0/ffdeebbc  spm
 7655   2371584  1202677644  722636800   ffe58700/ffe56b8c  sal
 7656  18264064  1486729395  719826944   fff3ede0/fff3ecc0  mrib
 7657  16396288  1075082201  365699072   ffbe9c60/ffbe809c  mfdm
 7658   2367488  680149184   342294528   ffafeb90/ffafdb10  mcm
 7659   2572288  943009267   342224896   ffce61e0/ffce464c  l2pt
 7660  19718144  818388313   458969088   ffc73470/ffc7188c  interface-vlan
 7668  21983232  962635660   415424512   ffc7dc60/ffc7c0ec  ufdm
 7669   8130560  1078978700  345919488   ffb1a360/ffb196b0  m2rib
 7672  11886592  1173634406  705236992   ffd32390/ffd322b0  mcastfwd
 7697         0  0           0                  0/0         wdpunch_thread
 7742         0  0           0                  0/0         bkncmd
 7743         0  0           0                  0/0         bknevt
 7783   3575808  0           287432704   ffa940f0/ffa9341c  bloggerd
 7863   1732608  0           286425088   ffff64a0/ffff6170  psshelper
 7864   1732608  0           286425088   ff840850/ff840520  psshelper
 7865  71057408  0           469352448   ffef8c60/ffef87c0  plog_lc
 7866    843776  0           280580096   fff4fc70/fff4f6e8  patch_installer
 7867    466944  0           280535040   ff8ec980/ff8ec038  obfl_lc
 7868   1183744  0           284327936   ffbe1af0/ffbe0870  mvsh
 7869   1253376  0           284467200   ffe03660/ffe02630  evmc
 7870  80203776  0           482349056   ff820ca0/ff82052c  dt_helper
 7871   3031040  0           348794880   ffca7d90/ffca6a40  diagclient
 7872  17182720  0           360239104   ffca8350/ffca7ac0  crdcfg_server
 7873    499712  0           280338432   ffa34260/ffa33ff8  capability
 7884  76320768  0           478597120   ffda3ed0/ffda346c  device_test
 7919    811008  0           345006080   ffcafbf0/ffcaf710  crdclient
 7921  200384512  0           634863616   ffd6ac20/ffd6a78c  t2usd
 7923  21573632  0           743190528   ffa62e00/ffa62160  nsusd
 8131  72155136  0           474206208   ffd3fb90/ffd3f6d0  dc3_sensor
 8141  91025408  0           499462144   ff9c1380/ff9c0e50  bfdc
 8142  143904768  0           556765184   ffa193b0/ffa1833c  iftmc
 8143  115154944  0           525111296   ffcb4bc0/ffcb3fbc  pixc
 8144  78970880  0           482656256   ffba7d20/ffba7120  port_client
 8145  83636224  0           489664512   ff8b66d0/ff8b6230  stats_client
 8146  81498112  0           490598400   ffe64520/ffe634ec  vntagc
 8147  113397760  0           524271616   ff80d3f0/ff80c8cc  mtm
 8153  175144960  0           597024768   ff8f8160/ff8f72fc  ipfib
 8156  118140928  0           537980928   ffea77d0/ffea658c  aclqos
 8157  81772544  0           505847808   ffeb2440/ffeb140c  ptplc
 8162  82186240  0           492183552   ff96d1a0/ff96c16c  monc
 8163  92790784  0           496664576   ffb8c260/ffb8b1cc  xbar_client
 8235         0  0           0                  0/0         ppm
 8935         0  0           0                  0/0         kworker/0:1
 8963    184320  0           4325376     7cc43360/7cc43248  klogd
 9835         0  0           0                  0/0         ppm
 9967         0  0           0                  0/0         ppm
12079         0  0           0                  0/0         ppm
12143         0  0           0                  0/0         ppm
12160   5550080  1210387545  767406080   fff523b0/fff4d74c  hsrp_engine
12539         0  0           0                  0/0         kworker/2:2
13039    188416  0           2572288     ff8fdf30/ff8fb888  getty
13364         0  0           0                  0/0         ppm
13381         0  0           0                  0/0         ppm
18573  153915392  758766694   496168960   ffe79940/ffe78b30  diag_port_lb
22872   1777664  0           335474688   ff9988f0/ff9941ac  dcos_sshd
22877  13152256  0           495513600   ffa5b610/ffa52f58  vsh
22930    184320  0           6586368     979b4a90/979b48c8  more
22931  13287424  0           495706112   ffa5b610/ffa52838  vsh
22932         0  0           0            f735960/f735658   ps
26436         0  0           0                  0/0         kworker/2:0
27712         0  0           0                  0/0         kworker/1:0
29466         0  0           0                  0/0         kworker/1:2
31800         0  0           0                  0/0         ppm

All processes: MemAlloc = 4278689792

[Gianluca Renzi]

As the Cisco TAC confirmed, there were some bugs causing memory leak. Rebooting the switches solved and now we are upgrading to a fixed version.

[Reza Sharifi]

Yeah, memory leaks are notorious for crashing the switch, and if you don't reboot in a planned maintenance window, it will get rebooted at a very bad time.

Good Luck with the upgrade!