cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1958
Views
0
Helpful
4
Replies

Nexus 5010 high CPU % issue

mjohnson22
Level 1
Level 1

Hi all, I'm having an issue with a Nexus 5010 running at almost 100% CPU utilization. I have ran the following commands to troubleshoot:

NX5010-2# show processes cpu history

    1111 1 1 1111111 1  111 11 1  1 11111 111111  11111111 11  

    000090909000000090990009009099090000090000009900000000900999

    000090909000000090990008009098090000070000009900000000900997

100 ############################################################

90 ############################################################

80 ############################################################

70 ############################################################

60 ############################################################

50 ############################################################

40 ############################################################

30 ############################################################

20 ############################################################

10 ############################################################

    0....5....1....1....2....2....3....3....4....4....5....5....

              0    5    0    5    0    5    0    5    0    5   

               CPU% per second (last 60 seconds)

                      # = average CPU%

    111111111111111111111111111111111111111111111111111111111111

    000000000000000000000000000000000000000000000000000000000000

    000000000000000000000000000000000000000000000000000000000000

100 ############################################################

90 ############################################################

80 ############################################################

70 ############################################################

60 ############################################################

50 ############################################################

40 ############################################################

30 ############################################################

20 ############################################################

10 ############################################################

    0....5....1....1....2....2....3....3....4....4....5....5....

              0    5    0    5    0    5    0    5    0    5   

               CPU% per minute (last 60 minutes)

              * = maximum CPU%   # = average CPU%

    111111111111111111111111111111111111111111111111111111111111111111111111

    000000000000000000000000000000000000000000000000000000000000000000000000

    000000000000000000000000000000000000000000000000000000000000000000000000

100 ########################################################################

90 ########################################################################

80 ########################################################################

70 ########################################################################

60 ########################################################################

50 ########################################################################

40 ########################################################################

30 ########################################################################

20 ########################################################################

10 ########################################################################

    0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.

              0    5    0    5    0    5    0    5    0    5    0    5    0

                   CPU% per hour (last 72 hours)

                  * = maximum CPU%   # = average CPU%

NX5010-2# show processes cpu sort

PID    Runtime(ms)  Invoked   uSecs  1Sec    Process

-----  -----------  --------  -----  ------  -----------

3221   1990734612  2147483647   3642   48.8%  snmpd

3163          350        53   6621   33.7%  netstack

3149   1169906661  1583992462    738    3.4%  gatosusd

3082    646529925  113123061   5715    1.7%  pfma

12673    217200755  165237584   1314    1.7%  udld

    1       766000   9251911     82    0.0%  init

    2            3       172     18    0.0%  kthreadd

    3            0         2      3    0.0%  migration/0

    4     52264184  2147483647     19    0.0%  ksoftirqd/0

    5       123129   2003480     61    0.0%  watchdog/0

    6      6628270  71771728     92    0.0%  events/0

    7            0        12     27    0.0%  khelper

    8        37858   2013467     18    0.0%  kblockd/0

    9            0         2      1    0.0%  kacpid

   10            0         2      2    0.0%  kacpi_notify

   11            0         3     11    0.0%  ksuspend_usbd

   12            3        46     67    0.0%  khubd

   14        64517   4767713     13    0.0%  pdflush

   15            0         2      2    0.0%  kswapd0

   16            0         2      2    0.0%  aio/0

   17            0         2      2    0.0%  nfsiod

   18            0         2      2    0.0%  SerrLogKthread

   19            0         2      3    0.0%  kide/0

   22            0         2      2    0.0%  scsi_eh_0

   23       123683  17581484      7    0.0%  usb-storage

   25            0        17     12    0.0%  rpciod/0

   27        68385   5026566     13    0.0%  pdflush

  426            6         2   3459    0.0%  jffs2_gcd_mtd2

  524        60059   1990810     30    0.0%  kjournald

  529        39230   1667256     23    0.0%  kjournald

1094        38828   1589932     24    0.0%  kjournald

1103        38696   1505577     25    0.0%  kjournald

1760            1        30     50    0.0%  portmap

1769            5         5   1070    0.0%  rpc.statd

1784            0         2      3    0.0%  lockd

1785           51     12553      4    0.0%  nfsd

1786           55     12554      4    0.0%  nfsd

1787           50     12550      4    0.0%  nfsd

1788           55     12554      4    0.0%  nfsd

1789           54     12555      4    0.0%  nfsd

1790           52     12554      4    0.0%  nfsd

1791           55     12556      4    0.0%  nfsd

1792           60     12555      4    0.0%  nfsd

1797            0         1    166    0.0%  rpc.mountd

1820     12997823  120010129    108    0.0%  sysmgr

2457            0         1     18    0.0%  mping-thread

2687        37775   4517992      8    0.0%  insmod

2769            0         2      5    0.0%  bond0

3078            9         2   4939    0.0%  xinetd

3079            4         2   2428    0.0%  tftpd

3080     26178733  2147483647     11    0.0%  syslogd

3081          100       317    318    0.0%  sdwrapd

3087       135140  22591058      5    0.0%  ls-notify-mts-t

3093      6445600  51482264    125    0.0%  usd_mts_kthread

3101       275833  11318427     24    0.0%  vshd

3102          109        18   6073    0.0%  smm

3103       609195  45196308     13    0.0%  psshelper

3104        53109    754120     70    0.0%  lmgrd

3105       259384   9050458     28    0.0%  licmgr

3106        80927   1029399     78    0.0%  fs-daemon

3107       112668   4829686     23    0.0%  feature-mgr

3108       196356   9045236     21    0.0%  fcfwd

3109          153      1286    119    0.0%  confcheck

3110       190900   7108991     26    0.0%  capability

3111       587267  45194943     12    0.0%  psshelper_gsvc

3118       201954    760669    265    0.0%  cisco

3122       164453   4530865     36    0.0%  clis

3124       453408  10352089     43    0.0%  port-profile

3125       223200   7110916     31    0.0%  provision

3127       424542  14945457     28    0.0%  zschk

3128       922620  59835987     15    0.0%  xmlma

3129       408575  14956490     27    0.0%  vmm

3130       355122  14955894     23    0.0%  vdc_mgr

3131       290982  22605382     12    0.0%  ttyd

3132        70987   4524569     15    0.0%  sysinfo

3133           71       325    218    0.0%  sksd

3134     76540364   3006046  25462    0.0%  sensor

3136        13562    862881     15    0.0%  res_mgr

3137        21536    309435     69    0.0%  ptplc

3138      1008742  45962133     21    0.0%  plugin

3139       940543  16306793     57    0.0%  pltfm_config

3140       308989   9066451     34    0.0%  plog_sup

3141      1521138  19524975     77    0.0%  pfstat

3142       221110   5674165     38    0.0%  obfl

3143          198        40   4955    0.0%  npacl

3144       197951   7110583     27    0.0%  nohms

3145       145971   4516709     32    0.0%  mvsh

3146       538704  14930296     36    0.0%  mcm

3147       907038  45216136     20    0.0%  ledmgr

3148       259708   7229740     35    0.0%  ifmgr

3150      9718306  39900332    243    0.0%  evmc

3151       401835  14981438     26    0.0%  ethpc

3152        39694    848006     46    0.0%  eth_dstats

3153          402       341   1181    0.0%  core-dmon

3154       307064  22609121     13    0.0%  bootvar

3155       159077   9041614     17    0.0%  bios_daemon

3156       431534   9035502     47    0.0%  athertonusd

3157       169080   5659420     29    0.0%  ascii-cfg

3158       387485  15058587     25    0.0%  altos

3159       839110  45207754     18    0.0%  securityd

3160       707091  45199702     15    0.0%  cert_enroll

3161       671116  45208625     14    0.0%  aaa

3162       262228   7244098     36    0.0%  eth_port_channel

3165     14041192  18798473    746    0.0%  afm

3166     30004122  183173383    163    0.0%  fwm

3168       402166  14974774     26    0.0%  qd

3169    664146542  154157076   4308    0.0%  statsclient

3170          131        28   4694    0.0%  l3vm

3171          135        48   2812    0.0%  u6rib

3172          144        45   3216    0.0%  urib

3197       163454   4241945     38    0.0%  m2rib

3212          203        46   4426    0.0%  igmp

3213           27       270    102    0.0%  ip_dummy

3214           28       271    103    0.0%  ipv6_dummy

3215           28       271    103    0.0%  pktmgr_dummy

3216           27       270    103    0.0%  tcpudp_dummy

3217     10283638  24808503    414    0.0%  cdp

3218     11852550  44179323    268    0.0%  lldp

3219        82666   1024823     80    0.0%  aclmgr

3220          663      2725    243    0.0%  dcos-xinetd

3231           10        13    794    0.0%  core-client

3258      1279223  45270831     28    0.0%  radius

3261          122        33   3726    0.0%  adjmgr

3262     70438662  471510502    149    0.0%  cfs

3263       414099  14964277     27    0.0%  otm

3264       419809  14950479     28    0.0%  sal

3291       308477   3296058     93    0.0%  ipqosmgr

3324       910441  45174733     20    0.0%  callhome

3326       220389   2639863     83    0.0%  ntp

3327     49209194  359862699    136    0.0%  stp

3329          173       112   1548    0.0%  arp

3330          152       136   1118    0.0%  icmpv6

3331       469796  14947506     31    0.0%  monitor

3337      3938489  111682351     35    0.0%  ntpd

3356          154        73   2109    0.0%  rpm

3370       225322   6505670     34    0.0%  vlan_mgr

3373          129        31   4181    0.0%  mcastfwd

3381       636746   8381228     75    0.0%  ethpm

3406        34591   3011894     11    0.0%  wdpunch_thread

3618            9        19    478    0.0%  getty

3720            0         2      4    0.0%  kauditd

4388          531      2819    188    0.0%  dcos_sshd

4399          378       988    382    0.0%  vsh

4696            5         4   1467    0.0%  more

4697            2         6    450    0.0%  vsh

4698           40        12   3387    0.0%  ps

10120            2         6    405    0.0%  klogd

10806      4724717  51495363     91    0.0%  tacacs

12654     12597967  139060733     90    0.0%  vpc

12675      7478554  24855426    300    0.0%  lacp

23954        92749   2768236     33    0.0%  vtp

23980       229886   4477800     51    0.0%  interface-vlan

29429            8         9    979    0.0%  tacacsd

CPU util  :   78.3% user,   20.7% kernel,   1.1% idle

Please note that only processes from the requested vdc are shown above

It looks like the snmpd service is using almost 50% of the CPU, does anyone know what could be the cause and how to fix it?

Thanks in advance!

-MJ

4 Replies 4

mjohnson22
Level 1
Level 1

Any suggestions?

Hi Michael,

What NX-OS version are you running as you may be hitting CSCtr65682. I've seen this cause high CPU on older NX-OS versions a number of times.

Also see SNMP on Nexus 5548 Failing for a few more details than the bug toolkit provides.

Regards

Hi Steve,

Thanks for the reply, this is the version currently running on the Nexus 5K:


Software

  BIOS:      version 1.5.0

  loader:    version N/A

  kickstart: version 5.1(3)N2(1b)

  system:    version 5.1(3)N2(1b)

  power-seq: Module 1: version v1.2

  BIOS compile time:       11/30/10

  kickstart image file is: bootflash:///n5000-uk9-kickstart.5.1.3.N2.1b.bin

  kickstart compile time:  8/31/2012 17:00:00 [08/31/2012 17:30:27]

  system image file is:    bootflash:///n5000-uk9.5.1.3.N2.1b.bin

  system compile time:     8/31/2012 17:00:00 [08/31/2012 18:29:34]

Regards,

-Michael

Hi Michael,

It's not stated in the bug whether the NX-OS 5.1(3)N2(1b) release you're running is susceptible to this bug. While it looks the same I would open a TAC case and have them confirm.

Regards

Review Cisco Networking for a $25 gift card