02-17-2014 11:04 AM - edited 03-07-2019 06:15 PM
Hi all, I'm having an issue with a Nexus 5010 running at almost 100% CPU utilization. I have ran the following commands to troubleshoot:
NX5010-2# show processes cpu history
1111 1 1 1111111 1 111 11 1 1 11111 111111 11111111 11
000090909000000090990009009099090000090000009900000000900999
000090909000000090990008009098090000070000009900000000900997
100 ############################################################
90 ############################################################
80 ############################################################
70 ############################################################
60 ############################################################
50 ############################################################
40 ############################################################
30 ############################################################
20 ############################################################
10 ############################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)
# = average CPU%
111111111111111111111111111111111111111111111111111111111111
000000000000000000000000000000000000000000000000000000000000
000000000000000000000000000000000000000000000000000000000000
100 ############################################################
90 ############################################################
80 ############################################################
70 ############################################################
60 ############################################################
50 ############################################################
40 ############################################################
30 ############################################################
20 ############################################################
10 ############################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
111111111111111111111111111111111111111111111111111111111111111111111111
000000000000000000000000000000000000000000000000000000000000000000000000
000000000000000000000000000000000000000000000000000000000000000000000000
100 ########################################################################
90 ########################################################################
80 ########################################################################
70 ########################################################################
60 ########################################################################
50 ########################################################################
40 ########################################################################
30 ########################################################################
20 ########################################################################
10 ########################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%
NX5010-2# show processes cpu sort
PID Runtime(ms) Invoked uSecs 1Sec Process
----- ----------- -------- ----- ------ -----------
3221 1990734612 2147483647 3642 48.8% snmpd
3163 350 53 6621 33.7% netstack
3149 1169906661 1583992462 738 3.4% gatosusd
3082 646529925 113123061 5715 1.7% pfma
12673 217200755 165237584 1314 1.7% udld
1 766000 9251911 82 0.0% init
2 3 172 18 0.0% kthreadd
3 0 2 3 0.0% migration/0
4 52264184 2147483647 19 0.0% ksoftirqd/0
5 123129 2003480 61 0.0% watchdog/0
6 6628270 71771728 92 0.0% events/0
7 0 12 27 0.0% khelper
8 37858 2013467 18 0.0% kblockd/0
9 0 2 1 0.0% kacpid
10 0 2 2 0.0% kacpi_notify
11 0 3 11 0.0% ksuspend_usbd
12 3 46 67 0.0% khubd
14 64517 4767713 13 0.0% pdflush
15 0 2 2 0.0% kswapd0
16 0 2 2 0.0% aio/0
17 0 2 2 0.0% nfsiod
18 0 2 2 0.0% SerrLogKthread
19 0 2 3 0.0% kide/0
22 0 2 2 0.0% scsi_eh_0
23 123683 17581484 7 0.0% usb-storage
25 0 17 12 0.0% rpciod/0
27 68385 5026566 13 0.0% pdflush
426 6 2 3459 0.0% jffs2_gcd_mtd2
524 60059 1990810 30 0.0% kjournald
529 39230 1667256 23 0.0% kjournald
1094 38828 1589932 24 0.0% kjournald
1103 38696 1505577 25 0.0% kjournald
1760 1 30 50 0.0% portmap
1769 5 5 1070 0.0% rpc.statd
1784 0 2 3 0.0% lockd
1785 51 12553 4 0.0% nfsd
1786 55 12554 4 0.0% nfsd
1787 50 12550 4 0.0% nfsd
1788 55 12554 4 0.0% nfsd
1789 54 12555 4 0.0% nfsd
1790 52 12554 4 0.0% nfsd
1791 55 12556 4 0.0% nfsd
1792 60 12555 4 0.0% nfsd
1797 0 1 166 0.0% rpc.mountd
1820 12997823 120010129 108 0.0% sysmgr
2457 0 1 18 0.0% mping-thread
2687 37775 4517992 8 0.0% insmod
2769 0 2 5 0.0% bond0
3078 9 2 4939 0.0% xinetd
3079 4 2 2428 0.0% tftpd
3080 26178733 2147483647 11 0.0% syslogd
3081 100 317 318 0.0% sdwrapd
3087 135140 22591058 5 0.0% ls-notify-mts-t
3093 6445600 51482264 125 0.0% usd_mts_kthread
3101 275833 11318427 24 0.0% vshd
3102 109 18 6073 0.0% smm
3103 609195 45196308 13 0.0% psshelper
3104 53109 754120 70 0.0% lmgrd
3105 259384 9050458 28 0.0% licmgr
3106 80927 1029399 78 0.0% fs-daemon
3107 112668 4829686 23 0.0% feature-mgr
3108 196356 9045236 21 0.0% fcfwd
3109 153 1286 119 0.0% confcheck
3110 190900 7108991 26 0.0% capability
3111 587267 45194943 12 0.0% psshelper_gsvc
3118 201954 760669 265 0.0% cisco
3122 164453 4530865 36 0.0% clis
3124 453408 10352089 43 0.0% port-profile
3125 223200 7110916 31 0.0% provision
3127 424542 14945457 28 0.0% zschk
3128 922620 59835987 15 0.0% xmlma
3129 408575 14956490 27 0.0% vmm
3130 355122 14955894 23 0.0% vdc_mgr
3131 290982 22605382 12 0.0% ttyd
3132 70987 4524569 15 0.0% sysinfo
3133 71 325 218 0.0% sksd
3134 76540364 3006046 25462 0.0% sensor
3136 13562 862881 15 0.0% res_mgr
3137 21536 309435 69 0.0% ptplc
3138 1008742 45962133 21 0.0% plugin
3139 940543 16306793 57 0.0% pltfm_config
3140 308989 9066451 34 0.0% plog_sup
3141 1521138 19524975 77 0.0% pfstat
3142 221110 5674165 38 0.0% obfl
3143 198 40 4955 0.0% npacl
3144 197951 7110583 27 0.0% nohms
3145 145971 4516709 32 0.0% mvsh
3146 538704 14930296 36 0.0% mcm
3147 907038 45216136 20 0.0% ledmgr
3148 259708 7229740 35 0.0% ifmgr
3150 9718306 39900332 243 0.0% evmc
3151 401835 14981438 26 0.0% ethpc
3152 39694 848006 46 0.0% eth_dstats
3153 402 341 1181 0.0% core-dmon
3154 307064 22609121 13 0.0% bootvar
3155 159077 9041614 17 0.0% bios_daemon
3156 431534 9035502 47 0.0% athertonusd
3157 169080 5659420 29 0.0% ascii-cfg
3158 387485 15058587 25 0.0% altos
3159 839110 45207754 18 0.0% securityd
3160 707091 45199702 15 0.0% cert_enroll
3161 671116 45208625 14 0.0% aaa
3162 262228 7244098 36 0.0% eth_port_channel
3165 14041192 18798473 746 0.0% afm
3166 30004122 183173383 163 0.0% fwm
3168 402166 14974774 26 0.0% qd
3169 664146542 154157076 4308 0.0% statsclient
3170 131 28 4694 0.0% l3vm
3171 135 48 2812 0.0% u6rib
3172 144 45 3216 0.0% urib
3197 163454 4241945 38 0.0% m2rib
3212 203 46 4426 0.0% igmp
3213 27 270 102 0.0% ip_dummy
3214 28 271 103 0.0% ipv6_dummy
3215 28 271 103 0.0% pktmgr_dummy
3216 27 270 103 0.0% tcpudp_dummy
3217 10283638 24808503 414 0.0% cdp
3218 11852550 44179323 268 0.0% lldp
3219 82666 1024823 80 0.0% aclmgr
3220 663 2725 243 0.0% dcos-xinetd
3231 10 13 794 0.0% core-client
3258 1279223 45270831 28 0.0% radius
3261 122 33 3726 0.0% adjmgr
3262 70438662 471510502 149 0.0% cfs
3263 414099 14964277 27 0.0% otm
3264 419809 14950479 28 0.0% sal
3291 308477 3296058 93 0.0% ipqosmgr
3324 910441 45174733 20 0.0% callhome
3326 220389 2639863 83 0.0% ntp
3327 49209194 359862699 136 0.0% stp
3329 173 112 1548 0.0% arp
3330 152 136 1118 0.0% icmpv6
3331 469796 14947506 31 0.0% monitor
3337 3938489 111682351 35 0.0% ntpd
3356 154 73 2109 0.0% rpm
3370 225322 6505670 34 0.0% vlan_mgr
3373 129 31 4181 0.0% mcastfwd
3381 636746 8381228 75 0.0% ethpm
3406 34591 3011894 11 0.0% wdpunch_thread
3618 9 19 478 0.0% getty
3720 0 2 4 0.0% kauditd
4388 531 2819 188 0.0% dcos_sshd
4399 378 988 382 0.0% vsh
4696 5 4 1467 0.0% more
4697 2 6 450 0.0% vsh
4698 40 12 3387 0.0% ps
10120 2 6 405 0.0% klogd
10806 4724717 51495363 91 0.0% tacacs
12654 12597967 139060733 90 0.0% vpc
12675 7478554 24855426 300 0.0% lacp
23954 92749 2768236 33 0.0% vtp
23980 229886 4477800 51 0.0% interface-vlan
29429 8 9 979 0.0% tacacsd
CPU util : 78.3% user, 20.7% kernel, 1.1% idle
Please note that only processes from the requested vdc are shown above
It looks like the snmpd service is using almost 50% of the CPU, does anyone know what could be the cause and how to fix it?
Thanks in advance!
-MJ
02-18-2014 08:30 AM
Any suggestions?
02-18-2014 11:35 AM
Hi Michael,
What NX-OS version are you running as you may be hitting CSCtr65682. I've seen this cause high CPU on older NX-OS versions a number of times.
Also see SNMP on Nexus 5548 Failing for a few more details than the bug toolkit provides.
Regards
02-18-2014 11:43 AM
Hi Steve,
Thanks for the reply, this is the version currently running on the Nexus 5K:
Software
BIOS: version 1.5.0
loader: version N/A
kickstart: version 5.1(3)N2(1b)
system: version 5.1(3)N2(1b)
power-seq: Module 1: version v1.2
BIOS compile time: 11/30/10
kickstart image file is: bootflash:///n5000-uk9-kickstart.5.1.3.N2.1b.bin
kickstart compile time: 8/31/2012 17:00:00 [08/31/2012 17:30:27]
system image file is: bootflash:///n5000-uk9.5.1.3.N2.1b.bin
system compile time: 8/31/2012 17:00:00 [08/31/2012 18:29:34]
Regards,
-Michael
02-18-2014 12:12 PM
Hi Michael,
It's not stated in the bug whether the NX-OS 5.1(3)N2(1b) release you're running is susceptible to this bug. While it looks the same I would open a TAC case and have them confirm.
Regards
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: