cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements

Community Helping Community

2047
Views
0
Helpful
2
Replies

Nexus 7K high CPU utilization

Hi,

I am trying to troubloeshoot the the High cpu utilization of N7K. The switch is connected in a POC setup and traffic load is very less. It is running with NXOS 6.0.4. For past 1 month I have observed the CPU utilization goes upto 80% without any traffic/load. I tried to capture the o/p of "show process cpu sort" through EEM script. while analyzing the o/p I can not determine the process which is contributing Th eutilization of user or kernel.

for example:

In Below out put

25.2% user, 31.0% kernel, 43.8% idle

Neither any process shown below, is using 25.2% or 31%, nor the aggrigation of all processes utilizaion is 56.2%. (I have 2 VDCs including default)

Please help me how to determine the process consuming cpu.

PID   Runtime(ms) Invoked   uSecs 1Sec   Process
----- ----------- -------- ----- ------ -----------
3296   120582933   8003274 15066   6.7% sac_usd
2924     85729099 13281994   6454   2.8% platform
3588     13699639 11361018   1205   0.9% udld
   1       42870   1269054     33   0.0% init
   2           3       308     12   0.0% kthreadd
   3         1062   167018     6   0.0% migration/0
   4       462583 133573370     3   0.0% ksoftirqd/0
   5       91988   955848     96   0.0% watchdog/0
   6         777   110442     7   0.0% migration/1
   7       325174 91256362     3   0.0% ksoftirqd/1
   8         3025   952785     3   0.0% watchdog/1
   9       36939   6700241     5   0.0% events/0
   10       30877   6396271     4   0.0% events/1
   11           0       10     19   0.0% khelper
   12         170       740   230   0.0% kblockd/0
   13         180       731   246   0.0% kblockd/1
   14            0         2     0   0.0% kacpid
   15           0         2     0   0.0% kacpi_notify
   16           28         8   3530   0.0% ksuspend_usbd
   17           0         6     3   0.0% khubd
   18       31730   778523     40   0.0% pdflush
   19       20043   569273     35   0.0% pdflush
   20           0         2     3   0.0% kswapd0
   21           0         2     3   0.0% aio/0
   22           0         2     2   0.0% aio/1
   23           0         2     2  0.0% nfsiod
   24           0         2     2   0.0% SerrLogKthread
   25       63668   279821   227   0.0% kide/0
   26       601880   1871286   321   0.0% kide/1
   29           0         4     10   0.0% rpciod/0
   30           0        17     9   0.0% rpciod/1
   31       14621   2521982     5   0.0% kirqd
652           89       22   4070   0.0% jffs2_gcd_mtd2
760       43958   208586   210   0.0% kjournald
765       31116   208347   149   0.0% kjournald
886       31349   208114   150   0.0% kjournald
1988       36029   237342   151   0.0% kjournald
2233           1       30     40   0.0% portmap
2242           3         5   696   0.0% rpc.statd
2257           0         2     11   0.0% lockd
2258           8     1728     4   0.0% nfsd
2259           8     1727     4   0.0% nfsd
2260           8     1727     4   0.0% nfsd
2261           9     1727     5   0.0% nfsd
2262           8     1728     4   0.0% nfsd
2263           8     1728     4   0.0% nfsd
2264           8     1729     4   0.0% nfsd
2265           8     1726     4   0.0% nfsd
2270           0         1   132   0.0% rpc.mountd
2290     2138348 13880127   154   0.0% sysmgr
2643           0         1     25   0.0% mping-thread
2718         7036   206917     34   0.0% redun_kthread
2920           43       224   194   0.0% xinetd
2921           50       479   104   0.0% tftpd
2922     1885755 308484299    6   0.0% syslogd
2923         181     1271   142   0.0% sdwrapd
2931           0         7     45   0.0% usd_mts_kthread
2944       16901   3100548     5   0.0% ls-notify-mts-t
3027     3660723   2618895   1397   0.0% cmond
3028        2900       625   4640   0.0% pfm_dummy
3035       19653   780705     25   0.0% vshd
3036           62       17   3669   0.0% smm
3037       53404   1650665     32   0.0% session-mgr
3038       94080   6211601     15   0.0% psshelper
3039         6963   103812     67   0.0% lmgrd
3040       65621   1441087     45   0.0% licmgr
3041         5362   139249     38   0.0% fs-daemon
3042       11122   520200     21   0.0% feature-mgr
3043         206       689   299   0.0% confcheck
3044       25743   826190     31   0.0% capability
3045       94664   6211930     15   0.0% psshelper_gsvc
3053       33425   207353   161   0.0% cisco
3056       40788   638596     63   0.0% clis
3057       59461   1636992     36   0.0% port-profile
3059       49041   2072373     23   0.0% xmlma
3060       53345   1696294     31   0.0% vmm
3061       48536   1719085     28   0.0% vdc_mgr
3062       43610   3106747     14   0.0% ttyd
3063       10612   628063     16   0.0% sysinfo
3064         293       486   603   0.0% sksd
3066         3262   123522     26   0.0% res_mgr
3067       134609   6231106     21   0.0% plugin
3068       467226   1380087   338   0.0% plog_sup
3069        128       15   8544   0.0% npacl
3070       18252   520815     35   0.0% mvsh
3071           26       75   347   0.0% mping_server
3072       159427   1250958   127   0.0% idehsd
3073       16671   522586     31   0.0% evms
3074        13941   397828     35   0.0% evmc
3075     49911293 16123404   3095   0.0% diagmgr
3076       45315   502617     90   0.0% crdcfg_server
3077         331     1017   325   0.0% core-dmon
3078       171639   1761631     97   0.0% cmpproxy
3079       497384   4187027   118   0.0% clk_mgr
3080       11388   621260     18   0.0% bloggerd
3081       22013   1242095     17   0.0% bios_daemon
3082       18112   777689     23   0.0% ascii-cfg
3083       111428   6210101     17   0.0% securityd
3084       98387   6211128     15   0.0% cert_enroll
3085       94987   6214320     15   0.0% aaa
3088       18690   777305     24   0.0% obfl
3090           85       28   3046   0.0% l3vm
3091         127        34   3752   0.0% urib
3093     33075502   6237681   5302   0.0% diagclient
3110           74       34   2196   0.0% adjmgr
3111           97       28   3481   0.0% u6rib
3120       32642   241081   135   0.0% aclmgr
3122         109       76   1446   0.0% arp
3124           91       20   4566   0.0% icmpv6
3161       33124   217673   152   0.0% kjournald
3169     2245769 64712179     34   0.0% mts-sync-thr
3242           3       87     38   0.0% dhcpd
3245      53250   1299891     40   0.0% cardclient
3246       409624   8403407     48   0.0% device_test
3247       55546   577909     96   0.0% xbar
3249     1302033   7717061   168   0.0% module
3254         208       398   523   0.0% ExceptionLog
3255       50295   3108982     16   0.0% bootvar
3256       27871   765502     36   0.0% ifmgr
3259         208       94   2218   0.0% netstack
3276       176505   6215803     28   0.0% radius
3279     1714777   4982359   344   0.0% cfs
3281           22       41   542   0.0% ip_dummy
3282           33       42   791   0.0% ipv6_dummy
3283       51426   738103     69   0.0% ntp
3284       54365   1727529     31   0.0% otm
3285           23       43   543  0.0% pktmgr_dummy
3286     1246677   4547384   274   0.0% snmpd
3287           23       44   544   0.0% tcpudp_dummy
3289       340737   3019215   112   0.0% cdp
3290           83       260   320   0.0% dcos-xinetd
3292       281988 11890167     23   0.0% ntpd
3293     11903804 14652688   812   0.0% R2D2_usd
3294       292360   3153368     92   0.0% arbiter
3295     1505005   3724037   404   0.0% babycaesar
3297     41721542 12143605   3435   0.0% sensor
3298     2892940   4943312   585   0.0% xbar_driver_usd
3317       115183   6208629     18   0.0% callhome
3358     54287184 13950490   3891   0.0% oc_usd
3359     3092574 15907454   194   0.0% metropolis_usd
3372           0         2     4   0.0% kauditd
3471       22801   1240441     18   0.0% sctpt_rx_thr
3472           0         1     5   0.0% sctpt_tx_thr
3508       25446   826409     30   0.0% u2
3509       28225   206525   136   0.0% spm
3510       55102   1725410    31   0.0% sal
3511         329     2197   149   0.0% rpm
3512       568064   2990164   189   0.0% pltfm_config
3513       55064   1715291     32   0.0% pixmc
3514       68105   1715304     39   0.0% pixm
3515       366788   1522226    240   0.0% pfstat
3516         475     2030   234   0.0% mrib
3517       53274   1720790     30   0.0% monitorc
3518       65252   1572542     41   0.0% mcm
3519         260     1059   245   0.0% m6rib
3522     3000740   7577947    395   0.0% l2pt
3524       28816   829700     34   0.0% eltm
3526       305549   416012   734   0.0% copp
3527       74173   1897493     39   0.0% acllog
3532       144421   1631647     88   0.0% mfdm
3533       63805   1638615     38   0.0% monitor
3534       19748   404008     48   0.0% m2rib
3536       105922   1906312     55   0.0% ufdm
3553         166       73   2286   0.0% igmp
3558         170       123   1384   0.0% mcastfwd
3559       30802   783951    39   0.0% eth_port_channel
3560       23603   689775     34   0.0% vlan_mgr
3569         3192   110681     28   0.0% eth_dstats
3570       249041   593924   419   0.0% ipqosmgr
3571       838078   3413023   245   0.0% lacp
3577      19102   203449     93   0.0% dhcp_snoop
3578     2855782 12674401   225   0.0% diag_port_lb
3579       39560   794428     49   0.0% ethpm
3580     2084026   4060385   513   0.0% l2fm
3581       46520   907379     51   0.0% nfm
3582         9639   249890     38   0.0% vtp
3589     1260301 17011436     74   0.0% vpc
3590     11393886 107080504   106   0.0% stp
3591       17574   366700     47   0.0% interface-vlan
3607       25561   826111     30   0.0% port_client
3609     17407279 10686606   1628   0.0% statsclient
3610       23619   650313     36   0.0% xbar_client
3627       55993   6205583     9   0.0% wdpunch_thread
3705           5       19   289   0.0% gettylogin1
3731           1        4   410   0.0% getty
4496           37       80   466   0.0% login
4591           8       60   135   0.0% klogd
4615       213619   8202090     26   0.0% ntpd
8405         1236     5666   218   0.0% vsh
12066           3        2   1535   0.0% vsh
12067           99         8 12416   0.0% telnet
12068         358     4626     77   0.0% dcos-telnet
12069         257     2980     86   0.0% in.dcos-telnetd
12086           35       42   839   0.0% login
12099        340       815   418   0.0% vsh
12673           3         3   1082   0.0% vsh
12696           2         4   578   0.0% vdclogin
12699           15         2   7789   0.0% login
12700         222       909   244   0.0% vsh
13059          3         7   510   0.0% mvsh
13060         179       35   5118   0.0% vsh
13096           9       210     43   0.0% vsh
13099           3         3   1197   0.0% mvsh
13100         136       45   3033   0.0% vsh
13103           8        9   987   0.0% tacacsd
13106           2         3   712   0.0% vsh
13107           39         3 13146   0.0% ps
17076       260337   6217630     41   0.0% tacacs

CPU util :   25.2% user,   31.0% kernel,   43.8% idle
Please note that only processes from the requested vdc are shown above

2 REPLIES 2
Highlighted
Beginner

Did you find the cause of the

Did you find the cause of the high CPU? If yes, what was it and how could you fix it?

Could you maybe provide a EEM script that you used for getting the info? Did you monitor for the CPU load of the sup or of the linecards itself?

Contributor

Re: Nexus 7K high CPU utilization


Netstack is a complete IP stack implemented in the user space of Nexus 7000. Components include a L2 Packet Manager, ARP, Adjacency Manager, IPv4, Internet Control Message Protocol v4 (ICMPv4), IPv6, ICMPv6, TCP/UDP, and socket library. When traffic to the CPU is triggering high CPU usage, you often see that Netstack and its respective process are running high.

I can see Avg CPU consumed by "Pktmgr" over 5 mins is  21.98%. viz. can be just because of your running config/features used.

But, if you still think , your problem is because of High CPU then below guide will help you - 

Nexus 7000 High CPU Usage Troubleshooting Guide

http://www.cisco.com/c/en/us/support/docs/interfaces-modules/nexus-7000-series-supervisor-1-module/116137-trouble-nexus7000-highcpu-00.html#anc29

CreatePlease to create content
Content for Community-Ad