Nexus 7K high CPU utilization

[Satya Siba Sundar Mishra]

Hi,

I am trying to troubloeshoot the the High cpu utilization of N7K. The switch is connected in a POC setup and traffic load is very less. It is running with NXOS 6.0.4. For past 1 month I have observed the CPU utilization goes upto 80% without any traffic/load. I tried to capture the o/p of "show process cpu sort" through EEM script. while analyzing the o/p I can not determine the process which is contributing Th eutilization of user or kernel.

for example:

In Below out put

25.2% user, 31.0% kernel, 43.8% idle

Neither any process shown below, is using 25.2% or 31%, nor the aggrigation of all processes utilizaion is 56.2%. (I have 2 VDCs including default)

Please help me how to determine the process consuming cpu.

PID   Runtime(ms) Invoked   uSecs 1Sec   Process
----- ----------- -------- ----- ------ -----------
3296   120582933   8003274 15066   6.7% sac_usd
2924     85729099 13281994   6454   2.8% platform
3588     13699639 11361018   1205   0.9% udld
   1       42870   1269054     33   0.0% init
   2           3       308     12   0.0% kthreadd
   3         1062   167018     6   0.0% migration/0
   4       462583 133573370     3   0.0% ksoftirqd/0
   5       91988   955848     96   0.0% watchdog/0
   6         777   110442     7   0.0% migration/1
   7       325174 91256362     3   0.0% ksoftirqd/1
   8         3025   952785     3   0.0% watchdog/1
   9       36939   6700241     5   0.0% events/0
   10       30877   6396271     4   0.0% events/1
   11           0       10     19   0.0% khelper
   12         170       740   230   0.0% kblockd/0
   13         180       731   246   0.0% kblockd/1
   14            0         2     0   0.0% kacpid
   15           0         2     0   0.0% kacpi_notify
   16           28         8   3530   0.0% ksuspend_usbd
   17           0         6     3   0.0% khubd
   18       31730   778523     40   0.0% pdflush
   19       20043   569273     35   0.0% pdflush
   20           0         2     3   0.0% kswapd0
   21           0         2     3   0.0% aio/0
   22           0         2     2   0.0% aio/1
   23           0         2     2  0.0% nfsiod
   24           0         2     2   0.0% SerrLogKthread
   25       63668   279821   227   0.0% kide/0
   26       601880   1871286   321   0.0% kide/1
   29           0         4     10   0.0% rpciod/0
   30           0        17     9   0.0% rpciod/1
   31       14621   2521982     5   0.0% kirqd
652           89       22   4070   0.0% jffs2_gcd_mtd2
760       43958   208586   210   0.0% kjournald
765       31116   208347   149   0.0% kjournald
886       31349   208114   150   0.0% kjournald
1988       36029   237342   151   0.0% kjournald
2233           1       30     40   0.0% portmap
2242           3         5   696   0.0% rpc.statd
2257           0         2     11   0.0% lockd
2258           8     1728     4   0.0% nfsd
2259           8     1727     4   0.0% nfsd
2260           8     1727     4   0.0% nfsd
2261           9     1727     5   0.0% nfsd
2262           8     1728     4   0.0% nfsd
2263           8     1728     4   0.0% nfsd
2264           8     1729     4   0.0% nfsd
2265           8     1726     4   0.0% nfsd
2270           0         1   132   0.0% rpc.mountd
2290     2138348 13880127   154   0.0% sysmgr
2643           0         1     25   0.0% mping-thread
2718         7036   206917     34   0.0% redun_kthread
2920           43       224   194   0.0% xinetd
2921           50       479   104   0.0% tftpd
2922     1885755 308484299    6   0.0% syslogd
2923         181     1271   142   0.0% sdwrapd
2931           0         7     45   0.0% usd_mts_kthread
2944       16901   3100548     5   0.0% ls-notify-mts-t
3027     3660723   2618895   1397   0.0% cmond
3028        2900       625   4640   0.0% pfm_dummy
3035       19653   780705     25   0.0% vshd
3036           62       17   3669   0.0% smm
3037       53404   1650665     32   0.0% session-mgr
3038       94080   6211601     15   0.0% psshelper
3039         6963   103812     67   0.0% lmgrd
3040       65621   1441087     45   0.0% licmgr
3041         5362   139249     38   0.0% fs-daemon
3042       11122   520200     21   0.0% feature-mgr
3043         206       689   299   0.0% confcheck
3044       25743   826190     31   0.0% capability
3045       94664   6211930     15   0.0% psshelper_gsvc
3053       33425   207353   161   0.0% cisco
3056       40788   638596     63   0.0% clis
3057       59461   1636992     36   0.0% port-profile
3059       49041   2072373     23   0.0% xmlma
3060       53345   1696294     31   0.0% vmm
3061       48536   1719085     28   0.0% vdc_mgr
3062       43610   3106747     14   0.0% ttyd
3063       10612   628063     16   0.0% sysinfo
3064         293       486   603   0.0% sksd
3066         3262   123522     26   0.0% res_mgr
3067       134609   6231106     21   0.0% plugin
3068       467226   1380087   338   0.0% plog_sup
3069        128       15   8544   0.0% npacl
3070       18252   520815     35   0.0% mvsh
3071           26       75   347   0.0% mping_server
3072       159427   1250958   127   0.0% idehsd
3073       16671   522586     31   0.0% evms
3074        13941   397828     35   0.0% evmc
3075     49911293 16123404   3095   0.0% diagmgr
3076       45315   502617     90   0.0% crdcfg_server
3077         331     1017   325   0.0% core-dmon
3078       171639   1761631     97   0.0% cmpproxy
3079       497384   4187027   118   0.0% clk_mgr
3080       11388   621260     18   0.0% bloggerd
3081       22013   1242095     17   0.0% bios_daemon
3082       18112   777689     23   0.0% ascii-cfg
3083       111428   6210101     17   0.0% securityd
3084       98387   6211128     15   0.0% cert_enroll
3085       94987   6214320     15   0.0% aaa
3088       18690   777305     24   0.0% obfl
3090           85       28   3046   0.0% l3vm
3091         127        34   3752   0.0% urib
3093     33075502   6237681   5302   0.0% diagclient
3110           74       34   2196   0.0% adjmgr
3111           97       28   3481   0.0% u6rib
3120       32642   241081   135   0.0% aclmgr
3122         109       76   1446   0.0% arp
3124           91       20   4566   0.0% icmpv6
3161       33124   217673   152   0.0% kjournald
3169     2245769 64712179     34   0.0% mts-sync-thr
3242           3       87     38   0.0% dhcpd
3245      53250   1299891     40   0.0% cardclient
3246       409624   8403407     48   0.0% device_test
3247       55546   577909     96   0.0% xbar
3249     1302033   7717061   168   0.0% module
3254         208       398   523   0.0% ExceptionLog
3255       50295   3108982     16   0.0% bootvar
3256       27871   765502     36   0.0% ifmgr
3259         208       94   2218   0.0% netstack
3276       176505   6215803     28   0.0% radius
3279     1714777   4982359   344   0.0% cfs
3281           22       41   542   0.0% ip_dummy
3282           33       42   791   0.0% ipv6_dummy
3283       51426   738103     69   0.0% ntp
3284       54365   1727529     31   0.0% otm
3285           23       43   543  0.0% pktmgr_dummy
3286     1246677   4547384   274   0.0% snmpd
3287           23       44   544   0.0% tcpudp_dummy
3289       340737   3019215   112   0.0% cdp
3290           83       260   320   0.0% dcos-xinetd
3292       281988 11890167     23   0.0% ntpd
3293     11903804 14652688   812   0.0% R2D2_usd
3294       292360   3153368     92   0.0% arbiter
3295     1505005   3724037   404   0.0% babycaesar
3297     41721542 12143605   3435   0.0% sensor
3298     2892940   4943312   585   0.0% xbar_driver_usd
3317       115183   6208629     18   0.0% callhome
3358     54287184 13950490   3891   0.0% oc_usd
3359     3092574 15907454   194   0.0% metropolis_usd
3372           0         2     4   0.0% kauditd
3471       22801   1240441     18   0.0% sctpt_rx_thr
3472           0         1     5   0.0% sctpt_tx_thr
3508       25446   826409     30   0.0% u2
3509       28225   206525   136   0.0% spm
3510       55102   1725410    31   0.0% sal
3511         329     2197   149   0.0% rpm
3512       568064   2990164   189   0.0% pltfm_config
3513       55064   1715291     32   0.0% pixmc
3514       68105   1715304     39   0.0% pixm
3515       366788   1522226    240   0.0% pfstat
3516         475     2030   234   0.0% mrib
3517       53274   1720790     30   0.0% monitorc
3518       65252   1572542     41   0.0% mcm
3519         260     1059   245   0.0% m6rib
3522     3000740   7577947    395   0.0% l2pt
3524       28816   829700     34   0.0% eltm
3526       305549   416012   734   0.0% copp
3527       74173   1897493     39   0.0% acllog
3532       144421   1631647     88   0.0% mfdm
3533       63805   1638615     38   0.0% monitor
3534       19748   404008     48   0.0% m2rib
3536       105922   1906312     55   0.0% ufdm
3553         166       73   2286   0.0% igmp
3558         170       123   1384   0.0% mcastfwd
3559       30802   783951    39   0.0% eth_port_channel
3560       23603   689775     34   0.0% vlan_mgr
3569         3192   110681     28   0.0% eth_dstats
3570       249041   593924   419   0.0% ipqosmgr
3571       838078   3413023   245   0.0% lacp
3577      19102   203449     93   0.0% dhcp_snoop
3578     2855782 12674401   225   0.0% diag_port_lb
3579       39560   794428     49   0.0% ethpm
3580     2084026   4060385   513   0.0% l2fm
3581       46520   907379     51   0.0% nfm
3582         9639   249890     38   0.0% vtp
3589     1260301 17011436     74   0.0% vpc
3590     11393886 107080504   106   0.0% stp
3591       17574   366700     47   0.0% interface-vlan
3607       25561   826111     30   0.0% port_client
3609     17407279 10686606   1628   0.0% statsclient
3610       23619   650313     36   0.0% xbar_client
3627       55993   6205583     9   0.0% wdpunch_thread
3705           5       19   289   0.0% gettylogin1
3731           1        4   410   0.0% getty
4496           37       80   466   0.0% login
4591           8       60   135   0.0% klogd
4615       213619   8202090     26   0.0% ntpd
8405         1236     5666   218   0.0% vsh
12066           3        2   1535   0.0% vsh
12067           99         8 12416   0.0% telnet
12068         358     4626     77   0.0% dcos-telnet
12069         257     2980     86   0.0% in.dcos-telnetd
12086           35       42   839   0.0% login
12099        340       815   418   0.0% vsh
12673           3         3   1082   0.0% vsh
12696           2         4   578   0.0% vdclogin
12699           15         2   7789   0.0% login
12700         222       909   244   0.0% vsh
13059          3         7   510   0.0% mvsh
13060         179       35   5118   0.0% vsh
13096           9       210     43   0.0% vsh
13099           3         3   1197   0.0% mvsh
13100         136       45   3033   0.0% vsh
13103           8        9   987   0.0% tacacsd
13106           2         3   712   0.0% vsh
13107           39         3 13146   0.0% ps
17076       260337   6217630     41   0.0% tacacs

CPU util :   25.2% user,   31.0% kernel,   43.8% idle
Please note that only processes from the requested vdc are shown above

[Raphael]

Did you find the cause of the high CPU? If yes, what was it and how could you fix it?

Could you maybe provide a EEM script that you used for getting the info? Did you monitor for the CPU load of the sup or of the linecards itself?

[sbhadrav@cisco.com]

Netstack is a complete IP stack implemented in the user space of Nexus 7000. Components include a L2 Packet Manager, ARP, Adjacency Manager, IPv4, Internet Control Message Protocol v4 (ICMPv4), IPv6, ICMPv6, TCP/UDP, and socket library. When traffic to the CPU is triggering high CPU usage, you often see that Netstack and its respective process are running high.

I can see Avg CPU consumed by "Pktmgr" over 5 mins is  21.98%. viz. can be just because of your running config/features used.

But, if you still think , your problem is because of High CPU then below guide will help you - 

Nexus 7000 High CPU Usage Troubleshooting Guide

http://www.cisco.com/c/en/us/support/docs/interfaces-modules/nexus-7000-series-supervisor-1-module/116137-trouble-nexus7000-highcpu-00.html#anc29