Nexus 3k High cpu utilization

[vinayjaiswal]

Hi Guys,

We are facing high cpu utilization issue on nexus 3k switch (Nexus 3048 Chassis)  due to netstack and bcm_usd processes. We observed cpu upto 70% due to these processes.

>> show version:

Software
  BIOS:      version 1.4.0
  loader:    version N/A
  kickstart: version 6.0(2)U3(7)
  system:    version 6.0(2)U3(7)
  Power Sequencer Firmware:
             Module 1: version v4.4
  BIOS compile time:       12/09/2013
  kickstart image file is: bootflash:///n3000-uk9-kickstart.6.0.2.U3.7.bin
  kickstart compile time:  2/19/2015 2:00:00 [02/19/2015 12:44:40]
  system image file is:    bootflash:///n3000-uk9.6.0.2.U3.7.bin
  system compile time:     2/19/2015 2:00:00 [02/19/2015 17:59:10]

>> 'show processes cpu history`
                                                                
      356555554212412                    1   1                  
    679714933882539105665646468666876567726570466556636883778855
100                                                             
 90                                                             
 80                                                             
 70                                                             
 60    ## #  #                                                  
 50    ########   #                                             
 40   #########   #                                             
 30   #########   #                                             
 20   ############# #                                           
 10 ###################### # ################# ###### ### ######
    0....5....1....1....2....2....3....3....4....4....5....5....
              0    5    0    5    0    5    0    5    0    5    

               CPU% per second (last 60 seconds)
                      # = average CPU%


                                                                
    11211111 2441444111111 11111111 111111 11111111 1 11111211 1
    206321709701483111011692300210281312109600301319391221371590
100                                                             
 90                                                             
 80                                                             
 70                                                             
 60                                                             
 50              *                                              
 40           ** ***                                            
 30   *      *#* *#*                                       *    
 20   *   *  *#* *#*     *                 *               * *  
 10 ############################################################
    0....5....1....1....2....2....3....3....4....4....5....5....
              0    5    0    5    0    5    0    5    0    5    

               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%


                                                                            
    213444545555743221213136735657886589875552231733777566858677957116261224
    766037227470352568784544567299921911681879039413535763780904272673839868
100                                                                         
 90                               *    **                 *     *           
 80                         *    ***  ****          * *   * *   *           
 70             *           *    ***  ****       *  *** * * ***** *         
 60         * * *          ** ************ **    *  ***************  * *    
 50      ** ******         ** ***************    *  ***************  * *   *
 40   ************         ******************    *  ***************  * *   *
 30 * *************** * * ******************** * ******************  *** ***
 20 ************************************##********************#*#***********
 10 ########################################################################
    0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
              0    5    0    5    0    5    0    5    0    5    0    5    0

                   CPU% per hour (last 72 hours)
                  * = maximum CPU%   # = average CPU%

Is there any bug with current Nx-OS version?  how to fix this issue.

Regards,

Vijay

[chintan_shah]

Hi, For bcm_usd process there was one bug CSCuh79034 which affected 5.0(3)U5(1d) release however this should be fixed with your current 6.0(2)U3(7) version.

Opening the TAC case will be your best bet.

[vinayjaiswal]

Hi Chintan,

Appreciate for your reply.

Can we go for NX-OS upgrade with latest version ie 6.0(2)U6(10) to fix this issue.

Regards,

Vijay

[Andrea Testino]

Hi there,

Per your graph, your average CPU utilization per minute and per hour is at a 10% average so you have nothing to worry about here:

    11211111 2441444111111 11111111 111111 11111111 1 11111211 1
    206321709701483111011692300210281312109600301319391221371590
100                                                             
 90                                                             
 80                                                             
 70                                                             
 60                                                             
 50              *                                              
 40           ** ***                                            
 30   *      *#* *#*                                       *    
 20   *   *  *#* *#*     *                 *               * *  
 10 ############################################################
    0....5....1....1....2....2....3....3....4....4....5....5....
              0    5    0    5    0    5    0    5    0    5    

               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%


                                                                            
    213444545555743221213136735657886589875552231733777566858677957116261224
    766037227470352568784544567299921911681879039413535763780904272673839868
100                                                                         
 90                               *    **                 *     *           
 80                         *    ***  ****          * *   * *   *           
 70             *           *    ***  ****       *  *** * * ***** *         
 60         * * *          ** ************ **    *  ***************  * *    
 50      ** ******         ** ***************    *  ***************  * *   *
 40   ************         ******************    *  ***************  * *   *
 30 * *************** * * ******************** * ******************  *** ***
 20 ************************************##********************#*#***********
 10 ########################################################################
    0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
              0    5    0    5    0    5    0    5    0    5    0    5    0 

One common mistake we see is generally a focus on the "per second" graph which does not truly reflect the status of your CPU - Recall that NX-OS is a Linux based system and thus has a preemptive scheduler that allows fair access to CPU resources for all processes. What this means is that any given process can take full advantage of an idle CPU in order to complete a single task much faster. Therefore, the "per second" history option on "show processes cpu history" will usually report CPU spikes that do not indicate a problem.

This behavior is also documented on our Nexus 7000 High CPU Troubleshooting guide but does apply to all Nexus platforms 

Hope that helps!

- Andrea

[vinayjaiswal]

Thanks Andrea,

Here is some sample logs.

VIMAN-NAGAR-CORE# show processes cpu sort | ex 0.00

PID    Runtime(ms)  Invoked   uSecs  1Sec    Process
-----  -----------  --------  -----  ------  -----------
 3291   1874779531  2147483647    550   25.5%  bcm_usd
 3478    107987418  1490625501     72    2.5%  ipfib
 3394          121        29   4189    2.0%  adjmgr
 3481          411      2521    163    1.5%  netstack
 3561          157        51   3091    1.5%  rpm
    4     10832928  589110362     18    0.5%  ksoftirqd/0
 3431          189       936    202    0.5%  arp

pls
    1       325764   5399810     60    0.0%  init
    2            3       322     11    0.0%  kthreadd
    3         2877    559654      5    0.0%  migration/0
    5       201257   3338846     60    0.0%  watchdog/0
    6         2256    426697      5    0.0%  migration/1
    7     10713556  385311488     27    0.0%  ksoftirqd/1
    8        20054   3338850      6    0.0%  watchdog/1
    9       136518  22220999      6    0.0%  events/0
   10      1657840  32950545     50    0.0%  events/1
   11            1        61     20    0.0%  khelper
   12           13        15    885    0.0%  netns
   13            6       661      9    0.0%  kblockd/0
   14            5       374     13    0.0%  kblockd/1
   15            0         2      0    0.0%  kacpid
   16            0         2      0    0.0%  kacpi_notify
   17            0         7     18    0.0%  ksuspend_usbd

Pls suggest.

Regards,

Vijay

[Andrea Testino]

Vijay,

Again, the output you are sharing is a "per second" snapshot of processes that are running. The baseline for your chassis (the average) is at a solid 10% for both per minute and per hour. In other words, there is no high CPU on your Nexus 3000.

NX-OS is Linux based and will utilize as much of the CPU as it can -per second- to complete any given task - Per second graphs/outputs should not be a point of concern unless your per minute/per hour outputs are also showing a high average which is not the case here. (More info on my previous post regarding this behavior).

Thanks,

- Andrea

[jkilleda]

Hello vinay,

 Nexus 3k High cpu utilization

High process due to BCM_USD is usually caused due to a bug , check if the issue prolongs

[jkilleda]

Hello vinay,

Nexus 3k High cpu utilization
High process due to BCM_USD is usually caused due to bug you can even check if the issue prolongs