cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
3741
Views
0
Helpful
4
Replies

Nexus 3064 High CPU utilization

Johnson Lew
Level 1
Level 1

We have many Nexus 3064 switches that have high CPU utilization.  The graph minute and hourly graphs show it.  We are on version 6.0(2)U6(6).  What can be the root cause and how do I fix it?

 

# sh proc cpu s | ex 0.0
Please note that only processes from the requested vdc are shown above

PID Runtime(ms) Invoked uSecs 1Sec Process
----- ----------- -------- ----- ------ -----------
3408 835847866 2147483647 372 11.05% bcm_usd
3372 1164350935 2079692536 559 8.64% syslogd
3759 625831156 2147483647 675 4.80% ntpd
3445 85645072 92374639 927 3.34% statsclient
3634 362 586 618 1.45% netstack
4 35623629 346157466 102 0.94% ksoftirqd/0
3514 36494770 172794685 211 0.50% snmpd
3572 198 1498 132 0.50% arp
CPU util : 24.50% user, 54.00% kernel, 21.50% idle
acc8-qtl1-atl# show process cpu history

677887867676677666897766876776667777766777878677787865768677
451754152337691927413066357422270377252153135787343503536448
100
90 ## # # # #
80 # ## # # ## ## ## # # # ## # # # # #
70 ######## ###### ############ ####### ############# # # ##
60 ##################################################### ######
50 ############################################################
40 ############################################################
30 ############################################################
20 ############################################################
10 ############################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5

CPU% per second (last 60 seconds)
# = average CPU%

 

999999999999889888998988998989999889899989998998998889998999
201113003354690789408479229093405994842090328109108984029115
100 * * *
90 ************************************************************
80 ****#*********#*********#*****#*#**********##*##**********#*
70 ############################################################
60 ############################################################
50 ############################################################
40 ############################################################
30 ############################################################
20 ############################################################
10 ############################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5

CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%

 

999999999999999999999999999999999999999999999999999999999999999999999999
554666475555566655565577565555544556556554656546455645545554644665656554
100 ** *** ************************ ******** **** * *** ** *** * ********
90 ************************************************************************
80 ************************************************************************
70 ########################################################################
60 ########################################################################
50 ########################################################################
40 ########################################################################
30 ########################################################################
20 ########################################################################
10 ########################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0

CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%

Software
BIOS: version 2.8.0
loader: version N/A
kickstart: version 6.0(2)U6(6)
system: version 6.0(2)U6(6)
Power Sequencer Firmware:
Module 1: version v1.0
SFP uC: version 2.12
BIOS compile time: 12/22/2015
kickstart image file is: bootflash:///n3000-uk9-kickstart.6.0.2.U6.6.bin
kickstart compile time: 2/16/2016 1:00:00 [02/16/2016 02:01:17]
system image file is: bootflash:///n3000-uk9.6.0.2.U6.6.bin
system compile time: 2/16/2016 1:00:00 [02/16/2016 02:27:15]


Hardware
cisco Nexus 3064 Chassis ("48x10GE + 4x40G Supervisor")
Intel(R) Celeron(R) CPU P4505 @ 1.87GHz with 3665240 kB of memory.
Processor Board ID FOC1636627J

Device name: acc8-qtl1-atl
bootflash: 2007040 kB

Kernel uptime is 114 day(s), 3 hour(s), 24 minute(s), 11 second(s)

1 Accepted Solution

Accepted Solutions

It was the enabled ntp feature that caused the high 70+ average percent CPU utilization.  I disabled it per your item no. 1 and it was drastically reduced.  

Thank you very much for your help!

# show process cpu history

1 2 11111211111121111212321 21 1 11 1 121
908862758657999673796687956808653593429397877656309881565018
100
90
80
70
60
50
40
30 # # ###
20 # ################ ###### # # ##
10 ################################### ### ####################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5

CPU% per second (last 60 seconds)
# = average CPU%

 

352999998989999989889999989998889988889989989999999988888999
321145109340121461966113380219961079791180343133201178577402
100 * *
90 ******* ******************************** ****************
80 **************#**#***#*******#****#*******###*******#****
70 *########################################################
60 *########################################################
50 * #########################################################
40 * #########################################################
30 ** #########################################################
20 ***#########################################################
10 ############################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5

CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%

 

999999999999999999999999999999999999999999999999999999999999999999999999
555655755656465556565654654436554446547655557556555546455675654445655455
100 ************ ********** ** *** ** ************** * ******* **** **
90 ************************************************************************
80 ************************************************************************
70 ###########################################*############################
60 ########################################################################
50 ########################################################################
40 ########################################################################
30 ########################################################################
20 ########################################################################
10 ########################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0

CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%

View solution in original post

4 Replies 4

balaji.bandi
Hall of Fame
Hall of Fame

Not sure this could be bug :

 

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuh79034/?rfs=iqvred

 

but can you post more information -

 

show processes cpu sort | ex 0.00

show policy-map interface control-plane
how spanning-tree detail | inc ieee|occurr|from

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

Johnson Lew
Level 1
Level 1

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuh79034/?rfs=iqvred

The bug referenced in CSCuh79034 was fixed in an earlier version than the version I use.

More info is posted below.

 

but can you post more information -

 

show processes cpu sort | ex 0.00
PID Runtime(ms) Invoked uSecs 1Sec Process
----- ----------- -------- ----- ------ -----------
3408 866822016 2147483647 316 9.19% bcm_usd
3372 1230314437 2147483647 558 8.69% syslogd
3759 660383384 2147483647 349 4.84% ntpd
3634 362 586 618 1.45% netstack
4 37975628 363966294 104 0.94% ksoftirqd/0
7 47849128 362440130 132 0.50% ksoftirqd/1
3374 13465786 57388247 234 0.50% pfma
3572 198 1498 132 0.50% arp

show policy-map interface control-plane

# show policy-map interface control-plane
Control Plane

service-policy input: copp-system-policy

class-map copp-s-selfIp (match-any)
police pps 500
OutPackets 3948273
DropPackets 0
class-map copp-s-default (match-any)
police pps 400
OutPackets 18
DropPackets 0
class-map copp-s-l2switched (match-any)
police pps 200
OutPackets 6185067
DropPackets 5920
class-map copp-s-ping (match-any)
match access-group name copp-system-acl-ping
police pps 100
OutPackets 1305749
DropPackets 0
class-map copp-s-l3destmiss (match-any)
police pps 100
OutPackets 0
DropPackets 0
class-map copp-s-glean (match-any)
police pps 500
OutPackets 0
DropPackets 0
class-map copp-s-l3mtufail (match-any)
police pps 100
OutPackets 0
DropPackets 0
class-map copp-s-ttl1 (match-any)
police pps 100
OutPackets 3293
DropPackets 0
class-map copp-s-ipmcmiss (match-any)
police pps 400
OutPackets 0
DropPackets 0
class-map copp-s-l3slowpath (match-any)
police pps 100
OutPackets 0
DropPackets 0
class-map copp-s-dhcpreq (match-any)
match access-group name copp-system-acl-dhcps6
police pps 300
OutPackets 0
DropPackets 0
class-map copp-s-dhcpresp (match-any)
match access-group name copp-system-acl-dhcpc6
match access-group name copp-system-dhcp-relay
police pps 300
OutPackets 0
DropPackets 0
class-map copp-s-dai (match-any)
police pps 300
OutPackets 0
DropPackets 0
class-map copp-s-igmp (match-any)
match access-group name copp-system-acl-igmp
police pps 400
OutPackets 75
DropPackets 0
class-map copp-s-routingProto2 (match-any)
match access-group name copp-system-acl-routingproto2
police pps 1300
OutPackets 2949711
DropPackets 0
class-map copp-s-v6routingProto2 (match-any)
match access-group name copp-system-acl-v6routingProto2
police pps 1300
OutPackets 0
DropPackets 0
class-map copp-s-eigrp (match-any)
match access-group name copp-system-acl-eigrp
match access-group name copp-system-acl-eigrp6
police pps 200
OutPackets 0
DropPackets 0
class-map copp-s-pimreg (match-any)
match access-group name copp-system-acl-pimreg
police pps 200
OutPackets 0
DropPackets 0
class-map copp-s-pimautorp (match-any)
police pps 200
OutPackets 0
DropPackets 0
class-map copp-s-routingProto1 (match-any)
match access-group name copp-system-acl-routingproto1
match access-group name copp-system-acl-v6routingproto1
police pps 1000
OutPackets 22611213
DropPackets 0
class-map copp-s-arp (match-any)
police pps 200
OutPackets 730097149
DropPackets 22372078
class-map copp-s-ptp (match-any)
police pps 1000
OutPackets 0
DropPackets 0
class-map copp-s-vxlan (match-any)
police pps 1000
OutPackets 0
DropPackets 0
class-map copp-s-bfd (match-any)
police pps 350
OutPackets 23
DropPackets 0
class-map copp-s-bpdu (match-any)
police pps 12000
OutPackets 15489073
DropPackets 0
class-map copp-s-dpss (match-any)
police pps 1000
OutPackets 0
DropPackets 0
class-map copp-s-mpls (match-any)
police pps 100
OutPackets 0
DropPackets 0
class-map copp-icmp (match-any)
match access-group name copp-system-acl-icmp
police pps 200
OutPackets 3936540
DropPackets 998355
class-map copp-telnet (match-any)
match access-group name copp-system-acl-telnet
police pps 500
OutPackets 48541
DropPackets 0
class-map copp-ssh (match-any)
match access-group name copp-system-acl-ssh
police pps 500
OutPackets 529097
DropPackets 0
class-map copp-snmp (match-any)
match access-group name copp-system-acl-snmp
police pps 500
OutPackets 5318742
DropPackets 0
class-map copp-ntp (match-any)
match access-group name copp-system-acl-ntp
police pps 100
OutPackets 794005338
DropPackets 3131999660
class-map copp-tacacsradius (match-any)
match access-group name copp-system-acl-tacacsradius
police pps 400
OutPackets 193905
DropPackets 0
class-map copp-stftp (match-any)
match access-group name copp-system-acl-stftp
police pps 400
OutPackets 493
DropPackets 0

show spanning-tree detail | inc ieee|occurr|from
Number of topology changes 239982 last change occurred 0:00:12 ago
from port-channel1

  

Sergiu.Daniluk
VIP Alumni
VIP Alumni

Hi @Johnson Lew :

By the look at the outputs:

 

class-map copp-s-arp (match-any)
police pps 200
OutPackets 730097149
DropPackets 22372078

class-map copp-ntp (match-any)
match access-group name copp-system-acl-ntp
police pps 100
OutPackets 794005338
DropPackets 3131999660

show spanning-tree detail | inc ieee|occurr|from
 Number of topology changes 239982 last change occurred 0:00:12 ago
from port-channel1

There are a couple of things:

 

1. Seems like the switch receives a loooooot of NTP packets to the CPU.

Do ethanalyzer to see which device is flooding the device with NTP (ctrl+c to stop it):

 

ethanalyzer local interface inband display-filter ntp limit-captured-frames 0

Find the source IP and make it stop.

 

If you are not able to stop it, you can disable the NTP feature

 

conf t
no feature ntp

2. There are also some ARPs which are busting your switch. Check ethanalyzer again, with "arp" filter. If there are too many requests/replies from a single source, try fix it as well.

 

3. STP is unstable. Continue with the same command (show spanning-tree detail | inc ieee|occurr|from) on the switches connected to interface listed in the output, until you find the source of the topology changes (most likely an interface flapping).

 

Stay safe,

Sergiu

It was the enabled ntp feature that caused the high 70+ average percent CPU utilization.  I disabled it per your item no. 1 and it was drastically reduced.  

Thank you very much for your help!

# show process cpu history

1 2 11111211111121111212321 21 1 11 1 121
908862758657999673796687956808653593429397877656309881565018
100
90
80
70
60
50
40
30 # # ###
20 # ################ ###### # # ##
10 ################################### ### ####################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5

CPU% per second (last 60 seconds)
# = average CPU%

 

352999998989999989889999989998889988889989989999999988888999
321145109340121461966113380219961079791180343133201178577402
100 * *
90 ******* ******************************** ****************
80 **************#**#***#*******#****#*******###*******#****
70 *########################################################
60 *########################################################
50 * #########################################################
40 * #########################################################
30 ** #########################################################
20 ***#########################################################
10 ############################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5

CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%

 

999999999999999999999999999999999999999999999999999999999999999999999999
555655755656465556565654654436554446547655557556555546455675654445655455
100 ************ ********** ** *** ** ************** * ******* **** **
90 ************************************************************************
80 ************************************************************************
70 ###########################################*############################
60 ########################################################################
50 ########################################################################
40 ########################################################################
30 ########################################################################
20 ########################################################################
10 ########################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0

CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: