05-26-2013 09:15 AM - edited 03-07-2019 01:33 PM
Dear All,
we are suffering from high CPU utilization with our Nexus switches in a persiodic manaer
please find the attached file contain output of
1- show process cpu hisotry
2- show process cpu sort
as i found AAA is consuming CPU cycles as attached file clarify
please help
Solved! Go to Solution.
05-26-2013 09:50 PM
Hi Ali,
Thanks for the information. Is there any management/snmp-enabled station which polls this switch (in any vdc)? This may be for example Cisco DCNM solution. spikes in aaa may be caused by regular polling of accounting log by snmp management application.
If you request output of aaa accounting log from N7k switch and this log is big enough, this may cause spike in aaa process.
As a workaround could you turn off DCNM-initiated snmp polling of N7k switch (for some time) to see if it makes any difference.
HTH
Regards
Inayath
05-26-2013 09:27 AM
Hi Ali,
Please furnish the following outputs:
show system resource ---> check CPU (as a %age) and memory resources. Load is defined as the number of running processes
show process cpu sort
show process cpu history
show system internal processes cpu ---> equivalent to Linux “TOP” tool. System processes across all VDC’s. Use to cross check “show system resource” accuracy
show system internal processes cpu
show system internal sysmgr service
Regards
Inayath
05-26-2013 09:59 AM
05-26-2013 09:50 PM
Hi Ali,
Thanks for the information. Is there any management/snmp-enabled station which polls this switch (in any vdc)? This may be for example Cisco DCNM solution. spikes in aaa may be caused by regular polling of accounting log by snmp management application.
If you request output of aaa accounting log from N7k switch and this log is big enough, this may cause spike in aaa process.
As a workaround could you turn off DCNM-initiated snmp polling of N7k switch (for some time) to see if it makes any difference.
HTH
Regards
Inayath
05-27-2013 12:06 AM
Dear Inayath,
Thanks for your help,
We already has DCNM monitoring our N7k , i also thuoght it may be DCNM so i did shutted down 10 hours ago and i find that CPU overall value decrease, but it is still high, i think it is a good start, but i don`t know what else may cause these CPU utilization values
do you have any ideas
Ali
05-27-2013 12:10 AM
Hi Ali,
So whats the current situtation ? Can you attach the latest outputs?
Also I would like to know the firmware running on the Nexus box? Show version outputs plz?
Regards
Inayath
05-27-2013 02:29 AM
Dear Inayath,
Now the CPU utilization values and version are as below ,
Nexus# show process cpu hi
12 1 1 11
766433471142343444333242334339453532559343424653365374376649
100
90
80
70
60
50
40
30
20 # # ##
10 ### ### # # # #### ## ## # ### #
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)
# = average CPU%
222224222122222222221222222422222324222222222221222232232222
463133179953433553319266426188137020601380223427000706333441
100
90
80
70
60
50
40 * * *
30 * * ** * ** ** **** ** ** * *** *
20 ************************************************************
10 ####################*#####*####*#**###########*#############
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
355545455555477696666558866665665656666566655666666589656666666666666666
715726822225323781532896171508118093732800198100675871490105227300611766
100 *
90 * * **
80 * ** **
70 **** * *** * * *** ** * * * ***
60 ** * * ***********************************************************
50 *** ******* ***********************************************************
40 ************************************************************************
30 ************************************************************************
20 ************************************************************************
10 ########################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%
Nexus# show ver
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Documents: http://www.cisco.com/en/US/products/ps9372/tsd_products_support_series_home.html
Copyright (c) 2002-2011, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php
Software
BIOS: version 3.22.0
kickstart: version 5.1(5)
system: version 5.1(5)
BIOS compile time: 02/20/10
kickstart image file is: bootflash:///n7000-s1-kickstart.5.1.5.bin
kickstart compile time: 12/25/2020 12:00:00 [09/18/2011 00:37:55]
system image file is: bootflash:///n7000-s1-dk9.5.1.5.bin
system compile time: 8/5/2011 13:00:00 [09/18/2011 01:23:19]
Hardware
cisco Nexus7000 C7010 (10 Slot) Chassis ("Supervisor module-1X")
Intel(R) Xeon(R) CPU with 4109560 kB of memory.
Processor Board ID JAF1507CCPR
Device name: Nexus
bootflash: 2029608 kB
slot0: 0 kB (expansion flash)
Kernel uptime is 162 day(s), 16 hour(s), 49 minute(s), 46 second(s)
Last reset
Reason: Unknown
System version: 5.1(5)
Service:
plugin
Core Plugin, Ethernet Plugin
CMP (Module 5) ok
CMP Software
CMP BIOS version: 02.01.05
CMP Image version: 5.1(1) [build 5.0(0.66)]
CMP BIOS compile time: 8/ 4/2008 19:39:40
CMP Image compile time: 8/5/2011 13:00:00
CMP (Module 6) ok
CMP Software
CMP BIOS version: 02.01.05
CMP Image version: 5.1(1) [build 5.0(0.66)]
CMP BIOS compile time: 8/ 4/2008 19:39:40
CMP Image compile time: 8/5/2011 13:00:00
Nexus#
Really thanks for your help
Ali
05-27-2013 04:35 AM
Hi Ali,
It looks good now. I would also consider in upgrading the firmware to the latest one.
I think we are good to close this thread now if you dont have any further queries on this.
HTH
Regards
Inayath.
**Please rate useful posts and remember to mark any solved questions as answered. Thank you.
06-17-2013 02:10 AM
Dear All,
Now after shutting DCNM down, the repeated polling stops and CPU overall decrease, but i installed LMS system that send me email notifications that CPU for Nexus reach 100%, i am not able to find out why that happens, i thuoght i may apply EEM to catch this as the below link clarify,
https://supportforums.cisco.com/docs/DOC-16708
Now my Question, is this EEM above has any impact that i should consider related to CPU ?
please help
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide