Detect high CPU with SNMP 3650/3850

[fsvensson]

Hello!

We have a problem with 3650 and 3850 with high CPU, the problem is that we cant detect it in our management systems until the CPU is so high that the switches is not responding to snmp. 

Our management systems is only reading the output on "sh process cpu platform" which is very different from "sh process cpu".

Does anyone has an idea how to be able to detect this? 

Example:

switch#sh processes cpu sort
CPU utilization for five seconds: 94%/27%; one minute: 95%; five minutes: 95%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
274 59773246 4506120 13264 37.64% 36.07% 36.31% 0 SISF Switcher Th
377 43504784 4954399 8781 28.89% 28.67% 28.75% 0 SISF Main Thread

switch#sh processes cpu plat sorted
CPU utilization for five seconds: 67%, one minute: 54%, five minutes: 52%
Core 0: CPU utilization for five seconds: 50%, one minute: 55%, five minutes: 52%
Core 1: CPU utilization for five seconds: 51%, one minute: 53%, five minutes: 52%
Core 2: CPU utilization for five seconds: 46%, one minute: 53%, five minutes: 51%
Core 3: CPU utilization for five seconds: 59%, one minute: 56%, five minutes: 52%
Pid PPid 5Sec 1Min 5Min Status Size Name
--------------------------------------------------------------------------------
3397 2793 96% 95% 96% R 2135900160 linux_iosd-imag
18694 17000 90% 89% 88% S 2625302528 fed main event

[balaji.bandi]

Not sure you hitting with Bug. Do you have IPv6 DHCP running or required in the network ? if not follow below steps to resolve the issue :

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvk32439/?rfs=iqvred

Workaround:
Remove all configuration related to ip dhcp snooping, run a different IOS version.
configure an IPv6 ACL to drop that traffic in the interfaces (specially the trunk/uplink ports).

ipv6 access-list DENY-IPV6
deny ipv6 any any
!
Interface gigx/x
ipv6 traffic-filter DENY-IPV6 in

[fsvensson]

Its definitly some kind of bug, disable dhcp snooping is our solution, but that wasnt the question. 

1: How can we detect high CPU since we cant read it with snmp

2: Why does cisco not sending the cpu value in snmp ? 

(disable ipv6 as mentioned isnt working, disable dhcp snooping is) 

[Leo Laohoo]

---

@fsvensson wrote:
3397 2793 96% 95% 96% R 2135900160 linux_iosd-imag

---

This is due to CSCus28046.
Raise a TAC Case and get TAC to verify this.

[fsvensson]

Thanks, but still not a solution to my question.

[Leo Laohoo]

The answer is:
1. Raise a TAC Case.
2. Upgrade to the version TAC recommends.

[fsvensson]

Please read the question again, its not related to WHY we have high CPU, its HOW we can detect it. 

[balaji.bandi]

To detect the problem and come to resolution Cisco TAC team need to investigate with you.

This technical Forum can help what best people can base on knowledge and experience.

TAC are the experts on a high level to debug the code and give you the reason why this was a High CPU since we do not have access to your device, Cisco TAC is the best for you.

If you have smartnet contract and its effecting rather than spending time over the forum, raise a TAC case as emergency for the solution.(if i were you i would do the same thing)

[fsvensson]

Can I be clearer? 

"base on knowledge and experience" is there a way to know if switches are having high CPU, or are the rest of you also sitting in the dark and waiting for an accident to happen? 

I want to be proactive and not be totally in the dark of theese things. 

[balaji.bandi]

Sure you can do many ways -

1. Send information to SYSLOG Server

2. use EEM Script to report over email or trap to SYSLOG

[Georg Pauwen]

Hello,

on a side note, you could use the script below to have the switch proactively send you an email when the CPU reaches a certain treshold (80 percent in the example, and the switch is polled every 5 seconds).

logging buffered 8192 informational
process cpu threshold type total rising 80 interval 5
!
event manager applet High_CPU authorization bypass
event syslog occurs 1 pattern "CPURISINGTHRESHOLD"
action 1.0 info type routername
action 2.0 syslog msg "High CPU utilization detected on $_info_routername"
action 3.0 mail server "201.222.223.12" to "user-to@domain.com" from "user-from@domain.com" subject "High CPU Alert on $_info_routername" body "$_syslog_msg"

[Joseph W. Doherty]

""base on knowledge and experience" is there a way to know if switches are having high CPU, or are the rest of you also sitting in the dark and waiting for an accident to happen? "

Why yes, either by polling the switch or using SNMP traps/informs or using EEM, as noted by myself and others.

BTW, as to knowledge and experience, high CPU on a switch often indicates an IOS bug or something "whacky" going on in your network (i.e. something control plane related). Remember, switches use dedicated hardware for their "normal" traffic forwarding (i.e. data plane), so high CPU is generally abnormal. Yea, I know you want to be proactive, and know of potential accidents before they happen, but again, we've described how that might be accomplished.

Also BTW, when a switch has a high CPU, often normal traffic forwarding isn't impacted, again because there's dedicated hardware for it and also because different software processes generally have different levels of priority. I.e. even with a switch's CPU at 100%, there might be no impact to traffic forwarding or critical control plane (e.g. STP processing, routing protocol processing, etc.; SNMP isn't a critical function). In other words, those of us with some knowledge and experience often find high switch CPU is often not critical, but when it is, other monitoring on/off the network often shows there's a problem. So, we're not always (totally) in the dark. ;)

[Leo Laohoo]

---

@fsvensson wrote:

its HOW we can detect it. 

---

Raise a TAC Case and ask TAC that very question.

[Joseph W. Doherty]

What it sounds like you want, is something on the switch to raise a trap or inform when CPU usage goes above a certain percentage. This link might help to get you started: https://community.cisco.com/t5/switching/configuring-snmp-trap-for-cpu-utilization/td-p/1606498

Later devices support EEM, which I recall (?) can generate SNMP traps too.

[Georg Pauwen]

Hello,

on a side note, you can proactively configure the below on the switch(es), and you will get an email from from the switch informing you of high CPU utilization. In the example, the threshold is set to 80 percent.

logging buffered 8192 informational
process cpu threshold type total rising 80 interval 5
!
event manager applet High_CPU authorization bypass
event syslog occurs 1 pattern "CPURISINGTHRESHOLD"
action 1.0 info type routername
action 2.0 syslog msg "High CPU utilization detected on $_info_routername"
action 3.0 mail server "201.222.223.12" to "user-to@domain.com" from "user-from@domain.com" subject "High CPU Alert on $_info_routername" body "$_syslog_msg"