Some help with High CPU utilization

[Thiago Henriques]

Folks,

     In some circumstances when we had 100% cpu utilization we ran 'show proc cpu'  and one process called "IP Host Track Pr" has shown some numbers in matter of utilization. It looks like this process only shows utilization higher than zero when CPU is high. Does any of you know WHAT PROCESS IS THIS?? I was told that this process is related to ip-source tracking, however we DO NOT have ip-source tracking enabled!!

Thanks in advance,

Thiago Henriques

[nkarpysh]

Hello,

I'm honestly not sure about that process. However as far as I have understood you see CPU up to 100% with this process holding just few percents. Am I right?

If so CPU can be driven by interrupts (traffic punted to processor instead of being HW switched). You can confirm it doing "show proc cpu sorted" and see if the highest process is quite low and also it will show you the total CPU value as X%/Y%. Where X - is total CPU and Y are the interrupts.

All the processes take X-Y=Z% CPU, so that can explain that all the proceeses are low but CPU is up to 100%.

To check what is the traffic  to processor you will need to check for interfaces and also do the CPU SPAN. One of the safe ways to do it is the following debug indeed that debug is safe to use when CPU is high):

- debug netdr cap rx

- show netdr cap

it will capture packets punted to CPU to special buffer which you will be able to read with show command. And you will see which packets are sent to processor instead of being HW switched and check those for root cause.

Hope this helps,

Nik

[Thiago Henriques]

Nik,

     The problem is because this "IP Host Track Tr" shows ONLY when we noticed high CPU. Under normal traffic pattern, cpu around 30~50% this "IP Host Track Tr" is 0.

Any ideas?

Thanks,

Thiago Henriques

[nkarpysh]

I would still stay for the same above. Ip host track can accompany traffic punted to CPU and can get minor cycles not affecting performance.

Please send "show proc cpu sort" during the problem to check that.

Nik

[ciscokid6969]

Hi there, i wanted to post that I haev the exact same issue but I am having a hard time finding information. We have a stack of 2960x switch (7 switches) and most of the time its at 99% CPU. When you add up all the process totals it is around 60% CPU. We reloaded the stack last night and its now down at 50% CPU which is great, however again, when you add all the processes together, you only get about half that   25% or so.

It was far worse yesterday and the last week with the 72 hour graph being all 99% or 100%. The reload seems to have temporarily solved the issue, but we are worried it will reoccur. We are running the latest firmware "15.2(5b)E             C2960X-UNIVERSALK9-M".

I would LOVE to try the command listed above, and this is why i am posting here, but the command

- debug netdr cap rx

- show netdr cap

does not seem to work anymore. Is there an equivalent command? how can i find these packets that are being switched in CPU instead of in hardware?

The other switches we have (some stacks as well) do not have seem to have this problem. I was thinking it was like traffic related, but from the "troubleshooting high cpu usage" whitepaper that cisco provides, the interrupt percentage is low, which indicates its not a network traffic problem (such as a loop).

please let me know if anyone has more info on how i can diagnose this problem. I believe the debug netdr command would be perfect however it does not work.