09-09-2014 02:51 AM - edited 03-07-2019 08:41 PM
Hi,
We have one Catalyst 3750 with higher CPU utilization. ( CPU utilization for five seconds: 36%/30%; one minute: 27%; five minutes: 28% ) The CPU utilization for the other Catalyst 3750 is below 10%
We execute "sh proc cpu sort | exc 0.00" many times and find process "Check heaps" with higher utilization :
switch#sh proc cpu sort
CPU utilization for five seconds: 36%/30%; one minute: 27%; five minutes: 28%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
4 1581593 71425 22143 8.94% 1.19% 0.91% 0 Check heaps
88 219985 14589 15078 0.79% 0.15% 0.12% 0 HULC Tcam Memory
207 76592 525547 145 0.15% 0.11% 0.07% 0 IP Input
289 100 315 317 0.15% 0.10% 0.02% 1 Virtual Exec
The IOS version is 122-55.SE8. Please help to give me some advise to find the higher utilization issue.
Best Regards,
09-09-2014 03:07 AM
28% CPU is not high.
09-09-2014 03:21 AM
Hi, Thanks for your reply, but the CPU utilization of the other Catalyst 3750 is below 10% ( almost the same configuration and same environment ) So we need to find the reason why this CPU utilization is higher than others.
09-09-2014 09:34 PM
Jackson,
Could you please have a look on my document which have been created for troubleshooting the high cpu on 3750 and see if that helps. Else capture the ouptuts which are been suggested and we will have a look:
https://supportforums.cisco.com/document/12298401/troubleshooting-high-cpu-3750
HTH
Inayath
09-10-2014 12:05 AM
09-10-2014 12:59 AM
Can you also send this:
- show sdm prefer
- show platform tcam util asic all
- show ip route summary
- show int | inc is up|drop
debug platform cpu-queues software-fwd-q
(Please be cautious using the debug, only run for a few seconds, then
'un all')
debug platform cpu-queues cpu heartbeat
(Please be cautious using the debug, only run for a few seconds, then
'un all')
09-10-2014 01:44 AM
Hi,
attached files are show command result. ( I have not collect debug result because this switch is in production and not allow any performance impact )
I also upload the running-config file. ( We used IPSLA, EEM, PBR to trace link status and change ip route dynamically )
This switch platform is Catalyst 3750G-12S, we have another one Catalyst 3750X-12S switch with almost same configuration, but the CPU utilization below 10%. I compare the "show controller cpu-interface" result, the queue "sw forwarding" for Catalyst 3750G-12S always increased, but the queue "sw forwarding" for Catalyst 3750X-12S not increased. Maybe it is why the Catalyst 3750G-12S CPU utilization higher than Catalyst 3750X-12S?
Is it the platform performance issue? If not, how to find the root cause?
09-10-2014 02:29 AM
Hi,
Below link might help something for you:
https://supportforums.cisco.com/discussion/11532116/high-cpu-usage-3750-x-stack
HTH
10-08-2014 08:49 PM
Hi,
I know the IP packet with option will forward by software, is there other traffic or function that cause traffic not forward via hardware and it is done by software?
( for example : PBR, IPSLA .. )
Maybe I can confirm with customer before I enable "debug platform cpu-queues software-fwd-q"?
Best Regards,
Jackson Ku
09-10-2014 02:42 AM
If you look at the process cpu the utilization is mainly by ip input process., Here is below explanation which I can think of right now:
The IP input process is merely traffic related. More specific layer 3
traffic. Normally layer 3 traffic goes to the CPU to be forwarded that is
why you will have an increase of the utilization.
When I say layer 3 traffic I mean traffic that need to be forwarded based on
its IP address. Traffic that is not forwarded via hardware and it is done
via software. Traffic that needs to be evaluated on the ACLs and routing
protocols instances. The SDM template set to default is a layer 2 template.
That means that the switch is not prepare for heavily loaded layer 3 traffic
forwarding.
What we can try right now is to set the SDM template to routing. You can do
that using the following command:
switch>enable
switch#config t
switch(config)#sdm prefer routing
After entering the command you will need to reload the device to make it
take effect.
Please apply these commands and let me know how it goes. However keep in
mind that IP input means that you are forwarding a lot of layer 3 (IP
address based) packets that are loading your switch too much.
HTH
Inayath
*Plz dont forget to rate all usefull posts.
09-10-2014 07:43 AM
Hi, The current sdm template was "aggregate routing". Thanks a lot.
09-10-2014 02:54 AM
Also show stacks can be used for diagnostic
09-11-2014 10:10 PM
09-11-2014 10:31 PM
IP_SA:192.168.1.77 and
Destination ip address: 10.1.1.171
As only this is th destination i paddress which I can see the src is punting the traffic.
Shut down the 192.168.1.77 and see if the cpu gets reduced?
09-12-2014 12:06 AM
Hi, the debug result was captured at our lab, not at real production network. Is there any doc that can help us to investigate it? ( we should arrange off-hour to capture the debug information )
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide