cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
3510
Views
0
Helpful
7
Replies

High CPU 4500X

nibinrodrigues
Level 1
Level 1

Dear All,

We are having high CPU on one of our Core Switch (C4500X-16). I am attaching the show process cpu output below.

I uploaded the show run on Cisco Output interpreter and got the suggestion to enable "ip cef" and "mls ip multicast non-rpf cef" but here my switch os 4500X so is this recommendation applicable.

I am having a GRE tunnel running in the Switch also multi-casting is enabled.  Is root cause of high CPU is multicasting and GRE if yes how can I verify this via log.  

"WARNING: The CEF non-RPF rate limiting has been disabled on the device.

CEF based non-RPF rate limiting should always be enabled on Catalyst 6000/6500 switches
with Supervisor 2 or Supervisor 720. If non-RPF traffic were allowed to be sent
to the MSFC it would significantly impact the CPU. The only time this command should
be used is for troubleshooting or as a workaround for a software defect and under
the direction of the Cisco TAC.
RECOMMENDED ACTION: Enter the global command, 'mls ip multicast non-rpf cef' to
reduce the CPU impact of non-rpf traffic to the MSFC."

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

CORE-2-FL04-DT#sh ip pim interface count

For switching state use "show ip mfib interface"
Address Interface Mpackets In/Out
172.19.20.13 Tunnel1 105722437765/0
172.19.20.13 Vlan9 0/466673
0.0.0.0 Vlan20 0/0
172.19.160.3 Vlan160 1507475/466673
172.19.162.3 Vlan162 251980/26364779238
172.19.164.3 Vlan164 8940053/466673
172.19.166.3 Vlan166 1778329/36397019052
172.19.168.3 Vlan168 3941068/466673
172.19.170.3 Vlan170 2286694/39584710961
172.19.172.3 Vlan172 8918385/466673
172.19.174.3 Vlan174 1787593/37942544424
172.19.176.3 Vlan176 11404030/466673
172.19.178.3 Vlan178 2808165/51469424652
172.19.180.3 Vlan180 11847745/466673
172.19.182.3 Vlan182 2297822/28919886063
172.19.184.3 Vlan184 64646/466673

CORE-2-FL04-DT#sh ver
Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500e-UNIVERSALK9-M), Version 03.06.00.E RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2014 by Cisco Systems, Inc.
Compiled Thu 26-Jun-14 12:52 by prod_rel_team

Cisco IOS-XE software, Copyright (c) 2005-2014 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0.
(http://www.gnu.org/licenses/gpl-2.0.html) For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: 15.0(1r)SG11
CORE-2-FL04-DT uptime is 11 weeks, 4 days, 6 hours, 35 minutes
System returned to ROM by power-on
System restarted at 06:14:42 Qatar Sun Oct 18 2015
Running default software
Jawa Revision 2, Winter Revision 0x0.0x41

Last reload reason: power-on

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.


License Information for 'WS-C4500X-16'
License Level: entservices Type: Permanent
Next reboot license Level: entservices

cisco WS-C4500X-16 (MPC8572) processor (revision 9) with 4194304K bytes of physical memory.
Processor board ID JAE1851010R
MPC8572 CPU at 1.5GHz, Cisco Catalyst 4500X
Last reset from PowerUp
33 Virtual Ethernet interfaces
24 Ten Gigabit Ethernet interfaces
511K bytes of non-volatile configuration memory.

Configuration register is 0x2101

CORE-2-FL04-DT#sh modu
CORE-2-FL04-DT#sh module
Chassis Type : WS-C4500X-16

Power consumed by backplane : 0 Watts

Mod Ports Card Type Model Serial No.
---+-----+--------------------------------------+------------
1 16 4500X-16 10GE (SFP+) WS-C4500X-16 J
2 8 10GE SFP+ C4KX-NM-8 

Thanks

Nibin

7 Replies 7

Leo Laohoo
Hall of Fame
Hall of Fame

Can you please post the complete output to the command "sh proc cpu sort | ex 0.00"?

Dear Leo,

Please find the inputs. I will post the "show proc cpu sort | ex 0.00" by tomorrow.

Thanks

Dear Leo,

Thanks for the prompt response. In the switch I can see 2 CPU Core0 and Core1. Core1 is always consuming full CPU and Core0 is free. Is there any way we can load balance the load of CPU.

Core 0: CPU utilization for five seconds: 16%; one minute: 15%; five minutes: 15%
Core 1: CPU utilization for five seconds: 96%; one minute: 95%; five minutes: 94% 

Thanks

Nibin

Hello, 

It is expected for two cores to have different utilization by same process say iosd. At any point of time, both the cores are utilized to a certain degree but their utilization need not be same.

HTH

Cheers

Dear Leo,

Please find the requested output

Thanks

Nibin

4657   3131834     29952109 652    55.24    54.33    54.26    34816 iosd    

Ok, this is the process that is chewing the CPU of the switch. 

However, instead of me pointing at the "usual suspect" of an IOS bug the process "iosd" means the switch is over-utilized.  

Look at each 10 Gbps interface and determine which one is dropping packets.  This is the culprit.

Eugen Serban
Level 1
Level 1

Edit 6h after :)

I have found the issue. The source was behind a nat and when I was doing a sh ip mroute I could see the group twice from both IPs (original and natted).

I suspect that's why the RPF failed. When I moved the source outside of the NAT, the CPU is normal.

Eugen

[ORIGINAL MESSAGE]

Hello,

I have the same problem with the same version, the differece is that I don't have any GRE tunnels on the switch.

When I do a show ip mfib, the (*,G) shows SW forwarding with RPF failed. I don't know if this might be a problem...

BTW, I have verified all the interfaces and there are no drops.

Have you found a solution?

Regards,

Eugen