01-09-2016 02:10 AM - edited 03-08-2019 03:21 AM
Dear All,
We are having high CPU on one of our Core Switch (C4500X-16). I am attaching the show process cpu output below.
I uploaded the show run on Cisco Output interpreter and got the suggestion to enable "ip cef" and "mls ip multicast non-rpf cef" but here my switch os 4500X so is this recommendation applicable.
I am having a GRE tunnel running in the Switch also multi-casting is enabled. Is root cause of high CPU is multicasting and GRE if yes how can I verify this via log.
"WARNING: The CEF non-RPF rate limiting has been disabled on the device.
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
CORE-2-FL04-DT#sh ip pim interface count
For switching state use "show ip mfib interface"
Address Interface Mpackets In/Out
172.19.20.13 Tunnel1 105722437765/0
172.19.20.13 Vlan9 0/466673
0.0.0.0 Vlan20 0/0
172.19.160.3 Vlan160 1507475/466673
172.19.162.3 Vlan162 251980/26364779238
172.19.164.3 Vlan164 8940053/466673
172.19.166.3 Vlan166 1778329/36397019052
172.19.168.3 Vlan168 3941068/466673
172.19.170.3 Vlan170 2286694/39584710961
172.19.172.3 Vlan172 8918385/466673
172.19.174.3 Vlan174 1787593/37942544424
172.19.176.3 Vlan176 11404030/466673
172.19.178.3 Vlan178 2808165/51469424652
172.19.180.3 Vlan180 11847745/466673
172.19.182.3 Vlan182 2297822/28919886063
172.19.184.3 Vlan184 64646/466673
CORE-2-FL04-DT#sh ver
Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500e-UNIVERSALK9-M), Version 03.06.00.E RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2014 by Cisco Systems, Inc.
Compiled Thu 26-Jun-14 12:52 by prod_rel_team
Cisco IOS-XE software, Copyright (c) 2005-2014 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0.
(http://www.gnu.org/licenses/gpl-2.0.html) For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.
ROM: 15.0(1r)SG11
CORE-2-FL04-DT uptime is 11 weeks, 4 days, 6 hours, 35 minutes
System returned to ROM by power-on
System restarted at 06:14:42 Qatar Sun Oct 18 2015
Running default software
Jawa Revision 2, Winter Revision 0x0.0x41
Last reload reason: power-on
This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.
A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
If you require further assistance please contact us by sending email to
export@cisco.com.
License Information for 'WS-C4500X-16'
License Level: entservices Type: Permanent
Next reboot license Level: entservices
cisco WS-C4500X-16 (MPC8572) processor (revision 9) with 4194304K bytes of physical memory.
Processor board ID JAE1851010R
MPC8572 CPU at 1.5GHz, Cisco Catalyst 4500X
Last reset from PowerUp
33 Virtual Ethernet interfaces
24 Ten Gigabit Ethernet interfaces
511K bytes of non-volatile configuration memory.
Configuration register is 0x2101
CORE-2-FL04-DT#sh modu
CORE-2-FL04-DT#sh module
Chassis Type : WS-C4500X-16
Power consumed by backplane : 0 Watts
Mod Ports Card Type Model Serial No.
---+-----+--------------------------------------+------------
1 16 4500X-16 10GE (SFP+) WS-C4500X-16 J
2 8 10GE SFP+ C4KX-NM-8
Thanks
Nibin
01-09-2016 03:12 AM
Can you please post the complete output to the command "sh proc cpu sort | ex 0.00"?
01-09-2016 03:45 AM
01-10-2016 01:56 AM
Dear Leo,
Thanks for the prompt response. In the switch I can see 2 CPU Core0 and Core1. Core1 is always consuming full CPU and Core0 is free. Is there any way we can load balance the load of CPU.
Core 0: CPU utilization for five seconds: 16%; one minute: 15%; five minutes: 15%
Core 1: CPU utilization for five seconds: 96%; one minute: 95%; five minutes: 94%
Thanks
Nibin
01-10-2016 06:49 AM
Hello,
It is expected for two cores to have different utilization by same process say iosd. At any point of time, both the cores are utilized to a certain degree but their utilization need not be same.
HTH
Cheers
01-10-2016 01:07 AM
01-10-2016 01:42 AM
4657 3131834 29952109 652 55.24 54.33 54.26 34816 iosd
Ok, this is the process that is chewing the CPU of the switch.
However, instead of me pointing at the "usual suspect" of an IOS bug the process "iosd" means the switch is over-utilized.
Look at each 10 Gbps interface and determine which one is dropping packets. This is the culprit.
04-12-2017 08:49 AM
Edit 6h after :)
I have found the issue. The source was behind a nat and when I was doing a sh ip mroute I could see the group twice from both IPs (original and natted).
I suspect that's why the RPF failed. When I moved the source outside of the NAT, the CPU is normal.
Eugen
[ORIGINAL MESSAGE]
Hello,
I have the same problem with the same version, the differece is that I don't have any GRE tunnels on the switch.
When I do a show ip mfib, the (*,G) shows SW forwarding with RPF failed. I don't know if this might be a problem...
BTW, I have verified all the interfaces and there are no drops.
Have you found a solution?
Regards,
Eugen
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide