High CPU utilization on C3560X caused by hl3mm process

[Suhel]

We've been experiencing high CPU utilization (99%) alerts for quite a while now.

Seems to be a bug described in the following link: https://quickview.cloudapps.cisco.com/quickview/bug/CSCut84989

Switch#sh process cpu sort 5sec
CPU utilization for five seconds: 99%/0%; one minute: 98%; five minutes: 98%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
174 1660524661 244591906 6788 62.27% 65.22% 66.58% 0 hl3mm
172 2423427203 783743222 3092 8.75% 9.53% 9.28% 0 Hulc LED Process
85 2003376480 352266897 5687 5.02% 4.64% 4.53% 0 RedEarth Tx Mana

The following error message is repeated in the logs :

Switch#sh log | i Oct 18
%PLATFORM_MCAST-3-IPV4_ERR: Cannot load multicast routing entry in hardware as corresponding TCAM region is full. (Condition hit 48120 times since last warning)
%PLATFORM_MCAST-5-IPV4_CLEAR: All outstanding multicast routing entries loaded in hardware as TCAM space freed up.

The current version on the switch is 15.0(2)SE9

Any suggestions for a workaround on this??

[Leo Laohoo]

Post the complete output to the following commands:
1. sh version; and
2. sh sdm template

[Suhel]

Show version output:

Cisco IOS Software, C3560E Software (C3560E-IPBASEK9-M), Version 15.0(2)SE9, REL EASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Tue 01-Dec-15 06:47 by prod_rel_team

ROM: Bootstrap program is C3560E boot loader
BOOTLDR: C3560E Boot Loader (C3560X-HBOOT-M) Version 15.0(2r)SE, RELEASE SOFTWAR E (fc1)

uptime is 1 year, 10 weeks, 2 days, 23 hours, 57 minutes
System returned to ROM by power-on
System restarted at 06:26:22 EDT Sun Aug 7 2016
System image file is "flash:c3560e-ipbasek9-mz.150-2.SE9.bin"

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

License Level: ipservices
License Type: Permanent
Next reload license Level: ipservices

cisco WS-C3560X-48U (PowerPC405) processor (revision A0) with 262144K bytes of memory.
Processor board ID FDO1730P1VD
Last reset from power-on
7 Virtual Ethernet interfaces
1 FastEthernet interface
52 Gigabit Ethernet interfaces
2 Ten Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.

Switch Ports Model SW Version SW Image
------ ----- ----- ---------- ----------
* 1 54 WS-C3560X-48U 15.0(2)SE9 C3560E-IPBASEK9-M

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Sh sdm prefer output:

The current template is "desktop default" template.
The selected template optimizes the resources in
the switch to support this level of features for
8 routed interfaces and 1024 VLANs.

number of unicast mac addresses: 6K
number of IPv4 IGMP groups + multicast routes: 1K
number of IPv4 unicast routes: 8K
number of directly-connected IPv4 hosts: 6K
number of indirect IPv4 routes: 2K
number of IPv6 multicast groups: 64
number of IPv6 unicast routes: 106
number of directly-connected IPv6 addresses: 74
number of indirect IPv6 unicast routes: 32
number of IPv4 policy based routing aces: 0
number of IPv4/MAC qos aces: 0.5K
number of IPv4/MAC security aces: 0.875k
number of IPv6 policy based routing aces: 0
number of IPv6 qos aces: 0
number of IPv6 security aces: 60

[Leo Laohoo]

---

@Suhel wrote:

uptime is 1 year, 10 weeks, 2 days, 23 hours, 57 minutes

---

Running 15.0(2)SE9 IOS is OK, but the uptime isn't.  

Reboot the switch is my recommendation.

[Suhel]

yes that's the first thing we're planning to do. Any workaround/fixes available for the bug?

[Leo Laohoo]

I've never used 15.0(2)SE9 but I have used 15.0(2)SE8.  And I've never had issues with "SE8" for the last three years (or so).  

Currently upgraded to 15.2(4)E5.  

My recommendation is to either upgrade the IOS or reboot the appliance once every 11 months.

[Suhel]

Thanks for the suggestion Leo! Appreciate your help

[divanko]

Out of curiosity, was the problem resolved?

What is the RP announce interval set on your RP?  I found several Nexus documents suggesting to set the interval to a minimum of 15 seconds.  We have a 65k as our RP and set the send-rp-announce interval to 15 seconds.  I found that by changing the hello interval to 15 on the RP a 3750x stack running layer 3 went from 99 percent to 14 percent.

Worth a shot ...

[fredrick.pettiford]

Ive had this issue on a few devices here on out network. A quick temporary fix is to clear all the mroutes with the " Clear ip mroute *" command. If fixes the issue for a few minutes before the cpu spikes back up....It wen from 98% on the cpu to 11%..It gave me enough time to do what i was trying to do....i will upgrade soon.