RedEarth Tx Mana - High CPU usage on Cisco 3560-X

mmendis · ‎08-22-2013

Hi Everyone,

We are doing a testing on Cisco 3560-X switch by generating 100M traffic. Now CPU usage of RedEarth Tx Mana

is high around 40%.

We have tweak the SDM templates for routing.

Can any one help to sort this out ??

Cisco IOS Software, C3560E Software (C3560E-UNIVERSALK9-M), Version 12.2(58)SE2, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Compiled Thu 21-Jul-11 01:34 by prod_rel_team

ROM: Bootstrap program is C3560E boot loader

BOOTLDR: C3560E Boot Loader (C3560X-HBOOT-M) Version 12.2(53r)SE2, RELEASE SOFTWARE (fc1)

NMC-0.77-3560 uptime is 1 day, 1 hour, 42 minutes

System returned to ROM by power-on

System restarted at 09:34:34 LKR Thu Aug 22 2013

System image file is "flash:/c3560e-universalk9-mz.122-58.SE2.bin"

This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply

third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for

compliance with U.S. and local country laws. By using this product you

agree to comply with applicable laws and regulations. If you are unable

to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to

export@cisco.com.

License Level: ipservices

License Type: Evaluation

Next reload license Level: ipservices

cisco WS-C3560X-24 (PowerPC405) processor (revision A0) with 262144K bytes of memory.

Processor board ID FDO1622R1X9

Last reset from power-on

1 Virtual Ethernet interface

1 FastEthernet interface

28 Gigabit Ethernet interfaces

2 Ten Gigabit Ethernet interfaces

The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.

Base ethernet MAC Address : D8:67:D9:CA:0D:80

Model revision number : A0

Motherboard revision number : C0

Model number : WS-C3560X-24T-L

Top Assembly Revision Number : E0

Version ID : V02

CLEI Code Number : COMJU00ARB

Hardware Board Revision Number : 0x03

Switch Ports Model SW Version SW Image

------ ----- ----- ---------- ----------

* 1 30 WS-C3560X-24 12.2(58)SE2 C3560E-UNIVERSALK9-M

Configuration register is 0xF

Leo Laohoo · ‎08-22-2013

Version 12.2(58)SE2

This is most likely your culprit.

Use 12.2(55)SE8 if you are running Layer 3, PoE and/or 802.1x.

You could also try 15.0(2)SE4 although I'd much prefer 12.2(55)SE8.

mmendis · ‎08-22-2013

Hi Leo,

Is this one will work ?

c3560e-universalk9-tar.150-2.SE4

Leo Laohoo · ‎08-23-2013

Without knowing what you've got, give it a try.

mmendis · ‎08-23-2013

I have tried with the new IOS , still the same issue.

Leo Laohoo · ‎08-23-2013

I have tried with the new IOS , still the same issue

Do you have the following: Layer 3 operation, PoE or 802.1x.

If you have any one of the above, then try 12.2(55)SE8.

mmendis · ‎08-23-2013

Is there any other work round to tackle the RedEarth Tx Mana high utilization issue ??

ahmed.gadi · ‎09-10-2013

I have same issue with our WS-C3750X-48 running 15.0(2)SE3 [c3750e-universalk9npe-mz.150-2.SE2] IOS.

------------------show process memory-------------------

PID TTY Allocated Freed Holding Getbufs Retbufs Process

!

89 0 0 0 7200 0 0 RedEarth I2C dri

90 0 34872 34816 7160 0 0 RedEarth Tx Mana

91 0 28600 35368 7160 2268 0 RedEarth Rx Mana

!

------------------ show process cpu ------------------

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

!

89 545506 2571645 212 0.15% 0.20% 0.17% 0 RedEarth I2C dri

90 9762656 1715940 5689 2.24% 2.46% 2.55% 0 RedEarth Tx Mana

91 93631 1285767 72 0.00% 0.01% 0.00% 0 RedEarth Rx Mana

!

Please advice!!!

mmendis · ‎09-10-2013

This HIGH CPU Usage might be due to TCAM Exhaustion. In our case we saw the same in our 3560 used in Router Mode with BGP + OSPF.

Use this command to check the utilization of TCAM.

show platform tcam utilization

You will get similar Output :

sh platform tcam utilization

CAM Utilization for ASIC# 0 Max Used

Masks/Values Masks/values

Unicast mac addresses: 2268/2268 39/39

IPv4 IGMP groups + multicast routes: 1072/1072 1/1

IPv4 unicast directly-connected routes: 2048/2048 34/34

IPv4 unicast indirectly-connected routes: 2048/2048 2047/2047

IPv6 Multicast groups: 1072/1072 11/11

IPv6 unicast directly-connected routes: 2048/2048 0/0

IPv6 unicast indirectly-connected routes: 3072/3072 3/3

IPv4 policy based routing aces: 128/128 13/13

IPv4 qos aces: 512/512 6/6

IPv4 security aces: 640/640 44/44

IPv6 policy based routing aces: 128/128 8/8

IPv6 qos aces: 128/128 5/5

IPv6 security aces: 128/128 17/17

Note: Allocation of TCAM entries per feature uses

a complex algorithm. The above information is meant

to provide an abstract view of the current TCAM utilization

in our testing the above high utilization of "IPv4 unicast indirectly-connected routes: 2048/2048 2047/2047"was the issue that high CPU usage occour.

As we have found, Router will forward packet to CPU when TCAM resources are full. We didn't found a good explanation for this, but i hope this will give you some idea.

ahmed.gadi · ‎09-10-2013

CAM Utilization for ASIC# 0 Max Used

Masks/Values Masks/values

Unicast mac addresses: 6364/6364 189/189

IPv4 IGMP groups + multicast routes: 1072/1072 1/1

IPv4 unicast directly-connected routes: 6144/6144 0/0

IPv4 unicast indirectly-connected routes: 2048/2048 34/34

IPv6 Multicast groups: 112/112 11/11

IPv6 unicast directly-connected routes: 74/74 0/0

IPv6 unicast indirectly-connected routes: 32/32 3/3

IPv4 policy based routing aces: 412/412 12/12

IPv4 qos aces: 512/512 21/21

IPv4 security aces: 924/924 44/44

IPv6 policy based routing aces: 18/18 8/8

IPv6 qos aces: 22/22 18/18

IPv6 security aces: 60/60 17/17

Note: Allocation of TCAM entries per feature uses

a complex algorithm. The above information is meant

to provide an abstract view of the current TCAM utilization

I do not see any thing abnormal here. Can you please advice from the above command ?

mmendis · ‎09-10-2013

Seems like any of your TCAM resorces are not much utilized. I did not find any good explanation about what RedEarth Rx Mana process actually does.

Better do some more search on this process.