TCAM ERROR

[Diana Karolina Rojas]

Good morning community,

I have a lot of logs in my core device (Cisco 3750), here is the output:  

%HARDWARE-1-TCAM_ERROR: Found error in HQATM TCAM Space and not able to recover the error

I searched for information about this but I could not find nothing clear or the resolution for this behavior. I know that this is a problem related to my Qos, PBR and ACL entries, and the memory the TCAM use for manage these processes, but I need to know how to solve it and why this happens. Here are the output if the command related to my TCAM.

T3DCSWCORE# show platform tcam utilization

CAM Utilization for ASIC# 0 Max Used
Masks/Values Masks/values

Unicast mac addresses: 544/4352 178/1347
IPv4 IGMP groups + multicast routes: 152/1216 6/26
IPv4 unicast directly-connected routes: 544/4352 178/1347
IPv4 unicast indirectly-connected routes: 176/1408 51/321
IPv6 Multicast groups: 544/4352 178/1347
IPv6 unicast directly-connected routes: 544/4352 178/1347
IPv6 unicast indirectly-connected routes: 262/2096 11/42
IPv4 policy based routing aces: 256/256 65/65
IPv4 qos aces: 768/768 388/388
IPv4 security aces: 512/512 236/236
IPv6 policy based routing aces: 0/0 0/0
IPv6 qos aces: 0/0 0/0
IPv6 security aces: 204/510 8/8

Note: Allocation of TCAM entries per feature uses
a complex algorithm. The above information is meant
to provide an abstract view of the current TCAM utilization

T3DCSWCORE#show platform tcam usage

=============================================================================
TCAM Table
TCAM / SSRAM Table TCAM SSRAM
Start Size X Start Size Y
=============================================================================
Local Forwarding Table: 0 15C0 1 0 15C0 4
Local Learning Table: 0 15C0 1 5700 15C0 2
Secondary Forwarding Table: 1100 1240 1 8280 1240 8
QoS Table: 2340 1800 1 16900 1800 4
ACL Table: 3B40 1000 1 1C900 1000 4
IPV6 Secondary Forwarding Tabl 4B40 1060 2 12780 830 8
IPV6 Classification Table: 5BA0 EC0 2 20940 760 4
IPV6 ACL Table: 6A60 CC0 2 226C0 660 4
Station Table: 0 0 0 24080 15C0 4
MAC Address Table: 0 0 0 29780 1200 8
Multicast Expansion Table: 0 0 0 32780 422 8
VLAN List Table: 0 0 0 35900 400 10
Equal Cost Route Table: 0 0 0 34900 80 20

X - Number of 144-bit TCAM entries per descriptor
Y - Number of bytes per descriptor
=============================================================================
SDM template realignment scheme-1 in effect

T3DCSWCORE#show platform tcam log-results

================================================================
CAM Log Results
----------------------------------------------------------------
Total Number of PortASICs: 7

ASIC 0
Lookup "Invalid", value 0
TCAM Index 131071, TCAM Table Index -1
Cam Log Keys
key-0: 00-80002020-00040000
key-1: 00-00008080-04800000
key-2: 00-60000000-00000000
key-3: 00-00001000-00000000

ASIC 1
Lookup "Invalid", value 0
TCAM Index 110335, TCAM Table Index -1
Cam Log Keys
key-0: 00-60000000-00180A00
key-1: 00-04800100-00000400
key-2: 00-00000000-00000200
key-3: 00-00000013-02800050

ASIC 2
Lookup "Invalid", value 0
TCAM Index 131071, TCAM Table Index -1
Cam Log Keys
key-0: 00-00113C00-C180C400
key-1: 00-00113C00-C180C400
key-2: 00-11000000-00818000
key-3: 40-00000800-00000000

ASIC 3
Lookup "Invalid", value 0
TCAM Index 24837, TCAM Table Index -1
Cam Log Keys
key-0: 00-00000400-08000000
key-1: 00-00000000-00082000
key-2: 00-00000000-00002000
key-3: 00-00000000-60000000

ASIC 4
Lookup "Invalid", value 0
TCAM Index 0, TCAM Table Index -1
Cam Log Keys
key-0: 30-F43086B4-E5543243
key-1: 07-4400C012-0802D083
key-2: 30-F43086B4-E5543243
key-3: 00-F43086B4-E5543243

ASIC 5
Lookup "Invalid", value 0
TCAM Index 131071, TCAM Table Index -1
Cam Log Keys
key-0: FF-F7F89DFF-FFF9CDFE
key-1: 00-00004000-00002004
key-2: 04-00000080-00000000
key-3: 08-10004400-00C40000

ASIC 6
Lookup "Invalid", value 0
TCAM Index 131071, TCAM Table Index -1
Cam Log Keys
key-0: 00-00000820-80000000
key-1: 00-00000820-80000000
key-2: 00-80000400-20000000
key-3: 00-00200004-00040008

Notes:
a) key-0 is most recent cam key
b) key-0 contains lsb's and key-3 contains msb's
c) watch for Lookup field in cam key for validating results
d) TCAM Table Index -1 indicates invalid results
================================================================

Thank you in advance if you can help me.

Regards,

[Mark Malone]

Hi Diana

Just reading a post here one of the Cisco guys updated on you may have already read it , its old enough but similar issue on 3560 but looks like could be  hardware , did you try reboot it clear all the processes etc

Heres the important bit suppose

"As i said earlier, this error message is thrown when there is a TCAM corruption and the TCAM consistency check routines on the switch haven't been able to correct the corruption that has happened. Hence the syslog is thrown.

70% of the time rebooting the switch corrects this function incase if this doesn't gets corrected you would need a hardware replacement."

https://supportforums.cisco.com/discussion/11353451/found-error-hqatm-tcam-space-and-not-able-recover-error-when-acl-label-128-use

[Julio E. Moisa]

Hi Diana,

I had similar problem a long time ago with a Cisco 4948 L3, for any reason we were reaching the maximum capacity of entries on the TCAM (I think we were receiving many same BGP entries with different next hops), If I'm not wrong these devices support around 64000 entries, I recommend to open a case with the Cisco TAC in order to discard any hardware problem or if you need update the IOS, the following command could be useful to get more details about TCAM:

show platform hardware ip route summary

Our solution was moved the BGP configuration to other robust Cisco device. 

Hope it is useful

:-)