TCAM Utilization Issue

[adonsius_artollo]

I have a Cisco WS-C3750G-24TS switch that is experiencing TCAM utilization issues, which in turn throttles the CPU, and the entire switch bogs down.

I'm running dual IPv4/IPv6 and have already changed the SDM template to "desktop IPv4 and IPv6 routing" template but the issue persists (we cannot use the aggregate template in this scenario). In the "show platform tcam utilization" output I notice three tables that are maxed-out, "IPv4 policy based routing aces", "IPv4 qos aces", and "IPv4 security aces". Why are these three tables maxed out, or how do I increase resources to these? I've examined all the remaining SDM templates and my current template assigns the maximum to these tables? Any input is greatly appreciated:

#show log | in Jul  7
Jul  7 02:09:23.578: %PLATFORM_UCAST-4-PREFIX:  One or more, more specific prefixes could not be programmed into TCAM and are being covered by a less specific prefix, and the packets may be software forwarded
Jul  7 02:09:23.586: %PLATFORM_UCAST-4-PREFIX:  One or more, more specific prefixes could not be programmed into TCAM and are being covered by a less specific prefix, and the packets may be software forwarded
Jul  7 02:51:24.311: %PLATFORM_UCAST-4-PREFIX:  One or more, more specific prefixes could not be programmed into TCAM and are being covered by a less specific prefix, and the packets may be software forwarded
Jul  7 02:51:24.319: %PLATFORM_UCAST-4-PREFIX:  One or more, more specific prefixes could not be programmed into TCAM and are being covered by a less specific prefix, and the packets may be software forwarded
Jul  7 04:00:59.656: %PLATFORM_UCAST-4-PREFIX:  One or more, more specific prefixes could not be programmed into TCAM and are being covered by a less specific prefix, and the packets may be software forwarded
Jul  7 04:00:59.656: %PLATFORM_UCAST-4-PREFIX:  One or more, more specific prefixes could not be programmed into TCAM and are being covered by a less specific prefix, and the packets may be software forwarded
Jul  7 04:17:16.879: %PLATFORM_UCAST-4-PREFIX:  One or more, more specific prefixes could not be programmed into TCAM and are being covered by a less specific prefix, and the packets may be software forwarded
Jul  7 04:17:16.904: %PLATFORM_UCAST-4-PREFIX:  One or more, more specific prefixes could not be programmed into TCAM and are being covered by a less specific prefix, and the packets may be software forwarded
Jul  7 04:20:42.456: %PLATFORM_UCAST-4-PREFIX:  One or more, more specific prefixes could not be programmed into TCAM and are being covered by a less specific prefix, and the packets may be software forwarded
Jul  7 04:20:42.453: %PLATFORM_UCAST-4-PREFIX:  One or more, more specific prefixes could not be programmed into TCAM and are being covered by a less specific prefix, and the packets may be software forwarded
Jul  7 04:42:25.224: %PLATFORM_UCAST-4-PREFIX:  One or more, more specific prefixes could not be programmed into TCAM and are being covered by a less specific prefix, and the packets may be software forwarded
Jul  7 04:42:25.070: %PLATFORM_UCAST-4-PREFIX:  One or more, more specific prefixes could not be programmed into TCAM and are being covered by a less specific prefix, and the packets may be software forwarded
Jul  7 04:46:51.057: %PLATFORM_UCAST-4-PREFIX:  One or more, more specific prefixes could not be programmed into TCAM and are being covered by a less specific prefix, and the packets may be software forwarded
Jul  7 04:46:51.071: %PLATFORM_UCAST-4-PREFIX:  One or more, more specific prefixes could not be programmed into TCAM and are being covered by a less specific prefix, and the packets may be software forwarded
Jul  7 04:57:22.476: %PLATFORM_UCAST-4-PREFIX:  One or more, more specific prefixes could not be programmed into TCAM and are being covered by a less specific prefix, and the packets may be software forwarded
Jul  7 04:57:22.483: %PLATFORM_UCAST-4-PREFIX:  One or more, more specific prefixes could not be programmed into TCAM and are being covered by a less specific prefix, and the packets may be software forwarded
Jul  7 15:53:37.610: %PLATFORM_UCAST-4-PREFIX:  One or more, more specific prefixes could not be programmed into TCAM and are being covered by a less specific prefix, and the packets may be software forwarded
Jul  7 15:53:37.626: %PLATFORM_UCAST-4-PREFIX:  One or more, more specific prefixes could not be programmed into TCAM and are being covered by a less specific prefix, and the packets may be software forwarded

#show platform tcam utili

CAM Utilization for ASIC# 0                                            Max                 Used
                                                                                  Masks/Values    Masks/values

 Unicast mac addresses:                                              544/4352         66/432  
 IPv4 IGMP groups + multicast routes:                         152/1216          6/26   
 IPv4 unicast directly-connected routes:                      544/4352         66/432  
 IPv4 unicast indirectly-connected routes:                  176/1408        163/1226 
 IPv6 Multicast groups:                                                544/4352         66/432  
 IPv6 unicast directly-connected routes:                      544/4352         66/432  
 IPv6 unicast indirectly-connected routes:                    262/2096         18/84   
 IPv4 policy based routing aces:                                 256/256           4/4    
 IPv4 qos aces:                                                           768/768         324/324  
 IPv4 security aces:                                                      512/512          39/39   
 IPv6 policy based routing aces:                                       0/0             0/0    
 IPv6 qos aces:                                                                 0/0             0/0    
 IPv6 security aces:                                                     204/510           8/8    

#show platform ip unicast counts
# of HL3U fibs 1565
# of HL3U adjs 85
# of HL3U mpaths 4
# of HL3U covering-fibs 17
# of HL3U fibs with adj failures 160
Fibs of Prefix length 0, with TCAM fails: 0
Fibs of Prefix length 1, with TCAM fails: 0
{OMITTED TEXT...}
Fibs of Prefix length 27, with TCAM fails: 0
Fibs of Prefix length 28, with TCAM fails: 0
Fibs of Prefix length 29, with TCAM fails: 710
Fibs of Prefix length 30, with TCAM fails: 529
Fibs of Prefix length 31, with TCAM fails: 0
Fibs of Prefix length 32, with TCAM fails: 52
Fibs of Prefix length 33, with TCAM fails: 0

#show sdm prefer
 The current template is "desktop IPv4 and IPv6 routing" template.
 The selected template optimizes the resources in
 the switch to support this level of features for
 8 routed interfaces and 1024 VLANs.

[Mark Malone · Accepted solution]

Hi

Your over utilizing the switch you need to reduce some of the prefixes , you cant increase it unless with better hardware, these are switches not routers so they cant handle the same amount of prefixes

This is a hardware boundary in SDM and when you max out the process overflow gets sent to cpu dramatically slowing the switch which your seeing

what does show ip route summary provide and match it against what the show sdm prefer allows in routes

and check below looks like theres way too many of these coming in , can you not summarize upstream into this router some of these subnets to reduce the load on the switch

show ip route | i /29     show ip route | i /30

what IGP protocol are you using

whats the cpu running at show proc cpu sorted and the show proc cpu history

[suresh natarajan]

Thanks

[ZAHIDHASEEB]

I have almost same issue. Below is the output.

sh platform tcam utilization

CAM Utilization for ASIC# 0 Max Used
Masks/Values Masks/values

Unicast mac addresses: 6364/6364 754/754
IPv4 IGMP groups + multicast routes: 1120/1120 1/1
IPv4 unicast directly-connected routes: 6144/6144 6144/6144
IPv4 unicast indirectly-connected routes: 2048/2048 172/172
IPv4 policy based routing aces: 442/442 12/12
IPv4 qos aces: 512/512 21/21
IPv4 security aces: 954/954 35/35

Note: Allocation of TCAM entries per feature uses
a complex algorithm. The above information is meant
to provide an abstract view of the current TCAM utilization

1- If we make switch stack with two switches(total two switches). Will this help us to cope the over utilization matter. 

2- is it possible to trace which subnet is eating limit the most TCAM storage (6144/6144 6144/6144)

[Joseph W. Doherty · Accepted solution]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages wha2tsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Mark has already identified you problem and possible remediation approaches, but to a bit more . . .

I believe you're miss reading your stats.  The problem, I believe, is:

#show platform tcam utili

CAM Utilization for ASIC# 0                                            Max                 Used
                                                                                  Masks/Values    Masks/values

 Unicast mac addresses:                                              544/4352         66/432  
 IPv4 IGMP groups + multicast routes:                         152/1216          6/26   
 IPv4 unicast directly-connected routes:                      544/4352         66/432  
 IPv4 unicast indirectly-connected routes:                  176/1408        163/1226 
 IPv6 Multicast groups:                                                544/4352         66/432  
 IPv6 unicast directly-connected routes:                      544/4352         66/432  
 IPv6 unicast indirectly-connected routes:                    262/2096         18/84   
 IPv4 policy based routing aces:                                 256/256           4/4    
 IPv4 qos aces:                                                           768/768         324/324  
 IPv4 security aces:                                                      512/512          39/39   
 IPv6 policy based routing aces:                                       0/0             0/0    
 IPv6 qos aces:                                                                 0/0             0/0    
 IPv6 security aces:                                                     204/510           8/8    

Unfortunately, 3750s do not have large TCAMs, and when enabling IPv6, TCAM resources become very limited.

The two ways to mitigate the issue are either use a better SDM template (believe you're using the best), or figure out how to reduce the number of routes "seen" by this device.

You might want to also review:

http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html#pgfId-1004066

http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html#pgfId-1004830

[adonsius_artollo]

Thanks for your response. You were absolutely right, I wasn't reading the tables correctly... for "IPv4 policy based routing aces" for example, I thought 256/256 meant I was using all of the 256. Thanks for your assistance.

[adonsius_artollo]

Thank you for your response. The issue is exactly what you stated... I have made the recommendation to summarize upstream or upgrade to better hardware, and they chose better hardware. Thanks for your help.