TCAM Utilization

[fgasimzade]

Hello everyone!

I have noticed on my 3750 stack these things:

sh platform tcam utilization

CAM Utilization for ASIC# 0                      Max            Used
                                             Masks/Values    Masks/values

 Unicast mac addresses:                                528/4224        508/3998  
 IPv4 IGMP groups + multicast routes:          144/1152         11/47    
 IPv4 unicast directly-connected routes:       528/4224        508/3998  
 IPv4 unicast indirectly-connected routes:     272/2176         64/451   
 IPv4 policy based routing aces:                     512/512           2/2     
 IPv4 qos aces:                                              528/528          82/82    
 IPv4 security aces:                                        1024/2048         27/27  

# of HL3U fibs 2229
# of HL3U adjs 1831
# of HL3U mpaths 2
# of HL3U covering-fibs 1
# of HL3U fibs with adj failures 4
Fibs of Prefix length 0, with TCAM fails: 0
Fibs of Prefix length 1, with TCAM fails: 0
Fibs of Prefix length 2, with TCAM fails: 0
Fibs of Prefix length 3, with TCAM fails: 0
Fibs of Prefix length 4, with TCAM fails: 0
Fibs of Prefix length 5, with TCAM fails: 0
Fibs of Prefix length 6, with TCAM fails: 0
Fibs of Prefix length 7, with TCAM fails: 0
Fibs of Prefix length 8, with TCAM fails: 0
Fibs of Prefix length 9, with TCAM fails: 0
Fibs of Prefix length 10, with TCAM fails: 0
Fibs of Prefix length 11, with TCAM fails: 0
Fibs of Prefix length 12, with TCAM fails: 0
Fibs of Prefix length 13, with TCAM fails: 0
Fibs of Prefix length 14, with TCAM fails: 0
Fibs of Prefix length 15, with TCAM fails: 0
Fibs of Prefix length 16, with TCAM fails: 0
Fibs of Prefix length 17, with TCAM fails: 0
Fibs of Prefix length 18, with TCAM fails: 0
Fibs of Prefix length 19, with TCAM fails: 0
Fibs of Prefix length 20, with TCAM fails: 0
Fibs of Prefix length 21, with TCAM fails: 0
Fibs of Prefix length 22, with TCAM fails: 0
Fibs of Prefix length 23, with TCAM fails: 0
Fibs of Prefix length 24, with TCAM fails: 0
Fibs of Prefix length 25, with TCAM fails: 0
Fibs of Prefix length 26, with TCAM fails: 0
Fibs of Prefix length 27, with TCAM fails: 0
Fibs of Prefix length 28, with TCAM fails: 0
Fibs of Prefix length 29, with TCAM fails: 0
Fibs of Prefix length 30, with TCAM fails: 0
Fibs of Prefix length 31, with TCAM fails: 0
Fibs of Prefix length 32, with TCAM fails: 107973
Fibs of Prefix length 33, with TCAM fails: 0

As far as I understand, MAC address table is full, but what does this mean?

Pv4 unicast directly-connected routes:       528/4224        508/3998  

Do I have a lot of connected routes? I dont have more than 20 connected routes

sh ip arp summary gives this:

1925 IP ARP entries, with 50 of them incomplete

What should I do to optimize it?

Thank you!

[Seb Rupik]

Hi there,

What SDM template are you running?

sh sdm prefer

Do you handle any IPv6 traffic on this switch?

Do you run VTP on this switch? If not, are there VLANs trunked to it that carry a lot of L2 traffic but are not used on any switchports other than the uplink? If so, your mac address table will get filled with entries created by broadcasts.

What is this switches function, edge, distribution, core? If it is used as a core switch can you push some of the SVIs out towards edge and have purely L3 links between edge and core?

cheers,

Seb.

[fgasimzade]

Hello Seb, thank you for your answer

SDM is

sh sdm prefer
 The current template is "desktop access IPv4" template.
 The selected template optimizes the resources in
 the switch to support this level of features for
 8 routed interfaces and 1024 VLANs.

  number of unicast mac addresses:                  4K
  number of IPv4 IGMP groups + multicast routes:    1K
  number of IPv4 unicast routes:                    6K
    number of directly-connected IPv4 hosts:        4K
    number of indirect IPv4 routes:                 2K
  number of IPv4 policy based routing aces:         0.5K
  number of IPv4/MAC qos aces:                      0.5K
  number of IPv4/MAC security aces:                 2K

No, IPv6 traffic, no VTP. And yes, there are VLANs trunked but not used on edge ports since this is our Bridge in MST

The switch is edge/distribution.

Is it an SVI problem? Do I have to minimize them?

[Seb Rupik]

According to:

http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3750/software/release/12-2_55_se/configuration/guide/scg3750/swsdm.html

...you will gain another 2k mac addresses by running:

!

sdm prefer default

!

This would require a reload of the switch.

The SVIs are only a problem if they are seeing a lot of MAC addresses, moving the routing of these towards your edge may free up some space.

For each routed VLAN, what is the output of:

sh mac address-table dynamic vlan <xx>

cheers,

Seb

[fgasimzade]

Thank you,

I know I can switch to Default but it requires restart of the switch stack which is the last option

I was wondering what can cause this

iPv4 unicast directly-connected routes:       528/4224        508/3998  

and this:

Fibs of Prefix length 32, with TCAM fails: 107973

Is it MAC addresses or something else?

[Seb Rupik]

Each host (/32) will occupy a FIB entry, these are the 'directly-connected routes'.

The TCAM fails for /32 indicate that the switch is unable to create a new FIB entry when it sees a new L3 address so packets must be punted to the CPU in order to route them (this will cause increase CPU ultilization).

Clearly for the subnets you are routing on the 3750 you are exceeding the maximum number hosts it can store in its FIB and efficiently route. You need to move (some of) your SVIs.
 

[fgasimzade]

Thank you Seb,

So just to make it clear each host address must be included in FIB right?

So directly connected routes are not only those I create with SVI, but also all the hosts IP addresses for these SVIs, right?

[Seb Rupik · Accepted solution]

Yes, for efficient routing the directly connected hosts must have a FIB entry.

cheers,

Seb.

[fgasimzade]

Dear Seb,

I have removed around 300 IP hosts from the switch. I used to have around 2100 ARP entries, no I have 1800

But it still shows

Fibs of Prefix length 32, with TCAM fails: 108689

And this number did not change

IPv4 unicast directly-connected routes:       528/4224        508/3996 

Am I doing something wrong?

[Seb Rupik]

Morning,

From the DHCP leases you offer on the remaing connected VLANs which you are still routing on the switch do you know if the maximum concurrent leases exceeds 528?

The TCAM fails will still increment. I don't think you can clear the counter without a switch reload. With the changes you have made so far I am sure the number will be climbing slower.

cheers,

Seb.

[fgasimzade]

Hello Seb,

Yes, it is more than 528

The TCAM fails number fluctuates, from more to less, but it is always around 108765

Do I still have to minimize my connected routes? When removed around 300 IP hosts from the switch, TCAM fails number did not change dramatically, sometime it is even more

[Seb Rupik]

How many DHCP leases are in use on these VLANs?

I struggle to belive the TCAM fails is not a counter, I would expect it to always climb.

You really need to consider chaging the SDM template. Can you schedule an outage for the weekend?

[fgasimzade]

DHCP leases is around 1300.

It is too late for a schedule on this weekend, I will have to postpone it till next week