Nexus 9000 TCAM exhausted but runs only 80%

PeterP11 · ‎07-26-2018

Hello,

i have a Nexus 93180LC-EX running on 7.0(3)I7(2).

Tcam is carved:

 NAT ACL[nat] size =    0
                        Ingress PACL [ing-ifacl] size =    0
                                     VACL [vacl] size =    0
                         Ingress RACL [ing-racl] size = 1792
                       Ingress RBACL [ing-rbacl] size =    0
                     Ingress L2 QOS [ing-l2-qos] size =  256
           Ingress L3/VLAN QOS [ing-l3-vlan-qos] size =  512
                           Ingress SUP [ing-sup] size =  512
     Ingress L2 SPAN filter [ing-l2-span-filter] size =  256
     Ingress L3 SPAN filter [ing-l3-span-filter] size =  256
                       Ingress FSTAT [ing-fstat] size =    0
                                     span [span] size =  512
                          Egress RACL [egr-racl] size =  512
                            Egress SUP [egr-sup] size =  256
                 Ingress Redirect [ing-redirect] size =    0
                      Egress L2 QOS [egr-l2-qos] size =  512
            Egress L3/VLAN QOS [egr-l3-vlan-qos] size =  768
                           Ingress NBM [ing-nbm] size =    0

although there still seems to be ressources left:

slot  1
=======



INSTANCE 0x0
-------------


         ACL Hardware Resource Utilization (Mod 1)
         ----------------------------------------------------------
                                        Used    Free    Percent
                                                        Utilization
-------------------------------------------------------------------

 Egress RACL OTHER                      0               0.00
Egress L3/VLAN QOS                      0       768     0.00
 Egress L3/VLAN QOS IPv4                0               0.00
 Egress L3/VLAN QOS IPv6                0               0.00
 Egress L3/VLAN QOS MAC                 0               0.00
 Egress L3/VLAN QOS ALL                 0               0.00
 Egress L3/VLAN QOS OTHER               0               0.00



INSTANCE 0x1
-------------


         ACL Hardware Resource Utilization (Mod 1)
         ----------------------------------------------------------
                                        Used    Free    Percent
                                                        Utilization
-------------------------------------------------------------------
Egress L3/VLAN QOS                      642     126     83.59
 Egress L3/VLAN QOS IPv4                633             82.42
 Egress L3/VLAN QOS IPv6                6               0.78
 Egress L3/VLAN QOS MAC                 3               0.39
 Egress L3/VLAN QOS ALL                 0               0.00
 Egress L3/VLAN QOS OTHER               0               0.00

an error comes up when editing the QOS-ACL

"%ACLQOS-SLOT1-2-ACLQOS_OOTR: Tcam resource exhausted: Egress L3/VLAN QOS [egr-l3-vlan-qos]"

Any Ideas or suggestions? Carving the TCAM should not be neccesarery because the the utilization is so low...

Thanks

Rajeshkumar Gatti · ‎07-26-2018

Peter,

Without knowing the existing qos configs and the additions you are making it would be difficult to provide a good explanation. My understanding would be that the changes you are making does not fit into the 20% free space. Would be good to share existing config and new additions that is trowing this error.

-Raj

PeterP11 · ‎07-31-2018

Hello,

i am not allowed to post the qos configuration, but what we want to do is something like that:

Nexus9k#show access-list

IP access list TEST-QOS-ACL
             statistics per-entry
             10 permit ip 10.1.1.1/24 any
             20 permit ip 10.2.2.2/24 any
             30 permit ip 10.3.4.3/24 any

Nexus9k#conf t
Nexus9k(config)#ip access-list TEST-QOS-ACL
Nexus9k(config-acl)#no 30
Sufficient free entries are available in TCAM bank

So the TCAM Ressources have 20 % Space left, but i am not able to remove a line in ACL which in my opinion should release some TCAM space...

Rajeshkumar Gatti · ‎07-31-2018

It is very likely that the reason for the error is due to atomic update which is enabled by default.

Check this link if you are not familiar with atomic update-

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-x/security/configuration/guide/b_Cisco_Nexus_9000_Series_NX-OS_Security_Configuration_Guide_7x/b_Cisco_Nexus_9000_Series_NX-OS_Security_Configuration_Guide_7x_chapter_01001.html...

Snippet below-

An atomic update requires that an I/O module that receives an ACL update has enough available resources to store each updated ACL entry in addition to all pre-existing entries in the affected ACL. After the update occurs, the additional resources used for the update are freed. If the I/O module lacks the required resources, the device generates an error message and the ACL update to the I/O module fails.

Hope this helps.

-Raj

Micheline Murphy · ‎07-31-2018

Hello and good morning--If you have a TCAM space that is being used by more than 50% the switch will not allow you to change any of the ACLs. Take a look at this document. The caveat is in the Guidelines and Limitations.

https://www.cisco.com/c/en/us/support/docs/switches/nexus-9000-series-switches/119032-nexus9k-tcam-00.html#anc6

HTH, MM