TCAM Carving on 9300EX

[j.a.m.e.s]

Dear All,

I would like to allocate some TCAM for arp-supression via this command:

hardware access-list tcam region arp-ether 256

I have read the TCAM Carving Guide, but it doesn't explain what impact turning off another TCAM region may have.

In my case, "ing-l2-span-filter" sounds like a reasonable feature to turn off but could anyone clarify what "ing-l2-span-filter" does?

switch# sho run all  | in tcam
hardware access-list tcam region ing-racl 1792
hardware access-list tcam region ing-l2-qos 256
hardware access-list tcam region ing-l3-vlan-qos 512
hardware access-list tcam region ing-sup 512
hardware access-list tcam region ing-l2-span-filter 256
hardware access-list tcam region ing-l3-span-filter 256
hardware access-list tcam region span 512
hardware access-list tcam region egr-racl 1792
hardware access-list tcam region egr-sup 256
[Everything else is 0]

The feature does not appear to be used:

switch# show hardware access-list resource utilization

slot  1
=======

INSTANCE 0x0
-------------
         ACL Hardware Resource Utilization (Mod 1)
         ----------------------------------------------------------
                                        Used    Free    Percent
                                                        Utilization
-------------------------------------------------------------------
Ingress L2 SPAN ACL                     0       256     0.00
 Ingress L2 SPAN ACL IPv4               0               0.00
 Ingress L2 SPAN ACL IPv6               0               0.00
 Ingress L2 SPAN ACL MAC                0               0.00
 Ingress L2 SPAN ACL ALL                0               0.00
 Ingress L2 SPAN ACL OTHER              0               0.00
[...]
Ingress L3/VLAN SPAN ACL                0       256     0.00
 Ingress L3/VLAN SPAN ACL IPv4          0               0.00
 Ingress L3/VLAN SPAN ACL IPv6          0               0.00
 Ingress L3/VLAN SPAN ACL MAC           0               0.00
 Ingress L3/VLAN SPAN ACL ALL           0               0.00
 Ingress L3/VLAN SPAN ACL OTHER         0               0.00
[...]
SPAN                                    0       512     0.00
 SPAN IPv4                              0               0.00
 SPAN IPv6                              0               0.00
 SPAN MAC                               0               0.00
 SPAN ALL                               0               0.00
 SPAN OTHER                             0               0.00
[...]
INSTANCE 0x1
-------------


         ACL Hardware Resource Utilization (Mod 1)
         ----------------------------------------------------------
                                        Used    Free    Percent
                                                        Utilization
-------------------------------------------------------------------

Ingress L2 SPAN ACL                     0       256     0.00
 Ingress L2 SPAN ACL IPv4               0               0.00
 Ingress L2 SPAN ACL IPv6               0               0.00
 Ingress L2 SPAN ACL MAC                0               0.00
 Ingress L2 SPAN ACL ALL                0               0.00
 Ingress L2 SPAN ACL OTHER              0               0.00
[...]
Ingress L3/VLAN SPAN ACL                0       256     0.00
 Ingress L3/VLAN SPAN ACL IPv4          0               0.00
 Ingress L3/VLAN SPAN ACL IPv6          0               0.00
 Ingress L3/VLAN SPAN ACL MAC           0               0.00
 Ingress L3/VLAN SPAN ACL ALL           0               0.00
 Ingress L3/VLAN SPAN ACL OTHER         0               0.00
[...]
SPAN                                    0       512     0.00
 SPAN IPv4                              0               0.00
 SPAN IPv6                              0               0.00
 SPAN MAC                               0               0.00
 SPAN ALL                               0               0.00
 SPAN OTHER                             0               0.00

I would rather know what I'm losing before I turn off a feature. Thanks for any guidance.

[educruz · Accepted solution]

Good day,

The switch TCAM is, in simple words, a memory where a few set of instructions can be allocated. As an example, if I create an Acess List, the switch will use its TCAM to allocate this order, and when a packet comes in, the switch will look up in such memory to see if there is a special instruction for such packet. 

Then, the TCAM memory is divided in parts, in order for features to install their respective set of instructions. When we ask the switch to carve the TCAM, it simply means that we want to distribute the memory space allocated to features differently. 

In the following guide, one can see that removing space from "ing-l2-span-filter" may cause the switch to lose its ability to use filters when one configures a SPAN session, or when one uses the Packet Tracer feature. 

ing-l2-span-filter: For filtering ingress
Layer 2 SPAN traffic (Cisco Nexus 9200
and 9300-EX Series switches only)

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-x/security/configuration/guide/b_Cisco_Nexus_9000_Series_NX-OS_Security_Configuration_Guide_7x/b_Cisco_Nexus_9000_Series_NX-OS_Security_Configuration_Guide_7x_chapter_01001.pdf

This may cause an issue if you would like to troubleshoot some other unrelated failure in the future, and a SPAN session is required. It would be advisable to browse the document above, and check which feature is used the least to make sure there are no undesired side effects.

Hope this helps.

Eduardo.

[j.a.m.e.s]

Thank you very much Eduardo, the guide provides a lot of answers.

This switch is a C93108TC-EX, and what I've noticed is that on trying to assign some TCAM to arp-ether (for EVPN ARP-suppression), I get:

switch(config)# hardware access-list tcam region arp-ether 256
                         ^
% Invalid command at '^' marker.

switch(config)# sho run all | in "arp-ether"
hardware access-list tcam region arp-ether 0

This is a bit weird, because I understood some of the earlier N9k switches needed this allocation.

[educruz]

Understood James, 

I believe in newer platforms, if you are to configure BGP EVPN (or VXLAN) in the long run, it would be necessary to enable “system routing template-vxlan-scale” which in turn allocates the required TCAM space. Some documentation about it can be found here.

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-x/vxlan/configuration/guide/b_Cisco_Nexus_9000_Series_NX-OS_VXLAN_Configuration_Guide_7x/b_Cisco_Nexus_9000_Series_NX-OS_VXLAN_Configuration_Guide_7x_chapter_011.html#concept_99...

My two theories right now would be either the main feature allocates the TCAM space automatically, or we may need to remove first some TCAM feature and try to reserve space for arp-ether again.

Also, could you check if "show logging last 10" threw any error message? 

Hope this helps.

Eduardo.

[j.a.m.e.s]

Hi Eduardo,

I've applied the "system routing template-vxlan-scale" and reloaded. Apparently this command is automated after 7.0(3)I5(1).

The ARP-Supression feature seems to enable with no errors:

interface nve1
  description VXLAN VTEP
  source-interface loopback2
  overlay-encapsulation vxlan
  host-reachability protocol bgp
  member vni 10328
    suppress-arp
    ingress-replication protocol bgp
  [...]
  member vni 103967 associate-vrf
exit

Still the arp-ether allocation shows up as zero:

switch# sho run all | in arp-ether
hardware access-list tcam region arp-ether 0

This looks unusual, but I guess it's best if I test it so see whether the feature works.

[educruz]

Hi James,

That looks good, so far. Are you able to go under configuration mode and try to carve the TCAM for "arp-ether" now? 

Otherwise, is there any way to test the feature to see if it allocates such space automatically?

Kind regards,

Eduardo.

[j.a.m.e.s]

Still the same result unfortunately:

switch(config)# hardware access-list tcam region arp-ether ^C
chges-d-falsa-01(config)# hardware access-list tcam region arp-ether 256
                           ^
% Invalid command at '^' marker.
switch(config)# hardware access-list tcam region ?
  egr-racl            Egress RACL
  egr-sup             Egress SUP
  ing-fstat           Ingress FSTAT
  ing-ifacl           Ingress PACL
  ing-l2-qos          Ingress L2 QOS
  ing-l2-span-filter  Ingress L2 SPAN filter
  ing-l3-span-filter  Ingress L3 SPAN filter
  ing-l3-vlan-qos     Ingress L3/VLAN QOS
  ing-racl            Ingress RACL
  ing-rbacl           Ingress RBACL
  ing-redirect        Ingress Redirect
  ing-sup             Ingress SUP
  nat                 Ingress NAT
  span                SPAN
  vacl                VACL

I'll be upgrading and testing these switches over the next few weeks, so will reply if the feature doesn't work.

[j.a.m.e.s]

Just to close down this thread, it appears that carving space for arp-ether is not required on the 9300-EX and FX series. You still have to do it on say the 9332PQ though:

conf ter
! Reduce RACL allocation:
hardware access-list tcam region racl 512
copy run start
! Provide ARP-Ether:
hardware access-list tcam region arp-ether 256 double-wide
copy run start
end

There is some documentation on this in the EVPN Config Example. by Lukas Krattiger.

[cha.raouf]

it works with me :)