cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
4601
Views
0
Helpful
1
Replies

Nexus 9300 - TCAM Carving

Robert Saurer
Level 1
Level 1

Hi!

I have a Nexus 9372PX vPC pair with Cisco recommended NX-OS 7.0(3)I1(3) and need to filter traffic with a MAC PACL on a specific port and I'd like to mark DSCP based on ACLs on 2248TP-E FEX HIF ports with QoS statistics (so no qos-lite or fex-qos-lite).

I am struggling to get enough ingress TCAM free for [fex-qos]:

dc1-n9k-2(config)# hardware access-list tcam region fex-qos 256
ERROR: Aggregate TCAM region configuration exceeded the available Ingress TCAM slices. Please re-configure.

I've already reduced other PACLs/VACLs/RACLs to 0.

The system tells me that I am currently using 3072 of 4096 possible TCAM entries in ingress direction for TCAM Instance 0.

Has anyone had a same or similar problem?

dc1-n9k-2# show run | i "hardware access-list"
hardware access-list tcam region ifacl 0
hardware access-list tcam region vacl 0
hardware access-list tcam region racl 0
hardware access-list tcam region mac-ifacl 256
dc1-n9k-2# show system internal access-list globals
slot  1
=======

  Atomic Update : ENABLED
  Default ACL   : DENY
  Bank Chaining : DISABLED
  Fabric path DNL : DISABLED
  NS Buffer Profile: Mesh optimized
  Min Buffer Profile: all
  EOQ Class Stats: qos-group-0
  NS MCQ3 Alias: qos-group-3
  Ing PG Share: ENABLED
  IPG in Shape: DISABLED
  LOU Threshold Value : 5
----------------------------------------------------------------            ----------------
         INSTANCE 0 TCAM Region Information:
----------------------------------------------------------------              ----------------
Ingress:
----------
                        Region                  GID     Base     Size     Width    
----------------------------------------------------------------              ----------------
                    IPV4 PACL [ifacl](   1)       3        0        0        1
               IPV6 PACL [ipv6-ifacl](   2)       4        0        0        2
                 MAC PACL [mac-ifacl](   3)       5     2048      256        2
                  IPV4 Port QoS [qos](   4)       6      512      256        2
             IPV6 Port QoS [ipv6-qos](   5)       7        0        0        2
               MAC Port QoS [mac-qos](   6)       8        0        0        2
            FEX IPV4 PACL [fex-ifacl](   7)       9        0        0        1
       FEX IPV6 PACL [fex-ipv6-ifacl](   8)      10        0        0        2
         FEX MAC PACL [fex-mac-ifacl](   9)      11        0        0        2
          FEX IPV4 Port QoS [fex-qos](  10)      12        0        0        2
     FEX IPV6 Port QoS [fex-ipv6-qos](  11)      13        0        0        2
       FEX MAC Port QoS [fex-mac-qos](  12)      14        0        0        2
                     IPV4 VACL [vacl](  13)      15        0        0        1
                IPV6 VACL [ipv6-vacl](  14)      16        0        0        2
                  MAC VACL [mac-vacl](  15)      17        0        0        2
                 IPV4 VLAN QoS [vqos](  16)      18        0        0        2
            IPV6 VLAN QoS [ipv6-vqos](  17)      19        0        0        2
              MAC VLAN QoS [mac-vqos](  18)      20        0        0        2
                     IPV4 RACL [racl](  19)      21        0        0        1
                IPV6 RACL [ipv6-racl](  20)      22        0        0        2
        IPV4 Port QoS Lite [qos-lite](  21)      63        0        0        1
FEX IPV4 Port QoS Lite [fex-qos-lite](  22)      64        0        0        1
       IPV4 VLAN QoS Lite [vqos-lite](  23)      65        0        0        1
        IPV4 L3 QoS Lite [l3qos-lite](  24)      66        0        0        1
                  IPV4 L3 QoS [l3qos](  34)      37        0        0        2
             IPV6 L3 QoS [ipv6-l3qos](  35)      38        0        0        2
               MAC L3 QoS [mac-l3qos](  36)      39        0        0        2
                       Ingress System(  37)       1     1024      256        2
                          SPAN [span](  39)       2     2560      256*       1
                  Ingress COPP [copp](  40)      40     1536      256        2
         Ingress Flow Counters [flow](  41)      43        0        0        1
           Ingress SVI Counters [svi](  43)      45        0        0        1
                  Redirect [redirect](  44)      46        0      512        1
            NS IPV4 Port QoS [ns-qos](  45)      47        0        0        1
       NS IPV6 Port QoS [ns-ipv6-qos](  46)      48        0        0        2
         NS MAC Port QoS [ns-mac-qos](  47)      49        0        0        1
           NS IPV4 VLAN QoS [ns-vqos](  48)      50        0        0        1
      NS IPV6 VLAN QoS [ns-ipv6-vqos](  49)      51        0        0        2
        NS MAC VLAN QoS [ns-mac-vqos](  50)      52        0        0        1
            NS IPV4 L3 QoS [ns-l3qos](  51)      53        0        0        1
       NS IPV6 L3 QoS [ns-ipv6-l3qos](  52)      54        0        0        2
         NS MAC L3 QoS [ns-mac-l3qos](  53)      55        0        0        1
    VPC Convergence [vpc-convergence](  54)      57     3328      256*       1
       IPSG SMAC-IP bind table [ipsg](  55)      59        0        0        1
    Ingress ARP-Ether ACL [arp-ether](  56)      62        0        0        1
----------------------------------------------------------------------
* - allocated 512 entry slice due to unavailability of 256 entry slices
----------------------------------------------------------------              ----------------
Total: 3072
----------------------------------------------------------------              ----------------
Egress
----------
                        Region                  GID     Base     Size     Width    
----------------------------------------------------------------              ----------------
              Egress IPV4 QoS [e-qos](  25)      28        0        0        2
         Egress IPV6 QoS [e-ipv6-qos](  26)      29        0        0        2
           Egress MAC QoS [e-mac-qos](  27)      30        0        0        2
              Egress IPV4 VACL [vacl](  28)      31        0        0        1
         Egress IPV6 VACL [ipv6-vacl](  29)      32        0        0        2
           Egress MAC VACL [mac-vacl](  30)      33        0        0        2
            Egress IPV4 RACL [e-racl](  31)      34     3840      256        1
       Egress IPV6 RACL [e-ipv6-racl](  32)      35        0        0        2
    Egress IPV4 QoS Lite [e-qos-lite](  33)      36        0        0        1
                        Egress System(  38)      24     3072      256        1
        Egress Flow Counters [e-flow](  42)      44        0        0        1
----------------------------------------------------------------------
Total: 512
----------------------------------------------------------------              ----------------
----------------------------------------------------------------            ----------------
         INSTANCE 1 TCAM Region Information:
----------------------------------------------------------------              ----------------
Ingress:
----------
                        Region                  GID     Base     Size     Width    
----------------------------------------------------------------              ----------------
                    IPV4 PACL [ifacl](   1)       3        0        0        1
               IPV6 PACL [ipv6-ifacl](   2)       4        0        0        2
                 MAC PACL [mac-ifacl](   3)       5        0        0        2
                  IPV4 Port QoS [qos](   4)       6        0        0        2
             IPV6 Port QoS [ipv6-qos](   5)       7        0        0        2
               MAC Port QoS [mac-qos](   6)       8        0        0        2
            FEX IPV4 PACL [fex-ifacl](   7)       9        0        0        1
       FEX IPV6 PACL [fex-ipv6-ifacl](   8)      10        0        0        2
         FEX MAC PACL [fex-mac-ifacl](   9)      11        0        0        2
          FEX IPV4 Port QoS [fex-qos](  10)      12        0        0        2
     FEX IPV6 Port QoS [fex-ipv6-qos](  11)      13        0        0        2
       FEX MAC Port QoS [fex-mac-qos](  12)      14        0        0        2
                     IPV4 VACL [vacl](  13)      15        0        0        1
                IPV6 VACL [ipv6-vacl](  14)      16        0        0        2
                  MAC VACL [mac-vacl](  15)      17        0        0        2
                 IPV4 VLAN QoS [vqos](  16)      18        0        0        2
            IPV6 VLAN QoS [ipv6-vqos](  17)      19        0        0        2
              MAC VLAN QoS [mac-vqos](  18)      20        0        0        2
                     IPV4 RACL [racl](  19)      21        0        0        1
                IPV6 RACL [ipv6-racl](  20)      22        0        0        2
        IPV4 Port QoS Lite [qos-lite](  21)      63        0        0        1
FEX IPV4 Port QoS Lite [fex-qos-lite](  22)      64        0        0        1
       IPV4 VLAN QoS Lite [vqos-lite](  23)      65        0        0        1
        IPV4 L3 QoS Lite [l3qos-lite](  24)      66        0        0        1
                  IPV4 L3 QoS [l3qos](  34)      37        0        0        2
             IPV6 L3 QoS [ipv6-l3qos](  35)      38        0        0        2
               MAC L3 QoS [mac-l3qos](  36)      39        0        0        2
                       Ingress System(  37)       1        0        0        2
                          SPAN [span](  39)       2        0        0        1
                  Ingress COPP [copp](  40)      40        0        0        2
         Ingress Flow Counters [flow](  41)      43        0        0        1
           Ingress SVI Counters [svi](  43)      45        0        0        1
                  Redirect [redirect](  44)      46        0        0        1
            NS IPV4 Port QoS [ns-qos](  45)      47      312      256        1
       NS IPV6 Port QoS [ns-ipv6-qos](  46)      48        0        0        2
         NS MAC Port QoS [ns-mac-qos](  47)      49        0        0        1
           NS IPV4 VLAN QoS [ns-vqos](  48)      50      568      256        1
      NS IPV6 VLAN QoS [ns-ipv6-vqos](  49)      51        0        0        2
        NS MAC VLAN QoS [ns-mac-vqos](  50)      52        0        0        1
            NS IPV4 L3 QoS [ns-l3qos](  51)      53       56      256        1
       NS IPV6 L3 QoS [ns-ipv6-l3qos](  52)      54        0        0        2
         NS MAC L3 QoS [ns-mac-l3qos](  53)      55        0        0        1
    VPC Convergence [vpc-convergence](  54)      57        0        0        1
       IPSG SMAC-IP bind table [ipsg](  55)      59        0        0        1
    Ingress ARP-Ether ACL [arp-ether](  56)      62        0        0        1
----------------------------------------------------------------------
Total: 768
----------------------------------------------------------------              ----------------
Egress
----------
                        Region                  GID     Base     Size     Width    
----------------------------------------------------------------              ----------------
              Egress IPV4 QoS [e-qos](  25)      28        0        0        2
         Egress IPV6 QoS [e-ipv6-qos](  26)      29        0        0        2
           Egress MAC QoS [e-mac-qos](  27)      30        0        0        2
              Egress IPV4 VACL [vacl](  28)      31        0        0        1
         Egress IPV6 VACL [ipv6-vacl](  29)      32        0        0        2
           Egress MAC VACL [mac-vacl](  30)      33        0        0        2
            Egress IPV4 RACL [e-racl](  31)      34        0        0        1
       Egress IPV6 RACL [e-ipv6-racl](  32)      35        0        0        2
    Egress IPV4 QoS Lite [e-qos-lite](  33)      36        0        0        1
                        Egress System(  38)      24        0        0        1
        Egress Flow Counters [e-flow](  42)      44        0        0        1
----------------------------------------------------------------------
Total: 0
----------------------------------------------------------------              ----------------
dc1-n9k-2# show hardware access-list tcam region
TCAM Region Sizes:
                               IPV4 PACL [ifacl] size =    0
                          IPV6 PACL [ipv6-ifacl] size =    0
                            MAC PACL [mac-ifacl] size =  256
                             IPV4 Port QoS [qos] size =  256
                        IPV6 Port QoS [ipv6-qos] size =    0
                          MAC Port QoS [mac-qos] size =    0
                       FEX IPV4 PACL [fex-ifacl] size =    0
                  FEX IPV6 PACL [fex-ipv6-ifacl] size =    0
                    FEX MAC PACL [fex-mac-ifacl] size =    0
                     FEX IPV4 Port QoS [fex-qos] size =    0
                FEX IPV6 Port QoS [fex-ipv6-qos] size =    0
                  FEX MAC Port QoS [fex-mac-qos] size =    0
                                IPV4 VACL [vacl] size =    0
                           IPV6 VACL [ipv6-vacl] size =    0
                             MAC VACL [mac-vacl] size =    0
                            IPV4 VLAN QoS [vqos] size =    0
                       IPV6 VLAN QoS [ipv6-vqos] size =    0
                         MAC VLAN QoS [mac-vqos] size =    0
                                IPV4 RACL [racl] size =    0
                           IPV6 RACL [ipv6-racl] size =    0
                   IPV4 Port QoS Lite [qos-lite] size =    0
           FEX IPV4 Port QoS Lite [fex-qos-lite] size =    0
                  IPV4 VLAN QoS Lite [vqos-lite] size =    0
                   IPV4 L3 QoS Lite [l3qos-lite] size =    0
                         Egress IPV4 QoS [e-qos] size =    0
                    Egress IPV6 QoS [e-ipv6-qos] size =    0
                      Egress MAC QoS [e-mac-qos] size =    0
                         Egress IPV4 VACL [vacl] size =    0
                    Egress IPV6 VACL [ipv6-vacl] size =    0
                      Egress MAC VACL [mac-vacl] size =    0
                       Egress IPV4 RACL [e-racl] size =  256
                  Egress IPV6 RACL [e-ipv6-racl] size =    0
               Egress IPV4 QoS Lite [e-qos-lite] size =    0
                             IPV4 L3 QoS [l3qos] size =    0
                        IPV6 L3 QoS [ipv6-l3qos] size =    0
                          MAC L3 QoS [mac-l3qos] size =    0
                                  Ingress System size =  256
                                   Egress System size =  256
                                     SPAN [span] size =  256
                             Ingress COPP [copp] size =  256
                    Ingress Flow Counters [flow] size =    0
                   Egress Flow Counters [e-flow] size =    0
                      Ingress SVI Counters [svi] size =    0
                             Redirect [redirect] size =  512
                       NS IPV4 Port QoS [ns-qos] size =  256
                  NS IPV6 Port QoS [ns-ipv6-qos] size =    0
                    NS MAC Port QoS [ns-mac-qos] size =    0
                      NS IPV4 VLAN QoS [ns-vqos] size =  256
                 NS IPV6 VLAN QoS [ns-ipv6-vqos] size =    0
                   NS MAC VLAN QoS [ns-mac-vqos] size =    0
                       NS IPV4 L3 QoS [ns-l3qos] size =  256
                  NS IPV6 L3 QoS [ns-ipv6-l3qos] size =    0
                    NS MAC L3 QoS [ns-mac-l3qos] size =    0
               VPC Convergence [vpc-convergence] size =  256
                  IPSG SMAC-IP bind table [ipsg] size =    0
               Ingress ARP-Ether ACL [arp-ether] size =    0
             ranger+ IPV4 QoS Lite [rp-qos-lite] size =    0
                       ranger+ IPV4 QoS [rp-qos] size =  256
                  ranger+ IPV6 QoS [rp-ipv6-qos] size =  256
                    ranger+ MAC QoS [rp-mac-qos] size =  256
1 Reply 1

Robert Parten
Level 1
Level 1

The width is double wide, so even though you may have freed up 256, it actually requires 512. 

This scenario is a "rob Peter to pay Paul" scenario, but weird.

You will need to free up BOTH space and slices. In your scenario, you need more "slices"; thus, more entries:

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-x/qos/configuration/guide/b_Cisco_Nexus_9000_Series_NX-OS_Quality_of_Service_Configuration_Guide_7x/b_Cisco_Nexus_9000_Series_NX-OS_Quality_of_Service_Configuration_Guide_7x_chap...

http://www.cisco.com/c/en/us/support/docs/switches/nexus-9000-series-switches/119032-nexus9k-tcam-00.html