cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2402
Views
5
Helpful
8
Replies

TCAM carving question for Cisco Nexus 9396PX

satish.txt1
Level 1
Level 1
Spoiler
 

I have Cisco Nexus 9396PX and configure for IPv4 with IPv4 RACL on SVI to block some basic traffic. Now i have configured IPv6 and trying to configure access-list but its saying you don't have TCAM space so i started looking around to see where i can borrow and this is what i have.

As per document i may need 512 slice for ipv6 doble-width.

Question:

  1. can i combine two 256 to create 512?

  2. what is IPV4 PACL (i don't know who is using it and how to find out if someone using it?)

  3. I am using BFD on this switch ( does BFD using redirect tcam space?)

 

# show hardware access-list tcam region | exclude 0
                               IPV4 PACL [ifacl] size =  512
                             IPV4 Port QoS [qos] size =  256
                                IPV4 RACL [racl] size =  512
                       Egress IPV4 RACL [e-racl] size =  256
                                  Ingress System size =  256
                                   Egress System size =  256
                             Ingress COPP [copp] size =  256
                             Redirect [redirect] size =  512
                       NS IPV4 Port QoS [ns-qos] size =  256
                      NS IPV4 VLAN QoS [ns-vqos] size =  256
                       NS IPV4 L3 QoS [ns-l3qos] size =  256
 VPC Convergence/ES-Multi Home [vpc-convergence] size =  256
               Ingress ARP-Ether ACL [arp-ether] size =  256
                       ranger+ IPV4 QoS [rp-qos] size =  256
                  ranger+ IPV6 QoS [rp-ipv6-qos] size =  256
                    ranger+ MAC QoS [rp-mac-qos] size =  256
                               sFlow ACL [sflow] size =  256

IPv6 has zero allocation

# show hardware access-list tcam region | grep IPV6
                          IPV6 PACL [ipv6-ifacl] size =    0
                        IPV6 Port QoS [ipv6-qos] size =    0
                  FEX IPV6 PACL [fex-ipv6-ifacl] size =    0
                FEX IPV6 Port QoS [fex-ipv6-qos] size =    0
                           IPV6 VACL [ipv6-vacl] size =    0
                       IPV6 VLAN QoS [ipv6-vqos] size =    0
                           IPV6 RACL [ipv6-racl] size =    0
                    Egress IPV6 QoS [e-ipv6-qos] size =    0
                    Egress IPV6 VACL [ipv6-vacl] size =    0
                  Egress IPV6 RACL [e-ipv6-racl] size =    0
                        IPV6 L3 QoS [ipv6-l3qos] size =    0
                  NS IPV6 Port QoS [ns-ipv6-qos] size =    0
                 NS IPV6 VLAN QoS [ns-ipv6-vqos] size =    0
                  NS IPV6 L3 QoS [ns-ipv6-l3qos] size =    0
                  ranger+ IPV6 QoS [rp-ipv6-qos] size =  256

This is what my utilization tables looks (its saying PACL used 3 does that means i can't take that slice?)

        ACL Hardware Resource Utilization (Mod 1)
         ----------------------------------------------------------
                                        Used    Free    Percent
                                                        Utilization
-------------------------------------------------------------------
Ingress IPv4 PACL                       3       509     0.58
Ingress IPv4 Port QoS                   4       252     1.56
Ingress IPv4 RACL                       32      480     6.25
Egress IPv4 RACL                        3       253     1.17
SUP COPP                                214     42      83.59
SUP COPP Reason Code TCAM               8       120     6.25
Redirect                                7       505     1.36
Ingress Ether ACL                       15      241     5.85
VPC Convergence                         1       255     0.39
sFlow Northstar ACL                     0       256     0.00

LOU                                     2       22      8.33
Both LOU Operands                       2
Single LOU Operands                     0
LOU L4 src port:                        1
LOU L4 dst port:                        1
LOU L3 packet len:                      0
LOU IP tos:                             0
LOU IP dscp:                            0
LOU ip precedence:                      0
LOU ip TTL:                             0
TCP Flags                               0       16      0.00

Protocol CAM                            2       244     0.81
Mac Etype/Proto CAM                     0       14      0.00

L4 op labels, Tcam 0                    0       1023    0.00
L4 op labels, Tcam 2                    1       62      1.58
L4 op labels, Tcam 6                    0       2047    0.00

 

1 Accepted Solution

Accepted Solutions

Sergiu.Daniluk
VIP Alumni
VIP Alumni

Hi @satish.txt1 

 


1. can i combine two 256 to create 512?

Yes, you can take 2x 256 from two different regions or 1x 512 from one single region

 

2. what is IPV4 PACL (i don't know who is using it and how to find out if someone using it?)

PACL or port ACL is an access list applied on L2 interfaces or L2 port-channels.

Command to apply PACLs is

interface eth x/y 
ip port access-group access-list in

Note: The PACLs are only supported on the ingress direction, on all Nexus platforms!

 

3.I am using BFD on this switch ( does BFD using redirect tcam space?)


Yes, the redirect space is being used for BFD and DHCP relay.

 

From what I see you can take 256 from redirect and 254 from PACL. I would not recommend taking all 512 from PACL as they are very useful in troubleshooting (very very useful).

 

Stay safe,

Sergiu

View solution in original post

8 Replies 8

Sergiu.Daniluk
VIP Alumni
VIP Alumni

Hi @satish.txt1 

 


1. can i combine two 256 to create 512?

Yes, you can take 2x 256 from two different regions or 1x 512 from one single region

 

2. what is IPV4 PACL (i don't know who is using it and how to find out if someone using it?)

PACL or port ACL is an access list applied on L2 interfaces or L2 port-channels.

Command to apply PACLs is

interface eth x/y 
ip port access-group access-list in

Note: The PACLs are only supported on the ingress direction, on all Nexus platforms!

 

3.I am using BFD on this switch ( does BFD using redirect tcam space?)


Yes, the redirect space is being used for BFD and DHCP relay.

 

From what I see you can take 256 from redirect and 254 from PACL. I would not recommend taking all 512 from PACL as they are very useful in troubleshooting (very very useful).

 

Stay safe,

Sergiu

Hi @Sergiu.Daniluk 

 

Beautiful answer, Let me tell you about function of this switch. This switch is border-leaf switch of EVPN+VxLAN fabric and its dedicated for Border-leaf (ISP connectivity). It has no server/host attached. 

 

As a border-leaf i need following features. 

 

IPv4 RACL - Basic IPv4 access-list to block bad traffic

IPv6 RACL - Basic IPv6 access-list to block bad traffic

BFD - Fast failover with ISP (Otherwise it will take 90 second to failover, I tried to reduce timer but still longer time)

ARP-ETHER - Arp Suppression (I am not sure useful for Border-Leaf because its bridge between L3 and L2 fabric, What do you think about it? )

sFlow - This is to collect some netflow data on Public Interface. 

 

This is what TCAM carving arrangement i came up with so please advise me if see something wrong. 

 

border-leaf-1# show run | grep tcam
hardware access-list tcam region ifacl 256
hardware access-list tcam region qos 0
hardware access-list tcam region vacl 0
hardware access-list tcam region span 0
hardware access-list tcam region redirect 256
hardware access-list tcam region vpc-convergence 0
hardware access-list tcam region ipv6-racl 512
hardware access-list tcam region arp-ether 256
hardware access-list tcam region sflow 256

VACL/SPAN/VPC - I don't need this function on border-leaf 

QOS - I don't have any QoS configured on this switch so i have removed it slice. 

 

Do you think ifacl is useful for border-leaf role because i am not using it anywhere for L2 function, This switch is pretty much L3 to just bridge traffic between other leaf switch with Internet. 

 

 

 

 

 

 

 

 

Hi @satish.txt1 

If you have any switchport interfaces (trunk or access) on the BL - like for example routing neighborship with the ISP over SVIs , then it's definitely good to have ifacl present. If all your interfaces are L3 interfaces (no switchport), then you do not need ifacl at the moment. However, if in the future you will change it's purpose, and connect any hosts/endpoints to it, then better to be prepared for the ifacl (or allocate the space at that time - although might be challenging if the switch is in production).

 

Stay safe,

Sergiu

Hi @Sergiu.Daniluk 

 

Thank you, my BL has all routed interface L3 (no switchport) so definitely i don't need ifacl at present but curious where we use ifacl for troubleshooting because i have one more datacenter running on Cisco nexus using all vPC design and i never ever used port acl for anything. just trying to understand why and where they are useful. if they are just for filtering traffic between VLAN then we don't do that because we have local firewall running on each host.

 

At present i have two option give 256 to ifacl or QoS (at present neither one i am using for my configuration). which one i should pick between both of them?

 

Hi @satish.txt1 

Suppose you have a scenario like this:

 

[10.1.1.1] ------- Router1 ----- Router2 --- [eth1/1] Nexus [eth1/2] ------ [20.1.1.1]

But ping does not work between 10.1.1.1 and 20.1.1.1.

 

And let's say from routing perspective everything looks ok, but you do not have access to 20.1.1.1 so confirm if at least the ICMP request reach destination, or if it responds with replies.

What you can do on the Nexus is:

1. Configure an ACL to use for thsooting, where you permit traffic (I selected IP which includes all traffic, but you can be more specific like ICMP or the desired traffic) from 10.1.1.1 to 20.1.1.1, the return traffic 20.1.1.1 to 10.1.1.1 and all traffic (to not impact the rest of the services) + you enable statistics per entry:

 

ip access-list tshoot
  statistics per-entry
  10 permit ip 10.1.1.1/32 20.1.1.1/32 
  20 permit ip 20.1.1.1/32 10.1.1.1/32 
  30 permit ip any any 

2. You configure the access list as PACL on the interface eth1/1 and eth1/2:

 

 

interface eth 1/1-2 
ip port access-group tshoot in

3. You verify if the statistics increase for each entry while you send ICMP or whatever traffic you want.

N9K(config-if)# show ip access-lists tshoot

IP access list tshoot
        statistics per-entry 
        10 permit ip 10.1.1.1/32 20.1.1.1/32 [match=0]
        20 permit ip 20.1.1.1/32 10.1.1.1/32 [match=0]
        30 permit ip any any [match=10122]

4. Depending on the results you move your tshoot on other devices or you continue it on the nexus. As you can see in my output, I do not receive any traffic from 10.1.1.1 destined to 20.1.1.1, which means the problem is not on nexus, but on devices before it:  R1 or R2.

 

Stay safe,

Sergiu

 

P.S. Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem

 

Hi @Sergiu.Daniluk 

 

Beautiful explanation, I can see it's very useful for L2 troubleshooting. Thank you. 

 

In my case at present i don't need ifacl because i am using all L3 interface so giving 256 slice to QoS incase i need to use QoS in future. I think you answer my all question nicely. 

Darian O'Dirling - TCE
Cisco Employee
Cisco Employee

Hi @satish.txt1,
In reference to making space and borrowing TCAM storage here is a video made by one of my teammates on Nexus 9000 TCAM carving. I would encourage you to watch it to see if it helps fill any knowledge gaps regarding TCAM carving. You can find the link here: https://www.youtube.com/watch?v=ISJ4C8ISKCw

An introduction to carving TCAM memory on Nexus 9000 series switches.

rendy.rinaldy
Level 1
Level 1

Thanks for the information - a good find.

I shall also try updating to the latest to see if it fixes it this time.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: