Cisco Nexus 9396PX TCAM carving issue with arp-ether size

[operations123]

I am in middle of deploying Spine-Leaf design using Cisco 9396PX switches for my leaf and when i am trying to configure arp-suppression for VNI i found this error.

Default vacl tcam size is 512 so i am freeing up all of that for arp-ether

hardware access-list tcam region vacl 0

Giving slice to arp-ether but its throwing error 

hardware access-list tcam region arp-ether 256 double-wide
ERROR: Aggregate TCAM region configuration exceeded the available Ingress TCAM slices. Please re-configure.

If i don't use "double-wide" then getting no error at all, why? and what is the best approche double-wide or no? 

# hardware access-list tcam region arp-ether 256
Warning: Please save config and reload the system for the configuration to take effect

[Ali Aghababaei]

Hi operations123 ,

1. For VXLAN IGMP snooping functionality, the ARP-ETHER TCAM must be configured in the double-wide mode using the hardware access-list tcam region arp-ether 256 double wide command for Cisco Nexus 9300 switches. This command is not required for Cisco Nexus 9300-EX switches.
2. You must decrease the size of an existing TCAM region before using this command
3. A slice can be allocated to one region only. For example, a 512-size slice cannot be used to configure two features of size 256 each. Similarly, a 256-size slice cannot be used to configure two features of size 128 each.
4. The IPv4 TCAM regions are single wide. The IPv6, QoS, MAC, CoPP, and system TCAM regions are double wide and consume double the physical TCAM entries. For example, a logical region size of 256 entries actually consumes 512 physical TCAM entries.
5. A separate TCAM in the Cisco Nexus C9396PX (uplink ports) and Cisco Nexus C93128TX (uplink ports) ASIC is used for the QoS classification policies applied on 40G uplink ports. By default, this separate TCAM is carved for Layer 3 QoS (IPV4), Layer 2 Port QoS (IPV4), and VLAN QoS (IPV4) with 256 entries each.

I hope this will be useful,

Thanks,

Ali

[operations123]

Thank you for reply,

I am new in EVPN+VxLAN design and we have very small datacenter only 6 VTEP and using Multicast with PIM, I have no idea what is IGMP Snooping.

Question on IGMP:

1. where do i use IGMP snooping? (we don't have any VMware or vMotion etc in our datacenter if it required IGMP)

2. If i am not using IGMP then can i like with arp-ether without double-wide ?

3. All i need arp-suppression to reduce my ARP flood for anycast gateway vni. 

What do you suggest?

[Ali Aghababaei]

Hi,

1)

• By default, multicast traffic over VXLAN is flooded in the VNI/VLAN like any broadcast and unknown unicast traffic. With IGMP snooping enabled, each VTEP can snoop IGMP reports and only forward multicast traffic towards interested receivers. The configuration of IGMP snooping is the same in VXLAN as in the configuration of IGMP snooping in a regular VLAN domain.
  
  
• Guidelines and limitations for IGMP snooping over VXLAN:

           IGMP snooping over VXLAN is not supported on VLANs with FEX member ports.

           IGMP snooping over VXLAN is supported with both IR and multicast underlay.

           IGMP snooping over VXLAN is supported in BGP EVPN topologies, not flood and learn topologies.

• When configuring SVI with flood and learn mode on the central gateway leaf, it is mandatory to configure hardware access-list tcam region arp-ether size double-wide .
  
  
• When configuring ARP suppression with BGP-EVPN, use the hardware access-list tcam region arp-ether size double-wide command to accommodate ARP in this region.

2)

Of Course you can. For the ACI leaf line card on Cisco Nexus 9300 Series switches, only the IPv6 TCAM regions consume double-wide entries. The rest of the TCAM regions consume single-wide entries. 

3)
It's enough to use this :

# hardware access-list tcam region arp-ether 256

I hope you find it helpful,

Thanks,

Ali

[operations123]

Thank you for detailed reply,

long story short, I am not using any IGMP snooping , IPv6 etc that means i don't need double-wide tcam for arp-ether. right?

I have following 3 counter question while we talking about arp-suppression.

1. I am trying to reduce one more 512 slice of tcam from RACL and i got following warning, I am not running any ACL or anything but its saying 1 entry is in tcam not sure what is that and how to verify if any one using any IPv4 ACL in tcam. Just wanted to make sure before i ignore this error

leaf-1-1(config)# hardware access-list tcam region racl 0
WARNING: On module 1, 1 entries are in use in the region IPV4 RACL [racl] on instance 0, but carving size is 0 [0*1].
Warning: Please save config and reload the system for the configuration to take effect

2. I have my leaf switches in Cisco vPC pair and when i am going to add or remove arp-suppression to one of my VNI and i am seeing it freezed traffic for 40sec and none of my host can ping in that VLAN (look like it bringing down whole NVE or vPC).  This is what i am going, as soon as i added suppress-arp 

leaf-1-1(config)# interface nve1
leaf-1-1(config-if-nve)# member vni 10100
leaf-1-1(config-if-nve-vni)# suppress-arp

My ping to host stopped and i am seeing following result on vPC (Per-vlan consistency status : failed)

leaf-1-1# show vpc
Legend:
                (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                     : 2
Peer status                       : peer adjacency formed ok
vPC keep-alive status             : peer is alive
Configuration consistency status  : success
Per-vlan consistency status       : failed

After 40 to 60s after my pings come back.. This is happening when i add suppress-arp or i remove in both condition my host stop pinging for some time until Per-vlan consistency status change to success. is this normal behavior ?

3. I have some VNI which doesn't have anycast-gatway SVI in that case if i enable suppress-arp does that work or not ? (These VNI default gateway is my Cisco ASA firewall)

[operations123]

Do you think if in future i deploy ipv6 then i need double-wide arp suppression ?

access-list tcam region arp-ether size double-wide

[Ali Aghababaei]

The IPv6, QoS, MAC, CoPP, and system TCAM regions are double wide and consume double the physical TCAM entries. For example, a logical region size of 256 entries actually consumes 512 physical TCAM entries.

Leaf Node Configuration – L2 & L3 VNI for IPv6

Spoiler

vlan 100
vn-segment 10000
vlan 200
vn-segment 20000
evpn
vni 10000 l2
rd 10000:1
route-target import 10000:1
route-target export 10000:1
!
vrf context EVPN-TENANT
vni 20000
rd 20000:1
address-family ipv4 unicast
route-target both 20000:1
route-target both 20000:1 evpn
address-family ipv6 unicast
route-target both 20000:1
route-target both 20000:1 evpn
!
interface Vlan200
no shutdown
vrf member EVPN-TENANT
ip forward
ipv6 address use-link-local-only
fabric forwarding anycast-gateway-mac 0001.0001.0001
interface Vlan100
no shutdown
vrf member EVPN-TENANT
ip address 100.1.1.254/24
ipv6 address 2001::1/64
fabric forwarding mode anycast-gateway
!
interface nve1
no shutdown
source-interface loopback0
host-reachability protocol bgp
member vni 10000
mcast-group 239.1.1.1
suppress-arp
member vni 20000 associate-vrf
!
router bgp 100
vrf EVPN-TENANT
address-family ipv4 unicast
advertise l2vpn evpn
address-family ipv6 unicast
advertise l2vpn evpn
!
vpc domain 10
ipv6 nd synchronize

I hope it will be helpful.

Regards,

Ali