- Cisco Community
- Technology and Support
- Data Center and Cloud
- Server Networking
- Nexus 9k TCAM %ACLQOS-SLOT1-2-ACLQOS_OOTR: Tcam resource?
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Nexus 9k TCAM %ACLQOS-SLOT1-2-ACLQOS_OOTR: Tcam resource?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-06-2017 03:07 AM - edited 03-01-2019 01:40 PM
Hi all,
Need help with our new Nexus switches in our data centre, 9k.
We are getting the following message come up and ports being disabled due to the following:
%ACLQOS-SLOT1-2-ACLQOS_OOTR: Tcam resource
exhausted: Ingress L2 QOS [ing-l2-qos]
On the Cisco site for these errors it states the action is:
No DDTS
I have no idea what No DDTS means? If this is the only resolution to this then that is a real problem as we have live services being affected by this TCAM resource being exhausted.
Anyone come across this or have a solution?
Thanks
- Labels:
-
Other Server Networking
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-06-2017 06:12 AM
Hi,
Are you seeing logs similar to the following in "show logging log":
2017 Mar 25 23:57:40 Nexus9K %ETHPORT-5-IF_DOWN_ERROR_DISABLED: Interface Ethernet1/10 is down (Error disabled. Reason:Sufficient free entries are not available in TCAM bank)
What you are seeing is not a defect (this is what NO DDTS also means, by the way). The switch is basically letting you know that you've exhausted the TCAM resources for a particular feature/configuration. In your case, the Ingress L2 QoS region.
You can view this via:
----`show system internal access-list resource utilization`----
slot 1
=======
INSTANCE 0x0
-------------
ACL Hardware Resource Utilization (Mod 1)
----------------------------------------------------------
Used Free Percent
Utilization
-------------------------------------------------------------------
Ingress L2 QOS 240 16 93.75
Ingress L2 QOS IPv4 240 93.75
The TCAM for the Nexus 9000 must be carved correctly for the features/configuration that are needed. Once the TCAM is carved, a chassis reload will be required to program the switch for the new carving and regions.
Here's the Nexus 9000 TCAM Carving Guide
Hope that helps.
- Andrea
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-06-2017 07:07 AM
We are applying QoS on both uplinks and access ports.
Do you think this is why we are running out of memory?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-06-2017 01:45 PM
Hi there,
Well, it is okay to apply QoS to Layer 2 interfaces; however, the TCAM would've needed to be carved correctly for Ingress Layer 2 QoS so the switch can support the configuration for said amount of interfaces.
Hope that helps.
- Andrea
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-06-2017 01:51 PM
We have attempted to prevent the exhausted tcam by taking memory from other services on the switches but concerned we may cause problems with other functions.
Could the issue be with the QoS policy itself do you think?
If we remove the QoS completely the issue subsides.
Do you have a guide as to the best way to carve the tcam for an intensive QoS policy at layer 2?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-06-2017 01:58 PM
Can you share the output to the following - I'll see what you have enabled and what is currently carved and make a suggestion based on that:
show module show run | i feature sh hardware access-list tcam region show hardware access-list resource utilization
Thanks,
- Andrea
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-07-2017 01:01 AM
HQDC-ACC-SW-02A# show module
Mod Ports Module-Type Model Status
--- ----- ------------------------------------- --------------------- ---------
1 54 48x10GT + 6x40G/100G Ethernet Module N9K-C93108TC-EX active *
Mod Sw Hw Slot
--- ---------------- ------ ----
1 7.0(3)I4(2) 1.0 NA
Mod MAC-Address(es) Serial-Num
--- -------------------------------------- ----------
1 00-a6-ca-09-77-90 to 00-a6-ca-09-77-df FDO203308SH
Mod Online Diag Status
--- ------------------
1 Pass
* this terminal session
HQDC-ACC-SW-02A# show run | i feature
feature scp-server
feature tacacs+
feature eigrp
feature udld
feature interface-vlan
feature lacp
feature vpc
feature lldp
feature vtp
HQDC-ACC-SW-02A# sh hardware access-list tcam region
NAT ACL[nat] size = 0
Ingress PACL [ing-ifacl] size = 0
VACL [vacl] size = 0
Ingress RACL [ing-racl] size = 512
Ingress RBACL [ing-rbacl] size = 0
Ingress L2 QOS [ing-l2-qos] size = 1536
Ingress L3/VLAN QOS [ing-l3-vlan-qos] size = 512
Ingress SUP [ing-sup] size = 512
Ingress L2 SPAN filter [ing-l2-span-filter] size = 256
Ingress L3 SPAN filter [ing-l3-span-filter] size = 256
Ingress FSTAT [ing-fstat] size = 0
span [span] size = 512
Egress RACL [egr-racl] size = 1792
Egress SUP [egr-sup] size = 256
Ingress Redirect [ing-redirect] size = 0
HQDC-ACC-SW-02A# show hardware access-list resource utilization
slot 1
=======
INSTANCE 0x0
-------------
ACL Hardware Resource Utilization (Mod 1)
----------------------------------------------------------
Used Free Percent
Utilization
-------------------------------------------------------------------
Ingress L2 QOS 1408 128 91.67
Ingress L2 QOS IPv4 1360 88.54
Ingress L2 QOS IPv6 32 2.08
Ingress L2 QOS MAC 16 1.04
Ingress L2 QOS ALL 0 0.00
Ingress L2 QOS OTHER 0 0.00
Ingress L2 SPAN ACL 0 256 0.00
Ingress L2 SPAN ACL IPv4 0 0.00
Ingress L2 SPAN ACL IPv6 0 0.00
Ingress L2 SPAN ACL MAC 0 0.00
Ingress L2 SPAN ACL ALL 0 0.00
Ingress L2 SPAN ACL OTHER 0 0.00
Ingress RACL 0 512 0.00
Ingress RACL IPv4 0 0.00
Ingress RACL IPv6 0 0.00
Ingress RACL MAC 0 0.00
Ingress RACL ALL 2 0.39
Ingress RACL OTHER 0 0.00
Ingress L3/VLAN QOS 4 508 0.78
Ingress L3/VLAN QOS IPv4 0 0.00
Ingress L3/VLAN QOS IPv6 0 0.00
Ingress L3/VLAN QOS MAC 0 0.00
Ingress L3/VLAN QOS ALL 4 0.78
Ingress L3/VLAN QOS OTHER 0 0.00
Ingress L3/VLAN SPAN ACL 0 256 0.00
Ingress L3/VLAN SPAN ACL IPv4 0 0.00
Ingress L3/VLAN SPAN ACL IPv6 0 0.00
Ingress L3/VLAN SPAN ACL MAC 0 0.00
Ingress L3/VLAN SPAN ACL ALL 0 0.00
Ingress L3/VLAN SPAN ACL OTHER 0 0.00
Ingress SUP 396 76 83.90
Ingress SUP IPv4 129 27.33
Ingress SUP IPv6 176 37.29
Ingress SUP MAC 35 7.42
Ingress SUP ALL 10 2.12
Ingress SUP OTHER 46 9.75
SPAN 0 512 0.00
SPAN IPv4 0 0.00
SPAN IPv6 0 0.00
SPAN MAC 0 0.00
SPAN ALL 0 0.00
SPAN OTHER 0 0.00
Egress RACL 0 1792 0.00
Egress RACL IPv4 0 0.00
Egress RACL IPv6 0 0.00
Egress RACL MAC 0 0.00
Egress RACL ALL 2 0.11
Egress RACL OTHER 0 0.00
Egress SUP 128 128 50.00
Egress SUP IPv4 0 0.00
Egress SUP IPv6 0 0.00
Egress SUP MAC 0 0.00
Egress SUP ALL 0 0.00
Egress SUP OTHER 128 50.00
Feature BFD 2 14 12.50
Feature BFD IPv4 0 0.00
Feature BFD IPv6 0 0.00
Feature BFD MAC 0 0.00
Feature BFD ALL 2 12.50
Feature BFD OTHER 0 0.00
Feature DHCP SNOOP 0 16 0.00
Feature DHCP SNOOP IPv4 0 0.00
Feature DHCP SNOOP IPv6 0 0.00
Feature DHCP SNOOP MAC 0 0.00
Feature DHCP SNOOP ALL 0 0.00
Feature DHCP SNOOP OTHER 0 0.00
Feature ARP SNOOP 0 2 0.00
Feature ARP SNOOP IPv4 0 0.00
Feature ARP SNOOP IPv6 0 0.00
Feature ARP SNOOP MAC 0 0.00
Feature ARP SNOOP ALL 0 0.00
Feature ARP SNOOP OTHER 0 0.00
Feature VxLAN OAM 0 2 0.00
Feature VxLAN OAM IPv4 0 0.00
Feature VxLAN OAM IPv6 0 0.00
Feature VxLAN OAM MAC 0 0.00
Feature VxLAN OAM ALL 0 0.00
Feature VxLAN OAM OTHER 0 0.00
Feature DHCPv6 RELAY 0 4 0.00
Feature DHCPv6 RELAY IPv4 0 0.00
Feature DHCPv6 RELAY IPv6 0 0.00
Feature DHCPv6 RELAY MAC 0 0.00
Feature DHCPv6 RELAY ALL 0 0.00
Feature DHCPv6 RELAY OTHER 0 0.00
LOU 8 8 50.00
Both LOU Operands 8
Single LOU Operands 0
LOU L4 src port: 3
LOU L4 dst port: 5
LOU L3 packet len: 0
LOU IP tos: 0
LOU IP dscp: 0
LOU ip precedence: 0
LOU ip TTL: 0
TCP Flags 0 16 0.00
Protocol CAM 2 244 0.81
Mac Etype/Proto CAM 0 14 0.00
Non L4op labels, Tcam 0 16 14 53.33
Non L4op labels, Tcam 1 0 62 0.00
Non L4op labels, Tcam 2 0 4095 0.00
Non L4op labels, Tcam 3 1 1022 0.09
Non L4op labels, Tcam 4 0 0 0.00
L4 op labels, Tcam 5 0 1023 0.00
L4 op labels, Tcam 6 0 1023 0.00
L4 op labels, Tcam 7 0 1024 0.00
L4 op labels, Tcam 8 0 254 0.00
L4 op labels, Tcam 9 0 1023 0.00
L4 op labels, Tcam 10 0 1023 0.00
L4 op labels, Tcam 11 0 1023 0.00
L4 op labels, Tcam 12 0 1023 0.00
L4 op labels, Tcam 13 0 1023 0.00
L4 op labels, Tcam 14 0 1023 0.00
L4 op labels, Tcam 15 0 1023 0.00
L4 op labels, Tcam 16 0 1023 0.00
Ingress Dest info table 0 512 0.00
Egress Dest info table 0 512 0.00
Note: Ingress SUP region includes the Redirect region
INSTANCE 0x1
-------------
ACL Hardware Resource Utilization (Mod 1)
----------------------------------------------------------
Used Free Percent
Utilization
-------------------------------------------------------------------
Ingress L2 QOS 1144 392 74.48
Ingress L2 QOS IPv4 1105 71.94
Ingress L2 QOS IPv6 26 1.69
Ingress L2 QOS MAC 13 0.85
Ingress L2 QOS ALL 0 0.00
Ingress L2 QOS OTHER 0 0.00
Ingress L2 SPAN ACL 0 256 0.00
Ingress L2 SPAN ACL IPv4 0 0.00
Ingress L2 SPAN ACL IPv6 0 0.00
Ingress L2 SPAN ACL MAC 0 0.00
Ingress L2 SPAN ACL ALL 0 0.00
Ingress L2 SPAN ACL OTHER 0 0.00
Ingress RACL 0 512 0.00
Ingress RACL IPv4 0 0.00
Ingress RACL IPv6 0 0.00
Ingress RACL MAC 0 0.00
Ingress RACL ALL 2 0.39
Ingress RACL OTHER 0 0.00
Ingress L3/VLAN QOS 4 508 0.78
Ingress L3/VLAN QOS IPv4 0 0.00
Ingress L3/VLAN QOS IPv6 0 0.00
Ingress L3/VLAN QOS MAC 0 0.00
Ingress L3/VLAN QOS ALL 4 0.78
Ingress L3/VLAN QOS OTHER 0 0.00
Ingress L3/VLAN SPAN ACL 0 256 0.00
Ingress L3/VLAN SPAN ACL IPv4 0 0.00
Ingress L3/VLAN SPAN ACL IPv6 0 0.00
Ingress L3/VLAN SPAN ACL MAC 0 0.00
Ingress L3/VLAN SPAN ACL ALL 0 0.00
Ingress L3/VLAN SPAN ACL OTHER 0 0.00
Ingress SUP 396 76 83.90
Ingress SUP IPv4 129 27.33
Ingress SUP IPv6 176 37.29
Ingress SUP MAC 35 7.42
Ingress SUP ALL 10 2.12
Ingress SUP OTHER 46 9.75
SPAN 0 512 0.00
SPAN IPv4 0 0.00
SPAN IPv6 0 0.00
SPAN MAC 0 0.00
SPAN ALL 0 0.00
SPAN OTHER 0 0.00
Egress RACL 0 1792 0.00
Egress RACL IPv4 0 0.00
Egress RACL IPv6 0 0.00
Egress RACL MAC 0 0.00
Egress RACL ALL 2 0.11
Egress RACL OTHER 0 0.00
Egress SUP 128 128 50.00
Egress SUP IPv4 0 0.00
Egress SUP IPv6 0 0.00
Egress SUP MAC 0 0.00
Egress SUP ALL 0 0.00
Egress SUP OTHER 128 50.00
Feature BFD 2 14 12.50
Feature BFD IPv4 0 0.00
Feature BFD IPv6 0 0.00
Feature BFD MAC 0 0.00
Feature BFD ALL 2 12.50
Feature BFD OTHER 0 0.00
Feature DHCP SNOOP 0 16 0.00
Feature DHCP SNOOP IPv4 0 0.00
Feature DHCP SNOOP IPv6 0 0.00
Feature DHCP SNOOP MAC 0 0.00
Feature DHCP SNOOP ALL 0 0.00
Feature DHCP SNOOP OTHER 0 0.00
Feature ARP SNOOP 0 2 0.00
Feature ARP SNOOP IPv4 0 0.00
Feature ARP SNOOP IPv6 0 0.00
Feature ARP SNOOP MAC 0 0.00
Feature ARP SNOOP ALL 0 0.00
Feature ARP SNOOP OTHER 0 0.00
Feature VxLAN OAM 0 2 0.00
Feature VxLAN OAM IPv4 0 0.00
Feature VxLAN OAM IPv6 0 0.00
Feature VxLAN OAM MAC 0 0.00
Feature VxLAN OAM ALL 0 0.00
Feature VxLAN OAM OTHER 0 0.00
Feature DHCPv6 RELAY 0 4 0.00
Feature DHCPv6 RELAY IPv4 0 0.00
Feature DHCPv6 RELAY IPv6 0 0.00
Feature DHCPv6 RELAY MAC 0 0.00
Feature DHCPv6 RELAY ALL 0 0.00
Feature DHCPv6 RELAY OTHER 0 0.00
LOU 8 8 50.00
Both LOU Operands 8
Single LOU Operands 0
LOU L4 src port: 3
LOU L4 dst port: 5
LOU L3 packet len: 0
LOU IP tos: 0
LOU IP dscp: 0
LOU ip precedence: 0
LOU ip TTL: 0
TCP Flags 0 16 0.00
Protocol CAM 2 244 0.81
Mac Etype/Proto CAM 0 14 0.00
Non L4op labels, Tcam 0 13 17 43.33
Non L4op labels, Tcam 1 0 62 0.00
Non L4op labels, Tcam 2 0 4095 0.00
Non L4op labels, Tcam 3 1 1022 0.09
Non L4op labels, Tcam 4 0 0 0.00
Non L4op labels, Tcam 5 0 1023 0.00
Non L4op labels, Tcam 6 0 1023 0.00
Non L4op labels, Tcam 7 0 1024 0.00
Non L4op labels, Tcam 8 0 254 0.00
Non L4op labels, Tcam 9 0 1023 0.00
Non L4op labels, Tcam 10 0 1023 0.00
Non L4op labels, Tcam 11 0 1023 0.00
Non L4op labels, Tcam 12 0 1023 0.00
Non L4op labels, Tcam 13 0 1023 0.00
Non L4op labels, Tcam 14 0 1023 0.00
Non L4op labels, Tcam 15 0 1023 0.00
Non L4op labels, Tcam 16 0 1023 0.00
L4 op labels, Tcam 0 13 17 43.33
L4 op labels, Tcam 1 0 62 0.00
L4 op labels, Tcam 2 0 4095 0.00
L4 op labels, Tcam 3 1 1022 0.09
L4 op labels, Tcam 4 0 0 0.00
L4 op labels, Tcam 5 0 1023 0.00
L4 op labels, Tcam 6 0 1023 0.00
L4 op labels, Tcam 7 0 1024 0.00
L4 op labels, Tcam 8 0 254 0.00
L4 op labels, Tcam 9 0 1023 0.00
L4 op labels, Tcam 10 0 1023 0.00
L4 op labels, Tcam 11 0 1023 0.00
L4 op labels, Tcam 12 0 1023 0.00
L4 op labels, Tcam 13 0 1023 0.00
L4 op labels, Tcam 14 0 1023 0.00
L4 op labels, Tcam 15 0 1023 0.00
L4 op labels, Tcam 16 0 1023 0.00
Ingress Dest info table 0 512 0.00
Egress Dest info table 0 512 0.00
Note: Ingress SUP region includes the Redirect region
Ideally we would like a recommendation on services we can afford to lose without causing us future problems or losing span (which we need)
thanks again
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-07-2017 06:57 AM
Hi there,
Below is a suggestion based on the utilization I see for your device currently:
Ingress RACL is not being utilized in this particular box:
HQDC-ACC-SW-02A# sh hardware access-list tcam region
NAT ACL[nat] size = 0
Ingress PACL [ing-ifacl] size = 0
VACL [vacl] size = 0
Ingress RACL [ing-racl] size = 512 <<< Ingress RACL
Ingress RBACL [ing-rbacl] size = 0
Ingress L2 QOS [ing-l2-qos] size = 1536
Ingress L3/VLAN QOS [ing-l3-vlan-qos] size = 512
Ingress SUP [ing-sup] size = 512
Ingress L2 SPAN filter [ing-l2-span-filter] size = 256
Ingress L3 SPAN filter [ing-l3-span-filter] size = 256
Ingress FSTAT [ing-fstat] size = 0
span [span] size = 512
Egress RACL [egr-racl] size = 1792
Egress SUP [egr-sup] size = 256
Ingress Redirect [ing-redirect] size = 0
HQDC-ACC-SW-02A# show hardware access-list resource utilization
ACL Hardware Resource Utilization (Mod 1)
----------------------------------------------------------
Used Free Percent
Utilization
-------------------------------------------------------------------
Ingress L2 QOS 1408 128 91.67 <<< Very close to 100%
Ingress L2 QOS IPv4 1360 88.54
Ingress L2 QOS IPv6 32 2.08
Ingress L2 QOS MAC 16 1.04
Ingress L2 QOS ALL 0 0.00
Ingress L2 QOS OTHER 0 0.00
<snip>
Ingress RACL 0 512 0.00 <<< Ingress RACL not being used at all
Ingress RACL IPv4 0 0.00
Ingress RACL IPv6 0 0.00
Ingress RACL MAC 0 0.00
Ingress RACL ALL 2 0.39
Ingress RACL OTHER 0 0.00
<snip>
You could then lower the Ingress RACL TCAM size significantly to allow Ingress L2 QoS to take more of it.
Example:
Nexus9K(config)# hardware access-list tcam region ing-racl 0
Warning: Please save config and reload the system for the configuration to take effect
Nexus9K(config)# hardware access-list tcam region ing-l2-qos 2048
Warning: Please save config and reload the system for the configuration to take
Nexus9K# copy r s
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete.
Nexus9K# reload
This command will reboot the system. (y/n)? [n] y
End Result:
Nexus9K# show hardware access-list tcam region
NAT ACL[nat] size = 0
Ingress PACL [ing-ifacl] size = 0
VACL [vacl] size = 0
Ingress RACL [ing-racl] size = 0 << Ing RACL is now 0
Ingress RBACL [ing-rbacl] size = 0
Ingress L2 QOS [ing-l2-qos] size = 2048 <<< Larger TCAM size
Ingress L3/VLAN QOS [ing-l3-vlan-qos] size = 512
Ingress SUP [ing-sup] size = 512
Ingress L2 SPAN filter [ing-l2-span-filter] size = 256
Ingress L3 SPAN filter [ing-l3-span-filter] size = 256
Ingress FSTAT [ing-fstat] size = 0
span [span] size = 512
Egress RACL [egr-racl] size = 1792
Egress SUP [egr-sup] size = 256
Ingress Redirect [ing-redirect] size = 0
Ingress NBM [ing-nbm] size = 0
Essentially, I've increased the Ingress L2 QoS TCAM size by 512 which should allow for more QoS configuration on L2 ports. Do keep in mind of course you can still hit the limit in the future as more L2 ports are configured with input service policies; however, this re-carve should allow you to "squeeze" a few more.
Hope that helps.
- Andrea
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-07-2017 07:05 AM
That's great thanks for your help.
Someone else has suggested we only configure the marking on the uplinks to the distribution switches.
Would you agree that this is ok, will it still mark correctly?
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide
