10-12-2014 08:29 PM - edited 03-07-2019 09:04 PM
Hi,
I'm working through a QoS configuration for the 6880-X-LE with 6800Ia FEXs. The QoS configuration, policymap, classes, ACLS etc have all been accepted fine.
I can apply a service policy to an interface but when I do I get the following errors come up:
*Oct 13 03:13:55.832: %EARL_CM-SW1-5-NOL4OP: Configured L4OPs exceeds the programmable limit for tcam= 0
*Oct 13 03:13:55.828: %EARL_CM-SW2_STBY-5-NOL4OP: Configured L4OPs exceeds the programmable limit for tcam= 0
*Oct 13 03:13:58.360: %QM-SW1-4-TCAM_ENTRY: Hardware TCAM entry programming failed for switch 1 slot 5 intf Gi141/1/0/1 dir IN: <CONFIG_UPDATE_REQ> TCAM Req Error: FAIL (4): Low TCAM Entries (1)
*Oct 13 03:13:58.360: %QM-SW1-4-TCAM_ENTRY: Hardware TCAM entry programming failed for switch 2 slot 5 intf Gi141/1/0/1 dir IN: <CONFIG_UPDATE_REQ> TCAM Req Error: FAIL (4): Low TCAM Entries (1)
*Oct 13 03:13:58.360: %QM-SW1-4-TCAM_ENTRY: Hardware TCAM entry programming failed for switch 1 slot 5 intf Gi141/1/0/1 dir IN: <CONFIG_UPDATE_REQ> TCAM Req Error: FAIL (4): Low TCAM Entries (1)
*Oct 13 03:13:58.360: %QM-SW1-4-TCAM_ENTRY: Hardware TCAM entry programming failed for switch 2 slot 5 intf Gi141/1/0/1 dir IN: <CONFIG_UPDATE_REQ> TCAM Req Error: FAIL (4): Low TCAM Entries (1)
*Oct 13 03:13:58.360: %QM-SW1-4-TCAM_ENTRY: Hardware TCAM entry programming failed for switch 1 slot 5 intf Gi141/1/0/1 dir IN: <CONFIG_UPDATE_REQ> TCAM Req Error: FAIL (4): Low TCAM Entries (1)
*Oct 13 03:13:58.360: %QM-SW1-4-TCAM_ENTRY: Hardware TCAM entry programming failed for switch 2 slot 5 intf Gi141/1/0/1 dir IN: <CONFIG_UPDATE_REQ> TCAM Req Error: FAIL (4): Low TCAM Entries (1)
*Oct 13 03:13:58.360: %QM-SW1-4-TCAM_ENTRY: Hardware TCAM entry programming failed for switch 1 slot 5 intf Gi141/1/0/1 dir IN: <CONFIG_UPDATE_REQ> TCAM Req Error: FAIL (4): Low TCAM Entries (1)
*Oct 13 03:13:58.360: %QM-SW1-4-TCAM_ENTRY: Hardware TCAM entry programming failed for switch 2 slot 5 intf Gi141/1/0/1 dir IN: <CONFIG_UPDATE_REQ> TCAM Req Error: FAIL (4): Low TCAM Entries (1)
*Oct 13 03:13:58.360: %QM-SW1-4-TCAM_ENTRY: Hardware TCAM entry programming failed for switch 1 slot 5 intf Gi141/1/0/1 dir IN: <CONFIG_UPDATE_REQ> TCAM Req Error: FAIL (4): Low TCAM Entries (1)
*Oct 13 03:13:58.360: %QM-SW1-4-TCAM_ENTRY: Hardware TCAM entry programming failed for switch 2 slot 5 intf Gi141/1/0/1 dir IN: <CONFIG_UPDATE_REQ> TCAM Req Error: FAIL (4): Low TCAM Entries (1)
I've checked the QoS policer quota's and they look OK. Is there something else I should be looking at??
NewLevel4Switch#sh platform hardware capacity qos
QoS Policer Resources
Aggregate policers: Sw/Mod Total Used %Used
1/5 16384 16 1%
2/5 16384 16 1%
Microflow policer configurations: Sw/Mod Total Used %Used
1/5 128 1 1%
2/5 128 1 1%
Netflow policer configurations: Sw/Mod Total Used %Used
1/5 384 0 0%
2/5 384 0 0%
Aggregate policer configurations: Sw/Mod Total Used %Used
1/5 1024 8 1%
2/5 1024 8 1%
Distributed policers: Total Used %Used
4096 1 1%
QoS Tcam Entries: Sw/Mod Total Used %Used
1/5 16384 1171 7%
2/5 16384 1171 7%
Thanks,
David.
Solved! Go to Solution.
10-13-2014 11:13 PM
Hi David,
I've confirmed the issue you see on your 6880s is because of the 9 L4Op limit. I put your config in the lab and see the same thing:
6800-VSS(config)#int ten1/5/1
6800-VSS(config-if)#service-policy input TAG-INBOUND-MARKING-AND-POLICING
6800-VSS(config-if)#end
6800-VSS#
*Oct 14 05:34:20.419: %SYS-SW2-5-CONFIG_I: Configured from console by console
*Oct 14 05:34:19.567: %EARL_CM-SW1_STBY-5-NOL4OP: Configured L4OPs exceeds the programmable limit for tcam= 0
*Oct 14 05:34:22.115: %QM-SW2-4-TCAM_ENTRY: Hardware TCAM entry programming failed for switch 1 slot 5 intf Te1/5/1 dir IN: <CONFIG_UPDATE_REQ> TCAM Req Error: FAIL (4): Low TCAM Entries (1)
*Oct 14 05:34:22.115: %QM-SW2-4-TCAM_ENTRY: Hardware TCAM entry programming failed for switch 1 slot 5 intf Te1/5/1 dir IN: <CONFIG_UPDATE_REQ> TCAM Req Error: FAIL (4): Low TCAM Entries (1)
*Oct 14 05:34:22.115: %QM-SW2-4-TCAM_ENTRY: Hardware TCAM entry programming failed for switch 1 slot 5 intf Te1/5/1 dir IN: <CONFIG_UPDATE_REQ> TCAM Req Error: FAIL (4): Low TCAM Entries (1)
*Oct 14 05:34:22.115: %QM-SW2-4-TCAM_ENTRY: Hardware TCAM entry programming failed for switch 1 slot 5 intf Te1/5/1 dir IN: <CONFIG_UPDATE_REQ> TCAM Req Error: FAIL (4): Low TCAM Entries (1)
*Oct 14 05:34:22.115: %QM-SW2-4-TCAM_ENTRY: Hardware TCAM entry programming failed for switch 1 slot 5 intf Te1/5/1 dir IN: <CONFIG_UPDATE_REQ> TCAM Req Error: FAIL (4): Low TCAM Entries (1)
*Oct 14 05:34:22.115: %FMCORE-SW2-6-RACL_ENABLED: Interface TenGigabitEthernet1/5/1 routed traffic is hardware switched in ingress direction
As an example, I went ahead and used only a subset of your policy-map (matched around 6 class-maps, each class-map matching one DSCP value).
6800-VSS#$e acl entry interface ten1/5/1 qos in ip switch 1 module 5
mls_if_index:8100000 dir:0 feature:1 proto:0
pass#0 features
fno:0
tcam:A, bank:0, prot:0 Aces
0x0000E010005D100B ip any any dscp eq 46
0x000100100131100B ip any any dscp eq 24
0x000120100245100B ip any any dscp eq 34
0x0000E0100349100B ip any any dscp eq 36
0x00014010044D100B ip any any dscp eq 38
0x000160100529100B ip any any dscp eq 20
0x000000000080D00B ip any any (3 matches)
I can see the entries installed in TCAM correctly. If you look at the capmap table now (the capmap table is what references the register index where your L4Ops are stored), you'd see 6 entries here:
6800-VSS#show platform software acl capmap tcam A label 2 switch 1 module 5
Shadow Capmap Table Entry For TCAM A
-----------------------------------------------------------------------
Output in a RST/INV/CNT format: RST - result value; INV - inverted;
CNT - aggregated reference account;
CBF - number of free cap bits (one per entry);
Free items are not shown
-----------------------------------------------------------------------
Index CBF [9] [8] [7] [6] [5] [4] [3] [2] [1] [0]
----- ----- ---------------- ---------------- ---------------- ---------------- ---------------- ---------------- ---------------- ---------------- ---------------- ----------------
2 3 212/0/1 Free Free Free 7/1/1 6/1/1 5/1/1 4/1/1 3/1/1 2/1/1
Ignore the 212/0/1 in the beginning - that is reserved and is used to specify direction of the installed feature.
Here you can see the limit as well - after 3 more entries with non-expandable L4Ops, you're out of space.
Please modify your policy-map in such a way that you're not going beyond 9 non-expandable L4Ops.
I'll work on finding out why the sup720s and the 32s behaved differently. Please give me some time.
Regards,
Aninda
10-12-2014 11:44 PM
Hi David,
Can you get me the output of 'show platform hardware capacity acl' and 'show platform hardware capacity qos' please? Additionally, do you mind giving me your QoS configuration? Shoot that over to my email.
Regards,
Aninda
10-13-2014 12:20 AM
Hi Aninda,
Details as requested:
#show platform hardware capacity acl
Classification Mgr Tcam Resources
Key: Ttlent - Total TCAM entries, QoSent - QoS TCAM entries, LOU - LOUs,
RBLent - RBACL TCAM entries, Lbl - Labels, TCP - TCP Flags,
Dstbl - Destinfo Table, Ethcam - Ethertype Cam Table,
ACTtbl - Accounting Table, V6ext - V6 Extn Hdr Table
Sw/Mod Ttlent QoSent RBLent Lbl LOU TCP Dstbl Ethcam ACTtbl V6ext
1/5 2% 4% 0% 1% 1% 0% 2% 0% 0% 0%
2/5 2% 4% 0% 1% 1% 0% 2% 0% 0% 0%
#show platform hardware capacity qos
QoS Policer Resources
Aggregate policers: Sw/Mod Total Used %Used
1/5 16384 16 1%
2/5 16384 16 1%
Microflow policer configurations: Sw/Mod Total Used %Used
1/5 128 1 1%
2/5 128 1 1%
Netflow policer configurations: Sw/Mod Total Used %Used
1/5 384 0 0%
2/5 384 0 0%
Aggregate policer configurations: Sw/Mod Total Used %Used
1/5 1024 8 1%
2/5 1024 8 1%
Distributed policers: Total Used %Used
4096 1 1%
QoS Tcam Entries: Sw/Mod Total Used %Used
1/5 32768 1192 3%
2/5 32768 1192 3%
Qos config is as follows:
auto qos default
ip access-list extended SIGNALING-ACL
remark Used for voice/video signaling
permit tcp any any eq 5060
permit udp any any eq 5060
permit udp any any eq 2427
permit udp any any eq 2727
permit tcp any any eq 2748
permit tcp any any eq 1720
permit tcp any any range 2000 2002
permit tcp any any range 11000 11999
class-map match-any TAG-VOIP
match ip dscp ef
class-map match-any TAG-SIGNALING-ACL
match access-group name SIGNALING-ACL
class-map match-any TAG-SIGNALING-CS3
match ip dscp cs3
class-map match-any TAG-VIDEO
match ip dscp af41
class-map match-any TAG-LYNC-VOICE
match ip dscp af42
class-map match-any TAG-LYNC-VIDEO
match ip dscp af43
class-map match-any TAG-LYNC-App-Sharing
match ip dscp af22
class-map match-any TAG-STREAM
match access-group name STREAM-ACL
class-map match-any TAG-REATIME-MULTIMEDIA-CS4
match ip dscp cs4
class-map match-any TAG-REATIME-MULTIMEDIA-CS5
match ip dscp cs5
class-map match-any TAG-NETWORK-CONTROL-CS6
match ip dscp cs6
class-map match-any TAG-NETWORK-CONTROL-CS7
match ip dscp cs7
class-map match-any TAG-NETWORK-MGT
match access-group name NETWORK-ACL
class-map match-any TAG-BUSINESS-CRITICAL-AF21
match ip dscp af21
class-map match-any TAG-BUSINESS-CRITICAL-AF23
match ip dscp af23
class-map match-any TAG-PROCESS
match access-group name PROCESS-ACL
class-map match-any TAG-BULK
match access-group name BULK-ACL
class-map match-any TAG-SCAVENGER
match access-group name SCAVENGER-ACL
!
ip access-list extended STREAM-ACL
remark Define any webcast traffic flows here
ip access-list extended NETWORK-ACL
remark Used to identify CRITICAL network management traffic
permit tcp any any eq 23
permit tcp any any eq 22
ip access-list extended PROCESS-ACL
remark Used to identify Process PCN traffic
ip access-list extended BULK-ACL
remark Define any bulk traffic flows here (Backups/Misc web surfing etc)
ip access-list extended SCAVENGER-ACL
remark Define any Scavenger/junk class traffic here
table-map policed-dscp-markdown
map from 8 to 0
map from 10 to 0
map from 16 to 0
map from 18 to 0
map from 20 to 0
default copy
table-map dscp2dscp
default copy
!
policy-map TAG-INBOUND-MARKING-AND-POLICING
class TAG-VOIP
police cir 128000 bc 8000
conform-action set-dscp-transmit ef
exceed-action drop
class TAG-SIGNALING-ACL
police cir 32000 bc 8000
conform-action set-dscp-transmit cs3
exceed-action drop
class TAG-SIGNALING-CS3
police cir 32000 bc 8000
conform-action set-dscp-transmit cs3
exceed-action drop
class TAG-VIDEO
police cir 7000000 bc 218750
conform-action set-dscp-transmit af41
exceed-action drop
class TAG-LYNC-VOICE
police cir 128000 bc 8000
conform-action set-dscp-transmit af42
exceed-action drop
class TAG-LYNC-VIDEO
police cir 5000000 bc 156250
conform-action set-dscp-transmit af43
exceed-action drop
class TAG-LYNC-App-Sharing
police cir 1000000 bc 31250
conform-action set-dscp-transmit af22
exceed-action drop
class TAG-STREAM
set ip dscp cs4
class TAG-REATIME-MULTIMEDIA-CS4
set ip dscp cs4
class TAG-REATIME-MULTIMEDIA-CS5
set ip dscp cs5
class TAG-NETWORK-CONTROL-CS6
set ip dscp cs6
class TAG-NETWORK-CONTROL-CS7
set ip dscp cs7
class TAG-NETWORK-MGT
police cir 1000000 bc 31250
conform-action set-dscp-transmit cs2
exceed-action policed-dscp-transmit
class TAG-BUSINESS-CRITICAL-AF21
set ip dscp af21
class TAG-BUSINESS-CRITICAL-AF23
set ip dscp af23
class TAG-PROCESS
set ip dscp af31
class TAG-BULK
set ip dscp af11
class TAG-SCAVENGER
set ip dscp cs1
class class-default
set dscp default
!
interface gigabitEthernet xx/xx/xx - xx
service-policy input TAG-INBOUND-MARKING-AND-POLICING
Thanks,
David
10-13-2014 02:25 AM
Thanks David. Give me a few hours - I'll get back to you on this.
Regards,
Aninda
10-13-2014 03:31 AM
Hi David,
Can you go through DDTS CSCuc81745 please? The behavior you see is described there, I believe.
Let me know if the defect does not make sense and I can explain further.
Regards,
Aninda
10-13-2014 05:20 PM
Hi Aninda,
Thanks, makes perfect sense and concurs with the TAC case we opened yesterday.
It says its fixed in a 15.1-2SY1, we're on 15.1-2SY3. Is there an alternative or better way we should be applying the configuration?
Thanks,
David.
10-13-2014 05:30 PM
Hi David,
What is the TAC SR number?
The DDTS I referred to is not a *fix*. It makes monitoring and being alerted to such a situation more robust. Before, it was just too convoluted to figure this out (the way it was coded).
The problem stems from the fact that you cannot have more than 9 non-expandable L4Ops (layer 4 operations) under an interface (more specifically, against a TCAM label, within the capmap table). As an example (taking your class-maps as an example), matching on a DSCP value is a non-expandable L4Op. You've clearly gone beyond the limit.
I have a 6880-X in my lab up and running; I can put your configuration in there later in the day and verify what I am saying.
Regards,
Aninda
10-13-2014 06:11 PM
Hi Aninda,
OK, we're just confused because we have applied a similar configuration to a 6500 (less the policing) which applies without issue. I've pasted it below, there are at least 12 match DSCP entries.
Does the 6500 range have a difference limit?
class-map match-all TAG-VOIP
match ip dscp ef
class-map match-all TAG-SIGNALING-ACL
match access-group name SIGNALING-ACL
class-map match-all TAG-SIGNALING-CS3
match ip dscp cs3
class-map match-any TAG-VIDEO
match ip dscp af41
class-map match-any TAG-LYNC-VOICE
match ip dscp af42
class-map match-any TAG-LYNC-VIDEO
match ip dscp af43
class-map match-any TAG-LYNC-App-Sharing
match ip dscp af22
class-map match-any TAG-STREAM
match access-group name STREAM-ACL
class-map match-any TAG-REATIME-MULTIMEDIA-CS4
match ip dscp cs4
class-map match-any TAG-REATIME-MULTIMEDIA-CS5
match ip dscp cs5
class-map match-any TAG-NETWORK-CONTROL-CS6
match ip dscp cs6
class-map match-any TAG-NETWORK-CONTROL-CS7
match ip dscp cs7
class-map match-any TAG-NETWORK-MGT
match access-group name NETWORK-ACL
class-map match-any TAG-BUSINESS-CRITICAL-AF21
match ip dscp af21
class-map match-any TAG-BUSINESS-CRITICAL-AF23
match ip dscp af23
class-map match-any TAG-PROCESS
match access-group name PROCESS-ACL
class-map match-any TAG-BULK
match access-group name BULK-ACL
class-map match-any TAG-SCAVENGER
match access-group name SCAVENGER-ACL
policy-map TAG-INBOUND-MARKING-AND-POLICING
class TAG-VOIP
set ip dscp ef
class TAG-SIGNALING-ACL
set ip dscp cs3
class TAG-SIGNALING-CS3
set ip dscp cs3
class TAG-VIDEO
set ip dscp af41
class TAG-LYNC-VOICE
set ip dscp af42
class TAG-LYNC-VIDEO
set ip dscp af43
class TAG-LYNC-App-Sharing
set ip dscp af22
class TAG-STREAM
set ip dscp cs4
class TAG-REATIME-MULTIMEDIA-CS4
set ip dscp cs4
class TAG-REATIME-MULTIMEDIA-CS5
set ip dscp cs5
class TAG-NETWORK-CONTROL-CS6
set ip dscp cs6
class TAG-NETWORK-CONTROL-CS7
set ip dscp cs7
class TAG-NETWORK-MGT
set ip dscp cs2
class TAG-BUSINESS-CRITICAL-AF21
set ip dscp af21
class TAG-BUSINESS-CRITICAL-AF23
set ip dscp af23
class TAG-PROCESS
set ip dscp af31
class TAG-BULK
set ip dscp af11
class TAG-SCAVENGER
set ip dscp cs1
class class-default
set dscp default
Thanks,
David
10-13-2014 06:17 PM
Hi David,
What supervisor are you using on the 6500 and what code are you on?
Regards,
Aninda
10-13-2014 06:20 PM
Hi Aninda,
We're using a mixture of Sup720s and Sup32s in our 6500 access switches.
An example IOS version is 122-33.SXJ4.
Thanks,
David.
10-13-2014 06:47 PM
10-13-2014 11:13 PM
Hi David,
I've confirmed the issue you see on your 6880s is because of the 9 L4Op limit. I put your config in the lab and see the same thing:
6800-VSS(config)#int ten1/5/1
6800-VSS(config-if)#service-policy input TAG-INBOUND-MARKING-AND-POLICING
6800-VSS(config-if)#end
6800-VSS#
*Oct 14 05:34:20.419: %SYS-SW2-5-CONFIG_I: Configured from console by console
*Oct 14 05:34:19.567: %EARL_CM-SW1_STBY-5-NOL4OP: Configured L4OPs exceeds the programmable limit for tcam= 0
*Oct 14 05:34:22.115: %QM-SW2-4-TCAM_ENTRY: Hardware TCAM entry programming failed for switch 1 slot 5 intf Te1/5/1 dir IN: <CONFIG_UPDATE_REQ> TCAM Req Error: FAIL (4): Low TCAM Entries (1)
*Oct 14 05:34:22.115: %QM-SW2-4-TCAM_ENTRY: Hardware TCAM entry programming failed for switch 1 slot 5 intf Te1/5/1 dir IN: <CONFIG_UPDATE_REQ> TCAM Req Error: FAIL (4): Low TCAM Entries (1)
*Oct 14 05:34:22.115: %QM-SW2-4-TCAM_ENTRY: Hardware TCAM entry programming failed for switch 1 slot 5 intf Te1/5/1 dir IN: <CONFIG_UPDATE_REQ> TCAM Req Error: FAIL (4): Low TCAM Entries (1)
*Oct 14 05:34:22.115: %QM-SW2-4-TCAM_ENTRY: Hardware TCAM entry programming failed for switch 1 slot 5 intf Te1/5/1 dir IN: <CONFIG_UPDATE_REQ> TCAM Req Error: FAIL (4): Low TCAM Entries (1)
*Oct 14 05:34:22.115: %QM-SW2-4-TCAM_ENTRY: Hardware TCAM entry programming failed for switch 1 slot 5 intf Te1/5/1 dir IN: <CONFIG_UPDATE_REQ> TCAM Req Error: FAIL (4): Low TCAM Entries (1)
*Oct 14 05:34:22.115: %FMCORE-SW2-6-RACL_ENABLED: Interface TenGigabitEthernet1/5/1 routed traffic is hardware switched in ingress direction
As an example, I went ahead and used only a subset of your policy-map (matched around 6 class-maps, each class-map matching one DSCP value).
6800-VSS#$e acl entry interface ten1/5/1 qos in ip switch 1 module 5
mls_if_index:8100000 dir:0 feature:1 proto:0
pass#0 features
fno:0
tcam:A, bank:0, prot:0 Aces
0x0000E010005D100B ip any any dscp eq 46
0x000100100131100B ip any any dscp eq 24
0x000120100245100B ip any any dscp eq 34
0x0000E0100349100B ip any any dscp eq 36
0x00014010044D100B ip any any dscp eq 38
0x000160100529100B ip any any dscp eq 20
0x000000000080D00B ip any any (3 matches)
I can see the entries installed in TCAM correctly. If you look at the capmap table now (the capmap table is what references the register index where your L4Ops are stored), you'd see 6 entries here:
6800-VSS#show platform software acl capmap tcam A label 2 switch 1 module 5
Shadow Capmap Table Entry For TCAM A
-----------------------------------------------------------------------
Output in a RST/INV/CNT format: RST - result value; INV - inverted;
CNT - aggregated reference account;
CBF - number of free cap bits (one per entry);
Free items are not shown
-----------------------------------------------------------------------
Index CBF [9] [8] [7] [6] [5] [4] [3] [2] [1] [0]
----- ----- ---------------- ---------------- ---------------- ---------------- ---------------- ---------------- ---------------- ---------------- ---------------- ----------------
2 3 212/0/1 Free Free Free 7/1/1 6/1/1 5/1/1 4/1/1 3/1/1 2/1/1
Ignore the 212/0/1 in the beginning - that is reserved and is used to specify direction of the installed feature.
Here you can see the limit as well - after 3 more entries with non-expandable L4Ops, you're out of space.
Please modify your policy-map in such a way that you're not going beyond 9 non-expandable L4Ops.
I'll work on finding out why the sup720s and the 32s behaved differently. Please give me some time.
Regards,
Aninda
10-14-2014 05:24 PM
Hi Aninda,
Thanks heaps. You are a legend.
Appreciate the ongoing support.
Cheers,
David.
10-17-2014 03:32 AM
Hi David,
I'm sorry, I've been caught up with some critical internal work. I'll look into this over the weekend and hopefully get back to you by Monday.
Regards,
Aninda
10-08-2015 08:32 AM
so something like this, is this 4 operations or 3? or is just match dscp an operation. Seems like a crazy limitation
Extended IP access list ACL_MISSION_CRITICAL_DATA
10 permit ip any any dscp af31
20 permit ip any any dscp af32
30 permit ip any any dscp af33
40 permit tcp any any eq 1494
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide