Catalyst 3650/3850 ：show mgmt-infra trace messages acl-events switch 1 ---Found an exception

[Rps-Cheers]

HI， I have a question that is very doubtful. I configured the ACL on the 3850 switch, but I have a different situation when I add an ACL entry and remove the ACL entry. I found it through "show mgmt-infra trace messages acl-events switch 1". Scenario 1: I add an entry to the ACL called "For_Mgmt". Through the above command you can see the following information: [07/27/18 12:15:55.345 east 391 6430] 5863: ACL_CHANGED: ACL change IPv4 [For_Mgmt] [07/27/18 12:15:55.637 east 392 6430] 3097: ACL_SEND_HARDWARE_WRITE_EVENT: START Input IPv4 L3 label_id 9 asic8 num_les 1 old_unload 0x0, cur_unloaded 0x0, trid 198 num_vmrs 300 [07/27/18 12:15:55.682 east 393 6430] 4236: ACL_HARDWARE_WRITE_RESULT_HANDLER_GUTS: Input IPv4 L3 label_id 9 hwlabel 9 asic8 status 0x0 old_unloaded 0x0 cur_unloaded 0x0 trid 198 [07/27/18 12:16:01.683 east 394 6430] 6015: ACL_RELOAD_LABELS: Reloading labels [07/27/18 12:16:01.683 east 395 6430] 6040: ACL_RELOAD_LABELS: DONE Reloading labels This took about 6 seconds. Scenario 2: When I remove an entry from the "For_Mgmt" ACL, use the same command to view the ACL events: [07/27/18 16:09:53.906 east 396 6430] 5863: ACL_CHANGED: ACL change IPv4 [For_Mgmt] [07/27/18 16:09:54.207 east 397 6430] 3097: ACL_SEND_HARDWARE_WRITE_EVENT: START Input IPv4 L3 label_id 9 asic8 num_les 1 old_unload 0x0, cur_unloaded 0x0, trid 199 num_vmrs 300 [07/27/18 16:09:54.224 east 398 6430] 5863: ACL_CHANGED: ACL change IPv4 [For_Mgmt] [07/27/18 16:09:54.531 east 399 6430] 3097: ACL_SEND_HARDWARE_WRITE_EVENT: START Input IPv4 L3 label_id 9 asic8 num_les 1 old_unload 0x0, cur_unloaded 0x0, trid 200 num_vmrs 301 [07/27/18 16:09:54.532 east 39a 6430] 4236: ACL_HARDWARE_WRITE_RESULT_HANDLER_GUTS: Input IPv4 L3 label_id 9 hwlabel 9 asic8 status 0x0 old_unloaded 0x0 cur_unloaded 0x0 trid 199 [07/27/18 16:09:54.583 east 39b 6430] 4236: ACL_HARDWARE_WRITE_RESULT_HANDLER_GUTS: Input IPv4 L3 label_id 9 hwlabel 9 asic8 status 0x0 old_unloaded 0x0 cur_unloaded 0x0 trid 200 [07/27/18 16:10:06.268 east 39c 6430] 6015: ACL_RELOAD_LABELS: Reloading labels [07/27/18 16:10:06.268 east 39d 6430] 6040: ACL_RELOAD_LABELS: DONE Reloading labels This process took about 12 seconds. Oddly enough, I found on other ACLs that it takes almost 6 seconds to add and remove ACL entries. It is this ACL that is unusual. In addition, this ACL called "ACL_Mgmt" I saw through the "show platform acl info switch 1" entry to the hardware has the following information: ====================================================== IPv4 ACL: ACL_Mgmt Aclinfo: 0x5bd89790 ASIC255 Input L3 labels: 9 Ipv4 Acl: CL_Mgmt Version 23351 Use Count 0 Clients 0x0 ................... Total Entries: 436 ====================================================== There are 436 entries here, is it too much? So my question is why is this ACL different from other ACLs when removing ACL entries? Is the Catalyst 3850 restricted for each ACL entry? What happens if it is exceeded? I hope you can help me analyze it. Thank you very much~

[andresfr]

Hello Moments,

There are several things to keep in mind regarding ACLs considering that they are processed and stored in hardware and also that there are different types of ACL. The behavior and restrictions of ACLs would depend on the platform capacity that could also be delimited by SDM templates on some switches.

Please check how SDM templates can affect/limit the number of ACEs depending on their type:

Understanding and Configuring Switching Database Manager on Catalyst 3750 Series Switches
https://www.cisco.com/c/en/us/support/docs/switches/catalyst-3750-series-switches/44921-swdatabase-3750ss-44921.html


SDM Templates  - Catalyst 3850
https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3850/software/release/16-6/configuration_guide/sys_mgmt/b_166_sys_mgmt_3850_cg/b_166_sys_mgmt_3850_cg_chapter_0110.html#con_1059809

Then, you will have to consider the limitations of having a mix of various types of ACLs and the associated use of TCAM resources:

Troubleshoot Security ACL TCAM Exhaustion on Catalyst 3850 Switches
https://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/118957-troubleshoot-sec-acl-tcam-cat3850.html

The previous would mean that there's no a simple answer to your question. I would not expect two ACLs to behave in the same way considering the might differ in the number of ACEs and also in the order, syntax, and way in which those entries are configured. Besides that, when considering the limitations imposed by hardware, you cannot think of the restrictions that will apply to a particular ACL without considering other existing ACLs.

So, if your ACLs are doing what they are supposed to do but you've some concerns regarding performance and ACLs with a lot of ACEs, then I would suggest checking if you can optimize those ACLs by checking things like:

• The order of the entries  (remember, once the device finds a match it will not continue to validate the rest of the entries in the ACL, so it's important to check order of specific ACEs in respect of more general ACEs).
• If there are any unused ACEs that can be removed from configuration.
• If there are similar ACEs that could be merged in a single one.
• If there are any duplicate ACEs. 

I hope you find this information useful.

Regards,

[Rps-Cheers]

Hi andresfr：

Thanks for your response！
the sdm templates is default，and support 3072 security ace，all acls are RACL，and used values is 1191.there is tcam utilization：
#show platform tcam utilization asic all
CAM Utilization for ASIC# 0
Table Max Values Used Values
--------------------------------------------------------------------------------
Unicast MAC addresses 32768/512 722/23
Directly or indirectly connected routes 16384/7168 857/122
L2 Multicast groups 8192/512 14/7
L3 Multicast groups 8192/512 4/99
QoS Access Control Entries 2816 52
Security Access Control Entries 3072 1191
Netflow ACEs 768 15
Input Microflow policer ACEs 256 7
Output Microflow policer ACEs 256 7
Flow SPAN ACEs 512 13
Control Plane Entries 512 270
Policy Based Routing ACEs 1024 9
Tunnels 256 12
Input Security Associations 256 4
SPD 256 2
Output Security Associations and Policies 256 9
SGT_DGT 4096/512 0/0
CLIENT_LE 4096/256 0/0
INPUT_GROUP_LE 6144 0
OUTPUT_GROUP_LE 6144 0

Is there any abnormal？I think the tcam utilization is not up to limitation.But，only this acl with 436 entries will cost twice time when remove a ace from it，and it will make traffic interruption about a few seconds.So，i want to know whether there will be a limitation about one acl.for example，up to 400 entries，the abnormal scene will be happened.

[andresfr · Accepted solution]

Hello Moments,

Did you take some time to review the provided links? I hope you can have a chance to go through those links.

The Catalyst 3850 Data Sheet suggests that 3,000 security ACL entries are supported. However, these rules define how these 3,000 ACEs can be configured:

VACL/vlmaps support a total of 1.5K entries as they can use only one of the two TAQs.
MAC VACL/vlmap needs three VMR/ACEs. This means 460 ACEs must be supported in each direction.
IPv4 VACL/vlmap needs two VMR/ACEs. This means 690 ACEs must be supported in each direction.
IPv4 PACL, RACL, and GACL need one VMR/ACE. This means 1,380 ACEs must be supported in each direction. <-------
MAC PACL, RACL, and GACL need two VMR/ACEs. This means 690 ACEs must be supported in each direction.
IPv6 PACL, RACL, and GACL need two VMR/ACEs. This means 690 ACEs must be supported in each direction.

Troubleshooting Security ACL TCAM on Catalyst 3850 Switches

• Check security TCAM utilization:

Note: Even though the installed security ACEs are less than 3,072, one of the limits previously mentioned might have been reached. For example, if a customer has most of the RACLs applied in the input direction, they can use up 1,380 entries available for the inbound RACL. However, TCAM exhaustion logs can show up before all 3,072 entries are used.

3850#show platform tcam utilization asic all
CAM Utilization for ASIC# 0
 Table                                              Max Values        Used Values
 --------------------------------------------------------------------------------
 Unicast MAC addresses                              32768/512          85/22  
 Directly or indirectly connected routes            32768/7680        125/127 
 IGMP and Multicast groups                          8192/512           0/16  
 QoS Access Control Entries                         3072                68
 Security Access Control Entries                    3072              1648 <--------
 Netflow ACEs                                       1024                15
 Input Microflow policer ACEs                        256                 7
 Output Microflow policer ACEs                       256                 7
 Flow SPAN ACEs                                      256                13
 Control Plane Entries                               512               195
 Policy Based Routing ACEs                          1024                 9
 Tunnels                                             256                12
 Input Security Associations                         256                 4
 Output Security Associations and Policies           256                 9
 SGT_DGT                                            4096/512           0/0   
 CLIENT_LE                                          4096/64            0/0   
 INPUT_GROUP_LE                                     6144                 0
 OUTPUT_GROUP_LE                                    6144                 0

http://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/I118957-troubleshoot-sec-acl-tcam-cat3850.html

I didn't find any information regarding the time that the switch will take to program the hardware when adding/removing ACEs from an ACL with a specific number of ACEs. However, logic dictates that it will take longer if the ACL have a big number of ACEs.

I hope that the information above results helpful to address your concerns.

[Rps-Cheers]

HI andresfr

Thank you for your patience, I have read the following documents many times before.

Http://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/I118957-troubleshoot-sec-acl-tcam-cat3850.html

But can't find an accurate answer. In this regard, the number of ACLs is definitely limited, we only found the total number, but there is no specific document to say how many entries in an ACL are the best. So, I summarize this question as follows:

1, according to the 3850 datasheet, support 3000 ACL entries；
2, TCAM utilization depends on all ACLs；
3. The addition and removal of ACLs will be rewritten into the hardware；
4. For longer ACLs, modifying the ACL may cause an exception. The traffic may be interrupted (but why is there a difference between adding and removing entries?)；
5. When modifying an ACL, it is best to remove the ACL from the applied interface first, and then modify the ACL before applying it to the corresponding interface.

Thank you again for your help!