Network transient interruption when removing an entry in the ACL under the Catalyst 3850 SVI interface

[Rps-Cheers]

Hello everyone,
I have a problem with the Catalyst 3850, about ACL.
On the 3850, I configured an extended ACL with more than 60 ACL entries, as follows:

interface Vlan110
ip address 10.110.10.1 255.255.255.0
ip access-group MGT-V110-ACL in
no ip redirects
no ip unreachables
no ip proxy-arp
ip pim sparse-mode

SW01#show inter vlan 110
Vlan110 is up, line protocol is up
  Hardware is Ethernet SVI, address is 2c86.d2c3.e84d (bia 2c86.d2c3.e84d)
  Internet address is 10.110.10.1/24
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 4/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not supported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output never, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/208/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 6178000 bits/sec, 1967 packets/sec
  5 minute output rate 17443000 bits/sec, 3078 packets/sec
     154138521294 packets input, 63430780136941 bytes, 0 no buffer
     Received 0 broadcasts (115009 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     333930193033 packets output, 393604448659988 bytes, 0 underruns
     0 output errors, 1 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out

ip access-list extended MGT-V110-ACL
permit icmp any any
permit udp any eq domain any
permit udp any any eq domain
permit ip any 224.0.0.0 0.0.0.255
permit udp any eq 5514 5515 any
remark SPLUNK-5
permit udp any any eq snmp
permit udp any eq snmp any
permit udp any any eq snmptrap
permit udp any eq snmptrap any
permit udp any any eq ntp
permit udp any eq ntp any
permit udp any any eq syslog
permit udp any eq syslog any
permit tcp any any established
permit tcp host 10.110.10.12 10.110.18.0 0.0.0.255 eq 22 443
permit tcp host 10.110.10.12 10.110.82.0 0.0.0.255 eq 22 telnet
permit tcp host 10.110.10.12 192.168.38.0 0.0.0.255 eq 22
permit tcp host 10.110.10.12 10.110.128.0 0.0.0.255 eq 22
remark IMCwangguan-2
permit ip host 10.110.10.82 10.110.18.0 0.0.0.255
permit ip host 10.110.10.82 10.110.82.0 0.0.0.255
permit ip host 10.110.10.82 192.168.38.0 0.0.0.255
permit ip host 10.110.10.82 10.110.128.0 0.0.0.255
remark SPLUNK-5
permit tcp host 10.110.10.8 10.110.0.0 0.0.0.255 eq 8089 9997
permit tcp host 10.110.10.8 10.110.1.0 0.0.0.255 eq 8089 9997
permit tcp host 10.110.10.8 10.110.3.0 0.0.0.255 eq 8089 9997
permit tcp host 10.110.10.9 10.110.0.0 0.0.0.255 eq 8089 9997
permit tcp host 10.110.10.9 10.110.1.0 0.0.0.255 eq 8089 9997
permit tcp host 10.110.10.9 10.110.3.0 0.0.0.255 eq 8089 9997
permit tcp host 10.110.10.10 10.110.0.0 0.0.0.255 eq 8089 9997
permit tcp host 10.110.10.10 10.110.1.0 0.0.0.255 eq 8089 9997
permit tcp host 10.110.10.10 10.110.3.0 0.0.0.255 eq 8089 9997
remark AllInOne-nbu
permit tcp host 10.110.10.13 any eq 22 3389 443 5899 telnet
permit tcp host 10.110.10.14 any eq 22 3389 443 5899 telnet
remark BaoLeiQianZhiJi-9
permit ip 10.110.10.116 0.0.0.1 host 10.110.8.30
permit ip 10.110.10.116 0.0.0.1 10.110.16.0 0.0.3.255
permit ip 10.110.10.116 0.0.0.1 10.110.8.0 0.0.1.255
permit ip 10.110.10.116 0.0.0.1 host 10.110.18.75
permit ip 10.110.10.116 0.0.0.1 host 10.110.1.129
remark NBUGuanLiJi-10
permit tcp host 10.110.10.35 object-group NBUGuanLiJi eq 1556 13724 13782 902 443
permit tcp host 10.110.10.99 object-group NBUGuanLiJi eq 1556 13724 13782 902 443
permit tcp host 10.110.10.122 object-group NBUGuanLiJi eq 1556 13724 13782 902 443
permit object-group Policy-11 object-group ZiDongHuaQieHuan host 10.110.10.159
remark QuShiShaDu-12
permit ip object-group QuShiShaDu object-group ShaDu
remark lenovoxclarityGuanLi-13
permit ip host 10.110.10.94 10.110.16.0 0.0.0.255
remark brocadebanGuanLi-15
permit ip host 10.110.10.97 10.110.17.0 0.0.0.255
remark ibm-tpcGuanLi-16
permit ip host 10.110.10.98 10.110.17.0 0.0.0.255
remark zabbix-vcenter-17
permit tcp host 10.110.10.109 host 10.110.8.30 eq 443
permit tcp host 10.110.10.137 host 10.110.8.30 eq 443
permit tcp host 10.110.10.138 host 10.110.8.30 eq 443
permit ip host 10.110.10.109 object-group agent
permit ip host 10.110.10.137 object-group agent
permit ip host 10.110.10.138 object-group agent
remark Policy-23
permit tcp host 10.110.10.103 host 10.110.0.135 eq www
permit tcp host 10.110.10.104 host 10.110.0.135 eq www
remark Policy-29
permit ip 10.110.10.86 0.0.0.1 any
remark Policy-30
permit ip any 10.110.10.86 0.0.0.1
remark YingFangTongBu-31
permit tcp host 10.110.10.163 object-group YingFangTongBu eq 26821
remark vplexZhongCai-32
permit ip host 10.110.10.136 10.110.17.0 0.0.0.255
remark exsiGuanLivcenter-38
permit ip host 10.110.10.115 host 10.110.8.30
remark waf-41
permit tcp host 10.110.10.58 10.110.18.164 0.0.0.1 eq 8080
remark ShaDu-42
permit tcp host 10.110.10.121 10.110.18.164 0.0.0.1 eq 8080
remark wsus-43
permit tcp host 10.110.10.90 10.110.18.164 0.0.0.1 eq 8080
remark VSM
permit ip host 10.110.10.123 10.110.8.0 0.0.0.255
permit ip host 10.110.10.123 10.110.9.0 0.0.0.255
remark waf-47
permit ip host 10.110.10.58 host 10.110.18.159
permit ip host 10.110.10.58 host 10.110.18.160
remark splunk-50
permit tcp host 10.110.10.8 host 10.110.3.8 eq 3306
remark nbu_XuJiHuiFu
permit tcp host 10.110.10.35 host 10.110.9.13 eq sunrpc 2049 7394
remark Policy-52
permit tcp host 10.110.10.8 host 10.110.0.41 eq smtp
remark itsm-mail
permit tcp 10.110.10.100 0.0.0.1 host 10.110.0.41 eq smtp
permit ip host 10.110.10.84 host 10.110.1.194
permit ip host 10.110.10.84 host 10.110.1.195
permit ip host 10.110.10.84 host 10.110.1.196
remark bst01App
permit ip host 10.110.10.36 host 10.110.1.194
permit ip host 10.110.10.36 host 10.110.1.195
permit ip host 10.110.10.36 host 10.110.1.196
permit ip host 10.110.10.84 host 10.110.1.90
permit ip host 10.110.10.84 host 10.110.1.91
permit ip host 10.110.10.84 host 10.110.1.92
permit tcp host 10.110.10.84 10.110.0.0 0.0.3.255 eq 1556 13720 13724 13782 13783 902 443
permit tcp host 10.110.10.84 10.110.8.0 0.0.1.255 eq 1556 13720 13724 13782 13783 902 443
remark AllInOne-nbu
permit tcp host 10.110.10.36 10.110.0.0 0.0.3.255 eq 1556 13720 13724 13782 13783 902 443
permit tcp host 10.110.10.36 10.110.8.0 0.0.1.255 eq 1556 13720 13724 13782 13783 902 443
permit ip host 10.110.10.139 host 10.110.18.166
permit ip host 10.110.10.84 host 10.110.1.190
permit ip host 10.110.10.84 host 10.110.1.192
permit ip host 10.110.10.84 host 10.110.1.193
permit ip host 10.110.10.84 host 10.110.1.191

If I remove an entry in the above ACL, the traffic of the SVI will be temporarily interrupted, including management traffic and normal traffic. What is especially strange is that I deleted some irrelevant items.
To this end, I also did a test, adding an arbitrary entry at the end of the ACL. For example, the following information:
===========================================
Ip access-list extended MGT-V110-ACL
     999 permit host 1.1.1.1 host 2.2.2.2
Then I delete it again.
Ip access-list extended MGT-V110-ACL
     no 999
===========================================
At this time, the traffic interruption occurred. So, I think this is very strange. Have you encountered such a problem? Can you help me solve this problem?
thank you very much.

[andresfr · Accepted solution]

Hello,

As you will find in the link below, the switch needs to program its hardware with the ACL information it processes:

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3850/software/release/3se/security/configuration_guide/b_sec_3se_3850_cg/b_sec_3se_3850_cg_chapter_01010.html#concept_2BC6AB2572F24E1AB9F04A062A16BAD2

Keeping that it mind, it's expected that the switch will need to re-program its hardware if you modify the access control entries in the ACL. So this interruption that you're reporting is an expected behavior considering the previous.

Please let me know if this information helps to address your concerns.

Regards,

[Rps-Cheers]

Hi andresfr ；

Thanks for your reply.
your meaning is that delete ACE will lead to the switch traffic interruption？As showed the doc：
“It programs its hardware with the ACL information it processes.”
If i add the ACE in a ACL,whether the same issue happen?In my case,adding ACE to a ACL will not mention.Is there any information to tell remove the ACE from a ACL will lead to the switch interruption?I have test it，the interruption state will stay seven seconds.

[Joseph W. Doherty]

As andresfr noted, if changing the ACL requires reprogramming the hardware (the ASICs), there usually going to be some traffic interruption. If changing the ACL doesn't require reprogramming the hardware, then there may not be a traffic interruption. The latter assumes the IOS realizes when the hardware needs to be reprogram vs. when it doesn't.

For example, if you add or remove a remark statement, I wouldn't expect the IOS to reprogram the hardware.

However suppose you replace your:

permit ip host 10.110.10.84 host 10.110.1.190
permit ip host 10.110.10.84 host 10.110.1.192
permit ip host 10.110.10.84 host 10.110.1.193
permit ip host 10.110.10.84 host 10.110.1.191

with:

permit ip host 10.110.10.84 10.110.1.190 0.0.0.3

If the IOS is clever, ii shouldn't need to reprogram the hardware, but it might not be that clever.

[Rps-Cheers]

Hello，
Before this,the Nexus switch can disable this feature,So,can we do it on Catalyst 3850?In other words,Is there any commands to make traffic don't be influenced when we remove ACE in ACL?

[Joseph W. Doherty]

Hmm, if you disable ACLs on a Nexus, you're sure it wouldn't impact any active flows subject to an ACL? (If it doesn't that might be because it's designed for DC usage.)

[Rps-Cheers]

Yes，there is a command to do it on nexus.

then，i want to know why the traffic interrupt when we remove ace，but cannot happening when we add ace？
thanks

[Joseph W. Doherty]

Again, you're sure adding a ACE won't interrupt traffic?

However, assuming there are some cases where adding or removing an ACE doesn't interrupt traffic, that would be likely due to a combination of the ACL itself and how the hardware (ASIC) is programmed.

Unless Cisco documents changing an ACL on a particular platform won't ever interrupt traffic, the safe assumption would be that it might.

[Rps-Cheers]

Yes,I am sure.
If we add a ACE,it won't interrupt traffic.And,the all acls at C3850 cannot up to 100 entries.We use command is too simply,that just add a ACE and remove it. for example:
ip access-list ex 100
200 permite host 1.1.1.1 host 2.2.2.2
then,we remove it:
ip access-list ex 100
no 200
now, the traffic will be interrupt about 7 seconds.So,i am very confused for that..that's why?

[Joseph W. Doherty]

Well again I would suspect it's just the way Cisco's IOS manages programming the hardware.

[Rps-Cheers]

at the beginning,i also suspect this due to Cisco's IOS manages program the hardware.and i have met a bug on C4500:
https://quickview.cloudapps.cisco.com/quickview/bug/CSCvi01706
i don't whether this bug also same to C3850....

[Joseph W. Doherty]

I would say a multi-second interruption is a long, long time. Usually such network interruptions might only interrupt a few frames/packets.

[Rps-Cheers]

So,i don't know the root reason....

[Joseph W. Doherty]

No, but if you have support contract is something you might bring to TAC.

[Rps-Cheers]

but we don't have support contract...
So we decided to upgrade the device