Strange ACL problem.

[hemmerling]

I'm trying to update an outbound ACL on a VLAN interface on a 6807 and I'm needing to remove the ACL from the interface, remove the ACL completely, create a new named iteration (let's say OUT_ACL_12), the new ACL contains similar lines from the older ACL, and these lines, despite being numbered differently and in a newly named ACL seem like they are stuck somewhere in the layer 3 switches routing instructions.

In that when I apply the new ACL, those lines that match the previous ACL no longer increase in hit count.

They are working, just in the background somehow. Adding "log-input" to the line won't create log entries, but removing it and modifying the line so it's different (expanding a subnet mask) will allow the line to start showing matches again, but if I repeat the process will have the same effect, the older line and the newer slightly modified line will work, but the match counts won't go up.

What am I seeing here? Is there a way to stop this from happening? The switch is in production and can't just be rebooted, and even if I did, it would happen again once I needed to update the ACLs again.

I'm having a hard time explaining what I'm seeing. It's almost like source and destination traffic that matches an ACL line gets cached somewhere in memory and as long as the ACL it came from remains in the system the match count and logging works, but when I detach and remove the ACL and create a similar line in a new ACL and attach it that the cached version remains internally and is not updating the match count on the new ACL, but is still working (albeit without showing that it is and logging if told to).

Any ideas?

[David Ruess]

Can you provide both ACLs to us for comparison?

Are both ACLS (new and old) the same type as in Standard or Extended?

[hemmerling]

They are hundreds of lines long, the old one would look like:

ip access-list extended TEST_OUT_ACL-A12
permit tcp any any established
permit tcp any host 10.10.10.0 0.0.0.255 eq 22

and then I remove that ACL from the VLAN interface with:
no ip access-group TEST_OUT_ACL-A12 out
and then delete the old ACL with the standard

no ip access-list extended TEST_OUT_ACL-A12

create a new one like normal:

ip access-list extended TEST_OUT_ACL-A13
permit tcp any any established
permit tcp any host 10.10.10.0 0.0.0.255 eq 22
permit udp any host 10.10.10.0 0.0.0.255 eq 69 514

and apply it to the VLAN interface.

The new line would increase in the match count, but the lines that were there before will still be working (I have tested that they are working) but their matches will never increase now, only the new line, and if I do the process over and add a new line, the same, the old ones will stop and the new ones will work.

In ACLs with denies that are seeing random high ports they continue to match and increase the match count, but any specific match doesn't.

Here is the output of the ACL being applied for an hour, and an example of the data moving through it, every one of the 190 lines you don't see is matching, just not increasing the match count.

SW#sh access-lists TEST_OUT_ACL-A13 | in match
110 permit ip any host xxx.xxx.xxx.xxx (471 matches)
500 deny tcp any host xxx.xxx.xxx.xxx eq 22 445 3389 (18 matches)
1240 permit udp xxx.xxx.xxx.xxx 0.0.0.255 xxx.xxx.xxx.xxx 0.0.0.255 eq 902 (515 matches)
1260 permit tcp xxx.xxx.xxx.xxx 0.0.0.255 xxx.xxx.xxx.xxx 0.0.0.255 eq www 443 (335 matches)
1630 deny tcp xxx.xxx.xxx.xxx 0.0.0.255 xxx.xxx.xxx.xxx 0.0.255.255 eq 22 445 (133 matches)
1640 deny udp any eq domain netbios-ns xxx.xxx.xxx.xxx 0.0.0.255 (388 matches)
1650 deny icmp xxx.xxx.xxx.xxx 0.255.255.255 xxx.xxx.xxx.xxx 0.0.0.255 (54 matches)
1680 permit ip any host xxx.xxx.xxx.xxx log-input (9 matches)
1690 permit ip any host xxx.xxx.xxx.xxx log-input (10 matches)
1750 permit ip any any log-input (30 matches)

SW#sh int vlan 666 | in put rate

5 minute input rate 10658000 bits/sec, 1445 packets/sec
5 minute output rate 3930000 bits/sec, 1154 packets/sec

and the ACLs do start with "permit tcp any any established" but it shows no matches either so it didn't match my filter.
That data rate is constant and has been for an hour, but of the 200+ lines, only those "match".

The other lines are matching, just not logging where instructed and are not increasing the match count while matching.

I have tried remove lines by number and adding them back with a new number, nothing, no match increase.

I have renumbered the ACLs, same effect, no increase in the older lines.

I'm sure some engineer will be able to tell me to clear some command related to CEF or something, but I'm at a loss  and don't know how to fix this short of rebooting.

[David Ruess]

You say "every one of the 190 lines you don't see is matching, just not increasing the match count". What are you using to measure or verify this. If its not increasing the count of the ACE then what is the basis for saying "its working". The flow could be hitting an ACL entry further down the line rather than the one it was intended for.

Changing ACE changes the meaning of ACLs as in maybe the packets are matching on another ACE that it wasn't before because either a more or less specific match  was added above the working ACE. Once a packet hits an ACL entry that matches it (regardless of specificity) it will use that entry even if more specific ones for that packet are below.

I see you have a permit ip any any at the end. Does that increase at all? Can you provide an example traffic flow of one of your tests? As in src/dst IP and port along with the ACE you think its hitting vs which one it is hitting based on counters incrementing if you can.

[hemmerling]

I test it by seeing the allow traffic reach the destination and not logging in the lower logging lines.

The volume of data from various sources passing through this ACL is massive, Windows and Linux servers serving thousands of different IPs every minute. The fact that the "permit tcp any any established" isn't counting up should be enough to show the issue I'm describing. 

Here is an ACL left to monitor overnight with the traffic running at the rate listed. The established line was even removed from 10 and was readded as 5 before being applied to see if it would start a match count, it hasn't gone up in 14 hours. It is some kind of bug, and I suspect it can be made to work by clearing some sort of cached routing/TCAM area of memory by some sort of unknown or never used command.

SW#sh access-lists TEST_OUT_ACL1
Extended IP access list TEST_OUT_ACL1
5 permit tcp any any established
20 permit ip any host xxx.xxx.xxx.123 log-input (6616 matches)
30 permit ip any host xxx.xxx.xxx.126 log-input (2234 matches)
40 permit ip any host xxx.xxx.xxx.120 log-input (2598 matches)
50 permit ip any host xxx.xxx.xxx.142 log-input (5315 matches)
60 permit ip any host xxx.xxx.xxx.15 log-input (2432 matches)
70 permit ip any host xxx.xxx.xxx.136 log-input
80 permit ip any host xxx.xxx.xxx.12 log-input
90 permit ip any host xxx.xxx.xxx.14 log-input
100 permit ip any host xxx.xxx.xxx.27 log-input
110 permit ip any any (50615 matches)

SW#sh int vlan 666 | in put rate
5 minute input rate 92280000 bits/sec, 8547 packets/sec
5 minute output rate 5603000 bits/sec, 2163 packets/sec

[David Ruess]

I wouldn't consider this a bug just yet as your ACL is working its just not hitting on some entries which usually means traffic isn't doing what you think its doing or the traffic isn't hitting the ACL  (not ruling out a bug either).

I do see that your ACL is on a VLAN interface and I assume by the name its applied in the out direction. That being said Outbound ACLs on VLAN interfaces means traffic must be going TO devices on VLAN 666 coming from outside that VLAN in order to hit the ACL. 

Can you telnet to a device from outside that VLAN to a device in that VLAN and do a packet capture as well as checking the TCP established line again after you do that?

-David

[MHM Cisco World]

One point 

The SVI VLAN acl is different than RACL.

The direction IN with RACL the sourc can be any and destiantion is IP behind interface you apply ACL under it.

The direction OUT with RACL is IP reach via interface to Any

That is totally reverse in vlan svi acl' the IN direction is for effect traffic generate from host in that vlan to any and OUT direction is from any to host in that vlan.

Just want to make you notice this behavior maybe it is issue here

MHM

[hemmerling]

Yes, it's an out ACL and it's on an interface vlan so the source is external to the vlan and the destination is inside the vlan.

It would be hard to do this on the production vlan in question and I can try to show the results on a different one if it's needed, but the "established" ACE alone should be enough to show that it's not working correctly. ACLs assigned to the inside seemingly function as expected, it does this on outbound (but really inbound) ACLs, and not on every line.

Are you saying that you think there is no established traffic so its match count isn't going up, or are you saying that you think there is and it's not passing that traffic and so not ticking up the match count, or like I thought, there is established traffic and the ACE is working, just not ticking up match counts. 
When I make an ACE (from the logged traffic that doesn't match an ACE already other than the logging one), that traffic is still getting to the device, it doesn't hit the logging ACE anymore, and its match count doesn't go up on the ACE it is now matching.
Is there a command to rebuild the TCAM or whatever holds the processed ACLs ACEs?

[David Ruess]

Until more troubleshooting it could be a couple of things. 

Does your permit ip any any statement at the end increment? If so like I said something isnt hitting the ACE but it IS being processed by the ACL. I've never seen an individual ACE just not work/increment.

You could also remove the ACL from the interface, and from the config. Then recreate it and apply it to the interface but I believe you have done that already.

Since you already have a permit ip any any as the last line you could change the ACL to be just the TCP established and the permit ip any any with the log keyword. If the ip any any statement increases in matches while the established doesnt then its not hitting that ACE.

You can also try clearing counters. I believe it also applies to ACLs.

Can you show the output of the command show run int vlan 666 (or whichever VLAN you're testing this on)

-David