Strange behaviour of ACL with Object Group

[Josh Mil]

Very strange behaviour on a layer 3 switch. The second entry can affect the first entry in an ACL.

A computer is in a vlan. The vlan has an ACL. The ACL references an object group to specify the protocol and port number.

interface vlan4
ip address 192.168.4.1 255.255.255.0
ip access-group acl_test in

interface Loopback0
ip address 192.168.0.1 255.255.255.255

ip access-list extended acl_test
10 deny ip any any
20 permit object-group og_test host 1.2.3.4 host 5.6.7.8

object-group service og_test
udp eq 53

The "10 deny ip any any" was added for testing purpose. I thought it should block all traffic entering the vlan. However, it doesn't. If I ping 192.168.0.1 from a computer 192.168.4.5, it's reachable! Actually the computer can browse www through this switch.

If I remove entry 20 from the ACL, ping is then blocked as expected, so are other types of traffic entering this vlan.

It is totally beyond what I can understand. How come the second entry of an ACL can affect the first entry? Shouldn't all traffic hit the first entry and get blocked? Not to mention the specified traffic type, source IP and destination IP in entry 20 are totally irrelevant to the traffic actually flows through. I literally put 1.2.3.4 and 5.6.7.8 in the entry.

Your help is much appreciated. Thank you!

[M02@rt37]

Hello @Josh Mil 

From my point of view, when an ACL contains object groups, the system recompiles the ACL into hardware, I mean TCAM, and the entry order may not match the text config!

So your “deny ip any any” at line 10 isn’t really evaluated first; that’s why unrelated trafic still passe until the object-group line is remove, and it can look like the second entry affect the first.

My suggestion is to rely on the implicit deny at the end instead of putting “deny any any” first, or to test with a simple ACL without object-groups, and see if the behavior persists.

Also, what is your ios version, I could check for known bugs with object-group ACL..

[Josh Mil]

Thank you.

This is ridiculous. Why would Cisco do this ?! The order of ACEs are critical to the logic. Shuffling ACEs destroys the logic, making the ACL unusable.

I'm not an IOS programmer, but based on my programming experience in other areas, it's not difficult to maintain the order when expanding a series of macros or compiling an ordered list. If entries in a normal ACL can maintain their order in TCAM, I don't see any reason why entries referencing object groups cannot.

I understand an object group ITSELF is an unordered list, just like an object in programming languages or a set in mathematics. But an ACL is an ordered list, just like an array.

Unfortunately I can't rely on the implicit deny rule, because I need to filter the traffic in a few steps. The code I posted is just a simplified example for testing. The original ACL actually doesn't have a "deny ip any any" entry. The original ACL permits something specific first, and then denies the rest of the type, and then permits all the rest not covered above. However I noticed the ACL is not working as expected. Thus I created a simplified code to test.

[Stefan Mihajlov]

@Josh Mil 

This isn’t a bug, it’s how IOS expands object-group ACLs. When you add that permit object-group line, IOS compiles it into extra hidden entries that can override your top deny ip any any. If you want “deny all unless permitted,” flip the order: put the permit first, then end with deny ip any any log. That way the logic works as expected.

Best regards,
Stefan Mihajlov

Mark this post as Helpful if it helped you, and Accept as Solution if it resolved your question.

[Josh Mil]

Thank you.

Does it mean an ACL is reversed if it references object groups? Would you mind sharing some sources regarding this design? I did some searches but couldn't find anything about this.

[MHM Cisco World]

Any ACL with object group is not effect.

Add normal ACL 

Add log to each entry 

Share log you get after remove onject group 

MHM

[Josh Mil]

Thank you.

Without referencing object groups, the ACL works fine.

If any ACL with object group does not work, why does Cisco provide object group in its products and mention it in its documentation...

[M02@rt37 · Accepted solution]

@Josh Mil 

Do you check Cisco’s bug toolkit or release notes for your exact ios-xe version ?

[Josh Mil]

I did some further searches. There are bugs reported with similar symptoms - when an ACL references an object group, the ACL behaves weird, such as allowing all traffic despite the deny rule in place, packet hit counter incorrect, etc.

Can't believe it but it's true. I just don't understand - Cisco is a leader of the networking industry, ACL is a basic feature of switches, and allowing all traffic through can cause serious security breaches. How come Cisco allow this to happen...

[MHM Cisco World]

It not bug' this behavior is normal for SW.

MHM

[Josh Mil]

My searches found some issues with object group which have been confirmed as bugs

CSCvy55497
CSCvw91713
CSCuq64938

Not exactly the same as the issue I encounter, but there are bugs making ACL malfunction when object groups are used.

[M02@rt37]

Hello @Josh Mil 

@Thanks for sharing that

[M02@rt37]

Hello @Josh Mil 

From my point of view, it is not unique to cisco. Any vendor that offloads ACL/firewall rules into hardware pipelines (like ASIC or TCAM) can encounter edge-case bugs where the compiled rules in hardware don’t align with the configuration set by the admin...

Cisco provide the feature because it is valuable and widely used, but just like any other vendor, certain releases can have defects that undermine its functionality until patched...

[MHM Cisco World]

Old SW and router use same IOS' so even if some command allow it not full work in SW

MHM