access-lists with object-group network not working on Catalyst 9500

Massimiliano Ballerini · ‎02-22-2022

Hello,

as part of our daily work, we are migrating a big hospital infrastructure (20k+ users), from Catalyst 6500 to new Catalyst 9500.

We have many L3 segment racks with subinterfaces on different VRFs. On some VRF we also have extendend access lists with object-groups on the subinterfaces. The access lists are working correctly in the 6500 switches.

I translated the the old access list and object groups in this way:

object-group ip address -> object-group network
host-info -> host
addrgroup any -> any
addrgroup -> object-group

Now I can paste all the configuration without syntax errors. But then it seems all the object group are just not matching at all in the access lists, while the rules without an object group matches.

I'm asking for help here because I can't find info about this behavior anywhere and seems like a big issue to me and It's strange that just noone is talking about that. We have C9500-40X VSS in pairs with suggested 17.3.4 firmware.

Here's an example of what I mean:

object-group network DENY_VRF_X
 172.23.0.0 255.255.0.0
ip access list extended RACK1_IN
 deny ip any object-group DENY_VRF_X
 deny ip any 172.23.0.0 0.0.255.255
 permit ip any any
ip access list extended RACK1_OUT
 deny ip object-group DENY_VRF_X any
 deny ip 172.23.0.0 0.0.255.255 any
 permit ip any any
interface te1/0/20.XXX
 encapsulation dot1Q XXX
 ip vrf forwarding vrf_x
 ip address 172.23.120.1 255.255.255.0
 ip access-group RACK1_IN in
 ip access-group RACK1_OUT out

In this scenario then:

Extended IP access list RACK1_OUT
 10 deny ip object-group DENY_VRF_X any <-- THIS NEVER MATCHES (why??)
 20 deny ip 172.23.0.0 0.0.255.255 any  <-- THIS MATCHES intra VRF traffic
 30 permit ip any any                   <-- THIS MATCHES extra VRF traffic

Let me know if anyone interested in this or need more info about the configuration. I just hope it's not a bug and we need to enable some hidden hardware related config line or such.

Massimiliano Ballerini · ‎02-22-2022

Hello,

searching in the documentation of the 17.3.4 firmware I found this page:

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9500/software/release/17-3/configuration_guide/sec/b_173_sec_9500_cg/object_groups_for_acls.html#concept_pdx_dk4_c3b

Where it states:

ACL statements using object groups will be ignored on packets that are sent to RP for processing

Could this be my problem? Maybe my configuration with VSS and MPLS is working in software instead of hardware?

Should I try to enable something like ip cef distributed? (Not clear to me if this is the default with VSS)

Massimiliano Ballerini · ‎02-24-2022

I tried with the command:

ip cef distributed

but it looks like it's the default.

I can't disable cef on single interfaces and no ip route-cache does not help. Also:

myswitch(config)#no ip cef distributed 
%Cannot disable CEF on this platform

How can a packet not go to the Route Processor?

coreystamp · ‎06-21-2022

I had a TAC case open for this issue on firmware 16.12.3a, and was informed that "Cisco is aware of this defect and is working on this. As of now, there are no known version that resolve this issue. The current workaround is to remove the “log” keyword from your ACL."