3650 IOS-XE Outbound access-lists not working

[nzmatto]

We have recently replaced two Cisco 3560 switches running IOS12.2(53r)SEY4 with two Cisco 3650-24TS switches running IOS-XE Version 03.06.04E(fc2). The configuration was copied from the old to the new switches with few minor changes, like the port names. Since the migration we have discovered the outbound access lists on our interfaces seem not to be working. There are no matches and data is flowing out the interfaces that should be being dropped. We would expect to see a huge number of drops and comparatively few permits.

The problem is the customer at the far end of the link described below are receiving UDP3033 traffic. They should only be receiving traffic on UDP3031. I have even gone as far as adding a specific deny statement (line 35) to test this but have no matches. NB inbound access-lists are all working as expected, it is only the outbound ones that have stopped working. 

interface GigabitEthernet1/0/19
description sed105
no switchport
ip address 10.45.203.27 255.255.255.248
ip access-group sed105_in in
ip access-group sed105_out out
ip helper-address 174.73.102.223
ip directed-broadcast
speed 100
duplex full

ip access-list extended sed105_out
permit icmp any any
permit udp 174.73.102.192 0.0.0.31 10.45.207.32 0.0.0.15 eq 3031
permit udp 174.73.102.192 0.0.0.31 10.45.207.160 0.0.0.15 eq 3031
deny udp 174.73.102.192 0.0.0.31 10.45.207.32 0.0.0.15 eq 3033
deny ip any any

show ip access-lists sed105_out
Extended IP access list sed105_out
10 permit icmp any any
20 permit udp 174.73.102.192 0.0.0.31 10.45.207.32 0.0.0.15 eq 3031
30 permit udp 174.73.102.192 0.0.0.31 10.45.207.160 0.0.0.15 eq 3031
35 deny udp 174.73.102.192 0.0.0.31 10.45.207.32 0.0.0.15 eq 3033
40 deny ip any any

As you can see there are no matches lists next to the show ip access-lists output, this is the same for a second customer on the same switch. As a comparison here is the output from our alternate site still running on the old switches.

Extended IP access list sed106_out
10 permit icmp any any
20 permit udp 174.73.102.192 0.0.0.31 10.45.207.32 0.0.0.15 eq 3031 (47646919 matches)
30 permit udp 174.73.102.192 0.0.0.31 10.45.207.160 0.0.0.15 eq 3031 (47646919 matches)
40 deny ip any any (641898070 matches)

Has there been some change in the functionality between IOS and IOS-XE with regards to how outbound access lists function?
Am I missing something else?

Of note, we are using a large number of helper addresses in this configuration. I know that locally created traffic is not subject to outbound access lists. Is there something that may be changing the source address within IOS-XE to make this traffic appear local?

Thanks Matt 

[Reza Sharifi]

Can you try adding a log to the end of each access-list and test again?

20 permit udp 174.73.102.192 0.0.0.31 10.45.207.32 0.0.0.15 eq 3031 log

permit or reject should also show up in "show logg"

HTH

[nzmatto]

I have added this on and are seeing no matches despite having about 15 packets per second flowing out the interface. Last night we were monitoring traffic out the interface and we're seeing it like this:

Jun 22 18:59:12.377 NZST: UDP: forwarding packet 255.255.255.255(58068) to 10.45.207.45(3031)

Due to the environment I am not easily able to connect to a monitor port and capture the data.

Our next step here is going to be to set up another identical switch in a lab environment with minimal config on it to test further. 

[Francesco Molino]

On IOS XE, you have wireshark embedded, take a look on monitor capture configuration.

In your lab environment, try to invert the source and destination to see if you have a match also.

Thanks 

PS: Please don't forget to rate and mark as correct answer if this solved your issue 

[nzmatto]

Thanks for that, I was unaware of the new wireshark capabilities on it. We are building this up now.

[Francesco Molino]

Ok. I'll answer you tomorrow as it's time to go bed for me :-)

[nzmatto]

After a lot more testing and research it appears what we are seeing is not a bug but a functional change to how the outbound access control lists are applied. 

On the old switches and IOS the IP helpers that forwarded the UDP broadcasts were on VLAN 1. They were routed out the interfaces, and outbound access list applied which filtered the unwanted UDP traffic.

On the new switches and IOSXE it treats VLAN 1 and the routed port as part of the control plane so will not apply outbound access-lists as it is treated as traffic that has originated on the switch.

It seems the last sentence of my initial query was very close to the cause. :-) 

"Is there something that may be changing the source address within IOS-XE to make this traffic appear local?"

So, the final question (for which I suspect the answer is no) would have to be....is there any way to force an outbound access-list to apply even for traffic originating from the control plane?
We do have another design solution that can address this, but I think it would be good to see is we can still achieve the desired result inside the switches. 
Thanks.

[Francesco Molino]

Hi

I'm not sure I get what you explained.

if you do a wireshark, you must see the ip of your svi as relay agent, again maybe I misunderstood what you explained.

acl on control plane could be done by using policing features in general or drop. Why do you want to authorize ip helper traffic over control plane if you don't apply any blocking acl on it?

Thanks

[nzmatto]

There is a behavior change between the switches.

The iphelper addresses, which are forwarded out vlan 1, used to be subject to the outbound access list of an interface in vlan1.

The same traffic is now seen as having been generated by the switch. Traffic generated by the switch is not subject to outbound access lists in the same manner as control plane traffic, so we are not able to block the unwanted traffic via an outbound access list. 

It is not a 'fault' per se, but rather a difference in behaviour between the old IOS and new IOS XE. 

[Stuart Holcombe]

Hi nzmatto

Were there any further conclusions to this? I'm encountering a very similar issue on outbound access-lists. 

Thanks

Stuart

[nzmatto]

No, there was no further update. This is a change in the behaviour between IOS and IOS-XE. 
As forwarded traffic appears as having come from the control plane it is not subject to outbound access lists. I am not expecting any further update from Cisco on this. 

[Francesco Molino]

Hi

In addition of Reza Sharifi tests, could you also do a monitor capture based on this acl?

Thanks