cancel
Showing results forĀ 
Search instead forĀ 
Did you mean:Ā 
cancel
930
Views
1
Helpful
20
Replies

Policy based routing with reflective ACL not working as expected

Karl3
Level 1
Level 1

Hi everyone. I have a 2960x acting as a "core" switch doing inter-vlan routing. Vlan 400 is for IoT. Other vlan hosts need to be able to access hosts in the iot vlan, no hosts in the iot vlan can access anything but internet. All hosts in the Iot vlan need to access the internet through an external VPN gateway on 172.16.30.42.
After configuring PBR, it works as expected. But when configured with reflactive ACL, things didn't work as expected.

configs:

ip access-list extended iot-1-in
5 evaluate iot-1-in-refl
10 deny ip any 10.0.0.0 0.255.255.255 log
20 deny ip any 172.16.0.0 0.15.255.255 log
30 deny ip any 192.168.0.0 0.0.255.255 log
40 permit ip any any

ip access-list extended iot-1-out
10 permit ip any any log reflect iot-1-in-refl

ip access-list extended vpn-pbr-acl1
10 deny ip any 10.0.0.0 0.255.255.255
20 deny ip any 172.16.0.0 0.15.255.255
30 deny ip any 192.168.0.0 0.0.255.255
40 permit ip any any

route-map vpn-pbr1 permit 10
match ip address pbr-acl1
set ip next-hop 172.16.30.42

interface Vlan400
ip address 172.16.4.1 255.255.255.240
ip access-group iot-1-in in
ip access-group iot-1-out out
ip policy route-map vpn-pbr1

 

The PBR config works as expected, but reflective ACL don't.

- Hosts in the IoT vlan can ping internet, and cannot ping LAN addresses.
- Hosts not in the IoT vlan cannot ping hosts in IoT vlan

When I remove `ip policy route-map vpn-pbr1` the reflective ACL works as expected, but internet traffic no longer goes to the VPN gateway

When the route-map is in place, this is what shows when showing access-lists

Extended IP access list iot-1-in
5 evaluate iot-1-in-refl
10 deny ip any 10.0.0.0 0.255.255.255 log
20 deny ip any 172.16.0.0 0.15.255.255 log (1041 matches)
30 deny ip any 192.168.0.0 0.0.255.255 log
40 permit ip any any
Reflexive IP access list iot-1-in-refl
permit icmp host 172.16.4.2 host 172.16.3.2 log (2037 matches) (time left 299)
Extended IP access list iot-1-out
10 permit ip any any reflect iot-1-in-refl log (1019 matches)
Extended IP access list vpn-pbr-acl1
10 deny ip any 10.0.0.0 0.255.255.255
20 deny ip any 172.16.0.0 0.15.255.255
30 deny ip any 192.168.0.0 0.0.255.255
40 permit ip any any

Why is it matching a permit on the reflexive ACL yet it is matched again on sequence number 20 on iot-1-in. Also one of the things I encountered is that the implicit deny seems to not exists(allowing all traffic on empty access-list)

What have I missed on these 2 components and why is have of the things configured not work as expected.

Version:

Cisco IOS Software, C2960X Software (C2960X-UNIVERSALK9-M), Version 15.2(7)E12, RELEASE SOFTWARE (fc5) on WS-C2960X-24PS-L
20 Replies 20

Hello
Your ace 20 in the ACLs is also capturing vlan 400 subnet, this need to be excluded 

Also a straight forward routed acl would be more applicable for this requirement

Note : SVI acl logic

IN = traffic originating from within vlan
OUT = external traffic towards vlan

IP access list extended vlan400_acl
deny ip any <vlan 1>
deny ip any <vlan 2>
etc
permit ip any

int vlan 400
no ip access-group iot-1-in in
no ip access-group iot-1-out out
ip access-group vlan400 IN

ip access-list extended vpn-pbr-acl1
no 10
no 20
no 30


Please rate and mark as an accepted solution if you have found any of the information provided useful.
This then could assist others on these forums to find a valuable answer and broadens the community’s global network.

Kind Regards
Paul

Karl3
Level 1
Level 1

Thank you Paul for the reply, I'm not sure if I understand why vpn-pbr-acl1 need to seq:10,20,30 removed. If the ACL is not excluded, wouldn't packets destinated for Local Addresses be also routed to the VPN Gateway?

Another thing that still don't make sense to me is the evaluation of the inbound access-list. According to https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/sec_data_acl/configuration/15-mt/sec-data-acl-15-mt-book/sec-cfg-ip-filter.html#GUID-28D4B3F2-E213-4DF5-BBCB-D12B55E857F7
After an entry is match according to the sequence of the entries in an ACL, no more entries is matched. How is the packet ICMP echo reply matched on both the entry in the reflective ACL(permit imcp) and the seq:20 deny to local network.


match ip address pbr-acl1<<- this name is wrong 

MHM 

Hello


@Karl3 wrote:

Thank you Paul for the reply, I'm not sure if I understand why vpn-pbr-acl1 need to seq:10,20,30 removed. If the ACL is not excluded, wouldn't packets destinated for Local Addresses be also routed to the VPN Gateway?

Hosts in the IoT vlan can ping internet, and cannot ping LAN addresses.
- Hosts not in the IoT vlan cannot ping hosts in IoT vlan


Yes it would, so apologies if that isnt what you intended  i must have misread your OP - but if your denying access to all other vlan users within your LAN to access to vlan 400 then basically that vlan 400 becomes just a guest vlan solely needing to reach the internet-  no ?


@Karl3 wrote:

Another thing that still don't make sense to me is the evaluation of the inbound access-list. 
After an entry is match according to the sequence of the entries in an ACL, no more entries is matched. How is the packet ICMP echo reply matched on both the entry in the reflective ACL(permit imcp) and the seq:20 deny to local network.


I would say as the initial icmp originated internally, this would trigger a state-full session thus allowing a reply in the refl acl

 

 


Please rate and mark as an accepted solution if you have found any of the information provided useful.
This then could assist others on these forums to find a valuable answer and broadens the community’s global network.

Kind Regards
Paul

srimal99
Level 1
Level 1
route-map vpn-pbr1 permit 10
match ip address pbr-acl1. <----no reference to acl as per your post. Did you mean to match vpn-pbr-acl1 ?

 In addition to above mentioned post check the acl is correct ?

Hi@Karl3 

Please change acl 

ip access-list extended iot-1-in
10 permit tcp any 10.0.0.0 0.255.255.255 established
20 deny ip any 10.0.0.0 0.255.255.255 log
30 deny ip any 172.16.0.0 0.15.255.255 log
40 deny ip any 192.168.0.0 0.0.255.255 log
50 permit ip any any

Thanks !

Karl3
Level 1
Level 1

Thank you to everyone's reply, I have been working on something else for the past day or so and didn't catch a change to further look into suggestions.

@paul driver  wrote:

Yes it would, so apologies if that isnt what you intended  i must have misread your OP - but if your denying access to all other vlan users within your LAN to access to vlan 400 then basically that vlan 400 becomes just a guest vlan solely needing to reach the internet-  no ?

Yes and No, yes because it is isolated from other vlans and have it's own internet outbound gateway, no, because it reports data to certain host in other vlan and some hosts not in the IoT vlan needs to communicate with host inside the IoT vlan for device control, hanse the need for stateful acl.

 

@srimal99  wrote:

route-map vpn-pbr1 permit 10
match ip address pbr-acl1. <----no reference to acl as per your post. Did you mean to match vpn-pbr-acl1 ?

 In addition to above mentioned post check the acl is correct ?

@MHM Cisco World  wrote:

match ip address pbr-acl1<<- this name is wrong 

MHM 

Thank you @MHM Cisco World@srimal99 for pointing out the misspelled ACL, unfortunately after double checking each ACL names and rules. That did not solve the problem. The issue did change a bit but not solved yet, will mention below

 


@Joshqun Ismayilov  wrote:

Hi@Karl3 

Please change acl 

ip access-list extended iot-1-in
10 permit tcp any 10.0.0.0 0.255.255.255 established
20 deny ip any 10.0.0.0 0.255.255.255 log
30 deny ip any 172.16.0.0 0.15.255.255 log
40 deny ip any 192.168.0.0 0.0.255.255 log
50 permit ip any any

Thanks !

I did thought of not using the tcp established to limit traffic and ditching reflective ACL, but there will be a mix of UDP and TCP traffic in and out of the vlan, so that won't do.


After fixing the misspelled ACL name and removing the log keyword so traffic won't all go through the CPU. Internet traffic from the IoT vlan get routed to the special gateway and LAN traffic are blocked from the IoT vlan. But when trying to ping from outside the IoT vlan, the icmp echo reply is blocked with maches for blocking LAN addresses counter rising

Extended IP access list CISCO-CWA-URL-REDIRECT-ACL
    100 deny udp any any eq domain
    101 deny tcp any any eq domain
    102 deny udp any eq bootps any
    103 deny udp any any eq bootpc
    104 deny udp any eq bootpc any
    105 permit tcp any any eq www
Extended IP access list iot-1-in
    10 evaluate iot-1-in-refl
    20 deny ip any 10.0.0.0 0.255.255.255
    30 deny ip any 172.16.0.0 0.15.255.255 (182 matches)
    40 deny ip any 192.168.0.0 0.0.255.255
    50 permit ip any any
Reflexive IP access list iot-1-in-refl
Extended IP access list iot-1-out
    10 permit ip any any reflect iot-1-in-refl
Extended IP access list preauth_ipv4_acl (per-user)
    10 permit udp any any eq domain
    20 permit tcp any any eq domain
    30 permit udp any eq bootps any
    40 permit udp any any eq bootpc
    50 permit udp any eq bootpc any
    60 deny ip any any
Extended IP access list vpn-pbr-acl1
    10 deny ip any 10.0.0.0 0.255.255.255
    20 deny ip any 172.16.0.0 0.15.255.255
    30 deny ip any 192.168.0.0 0.0.255.255
    40 permit ip any any

I also noticed the outbound traffic was not reflecting onto the iot-1-in-refl acl so changed the ACL to 

ip access-list ext iot-1-out
 no 10
 10 permit ip any any log reflect iot-1-in-refl

And everything worked, but now I am getting invalid ACL logs

Aug  8 11:32:29.598: %ACLMGR-3-INVALIDPARAM: Invalid ACL type 5 encountered
-Traceback= 7E98DCz 33BDF60z 3385AA0z 3B94FDCz 3B90CCCz

It is nice that things work, but not when there are warning about invalid configs.

After googling around, I came to the conclusion that this switch does not support reflective ACL on a hardware level, on software it would allow it, just doesn't work, by placing a log keyword on the outbound ACL, the ACL is being processed on the CPU instead of TCAM(which it can somewhat handle?), the ACL works as expected, in the expense of unknow outcome the more heavy traffic gets routed between the vlan.

I know, on paper this switch does not support advanced routing feature, but also a lot of what the doc says is not supported works, so ĀÆ\_(惄)_/ĀÆ


I would really like to know if anyone have encountered things like this in the past, any idea on what I should try next? Thank you to everyone for your input.

let me see ACL and route-map and policy config 

MHM

Karl3
Level 1
Level 1
interface Vlan400
 ip address 172.16.4.1 255.255.255.240
 ip access-group iot-1-in in
 ip access-group iot-1-out out
 ip policy route-map vpn-pbr1

ip access-list extended iot-1-in
 evaluate iot-1-in-refl
 deny   ip any 10.0.0.0 0.255.255.255
 deny   ip any 172.16.0.0 0.15.255.255
 deny   ip any 192.168.0.0 0.0.255.255
 permit ip any any
ip access-list extended iot-1-out
 permit ip any any reflect iot-1-in-refl log
ip access-list extended vpn-pbr-acl1
 deny   ip any 10.0.0.0 0.255.255.255
 deny   ip any 172.16.0.0 0.15.255.255
 deny   ip any 192.168.0.0 0.0.255.255
 permit ip any any

route-map vpn-pbr1 permit 10
 match ip address vpn-pbr-acl1
 set ip next-hop 172.16.30.42

The log keyword when applying the config is flipped with the reflect keyword, this is what's get outputted when doing sh run.

ip access-list extended iot-1-out
 permit ip any any log reflect iot-1-in-refl

 

MHM

Karl3
Level 1
Level 1

Tried, now hosts outside the IoT vlan cannot ping hosts inside it. This is what shows when hosts outside the IoT vlan cannot ping hosts inside.

Extended IP access list iot-1-in
    10 deny ip any 10.0.0.0 0.255.255.255
    20 deny ip any 172.16.0.0 0.15.255.255 (4 matches)
    30 deny ip any 192.168.0.0 0.0.255.255
    40 permit ip any any
    50 evaluate iot-1-in-refl
Reflexive IP access list iot-1-in-refl
     permit icmp host 172.16.4.2 host 172.16.3.2  log (4 matches) (time left 299)
Extended IP access list iot-1-out
    10 permit ip any any reflect iot-1-in-refl log (4 matches)
Extended IP access list vpn-pbr-acl1
    10 deny ip any 10.0.0.0 0.255.255.255
    20 deny ip any 172.16.0.0 0.15.255.255
    30 deny ip any 192.168.0.0 0.0.255.255
    40 permit ip any any

 

ip access-list extended iot-1-in
 evaluate iot-1-in-refl
 deny   ip any 10.0.0.0 0.255.255.255
 deny   ip any 172.16.0.0 0.15.255.255
 deny   ip any 192.168.0.0 0.0.255.255
 permit ip any any
ip access-list extended iot-1-out
 permit ip any any reflect iot-1-in-refl log

This config is correct' evaluation must be above deny 

But I think you can not use icmp for check this type of ACL' icmp is stateless not statefull 

You see automatic reflexive acl add it include icmp that wrong it need to be icmp reply.

Reflexive IP access list iot-1-in-refl
     permit icmp host 172.16.4.2 host 172.16.3.2  log (4 matches) (time left 299)

One more note:- remove log from outbound access-list.

MHM

Karl3
Level 1
Level 1

@MHM Cisco World wrote:

You see automatic reflexive acl add it include icmp that wrong it need to be icmp reply.

Reflexive IP access list iot-1-in-refl
     permit icmp host 172.16.4.2 host 172.16.3.2  log (4 matches) (time left 299)

One more note:- remove log from outbound access-list.

I think it is just not displaying it properly, when I tried pinging back from 172.16.4.2 to 172.16.3.2, the icmp message is blocked. I forgot to mention, when an ICMP is packet is blocked, I get this message back

From 172.16.4.1 icmp_seq={} Packet filtered

so I think the switch is not displaying the entries properly. I though this Packet filtered reply was normal so I didn't mention it before. The reason for the log is to make it work, without it, the ACL would not work. Without the log arg, pinging from outside the IoT vlan does not trigger the Packet filtered message, just no reply, also the matches on seq:20 in the iot-1-in ACL is going up, so I think the ICMP reply message is blocked by that rule. The seq:10 rule in the iot-1-out ACL is not matching anything.

For a sanity check, I created a web server on 2 host, one inside the vlan, one outside.
With the setup 

ip access-list extended iot-1-out
 permit ip any any reflect iot-1-in-refl

The host outside the IoT vlan times out when trying to access the web server

Extended IP access list iot-1-in
    10 evaluate iot-1-in-refl
    20 deny ip any 10.0.0.0 0.255.255.255
    30 deny ip any 172.16.0.0 0.15.255.255 (42 matches)
    40 deny ip any 192.168.0.0 0.0.255.255
    50 permit ip any any
Reflexive IP access list iot-1-in-refl
Extended IP access list iot-1-out
    10 permit ip any any reflect iot-1-in-refl

When using

ip access-list extended iot-1-out
 permit ip any any log reflect iot-1-in-refl

The host outside the IoT vlan, can access the web server inside the IoT vlan

Extended IP access list iot-1-in
    10 evaluate iot-1-in-refl
    20 deny ip any 10.0.0.0 0.255.255.255
    30 deny ip any 172.16.0.0 0.15.255.255 (1 match)
    40 deny ip any 192.168.0.0 0.0.255.255
    50 permit ip any any
Reflexive IP access list iot-1-in-refl
     permit tcp host 172.16.4.2 eq www host 172.16.3.2 eq 52616 log (13 matches) (time left 3)
Extended IP access list iot-1-out
    10 permit ip any any reflect iot-1-in-refl log (8 matches)

The reason, I think, in theory is that the ASIC that process ACLs cannot handle reflective-acl, but the CPU can, by having the log argument, the frame/packet is not directly handled by the ASICs, but is directed to the CPU for processing. This would explain why

%ACLMGR-3-INVALIDPARAM: Invalid ACL type 5 encountered

not appearing when, the log arg is not set. Apologies if I didn't mention that before. I think when the ACL filtering is done on the ASIC, the CPU is not involved, the ASIC not being able to handle reflective-acl would also explain why the frame/packet gets match both on both seq:5,20 on the first post


@Karl3 wrote:
Extended IP access list iot-1-in
5 evaluate iot-1-in-refl
10 deny ip any 10.0.0.0 0.255.255.255 log
20 deny ip any 172.16.0.0 0.15.255.255 log (1041 matches)
30 deny ip any 192.168.0.0 0.0.255.255 log
40 permit ip any any
Reflexive IP access list iot-1-in-refl
permit icmp host 172.16.4.2 host 172.16.3.2 log (2037 matches) (time left 299)
Extended IP access list iot-1-out
10 permit ip any any reflect iot-1-in-refl log (1019 matches)
Extended IP access list vpn-pbr-acl1
10 deny ip any 10.0.0.0 0.255.255.255
20 deny ip any 172.16.0.0 0.15.255.255
30 deny ip any 192.168.0.0 0.0.255.255
40 permit ip any any

The ASIC treated the ACL like a normal ACL, incrementing the counter on the rule in iot-1-in-refl but also not knowing what to do with it and continue on matching seq:20. 

 

Another thing that seems similar is when using object-group to define networks.

object-group network rfc1918
 10.0.0.0 255.255.255.255
 172.16.0.0 255.240.0.0
 192.168.0.0 255.255.255.0

ip access-list extended iot-1-in
 evaluate iot-1-in-refl
 deny   ip any object-group rfc1918
 permit ip any any
ip access-list extended iot-1-out
 permit ip any any log reflect iot-1-in-refl
ip access-list extended vpn-pbr-acl1
 deny   ip any object-group rfc1918
 permit ip any any

With this applied, the both inbound and outbound acl gets ignored, with the PBR acl getting matched on permit ip any any when fetching the web server in the IoT vlan.

Extended IP access list iot-1-in
    10 evaluate iot-1-in-refl
    20 deny ip any object-group rfc1918
    30 permit ip any any
Reflexive IP access list iot-1-in-refl
Extended IP access list iot-1-out
    10 permit ip any any reflect iot-1-in-refl
Extended IP access list vpn-pbr-acl1
    10 deny ip any object-group rfc1918
    20 permit ip any any (12 matches)

I proceeded to add the log argument to the rules with object-group

ip access-list extended iot-1-in
 evaluate iot-1-in-refl
 deny   ip any object-group rfc1918 log
 permit ip any any
ip access-list extended iot-1-out
 permit ip any any log reflect iot-1-in-refl
ip access-list extended vpn-pbr-acl1
 deny   ip any object-group rfc1918 log
 permit ip any any

The accessing the web server in the IoT vlan did not work and same result as before, the host in the IoT vlan can access any ware without restriction as if the ACLs didn't exists.

Extended IP access list iot-1-in
    10 evaluate iot-1-in-refl
    20 deny ip any object-group rfc1918 log
    30 permit ip any any (4 matches)
Reflexive IP access list iot-1-in-refl
     permit tcp host 172.16.4.2 eq www host 172.16.3.2 eq 34080 log (5 matches) (time left 288)
     permit tcp host 172.16.4.2 eq www host 172.16.3.2 eq 47350 log (31 matches) (time left 290)
     permit tcp host 172.16.4.2 eq www host 172.16.3.2 eq 52086 log (26 matches) (time left 143)
     permit tcp host 172.16.4.2 eq www host 172.16.3.2 eq 34026 log (1 match) (time left 77)
Extended IP access list iot-1-out
    10 permit ip any any reflect iot-1-in-refl log (24 matches)
Extended IP access list vpn-pbr-acl1
    10 deny ip any object-group rfc1918 log
    20 permit ip any any (43 matches)

PBR no longer routes internet traffic to the gateway, but that's some what to be expected, PBR need IP ACL, so I switch back to not using object-group

ip access-list extended iot-1-in
 evaluate iot-1-in-refl
 deny   ip any object-group rfc1918 log
 permit ip any any
ip access-list extended iot-1-out
 permit ip any any log reflect iot-1-in-refl
ip access-list extended vpn-pbr-acl1
 deny   ip any 10.0.0.0 0.255.255.255
 deny   ip any 172.16.0.0 0.15.255.255
 deny   ip any 192.168.0.0 0.0.255.255
 permit ip any any

The internet traffic from the IoT vlan gets routed to the gateway, but the 2 in/out ACL no longer have any effecy. None of the ACL hava any effects as if they don't even exists if object-group is defined.

So far the only config that works is

ip access-list extended iot-1-in
 evaluate iot-1-in-refl
 deny   ip any 10.0.0.0 0.255.255.255
 deny   ip any 172.16.0.0 0.15.255.255
 deny   ip any 192.168.0.0 0.0.255.255
 permit ip any any
ip access-list extended iot-1-out
 permit ip any any log reflect iot-1-in-refl
ip access-list extended vpn-pbr-acl1
 deny   ip any 10.0.0.0 0.255.255.255
 deny   ip any 172.16.0.0 0.15.255.255
 deny   ip any 192.168.0.0 0.0.255.255
 permit ip any any

 

I check yesterday there is no need for log 

But this point maybe answer here

Indeed log make acl handle by cpu not tcam (ASIC)

So it can that tcam have no space to have ACL of Policy route and reflexive and IN and OUT acl

What SW platform you use?

MHM