c9800 Flex Connect Post Auth ACL Issue

[xxkozxx]

I already have a TAC case open on this but I wanted to see if anyone else has run across this...

I am deploying new c9800-40's and have a requirement to do flex connect with CWA. I had this set up with my 5520's and it worked without issue but for some reason I am hitting a wall with the new configs.

I have the SSID set up and the CWA redirect and subsequent auth happens without issue. However when the post auth "internet-only" ACL gets applied on the AP, the client has no network access. The exact same ACL is used on a centrally switch WLAN and works without issue.

I've test this on both 3802i and 9136i AP's and it made no difference.

I have found plenty of documentation on how to set up the redirect (much of it vague) and none seem to actually discuss the post-auth ACL portion of the configuration. Any help would be greatly appreciated.

EDIT: In testing with TAC, we found that the ACL that is downloaded to the AP is blocking the user. It is not adding an exception for the client so the line in the ACL which blocks the 10.x network is effectively blocking the client and access to the gateway. We were able to test this theory by removing the line referencing the 10.x network and the client had access. However, this negates the security control by also allowing the client access to internal resources on a guest network. The same ACL processed by the controller (centrally switched vs. flex connect) works without issue. 

Post-Mortem: I worked with my account team, TAC and a wireless engineer to test this both on AIR-OS and on the IOS-XE controllers. The behavior is the same. After much debate, it was determined that the ACL behavior is by design as the AP Operating System does not currently process ACL's in the same manner as switches, routers or wireless controllers. As a result you are left with solutions that require additional configuration and or infrastructure to support it. So, you can leave it centrally switched or add infrastructure and or configuration to accommodate security policy around locally switched networks (i.e. ACL's on upstream switch, firewall gateway, VRF's etc...) 

I have requested that a feature request be opened on this issue as I firmly believe that if I can push policy via a NAC or SGT type solution to pretty much any other Cisco Product that the AP should behave in the same manner. Hopefully, this functionality will come in later releases. 

Here is the Wireless Configuration:

========
  WLAN
========
wlan flex-test 7 flex-test
 assisted-roaming prediction
 dot11ax target-waketime
 dot11ax twt-broadcast-support
 mac-filtering ise-radius-aaa
 scan-report association
 no security ft adaptive
 no security wpa
 no security wpa wpa2
 no security wpa wpa2 ciphers aes
 no security wpa akm dot1x
 security dot1x authentication-list ise-radius-aaa
 no shutdown

=============
 FLEX PROFILE
=============
wireless profile flex cwa-flex-profile
 acl-policy PERMIT-ANY
 acl-policy ACL-INTERNET-ONLY
 acl-policy ACL-WEBAUTH-REDIRECT
  central-webauth
 ip http client proxy 0.0.0.0 0
 native-vlan-id 50
 vlan-name flex-client-wireless
  vlan-id 50

===================
 FLEX POLICY  PROFILE
===================
wireless profile policy cwa-portal-flex-policy-profile
 aaa-override
 aaa-policy dvn-aaa-policy
 no accounting-interim
 accounting-list ise-radius-aaa
 no central dhcp
 no central switching
 no flex umbrella dhcp-dns-option
 http-tlv-caching
 ipv4 flow monitor wireless-avc-basic input
 ipv4 flow monitor wireless-avc-basic output
 ipv6 flow monitor wireless-avc-basic-ipv6 input
 ipv6 flow monitor wireless-avc-basic-ipv6 output
 nac
 passive-client
 radius-profiling
 vlan 50
 no shutdown

================
 AP JOIN SITE TAG
================
wireless tag site ap-join-flex-site-tag
 ap-profile flex-test-ap-join-profile
 flex-profile cwa-flex-profile
 no local-site

===============
 AP JOIN PROFILE
===============
ap profile flex-test-ap-join-profile
 country US
 mgmtuser username [REDACTED] password [REDACTED] secret [REDACTED]
 ntp ip 0.0.0.0
 no oeap link-encryption
 no oeap local-access
 no oeap provisioning-ssid
 preferred-mode ipv4
 ssh
 statistics ap-system-monitoring alarm-enable
 statistics ap-system-monitoring enable
 statistics ap-radio-monitoring action radio-reset
 statistics ap-radio-monitoring alarm-enable
 statistics ap-radio-monitoring enable
 syslog host 255.255.255.255

==================
 AP THAT IS TAGGED
==================
ap 6871.61f2.2a04
 policy-tag flex-test-policy-tag
 rf-tag dvn-campus-rf-tag
 site-tag ap-join-flex-site-tag

Here are my ACL's:

ip access-list extended ACL-INTERNET-ONLY
 10 permit udp any any eq bootps
 20 permit udp any any eq bootpc
 30 permit udp any any eq domain
 40 permit tcp any 172.17.242.0 0.0.1.255 <---ACCESS TO ISE PSN's Post-Auth
 50 permit tcp 172.17.242.0 0.0.1.255 any <---ACCESS FROM ISE PSN's Post-Auth
 60 deny ip any 192.168.0.0 0.0.255.255
 70 deny ip any 172.16.0.0 0.15.255.255
 80 deny ip any 10.0.0.0 0.255.255.255
 90 permit ip any any

ip access-list extended ACL-WEBAUTH-REDIRECT
 10 deny ip any 172.17.242.0 0.0.1.255
 20 deny ip 172.17.242.0 0.0.1.255 any
 30 deny udp any any eq domain
 40 deny udp any eq domain any
 50 permit tcp any any eq www

I see the client in a run state and in ISE I see a full complete auth against the CWA portal. 

MAC Address        AP Name        Type ID       State         Protocol Method     Role
------------------------------------------------------------------------------------------------------------
2222.98b7.0b43     AP6871-61F2-2A04   WLAN 7    Run           11ax(5)  MAB        Local 

Here I see the association and auth applied:

WLC1# sh wireless client mac-address 2222.98b7.0b43 detail 
Client MAC Address : 2222.98b7.0b43
Client MAC Type : Locally Administered Address
Client DUID: NA
Client IPv4 Address : 10.2.18.188
Client IPv6 Addresses : fe80::41f:24f5:bbf7:850
Client Username : XXXXXXXXXXXXXX
AP MAC Address : 6871.6196.0630
AP Name: AP6871-61F2-2A04
AP slot : 1
Client State : Associated
Policy Profile : cwa-portal-flex-policy-profile
Flex Profile : cwa-flex-profile
Wireless LAN Id: 7
WLAN Profile Name: flex-test
Wireless LAN Network Name (SSID): flex-test
BSSID : 6871.6196.063f
Connected For : 150 seconds 
Protocol : 802.11ax - 5 GHz
Channel : 161
Client IIF-ID : xxx
Association Id : 2
Authentication Algorithm : Open System
Idle state timeout : N/A
Session Timeout : 1800 sec (Remaining time: 1594 sec)
Session Warning Time : Timer not running
Input Policy Name  : None
Input Policy State : None
Input Policy Source : None
Output Policy Name  : None
Output Policy State : None
Output Policy Source : None
WMM Support : Enabled
U-APSD Support : Disabled
Fastlane Support : Enabled
Client Active State : Active
Power Save : OFF
Current Rate : 24.0
Supported Rates : 6.0,9.0,12.0,18.0,24.0,36.0,48.0,54.0
AAA QoS Rate Limit Parameters:
  QoS Average Data Rate Upstream             : 0 (kbps)
  QoS Realtime Average Data Rate Upstream    : 0 (kbps)
  QoS Burst Data Rate Upstream               : 0 (kbps)
  QoS Realtime Burst Data Rate Upstream      : 0 (kbps)
  QoS Average Data Rate Downstream           : 0 (kbps)
  QoS Realtime Average Data Rate Downstream  : 0 (kbps)
  QoS Burst Data Rate Downstream             : 0 (kbps)
  QoS Realtime Burst Data Rate Downstream    : 0 (kbps)
Mobility:
  Move Count                  : 0
  Mobility Role               : Local
  Mobility Roam Type          : None
  Mobility Complete Timestamp : 07/11/2023 12:39:47 CDT
Client Join Time:
  Join Time Of Client : 07/11/2023 12:39:47 CDT
Client State Servers : None
Client ACLs : None
Policy Manager State: Run
Last Policy Manager State : IP Learn Complete
Client Entry Create Time : 343 seconds 
Policy Type : N/A
Encryption Cipher : None
Transition Disable Bitmap : 0x00
User Defined (Private) Network : Disabled
User Defined (Private) Network Drop Unicast : Disabled
Encrypted Traffic Analytics : No
Protected Management Frame - 802.11w : No
EAP Type : Not Applicable
VLAN Override after Webauth : No
VLAN : 50
Multicast VLAN : 0
WiFi Direct Capabilities:
  WiFi Direct Capable           : No
Central NAT : DISABLED
Session Manager:
  Point of Attachment : capwap_9040000e
  IIF ID             : xxx
  Authorized         : TRUE
  Session timeout    : 1800
  Common Session ID: xxx
  Acct Session ID  : xxx
  Last Tried Aaa Server Details:
        Server IP : 172.17.243.40
  Auth Method Status List
        Method : MAB
                SM State        : TERMINATE
                Authen Status   : Success
  Local Policies:
        Service Template : wlan_svc_cwa-portal-flex-policy-profile (priority 254)
                VLAN             : 50
                Absolute-Timer   : 1800
  Server Policies:
                Filter-ID        : ACL-INTERNET-ONLY
  Resultant Policies:
                Filter-ID        : ACL-INTERNET-ONLY
                VLAN             : 50
                Absolute-Timer   : 1800
DNS Snooped IPv4 Addresses : None
DNS Snooped IPv6 Addresses : None
Client Capabilities
  CF Pollable : Not implemented
  CF Poll Request : Not implemented
  Short Preamble : Not implemented
  PBCC : Not implemented
  Channel Agility : Not implemented
  Listen Interval : 0
Fast BSS Transition Details :
  Reassociation Timeout : 0
11v BSS Transition : Implemented
11v DMS Capable : No
QoS Map Capable : No
FlexConnect Data Switching : Local
FlexConnect Dhcp Status : Local
FlexConnect Authentication : Central
Client Statistics:
  Number of Bytes Received from Client : 299981
  Number of Bytes Sent to Client : 2507060
  Number of Packets Received from Client : 1695
  Number of Packets Sent to Client : 2121
  Number of Policy Errors : 0
  Radio Signal Strength Indicator : -32 dBm
  Signal to Noise Ratio : 64 dB
Fabric status : Disabled
Radio Measurement Enabled Capabilities
  Capabilities: Link Measurement, Passive Beacon Measurement, Active Beacon Measurement, Table Beacon Measurement, Statistics Measurement, AP Channel Report
Client Scan Report Time : Timer not running
Client Scan Reports 
  Last Report @: 07/11/2023 12:43:00
Assisted Roaming Neighbor List 
Nearby AP Statistics:
EoGRE : Pending Classification
  Device Protocol  : HTTP
    Type             : 1    115 
    Data             : 73
    00000000  00 01 00 6f 4d 6f 7a 69  6c 6c 61 2f 35 2e 30 20  |...oMozilla/5.0 |
    00000010  28 69 50 68 6f 6e 65 3b  20 43 50 55 20 69 50 68  |(iPhone; CPU iPh|
    00000020  6f 6e 65 20 4f 53 20 31  36 5f 35 5f 31 20 6c 69  |one OS 16_5_1 li|
    00000030  6b 65 20 4d 61 63 20 4f  53 20 58 29 20 41 70 70  |ke Mac OS X) App|
    00000040  6c 65 57 65 62 4b 69 74  2f 36 30 35 2e 31 2e 31  |leWebKit/605.1.1|
    00000050  35 20 28 4b 48 54 4d 4c  2c 20 6c 69 6b 65 20 47  |5 (KHTML, like G|
    00000060  65 63 6b 6f 29 20 4d 6f  62 69 6c 65 2f 31 35 45  |ecko) Mobile/15E|
    00000070  31 34 38                                          |148             |
Max Client Protocol Capability: Wi-Fi6 (802.11ax)
WiFi to Cellular Steering : Not implemented
Cellular Capability : N/A
Advanced Scheduling Requests Details:
  Apple Specific Requests(ASR) Capabilities/Statistics:
    Regular ASR support: DISABLED

And on the AP I can see the ACL being applied to the client but it shows a bunch of drops:

AP6871-61F2-2A04#sh client access-lists post-auth all 2222.98b7.0b43
Post-Auth URL ACLs for Client: 22:22:98:B7:0B:43
IPv4 ACL: ACL-INTERNET-ONLY

IPv6 ACL: 

ACTION  URL-LIST
Resolved IPs for Client: 22:22:98:B7:0B:43
HIT-COUNT       URL             ACTION  IP-LIST

ACL-INTERNET-ONLY
        rule 0: allow true and ip proto 17 and dst port 67
        rule 1: allow true and ip proto 17 and dst port 68
        rule 2: allow true and ip proto 17 and dst port 53
        rule 3: allow true and dst 172.17.242.0 mask 255.255.254.0 and ip proto 6
        rule 4: allow true and src 172.17.242.0 mask 255.255.254.0 and ip proto 6
        rule 5: deny true and dst 192.168.0.0 mask 255.255.0.0
        rule 6: deny true and dst 172.16.0.0 mask 255.240.0.0
        rule 7: deny true and dst 10.0.0.0 mask 255.0.0.0
        rule 8: allow true
No IPv6 ACL found
         Acl name Quota Bytes left In bytes Out bytes In pkts Out pkts Drops-in Drops-out
ACL-INTERNET-ONLY     0          0     1756     38725       8      374      348         9
CLIENT STATE: FWD
WEBAUTH_REQUIRED: FALSE
DNS POST AUTH:  FALSE
PREAUTH ENABLED: FALSE
POSTAUTH ENABLED: TRUE

[marce1000]

   - Have a checkup of the 9800-40's configuration(s) with the CLI command show tech wireless ; feed the output into : 
                              https://cway.cisco.com/wireless-config-analyzer/
 To begin with this proceduren is strongly advised 

   - Make sure that for flexconntect APs , the necessary VLANs corresponding to WLANs are arriving at the APs through trunking
  

   For client debugging use : https://logadvisor.cisco.com/logadvisor/wireless/9800/9800CWA
                                           https://logadvisor.cisco.com/logadvisor/wireless/9800/9800ClientConnectivity
  The latter (full client debugs) can be analyzed with : https://cway.cisco.com/tools/WirelessDebugAnalyzer/

   Also noted   : https://bst.cisco.com/bugsearch/bug/CSCvr58194

 M.
                                            

[xxkozxx]

So question about your statement:

- Make sure that for flexconntect APs , the necessary VLANs corresponding to WLANs are arriving at the APs through trunking

I have only one VLAN for wireless and all my AP's have always been access ports with no specific tags on the AP themselves. Is it really necessary to trunk the AP if you only have one vlan and that native vlan is set in the wireless config?

I should also note that in testing, I changed the ACL to a permit any ACL and that gives network access.

[marce1000]

                                     >... Is it really necessary to trunk the AP
  - If you have only one WLAN that would not be needed , use the WirelessAnalyzer procedure (first step from my reply ) too !

 M.

[xxkozxx]

@marce1000 

Configuration Analyzer came back with nothing related to a WLAN configuration issue. Doing some additional testing I trimmed the ACL down to just deny RFC1918 addresses and permit everything else. This blocked the system from even reconnecting after CoA. 

It's as though the ACL gets applied after CoA but it is not properly processing traffic. This whole issue is related to the post-auth ACL on flex connect. I see good association and CWA is properly auth'ing. I see the post-auth ACL get applied on the AP itself for that client MAC. But once the ACL is applied, the client is dead in the water (with the exception of the DNS, DHCP and ISE PSN allows I have above the RFC1918 denies). 

Conversely, in a centrally switched scenario, the ACL works without issue. There seems to be a difference in how the AP is applying the ACL to the client versus how the WLC handles it. 

I also have a dot1x wlan that is configured on flex connect and uses a permit any. That wlan has no issues. If I change the CWA wlan to use a permit any, the client has full access. It only seems to have an issue when an ACL is applied that has denies in it. And I can clearly see in the ACL stats, that it is dropping packets both in and out.

[marce1000]

 -  What software version is the controller on and or consider :
                          https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/214749-tac-recommended-ios-xe-builds-for-wirele.html
       if applicable , 

 M.

[xxkozxx]

17.9.3

[marce1000]

            - Looks good (recent) ; for the rest it is out of my capacities ; you may want to engage Cisco on the matter (TAC) , 

 M.

[Rich R · Accepted solution]

Yes I think you need to keep pushing TAC to open a bug for it.  They will want to repro it first which should be pretty easy for them to do.  Get the bug opened then the BU assign a dev to look at it.  They will probably come back and say "feature enhancement" because that seems to be standard procedure for many bugs these days.  That's when you quote "feature parity" because it already works fine on AireOS so it's not an enhancement at all - they've just broken it on IOS-XE.  It's good that you're already on 17.9.3 because that's effectively the most up to date release (discounting 17.10 and 17.11 because they're limited support releases).

I suspect very few people use post-auth ACL on flex local switching so you're probably the first to discover and report this!  Most would put guest traffic in a separate VLAN so the enforcement can be done at the next hop rather than relying on AP ACL.  And that might be the workaround you have to consider because a fix won't be available before 17.9.5 at the earliest unless you can persuade them to produce a APSP or SMU for you once they have the fix - which would likely be based on 17.9.4 which should be out soon.

[xxkozxx]

Rich,

Thanks for the reply. I am currently doing exactly as you suggested. It appears that the ACL's are improperly applied in this manner. I suspect you are correct in that very few people are using post-auth ACL's in flex connect. However, it's much more scalable than having to set up infrastructure at several locations (such as branch offices). Though adding next-hop infrastructure is an option and a workaround I have considered, it's not the preferred method. For now, I have left that particular WLAN in centrally switched until this issue can be resolved. I will follow-up with my account team and TAC to see what next steps are...

[Rich R]

Cool - let us know the bug id once you get it.

[Erich Schommarz]

We have exactly the same issue. I was trying out to find out how the acl / where the acl's are assigned to -> Ingress Traffic, Egress Traffic, to or from Client.

Today I did a lot of testing and for me it seems that the ACL is applied in both directions. So I adapted my ACL to reflect this and now it works for me. Couldn't find any actual documentation except AirOS 8.2. There this behaviour is mentioned. But since 8.2 a lot of things have changed and added with Flexconnect: https://www.cisco.com/c/en/us/td/docs/wireless/controller/8-2/config-guide/b_cg82/b_cg82_chapter_010010110.pdf

Interesting that TAC doesn't even know how things are applied / should work!

My ACL for the moment to block communication internal and allow traffic to Internet:

ip access-list extended BLOCK-INTRAZONE-188
permit icmp any any
permit udp any any eq bootps
permit udp any any eq bootpc
deny ip 10.0.188.0 0.0.1.255 10.0.188.0 0.0.1.255
permit ip any any

[Erich Schommarz]

moved to the end

[Aomar bahloul]

Hi,

I'm having the exact same issue. My use case is limiting access for some clients to only internet and few internal resources I don't want to create a separate WLAN just for that. Access to internal resources works but Internet access doesn't. I suspect it's because of the "permit ip any any" ACE at the end if the Felxconnect ACL. I do see it replaced by "allow true" statement on the AP client ACL but that doesn't seem to work for returning traffic since I can see outbound traffic reach the upstream firewall. 

Were you able to file a Bug with TAC, if so, what's the Bug ID I would like to get updates when it's fixed. 

FYI I'm running version 17.9.4 and this still hasn't been fixed. 

Thank you. 

Aomar.

[Rich R]

@Aomar bahloul - @xxkozxx said above "After much debate, it was determined that the ACL behavior is by design as the AP Operating System" so as far as Cisco is concerned this is not a bug.

@Erich Schommarz confirmed above that it is working (once you understand how it is applied) and gives an example of a working ACL.  Have you followed that guidance?

I don't think it's your permit ip any any causing the problem - it's more likely that you've misunderstood how the prior deny statements are being applied to traffic in both directions.

> FYI I'm running version 17.9.4 and this still hasn't been fixed. 
Well since it's not considered to be a bug, it will not be "fixed".
If you can persuade your Cisco account team, and have a large enough business justification, you could raise an enhancement request to have the behaviour changed.  But realistically I don't think there is much chance of that happening.  You might get more mileage out of getting a documentation bug opened to have the behaviour documented and explained fully in the config guides.