topic Re: Policy Extended Node not receiving TrustSec SGACLs in Software-Defined Access (SD-Access)

Policy Extended Node not receiving TrustSec SGACLs

Joshua Marks — Wed, 25 Aug 2021 01:49:59 GMT

Hi community,

We have set up a 9200L as a policy extended node in our network, hoping to leverage the micro-segmentation policy defined in ISE and applied to the rest of the network. When verifying that the automated onboarding and provisioning of this switch has TrustSec working, we have found that everything looks good except the SGACLs are requested by the 9200L and pushed by ISE, except the switch is not applying the SGACLs.

Note: all switch configuration has been automated by DNA Center 2.2.2.3, switch is running IOS-XE 17.3.3. No manual configuration has been applied to the switch.

On the switch we can see that it has SGTs applied statically to switchports:

sh run | in interface G|cts|sgt

interface GigabitEthernet1/0/1
cts manual
policy static sgt 4
no propagate sgt
interface GigabitEthernet1/0/2
cts manual
policy static sgt 255
no propagate sgt
interface GigabitEthernet1/0/3
cts manual
policy static sgt 4
no propagate sgt
interface GigabitEthernet1/0/4
cts manual
policy static sgt 4
no propagate sgt

-----------------------------brevity

interface GigabitEthernet1/0/22
cts manual
policy static sgt 4
no propagate sgt

When refreshing the cts environment data, we can see it has all of our defined SGTs in its table:

EX-01#show cts environment-data

CTS Environment Data
====================
Current state = COMPLETE
Last status = Successful
Local Device SGT:
SGT tag = 0-00:Unknown
Server List Info:
Installed list: CTSServerList1-0001, 1 server(s):
*Server: 192.168.0.220, port 1812, A-ID 12C8BD4B21731585D043636C81DAE9FF
Status = ALIVE
auto-test = TRUE, keywrap-enable = FALSE, idle-time = 60 mins, deadtime = 20 secs
Security Group Name Table:
0-00:Unknown
3-05:IoT
4-20:Employees
5-35:Lab_Employees
6-07:Guests
7-02:Lab_Guests
8-02:Lab_Isolated_Devices
9-05:Lab_Grouped_Devices
10-05:Printers
11-06:PSK_Devices
15-07:BYOD
255-05:Quarantined_Systems
Environment Data Lifetime = 86400 secs
Last update time = 23:28:53 UTC Tue Aug 24 2021
Env-data expires in 0:22:19:52 (dd:hr:mm:sec)
Env-data refreshes in 0:22:19:52 (dd:hr:mm:sec)
Cache data applied = NONE
State Machine is running

However, when refreshing the policy it is only receiving the default SGACL:

EX-01#show cts role-based permissions
IPv4 Role-based permissions from group Invalid to group Invalid:
Permit IP-00
RBACL Monitor All for Dynamic Policies : FALSE
RBACL Monitor All for Configured Policies : FALSE

This is compared to the fabric edge nodes (9300Ls) in our network which show the SGACLs related to their applied SGTs:

FE-01#sh cts role-based permissions
IPv4 Role-based permissions default:
Permit IP-00
IPv4 Role-based permissions from group 3:IoT to group 4:Employees:
Deny IP-00
IPv4 Role-based permissions from group 4:Employees to group 4:Employees:
Deny IP-00
IPv4 Role-based permissions from group 10:Printers to group 4:Employees:
Permit IP-00
IPv4 Role-based permissions from group 11:PSK_Devices to group 4:Employees:
Deny IP-00
IPv4 Role-based permissions from group 255:Quarantined_Systems to group 4:Employees:
Deny IP-00
IPv4 Role-based permissions from group 3:IoT to group 10:Printers:
Deny IP-00
IPv4 Role-based permissions from group 4:Employees to group 10:Printers:
Permit IP-00
IPv4 Role-based permissions from group 10:Printers to group 10:Printers:
Permit IP-00
IPv4 Role-based permissions from group 11:PSK_Devices to group 10:Printers:
Deny IP-00
IPv4 Role-based permissions from group 255:Quarantined_Systems to group 10:Printers:
Deny IP-00
RBACL Monitor All for Dynamic Policies : FALSE
RBACL Monitor All for Configured Policies : FALSE

When we force a policy refresh from the switch, we can see in the ISE live logs that the policy extended node is only being refreshed with the default SGACL:

Also, when we adjust the TrustSec policy in DNAC and deploy to the fabric, we can see that all switches except this policy extended node are receiving the updated SGT mappings, despite the fact that it has the relevant SGTs applied to switchports. Note that the device is included in update in ISE (this is configured by DNAC):

We have also noted that the policy extended node does not have any IP to SGT mappings as we would on a fabric edge switch:

EX-01#sh cts role-based sgt-map platform | ex 0 0
vrf dest htm flags SGT DGID MPLS Last-modified SecsSinceHit
--- ---- --- ----- --- ---- ---- ------------------------ ------------
vrf dest htm flags SGT DGID MPLS Last-modified SecsSinceHit
--- ---- --- ----- --- ---- ---- ------------------------ ------------

Compared to fabric edge switch:

FE-01# sh cts role-based sgt-map platform | ex 0 0
vrf dest htm flags SGT DGID MPLS Last-modified SecsSinceHit
--- ---- --- ----- --- ---- ---- ------------------------ ------------
3 192.168.48.26/32 0x7f5080e09678 0x0 10 2 2021/08/23 14:58:01.023 47
3 192.168.48.42/32 0x7f5080dd5218 0x4 4 1 2021/08/25 01:40:24.374 124
3 192.168.48.29/32 0x7f5080e230f8 0x4 4 1 2021/08/25 01:40:09.980 134
3 192.168.48.35/32 0x7f5080dc99b8 0x0 4 1 2021/08/25 01:14:20.382 0

In case licensing plays a factor, this is what is applied:

EX-01#show license summary
License Usage:
License Entitlement Tag Count Status
-----------------------------------------------------------------------------
network-advantage (C9200L-NW-A-24) 1 IN USE
dna-advantage (C9200L-DNA-A-24) 1 IN USE

Because the configuration is automated and policy extended nodes are supposed to apply and enforce SGT policy, this should "just work". Any assistance is greatly appreciated.

Josh

Re: Policy Extended Node not receiving TrustSec SGACLs

jalejand — Wed, 25 Aug 2021 04:27:05 GMT

SGT Rules / SGACLs will only be downloaded on the switch in case SGTs are assigned to endpoints, and it will only download the rules that contain such SGTs as destination, if your SGT mappings are null, then your rules will be also 0.

As you have static mappings on these ports, I imagine these ports already have endpoints attached to them.
Can you please get the following from the Extended Node:

show run

show version

show cts pac
show cts role-based sgt map all

show device-tracking database

Regards

Re: Policy Extended Node not receiving TrustSec SGACLs

Joshua Marks — Wed, 25 Aug 2021 04:53:09 GMT

Hi jalejand, thanks for your response.

The show run is attached. Remaining commands are below.

show version
Cisco IOS XE Software, Version 17.03.03
Cisco IOS Software [Amsterdam], Catalyst L3 Switch Software (CAT9K_LITE_IOSXE), Version 17.3.3, RELEASE SOFTWARE (fc7)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2021 by Cisco Systems, Inc.
Compiled Thu 04-Mar-21 08:48 by mcpre

Cisco IOS-XE software, Copyright (c) 2005-2021 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: IOS-XE ROMMON
BOOTLDR: System Bootstrap, Version 17.5.1r [FC4], RELEASE SOFTWARE (P)

PROD-EX-01 uptime is 1 hour, 38 minutes
Uptime for this control processor is 1 hour, 40 minutes
System returned to ROM by Power Failure or Unknown at 02:57:07 UTC Sat Jul 3 2021
System restarted at 03:00:55 UTC Wed Aug 25 2021
System image file is "flash:cat9k_lite_iosxe.17.03.03.SPA.bin"
Last reload reason: Power Failure or Unknown

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Technology Package License Information:

------------------------------------------------------------------------------
Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------------------
network-advantage Smart License network-advantage
dna-advantage Subscription Smart License dna-advantage

Smart Licensing Status: Registration Not Applicable/Not Applicable

cisco C9200L-24P-4G (ARM64) processor with 523553K/3071K bytes of memory.
Processor board ID xxxxxxxxxx
2 Virtual Ethernet interfaces
28 Gigabit Ethernet interfaces
2048K bytes of non-volatile configuration memory.
1984368K bytes of physical memory.
819200K bytes of Crash Files at crashinfo:.
1941504K bytes of Flash at flash:.

Base Ethernet MAC Address : 74:xxxxxxxxxx:00
Motherboard Assembly Number : xxxxxxxxxx
Motherboard Serial Number : xxxxxxxxxx
Model Revision Number : A0
Motherboard Revision Number : A0
Model Number : C9200L-24P-4G
System Serial Number : xxxxxxxxxx
CLEI Code Number : xxxxxxxxxx

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 28 C9200L-24P-4G 17.03.03 CAT9K_LITE_IOSXE BUNDLE

Configuration register is 0x102

show cts pac
AID: 12C8BDxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxC81DAE9FF
PAC-Info:
PAC-type = Cisco Trustsec
AID: 12C8BDxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxC81DAE9FF
I-ID: xxxxxxxxxx
A-ID-Info: Identity Services Engine
Credential Lifetime: 03:04:58 UTC Tue Nov 23 2021
PAC-Opaque: 000200B8000300010004001012C8BD4B21731585D043636C81DAE9FF0006009C00030100FAB35E4A6548E5C7BDA16B86E51D601C00000013xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxCAC13373AD1D0F5E6BA2111C5D47C1FEAA1ADC9D330800E4C0E65CDE994D59C4FB12A9F066D603AAD5EBBC603CEB762E23060BB2FF31420558F1
Refresh timer is set for 12w4d

show cts role-based sgt-map all
Active IPv4-SGT Bindings Information

IP Address SGT Source
============================================
Active IPv6-SGT Bindings Information

IP Address SGT Source
================================================================

show device-tracking database
Binding Table has 2 entries, 2 dynamic (limit 200000)
Codes: L - Local, S - Static, ND - Neighbor Discovery, ARP - Address Resolution Protocol, DH4 - IPv4 DHCP, DH6 - IPv6 DHCP, PKT - Other Packet, API - API created
Preflevel flags (prlvl):
0001:MAC and LLA match 0002:Orig trunk 0004:Orig access
0008:Orig trusted trunk 0010:Orig trusted access 0020:DHCP assigned
0040:Cga authenticated 0080:Cert authenticated 0100:Statically assigned

Network Layer Address Link Layer Address Interface vlan prlvl age state Time left
ARP 192.168.41.13 70f3.5a80.2ad8 Gi1/0/5 41 0005 7s REACHABLE 303 s
ND FE80::72F3:5AFF:FE80:2AD8 70f3.5a80.2ad8 Gi1/0/5 41 0005 170s REACHABLE 141 s try 0

Re: Policy Extended Node not receiving TrustSec SGACLs

jalejand — Wed, 25 Aug 2021 17:31:34 GMT

Ok, no bindings, CTS config looks good, but you only have one endpoint which seems to be Gi1/0/5 with no SGT config on it:

interface GigabitEthernet1/0/5
switchport access vlan 41
switchport mode access
device-tracking attach-policy IPDT_POLICY
load-interval 30
access-session inherit disable interface-template-sticky
access-session inherit disable autoconf
no macro auto processing
spanning-tree portfast
spanning-tree bpduguard enable
service-policy input DNA-MARKING_IN
service-policy output DNA-dscp#APIC_QOS_Q_OUT
ip nbar protocol-discovery
!

What happens if you assign an SGT to 1/0/5 and check if there is a client on device-tracking for that port?
I would expect to have a new binding on:

show cts role-based sgt-map all

For example:

Edge1#show cts role-based sgt-map vrf Campus all
%IPv6 protocol is not enabled in VRF Campus
Active IPv4-SGT Bindings Information

IP Address SGT Source
============================================
172.19.10.12 4 LOCAL

In this case, I have an SGT mapping on 172.19.10.12, which is on DeviceTracking (ignore the vrf awareness for simplicity, you will just use the global RIB). But I have two interfaces with static SGT enabled and they are in connected state:

Edge1#show pla sof fed sw active sgacl port | i 4

Port             Status     Port-SGT    Trust    Propagate IngressCache EgressCache
-------------------------------------------------------------------------------
Te1/0/4     Enabled        4         No    No           No                No
Te1/0/5 Enabled                4         No                No                 No                No

Edge1#show run int te1/0/4 | se cts
cts manual
policy static sgt 4
no propagate sgt

Edge1#show run int te1/0/5 | se cts
cts manual
policy static sgt 4
no propagate sgt

But onle 1 of them shows in device-tracking

Edge1#show device-tracking data int te1/0/4 | be Network
Network Layer Address Link Layer Address Interface vlan prlvl age state Time left
API 172.19.10.12 d4e8.801f.4876 Te1/0/4 1021 0005 171s REACHABLE 75 s

Edge1#show device-tracking data int te1/0/5
Edge1#

Re: Policy Extended Node not receiving TrustSec SGACLs

Joshua Marks — Wed, 25 Aug 2021 22:32:58 GMT

Hi jalejand,

Gi1/0/4 is an AP port which is why it has no SGT assigned:

Here is the RIB SGT mapping information:

EX-01#show cts role-based sgt-map all
Active IPv4-SGT Bindings Information

IP Address SGT Source
============================================

Here are the device-tracking applied interfaces:

EX-01#show device-tracking policies
Target Type Policy Feature Target range
Gi1/0/1 PORT IPDT_POLICY Device-tracking vlan all
Gi1/0/2 PORT IPDT_POLICY Device-tracking vlan all
Gi1/0/4 PORT IPDT_POLICY Device-tracking vlan all
Gi1/0/5 PORT IPDT_POLICY Device-tracking vlan all
Gi1/0/6 PORT IPDT_POLICY Device-tracking vlan all
Gi1/0/8 PORT IPDT_POLICY Device-tracking vlan all
Gi1/0/13 PORT IPDT_POLICY Device-tracking vlan all
Gi1/0/17 PORT IPDT_POLICY Device-tracking vlan all
Gi1/0/18 PORT IPDT_POLICY Device-tracking vlan all
Gi1/0/19 PORT IPDT_POLICY Device-tracking vlan all
Gi1/0/20 PORT IPDT_POLICY Device-tracking vlan all
Gi1/0/21 PORT IPDT_POLICY Device-tracking vlan all
Gi1/0/22 PORT IPDT_POLICY Device-tracking vlan all
Gi1/1/1 PORT IPDT_POLICY Device-tracking vlan all
Gi1/1/2 PORT IPDT_POLICY Device-tracking vlan all
Gi1/1/3 PORT IPDT_POLICY Device-tracking vlan all
Gi1/1/4 PORT IPDT_POLICY Device-tracking vlan all

See here that only Gi1/0/5 (the AP port) has any device tracking data, none of the data ports with SGTs assigned do:

EX-01#show device-tracking data | be Network
Network Layer Address Link Layer Address Interface vlan prlvl age state Time left
ARP 192.168.41.13 70f3.5a80.2ad8 Gi1/0/5 41 0005 11s REACHABLE 300 s
ND FE80::72F3:5AFF:FE80:2AD8 70f3.5a80.2ad8 Gi1/0/5 41 0005 122s REACHABLE 182 s try 0

Here are the port SGTs:

EX-01#show platform software fed switch active sgacl port | in 4
Gi1/0/3 Enabled 4 No No No No
Gi1/0/4 Enabled 0 No No No No
Gi1/0/7 Enabled 4 No No No No
Gi1/0/9 Enabled 4 No No No No
Gi1/0/12 Enabled 4 No No No No
Gi1/0/14 Enabled 4 No No No No
Gi1/0/15 Enabled 4 No No No No
Gi1/0/16 Enabled 4 No No No No
Gi1/0/24 Enabled 8000 Yes Yes No No
Gi1/1/4 Enabled 0 No No No No

It looks to me that the switch isn't aware of endpoint IP addresses and therefore isn't mapping the SGT to the endpoint...

Any thoughts?

Re: Policy Extended Node not receiving TrustSec SGACLs

jalejand — Thu, 26 Aug 2021 00:45:35 GMT

Exactly, if no DT mapping exists, no SGT map exists either. For now, we can focus on allowing DT to discover the rest of devices.
Are your endpoints in 1/0/1, 2, 3 etc getting IP via DHCP? If so try enabling DHCP snooping manually so DT can have a table based on DHCP information (I would call this a workaround, ideally you should have an ARP entry on the DT table of the PEN, but who knows how silent are the devices connected there)

conf t

no ip dhcp snooping information option

ip dhcp snooping

ip dhcp snooping vlan (client vlan)

int po 1

ip dhcp snooping trust

Then bounce a client port to force it to get an IP again, and if it does (implying that the DHCP snoop config didn't break DHCP as a possibility), after that, check if the client is shown on DT and check its SGT binding.

I'll test this myself on my lab and get back to you 🙂

Re: Policy Extended Node not receiving TrustSec SGACLs

Joshua Marks — Thu, 26 Aug 2021 10:15:29 GMT

Hi jalejand,

Your change suggestion appears to have worked well as I can now see the SGT mappings:

EX-01#show cts role-based sgt-map all
Active IPv4-SGT Bindings Information

IP Address SGT Source
============================================

conf t

no ip dhcp snooping information option

ip dhcp snooping

ip dhcp snooping vlan 41,42,48,50,51,55,56

int po 1

ip dhcp snooping trust

###After a minute or so###

show cts role-based sgt-map all
Active IPv4-SGT Bindings Information

IP Address SGT Source
============================================
192.168.50.25 4 LOCAL
192.168.50.26 4 LOCAL
192.168.50.27 4 LOCAL
192.168.50.28 4 LOCAL
192.168.50.29 4 LOCAL
192.168.50.30 4 LOCAL
192.168.50.31 4 LOCAL

Now the switch knows that it has clients with SGTs applied, it has requested the SGACLs from ISE:

show cts role-based permissions
IPv4 Role-based permissions default:
Permit IP-00
IPv4 Role-based permissions from group 3:IoT to group 4:Employees:
Deny IP-00
IPv4 Role-based permissions from group 4:Employees to group 4:Employees:
Allow_VoIP-01
IPv4 Role-based permissions from group 10:Printers to group 4:Employees:
Permit IP-00
IPv4 Role-based permissions from group 11:PSK_Devices to group 4:Employees:
Deny IP-00
IPv4 Role-based permissions from group 255:Quarantined_Systems to group 4:Employees:
Deny IP-00
RBACL Monitor All for Dynamic Policies : FALSE
RBACL Monitor All for Configured Policies : FALSE

Thanks for the help. Why would the automated config pushed by DNAC not include DHCP snooping? Is this normal for policy extended nodes?

Josh

Re: Policy Extended Node not receiving TrustSec SGACLs

jalejand — Thu, 26 Aug 2021 16:26:13 GMT

DNAC doesn't push DHCP snooping by default, there are some enhancement bug for this but are not yet customer visibile.

I would say that for now is normal, I don't have the information about when or how DHCP snooping will be implemented on PENs (or if such enhancement will be resolved as a valid enhancement). However it does two things.

1) It adds the security benefits of DHCP snooping to the extended access layer, which is currently not implemented

2) By snooping DHCP packets which are not processed by CPU by default, device-tracking gains the ability of a more "trusted" source of information which are DHCP packets + DHCP binding table instead of only relying on ARP to get such entries.

Re: Policy Extended Node not receiving TrustSec SGACLs

Joshua Marks — Fri, 27 Aug 2021 00:24:50 GMT

Hi jalejand,

This is my first experience with a policy extended node, but my understanding was that these devices should be able to apply, track and enforce SGTs/contracts between endpoints. If this was not happening on our switch until we enabled DHCP snooping, does that mean the policy extended nodes in general do not enforce TrustSec right now, or is it just something off with our switch?

Thanks again for your great assistance,

Josh