Trustsec POC strange behavior, removes dynamic ACL

[joeharb]

I am doing a POC for trustsec SGT and SGACL's and noticing very odd behavior.  I am testing with a simple deny icmp SGACL, and have my machine as a static mapping and a device on the switch receiving a SGT of 5.  The SGTACL is applied from my security group (tag 3) to the SGT 5.  Looking at the switch the DENY_ICMP is not present but ISE doesn't think anything is needed to be pushed.  I do a cts refresh policy and after a few minutes the acl is present and I can't ping the destination IP address.  After some time (not sure the exact time) but I see an event from the interface of the device I am testing with and then the ACL is deleted from the switch, the only way to get it back is to do a cts refresh again.

Here are the log entries: (Gi1/0/4) is the device that is being tagged with SGT 5.

Jun 8 21:24:52.154: CTS-rcl-server-events:Receieved idb for the session: [Gi1/0/4]
Jun 8 21:24:52.154: CTS-rcl-server-events:Vlan ID to be sent to rbm is 10, mac=3448.ed72.9d49
Jun 8 21:24:52.154: CTS-rcl-server-events:eac bindings message handled successfully: add_ip 0.0.0.0 remove_ip 10.33.0.112 ipv6_add_count 0 ipv6_remove_count 0
Jun 8 21:24:52.154: CTS-ha-binding-event:CTS IP-SGT binding incremental sync entry
Jun 8 21:24:52.155: CTS-ha-binding-event: CTS IP-SGT binding incremental sync not allowed
Jun 8 21:24:52.155: CTS-rcl-server-events:IPSGT binding sync failed
Jun 8 21:24:52.155: CTS-SXP-MDB:sxp_export_ipsgt_change_enhanced 10.33.0.112/32 del 0 sgt 5 source 6
Jun 8 21:24:52.156: CTS-l3s:cts_l3s_ipv4_acl_modified(acl DENY_ICMP-01, type 21, deleted=0)
Jun 8 21:24:52.156: CTS-l3s:cts_l3s_ipv4_acl_modified(acl DENY_ICMP-01, type 21, deleted=1)
Jun 8 21:24:52.156: CTS authz entry ev (Unknown-5): Removed session hdl(DF00000E) from entry.
Jun 8 21:24:52.157: CTS-core-ha-ev:cts_coa_end_status_sync: status(INCOMPLETE), context(0), forced(1)
Jun 8 21:24:52.157: CTS-core-ha-ev:
cts_coa_end_status_sync: contextp is NULL, Return False

Please advise,

Thanks,

Joe

[joeharb]

Update:

After posting I wanted to make sure the ACL was still gone but it is now back and looking at the logs there was another event on the port that triggered an update:

Jun 9 16:42:19.025: CTS-rcl-server-events:Receieved idb for the session: [Gi1/0/4]
Jun 9 16:42:19.025: CTS-rcl-server-events:Vlan retrieved using dot1x switch API is 10, mac=3448.ed72.9d49
Jun 9 16:42:19.026: CTS-rcl-server-events:Marshalling done, sending msg to BINOS
Jun 9 16:42:19.026: CTS-rcl-server-events:Update vlan tdl message is sent successfully to EPM Plugin in SMD, vlan:10
Jun 9 16:42:19.026: CTS-rcl-server-events:eac bindings message handled successfully: add_ip 10.33.0.112 remove_ip 0.0.0.0 ipv6_add_count 0 ipv6_remove_count 0
Jun 9 16:42:19.026: CTS-ha-binding-event:CTS IP-SGT binding incremental sync entry
Jun 9 16:42:19.026: CTS-ha-binding-event: CTS IP-SGT binding incremental sync not allowed
Jun 9 16:42:19.026: CTS-rcl-server-events:IPSGT binding sync failed
Jun 9 16:42:19.026: CTS-SXP-MDB:sxp_export_ipsgt_change_enhanced 10.33.0.112/32 add 0 sgt 5 source 6
Jun 9 16:42:19.027: cts_aaa_is_fragmented: (Unknown-5)NOT-FRAG attr_q(0)
Jun 9 16:42:19.027: cts_aaa_req_setup: (Unknown-5)private server(s) exist
Jun 9 16:42:19.027: cts_aaa_req_setup: (Unknown-5)Using private server group
Jun 9 16:42:19.027: cts_aaa_req_setup: (Unknown-5)CTS_TRANSPORT_IP_UDP
Jun 9 16:42:19.027: cts_aaa_req_setup: (Unknown-5)AAA req(x7F49CEBCD8F0)
Jun 9 16:42:19.027: cts_aaa_attr_add: AAA req(0x7F49CEBCD8F0)
Jun 9 16:42:19.027: username = #CTSREQUEST#
Jun 9 16:42:19.027: password = *****
Jun 9 16:42:19.027: AAA Context Add Attribute: (Unknown-5)attr(0005-v4v6)
Jun 9 16:42:19.027: cts-rbacl-source-list = 0005-v4v6
Jun 9 16:42:19.027: cts_aaa_attr_add: AAA req(0x7F49CEBCD8F0)
Jun 9 16:42:19.027: AAA Context Add Attribute: (Unknown-5)attr(monitor)
Jun 9 16:42:19.027: cts-device-capability = monitor
Jun 9 16:42:19.027: cts_aaa_req_send: AAA req(0x7F49CEBCD8F0) successfully sent to AAA.
Jun 9 16:42:19.073: cts_aaa_callback: (Unknown-5)AAA req(0x7F49CEBCD8F0) response success
Jun 9 16:42:19.073: AAA CTX FRAG CLEAN: (Unknown-5)attr(0005-v4v6)
Jun 9 16:42:19.073: AAA CTX FRAG CLEAN: (Unknown-5)attr(monitor)
Jun 9 16:42:19.073: AAA attr: Unknown type (450).
Jun 9 16:42:19.073: AAA attr: Unknown type (274).
Jun 9 16:42:19.073: AAA attr: src-dst-rbacl = 0003-00-00-0005-05-00-v4-DENY_ICMP-1.
Jun 9 16:42:19.073: cts_aaa_is_fragmented: (Unknown-5)NOT-FRAG attr_q(0)
Jun 9 16:42:19.073: AAA attr: rbacl-monitor-all = OFF.
Jun 9 16:42:19.073: AAA attr: authorization-expiry = 86400.
Jun 9 16:42:19.073: cts_aaa_is_fragmented: (Unknown-5)NOT-FRAG attr_q(0)
Jun 9 16:42:19.073: cts_aaa_is_fragmented: (Unknown-5)NOT-FRAG attr_q(0)
Jun 9 16:42:19.073: cts_aaa_req_setup: (Unknown-5)private server(s) exist
Jun 9 16:42:19.073: cts_aaa_req_setup: (Unknown-5)Using private server group
Jun 9 16:42:19.073: cts_aaa_req_setup: (Unknown-5)CTS_TRANSPORT_IP_UDP
Jun 9 16:42:19.074: cts_aaa_req_setup: (Unknown-5)AAA req(x7F49CE349930)
Jun 9 16:42:19.074: cts_aaa_attr_add: AAA req(0x7F49CE349930)
Jun 9 16:42:19.074: username = #CTSREQUEST#
Jun 9 16:42:19.074: password = *****
Jun 9 16:42:19.074: AAA Context Add Attribute: (Unknown-5)attr(DENY_ICMP)
Jun 9 16:42:19.074: cts-rbacl = DENY_ICMP
Jun 9 16:42:19.074: cts_aaa_req_send: AAA req(0x7F49CE349930) successfully sent to AAA.
Jun 9 16:42:19.102: cts_aaa_callback: (Unknown-5)AAA req(0x7F49CE349930) response success
Jun 9 16:42:19.102: AAA CTX FRAG CLEAN: (Unknown-5)attr(DENY_ICMP)
Jun 9 16:42:19.102: AAA attr: Unknown type (450).
Jun 9 16:42:19.102: AAA attr: Unknown type (274).
Jun 9 16:42:19.102: AAA attr: rbacl = DENY_ICMP-1.
Jun 9 16:42:19.103: AAA attr: rbacl-ace = deny icmp.

Is there some type of timer that is causing this?

Thanks,

Joe

[andrewswanson]

So

one device on the switch has a static SGT assignment (SGT 3)
one device on the switch has an SXP SGT assignment (SGT 5)

Does the SGACL download issue still happen if both devices have static SGT assignments?

What is the output of "show cts role-based sgt-map all"? Can you see both tagged IP addresses? (I assume SGT3 will be CLI and SGT5 SXP). If the SGT 5 device isn't listed, what is the output of "show cts sxp connections"

hth
Andy

[joeharb]

Yes I see everything correctly but it seems like every 10 minutes the interface (SGT 5) will trigger some type of update and the ACL will be deleted and the device is no longer in table:
Here is the event:
Jun 9 19:07:10.623: CTS-rcl-server-events:Receieved idb for the session: [Gi1/0/8]
Jun 9 19:07:10.623: CTS-rcl-server-events:Vlan ID to be sent to rbm is 10, mac=a029.199f.e04f
Jun 9 19:07:10.623: CTS-rcl-server-events:eac bindings message handled successfully: add_ip 0.0.0.0 remove_ip 10.33.0.106 ipv6_add_count 0 ipv6_remove_count 0
Jun 9 19:07:10.624: CTS-ha-binding-event:CTS IP-SGT binding incremental sync entry
Jun 9 19:07:10.624: CTS-ha-binding-event: CTS IP-SGT binding incremental sync not allowed
Jun 9 19:07:10.624: CTS-rcl-server-events:IPSGT binding sync failed
Jun 9 19:07:10.624: CTS-SXP-MDB:sxp_export_ipsgt_change_enhanced 10.33.0.106/32 del 0 sgt 5 source 6
Jun 9 19:07:10.625: CTS-l3s:cts_l3s_ipv4_acl_modified(acl DENY_ICMP-01, type 21, deleted=0)
Jun 9 19:07:10.625: CTS-l3s:cts_l3s_ipv4_acl_modified(acl DENY_ICMP-01, type 21, deleted=1)
Jun 9 19:07:10.626: CTS authz entry ev (Unknown-5): Removed session hdl(C000024) from entry.
Jun 9 19:07:10.627: CTS-core-ha-ev:cts_coa_end_status_sync: status(INCOMPLETE), context(0), forced(1)
Jun 9 19:07:10.627: CTS-core-ha-ev:

[andrewswanson]

Do the clients appear ok in the ip dhcp snooping binding (what is the lease time?) and ip device tracking tables?

[joeharb]

Lease time is 21 days
show device-tracking database
Binding Table has 14 entries, 14 dynamic (limit 100000)
Codes: L - Local, S - Static, ND - Neighbor Discovery, ARP - Address Resolution Protocol, DH4 - IPv4 DHCP, DH6 - IPv6 DHCP, PKT - Other Packet, API - API created
Preflevel flags (prlvl):
0001:MAC and LLA match 0002:Orig trunk 0004:Orig access
0008:Orig trusted trunk 0010:Orig trusted access 0020:DHCP assigned
0040:Cga authenticated 0080:Cert authenticated 0100:Statically assigned


Network Layer Address Link Layer Address Interface vlan prlvl age state Time left
ARP 10.33.1.112 848a.8d68.a176 Gi1/0/10 20 0005 216mn STALE 77653 s
ARP 10.33.1.111 848a.8d68.86ee Gi1/0/12 20 0005 215mn STALE 74802 s
ARP 10.33.1.108 001f.9e25.93e9 Gi1/0/9 20 0005 9mn STALE 87804 s
ARP 10.33.1.107 848a.8d68.a349 Gi1/0/8 20 0005 212mn STALE 78060 s
ARP 10.33.1.106 848a.8d68.a7cb Gi1/0/18 20 0005 102mn STALE 84727 s
ARP 10.33.1.105 00eb.d5cd.982f Gi1/0/4 20 0005 63mn STALE 87276 s
ARP 10.33.1.103 6c41.0e5f.4ef0 Gi1/0/7 20 0005 53mn STALE 86711 s
ARP 10.33.1.100 848a.8d68.a17b Gi1/0/5 20 0005 24mn STALE 88754 s
ARP 10.33.0.148 488b.0a4e.9050 Gi1/0/15 10 0005 2s REACHABLE 308 s
ARP 10.33.0.138 000b.9423.5d96 Gi1/0/19 10 0005 181mn STALE 80047 s
ARP 10.33.0.117 908d.6e27.5850 Gi1/0/12 10 0005 1332mn STALE 7979 s
ARP 10.33.0.112 3448.ed72.9d49 Gi1/0/4 10 0005 220mn STALE 76667 s
ARP 10.33.0.106 a029.199f.e04f Gi1/0/8 10 0005 20mn STALE 88992 s
ARP 10.33.0.101 f430.b972.1eca Gi1/0/16 10 0005 145s REACHABLE 170 s

[joeharb]

 

We are using EAP-TLS for the user authentication, looking at ISE could this be an issue?

[joeharb · Accepted solution]

I am pretty sure the issue I was having was due to a bug CSCvh70725.  I created a new device tracking policy as is recommended and have not had the issue since.  The log files were very similar to the bug.

Hope this helps others,

Joe