topic Re: Cisco ISE / CTS switch Issues in Network Access Control

Cisco ISE / CTS switch Issues

Jay233 — Thu, 11 Jun 2020 11:32:50 GMT

Recently noticed a strange issue with a few switches in our network.

Using SGT/CTS with ISE 2.4.

Switches are 9200 series, working ok until several switches started to show an error with CTS server info list I.E. marking the ISE servers as down?

2 switch outputs below (sw1 not working, sw2 working). The switches have the same config and in the same location, able to refresh env data and also PAC files on both switches without error.

The only difference I can see is info output for TAG 0:Unknown

The working switch shows "status alive" with auto-test=false?

The none working switch shows "status dead" with auto-test=true?

Can anyone explain this auto-test feature please.

Output for sw1 (error switch):

SW1#sh cts environment-data
CTS Environment Data
====================
Current state = COMPLETE
Last status = Successful
Local Device SGT:
  SGT tag = 2-01:Infrastructure
Server List Info:
Installed list: CTSServerList1-0004, 2 server(s):
*Server: 10.X.X.X, port 1812, A-ID C5E76EXXXXXXXXXXXX
          Status = DEAD
          auto-test = TRUE, keywrap-enable = FALSE, idle-time = 60 mins, deadtime = 20 secs
*Server: 10.X.X.X, port 1812, A-ID C5E76EXXXXXXXXXXXX
          Status = DEAD
          auto-test = TRUE, keywrap-enable = FALSE, idle-time = 60 mins, deadtime = 20 secs
Security Group Name Table:
    0-01:Unknown
    2-01:Infrastructure
    3-00:Network_Services
    4-00:Employees

Output for sw2 (working switch):

SW2#sh cts environment-data
CTS Environment Data
====================
Current state = COMPLETE
Last status = Successful
Local Device SGT:
  SGT tag = 2-01:Infrastructure
Server List Info:
Installed list: CTSServerList1-0004, 2 server(s):
  Server: 10.X.X.X, port 1812, A-ID C5E76EXXXXXXXXXXXX
          Status = ALIVE
          auto-test = FALSE, keywrap-enable = FALSE, idle-time = 60 mins, deadtime = 20 secs
  Server: 10.X.X.X, port 1812, A-ID C5E76EXXXXXXXXXXXX
          Status = ALIVE
          auto-test = FALSE, keywrap-enable = FALSE, idle-time = 60 mins, deadtime = 20 secs
Security Group Name Table:
    0-03:Unknown
    2-01:Infrastructure
    3-00:Network_Services
    4-00:Employees

Appreciate any help on this, not sure if its a bug or not.

Cheers,

Re: Cisco ISE / CTS switch Issues

Jay233 — Thu, 11 Jun 2020 11:48:42 GMT

Quick update: After a reboot on sw1# (No config change at all) the switch is now marking the ISE servers as "alive" when I do sw1#show cts env data?

What is causing the switch to previously report the severs as "dead"?

Reboot and the issue disappears but for how long is the question.

Could this be an auth time type loop issue?

My radius servers are local PSN's while my CTS AAA is using the PAN and SPAN, I dont believe this makes any difference but should the ISE servers be the same targets for radius and CTS trusted AAA?

If anyone has a known working CTS switch config and willing to post that would be great.

Thanks,

Re: Cisco ISE / CTS switch Issues

Greg Gibbs — Thu, 11 Jun 2020 23:27:24 GMT

The switch will mark the RADIUS servers as DEAD if it does not receive a response from the server within it's configured dead-criteria timers. See Demystifying RADIUS Server Configurations for more information. This could be due to either a misconfiguration or a network issue.

As per your comment "my CTS AAA is using the PAN and SPAN", unless you have the PSN role enabled on the PAN nodes (not recommended in a distributed environment), this will not work. The CTS AAA servers should be your PSNs.

See Group Based Policy Fundamentals for more info and example configurations.

You might also have a look at the TrustSec lab examples available on LabMinutes

Re: Cisco ISE / CTS switch Issues

Damien Miller — Fri, 12 Jun 2020 01:22:04 GMT

I can't speak for all switch types and software versions, but the ones I am familiar with have auto test enabled by default when you enable CTS and you cannot configure the user. It won't show up in the config, but you can see it with a "show run all | inc cts". While it doesn't appear to be related to your issue, the test comes through with the user as "CTS-Test-Server", and if you strip down your policy sets and disable the defaults, you can sometimes have to define this user in the local identities store to get a radius response.

cts server test all enable
cts server test all idle-time 60
cts server test all deadtime 20

Out of curiosity, what version IOS and platform were you having this issue on? I've had good results with TrustSec on 16.9.5 with 3k/9k platforms.

Re: Cisco ISE / CTS switch Issues

Jay233 — Fri, 12 Jun 2020 11:30:53 GMT

Hi Damien,

Many thanks for the reply.

As requested:

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- --------
* 1 52 C9200L-48PXG-4X 16.12.1 CAT9K_LITE_IOSXE INSTALL
2 52 C9200L-48P-4X 16.12.1 CAT9K_LITE_IOSXE INSTALL

-----------------------------------------------

So do I need a positive ISE (AAA server) response to "CTS-Test-Server" username for the auto server check to work?

I dont see "CTS-Test-Server" username but do see username = #CTSREQUEST# when I debug CTS env data, is this correct?

SW1#debug cts environment-data all
All cts environment data debugging is on

SW1#cts refresh environment-data
Environment data download in progress
SW1#
May 21 08:52:24.467: CTS env-data: Force environment-data refresh
May 21 08:52:24.467: CTS env-data: download transport-type = CTS_TRANSPORT_IP_UDP
May 21 08:52:24.467: cts_env_data COMPLETE: during state env_data_complete, got event 0(env_data_request)
May 21 08:52:24.467: @@@ cts_env_data COMPLETE: env_data_complete -> env_data_waiting_rsp
May 21 08:52:24.467: env_data_waiting_rsp_enter: state = WAITING_RESPONSE
May 21 08:52:24.467: cts_aaa_is_fragmented: (CTS env-data SM)NOT-FRAG attr_q(0)
May 21 08:52:24.467: env_data_request_action: state = WAITING_RESPONSE
May 21 08:52:24.467: cts_env_data_is_complete: FALSE, req(x0), rec(x0)
May 21 08:52:24.467: FALSE, req(x0), rec(x0), expect(x81), complete1(x85), complete2(xB5), complete3(x1485), complete4(x18085)complete5(xC0085), complete6(x600085)
May 21 08:52:24.467: env_data_request_action: state = WAITING_RESPONSE, received = 0x0 request = 0x0

May 21 08:52:24.467: cts_env_data_aaa_req_setup : aaa_id = 11
May 21 08:52:24.467: cts_aaa_req_setup: (CTS env-data SM)Private group appears DEAD, attempt public group
May 21 08:52:24.467: cts_aaa_req_setup: (CTS env-data SM)CTS_TRANSPORT_IP_UDP
May 21 08:52:24.467: cts_aaa_req_setup: (CTS env-data SM)AAA req(x4EEB07D8)
May 21 08:52:24.468: cts_aaa_attr_add: AAA req(0x4EEB07D8)
May 21 08:52:24.468: username = #CTSREQUEST#
Cheers,

Re: Cisco ISE / CTS switch Issues

Jay233 — Fri, 12 Jun 2020 12:33:38 GMT

Hi Greg,

Thank you for the reply, confused a little on "configured dead-criteria timers" as I assumed this was for general radius communications? When I do a #show aaa servers all severs are showing as up? So are you saying that the CTS aaa server function is using the dead-criteria? Is this the problem that I'm targeting different server IP's? (PSN and direct to PAN)?

What is the below CTS dead server group/global time based on?

CTS Server Radius Load Balance = DISABLED
Server Group Deadtime = 20 secs (default)
Global Server Liveness Automated Test Deadtime = 20 secs
Global Server Liveness Automated Test Idle Time = 60 mins
Global Server Liveness Automated Test = ENABLED (default)

Another suggestion I have seen is to separate general radius and CTS communications using 2 groups and different ports (1812, 1645)? Is this a valid solution?

"Configure RADIUS server for TrustSec but use different ports to avoid conflict"

What conflicts do we see?

Cheers,

Re: Cisco ISE / CTS switch Issues

Greg Gibbs — Sun, 14 Jun 2020 23:04:51 GMT

Another suggestion I have seen is to separate general radius and CTS communications using 2 groups and different ports (1812, 1645)? Is this a valid solution?

This is a common approach to workaround the known behaviour with RADIUS Accounting referenced in CSCtw56571

So do I need a positive ISE (AAA server) response to "CTS-Test-Server" username for the auto server check to work?

As long as the switch receives a response to the test keepalives (accept or reject) it knows the RADIUS server is alive. You can also hide the fail logs using a Collection Filter.

The debug logs appear to indicate that the environment data request is not getting a response. This could result in the CTS server being marked DEAD. If you're pointing to the PAN for the CTS server, that is likely at least part of the problem. The CTS server is still your RADIUS server (plus the PAC that is negotiated), so it must be using your PSNs.

Re: Cisco ISE / CTS switch Issues

Jay233 — Mon, 15 Jun 2020 08:05:01 GMT

Cheers Greg really helpful.

Re: Cisco ISE / CTS switch Issues

Jay233 — Mon, 15 Jun 2020 11:02:53 GMT

Hi Damien,

Out of curiosity I did a sh run | inc cts on one of the edge switches (9200-L).

Did not see the user name of "CTS-Test-Server", in fact don't see any user name for CTS?

SW1# sh run all | inc cts
cts server deadtime 20
cts server test all enable
cts server test all idle-time 60
cts server test all deadtime 20
no cts server key-wrap enable
cts authorization list iselist
no cts logging verbose
no cts sg-epg translation
no cts sxp enable
cts sxp retry period 120
cts sxp reconciliation period 120
no cts sxp log binding-changes
cts sxp mapping network-map 0
cts sxp speaker hold-time 120
cts sxp listener hold-time 90 180
cts sxp node-id 0
no cts sxp filter-enable
ipv6 redirects
ip redirects
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts manual
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts manual
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ip redirects
ipv6 redirects
ip redirects
ipv6 redirects
cts role-based sgt-map sgt 2
cts role-based enforcement
cts role-based enforcement vlan-list 1-4094