06-11-2020 04:32 AM
Recently noticed a strange issue with a few switches in our network.
Using SGT/CTS with ISE 2.4.
Switches are 9200 series, working ok until several switches started to show an error with CTS server info list I.E. marking the ISE servers as down?
2 switch outputs below (sw1 not working, sw2 working). The switches have the same config and in the same location, able to refresh env data and also PAC files on both switches without error.
The only difference I can see is info output for TAG 0:Unknown
The working switch shows "status alive" with auto-test=false?
The none working switch shows "status dead" with auto-test=true?
Can anyone explain this auto-test feature please.
Output for sw1 (error switch):
Output for sw2 (working switch):
06-11-2020 04:48 AM
Quick update: After a reboot on sw1# (No config change at all) the switch is now marking the ISE servers as "alive" when I do sw1#show cts env data?
What is causing the switch to previously report the severs as "dead"?
Reboot and the issue disappears but for how long is the question.
Could this be an auth time type loop issue?
My radius servers are local PSN's while my CTS AAA is using the PAN and SPAN, I dont believe this makes any difference but should the ISE servers be the same targets for radius and CTS trusted AAA?
If anyone has a known working CTS switch config and willing to post that would be great.
Thanks,
06-11-2020 04:27 PM
The switch will mark the RADIUS servers as DEAD if it does not receive a response from the server within it's configured dead-criteria timers. See Demystifying RADIUS Server Configurations for more information. This could be due to either a misconfiguration or a network issue.
As per your comment "my CTS AAA is using the PAN and SPAN", unless you have the PSN role enabled on the PAN nodes (not recommended in a distributed environment), this will not work. The CTS AAA servers should be your PSNs.
See Group Based Policy Fundamentals for more info and example configurations.
You might also have a look at the TrustSec lab examples available on LabMinutes
06-12-2020 05:33 AM
Hi Greg,
Thank you for the reply, confused a little on "configured dead-criteria timers" as I assumed this was for general radius communications? When I do a #show aaa servers all severs are showing as up? So are you saying that the CTS aaa server function is using the dead-criteria? Is this the problem that I'm targeting different server IP's? (PSN and direct to PAN)?
What is the below CTS dead server group/global time based on?
CTS Server Radius Load Balance = DISABLED
Server Group Deadtime = 20 secs (default)
Global Server Liveness Automated Test Deadtime = 20 secs
Global Server Liveness Automated Test Idle Time = 60 mins
Global Server Liveness Automated Test = ENABLED (default)
Another suggestion I have seen is to separate general radius and CTS communications using 2 groups and different ports (1812, 1645)? Is this a valid solution?
"Configure RADIUS server for TrustSec but use different ports to avoid conflict"
What conflicts do we see?
Cheers,
06-11-2020 06:22 PM
06-12-2020 04:30 AM
Hi Damien,
Many thanks for the reply.
As requested:
Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- --------
* 1 52 C9200L-48PXG-4X 16.12.1 CAT9K_LITE_IOSXE INSTALL
2 52 C9200L-48P-4X 16.12.1 CAT9K_LITE_IOSXE INSTALL
-----------------------------------------------
So do I need a positive ISE (AAA server) response to "CTS-Test-Server" username for the auto server check to work?
I dont see "CTS-Test-Server" username but do see username = #CTSREQUEST# when I debug CTS env data, is this correct?
SW1#debug cts environment-data all
All cts environment data debugging is on
SW1#cts refresh environment-data
Environment data download in progress
SW1#
May 21 08:52:24.467: CTS env-data: Force environment-data refresh
May 21 08:52:24.467: CTS env-data: download transport-type = CTS_TRANSPORT_IP_UDP
May 21 08:52:24.467: cts_env_data COMPLETE: during state env_data_complete, got event 0(env_data_request)
May 21 08:52:24.467: @@@ cts_env_data COMPLETE: env_data_complete -> env_data_waiting_rsp
May 21 08:52:24.467: env_data_waiting_rsp_enter: state = WAITING_RESPONSE
May 21 08:52:24.467: cts_aaa_is_fragmented: (CTS env-data SM)NOT-FRAG attr_q(0)
May 21 08:52:24.467: env_data_request_action: state = WAITING_RESPONSE
May 21 08:52:24.467: cts_env_data_is_complete: FALSE, req(x0), rec(x0)
May 21 08:52:24.467: FALSE, req(x0), rec(x0), expect(x81), complete1(x85), complete2(xB5), complete3(x1485), complete4(x18085)complete5(xC0085), complete6(x600085)
May 21 08:52:24.467: env_data_request_action: state = WAITING_RESPONSE, received = 0x0 request = 0x0
May 21 08:52:24.467: cts_env_data_aaa_req_setup : aaa_id = 11
May 21 08:52:24.467: cts_aaa_req_setup: (CTS env-data SM)Private group appears DEAD, attempt public group
May 21 08:52:24.467: cts_aaa_req_setup: (CTS env-data SM)CTS_TRANSPORT_IP_UDP
May 21 08:52:24.467: cts_aaa_req_setup: (CTS env-data SM)AAA req(x4EEB07D8)
May 21 08:52:24.468: cts_aaa_attr_add: AAA req(0x4EEB07D8)
May 21 08:52:24.468: username = #CTSREQUEST#
Cheers,
06-14-2020 04:04 PM
Another suggestion I have seen is to separate general radius and CTS communications using 2 groups and different ports (1812, 1645)? Is this a valid solution?
This is a common approach to workaround the known behaviour with RADIUS Accounting referenced in CSCtw56571
So do I need a positive ISE (AAA server) response to "CTS-Test-Server" username for the auto server check to work?
As long as the switch receives a response to the test keepalives (accept or reject) it knows the RADIUS server is alive. You can also hide the fail logs using a Collection Filter.
The debug logs appear to indicate that the environment data request is not getting a response. This could result in the CTS server being marked DEAD. If you're pointing to the PAN for the CTS server, that is likely at least part of the problem. The CTS server is still your RADIUS server (plus the PAC that is negotiated), so it must be using your PSNs.
06-15-2020 01:05 AM
Cheers Greg really helpful.
06-15-2020 04:02 AM
Hi Damien,
Out of curiosity I did a sh run | inc cts on one of the edge switches (9200-L).
Did not see the user name of "CTS-Test-Server", in fact don't see any user name for CTS?
SW1# sh run all | inc cts
cts server deadtime 20
cts server test all enable
cts server test all idle-time 60
cts server test all deadtime 20
no cts server key-wrap enable
cts authorization list iselist
no cts logging verbose
no cts sg-epg translation
no cts sxp enable
cts sxp retry period 120
cts sxp reconciliation period 120
no cts sxp log binding-changes
cts sxp mapping network-map 0
cts sxp speaker hold-time 120
cts sxp listener hold-time 90 180
cts sxp node-id 0
no cts sxp filter-enable
ipv6 redirects
ip redirects
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts manual
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts manual
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ipv6 redirects
cts role-based enforcement
ip redirects
ipv6 redirects
ip redirects
ipv6 redirects
cts role-based sgt-map sgt 2
cts role-based enforcement
cts role-based enforcement vlan-list 1-4094
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide