topic Re: Problem with TrustSec ISE in Network Access Control

Problem with TrustSec ISE

Niklas.D — Fri, 09 Jul 2021 09:17:38 GMT

So i am trying implement trustsec at the company.

I have started by setting up ISE 3.0 Patch 1,2 and a singel switch.

I used this guide:
https://community.cisco.com/t5/security-documents/trustsec-troubleshooting-guide/ta-p/3647576#toc-hId-1865711122
And a few Youtube videos by Kataherine https:/ /www.youtube .com/watch?v=j2i7U5StOYE

Jul 9 11:00:26.137 MET-DST: %RADIUS-4-RADIUS_ALIVE: RADIUS server 10.0.4.174:1812,1813 is being marked alive.
Jul 9 11:00:36.177 MET-DST: %RADIUS-4-RADIUS_ALIVE: RADIUS server 10.0.5.9:1812,1813 is being marked alive.
Jul 9 11:00:40.838 MET-DST: %CTS-3-PAC_PROVI_FAIL: PAC Provisioning failed for 10.0.4.174
Jul 9 11:01:16.087 MET-DST: %CTS-6-ENV_DATA_WAIT_RESP_STATE: Environment Data Download in wait response state
Jul 9 11:01:26.154 MET-DST: %RADIUS-4-RADIUS_DEAD: RADIUS server 10.0.4.174:1812,1813 is not responding.
Jul 9 11:01:36.262 MET-DST: %RADIUS-4-RADIUS_DEAD: RADIUS server 10.0.5.9:1812,1813 is not responding.
Jul 9 11:02:16.093 MET-DST: %CTS-6-ENV_DATA_START_STATE: Environment Data Download in start state
Jul 9 11:02:16.093 MET-DST: %CTS-6-ENV_DATA_WAIT_RESP_STATE: Environment Data Download in wait response state
Jul 9 11:02:16.093 MET-DST: %CTS-3-AAA_NO_RADIUS_SERVER: No RADIUS servers available for CTS AAA request for CTS env-data SM
Jul 9 11:02:16.680 MET-DST: %RADIUS-3-ALLDEADSERVER: Group ISE: No active radius servers found. Id 2.
Jul 9 11:02:25.851 MET-DST: %CTS-3-PAC_PROVI_FAIL: PAC Provisioning failed for 10.0.4.174
Jul 9 11:02:26.160 MET-DST: %RADIUS-6-SERVERALIVE: Group ISE: Radius server 10.0.4.174:1812,1813 is responding again (previously dead).
Jul 9 11:02:26.160 MET-DST: %RADIUS-4-RADIUS_ALIVE: RADIUS server 10.0.4.174:1812,1813 is being marked alive.
Jul 9 11:02:36.268 MET-DST: %RADIUS-4-RADIUS_ALIVE: RADIUS server 10.0.5.9:1812,1813 is being marked alive.
Jul 9 11:03:16.099 MET-DST: %CTS-6-ENV_DATA_WAIT_RESP_STATE: Environment Data Download in wait response state
Jul 9 11:03:26.203 MET-DST: %RADIUS-4-RADIUS_DEAD: RADIUS server 10.0.4.174:1812,1813 is not responding.
Jul 9 11:03:36.243 MET-DST: %RADIUS-4-RADIUS_DEAD: RADIUS server 10.0.5.9:1812,1813 is not responding.
Jul 9 11:04:10.857 MET-DST: %CTS-3-PAC_PROVI_FAIL: PAC Provisioning failed for 10.0.4.174
Jul 9 11:04:16.100 MET-DST: %CTS-6-ENV_DATA_START_STATE: Environment Data Download in start state
Jul 9 11:04:16.100 MET-DST: %CTS-6-ENV_DATA_WAIT_RESP_STATE: Environment Data Download in wait response state
Jul 9 11:04:16.100 MET-DST: %CTS-3-AAA_NO_RADIUS_SERVER: No RADIUS servers available for CTS AAA request for CTS env-data SM
Jul 9 11:04:16.509 MET-DST: %RADIUS-3-ALLDEADSERVER: Group ISE: No active radius servers found. Id 3.
Jul 9 11:04:26.209 MET-DST: %RADIUS-6-SERVERALIVE: Group ISE: Radius server 10.0.4.174:1812,1813 is responding again (previously dead).
Jul 9 11:04:26.209 MET-DST: %RADIUS-4-RADIUS_ALIVE: RADIUS server 10.0.4.174:1812,1813 is being marked alive.
Jul 9 11:04:36.249 MET-DST: %RADIUS-4-RADIUS_ALIVE: RADIUS server 10.0.5.9:1812,1813 is being marked alive.

Before this config i was useing a Radius config also via ISE, so i know that the server and the switch has connecction and the ports are open thrue the Firewall.

i checked and re checked the Password and the device-ID and they match

Switch config:

aaa group server radius ISE
server name vaclscise01
server name vaclscise03
!
aaa authentication login default group ISE local
aaa authentication enable default group ISE enable
aaa authentication dot1x default group ISE
aaa authorization console
aaa authorization config-commands
aaa authorization exec default group ISE local
aaa authorization network default group ISE
aaa authorization network VASYD-CTS-LIST group ISE
aaa accounting update newinfo periodic 2440
aaa accounting dot1x default start-stop group ISE
aaa accounting network default start-stop group ISE
!

aaa server radius dynamic-author
client x.x.x.x server-key 7 120901344A29260C2E1D
client x.x.x.x server-key 7 051B022C796E64011D33
auth-type any

radius-server attribute 6 on-for-login-auth
radius-server attribute 8 include-in-access-req
radius-server attribute 25 access-request include
radius-server attribute 31 mac format ietf upper-case
radius-server attribute 31 send nas-port-detail mac-only
radius-server dead-criteria time 30 tries 2
radius-server host x.x.x.x key 7 073C0F7F7E5B2E1C414A2716307922062C01340043
radius-server deadtime 1
radius-server accounting system host-config
!
radius server vaclscise01
address ipv4 x.x.x.x auth-port 1812 acct-port 1813
pac key 7 105E0D3A5D3538030832
!
radius server vaclscise03
address ipv4 x.x.x.x auth-port 1812 acct-port 1813
pac key 7 13151331532E2E222F12

Re: Problem with TrustSec ISE

Rob Ingram — Sun, 11 Jul 2021 07:56:02 GMT

@Niklas.D

Do you have the cts authorisation list command configured?I don't see it in your output above.

cts authorization list <LIST NAME>

Re: Problem with TrustSec ISE

Niklas.D — Tue, 13 Jul 2021 06:33:25 GMT

Hi Rob

i do indeed have the line

cts authorization list CTS-LIST

also: aaa authorization network CTS-LIST group ISE

Re: Problem with TrustSec ISE

Greg Gibbs — Wed, 14 Jul 2021 00:02:43 GMT

It's difficult to say without more info on what the ISE detailed logs show, what switch hardware/software, what the switch debugs show, etc. However, one of the issues I've seen with older switch code is that you cannot use the same RADIUS server for PAC and non-PAC communications. The ISE detailed logs would usually point to an issue with 'pac-opaque'

You might try the following:

Reconfigure your original RADIUS server to use a non-PAC key

radius server vaclscise01
address ipv4 x.x.x.x auth-port 1812 acct-port 1813
key xxx

Configure a new RADIUS server using the same IP but different ports and a PAC key

radius server vaclscise01-PAC
address ipv4 x.x.x.x auth-port 1645 acct-port 1646
pac key xxx

Create a new aaa server group for the PAC-based RADIUS server and update your aaa authz to use that group

aaa group server radius ISE+PAC
server name vaclscise01-PAC
!
aaa authorization network CTS-LIST group ISE+PAC

Clear your cts credentials and reapply

Re: Problem with TrustSec ISE

Niklas.D — Wed, 14 Jul 2021 06:51:08 GMT

Hey Greg thank you for the time!

radius server vaclscise01
address ipv4 10.0.4.174 auth-port 1812 acct-port 1813
key 7 107D272A363453
!
radius server vaclscise03
address ipv4 10.0.5.9 auth-port 1812 acct-port 1813
key 7 097F46
!
radius server pac-vaclscise01
address ipv4 10.0.4.174 auth-port 1645 acct-port 1646
pac key 7 12E1D
!
radius server pac-vaclscise03
address ipv4 10.0.5.9 auth-port 1645 acct-port 1646
pac key 7 07324

aaa new-model
!
!
aaa group server radius ISE
server name vaclscise01
server name vaclscise03
!
aaa group server radius ISE+PAC
server name pac-vaclscise03
server name pac-vaclscise01
!
aaa authentication login default group ISE local
aaa authentication enable default group ISE enable
aaa authentication dot1x default group ISE
aaa authorization console
aaa authorization config-commands
aaa authorization exec default group ISE local
aaa authorization network default group ISE
aaa authorization network CTS-LIST group ISE+PAC
aaa accounting update newinfo periodic 2440
aaa accounting dot1x default start-stop group ISE
aaa accounting network default start-stop group ISE
!
!
CTS password is defined in keystore, device-id = MOITCHPTEST
cts authorization list CTS-LIST
cts logging verbose
!
aaa server radius dynamic-author
client 10.0.5.9 server-key 7 107D272A53
client 10.0.4.174 server-key 7 3E05087B

This at least lets me use radius to login again that is nice 🙂

But still same problem.

Jul 14 08:39:11.204 MET-DST: %RADIUS-4-RADIUS_DEAD: RADIUS server 10.0.5.9:1645,1646 is not responding.
Jul 14 08:39:21.281 MET-DST: %RADIUS-4-RADIUS_DEAD: RADIUS server 10.0.4.174:1645,1646 is not responding.
Jul 14 08:40:01.128 MET-DST: %CTS-6-ENV_DATA_START_STATE: Environment Data Download in start state
Jul 14 08:40:01.128 MET-DST: %CTS-6-ENV_DATA_WAIT_RESP_STATE: Environment Data Download in wait response state
Jul 14 08:40:01.128 MET-DST: %CTS-3-AAA_NO_RADIUS_SERVER: No RADIUS servers available for CTS AAA request for CTS env-data SM
Jul 14 08:40:01.579 MET-DST: %RADIUS-3-ALLDEADSERVER: Group ISE+PAC: No active radius servers found. Id 7.
Jul 14 08:40:11.211 MET-DST: %RADIUS-6-SERVERALIVE: Group ISE+PAC: Radius server 10.0.5.9:1645,1646 is responding again (previously dead).
Jul 14 08:40:11.211 MET-DST: %RADIUS-4-RADIUS_ALIVE: RADIUS server 10.0.5.9:1645,1646 is being marked alive.
Jul 14 08:40:15.515 MET-DST: %CTS-3-PAC_PROVI_FAIL: PAC Provisioning failed for 10.0.4.174
Jul 14 08:40:21.298 MET-DST: %RADIUS-4-RADIUS_ALIVE: RADIUS server 10.0.4.174:1645,1646 is being marked alive.
Jul 14 08:41:01.140 MET-DST: %CTS-6-ENV_DATA_WAIT_RESP_STATE: Environment Data Download in wait response state
Jul 14 08:41:11.180 MET-DST: %RADIUS-4-RADIUS_DEAD: RADIUS server 10.0.5.9:1645,1646 is not responding.
Jul 14 08:41:21.231 MET-DST: %RADIUS-4-RADIUS_DEAD: RADIUS server 10.0.4.174:1645,1646 is not responding.

Re: Problem with TrustSec ISE

Greg Gibbs — Wed, 14 Jul 2021 23:18:06 GMT

See the TrustSec Troubleshooting Guide. The error you're seeing in the ISE detailed logs indicates that the PAC is not being provisioned. This is normally due to an error in the credential matching between ISE and the switch.

I would suggest the following:

On the switch clear your CTS credentials, remove your CTS authorization and aaa configuration.

clear cts credentials
!
no cts authorization list CTS-LIST
no aaa authorization network CTS-LIST group ISE+PAC

In ISE, ensure your NAD is configured with the name MOITCHPTEST, under the Advanced TrustSec Settings section, you have ticked the 'Use Device ID for TrustSec Identification' and re-apply the password is correct for the Device ID (ensuring that no special characters are used). In the TrustSec Notifications and Updates section, use the 'Test Connection' button to verify connectivity.

On the switch, re-apply the pac key in the RADIUS server configuration (using 'pac key 0' for non-encrypted string and again ensuring no special characters are used). Ensure this key string matches the one configured for RADIUS auth in ISE.

Re-apply the 'aaa authorization' and 'cts authorization' config. Re-apply the 'cts credentials' config ensuring that it matches the TrustSec Device ID and password configured in the NAD on ISE (no special characters).

Use the 'show cts pacs' command to ensure that the PAC is provisioned.

If all else fails, you might need to open a TAC case to investigate further.

If you're using a Cat9k switch, another option is to use the newer method of HTTP REST API as it removes the need for a EAP-FAST and PAC.

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9500/software/release/17-1/configuration_guide/cts/b_171_cts_9500_cg/cts_sgacl_env_data_dwnload.html

Re: Problem with TrustSec ISE

Niklas.D — Thu, 15 Jul 2021 12:38:41 GMT

Yeah tried this now, made a easy password. still same problem, marked as not alive. going to TAC

thank you!

Re: Problem with TrustSec ISE

Blackhawk1278 — Mon, 13 Jun 2022 19:46:17 GMT

I think I figured out what your issue is as I am having the same one, I even used Katherine Mcnamaras videos to setup trustsec. Pac provisioning uses TLS 1.0 so if you don't support that in ISE than the pac provision will fail causing the entire process to fail. If you have switched over to CAT9000s then you will need to use the Rest API method, if you still have 3k switches well your only choice is support TLS 1.0.

Re: Problem with TrustSec ISE

dddd2 — Tue, 29 Oct 2024 14:07:33 GMT

This is a great suggestion.

I've found that switch configuration that worked without issue using ISE 2.6 no longer works with ISE 3.3 because TLS1.0 was not enabled. Enabling TLS1.0 via Administation > System > Settings > Security Settings allow the switch to retrieve its PAC and download environment settings.

Re: Problem with TrustSec ISE

Arne Bier — Tue, 29 Oct 2024 22:31:52 GMT

I'm also eagerly trying to disable TLS 1.0 in ISE but ran into the same gotcha with customers who provision their IOS-XE devices with DNAC/Cat Center.

ISE 3.4 supports PAC-less provisioning for IOS-XE 17.15.12 and later. But there is no version of DNAC/Catalyst Center that supports PAC-less provisioning. Therefore if you are provisioning IOS-XE devices with DNAC/Cat Center, then you will still get the old style PAC based configs for your RADIUS server configs.