CTS PACS auto-renew issue

[Turbo727]

Hi Experts,

We have a number of DNAC provisioned 9300 switches, everything is working fine. SSH to the devices is working as well.

show cts pacs, shows refresh timer is set to12w4d. However, after 12w4d it does not auto-renew, which resulted in being unable to ssh to the devices and ISE is marked down on the switch. we are on IOS XE 17.9.5, ISE is 3.2 patch 6. Is this a bug or some settings that can be changed on DNA/ISE?

[ahollifield]

Thankfully Cisco is completely moving away from PAC provisioning.  https://www.cisco.com/c/en/us/td/docs/security/ise/3-4/release_notes/b_ise_34_RN.html#concept_czx_1nt_11c

[Ben Walters]

We are seeing the same issues, running the same ISE version and switch model running 17.12.03. 

We are working with TAC on it, I'll update here if we get any useful information to fix this. 

[Arne Bier]

@Ben Walters what impact does this bug have on you?  Do you use TrustSec? Do you know if customers who don't use TrustSec, but who have provisioned their IOS-XE devices through DNAC,  are affected by this?  I have a bunch of switches that use PAC in the RADIUS config - but apart from the CTS constantly not being happy about one thing or another, the RADIUS process is unaffected.

Good riddance to PAC and I get annoyed thinking about the hours I have spent fixing stuff that is there on the switches due to DNAC, and DNAC's inability to think like a real engineer (it's buggy, it doesn't handle VRF logic, it introduced PAC, it's overly complex to drive, etc.). In hindsight, for non SDA environments, I'd have been better off with a config generator spreadsheet to manage all my device configs. And I could have killed off TLS 1.0 in ISE years ago. 

[Ben Walters]

We do not use trustsec at all, the settings are enabled by default when we enter ISE config in DNAC and provision switches. Right now we have only provisioned catalyst 9300s and have seen the issue across multiple devices. 

When the switches fail to update their PAC and it expires we lose all SSH access to the device and need to console in and run the cts refresh pac command before SSH access works again. Even if TAC does provide a fix we are really considering removing the ISE config from DNAC and setting the radius config from a template in DNAC.  

[ahollifield]

Is this not an SDA environment? Are you using ISE at all?

[Ben Walters]

No SDA, right now we use ISE for 802.1x wired/wireless and radius authentication for various devices. 

[andrewswanson]

Had to check my environment when I saw this - not seeing this issue with ISE 3.2 patch 3 and C9500-48Y4C 17.03.03.

Are you using a loadbalancer for ISE? If so, the bug below applies for both 17.9.5 and 17.12.3:

https://bst.cisco.com/quickview/bug/CSCwk69873

hth
Andy

[Ben Walters]

No load balancer for ISE in our environment.

[Aref Alsouqi]

Are you using RADIUS for SSH authentication to the devices? if not I can't see how and expired PAC could relate to deny SSH access. Instead if you are using RADIUS I think the expired PAC causes this issue because PAC in this case would be used to secure RADIUS traffic between the network devices and ISE. Why it is not auto-renewing? I think this would be a bug. My recommendation would be to move to TACACS for devices accesses to remove this dependency on the PAC files.

[Ben Walters]

Yes we are using RADIUS, the errors in the radius messages seen in ISE are because of the expired PAC. 

The real issue is that some of our DNAC provisioned switches renew their PAC some don't, sometimes a switch that normally renews it's PAC could not renew and have it become expired. We never used trustsec settings for RADIUS before we installed DNAC, but DNAC automatically enabled the trustsec settings when we added our ISE servers into the DNAC configuration. We just recently got DNAC as part of a hardware refresh for our access layer.

I was trying to provide more info for the original thread starter since our situation sounds extremely similar. Our "fix" for this would be to remove the ISE settings in DNAC, disable the trustsec settings that we never used before anyway and just manually add devices in ISE as they are deployed as we have done in the past. We are working with TAC to try and find a solution to keep using DNAC with ISE settings and either disable the trustsec settings, which we can't seem to do, or have the PACs renew properly so we don't see this out at our remote sites.    

[Aref Alsouqi]

If moving to TACACS for NADs management or turning off TrustSec on DNA-C are not options, then probably one thing you can consider would be moving to HTTPS for PACs exchange between ISE and the NADs. This will basically move away from the traditional way of exchanging the PACs over RADIUS to HTTPS, one of its advantages is moving away from TLS 1.0 which is used by EAP-FAST to TLS 1.1. However, it requires some not simple configs on the NADs. One of the cons I found with this though is that it requires a unique user account for each individual NAD. So, in a big environment this could be a cumbersome task, but that applies only first time you configure it. Maybe you can discuss this option with TAC and evaluate it. Check this link please to see how it could be configured:

ISE TrustSec using RESTAPI – integrating IT

[Ben Walters]

Here was the workaround from TAC if you can't physically console in to the device to reset the PAC.

• Edit the Network Device in ISE
• Under Advanced Trustsec Settings take note of the device ID and password.
• Uncheck Advanced Trustec Settings
• Click Save
• Access network device using local credentials.
• Enable Advanced Trustsec Settings
• Populate the device ID and password
• Click Save
• From CLI of network device perform cts refresh pacHopefully more info on an actual fix is on it's way.

[Aref Alsouqi]

Thanks for sharing that, but tbh it's kinda not scalable at all especially if you have hundreds of devices affected by this bug/behaviour. Hopefully this will be fixed in one of the next releases.

[Ben Walters]

We finally have confirmation from TAC on how to fix this long term. First we had to confirm that our ISE PSNs were in the trustsec server list in ISE. Workcenters > Trustsec Components > Trustsec Servers > Trustsec AAA servers. 

We then had to add the following commands to our switches:

aaa authorization network dnac-cts-list group dnac-network-radius-group
cts authorization list dnac-cts-list

TAC said this was a bug/missing feature, since not having those commands on the switch side would eventually lead to expired PACs like we see. They are sending a feature request to the DNAC devs to add these commands in as default config for DNAC when it has ISE settings configured on a switch.