CTS Request before PAC provisioning making RADIUS server DEAD

[Michal Olsovsky]

Hi,

after recent upgrade of C3650s from 16.6.4 to 16.6.6 switches started requesting CTS data before PAC is provisioned. Because of this ISE is dropping RADIUS messages with the error message 11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute. These silent drops are effectively marking the RADIUS server "DEAD" and because of "radius-server deadtime 15" making it unusable for some time. 

Does anyone else also observed this change of CTS request behavior? Is this now new expected behavior? Is there a way to force the switch to ask for CTS data only once the PAC is provisioned or change the ISE not to silently drop the requests but reply with access reject message?

Thank you.

[Mike.Cifelli]

I had the exact same issue in my SDA fabric across the campus on both 9K and 3K series switches to include edge nodes and intermediary nodes. The nodes experienced the same issue even when testing other SW releases too. I had a TAC ticket and several calls with the BU on the issue and they were unable to replicate the issue so the case was left as a monitoring case. Since then we have upgraded both our DNAC and ISE cluster and I have not seen the issue since (knock on wood). To further expand on the issue we were facing:
Switches would have two hung CTS provisioning jobs, and no PACs. The NAD in ISE would show valid PAC, but drop requests. Therefore, as you also mentioned on the NAD itself radius server would be found as dead.
Our workarounds used to temporarily fix the issue relating to our unique scenario in SDA was to go into DNAC, re-provision the device (this re-provision would terminate one of the two hung CTS jobs), then a fresh reboot of device would make the device happy. And by happy I mean no more hung jobs, and it would successfully pull down its PAC from ISE. If you are not running SDA I have found that removing your radius server configs and re-applying may do the trick for you.
I know this does not directly answer your question, but I wanted to share that I have found pretty much the same issue. My recommendation would be to contact TAC to make them aware, work that angle, and potentially test some upgrades. Good luck & HTH!

[Damien Miller]

I have found that leveraging the automated-tester feature reduces the resolution time of this.

Mike, I had the same hung CTS pac provisioning in earlier releases or when load balancer persistence settings were misconfigured.

[Michal Olsovsky]

Hi,

RADIUS servers are probed every 1 with the automate-tester feature and ISE is sending back access-reject messages however this doesn't bring the RADIUS servers UP again. I have also tried to remove the whole RADIUS config and applied back but no difference. Looks that logging a case is the only option left.

Thank you!