I recently posted something to do with CTS issues we are facing in our production SDA fabric. I want to bring this issue back because we are experiencing it again.
Versions:
ISE 2.3p5
DNAC 1.2.8
Overview: Cat 3850s running 16.9.2(s) and 9300s running 16.9.2 &/OR 16.6.5 are having CTS provisioning job hang ups. ISE radius live logs get flooded with request drops for CTS PAC provisioning. ISE NADs show the troublesome devices with PACs. However, when checking via CLI on the NADs they have no PACs upon a reboot. Each issue case the NADs have 2 hung provisioning jobs. I have to go into DNAC, re-provision the device in order for the NADs to get a new PAC. Note that there is then 1 CTS provisioning job still hung. Issuing another reboot takes care of the problem. In a previous post apparently manually removing radius server configs and re-adding them also fixes the issue.
Here are the steps and what I see after a reboot:
#sh cts pac
No PACs found in the key store.
#sh cts provisioning
A-ID: Unknown
Server xxxx, using shared secret
Req-ID 42460001: callback func 0x7f3daee5e810, context 0x54000005
A-ID: Unknown
Server xxxx, using shared secret
Req-ID 3e5e0002: callback func 0x7f3daee5e810, context 0xf8000006
Req-ID 70450031: callback func 0x7f3daf9317f0, context (nil)
%RADIUS-3-NOSERVERS: No Radius hosts configured or no valid server present in the server group dnac-network-radius-group
One difference between IOSs on separate test cases using 9300s (16.9.2 & 16.6.5) is that the issue is the same except the 16.6.5 host attempts to reach out to ISE authenticating with an identity known as "CTS-Test-Server".
If anyone can provide any insight it would be greatly appreciated. I have TAC engaged & I am working through my reps to get the BU involved. Thanks in advance.
