Just wanted to update,
I did redeploy ISE 2.7 P10 without it been restored from backups. Here is order of operations I carried out:
• On network switch where Cisco ISE is attached, I created ACL and applied them tot he ports where ISE Prod NIC was attached. ACL is there to prevents OLD ISE to be able to contact the new one (Had this some time ago, where I built new node and as it was using same credentials, was already picked up by old deployment that synced corrupt state).
It also was there to prevent NAD to start contacting newly built ISE and encountering default policies, thus causing authentication problems. I also included ACE's to prevent DNAC center reaching it. Thing to keep in mind is that ACL needs to be quite specific, as you do not want to block ISE to speak to DNS, Default Gateway, NTP and Internet destinations. I went overboard and did both Inbound and Outbound ACLs.
• Built ISE node from USB and applied the Patch 10.
• Updated Profiler Feed (this could sometimes be issue as I had seen where node profiling databases were not syncing up and if new node is added to deployment, it could remain with outdated profiling policies. This however could be just my bad luck and corrupt ISE I had running)
• Imported Certificates and assigned them to the services.
• Exported from old ISE any Profiling Policies that were created, look for any System State = Administrator Created or Administrator Modified. This if configured, will go about re-creating Endpoint Identity Groups.
• Manually created any missed Endpoint Identity Groups.
• Exported and imported Network Device Profiles, Network Device Groups and Network Devices.
• Configured System Settings to match those of old ISE.
• Joined ISE to AD and matched against what Security Groups should be pulled in.
• Sorted all Identity Source Sequences to match those of old node.
• Exported and Imported User Identity Groups and local ISE Identities.
• Created Policy Authentication Protocols to match the old deployment.
• Recreated Guest Portal and matched all the settings.
• Recreated all DACLs and Authorization Profiles, Library Conditions.
• Recreated all Policies, their AuthC and AuthZ rules.
• Exported from old node and imported into new one all Endpoints.
• Checked that I have enabled ERS and configured pxGrid setting to true for "Automatically approve new certificate-based accounts"
Now that ISE was close of its older self, I took it over weekend to carry our reintegration with DNA Center, here are some of the issues I encountered and managed to resolve.
• I shut down old ISE node, it had actually died over night from Friday to Saturday as its Application Server had been stuck in Initializing when I checked it.
• Removed the ACL from network switch port where new ISE node was attached.
• I did start to see authentication attempts and some sessions been picked up, I did have issues with ISE reporting that TrustSec was not working correctly and that there SGT's asked to be assigned that did not exist in ISE.
NOTE: I did not create any of the SGT's in ISE as my deployment, they were managed by DNAC.
This is where some of the problems started, DNA center would of course report that it is able to see one node and not both (DNAC System 360), I hoped that by just going into Settings and under Authentication and Policy Servers, edit the ISE server, even so I updated the username and password (which were the same on old ISE), FQDN, and be greeted with popup where it goes through the stages of integration, it promptly displayed it as all done and dusted with green circles for each step. DNAC however would still report that pxGrid is Unavailable and I did not see a sign of ISE listing any clients which would match the DNAC.
I switched to troubleshoot switch issues relating to CTS and PACs, I redeployed all my switches with only one ISE aka Radius server, hoping that this will resolve the issues I was seeing with TrustSec on the switches, but this did not help, I should have done the following step sooner.
As I was not having too much of a luck with DNAC interacting with ISE in expected manner, that is updating PACs for NADs and repopulating TrustSec, I ended up adding bogus Radius Server under Authentication and Policy Servers.
This allowed me to go into Design - Network Settings and update AAA settings and remove any mention of ISE IP addresses, I had to do this for Wireless AAA servers as well.
This now allowed me to completely remove ISE under Authentication and Policy Servers, I did also clear out any trusted certificates that referenced ISE deployment (on this I was not entirely sure why they were there in first place as later when I re-added ISE, they were none to be found).
Now I was able to add ISE from scratch into DNAC, integration worked, pxGrid was marked as Available, and I was able to see client in ISE pxGrid dashboard. I then went back to DNAC and navigated Policy - Group-Based Access Control that presented me with integration steps that allowed to sync DNAC and ISE SGT's and policy matrix (This area previously would just in read say that Integration is not implemented and there was nothing I could do apart from viewing what SGT's and Policy Matrix is in place)
Then I proceeded with again reconfiguring all my Network Setting and Wireless AAA servers, removed the bogus Radius server I created earlier and was able to re-provision all network devices with this one ISE node.
I had to go about connecting to every switch I had and carry out clearing of old PACs and refresh CTS Environment-data:
clear cts pac all
clear cts environment-data
cts refresh pac
cts refresh environment-data
This was due to following Alert being logged in ISE:
=================================================
Trustsec PAC validation failed
Details :
TrustSec recieved invalid PAC';'ISE identified Trustsec request with invalid PAC. Verify network device uses a pac derived for the current ISE deployment. Try to refresh the PAC on the network device.
=================================================
I still believe I have some PAC issues as on my WLC5520, I had to remove "PAC Provisioning" under Radius server or I would not have even single authentication request come in from this WLC.
Now that I had both the Wired and Wireless authentication working for MAB and EAP-TLS, I proceed with second ISE node rebuild, patching, updating Profiling Feed and joining the new deployment. I was able to again re-provision all network devices to have both ISE nodes for AAA activities.
One thing I did see, which I believe is a bug on DNAC, it would not store the Secret field populated, It would allow me to type it in both for network and client section, allow me to save without the issues, but would disappear when I come back to check the Network Settings again.
I am intending to build a virtual ISE 2.7 in isolated environment where I could restore my old deployments back, just in case there is something crucial I have forgotten about setting wise, but most should be fixable once I knw what is not working.
Hope this serves as a reminded, keep your platform in support and remove it time before anything goes EoL. For 10,000 endpoints we have with active session being around 2000 daily, this took me 4 days to complete, longest part of course was recreating Authorization Profiles and Policies and I was fortunate that I had a weekend where I could do switchover of the deployment.
