Hi All,
I am fully aware that 2.7 is EoL, but wanted to seek some guidance.
We have 2 Cisco ISE nodes running on EoL 3515-K9, I recall that I posted some time ago issues I observed with deployment when it was still running Patch 7, I believe this was just around the time when Log4J vulnerability was around. Either way I believe Hot fix for Log4J was installed and later deployment was patched version 7.
Issues observed were around High Resource Utilization that caused High Latency and DNAC reporting ISE server down every morning at around 03:40. Along with Replication Errors and Queue Link errors etc.
This subsequently at one point led down to the issue which I have observed 4 times now, with last one happening right now.
ISE is setup as followed, 1x PAN,pMnT, PSN and 1x sPAN, sMnT, PSN and PxGrid. It works fine until, one day when you log onto Admin page, I get error message " Oops, Something went wrong, Invalid Request" I am able to bypass this message by just clearing the URL in browser, by removing anything past the FQDN.
When logged in I do see same alerts, Slow Replication Error, High Load Average, Replication Failed, Slow Replication Warning and others. If I was to clear any alert and try to acknowledge them I get a popup at bottom right corner that "Alert(s) could not be acknowledged because you do not have enough permissions or due to an unexpected error", if I refresh the Alerts, I actually see that it is removed.
Next, deployment is eventually stating that it needs to be re-synced, if I look in Deployment, it does state Not in Sync, pressing syncup, it gives an error "Unable to sync node NODE-NAME. Sync may already be in progress".
Service restart does not help, cold boot of both nodes does not do it either.
Issues are more serious as I am no longer able to add new endpoints and it plainly says so Unable to create the endpoint, but it appears as created if I try again to add it, if I then look in Context Visibility for the endpoints, it is not displayed. Other issues are that I am not able to see either one of my superadmin accounts even so I am able to log in with them.
When this happened first at Path7, case was raised with TAC, they looked at it and said that yes, issue could be with the Log4J Hot fix and that it is corrupting things, they advised to rebuild and then restore the backup, did that, patched to version 7 and then they followed up and advised that Patch 8 was released and that it should be applied as well. It was followed through, and some 6 month later this same issue happened again, TAC said that this time Patch 8 was completely recalled and I should rebuild, go to Patch version 9 directly and do a restore. I did this and again it was fine for some time until this happened again and again, now even with Patch 10.
I have a reason to believe that my entire 2 node setup is just corrupted beyond repair, doubt that TAC would touch it, but worst part is, it appears this issue follows the backup. Backup as well has issues, even at the first rebuild, issue was that after restore, it appears that Certs I exported are somehow stuck in DB even so not visible, I am not able to import them back. This required TAC to root the access to the ISE and delete them from DB. I cannot do it without TAC help, and last time I rebuilt, I was lucky that external cert used for the Guest Portal was due for renewal in couple of weeks and I generated new CSR for it and got new cert. I had to generate and apply new certs for PSN, Admin, PxGrid, but as they are on internal CA, it is not much and issue.
So what are my options, how can I get rid of this issue before I get my new hardware and support? I would need to install ISE 3.x and then perform a restore of my old 2.7 deployment, but I do not want to inherit same issues with new deployment.
How feasible is to extract every piece of information of the existing ISE and just build ISE 2.7 Patch 10 on a side from scratch and import endpoints, Endpoint Identity Groups, Policies and NAD.
Worth to mention that I also need to keep DNAC integration for the SDA,
What would you advise? Please do tell me for much pain I am in for?
