Troubleshooting a failed ISE Upgrade Readiness Tool Assessment

[RichardAtkin]

Troubleshooting a failed ISE Upgrade Readiness Tool Assessment

About the Author

Richard Atkin is an Engineer at a Cisco Partner in the UK, specialising in ISE, WiFi & Prime... I'm also currently learning about APIs, Webhooks, AWS, (etc etc) having been inspired somewhat by the Evolving Technologies component of the CCIE program.

Situation

You've started an ISE upgrade and you've decided to run the Upgrade Readiness Assessment Tool before doing the upgrade, but unfortunately it got part way through and failed.  It doesn't say why it failed but it says you should call Cisco TAC for support.  By all means call Cisco TAC if you want, but there are still a few things you can do yourself.

Resolution

As you might expect, there are a whole bunch of logs associated with all of this, but unfortunately they don't seem to default to logging a useful amount of information, so what you need to do is increase the logging levels, repeat the URT assessment exercise again, then we can offload the logs and see what's going on.  Here's the step-by-step;

1. SSH to the ISE node you are running the URT on

2. Increase the logging level;

conf t
logging level 7

3. Verify the change via Show Run, and you should see this towards the end of your Sh Ru output;

!
logging loglevel 7
!

4. Repeat the URT exercise;

application install ise-urtbundle-[version numbers].SPA.x86_64.tar.gz [Your Repository]
yes
y
y

5. Let it run until it fails again - it could fail anywhere but this is where it failed for me.  Enter a password when prompted.

6.  Now transfer the URT log files off ISE and to your machine.  Note that when you do this it doesn't seem to let you specify a repository so you have to enter a destination URL.  If the destination requires credentials ISE will prompt you to enter them.

7.  The log file is a compressed, encrypted, GPG file.  Windows can't extract these natively so download something via Google and enter the encryption password you provided earlier.

8.  Navigate through a series of empty folders until you get to the logs.  The log contents aren't documented (or at least, not that I can find) so I had to trawl through them all.  Eventually I found the log called 'dbupgrade-data-global....' contained something useful.

 9.  The dbupgrade-data-global.... log seems to document the upgrade process the URT uses.  I was able to follow this log through to the point at which it failed.  Fortunately for me, it showed the name of a particular Condition that I knew was in the ISE.  I also knew it contained all sorts of potentially unexpected stuff like "@", "..", "://", and so on... so for me, this made sense as a potential failure reason.

10.  Rather than try to work out what exactly ISE didn't like about my Eduroam condition, I simply deleted it from the config via the ISE GUI and made a note to myself to re-enter it in to the config again after the upgrade.  Hopefully you are able to identify and rectify what is causing you issue in a similar way.

11.  Repeat the URT Assessment now that you think you have resolved the issue.  Ideally it all works, or if it fails again, either your remedy didn't work, or you've hit a second issue.

12. Repeat the process until you get bored / stuck / fix the issue.  Failing that, call Cisco TAC.