ISE 2.0 distributed deployment upgrade experiences

[Ben.Levin]

I'm getting ready to upgrade our 8 node ISE 1.3 deployment to 2.0.  I've followed the upgrade documentation to prepare for this but I was wondering if anyone has experience doing the 2.0 upgrade on a similar setup.   Do you have any experiences, issues, etc, you can share?  I'm particularly interested in how long it took.  We did set up a 2 node deployment in our lab but the upgrade was pretty quick, about 1 hour per server.

Thank you.

[Martin Bosch]

In the last 2 weeks I have upgraded from ISE 1.2 -> 1.3

then 1.3 -> 2.0. finally 2.0 - 2.1.

I used the same repository where I keep my backups.

So in mycase CiscoISE is my repository

"application upgrade prepare ise-upgradebundle-1.3.x-and-1.4.x-to-2.0.0.306.x86_64.tar.gz CiscoISE"

once uploaded

"application upgrade proceed"

but if you have done a upgrade in between remember to use

"application upgrade cleanup"

Make sure you have a backup before you start, including your cert's.

[joevimal01]

Hi Martin,

Thanks. But Cisco is recommending to load the IOS to local Disk before the upgrade to reduce the time.

I would request you to share steps for above my query if anyone has experience in this regard.

1st copying IOS bundle to local Disck 

2nd upgrade it.

[joevimal01]

Hi 

i am getting this error message when i try to copy the file to local disk.

admin# copy ftp://<username:password ftpserverip>:/ise-upgradebundle-1.3.x-and-1.4.x-to-2.1.0.474.x86_64.tar.gz disk:/
Username: joe
Password:
% Error: Transfer failed

[Marvin Rhoads]

joevimal01  ,

If you're running your FTP server on Windows, verify your Windows firewall is turned off temporarily or configured to explicitly allow incoming ftp connections.

Open the ftp server console to make sure you see the incoming requests from ISE. It will typically give you more verbose information about the reason for transfer failures.

[joevimal01]

Hi Marvin,

There was problem in syntax.

Its fixed after removing the colon following ftpserverip . 

Working Syntax:

admin# copy ftp://<username:password ftpserverip>/ise-upgradebundle-1.3.x-and-1.4.x-to-2.1.0.474.x86_64.tar.gz disk:/

Is there a way to check MD5 on ISE CLI?

Thanks for your support

[Marvin Rhoads]

When you move to the next step of preparing the upgrade, ISE will prompt you to confirm that the MD5 checksum and SHA-256 hash are correct. the correct values are posted on Cisco's download page for the upgrade files.

ise/admin# application upgrade prepare ise-upgradebundle-1.3.x-and-1.4.x-to-2.1.0.x.x86_64.tar.gz upgrade
Getting bundle to local machine...
md5: 35a159416afd0900c9da7b3dc6c72043
sha256: e3358ca424d977af67f8bb2bb3574b3e559ce9578d2f36c44cd8ba9e6dddfefd
% Please confirm above crypto hash matches what is posted on Cisco download site.
% Continue? Y/N [Y]

[joevimal01]

Hi Marvin,

I actually planned to execute the below command as we discussed earlier.

application upgrade ise-upgradebundle-1.3.x-and-1.4.x-to-2.1.0.x.x86_64.tar.gz upgrade

Please help which one i have to use?

i hope the above command is to copy the file to local disk and then , we need execute next command as application upgrade proceed.

[Marvin Rhoads]

You can do it either way. The "application prepare" method will combine copying from repository, doing file check and extracting the files in one step. This is described in the 2.1 Upgrade Guide here:

http://www.cisco.com/c/en/us/td/docs/security/ise/2-1/upgrade_guide/b_ise_upgrade_guide_21/b_ise_upgrade_guide_21_chapter_011.html

If you just do a manual copy and then proceed, you do not have the option of checking the MD5 directly as the script calls the checking processes from the underlying Linux OS to which interactive access is blocked.

You can proceed as you planned without doing the prepare step - you will just have to bet on the MD5 being fine. If it got corrupted, the upgrade will fail and the node should revert.

[joevimal01]

Thanks Marvin:)

My concern is . Since i already copied the IOS bundle to local disk, don't have an option to check MD5? or shall i still run application upgrade prepare and application upgrade proceed ?

[Marvin Rhoads]

If you're cautious, I believe you can still run the prepare bit.

I have never had an ISE upgrade fail due to incorrect MD5 hash - but there's always a first time.

(I have had Prime Infrastructure upgrade fail due to that but there I had to do a manual ftp from the cli and omitting the "binary " mode command would cause the transfer to be inexact.)

[joevimal01]

hi ,

Any one know how long will take for ISE Backup in distributed environment.

i see the below status for 30mins

admin# backup config repository ISE-Configbkp-1 ise-config encryption-key plain <key>
% Internal CA Store is not included in this backup. It is recommended to export it using "application configure ise" CLI command
% Creating backup with timestamped filename: config-CFG-160916-1932.tar.gpg
% backup in progress: Starting Backup...10% completed
% backup in progress: Validating ISE Node Role...15% completed
% backup in progress: Backing up ISE Configuration Data...20% completed

[Marvin Rhoads]

I haven't measured it, but a configuration backup seems to typically take about 10-20 minutes in the environments I've worked with.

[joevimal01]

Hi Marvin

Its end with error.

admin# backup config repository ISE-Configbkp-1 ise-config encryption-key plain <key>
% Internal CA Store is not included in this backup. It is recommended to export it using "application configure ise" CLI command
% Creating backup with timestamped filename: config-CFG-160916-1932.tar.gpg
% backup in progress: Starting Backup...10% completed
% backup in progress: Validating ISE Node Role...15% completed
% backup in progress: Backing up ISE Configuration Data...20% completed
% backup in progress: Backing up ISE Logs...45% completed
% backup in progress: Completing ISE Backup Staging...50% completed
% backup in progress: Backing up ADEOS configuration...55% completed
% backup in progress: Moving Backup file to the repository...75% completed
% File transfer error

am i doing in the right way..is there any impact to node performing this?

[Marvin Rhoads]

Where is your repository - remote ftp? If so, check the ftp server log for errors encountered.

There should be no functional impact to the node performing backup

[joevimal01]

Hi Marvin,

yes, repository is Remote FTP server. but i do not see any logs.i am using FTP 3Cdaemon. i see only "listening for FTP  request on ip address<>