URT success but time estimate failed (2.1 -> 2.3)

[kevinchan]

Hi all,

    Just looking for comment if I should continue with the actual upgrade from ISE 2.1 to 2.3.

    Looks like URT to run successfully (after all the kinks), but the 'Time Estimated to Upgrade" gave an error.

thanks in advance!

Kevin

MyISEBox/admin# application install ise-urtbundle-2.3.0.298-1.1.0.SPA.x86_64.tar.gz localDisk

Save the current ADE-OS running configuration? (yes/no) [yes] ? yes

Generating configuration...

Saved the ADE-OS running configuration to startup successfully

Getting bundle to local machine...

Unbundling Application Package...

Verifying Application Signature...

Initiating Application Install...

###########################################

# Installing Upgrade Readiness Tool (URT) #

###########################################

Checking ISE version compatibility

- Successful

Checking ISE persona

- Successful

Installing and running URT might consume additional resources.

Do you want to proceed with installing and running URT now (y/n):y

Checking if URT is recent(<45 days old)

- Note: URT is 216 days old and its version is 1.1.0. There might be a recent URT bundle on CCO, please verify on CCO

Do you want to proceed with this version which is 216 days old (y/n):y

Proceeding with this version of URT itself

Installing URT bundle

- Successful

########################################

# Running Upgrade Readiness Tool (URT) #

########################################

This tool will perform following tasks:

1. Pre-requisite checks

2. Clone config database

3. Copy upgrade files

4. Data upgrade on cloned database

5. Time estimate for upgrade

Pre-requisite checks

====================

Disk Space sanity check

- Successful

NTP sanity

- Successful

Appliance/VM compatibility

- Successful

Trust Cert Validation

- Successful

System Cert Validation

- Successful

5 out of 5 pre-requisite checks passed

Clone config database

=====================

[########################################] 100%  Successful                                        

Copy upgrade files

==================

- N/A

Data upgrade on cloned database

===============================

Modifying upgrade scripts to run on cloned database

- Successful

Running schema upgrade on cloned database

- Running db sanity to check and fix if any index corruption

- Auto Upgrading Schema for UPS Model

- Upgrading Schema completed for UPS Model

- Successful

Running sanity after schema upgrade on cloned database

- Successful

Running data upgrade on cloned database

- Data upgrade step 1/60, NSFUpgradeService(2.1.101.145)... Done in 29 seconds.

- Data upgrade step 2/60, ProfilerUpgradeService(2.1.101.145)... Done in 0 seconds.

- Data upgrade step 3/60, UPSUpgradeHandler(2.1.101.188)... Done in 12 seconds.

- Data upgrade step 4/60, NetworkAccessUpgrade(2.2.0.007)... Done in 1 seconds.

- Data upgrade step 5/60, UPSUpgradeHandler(2.2.0.118)... Done in 5 seconds.

- Data upgrade step 6/60, GuestAccessUpgradeService(2.2.0.124)... Done in 14 seconds.

- Data upgrade step 7/60, NSFUpgradeService(2.2.0.135)... Done in 0 seconds.

- Data upgrade step 8/60, NSFUpgradeService(2.2.0.136)... Done in 0 seconds.

- Data upgrade step 9/60, NetworkAccessUpgrade(2.2.0.137)... Done in 0 seconds.

- Data upgrade step 10/60, NetworkAccessUpgrade(2.2.0.143)... Done in 15 seconds.

- Data upgrade step 11/60, NSFUpgradeService(2.2.0.145)... Done in 4 seconds.

- Data upgrade step 12/60, NSFUpgradeService(2.2.0.146)... Done in 2 seconds.

- Data upgrade step 13/60, NetworkAccessUpgrade(2.2.0.155)... Done in 0 seconds.

- Data upgrade step 14/60, CdaRegistration(2.2.0.156)... Done in 2 seconds.

- Data upgrade step 15/60, NetworkAccessUpgrade(2.2.0.161)... Done in 0 seconds.

- Data upgrade step 16/60, UPSUpgradeHandler(2.2.0.166)... Done in 0 seconds.

- Data upgrade step 17/60, NetworkAccessUpgrade(2.2.0.169)... Done in 0 seconds.

- Data upgrade step 18/60, UPSUpgradeHandler(2.2.0.169)... Done in 0 seconds.

- Data upgrade step 19/60, NetworkAccessUpgrade(2.2.0.180)... Done in 0 seconds.

- Data upgrade step 20/60, CertMgmtUpgradeService(2.2.0.200)... Done in 0 seconds.

- Data upgrade step 21/60, NetworkAccessUpgrade(2.2.0.208)... Done in 0 seconds.

- Data upgrade step 22/60, RegisterPostureTypes(2.2.0.218)... Done in 3 seconds.

- Data upgrade step 23/60, NetworkAccessUpgrade(2.2.0.218)... Done in 2 seconds.

- Data upgrade step 24/60, NetworkAccessUpgrade(2.2.0.222)... Done in 6 seconds.

- Data upgrade step 25/60, NetworkAccessUpgrade(2.2.0.223)... Done in 0 seconds.

- Data upgrade step 26/60, NetworkAccessUpgrade(2.2.0.224)... Done in 2 seconds.

- Data upgrade step 27/60, SyslogTemplatesRegistration(2.2.0.224)... Done in 0 seconds.

- Data upgrade step 28/60, ReportUpgradeHandler(2.2.0.242)... Done in 0 seconds.

- Data upgrade step 29/60, IRFUpgradeService(2.2.0.242)... Done in 0 seconds.

- Data upgrade step 30/60, LocalHostNADRegistrationService(2.2.0.261)... Done in 0 seconds.

- Data upgrade step 31/60, NetworkAccessUpgrade(2.2.0.300)... Done in 0 seconds.

- Data upgrade step 32/60, CertMgmtUpgradeService(2.2.0.300)... Done in 5 seconds.

- Data upgrade step 33/60, NSFUpgradeService(2.2.0.323)... Done in 0 seconds.

- Data upgrade step 34/60, NetworkAccessUpgrade(2.2.0.330)... Done in 0 seconds.

- Data upgrade step 35/60, NSFUpgradeService(2.2.0.340)... Done in 0 seconds.

- Data upgrade step 36/60, NetworkAccessUpgrade(2.2.0.340)... Done in 0 seconds.

- Data upgrade step 37/60, NetworkAccessUpgrade(2.2.0.342)... Done in 0 seconds.

- Data upgrade step 38/60, AuthzUpgradeService(2.2.0.344)... Done in 0 seconds.

- Data upgrade step 39/60, RegisterPostureTypes(2.2.0.350)... Done in 23 seconds.

- Data upgrade step 40/60, DictionaryUpgradeRegistration(2.2.0.374)... Done in 19 seconds.

- Data upgrade step 41/60, UPSUpgradeHandler(2.2.0.403)... Done in 0 seconds.

- Data upgrade step 42/60, DictionaryUpgradeRegistration(2.2.0.410)... Done in 0 seconds.

- Data upgrade step 43/60, UPSUpgradeHandler(2.3.0.100)... Done in 56 seconds.

- Data upgrade step 44/60, UPSUpgradeHandler(2.3.0.110)... Done in 0 seconds.

- Data upgrade step 45/60, NetworkAccessUpgrade(2.3.0.145)... Done in 0 seconds.

- Data upgrade step 46/60, NodeGroupUpgradeService(2.3.0.155)... Done in 5 seconds.

- Data upgrade step 47/60, IRFUpgradeService(2.3.0.155)... Done in 0 seconds.

- Data upgrade step 48/60, UPSUpgradeHandler(2.3.0.158)... Done in 0 seconds.

- Data upgrade step 49/60, NetworkAccessUpgrade(2.3.0.178)... Done in 0 seconds.

- Data upgrade step 50/60, NetworkAccessUpgrade(2.3.0.182)... Done in 0 seconds.

- Data upgrade step 51/60, RegisterPostureTypes(2.3.0.189)... Done in 0 seconds.

- Data upgrade step 52/60, CertMgmtUpgradeService(2.3.0.194)... Done in 2 seconds.

- Data upgrade step 53/60, UPSUpgradeHandler(2.3.0.201)... Done in 0 seconds.

- Data upgrade step 54/60, ProfilerUpgradeService(2.3.0.232)... Done in 0 seconds.

- Data upgrade step 55/60, GuestAccessUpgradeService(2.3.0.243)... Done in 0 seconds.

- Data upgrade step 56/60, NSFUpgradeService(2.3.0.249)... Done in 0 seconds.

- Data upgrade step 57/60, NSFUpgradeService(2.3.0.273)... Done in 4 seconds.

- Data upgrade step 58/60, NSFUpgradeService(2.3.0.298)... Done in 0 seconds.

- Data upgrade step 59/60, ProfilerUpgradeService(2.3.0.298)... Done in 0 seconds.

- Data upgrade step 60/60, GuestAccessUpgradeService(2.3.0.298)... Done in 5 seconds.

- Successful

Running data upgrade for node specific data on cloned database

- Successful

Time estimate for upgrade

=========================

(Estimates are calculated based on size of config and mnt data only. Network latency between PAN and other nodes is not considered in calculating estimates)

Estimated time for each node (in mins):

/opt/urt/estimate_time.sh: line 17: ERROR:

ORA-12545: Connect failed because target host or object does not exist

SP2-0306: Invalid option.

Usage: CONN[ECT] [{logon|/|proxy} [AS {SYSDBA|SYSOPER|SYSASM|SYSBACKUP|SYSDG|SYSKM}] [edition=value]]

where <logon> ::= <username>[/<password>][@<connect_identifier>]

      <proxy> ::= <proxyuser>[<username>][/<password>][@<connect_identifier>]

SP2-0306: Invalid option.

Usage: CONN[ECT] [{logon|/|proxy} [AS {SYSDBA|SYSOPER|SYSASM|SYSBACKUP|SYSDG|SYSKM}] [edition=value]]

where <logon> ::= <username>[/<password>][@<connect_identifier>]

      <proxy> ::= <proxyuser>[<username>][/<password>][@<connect_identifier>]

SP2-0157: unable to CONNECT to ORACLE after 3 attempts, exiting SQL*Plus*5: syntax error in expression (error token is ":

ORA-12545: Connect failed because target host or object does not exist

SP2-0306: Invalid option.

Usage: CONN[ECT] [{logon|/|proxy} [AS {SYSDBA|SYSOPER|SYSASM|SYSBACKUP|SYSDG|SYSKM}] [edition=value]]

where <logon> ::= <username>[/<password>][@<connect_identifier>]

      <proxy> ::= <proxyuser>[<username>][/<password>][@<connect_identifier>]

SP2-0306: Invalid option.

Usage: CONN[ECT] [{logon|/|proxy} [AS {SYSDBA|SYSOPER|SYSASM|SYSBACKUP|SYSDG|SYSKM}] [edition=value]]

where <logon> ::= <username>[/<password>][@<connect_identifier>]

      <proxy> ::= <proxyuser>[<username>][/<password>][@<connect_identifier>]

SP2-0157: unable to CONNECT to ORACLE after 3 attempts, exiting SQL*Plus*5")

/opt/urt/estimate_time.sh: line 114: 63 + : syntax error: operand expected (error token is "+ ")

Final cleanup before exiting...

^C

Application successfully installed

sns-ksl-admin/admin# show application

<name>          <Description>

ise             Cisco Identity Services Engine

                Patches: 3

urt             Cisco ISE - Upgrade Readiness Tool

sns-ksl-admin/admin# ^C

sns-ksl-admin/admin#

sns-ksl-admin/admin#

sns-ksl-admin/admin#

sns-ksl-admin/admin# dir

Directory of disk:/

         69 Jul 07 2016 16:45:43  1

       4096 Jul 07 2016 11:56:51  corefiles/

          0 Apr 15 2018 13:21:25  err.out

  483335487 Jun 14 2017 19:58:48  ise-patchbundle-2.1.0.474-Patch3-201683.SPA.x86_64.tar.gz

6581931913 Jul 07 2016 16:45:43  ise-upgradebundle-2.0.x-to-2.1.0.474.SPA.x86_64.tar.gz

  440632959 Apr 12 2018 23:52:48  ise-urtbundle-2.3.0.298-1.1.0.SPA.x86_64.tar.gz

       4096 Jul 07 2016 16:45:43  lost+found/

    1587285 Apr 15 2018 12:36:25  urt_logs.tar.gpg

           Usage for disk: filesystem

                92201902080 bytes total used

                442204577792 bytes free

                563030368256 bytes available

sns-ksl-admin/admin# show application

<name>          <Description>

ise             Cisco Identity Services Engine

                Patches: 3

urt             Cisco ISE - Upgrade Readiness Tool

sns-ksl-admin/admin# application remove ?

  <WORD>  Application name to be removed (Max Size - 255)

sns-ksl-admin/admin# application remove urt

Continue with application removal? (y/n) [n] ? y

Application successfully uninstalled

sns-ksl-admin/admin# show application

<name>          <Description>

ise             Cisco Identity Services Engine

                Patches: 3

sns-ksl-admin/admin#

[hslai]

If the URT is run for a multi-node deployment, the error appears it unable to connect to the oracle database of the active M&T node, which it expected to be a secondary ISE node. Is your M&T on the primary ISE node?

I would suggest to engage TAC if not already done so TAC may investigate further.

[kevinchan]

HI hslai,

       Yes .. we have a multinode deployment.  2 PAN, 2 monitor and 4 PSN.

       I ran URT on secondary PAN.

        So you saying this is expected result for multiple node deployment ( when URT ran on secondary PAN?)

thanks!

Kevin

[hslai · Accepted solution]

Nope. URT should run on the secondary PAN.

From my read of the script, it's expecting the active M&T on a secondary ISE node. Thus, I think it would fail if your primary PAN happens also the active M&T. And, you should be able to work around it by moving the active M&T to a secondary ISE node in the deployment.

[kevinchan]

I see...

So temporarily set Secondary PAN to be the 'primary' Monitoring node also and then run URT again?

Kevin

[hslai · Accepted solution]

Right. That should do it.

[hslai]

Please also review this doc ISE Upgrades - Best Practices, if not already done.

[hslai]

CSCvi99139 opened to track this issue. It will take a couple of days for reviews before visible.

[victormanuelsolis]

Hello,

the workaround for this issue is the following according to CSCvi99139  

Workaround: Move the active M&T role to a secondary ISE node

This mean I need to promote the secondary admin node to primary and run again the URT from the new secondary?

[ahollifield]

The workaround means you need to move the Primary Monitoring Persona to a node that also isn't the Primary Administration Node.  Are you sure this still applies to your version though?  What is your current version of ISE and what version are you upgrading to?

[victormanuelsolis]

Hello sir, yeap, I try to upgrade from 2.4 to 2.7

[ahollifield · Accepted solution]

I would say TAC would be the best place to go next.  Also: https://www.cisco.com/c/en/us/products/collateral/security/identity-services-engine/bulletin-c25-2943876.html 

[victormanuelsolis]

Thank you!

[Marcelo Morais · Accepted solution]

Hi @victormanuelsolis ,

 remember that ISE 2.7 supports restore from backups obtained from ISE 2.2 and later.

 In other words, you are able to:

1st. de-register the SPAN from your Cluster

2nd backup the SPAN (at this time it's a Standalone Node)

3rd install from scratch ISE 2.7 P7 on SPAN

4th restore the backup

Repeat the process to other Nodes and create a new Cluster (ISE 2.7 P7)

Hope this helps !!!