cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
441
Views
0
Helpful
2
Replies
Highlighted
Beginner

ISE 2.0 URT tool failing ProfilerUpgradeService on Cloned Database

Hi,

 

The issue almost resembles this post, although I am running the ISE URT tool first.

https://supportforums.cisco.com/t5/aaa-identity-and-nac/profilerupgradeservice-2-1-0-166-error-ise-global-data-upgrade/m-p/3318564#M67457

 

The URT fails at this point:

    ...
...
Running data upgrade on cloned database - Data upgrade step 1/100, AuthzUpgradeService(2.0.0.308)... ..Done in 169 seconds. - Data upgrade step 2/100, NSFUpgradeService(2.1.0.102)... Done in 1 seconds. - Data upgrade step 3/100, UPSUpgradeHandler(2.1.0.105)... ..Done in 131 seconds. - Data upgrade step 4/100, UPSUpgradeHandler(2.1.0.107)... Done in 2 seconds. - Data upgrade step 5/100, NSFUpgradeService(2.1.0.109)... Done in 0 seconds. - Data upgrade step 6/100, NSFUpgradeService(2.1.0.126)... Done in 0 seconds. - Data upgrade step 7/100, NetworkAccessUpgrade(2.1.0.127)... Done in 1 seconds. - Data upgrade step 8/100, ProfilerUpgradeService(2.1.0.134)... Done in 0 seconds. - Data upgrade step 9/100, ProfilerUpgradeService(2.1.0.139)... Done in 0 seconds. - Data upgrade step 10/100, ProfilerUpgradeService(2.1.0.166)... ................- Failed Final cleanup before exiting... Collecting log files ... - Encrypting logs bundle... Please enter encryption password: Please enter encryption password again to verify: Encrypted URT logs(urt_logs.tar.gpg) are available in localdisk. Please reach out to Cisco TAC to debug % Post-install step failed. Please check the logs for more details.

In the generated log file, I can see an OutOfMemoryError exception:

Instance details com.cisco.profiler.upgrade.ProfilerUpgradeService@27df971b
ProfilerUpgradeService.upgrade for version 2.1.0.166
java.lang.OutOfMemoryError: GC overhead limit exceeded
Dumping heap to /var/tmp/java_pid25697.hprof ...
Heap dump file created [2017383231 bytes in 18.479 secs]
Terminated with exception java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.util.Arrays.copyOfRange(Arrays.java:3664)
	at java.lang.String.<init>(String.java:207)
	at oracle.jdbc.driver.CharCommonAccessor.getString(CharCommonAccessor.java:463)
	at oracle.jdbc.driver.T4CVarcharAccessor.getString(T4CVarcharAccessor.java:464)
	at oracle.jdbc.driver.OracleResultSetImpl.getString(OracleResultSetImpl.java:1297)
	at org.apache.commons.dbcp.DelegatingResultSet.getString(DelegatingResultSet.java:213)
	at org.apache.commons.dbcp.DelegatingResultSet.getString(DelegatingResultSet.java:213)
	at com.cisco.epm.pap.api.services.persistance.dao.ResourceDAO.searchEntities(ResourceDAO.java:6922)
	at com.cisco.epm.pap.api.services.server.resource.ResourceImpl.searchEntities(ResourceImpl.java:2016)
	at com.cisco.cpm.nsf.impl.NSFGroup.load(NSFGroup.java:212)
	at com.cisco.cpm.nsf.impl.NSFGroup.load(NSFGroup.java:158)
	at com.cisco.cpm.nsf.impl.policy.NSFPolicyImpl.loadAssociatedSubjects(NSFPolicyImpl.java:2171)
	at com.cisco.cpm.nsf.impl.policy.NSFPolicyImpl.loadPolicy(NSFPolicyImpl.java:1841)
	at com.cisco.cpm.nsf.impl.policy.NSFPolicyImpl.<init>(NSFPolicyImpl.java:233)
	at com.cisco.cpm.nsf.impl.policy.NSFPolicyImpl.retrieveAll(NSFPolicyImpl.java:1600)
	at com.cisco.profiler.api.EndpointPolicyHandler.getAll(EndpointPolicyHandler.java:238)
	at com.cisco.profiler.api.LogicalProfileImporter.<init>(LogicalProfileImporter.java:48)
	at com.cisco.profiler.api.ProfilerRegistration.defaultLogicalProfiles(ProfilerRegistration.java:703)
	at com.cisco.profiler.upgrade.DefaultLogicalProfileUpgrade.execUpgrade(DefaultLogicalProfileUpgrade.java:17)
	at com.cisco.profiler.upgrade.ProfilerUpgradeService.upgrade(ProfilerUpgradeService.java:107)
	at com.cisco.cpm.infrastructure.upgrade.impl.UpgradeServiceRegistrar.UpgradeServices(UpgradeServiceRegistrar.java:132)
	at com.cisco.cpm.infrastructure.upgrade.impl.UpgradeServiceRegistrar.main(UpgradeServiceRegistrar.java:185)
ERROR! isedataupgrade.sh FAILED. ISE GLOBAL DATA UPGRADE FAILED

The ISE SAN node had 16G memory on first URT run. Thought that may not be enough. Increased it to 24G and still, URT fails at the exact same point. "show inventory" and "show memory" confirms that additional memory is all available.

 

Seems I am also hitting bug CSCvh13873. Open a Cisco TAC case ?

 

ISE 2.0 patch 5, 2 nodes deployment.

 

Regards,

Rick.

Everyone's tags (2)
2 REPLIES 2
Cisco Employee

Re: ISE 2.0 URT tool failing ProfilerUpgradeService on Cloned Database

Hello Rick,

 

Hope you are doing well.

 

Yes, before the upgrade you need to install the hotfix application to solve this issue (CSCvh13873)  Please open a service request, we will contact you and provide the file.

 

Best regards,

-dacabrer

Beginner

Re: ISE 2.0 URT tool failing ProfilerUpgradeService on Cloned Database

Thanks dacabrer,

 

Opened a case right after, got the patch and following that upgrade was successful.

 

Regards, 

Rick.