Hi,
The issue almost resembles this post, although I am running the ISE URT tool first.
https://supportforums.cisco.com/t5/aaa-identity-and-nac/profilerupgradeservice-2-1-0-166-error-ise-global-data-upgrade/m-p/3318564#M67457
The URT fails at this point:
...
...
Running data upgrade on cloned database
- Data upgrade step 1/100, AuthzUpgradeService(2.0.0.308)... ..Done in 169 seconds.
- Data upgrade step 2/100, NSFUpgradeService(2.1.0.102)... Done in 1 seconds.
- Data upgrade step 3/100, UPSUpgradeHandler(2.1.0.105)... ..Done in 131 seconds.
- Data upgrade step 4/100, UPSUpgradeHandler(2.1.0.107)... Done in 2 seconds.
- Data upgrade step 5/100, NSFUpgradeService(2.1.0.109)... Done in 0 seconds.
- Data upgrade step 6/100, NSFUpgradeService(2.1.0.126)... Done in 0 seconds.
- Data upgrade step 7/100, NetworkAccessUpgrade(2.1.0.127)... Done in 1 seconds.
- Data upgrade step 8/100, ProfilerUpgradeService(2.1.0.134)... Done in 0 seconds.
- Data upgrade step 9/100, ProfilerUpgradeService(2.1.0.139)... Done in 0 seconds.
- Data upgrade step 10/100, ProfilerUpgradeService(2.1.0.166)... ................- Failed
Final cleanup before exiting...
Collecting log files ...
- Encrypting logs bundle...
Please enter encryption password:
Please enter encryption password again to verify:
Encrypted URT logs(urt_logs.tar.gpg) are available in localdisk. Please reach out to Cisco TAC to debug
% Post-install step failed. Please check the logs for more details.
In the generated log file, I can see an OutOfMemoryError exception:
Instance details com.cisco.profiler.upgrade.ProfilerUpgradeService@27df971b
ProfilerUpgradeService.upgrade for version 2.1.0.166
java.lang.OutOfMemoryError: GC overhead limit exceeded
Dumping heap to /var/tmp/java_pid25697.hprof ...
Heap dump file created [2017383231 bytes in 18.479 secs]
Terminated with exception java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOfRange(Arrays.java:3664)
at java.lang.String.<init>(String.java:207)
at oracle.jdbc.driver.CharCommonAccessor.getString(CharCommonAccessor.java:463)
at oracle.jdbc.driver.T4CVarcharAccessor.getString(T4CVarcharAccessor.java:464)
at oracle.jdbc.driver.OracleResultSetImpl.getString(OracleResultSetImpl.java:1297)
at org.apache.commons.dbcp.DelegatingResultSet.getString(DelegatingResultSet.java:213)
at org.apache.commons.dbcp.DelegatingResultSet.getString(DelegatingResultSet.java:213)
at com.cisco.epm.pap.api.services.persistance.dao.ResourceDAO.searchEntities(ResourceDAO.java:6922)
at com.cisco.epm.pap.api.services.server.resource.ResourceImpl.searchEntities(ResourceImpl.java:2016)
at com.cisco.cpm.nsf.impl.NSFGroup.load(NSFGroup.java:212)
at com.cisco.cpm.nsf.impl.NSFGroup.load(NSFGroup.java:158)
at com.cisco.cpm.nsf.impl.policy.NSFPolicyImpl.loadAssociatedSubjects(NSFPolicyImpl.java:2171)
at com.cisco.cpm.nsf.impl.policy.NSFPolicyImpl.loadPolicy(NSFPolicyImpl.java:1841)
at com.cisco.cpm.nsf.impl.policy.NSFPolicyImpl.<init>(NSFPolicyImpl.java:233)
at com.cisco.cpm.nsf.impl.policy.NSFPolicyImpl.retrieveAll(NSFPolicyImpl.java:1600)
at com.cisco.profiler.api.EndpointPolicyHandler.getAll(EndpointPolicyHandler.java:238)
at com.cisco.profiler.api.LogicalProfileImporter.<init>(LogicalProfileImporter.java:48)
at com.cisco.profiler.api.ProfilerRegistration.defaultLogicalProfiles(ProfilerRegistration.java:703)
at com.cisco.profiler.upgrade.DefaultLogicalProfileUpgrade.execUpgrade(DefaultLogicalProfileUpgrade.java:17)
at com.cisco.profiler.upgrade.ProfilerUpgradeService.upgrade(ProfilerUpgradeService.java:107)
at com.cisco.cpm.infrastructure.upgrade.impl.UpgradeServiceRegistrar.UpgradeServices(UpgradeServiceRegistrar.java:132)
at com.cisco.cpm.infrastructure.upgrade.impl.UpgradeServiceRegistrar.main(UpgradeServiceRegistrar.java:185)
ERROR! isedataupgrade.sh FAILED. ISE GLOBAL DATA UPGRADE FAILED
The ISE SAN node had 16G memory on first URT run. Thought that may not be enough. Increased it to 24G and still, URT fails at the exact same point. "show inventory" and "show memory" confirms that additional memory is all available.
Seems I am also hitting bug CSCvh13873. Open a Cisco TAC case ?
ISE 2.0 patch 5, 2 nodes deployment.
Regards,
Rick.
