topic Re: Upgrade to ise 2.3 failed in Network Access Control

Upgrade to ise 2.3 failed

alyautdinov — Tue, 08 Aug 2017 15:20:40 GMT

Hello team.

We tryed to update ISE 2.2 to 2.3 and got such error:

UPGRADE STEP 1: Running ISE configuration database schema upgrade...
- Running db sanity to check and fix if any index corruption
- Auto Upgrading Schema for UPS Model
- Upgrading Schema completed for UPS Model
UPGRADE STEP 2: Running ISE configuration data upgrade...
- Data upgrade step 1/18, UPSUpgradeHandler(2.3.0.100)... .Failed.
% Error: ISE Global data upgrade failed!

After, we installed 2.3 from stratch version and tryed to restore configuration backup from ise 2.2 but still same error

Logs from dbupgrade-data-global-.log

Retrived the data from Handlercom.cisco.cpm.ups.upgrade.impl.PolicyUpgradeHandler]

com.cisco.cpm.infrastructure.upgrade.api.UpgradeFailureException: java.lang.NullPointerException

at com.cisco.cpm.ups.upgrade.UpgradeHandler.exportAndImport(UpgradeHandler.java:41)

at com.cisco.cpm.ups.upgrade.UpgradeHandler.execUpgrade(UpgradeHandler.java:29)

at com.cisco.cpm.ups.upgrade.impl.UPSUpgradeHandler.upgrade(UPSUpgradeHandler.java:151)

at com.cisco.cpm.infrastructure.upgrade.impl.UpgradeServiceRegistrar.UpgradeServices(UpgradeServiceRegistrar.java:132)

at com.cisco.cpm.infrastructure.upgrade.impl.UpgradeServiceRegistrar.main(UpgradeServiceRegistrar.java:185)

Caused by: java.lang.NullPointerException

at com.cisco.cpm.policy.configuration.upgrade.builder.AbstractUpgradePolicyDataBuilder.buildAuthenticationRuleResultDataForOuterDefaultRule(AbstractUpgradePolicyDataBuilder.java:284)

at com.cisco.cpm.policy.configuration.upgrade.builder.AbstractUpgradePolicyDataBuilder.buildAuthenticationInnerRules(AbstractUpgradePolicyDataBuilder.java:182)

at com.cisco.cpm.policy.configuration.upgrade.builder.UpgradePolicyDataBuilderRadius.buildUpgradeData(UpgradePolicyDataBuilderRadius.java:99)

at com.cisco.cpm.policy.configuration.upgrade.PolicyUpgrade.upgradeLegacySetRadius(PolicyUpgrade.java:394)

at com.cisco.cpm.policy.configuration.upgrade.PolicyUpgrade.upgradeLegacySet(PolicyUpgrade.java:337)

at com.cisco.cpm.policy.configuration.upgrade.PolicyUpgrade.upgradeLegacySets(PolicyUpgrade.java:213)

at com.cisco.cpm.ups.upgrade.impl.PolicyUpgradeHandler.importData(PolicyUpgradeHandler.java:67)

at com.cisco.cpm.ups.upgrade.UpgradeHandler.exportAndImport(UpgradeHandler.java:38)

... 4 more

Error while applying changes in version: 2.3.0.100 class: com.cisco.cpm.ups.upgrade.impl.UPSUpgradeHandler

com.cisco.cpm.infrastructure.upgrade.api.UpgradeFailureException: Failed to upgrade to version 2.3.0.100: java.lang.NullPointerException

at com.cisco.cpm.ups.upgrade.impl.UPSUpgradeHandler.upgrade(UPSUpgradeHandler.java:159)

at com.cisco.cpm.infrastructure.upgrade.impl.UpgradeServiceRegistrar.UpgradeServices(UpgradeServiceRegistrar.java:132)

at com.cisco.cpm.infrastructure.upgrade.impl.UpgradeServiceRegistrar.main(UpgradeServiceRegistrar.java:185)

ERROR! isedataupgrade.sh FAILED. ISE GLOBAL DATA UPGRADE FAILED

What is the reason?

Re: Upgrade to ise 2.3 failed

scamarda — Tue, 08 Aug 2017 15:28:48 GMT

I had the same issue with my 2 node 2.2P2 setup. Same exact error on both nodes. I am in the process of creating new 2.3 machines as well.

Re: Upgrade to ise 2.3 failed

Timothy Abbott — Tue, 08 Aug 2017 16:57:37 GMT

Hi,

Please work with TAC to find out why the upgrade failed.

Regards,

-Tim

Re: Upgrade to ise 2.3 failed

alyautdinov — Tue, 08 Aug 2017 17:40:19 GMT

This is test environment. Lab version without license.

Re: Upgrade to ise 2.3 failed

hslai — Tue, 08 Aug 2017 18:14:53 GMT

Sam's failure has a different text so it's not the same as your failure.

Re: Upgrade to ise 2.3 failed

paul — Tue, 08 Aug 2017 19:05:25 GMT

Rebuild with new version and restore backup is the best way to upgrade an ISE deploiyment. I have done 50+ this way. Every time I have tried either the CLI ugprade process something has blown up. The rebuild/restore method is very predictable and offers maximum control over the upgrade process.

What you you are finding out is that when the automated process blows up you are going to spend days trying to fix it or investigate what went wrong when the whole process could have been down in a few hours using rebuild/restore method.

Re: Upgrade to ise 2.3 failed

Jason Kunst — Tue, 08 Aug 2017 19:12:28 GMT

Have you tried the upgrade readiness tool to see what it says as well?

https://www.cisco.com/c/en/us/td/docs/security/ise/2-3/upgrade_guide/b_ise_upgrade_guide_23/b_ise_upgrade_guide_23_chapter_01.html

Re: Upgrade to ise 2.3 failed

hslai — Tue, 08 Aug 2017 19:13:03 GMT

The failure seems something to do with authentication policy outer rules for Network Access. Could you post the screenshots of your policy sets, if any, and authentication policy rules?

Re: Upgrade to ise 2.3 failed

alyautdinov — Tue, 08 Aug 2017 19:15:09 GMT

We tried to restore backup 2.2 on 2.3 version but have same error:

Re: Upgrade to ise 2.3 failed

alyautdinov — Tue, 08 Aug 2017 19:16:40 GMT

Yes we tried with URT:

com.cisco.cpm.ups.upgrade.impl.PolicyUpgradeHandler.importData(PolicyUpgradeHandler.java:67)

at com.cisco.cpm.ups.upgrade.UpgradeHandler.exportAndImport(UpgradeHandler.java:38)

... 4 more

Error while applying changes in version: 2.3.0.100 class: com.cisco.cpm.ups.upgrade.impl.UPSUpgradeHandler

com.cisco.cpm.infrastructure.upgrade.api.UpgradeFailureException: Failed to upgrade to version 2.3.0.100: java.lang.NullPointerException

at com.cisco.cpm.ups.upgrade.impl.UPSUpgradeHandler.upgrade(UPSUpgradeHandler.java:159)

at com.cisco.cpm.infrastructure.upgrade.impl.UpgradeServiceRegistrar.UpgradeServices(UpgradeServiceRegistrar.java:132)

at com.cisco.cpm.infrastructure.upgrade.impl.UpgradeServiceRegistrar.main(UpgradeServiceRegistrar.java:185)

ERROR! isedataupgrade.sh FAILED. ISE GLOBAL DATA UPGRADE FAILED

Re: Upgrade to ise 2.3 failed

hslai — Wed, 09 Aug 2017 04:57:45 GMT

Please let us know whether you may provide the CFG backup for more investigation.

Re: Upgrade to ise 2.3 failed

hslai — Sat, 12 Aug 2017 05:50:37 GMT

I am still waiting on our dev team's analysis, but I found two issues:

In the RADIUS policy sets, the Easy Connect policy set has the same condition for the policy set itself and for the non-default authentication policy rule MAB. And, the "Default Rule (if no match)" will never match. After combining the two rules into one -- the "Default Rule (if no match)" to use Default Network Access as the allowed protocols and Internal Endpoints as the ID source, URT able to complete the sanity tests for the RADIUS policy sets.

In the T+ policy sets, the Test_admin one has the condition "Network Access:Protocol EQUALS TACACS+". This is odd because it's always true for T+ auth. After merging it into one -- the "Default Rule (if no match)" to use Default Device Admin as the allowed protocols and PC_ISE_Ebusiness as the ID source, URT able to complete the sanity for T+ policy sets.

After these two edits, URT completed successfully.

Re: Upgrade to ise 2.3 failed

alyautdinov — Mon, 14 Aug 2017 11:19:48 GMT

Hi,

Unfortunately, URT failed for PS:Checkpoint.

For PS:Easyconnect Test migration was succesful:

@@@ PsUpgrade: info- :***** Upgrade process for the legacy PS:Easyconnect Test was finished with the result:PolicyUpgradeResult status:SUCESS...Hooray! Policy Id:84438d00-80cd-11e7-b4bf-02427242cd9c Policy Name:Easyconnect Test

Full dbupgrade-data.log for PS:Checkpoint:

@@@ PsUpgrade: info- :*** Starting an upgrade process for the Radius legacy PS:Checkpoint

@@@ PsUpgrade: debug- :Build PS level condition for PS: Checkpoint

@@@ PsUpgrade: debug- :About to get condition RHS display value for Network Access with attribute Protocol

@@@ PsUpgrade: debug- :Network Access:Protocol has allow values enumeration

@@@ PsUpgrade: debug- : Found allow value for Network Access:Protocol0:RADIUS

@@@ PsUpgrade: warn- :Couldn't buildConditionDataForNameValue for: lhsAttrId:DEVICE.Migrated_NDGs rhsString:Migrated_NDGs#All Migrated_NDGs#CheckPoint#CP_TEST, Will try to build it from rhs value

com.cisco.cpm.policy.pal.PalException: Value for attribute is not a permitted option

at com.cisco.cpm.policy.pal.policyCondition.ConditionsData.validateAllowedValues(ConditionsData.java:510)

at com.cisco.cpm.policy.pal.policyCondition.ConditionsData.initSimple(ConditionsData.java:425)

at com.cisco.cpm.policy.pal.policyCondition.ConditionsData.<init>(ConditionsData.java:290)

at com.cisco.cpm.policy.configuration.upgrade.PolicyUpgradeUtil.buildConditionDataForNameValue(PolicyUpgradeUtil.java:947)

at com.cisco.cpm.policy.configuration.upgrade.builder.UpgradeNetAccessRuleBuilder.buildConditionDataClauseSimple(UpgradeNetAccessRuleBuilder.java:139)

at com.cisco.cpm.policy.configuration.upgrade.builder.UpgradeNetAccessRuleBuilder.buildConditionDataClauses(UpgradeNetAccessRuleBuilder.java:99)

at com.cisco.cpm.policy.configuration.upgrade.builder.UpgradeNetAccessRuleBuilder.buildRuleConditionData(UpgradeNetAccessRuleBuilder.java:70)

at com.cisco.cpm.policy.configuration.upgrade.builder.AbstractUpgradePolicyDataBuilder.buildNetAccessRuleConditionData(AbstractUpgradePolicyDataBuilder.java:78)

at com.cisco.cpm.policy.configuration.upgrade.builder.UpgradePolicyDataBuilderRadius.buildNetAccessRuleConditionData(UpgradePolicyDataBuilderRadius.java:200)

at com.cisco.cpm.policy.configuration.upgrade.builder.AbstractUpgradePolicyDataBuilder.buildPSLevelConditionsData(AbstractUpgradePolicyDataBuilder.java:64)

at com.cisco.cpm.policy.configuration.upgrade.builder.UpgradePolicyDataBuilderRadius.buildUpgradeData(UpgradePolicyDataBuilderRadius.java:76)