PCD: Migration failed from CUCM 7.1.5 to CUCM 11.0

[Laith Ibrahim]

Dears,

I am in a project to migrate Cisco CUCM 7.1.5 to version 11.0,

as recommended from Cisco, i used PCD version 11 in the migration process, but the migration failed,

i tried the same process with PCD version 10.5 and same error happened,

Here are the PCD logs:

2016-02-26 21:50:32,019 INFO  [DefaultQuartzScheduler_Worker-2] inventory.Migration.dataExport - installCop return cmdOutput
2016-02-26 21:50:32,019 INFO  [DefaultQuartzScheduler_Worker-2] inventory.Migration.dataExport - return 134
2016-02-26 21:50:32,019 INFO  [DefaultQuartzScheduler_Worker-2] inventory.Migration.nodeExport - export errorCode 134

options: q=quit, n=next, p=prev, b=begin, e=end (lines 2221 - 2240 of 2337) :
2016-02-26 21:50:32,019 INFO  [DefaultQuartzScheduler_Worker-2] inventory.Migration.nodeExport - Return nodeExport result 134
2016-02-26 21:50:32,020 INFO  [DefaultQuartzScheduler_Worker-2] scheduler.ExportJob.executeJob - node 73 export failed with errorcode 134
2016-02-26 21:50:32,020 INFO  [DefaultQuartzScheduler_Worker-2] scheduler.CustomJob.acquireLock - T122:TA123:N73 waiting for DB lock
2016-02-26 21:50:32,020 INFO  [DefaultQuartzScheduler_Worker-2] scheduler.CustomJob.acquireLock - T122:TA123:N73 acquired lock
2016-02-26 21:50:32,025 INFO  [DefaultQuartzScheduler_Worker-2] scheduler.Scheduler.addStatusMessage - Adding ERROR message [migration_errorCode_134] to task action 123
2016-02-26 21:50:32,030 INFO  [DefaultQuartzScheduler_Worker-2] db.EntityBroker.createStatusMessage - Created 147
2016-02-26 21:50:32,031 INFO  [DefaultQuartzScheduler_Worker-2] db.EntityBroker.updateTaskAction - Updated 123 to v9
2016-02-26 21:50:32,031 INFO  [DefaultQuartzScheduler_Worker-2] scheduler.CustomJob.releaseLock - T122:TA123:N73 releasing DB lock
2016-02-26 21:50:32,031 INFO  [DefaultQuartzScheduler_Worker-2] scheduler.CustomJob.acquireLock - T122:TA123:N73 waiting for DB lock
2016-02-26 21:50:32,031 INFO  [DefaultQuartzScheduler_Worker-2] scheduler.CustomJob.acquireLock - T122:TA123:N73 acquired lock
2016-02-26 21:50:32,031 INFO  [DefaultQuartzScheduler_Worker-2] scheduler.Scheduler.isJobActive - Searching for job T122:TA123:N73
2016-02-26 21:50:32,031 INFO  [DefaultQuartzScheduler_Worker-2] scheduler.Scheduler.isJobActive - Job T122:TA123:N73 active = true
2016-02-26 21:50:32,031 INFO  [DefaultQuartzScheduler_Worker-2] scheduler.CustomJob.setNodeStatus - Updating node 73's status to FAILED
2016-02-26 21:50:32,044 INFO  [DefaultQuartzScheduler_Worker-2] scheduler.CustomJob.findNodeStatus - Looking for hal-ucm-jed-p1 or 10.1.1.71
2016-02-26 21:50:32,044 INFO  [DefaultQuartzScheduler_Worker-2] scheduler.CustomJob.findNodeStatus - Comparing hal-ucm-jed-p1
2016-02-26 21:50:32,044 INFO  [DefaultQuartzScheduler_Worker-2] db.EntityBroker.updateNodeStatus - Updated 128 to v3
2016-02-26 21:50:32,048 INFO  [DefaultQuartzScheduler_Worker-2] scheduler.CustomJob.finishJob - Node 73 successful = false
2016-02-26 21:50:32,050 INFO  [DefaultQuartzScheduler_Worker-2] scheduler.Scheduler.addStatusMessage - Adding ERROR message [scheduler.node.fail] to task action 123
2016-02-26 21:50:32,062 INFO  [DefaultQuartzScheduler_Worker-2] db.EntityBroker.createStatusMessage - Created 150
2016-02-26 21:50:32,063 INFO  [DefaultQuartzScheduler_Worker-2] db.EntityBroker.updateTaskAction - Updated 123 to v10

[Aman Soi]

Hi,

the message points to DB lock.

Did u check the DB replication on CUCM 7.1.5?

regds,

aman

[Laith Ibrahim]

Dear Aman,

I have only a PUB on the cluster, there is no SUB,

There was before a SUB and i deleted it from the cluster,

Could this cause a DB lock ?

[Aman Soi]

It is hard to say.

What does the show version active of CUCM 7.x show?

regds,

aman

[Laith Ibrahim]

It says that there is no db replication because it is a single node cluster.

[Aman Soi]

Hi,

Did u check the DB replication on CUCM 7.1.5 ?

regds,

aman

[Jaime Valencia]

FYI if you review the PCD documentation, you'll notice that PCD 10.5 CANNOT upgrade/migrate to 11.x releases, you need PCD 11 for that.

Code 134 means: Data export failed for node {0}.

That can be for many reasons, from simply it taking too long, and it might work on the next try, or some bugs which require root access to be fixed, I'd suggest you open a TAC for assistance with that. I even found some cases in which the problem was a parameter in CAR mail config using a / and it not being parsed correctly.

[Laith Ibrahim]

Dear Jaime,

Thank you for your response,

Yeah I know about PCD 10.5, i used it to migrate to CUCM 10.5,

So for this case there is too many reasons to determine the cause issue.

[Laith Ibrahim]

I noticed these errors on the server CLI:

Message from syslogd@HAL-UCM-JED-P1 at Mon Feb 29 10:07:04 2016 ...
HAL-UCM-JED-P1 local7 0 : 969: Feb 29 07:00:00.49 UTC :  %CCM_UNKNOWN-CERT-0-CertExpiryEmergency: Certificate Expiry EMERGENCY_ALARM  Message:Certificate expiration Notification. Certificate name:tomcat Unit:tomcat Type:own-cert Expiration:Sun Nov 8 19:51:18:000 GMT+03:0 App ID:Cisco Certificate Monitor Cluster ID: Node ID:HAL-UCM-JED-P1

Message from syslogd@HAL-UCM-JED-P1 at Mon Feb 29 10:07:04 2016 ...
HAL-UCM-JED-P1 local7 0 : 970: Feb 29 07:00:00.49 UTC :  %CCM_UNKNOWN-CERT-0-CertExpiryEmergency: Certificate Expiry EMERGENCY_ALARM  Message:Certificate expiration Notification. Certificate name:ipsec Unit:ipsec Type:own-cert Expiration:Sun Nov 8 19:51:21:000 GMT+03:07  App ID:Cisco Certificate Monitor Cluster ID: Node ID:HAL-UCM-JED-P1

Message from syslogd@HAL-UCM-JED-P1 at Mon Feb 29 10:07:04 2016 ...
HAL-UCM-JED-P1 local7 0 : 971: Feb 29 07:00:00.50 UTC :  %CCM_UNKNOWN-CERT-0-CertExpiryEmergency: Certificate Expiry EMERGENCY_ALARM  Message:Certificate expiration Notification. Certificate name:CallManager Unit:CallManager Type:own-cert Expiration:Sun Nov 8 19:51:23:00 App ID:Cisco Certificate Monitor Cluster ID: Node ID:HAL-UCM-JED-P1

Message from syslogd@HAL-UCM-JED-P1 at Mon Feb 29 10:07:04 2016 ...
HAL-UCM-JED-P1 local7 0 : 972: Feb 29 07:00:00.50 UTC :  %CCM_UNKNOWN-CERT-0-CertExpiryEmergency: Certificate Expiry EMERGENCY_ALARM  Message:Certificate expiration Notification. Certificate name:CAPF Unit:CAPF Type:own-cert Expiration:Sun Nov 8 19:51:25:000 GMT+03:07 20 App ID:Cisco Certificate Monitor Cluster ID: Node ID:HAL-UCM-JED-P1

Message from syslogd@HAL-UCM-JED-P1 at Mon Feb 29 10:07:04 2016 ...
HAL-UCM-JED-P1 local7 0 : 973: Feb 29 07:00:00.51 UTC :  %CCM_UNKNOWN-CERT-0-CertExpiryEmergency: Certificate Expiry EMERGENCY_ALARM  Message:Certificate expiration Notification. Certificate name:CAPF-6ffef25a Unit:CallManager-trust Type:trust-cert Expiration:Sun Nov 8 1 App ID:Cisco Certificate Monitor Cluster ID: Node ID:HAL-UCM-JED-P1

Message from syslogd@HAL-UCM-JED-P1 at Mon Feb 29 10:07:04 2016 ...
HAL-UCM-JED-P1 local7 0 : 974: Feb 29 07:00:00.51 UTC :  %CCM_UNKNOWN-CERT-0-CertExpiryEmergency: Certificate Expiry EMERGENCY_ALARM  Message:Certificate expiration Notification. Certificate name:CAPF-02d0cc0a Unit:CallManager-trust Type:trust-cert Expiration:Sun Nov 8 1 App ID:Cisco Certificate Monitor Cluster ID: Node ID:HAL-UCM-JED-P1

Message from syslogd@HAL-UCM-JED-P1 at Mon Feb 29 10:07:04 2016 ...
HAL-UCM-JED-P1 local7 0 : 975: Feb 29 07:00:00.52 UTC :  %CCM_UNKNOWN-CERT-0-CertExpiryEmergency: Certificate Expiry EMERGENCY_ALARM  Message:Certificate expiration Notification. Certificate name:CAPF-e457ff9f Unit:CallManager-trust Type:trust-cert Expiration:Sun Nov 8 1 App ID:Cisco Certificate Monitor Cluster ID: Node ID:HAL-UCM-JED-P1

Message from syslogd@HAL-UCM-JED-P1 at Mon Feb 29 10:07:04 2016 ...
HAL-UCM-JED-P1 local7 0 : 976: Feb 29 07:00:00.52 UTC :  %CCM_UNKNOWN-CERT-0-CertExpiryEmergency: Certificate Expiry EMERGENCY_ALARM  Message:Certificate expiration Notification. Certificate name:CAPF-78ea5415 Unit:CallManager-trust Type:trust-cert Expiration:Sat Dec 13  App ID:Cisco Certificate Monitor Cluster ID: Node ID:HAL-UCM-JED-P1

Message from syslogd@HAL-UCM-JED-P1 at Mon Feb 29 10:07:04 2016 ...
HAL-UCM-JED-P1 local7 0 : 977: Feb 29 07:00:00.53 UTC :  %CCM_UNKNOWN-CERT-0-CertExpiryEmergency: Certificate Expiry EMERGENCY_ALARM  Message:Certificate expiration Notification. Certificate name:CAPF-5fc84a2a Unit:CallManager-trust Type:trust-cert Expiration:Wed Jan 21  App ID:Cisco Certificate Monitor Cluster ID: Node ID:HAL-UCM-JED-P1

Message from syslogd@HAL-UCM-JED-P1 at Mon Feb 29 10:07:04 2016 ...
HAL-UCM-JED-P1 local7 0 : 978: Feb 29 07:00:00.53 UTC :  %CCM_UNKNOWN-CERT-0-CertExpiryEmergency: Certificate Expiry EMERGENCY_ALARM  Message:Certificate expiration Notification. Certificate name:CAPF-f43411d8 Unit:CallManager-trust Type:trust-cert Expiration:Fri Nov 6 2 App ID:Cisco Certificate Monitor Cluster ID: Node ID:HAL-UCM-JED-P1

Message from syslogd@HAL-UCM-JED-P1 at Mon Feb 29 10:07:04 2016 ...
HAL-UCM-JED-P1 local7 0 : 979: Feb 29 07:00:00.54 UTC :  %CCM_UNKNOWN-CERT-0-CertExpiryEmergency: Certificate Expiry EMERGENCY_ALARM  Message:Certificate expiration Notification. Certificate name:CAPF-c09aba6c Unit:CallManager-trust Type:trust-cert Expiration:Fri Nov 6 2 App ID:Cisco Certificate Monitor Cluster ID: Node ID:HAL-UCM-JED-P1

Message from syslogd@HAL-UCM-JED-P1 at Mon Feb 29 10:07:04 2016 ...
HAL-UCM-JED-P1 local7 0 : 980: Feb 29 07:00:00.54 UTC :  %CCM_UNKNOWN-CERT-0-CertExpiryEmergency: Certificate Expiry EMERGENCY_ALARM  Message:Certificate expiration Notification. Certificate name:CAPF-6ffef25a Unit:CAPF-trust Type:trust-cert Expiration:Sun Nov 8 19:09:01 App ID:Cisco Certificate Monitor Cluster ID: Node ID:HAL-UCM-JED-P1

Message from syslogd@HAL-UCM-JED-P1 at Mon Feb 29 10:07:04 2016 ...
HAL-UCM-JED-P1 local7 0 : 981: Feb 29 07:00:00.55 UTC :  %CCM_UNKNOWN-CERT-0-CertExpiryEmergency: Certificate Expiry EMERGENCY_ALARM  Message:Certificate expiration Notification. Certificate name:CAPF-02d0cc0a Unit:CAPF-trust Type:trust-cert Expiration:Sun Nov 8 18:45:26 App ID:Cisco Certificate Monitor Cluster ID: Node ID:HAL-UCM-JED-P1

Message from syslogd@HAL-UCM-JED-P1 at Mon Feb 29 10:07:04 2016 ...
HAL-UCM-JED-P1 local7 0 : 982: Feb 29 07:00:00.55 UTC :  %CCM_UNKNOWN-CERT-0-CertExpiryEmergency: Certificate Expiry EMERGENCY_ALARM  Message:Certificate expiration Notification. Certificate name:CAPF-e457ff9f Unit:CAPF-trust Type:trust-cert Expiration:Sun Nov 8 19:51:25 App ID:Cisco Certificate Monitor Cluster ID: Node ID:HAL-UCM-JED-P1

Message from syslogd@HAL-UCM-JED-P1 at Mon Feb 29 10:07:04 2016 ...
HAL-UCM-JED-P1 local7 0 : 983: Feb 29 07:00:00.56 UTC :  %CCM_UNKNOWN-CERT-0-CertExpiryEmergency: Certificate Expiry EMERGENCY_ALARM  Message:Certificate expiration Notification. Certificate name:CAPF-78ea5415 Unit:CAPF-trust Type:trust-cert Expiration:Sat Dec 13 22:52:1 App ID:Cisco Certificate Monitor Cluster ID: Node ID:HAL-UCM-JED-P1

Message from syslogd@HAL-UCM-JED-P1 at Mon Feb 29 10:07:04 2016 ...
HAL-UCM-JED-P1 local7 0 : 984: Feb 29 07:00:00.56 UTC :  %CCM_UNKNOWN-CERT-0-CertExpiryEmergency: Certificate Expiry EMERGENCY_ALARM  Message:Certificate expiration Notification. Certificate name:CAPF-f43411d8 Unit:CAPF-trust Type:trust-cert Expiration:Fri Nov 6 23:34:40 App ID:Cisco Certificate Monitor Cluster ID: Node ID:HAL-UCM-JED-P1

Message from syslogd@HAL-UCM-JED-P1 at Mon Feb 29 10:07:04 2016 ...
HAL-UCM-JED-P1 local7 0 : 985: Feb 29 07:00:00.57 UTC :  %CCM_UNKNOWN-CERT-0-CertExpiryEmergency: Certificate Expiry EMERGENCY_ALARM  Message:Certificate expiration Notification. Certificate name:CAPF-c09aba6c Unit:CAPF-trust Type:trust-cert Expiration:Fri Nov 6 23:55:17 App ID:Cisco Certificate Monitor Cluster ID: Node ID:HAL-UCM-JED-P1

Could these be the reason why the migration is failing?

[Aman Soi]

Hi Ibrahim,

Could be related to this bug

https://tools.cisco.com/bugsearch/bug/CSCun76709

PCD export migration warning copying uc platform post failed

CSCun76709

Description

Symptom:
The Prime Collaboration Deployment (PCD) export migration step fails.

Conditions:
PCD 10.0.1.10000-14

The failing Cisco Unified Communications Manager (CUCM) migration log displays the following:

# file list install / date detail
# file view install /migrate_export_
03/15/2014 14:35:39 migrate_export|Exporting install log files|<:INFO>
03/15/2014 14:35:42 migrate_export|platformConfig.xml export successful.|<:INFO>
03/15/2014 14:35:42 migrate_export|warning Copying uc platform post failed.|<:INFO>
03/15/2014 14:35:42 migrate_export|Acquired LOCK file: /var/lock/update_migrate_state.lock for PID: 12638|<:INFO>
03/15/2014 14:35:43 migrate_export|Updated status file: SYSTEM_MIGRATE_EXPORT_STATUS=TERMINATED ()|<:INFO>
03/15/2014 14:35:43 migrate_export|Updated status file: MSG=System migrate export failed while creating the requried post platform configuration directory. ()|<:INFO>
03/15/2014 14:35:43 migrate_export|Removed LOCK file: /var/lock/update_migrate_state.lock for PID: 12638|<:INFO>
03/15/2014 14:35:43 migrate_export|Perform cleanup|<:INFO>
03/15/2014 14:35:43 migrate_export|Failed to remove the PID file () for PID: 12638|<:ERROR>
03/15/2014 14:35:43 migrate_export|Removed LOCK file: /var/lock/migrate_export.lock for PID: 12638|<:INFO>
03/15/2014 14:35:43 migrate_export|Removed PID file: /var/run/migrate_export.pid for PID: 12638|<:INFO>
03/15/2014 14:35:43 migrate_export|Updated status file: SCRIPT_STOP_TIME=Sat Mar 15 14:35:43 2014 ()|<:INFO>
03/15/2014 14:35:43 migrate_export|Contents of the status file:|<:INFO>
03/15/2014 14:35:43 migrate_export|MIGRATE_EXPORT_TYPE=STATIC
SYSTEM_MIGRATE_EXPORT_STATUS=TERMINATED
PID=12638
LOG_FILE=/var/log/install/migrate_export_2014-03-15-14-29-38.log
SCRIPT_START_TIME=Sat-Mar-15-14:29:38-2014
SCRIPT_STOP_TIME=Sat-Mar-15-14:35:43-2014
MSG=System-migrate-export-failed-while-creating-the-requried-post-platform-configuration-directory.
SERVER=192.168.1.2
DIRECTORY=/export/
USER=adminsftp|<:INFO>


The PCD logs display the following:

##file list activelog tomcat/logs/ucmap/log4j date detail
##file view activelog tomcat/logs/ucmap/log4j/
2014-03-12 16:45:35,216 ERROR [http-bio-443-exec-16] scheduler.SchedulerException. - SchedulerException: scheduler.task.notfound
2014-03-12 16:45:35,216 ERROR [http-bio-443-exec-16] resources.CancelTaskHandler.cancelTask - com.cisco.ucmap.scheduler.SchedulerException: scheduler.task.notfound
2014-03-12 16:45:41,265 ERROR [pool-1-thread-3] custom.PreDiscoveryTaskCreate.firstTimeDiscovery - getException
2014-03-15 13:45:20,874 INFO [DefaultQuartzScheduler_Worker-4] scheduler.Scheduler.addStatusMessage - Adding ERROR message [migration_errorCode_134] to task action 202
2014-03-15 13:45:20,937 INFO [DefaultQuartzScheduler_Worker-4] scheduler.Scheduler.addStatusMessage - Adding ERROR message [scheduler.node.fail] to task action 202
2014-03-15 13:45:21,038 INFO [DefaultQuartzScheduler_Worker-4] scheduler.Scheduler.addStatusMessage - Adding ERROR message [scheduler.taskaction.failednodes] to task action 202
2014-03-15 13:45:21,134 INFO [DefaultQuartzScheduler_Worker-4] scheduler.Scheduler.addStatusMessage - Adding ERROR message [scheduler.task.pause] to task 201
2014-03-15 13:45:21,645 INFO [DefaultQuartzScheduler_Worker-4] scheduler.Scheduler.updateTaskStatus - task 201 set to PAUSE_ERROR
2014-03-15 13:47:36,724 INFO [http-bio-443-exec-12] scheduler.Scheduler.scheduleTask - task 201 status = PAUSE_ERROR
2014-03-15 13:49:23,728 ERROR [DefaultQuartzScheduler_Worker-5] db.EntityBroker.getNotifications - Query exception
2014-03-15 13:49:23,731 ERROR [DefaultQuartzScheduler_Worker-5] scheduler.Scheduler.addStatusMessage - Email notification for task 250 was not sent
2014-03-15 14:03:43,562 ERROR [DefaultQuartzScheduler_Worker-6] db.EntityBroker.getMigrationTask - Query exception finding 201
2014-03-15 14:03:43,569 ERROR [DefaultQuartzScheduler_Worker-6] scheduler.ExportJob.createFloppyImage - Failed to get migration map or add status
2014-03-15 14:17:02,931 INFO [DefaultQuartzScheduler_Worker-7] scheduler.Scheduler.addStatusMessage - Adding ERROR message [migration_errorCode_134] to task action 427
2014-03-15 14:17:02,989 INFO [DefaultQuartzScheduler_Worker-7] scheduler.Scheduler.addStatusMessage - Adding ERROR message [scheduler.node.fail] to task action 427
2014-03-15 14:17:03,094 INFO [DefaultQuartzScheduler_Worker-7] scheduler.Scheduler.addStatusMessage - Adding ERROR message [scheduler.taskaction.failednodes] to task action 427
2014-03-15 14:17:03,205 INFO [DefaultQuartzScheduler_Worker-7] scheduler.Scheduler.addStatusMessage - Adding ERROR message [scheduler.task.pause] to task 426
2014-03-15 14:17:03,768 INFO [DefaultQuartzScheduler_Worker-7] scheduler.Scheduler.updateTaskStatus - task 426 set to PAUSE_ERROR
2014-03-15 14:18:55,709 INFO [http-bio-443-exec-9] scheduler.Scheduler.scheduleTask - task 426 status = PAUSE_ERROR
2014-03-15 14:35:46,949 INFO [DefaultQuartzScheduler_Worker-9] scheduler.Scheduler.addStatusMessage - Adding ERROR message [migration_errorCode_134] to task action 427
2014-03-15 14:35:47,029 INFO [DefaultQuartzScheduler_Worker-9] scheduler.Scheduler.addStatusMessage - Adding ERROR message [scheduler.node.fail] to task action 427
2014-03-15 14:35:47,141 INFO [DefaultQuartzScheduler_Worker-9] scheduler.Scheduler.addStatusMessage - Adding ERROR message [scheduler.taskaction.failednodes] to task action 427
2014-03-15 14:35:47,224 INFO [DefaultQuartzScheduler_Worker-9] scheduler.Scheduler.addStatusMessage - Adding ERROR message [scheduler.task.pause] to task 426
2014-03-15 14:35:47,768 INFO [DefaultQuartzScheduler_Worker-9] scheduler.Scheduler.updateTaskStatus - task 426 set to PAUSE_ERROR

Workaround:
The underlying permissions of some of the certificates cause the failure. Open a service request with the Cisco TAC to have them fix the file permissions

suggest opening TAC case.

regds,

aman

[Laith Ibrahim]

I opened a Cisco TAC case,

After troubleshooting the migration logs, it turns out that the problem is the timezone,

The CUCM 7 has some timezones that doesn't exist in CUCM 11, so you have to make sure that the timezone exists in it to avoid this error,

Thanks.

[Aman Soi · Accepted solution]

HI Laith,

thanks for update[+5]

are u referring this bug  even though versions are different.?

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCup50058

Migration issues from CUCM 6.x to 10.x due to timezones changes

CSCup50058

Description

Symptom:
Migration from CUCM 6.x to CUCM 10.x fails using PCD showing an error regarding timezone

"invalid Timezone US pacific"

Conditions:
CUCM 6.x to CUCM 10.x migration

Any servers in the CUCM 6.x cluster is using one of the following timezones:

regds,

aman