UCSPM 2.0 Upgrade Failure

[Paul O'Leary]

Anyone have luck getting the export4 to work in UCSPM 1.1 for upgrading to UCSPM 2.0?

Mine completes but utterly fails to create the correct tar file with the zenoss backup and ancillary (dmduuid, flexera etc) files.

It fails to create even the  4x-export-%date%.tar file in the right place, and leaves the zenbackup_4x.tgz in the temp location.

Here's the output from my attempts

2015-12-18 04:13:08,821 INFO CoreBackup: Backing up ZenPack contents.

2015-12-18 04:13:08,880 INFO CoreBackup: Performing an offline backup of zencatalogservice.

2015-12-18 04:13:08,955 INFO CoreBackup: Backup of zencatalogservice complete.

2015-12-18 04:13:08,964 INFO CoreBackup: Backup of ZenPack contents complete.

2015-12-18 04:13:08,965 INFO zenbackup: Backup of ZenPacks completed in 5 seconds.

2015-12-18 04:13:08,965 INFO CoreBackup: Backing up performance data (RRDs).

2015-12-18 04:26:50,053 INFO zenbackup: Packaging backup file.

2015-12-18 04:35:35,067 INFO zenbackup: Backup written to /tmp/tmpLo5fWW/backups/zenbackup_4x.tgz

2015-12-18 04:35:35,067 INFO zenbackup: Cleaning up staging directory /tmp/tmpLo5fWW/5KOeNr

2015-12-18 04:35:38,290 INFO zenbackup: Backup completed successfully in 22 minutes.

And the directories on the VMDK for the appliance after finished:

./export4

./export4/lost+found

./export4/4x-backup

./export4/4x-backup/tmprDI3xg

./export4/4x-backup/tmprDI3xg/backups

./export4/4x-backup/tmprDI3xg/backups/zenbackup_4x.tgz

./export4/4x-backup/tmprDI3xg/componentList.txt

[jasonsha]

Hi Paul,

Are you following the steps from the UCSPM 2.0 Migration Guide?

- and running from a UCSPM v1.1, or v1.1.1 appliance?

Should be detecting the newly attached VMware Disk, partitioning/formatting/mounting - and then creating the backup on the second disk.

Thx,

Jason

[Paul O'Leary]

5 Or 6 times with the same result - as above.

FInishes and unmounts but no files.

Here's the full log:

[zenoss@ucspm ~]$ export4 -s

Calculating free space needed for backup

Local catalog data estimated to need 4 MB

Local zenpack data estimated to need 113 MB

Estimated zeneventserver database dump size is 4 MB

Estimated zodb database dump size is 38 MB

Estimated zodb session db dump size is 28 MB

Zeneventserver indexes estimated to need 46 MB

Local performance data estimated to need 18821 MB

Total estimated free space needed for export is up to 58 GB

Expected export disk size should be at least: 61 GiB

Running validations

2015-12-19 02:04:20,846 INFO zenoss.toolbox.import4tools.validate4import: Starting validation ZenPackValidation

2015-12-19 02:04:25,446 INFO zenoss.toolbox.import4tools.validations.ZenPackValidation: ZenPack states are valid for upgrade

2015-12-19 02:04:25,452 INFO zenoss.toolbox.import4tools.validate4import: Validation successful

Verify that the export disk is not added to the virtual machine ...

Enter root password when prompted ->

Password:

Add the export disk to the virtual machine now ...

<ENTER> when added, <CTRL+C> to quit ...

New device identified -> /dev/sdb ...

WARNING: Ready to prepare /dev/sdb [100 GiB] for export ...

<ENTER> to continue, <CTRL+C> to quit ...

Preparing disk device /dev/sdb ...

1+0 records in

1+0 records out

512 bytes (512 B) copied, 0.0687251 s, 7.4 kB/s

Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel

Building a new DOS disklabel with disk identifier 0x612e776c.

Changes will remain in memory only, until you decide to write them.

After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to

         switch off the mode (command 'c') and change display units to

         sectors (command 'u').

Command (m for help): Command action

   e   extended

   p   primary partition (1-4)

Partition number (1-4): First cylinder (1-13054, default 1): Using default value 1

Last cylinder, +cylinders or +size{K,M,G} (1-13054, default 13054): Using default value 13054

Command (m for help):

Disk /dev/sdb: 107.4 GB, 107374182400 bytes

255 heads, 63 sectors/track, 13054 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x612e776c

   Device Boot      Start         End      Blocks   Id  System

/dev/sdb1               1       13054   104856223+  83  Linux

Command (m for help): The partition table has been altered!

Calling ioctl() to re-read partition table.

Syncing disks.

mke2fs 1.41.12 (17-May-2010)

Filesystem label=IMPORT4

OS type: Linux

Block size=4096 (log=2)

Fragment size=4096 (log=2)

Stride=0 blocks, Stripe width=0 blocks

6553600 inodes, 26214055 blocks

1310702 blocks (5.00%) reserved for the super user

First data block=0

Maximum filesystem blocks=4294967296

800 block groups

32768 blocks per group, 32768 fragments per group

8192 inodes per group

Superblock backups stored on blocks:

32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,

4096000, 7962624, 11239424, 20480000, 23887872

Writing inode tables:  <SNIP>                         

Creating journal (32768 blocks): done

Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 34 mounts or

180 days, whichever comes first.  Use tune2fs -c or -i to override.

mode of `/mnt/export4' changed to 0777 (rwxrwxrwx)

/mnt/export4 mounted OK.

mode of `/mnt/export4/4x-backup' changed to 0777 (rwxrwxrwx)

Export space prepared...

Checking accessibility of 4x-export-20151219-020416.tar ...

4x-export-20151219-020416.tar is accessible ...

Checking platform ...

UCSPM version 1.1.0 is supported.

Available temporary space of /mnt/export4/4x-backup/tmpUO0vRM: 100 GB.

Available backup space of /mnt/export4: 100 GB.

Getting remote collector information.

Not using distributed collectors.

making new backup ...

All Zenoss services will be stopped to export data for migration.

The services are restarted once the backup completes.

Press ENTER to continue or <CTRL+C> to quit

Daemon: zenucsevents stopping...

Daemon: zenvsphere stopping...

Daemon: zenjserver stopping...

Daemon: zenpython stopping...

Daemon: zencommand stopping...

Daemon: zenperfsnmp stopping...

Daemon: zenmodeler stopping...

Daemon: zenactiond stopping...

Daemon: zenping stopping...

Daemon: zeneventd stopping...

Daemon: zenjobs stopping...

Daemon: zenhub stopping...

Daemon: zredis stopping...

Daemon: zenrrdcached stopping...

Daemon: zenwebserver stopping...

Daemon: zeneventserver stopping...

Daemon: zencatalogservice stopping...

INFO:zenbackup:Getting ZEP dbname, user, password, port from configuration files.

INFO:CoreBackup:Backing up the events database.

INFO:CoreBackup:Backing up event indexes.

INFO:CoreBackup:Backing up event indexes completed.

INFO:zenbackup:Backup of events database completed in 5 seconds.

INFO:CoreBackup:Backing up config files.

INFO:CoreBackup:Backup of config files completed.

INFO:zenbackup:Skipping backup MIBs.

INFO:zenbackup:Skipping backup $ZENHOME/libexec.

INFO:CoreBackup:Backing up the ZODB.

INFO:zenbackup:Backup of ZODB database completed in 20 seconds.

INFO:CoreBackup:Backing up ZenPacks.

INFO:CoreBackup:Backup of ZenPacks completed.

INFO:CoreBackup:Backing up bin dir.

INFO:CoreBackup:Backup of bin completed.

2015-12-19 02:10:46,098 INFO CoreBackup: Backing up ZenPack contents.

2015-12-19 02:10:46,157 INFO CoreBackup: Performing an offline backup of zencatalogservice.

2015-12-19 02:10:46,180 INFO CoreBackup: Backup of zencatalogservice complete.

2015-12-19 02:10:46,189 INFO CoreBackup: Backup of ZenPack contents complete.

2015-12-19 02:10:46,189 INFO zenbackup: Backup of ZenPacks completed in 5 seconds.

2015-12-19 02:10:46,190 INFO CoreBackup: Backing up performance data (RRDs).

2015-12-19 02:23:53,416 INFO zenbackup: Packaging backup file.

2015-12-19 02:32:43,210 INFO zenbackup: Backup written to /mnt/export4/4x-backup/tmpUO0vRM/backups/zenbackup_4x.tgz

2015-12-19 02:32:43,210 INFO zenbackup: Cleaning up staging directory /mnt/export4/4x-backup/tmpUO0vRM/fvjPFj

2015-12-19 02:32:46,090 INFO zenbackup: Backup completed successfully in 22 minutes.

Daemon: zencatalogservice starting...

Daemon: zeneventserver starting...

Waiting for zeneventserver to start......

Daemon: zenwebserver starting...

Daemon: zenrrdcached starting...

Daemon: zredis starting...

Daemon: zenhub starting...

Daemon: zenjobs starting...

Daemon: zeneventd starting...

Daemon: zenping starting...

Daemon: zenactiond starting...

Daemon: zenmodeler starting...

Daemon: zenperfsnmp starting...

Daemon: zencommand starting...

Daemon: zenpython starting...

Daemon: zenjserver starting...

Daemon: zenvsphere starting...

Daemon: zenucsevents starting...

exporting component list ...

0x05f624

Unmounting - enter root password when prompted ->

Password:

Please remove the export disk from the virtual machine ...

<ENTER> after removed ...

Exiting ...

[Paul O'Leary]

And then the import errors:

Dec 18 15:50:51 UCSPM gunicorn: starting validation

Dec 18 15:50:51 UCSPM gunicorn: ['2015-12-18 15:50:51'] running command: ['bash', '-c', 'exec /opt/import4/bin/run_import4.sh util mountSCSI']

Dec 18 15:50:51 UCSPM gunicorn: Entering _poll_stdout

Dec 18 15:50:51 UCSPM kernel: ata1: soft resetting link

Dec 18 15:50:51 UCSPM kernel: ata1: EH complete

Dec 18 15:50:51 UCSPM kernel: ata2: soft resetting link

Dec 18 15:50:51 UCSPM docker: time="2015-12-18T15:50:51.758079027Z" level=info msg="GET /containers/serviced-isvcs_celery/json"

Dec 18 15:50:51 UCSPM kernel: ata2.00: configured for UDMA/33

Dec 18 15:50:51 UCSPM kernel: ata2: EH complete

Dec 18 15:50:51 UCSPM kernel: scsi 2:0:1:0: Direct-Access    VMware  Virtual disk    1.0  PQ: 0 ANSI: 2

Dec 18 15:50:51 UCSPM kernel: scsi target2:0:1: Beginning Domain Validation

Dec 18 15:50:51 UCSPM kernel: scsi target2:0:1: Domain Validation skipping write tests

Dec 18 15:50:51 UCSPM kernel: scsi target2:0:1: Ending Domain Validation

Dec 18 15:50:51 UCSPM kernel: scsi target2:0:1: FAST-40 WIDE SCSI 80.0 MB/s ST (25 ns, offset 127)

Dec 18 15:50:51 UCSPM kernel: sd 2:0:1:0: Attached scsi generic sg2 type 0

Dec 18 15:50:51 UCSPM kernel: sd 2:0:1:0: [sdb] 209715200 512-byte logical blocks: (107 GB/100 GiB)

Dec 18 15:50:51 UCSPM kernel: sd 2:0:1:0: [sdb] Write Protect is off

Dec 18 15:50:51 UCSPM kernel: sd 2:0:1:0: [sdb] Cache data unavailable

Dec 18 15:50:51 UCSPM kernel: sd 2:0:1:0: [sdb] Assuming drive cache: write through

Dec 18 15:50:51 UCSPM kernel: sdb: sdb1

Dec 18 15:50:51 UCSPM kernel: sd 2:0:1:0: [sdb] Attached SCSI disk

Dec 18 15:50:52 UCSPM docker: time="2015-12-18T15:50:52.388033569Z" level=info msg="GET /containers/serviced-isvcs_opentsdb/json"

Dec 18 15:50:53 UCSPM kernel: EXT4-fs (sdb1): mounted filesystem with ordered data mode. Opts: (null)

Dec 18 15:50:54 UCSPM gunicorn: stdout: mode of ./mnt/import4. retained as 0777 (rwxrwxrwx)

Dec 18 15:50:54 UCSPM gunicorn: stdout: mode of ./mnt/import4/4x-backup. retained as 0777 (rwxrwxrwx)

Dec 18 15:50:54 UCSPM gunicorn: stdout: mode of ./mnt/import4/volume. retained as 0777 (rwxrwxrwx)

Dec 18 15:50:54 UCSPM gunicorn: stderr: ls: cannot access /mnt/import4/4x-backup/4x-export-*.tar: No such file or directory

Dec 18 15:50:54 UCSPM gunicorn: stdout: {"imp4_error": {"_prep_scsi": "Cannot find import file on the SCSI device"}}

Dec 18 15:50:54 UCSPM gunicorn: stderr: ERROR: Cannot find import file on the SCSI device

Dec 18 15:50:54 UCSPM gunicorn: Exiting _poll_stdout

Dec 18 15:50:54 UCSPM gunicorn: Finding and mounting IMPORT4 disk returned 1

Dec 18 15:50:54 UCSPM gunicorn: ['2015-12-18 15:50:54'] running command: ['bash', '-c', 'exec /opt/import4/bin/run_import4.sh util umountSCSI']

Dec 18 15:50:54 UCSPM gunicorn: Entering _poll_stdout

Dec 18 15:50:54 UCSPM gunicorn: Exiting _poll_stdout

Dec 18 15:50:54 UCSPM gunicorn: Unmounting IMPORT4 disk returned 0

Dec 18 15:50:54 UCSPM gunicorn: IMPORT4 disk unmounted correctly

Dec 18 15:50:54 UCSPM gunicorn: Validation failed Finding and mounting IMPORT4 disk failed

[jasonsha]

Hi Paul,

I would recommend opening a TAC Case so we can investigate.

Thanks,

Jason

[Paul O'Leary]

For those playing at home - i had a bad device object in the schema and we had to run toolbox tools to repair and then correct the DB, device info and more.

Fixed it now and upgraded no problem