VM's disks are not on HyperFlex Datastore

[iamtheroot]

Hi,

When we take snapshots for the VM  in a Datastore created by HX it's reporting below

"VM's disks are not on HyperFlex Datastore".   We have Vmware Hypervisor on HX platform. Taking snapshot via plugin and directly via HX gui fails.

We use Veeam as backup software but on Veeam end seeing this 
<62> Error {"message":"Failed to take snapshot for vm vm-XXX, serverUuid . Invalid argument","messageId":0}

In the /var/log/springpath/stDataSvcMgr_VirtPlatform.log file it reports below as well

==========

 [snapshot-virt-pool-5] DEBUG c.s.s.v.VirtPlatformUtil$ - fromTryCatch got exception: StException(InvalidArgument,None,Some(EntityRef(vm-XXX,Virtmachine,Some(Linux-blalbla),None,None)),Some(Invalid argument),None,None,Error,Some(LocalizableMessage(VM's disks are not on HyperFlex Datastore,None,None,VM's disks are not on HyperFlex Datastore,ArrayBuffer())))

=====================

Anyone come across this ?

[RedNectar]

Has the VM been allocated additional disk that is on a NAS somewhere?  For a HX snapshot to work , all disk drives must be on the same HX data store

[iamtheroot]

I did check and all the disks for VM are in HX datastore and still failing.

[Kirk J]

Can you manually trigger a snapshot from the HX vcenter plugin for that VM?

Unfortunately that error is somewhat generic and gets triggered in a number of circumstances.  Resolution in cases with that error typically  range from vcenter reboot, reinstall of HX vcenter plugin, etc

Kirk...

[iamtheroot]

Thanks Krik.  Yes I tried from vcentre plugin and here is what I got 

Failed to take snapshot for vm vm-XXX, serverUuid . Invalid argument","messageId":0

From HX gui itself I am getting below

VM's disks are not on HyperFlex Datastore

[Kirk J]

Please confirm you have reinstalled the 2.2 plugin 

https://software.cisco.com/download/home/286305544/type/286305994/release/5.0(2a) 

HyperFlex-VC-HTML-Plugin-2.2.0.zip

https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_DataPlatformSoftware/AdminGuide/4-5/b-hxdp-admin-guide-4-5/m-html-plug-in-for-vmware-vcenter.html#Cisco_Task.dita_4b00c229-6eb0-4f08-ae03-fdee71fe79a4

Kirk...

[iamtheroot]

Yep reinstalled the plugin 2.2 from hx 4.52b release as we use that.

Tried re-register vcentre and no luck.  Even tried removing mob domain for this vecentre and re-registered. No luck.

[KyoCode]

Hi ! It looks like you need assistance registering vCenter to HX if it failed, can you open a ticket with Cisco TAC? most of the time it will require to reset some backend services or initiate external services, sometimes these tasks will require SSH to each ESXi and other issues which are not customer facing / public to troubleshoot.

AIV

[iamtheroot]

Issue details ( just in case anyone have similar issue later on):

We had one Vcentre ( example: ausvc01.lab.local). This one vcentre had 2 HX Clusters   ( example: ClusterA and ClusterB).

Steps to recreate issue in our case:

Make a datastore ( example: DS01) on ClusterA and mount it to all hosts.

Unmount DS01 and remove datastore from HX

Now create same name datastore DS01 on ClusterB and mount on all hosts.

Now take HX snap for any VM on ClusterA and it fails.

Quick Fix for this issue :

Just create a test datastore one ClusterA and mount it all hosts. then take HX snap on vm's on ClusterA  and it works 

Now safe to unmount test datastore and do test again it works fine.

Tac was able to reproduce issue  in his lab and provided this workaround. Root cause is yet to be found, I assume some bug with hx plugin or vcentre DB stale entry. Just creating another Datastore fix it.

Note: Re-register with vcentre , HX plugin re-install and restarting vcentre-UI  didn't help

Hope this helps.