Hyperflex Cluster cannot start

[lkamal]

Hello ,

I have a Hyperflex cluster  which consists of three nodes, and we made a physical migration from place to another place , and I stoped the cluster and put the cluster esxi hosts on maintenance mode, and after relocating the cluster and powered on the nodes, then exited from maintenance mode , the controllers started on the three nodes but couldn't remount the cluster datastore , and there is an error on one node , and after login SSH to the Node controller VM  and input (HXCLI ABOUT ) command it display : ( Internal Server Error (500) : could not connect to internal service. please check configuration )

[Kirk J]

Cluster is not healthy (resiliency health, # of failures tolerable).

You might want to run commands:

sysmtool --ns cluster --cmd healthdetail
sysmtool --ns cluster --cmd offlineresources

It is possible the cluster simply needs more time to heal.  Also, you may also need to reregister the cluster with vcenter.

Kirk...

[lkamal]

HI Kirk  , 

- I waited for about one hour  but with  the same result  , do I need to wait for more time ?, also I need to know the result I should expect from the two commands  and is there any command I should input after both commands  results?  :.

sysmtool --ns cluster --cmd healthdetail
sysmtool --ns cluster --cmd offlineresources

- Regarding the Vcentre it is a nested Vcentre, so the Vcentre will be online after I start the cluster.

[Steven Tardy]

May need to run those `sysmtool` commands from one of the other two nodes.

My guess is those errors are symptoms and not causes.
Could deep dive into application (tomcat) logs to figure out what isn't working, but again that may be symptoms.

I would check basic things.

Does `df` show SCVM boot disk and housekeeping disk?

Does `df` show HX NFS mount?

Are all SCVM IPs up and reachable (`ping`)? Both management and storage-data?
My guess is this one SCVM storage-data IP isn't reachable.

What generation servers? (M4/M5/M6)?