Hyperflex 3 Node Unhealthy

[Sascha Mueller]

Hey Guys,

our Hyperflex 3 Node Lab Cluster is unhealthy. One Node moved to an very very irritating state.

The stctlVM was downsized to 1GB, had a snapshot (holy moly? i dont know why, it was an eam snapshot= and was not part of the ESX Agents in vCenter anymore. Booting up the stctlVM was not possible. 

We decided to take it down, reinstall esxi manually and configure it with hx installer 2.0.1a.

Software Reinstallation and Configuration was possible, but we were not able to reintegrate, replace or delete the old node in the Cluster, because it is unhealty.

admin@hx-esxi2-sc:~$ stcli node remove --id-1 nodeID -f
Failed to remove node
severity: error
_message: Cluster insufficient number of nodes
id: 103

Is there anyone out there with experience in that bloddy mess?

Regards,
Sascha

[Clifford Aldan]

Sascha,

Adding a node back into the cluster is a TAC-assisted task

Please open a case with Cisco TAC to get assistance with integrating the same HX node back into the 3-node cluster

[sbhadrav@cisco.com]

Replacing a node uses Expand Cluster to add the replacement node after removing the failed node. Replacing a node can be performed while the HX storage cluster is online or offline, provided the requirements are met. Replacing a node in a storage cluster typically requires TAC assistance.

Conditions that require TAC assistance to replace a node.

3 node cluster

All 3 node clusters require TAC assistance to replace a node. Replace the node during cluster maintenance.

4 node cluster

Storage cluster is unhealthy.

Storage cluster will become unhealthy if a node is removed.

2 or more nodes have failed.

Disks on the replaced node will be reused.

When a node is added to a storage cluster, the HX Data Platform associates each disk UUID with the node UUID. This is a fixed relationship for the life of the storage cluster. Disks cannot be reassigned to nodes with different UUIDs. TAC will work with you to assign the old node's UUID to the new node to ensure the disk UUID to node UUID association.

Storage cluster to remain online while replacing a node.

5 node cluster

Storage cluster is unhealthy.

Storage cluster will become unhealthy if a node is removed.

3 or more nodes have failed.

Disks on the replaced node will be reused.

When a node is added to a storage cluster, the HX Data Platform associates each disk UUID with the node UUID. This is a fixed relationship for the life of the storage cluster. Disks cannot be reassigned to nodes with different UUIDs. TAC will work with you to assign the old node's UUID to the new node to ensure the disk UUID to node UUID association.

Storage cluster to remain online while replacing 2 nodes.

Storage cluster to remain online and the cluster was initially 3 or 4 nodes.

If your storage cluster was initially configured with either 3 or 4 nodes, adding nodes to make a total of 5 keeps your cluster as a 3+2 or 4+1 cluster. To keep the cluster online while replacing a node requires TAC assistance.