Safely shut down all nodes in one site of a HyperFlex stretch cluster

[riaan.vanniekerk]

We need to temporarily shut down one of the sites of our HXDP 5.0(2e) based stretch cluster (with RF 2+2). The cluster has plenty of compute capacity (all VMs can easily run on one site). I am hoping it is as easy as:

1. Confirm in HX Connect that cluster is healthy, has no major issues.
2. Put all hosts in Site B in HX Maintenance Mode.
3. Shut down all ESXi hosts.
4. (When done with our maintenance operation), power on all ESXi hosts.
5. Take all hosts out of HX MM.

Is that it, or am I missing something?

[marce1000]

     - FYI :  https://www.youtube.com/watch?v=XL495PUflbI

 M.

[riaan.vanniekerk]

I am aware of the above procedure for total shutdown of an HX cluster (and have used it to physically move a cluster between DCs). We had downtime for all VMs in the maintenance window.

My scenario is different: I need to shut down one half/site of a stretch cluster without impacting the VMs/workload.

[RedNectar · Accepted solution]

Hi @riaan.vanniekerk ,

Your process looks fine to me - but I'd add the following checks.

1. Make sure the Witness VM is healthy, and the 2nd site has visibility to the Witness
2. If you are running HA and DSR on the cluster
   • put the ESXi hosts into Maintenance Mode one-at-a-time,
   • and make sure all VMs move gracefully to the 2nd site

• If not (surely you are..), then do that manually

But I realise you probably already have taken that into consideration.  And as always - do a full backup first!

[riaan.vanniekerk]

Hi @RedNectar 

Thank you very much. Agree with your suggestions. 

W.r.t. backups - what should I back up of? VC? UCS? HX cluster config export? Which specific failure scenarios should I plan for and would I be able to recover from with a backup? 

[RedNectar]

Hi @riaan.vanniekerk ,

Not much point in backing up VC or UCS - and the HX Cluster config backup should have been taken from the Installer VM at the end of the install. Not easy to get that HX cluster config backup after the event, but then if you kept your pre-install checklist values somewhere it's not hard to recreate.

No - I was just being exxxxtra cautious suggesting that all of your actual HX datastores are fully backed up - typically customers implement HX Clusters with an integrated Veeam, Commvault or Cohesity backup system in case you experience a TOTAL failure.  TBH, I think the chances of such a disaster would be remote, but if your primary data centre went up in flames just after you'd shut down your secondary site, you might need those backups after you'd rebuilt the cluster.