Error when reregistering storage cluster with vCenter

[mic3]

Long story short: I let my Root CA certificate expire and along with it all of the certs it had issued over the last 5 years.

Don't flame me for that I already have felt / am feeling the pain. One of the big things is getting my VM environment working again.

After working with VMware we have new Platform Services Controller and vCenter servers. They were not able to help me recover from the expired certificates. Recreating these servers allowed access to the Web Clients. I was then able to get the PSC to function as a subordinate CA and issue certificates to vCenter and ESXi hosts.

I now need to reregister the Storage Cluster with vCenter. When I issue the following command, replacing placeholders in <> with environment targets, I receive an error:

Command: stcli cluster reregister --vcenter-datacenter <DATACENTER> --vcenter-cluster <CLUSTER> --vcenter-url <URL> --vcenter-user <USER>

Prompted for password, supply password.

After about 3 - 5 minutes Error: storage cluster reregistration with a new vCenter failed: StException (severity=2, data=None, _message='VI SDK invoke exception:; nested exception is: \n\tcom.vmware.eam.EamRuntimeFault', lcmessage=None, _DO_NOT_USE_localizedMessage=None, cause=None, id=204, entityRef=None) VI SDK invoke exception:; nested exception is: com.vmware.eam.EamruntimeFault

If anyone has any ideas I would appreciate them.

[mojafri · Accepted solution]

Hi, 

Seems to be an EAM issue. Could you please check the below KB it you are hitting the same? 

https://kb.vmware.com/s/article/2112577

Also provide the stMrg.log output which doing re-register - /var/log/springpath/stMrg.log. Hopefully you have already removed the MOB entry from VC is your using the same VC.

Please rate if you find it helpful.

Regards,

MJ 

[mic3]

Mojafri,

Thank you for pointing me to that KB. I think it did the trick to solve the VI SDK invoke exception error. However, I now have a new error.

storage cluster reregistration with a new vCenter failed: StException (severity=2, data=None, _message='java.net.ConnectException: Connection refused (Connection refused)', lcmessage=None, _DO_NOT_USE_localizedMessage=None, cause=None, id=1, entityRef=None) 

java.net.ConnectionException: Connection refused (Connection refused)

Unfortunately, due to secury policy I cannot export log files. I can grab snipits but I am restricted to writing them down by hand and transcribing them.

I am not using the same VC we created a new one, however I used the same name and IP address in an attempt to simplify things ... perhaps that was a poor choice.

[mojafri]

I need more info...what output is there after this line? 

storage cluster reregistration with a new vCenter failed: StException (severity=2, data=None, _message='java.net.ConnectException: Connection refused (Connection refused)', lcmessage=None, _DO_NOT_USE_localizedMessage=None, cause=None, id=1, entityRef=None) 

We are getting connection refuse - could be because of several reasons: 

1. Check connectivity from HX connect IP ssh session:  

# nc -zv <vCenter-IP> 443

# nc -zv <vCenter-IP> 80

2. Are you using local administrator account for VC while doing re-register? If not, try that 

3. Check if all HX ESXi node have "hxuser"

ESXi# esxcli system account list

ESXi# esxcli system permission list 

Regards,

MJ

[mic3]

MJ,

Here are the results from your inquiries.

1. Check connectivity from HX connect IP ssh session:  

# nc -zv <vCenter-IP> 443 - Success

# nc -zv <vCenter-IP> 80 - Success

2. Are you using local administrator account for VC while doing re-register? If not, try that -  Yes I am using administrator@vsphere.local; I also tried my domain account which is in the Administrators group in VC, but that failed as well.

3. Check if all HX ESXi node have "hxuser"

ESXi# esxcli system account list - Yes

ESXi# esxcli system permission list - Yes

[mojafri]

Hi,

Unfortunately i can't help you more. I need stMgr.log & virgo (vsphere client) logs to find whats going on. Better to open TAC case where you can upload logs securely. 

Regards,

MJ

[mic3 · Accepted solution]

-Solved-

The finally piece fell into place. It was suggested that I check that ESXi shell and SSH were enabled on all hosts. SSH service was stopped on all hosts. I went to the configuration tab, security profile, and started the SSH service. I then tried the reregister command and it worked. Well I was given a success message but the environment was dirty from previous unsuccessful attempts, so I ran through the unregistering process again in this document, https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_DataPlatformSoftware/AdminGuide/2_0/b_HyperFlexSystems_AdministrationGuide_2_0/b_HyperFlexSystems_AdministrationGuide_2_0_chapter_011.html, and then did the reregister process again. The only thing missing from the above document is to restart vsphere-client service in vCenter after the reregister is complete; I also had to issue stcli cluster start to get the storage cluster to recheck its state.

[mojafri]

Glad that issue is resolved. Cheers !!