Waiting for stmgr management server on port 9333 to get ready

[mohamed_elmohamady]

I have hyperflex cluster with 3 nodes built with replication factor 2, the SCVM cluster is down and can't get access to any on the datastores, tried to start SCVM services but got the following,

Waiting for stmgr management server on port 9333 to get ready . . . . . . . . . . . . . . . . . . . . Traceback (most recent call last):
File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/opt/springpath/storfs-mgmt-cli/stCli-1.0-py3.5.egg/stCli/stCli.py", line 13400, in <module>
File "/opt/springpath/storfs-mgmt-cli/stCli-1.0-py3.5.egg/stCli/stCli.py", line 835, in __init__
File "/opt/springpath/storfs-mgmt-cli/stCli-1.0-py3.5.egg/stCli/stCli.py", line 924, in __init__
File "/opt/springpath/storfs-mgmt-cli/stCli-1.0-py3.5.egg/stCli/StMgrTransport.py", line 38, in __init__
File "/opt/springpath/storfs-mgmt-cli/stCli-1.0-py3.5.egg/stCli/StTransportBase.py", line 124, in test_connection
ConnectionRefusedError: [Errno 111] Connection refused

tried to restrart all SCVM with no luck. 

[mohamed_elmohamady]

tried to run 

admin:~$ ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
ntp-ip  .XFAC. 16 u - 1024 0 0.000 0.000 0.000

and the same output on all SCVMs 
none of the SCVMs have the cluster IP

[mohamed_elmohamady]

tried also the following,

admin:~$ priv service stMgr restart
stMgr stop/waiting
stMgr start/running, process 334

admin:~$ sysmtool --ns cluster --cmd info
Thrift: Sat Jan 14 13:17:33 2023 TSocket::open() connect() <Host: localhost Port: 9090>Connection refused
Thrift: Sat Jan 14 13:17:33 2023 TSocket::open() connect() <Host: localhost Port: 9090>Connection refused
Caught Exception: connect() failed: Connection refused

admin:~$ sysmtool --ns cluster --cmd info
Thrift: Sat Jan 14 13:18:08 2023 TSocket::open() connect() <Host: localhost Port: 9090>Connection refused
Thrift: Sat Jan 14 13:18:08 2023 TSocket::open() connect() <Host: localhost Port: 9090>Connection refused
Caught Exception: connect() failed: Connection refused

admin:~$ echo srvr | nc localhost
This is nc from the netcat-openbsd package. An alternative nc is available
in the netcat-traditional package.
usage: nc [-46bCDdhjklnrStUuvZz] [-I length] [-i interval] [-O length]
[-P proxy_username] [-p source_port] [-q seconds] [-s source]
[-T toskeyword] [-V rtable] [-w timeout] [-X proxy_protocol]
[-x proxy_address[:port]] [destination] [port]
admin:~$ stcli cluster info | grep –i domain
grep: domain: No such file or directory
Traceback (most recent call last):
File "/opt/springpath/storfs-mgmt-cli/stCli-1.0-py3.5.egg/stCli/StTransportBase.py", line 124, in test_connection
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/opt/springpath/storfs-mgmt-cli/stCli-1.0-py3.5.egg/stCli/stCli.py", line 13400, in <module>
File "/opt/springpath/storfs-mgmt-cli/stCli-1.0-py3.5.egg/stCli/stCli.py", line 835, in __init__
File "/opt/springpath/storfs-mgmt-cli/stCli-1.0-py3.5.egg/stCli/stCli.py", line 924, in __init__
File "/opt/springpath/storfs-mgmt-cli/stCli-1.0-py3.5.egg/stCli/StMgrTransport.py", line 38, in __init__
File "/opt/springpath/storfs-mgmt-cli/stCli-1.0-py3.5.egg/stCli/StTransportBase.py", line 136, in test_connection
BrokenPipeError: [Errno 32] Broken pipe
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='ISO-8859-1'>
BrokenPipeError: [Errno 32] Broken pipe