Unable to deploy 3 master nodes on 3 different sites - KMS not working

justanetworkengineer — Tue, 28 Dec 2021 08:19:22 GMT

Hi,

i'm trying to deploy 3 master nodes on 3 different site, so with different management and data IPs each one.

The version I'm using is 2.1.2d (but it was the same with the previous one)

I have problems to bring up the cluster and looking at logs i have errors like these ones

{"level":"warn","name":"installerd","error":"etcd file /data/services/kms_etcd/certs/v1/clients/installerd/etcd/cacerts.crt not found","retries":60,"time":"2021-12-28T07:56:41.306317904Z","message":"cannot create etcd client. retrying"}
{"level":"warn","name":"installerd","error":"etcd file /data/services/kms_etcd/certs/v1/clients/installerd/etcd/cacerts.crt not found","retries":59,"time":"2021-12-28T07:57:11.306791644Z","message":"cannot create etcd client. retrying"}
{"level":"warn","name":"installerd","error":"etcd file /data/services/kms_etcd/certs/v1/clients/installerd/etcd/cacerts.crt not found","retries":58,"time":"2021-12-28T07:57:41.307813936Z","message":"cannot create etcd client. retrying"}
{"level":"warn","name":"installerd","error":"etcd file /data/services/kms_etcd/certs/v1/clients/installerd/etcd/cacerts.crt not found","retries":57,"time":"2021-12-28T07:58:11.308015335Z","message":"cannot create etcd client. retrying"}
{"level":"warn","name":"installerd","error":"etcd file /data/services/kms_etcd/certs/v1/clients/installerd/etcd/cacerts.crt not found","retries":56,"time":"2021-12-28T07:58:41.308800404Z","message":"cannot create etcd client. retrying"}
{"level":"warn","name":"installerd","error":"etcd file /data/services/kms_etcd/certs/v1/clients/installerd/etcd/cacerts.crt not found","retries":55,"time":"2021-12-28T07:59:11.308987232Z","message":"cannot create etcd client. retrying"}
{"level":"warn","name":"installerd","error":"etcd file /data/services/kms_etcd/certs/v1/clients/installerd/etcd/cacerts.crt not found","retries":54,"time":"2021-12-28T07:59:41.309254666Z","message":"cannot create etcd client. retrying"}
{"level":"warn","name":"installerd","error":"etcd file /data/services/kms_etcd/certs/v1/clients/installerd/etcd/cacerts.crt not found","retries":53,"time":"2021-12-28T08:00:11.310090327Z","message":"cannot create etcd client. retrying"}

{"level":"warn","name":"clusterd","topic":"RESTCLIENT","time":"2021-12-28T07:56:07.373427088Z","message":"Certs are not generated, waiting for 10 seconds"}
{"level":"warn","name":"clusterd","topic":"RESTCLIENT","time":"2021-12-28T07:56:17.382635292Z","message":"Certs are not generated, waiting for 10 seconds"}
{"level":"warn","name":"clusterd","topic":"RESTCLIENT","time":"2021-12-28T07:56:27.39177669Z","message":"Certs are not generated, waiting for 10 seconds"}
{"level":"warn","name":"clusterd","topic":"RESTCLIENT","time":"2021-12-28T07:56:37.40090938Z","message":"Certs are not generated, waiting for 10 seconds"}
{"level":"warn","name":"clusterd","topic":"RESTCLIENT","time":"2021-12-28T07:56:47.410054491Z","message":"Certs are not generated, waiting for 10 seconds"}
{"level":"warn","name":"clusterd","topic":"RESTCLIENT","time":"2021-12-28T07:56:57.419215527Z","message":"Certs are not generated, waiting for 10 seconds"}
{"level":"warn","name":"clusterd","topic":"RESTCLIENT","time":"2021-12-28T07:57:07.428415873Z","message":"Certs are not generated, waiting for 10 seconds"}
{"level":"warn","name":"clusterd","topic":"RESTCLIENT","time":"2021-12-28T07:57:17.437551105Z","message":"Certs are not generated, waiting for 10 seconds"}
{"level":"warn","name":"clusterd","topic":"RESTCLIENT","time":"2021-12-28T07:57:27.446761716Z","message":"Certs are not generated, waiting for 10 seconds"}
{"level":"warn","name":"clusterd","topic":"RESTCLIENT","time":"2021-12-28T07:57:37.455904832Z","message":"Certs are not generated, waiting for 10 seconds"}
{"level":"warn","name":"clusterd","topic":"RESTCLIENT","time":"2021-12-28T07:57:47.465054362Z","message":"Certs are not generated, waiting for 10 seconds"}
{"level":"warn","name":"clusterd","topic":"RESTCLIENT","time":"2021-12-28T07:57:57.4742688Z","message":"Certs are not generated, waiting for 10 seconds"}
{"level":"warn","name":"clusterd","topic":"RESTCLIENT","time":"2021-12-28T07:58:07.48341853Z","message":"Certs are not generated, waiting for 10 seconds"}
{"level":"warn","name":"clusterd","topic":"RESTCLIENT","time":"2021-12-28T07:58:17.492564934Z","message":"Certs are not generated, waiting for 10 seconds"}
{"level":"warn","name":"clusterd","topic":"RESTCLIENT","time":"2021-12-28T07:58:27.501727984Z","message":"Certs are not generated, waiting for 10 seconds"}
{"level":"warn","name":"clusterd","topic":"RESTCLIENT","time":"2021-12-28T07:58:37.51087376Z","message":"Certs are not generated, waiting for 10 seconds"}
{"level":"warn","name":"clusterd","topic":"RESTCLIENT","time":"2021-12-28T07:58:47.520432748Z","message":"Certs are not generated, waiting for 10 seconds"}
{"level":"warn","name":"clusterd","topic":"RESTCLIENT","time":"2021-12-28T07:58:57.529573818Z","message":"Certs are not generated, waiting for 10 seconds"}
{"level":"warn","name":"clusterd","topic":"RESTCLIENT","time":"2021-12-28T07:59:07.53882486Z","message":"Certs are not generated, waiting for 10 seconds"}
{"level":"warn","name":"clusterd","topic":"RESTCLIENT","time":"2021-12-28T07:59:17.547986467Z","message":"Certs are not generated, waiting for 10 seconds"}
{"level":"warn","name":"clusterd","topic":"RESTCLIENT","time":"2021-12-28T07:59:27.55714265Z","message":"Certs are not generated, waiting for 10 seconds"}

2021-12-28 08:07:38.450388 I | embed: peerTLS: cert = /certs/etcd/peer.crt, key = /certs/etcd/peer.key, ca = , trusted-ca = /certs/etcd/cacerts.crt, client-cert-auth = true, crl-file =
2021-12-28 08:07:38.450431 C | etcdmain: open /certs/etcd/peer.crt: no such file or directory

Any idea except open TAC ?

Re: Unable to deploy 3 master nodes on 3 different sites - KMS not wor

AJ Cruz — Wed, 12 Jan 2022 21:15:22 GMT

No idea, but how are you seeing the cluster formation logs? That would be helpful for my problem.

Re: Unable to deploy 3 master nodes on 3 different sites - KMS not wor

justanetworkengineer — Tue, 18 Jan 2022 07:14:48 GMT

Go in SSH on the nodes,

there are several folders in /logs/

go check the files in those folders

For my problem...There is a wrong port to open on the deployment guide, there is 15233 but it's 15223.

topic Re: Unable to deploy 3 master nodes on 3 different sites - KMS not wor in Nexus Dashboard

Unable to deploy 3 master nodes on 3 different sites - KMS not working

Re: Unable to deploy 3 master nodes on 3 different sites - KMS not wor

Re: Unable to deploy 3 master nodes on 3 different sites - KMS not wor