cockroachdb failed in new vND 2.3.2d and NDFC 12.1.2e installation

[BigDalton]

Fresh new installation of 3 vND 2.3.2d master nodes in a cluster with no error. New NDFC app version 12.1.2e installed but this error appeared on one of the three nodes. Tried to delete all VMs and re-create the vND cluster and install NDFC, got the same error on a different node. I cannot find any relevant document online to help with troubleshooting. We plan to migrate DCNM to NDFC but this error is causing big concern around stability of NDFC: how can a new installation have the same error again and again?

cockroachdb-2

General

• Status
  Failed
  Back-off restarting failed container
• IP Address
  172.17.190.85
• Node
  xxnd02
• Restart Count
  1370
• Container Count
  2
• Age
  5d18h

[BigDalton]

It turned out to be a NTP issue. The configured NTP server is having intermittent connectivity problem, and running acs health command on the servers multiple times sometimes reports NTP synchronisation error. Once I changed NTP server to a different one and verified connection was stable, re-installing NDFC was successful. Wish there could be more indicative information in Nexus Dashboard GUI to quickly pinpoint the NTP issue.