NDFC Greenfield Lab Deployment Issues

[RobertCO]

I'm investigating using NDFC for a new data center and am relatively new to the product.

Working with a lab setup, I'm finding a check-or-the-egg type scenario:

I've deployed a 3-node cluster of ND and added NDFC onto the cluster.  The MGMT interfaces are on one (traditional) network.  The data/fabric interfaces are connected to leaf pairs for the fabric I am trying to build.  The leaf pairs are connected to a spine.  The leaf and spines are N9Ks with no config.  The nodes can all reach each other on both the fabric interfaces and MGMT interfaces.  My setup NDCF connectivity is exactly link this (https://www.ciscolive.com/c/dam/r/ciscolive/global-event/docs/2022/pdf/BRKDCN-1119.pdf option #1 on page 19):  

However, once I discover the switches on the data network where the NDFC controllers are connected to, it reboots them, and NDFC goes haywire.  The ND controllers are still accessible via the MGMT interfaces, but the NDFC process/web pages becomes completely inaccessible.  The network eventually recovers, but the switches never successfully complete the onboarding process within NDFC.  It appears NDFC times-out during the reboot process when the controllers can't reach each other over the data/fabric interfaces?

So, in a greenfield build, how is one supposed to connect their ND cluster to the fabric which they are trying to build when during the process of onboarding the fabric switches they connect to it fails? 

[RobertCO]

As a work around, I've put a regular L2 switch (non-fabric) between the controllers and leafs.  Just seems like there would be a better way and not so much reliance for NDFC on the fabric interfaces for discovery.

I'm still looking for official Cisco guidance here, so if anyone stumble across this and has a better, more correct way, it would be much appreciated!

[Petar Rusev]

Hello Robert,

Have you tried to distribute the data connections across more than a single leaf switch? I would try connecting the data ports with two legs, each leg to different switch.

The ND goes crazy most probably because your cluster is totally disconnected. You need to make sure that at least two of the nodes can see each other so the cluster can operate even with one node disconnected. Consider the ND cluster like ACI APIC cluster, you need to have the quorum.

Then you discover the switches one by one.