So I don't know how this happened but despite all leaf/spines showing "Active" via acidiag fnvread, the third APIC is stuck at "Setting up network configuration". Rebooting just takes me right back to this screen again. avread alone shows that the cluster size is 3 but 2 APICs are active. Otherwise all 2 APICs + nodes show Healthy.
The third APIC is plugged in with CIMC access but the two uplinks to the leaf switches are down. The leaf says "link not connected" but that's only because the APIC has not fully booted up.
How do I recover when every time I reboot, the APIC goes straight to setting up network configuration... ?
Cluster fault says F1410. The usual, cluster size is 3, actual size is 2. No contenders blah blah. But this is because the APIC is stuck and won't seem to pass go. Image attached. Basically, I suspect when the cluster was created, the third one never fully finished setting up and nobody noticed.
