New Fabric - APIC1 cannot see APIC2 or APIC3

[Dan Peronto]

APIC1 is up and discovered *most* of the fabric.  2 Leafs missing, but that can be seperate discussion.

All of the APICs are connected to Leaf3 & Leaf4.  APIC1 sees Leaf3&4.  I can reach all 3 APICs via SSH to OOB Mgmt (rescue-user only on APIC 2&3).  Yet the APICs are not clustering. 

Any Ideas?  Is it possible I fatfingered the fabric name?  How can I verify the setup on each is correct?

Thanks

[lpember · Accepted solution]

Dan,

Here are a few things you can check:

• 'show lldp neighbors' on the leafs. Do you see the APICs that are connected? Are the APIC hostnames showing up?
• 'show vlan extended' on the leafs. Is the infra VLAN programmed on the APIC-facing ports?
• 'ping X.X.X.X'. Can the APICs ping each other via the infra network? They will not be able to cluster unless they can reach other. You can see the infra VTEP IP by issuing 'acidiag avread' or 'ifconfig' on the APICs.
• 'acidiag avread' on each APIC. This will verify the fabric name you have configured.
• 'show lldptool in eth2-1' and 'show lldptool in eth2-2'. Check all the parameters that the leaf is sending to the APICs and ensure they match.
• 'version' on APIC 1. Are the leafs/spines/APICs all on the same version?

Let me know what your search turns up and I can recommend some more things to check.

[Dan Peronto]

Thanks again Ipember!

I did see the unclustered APICs as lldp neighbors

The infra vlan was missing from the ports facing those APICs

acidiag avread showed I did indeed mess up the fabric name on those APICs (mismatched CAPS!)

Corrected fabric name after running eraseconfig setup.

All APICs are now up and clustered.

This forum is a lifesaver!

[vladimirgomez1]

Hello, I am experiencing the very same issue.
- Switches are using the very same version "14.0(3d)"
- I can see the Leaf directly connected to the APICs via LLDP and from Leaf side the parameters, infra vlan seems to be ok
- APICs are running the same version "4.1.1i" Cluster ID is the same on all of them, Cluster name as well.
However the cluster is not being formed, i can see the following error on APIC1:
"Cluster cannot continue to Expand from its current size 1 for the following reason: No expansion contender. The controller ID is 2, the service ID is None, the internal database partition (shard) ID is Undefined and the replica ID is Undefined"
- I am not able to log into APIC2 and APIC3 i got the error:
"REST Endpoint user authorization datastore is not initialized - Check Fabric Membership Status of this fabric node"
But using the rescue-user via CLI i can see all the parameters match - via "acidiag avread" they do not see each other.

Any ideas what could i check further ?

Many thanks - i can provide outputs if needed.

Vlad.

[sbdladla1]

Hi I have run the command you have recommended and included the result

[sbdladla1]

I am having the same problem:

I have included the result of the command