APIC not joining to the cluster

[Gopi krishnan]

One of the controller is failing to join to the cluster. But at the same time i can see the controller under Unauthorized controllers and the state is showing as APPROVED.

What could be the reason? 

How i can add this controller to the cluster?

[Rick1776]

Make sure the cluster size is appropriate for your environment. Could also be the transceiver on the server. Are all the other controllers discovered and fully-fit n the APIC manager?

[Rick1776]

Also you have to disable the LLDP via the CIMC on the VIC. Some APIC's where shipped with it enabled. The ACI uses it own LLDP process for the fabric discovery.

https://supportforums.cisco.com/kxiwq67737/attachments/kxiwq67737/4436-docs-data-center/919/1/apic-vic-lldp-fn.pdf

[gmonroy]

Gopi,

Some other things to check:

1. What version is this apic running when compared to the other two? We will want all of them to be on the same version.
   1. "show version" or "acidiag version" will give you this depending on the APIC ver.
2. Does the APIC information match? This is referring to the values provided in the startupscript.
   1. "acidiag avread" on all 3 apics for comparison.
3. check lldp on the APICs:
   1. cat /proc/net/bonding/bond0  (to get active currently active slave)
   2. bash
   3. show lldptool in ethx-x (on currently active slave)
   4. show lldptool out ethx-x (same)
4. Are there any faults seen from any of the apics in reference to cluster expansion, or relating to the other APICs?

-Gabriel

[Gopi krishnan]

Thank you guys, for your support…….

Unfortunately, the network card installed in the APIC chassis got failed. Currently the cluster contains only one controller, which is up and running. So I tried to bring up the other APIC .The second APIC got registered to the cluster. But the health status is showing “data layer partially diverged”. Also, I am not able to login to the second controller and it is giving the error “REST Endpoint user authorization datastore is not initialized - Check Fabric Membership Status of this fabric node”.

Do guys know what could be the reason?

Is it advisable to run two APICs in a cluster?

[Rick1776]

It's always best practice to have three controllers in a production environment. In our LAB we only have one APIC and it works just fine you just get an error every time you log in stating that you only have one APIC.

See the following thread that other people that had the same error on the APIC.

https://supportforums.cisco.com/t5/application-centric/health-status-change-to-quot-data-layer-partially-diverged-quot/td-p/2688763

Looks like there is a bug on this as well...

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCus45262/?referring_site=bugquickviewredir

Follow the steps in this link to get the second APIC online again.
https://www.cisco.com/c/en/us/support/docs/cloud-systems-management/application-policy-infrastructure-controller-apic/118918-technote-aci-00.html