New APIC not able to join cluster.

[JamesB11]

Hi,

I'm replacing our APICs from M2 to M4.

I followed this guide: Cisco APIC M1/M2/M3/L1/L2/L3 to M4/L4 Cluster Migration, Release 6.0(2) - Cisco

The new APICs were shipped with 6.0.2, while our cluster is running 6.0(7e).

The first one (APIC1-NODE3) was successfully upgraded with auto-upgrade and joined the clutster after decommissioning the old, and commissioning the new one by CIMC ip.

The second one (APIC2-NODE2) failed the auto-upgrade, so I had to upgrade it manually, and then it joined the cluster.

The APIC2's interface towards L131 is still in Discovery state but shows Up, Up  - switching state disabled.

Its interface towards L132 is fine.

Anyway, I proceeded with the last APIC (APIC3-NODE1), manually upgraded, decommissioned the old one, and commissioned the new one with its CIMC ip. However, I can't get it to join the cluster.

Both its interfaces towards the Leaf switches is orange:

it sees it with LLDP:

I can access the GUI via the OOB ip that it has taken over. The hostname is correct, but it is in Cluster Bringup.

So, it has kinda received the configuration, but it can't join the cluster.

The Health State for APIC3 sometimes changes to: "Data Layer Synchronization in Progress…"

Also, when i reboot the APIC3 (which is not part of the cluster, yet), it looks like APIC2 also become unavaiable seen from APIC1's GUI.

Errors for the interfaces connected to APIC3:

Towards L131 (same errors toward L132).

KVM:

[AshSe]

Hello @JamesB11 

Sharing below checklist to ensure all APICs are configured properly:

1. APIC IP addresses uniqueness
2. Gateway check
3. Infra VLAN identical
4. DNS resolution by APIC
5. NTP reachable by APIC
6. Check Physical connectivity
7. Check Switch port configuration
8. Check firewall (if any)
9. CIMC/BIOS/APIC Software Incompatibilities 
10. Cluster ID Mismatch
11. Hardware Issues

Hope this will help!

AshSe

Community Etiquette: 

1. Insert photos/images inline - don't attach.
2. Always mark helpful and correct answers, it helps others find what they need.
3. For a prompt reply, kindly tag @name. An email will be automatically sent to the member.

[Wassim Aouadi]

Hello,

The following is an excerpt from Cisco ACI System Messages Reference Guide - ACI F0x Messages [Cisco Application Policy Infrastructure Controller (APIC)] - Cisco

Explanation: This fault occurs when a port is out-of-service because of a wiring error.

Recommended Action: If you see this fault, take the following actions:

1. Ensure the device connected to the port is the correct device.
2. Ensure the device connected to the port is configured properly for the fabric it is being connected to.
3. If the device being connected is being reprovisioned from another fabric, ensure that the config on that device has been factory reset
4. If above actions did not resolve the issue, create a tech-support file and contact Cisco TAC.

[AlexI1978]

We had a similar issue and resolved by fixing uplink from the APIC (using correct dedicated interface on APIC) to the LF and rebooted all the spines.

[Hefe2]

Hi @JamesB11 ,

Did u already check your discovery mode settings? (can either be "strict" or "permissive")

System->Controllers-> APIC1 --> Cluster as Seen By Node

regards