Split Brain on 3 node APIC cluster after one APIC HW faul

[petar.forai1]

Hi guys!

We're hitting a 'data plane partially diverged' on two controller nodes. The initial situation was that one of the APICs  (APIC ID1)had died after a reboot due to HW faults ( after a reboot to with FW upgrade to 2.0). We decommissioned the faulty controller and powered it off and unplugged power. The remaining two APICs then got into split brain/'data plane partially diverged'. After rebooting both APICs sequentially the problem was still present and we decided to decomission and recomission APIC ID 3 but to no avail. 

Any ideas?

P

[Tomas de Leon]

Petar,

Thank you for contacting the Cisco Support Community for ACI.  I am sorry that you are having an issue with the APIC Cluster.  I am going to summarize the issue so please correct me if my understanding is incorrect.

• you had a 3 APIC cluster the was working and fully fit
• you scheduled a "Controller" upgrade for all 3 APICs. APIC 1 started the Upgrade while the other two APICs were a queued or in-progress state. APIC 2 & 3 should have started the upgrade and waited at around 75% for APIC 1 to complete.
• APIC 1 completed the upgrade and rebooted. Then it came up with some Hardware faults.  What were the faults seen? Please be specific?
• What was the "Running" ACI version for the APICs prior to the upgrade to 2.0?  What was the target version for 2.0?
• Did APIC2 and APIC3 complete their Upgrades?
• What is the current state of APIC 1?
• Please gather a "show controller" and an "acidiag avread" on APIC 2 & APIC 3 and copy the display results into a text file and attach here.
• Have you opened a Cisco TAC Case yet?

This will most likely be the best way for us to determine the next plan of action.

Thanks

T.

[petar.forai1]

We resolved the problem by decomissioning the HW faulty APIC, and by resetting the non-leader to the setup wizzard. That resolved the split brain by some force :)