ACI APIC Recovery

[Whiplash2013]

Current setup and situation:

We currently have a Mini Fabric deployment (2 virtual APICs, 1 Physical, 4 Leafs, and 2 Spines). One of our engineers accidentally changed the port speed on the uplink interface that connect our Leaf switch and Virtual environment. He changed it to 10Gb when the port on the ESXi will only accept 40GB/100GB. When this uplink went down, so did 2 of our APICS. Now the 3rd is in Read-Only mode due to it being the only one in the cluster.

We are unable to make any changes on the Leafs to fix this mistake. 

Any ideas? We are currently at a stand still with both our APICs and entire Virtual Environment down.

Thank you ahead of time!

[Robert Burns]

Recommend you open a TAC case, they can assist w/ recovery.  Especially if this is affecting a prod deployment.  I wont ask by both virtual APICs are unavailable, seeing as they should be hosted on separate ESX servers per best practice.  Can you vmotion the APIC to another ESX host that still has it's original connectivity to restore cluster majority?

Robert

[Whiplash2013]

They were hosted on separate ESX hosts, however when the change was made to the VPC_IPG that controlled the duel-homed uplinks, it didn't matter.

[Robert Burns]

Alternately if this is a lab enviornment, you can shrink the cluster size to 1, remove the two inacceible APICs, fix your ESX host connectivity after re-gaining R/W control and then redeploy the vAPICs.

Robert

[Whiplash2013]

Any idea on how to shrink the cluster size to 1 on the remaining Physical APIC with it being in read-only mode?