APIC database

[patrikvarga11]

APIC cluster "Data Layer Partially Diverged" after RMA of APIC2

Three-node cluster, APIC-M3, version 6.0(5h). APIC2 had a disk failure, we got an RMA.

Followed the procedure: decommissioned APIC2 from APIC1, physically swapped, ran the setup script with the same node ID 2,
same fabric name, same TEP pool etc.

System > Controllers the Data Layer state has been "Partially Diverged" for a while now.

Fabric is fully operational, no impact, but I don't want to leave it like this.
acidiag cluster on APIC1 shows all checks pass except "Data Layer Partially Diverged".

Do I just wait? Is there a point at which I should decommission it again and retry?

[Daniel Slyk · Accepted solution]

Hi,

how long has it been? It is expected to see that, but it should not stay like this.

If it is fresh, just "now", I would wait.

It's normal for a freshly commissioned replacement APIC, and I would not decommission it again.
The APIC distributed DB is split into shards, and in a 3 node cluster every shard has 3 replicas one per APIC.
When you decommissioned APIC2, the cluster dropped to 2 replicas per shard.

When you commissioned the RMA unit, APIC2 has to receive a full copy of every shard replica from the other nodes.

Data Layer Partially Diverged means some replicas are not yet in sync with their leader.

Check the replicas directly from APIC: acidiag rvread

Run it and wait 10-15 minutes, run it again. If it's improving leave it alone.

This can take a minute, the worst I have seen was maybe an hour.

If it is more then that, I would reach out to Cisco TAC.

[patrikvarga11]

Thank you, it actually synced and show as fully-fit. Took almost three hours.