Issues with "partial inventory sync"between ACI and VCenter F0132

[joshua.james]

I'm receiving an error that states "Received partial inventory in the last inventory sync". There seems to be little about this in various searches. It is accompanied by an error code: F0132

I have, from all results I've seen in ACI and with the VMWare admins, good connectivity to Vcenter from the APIC as well as ongoing communication. I can see VMs added, and I see all my 216 port-groups in Vcenter and I'm able to add and move VMs between them and operate in the fabric.

Any ideas on how to test or troubleshoot further, if not actually resolve the issue or error message?

[gmonroy]

joshua.james,

    If you haven't already opened a TAC case, I would definitely recommend you do so in order to further isolate the cause of those faults. In a nutshell, connectivity is not the only factor when it comes to seeing the "Partial Inventory Sync" fault. 

Most of what we see in the VMM Networking tab is populated by information sent from vCenter (VMM) to the APIC. The APIC attempts to keep track of information it is told about via MOs and faults get flagged if this information changes, or if vCenter reports some issue with its inventory (depending on monitoring/statistics policy). 

What this means is we can see this fault if some change was made to vCenter that the APIC is unable to account for it. One example of such instance is if Vms migrated into DVS still have pointers/vmnics that are trying to resolve to an older/non-APIC managed DVS. The APIC will see this and flag it, and in some instances, can be flagged with that "partial inventory sync" fault.

If you haven't opened a TAC case and really want to start having an idea of where to dig, we will want to focus on the vmmmgr logging of all 3 APICs. Since only one will be leader, each DVS (if we have more than 1) can have a different APIC as its leader, and subsequently needs to be checked for its logs.

From an APIC:

# grep -C 2 "SOAP API" /var/log/dme/log/svc_ifc_vmmmgr.bin.log

Essentially, /var/log/dme/log/svc_ifc_vmmmgr.bin.log will contain some information relevant to troubleshooting the fault. For reference, this file is collected in the Logs (3of3) file of an APIC generated techsupport:

https://supportforums.cisco.com/t5/data-center-documents/aci-on-demand-techsupport-collection-when-first-opening-an-sr/ta-p/3215947

Cheers,

Gabriel

[rodukov5555]

Same problem appeared when we deleted and recreated VMM integration. It reproduces even when we made brand integration with the new VDS name.

Should we reboot VMM-Leader APIC to fix it?

Or what should we do to make a fix?

[Robert Burns]

Rebooting the controllers (APICs) doesn't typically fix this.  One thing to check is that all your VMs have a valid NIC binding (ensure all VMs under the VMware Datacenter have a valid network port group assigned, including vmkernel interfaces).   This can also lead to this partial sync issue - if a VMs network config is unable to resolve to a valid DVS and/or Port Group ID.  More details on this here, but it's a VMware-side issue that needs fixing.  https://kb.vmware.com/s/article/2008092

If this isn't the issue, the only other way to determine the problematic VM is to dig through logs - which you'd likely need TAC support to do.  

Robert 

[rodukov5555 · Accepted solution]

Thanks for your reply!

What actually "fix" our problem:

1. We opened a case in the TAC

2. Cisco engineer made WebEx-session with us and made described below steps

3. He connected to the CLI of our 1 of 3 APIC (randomly selected) and determined which one is Leader in our VMM intergation:

show vmware domain name [NAME]

4. He run on Leader APIC CLI tail output and few second after made manual VMM resync via APIC GUI:

cd /var/log/dme/log
tail -f svc_ifc_vmmmgr.bin.log | egrep -i "Inventory pull|failed to update"

5. He observed the output in CLI and, also, he observed in APIC GUI the fact that there is no date changes in error, after succesfully resync.

6. Also he observer absence of any errors on our vSphere.

7. After this he said, that this is a "Stale" error, and it need to be removed from APIC database.

8. He login in APIC CLI under root (password was generated by him earlier with obtainig output of command "acidiag dbtoken"):

ssh root@0

9. He run test API for an 1 hour:

enable_testapi.bin 3600

10. He made an XML file with the properties of our error:

mkdir /data/techsupport/temp
cd /data/techsupport/temp
echo '<faultInst code="F0132" severity="major" status="deleted"/>' > staleFaultsToBeDeleted

11. Finally he deleted our error, made after this secnd VMM resync, and the error doesn't reappear!

moquery -c faultInst -f 'fault.Inst.code=="F0132"' | egrep '^dn.*:.' | sed 's/dn.*:.//g' | xargs -n 1 -I {} icurl -g -X POST http://localhost:7777/testapi/mo/{}.xml -d @staleFaultsToBeDeleted.xml