Tunning APIC-EM

[JPvisltgb]

Hello, 

We started to use APIC-EM for massive hardware lifecycle and now we have more that 1000 devices in inventory. 

APIC-EM server is a VM : 12 vCPU's, 64 GB RAM and 500 GB HDD. APIC-EM Version 1.6.3.30018.  However after about 700 devices are being provisioned system performance started to degrade. Now I see that  CPU is 100 % and RAM almost 100% in use.

Question: is there way to tune APIC-EM? Turn off all services except PNP, device discovery and device inventory? 

Another question might be related to that issue : As number of APIC-EM administrators increased so someone removed Project with 70% of all provisioned devices and now all these devices are in Unclaimed state. Is there way to bring them back via API somehow ? 

[Seb Rupik]

Hi there,

I can't help with your performance question, but I would instinctively say no. @aradford might be able to advise when he's online.

Regarding the the lost project issue. If you do a GET /pnp-search with a 'state': 'UNCLAIMED', do the device objects contain a 'site' object referencing the old project name? If so, try recreating the old project with exactly the same name and it *may* repopulate. 

I had a similar experience with APIC-EM when I was using it a while back, and it seemed very sloppy in the way it tidied objects when parent groups where removed:

https://community.cisco.com/t5/automation-and-analytics/devices-in-projects-stale-entries/td-p/3474062

...you might be lucky!

cheers,

Seb.

[JPvisltgb]

Unfortunately this doesn't work. I created same project name but all devices appeared in Unclaimed state.

[JPvisltgb]

So I was able to add Unclaimed device  to Project from API using :  POST /pnp-project/{projectId}/device. I was using payload '

[{"serialNumber": "FCW2XXXXXX", "hostName": "switch-1", "platformId": "WS-C2960L-48PS-LL", "state":"PROVISIONED"}]

But state of this devices changed to Error  - "ERROR_DURING_TRUSTPOINT_PROVISIONING"  I'm wonder is there a way to change this status to Provisioned ? 

I tried to update device using  PUT /pnp-project/{projectId}/device using payload : 

[{"state":"PROVISIONED",
"stateDisplay":"Provisioned",
"id":"558590ef-50dc-4f7c-b6a5-XXXXXXXXX"}]

But nothing changed. 

[balaji.bandi]

here is the Troubleshooting Guide to start with :

https://www.cisco.com/c/en/us/td/docs/cloud-systems-management/application-policy-infrastructure-controller-enterprise-module/1-6-x/troubleshoot-guide/b_apic_em_trouble_guide_v_1-6-x/b_apic_em_trouble_guide_v_1-6-x_chapter_01000.html

[aradford]

re performance.  What is the Disk IO?

Most of the issues i have seen have been IO related.

If you run the following command on the APIC-EM CLi.  you should get the disk IO.  Below is a bad example of 23M of IO.

$ dd if=/dev/zero of=/tmp/foo bs=1M count=512 conv=fdatasync

512+0 records in
512+0 records out

536870912 bytes (537 MB) copied, 23.7006 s, 22.7 MB/s

[JPvisltgb]

Got this:

512+0 records in
512+0 records out
536870912 bytes (537 MB) copied, 1.19586 s, 449 MB/s

I think it is not too bad.

[aradford]

Yeap… that looks fine. Just needs to be more than 200M.

I have customers who have loaded > 1000 switches with pnp.

Can you open a TAC case?

[JPvisltgb]

I tried, but I don't have support contract for that.