topic Re: Devices stuck in UNKNOWN state in Controllers

Devices stuck in UNKNOWN state

Vag — Fri, 01 Mar 2019 12:38:39 GMT

I have 2 devices stuck in unknown state. I've tried to re-discover them, but there was no change in their status.

Then I tried to delete them and I got the following error message:

I have restarted the APIC-EM server, but I still get the same error messages. The discovery doesn't show any errors.

Any ideas how to fix this?

Re: Devices stuck in UNKNOWN state

pmuthuva — Tue, 06 Jun 2017 09:09:09 GMT

Hi,

Can you please share the cluster setup or the inventory-service logs to look into the issue.

Thanks,

Pragatheeswary M

Re: Devices stuck in UNKNOWN state

ngoldwat — Tue, 06 Jun 2017 13:53:52 GMT

What displays when you hover over the 'i' next to unknown?

Re: Devices stuck in UNKNOWN state

Vag — Wed, 07 Jun 2017 00:33:11 GMT

Re: Devices stuck in UNKNOWN state

Vag — Wed, 07 Jun 2017 00:57:13 GMT

Hi Pragatheeswary,

It's a single server setup. Do you mean the apic-em-inventory-manager-service Instance Logs output from controllerDevelopment tab?

Re: Devices stuck in UNKNOWN state

ngoldwat — Wed, 07 Jun 2017 12:22:46 GMT

execute: cd bin

execute: grape instance status

This will display all of the services. Look for the apic-em-inventory-manager-service and note the ip address.

execute: ssh <ip of inventory service>

execute: tar czvf logs4tac.tgz /var/log/grapevine/services/apic-em-inventory-manager-service/5.1.3.1009/

NOTE: the version at the end of the path differs depending on what you are running

execute: exit

execute: scp <ip of inventory service>:logs4tac.tgz .

execute: exit

Now the file can be retrieved with filezilla or such.

sftp://<ip of inventory service>:logs4tac.tgz .

Re: Devices stuck in UNKNOWN state

pmuthuva — Wed, 07 Jun 2017 16:14:53 GMT

Hi Vag Le,

I meant the instance logs of apic-em-inventory-manager-service.

The steps are as shared by Nicholas.

Re: Devices stuck in UNKNOWN state

Vag — Thu, 08 Jun 2017 00:17:00 GMT

Thanks for the answers everyone,

I've followed the instructions and produced the logs4tac.tgz file with a size of 85.9MB (1.34GB uncompressed).

Some of the log files (43) are empty, some are up to 100MB and some of them have quite a bit of information on the network topology..

Are we looking for a specific file that might give an indication on the nature of the issue?

Re: Devices stuck in UNKNOWN state

pmuthuva — Thu, 08 Jun 2017 13:06:58 GMT

Please share apic-em-inventory-manager-service.log* logs.(assuming it is 2.0.X build)

If it is earlier build (1.4.X) , please share apic-em-inventory-manager-service.log, apic-inventory.log, xde.log, existenceInventory.log, inventory.log

Re: Devices stuck in UNKNOWN state

Vag — Thu, 08 Jun 2017 23:58:34 GMT

Assuming that you are referring to apic-em-inventory-manager-service build (unless I'm missing something and APIC-EM v2.0.x is out somewhere), it's version 5.1.3.1009 (APIC-EM Version 1.4.3.1009).

I have attached the apic-em-inventory-manager-service.log

Re: Devices stuck in UNKNOWN state

pmuthuva — Fri, 09 Jun 2017 02:17:57 GMT

I was mentioning about the APIC-EM version. Since it is 1.4.X please share apic-inventory.log, xde.log, existenceInventory.log, inventory.log logs too.

Re: Devices stuck in UNKNOWN state

pmuthuva — Fri, 09 Jun 2017 02:20:52 GMT

Please ignore. Looks the attachment has all log files. Will take a look and update you.

Re: Devices stuck in UNKNOWN state

Vag — Fri, 09 Jun 2017 03:16:23 GMT

Is there an APIC-EM v2.0.x out? Where can I get it?

Re: Devices stuck in UNKNOWN state

aradford — Fri, 09 Jun 2017 03:19:58 GMT

no. it has not been released yet.

there will be a 1.5 before 2.x

Re: Devices stuck in UNKNOWN state

pmuthuva — Fri, 09 Jun 2017 11:36:28 GMT

Please share inventory.log, apic-inventory.log, xde.log and existenceInventory.log too. I thought all these files were in the attachment. But only apic-em-inventory-manager-service.log is available and that does not have the info on the issue.

Re: Devices stuck in UNKNOWN state

Vag — Tue, 13 Jun 2017 00:42:44 GMT

I went through the requested logs and they had an enormous size, so I tried to narrow it down for the sake of making the whole troubleshooting process easier.

I took a backup of APIC-EM and a snapshot in VMWare, deleted all of the devices but those that could not be deleted, cleared the logs and then performed discovery, resync and delete operations on those remaining devices that were stuck.

Then I restarted the APIC-EM server and repeated.

I attached the requested log files that were produced.

Re: Devices stuck in UNKNOWN state

pmuthuva — Tue, 13 Jun 2017 10:23:12 GMT

Hi Vag,

There are some exceptions seen with respect to RawCliInfo table entries and found that we have a defect open for this issue of devices being stuck at 'Unknown' state.

To check the delete device issue, there is no relevant logs available. Enabling DEBUG logs and sharing the same set of logs would help to debug the issue with delete.

Command to enable DEBUG logs:

sudo /opt/CSCOlumos/bin/setLogLevel.sh inventory DEBUG

sudo /opt/CSCOlumos/bin/setLogLevel.sh disocvery DEBUG

This has to be run on the VM where apic-em-inventory-manager-service is running.

Re: Devices stuck in UNKNOWN state

Vag — Sun, 18 Jun 2017 23:27:16 GMT

I have attached the requested logs, produced by following the same procedure and enabling the debug options (delete, discovery etc).

Re: Devices stuck in UNKNOWN state

pmuthuva — Mon, 19 Jun 2017 11:35:21 GMT

Looking at the logs, the rootcause for both the issues that Device goes to 'Unknown' state and unable to delete device is same. There is a open defect for this issue.

Re: Devices stuck in UNKNOWN state

Vag — Wed, 21 Jun 2017 03:55:31 GMT

Is this related to an issue that is expected to be fixed in the next update release?