APIC 2 stuck in unavailable.

Amaya · ‎08-17-2023

Hi,

I have a new deployment, a typical setup, my second APIC out of 3 is stuck in unavailable. I have blown the fabric up and tried it again based on TACs recommendations. I have upgraded the Leaf switches to the same version as the primary APIC and so far we are still seeing the same issue.

I am wondering if anyone has seen this issue and what could help us come out of the situation. Also, is it an issue to proceed with the configuration on just the primary APiC,

Thank you,

Robert Burns · ‎08-21-2023

1. What version are you running.
2. Which model of APIC
3. What's the topology - Is this multipod/single pod fabric?
4. Is there any message on the CLI of the faulty APIC?
5. Are you seeing LLDP info being Sent/Received by the APIC?

If you already consulted/engaged TAC, they should be able to resolve this or replace the controller.

Robert

Amaya · ‎08-22-2023

Hi Robert,

1. What version are you running.

- Running 5.2(5c), upgraded switches to the same version as well.

2. Which model of APIC

- APIC-SERVER=L3
3. What's the topology - Is this multipod/single pod fabric?

- Single pod

4. Is there any message on the CLI of the faulty APIC?

-

the messaging layer was unable to deliver the stimulus (no replica is available)

1 xxxxxxxAPIC101 172.17.0.1 in-service unavailable data-layer-
synchronization-in-
progress...
2 xxxxxxAPIC102 172.17.0.2 in-service available data-layer-partially-
degraded-leadership

5. Are you seeing LLDP info being Sent/Received by the APIC?

Yes, LLDP is available as it see's the connections to the attached leaf switches.

The above error leads to something similar as the following. However we have checked all of the cabling on the back of the APIC.

https://community.cisco.com/t5/application-centric-infrastructure/aci-apic-error-the-messaging-layer-was-unable-to-deliver-the/td-p/4270831

I am still working with TAC, waiting to hear back from them.

Robert Burns · ‎08-22-2023

Gather and share the following commands from the problem APIC and from one of the healthy APICs (capture it to a log file and attach to this post):

acidiag verifyapic
acidiag rvread
acidiag avread
acidiag run lldptool in eth2-1
acidiag run lldptool in eth2-2
date

Robert