APIC Failing Ping

[Rem Markov]

Hey! 

So we have 3 apics that are extermly slow, both the CLI and the GUI  so we wanted to upgrade them, but the upgrade failed due to unknown reasons.

When trying to understand and troubleshoot We ran into something weird.

phmoapc-166014-24# acidiag cluster
Admin password:

Running...

Checking Wiring and UUID: OK
Checking AD Processes: Running
Checking All Apics in Commission State: OK
Checking All Apics in Active State: OK
Checking Fabric Nodes: OK
Checking Apic Fully-Fit: Not Fully Fit Apics: IFC-1 IFC-2 IFC-3
Checking Shard Convergence: OK
Checking Leadership Degration: Non optimal leader for shards : 3:1,3:2,3:4,3:5,3:7,3:8,3:10,3:11,3:13,3:16,3:17,3:19,3:20,3:22,3:23,3:25,3:26,3:28,3:31,6:1,6:2,6:4,6:5,6:7,6:8,6:10,6:11,6:13,6:16,6:17,6:19,6:20,6:22,6:23,6:25,6:26,6:28,6:31,6:32,9:1,9:2,9:4,9:5,9:7,9:8,9:10,9:11,9:13,9:16,9:17,9:19,9:20,9:22,9:23,9:25,9:26,9:28,9:29,9:31,10:1,10:2,10:4,10:5,10:7,10:8,10:10,10:11,10:13,10:14,10:16,10:17,10:19,10:20,10:22,10:23,10:25,10:26,10:28,10:31,11:1,11:2,11:4,11:5,11:7,11:8,11:10,11:11,11:13,11:14,11:16,11:17,11:19,11:20,11:22,11:23,11:25,11:26,11:28,11:31,14:1,14:2,14:4,14:5,14:7,14:8,14:10,14:11,14:13,14:14,14:16,14:17,14:19,14:20,14:22,14:23,14:25,14:26,14:28,14:31,16:1,16:2,16:4,16:5,16:7,16:10,16:11,16:13,16:14,16:16,16:17,16:19,16:20,16:22,16:23,16:25,16:26,16:28,16:31,22:1,22:2,22:4,22:5,22:7,22:8,22:10,22:11,22:13,22:14,22:16,22:17,22:19,22:20,22:22,22:25,22:28,22:31,22:32,23:1,23:2,23:4,23:5,23:7,23:8,23:10,23:11,23:13,23:14,23:16,23:17,23:19,23:20,23:22,23:23,23:25,23:26,23:28,23:31,33:1,34:1,34:2,34:4,34:5,34:7,34:8,34:10,34:11,34:13,34:16,34:17,34:19,34:20,34:22,34:23,34:25,34:26,34:28,34:31,35:1,35:2,35:4,35:5,35:7,35:8,35:10,35:11,35:13,35:14,35:16,35:17,35:19,35:22,35:25,35:26,35:28,35:31,36:1,39:1,39:2,39:4,39:5,39:7,39:8,39:10,39:11,39:13,39:14,39:16,39:17,39:19,39:20,39:22,39:23,39:25,39:26,39:28,39:31
Ping OOB IPs:
APIC-1: 192.168.199.9 - OK
APIC-2: 192.168.199.10 - OK
APIC-3: 192.168.199.11 - OK
Ping Infra IPs:
APIC-1: 10.0.0.1 - OK
APIC-2: 10.0.0.2 - OK
APIC-3: 10.0.0.3 - Ping failed
Checking APIC Versions: Cluster Version:5.2(7g) Imcompatible Apics: IFC-1(5.2(4e)) IFC-2(5.2(4e)) IFC-3(5.2(4e))
Checking SSL: OK
Full file system(s): None

Done!

For some reason the ping between the infra IPs fails, does someone has a clue where to begin searching for the error?

[Rem Markov]

We restarted one of the APICs once and it took two days, is there anything else I can try before?

[Robert Burns]

Complete the ping test above.  If APIC2 can reach APIC1, then we know the physical layer should be fine, and it's likely a forwarding issue on the fabric.  Another less intrusive test could be to disconnect the current active fabric link for APIC3 (bringing the standby online).  See if that has any impact.  Short of that, you're getting into TAC terriritory if you want a proper RCA.  I'd just bounce it myself.  If your APIC is taking more than 30mins to come back healthy, that's a whole other problem you should pursue.  How large is this fabric btw?

Robert

[Rem Markov]

we have 2 spines and 4 leafs . 
which I'm trying to upgrade now to the latest version and it takes up until now more then 30 hours.

When I tried restarts apic1 via the CiMC  it took almost a day, I'll to restart APIC2 tomorrow.

[Robert Burns]

Yeah sorry meant to say testing both directions APIC1 > 3, then APIC3 > 1.....

[Rem Markov]

Yea , it doesn't work sadly.

[Rem Markov]

I have noticed something weird.
When running show lldp on apic3 it returned the eth2/1 and eth2/2 are connected to eth1/2 on 2 leafs.
however, when I shut down the port via the GUI and rerun the lldp once again it showed that  eth2/1 and eth2/2 are connected to eth1/1 on 2 leafs.

Now does it mean that there is a bond? or is there an extra cable?

[Rem Markov · Accepted solution]

I finally found the problem.

I had 4 connections between APIC3&1 and the leafs instead of 2.

VIC 1455 is used on the APICs. All  4-ports on this NIC were cabled to the Leaf.

after shut 2 ports it worked!