Fabric Discovery Issues

[ZacharyDunphy1649]

I want to start by saying this is my last resort.  I have been trouble shooting for close to a month now using the resources I have via google, Cisco boards, and the limited support our Cisco partners have been able to offer because of how busy they are with other projects.  I'll do my best to provide all the information necessary, but please ask questions if I leave anything out.

Software Files:

APIC Version: 9.4.1.1

Switch Versions: 9.14.1.1

"Show lldp neighbor" output on the leaf on the APIC active slave port (eth2-1):

Capability codes:
(R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
(W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other
Device ID Local Intf Hold-time Capability Port ID
APIC1 Eth1/1 120 eth2-1
switch Eth1/49 120 BR Eth1/1
Total entries displayed: 2

(the switch on eth1/49 is the spine)

"cat /proc/net/bonding/bond0" output from APIC:

Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth2-1
MII Status: up
MII Polling Interval (ms): 60
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth2-1
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 3
Permanent HW addr: 00:fd:22:98:b7:9a
Slave queue ID: 0

Slave Interface: eth2-2
MII Status: down
Speed: Unknown
Duplex: Unknown
Link Failure Count: 1
Permanent HW addr: 00:fd:22:98:b7:9b
Slave queue ID: 0

Product # for the cabling between APIC and Leaves:

SFP-H10GB-CU1M=

Time/date on both Leaves and APIC is withing 1-2 seconds of eachother.

APIC has been Touch Cleaned and rebooted at least twice.  And the Leaves have had boot Clean configs on several occasions with no change.

APIC NIC mode is dedicated.

LLDP is disabled on the APIC vNIC.

There are no "wiring issues" outputs from any of the moquery commands on the APIC

There is no vlan mismatch on the leaf switches.

I may have missed some of the troubleshooting information, but that should be a decent summary of what I know and what I have tried.  I am just looking for a way forward with troubleshooting.  Any help and/or direction is appreciated. 

P.S. I've also attached a drawing of our basic lab layout.  The pertinent information being, we are running 1 spine and 2 leaves.

V/R,

Zachary J. Dunphy

Network Engineer

Integration Services Inc.

zdunphy@go-isi.com

[Claudia de Luna]

Hey @ZacharyDunphy1649, I think I may have missed it, but what issue are you seeing?  Also, ACI uses LLDP for discovery so I'm not sure I'd be disabling it anywhere.  I'm assuming this is a mini fabric?

[ZacharyDunphy1649]

The issue is in the title i suppose. I am having issues discoverying my Leaves.  I am well aware LLDP is used for discovery.  Disabling LLDP on the vNIC is a common troubleshooting step found in most basic troubleshooting guides. Not sure what you mean by Mini-fabric, but it is two leave and one spine in a lab that will primarily be used for training once is it up and running.

[Claudia de Luna]

So you've checked version, time, optics compatability, BIOS, FPGA, EPLD versions I'm assuming?

how about the full output including the prompt of acidic avread from apic and leafs?

#acidiag avread

[ZacharyDunphy1649]

APIC1# acidiag avread
Local appliance ID=1 ADDRESS=10.0.0.1 TEP ADDRESS=10.0.0.0/16 ROUTABLE IP ADDRESS=0.0.0.0 CHASSIS_ID=398f106a-1129-11ea-9a88-4d15e49c1f9d
Cluster of 1 lm(t):1(zeroTime) appliances (out of targeted 1 lm(t):1(2019-12-10T15:40:57.860+00:00)) with FABRIC_DOMAIN name=ISI_Fabric set to version=4.1(1l) lm(t):1(zeroTime); discoveryMode=PERMISSIVE lm(t):0(1970-01-01T00:00:00.001+00:00); drrMode=OFF lm(t):0(1970-01-01T00:00:00.001+00:00)
appliance id=1 address=10.0.0.1 lm(t):1(2019-12-10T15:40:49.310+00:00) tep address=10.0.0.0/16 lm(t):1(2019-12-10T15:40:49.310+00:00) routable address=0.0.0.0 lm(t):1(zeroTime) oob address=192.168.80.101/24 lm(t):1(2019-12-10T15:40:56.154+00:00) version=4.1(1l) lm(t):1(2019-12-10T15:40:56.380+00:00) chassisId=398f106a-1129-11ea-9a88-4d15e49c1f9d lm(t):1(2019-12-10T15:40:56.380+00:00) capabilities=0X3EEFFFFFFFFF--0X2020--0X1 lm(t):1(2019-12-10T15:45:59.934+00:00) rK=(stable,present,0X206173722D687373) lm(t):1(2019-12-10T15:40:56.158+00:00) aK=(stable,present,0X206173722D687373) lm(t):1(2019-12-10T15:40:56.158+00:00) oobrK=(stable,present,0X206173722D687373) lm(t):1(2019-12-10T15:40:56.158+00:00) oobaK=(stable,present,0X206173722D687373) lm(t):1(2019-12-10T15:40:56.158+00:00) cntrlSbst=(APPROVED, WZP22480273) lm(t):1(2019-12-10T15:40:56.380+00:00) (targetMbSn= lm(t):0(zeroTime), failoverStatus=0 lm(t):0(zeroTime)) podId=1 lm(t):1(2019-12-10T15:40:49.310+00:00) commissioned=YES lm(t):1(zeroTime) registered=YES lm(t):1(2019-12-10T15:40:49.310+00:00) standby=NO lm(t):1(2019-12-10T15:40:49.310+00:00) DRR=NO lm(t):0(zeroTime) apicX=NO lm(t):1(2019-12-10T15:40:49.310+00:00) virtual=NO lm(t):1(2019-12-10T15:40:49.310+00:00) active=YES(2019-12-10T15:40:49.310+00:00) health=(applnc:255 lm(t):1(2019-12-10T15:42:34.121+00:00) svc's)
---------------------------------------------
clusterTime=<diff=1 common=2019-12-12T17:14:42.592+00:00 local=2019-12-12T17:14:42.591+00:00 pF=<displForm=0 offsSt=0 offsVlu=0 lm(t):1(2019-11-27T15:26:17.646+00:00)>>
---------------------------------------------

[ZacharyDunphy1649 · Accepted solution]

Solved.