Hi Ju,
As a first step please log into the APIC GUI and verify that the Fabric -> Inventory -> Fabric Membership page is empty. If you see one serial number, then double click the serial number to assign the serial number a node ID and node name.
If the page is empty, then read further on for troubleshooting.
Verify that the date/time are within a close range. Run the date command on the APIC and leaf.
apic# date
Tue Sep 13 06:14:34 ART 2016
leaf# date
Tue Sep 13 06:14:55 ART 2016
If they are more than a few hours apart, then this may cause issues with fabric discovery.
Verify that both the APIC and leaf are running same code or at least within the relevant code number [2.0(2)]
Fabric discovery also requires LLDP then DHCP. According to your post LLDP appears to be fine on the leaf side, but lets verify on the APIC.
The APIC interfaces going to leaf nodes are in a active-standby bond (bond0). We should find out the active interface of the bond to begin troubleshooting. To find the active interface, try running cat /proc/net/bonding/bond0
admin@APIC1:~> cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth2-2 <-- Eth2-2 is active
MII Status: up
MII Polling Interval (ms): 60
Up Delay (ms): 0
Down Delay (ms): 0
Slave Interface: eth2-1
MII Status: down
Speed: Unknown
Duplex: Unknown
Link Failure Count: 0
Permanent HW addr: b8:38:61:f7:05:b1
Slave queue ID: 0
Slave Interface: eth2-2
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 1
Permanent HW addr: b8:38:61:f7:05:b2
Slave queue ID: 0
Check LLDP on the active interface of the bond0 using show lldptool in ethx-y
This will display information about the neighbor switch
admin@apic:~> show lldptool in eth2-2
Chassis ID TLV
MAC: 10:05:ca:f5:bb:71
Port ID TLV
Local: Eth1/1
Time to Live TLV
120
Port Description TLV
System Name TLV
System Description TLV
System Capabilities TLV
System capabilities: Bridge, Router
Enabled capabilities: Bridge, Router
Management Address TLV
MAC: 10:05:ca:f5:bb:71
Ifindex: 83886080
Cisco 4-wire Power-via-MDI TLV
4-Pair PoE not supported
Spare pair Detection/Classification not required
PD Spare pair Desired State: Disabled
PSE Spare pair Operational State: Disabled
Cisco Port Mode TLV
0
Cisco Port State TLV
1
Cisco Serial Number TLV
xxxx <--- Serial #
Cisco Model TLV
N9K-C9396PX <--- Model of neighbor
Cisco Firmware Version TLV
n9000-12.0(1q)
Cisco Node Role TLV
1
Cisco Infra VLAN TLV
xxxx
Cisco Node IP TLV
xxxxxxx
Cisco Name TLV
xxxxxxx
Cisco Fabric Name TLV
xxxxxxx
Cisco Node ID TLV
xxxxxxx
Cisco POD ID TLV
1
Cisco Appliance Vector TLV
If LLDP looks okay, then look into DHCP.
admin@apic:~> moquery -c dhcpPool | grep -B 8 -A 5 2016-02
# dhcp.Pool
id : 2
childAction :
className : pod
dn : prov-1/net-[10.0.0.0/16]/pool-2
endIp : 10.0.128.95
freeIPs : 4294967295
lcOwn : local
modTs : 2016-09-10T13:09:08.203+00:00
rn : pool-2
startIp : 10.0.128.64
status :
type : normal
# dhcp.Pool
id : 3
childAction :
className : protectionchain
dn : prov-1/net-[10.0.0.0/16]/pool-3
endIp : 10.0.48.95
freeIPs : 4294967280
lcOwn : local
modTs : 2016-09-10T13:11:33.392+00:00
rn : pool-3
startIp : 10.0.48.64
status :
type : normal
--
# dhcp.Pool
id : 1
childAction :
className : vip
dn : prov-1/net-[10.0.0.0/16]/pool-1
endIp : 10.0.160.95
freeIPs : 4294967288
lcOwn : local
modTs : 2016-09-09T18:18:37.674+00:00
rn : pool-1
startIp : 10.0.160.64
status :
type : normal
===============================================
Gotchas: Make sure that your CIMC NIC Mode is not set to Shared LOM extended since this impacts your VIC interfaces. VIC interfaces are used to connect to the leaf ports.
