Hi guys,
I am setting a fabric with 3 APIC M4 having VIC 1455 cards ... Each APIC VIC port-3 and VIC port-1 goes to Leaf 1 and 2 according to the topology below. Problem : APIC 1 can't see other 2 APICs. APIC 1 registers Leaf 1 and Leaf 2 but each remains with status inactive in "Fabric membership".
According to the guide : https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/server/m4-l4-server/APIC-M4-L4-Server/m_server_specifications.html The Cisco VIC 1455 has 4 ports: port-1, port-2, port-3, and port-4, from left to right.Port 1 and port 2 is one pair, corresponding to eth2-1 on Cisco APIC. Port 3 and port 4 is another pair, corresponding to eth2-2 on Cisco APIC.
Wondering why during the setup of APIC 1, APIC 2 and 3 can't be detected. + I can't ping the bond0.INFRA_VLAN IP of the other 2 APICs from APIC 1.
Sharing the VIC type as well (from CIMC)
How come that there is eth2-1, eth2-2, eth2-3 and eth2-4 ?
I am thinking that there is a problem on the APIC to Leaf connectivity.
See the output from APIC 1 GUI / system / controller
Does anyone have an idea on how this can be fixed ?
Thanks Guys
Hi @Nabari94 ,
Your cabling is spot on, so you can back out of that rabbit-hole. To me, it looks like no nodes have been registered.
Did you assign an ID to the first leaf discovered? That leaf will NOT be allocated a VTEP IP address until it has been given a node ID, and so no other nodes will be discovered until that is done. Similarly, once the first leaf has been given a VTEP address, the spines will then be discovered and will need to be given unique node IDs, they will then get their VTEP addresses, then the remaining leaves go through the same process and finally, the other APICs will then be automatically discovered and the cluster will form and THEN you'll be able to ping the other APICs.
So - start by checking in the APIC GUI, Fabric > Inventory > Fabric Membership >| [Nodes Pending Registration] - my bet is that you'll see either Leaf LF1001 or Leaf LF1002 sitting there waiting to be registered! (It will show the serial number of the leaf which will help you figure out if it should be given node ID 1001 or 1002)
You can see the registration process being demonstrated here: https://youtu.be/3_P8Rs7HkaE?t=166
Thanks for the confirmation on the cabling, i was worried ...
I went through the Fabric membership tab and did a couple actions even before. bottom line is I can register Leaf 1002 but the status remains inactive and never changes.
Did assign ID 1002 to the first leaf discovered (Leaf 1002). You can see the VTEP IP above, guessing that ID is valid.
LF1002 VTEP IP is even reachable from all 3 APICs (eKVM, just pasted 2 images below) but I am wondering why LF1002 remains inactive on APIC GUI ? I am thinking that if it becomes active then other 2 APICs will join the cluster ... I already did a full reset (touch clean/touch setup) multiple times... Same result. Any
Regards,
Nabari94
Hi @Nabari94 ,
Firstly I forgot to thank you for writing your questions so clearly - and with screendumps (Tip: Drag the corner of the screendump to make it bigger before posting, makes them much easier to read)
Regarding the inactive leaf - my first inclination is to just reboot it.
But first I think I'd try (from one of the APICs)
ACI-APIC-1# fabric 1002 show lldp neighbors
and hopefully you'll see your APICs as neighbours - something like
ACI-APIC-1# fabric 1002 show lldp neighbors ---------------------------------------------------------------- Node 1002 (Leaf1002) ---------------------------------------------------------------- Capability codes: (R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device (W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other Device ID Local Intf Hold-time Capability Port ID ACI-APIC-1 Eth1/46 120 eth2-1 ACI-APIC-2 Eth1/47 120 eth2-1 ACI-APIC-3 Eth1/48 120 eth2-1
You could also try and SSH to the leaf from one of the APICs as a connectivity test too,
But at the end of the day, I'd probably still reboot the leaf!
Now, I also have an apology, I missed something in your original post:
How come that there is eth2-1, eth2-2, eth2-3 and eth2-4 ?
and in fact your APIC should only show eth2-1 and eth2-2 when you look at /proc/net/bonding/bond0 - at least that's what my APIC shows
apic1# cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v5.15.126atom-1 Bonding Mode: fault-tolerance (active-backup) Primary Slave: None Currently Active Slave: eth2-2 MII Status: up MII Polling Interval (ms): 60 Up Delay (ms): 0 Down Delay (ms): 0 Peer Notification Delay (ms): 0 Slave Interface: eth2-1 MII Status: up Speed: 10000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: f4:ee:31:10:6c:50 Slave queue ID: 0 Slave Interface: eth2-2 MII Status: up Speed: 10000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: f4:ee:31:10:6c:51 Slave queue ID: 0
It worries me a bit when I see all four interfaces listed in your example above!
To actually see which interfaces are bonded is pretty hard to do, AFAIK you just have to trust the document you mentioned before that says ports 1 & 2 are bundled into one vNIC and ports 3 & 4 are bundled into the other vNIC.
Anyway - it might be worth checking the CIMC config too:
You should see two vNICs
