APIC-M4 doesn't detect adjacent Leafs during APIC cluster setup

[Min Myo]

Hello, Need help with M4 (C225 M6) loading APIC 6.0.2h. Have both 1gig onboard and Intel 10/25g SFP NIC. CIMC doesn't show VIC nor Intel-NIC option under CIMC|Networking.  The issue I have now is APIC can't detect adjacent Leaf Switches.

• is it firmware related? 4.2(2g) is the current firmware running. The release note 4.2(2g) says should be fine. 
• physical layer issue - SFP, cable, etc.?  The cables are going right places, but NO link-lights are shown on both ends. Tried swapping out different SFPs as well as cables.

The same code level loaded on both APIC and fabric switches. Any clue/tip? Appreciated. 

@Robert Burns thank you for a suggesting to post as new.

[Robert Burns]

Did the fabric uplinks on this APIC ever work?
Which SFP and Cable are you using exactly on both sides?
Have you tried connecting to a different Leaf port?

Robert

[Min Myo]

They are brand new equipments and 6.0.2x is the first time we're putting together.  Never have the links on fabric switches nor APIC itself.

Cisco SFP-25G-SR-S MM are used on both over MMF cables.  No, haven't tried on different ports yet. 

-Min

[Robert Burns]

Do you have more than 1 APIC exhibiting this behavior?

Robert

[Min Myo]

Yes, I do have 2 other racks with the same config, having the same behavior. Total of 3 APICs, 6 Leafs and 2 Spines. 

[RedNectar]

Hi @Min Myo ,

It worries me that you say

CIMC doesn't show VIC nor Intel-NIC option under CIMC|Networking. 

---

I would expect that you'd at least see something like (this is an APIC M3 running APIC v5.2 CIMC v5.1(3a))

If you DON'T see something like this, and the system is new, I'd be raising a TAC case.

Apart from that - make sure the cables are connected to ports 1 and 3 of the VIC (2 and 4 might work too, but 1 & 3 is recommended. Connecting ports 1 &2 or 3&4 WON'T work)

[Min Myo]

Thanks @Robert Burns. Yes, our APIC-M3 does show exactly like that in CIMC and use of 1&3 are our standard config. However, new M4 we have doesn't come with VIC and uses Intel NIC instead, which comes with only 2-ports, instead of 4. Apparently, that might be new architecture, just assuming. 

We're working /w TAC now and they're scratching their head, too. Thanks for your quick replies. Appreciate it.

-Min 

[Petar Rusev]

@Min Myo Have you found solution with TAC?

I have the same issue running APIC-L4 with Intel E810XXVDA2, we use 10G MM SFPs, running version 6.0.2h
What I see that again in CIMC-> Networking there is nothing. Initially I was thinking it is a CIMC bug, but after testing with 4.2.3e and 4.3.3.240022 I still don't see anything.

What I also see is strange behavior in the NIC redundancy. I have followed the traditional config: NIC Mode: Dedicated and NIC Redundancy: None, but the Intel card is changing the active link in bond0 quite randomly and I have couple of times the situation where I see both Leaf switches connected to the same APIC waiting in Nodes Pending Registration. I haven't seen this so far with M3/L3s.

I tried couple of times registering Leafs: 1st checking the active NIC in bond0, verifying the SN via LLDP, registering the correct SN and it always hangs and the DiscoveryIssues say it cannot download the BootScript.

I am kind of stuck and looking for any advice.