VIC 1385 in a UCS C220 M4 Without FI possible?

[Martinisoft1]

Dear all,

I have two UCSC220 M4 Rack servers and added a VIC1385 to each of them under the assumption that I could directly connect the servers trough them using a supported DAC cable.

The thing is that the cards are detecting the cable just fine but the link stays down.

When I do the same with two 10gb VIC's they do connect without any issue.

After trying to find info on how to fix this I found that in almost every scenario that I found on the internet a Fabric Interconnect is involved. Is it even possible to use a VIC 1385 without a FI, and if so what could I be doing wrong?

thanks in advance for thinking with me,

Martijn.

The Netherlands.

[Steven Tardy]

It's not clear from your description.

Are the two UCS c-series servers connected to a UCS Fabric Interconnect (with the MLOM VIC)?

When doing "Cisco UCS C-Series Integration with Cisco UCS Manager" ALL Cisco VICs MUST be connected to BOTH FIs.

https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/c-series_integration/ucsm4-0/b_C-Series-Integration_UCSM4-0/b_C-Series-Integration_UCSM4-0_chapter_01.html

For Cisco UCS C-Series Rack-Mount Servers integration with Cisco UCS Manager, 
Cisco VIC adapters can only be connected to Cisco UCS Fabric Interconnects 
and not to third-party switches. 

If these are stand-alone servers (no UCS Fabric Interconnects in the mix). . . I've never connected VIC 1385s together personally so unsure.

[RedNectar]

Hi @Martinisoft1,

I have never done what you are attempting, but if I were I'd make sure I checked the BIOS setting first to make sure that the CIMC management is set to Dedicated. (Sorry for the mocked up screen dump, my original showed Dedicated unset)

[Edit: Ooops - the following is only relevant for  the four port 10/25GB VIC 1457 - NOT the 2 port 40GB VIC 1385]

Next, I'd make sure that I only connect ports 1 & 3 to upstream switches - OR if I were to connect all four ports, I'd configure the ports that ports 1&2 go to as a trunk and ditto for 3&4

[Edit: Question: @Martinisoft1, what upstream device are you connecting the VIC 1385 to? And is configured for 40Gbps rather than 4x10Gbps?]

I hope this helps.

---

Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem.

---

[Martinisoft1]

Hello Chris and Steven,

Many thanks for your quick and extensive reply.

@Steven Tardy , there are no FI's involved what so ever. The Servers are configured as stand alone servers and are managed trough IMC, not UCS Manager.

@RedNectar I double checked the IMC network setting but it already is on dedicated.

Full disclosure, what I am trying to do here is to connect the two servers directly together with DAC cables without any switch infrastructure in between.

The VIC 1385's do see the cable and its firmware but the link stays down.

Maybe I am trying to get a scenario going that just cannot work.

But what I find strange about it is that connecting the 10gb MLOM NIC's together in the same way just works fine, and other card vendors like Mellanox and Intel all seem to be able to work just fine in a Switchless config with their 40GB QSFP+ cards (based on info I found online).

What I tried so far:

- Pinned the speed on both cards (tried both 40GB and 4x10GB)

- Removed the MLOM and made sure that the 1385 is seated in slot 1 (seems to be a requirement when using a direct connect scenario with FI's when using a 1385 in a C220 M4 so I thought maybe it would help to get my scenario running)

- Updated all the firmware

- Swapped the VIC 1385's to make sure that they are ok.

At the moment I run two DAC's in between the 1385's that I ordered from FS.com which are compatible with the DAC's on the 1385 support list. The cables are detected by the cards including all info like serial number and so on. Based on this is it safe to assume that the cables are not the issue or is there still a chance that original Cisco cables could solve the connection issue?

If you are sure that my scenario cannot work please let me know, than I would need to find other options to get this running, for instance with 3th party NIC's or by introducing FI's or switches.

My goal is to get Storage Spaces Direct on these servers, which is the storage component of Microsoft Azurestack HCI. In a two node scenario like mine a direct connection between the NIC's that handle the storage is preferred as this is the simplest and most robust scenario.

thanks again for being with me an this one,

Martijn.

[RedNectar]

Hi Martin,

OK, I see the picture much better now, but have never tried to do that. So I don't think there's any more I can do to help - I'm going to watch from the sidelines now.

[Martinisoft1]

Hello Chris,

thanks for your followup, I understand that my scenario is not common ;). Do you happen to have any thoughts on the DAC cable part of my question? Is it save to assume that a Cable that is detected by the NIC could not be the source of the issue or is there still a chance that introducing original Cisco cables can make a difference? 

kind regards,

Martijn.

[RedNectar]

Hi Martin,

Well, my official advice is of course is that you should definitely be using Cisco branded everything.

But it is fair to say that on occasions, I have not been able to procure such items in the constraint of time

And when it comes to DAC cables, I've not had any troubles, although I can only think of one instance where the 40G cables were involved.

And keep in mind my work is mostly training rather than installing, plus a bit of lab configuration where we keep costs as low as possible.

So on balance, I suspect it is NOT the cable, but there's only one way to find out.

[Wes Austin]

What you are mentioning is possible, so lets start with confirming the PID on the DAC cable that you are using? Is it a supported Cisco cable? Can you share the part number?

Can you also send a screen capture from your CIMC like this one?

[Martinisoft1]

Hello Wes,

Thanks for the hopeful response. At the moment we are using 3th party cables from FS.com that should be compatible, but I think I will go ahead and order original Cisco cables since the issue could very well leigh in the cables now I know that this setup should be able to run.

This is the cable we are currently using: Cisco QSFP-H40G-CU50CM Compatible 40G QSFP+ DAC Twinax Cable - FS Germany

adapter (mcp):2# dcem-port 1
  uif :           1
  port:           0
  mac:            28:ac:9e:c4:bf:ac
  adminstate:     ENABLED
  operstate:      DOWN
  linkdowncnt:    0
  adminspeed:     4x10g
  maxframesize:   9216
  pausetype:      STD
  pausemap:       0xa0
  xcvr_status:    OK
  xcvr_flags:     MODPRESENT, ROMLOADED
  sfp id :        1
  xcvr_type:      UNKNOWN
  sfprom:
    [00] 0d 00 04 00 00 00 00 00 00 00 00 00 00 00 00 00  |................|
    [10] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  |................|
    [20] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  |................|
    [30] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  |................|
    [40] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  |................|
    [50] 00 00 00 00 00 00 0f 00 00 00 00 00 00 00 00 00  |................|
    [60] 00 00 00 00 f0 00 00 f0 f0 00 00 00 00 00 00 00  |................|
    [70] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 02  |................|
    [80] 0d 10 23 08 00 00 00 00 00 00 00 05 67 00 00 00  |..#.........g...|
    [90] 00 00 01 a0 46 53 20 20 20 20 20 20 20 20 20 20  |....FS          |
    [a0] 20 20 20 20 00 00 09 3a 51 53 46 50 2d 50 43 30  |    ...:QSFP-PC0|
    [b0] 30 35 20 20 20 20 20 20 30 33 04 06 00 00 46 f3  |05      03....F.|
    [c0] 00 00 00 c0 46 31 39 34 30 31 37 38 37 38 30 2d  |....F1940178780-|
    [d0] 31 20 20 20 32 30 32 30 31 31 32 37 00 00 00 60  |1   20201127...`|
    [e0] 00 00 08 3d 1b 51 38 bc 74 ef c6 58 fa 5f 59 03  |...=.Q8.t..X._Y.|
    [f0] ca 5a ba 00 00 00 00 00 00 00 00 00 92 74 19 c1  |.Z...........t..|
    [100] 0d 10 23 08 00 00 00 00 00 00 00 05 67 00 00 00  |..#.........g...|
    [110] 00 00 01 a0 46 53 20 20 20 20 20 20 20 20 20 20  |....FS          |
    [120] 20 20 20 20 00 00 09 3a 51 53 46 50 2d 50 43 30  |    ...:QSFP-PC0|
    [130] 30 35 20 20 20 20 20 20 30 33 04 06 00 00 46 f3  |05      03....F.|
    [140] 00 00 00 c0 46 31 39 34 30 31 37 38 37 38 30 2d  |....F1940178780-|
    [150] 31 20 20 20 32 30 32 30 31 31 32 37 00 00 00 60  |1   20201127...`|
    [160] 00 00 08 3d 1b 51 38 bc 74 ef c6 58 fa 5f 59 03  |...=.Q8.t..X._Y.|
    [170] ca 5a ba 00 00 00 00 00 00 00 00 00 92 74 19 c1  |.Z...........t..|

In the above info you will find that the admin speed is 4x10g but we also tried Auto and 40g on both ends.

Thanks to you all, I really appreciate all the help.

Martijn.

[Wes Austin · Accepted solution]

Yeah I would start with getting an authentic supported cable. Unfortunately I do not have any experience with those third party cables, but I do know that a lot of vendors like to say they are "compatible" but doesn't necessarily mean they are going to work.

Please update once you are able to test with a Cisco cable.

[Martinisoft1]

Thanks Wes,

The cable is ordered and on its way. As soon as I have results I will post them here.

kind regards,

Martijn.

[Martinisoft1]

After replacing the DAC Cable for an original Cisco Cable the link came up immediately. So no 3th Party cables anymore for me

So for the record, connecting two VIC 1385's together without any switch or FI works without any issue's.

thanks again to you all,

Martijn.