S3260 M4 DAC No Link

[matthias8422]

I bought a s3260 with 2 M4 nodes as a home server (currently only using 1 the other is currently not plugged in), I am attempting to get it connected to my Microtik router but the port says No Link. I am using a qsfp+ to 4x sfp+ breakout DAC cable. I have tried 10gtec and FS, I have not yet tried a Cisco branded breakout cable but both of the ones i did try displayed as compatible. I have tried many different settings, auto negotiation as well as setting it manually to 4x 10G. Nothing seems to work. Im at a total loss atm and no amount of googling seems to help. Any incite as to what i could be doing wrong would be great. I have attached screenshots and other stuff that i thought might be needed.

[Steven Tardy]

On the Network / SIOC1 / External Ethernet Interfaces tab:

Try changing Port-0 to:

• Admin Speed = 4x10Gbps
• Link Training = Off

From that same External Ethernet Interfaces tab you should see Link State column change to "Link Up" when working.

The logs show the SFP/cable in Port-0 has Serial Number G2220234174 which is not a Cisco cable but is a FS cable.
Not saying unsupported cable is the issue, but could be the issue.
Some products (on either side) don't have enough (oomph) power to transmit over long(er) DAC cables.
Also switch compatibility (link training features) may come into play here.

The S3260 spec sheet:

• https://www.cisco.com/c/dam/en/us/products/collateral/servers-unified-computing/ucs-s-series-storage-servers/s3260-specsheet.pdf

Calls out PID CVR-QSFP-SFP10G as a way to convert the 40G port to 10G for external connectivity.

[matthias8422]

I am unable to change the link training state, the option is greyed out for me, is there a different setting i have too change first to enable that drop-down? I have looked at that spec sheet before and did search for that particular cable but could not find the Cisco branded one without paying almost $500. The FS cable should be compatible but if you have a link to a reseller that's selling the Cisco branded one id be willing to try that as long as the price isn't too outrageous (just a home server after all.

[Steven Tardy]

First change the Admin Speed drop down.
I'm not 100% sure what 4x10Gbps options there are and which 4x10Gbps options work as I was usually changing to 40Gbps.

Screenshots from some internal docs I wrote on the opposite issue (making 40Gbps work):

Change Admin Speed

Change Link Training

[matthias8422]

I think perhaps that this adapter card does not have that option. as the option is greyed out. I also tried changing it via ssh and the option does not exist there either as shown in the last screenshot link-training isnt an auto complete option

[Steven Tardy]

Your VIC logs show what seems to be receiving something which is not expected:
Sync_OK, then immediately NOT_Sync_OK.

230524-08:04:04.716813 paloerrd.paloerrd ERROR: INTR: discovered interrupt on Module NET Register MREGS_BMC_INT_PCS_RX value 0x1 on 32 bits & bit name: NET_MREGS_BMC_INT_PCS_RX_INVALID_SYMBOL (masked bit)
230524-08:04:04.716871 paloerrd.paloerrd ERROR: INTR: discovered interrupt on Module NET Register MREGS_BMC_INT_PCS_RX value 0x4 on 32 bits & bit name: NET_MREGS_BMC_INT_PCS_RX_SYNC_OK (masked bit)
230524-08:04:04.716943 paloerrd.paloerrd ERROR: INTR: discovered interrupt on Module NET Register MREGS_BMC_INT_PCS_RX value 0x8 on 32 bits & bit name: NET_MREGS_BMC_INT_PCS_RX_NOT_SYNC_OK (masked bit)

Still may be something in the cable or the switch side.

What is the switching device and how is the switchport configured related to speed?

40Gbps Ethernet does feel like Ethernet reverted to 1995 having to match speed and duplex settings (10Mbps vs 100Mbps).
Both sides need to speak the same dialect of 40Gbps (or 4x10Gbps or 10Gbps or Auto) Ethernet (link training vs no link training) for links to come up.
Both sides claiming the "other" side is to blame. *sigh*.
Throw in 40Gbps - 4x10Gbps cables just to confuse everything.

[matthias8422]

The switch im using is a microtik CRS328-24P-4S+RM. As for the speed i have tried auto, 40Gbps full duplex, 40Gbps half, 10Gbps Full and 10Gbps half. Rate select low and high, changing the tx and rx flow control to off, on and auto and Loop protect to on and off

[Steven Tardy]

Digging around I found this doc:

• https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/s/hw/S3260/installb/S3260/replace.html#24127

Which details:

Setting SIOC QSFP Port Speed

Optional: Set the port speeds for the SIOC QSFP ports:
Step 1 Connect a keyboard and console to the system or log in remotely and view a virtual KVM window.
Step 2 Reboot the system and press F8 when prompted to launch the Cisco IMC Configuration Utility.
Step 3 After the first utility screen is displayed, press F1 twice to go to the third utility screen, which has the settings for Adapter Port Speeds.
Step 4 Set the desired port speeds. “Adapter-1” is SIOC 1; “Adapter-2” is SIOC 2, if present.
Step 5 Press F10 to save your changes and exit the utility.

Maybe on the S3260 (vs I am most familiar with C240) this External Ethernet Interface must be configured via the BIOS screen and not CIMC?

Hope that helps.

[matthias8422]

They were initially blank so i set them to 4x10Gbps, still no connection on any of the 4 ports. I restarted and set them to auto, 4x10 and 40 and tried again, still nothing. I have a new switch of the same type ariving next week, as this one was dinged during shipping. Its possible that did something to the SFP ports, though it seemed to just be cosmetic. Ill try with the new one when it gets here. Also going to look more into a cisco branded DAC cable just cant seem to find them anywhere, everything i found so far is "compatible"