Cisco hardware and RoCE (RDMA)

[Dmitry Shleg]

Hello everyone, I'm testing the technology of Microsoft Storage Spaces Direct, the technology vendor recommends the use RDMA network cards, but I could not find the Cisco switch with RDMA support (RoCE)!
Of all the materials that I found on the network, the network equipment should run at a speed of at least 10G, support the Priority Flow Control and Jumbo Frames.
What equipment does Cisco support for RDMA? Please explain, is it necessary to have a RoCE (RDMA) support in the description of the network switch?
I already have Cisco equipment and I do not want to change the manufacturer!

Thx, for help... -)

[Philip D'Ath]

From what I understand RoCEv1 uses standard Ethernet frames with an ethertype of 0x8915.  RoCEv2 uses UDP.

So pretty much either should work on any Cisco switch.

A lower cost switch with 24 to 48 10Gbe ports is a Cisco Nexus 3524.
http://www.cisco.com/c/en/us/products/switches/nexus-3524-switch/index.html

You could also use the Cisco 3850 family, the Cisco 4500 family, Cisco 6800 family, or most of the Nexus family.  Pretty much anything with 10Gbe support.

[MrStinch01]

The Nexus 3524 does not support PFC which is required for RoCE. We have just got stung by this. The QoS guide says PFC is supported on the 3000 series but if you look carefully you find a caveat that the 3548 does not support PFC. A 3524 is just a 3548 but licensed for 24 ports. Without PFC you can't pause the frames which you need to do. RoCE stands for RDMA over Converged Ethernet, the RDMA part is Remote Direct Memory Access. This needs the pause frame to work for the traffic which is bypassing the hosts CPU. We are now looking at the Nexus 93180YC-EX as a possible solution.