Linux VIC enic driver SR-IOV sysfs support?

[bpkroth]

Hi, I have a pile of Cisco C220M4 and C240M4 servers running in a testbed for researchers to use.  They each have a VIC 1227 card in them.  The machines are running in standalone mode (there's some other software managing certain aspects of them and the switches they're attached to, so we can't use UCS).  The primarily run Linux (typically either Ubuntu 14.04, or Centos 6.6 - an RHEL variant).  The kernel version involved in the Ubuntu case is 3.13.  In my tests, I've enabled VTd in the BIOS and booted the kernel with the intel_iommu=on parameter.

I want to be able to allow individual experimenters to configure some aspects of the VIC for themselves, but since I need to be able to reset them to a clean state after they're done using them, I can't give them access to the CIMC web or XML interface.  They are allowed to interact with the BIOS and other controller settings via an automatic Serial over IPMI tunnel, but I haven't seen a way to configure the VIC from any of those views.

Ideally, there would be a way for them to control those bits for themselves from within the OS itself.

Short of giving them full control, I thought I would start with getting SR-IOV support working on the enic driver.  Aside from that, it would benefit our libvirt/kvm infrastructure as well.

The 1GB igb driver nics on the machines show support for SR-IOV, however, the 10GB enic driver VIC vNics do not.

From my quick skim of the enic driver code, this should have some support, though I didn't see any reference to any sysfs exposure as described at [1].  However, when I examine the vNics exposed via sysfs, ethtool, virsh nodedev-dumpxml, etc. interfaces, they don't appear to provide that support.

Just to see, I tried enabling a usnic and some vmfex interfaces on the VIC as well.  Interestingly, the usnic device does actually claim to support SR-IOV.  The vmfex interfaces do not, but I don't think that would help me even if it did, since we're not running in UCS mode and the switches are configuring the client ports.  Since usnic doesn't appear to be supported in Ubuntu, and it doesn't have full support for all of the other features that we'd normally want to use, I'm not sure this is helpful either.

Can someone please confirm for me whether or not SR-IOV is actually supported by the vNics (enic driver) on a VIC for Linux, and if so, how to set it up?  Also, is my read on vmfex off?

Thanks,
Brian

[1] <http://lxr.free-electrons.com/source/Documentation/PCI/pci-iov-howto.txt>

[bpkroth]

For what it's worth, I was told that the VIC/vNIC/enic can't currently do this.  "sudo lspci -vvv" is really the most definitive way to see this.  The standard "VIC Ethernet NIC" interface shows only one type of VF Device ID: 00cf, which corresponds to the "VIC Userspace NIC", which isn't really an ethernet style device I'd expect to give to a VM.  The vmfex devices also do not show any VF information (even leading backwards to a PF), which is less helpful since I can't tell which physical interface they'd be assigned to, for instance.

Anyways, Intel X520s seem to work nicely via sysfs so far.  Hopefully that helps someone else out at some point.

Cheers,
Brian