Libvirt SR-IOV pools on a Cisco UCS in the CentOS7

Libvirt allows you to create a network definition with all the virtual functions (VF) from the specified SR-IOV physical function (PF). It will assign one of the available VFs to the guest VM connected to such network automatically when it’s powered on. This allows you to avoid hardcoding of the VF PCI addresses into the configuration of the guest.

To enable SR-IOV for the static vNIC on the Cisco UCS You need to create a Dynamic vNIC Connection Policy with the desired number of the dynamic vNICs and assign it to the static vNIC.

You will also need to enable VT-d in the BIOS policy of the service profile (by selecting the predefined SRIOV BIOS policy for example) and enable the IOMMU support for your Linux kernel (see the “17.1.6.2. Using SR-IOV” section of the “Guest virtual machine device configuration” chapter from the RH Virtualization Deployment and Administration Guide)

Typical example, like the one from the section “17.1.6.4. Setting PCI device assignment from a pool of SR-IOV virtual functions” of the mentioned chapter might not work on the UCS:

<network>
   <name>passthrough</name> <!-- This is the name of the file you created -->
   <forward mode='hostdev' managed='yes'>
     <pf dev='enpXs0f0'/>  <!-- Use the netdev name of your SR-IOV devices PF here -->
   </forward>
</network>

In my case I was getting the internal error: missing IFLA_VF_INFO in netlink response error after trying to power on the VM connected to the “passthrough” network.

In order to work properly, the SR-IOV VF needs the Port Profile and the Port Profile Client created on the UCS (VM tab in the GUI, see the Cisco UCS Manager VM-FEX for KVM GUI Configuration Guide for more details on configuring port profiles on the UCS) and specified in the Libvirt network or interface definition. This allows you to specify the allowed VLANs for the VM configured to use that port profile.

Working Libvirt network definition looks like this (notice the <driver name='vfio'/> I’ve had to add as well):

<network>
   <name>passthrough</name>
   <forward mode='hostdev' managed='yes'>
     <pf dev='enpXs0f0'/>
     <driver name='vfio'/>
   </forward>
  <portgroup name='libvirt_port_group1' default='yes'>
    <virtualport type='802.1Qbh'>
      <parameters profileid='UCS_Port_Profile'/>
    </virtualport>
  </portgroup>
 </network>

And the network interface for the VM can be defined like:

<interface type='network'>
  <mac address='52:54:00:ec:24:f1'/>
  <source network='passthrough' portgroup='libvirt_port_group1'/>
</interface>

One of the enpXs0f0 virtual functions (total number depends on the Dynamic vNIC Connection Policy assigned to the static vNIC) will be consumed by that VM as soon as you power it on.