path: root/wiki/Virtual_IO_Internals.md

::: {#mw-page-base .noprint}
:::

::: {#mw-head-base .noprint}
:::

::: {#content .mw-body role="main"}
[]{#top}

::: {#siteNotice}
:::

::: mw-indicators
:::

# Virtual I/O Internals {#firstHeading .firstHeading}

> **Imported 2023 material — superseded.** This page is an archived copy of the
> previous project wiki (circa 2023) and is kept for historical reference.
> Current documentation:
> [`../docs/03-architecture/`](../docs/03-architecture/) and
> [`../docs/09-reference/`](../docs/09-reference/).

::: {#bodyContent .vector-body}
::: {#siteSub .noprint}
From Open-IOV
:::

::: {#contentSub}
::: {.mw-revision .warningbox}
::: {#mw-revision-info}
Revision as of 17:54, 19 June 2023 by
[Arthur](./index.php%3Ftitle=User:Arthur&action=edit&redlink=1.html "User:Arthur (page does not exist)"){.new
.mw-userlink}
[([talk](./index.php%3Ftitle=User_talk:Arthur&action=edit&redlink=1.html "User talk:Arthur (page does not exist)"){.new
.mw-usertoollinks-talk} \|
[contribs](index.php/Special:Contributions/Arthur.html "Special:Contributions/Arthur"){.mw-usertoollinks-contribs})]{.mw-usertoollinks}
[([[[→‎References (Talks & Reading
Material)](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#References_(Talks_&_Reading_Material))]{.autocomment}]{dir="auto"})]{.comment}
:::

::: {#mw-revision-nav}
([diff](https://open-iov.org/index.php?title=Virtual_I/O_Internals&diff=prev&oldid=23792 "Virtual I/O Internals"))
[← Older
revision](https://open-iov.org/index.php?title=Virtual_I/O_Internals&direction=prev&oldid=23792 "Virtual I/O Internals")
\| Latest revision (diff) \| Newer revision → (diff)
:::
:::
:::

::: {#contentSub2}
:::

::: {#jump-to-nav}
:::

[Jump to
navigation](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#mw-head){.mw-jump-link}
[Jump to
search](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#searchInput){.mw-jump-link}

::: {#mw-content-text .mw-body-content .mw-content-ltr lang="en" dir="ltr"}
::: mw-parser-output
The following document details the internals of a **VFIO (Virtual
Function I/O)** driven **Shared** **I/O Device.**

> An absence of critical technical documentation has historically slowed
> growth and adoption of developer ecosystems for GPU virtualization.
>
> This
> [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/){.external
> .text rel="nofollow"} licensed content can either be used with
> attribution, or used as inspiration for new documentation, created by
> GPU vendors for public commercial distribution as developer
> documentation.
>
> Where possible, this documentation will clearly label dates and
> versions of observed-but-not-guaranteed behaviour vs.
> vendor-documented stable interfaces/behaviour with guarantees of
> forward or backward compatibility.

This article places emphasis on the **Virtual GPU (vGPU)** use case
however these concepts apply generically to virtualization of I/O
devices (TPUs, NICs, storage peripherals, ect..).

  -------------------------------------------------------------------------------------- ------------------------------------------------------------------------- ---------------------------------------------
  Mdev Mode                                                                              SR-IOV Mode                                                               SIOV Mode
  No hardware assistance needed.                                                         Hardware assistance needed.                                               Hardware assistance needed.
  Host requires insight about guest workload.                                            Host ignorance of guest workload.                                         Host requires insight about guest workload.
  Error reporting.                                                                       No guest driver error reporting.                                          Error reporting.
  In depth dynamic monitoring.                                                           Basic dynamic monitoring.                                                 In depth dynamic monitoring.
  Software defined MMU guest separation.                                                 Firmware defined MMU guest separation.                                    Firmware defined MMU guest separation.
  Requires deferred instructions to be supported by host software (support libraries).   Guest is ignorant of host supported software such as support libraries.   
  Routing interrupts.                                                                    Routing interrupts.                                                       Routing interrupts.
  Device reset.                                                                          Device reset.                                                             Device reset.
  Enable/Disable device.                                                                 Enable/Disable device.                                                    Enable/Disable device.
  Support for multiple scheduling techniques.                                            Support for multiple scheduling techniques.                               Support for multiple scheduling techniques.
  Single PCI requester ID.                                                               Multiple PCI requester IDs.                                               Multiple PCI requester IDs.
  -------------------------------------------------------------------------------------- ------------------------------------------------------------------------- ---------------------------------------------

  : Comparison of Assistance Modes

::: {#toc .toc role="navigation" aria-labelledby="mw-toc-heading"}
::: {.toctitle lang="en" dir="ltr"}
## Contents {#mw-toc-heading}

[]{.toctogglespan}
:::

-   [[1]{.tocnumber} [All
    Modes]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#All_Modes)
    -   [[1.1]{.tocnumber} [Knowledge Resources
        Used]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Knowledge_Resources_Used)
    -   [[1.2]{.tocnumber} [Binding VFIO
        devices]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Binding_VFIO_devices)
    -   [[1.3]{.tocnumber} [Programming the
        IOMMU]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Programming_the_IOMMU)
    -   [[1.4]{.tocnumber} [VFIO Memory Mapped
        IO]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#VFIO_Memory_Mapped_IO)
    -   [[1.5]{.tocnumber} [Getting the VFIO Group File
        Descriptor]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Getting_the_VFIO_Group_File_Descriptor)
    -   [[1.6]{.tocnumber} [VFIO device file
        descriptor]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#VFIO_device_file_descriptor)
    -   [[1.7]{.tocnumber} [VFIO
        Interrupts]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#VFIO_Interrupts)
        -   [[1.7.1]{.tocnumber}
            [IRQs]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#IRQs)
            -   [[1.7.1.1]{.tocnumber} [Get Device
                Info]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Get_Device_Info)
            -   [[1.7.1.2]{.tocnumber} [Get Region
                Info]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Get_Region_Info)
            -   [[1.7.1.3]{.tocnumber} [Get IRQ
                Info]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Get_IRQ_Info)
            -   [[1.7.1.4]{.tocnumber} [Set
                IRQs]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Set_IRQs)
    -   [[1.8]{.tocnumber} [Device
        Decomposure/Recomposure]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Device_Decomposure/Recomposure)
    -   [[1.9]{.tocnumber} [Memory Management Unit
        (MMU)]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Memory_Management_Unit_(MMU))
        -   [[1.9.1]{.tocnumber} [MMIO Isolation (Platform
            MMU)]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#MMIO_Isolation_(Platform_MMU))
        -   [[1.9.2]{.tocnumber} [VRAM Isolation (GPU
            GMMU)]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#VRAM_Isolation_(GPU_GMMU))
        -   [[1.9.3]{.tocnumber} [Platform MMIO \<-\> GPU Shadow
            Pages]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Platform_MMIO_%3C-%3E_GPU_Shadow_Pages)
    -   [[1.10]{.tocnumber} [VFIO Quirks (region
        traps)]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#VFIO_Quirks_(region_traps))
    -   [[1.11]{.tocnumber} [Known IOMMU
        Issues]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Known_IOMMU_Issues)
-   [[2]{.tocnumber} [Mdev
    Mode]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Mdev_Mode)
    -   [[2.1]{.tocnumber} [Knowledge Resources
        Used]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Knowledge_Resources_Used_2)
    -   [[2.2]{.tocnumber} [Mediated
        Core]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Mediated_Core)
    -   [[2.3]{.tocnumber} [Device
        Initialization]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Device_Initialization)
        -   [[2.3.1]{.tocnumber} [Registering VFIO MDEV as a
            driver]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Registering_VFIO_MDEV_as_a_driver)
        -   [[2.3.2]{.tocnumber} [Registering PCIE Device with the
            Mediated
            Core]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Registering_PCIE_Device_with_the_Mediated_Core)
        -   [[2.3.3]{.tocnumber} [Registering Mediated Callbacks
            (CBs)]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Registering_Mediated_Callbacks_(CBs))
        -   [[2.3.4]{.tocnumber} [Creating a Mediated Device via
            mdev-sysfsdev
            API]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Creating_a_Mediated_Device_via_mdev-sysfsdev_API)
        -   [[2.3.5]{.tocnumber} [QEMU adds VFIO device to IOMMU
            container-group]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#QEMU_adds_VFIO_device_to_IOMMU_container-group)
        -   [[2.3.6]{.tocnumber} [QEMU passes mdev device file
            descriptor to
            VM]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#QEMU_passes_mdev_device_file_descriptor_to_VM)
    -   [[2.4]{.tocnumber} [Instruction
        Execution]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Instruction_Execution)
        -   [[2.4.1]{.tocnumber} [IRQ
            remapping]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#IRQ_remapping)
        -   [[2.4.2]{.tocnumber} [Interrupt
            Injection]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Interrupt_Injection)
    -   [[2.5]{.tocnumber} [Memory
        Management]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Memory_Management)
        -   [[2.5.1]{.tocnumber} [Region
            Passthrough]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Region_Passthrough)
            -   [[2.5.1.1]{.tocnumber} [Emulated
                Regions]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Emulated_Regions)
            -   [[2.5.1.2]{.tocnumber} [Passthrough
                Regions]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Passthrough_Regions)
            -   [[2.5.1.3]{.tocnumber} [Region
                Access]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Region_Access)
        -   [[2.5.2]{.tocnumber} [EPT Page
            Violations]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#EPT_Page_Violations)
        -   [[2.5.3]{.tocnumber} [Mediated DMA
            Translations]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Mediated_DMA_Translations)
    -   [[2.6]{.tocnumber}
        [Scheduling]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Scheduling)
    -   [[2.7]{.tocnumber} [Kernel
        API]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Kernel_API)
    -   [[2.8]{.tocnumber} [Sample
        Code]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Sample_Code)
        -   [[2.8.1]{.tocnumber}
            [Compilation]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Compilation)
        -   [[2.8.2]{.tocnumber} [Loading Kernel
            Modules]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Loading_Kernel_Modules)
        -   [[2.8.3]{.tocnumber} [Unloading Kernel
            Modules]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Unloading_Kernel_Modules)
        -   [[2.8.4]{.tocnumber} [Additional
            Documentation]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Additional_Documentation)
    -   [[2.9]{.tocnumber} [Mdev Mode
        Requirements]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Mdev_Mode_Requirements)
-   [[3]{.tocnumber} [SR-IOV
    Mode]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#SR-IOV_Mode)
    -   [[3.1]{.tocnumber} [Knowledge Resources
        Used]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Knowledge_Resources_Used_3)
    -   [[3.2]{.tocnumber} [Instruction
        Execution]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Instruction_Execution_2)
    -   [[3.3]{.tocnumber} [Memory
        Management]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Memory_Management_2)
    -   [[3.4]{.tocnumber}
        [Scheduling]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Scheduling_2)
    -   [[3.5]{.tocnumber} [SR-IOV Mode
        Requirements]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#SR-IOV_Mode_Requirements)
-   [[4]{.tocnumber} [SIOV
    Mode]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#SIOV_Mode)
    -   [[4.1]{.tocnumber} [Knowledge Resources
        Used]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Knowledge_Resources_Used_4)
    -   [[4.2]{.tocnumber} [Memory
        Management]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Memory_Management_3)
        -   [[4.2.1]{.tocnumber} [IOMMU Aware Mediated
            Device]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#IOMMU_Aware_Mediated_Device)
        -   [[4.2.2]{.tocnumber} [Shared Hardware
            Workqueues]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Shared_Hardware_Workqueues)
    -   [[4.3]{.tocnumber} [SIOV Mode
        Requirements]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#SIOV_Mode_Requirements)
-   [[5]{.tocnumber} [References (Talks & Reading
    Material)]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#References_(Talks_&_Reading_Material))
:::

## [All Modes]{#All_Modes .mw-headline}

This section will cover concepts which apply both to [Mdev
Mode](index.php/Mediated_Device_Internals.html#Mdev_Mode){.external
.text rel="nofollow"}, [SR-IOV
Mode](index.php/Mediated_Device_Internals.html#SR-IOV_Mode){.external
.text rel="nofollow"} & [SIOV
Mode](index.php/Mediated_Device_Internals.html#SIOV_Mode){.external
.text rel="nofollow"}.

### [Knowledge Resources Used]{#Knowledge_Resources_Used .mw-headline}

This section is supported by significant contributions in open source by
Alex Williamson.

See references 2, 14, & 22 in the [References (Talks & Reading
Material)](index.php/Virtual_IO_Internals.html#References_(Talks_&_Reading_Material)){.external
.text rel="nofollow"} section.

### [Binding VFIO devices]{#Binding_VFIO_devices .mw-headline}

::: {.thumb .tright}
::: {.thumbinner style="width:302px;"}
[![](images/thumb/9/9e/Vfio-pci_driver_bindings.png/300px-Vfio-pci_driver_bindings.png){.thumbimage
decoding="async" width="300" height="218"
srcset="images/thumb/9/9e/Vfio-pci_driver_bindings.png/450px-Vfio-pci_driver_bindings.png 1.5x, images/thumb/9/9e/Vfio-pci_driver_bindings.png/600px-Vfio-pci_driver_bindings.png 2x"}](index.php/File:Vfio-pci_driver_bindings.png.html){.image}

::: thumbcaption
::: magnify
[](index.php/File:Vfio-pci_driver_bindings.png.html "Enlarge"){.internal}
:::

**Figure 1:** VFIO group nodes are unit of ownership that VFIO uses. See
slides from:
[\[2\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external
.text rel="nofollow"}
:::
:::
:::

::: {.thumb .tright}
::: {.thumbinner style="width:302px;"}
[![](images/thumb/3/33/IOCTL_set_VFIO_container.png/300px-IOCTL_set_VFIO_container.png){.thumbimage
decoding="async" width="300" height="242"
srcset="images/thumb/3/33/IOCTL_set_VFIO_container.png/450px-IOCTL_set_VFIO_container.png 1.5x, images/thumb/3/33/IOCTL_set_VFIO_container.png/600px-IOCTL_set_VFIO_container.png 2x"}](index.php/File:IOCTL_set_VFIO_container.png.html){.image}

::: thumbcaption
::: magnify
[](index.php/File:IOCTL_set_VFIO_container.png.html "Enlarge"){.internal}
:::

**Figure 2:** IOCTL(GROUP, VFIO_GROUP_SET_CONTAINER, &CONTAINER) places
the VFIO Group inside the VFIO Container. See slides from:
[\[2\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external
.text rel="nofollow"}
:::
:::
:::

::: {.thumb .tright}
::: {.thumbinner style="width:302px;"}
[![](images/thumb/4/40/Ioctl-VFIO_IOMMU_MAP-UNMAP_DMA.png/300px-Ioctl-VFIO_IOMMU_MAP-UNMAP_DMA.png){.thumbimage
decoding="async" width="300" height="214"
srcset="images/thumb/4/40/Ioctl-VFIO_IOMMU_MAP-UNMAP_DMA.png/450px-Ioctl-VFIO_IOMMU_MAP-UNMAP_DMA.png 1.5x, images/thumb/4/40/Ioctl-VFIO_IOMMU_MAP-UNMAP_DMA.png/600px-Ioctl-VFIO_IOMMU_MAP-UNMAP_DMA.png 2x"}](index.php/File:Ioctl-VFIO_IOMMU_MAP-UNMAP_DMA.png.html){.image}

::: thumbcaption
::: magnify
[](index.php/File:Ioctl-VFIO_IOMMU_MAP-UNMAP_DMA.png.html "Enlarge"){.internal}
:::

**Figure 3:** Using interrupts IOCTL(CONTAINER, VFIO_IOMMU_MAP_DMA,
&MAP), IOCTL(CONTAINER, VFIO_IOMMU_UNMAP_DMA, &UNMAP) to map memory and
pin pages. See slides from:
[\[2\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external
.text rel="nofollow"}
:::
:::
:::

::: {.thumb .tright}
::: {.thumbinner style="width:302px;"}
[![](images/thumb/3/3f/Ioctl-VFIO_GROUP_GET_FD.png/300px-Ioctl-VFIO_GROUP_GET_FD.png){.thumbimage
decoding="async" width="300" height="204"
srcset="images/thumb/3/3f/Ioctl-VFIO_GROUP_GET_FD.png/450px-Ioctl-VFIO_GROUP_GET_FD.png 1.5x, images/thumb/3/3f/Ioctl-VFIO_GROUP_GET_FD.png/600px-Ioctl-VFIO_GROUP_GET_FD.png 2x"}](index.php/File:Ioctl-VFIO_GROUP_GET_FD.png.html){.image}

::: thumbcaption
::: magnify
[](index.php/File:Ioctl-VFIO_GROUP_GET_FD.png.html "Enlarge"){.internal}
:::

**Figure 4:** Using interrupt IOCTL(GROUP2, VFIO_GROUP_GET_FD,
\"0000:01:00.0\") to obtain the VFIO Group file descriptor. See slides
from:
[\[2\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external
.text rel="nofollow"}
:::
:::
:::

**Figure 1:** Binding devices to the vfio-pci driver results in VFIO
group nodes.

Opening the file \"/dev/vfio/vfio\" creates a VFIO Container.

**Figure 2:** The interrupt routine
**`IOCTL(GROUP, VFIO_GROUP_SET_CONTAINER, &CONTAINER)`** places the VFIO
group inside the VFIO container.

### [Programming the IOMMU]{#Programming_the_IOMMU .mw-headline}

When this has been done
**`IOCTL(CONTAINER, VFIO_SET_IOMMU, VFIO_TYPE1_IOMMU)`** can then be
used to set an IOMMU type for the container which places it in a user
interactable state.

Once this IOMMU type state has been set and the VFIO container has been
made interactable additional VFIO groups may be added to the container
without requiring that the group\'s IOMMU type be set again as newly
added groups automatically inherit the container\'s IOMMU context.

### [VFIO Memory Mapped IO]{#VFIO_Memory_Mapped_IO .mw-headline}

**Figure 3:** Once the VFIO Groups have been placed inside the VFIO
container and the IOMMU type has been set the user may then map and
unmap which will automatically inserts Memory Mapped IO (MMIO) entries
into the IOMMU as well as pin/unpin pages as necessary. This can be
accomplished using **`IOCTL(CONTAINER, VFIO_IOMMU_MAP_DMA, &MAP)`** for
map/pin and **`IOCTL(CONTAINER, VFIO_IOMMU_UNMAP_DMA, &UNMAP)`** for
unmap/unpin.

### [Getting the VFIO Group File Descriptor]{#Getting_the_VFIO_Group_File_Descriptor .mw-headline}

**Figure 4:** Once the device has been bound to a VFIO driver, set in a
VFIO container, the VFIO container has it\'s IOMMU type set, and a
memory map/page pin of the VFIO device has been completed a file
descriptor can then be obtained for the device. This file descriptor can
be used for interrupts (ioctls), to probe for information about the BAR
regions, and configure the IRQs.

### [VFIO device file descriptor]{#VFIO_device_file_descriptor .mw-headline}

VFIO device file descriptors are divided into regions and each region is
mapped into a device resource. Region count and info (file offset,
allowable access, ect..) can be discovered through interrupt (IOCTL).
Each file descriptor region corresponding to a PCI resource is
represented as a file offset.

In the case of RPC Mode this structure is emulated whereas in SR-IOV
Mode the structure is mapped to a real PCI resource.

  ---------------------------------------------------
  00:00.0 VGA compatible controller
  Region 0 Bar0 (Config Space) starts at offset 0
  Region 1 Bar1 (MSI - Message Signaled Interrupts)
  Region 2 Bar2 (MSIX)
  Region 3 Bar3
  Region 4 Bar4
  Region 5 Bar5 (IO port space)
  Expansion ROM
  ---------------------------------------------------

  : BAR regions in a VGA PCI device.

Below is what the file offsets looks like internally for each BAR region
starting from address 0 and growing with the addition of former regions
as you progress through the file.

  --------------------- ------------------- ------------------- ----------------------- ------
  \<- File Offset -\>                                                                   
  0 -\> A               A -\> (A+B)         (A+B) -\> (A+B+C)   (A+B+C) -\> (A+B+C+D)   \...
  Region 0 (size A)     Region 1 (size B)   Region 2 (size C)   Region 3 (size D)       \...
  --------------------- ------------------- ------------------- ----------------------- ------

  : VFIO representation of PCI BAR regions offsets.

### [VFIO Interrupts]{#VFIO_Interrupts .mw-headline}

Guests communicate with the host via VFIO Interrupt Requests
([IRQs](https://infogalactic.com/info/Interrupt_request_(PC_architecture)){.external
.text rel="nofollow"}). These are sent via an irqfd (IRQ [File
Descriptor](https://infogalactic.com/info/File_descriptor){.external
.text rel="nofollow"}). Similarly, the host receives these interrupts
via
[eventfd](https://man7.org/linux/man-pages/man2/eventfd.2.html){.external
.text rel="nofollow"} (Event File Descriptor). The resulting data can be
returned via a
[callback](https://infogalactic.com/info/Callback_(computer_programming)){.external
.text rel="nofollow"}.

#### [IRQs]{#IRQs .mw-headline}

Device properties discovered via interrupt (IOCTL).

##### [Get Device Info]{#Get_Device_Info .mw-headline}

  ------------------------- ------------- ----------------------------
  VFIO_DEVICE_GET_INFO                    
  struct vfio_device_info                 
                            argz          
                            flags         
                                          VFIO_DEVICE_FLAGS_PCI
                                          VFIO_DEVICE_FLAGS_PLATFORM
                                          VFIO_DEVICE_FLAGS_RESET
                            num_irqs      
                            num_regions   
  ------------------------- ------------- ----------------------------

The IRQ **`VFIO_DEVICE_GET_INFO`** can provide information to
distinguish between PCI and platform devices as well as the number of
regions and IRQs for a particular device.

The upstream API can be read **[here
(vfio.h)](https://github.com/torvalds/linux/blob/47700948a4abb4a5ae13ef943ff682a7f327547a/include/uapi/linux/vfio.h#L194){.external
.text rel="nofollow"}**.

**Sample mdev code** to service this request can be found
**[here](https://github.com/Open-IOV/VFIO-Mdev_Samples/blob/c0adb9957c0b22bfdbc18395e81dd2b476addac5/mdpy.c#L470){.external
.text rel="nofollow"}**, and
**[here](https://github.com/Open-IOV/VFIO-Mdev_Samples/blob/c0adb9957c0b22bfdbc18395e81dd2b476addac5/mdpy.c#L522){.external
.text rel="nofollow"}**.

##### [Get Region Info]{#Get_Region_Info .mw-headline}

  ----------------------------- ------------ -----------------------------
  VFIO_DEVICE_GET_REGION_INFO                
  struct vfio_region_info                    
                                argz         
                                cap_offset   
                                flags        
                                             VFIO_REGION_INFO_FLAG_CAPS
                                             VFIO_REGION_INFO_FLAG_MMAP
                                             VFIO_REGION_INFO_FLAG_READ
                                             VFIO_REGION_INFO_FLAG_WRITE
                                index        
                                offset       
                                size         
  ----------------------------- ------------ -----------------------------

Once the interrupt user knows the number of regions within a VFIO device
they can use IRQ **`VFIO_DEVICE_GET_REGION_INFO`** to probe each region
for additional information. This interrupt will return information such
as if it can be read from or written to, if the device supports MMAP, as
well as what the offset and size of the region is within the VFIO file
descriptor.

The upstream API can be read [**here
(vfio.h)**](https://github.com/torvalds/linux/blob/47700948a4abb4a5ae13ef943ff682a7f327547a/include/uapi/linux/vfio.h#L240){.external
.text rel="nofollow"}.

**Sample mdev code** to service this request can be found
[**here**](https://github.com/Open-IOV/VFIO-Mdev_Samples/blob/c0adb9957c0b22bfdbc18395e81dd2b476addac5/mdpy.c#L425){.external
.text rel="nofollow"}, and
[**here**](https://github.com/Open-IOV/VFIO-Mdev_Samples/blob/c0adb9957c0b22bfdbc18395e81dd2b476addac5/mdpy.c#L545){.external
.text rel="nofollow"}.

##### [Get IRQ Info]{#Get_IRQ_Info .mw-headline}

  -------------------------- ------- --------------------------
  VFIO_DEVICE_GET_IRQ_INFO           
  struct vfio_irq_info               
                             argz    
                             count   
                             flags   
                                     VFIO_IRQ_INFO_AUTOMASKED
                                     VFIO_IRQ_INFO_EVENTFD
                                     VFIO_IRQ_INFO_MASKABLE
                                     VFIO_IRQ_INFO_NORESIZE
                             index   
  -------------------------- ------- --------------------------

**`VFIO_DEVICE_GET_IRQ_INFO`** is used to retrieve information about a
device IRQ.

**`VFIO_IRQ_INFO_AUTOMASKED`** is used to mask interrupts when they
occur to protect the host.

The upstream API can be read **[here
(vfio.h)](https://github.com/torvalds/linux/blob/47700948a4abb4a5ae13ef943ff682a7f327547a/include/uapi/linux/vfio.h#L480){.external
.text rel="nofollow"}**

**Sample mdev code** to service this request can be found
[**here**](https://github.com/Open-IOV/VFIO-Mdev_Samples/blob/c0adb9957c0b22bfdbc18395e81dd2b476addac5/mdpy.c#L463){.external
.text rel="nofollow"}, and
[**here**](https://github.com/Open-IOV/VFIO-Mdev_Samples/blob/c0adb9957c0b22bfdbc18395e81dd2b476addac5/mdpy.c#L570){.external
.text rel="nofollow"}.

##### [Set IRQs]{#Set_IRQs .mw-headline}

  ---------------------- ---------- -----------------------------
  VFIO_DEVICE_SET_IRQS              
  struct vfio_irq_set               
                         argz       
                         count      
                         data\[\]   
                         flags      
                                    VFIO_IRQ_SET_ACTION_MASK
                                    VFIO_IRQ_SET_ACTION_TRIGGER
                                    VFIO_IRQ_SET_ACTION_UNMASK
                                    VFIO_IRQ_SET_DATA_BOOL
                                    VFIO_IRQ_SET_DATA_EVENTFD
                                    VFIO_IRQ_SET_DATA_NONE
                         index      
                         start      
  ---------------------- ---------- -----------------------------

**`VFIO_DEVICE_SET_IRQS`** is used to setup IRQs. Actions can be
configured such as trigger which is when the device triggers an
interrupt (IOCTL), masking and unmasking actions can be set. Bool and
None data types are used for loopback testing of the device. Start and
index may be used to modify subregions.

The upstream API can be read **[here
(vfio.h)](https://github.com/torvalds/linux/blob/47700948a4abb4a5ae13ef943ff682a7f327547a/include/uapi/linux/vfio.h#L524){.external
.text rel="nofollow"}**.

### []{#Device_Decomposure.2FRecomposure}[Device Decomposure/Recomposure]{#Device_Decomposure/Recomposure .mw-headline}

::: {.thumb .tright}
::: {.thumbinner style="width:302px;"}
[![Figure 5: Device Decomposure and Recomposure via
VFIO.](images/thumb/9/90/Device_Decomposure_and_Recomposure_via_VFIO.png/300px-Device_Decomposure_and_Recomposure_via_VFIO.png){.thumbimage
decoding="async" width="300" height="147"
srcset="images/thumb/9/90/Device_Decomposure_and_Recomposure_via_VFIO.png/450px-Device_Decomposure_and_Recomposure_via_VFIO.png 1.5x, images/thumb/9/90/Device_Decomposure_and_Recomposure_via_VFIO.png/600px-Device_Decomposure_and_Recomposure_via_VFIO.png 2x"}](index.php/File:Device_Decomposure_and_Recomposure_via_VFIO.png.html){.image}

::: thumbcaption
::: magnify
[](index.php/File:Device_Decomposure_and_Recomposure_via_VFIO.png.html "Enlarge"){.internal}
:::

**Figure 5:** Device Decomposure and Recomposure via VFIO. See slides
from:
[\[2\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external
.text rel="nofollow"}
:::
:::
:::

**Figure 5:** Virtual Function IO (VFIO) devices are deconstructed in
userspace into a set of VFIO primitives (MMIO pages, VFIO/IOMMU Groups,
VFIO IRQs, File Descriptors). Recomposure of these devices occurs upon
assignment of a Virtual Function (VF) to a QEMU virtual machine.

### []{#Memory_Management_Unit_.28MMU.29}[Memory Management Unit (MMU)]{#Memory_Management_Unit_(MMU) .mw-headline}

::: {.thumb .tright}
::: {.thumbinner style="width:302px;"}
::: {.MediaTransformError style="width: 300px; height: 275px; display:inline-block;"}
Error creating thumbnail: File missing
:::

::: thumbcaption
::: magnify
[](index.php/File:Figure_6-_MMU_-_GMMU_IOVA_translation-isolation..png.html "Enlarge"){.internal}
:::

**Figure 6:** MMU - GMMU IOVA translation/isolation.
:::
:::
:::

**Figure 6:** This section will touch upon the mechanisms used for
enforcement of Host Physical Address (HPA) to Guest Physical Address
(GPA) isolation.

#### []{#MMIO_Isolation_.28Platform_MMU.29}[MMIO Isolation (Platform MMU)]{#MMIO_Isolation_(Platform_MMU) .mw-headline}

The platform\'s CPU communicates with the GPU by reading/writing to and
from pinned MMIO pages in [Random Access Memory
(RAM)](https://infogalactic.com/info/Random-access_memory){.external
.text rel="nofollow"}. MMIO pages within the RAM are subject to IO
Virtual Address (IOVA) translations by the platform\'s discrete MMU
controller which is programmed by the CPU. These IOVA translations serve
as a mechanism to enforce HPA to GPA isolation in the context of the
platform.

#### []{#VRAM_Isolation_.28GPU_GMMU.29}[VRAM Isolation (GPU GMMU)]{#VRAM_Isolation_(GPU_GMMU) .mw-headline}

The GPU core performs virtualized operations by reading/writing to and
from shadow page tables in onboard [Video Random Access Memory
(VRAM)](https://en.wikipedia.org/wiki/Video_random_access_memory "wikipedia:Video random access memory"){.extiw}.
Shadow pages within the VRAM are subject to IO Virtual Address (IOVA)
translations by the GPU\'s discrete GPU MMU controller (GMMU) which is
programmed by the Embedded CPU (GPU co-processor). These IOVA
translations serve as a mechanism to enforce HPA to GPA isolation in the
context of the virtual GPUs and multi-process isolation in single user
environments.

#### []{#Platform_MMIO_.3C-.3E_GPU_Shadow_Pages}[Platform MMIO \<-\> GPU Shadow Pages]{#Platform_MMIO_<->_GPU_Shadow_Pages .mw-headline}

In the context of VFIO pinned MMIO pages in RAM act as an interface to
communicate with VRAM shadow pages allowing GPU drivers on the platform
to send instructions to the GPU. When the GPU or Platform alters memory
contained in a shadow page or pinned MMIO page the change is mirrored in
the corresponding IO Virtual Address (IOVA). For example if shadow page
0 is changed by the GPU this change is mirrored in MMIO page 0 on the
platform (the reverse example also applies). When communications occur
between the platform and GPU the information first moves through the
MMU/GMMU and is then written to RAM/VRAM.

::: {.thumb .tright}
::: {.thumbinner style="width:302px;"}
[![Figure 7: A depiction of region overlays within a VFIO file
descriptor.](images/thumb/4/48/Figure_7-_A_depiction_of_region_overlays_within_a_VFIO_file_descriptor..png/300px-Figure_7-_A_depiction_of_region_overlays_within_a_VFIO_file_descriptor..png){.thumbimage
decoding="async" width="300" height="268"
srcset="images/thumb/4/48/Figure_7-_A_depiction_of_region_overlays_within_a_VFIO_file_descriptor..png/450px-Figure_7-_A_depiction_of_region_overlays_within_a_VFIO_file_descriptor..png 1.5x, images/thumb/4/48/Figure_7-_A_depiction_of_region_overlays_within_a_VFIO_file_descriptor..png/600px-Figure_7-_A_depiction_of_region_overlays_within_a_VFIO_file_descriptor..png 2x"}](index.php/File:Figure_7-_A_depiction_of_region_overlays_within_a_VFIO_file_descriptor..png.html){.image}

::: thumbcaption
::: magnify
[](index.php/File:Figure_7-_A_depiction_of_region_overlays_within_a_VFIO_file_descriptor..png.html "Enlarge"){.internal}
:::

**Figure 7:** A depiction of region overlays within a VFIO file
descriptor. See slides from:
[\[2\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external
.text rel="nofollow"}
:::
:::
:::

### []{#VFIO_Quirks_.28region_traps.29}[VFIO Quirks (region traps)]{#VFIO_Quirks_(region_traps) .mw-headline}

**Figure 7:** Regions of the PCI BAR require emulation (slow path) via
emulated traps (known as quirks). These regions are primarily in PCI
configuration space. QEMU may overlap a region overlay which when
read/written to/from triggers a VM-exit to trap and emulate the region
in order that appropriate translations may occur (such as those
concerned with IO Virtual Address - IOVA).

::: {.thumb .tright}
::: {.thumbinner style="width:302px;"}
[![Peer-to-Peer DMA Isolation under
IOMMU](images/thumb/1/13/Screen_Shot_2022-05-06_at_10.46.59_AM.png/300px-Screen_Shot_2022-05-06_at_10.46.59_AM.png){.thumbimage
decoding="async" width="300" height="224"
srcset="images/thumb/1/13/Screen_Shot_2022-05-06_at_10.46.59_AM.png/450px-Screen_Shot_2022-05-06_at_10.46.59_AM.png 1.5x, images/1/13/Screen_Shot_2022-05-06_at_10.46.59_AM.png 2x"}](index.php/File:Screen_Shot_2022-05-06_at_10.46.59_AM.png.html){.image}

::: thumbcaption
::: magnify
[](index.php/File:Screen_Shot_2022-05-06_at_10.46.59_AM.png.html "Enlarge"){.internal}
:::

**Figure 8:** Peer-to-Peer DMA Isolation under IOMMU. See slides from:
[\[2\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external
.text rel="nofollow"}
:::
:::
:::

### [Known IOMMU Issues]{#Known_IOMMU_Issues .mw-headline}

**DMA Aliasing**

-   Not all devices generate unique IDs.
-   Not all devices generate IDs they should.\

**DMA Isolation**

-   **Figure 8:** Peer-to-Peer DMA Isolation. In many circumstances IO
    Virtual Address (IOVA) translations do not occur properly in the
    context of DMA peering. Transactions that occur through the IOMMU
    are unaffected.

::: {.thumb .tright}
::: {.thumbinner style="width:302px;"}
[![Figure 0: Approches to GPU
Virtualization.](images/thumb/6/6a/Figure_0-_Approches_to_GPU_Virtualization..png/300px-Figure_0-_Approches_to_GPU_Virtualization..png){.thumbimage
decoding="async" width="300" height="153"
srcset="images/thumb/6/6a/Figure_0-_Approches_to_GPU_Virtualization..png/450px-Figure_0-_Approches_to_GPU_Virtualization..png 1.5x, images/thumb/6/6a/Figure_0-_Approches_to_GPU_Virtualization..png/600px-Figure_0-_Approches_to_GPU_Virtualization..png 2x"}](index.php/File:Figure_0-_Approches_to_GPU_Virtualization..png.html){.image}

::: thumbcaption
::: magnify
[](index.php/File:Figure_0-_Approches_to_GPU_Virtualization..png.html "Enlarge"){.internal}
:::

**Figure 0:** Approaches to GPU Virtualization. See slides from:
[\[61\]](index.php/Virtual_I/O_Internals.html#References_(Talks_&_Reading_Material)){.external
.text rel="nofollow"}
:::
:::
:::

## [Mdev Mode]{#Mdev_Mode .mw-headline}

**Mdev Mode (VFIO Mediated Device)** is a method of virtualizing I/O
devices enabling full API capabilities without the requirement for
hardware assistance.

### [Knowledge Resources Used]{#Knowledge_Resources_Used_2 .mw-headline}

This section is supported by significant contributions in open source by
Neo Jia, Kirti Wankhede, Kevin Tian, Yiying Zhang, David Cowperthwaite,
Kun Tian, Yaozu Dong, Tina Zhang, Gerd Hoffmann, & Zhi Wang.

See references 1, 7, 8, 17, 18, 19, 70 in the [References (Talks &
Reading
Material)](index.php/Virtual_IO_Internals.html#References_(Talks_&_Reading_Material)){.external
.text rel="nofollow"} section.

### [Mediated Core]{#Mediated_Core .mw-headline}

The mediated device framework made 3 major changes to the VFIO driver.

-   **1: [Mediated core module
    (new)](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/vfio/mdev/mdev_core.c){.external
    .text rel="nofollow"}**\
    -Mediated bus driver, create mediated device.\
    -Physical device interface for vendor callbacks.\
    -Generic Mediated device management user interface (sysfs).
-   **2: [Mediated device module
    (new)](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/vfio/mdev/mdev_driver.c){.external
    .text rel="nofollow"}**\
    -Manage created mediated device, fully compatible with VFIO user API
    (UAPI).
-   **3: [VFIO IOMMU driver
    (enhancement)](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/vfio/vfio_iommu_type1.c){.external
    .text rel="nofollow"}**\
    -VFIO IOMMU API Type1 compatible, easy to extend to non-Type1.

The full list of these changes can be seen in the [**lists.gnu.org
Qemu-devel** mailing list
archive](https://lists.gnu.org/archive/html/qemu-devel/2016-10/msg03922.html){.external
.text rel="nofollow"}.

### [Device Initialization]{#Device_Initialization .mw-headline}

This section will deal with how an Mdev driver is initialized.

**Sample mdev code** for device initialization can be found
**[here](https://github.com/Open-IOV/VFIO-Mdev_Samples/blob/c0adb9957c0b22bfdbc18395e81dd2b476addac5/mdpy.c#L229){.external
.text rel="nofollow"}**.

#### [Registering VFIO MDEV as a driver]{#Registering_VFIO_MDEV_as_a_driver .mw-headline}

VFIO Mdev must be registered as a driver. This register event occurs
between the Mdev Driver Register Interface and the VFIO MDEV interface.

#### [Registering PCIE Device with the Mediated Core]{#Registering_PCIE_Device_with_the_Mediated_Core .mw-headline}

::: {.thumb .tright}
::: {.thumbinner style="width:302px;"}
[![Registering the mediated
device.](images/thumb/7/75/Mdev-gpu.png/300px-Mdev-gpu.png){.thumbimage
decoding="async" width="300" height="208"
srcset="images/thumb/7/75/Mdev-gpu.png/450px-Mdev-gpu.png 1.5x, images/thumb/7/75/Mdev-gpu.png/600px-Mdev-gpu.png 2x"}](index.php/File:Mdev-gpu.png.html){.image}

::: thumbcaption
::: magnify
[](index.php/File:Mdev-gpu.png.html "Enlarge"){.internal}
:::

**Figure 9:** Registering the mediated device. This step may be
accomplished via the vendor driver or via
[GVM/Mdev-GPU](){.external .text
rel="nofollow"} for drivers which do not include support for Mdev
functionality and/or do not support arbitrary Mdev types.
:::
:::
:::

**Figure 9:** The vendor driver must register with the Mediated Core\'s
Device Register Interface. The example shown uses the [GVM
Project](){.external .text rel="nofollow"}\'s
[Mdev-GPU](){.external .text
rel="nofollow"} component to accomplish this step on behalf of the
vendor driver.

#### []{#Registering_Mediated_Callbacks_.28CBs.29}[Registering Mediated Callbacks (CBs)]{#Registering_Mediated_Callbacks_(CBs) .mw-headline}

**Figure 9:** The vendor driver must now register Mediated Callbacks
which it expects to receive from Mdev devices. The example shown uses
the [GVM Project](){.external .text
rel="nofollow"}\'s
[Mdev-GPU](){.external .text
rel="nofollow"} component to accomplish this step on behalf of the
vendor driver.

#### [Creating a Mediated Device via mdev-sysfsdev API]{#Creating_a_Mediated_Device_via_mdev-sysfsdev_API .mw-headline}

::: {.thumb .tright}
::: {.thumbinner style="width:302px;"}
[![](images/thumb/e/ef/Echoing_a_UUID_into_the_sysfsdev_API.png/300px-Echoing_a_UUID_into_the_sysfsdev_API.png){.thumbimage
decoding="async" width="300" height="235"
srcset="images/thumb/e/ef/Echoing_a_UUID_into_the_sysfsdev_API.png/450px-Echoing_a_UUID_into_the_sysfsdev_API.png 1.5x, images/thumb/e/ef/Echoing_a_UUID_into_the_sysfsdev_API.png/600px-Echoing_a_UUID_into_the_sysfsdev_API.png 2x"}](index.php/File:Echoing_a_UUID_into_the_sysfsdev_API.png.html){.image}

::: thumbcaption
::: magnify
[](index.php/File:Echoing_a_UUID_into_the_sysfsdev_API.png.html "Enlarge"){.internal}
:::

**Figure 10:** Echoing a UUID into the mdev-sysfsdev API. This
functionality is included in
[mdevctl](https://github.com/mdevctl/mdevctl){.external .text
rel="nofollow"} and in [](){.external .text
rel="nofollow"}. See slides from:
[\[1\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external
.text rel="nofollow"}
:::
:::
:::

**Figure 10:** The user of an mdev capable device driver may echo values
such as a UUID into the mdev-sysfsdev interface to create a unique
mediated device. UUIDs must be unique per mdev device within a host.

::: {.thumb .tright}
::: {.thumbinner style="width:302px;"}
[![Figure 10: QEMU creates VFIO-mdev IOMMU binding and acquires mdev
file
descriptor.](images/thumb/6/63/QEMU_creates_VFIO-mdev_IOMMU_binding_and_acquires_mdev_file_descriptor..png/300px-QEMU_creates_VFIO-mdev_IOMMU_binding_and_acquires_mdev_file_descriptor..png){.thumbimage
decoding="async" width="300" height="231"
srcset="images/thumb/6/63/QEMU_creates_VFIO-mdev_IOMMU_binding_and_acquires_mdev_file_descriptor..png/450px-QEMU_creates_VFIO-mdev_IOMMU_binding_and_acquires_mdev_file_descriptor..png 1.5x, images/thumb/6/63/QEMU_creates_VFIO-mdev_IOMMU_binding_and_acquires_mdev_file_descriptor..png/600px-QEMU_creates_VFIO-mdev_IOMMU_binding_and_acquires_mdev_file_descriptor..png 2x"}](index.php/File:QEMU_creates_VFIO-mdev_IOMMU_binding_and_acquires_mdev_file_descriptor..png.html){.image}

::: thumbcaption
::: magnify
[](index.php/File:QEMU_creates_VFIO-mdev_IOMMU_binding_and_acquires_mdev_file_descriptor..png.html "Enlarge"){.internal}
:::

**Figure 11:** QEMU creates VFIO-mdev IOMMU binding and acquires mdev
file descriptor. See slides from:
[\[1\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external
.text rel="nofollow"}
:::
:::
:::

#### [QEMU adds VFIO device to IOMMU container-group]{#QEMU_adds_VFIO_device_to_IOMMU_container-group .mw-headline}

**Figure 11:** When starting a QEMU process with a VFIO-mdev attached
QEMU calls the VFIO API to add the VFIO device to an IOMMU
container/group. QEMU then runs the IOCTL to obtain a file descriptor
for the device.

#### [QEMU passes mdev device file descriptor to VM]{#QEMU_passes_mdev_device_file_descriptor_to_VM .mw-headline}

::: {.thumb .tright}
::: {.thumbinner style="width:302px;"}
[![Figure 11: QEMU presenting the VFIO file descriptor into the virtual
machine.](images/thumb/7/7c/QEMU_presenting_the_VFIO_file_descriptor_into_the_virtual_machine..png/300px-QEMU_presenting_the_VFIO_file_descriptor_into_the_virtual_machine..png){.thumbimage
decoding="async" width="300" height="234"
srcset="images/thumb/7/7c/QEMU_presenting_the_VFIO_file_descriptor_into_the_virtual_machine..png/450px-QEMU_presenting_the_VFIO_file_descriptor_into_the_virtual_machine..png 1.5x, images/thumb/7/7c/QEMU_presenting_the_VFIO_file_descriptor_into_the_virtual_machine..png/600px-QEMU_presenting_the_VFIO_file_descriptor_into_the_virtual_machine..png 2x"}](index.php/File:QEMU_presenting_the_VFIO_file_descriptor_into_the_virtual_machine..png.html){.image}

::: thumbcaption
::: magnify
[](index.php/File:QEMU_presenting_the_VFIO_file_descriptor_into_the_virtual_machine..png.html "Enlarge"){.internal}
:::

**Figure 12:** QEMU presenting the VFIO file descriptor into the virtual
machine. See slides from:
[\[1\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external
.text rel="nofollow"}
:::
:::
:::

**Figure 12:** Once QEMU has obtained the VFIO file descriptor for the
Mdev device via IOCTL it is then QEMU\'s job to present the file
descriptor into the virtual machine so that the mdev may be used by the
guest.

### [Instruction Execution]{#Instruction_Execution .mw-headline}

::: {.thumb .tright}
::: {.thumbinner style="width:302px;"}
[![](images/thumb/a/aa/Ioeventfd-and-irqfd.png/300px-Ioeventfd-and-irqfd.png){.thumbimage
decoding="async" width="300" height="215"
srcset="images/thumb/a/aa/Ioeventfd-and-irqfd.png/450px-Ioeventfd-and-irqfd.png 1.5x, images/thumb/a/aa/Ioeventfd-and-irqfd.png/600px-Ioeventfd-and-irqfd.png 2x"}](index.php/File:Ioeventfd-and-irqfd.png.html){.image}

::: thumbcaption
::: magnify
[](index.php/File:Ioeventfd-and-irqfd.png.html "Enlarge"){.internal}
:::

**Figure 13:** A simple diagram of signalling from host to guest (via
irqfd) & guest to host (via ioeventfd) From
[blog.allenx](http://blog.allenx.org/2015/07/05/kvm-irqfd-and-ioeventfd){.external
.text rel="nofollow"}
:::
:::
:::

**Figure 13:** Mdev Mode moves instruction information across a virtual
function (VF) device using [Remote Procedure
Calls](https://infogalactic.com/info/Remote_procedure_call){.external
.text rel="nofollow"} generally by way of [software
interrupt](https://infogalactic.com/info/Interrupt){.external .text
rel="nofollow"}
([IOCTL](https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core.git/tree/Documentation/driver-api/ioctl.rst){.external
.text rel="nofollow"}). Signals to and from the guest and the host GPU
driver may be passed over file descriptors such as the [Interrupt
Request File Descriptor
(irqfd)](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=721eecbf4fe995ca94a9edec0c9843b1cc0eaaf3){.external
.text rel="nofollow"} and [IO Event File Descriptor
(ioeventfd)](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d34e6b175e61821026893ec5298cc8e7558df43a){.external
.text rel="nofollow"}. The irqfd may be used to signal from the host
into the guest whereas the ioeventfd may be used to signal from the
guest into the host. Guest GPU instructions which would normally
serialize as pRPCs (physical Remote Procedure Calls) are instead
serialized from the guest as vRPCs (virtual Remote Procedure Calls)
which are executed by the host mediated driver.

#### [IRQ remapping]{#IRQ_remapping .mw-headline}

Interrupt Requests (IRQs) must be remapped (trapped for virtualized
execution) to protect the host from sensitive instructions which may
affect global memory state.

#### [Interrupt Injection]{#Interrupt_Injection .mw-headline}

**\--**

**\--Researching\--**

**\--**

### [Memory Management]{#Memory_Management .mw-headline}

Mdev memory management is handled by vendor driver software.

#### [Region Passthrough]{#Region_Passthrough .mw-headline}

Guests may be presented with emulated memory regions or via passthrough
regions or a mixture of the two such as in the case of passthrough
regions with BAR 0 configuration space emulation while other regions are
passthrough\'d.

##### [Emulated Regions]{#Emulated_Regions .mw-headline}

Emulated memory regions use indirect emulated communication requiring a
VM-exit (slow). These regions are often used for virtual PCI config
space such as [in this sample
code](https://github.com/Open-IOV/VFIO-Mdev_Samples/blob/c0adb9957c0b22bfdbc18395e81dd2b476addac5/mdpy.c#L112){.external
.text rel="nofollow"}.

##### [Passthrough Regions]{#Passthrough_Regions .mw-headline}

Passthrough memory regions use direct communication requiring no VM-exit
(fast).

##### [Region Access]{#Region_Access .mw-headline}

::: {.thumb .tright}
::: {.thumbinner style="width:302px;"}
[![Figure 13: QEMU gets region info via VFIO UAPI from vendor driver
through vfio-mdev and Mediated
CBs](images/thumb/f/fa/QEMU_gets_region_info_via_VFIO_UAPI_from_vendor_driver_through_vfio-mdev_and_Mediated_CBs.png/300px-QEMU_gets_region_info_via_VFIO_UAPI_from_vendor_driver_through_vfio-mdev_and_Mediated_CBs.png){.thumbimage
decoding="async" width="300" height="236"
srcset="images/thumb/f/fa/QEMU_gets_region_info_via_VFIO_UAPI_from_vendor_driver_through_vfio-mdev_and_Mediated_CBs.png/450px-QEMU_gets_region_info_via_VFIO_UAPI_from_vendor_driver_through_vfio-mdev_and_Mediated_CBs.png 1.5x, images/thumb/f/fa/QEMU_gets_region_info_via_VFIO_UAPI_from_vendor_driver_through_vfio-mdev_and_Mediated_CBs.png/600px-QEMU_gets_region_info_via_VFIO_UAPI_from_vendor_driver_through_vfio-mdev_and_Mediated_CBs.png 2x"}](index.php/File:QEMU_gets_region_info_via_VFIO_UAPI_from_vendor_driver_through_vfio-mdev_and_Mediated_CBs.png.html){.image}

::: thumbcaption
::: magnify
[](index.php/File:QEMU_gets_region_info_via_VFIO_UAPI_from_vendor_driver_through_vfio-mdev_and_Mediated_CBs.png.html "Enlarge"){.internal}
:::

**Figure 14:** QEMU gets region info via VFIO UAPI from vendor driver
through vfio-mdev and Mediated CBs. See slides from:
[\[1\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external
.text rel="nofollow"}
:::
:::
:::

**Figure 14:** QEMU gets region information via VFIO User API (UAPI)
from the vendor driver through VFIO-mdev and Mediated Callbacks.

::: {.thumb .tright}
::: {.thumbinner style="width:302px;"}
::: {.MediaTransformError style="width: 300px; height: 238px; display:inline-block;"}
Error creating thumbnail: File missing
:::

::: thumbcaption
::: magnify
[](index.php/File:Guest_driver_accessing_Mdev_MMIO_space_backed_by_mdev_file_descriptor_triggers_EPT_violation..png.html "Enlarge"){.internal}
:::

**Figure 15:** Guest driver accessing Mdev MMIO space backed by mdev
file descriptor triggers EPT violation. See slides from:
[\[1\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external
.text rel="nofollow"}
:::
:::
:::

**Figure 15:** The guest\'s vendor driver accesses the Mdev MMIO trapped
region backed by a mdev file descriptor (fd) which triggers an Extended
Page Table (EPT) violation.

::: {.thumb .tright}
::: {.thumbinner style="width:302px;"}
[![Figure 15: KVM services EPT violation and forwards to QEMU VFIO PCI
driver.](images/thumb/4/43/KVM_services_EPT_violation_and_forwards_to_QEMU_VFIO_PCI_driver..png/300px-KVM_services_EPT_violation_and_forwards_to_QEMU_VFIO_PCI_driver..png){.thumbimage
decoding="async" width="300" height="235"
srcset="images/thumb/4/43/KVM_services_EPT_violation_and_forwards_to_QEMU_VFIO_PCI_driver..png/450px-KVM_services_EPT_violation_and_forwards_to_QEMU_VFIO_PCI_driver..png 1.5x, images/thumb/4/43/KVM_services_EPT_violation_and_forwards_to_QEMU_VFIO_PCI_driver..png/600px-KVM_services_EPT_violation_and_forwards_to_QEMU_VFIO_PCI_driver..png 2x"}](index.php/File:KVM_services_EPT_violation_and_forwards_to_QEMU_VFIO_PCI_driver..png.html){.image}

::: thumbcaption
::: magnify
[](index.php/File:KVM_services_EPT_violation_and_forwards_to_QEMU_VFIO_PCI_driver..png.html "Enlarge"){.internal}
:::

**Figure 16:** KVM services EPT violation and forwards to QEMU VFIO PCI
driver. See slides from:
[\[1\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external
.text rel="nofollow"}
:::
:::
:::

**Figure 16:** KVM services EPT violation and forwards to QEMU VFIO PCI
driver.

::: {.thumb .tright}
::: {.thumbinner style="width:302px;"}
[![Figure 16: QEMU convert request from KVM to Read/Write acess to Mdev
file
descriptor.](images/thumb/2/27/QEMU_convert_request_from_KVM_to_Read-Write_acess_to_Mdev_file_descriptor..png/300px-QEMU_convert_request_from_KVM_to_Read-Write_acess_to_Mdev_file_descriptor..png){.thumbimage
decoding="async" width="300" height="235"
srcset="images/thumb/2/27/QEMU_convert_request_from_KVM_to_Read-Write_acess_to_Mdev_file_descriptor..png/450px-QEMU_convert_request_from_KVM_to_Read-Write_acess_to_Mdev_file_descriptor..png 1.5x, images/thumb/2/27/QEMU_convert_request_from_KVM_to_Read-Write_acess_to_Mdev_file_descriptor..png/600px-QEMU_convert_request_from_KVM_to_Read-Write_acess_to_Mdev_file_descriptor..png 2x"}](index.php/File:QEMU_convert_request_from_KVM_to_Read-Write_acess_to_Mdev_file_descriptor..png.html){.image}

::: thumbcaption
::: magnify
[](index.php/File:QEMU_convert_request_from_KVM_to_Read-Write_acess_to_Mdev_file_descriptor..png.html "Enlarge"){.internal}
:::

**Figure 17:** QEMU convert request from KVM to Read/Write acess to Mdev
file descriptor. See slides from:
[\[1\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external
.text rel="nofollow"}
:::
:::
:::

**Figure 17:** QEMU convert request from KVM to R/W access to Mdev file
descriptor.

#### [EPT Page Violations]{#EPT_Page_Violations .mw-headline}

Guest [Memory Mapped IO
(MMIO)](https://infogalactic.com/info/Memory-mapped_I/O){.external .text
rel="nofollow"} trips Extended Page Table (EPT) violations which are
trapped by the host MMU. KVM services EPT violations and forwards to
QEMU VFIO PCI driver. QEMU then converts the request from KVM to R/W
access to the [Mdev File Descriptor
(FD)](https://infogalactic.com/info/File_descriptor){.external .text
rel="nofollow"}. Reads and writes are then handled by the host GPU
device driver via mediated [callbacks
(CBs)](https://infogalactic.com/info/Callback_(computer_programming)){.external
.text rel="nofollow"} and
[VFIO-mdev](https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core.git/tree/Documentation/driver-api/vfio-mediated-device.rst){.external
.text rel="nofollow"}.

#### [Mediated DMA Translations]{#Mediated_DMA_Translations .mw-headline}

**\--**

**\--Researching\--**

**\--**

Because memory is not statically allocated by the vendor driver under
the mediated device framework there is no requirement to make use of
traditional VFIO pinned pages (via the vfio-pci module) rather MMIO
memory can be mapped at runtime incrementally. As a result non-standard
mediated device vfio stub modules may be used.

**Figure 18:** Memory regions get added by QEMU.

::: {.thumb .tright}
::: {.thumbinner style="width:302px;"}
[![Memory Regions Added by
QEMU](images/thumb/5/58/Memory_Regions_Added_by_QEMU.png/300px-Memory_Regions_Added_by_QEMU.png){.thumbimage
decoding="async" width="300" height="225"
srcset="images/thumb/5/58/Memory_Regions_Added_by_QEMU.png/450px-Memory_Regions_Added_by_QEMU.png 1.5x, images/thumb/5/58/Memory_Regions_Added_by_QEMU.png/600px-Memory_Regions_Added_by_QEMU.png 2x"}](index.php/File:Memory_Regions_Added_by_QEMU.png.html){.image}

::: thumbcaption
::: magnify
[](index.php/File:Memory_Regions_Added_by_QEMU.png.html "Enlarge"){.internal}
:::

**Figure 18:** Memory Regions Added by QEMU. See slides from:
[\[1\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external
.text rel="nofollow"}
:::
:::
:::

**Figure 19:** QEMU calls VFIO_DMA_MAP via Memory listener. (not just
guest physical memory but also device memory will be added through this
memory listener)

::: {.thumb .tright}
::: {.thumbinner style="width:302px;"}
[![QEMU calls VFIO DMA MAP via Memory
listener](images/thumb/2/21/QEMU_calls_VFIO_DMA_MAP_via_Memory_listener.png/300px-QEMU_calls_VFIO_DMA_MAP_via_Memory_listener.png){.thumbimage
decoding="async" width="300" height="237"
srcset="images/thumb/2/21/QEMU_calls_VFIO_DMA_MAP_via_Memory_listener.png/450px-QEMU_calls_VFIO_DMA_MAP_via_Memory_listener.png 1.5x, images/thumb/2/21/QEMU_calls_VFIO_DMA_MAP_via_Memory_listener.png/600px-QEMU_calls_VFIO_DMA_MAP_via_Memory_listener.png 2x"}](index.php/File:QEMU_calls_VFIO_DMA_MAP_via_Memory_listener.png.html){.image}

::: thumbcaption
::: magnify
[](index.php/File:QEMU_calls_VFIO_DMA_MAP_via_Memory_listener.png.html "Enlarge"){.internal}
:::

**Figure 19:** QEMU calls VFIO DMA MAP via Memory listener. See slides
from:
[\[1\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external
.text rel="nofollow"}
:::
:::
:::

::: {.thumb .tright}
::: {.thumbinner style="width:302px;"}
[![Type 1 IOMMU Tracks
VA-GFN](images/thumb/1/1d/Type_1_IOMMU_Tracks_VA-GFN.png/300px-Type_1_IOMMU_Tracks_VA-GFN.png){.thumbimage
decoding="async" width="300" height="237"
srcset="images/thumb/1/1d/Type_1_IOMMU_Tracks_VA-GFN.png/450px-Type_1_IOMMU_Tracks_VA-GFN.png 1.5x, images/thumb/1/1d/Type_1_IOMMU_Tracks_VA-GFN.png/600px-Type_1_IOMMU_Tracks_VA-GFN.png 2x"}](index.php/File:Type_1_IOMMU_Tracks_VA-GFN.png.html){.image}

::: thumbcaption
::: magnify
[](index.php/File:Type_1_IOMMU_Tracks_VA-GFN.png.html "Enlarge"){.internal}
:::

**Figure 20:** Type 1 IOMMU Tracks \<VA, GFN\>. See slides from:
[\[1\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external
.text rel="nofollow"}
:::
:::
:::

**Figure 20:** Type 1 IOMMU tracks \<VA, GFN (Guest Frame Number)\>. We
build a table to list the QEMU VAs and track the their mapping relation
to Guest Frame Numbers (GFNs).

### [Scheduling]{#Scheduling .mw-headline}

Scheduling is handled by the host mdev driver.

### [Kernel API]{#Kernel_API .mw-headline}

Kernel documentation used for implementing a VFIO Mediated Device may be
found at
[kernel.org](https://www.kernel.org/doc/html/latest/driver-api/vfio-mediated-device.html){.external
.text rel="nofollow"}.

### [Sample Code]{#Sample_Code .mw-headline}

Sample code for various mdev implementations may be found below:

**Mediated Virtual PCI Display Host Device:**

1.  [mdpy.c](https://github.com/Open-IOV/VFIO-Mdev_Samples/blob/master/mdpy.c){.external
    .text rel="nofollow"}
2.  [mdpy-fb.c](https://github.com/Open-IOV/VFIO-Mdev_Samples/blob/master/mdpy-fb.c){.external
    .text rel="nofollow"}
3.  [mbochs.c](https://github.com/Open-IOV/VFIO-Mdev_Samples/blob/master/mbochs.c){.external
    .text rel="nofollow"}.

**Serial PCI Port-based Mediated Device:**

1.  [mtty.c](https://github.com/Open-IOV/VFIO-Mdev_Samples/blob/master/mtty.c){.external
    .text rel="nofollow"}.

###### [Compilation]{#Compilation .mw-headline}

To compile the kernel modules linked above you should have your
distro\'s equivalent of the build-essential package installed. The
included Makefile should provide all that is necessary to successfully
compile the kernel modules. You can type the following command to
compile the modules from within the directory:

`make`

When the make operation has been completed successfully the directory
will now contain .ko files. These files are the binary kernel modules.

###### [Loading Kernel Modules]{#Loading_Kernel_Modules .mw-headline}

Now that you have compiled the kernel modules you may load them via the
insmod command.

For example to load the **mtty.ko** module run the following command
from within the directory where you built the modules:

`insmod mtty.ko`

###### [Unloading Kernel Modules]{#Unloading_Kernel_Modules .mw-headline}

To unload any of the kernel modules you may make use of the rmmod
command.

For example to unload the **mtty.ko** module run the following command:

`rmmod mtty.ko`

###### [Additional Documentation]{#Additional_Documentation .mw-headline}

An additional guide explaining how to make use of the mtty.c sample code
may be found at [line
308](https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core.git/tree/Documentation/driver-api/vfio-mediated-device.rst?h=driver-core-next&id=7de3697e9cbd4bd3d62bafa249d57990e1b8f294#n308){.external
.text rel="nofollow"} of the kernel.org VFIO Mediated Device
documentation.

### [Mdev Mode Requirements]{#Mdev_Mode_Requirements .mw-headline}

Driver Support.

Software HPA\<-\>GPA Boundary Enforcement.

## [SR-IOV Mode]{#SR-IOV_Mode .mw-headline}

**SR-IOV Mode (Single Root I/O Virtualization)** involves hardware
assisted virtualization on I/O peripherals.

### [Knowledge Resources Used]{#Knowledge_Resources_Used_3 .mw-headline}

This section is supported by significant contributions in open source by
Zheng Xiao, Jerry Jiang, & Ken Xue.

See reference 6 in the [References (Talks & Reading
Material)](index.php/Virtual_IO_Internals.html#References_(Talks_&_Reading_Material)){.external
.text rel="nofollow"} section.

### [Instruction Execution]{#Instruction_Execution_2 .mw-headline}

SR-IOV communicates instructions from a virtual function (VF) directly
to the [PCI
BAR](https://infogalactic.com/info/PCI_configuration_space){.external
.text rel="nofollow"}.

### [Memory Management]{#Memory_Management_2 .mw-headline}

Guests are presenting with passthrough memory regions by the device
firmware.

### [Scheduling]{#Scheduling_2 .mw-headline}

Scheduling may be handled by the host mdev driver and/or the device
firmware.

### [SR-IOV Mode Requirements]{#SR-IOV_Mode_Requirements .mw-headline}

Driver Support.

Device SR-IOV support.

Firmware HPA\<-\>GPA Boundary Enforcement.

## [SIOV Mode]{#SIOV_Mode .mw-headline}

**SIOV (Scalable I/O Virtualization)** involves the combination of
concepts form both [SR-IOV
Mode](index.php/Mediated_Device_Internals.html#SR-IOV_Mode){.external
.text rel="nofollow"} and [Mdev
Mode](index.php/Mediated_Device_Internals.html#Mdev_Mode){.external
.text rel="nofollow"} as well as novel concepts like shared IOMMU aware
buffers and offloading of PCI config space VM-exits (slow path) to a
discrete controller.

**Revision 1.0 of the SIOV specification** can be read on the [Open
Compute Project
website](https://www.opencompute.org/documents/ocp-scalable-io-virtualization-technical-specification-revision-1-v1-2-pdf){.external
.text rel="nofollow"}.

### [Knowledge Resources Used]{#Knowledge_Resources_Used_4 .mw-headline}

This section is supported by significant contributions in open source by
Kevin Tian, Tina Zhang, Xin Zeng, Yi Liu.

See references 3, 4, 5, 16, 17, 18, 19, 20, 21 in the [References (Talks
& Reading
Material)](index.php/Virtual_IO_Internals.html#References_(Talks_&_Reading_Material)){.external
.text rel="nofollow"} section.

### [Memory Management]{#Memory_Management_3 .mw-headline}

Enhancements to [Intel
VT-d](https://edc.intel.com/content/www/us/en/design/ipla/software-development-platforms/client/platforms/alder-lake-desktop/12th-generation-intel-core-processors-datasheet-volume-1-of-2/002/intel-virtualization-technology-for-directed-i-o/){.external
.text rel="nofollow"} introduce new \'Scalable Mode\' to allow the
platform to assign more granular IOMMU allocations to mediated devices
([mdev memory
management](index.php/Mediated_Device_Internals.html#Memory_Management){.external
.text rel="nofollow"}). This change is referred to as an IOMMU Aware
Mediated Device.

#### [IOMMU Aware Mediated Device]{#IOMMU_Aware_Mediated_Device .mw-headline}

SIOV made several changes to the VFIO driver, Intel IOMMU, and Mediated
Device Framework.

The full list of these changes can be seen in the [mailing list archive
on **lwn.net** under **(vfio/mdev: IOMMU aware mediated
device)**](https://lwn.net/ml/linux-kernel/20190222021927.13132-1-baolu.lu@linux.intel.com/){.external
.text rel="nofollow"}.

#### [Shared Hardware Workqueues]{#Shared_Hardware_Workqueues .mw-headline}

SIOV makes use of Shared Hardware Workqueues which may be accessed by
processes or Virtual Machines.

[According to
**kernel.org**](https://www.kernel.org/doc/html/latest/x86/sva.html){.external
.text rel="nofollow"}: *\"In order to allow the hardware to distinguish
the context for which work is being executed in the hardware by SWQ
interface, SIOV uses Process Address Space ID (PASID), which is a 20-bit
number defined by the PCIe SIG. PASID value is encoded in all
transactions from the device. This allows the IOMMU to track I/O on a
per-PASID granularity in addition to using the PCIe Resource Identifier
(RID) which is the Bus/Device/Function.\"*

### [SIOV Mode Requirements]{#SIOV_Mode_Requirements .mw-headline}

Driver Support.

Device SIOV support.

Firmware HPA\<-\>GPA Boundary Enforcement.

## []{#References_.28Talks_.26_Reading_Material.29}[References (Talks & Reading Material)]{#References_(Talks_&_Reading_Material) .mw-headline}

1.  [\[2016\] vGPU on KVM - A VFIO Based Framework by Neo Jia & Kirti
    Wankhede](https://www.youtube.com/watch?v=Xs0TJU_sIPc){.external
    .text rel="nofollow"} -
    [slides](https://www.linux-kvm.org/images/5/59/02x03-Neo_Jia_and_Kirti_Wankhede-vGPU_on_KVM-A_VFIO_based_Framework.pdf){.external
    .text rel="nofollow"}
2.  [\[2016\] An Introduction to PCI Device Assignment with VFIO by Alex
    Williamson](https://www.youtube.com/watch?v=WFkdTFTOTpA){.external
    .text rel="nofollow"} -
    [slides](http://events17.linuxfoundation.org/sites/events/files/slides/An%20Introduction%20to%20PCI%20Device%20Assignment%20with%20VFIO%20-%20Williamson%20-%202016-08-30_0.pdf){.external
    .text rel="nofollow"}
3.  [\[2017\] Scalable I/O Virtualization by Kevin
    Tian](https://events19.linuxfoundation.cn/wp-content/uploads/2017/11/Intel%C2%AE-Scalable-I_O-Virtualization_Kevin-Tian.pdf){.external
    .text rel="nofollow"}
4.  [\[2019\] Bring a Scalable IOV Capable Device into Linux World by
    Xin Zeng & Yi
    Liu](https://www.youtube.com/watch?v=G6D-jaCs6sc){.external .text
    rel="nofollow"} -
    [slides](https://static.sched.com/hosted_files/kvmforum2019/5e/Bring%20a%20scalable%20IOV%20capable%20device%20into%20Linux%20-%20KVM%202019.pdf){.external
    .text rel="nofollow"}
5.  [Scalable I/O Virtualization Revision
    1.0](https://www.opencompute.org/documents/ocp-scalable-io-virtualization-technical-specification-revision-1-v1-2-pdf){.external
    .text rel="nofollow"}
6.  [\[2018\] Live Migration Support for GPU with SRIOV by Zheng Xiao,
    Jerry Jiang & Ken
    Xue](https://www.youtube.com/watch?v=_tB3EbFDcRQ){.external .text
    rel="nofollow"} -
    [slides](https://events19.linuxfoundation.org/wp-content/uploads/2017/12/Live-Migration-Support-for-GPU-with-SRIOV-Challenges-and-Solution-Zheng-Xiao-Alibaba-Cloud-Jerry-Jiang-Ken-Xue-AMD.pdf){.external
    .text rel="nofollow"}
7.  [\[2017\] Intel GVT-g: From Production to Upstream - Zhi Wang,
    Intel](https://www.youtube.com/watch?v=UODxW1opfn0){.external .text
    rel="nofollow"}
8.  [\[2016\] Qemu Graphics Update 2016 by Gerd
    Hoffmann](https://www.youtube.com/watch?v=DKYvQ3FdFeo){.external
    .text rel="nofollow"}
9.  [IOCTL](https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core.git/tree/Documentation/driver-api/ioctl.rst){.external
    .text rel="nofollow"}
10. [eventfd](https://man7.org/linux/man-pages/man2/eventfd.2.html){.external
    .text rel="nofollow"} -
    [root/virt/kvm/eventfd.c](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/virt/kvm/eventfd.c){.external
    .text rel="nofollow"}
11. (kernel diff:: [KVM:
    irqfd](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=721eecbf4fe995ca94a9edec0c9843b1cc0eaaf3){.external
    .text rel="nofollow"})
12. (kernel diff:: [KVM: add ioeventfd
    support](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d34e6b175e61821026893ec5298cc8e7558df43a){.external
    .text rel="nofollow"})
13. [KVM irqfd &
    ioeventfd](https://web.archive.org/web/20220120223711/http://blog.allenx.org/2015/07/05/kvm-irqfd-and-ioeventfd){.external
    .text rel="nofollow"}
14. [VFIO - Virtual Function
    I/O](https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core.git/tree/Documentation/driver-api/vfio.rst){.external
    .text rel="nofollow"} -
    [root/virt/kvm/vfio.c](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/virt/kvm/vfio.c){.external
    .text rel="nofollow"}
15. [VFIO Mediated
    Devices](https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core.git/tree/Documentation/driver-api/vfio-mediated-device.rst){.external
    .text rel="nofollow"}
16. [IOMMU Aware Mediated
    Device](https://lwn.net/ml/linux-kernel/20190222021927.13132-1-baolu.lu@linux.intel.com/){.external
    .text rel="nofollow"}
17. [Hardware-Assisted Mediated Pass-Through with VFIO by Kevin
    Tian](https://www.youtube.com/watch?v=95KSKrZM8oQ){.external .text
    rel="nofollow"}
18. [\[2017\] Generic Buffer Sharing Mechanism for Mediated Devices by
    Tina Zhang](https://www.youtube.com/watch?v=cHMLBcHplhk){.external
    .text rel="nofollow"}
19. [\[2019\] Toward a Virtualization World Built on Mediated
    Pass-Through - Kevin
    Tian](https://www.youtube.com/watch?v=KWRKx_uxUDI){.external .text
    rel="nofollow"}
20. [Auxiliary
    Bus](https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core.git/tree/Documentation/driver-api/auxiliary_bus.rst?h=driver-core-next&id=7de3697e9cbd4bd3d62bafa249d57990e1b8f294){.external
    .text rel="nofollow"}
21. [Shared Virtual
    Addressing](https://www.kernel.org/doc/html/latest/x86/sva.html){.external
    .text rel="nofollow"}
22. [vfio.blogspot.com](https://vfio.blogspot.com/){.external .text
    rel="nofollow"}
23. [XenGT by Kevin
    Tian](https://01.org/sites/default/files/xengt.pdf){.external .text
    rel="nofollow"}
24. [VMWare Academic Publications: GPU Virtualization by Micah Dowty &
    Jeremy
    Sugerman](https://web.archive.org/web/20130721094438/http://labs.vmware.com/academic/publications/gpu-virtualization){.external
    .text rel="nofollow"}
25. [PCI SIG I/O
    Virtualization](https://pcisig.com/specifications/iov){.external
    .text rel="nofollow"}
26. [VMGL by the University of Toronto Computer Science
    Department](https://web.archive.org/web/20120108034526/http://sysweb.cs.toronto.edu/vmgl){.external
    .text rel="nofollow"}
27. [Kepler VGX
    Hypervisor](https://web.archive.org/web/20120518125815/http://www.nvidia.com/object/vgx-hypervisor.html){.external
    .text rel="nofollow"}
28. [Intel Virtualization Technology for Directed I/O (VT-d): Enhancing
    Intel platforms for efficient virtualization of I/O
    devices](https://web.archive.org/web/20090218010221/http://software.intel.com/en-us/articles/intel-virtualization-technology-for-directed-io-vt-d-enhancing-intel-platforms-for-efficient-virtualization-of-io-devices){.external
    .text rel="nofollow"}
29. [DMA-Mapping.txt (Dynamic DMA
    Mapping)](https://lwn.net/2001/0712/a/dma-interface.php3){.external
    .text rel="nofollow"}
30. [KVM MMU Virtualization by Xiao
    Guangrong](https://events.static.linuxfound.org/slides/2011/linuxcon-japan/lcj2011_guangrong.pdf){.external
    .text rel="nofollow"}
31. [OSDEV: PCI](https://wiki.osdev.org/PCI){.external .text
    rel="nofollow"}
32. [Down to the TLP: How PCI express devices talk (Part
    I)](http://xillybus.com/tutorials/pci-express-tlp-pcie-primer-tutorial-guide-1){.external
    .text rel="nofollow"}
33. [Down to the TLP: How PCI express devices talk (Part
    II)](http://xillybus.com/tutorials/pci-express-tlp-pcie-primer-tutorial-guide-2){.external
    .text rel="nofollow"}
34. [Intel ACRN: Enabling SR-IOV
    Virtualization](https://projectacrn.github.io/latest/tutorials/sriov_virtualization.html){.external
    .text rel="nofollow"}
35. [Kernel.org
    sysfs-bus-pci](https://docs.kernel.org/admin-guide/abi-testing.html#abi-file-testing-sysfs-bus-pci){.external
    .text rel="nofollow"}
36. [Linux Kernel Labs: Memory
    Mapping](https://linux-kernel-labs.github.io/refs/heads/master/labs/memory_mapping.html){.external
    .text rel="nofollow"}
37. [Inside Windows Page Frame Number (PFN) - Part
    1](https://rayanfam.com/topics/inside-windows-page-frame-number-part1/){.external
    .text rel="nofollow"}
38. [Direct Rendering Infrastructure
    (DRI)](https://en.wikipedia.org/wiki/Direct_Rendering_Infrastructure){.external
    .text rel="nofollow"}
39. [DRI User
    Guide](https://dri.sourceforge.net/doc/DRIuserguide.html){.external
    .text rel="nofollow"}
40. [Linux Kernel Labs: IO
    Virtualization](https://linux-kernel-labs.github.io/refs/heads/master/lectures/virt.html#i-o-virtualization){.external
    .text rel="nofollow"}
41. [Gallium3D: Main
    Page](https://dri.freedesktop.org/doxygen/gallium/){.external .text
    rel="nofollow"}
42. [Gallium3D
    Wiki](https://web.archive.org/web/20080107043445/http://www.tungstengraphics.com/wiki/index.php/Gallium3D){.external
    .text rel="nofollow"}
43. [Gallium3D talk from XDS
    2007](https://web.archive.org/web/20090219182518/http://www.tungstengraphics.com/wiki/files/gallium3d-xds2007.pdf){.external
    .text rel="nofollow"}
44. [Memory Management High Level Design
    (ARCN)](https://projectacrn.github.io/latest/developer-guides/hld/hv-memmgt.html){.external
    .text rel="nofollow"}
45. [Block Architecture Diagrams for Geforce
    series](https://web.archive.org/web/20080111122628/http://www.digit-life.com/articles2/gffx/nv40-part1-a.html){.external
    .text rel="nofollow"}
46. [Block diagrams for 40 series instruction
    format](http://freenv.svn.sourceforge.net/viewvc/freenv/doc/shaderinsnformat/bitdiagen/){.external
    .text rel="nofollow"}
47. [PCI Configuration
    Space](https://en.wikipedia.org/wiki/PCI_configuration_space "wikipedia:PCI configuration space"){.extiw}
48. [GPU virtual memory in WDDM
    2.0](https://learn.microsoft.com/en-us/windows-hardware/drivers/display/gpu-virtual-memory-in-wddm-2-0){.external
    .text rel="nofollow"}
49. [Intel Graphics Programmer\'s Reference Manuals
    (PRM)](https://01.org/linuxgraphics/documentation/hardware-specification-prms){.external
    .text rel="nofollow"}
50. [i915: Hardware Contexts (and some bits about
    batchbuffers)](https://bwidawsk.net/blog/2013/1/i915-hardware-contexts-and-some-bits-about-batchbuffers/){.external
    .text rel="nofollow"}
51. [i915: The Global GTT Part
    1](https://bwidawsk.net/blog/2014/6/the-global-gtt-part-1/){.external
    .text rel="nofollow"}
52. [i915: Aliasing PPGTT Part
    2](https://bwidawsk.net/blog/2014/6/aliasing-ppgtt-part-2/){.external
    .text rel="nofollow"}
53. [i915: True PPGTT Part
    3](https://bwidawsk.net/blog/2014/7/true-ppgtt-part-3/){.external
    .text rel="nofollow"}
54. [i915: Future PPGTT Part 4 (Dynamic page table allocations, 64 bit
    address space, GPU \"mirroring\", and yeah, something about relocs
    too)](https://bwidawsk.net/blog/2014/7/future-ppgtt-part-4-dynamic-page-table-allocations-64-bit-address-space-gpu-mirroring-and-yeah-something-about-relocs-too/){.external
    .text rel="nofollow"}
55. [i915: Security of the Intel Graphics Stack - Part 1 -
    Introduction](https://igor-blue.github.io/2021/02/10/graphics-part1.html){.external
    .text rel="nofollow"}
56. [i915: Security of the Intel Graphics Stack - Part 2 - FW \<-\>
    GuC](https://igor-blue.github.io/2021/02/24/graphics-part2.html){.external
    .text rel="nofollow"}
57. [A deep dive into QEMU: PCI slave
    devices](https://airbus-seclab.github.io/qemu_blog/pci_slave.html){.external
    .text rel="nofollow"}
58. [Debugging QEMU Guests with GDB (start & stop, examine state like
    registers & memory, set breakpoints &
    watchpoints)](https://qemu-project.gitlab.io/qemu/system/gdb.html){.external
    .text rel="nofollow"}
59. [IOMMU
    Introduction](https://terenceli.github.io/%E6%8A%80%E6%9C%AF/2019/08/04/iommu-introduction){.external
    .text rel="nofollow"}
60. [Arch Wiki: Intel
    Graphics](https://wiki.archlinux.org/title/intel_graphics){.external
    .text rel="nofollow"}
61. [An Introduction to Intel Graphics Virtualization Technology (legacy
    GVT-g) by Zhi
    Wang](https://01.org/sites/default/files/documentation/an_introduction_to_intel_gvt-g_for_external.pdf){.external
    .text rel="nofollow"}
62. [Kernel.org: vfio iommu type1: Add support for mediated
    devices](https://patchwork.kernel.org/project/qemu-devel/patch/1478293856-8191-11-git-send-email-kwankhede@nvidia.com/){.external
    .text rel="nofollow"}
63. [Hardware and Compute Abstraction Layers For Accelerated Computing
    Using Graphics Hardware and Conventional
    CPUs](https://archive.ll.mit.edu/HPEC/agendas/proc07/Day3/10_Hensley_Abstract.pdf){.external
    .text rel="nofollow"}
64. [More Shadow Walker: The Progression of TLB-Splitting on
    x86](https://www.blackhat.com/docs/us-14/materials/us-14-Torrey-MoRE-Shadow-Walker-The-Progression-Of-TLB-Splitting-On-x86.pdf){.external
    .text rel="nofollow"}
65. [MMU Virtualization Via Intel EPT: Technical
    Details](https://revers.engineering/mmu-ept-technical-details/){.external
    .text rel="nofollow"}
66. [Alex Williamson Github: VFIO Development (tree
    next)](https://github.com/awilliam/linux-vfio/tree/next){.external
    .text rel="nofollow"}
67. [GPU PMU: performance monitoring with perf
    event](https://events.static.linuxfound.org/slides/2011/linuxcon-japan/lcj2011_linming.pdf){.external
    .text rel="nofollow"}
68. [Remote Procedure Call
    (RPC)](https://book.systemsapproach.org/e2e/rpc.html){.external
    .text rel="nofollow"}
69. [Virtual Open Systems: API
    Remoting](http://www.virtualopensystems.com/en/products/api-remoting/){.external
    .text rel="nofollow"}
70. [A Full GPU Virtualization Solution with Mediated Pass-Through by
    Yiying Zhang, David Cowperthwaite, Kun Tian, and Yaozu
    Dong](https://www.usenix.org/conference/atc14/technical-sessions/presentation/tian){.external
    .text rel="nofollow"} -
    [\[Paper\]](https://www.usenix.org/system/files/conference/atc14/atc14-paper-tian.pdf){.external
    .text rel="nofollow"} -
    [\[Video\]](https://www.youtube.com/watch?v=vvYmDQKZ6MQ){.external
    .text rel="nofollow"} - [\[Slides
    1\]](https://www.usenix.org/sites/default/files/conference/protected-files/atc14_slides_tian.pdf){.external
    .text rel="nofollow"} - [\[Slides
    2\]](https://cseweb.ucsd.edu/~yiying/cse291j-winter20/reading/GPU-Virtualization.pdf){.external
    .text rel="nofollow"}
71. [\[2014\] KvmGT: A Full GPU Virtualization Solution by Jike
    Song](https://www.youtube.com/watch?v=-iuIu7_GuEo){.external .text
    rel="nofollow"}
72. [docs.kernel.org: DRM Memory
    Management](https://docs.kernel.org/gpu/drm-mm.html){.external .text
    rel="nofollow"}
73. [docs.kernel.org: PCI Bus
    Subsystem](https://docs.kernel.org/PCI/index.html){.external .text
    rel="nofollow"}
74. [openglbook.com: What is
    OpenGL?](https://openglbook.com/chapter-0-preface-what-is-opengl.html){.external
    .text rel="nofollow"}
75. [\[2020\] Getting pixels on screen: introduction to Kernel Mode
    Setting (KMS) by Simon
    Ser](https://yewtu.be/watch?v=haes4_Xnc5Q){.external .text
    rel="nofollow"} -
    [slides](https://fs.emersion.fr/protected/presentations/present.html?src=kms-foss-north/index.md#1){.external
    .text rel="nofollow"}
76. [Dynamic Mediation for Live Migration
    VFIO-PCI](https://sdic.sjtu.edu.cn/wp-content/uploads/2019/12/Mediated-Pass-Through-MPT-NVIDIA-vGPU.pptx){.external
    .text rel="nofollow"}
77. [DRM Management Via
    CGroups](https://lwn.net/Articles/746127/){.external .text
    rel="nofollow"}
78. [DRM: Add GEM (\"graphics execution manager\") to i915
    driver.](https://lwn.net/Articles/292583/){.external .text
    rel="nofollow"}
:::

::: printfooter
Retrieved from
\"[https://open-iov.org/index.php?title=Virtual_I/O_Internals&oldid=23792](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html){dir="ltr"}\"
:::
:::

::: {#catlinks .catlinks .catlinks-allhidden mw="interface"}
:::
:::
:::

::: {#mw-navigation}
## Navigation menu

::: {#mw-head}
### Personal tools {#p-personal-label .vector-menu-heading}

::: vector-menu-content
-   [[Log
    in](https://open-iov.org/index.php?title=Special:UserLogin&returnto=Virtual+I%2FO+Internals&returntoquery=oldid%3D23792 "You are encouraged to log in; however, it is not mandatory [o]"){accesskey="o"}]{#pt-login}
:::

::: {#left-navigation}
### Namespaces {#p-namespaces-label .vector-menu-heading}

::: vector-menu-content
-   [[Page](index.php/Virtual_I/O_Internals.html "View the content page [c]"){accesskey="c"}]{#ca-nstab-main}
-   [[Discussion](https://open-iov.org/index.php?title=Talk:Virtual_I/O_Internals&action=edit&redlink=1 "Discussion about the content page (page does not exist) [t]"){rel="discussion"
    accesskey="t"}]{#ca-talk}
:::

### Variants [expanded]{.vector-menu-checkbox-expanded} [collapsed]{.vector-menu-checkbox-collapsed} {#p-variants-label .vector-menu-heading}

::: vector-menu-content
:::
:::

::: {#right-navigation}
### Views {#p-views-label .vector-menu-heading}

::: vector-menu-content
-   [[Read](index.php/Virtual_I/O_Internals.html)]{#ca-view}
-   [[View
    source](index.php%3Ftitle=Virtual_I%252FO_Internals&action=edit.html "This page is protected.
    You can view its source [e]"){accesskey="e"}]{#ca-viewsource}
-   [[View
    history](index.php%3Ftitle=Virtual_I%252FO_Internals&action=history.html "Past revisions of this page [h]"){accesskey="h"}]{#ca-history}
:::

### More [expanded]{.vector-menu-checkbox-expanded} [collapsed]{.vector-menu-checkbox-collapsed} {#p-cactions-label .vector-menu-heading}

::: vector-menu-content
:::

::: {#p-search .vector-search-box role="search"}
<div>

### Search

::: {#simpleSearch search-loc="header-navigation"}
:::

</div>
:::
:::
:::

::: {#mw-panel}
::: {#p-logo role="banner"}
[](images/archive/c/c3/20221214193624!I915_Scheduling_Events_and_Requests.png.html "Visit the main page"){.mw-wiki-logo}
:::

### Navigation {#p-navigation-label .vector-menu-heading}

::: vector-menu-content
-   [[Main
    page](images/archive/c/c3/20221214193624!I915_Scheduling_Events_and_Requests.png.html "Visit the main page [z]"){accesskey="z"}]{#n-mainpage-description}
-   [[Recent
    changes](https://open-iov.org/index.php/Special:RecentChanges "A list of recent changes in the wiki [r]"){accesskey="r"}]{#n-recentchanges}
-   [[Random
    page](index.php/Special:Random.html "Load a random page [x]"){accesskey="x"}]{#n-randompage}
-   [[Help about
    MediaWiki](https://www.mediawiki.org/wiki/Special:MyLanguage/Help:Contents)]{#n-help-mediawiki}
:::

### Tools {#p-tb-label .vector-menu-heading}

::: vector-menu-content
-   [[What links
    here](index.php/Special:WhatLinksHere/Virtual_I/O_Internals.html "A list of all wiki pages that link here [j]"){accesskey="j"}]{#t-whatlinkshere}
-   [[Related
    changes](https://open-iov.org/index.php/Special:RecentChangesLinked/Virtual_I/O_Internals "Recent changes in pages linked from this page [k]"){rel="nofollow"
    accesskey="k"}]{#t-recentchangeslinked}
-   [[Special
    pages](index.php/Special:SpecialPages.html "A list of all special pages [q]"){accesskey="q"}]{#t-specialpages}
-   [[Printable
    version](javascript:print(); "Printable version of this page [p]"){rel="alternate"
    accesskey="p"}]{#t-print}
-   [[Permanent
    link](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html "Permanent link to this revision of the page")]{#t-permalink}
-   [[Page
    information](index.php%3Ftitle=Virtual_I%252FO_Internals&action=info.html "More information about this page")]{#t-info}
:::
:::
:::

-   [This page was last edited on 19 June 2023, at
    17:54.]{#footer-info-lastmod}

```{=html}
<!-- -->
```
-   [[Privacy
    policy](https://open-iov.org/index.php/Open-IOV:Privacy_policy "Open-IOV:Privacy policy")]{#footer-places-privacy}
-   [[About
    Open-IOV](index.php/Open-IOV:About.html "Open-IOV:About")]{#footer-places-about}
-   [[Disclaimers](https://open-iov.org/index.php/Open-IOV:General_disclaimer "Open-IOV:General disclaimer")]{#footer-places-disclaimer}

```{=html}
<!-- -->
```
-   [[![Powered by
    MediaWiki](resources/assets/poweredby_mediawiki_88x31.png){srcset="resources/assets/poweredby_mediawiki_132x47.png 1.5x, resources/assets/poweredby_mediawiki_176x62.png 2x"
    width="88" height="31"
    loading="lazy"}](https://www.mediawiki.org/)]{#footer-poweredbyico}