::: {#mw-page-base .noprint} ::: ::: {#mw-head-base .noprint} ::: ::: {#content .mw-body role="main"} []{#top} ::: {#siteNotice} ::: ::: mw-indicators ::: # Virtual I/O Internals {#firstHeading .firstHeading} ::: {#bodyContent .vector-body} ::: {#siteSub .noprint} From Open-IOV ::: ::: {#contentSub} ::: {.mw-revision .warningbox} ::: {#mw-revision-info} Revision as of 17:54, 19 June 2023 by [Arthur](./index.php%3Ftitle=User:Arthur&action=edit&redlink=1.html "User:Arthur (page does not exist)"){.new .mw-userlink} [([talk](./index.php%3Ftitle=User_talk:Arthur&action=edit&redlink=1.html "User talk:Arthur (page does not exist)"){.new .mw-usertoollinks-talk} \| [contribs](index.php/Special:Contributions/Arthur.html "Special:Contributions/Arthur"){.mw-usertoollinks-contribs})]{.mw-usertoollinks} [([[[→‎References (Talks & Reading Material)](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#References_(Talks_&_Reading_Material))]{.autocomment}]{dir="auto"})]{.comment} ::: ::: {#mw-revision-nav} ([diff](https://open-iov.org/index.php?title=Virtual_I/O_Internals&diff=prev&oldid=23792 "Virtual I/O Internals")) [← Older revision](https://open-iov.org/index.php?title=Virtual_I/O_Internals&direction=prev&oldid=23792 "Virtual I/O Internals") \| Latest revision (diff) \| Newer revision → (diff) ::: ::: ::: ::: {#contentSub2} ::: ::: {#jump-to-nav} ::: [Jump to navigation](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#mw-head){.mw-jump-link} [Jump to search](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#searchInput){.mw-jump-link} ::: {#mw-content-text .mw-body-content .mw-content-ltr lang="en" dir="ltr"} ::: mw-parser-output The following document details the internals of a **VFIO (Virtual Function I/O)** driven **Shared** **I/O Device.** > An absence of critical technical documentation has historically slowed > growth and adoption of developer ecosystems for GPU virtualization. > > This > [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/){.external > .text rel="nofollow"} licensed content can either be used with > attribution, or used as inspiration for new documentation, created by > GPU vendors for public commercial distribution as developer > documentation. > > Where possible, this documentation will clearly label dates and > versions of observed-but-not-guaranteed behaviour vs. > vendor-documented stable interfaces/behaviour with guarantees of > forward or backward compatibility. This article places emphasis on the **Virtual GPU (vGPU)** use case however these concepts apply generically to virtualization of I/O devices (TPUs, NICs, storage peripherals, ect..). -------------------------------------------------------------------------------------- ------------------------------------------------------------------------- --------------------------------------------- Mdev Mode SR-IOV Mode SIOV Mode No hardware assistance needed. Hardware assistance needed. Hardware assistance needed. Host requires insight about guest workload. Host ignorance of guest workload. Host requires insight about guest workload. Error reporting. No guest driver error reporting. Error reporting. In depth dynamic monitoring. Basic dynamic monitoring. In depth dynamic monitoring. Software defined MMU guest separation. Firmware defined MMU guest separation. Firmware defined MMU guest separation. Requires deferred instructions to be supported by host software (support libraries). Guest is ignorant of host supported software such as support libraries. Routing interrupts. Routing interrupts. Routing interrupts. Device reset. Device reset. Device reset. Enable/Disable device. Enable/Disable device. Enable/Disable device. Support for multiple scheduling techniques. Support for multiple scheduling techniques. Support for multiple scheduling techniques. Single PCI requester ID. Multiple PCI requester IDs. Multiple PCI requester IDs. -------------------------------------------------------------------------------------- ------------------------------------------------------------------------- --------------------------------------------- : Comparison of Assistance Modes ::: {#toc .toc role="navigation" aria-labelledby="mw-toc-heading"} ::: {.toctitle lang="en" dir="ltr"} ## Contents {#mw-toc-heading} []{.toctogglespan} ::: - [[1]{.tocnumber} [All Modes]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#All_Modes) - [[1.1]{.tocnumber} [Knowledge Resources Used]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Knowledge_Resources_Used) - [[1.2]{.tocnumber} [Binding VFIO devices]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Binding_VFIO_devices) - [[1.3]{.tocnumber} [Programming the IOMMU]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Programming_the_IOMMU) - [[1.4]{.tocnumber} [VFIO Memory Mapped IO]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#VFIO_Memory_Mapped_IO) - [[1.5]{.tocnumber} [Getting the VFIO Group File Descriptor]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Getting_the_VFIO_Group_File_Descriptor) - [[1.6]{.tocnumber} [VFIO device file descriptor]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#VFIO_device_file_descriptor) - [[1.7]{.tocnumber} [VFIO Interrupts]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#VFIO_Interrupts) - [[1.7.1]{.tocnumber} [IRQs]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#IRQs) - [[1.7.1.1]{.tocnumber} [Get Device Info]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Get_Device_Info) - [[1.7.1.2]{.tocnumber} [Get Region Info]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Get_Region_Info) - [[1.7.1.3]{.tocnumber} [Get IRQ Info]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Get_IRQ_Info) - [[1.7.1.4]{.tocnumber} [Set IRQs]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Set_IRQs) - [[1.8]{.tocnumber} [Device Decomposure/Recomposure]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Device_Decomposure/Recomposure) - [[1.9]{.tocnumber} [Memory Management Unit (MMU)]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Memory_Management_Unit_(MMU)) - [[1.9.1]{.tocnumber} [MMIO Isolation (Platform MMU)]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#MMIO_Isolation_(Platform_MMU)) - [[1.9.2]{.tocnumber} [VRAM Isolation (GPU GMMU)]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#VRAM_Isolation_(GPU_GMMU)) - [[1.9.3]{.tocnumber} [Platform MMIO \<-\> GPU Shadow Pages]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Platform_MMIO_%3C-%3E_GPU_Shadow_Pages) - [[1.10]{.tocnumber} [VFIO Quirks (region traps)]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#VFIO_Quirks_(region_traps)) - [[1.11]{.tocnumber} [Known IOMMU Issues]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Known_IOMMU_Issues) - [[2]{.tocnumber} [Mdev Mode]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Mdev_Mode) - [[2.1]{.tocnumber} [Knowledge Resources Used]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Knowledge_Resources_Used_2) - [[2.2]{.tocnumber} [Mediated Core]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Mediated_Core) - [[2.3]{.tocnumber} [Device Initialization]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Device_Initialization) - [[2.3.1]{.tocnumber} [Registering VFIO MDEV as a driver]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Registering_VFIO_MDEV_as_a_driver) - [[2.3.2]{.tocnumber} [Registering PCIE Device with the Mediated Core]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Registering_PCIE_Device_with_the_Mediated_Core) - [[2.3.3]{.tocnumber} [Registering Mediated Callbacks (CBs)]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Registering_Mediated_Callbacks_(CBs)) - [[2.3.4]{.tocnumber} [Creating a Mediated Device via mdev-sysfsdev API]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Creating_a_Mediated_Device_via_mdev-sysfsdev_API) - [[2.3.5]{.tocnumber} [QEMU adds VFIO device to IOMMU container-group]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#QEMU_adds_VFIO_device_to_IOMMU_container-group) - [[2.3.6]{.tocnumber} [QEMU passes mdev device file descriptor to VM]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#QEMU_passes_mdev_device_file_descriptor_to_VM) - [[2.4]{.tocnumber} [Instruction Execution]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Instruction_Execution) - [[2.4.1]{.tocnumber} [IRQ remapping]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#IRQ_remapping) - [[2.4.2]{.tocnumber} [Interrupt Injection]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Interrupt_Injection) - [[2.5]{.tocnumber} [Memory Management]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Memory_Management) - [[2.5.1]{.tocnumber} [Region Passthrough]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Region_Passthrough) - [[2.5.1.1]{.tocnumber} [Emulated Regions]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Emulated_Regions) - [[2.5.1.2]{.tocnumber} [Passthrough Regions]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Passthrough_Regions) - [[2.5.1.3]{.tocnumber} [Region Access]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Region_Access) - [[2.5.2]{.tocnumber} [EPT Page Violations]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#EPT_Page_Violations) - [[2.5.3]{.tocnumber} [Mediated DMA Translations]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Mediated_DMA_Translations) - [[2.6]{.tocnumber} [Scheduling]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Scheduling) - [[2.7]{.tocnumber} [Kernel API]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Kernel_API) - [[2.8]{.tocnumber} [Sample Code]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Sample_Code) - [[2.8.1]{.tocnumber} [Compilation]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Compilation) - [[2.8.2]{.tocnumber} [Loading Kernel Modules]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Loading_Kernel_Modules) - [[2.8.3]{.tocnumber} [Unloading Kernel Modules]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Unloading_Kernel_Modules) - [[2.8.4]{.tocnumber} [Additional Documentation]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Additional_Documentation) - [[2.9]{.tocnumber} [Mdev Mode Requirements]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Mdev_Mode_Requirements) - [[3]{.tocnumber} [SR-IOV Mode]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#SR-IOV_Mode) - [[3.1]{.tocnumber} [Knowledge Resources Used]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Knowledge_Resources_Used_3) - [[3.2]{.tocnumber} [Instruction Execution]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Instruction_Execution_2) - [[3.3]{.tocnumber} [Memory Management]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Memory_Management_2) - [[3.4]{.tocnumber} [Scheduling]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Scheduling_2) - [[3.5]{.tocnumber} [SR-IOV Mode Requirements]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#SR-IOV_Mode_Requirements) - [[4]{.tocnumber} [SIOV Mode]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#SIOV_Mode) - [[4.1]{.tocnumber} [Knowledge Resources Used]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Knowledge_Resources_Used_4) - [[4.2]{.tocnumber} [Memory Management]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Memory_Management_3) - [[4.2.1]{.tocnumber} [IOMMU Aware Mediated Device]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#IOMMU_Aware_Mediated_Device) - [[4.2.2]{.tocnumber} [Shared Hardware Workqueues]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Shared_Hardware_Workqueues) - [[4.3]{.tocnumber} [SIOV Mode Requirements]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#SIOV_Mode_Requirements) - [[5]{.tocnumber} [References (Talks & Reading Material)]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#References_(Talks_&_Reading_Material)) ::: ## [All Modes]{#All_Modes .mw-headline} This section will cover concepts which apply both to [Mdev Mode](index.php/Mediated_Device_Internals.html#Mdev_Mode){.external .text rel="nofollow"}, [SR-IOV Mode](index.php/Mediated_Device_Internals.html#SR-IOV_Mode){.external .text rel="nofollow"} & [SIOV Mode](index.php/Mediated_Device_Internals.html#SIOV_Mode){.external .text rel="nofollow"}. ### [Knowledge Resources Used]{#Knowledge_Resources_Used .mw-headline} This section is supported by significant contributions in open source by Alex Williamson. See references 2, 14, & 22 in the [References (Talks & Reading Material)](index.php/Virtual_IO_Internals.html#References_(Talks_&_Reading_Material)){.external .text rel="nofollow"} section. ### [Binding VFIO devices]{#Binding_VFIO_devices .mw-headline} ::: {.thumb .tright} ::: {.thumbinner style="width:302px;"} [![](images/thumb/9/9e/Vfio-pci_driver_bindings.png/300px-Vfio-pci_driver_bindings.png){.thumbimage decoding="async" width="300" height="218" srcset="images/thumb/9/9e/Vfio-pci_driver_bindings.png/450px-Vfio-pci_driver_bindings.png 1.5x, images/thumb/9/9e/Vfio-pci_driver_bindings.png/600px-Vfio-pci_driver_bindings.png 2x"}](index.php/File:Vfio-pci_driver_bindings.png.html){.image} ::: thumbcaption ::: magnify [](index.php/File:Vfio-pci_driver_bindings.png.html "Enlarge"){.internal} ::: **Figure 1:** VFIO group nodes are unit of ownership that VFIO uses. See slides from: [\[2\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external .text rel="nofollow"} ::: ::: ::: ::: {.thumb .tright} ::: {.thumbinner style="width:302px;"} [![](images/thumb/3/33/IOCTL_set_VFIO_container.png/300px-IOCTL_set_VFIO_container.png){.thumbimage decoding="async" width="300" height="242" srcset="images/thumb/3/33/IOCTL_set_VFIO_container.png/450px-IOCTL_set_VFIO_container.png 1.5x, images/thumb/3/33/IOCTL_set_VFIO_container.png/600px-IOCTL_set_VFIO_container.png 2x"}](index.php/File:IOCTL_set_VFIO_container.png.html){.image} ::: thumbcaption ::: magnify [](index.php/File:IOCTL_set_VFIO_container.png.html "Enlarge"){.internal} ::: **Figure 2:** IOCTL(GROUP, VFIO_GROUP_SET_CONTAINER, &CONTAINER) places the VFIO Group inside the VFIO Container. See slides from: [\[2\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external .text rel="nofollow"} ::: ::: ::: ::: {.thumb .tright} ::: {.thumbinner style="width:302px;"} [![](images/thumb/4/40/Ioctl-VFIO_IOMMU_MAP-UNMAP_DMA.png/300px-Ioctl-VFIO_IOMMU_MAP-UNMAP_DMA.png){.thumbimage decoding="async" width="300" height="214" srcset="images/thumb/4/40/Ioctl-VFIO_IOMMU_MAP-UNMAP_DMA.png/450px-Ioctl-VFIO_IOMMU_MAP-UNMAP_DMA.png 1.5x, images/thumb/4/40/Ioctl-VFIO_IOMMU_MAP-UNMAP_DMA.png/600px-Ioctl-VFIO_IOMMU_MAP-UNMAP_DMA.png 2x"}](index.php/File:Ioctl-VFIO_IOMMU_MAP-UNMAP_DMA.png.html){.image} ::: thumbcaption ::: magnify [](index.php/File:Ioctl-VFIO_IOMMU_MAP-UNMAP_DMA.png.html "Enlarge"){.internal} ::: **Figure 3:** Using interrupts IOCTL(CONTAINER, VFIO_IOMMU_MAP_DMA, &MAP), IOCTL(CONTAINER, VFIO_IOMMU_UNMAP_DMA, &UNMAP) to map memory and pin pages. See slides from: [\[2\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external .text rel="nofollow"} ::: ::: ::: ::: {.thumb .tright} ::: {.thumbinner style="width:302px;"} [![](images/thumb/3/3f/Ioctl-VFIO_GROUP_GET_FD.png/300px-Ioctl-VFIO_GROUP_GET_FD.png){.thumbimage decoding="async" width="300" height="204" srcset="images/thumb/3/3f/Ioctl-VFIO_GROUP_GET_FD.png/450px-Ioctl-VFIO_GROUP_GET_FD.png 1.5x, images/thumb/3/3f/Ioctl-VFIO_GROUP_GET_FD.png/600px-Ioctl-VFIO_GROUP_GET_FD.png 2x"}](index.php/File:Ioctl-VFIO_GROUP_GET_FD.png.html){.image} ::: thumbcaption ::: magnify [](index.php/File:Ioctl-VFIO_GROUP_GET_FD.png.html "Enlarge"){.internal} ::: **Figure 4:** Using interrupt IOCTL(GROUP2, VFIO_GROUP_GET_FD, \"0000:01:00.0\") to obtain the VFIO Group file descriptor. See slides from: [\[2\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external .text rel="nofollow"} ::: ::: ::: **Figure 1:** Binding devices to the vfio-pci driver results in VFIO group nodes. Opening the file \"/dev/vfio/vfio\" creates a VFIO Container. **Figure 2:** The interrupt routine **`IOCTL(GROUP, VFIO_GROUP_SET_CONTAINER, &CONTAINER)`** places the VFIO group inside the VFIO container. ### [Programming the IOMMU]{#Programming_the_IOMMU .mw-headline} When this has been done **`IOCTL(CONTAINER, VFIO_SET_IOMMU, VFIO_TYPE1_IOMMU)`** can then be used to set an IOMMU type for the container which places it in a user interactable state. Once this IOMMU type state has been set and the VFIO container has been made interactable additional VFIO groups may be added to the container without requiring that the group\'s IOMMU type be set again as newly added groups automatically inherit the container\'s IOMMU context. ### [VFIO Memory Mapped IO]{#VFIO_Memory_Mapped_IO .mw-headline} **Figure 3:** Once the VFIO Groups have been placed inside the VFIO container and the IOMMU type has been set the user may then map and unmap which will automatically inserts Memory Mapped IO (MMIO) entries into the IOMMU as well as pin/unpin pages as necessary. This can be accomplished using **`IOCTL(CONTAINER, VFIO_IOMMU_MAP_DMA, &MAP)`** for map/pin and **`IOCTL(CONTAINER, VFIO_IOMMU_UNMAP_DMA, &UNMAP)`** for unmap/unpin. ### [Getting the VFIO Group File Descriptor]{#Getting_the_VFIO_Group_File_Descriptor .mw-headline} **Figure 4:** Once the device has been bound to a VFIO driver, set in a VFIO container, the VFIO container has it\'s IOMMU type set, and a memory map/page pin of the VFIO device has been completed a file descriptor can then be obtained for the device. This file descriptor can be used for interrupts (ioctls), to probe for information about the BAR regions, and configure the IRQs. ### [VFIO device file descriptor]{#VFIO_device_file_descriptor .mw-headline} VFIO device file descriptors are divided into regions and each region is mapped into a device resource. Region count and info (file offset, allowable access, ect..) can be discovered through interrupt (IOCTL). Each file descriptor region corresponding to a PCI resource is represented as a file offset. In the case of RPC Mode this structure is emulated whereas in SR-IOV Mode the structure is mapped to a real PCI resource. --------------------------------------------------- 00:00.0 VGA compatible controller Region 0 Bar0 (Config Space) starts at offset 0 Region 1 Bar1 (MSI - Message Signaled Interrupts) Region 2 Bar2 (MSIX) Region 3 Bar3 Region 4 Bar4 Region 5 Bar5 (IO port space) Expansion ROM --------------------------------------------------- : BAR regions in a VGA PCI device. Below is what the file offsets looks like internally for each BAR region starting from address 0 and growing with the addition of former regions as you progress through the file. --------------------- ------------------- ------------------- ----------------------- ------ \<- File Offset -\> 0 -\> A A -\> (A+B) (A+B) -\> (A+B+C) (A+B+C) -\> (A+B+C+D) \... Region 0 (size A) Region 1 (size B) Region 2 (size C) Region 3 (size D) \... --------------------- ------------------- ------------------- ----------------------- ------ : VFIO representation of PCI BAR regions offsets. ### [VFIO Interrupts]{#VFIO_Interrupts .mw-headline} Guests communicate with the host via VFIO Interrupt Requests ([IRQs](https://infogalactic.com/info/Interrupt_request_(PC_architecture)){.external .text rel="nofollow"}). These are sent via an irqfd (IRQ [File Descriptor](https://infogalactic.com/info/File_descriptor){.external .text rel="nofollow"}). Similarly, the host receives these interrupts via [eventfd](https://man7.org/linux/man-pages/man2/eventfd.2.html){.external .text rel="nofollow"} (Event File Descriptor). The resulting data can be returned via a [callback](https://infogalactic.com/info/Callback_(computer_programming)){.external .text rel="nofollow"}. #### [IRQs]{#IRQs .mw-headline} Device properties discovered via interrupt (IOCTL). ##### [Get Device Info]{#Get_Device_Info .mw-headline} ------------------------- ------------- ---------------------------- VFIO_DEVICE_GET_INFO struct vfio_device_info argz flags VFIO_DEVICE_FLAGS_PCI VFIO_DEVICE_FLAGS_PLATFORM VFIO_DEVICE_FLAGS_RESET num_irqs num_regions ------------------------- ------------- ---------------------------- The IRQ **`VFIO_DEVICE_GET_INFO`** can provide information to distinguish between PCI and platform devices as well as the number of regions and IRQs for a particular device. The upstream API can be read **[here (vfio.h)](https://github.com/torvalds/linux/blob/47700948a4abb4a5ae13ef943ff682a7f327547a/include/uapi/linux/vfio.h#L194){.external .text rel="nofollow"}**. **Sample mdev code** to service this request can be found **[here](https://github.com/Open-IOV/VFIO-Mdev_Samples/blob/c0adb9957c0b22bfdbc18395e81dd2b476addac5/mdpy.c#L470){.external .text rel="nofollow"}**, and **[here](https://github.com/Open-IOV/VFIO-Mdev_Samples/blob/c0adb9957c0b22bfdbc18395e81dd2b476addac5/mdpy.c#L522){.external .text rel="nofollow"}**. ##### [Get Region Info]{#Get_Region_Info .mw-headline} ----------------------------- ------------ ----------------------------- VFIO_DEVICE_GET_REGION_INFO struct vfio_region_info argz cap_offset flags VFIO_REGION_INFO_FLAG_CAPS VFIO_REGION_INFO_FLAG_MMAP VFIO_REGION_INFO_FLAG_READ VFIO_REGION_INFO_FLAG_WRITE index offset size ----------------------------- ------------ ----------------------------- Once the interrupt user knows the number of regions within a VFIO device they can use IRQ **`VFIO_DEVICE_GET_REGION_INFO`** to probe each region for additional information. This interrupt will return information such as if it can be read from or written to, if the device supports MMAP, as well as what the offset and size of the region is within the VFIO file descriptor. The upstream API can be read [**here (vfio.h)**](https://github.com/torvalds/linux/blob/47700948a4abb4a5ae13ef943ff682a7f327547a/include/uapi/linux/vfio.h#L240){.external .text rel="nofollow"}. **Sample mdev code** to service this request can be found [**here**](https://github.com/Open-IOV/VFIO-Mdev_Samples/blob/c0adb9957c0b22bfdbc18395e81dd2b476addac5/mdpy.c#L425){.external .text rel="nofollow"}, and [**here**](https://github.com/Open-IOV/VFIO-Mdev_Samples/blob/c0adb9957c0b22bfdbc18395e81dd2b476addac5/mdpy.c#L545){.external .text rel="nofollow"}. ##### [Get IRQ Info]{#Get_IRQ_Info .mw-headline} -------------------------- ------- -------------------------- VFIO_DEVICE_GET_IRQ_INFO struct vfio_irq_info argz count flags VFIO_IRQ_INFO_AUTOMASKED VFIO_IRQ_INFO_EVENTFD VFIO_IRQ_INFO_MASKABLE VFIO_IRQ_INFO_NORESIZE index -------------------------- ------- -------------------------- **`VFIO_DEVICE_GET_IRQ_INFO`** is used to retrieve information about a device IRQ. **`VFIO_IRQ_INFO_AUTOMASKED`** is used to mask interrupts when they occur to protect the host. The upstream API can be read **[here (vfio.h)](https://github.com/torvalds/linux/blob/47700948a4abb4a5ae13ef943ff682a7f327547a/include/uapi/linux/vfio.h#L480){.external .text rel="nofollow"}** **Sample mdev code** to service this request can be found [**here**](https://github.com/Open-IOV/VFIO-Mdev_Samples/blob/c0adb9957c0b22bfdbc18395e81dd2b476addac5/mdpy.c#L463){.external .text rel="nofollow"}, and [**here**](https://github.com/Open-IOV/VFIO-Mdev_Samples/blob/c0adb9957c0b22bfdbc18395e81dd2b476addac5/mdpy.c#L570){.external .text rel="nofollow"}. ##### [Set IRQs]{#Set_IRQs .mw-headline} ---------------------- ---------- ----------------------------- VFIO_DEVICE_SET_IRQS struct vfio_irq_set argz count data\[\] flags VFIO_IRQ_SET_ACTION_MASK VFIO_IRQ_SET_ACTION_TRIGGER VFIO_IRQ_SET_ACTION_UNMASK VFIO_IRQ_SET_DATA_BOOL VFIO_IRQ_SET_DATA_EVENTFD VFIO_IRQ_SET_DATA_NONE index start ---------------------- ---------- ----------------------------- **`VFIO_DEVICE_SET_IRQS`** is used to setup IRQs. Actions can be configured such as trigger which is when the device triggers an interrupt (IOCTL), masking and unmasking actions can be set. Bool and None data types are used for loopback testing of the device. Start and index may be used to modify subregions. The upstream API can be read **[here (vfio.h)](https://github.com/torvalds/linux/blob/47700948a4abb4a5ae13ef943ff682a7f327547a/include/uapi/linux/vfio.h#L524){.external .text rel="nofollow"}**. ### []{#Device_Decomposure.2FRecomposure}[Device Decomposure/Recomposure]{#Device_Decomposure/Recomposure .mw-headline} ::: {.thumb .tright} ::: {.thumbinner style="width:302px;"} [![Figure 5: Device Decomposure and Recomposure via VFIO.](images/thumb/9/90/Device_Decomposure_and_Recomposure_via_VFIO.png/300px-Device_Decomposure_and_Recomposure_via_VFIO.png){.thumbimage decoding="async" width="300" height="147" srcset="images/thumb/9/90/Device_Decomposure_and_Recomposure_via_VFIO.png/450px-Device_Decomposure_and_Recomposure_via_VFIO.png 1.5x, images/thumb/9/90/Device_Decomposure_and_Recomposure_via_VFIO.png/600px-Device_Decomposure_and_Recomposure_via_VFIO.png 2x"}](index.php/File:Device_Decomposure_and_Recomposure_via_VFIO.png.html){.image} ::: thumbcaption ::: magnify [](index.php/File:Device_Decomposure_and_Recomposure_via_VFIO.png.html "Enlarge"){.internal} ::: **Figure 5:** Device Decomposure and Recomposure via VFIO. See slides from: [\[2\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external .text rel="nofollow"} ::: ::: ::: **Figure 5:** Virtual Function IO (VFIO) devices are deconstructed in userspace into a set of VFIO primitives (MMIO pages, VFIO/IOMMU Groups, VFIO IRQs, File Descriptors). Recomposure of these devices occurs upon assignment of a Virtual Function (VF) to a QEMU virtual machine. ### []{#Memory_Management_Unit_.28MMU.29}[Memory Management Unit (MMU)]{#Memory_Management_Unit_(MMU) .mw-headline} ::: {.thumb .tright} ::: {.thumbinner style="width:302px;"} ::: {.MediaTransformError style="width: 300px; height: 275px; display:inline-block;"} Error creating thumbnail: File missing ::: ::: thumbcaption ::: magnify [](index.php/File:Figure_6-_MMU_-_GMMU_IOVA_translation-isolation..png.html "Enlarge"){.internal} ::: **Figure 6:** MMU - GMMU IOVA translation/isolation. ::: ::: ::: **Figure 6:** This section will touch upon the mechanisms used for enforcement of Host Physical Address (HPA) to Guest Physical Address (GPA) isolation. #### []{#MMIO_Isolation_.28Platform_MMU.29}[MMIO Isolation (Platform MMU)]{#MMIO_Isolation_(Platform_MMU) .mw-headline} The platform\'s CPU communicates with the GPU by reading/writing to and from pinned MMIO pages in [Random Access Memory (RAM)](https://infogalactic.com/info/Random-access_memory){.external .text rel="nofollow"}. MMIO pages within the RAM are subject to IO Virtual Address (IOVA) translations by the platform\'s discrete MMU controller which is programmed by the CPU. These IOVA translations serve as a mechanism to enforce HPA to GPA isolation in the context of the platform. #### []{#VRAM_Isolation_.28GPU_GMMU.29}[VRAM Isolation (GPU GMMU)]{#VRAM_Isolation_(GPU_GMMU) .mw-headline} The GPU core performs virtualized operations by reading/writing to and from shadow page tables in onboard [Video Random Access Memory (VRAM)](https://en.wikipedia.org/wiki/Video_random_access_memory "wikipedia:Video random access memory"){.extiw}. Shadow pages within the VRAM are subject to IO Virtual Address (IOVA) translations by the GPU\'s discrete GPU MMU controller (GMMU) which is programmed by the Embedded CPU (GPU co-processor). These IOVA translations serve as a mechanism to enforce HPA to GPA isolation in the context of the virtual GPUs and multi-process isolation in single user environments. #### []{#Platform_MMIO_.3C-.3E_GPU_Shadow_Pages}[Platform MMIO \<-\> GPU Shadow Pages]{#Platform_MMIO_<->_GPU_Shadow_Pages .mw-headline} In the context of VFIO pinned MMIO pages in RAM act as an interface to communicate with VRAM shadow pages allowing GPU drivers on the platform to send instructions to the GPU. When the GPU or Platform alters memory contained in a shadow page or pinned MMIO page the change is mirrored in the corresponding IO Virtual Address (IOVA). For example if shadow page 0 is changed by the GPU this change is mirrored in MMIO page 0 on the platform (the reverse example also applies). When communications occur between the platform and GPU the information first moves through the MMU/GMMU and is then written to RAM/VRAM. ::: {.thumb .tright} ::: {.thumbinner style="width:302px;"} [![Figure 7: A depiction of region overlays within a VFIO file descriptor.](images/thumb/4/48/Figure_7-_A_depiction_of_region_overlays_within_a_VFIO_file_descriptor..png/300px-Figure_7-_A_depiction_of_region_overlays_within_a_VFIO_file_descriptor..png){.thumbimage decoding="async" width="300" height="268" srcset="images/thumb/4/48/Figure_7-_A_depiction_of_region_overlays_within_a_VFIO_file_descriptor..png/450px-Figure_7-_A_depiction_of_region_overlays_within_a_VFIO_file_descriptor..png 1.5x, images/thumb/4/48/Figure_7-_A_depiction_of_region_overlays_within_a_VFIO_file_descriptor..png/600px-Figure_7-_A_depiction_of_region_overlays_within_a_VFIO_file_descriptor..png 2x"}](index.php/File:Figure_7-_A_depiction_of_region_overlays_within_a_VFIO_file_descriptor..png.html){.image} ::: thumbcaption ::: magnify [](index.php/File:Figure_7-_A_depiction_of_region_overlays_within_a_VFIO_file_descriptor..png.html "Enlarge"){.internal} ::: **Figure 7:** A depiction of region overlays within a VFIO file descriptor. See slides from: [\[2\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external .text rel="nofollow"} ::: ::: ::: ### []{#VFIO_Quirks_.28region_traps.29}[VFIO Quirks (region traps)]{#VFIO_Quirks_(region_traps) .mw-headline} **Figure 7:** Regions of the PCI BAR require emulation (slow path) via emulated traps (known as quirks). These regions are primarily in PCI configuration space. QEMU may overlap a region overlay which when read/written to/from triggers a VM-exit to trap and emulate the region in order that appropriate translations may occur (such as those concerned with IO Virtual Address - IOVA). ::: {.thumb .tright} ::: {.thumbinner style="width:302px;"} [![Peer-to-Peer DMA Isolation under IOMMU](images/thumb/1/13/Screen_Shot_2022-05-06_at_10.46.59_AM.png/300px-Screen_Shot_2022-05-06_at_10.46.59_AM.png){.thumbimage decoding="async" width="300" height="224" srcset="images/thumb/1/13/Screen_Shot_2022-05-06_at_10.46.59_AM.png/450px-Screen_Shot_2022-05-06_at_10.46.59_AM.png 1.5x, images/1/13/Screen_Shot_2022-05-06_at_10.46.59_AM.png 2x"}](index.php/File:Screen_Shot_2022-05-06_at_10.46.59_AM.png.html){.image} ::: thumbcaption ::: magnify [](index.php/File:Screen_Shot_2022-05-06_at_10.46.59_AM.png.html "Enlarge"){.internal} ::: **Figure 8:** Peer-to-Peer DMA Isolation under IOMMU. See slides from: [\[2\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external .text rel="nofollow"} ::: ::: ::: ### [Known IOMMU Issues]{#Known_IOMMU_Issues .mw-headline} **DMA Aliasing** - Not all devices generate unique IDs. - Not all devices generate IDs they should.\ **DMA Isolation** - **Figure 8:** Peer-to-Peer DMA Isolation. In many circumstances IO Virtual Address (IOVA) translations do not occur properly in the context of DMA peering. Transactions that occur through the IOMMU are unaffected. ::: {.thumb .tright} ::: {.thumbinner style="width:302px;"} [![Figure 0: Approches to GPU Virtualization.](images/thumb/6/6a/Figure_0-_Approches_to_GPU_Virtualization..png/300px-Figure_0-_Approches_to_GPU_Virtualization..png){.thumbimage decoding="async" width="300" height="153" srcset="images/thumb/6/6a/Figure_0-_Approches_to_GPU_Virtualization..png/450px-Figure_0-_Approches_to_GPU_Virtualization..png 1.5x, images/thumb/6/6a/Figure_0-_Approches_to_GPU_Virtualization..png/600px-Figure_0-_Approches_to_GPU_Virtualization..png 2x"}](index.php/File:Figure_0-_Approches_to_GPU_Virtualization..png.html){.image} ::: thumbcaption ::: magnify [](index.php/File:Figure_0-_Approches_to_GPU_Virtualization..png.html "Enlarge"){.internal} ::: **Figure 0:** Approaches to GPU Virtualization. See slides from: [\[61\]](index.php/Virtual_I/O_Internals.html#References_(Talks_&_Reading_Material)){.external .text rel="nofollow"} ::: ::: ::: ## [Mdev Mode]{#Mdev_Mode .mw-headline} **Mdev Mode (VFIO Mediated Device)** is a method of virtualizing I/O devices enabling full API capabilities without the requirement for hardware assistance. ### [Knowledge Resources Used]{#Knowledge_Resources_Used_2 .mw-headline} This section is supported by significant contributions in open source by Neo Jia, Kirti Wankhede, Kevin Tian, Yiying Zhang, David Cowperthwaite, Kun Tian, Yaozu Dong, Tina Zhang, Gerd Hoffmann, & Zhi Wang. See references 1, 7, 8, 17, 18, 19, 70 in the [References (Talks & Reading Material)](index.php/Virtual_IO_Internals.html#References_(Talks_&_Reading_Material)){.external .text rel="nofollow"} section. ### [Mediated Core]{#Mediated_Core .mw-headline} The mediated device framework made 3 major changes to the VFIO driver. - **1: [Mediated core module (new)](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/vfio/mdev/mdev_core.c){.external .text rel="nofollow"}**\ -Mediated bus driver, create mediated device.\ -Physical device interface for vendor callbacks.\ -Generic Mediated device management user interface (sysfs). - **2: [Mediated device module (new)](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/vfio/mdev/mdev_driver.c){.external .text rel="nofollow"}**\ -Manage created mediated device, fully compatible with VFIO user API (UAPI). - **3: [VFIO IOMMU driver (enhancement)](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/vfio/vfio_iommu_type1.c){.external .text rel="nofollow"}**\ -VFIO IOMMU API Type1 compatible, easy to extend to non-Type1. The full list of these changes can be seen in the [**lists.gnu.org Qemu-devel** mailing list archive](https://lists.gnu.org/archive/html/qemu-devel/2016-10/msg03922.html){.external .text rel="nofollow"}. ### [Device Initialization]{#Device_Initialization .mw-headline} This section will deal with how an Mdev driver is initialized. **Sample mdev code** for device initialization can be found **[here](https://github.com/Open-IOV/VFIO-Mdev_Samples/blob/c0adb9957c0b22bfdbc18395e81dd2b476addac5/mdpy.c#L229){.external .text rel="nofollow"}**. #### [Registering VFIO MDEV as a driver]{#Registering_VFIO_MDEV_as_a_driver .mw-headline} VFIO Mdev must be registered as a driver. This register event occurs between the Mdev Driver Register Interface and the VFIO MDEV interface. #### [Registering PCIE Device with the Mediated Core]{#Registering_PCIE_Device_with_the_Mediated_Core .mw-headline} ::: {.thumb .tright} ::: {.thumbinner style="width:302px;"} [![Registering the mediated device.](images/thumb/7/75/Mdev-gpu.png/300px-Mdev-gpu.png){.thumbimage decoding="async" width="300" height="208" srcset="images/thumb/7/75/Mdev-gpu.png/450px-Mdev-gpu.png 1.5x, images/thumb/7/75/Mdev-gpu.png/600px-Mdev-gpu.png 2x"}](index.php/File:Mdev-gpu.png.html){.image} ::: thumbcaption ::: magnify [](index.php/File:Mdev-gpu.png.html "Enlarge"){.internal} ::: **Figure 9:** Registering the mediated device. This step may be accomplished via the vendor driver or via [GVM/Mdev-GPU](){.external .text rel="nofollow"} for drivers which do not include support for Mdev functionality and/or do not support arbitrary Mdev types. ::: ::: ::: **Figure 9:** The vendor driver must register with the Mediated Core\'s Device Register Interface. The example shown uses the [GVM Project](){.external .text rel="nofollow"}\'s [Mdev-GPU](){.external .text rel="nofollow"} component to accomplish this step on behalf of the vendor driver. #### []{#Registering_Mediated_Callbacks_.28CBs.29}[Registering Mediated Callbacks (CBs)]{#Registering_Mediated_Callbacks_(CBs) .mw-headline} **Figure 9:** The vendor driver must now register Mediated Callbacks which it expects to receive from Mdev devices. The example shown uses the [GVM Project](){.external .text rel="nofollow"}\'s [Mdev-GPU](){.external .text rel="nofollow"} component to accomplish this step on behalf of the vendor driver. #### [Creating a Mediated Device via mdev-sysfsdev API]{#Creating_a_Mediated_Device_via_mdev-sysfsdev_API .mw-headline} ::: {.thumb .tright} ::: {.thumbinner style="width:302px;"} [![](images/thumb/e/ef/Echoing_a_UUID_into_the_sysfsdev_API.png/300px-Echoing_a_UUID_into_the_sysfsdev_API.png){.thumbimage decoding="async" width="300" height="235" srcset="images/thumb/e/ef/Echoing_a_UUID_into_the_sysfsdev_API.png/450px-Echoing_a_UUID_into_the_sysfsdev_API.png 1.5x, images/thumb/e/ef/Echoing_a_UUID_into_the_sysfsdev_API.png/600px-Echoing_a_UUID_into_the_sysfsdev_API.png 2x"}](index.php/File:Echoing_a_UUID_into_the_sysfsdev_API.png.html){.image} ::: thumbcaption ::: magnify [](index.php/File:Echoing_a_UUID_into_the_sysfsdev_API.png.html "Enlarge"){.internal} ::: **Figure 10:** Echoing a UUID into the mdev-sysfsdev API. This functionality is included in [mdevctl](https://github.com/mdevctl/mdevctl){.external .text rel="nofollow"} and in [](){.external .text rel="nofollow"}. See slides from: [\[1\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external .text rel="nofollow"} ::: ::: ::: **Figure 10:** The user of an mdev capable device driver may echo values such as a UUID into the mdev-sysfsdev interface to create a unique mediated device. UUIDs must be unique per mdev device within a host. ::: {.thumb .tright} ::: {.thumbinner style="width:302px;"} [![Figure 10: QEMU creates VFIO-mdev IOMMU binding and acquires mdev file descriptor.](images/thumb/6/63/QEMU_creates_VFIO-mdev_IOMMU_binding_and_acquires_mdev_file_descriptor..png/300px-QEMU_creates_VFIO-mdev_IOMMU_binding_and_acquires_mdev_file_descriptor..png){.thumbimage decoding="async" width="300" height="231" srcset="images/thumb/6/63/QEMU_creates_VFIO-mdev_IOMMU_binding_and_acquires_mdev_file_descriptor..png/450px-QEMU_creates_VFIO-mdev_IOMMU_binding_and_acquires_mdev_file_descriptor..png 1.5x, images/thumb/6/63/QEMU_creates_VFIO-mdev_IOMMU_binding_and_acquires_mdev_file_descriptor..png/600px-QEMU_creates_VFIO-mdev_IOMMU_binding_and_acquires_mdev_file_descriptor..png 2x"}](index.php/File:QEMU_creates_VFIO-mdev_IOMMU_binding_and_acquires_mdev_file_descriptor..png.html){.image} ::: thumbcaption ::: magnify [](index.php/File:QEMU_creates_VFIO-mdev_IOMMU_binding_and_acquires_mdev_file_descriptor..png.html "Enlarge"){.internal} ::: **Figure 11:** QEMU creates VFIO-mdev IOMMU binding and acquires mdev file descriptor. See slides from: [\[1\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external .text rel="nofollow"} ::: ::: ::: #### [QEMU adds VFIO device to IOMMU container-group]{#QEMU_adds_VFIO_device_to_IOMMU_container-group .mw-headline} **Figure 11:** When starting a QEMU process with a VFIO-mdev attached QEMU calls the VFIO API to add the VFIO device to an IOMMU container/group. QEMU then runs the IOCTL to obtain a file descriptor for the device. #### [QEMU passes mdev device file descriptor to VM]{#QEMU_passes_mdev_device_file_descriptor_to_VM .mw-headline} ::: {.thumb .tright} ::: {.thumbinner style="width:302px;"} [![Figure 11: QEMU presenting the VFIO file descriptor into the virtual machine.](images/thumb/7/7c/QEMU_presenting_the_VFIO_file_descriptor_into_the_virtual_machine..png/300px-QEMU_presenting_the_VFIO_file_descriptor_into_the_virtual_machine..png){.thumbimage decoding="async" width="300" height="234" srcset="images/thumb/7/7c/QEMU_presenting_the_VFIO_file_descriptor_into_the_virtual_machine..png/450px-QEMU_presenting_the_VFIO_file_descriptor_into_the_virtual_machine..png 1.5x, images/thumb/7/7c/QEMU_presenting_the_VFIO_file_descriptor_into_the_virtual_machine..png/600px-QEMU_presenting_the_VFIO_file_descriptor_into_the_virtual_machine..png 2x"}](index.php/File:QEMU_presenting_the_VFIO_file_descriptor_into_the_virtual_machine..png.html){.image} ::: thumbcaption ::: magnify [](index.php/File:QEMU_presenting_the_VFIO_file_descriptor_into_the_virtual_machine..png.html "Enlarge"){.internal} ::: **Figure 12:** QEMU presenting the VFIO file descriptor into the virtual machine. See slides from: [\[1\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external .text rel="nofollow"} ::: ::: ::: **Figure 12:** Once QEMU has obtained the VFIO file descriptor for the Mdev device via IOCTL it is then QEMU\'s job to present the file descriptor into the virtual machine so that the mdev may be used by the guest. ### [Instruction Execution]{#Instruction_Execution .mw-headline} ::: {.thumb .tright} ::: {.thumbinner style="width:302px;"} [![](images/thumb/a/aa/Ioeventfd-and-irqfd.png/300px-Ioeventfd-and-irqfd.png){.thumbimage decoding="async" width="300" height="215" srcset="images/thumb/a/aa/Ioeventfd-and-irqfd.png/450px-Ioeventfd-and-irqfd.png 1.5x, images/thumb/a/aa/Ioeventfd-and-irqfd.png/600px-Ioeventfd-and-irqfd.png 2x"}](index.php/File:Ioeventfd-and-irqfd.png.html){.image} ::: thumbcaption ::: magnify [](index.php/File:Ioeventfd-and-irqfd.png.html "Enlarge"){.internal} ::: **Figure 13:** A simple diagram of signalling from host to guest (via irqfd) & guest to host (via ioeventfd) From [blog.allenx](http://blog.allenx.org/2015/07/05/kvm-irqfd-and-ioeventfd){.external .text rel="nofollow"} ::: ::: ::: **Figure 13:** Mdev Mode moves instruction information across a virtual function (VF) device using [Remote Procedure Calls](https://infogalactic.com/info/Remote_procedure_call){.external .text rel="nofollow"} generally by way of [software interrupt](https://infogalactic.com/info/Interrupt){.external .text rel="nofollow"} ([IOCTL](https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core.git/tree/Documentation/driver-api/ioctl.rst){.external .text rel="nofollow"}). Signals to and from the guest and the host GPU driver may be passed over file descriptors such as the [Interrupt Request File Descriptor (irqfd)](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=721eecbf4fe995ca94a9edec0c9843b1cc0eaaf3){.external .text rel="nofollow"} and [IO Event File Descriptor (ioeventfd)](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d34e6b175e61821026893ec5298cc8e7558df43a){.external .text rel="nofollow"}. The irqfd may be used to signal from the host into the guest whereas the ioeventfd may be used to signal from the guest into the host. Guest GPU instructions which would normally serialize as pRPCs (physical Remote Procedure Calls) are instead serialized from the guest as vRPCs (virtual Remote Procedure Calls) which are executed by the host mediated driver. #### [IRQ remapping]{#IRQ_remapping .mw-headline} Interrupt Requests (IRQs) must be remapped (trapped for virtualized execution) to protect the host from sensitive instructions which may affect global memory state. #### [Interrupt Injection]{#Interrupt_Injection .mw-headline} **\--** **\--Researching\--** **\--** ### [Memory Management]{#Memory_Management .mw-headline} Mdev memory management is handled by vendor driver software. #### [Region Passthrough]{#Region_Passthrough .mw-headline} Guests may be presented with emulated memory regions or via passthrough regions or a mixture of the two such as in the case of passthrough regions with BAR 0 configuration space emulation while other regions are passthrough\'d. ##### [Emulated Regions]{#Emulated_Regions .mw-headline} Emulated memory regions use indirect emulated communication requiring a VM-exit (slow). These regions are often used for virtual PCI config space such as [in this sample code](https://github.com/Open-IOV/VFIO-Mdev_Samples/blob/c0adb9957c0b22bfdbc18395e81dd2b476addac5/mdpy.c#L112){.external .text rel="nofollow"}. ##### [Passthrough Regions]{#Passthrough_Regions .mw-headline} Passthrough memory regions use direct communication requiring no VM-exit (fast). ##### [Region Access]{#Region_Access .mw-headline} ::: {.thumb .tright} ::: {.thumbinner style="width:302px;"} [![Figure 13: QEMU gets region info via VFIO UAPI from vendor driver through vfio-mdev and Mediated CBs](images/thumb/f/fa/QEMU_gets_region_info_via_VFIO_UAPI_from_vendor_driver_through_vfio-mdev_and_Mediated_CBs.png/300px-QEMU_gets_region_info_via_VFIO_UAPI_from_vendor_driver_through_vfio-mdev_and_Mediated_CBs.png){.thumbimage decoding="async" width="300" height="236" srcset="images/thumb/f/fa/QEMU_gets_region_info_via_VFIO_UAPI_from_vendor_driver_through_vfio-mdev_and_Mediated_CBs.png/450px-QEMU_gets_region_info_via_VFIO_UAPI_from_vendor_driver_through_vfio-mdev_and_Mediated_CBs.png 1.5x, images/thumb/f/fa/QEMU_gets_region_info_via_VFIO_UAPI_from_vendor_driver_through_vfio-mdev_and_Mediated_CBs.png/600px-QEMU_gets_region_info_via_VFIO_UAPI_from_vendor_driver_through_vfio-mdev_and_Mediated_CBs.png 2x"}](index.php/File:QEMU_gets_region_info_via_VFIO_UAPI_from_vendor_driver_through_vfio-mdev_and_Mediated_CBs.png.html){.image} ::: thumbcaption ::: magnify [](index.php/File:QEMU_gets_region_info_via_VFIO_UAPI_from_vendor_driver_through_vfio-mdev_and_Mediated_CBs.png.html "Enlarge"){.internal} ::: **Figure 14:** QEMU gets region info via VFIO UAPI from vendor driver through vfio-mdev and Mediated CBs. See slides from: [\[1\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external .text rel="nofollow"} ::: ::: ::: **Figure 14:** QEMU gets region information via VFIO User API (UAPI) from the vendor driver through VFIO-mdev and Mediated Callbacks. ::: {.thumb .tright} ::: {.thumbinner style="width:302px;"} ::: {.MediaTransformError style="width: 300px; height: 238px; display:inline-block;"} Error creating thumbnail: File missing ::: ::: thumbcaption ::: magnify [](index.php/File:Guest_driver_accessing_Mdev_MMIO_space_backed_by_mdev_file_descriptor_triggers_EPT_violation..png.html "Enlarge"){.internal} ::: **Figure 15:** Guest driver accessing Mdev MMIO space backed by mdev file descriptor triggers EPT violation. See slides from: [\[1\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external .text rel="nofollow"} ::: ::: ::: **Figure 15:** The guest\'s vendor driver accesses the Mdev MMIO trapped region backed by a mdev file descriptor (fd) which triggers an Extended Page Table (EPT) violation. ::: {.thumb .tright} ::: {.thumbinner style="width:302px;"} [![Figure 15: KVM services EPT violation and forwards to QEMU VFIO PCI driver.](images/thumb/4/43/KVM_services_EPT_violation_and_forwards_to_QEMU_VFIO_PCI_driver..png/300px-KVM_services_EPT_violation_and_forwards_to_QEMU_VFIO_PCI_driver..png){.thumbimage decoding="async" width="300" height="235" srcset="images/thumb/4/43/KVM_services_EPT_violation_and_forwards_to_QEMU_VFIO_PCI_driver..png/450px-KVM_services_EPT_violation_and_forwards_to_QEMU_VFIO_PCI_driver..png 1.5x, images/thumb/4/43/KVM_services_EPT_violation_and_forwards_to_QEMU_VFIO_PCI_driver..png/600px-KVM_services_EPT_violation_and_forwards_to_QEMU_VFIO_PCI_driver..png 2x"}](index.php/File:KVM_services_EPT_violation_and_forwards_to_QEMU_VFIO_PCI_driver..png.html){.image} ::: thumbcaption ::: magnify [](index.php/File:KVM_services_EPT_violation_and_forwards_to_QEMU_VFIO_PCI_driver..png.html "Enlarge"){.internal} ::: **Figure 16:** KVM services EPT violation and forwards to QEMU VFIO PCI driver. See slides from: [\[1\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external .text rel="nofollow"} ::: ::: ::: **Figure 16:** KVM services EPT violation and forwards to QEMU VFIO PCI driver. ::: {.thumb .tright} ::: {.thumbinner style="width:302px;"} [![Figure 16: QEMU convert request from KVM to Read/Write acess to Mdev file descriptor.](images/thumb/2/27/QEMU_convert_request_from_KVM_to_Read-Write_acess_to_Mdev_file_descriptor..png/300px-QEMU_convert_request_from_KVM_to_Read-Write_acess_to_Mdev_file_descriptor..png){.thumbimage decoding="async" width="300" height="235" srcset="images/thumb/2/27/QEMU_convert_request_from_KVM_to_Read-Write_acess_to_Mdev_file_descriptor..png/450px-QEMU_convert_request_from_KVM_to_Read-Write_acess_to_Mdev_file_descriptor..png 1.5x, images/thumb/2/27/QEMU_convert_request_from_KVM_to_Read-Write_acess_to_Mdev_file_descriptor..png/600px-QEMU_convert_request_from_KVM_to_Read-Write_acess_to_Mdev_file_descriptor..png 2x"}](index.php/File:QEMU_convert_request_from_KVM_to_Read-Write_acess_to_Mdev_file_descriptor..png.html){.image} ::: thumbcaption ::: magnify [](index.php/File:QEMU_convert_request_from_KVM_to_Read-Write_acess_to_Mdev_file_descriptor..png.html "Enlarge"){.internal} ::: **Figure 17:** QEMU convert request from KVM to Read/Write acess to Mdev file descriptor. See slides from: [\[1\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external .text rel="nofollow"} ::: ::: ::: **Figure 17:** QEMU convert request from KVM to R/W access to Mdev file descriptor. #### [EPT Page Violations]{#EPT_Page_Violations .mw-headline} Guest [Memory Mapped IO (MMIO)](https://infogalactic.com/info/Memory-mapped_I/O){.external .text rel="nofollow"} trips Extended Page Table (EPT) violations which are trapped by the host MMU. KVM services EPT violations and forwards to QEMU VFIO PCI driver. QEMU then converts the request from KVM to R/W access to the [Mdev File Descriptor (FD)](https://infogalactic.com/info/File_descriptor){.external .text rel="nofollow"}. Reads and writes are then handled by the host GPU device driver via mediated [callbacks (CBs)](https://infogalactic.com/info/Callback_(computer_programming)){.external .text rel="nofollow"} and [VFIO-mdev](https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core.git/tree/Documentation/driver-api/vfio-mediated-device.rst){.external .text rel="nofollow"}. #### [Mediated DMA Translations]{#Mediated_DMA_Translations .mw-headline} **\--** **\--Researching\--** **\--** Because memory is not statically allocated by the vendor driver under the mediated device framework there is no requirement to make use of traditional VFIO pinned pages (via the vfio-pci module) rather MMIO memory can be mapped at runtime incrementally. As a result non-standard mediated device vfio stub modules may be used. **Figure 18:** Memory regions get added by QEMU. ::: {.thumb .tright} ::: {.thumbinner style="width:302px;"} [![Memory Regions Added by QEMU](images/thumb/5/58/Memory_Regions_Added_by_QEMU.png/300px-Memory_Regions_Added_by_QEMU.png){.thumbimage decoding="async" width="300" height="225" srcset="images/thumb/5/58/Memory_Regions_Added_by_QEMU.png/450px-Memory_Regions_Added_by_QEMU.png 1.5x, images/thumb/5/58/Memory_Regions_Added_by_QEMU.png/600px-Memory_Regions_Added_by_QEMU.png 2x"}](index.php/File:Memory_Regions_Added_by_QEMU.png.html){.image} ::: thumbcaption ::: magnify [](index.php/File:Memory_Regions_Added_by_QEMU.png.html "Enlarge"){.internal} ::: **Figure 18:** Memory Regions Added by QEMU. See slides from: [\[1\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external .text rel="nofollow"} ::: ::: ::: **Figure 19:** QEMU calls VFIO_DMA_MAP via Memory listener. (not just guest physical memory but also device memory will be added through this memory listener) ::: {.thumb .tright} ::: {.thumbinner style="width:302px;"} [![QEMU calls VFIO DMA MAP via Memory listener](images/thumb/2/21/QEMU_calls_VFIO_DMA_MAP_via_Memory_listener.png/300px-QEMU_calls_VFIO_DMA_MAP_via_Memory_listener.png){.thumbimage decoding="async" width="300" height="237" srcset="images/thumb/2/21/QEMU_calls_VFIO_DMA_MAP_via_Memory_listener.png/450px-QEMU_calls_VFIO_DMA_MAP_via_Memory_listener.png 1.5x, images/thumb/2/21/QEMU_calls_VFIO_DMA_MAP_via_Memory_listener.png/600px-QEMU_calls_VFIO_DMA_MAP_via_Memory_listener.png 2x"}](index.php/File:QEMU_calls_VFIO_DMA_MAP_via_Memory_listener.png.html){.image} ::: thumbcaption ::: magnify [](index.php/File:QEMU_calls_VFIO_DMA_MAP_via_Memory_listener.png.html "Enlarge"){.internal} ::: **Figure 19:** QEMU calls VFIO DMA MAP via Memory listener. See slides from: [\[1\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external .text rel="nofollow"} ::: ::: ::: ::: {.thumb .tright} ::: {.thumbinner style="width:302px;"} [![Type 1 IOMMU Tracks VA-GFN](images/thumb/1/1d/Type_1_IOMMU_Tracks_VA-GFN.png/300px-Type_1_IOMMU_Tracks_VA-GFN.png){.thumbimage decoding="async" width="300" height="237" srcset="images/thumb/1/1d/Type_1_IOMMU_Tracks_VA-GFN.png/450px-Type_1_IOMMU_Tracks_VA-GFN.png 1.5x, images/thumb/1/1d/Type_1_IOMMU_Tracks_VA-GFN.png/600px-Type_1_IOMMU_Tracks_VA-GFN.png 2x"}](index.php/File:Type_1_IOMMU_Tracks_VA-GFN.png.html){.image} ::: thumbcaption ::: magnify [](index.php/File:Type_1_IOMMU_Tracks_VA-GFN.png.html "Enlarge"){.internal} ::: **Figure 20:** Type 1 IOMMU Tracks \. See slides from: [\[1\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external .text rel="nofollow"} ::: ::: ::: **Figure 20:** Type 1 IOMMU tracks \. We build a table to list the QEMU VAs and track the their mapping relation to Guest Frame Numbers (GFNs). ### [Scheduling]{#Scheduling .mw-headline} Scheduling is handled by the host mdev driver. ### [Kernel API]{#Kernel_API .mw-headline} Kernel documentation used for implementing a VFIO Mediated Device may be found at [kernel.org](https://www.kernel.org/doc/html/latest/driver-api/vfio-mediated-device.html){.external .text rel="nofollow"}. ### [Sample Code]{#Sample_Code .mw-headline} Sample code for various mdev implementations may be found below: **Mediated Virtual PCI Display Host Device:** 1. [mdpy.c](https://github.com/Open-IOV/VFIO-Mdev_Samples/blob/master/mdpy.c){.external .text rel="nofollow"} 2. [mdpy-fb.c](https://github.com/Open-IOV/VFIO-Mdev_Samples/blob/master/mdpy-fb.c){.external .text rel="nofollow"} 3. [mbochs.c](https://github.com/Open-IOV/VFIO-Mdev_Samples/blob/master/mbochs.c){.external .text rel="nofollow"}. **Serial PCI Port-based Mediated Device:** 1. [mtty.c](https://github.com/Open-IOV/VFIO-Mdev_Samples/blob/master/mtty.c){.external .text rel="nofollow"}. ###### [Compilation]{#Compilation .mw-headline} To compile the kernel modules linked above you should have your distro\'s equivalent of the build-essential package installed. The included Makefile should provide all that is necessary to successfully compile the kernel modules. You can type the following command to compile the modules from within the directory: `make` When the make operation has been completed successfully the directory will now contain .ko files. These files are the binary kernel modules. ###### [Loading Kernel Modules]{#Loading_Kernel_Modules .mw-headline} Now that you have compiled the kernel modules you may load them via the insmod command. For example to load the **mtty.ko** module run the following command from within the directory where you built the modules: `insmod mtty.ko` ###### [Unloading Kernel Modules]{#Unloading_Kernel_Modules .mw-headline} To unload any of the kernel modules you may make use of the rmmod command. For example to unload the **mtty.ko** module run the following command: `rmmod mtty.ko` ###### [Additional Documentation]{#Additional_Documentation .mw-headline} An additional guide explaining how to make use of the mtty.c sample code may be found at [line 308](https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core.git/tree/Documentation/driver-api/vfio-mediated-device.rst?h=driver-core-next&id=7de3697e9cbd4bd3d62bafa249d57990e1b8f294#n308){.external .text rel="nofollow"} of the kernel.org VFIO Mediated Device documentation. ### [Mdev Mode Requirements]{#Mdev_Mode_Requirements .mw-headline} Driver Support. Software HPA\<-\>GPA Boundary Enforcement. ## [SR-IOV Mode]{#SR-IOV_Mode .mw-headline} **SR-IOV Mode (Single Root I/O Virtualization)** involves hardware assisted virtualization on I/O peripherals. ### [Knowledge Resources Used]{#Knowledge_Resources_Used_3 .mw-headline} This section is supported by significant contributions in open source by Zheng Xiao, Jerry Jiang, & Ken Xue. See reference 6 in the [References (Talks & Reading Material)](index.php/Virtual_IO_Internals.html#References_(Talks_&_Reading_Material)){.external .text rel="nofollow"} section. ### [Instruction Execution]{#Instruction_Execution_2 .mw-headline} SR-IOV communicates instructions from a virtual function (VF) directly to the [PCI BAR](https://infogalactic.com/info/PCI_configuration_space){.external .text rel="nofollow"}. ### [Memory Management]{#Memory_Management_2 .mw-headline} Guests are presenting with passthrough memory regions by the device firmware. ### [Scheduling]{#Scheduling_2 .mw-headline} Scheduling may be handled by the host mdev driver and/or the device firmware. ### [SR-IOV Mode Requirements]{#SR-IOV_Mode_Requirements .mw-headline} Driver Support. Device SR-IOV support. Firmware HPA\<-\>GPA Boundary Enforcement. ## [SIOV Mode]{#SIOV_Mode .mw-headline} **SIOV (Scalable I/O Virtualization)** involves the combination of concepts form both [SR-IOV Mode](index.php/Mediated_Device_Internals.html#SR-IOV_Mode){.external .text rel="nofollow"} and [Mdev Mode](index.php/Mediated_Device_Internals.html#Mdev_Mode){.external .text rel="nofollow"} as well as novel concepts like shared IOMMU aware buffers and offloading of PCI config space VM-exits (slow path) to a discrete controller. **Revision 1.0 of the SIOV specification** can be read on the [Open Compute Project website](https://www.opencompute.org/documents/ocp-scalable-io-virtualization-technical-specification-revision-1-v1-2-pdf){.external .text rel="nofollow"}. ### [Knowledge Resources Used]{#Knowledge_Resources_Used_4 .mw-headline} This section is supported by significant contributions in open source by Kevin Tian, Tina Zhang, Xin Zeng, Yi Liu. See references 3, 4, 5, 16, 17, 18, 19, 20, 21 in the [References (Talks & Reading Material)](index.php/Virtual_IO_Internals.html#References_(Talks_&_Reading_Material)){.external .text rel="nofollow"} section. ### [Memory Management]{#Memory_Management_3 .mw-headline} Enhancements to [Intel VT-d](https://edc.intel.com/content/www/us/en/design/ipla/software-development-platforms/client/platforms/alder-lake-desktop/12th-generation-intel-core-processors-datasheet-volume-1-of-2/002/intel-virtualization-technology-for-directed-i-o/){.external .text rel="nofollow"} introduce new \'Scalable Mode\' to allow the platform to assign more granular IOMMU allocations to mediated devices ([mdev memory management](index.php/Mediated_Device_Internals.html#Memory_Management){.external .text rel="nofollow"}). This change is referred to as an IOMMU Aware Mediated Device. #### [IOMMU Aware Mediated Device]{#IOMMU_Aware_Mediated_Device .mw-headline} SIOV made several changes to the VFIO driver, Intel IOMMU, and Mediated Device Framework. The full list of these changes can be seen in the [mailing list archive on **lwn.net** under **(vfio/mdev: IOMMU aware mediated device)**](https://lwn.net/ml/linux-kernel/20190222021927.13132-1-baolu.lu@linux.intel.com/){.external .text rel="nofollow"}. #### [Shared Hardware Workqueues]{#Shared_Hardware_Workqueues .mw-headline} SIOV makes use of Shared Hardware Workqueues which may be accessed by processes or Virtual Machines. [According to **kernel.org**](https://www.kernel.org/doc/html/latest/x86/sva.html){.external .text rel="nofollow"}: *\"In order to allow the hardware to distinguish the context for which work is being executed in the hardware by SWQ interface, SIOV uses Process Address Space ID (PASID), which is a 20-bit number defined by the PCIe SIG. PASID value is encoded in all transactions from the device. This allows the IOMMU to track I/O on a per-PASID granularity in addition to using the PCIe Resource Identifier (RID) which is the Bus/Device/Function.\"* ### [SIOV Mode Requirements]{#SIOV_Mode_Requirements .mw-headline} Driver Support. Device SIOV support. Firmware HPA\<-\>GPA Boundary Enforcement. ## []{#References_.28Talks_.26_Reading_Material.29}[References (Talks & Reading Material)]{#References_(Talks_&_Reading_Material) .mw-headline} 1. [\[2016\] vGPU on KVM - A VFIO Based Framework by Neo Jia & Kirti Wankhede](https://www.youtube.com/watch?v=Xs0TJU_sIPc){.external .text rel="nofollow"} - [slides](https://www.linux-kvm.org/images/5/59/02x03-Neo_Jia_and_Kirti_Wankhede-vGPU_on_KVM-A_VFIO_based_Framework.pdf){.external .text rel="nofollow"} 2. [\[2016\] An Introduction to PCI Device Assignment with VFIO by Alex Williamson](https://www.youtube.com/watch?v=WFkdTFTOTpA){.external .text rel="nofollow"} - [slides](http://events17.linuxfoundation.org/sites/events/files/slides/An%20Introduction%20to%20PCI%20Device%20Assignment%20with%20VFIO%20-%20Williamson%20-%202016-08-30_0.pdf){.external .text rel="nofollow"} 3. [\[2017\] Scalable I/O Virtualization by Kevin Tian](https://events19.linuxfoundation.cn/wp-content/uploads/2017/11/Intel%C2%AE-Scalable-I_O-Virtualization_Kevin-Tian.pdf){.external .text rel="nofollow"} 4. [\[2019\] Bring a Scalable IOV Capable Device into Linux World by Xin Zeng & Yi Liu](https://www.youtube.com/watch?v=G6D-jaCs6sc){.external .text rel="nofollow"} - [slides](https://static.sched.com/hosted_files/kvmforum2019/5e/Bring%20a%20scalable%20IOV%20capable%20device%20into%20Linux%20-%20KVM%202019.pdf){.external .text rel="nofollow"} 5. [Scalable I/O Virtualization Revision 1.0](https://www.opencompute.org/documents/ocp-scalable-io-virtualization-technical-specification-revision-1-v1-2-pdf){.external .text rel="nofollow"} 6. [\[2018\] Live Migration Support for GPU with SRIOV by Zheng Xiao, Jerry Jiang & Ken Xue](https://www.youtube.com/watch?v=_tB3EbFDcRQ){.external .text rel="nofollow"} - [slides](https://events19.linuxfoundation.org/wp-content/uploads/2017/12/Live-Migration-Support-for-GPU-with-SRIOV-Challenges-and-Solution-Zheng-Xiao-Alibaba-Cloud-Jerry-Jiang-Ken-Xue-AMD.pdf){.external .text rel="nofollow"} 7. [\[2017\] Intel GVT-g: From Production to Upstream - Zhi Wang, Intel](https://www.youtube.com/watch?v=UODxW1opfn0){.external .text rel="nofollow"} 8. [\[2016\] Qemu Graphics Update 2016 by Gerd Hoffmann](https://www.youtube.com/watch?v=DKYvQ3FdFeo){.external .text rel="nofollow"} 9. [IOCTL](https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core.git/tree/Documentation/driver-api/ioctl.rst){.external .text rel="nofollow"} 10. [eventfd](https://man7.org/linux/man-pages/man2/eventfd.2.html){.external .text rel="nofollow"} - [root/virt/kvm/eventfd.c](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/virt/kvm/eventfd.c){.external .text rel="nofollow"} 11. (kernel diff:: [KVM: irqfd](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=721eecbf4fe995ca94a9edec0c9843b1cc0eaaf3){.external .text rel="nofollow"}) 12. (kernel diff:: [KVM: add ioeventfd support](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d34e6b175e61821026893ec5298cc8e7558df43a){.external .text rel="nofollow"}) 13. [KVM irqfd & ioeventfd](https://web.archive.org/web/20220120223711/http://blog.allenx.org/2015/07/05/kvm-irqfd-and-ioeventfd){.external .text rel="nofollow"} 14. [VFIO - Virtual Function I/O](https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core.git/tree/Documentation/driver-api/vfio.rst){.external .text rel="nofollow"} - [root/virt/kvm/vfio.c](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/virt/kvm/vfio.c){.external .text rel="nofollow"} 15. [VFIO Mediated Devices](https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core.git/tree/Documentation/driver-api/vfio-mediated-device.rst){.external .text rel="nofollow"} 16. [IOMMU Aware Mediated Device](https://lwn.net/ml/linux-kernel/20190222021927.13132-1-baolu.lu@linux.intel.com/){.external .text rel="nofollow"} 17. [Hardware-Assisted Mediated Pass-Through with VFIO by Kevin Tian](https://www.youtube.com/watch?v=95KSKrZM8oQ){.external .text rel="nofollow"} 18. [\[2017\] Generic Buffer Sharing Mechanism for Mediated Devices by Tina Zhang](https://www.youtube.com/watch?v=cHMLBcHplhk){.external .text rel="nofollow"} 19. [\[2019\] Toward a Virtualization World Built on Mediated Pass-Through - Kevin Tian](https://www.youtube.com/watch?v=KWRKx_uxUDI){.external .text rel="nofollow"} 20. [Auxiliary Bus](https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core.git/tree/Documentation/driver-api/auxiliary_bus.rst?h=driver-core-next&id=7de3697e9cbd4bd3d62bafa249d57990e1b8f294){.external .text rel="nofollow"} 21. [Shared Virtual Addressing](https://www.kernel.org/doc/html/latest/x86/sva.html){.external .text rel="nofollow"} 22. [vfio.blogspot.com](https://vfio.blogspot.com/){.external .text rel="nofollow"} 23. [XenGT by Kevin Tian](https://01.org/sites/default/files/xengt.pdf){.external .text rel="nofollow"} 24. [VMWare Academic Publications: GPU Virtualization by Micah Dowty & Jeremy Sugerman](https://web.archive.org/web/20130721094438/http://labs.vmware.com/academic/publications/gpu-virtualization){.external .text rel="nofollow"} 25. [PCI SIG I/O Virtualization](https://pcisig.com/specifications/iov){.external .text rel="nofollow"} 26. [VMGL by the University of Toronto Computer Science Department](https://web.archive.org/web/20120108034526/http://sysweb.cs.toronto.edu/vmgl){.external .text rel="nofollow"} 27. [Kepler VGX Hypervisor](https://web.archive.org/web/20120518125815/http://www.nvidia.com/object/vgx-hypervisor.html){.external .text rel="nofollow"} 28. [Intel Virtualization Technology for Directed I/O (VT-d): Enhancing Intel platforms for efficient virtualization of I/O devices](https://web.archive.org/web/20090218010221/http://software.intel.com/en-us/articles/intel-virtualization-technology-for-directed-io-vt-d-enhancing-intel-platforms-for-efficient-virtualization-of-io-devices){.external .text rel="nofollow"} 29. [DMA-Mapping.txt (Dynamic DMA Mapping)](https://lwn.net/2001/0712/a/dma-interface.php3){.external .text rel="nofollow"} 30. [KVM MMU Virtualization by Xiao Guangrong](https://events.static.linuxfound.org/slides/2011/linuxcon-japan/lcj2011_guangrong.pdf){.external .text rel="nofollow"} 31. [OSDEV: PCI](https://wiki.osdev.org/PCI){.external .text rel="nofollow"} 32. [Down to the TLP: How PCI express devices talk (Part I)](http://xillybus.com/tutorials/pci-express-tlp-pcie-primer-tutorial-guide-1){.external .text rel="nofollow"} 33. [Down to the TLP: How PCI express devices talk (Part II)](http://xillybus.com/tutorials/pci-express-tlp-pcie-primer-tutorial-guide-2){.external .text rel="nofollow"} 34. [Intel ACRN: Enabling SR-IOV Virtualization](https://projectacrn.github.io/latest/tutorials/sriov_virtualization.html){.external .text rel="nofollow"} 35. [Kernel.org sysfs-bus-pci](https://docs.kernel.org/admin-guide/abi-testing.html#abi-file-testing-sysfs-bus-pci){.external .text rel="nofollow"} 36. [Linux Kernel Labs: Memory Mapping](https://linux-kernel-labs.github.io/refs/heads/master/labs/memory_mapping.html){.external .text rel="nofollow"} 37. [Inside Windows Page Frame Number (PFN) - Part 1](https://rayanfam.com/topics/inside-windows-page-frame-number-part1/){.external .text rel="nofollow"} 38. [Direct Rendering Infrastructure (DRI)](https://en.wikipedia.org/wiki/Direct_Rendering_Infrastructure){.external .text rel="nofollow"} 39. [DRI User Guide](https://dri.sourceforge.net/doc/DRIuserguide.html){.external .text rel="nofollow"} 40. [Linux Kernel Labs: IO Virtualization](https://linux-kernel-labs.github.io/refs/heads/master/lectures/virt.html#i-o-virtualization){.external .text rel="nofollow"} 41. [Gallium3D: Main Page](https://dri.freedesktop.org/doxygen/gallium/){.external .text rel="nofollow"} 42. [Gallium3D Wiki](https://web.archive.org/web/20080107043445/http://www.tungstengraphics.com/wiki/index.php/Gallium3D){.external .text rel="nofollow"} 43. [Gallium3D talk from XDS 2007](https://web.archive.org/web/20090219182518/http://www.tungstengraphics.com/wiki/files/gallium3d-xds2007.pdf){.external .text rel="nofollow"} 44. [Memory Management High Level Design (ARCN)](https://projectacrn.github.io/latest/developer-guides/hld/hv-memmgt.html){.external .text rel="nofollow"} 45. [Block Architecture Diagrams for Geforce series](https://web.archive.org/web/20080111122628/http://www.digit-life.com/articles2/gffx/nv40-part1-a.html){.external .text rel="nofollow"} 46. [Block diagrams for 40 series instruction format](http://freenv.svn.sourceforge.net/viewvc/freenv/doc/shaderinsnformat/bitdiagen/){.external .text rel="nofollow"} 47. [PCI Configuration Space](https://en.wikipedia.org/wiki/PCI_configuration_space "wikipedia:PCI configuration space"){.extiw} 48. [GPU virtual memory in WDDM 2.0](https://learn.microsoft.com/en-us/windows-hardware/drivers/display/gpu-virtual-memory-in-wddm-2-0){.external .text rel="nofollow"} 49. [Intel Graphics Programmer\'s Reference Manuals (PRM)](https://01.org/linuxgraphics/documentation/hardware-specification-prms){.external .text rel="nofollow"} 50. [i915: Hardware Contexts (and some bits about batchbuffers)](https://bwidawsk.net/blog/2013/1/i915-hardware-contexts-and-some-bits-about-batchbuffers/){.external .text rel="nofollow"} 51. [i915: The Global GTT Part 1](https://bwidawsk.net/blog/2014/6/the-global-gtt-part-1/){.external .text rel="nofollow"} 52. [i915: Aliasing PPGTT Part 2](https://bwidawsk.net/blog/2014/6/aliasing-ppgtt-part-2/){.external .text rel="nofollow"} 53. [i915: True PPGTT Part 3](https://bwidawsk.net/blog/2014/7/true-ppgtt-part-3/){.external .text rel="nofollow"} 54. [i915: Future PPGTT Part 4 (Dynamic page table allocations, 64 bit address space, GPU \"mirroring\", and yeah, something about relocs too)](https://bwidawsk.net/blog/2014/7/future-ppgtt-part-4-dynamic-page-table-allocations-64-bit-address-space-gpu-mirroring-and-yeah-something-about-relocs-too/){.external .text rel="nofollow"} 55. [i915: Security of the Intel Graphics Stack - Part 1 - Introduction](https://igor-blue.github.io/2021/02/10/graphics-part1.html){.external .text rel="nofollow"} 56. [i915: Security of the Intel Graphics Stack - Part 2 - FW \<-\> GuC](https://igor-blue.github.io/2021/02/24/graphics-part2.html){.external .text rel="nofollow"} 57. [A deep dive into QEMU: PCI slave devices](https://airbus-seclab.github.io/qemu_blog/pci_slave.html){.external .text rel="nofollow"} 58. [Debugging QEMU Guests with GDB (start & stop, examine state like registers & memory, set breakpoints & watchpoints)](https://qemu-project.gitlab.io/qemu/system/gdb.html){.external .text rel="nofollow"} 59. [IOMMU Introduction](https://terenceli.github.io/%E6%8A%80%E6%9C%AF/2019/08/04/iommu-introduction){.external .text rel="nofollow"} 60. [Arch Wiki: Intel Graphics](https://wiki.archlinux.org/title/intel_graphics){.external .text rel="nofollow"} 61. [An Introduction to Intel Graphics Virtualization Technology (legacy GVT-g) by Zhi Wang](https://01.org/sites/default/files/documentation/an_introduction_to_intel_gvt-g_for_external.pdf){.external .text rel="nofollow"} 62. [Kernel.org: vfio iommu type1: Add support for mediated devices](https://patchwork.kernel.org/project/qemu-devel/patch/1478293856-8191-11-git-send-email-kwankhede@nvidia.com/){.external .text rel="nofollow"} 63. [Hardware and Compute Abstraction Layers For Accelerated Computing Using Graphics Hardware and Conventional CPUs](https://archive.ll.mit.edu/HPEC/agendas/proc07/Day3/10_Hensley_Abstract.pdf){.external .text rel="nofollow"} 64. [More Shadow Walker: The Progression of TLB-Splitting on x86](https://www.blackhat.com/docs/us-14/materials/us-14-Torrey-MoRE-Shadow-Walker-The-Progression-Of-TLB-Splitting-On-x86.pdf){.external .text rel="nofollow"} 65. [MMU Virtualization Via Intel EPT: Technical Details](https://revers.engineering/mmu-ept-technical-details/){.external .text rel="nofollow"} 66. [Alex Williamson Github: VFIO Development (tree next)](https://github.com/awilliam/linux-vfio/tree/next){.external .text rel="nofollow"} 67. [GPU PMU: performance monitoring with perf event](https://events.static.linuxfound.org/slides/2011/linuxcon-japan/lcj2011_linming.pdf){.external .text rel="nofollow"} 68. [Remote Procedure Call (RPC)](https://book.systemsapproach.org/e2e/rpc.html){.external .text rel="nofollow"} 69. [Virtual Open Systems: API Remoting](http://www.virtualopensystems.com/en/products/api-remoting/){.external .text rel="nofollow"} 70. [A Full GPU Virtualization Solution with Mediated Pass-Through by Yiying Zhang, David Cowperthwaite, Kun Tian, and Yaozu Dong](https://www.usenix.org/conference/atc14/technical-sessions/presentation/tian){.external .text rel="nofollow"} - [\[Paper\]](https://www.usenix.org/system/files/conference/atc14/atc14-paper-tian.pdf){.external .text rel="nofollow"} - [\[Video\]](https://www.youtube.com/watch?v=vvYmDQKZ6MQ){.external .text rel="nofollow"} - [\[Slides 1\]](https://www.usenix.org/sites/default/files/conference/protected-files/atc14_slides_tian.pdf){.external .text rel="nofollow"} - [\[Slides 2\]](https://cseweb.ucsd.edu/~yiying/cse291j-winter20/reading/GPU-Virtualization.pdf){.external .text rel="nofollow"} 71. [\[2014\] KvmGT: A Full GPU Virtualization Solution by Jike Song](https://www.youtube.com/watch?v=-iuIu7_GuEo){.external .text rel="nofollow"} 72. [docs.kernel.org: DRM Memory Management](https://docs.kernel.org/gpu/drm-mm.html){.external .text rel="nofollow"} 73. [docs.kernel.org: PCI Bus Subsystem](https://docs.kernel.org/PCI/index.html){.external .text rel="nofollow"} 74. [openglbook.com: What is OpenGL?](https://openglbook.com/chapter-0-preface-what-is-opengl.html){.external .text rel="nofollow"} 75. [\[2020\] Getting pixels on screen: introduction to Kernel Mode Setting (KMS) by Simon Ser](https://yewtu.be/watch?v=haes4_Xnc5Q){.external .text rel="nofollow"} - [slides](https://fs.emersion.fr/protected/presentations/present.html?src=kms-foss-north/index.md#1){.external .text rel="nofollow"} 76. [Dynamic Mediation for Live Migration VFIO-PCI](https://sdic.sjtu.edu.cn/wp-content/uploads/2019/12/Mediated-Pass-Through-MPT-NVIDIA-vGPU.pptx){.external .text rel="nofollow"} 77. [DRM Management Via CGroups](https://lwn.net/Articles/746127/){.external .text rel="nofollow"} 78. [DRM: Add GEM (\"graphics execution manager\") to i915 driver.](https://lwn.net/Articles/292583/){.external .text rel="nofollow"} ::: ::: printfooter Retrieved from \"[https://open-iov.org/index.php?title=Virtual_I/O_Internals&oldid=23792](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html){dir="ltr"}\" ::: ::: ::: {#catlinks .catlinks .catlinks-allhidden mw="interface"} ::: ::: ::: ::: {#mw-navigation} ## Navigation menu ::: {#mw-head} ### Personal tools {#p-personal-label .vector-menu-heading} ::: vector-menu-content - [[Log in](https://open-iov.org/index.php?title=Special:UserLogin&returnto=Virtual+I%2FO+Internals&returntoquery=oldid%3D23792 "You are encouraged to log in; however, it is not mandatory [o]"){accesskey="o"}]{#pt-login} ::: ::: {#left-navigation} ### Namespaces {#p-namespaces-label .vector-menu-heading} ::: vector-menu-content - [[Page](index.php/Virtual_I/O_Internals.html "View the content page [c]"){accesskey="c"}]{#ca-nstab-main} - [[Discussion](https://open-iov.org/index.php?title=Talk:Virtual_I/O_Internals&action=edit&redlink=1 "Discussion about the content page (page does not exist) [t]"){rel="discussion" accesskey="t"}]{#ca-talk} ::: ### Variants [expanded]{.vector-menu-checkbox-expanded} [collapsed]{.vector-menu-checkbox-collapsed} {#p-variants-label .vector-menu-heading} ::: vector-menu-content ::: ::: ::: {#right-navigation} ### Views {#p-views-label .vector-menu-heading} ::: vector-menu-content - [[Read](index.php/Virtual_I/O_Internals.html)]{#ca-view} - [[View source](index.php%3Ftitle=Virtual_I%252FO_Internals&action=edit.html "This page is protected. You can view its source [e]"){accesskey="e"}]{#ca-viewsource} - [[View history](index.php%3Ftitle=Virtual_I%252FO_Internals&action=history.html "Past revisions of this page [h]"){accesskey="h"}]{#ca-history} ::: ### More [expanded]{.vector-menu-checkbox-expanded} [collapsed]{.vector-menu-checkbox-collapsed} {#p-cactions-label .vector-menu-heading} ::: vector-menu-content ::: ::: {#p-search .vector-search-box role="search"}
### Search ::: {#simpleSearch search-loc="header-navigation"} :::
::: ::: ::: ::: {#mw-panel} ::: {#p-logo role="banner"} [](images/archive/c/c3/20221214193624!I915_Scheduling_Events_and_Requests.png.html "Visit the main page"){.mw-wiki-logo} ::: ### Navigation {#p-navigation-label .vector-menu-heading} ::: vector-menu-content - [[Main page](images/archive/c/c3/20221214193624!I915_Scheduling_Events_and_Requests.png.html "Visit the main page [z]"){accesskey="z"}]{#n-mainpage-description} - [[Recent changes](https://open-iov.org/index.php/Special:RecentChanges "A list of recent changes in the wiki [r]"){accesskey="r"}]{#n-recentchanges} - [[Random page](index.php/Special:Random.html "Load a random page [x]"){accesskey="x"}]{#n-randompage} - [[Help about MediaWiki](https://www.mediawiki.org/wiki/Special:MyLanguage/Help:Contents)]{#n-help-mediawiki} ::: ### Tools {#p-tb-label .vector-menu-heading} ::: vector-menu-content - [[What links here](index.php/Special:WhatLinksHere/Virtual_I/O_Internals.html "A list of all wiki pages that link here [j]"){accesskey="j"}]{#t-whatlinkshere} - [[Related changes](https://open-iov.org/index.php/Special:RecentChangesLinked/Virtual_I/O_Internals "Recent changes in pages linked from this page [k]"){rel="nofollow" accesskey="k"}]{#t-recentchangeslinked} - [[Special pages](index.php/Special:SpecialPages.html "A list of all special pages [q]"){accesskey="q"}]{#t-specialpages} - [[Printable version](javascript:print(); "Printable version of this page [p]"){rel="alternate" accesskey="p"}]{#t-print} - [[Permanent link](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html "Permanent link to this revision of the page")]{#t-permalink} - [[Page information](index.php%3Ftitle=Virtual_I%252FO_Internals&action=info.html "More information about this page")]{#t-info} ::: ::: ::: - [This page was last edited on 19 June 2023, at 17:54.]{#footer-info-lastmod} ```{=html} ``` - [[Privacy policy](https://open-iov.org/index.php/Open-IOV:Privacy_policy "Open-IOV:Privacy policy")]{#footer-places-privacy} - [[About Open-IOV](index.php/Open-IOV:About.html "Open-IOV:About")]{#footer-places-about} - [[Disclaimers](https://open-iov.org/index.php/Open-IOV:General_disclaimer "Open-IOV:General disclaimer")]{#footer-places-disclaimer} ```{=html} ``` - [[![Powered by MediaWiki](resources/assets/poweredby_mediawiki_88x31.png){srcset="resources/assets/poweredby_mediawiki_132x47.png 1.5x, resources/assets/poweredby_mediawiki_176x62.png 2x" width="88" height="31" loading="lazy"}](https://www.mediawiki.org/)]{#footer-poweredbyico}