diff options
| author | Open-IOV <dev@open-iov.org> | 2026-09-23 05:44:08 -0400 |
|---|---|---|
| committer | Open-IOV <dev@open-iov.org> | 2026-09-23 05:46:43 -0400 |
| commit | 68cdd7728a7cc6715bbbff7decfe2b11c3bdf558 (patch) | |
| tree | 3dc8ce2e6af313a97ec6852f5c663a9a93665d2e /wiki/Virtual_IO_Internals.md | |
| download | open-iov-68cdd7728a7cc6715bbbff7decfe2b11c3bdf558.tar.gz open-iov-68cdd7728a7cc6715bbbff7decfe2b11c3bdf558.zip | |
Open-IOV: initial public release
Open-IOV virtualizes NVIDIA GPUs so that unmodified guest drivers run with
real GPU functionality - CUDA first, 3D and video to follow - on unmodified
hosts, without a proprietary vGPU license and without patching any driver,
firmware or hypervisor.
Core principle: emulate the interface, pass through the work.
kernel/ open-iov-vfio.ko - host VFIO mdev driver: virtual GPU device
(PCI config space, BAR shadows, MSI-X), GSP boot emulation,
GSP-RM message-queue/RPC emulation, memory model (GPU-VA to
guest-PA discovery) and the semaphore-completion service that
advances guest UVM progress.
userspace/ libiov.so management API and examples.
tools/ guest diagnostics, firmware analysis wrappers, RM probes.
scripts/ stack manager, VM harness, build/load/test automation.
config/ hardware/version profiles (the portability surface).
docs/ documentation tree - start at docs/README.md.
wiki/ long-form firmware and protocol deep dives.
Status: initialization works end to end for a stock 570.144 Linux guest on
Turing (TU104): driver probe, nvidia-smi, the full GSP-RM control battery,
object creation and the early CUDA/UVM start-up path. GPU work forwarding -
the step that makes CUDA kernels execute on real hardware - is the active
work item; see docs/04-implementation/01-known-knowns-and-unknowns.md.
License: BSD-3-Clause for the project; GPL-2.0 for kernel modules that link
GPL-only kernel interfaces. See LICENSE and docs/09-reference/legal-and-licensing.md.
Diffstat (limited to 'wiki/Virtual_IO_Internals.md')
| -rw-r--r-- | wiki/Virtual_IO_Internals.md | 1736 |
1 files changed, 1736 insertions, 0 deletions
diff --git a/wiki/Virtual_IO_Internals.md b/wiki/Virtual_IO_Internals.md new file mode 100644 index 0000000..d97344e --- /dev/null +++ b/wiki/Virtual_IO_Internals.md @@ -0,0 +1,1736 @@ +::: {#mw-page-base .noprint} +::: + +::: {#mw-head-base .noprint} +::: + +::: {#content .mw-body role="main"} +[]{#top} + +::: {#siteNotice} +::: + +::: mw-indicators +::: + +# Virtual I/O Internals {#firstHeading .firstHeading} + +::: {#bodyContent .vector-body} +::: {#siteSub .noprint} +From Open-IOV +::: + +::: {#contentSub} +::: {.mw-revision .warningbox} +::: {#mw-revision-info} +Revision as of 17:54, 19 June 2023 by +[Arthur](./index.php%3Ftitle=User:Arthur&action=edit&redlink=1.html "User:Arthur (page does not exist)"){.new +.mw-userlink} +[([talk](./index.php%3Ftitle=User_talk:Arthur&action=edit&redlink=1.html "User talk:Arthur (page does not exist)"){.new +.mw-usertoollinks-talk} \| +[contribs](index.php/Special:Contributions/Arthur.html "Special:Contributions/Arthur"){.mw-usertoollinks-contribs})]{.mw-usertoollinks} +[([[[→References (Talks & Reading +Material)](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#References_(Talks_&_Reading_Material))]{.autocomment}]{dir="auto"})]{.comment} +::: + +::: {#mw-revision-nav} +([diff](https://open-iov.org/index.php?title=Virtual_I/O_Internals&diff=prev&oldid=23792 "Virtual I/O Internals")) +[← Older +revision](https://open-iov.org/index.php?title=Virtual_I/O_Internals&direction=prev&oldid=23792 "Virtual I/O Internals") +\| Latest revision (diff) \| Newer revision → (diff) +::: +::: +::: + +::: {#contentSub2} +::: + +::: {#jump-to-nav} +::: + +[Jump to +navigation](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#mw-head){.mw-jump-link} +[Jump to +search](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#searchInput){.mw-jump-link} + +::: {#mw-content-text .mw-body-content .mw-content-ltr lang="en" dir="ltr"} +::: mw-parser-output +The following document details the internals of a **VFIO (Virtual +Function I/O)** driven **Shared** **I/O Device.** + +> An absence of critical technical documentation has historically slowed +> growth and adoption of developer ecosystems for GPU virtualization. +> +> This +> [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/){.external +> .text rel="nofollow"} licensed content can either be used with +> attribution, or used as inspiration for new documentation, created by +> GPU vendors for public commercial distribution as developer +> documentation. +> +> Where possible, this documentation will clearly label dates and +> versions of observed-but-not-guaranteed behaviour vs. +> vendor-documented stable interfaces/behaviour with guarantees of +> forward or backward compatibility. + +This article places emphasis on the **Virtual GPU (vGPU)** use case +however these concepts apply generically to virtualization of I/O +devices (TPUs, NICs, storage peripherals, ect..). + + -------------------------------------------------------------------------------------- ------------------------------------------------------------------------- --------------------------------------------- + Mdev Mode SR-IOV Mode SIOV Mode + No hardware assistance needed. Hardware assistance needed. Hardware assistance needed. + Host requires insight about guest workload. Host ignorance of guest workload. Host requires insight about guest workload. + Error reporting. No guest driver error reporting. Error reporting. + In depth dynamic monitoring. Basic dynamic monitoring. In depth dynamic monitoring. + Software defined MMU guest separation. Firmware defined MMU guest separation. Firmware defined MMU guest separation. + Requires deferred instructions to be supported by host software (support libraries). Guest is ignorant of host supported software such as support libraries. + Routing interrupts. Routing interrupts. Routing interrupts. + Device reset. Device reset. Device reset. + Enable/Disable device. Enable/Disable device. Enable/Disable device. + Support for multiple scheduling techniques. Support for multiple scheduling techniques. Support for multiple scheduling techniques. + Single PCI requester ID. Multiple PCI requester IDs. Multiple PCI requester IDs. + -------------------------------------------------------------------------------------- ------------------------------------------------------------------------- --------------------------------------------- + + : Comparison of Assistance Modes + +::: {#toc .toc role="navigation" aria-labelledby="mw-toc-heading"} +::: {.toctitle lang="en" dir="ltr"} +## Contents {#mw-toc-heading} + +[]{.toctogglespan} +::: + +- [[1]{.tocnumber} [All + Modes]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#All_Modes) + - [[1.1]{.tocnumber} [Knowledge Resources + Used]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Knowledge_Resources_Used) + - [[1.2]{.tocnumber} [Binding VFIO + devices]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Binding_VFIO_devices) + - [[1.3]{.tocnumber} [Programming the + IOMMU]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Programming_the_IOMMU) + - [[1.4]{.tocnumber} [VFIO Memory Mapped + IO]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#VFIO_Memory_Mapped_IO) + - [[1.5]{.tocnumber} [Getting the VFIO Group File + Descriptor]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Getting_the_VFIO_Group_File_Descriptor) + - [[1.6]{.tocnumber} [VFIO device file + descriptor]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#VFIO_device_file_descriptor) + - [[1.7]{.tocnumber} [VFIO + Interrupts]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#VFIO_Interrupts) + - [[1.7.1]{.tocnumber} + [IRQs]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#IRQs) + - [[1.7.1.1]{.tocnumber} [Get Device + Info]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Get_Device_Info) + - [[1.7.1.2]{.tocnumber} [Get Region + Info]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Get_Region_Info) + - [[1.7.1.3]{.tocnumber} [Get IRQ + Info]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Get_IRQ_Info) + - [[1.7.1.4]{.tocnumber} [Set + IRQs]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Set_IRQs) + - [[1.8]{.tocnumber} [Device + Decomposure/Recomposure]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Device_Decomposure/Recomposure) + - [[1.9]{.tocnumber} [Memory Management Unit + (MMU)]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Memory_Management_Unit_(MMU)) + - [[1.9.1]{.tocnumber} [MMIO Isolation (Platform + MMU)]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#MMIO_Isolation_(Platform_MMU)) + - [[1.9.2]{.tocnumber} [VRAM Isolation (GPU + GMMU)]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#VRAM_Isolation_(GPU_GMMU)) + - [[1.9.3]{.tocnumber} [Platform MMIO \<-\> GPU Shadow + Pages]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Platform_MMIO_%3C-%3E_GPU_Shadow_Pages) + - [[1.10]{.tocnumber} [VFIO Quirks (region + traps)]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#VFIO_Quirks_(region_traps)) + - [[1.11]{.tocnumber} [Known IOMMU + Issues]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Known_IOMMU_Issues) +- [[2]{.tocnumber} [Mdev + Mode]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Mdev_Mode) + - [[2.1]{.tocnumber} [Knowledge Resources + Used]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Knowledge_Resources_Used_2) + - [[2.2]{.tocnumber} [Mediated + Core]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Mediated_Core) + - [[2.3]{.tocnumber} [Device + Initialization]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Device_Initialization) + - [[2.3.1]{.tocnumber} [Registering VFIO MDEV as a + driver]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Registering_VFIO_MDEV_as_a_driver) + - [[2.3.2]{.tocnumber} [Registering PCIE Device with the + Mediated + Core]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Registering_PCIE_Device_with_the_Mediated_Core) + - [[2.3.3]{.tocnumber} [Registering Mediated Callbacks + (CBs)]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Registering_Mediated_Callbacks_(CBs)) + - [[2.3.4]{.tocnumber} [Creating a Mediated Device via + mdev-sysfsdev + API]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Creating_a_Mediated_Device_via_mdev-sysfsdev_API) + - [[2.3.5]{.tocnumber} [QEMU adds VFIO device to IOMMU + container-group]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#QEMU_adds_VFIO_device_to_IOMMU_container-group) + - [[2.3.6]{.tocnumber} [QEMU passes mdev device file + descriptor to + VM]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#QEMU_passes_mdev_device_file_descriptor_to_VM) + - [[2.4]{.tocnumber} [Instruction + Execution]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Instruction_Execution) + - [[2.4.1]{.tocnumber} [IRQ + remapping]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#IRQ_remapping) + - [[2.4.2]{.tocnumber} [Interrupt + Injection]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Interrupt_Injection) + - [[2.5]{.tocnumber} [Memory + Management]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Memory_Management) + - [[2.5.1]{.tocnumber} [Region + Passthrough]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Region_Passthrough) + - [[2.5.1.1]{.tocnumber} [Emulated + Regions]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Emulated_Regions) + - [[2.5.1.2]{.tocnumber} [Passthrough + Regions]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Passthrough_Regions) + - [[2.5.1.3]{.tocnumber} [Region + Access]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Region_Access) + - [[2.5.2]{.tocnumber} [EPT Page + Violations]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#EPT_Page_Violations) + - [[2.5.3]{.tocnumber} [Mediated DMA + Translations]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Mediated_DMA_Translations) + - [[2.6]{.tocnumber} + [Scheduling]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Scheduling) + - [[2.7]{.tocnumber} [Kernel + API]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Kernel_API) + - [[2.8]{.tocnumber} [Sample + Code]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Sample_Code) + - [[2.8.1]{.tocnumber} + [Compilation]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Compilation) + - [[2.8.2]{.tocnumber} [Loading Kernel + Modules]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Loading_Kernel_Modules) + - [[2.8.3]{.tocnumber} [Unloading Kernel + Modules]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Unloading_Kernel_Modules) + - [[2.8.4]{.tocnumber} [Additional + Documentation]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Additional_Documentation) + - [[2.9]{.tocnumber} [Mdev Mode + Requirements]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Mdev_Mode_Requirements) +- [[3]{.tocnumber} [SR-IOV + Mode]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#SR-IOV_Mode) + - [[3.1]{.tocnumber} [Knowledge Resources + Used]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Knowledge_Resources_Used_3) + - [[3.2]{.tocnumber} [Instruction + Execution]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Instruction_Execution_2) + - [[3.3]{.tocnumber} [Memory + Management]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Memory_Management_2) + - [[3.4]{.tocnumber} + [Scheduling]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Scheduling_2) + - [[3.5]{.tocnumber} [SR-IOV Mode + Requirements]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#SR-IOV_Mode_Requirements) +- [[4]{.tocnumber} [SIOV + Mode]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#SIOV_Mode) + - [[4.1]{.tocnumber} [Knowledge Resources + Used]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Knowledge_Resources_Used_4) + - [[4.2]{.tocnumber} [Memory + Management]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Memory_Management_3) + - [[4.2.1]{.tocnumber} [IOMMU Aware Mediated + Device]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#IOMMU_Aware_Mediated_Device) + - [[4.2.2]{.tocnumber} [Shared Hardware + Workqueues]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#Shared_Hardware_Workqueues) + - [[4.3]{.tocnumber} [SIOV Mode + Requirements]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#SIOV_Mode_Requirements) +- [[5]{.tocnumber} [References (Talks & Reading + Material)]{.toctext}](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html#References_(Talks_&_Reading_Material)) +::: + +## [All Modes]{#All_Modes .mw-headline} + +This section will cover concepts which apply both to [Mdev +Mode](index.php/Mediated_Device_Internals.html#Mdev_Mode){.external +.text rel="nofollow"}, [SR-IOV +Mode](index.php/Mediated_Device_Internals.html#SR-IOV_Mode){.external +.text rel="nofollow"} & [SIOV +Mode](index.php/Mediated_Device_Internals.html#SIOV_Mode){.external +.text rel="nofollow"}. + +### [Knowledge Resources Used]{#Knowledge_Resources_Used .mw-headline} + +This section is supported by significant contributions in open source by +Alex Williamson. + +See references 2, 14, & 22 in the [References (Talks & Reading +Material)](index.php/Virtual_IO_Internals.html#References_(Talks_&_Reading_Material)){.external +.text rel="nofollow"} section. + +### [Binding VFIO devices]{#Binding_VFIO_devices .mw-headline} + +::: {.thumb .tright} +::: {.thumbinner style="width:302px;"} +[{.thumbimage +decoding="async" width="300" height="218" +srcset="images/thumb/9/9e/Vfio-pci_driver_bindings.png/450px-Vfio-pci_driver_bindings.png 1.5x, images/thumb/9/9e/Vfio-pci_driver_bindings.png/600px-Vfio-pci_driver_bindings.png 2x"}](index.php/File:Vfio-pci_driver_bindings.png.html){.image} + +::: thumbcaption +::: magnify +[](index.php/File:Vfio-pci_driver_bindings.png.html "Enlarge"){.internal} +::: + +**Figure 1:** VFIO group nodes are unit of ownership that VFIO uses. See +slides from: +[\[2\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external +.text rel="nofollow"} +::: +::: +::: + +::: {.thumb .tright} +::: {.thumbinner style="width:302px;"} +[{.thumbimage +decoding="async" width="300" height="242" +srcset="images/thumb/3/33/IOCTL_set_VFIO_container.png/450px-IOCTL_set_VFIO_container.png 1.5x, images/thumb/3/33/IOCTL_set_VFIO_container.png/600px-IOCTL_set_VFIO_container.png 2x"}](index.php/File:IOCTL_set_VFIO_container.png.html){.image} + +::: thumbcaption +::: magnify +[](index.php/File:IOCTL_set_VFIO_container.png.html "Enlarge"){.internal} +::: + +**Figure 2:** IOCTL(GROUP, VFIO_GROUP_SET_CONTAINER, &CONTAINER) places +the VFIO Group inside the VFIO Container. See slides from: +[\[2\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external +.text rel="nofollow"} +::: +::: +::: + +::: {.thumb .tright} +::: {.thumbinner style="width:302px;"} +[{.thumbimage +decoding="async" width="300" height="214" +srcset="images/thumb/4/40/Ioctl-VFIO_IOMMU_MAP-UNMAP_DMA.png/450px-Ioctl-VFIO_IOMMU_MAP-UNMAP_DMA.png 1.5x, images/thumb/4/40/Ioctl-VFIO_IOMMU_MAP-UNMAP_DMA.png/600px-Ioctl-VFIO_IOMMU_MAP-UNMAP_DMA.png 2x"}](index.php/File:Ioctl-VFIO_IOMMU_MAP-UNMAP_DMA.png.html){.image} + +::: thumbcaption +::: magnify +[](index.php/File:Ioctl-VFIO_IOMMU_MAP-UNMAP_DMA.png.html "Enlarge"){.internal} +::: + +**Figure 3:** Using interrupts IOCTL(CONTAINER, VFIO_IOMMU_MAP_DMA, +&MAP), IOCTL(CONTAINER, VFIO_IOMMU_UNMAP_DMA, &UNMAP) to map memory and +pin pages. See slides from: +[\[2\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external +.text rel="nofollow"} +::: +::: +::: + +::: {.thumb .tright} +::: {.thumbinner style="width:302px;"} +[{.thumbimage +decoding="async" width="300" height="204" +srcset="images/thumb/3/3f/Ioctl-VFIO_GROUP_GET_FD.png/450px-Ioctl-VFIO_GROUP_GET_FD.png 1.5x, images/thumb/3/3f/Ioctl-VFIO_GROUP_GET_FD.png/600px-Ioctl-VFIO_GROUP_GET_FD.png 2x"}](index.php/File:Ioctl-VFIO_GROUP_GET_FD.png.html){.image} + +::: thumbcaption +::: magnify +[](index.php/File:Ioctl-VFIO_GROUP_GET_FD.png.html "Enlarge"){.internal} +::: + +**Figure 4:** Using interrupt IOCTL(GROUP2, VFIO_GROUP_GET_FD, +\"0000:01:00.0\") to obtain the VFIO Group file descriptor. See slides +from: +[\[2\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external +.text rel="nofollow"} +::: +::: +::: + +**Figure 1:** Binding devices to the vfio-pci driver results in VFIO +group nodes. + +Opening the file \"/dev/vfio/vfio\" creates a VFIO Container. + +**Figure 2:** The interrupt routine +**`IOCTL(GROUP, VFIO_GROUP_SET_CONTAINER, &CONTAINER)`** places the VFIO +group inside the VFIO container. + +### [Programming the IOMMU]{#Programming_the_IOMMU .mw-headline} + +When this has been done +**`IOCTL(CONTAINER, VFIO_SET_IOMMU, VFIO_TYPE1_IOMMU)`** can then be +used to set an IOMMU type for the container which places it in a user +interactable state. + +Once this IOMMU type state has been set and the VFIO container has been +made interactable additional VFIO groups may be added to the container +without requiring that the group\'s IOMMU type be set again as newly +added groups automatically inherit the container\'s IOMMU context. + +### [VFIO Memory Mapped IO]{#VFIO_Memory_Mapped_IO .mw-headline} + +**Figure 3:** Once the VFIO Groups have been placed inside the VFIO +container and the IOMMU type has been set the user may then map and +unmap which will automatically inserts Memory Mapped IO (MMIO) entries +into the IOMMU as well as pin/unpin pages as necessary. This can be +accomplished using **`IOCTL(CONTAINER, VFIO_IOMMU_MAP_DMA, &MAP)`** for +map/pin and **`IOCTL(CONTAINER, VFIO_IOMMU_UNMAP_DMA, &UNMAP)`** for +unmap/unpin. + +### [Getting the VFIO Group File Descriptor]{#Getting_the_VFIO_Group_File_Descriptor .mw-headline} + +**Figure 4:** Once the device has been bound to a VFIO driver, set in a +VFIO container, the VFIO container has it\'s IOMMU type set, and a +memory map/page pin of the VFIO device has been completed a file +descriptor can then be obtained for the device. This file descriptor can +be used for interrupts (ioctls), to probe for information about the BAR +regions, and configure the IRQs. + +### [VFIO device file descriptor]{#VFIO_device_file_descriptor .mw-headline} + +VFIO device file descriptors are divided into regions and each region is +mapped into a device resource. Region count and info (file offset, +allowable access, ect..) can be discovered through interrupt (IOCTL). +Each file descriptor region corresponding to a PCI resource is +represented as a file offset. + +In the case of RPC Mode this structure is emulated whereas in SR-IOV +Mode the structure is mapped to a real PCI resource. + + --------------------------------------------------- + 00:00.0 VGA compatible controller + Region 0 Bar0 (Config Space) starts at offset 0 + Region 1 Bar1 (MSI - Message Signaled Interrupts) + Region 2 Bar2 (MSIX) + Region 3 Bar3 + Region 4 Bar4 + Region 5 Bar5 (IO port space) + Expansion ROM + --------------------------------------------------- + + : BAR regions in a VGA PCI device. + +Below is what the file offsets looks like internally for each BAR region +starting from address 0 and growing with the addition of former regions +as you progress through the file. + + --------------------- ------------------- ------------------- ----------------------- ------ + \<- File Offset -\> + 0 -\> A A -\> (A+B) (A+B) -\> (A+B+C) (A+B+C) -\> (A+B+C+D) \... + Region 0 (size A) Region 1 (size B) Region 2 (size C) Region 3 (size D) \... + --------------------- ------------------- ------------------- ----------------------- ------ + + : VFIO representation of PCI BAR regions offsets. + +### [VFIO Interrupts]{#VFIO_Interrupts .mw-headline} + +Guests communicate with the host via VFIO Interrupt Requests +([IRQs](https://infogalactic.com/info/Interrupt_request_(PC_architecture)){.external +.text rel="nofollow"}). These are sent via an irqfd (IRQ [File +Descriptor](https://infogalactic.com/info/File_descriptor){.external +.text rel="nofollow"}). Similarly, the host receives these interrupts +via +[eventfd](https://man7.org/linux/man-pages/man2/eventfd.2.html){.external +.text rel="nofollow"} (Event File Descriptor). The resulting data can be +returned via a +[callback](https://infogalactic.com/info/Callback_(computer_programming)){.external +.text rel="nofollow"}. + +#### [IRQs]{#IRQs .mw-headline} + +Device properties discovered via interrupt (IOCTL). + +##### [Get Device Info]{#Get_Device_Info .mw-headline} + + ------------------------- ------------- ---------------------------- + VFIO_DEVICE_GET_INFO + struct vfio_device_info + argz + flags + VFIO_DEVICE_FLAGS_PCI + VFIO_DEVICE_FLAGS_PLATFORM + VFIO_DEVICE_FLAGS_RESET + num_irqs + num_regions + ------------------------- ------------- ---------------------------- + +The IRQ **`VFIO_DEVICE_GET_INFO`** can provide information to +distinguish between PCI and platform devices as well as the number of +regions and IRQs for a particular device. + +The upstream API can be read **[here +(vfio.h)](https://github.com/torvalds/linux/blob/47700948a4abb4a5ae13ef943ff682a7f327547a/include/uapi/linux/vfio.h#L194){.external +.text rel="nofollow"}**. + +**Sample mdev code** to service this request can be found +**[here](https://github.com/Open-IOV/VFIO-Mdev_Samples/blob/c0adb9957c0b22bfdbc18395e81dd2b476addac5/mdpy.c#L470){.external +.text rel="nofollow"}**, and +**[here](https://github.com/Open-IOV/VFIO-Mdev_Samples/blob/c0adb9957c0b22bfdbc18395e81dd2b476addac5/mdpy.c#L522){.external +.text rel="nofollow"}**. + +##### [Get Region Info]{#Get_Region_Info .mw-headline} + + ----------------------------- ------------ ----------------------------- + VFIO_DEVICE_GET_REGION_INFO + struct vfio_region_info + argz + cap_offset + flags + VFIO_REGION_INFO_FLAG_CAPS + VFIO_REGION_INFO_FLAG_MMAP + VFIO_REGION_INFO_FLAG_READ + VFIO_REGION_INFO_FLAG_WRITE + index + offset + size + ----------------------------- ------------ ----------------------------- + +Once the interrupt user knows the number of regions within a VFIO device +they can use IRQ **`VFIO_DEVICE_GET_REGION_INFO`** to probe each region +for additional information. This interrupt will return information such +as if it can be read from or written to, if the device supports MMAP, as +well as what the offset and size of the region is within the VFIO file +descriptor. + +The upstream API can be read [**here +(vfio.h)**](https://github.com/torvalds/linux/blob/47700948a4abb4a5ae13ef943ff682a7f327547a/include/uapi/linux/vfio.h#L240){.external +.text rel="nofollow"}. + +**Sample mdev code** to service this request can be found +[**here**](https://github.com/Open-IOV/VFIO-Mdev_Samples/blob/c0adb9957c0b22bfdbc18395e81dd2b476addac5/mdpy.c#L425){.external +.text rel="nofollow"}, and +[**here**](https://github.com/Open-IOV/VFIO-Mdev_Samples/blob/c0adb9957c0b22bfdbc18395e81dd2b476addac5/mdpy.c#L545){.external +.text rel="nofollow"}. + +##### [Get IRQ Info]{#Get_IRQ_Info .mw-headline} + + -------------------------- ------- -------------------------- + VFIO_DEVICE_GET_IRQ_INFO + struct vfio_irq_info + argz + count + flags + VFIO_IRQ_INFO_AUTOMASKED + VFIO_IRQ_INFO_EVENTFD + VFIO_IRQ_INFO_MASKABLE + VFIO_IRQ_INFO_NORESIZE + index + -------------------------- ------- -------------------------- + +**`VFIO_DEVICE_GET_IRQ_INFO`** is used to retrieve information about a +device IRQ. + +**`VFIO_IRQ_INFO_AUTOMASKED`** is used to mask interrupts when they +occur to protect the host. + +The upstream API can be read **[here +(vfio.h)](https://github.com/torvalds/linux/blob/47700948a4abb4a5ae13ef943ff682a7f327547a/include/uapi/linux/vfio.h#L480){.external +.text rel="nofollow"}** + +**Sample mdev code** to service this request can be found +[**here**](https://github.com/Open-IOV/VFIO-Mdev_Samples/blob/c0adb9957c0b22bfdbc18395e81dd2b476addac5/mdpy.c#L463){.external +.text rel="nofollow"}, and +[**here**](https://github.com/Open-IOV/VFIO-Mdev_Samples/blob/c0adb9957c0b22bfdbc18395e81dd2b476addac5/mdpy.c#L570){.external +.text rel="nofollow"}. + +##### [Set IRQs]{#Set_IRQs .mw-headline} + + ---------------------- ---------- ----------------------------- + VFIO_DEVICE_SET_IRQS + struct vfio_irq_set + argz + count + data\[\] + flags + VFIO_IRQ_SET_ACTION_MASK + VFIO_IRQ_SET_ACTION_TRIGGER + VFIO_IRQ_SET_ACTION_UNMASK + VFIO_IRQ_SET_DATA_BOOL + VFIO_IRQ_SET_DATA_EVENTFD + VFIO_IRQ_SET_DATA_NONE + index + start + ---------------------- ---------- ----------------------------- + +**`VFIO_DEVICE_SET_IRQS`** is used to setup IRQs. Actions can be +configured such as trigger which is when the device triggers an +interrupt (IOCTL), masking and unmasking actions can be set. Bool and +None data types are used for loopback testing of the device. Start and +index may be used to modify subregions. + +The upstream API can be read **[here +(vfio.h)](https://github.com/torvalds/linux/blob/47700948a4abb4a5ae13ef943ff682a7f327547a/include/uapi/linux/vfio.h#L524){.external +.text rel="nofollow"}**. + +### []{#Device_Decomposure.2FRecomposure}[Device Decomposure/Recomposure]{#Device_Decomposure/Recomposure .mw-headline} + +::: {.thumb .tright} +::: {.thumbinner style="width:302px;"} +[{.thumbimage +decoding="async" width="300" height="147" +srcset="images/thumb/9/90/Device_Decomposure_and_Recomposure_via_VFIO.png/450px-Device_Decomposure_and_Recomposure_via_VFIO.png 1.5x, images/thumb/9/90/Device_Decomposure_and_Recomposure_via_VFIO.png/600px-Device_Decomposure_and_Recomposure_via_VFIO.png 2x"}](index.php/File:Device_Decomposure_and_Recomposure_via_VFIO.png.html){.image} + +::: thumbcaption +::: magnify +[](index.php/File:Device_Decomposure_and_Recomposure_via_VFIO.png.html "Enlarge"){.internal} +::: + +**Figure 5:** Device Decomposure and Recomposure via VFIO. See slides +from: +[\[2\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external +.text rel="nofollow"} +::: +::: +::: + +**Figure 5:** Virtual Function IO (VFIO) devices are deconstructed in +userspace into a set of VFIO primitives (MMIO pages, VFIO/IOMMU Groups, +VFIO IRQs, File Descriptors). Recomposure of these devices occurs upon +assignment of a Virtual Function (VF) to a QEMU virtual machine. + +### []{#Memory_Management_Unit_.28MMU.29}[Memory Management Unit (MMU)]{#Memory_Management_Unit_(MMU) .mw-headline} + +::: {.thumb .tright} +::: {.thumbinner style="width:302px;"} +::: {.MediaTransformError style="width: 300px; height: 275px; display:inline-block;"} +Error creating thumbnail: File missing +::: + +::: thumbcaption +::: magnify +[](index.php/File:Figure_6-_MMU_-_GMMU_IOVA_translation-isolation..png.html "Enlarge"){.internal} +::: + +**Figure 6:** MMU - GMMU IOVA translation/isolation. +::: +::: +::: + +**Figure 6:** This section will touch upon the mechanisms used for +enforcement of Host Physical Address (HPA) to Guest Physical Address +(GPA) isolation. + +#### []{#MMIO_Isolation_.28Platform_MMU.29}[MMIO Isolation (Platform MMU)]{#MMIO_Isolation_(Platform_MMU) .mw-headline} + +The platform\'s CPU communicates with the GPU by reading/writing to and +from pinned MMIO pages in [Random Access Memory +(RAM)](https://infogalactic.com/info/Random-access_memory){.external +.text rel="nofollow"}. MMIO pages within the RAM are subject to IO +Virtual Address (IOVA) translations by the platform\'s discrete MMU +controller which is programmed by the CPU. These IOVA translations serve +as a mechanism to enforce HPA to GPA isolation in the context of the +platform. + +#### []{#VRAM_Isolation_.28GPU_GMMU.29}[VRAM Isolation (GPU GMMU)]{#VRAM_Isolation_(GPU_GMMU) .mw-headline} + +The GPU core performs virtualized operations by reading/writing to and +from shadow page tables in onboard [Video Random Access Memory +(VRAM)](https://en.wikipedia.org/wiki/Video_random_access_memory "wikipedia:Video random access memory"){.extiw}. +Shadow pages within the VRAM are subject to IO Virtual Address (IOVA) +translations by the GPU\'s discrete GPU MMU controller (GMMU) which is +programmed by the Embedded CPU (GPU co-processor). These IOVA +translations serve as a mechanism to enforce HPA to GPA isolation in the +context of the virtual GPUs and multi-process isolation in single user +environments. + +#### []{#Platform_MMIO_.3C-.3E_GPU_Shadow_Pages}[Platform MMIO \<-\> GPU Shadow Pages]{#Platform_MMIO_<->_GPU_Shadow_Pages .mw-headline} + +In the context of VFIO pinned MMIO pages in RAM act as an interface to +communicate with VRAM shadow pages allowing GPU drivers on the platform +to send instructions to the GPU. When the GPU or Platform alters memory +contained in a shadow page or pinned MMIO page the change is mirrored in +the corresponding IO Virtual Address (IOVA). For example if shadow page +0 is changed by the GPU this change is mirrored in MMIO page 0 on the +platform (the reverse example also applies). When communications occur +between the platform and GPU the information first moves through the +MMU/GMMU and is then written to RAM/VRAM. + +::: {.thumb .tright} +::: {.thumbinner style="width:302px;"} +[{.thumbimage +decoding="async" width="300" height="268" +srcset="images/thumb/4/48/Figure_7-_A_depiction_of_region_overlays_within_a_VFIO_file_descriptor..png/450px-Figure_7-_A_depiction_of_region_overlays_within_a_VFIO_file_descriptor..png 1.5x, images/thumb/4/48/Figure_7-_A_depiction_of_region_overlays_within_a_VFIO_file_descriptor..png/600px-Figure_7-_A_depiction_of_region_overlays_within_a_VFIO_file_descriptor..png 2x"}](index.php/File:Figure_7-_A_depiction_of_region_overlays_within_a_VFIO_file_descriptor..png.html){.image} + +::: thumbcaption +::: magnify +[](index.php/File:Figure_7-_A_depiction_of_region_overlays_within_a_VFIO_file_descriptor..png.html "Enlarge"){.internal} +::: + +**Figure 7:** A depiction of region overlays within a VFIO file +descriptor. See slides from: +[\[2\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external +.text rel="nofollow"} +::: +::: +::: + +### []{#VFIO_Quirks_.28region_traps.29}[VFIO Quirks (region traps)]{#VFIO_Quirks_(region_traps) .mw-headline} + +**Figure 7:** Regions of the PCI BAR require emulation (slow path) via +emulated traps (known as quirks). These regions are primarily in PCI +configuration space. QEMU may overlap a region overlay which when +read/written to/from triggers a VM-exit to trap and emulate the region +in order that appropriate translations may occur (such as those +concerned with IO Virtual Address - IOVA). + +::: {.thumb .tright} +::: {.thumbinner style="width:302px;"} +[{.thumbimage +decoding="async" width="300" height="224" +srcset="images/thumb/1/13/Screen_Shot_2022-05-06_at_10.46.59_AM.png/450px-Screen_Shot_2022-05-06_at_10.46.59_AM.png 1.5x, images/1/13/Screen_Shot_2022-05-06_at_10.46.59_AM.png 2x"}](index.php/File:Screen_Shot_2022-05-06_at_10.46.59_AM.png.html){.image} + +::: thumbcaption +::: magnify +[](index.php/File:Screen_Shot_2022-05-06_at_10.46.59_AM.png.html "Enlarge"){.internal} +::: + +**Figure 8:** Peer-to-Peer DMA Isolation under IOMMU. See slides from: +[\[2\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external +.text rel="nofollow"} +::: +::: +::: + +### [Known IOMMU Issues]{#Known_IOMMU_Issues .mw-headline} + +**DMA Aliasing** + +- Not all devices generate unique IDs. +- Not all devices generate IDs they should.\ + +**DMA Isolation** + +- **Figure 8:** Peer-to-Peer DMA Isolation. In many circumstances IO + Virtual Address (IOVA) translations do not occur properly in the + context of DMA peering. Transactions that occur through the IOMMU + are unaffected. + +::: {.thumb .tright} +::: {.thumbinner style="width:302px;"} +[{.thumbimage +decoding="async" width="300" height="153" +srcset="images/thumb/6/6a/Figure_0-_Approches_to_GPU_Virtualization..png/450px-Figure_0-_Approches_to_GPU_Virtualization..png 1.5x, images/thumb/6/6a/Figure_0-_Approches_to_GPU_Virtualization..png/600px-Figure_0-_Approches_to_GPU_Virtualization..png 2x"}](index.php/File:Figure_0-_Approches_to_GPU_Virtualization..png.html){.image} + +::: thumbcaption +::: magnify +[](index.php/File:Figure_0-_Approches_to_GPU_Virtualization..png.html "Enlarge"){.internal} +::: + +**Figure 0:** Approaches to GPU Virtualization. See slides from: +[\[61\]](index.php/Virtual_I/O_Internals.html#References_(Talks_&_Reading_Material)){.external +.text rel="nofollow"} +::: +::: +::: + +## [Mdev Mode]{#Mdev_Mode .mw-headline} + +**Mdev Mode (VFIO Mediated Device)** is a method of virtualizing I/O +devices enabling full API capabilities without the requirement for +hardware assistance. + +### [Knowledge Resources Used]{#Knowledge_Resources_Used_2 .mw-headline} + +This section is supported by significant contributions in open source by +Neo Jia, Kirti Wankhede, Kevin Tian, Yiying Zhang, David Cowperthwaite, +Kun Tian, Yaozu Dong, Tina Zhang, Gerd Hoffmann, & Zhi Wang. + +See references 1, 7, 8, 17, 18, 19, 70 in the [References (Talks & +Reading +Material)](index.php/Virtual_IO_Internals.html#References_(Talks_&_Reading_Material)){.external +.text rel="nofollow"} section. + +### [Mediated Core]{#Mediated_Core .mw-headline} + +The mediated device framework made 3 major changes to the VFIO driver. + +- **1: [Mediated core module + (new)](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/vfio/mdev/mdev_core.c){.external + .text rel="nofollow"}**\ + -Mediated bus driver, create mediated device.\ + -Physical device interface for vendor callbacks.\ + -Generic Mediated device management user interface (sysfs). +- **2: [Mediated device module + (new)](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/vfio/mdev/mdev_driver.c){.external + .text rel="nofollow"}**\ + -Manage created mediated device, fully compatible with VFIO user API + (UAPI). +- **3: [VFIO IOMMU driver + (enhancement)](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/vfio/vfio_iommu_type1.c){.external + .text rel="nofollow"}**\ + -VFIO IOMMU API Type1 compatible, easy to extend to non-Type1. + +The full list of these changes can be seen in the [**lists.gnu.org +Qemu-devel** mailing list +archive](https://lists.gnu.org/archive/html/qemu-devel/2016-10/msg03922.html){.external +.text rel="nofollow"}. + +### [Device Initialization]{#Device_Initialization .mw-headline} + +This section will deal with how an Mdev driver is initialized. + +**Sample mdev code** for device initialization can be found +**[here](https://github.com/Open-IOV/VFIO-Mdev_Samples/blob/c0adb9957c0b22bfdbc18395e81dd2b476addac5/mdpy.c#L229){.external +.text rel="nofollow"}**. + +#### [Registering VFIO MDEV as a driver]{#Registering_VFIO_MDEV_as_a_driver .mw-headline} + +VFIO Mdev must be registered as a driver. This register event occurs +between the Mdev Driver Register Interface and the VFIO MDEV interface. + +#### [Registering PCIE Device with the Mediated Core]{#Registering_PCIE_Device_with_the_Mediated_Core .mw-headline} + +::: {.thumb .tright} +::: {.thumbinner style="width:302px;"} +[{.thumbimage +decoding="async" width="300" height="208" +srcset="images/thumb/7/75/Mdev-gpu.png/450px-Mdev-gpu.png 1.5x, images/thumb/7/75/Mdev-gpu.png/600px-Mdev-gpu.png 2x"}](index.php/File:Mdev-gpu.png.html){.image} + +::: thumbcaption +::: magnify +[](index.php/File:Mdev-gpu.png.html "Enlarge"){.internal} +::: + +**Figure 9:** Registering the mediated device. This step may be +accomplished via the vendor driver or via +[GVM/Mdev-GPU](){.external .text +rel="nofollow"} for drivers which do not include support for Mdev +functionality and/or do not support arbitrary Mdev types. +::: +::: +::: + +**Figure 9:** The vendor driver must register with the Mediated Core\'s +Device Register Interface. The example shown uses the [GVM +Project](){.external .text rel="nofollow"}\'s +[Mdev-GPU](){.external .text +rel="nofollow"} component to accomplish this step on behalf of the +vendor driver. + +#### []{#Registering_Mediated_Callbacks_.28CBs.29}[Registering Mediated Callbacks (CBs)]{#Registering_Mediated_Callbacks_(CBs) .mw-headline} + +**Figure 9:** The vendor driver must now register Mediated Callbacks +which it expects to receive from Mdev devices. The example shown uses +the [GVM Project](){.external .text +rel="nofollow"}\'s +[Mdev-GPU](){.external .text +rel="nofollow"} component to accomplish this step on behalf of the +vendor driver. + +#### [Creating a Mediated Device via mdev-sysfsdev API]{#Creating_a_Mediated_Device_via_mdev-sysfsdev_API .mw-headline} + +::: {.thumb .tright} +::: {.thumbinner style="width:302px;"} +[{.thumbimage +decoding="async" width="300" height="235" +srcset="images/thumb/e/ef/Echoing_a_UUID_into_the_sysfsdev_API.png/450px-Echoing_a_UUID_into_the_sysfsdev_API.png 1.5x, images/thumb/e/ef/Echoing_a_UUID_into_the_sysfsdev_API.png/600px-Echoing_a_UUID_into_the_sysfsdev_API.png 2x"}](index.php/File:Echoing_a_UUID_into_the_sysfsdev_API.png.html){.image} + +::: thumbcaption +::: magnify +[](index.php/File:Echoing_a_UUID_into_the_sysfsdev_API.png.html "Enlarge"){.internal} +::: + +**Figure 10:** Echoing a UUID into the mdev-sysfsdev API. This +functionality is included in +[mdevctl](https://github.com/mdevctl/mdevctl){.external .text +rel="nofollow"} and in [](){.external .text +rel="nofollow"}. See slides from: +[\[1\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external +.text rel="nofollow"} +::: +::: +::: + +**Figure 10:** The user of an mdev capable device driver may echo values +such as a UUID into the mdev-sysfsdev interface to create a unique +mediated device. UUIDs must be unique per mdev device within a host. + +::: {.thumb .tright} +::: {.thumbinner style="width:302px;"} +[{.thumbimage +decoding="async" width="300" height="231" +srcset="images/thumb/6/63/QEMU_creates_VFIO-mdev_IOMMU_binding_and_acquires_mdev_file_descriptor..png/450px-QEMU_creates_VFIO-mdev_IOMMU_binding_and_acquires_mdev_file_descriptor..png 1.5x, images/thumb/6/63/QEMU_creates_VFIO-mdev_IOMMU_binding_and_acquires_mdev_file_descriptor..png/600px-QEMU_creates_VFIO-mdev_IOMMU_binding_and_acquires_mdev_file_descriptor..png 2x"}](index.php/File:QEMU_creates_VFIO-mdev_IOMMU_binding_and_acquires_mdev_file_descriptor..png.html){.image} + +::: thumbcaption +::: magnify +[](index.php/File:QEMU_creates_VFIO-mdev_IOMMU_binding_and_acquires_mdev_file_descriptor..png.html "Enlarge"){.internal} +::: + +**Figure 11:** QEMU creates VFIO-mdev IOMMU binding and acquires mdev +file descriptor. See slides from: +[\[1\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external +.text rel="nofollow"} +::: +::: +::: + +#### [QEMU adds VFIO device to IOMMU container-group]{#QEMU_adds_VFIO_device_to_IOMMU_container-group .mw-headline} + +**Figure 11:** When starting a QEMU process with a VFIO-mdev attached +QEMU calls the VFIO API to add the VFIO device to an IOMMU +container/group. QEMU then runs the IOCTL to obtain a file descriptor +for the device. + +#### [QEMU passes mdev device file descriptor to VM]{#QEMU_passes_mdev_device_file_descriptor_to_VM .mw-headline} + +::: {.thumb .tright} +::: {.thumbinner style="width:302px;"} +[{.thumbimage +decoding="async" width="300" height="234" +srcset="images/thumb/7/7c/QEMU_presenting_the_VFIO_file_descriptor_into_the_virtual_machine..png/450px-QEMU_presenting_the_VFIO_file_descriptor_into_the_virtual_machine..png 1.5x, images/thumb/7/7c/QEMU_presenting_the_VFIO_file_descriptor_into_the_virtual_machine..png/600px-QEMU_presenting_the_VFIO_file_descriptor_into_the_virtual_machine..png 2x"}](index.php/File:QEMU_presenting_the_VFIO_file_descriptor_into_the_virtual_machine..png.html){.image} + +::: thumbcaption +::: magnify +[](index.php/File:QEMU_presenting_the_VFIO_file_descriptor_into_the_virtual_machine..png.html "Enlarge"){.internal} +::: + +**Figure 12:** QEMU presenting the VFIO file descriptor into the virtual +machine. See slides from: +[\[1\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external +.text rel="nofollow"} +::: +::: +::: + +**Figure 12:** Once QEMU has obtained the VFIO file descriptor for the +Mdev device via IOCTL it is then QEMU\'s job to present the file +descriptor into the virtual machine so that the mdev may be used by the +guest. + +### [Instruction Execution]{#Instruction_Execution .mw-headline} + +::: {.thumb .tright} +::: {.thumbinner style="width:302px;"} +[{.thumbimage +decoding="async" width="300" height="215" +srcset="images/thumb/a/aa/Ioeventfd-and-irqfd.png/450px-Ioeventfd-and-irqfd.png 1.5x, images/thumb/a/aa/Ioeventfd-and-irqfd.png/600px-Ioeventfd-and-irqfd.png 2x"}](index.php/File:Ioeventfd-and-irqfd.png.html){.image} + +::: thumbcaption +::: magnify +[](index.php/File:Ioeventfd-and-irqfd.png.html "Enlarge"){.internal} +::: + +**Figure 13:** A simple diagram of signalling from host to guest (via +irqfd) & guest to host (via ioeventfd) From +[blog.allenx](http://blog.allenx.org/2015/07/05/kvm-irqfd-and-ioeventfd){.external +.text rel="nofollow"} +::: +::: +::: + +**Figure 13:** Mdev Mode moves instruction information across a virtual +function (VF) device using [Remote Procedure +Calls](https://infogalactic.com/info/Remote_procedure_call){.external +.text rel="nofollow"} generally by way of [software +interrupt](https://infogalactic.com/info/Interrupt){.external .text +rel="nofollow"} +([IOCTL](https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core.git/tree/Documentation/driver-api/ioctl.rst){.external +.text rel="nofollow"}). Signals to and from the guest and the host GPU +driver may be passed over file descriptors such as the [Interrupt +Request File Descriptor +(irqfd)](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=721eecbf4fe995ca94a9edec0c9843b1cc0eaaf3){.external +.text rel="nofollow"} and [IO Event File Descriptor +(ioeventfd)](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d34e6b175e61821026893ec5298cc8e7558df43a){.external +.text rel="nofollow"}. The irqfd may be used to signal from the host +into the guest whereas the ioeventfd may be used to signal from the +guest into the host. Guest GPU instructions which would normally +serialize as pRPCs (physical Remote Procedure Calls) are instead +serialized from the guest as vRPCs (virtual Remote Procedure Calls) +which are executed by the host mediated driver. + +#### [IRQ remapping]{#IRQ_remapping .mw-headline} + +Interrupt Requests (IRQs) must be remapped (trapped for virtualized +execution) to protect the host from sensitive instructions which may +affect global memory state. + +#### [Interrupt Injection]{#Interrupt_Injection .mw-headline} + +**\--** + +**\--Researching\--** + +**\--** + +### [Memory Management]{#Memory_Management .mw-headline} + +Mdev memory management is handled by vendor driver software. + +#### [Region Passthrough]{#Region_Passthrough .mw-headline} + +Guests may be presented with emulated memory regions or via passthrough +regions or a mixture of the two such as in the case of passthrough +regions with BAR 0 configuration space emulation while other regions are +passthrough\'d. + +##### [Emulated Regions]{#Emulated_Regions .mw-headline} + +Emulated memory regions use indirect emulated communication requiring a +VM-exit (slow). These regions are often used for virtual PCI config +space such as [in this sample +code](https://github.com/Open-IOV/VFIO-Mdev_Samples/blob/c0adb9957c0b22bfdbc18395e81dd2b476addac5/mdpy.c#L112){.external +.text rel="nofollow"}. + +##### [Passthrough Regions]{#Passthrough_Regions .mw-headline} + +Passthrough memory regions use direct communication requiring no VM-exit +(fast). + +##### [Region Access]{#Region_Access .mw-headline} + +::: {.thumb .tright} +::: {.thumbinner style="width:302px;"} +[{.thumbimage +decoding="async" width="300" height="236" +srcset="images/thumb/f/fa/QEMU_gets_region_info_via_VFIO_UAPI_from_vendor_driver_through_vfio-mdev_and_Mediated_CBs.png/450px-QEMU_gets_region_info_via_VFIO_UAPI_from_vendor_driver_through_vfio-mdev_and_Mediated_CBs.png 1.5x, images/thumb/f/fa/QEMU_gets_region_info_via_VFIO_UAPI_from_vendor_driver_through_vfio-mdev_and_Mediated_CBs.png/600px-QEMU_gets_region_info_via_VFIO_UAPI_from_vendor_driver_through_vfio-mdev_and_Mediated_CBs.png 2x"}](index.php/File:QEMU_gets_region_info_via_VFIO_UAPI_from_vendor_driver_through_vfio-mdev_and_Mediated_CBs.png.html){.image} + +::: thumbcaption +::: magnify +[](index.php/File:QEMU_gets_region_info_via_VFIO_UAPI_from_vendor_driver_through_vfio-mdev_and_Mediated_CBs.png.html "Enlarge"){.internal} +::: + +**Figure 14:** QEMU gets region info via VFIO UAPI from vendor driver +through vfio-mdev and Mediated CBs. See slides from: +[\[1\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external +.text rel="nofollow"} +::: +::: +::: + +**Figure 14:** QEMU gets region information via VFIO User API (UAPI) +from the vendor driver through VFIO-mdev and Mediated Callbacks. + +::: {.thumb .tright} +::: {.thumbinner style="width:302px;"} +::: {.MediaTransformError style="width: 300px; height: 238px; display:inline-block;"} +Error creating thumbnail: File missing +::: + +::: thumbcaption +::: magnify +[](index.php/File:Guest_driver_accessing_Mdev_MMIO_space_backed_by_mdev_file_descriptor_triggers_EPT_violation..png.html "Enlarge"){.internal} +::: + +**Figure 15:** Guest driver accessing Mdev MMIO space backed by mdev +file descriptor triggers EPT violation. See slides from: +[\[1\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external +.text rel="nofollow"} +::: +::: +::: + +**Figure 15:** The guest\'s vendor driver accesses the Mdev MMIO trapped +region backed by a mdev file descriptor (fd) which triggers an Extended +Page Table (EPT) violation. + +::: {.thumb .tright} +::: {.thumbinner style="width:302px;"} +[{.thumbimage +decoding="async" width="300" height="235" +srcset="images/thumb/4/43/KVM_services_EPT_violation_and_forwards_to_QEMU_VFIO_PCI_driver..png/450px-KVM_services_EPT_violation_and_forwards_to_QEMU_VFIO_PCI_driver..png 1.5x, images/thumb/4/43/KVM_services_EPT_violation_and_forwards_to_QEMU_VFIO_PCI_driver..png/600px-KVM_services_EPT_violation_and_forwards_to_QEMU_VFIO_PCI_driver..png 2x"}](index.php/File:KVM_services_EPT_violation_and_forwards_to_QEMU_VFIO_PCI_driver..png.html){.image} + +::: thumbcaption +::: magnify +[](index.php/File:KVM_services_EPT_violation_and_forwards_to_QEMU_VFIO_PCI_driver..png.html "Enlarge"){.internal} +::: + +**Figure 16:** KVM services EPT violation and forwards to QEMU VFIO PCI +driver. See slides from: +[\[1\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external +.text rel="nofollow"} +::: +::: +::: + +**Figure 16:** KVM services EPT violation and forwards to QEMU VFIO PCI +driver. + +::: {.thumb .tright} +::: {.thumbinner style="width:302px;"} +[{.thumbimage +decoding="async" width="300" height="235" +srcset="images/thumb/2/27/QEMU_convert_request_from_KVM_to_Read-Write_acess_to_Mdev_file_descriptor..png/450px-QEMU_convert_request_from_KVM_to_Read-Write_acess_to_Mdev_file_descriptor..png 1.5x, images/thumb/2/27/QEMU_convert_request_from_KVM_to_Read-Write_acess_to_Mdev_file_descriptor..png/600px-QEMU_convert_request_from_KVM_to_Read-Write_acess_to_Mdev_file_descriptor..png 2x"}](index.php/File:QEMU_convert_request_from_KVM_to_Read-Write_acess_to_Mdev_file_descriptor..png.html){.image} + +::: thumbcaption +::: magnify +[](index.php/File:QEMU_convert_request_from_KVM_to_Read-Write_acess_to_Mdev_file_descriptor..png.html "Enlarge"){.internal} +::: + +**Figure 17:** QEMU convert request from KVM to Read/Write acess to Mdev +file descriptor. See slides from: +[\[1\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external +.text rel="nofollow"} +::: +::: +::: + +**Figure 17:** QEMU convert request from KVM to R/W access to Mdev file +descriptor. + +#### [EPT Page Violations]{#EPT_Page_Violations .mw-headline} + +Guest [Memory Mapped IO +(MMIO)](https://infogalactic.com/info/Memory-mapped_I/O){.external .text +rel="nofollow"} trips Extended Page Table (EPT) violations which are +trapped by the host MMU. KVM services EPT violations and forwards to +QEMU VFIO PCI driver. QEMU then converts the request from KVM to R/W +access to the [Mdev File Descriptor +(FD)](https://infogalactic.com/info/File_descriptor){.external .text +rel="nofollow"}. Reads and writes are then handled by the host GPU +device driver via mediated [callbacks +(CBs)](https://infogalactic.com/info/Callback_(computer_programming)){.external +.text rel="nofollow"} and +[VFIO-mdev](https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core.git/tree/Documentation/driver-api/vfio-mediated-device.rst){.external +.text rel="nofollow"}. + +#### [Mediated DMA Translations]{#Mediated_DMA_Translations .mw-headline} + +**\--** + +**\--Researching\--** + +**\--** + +Because memory is not statically allocated by the vendor driver under +the mediated device framework there is no requirement to make use of +traditional VFIO pinned pages (via the vfio-pci module) rather MMIO +memory can be mapped at runtime incrementally. As a result non-standard +mediated device vfio stub modules may be used. + +**Figure 18:** Memory regions get added by QEMU. + +::: {.thumb .tright} +::: {.thumbinner style="width:302px;"} +[{.thumbimage +decoding="async" width="300" height="225" +srcset="images/thumb/5/58/Memory_Regions_Added_by_QEMU.png/450px-Memory_Regions_Added_by_QEMU.png 1.5x, images/thumb/5/58/Memory_Regions_Added_by_QEMU.png/600px-Memory_Regions_Added_by_QEMU.png 2x"}](index.php/File:Memory_Regions_Added_by_QEMU.png.html){.image} + +::: thumbcaption +::: magnify +[](index.php/File:Memory_Regions_Added_by_QEMU.png.html "Enlarge"){.internal} +::: + +**Figure 18:** Memory Regions Added by QEMU. See slides from: +[\[1\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external +.text rel="nofollow"} +::: +::: +::: + +**Figure 19:** QEMU calls VFIO_DMA_MAP via Memory listener. (not just +guest physical memory but also device memory will be added through this +memory listener) + +::: {.thumb .tright} +::: {.thumbinner style="width:302px;"} +[{.thumbimage +decoding="async" width="300" height="237" +srcset="images/thumb/2/21/QEMU_calls_VFIO_DMA_MAP_via_Memory_listener.png/450px-QEMU_calls_VFIO_DMA_MAP_via_Memory_listener.png 1.5x, images/thumb/2/21/QEMU_calls_VFIO_DMA_MAP_via_Memory_listener.png/600px-QEMU_calls_VFIO_DMA_MAP_via_Memory_listener.png 2x"}](index.php/File:QEMU_calls_VFIO_DMA_MAP_via_Memory_listener.png.html){.image} + +::: thumbcaption +::: magnify +[](index.php/File:QEMU_calls_VFIO_DMA_MAP_via_Memory_listener.png.html "Enlarge"){.internal} +::: + +**Figure 19:** QEMU calls VFIO DMA MAP via Memory listener. See slides +from: +[\[1\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external +.text rel="nofollow"} +::: +::: +::: + +::: {.thumb .tright} +::: {.thumbinner style="width:302px;"} +[{.thumbimage +decoding="async" width="300" height="237" +srcset="images/thumb/1/1d/Type_1_IOMMU_Tracks_VA-GFN.png/450px-Type_1_IOMMU_Tracks_VA-GFN.png 1.5x, images/thumb/1/1d/Type_1_IOMMU_Tracks_VA-GFN.png/600px-Type_1_IOMMU_Tracks_VA-GFN.png 2x"}](index.php/File:Type_1_IOMMU_Tracks_VA-GFN.png.html){.image} + +::: thumbcaption +::: magnify +[](index.php/File:Type_1_IOMMU_Tracks_VA-GFN.png.html "Enlarge"){.internal} +::: + +**Figure 20:** Type 1 IOMMU Tracks \<VA, GFN\>. See slides from: +[\[1\]](index.php/Mediated_Device_Internals.html#References_(Talks_&_Reading_Material)){.external +.text rel="nofollow"} +::: +::: +::: + +**Figure 20:** Type 1 IOMMU tracks \<VA, GFN (Guest Frame Number)\>. We +build a table to list the QEMU VAs and track the their mapping relation +to Guest Frame Numbers (GFNs). + +### [Scheduling]{#Scheduling .mw-headline} + +Scheduling is handled by the host mdev driver. + +### [Kernel API]{#Kernel_API .mw-headline} + +Kernel documentation used for implementing a VFIO Mediated Device may be +found at +[kernel.org](https://www.kernel.org/doc/html/latest/driver-api/vfio-mediated-device.html){.external +.text rel="nofollow"}. + +### [Sample Code]{#Sample_Code .mw-headline} + +Sample code for various mdev implementations may be found below: + +**Mediated Virtual PCI Display Host Device:** + +1. [mdpy.c](https://github.com/Open-IOV/VFIO-Mdev_Samples/blob/master/mdpy.c){.external + .text rel="nofollow"} +2. [mdpy-fb.c](https://github.com/Open-IOV/VFIO-Mdev_Samples/blob/master/mdpy-fb.c){.external + .text rel="nofollow"} +3. [mbochs.c](https://github.com/Open-IOV/VFIO-Mdev_Samples/blob/master/mbochs.c){.external + .text rel="nofollow"}. + +**Serial PCI Port-based Mediated Device:** + +1. [mtty.c](https://github.com/Open-IOV/VFIO-Mdev_Samples/blob/master/mtty.c){.external + .text rel="nofollow"}. + +###### [Compilation]{#Compilation .mw-headline} + +To compile the kernel modules linked above you should have your +distro\'s equivalent of the build-essential package installed. The +included Makefile should provide all that is necessary to successfully +compile the kernel modules. You can type the following command to +compile the modules from within the directory: + +`make` + +When the make operation has been completed successfully the directory +will now contain .ko files. These files are the binary kernel modules. + +###### [Loading Kernel Modules]{#Loading_Kernel_Modules .mw-headline} + +Now that you have compiled the kernel modules you may load them via the +insmod command. + +For example to load the **mtty.ko** module run the following command +from within the directory where you built the modules: + +`insmod mtty.ko` + +###### [Unloading Kernel Modules]{#Unloading_Kernel_Modules .mw-headline} + +To unload any of the kernel modules you may make use of the rmmod +command. + +For example to unload the **mtty.ko** module run the following command: + +`rmmod mtty.ko` + +###### [Additional Documentation]{#Additional_Documentation .mw-headline} + +An additional guide explaining how to make use of the mtty.c sample code +may be found at [line +308](https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core.git/tree/Documentation/driver-api/vfio-mediated-device.rst?h=driver-core-next&id=7de3697e9cbd4bd3d62bafa249d57990e1b8f294#n308){.external +.text rel="nofollow"} of the kernel.org VFIO Mediated Device +documentation. + +### [Mdev Mode Requirements]{#Mdev_Mode_Requirements .mw-headline} + +Driver Support. + +Software HPA\<-\>GPA Boundary Enforcement. + +## [SR-IOV Mode]{#SR-IOV_Mode .mw-headline} + +**SR-IOV Mode (Single Root I/O Virtualization)** involves hardware +assisted virtualization on I/O peripherals. + +### [Knowledge Resources Used]{#Knowledge_Resources_Used_3 .mw-headline} + +This section is supported by significant contributions in open source by +Zheng Xiao, Jerry Jiang, & Ken Xue. + +See reference 6 in the [References (Talks & Reading +Material)](index.php/Virtual_IO_Internals.html#References_(Talks_&_Reading_Material)){.external +.text rel="nofollow"} section. + +### [Instruction Execution]{#Instruction_Execution_2 .mw-headline} + +SR-IOV communicates instructions from a virtual function (VF) directly +to the [PCI +BAR](https://infogalactic.com/info/PCI_configuration_space){.external +.text rel="nofollow"}. + +### [Memory Management]{#Memory_Management_2 .mw-headline} + +Guests are presenting with passthrough memory regions by the device +firmware. + +### [Scheduling]{#Scheduling_2 .mw-headline} + +Scheduling may be handled by the host mdev driver and/or the device +firmware. + +### [SR-IOV Mode Requirements]{#SR-IOV_Mode_Requirements .mw-headline} + +Driver Support. + +Device SR-IOV support. + +Firmware HPA\<-\>GPA Boundary Enforcement. + +## [SIOV Mode]{#SIOV_Mode .mw-headline} + +**SIOV (Scalable I/O Virtualization)** involves the combination of +concepts form both [SR-IOV +Mode](index.php/Mediated_Device_Internals.html#SR-IOV_Mode){.external +.text rel="nofollow"} and [Mdev +Mode](index.php/Mediated_Device_Internals.html#Mdev_Mode){.external +.text rel="nofollow"} as well as novel concepts like shared IOMMU aware +buffers and offloading of PCI config space VM-exits (slow path) to a +discrete controller. + +**Revision 1.0 of the SIOV specification** can be read on the [Open +Compute Project +website](https://www.opencompute.org/documents/ocp-scalable-io-virtualization-technical-specification-revision-1-v1-2-pdf){.external +.text rel="nofollow"}. + +### [Knowledge Resources Used]{#Knowledge_Resources_Used_4 .mw-headline} + +This section is supported by significant contributions in open source by +Kevin Tian, Tina Zhang, Xin Zeng, Yi Liu. + +See references 3, 4, 5, 16, 17, 18, 19, 20, 21 in the [References (Talks +& Reading +Material)](index.php/Virtual_IO_Internals.html#References_(Talks_&_Reading_Material)){.external +.text rel="nofollow"} section. + +### [Memory Management]{#Memory_Management_3 .mw-headline} + +Enhancements to [Intel +VT-d](https://edc.intel.com/content/www/us/en/design/ipla/software-development-platforms/client/platforms/alder-lake-desktop/12th-generation-intel-core-processors-datasheet-volume-1-of-2/002/intel-virtualization-technology-for-directed-i-o/){.external +.text rel="nofollow"} introduce new \'Scalable Mode\' to allow the +platform to assign more granular IOMMU allocations to mediated devices +([mdev memory +management](index.php/Mediated_Device_Internals.html#Memory_Management){.external +.text rel="nofollow"}). This change is referred to as an IOMMU Aware +Mediated Device. + +#### [IOMMU Aware Mediated Device]{#IOMMU_Aware_Mediated_Device .mw-headline} + +SIOV made several changes to the VFIO driver, Intel IOMMU, and Mediated +Device Framework. + +The full list of these changes can be seen in the [mailing list archive +on **lwn.net** under **(vfio/mdev: IOMMU aware mediated +device)**](https://lwn.net/ml/linux-kernel/20190222021927.13132-1-baolu.lu@linux.intel.com/){.external +.text rel="nofollow"}. + +#### [Shared Hardware Workqueues]{#Shared_Hardware_Workqueues .mw-headline} + +SIOV makes use of Shared Hardware Workqueues which may be accessed by +processes or Virtual Machines. + +[According to +**kernel.org**](https://www.kernel.org/doc/html/latest/x86/sva.html){.external +.text rel="nofollow"}: *\"In order to allow the hardware to distinguish +the context for which work is being executed in the hardware by SWQ +interface, SIOV uses Process Address Space ID (PASID), which is a 20-bit +number defined by the PCIe SIG. PASID value is encoded in all +transactions from the device. This allows the IOMMU to track I/O on a +per-PASID granularity in addition to using the PCIe Resource Identifier +(RID) which is the Bus/Device/Function.\"* + +### [SIOV Mode Requirements]{#SIOV_Mode_Requirements .mw-headline} + +Driver Support. + +Device SIOV support. + +Firmware HPA\<-\>GPA Boundary Enforcement. + +## []{#References_.28Talks_.26_Reading_Material.29}[References (Talks & Reading Material)]{#References_(Talks_&_Reading_Material) .mw-headline} + +1. [\[2016\] vGPU on KVM - A VFIO Based Framework by Neo Jia & Kirti + Wankhede](https://www.youtube.com/watch?v=Xs0TJU_sIPc){.external + .text rel="nofollow"} - + [slides](https://www.linux-kvm.org/images/5/59/02x03-Neo_Jia_and_Kirti_Wankhede-vGPU_on_KVM-A_VFIO_based_Framework.pdf){.external + .text rel="nofollow"} +2. [\[2016\] An Introduction to PCI Device Assignment with VFIO by Alex + Williamson](https://www.youtube.com/watch?v=WFkdTFTOTpA){.external + .text rel="nofollow"} - + [slides](http://events17.linuxfoundation.org/sites/events/files/slides/An%20Introduction%20to%20PCI%20Device%20Assignment%20with%20VFIO%20-%20Williamson%20-%202016-08-30_0.pdf){.external + .text rel="nofollow"} +3. [\[2017\] Scalable I/O Virtualization by Kevin + Tian](https://events19.linuxfoundation.cn/wp-content/uploads/2017/11/Intel%C2%AE-Scalable-I_O-Virtualization_Kevin-Tian.pdf){.external + .text rel="nofollow"} +4. [\[2019\] Bring a Scalable IOV Capable Device into Linux World by + Xin Zeng & Yi + Liu](https://www.youtube.com/watch?v=G6D-jaCs6sc){.external .text + rel="nofollow"} - + [slides](https://static.sched.com/hosted_files/kvmforum2019/5e/Bring%20a%20scalable%20IOV%20capable%20device%20into%20Linux%20-%20KVM%202019.pdf){.external + .text rel="nofollow"} +5. [Scalable I/O Virtualization Revision + 1.0](https://www.opencompute.org/documents/ocp-scalable-io-virtualization-technical-specification-revision-1-v1-2-pdf){.external + .text rel="nofollow"} +6. [\[2018\] Live Migration Support for GPU with SRIOV by Zheng Xiao, + Jerry Jiang & Ken + Xue](https://www.youtube.com/watch?v=_tB3EbFDcRQ){.external .text + rel="nofollow"} - + [slides](https://events19.linuxfoundation.org/wp-content/uploads/2017/12/Live-Migration-Support-for-GPU-with-SRIOV-Challenges-and-Solution-Zheng-Xiao-Alibaba-Cloud-Jerry-Jiang-Ken-Xue-AMD.pdf){.external + .text rel="nofollow"} +7. [\[2017\] Intel GVT-g: From Production to Upstream - Zhi Wang, + Intel](https://www.youtube.com/watch?v=UODxW1opfn0){.external .text + rel="nofollow"} +8. [\[2016\] Qemu Graphics Update 2016 by Gerd + Hoffmann](https://www.youtube.com/watch?v=DKYvQ3FdFeo){.external + .text rel="nofollow"} +9. [IOCTL](https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core.git/tree/Documentation/driver-api/ioctl.rst){.external + .text rel="nofollow"} +10. [eventfd](https://man7.org/linux/man-pages/man2/eventfd.2.html){.external + .text rel="nofollow"} - + [root/virt/kvm/eventfd.c](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/virt/kvm/eventfd.c){.external + .text rel="nofollow"} +11. (kernel diff:: [KVM: + irqfd](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=721eecbf4fe995ca94a9edec0c9843b1cc0eaaf3){.external + .text rel="nofollow"}) +12. (kernel diff:: [KVM: add ioeventfd + support](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d34e6b175e61821026893ec5298cc8e7558df43a){.external + .text rel="nofollow"}) +13. [KVM irqfd & + ioeventfd](https://web.archive.org/web/20220120223711/http://blog.allenx.org/2015/07/05/kvm-irqfd-and-ioeventfd){.external + .text rel="nofollow"} +14. [VFIO - Virtual Function + I/O](https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core.git/tree/Documentation/driver-api/vfio.rst){.external + .text rel="nofollow"} - + [root/virt/kvm/vfio.c](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/virt/kvm/vfio.c){.external + .text rel="nofollow"} +15. [VFIO Mediated + Devices](https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core.git/tree/Documentation/driver-api/vfio-mediated-device.rst){.external + .text rel="nofollow"} +16. [IOMMU Aware Mediated + Device](https://lwn.net/ml/linux-kernel/20190222021927.13132-1-baolu.lu@linux.intel.com/){.external + .text rel="nofollow"} +17. [Hardware-Assisted Mediated Pass-Through with VFIO by Kevin + Tian](https://www.youtube.com/watch?v=95KSKrZM8oQ){.external .text + rel="nofollow"} +18. [\[2017\] Generic Buffer Sharing Mechanism for Mediated Devices by + Tina Zhang](https://www.youtube.com/watch?v=cHMLBcHplhk){.external + .text rel="nofollow"} +19. [\[2019\] Toward a Virtualization World Built on Mediated + Pass-Through - Kevin + Tian](https://www.youtube.com/watch?v=KWRKx_uxUDI){.external .text + rel="nofollow"} +20. [Auxiliary + Bus](https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core.git/tree/Documentation/driver-api/auxiliary_bus.rst?h=driver-core-next&id=7de3697e9cbd4bd3d62bafa249d57990e1b8f294){.external + .text rel="nofollow"} +21. [Shared Virtual + Addressing](https://www.kernel.org/doc/html/latest/x86/sva.html){.external + .text rel="nofollow"} +22. [vfio.blogspot.com](https://vfio.blogspot.com/){.external .text + rel="nofollow"} +23. [XenGT by Kevin + Tian](https://01.org/sites/default/files/xengt.pdf){.external .text + rel="nofollow"} +24. [VMWare Academic Publications: GPU Virtualization by Micah Dowty & + Jeremy + Sugerman](https://web.archive.org/web/20130721094438/http://labs.vmware.com/academic/publications/gpu-virtualization){.external + .text rel="nofollow"} +25. [PCI SIG I/O + Virtualization](https://pcisig.com/specifications/iov){.external + .text rel="nofollow"} +26. [VMGL by the University of Toronto Computer Science + Department](https://web.archive.org/web/20120108034526/http://sysweb.cs.toronto.edu/vmgl){.external + .text rel="nofollow"} +27. [Kepler VGX + Hypervisor](https://web.archive.org/web/20120518125815/http://www.nvidia.com/object/vgx-hypervisor.html){.external + .text rel="nofollow"} +28. [Intel Virtualization Technology for Directed I/O (VT-d): Enhancing + Intel platforms for efficient virtualization of I/O + devices](https://web.archive.org/web/20090218010221/http://software.intel.com/en-us/articles/intel-virtualization-technology-for-directed-io-vt-d-enhancing-intel-platforms-for-efficient-virtualization-of-io-devices){.external + .text rel="nofollow"} +29. [DMA-Mapping.txt (Dynamic DMA + Mapping)](https://lwn.net/2001/0712/a/dma-interface.php3){.external + .text rel="nofollow"} +30. [KVM MMU Virtualization by Xiao + Guangrong](https://events.static.linuxfound.org/slides/2011/linuxcon-japan/lcj2011_guangrong.pdf){.external + .text rel="nofollow"} +31. [OSDEV: PCI](https://wiki.osdev.org/PCI){.external .text + rel="nofollow"} +32. [Down to the TLP: How PCI express devices talk (Part + I)](http://xillybus.com/tutorials/pci-express-tlp-pcie-primer-tutorial-guide-1){.external + .text rel="nofollow"} +33. [Down to the TLP: How PCI express devices talk (Part + II)](http://xillybus.com/tutorials/pci-express-tlp-pcie-primer-tutorial-guide-2){.external + .text rel="nofollow"} +34. [Intel ACRN: Enabling SR-IOV + Virtualization](https://projectacrn.github.io/latest/tutorials/sriov_virtualization.html){.external + .text rel="nofollow"} +35. [Kernel.org + sysfs-bus-pci](https://docs.kernel.org/admin-guide/abi-testing.html#abi-file-testing-sysfs-bus-pci){.external + .text rel="nofollow"} +36. [Linux Kernel Labs: Memory + Mapping](https://linux-kernel-labs.github.io/refs/heads/master/labs/memory_mapping.html){.external + .text rel="nofollow"} +37. [Inside Windows Page Frame Number (PFN) - Part + 1](https://rayanfam.com/topics/inside-windows-page-frame-number-part1/){.external + .text rel="nofollow"} +38. [Direct Rendering Infrastructure + (DRI)](https://en.wikipedia.org/wiki/Direct_Rendering_Infrastructure){.external + .text rel="nofollow"} +39. [DRI User + Guide](https://dri.sourceforge.net/doc/DRIuserguide.html){.external + .text rel="nofollow"} +40. [Linux Kernel Labs: IO + Virtualization](https://linux-kernel-labs.github.io/refs/heads/master/lectures/virt.html#i-o-virtualization){.external + .text rel="nofollow"} +41. [Gallium3D: Main + Page](https://dri.freedesktop.org/doxygen/gallium/){.external .text + rel="nofollow"} +42. [Gallium3D + Wiki](https://web.archive.org/web/20080107043445/http://www.tungstengraphics.com/wiki/index.php/Gallium3D){.external + .text rel="nofollow"} +43. [Gallium3D talk from XDS + 2007](https://web.archive.org/web/20090219182518/http://www.tungstengraphics.com/wiki/files/gallium3d-xds2007.pdf){.external + .text rel="nofollow"} +44. [Memory Management High Level Design + (ARCN)](https://projectacrn.github.io/latest/developer-guides/hld/hv-memmgt.html){.external + .text rel="nofollow"} +45. [Block Architecture Diagrams for Geforce + series](https://web.archive.org/web/20080111122628/http://www.digit-life.com/articles2/gffx/nv40-part1-a.html){.external + .text rel="nofollow"} +46. [Block diagrams for 40 series instruction + format](http://freenv.svn.sourceforge.net/viewvc/freenv/doc/shaderinsnformat/bitdiagen/){.external + .text rel="nofollow"} +47. [PCI Configuration + Space](https://en.wikipedia.org/wiki/PCI_configuration_space "wikipedia:PCI configuration space"){.extiw} +48. [GPU virtual memory in WDDM + 2.0](https://learn.microsoft.com/en-us/windows-hardware/drivers/display/gpu-virtual-memory-in-wddm-2-0){.external + .text rel="nofollow"} +49. [Intel Graphics Programmer\'s Reference Manuals + (PRM)](https://01.org/linuxgraphics/documentation/hardware-specification-prms){.external + .text rel="nofollow"} +50. [i915: Hardware Contexts (and some bits about + batchbuffers)](https://bwidawsk.net/blog/2013/1/i915-hardware-contexts-and-some-bits-about-batchbuffers/){.external + .text rel="nofollow"} +51. [i915: The Global GTT Part + 1](https://bwidawsk.net/blog/2014/6/the-global-gtt-part-1/){.external + .text rel="nofollow"} +52. [i915: Aliasing PPGTT Part + 2](https://bwidawsk.net/blog/2014/6/aliasing-ppgtt-part-2/){.external + .text rel="nofollow"} +53. [i915: True PPGTT Part + 3](https://bwidawsk.net/blog/2014/7/true-ppgtt-part-3/){.external + .text rel="nofollow"} +54. [i915: Future PPGTT Part 4 (Dynamic page table allocations, 64 bit + address space, GPU \"mirroring\", and yeah, something about relocs + too)](https://bwidawsk.net/blog/2014/7/future-ppgtt-part-4-dynamic-page-table-allocations-64-bit-address-space-gpu-mirroring-and-yeah-something-about-relocs-too/){.external + .text rel="nofollow"} +55. [i915: Security of the Intel Graphics Stack - Part 1 - + Introduction](https://igor-blue.github.io/2021/02/10/graphics-part1.html){.external + .text rel="nofollow"} +56. [i915: Security of the Intel Graphics Stack - Part 2 - FW \<-\> + GuC](https://igor-blue.github.io/2021/02/24/graphics-part2.html){.external + .text rel="nofollow"} +57. [A deep dive into QEMU: PCI slave + devices](https://airbus-seclab.github.io/qemu_blog/pci_slave.html){.external + .text rel="nofollow"} +58. [Debugging QEMU Guests with GDB (start & stop, examine state like + registers & memory, set breakpoints & + watchpoints)](https://qemu-project.gitlab.io/qemu/system/gdb.html){.external + .text rel="nofollow"} +59. [IOMMU + Introduction](https://terenceli.github.io/%E6%8A%80%E6%9C%AF/2019/08/04/iommu-introduction){.external + .text rel="nofollow"} +60. [Arch Wiki: Intel + Graphics](https://wiki.archlinux.org/title/intel_graphics){.external + .text rel="nofollow"} +61. [An Introduction to Intel Graphics Virtualization Technology (legacy + GVT-g) by Zhi + Wang](https://01.org/sites/default/files/documentation/an_introduction_to_intel_gvt-g_for_external.pdf){.external + .text rel="nofollow"} +62. [Kernel.org: vfio iommu type1: Add support for mediated + devices](https://patchwork.kernel.org/project/qemu-devel/patch/1478293856-8191-11-git-send-email-kwankhede@nvidia.com/){.external + .text rel="nofollow"} +63. [Hardware and Compute Abstraction Layers For Accelerated Computing + Using Graphics Hardware and Conventional + CPUs](https://archive.ll.mit.edu/HPEC/agendas/proc07/Day3/10_Hensley_Abstract.pdf){.external + .text rel="nofollow"} +64. [More Shadow Walker: The Progression of TLB-Splitting on + x86](https://www.blackhat.com/docs/us-14/materials/us-14-Torrey-MoRE-Shadow-Walker-The-Progression-Of-TLB-Splitting-On-x86.pdf){.external + .text rel="nofollow"} +65. [MMU Virtualization Via Intel EPT: Technical + Details](https://revers.engineering/mmu-ept-technical-details/){.external + .text rel="nofollow"} +66. [Alex Williamson Github: VFIO Development (tree + next)](https://github.com/awilliam/linux-vfio/tree/next){.external + .text rel="nofollow"} +67. [GPU PMU: performance monitoring with perf + event](https://events.static.linuxfound.org/slides/2011/linuxcon-japan/lcj2011_linming.pdf){.external + .text rel="nofollow"} +68. [Remote Procedure Call + (RPC)](https://book.systemsapproach.org/e2e/rpc.html){.external + .text rel="nofollow"} +69. [Virtual Open Systems: API + Remoting](http://www.virtualopensystems.com/en/products/api-remoting/){.external + .text rel="nofollow"} +70. [A Full GPU Virtualization Solution with Mediated Pass-Through by + Yiying Zhang, David Cowperthwaite, Kun Tian, and Yaozu + Dong](https://www.usenix.org/conference/atc14/technical-sessions/presentation/tian){.external + .text rel="nofollow"} - + [\[Paper\]](https://www.usenix.org/system/files/conference/atc14/atc14-paper-tian.pdf){.external + .text rel="nofollow"} - + [\[Video\]](https://www.youtube.com/watch?v=vvYmDQKZ6MQ){.external + .text rel="nofollow"} - [\[Slides + 1\]](https://www.usenix.org/sites/default/files/conference/protected-files/atc14_slides_tian.pdf){.external + .text rel="nofollow"} - [\[Slides + 2\]](https://cseweb.ucsd.edu/~yiying/cse291j-winter20/reading/GPU-Virtualization.pdf){.external + .text rel="nofollow"} +71. [\[2014\] KvmGT: A Full GPU Virtualization Solution by Jike + Song](https://www.youtube.com/watch?v=-iuIu7_GuEo){.external .text + rel="nofollow"} +72. [docs.kernel.org: DRM Memory + Management](https://docs.kernel.org/gpu/drm-mm.html){.external .text + rel="nofollow"} +73. [docs.kernel.org: PCI Bus + Subsystem](https://docs.kernel.org/PCI/index.html){.external .text + rel="nofollow"} +74. [openglbook.com: What is + OpenGL?](https://openglbook.com/chapter-0-preface-what-is-opengl.html){.external + .text rel="nofollow"} +75. [\[2020\] Getting pixels on screen: introduction to Kernel Mode + Setting (KMS) by Simon + Ser](https://yewtu.be/watch?v=haes4_Xnc5Q){.external .text + rel="nofollow"} - + [slides](https://fs.emersion.fr/protected/presentations/present.html?src=kms-foss-north/index.md#1){.external + .text rel="nofollow"} +76. [Dynamic Mediation for Live Migration + VFIO-PCI](https://sdic.sjtu.edu.cn/wp-content/uploads/2019/12/Mediated-Pass-Through-MPT-NVIDIA-vGPU.pptx){.external + .text rel="nofollow"} +77. [DRM Management Via + CGroups](https://lwn.net/Articles/746127/){.external .text + rel="nofollow"} +78. [DRM: Add GEM (\"graphics execution manager\") to i915 + driver.](https://lwn.net/Articles/292583/){.external .text + rel="nofollow"} +::: + +::: printfooter +Retrieved from +\"[https://open-iov.org/index.php?title=Virtual_I/O_Internals&oldid=23792](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html){dir="ltr"}\" +::: +::: + +::: {#catlinks .catlinks .catlinks-allhidden mw="interface"} +::: +::: +::: + +::: {#mw-navigation} +## Navigation menu + +::: {#mw-head} +### Personal tools {#p-personal-label .vector-menu-heading} + +::: vector-menu-content +- [[Log + in](https://open-iov.org/index.php?title=Special:UserLogin&returnto=Virtual+I%2FO+Internals&returntoquery=oldid%3D23792 "You are encouraged to log in; however, it is not mandatory [o]"){accesskey="o"}]{#pt-login} +::: + +::: {#left-navigation} +### Namespaces {#p-namespaces-label .vector-menu-heading} + +::: vector-menu-content +- [[Page](index.php/Virtual_I/O_Internals.html "View the content page [c]"){accesskey="c"}]{#ca-nstab-main} +- [[Discussion](https://open-iov.org/index.php?title=Talk:Virtual_I/O_Internals&action=edit&redlink=1 "Discussion about the content page (page does not exist) [t]"){rel="discussion" + accesskey="t"}]{#ca-talk} +::: + +### Variants [expanded]{.vector-menu-checkbox-expanded} [collapsed]{.vector-menu-checkbox-collapsed} {#p-variants-label .vector-menu-heading} + +::: vector-menu-content +::: +::: + +::: {#right-navigation} +### Views {#p-views-label .vector-menu-heading} + +::: vector-menu-content +- [[Read](index.php/Virtual_I/O_Internals.html)]{#ca-view} +- [[View + source](index.php%3Ftitle=Virtual_I%252FO_Internals&action=edit.html "This page is protected. + You can view its source [e]"){accesskey="e"}]{#ca-viewsource} +- [[View + history](index.php%3Ftitle=Virtual_I%252FO_Internals&action=history.html "Past revisions of this page [h]"){accesskey="h"}]{#ca-history} +::: + +### More [expanded]{.vector-menu-checkbox-expanded} [collapsed]{.vector-menu-checkbox-collapsed} {#p-cactions-label .vector-menu-heading} + +::: vector-menu-content +::: + +::: {#p-search .vector-search-box role="search"} +<div> + +### Search + +::: {#simpleSearch search-loc="header-navigation"} +::: + +</div> +::: +::: +::: + +::: {#mw-panel} +::: {#p-logo role="banner"} +[](images/archive/c/c3/20221214193624!I915_Scheduling_Events_and_Requests.png.html "Visit the main page"){.mw-wiki-logo} +::: + +### Navigation {#p-navigation-label .vector-menu-heading} + +::: vector-menu-content +- [[Main + page](images/archive/c/c3/20221214193624!I915_Scheduling_Events_and_Requests.png.html "Visit the main page [z]"){accesskey="z"}]{#n-mainpage-description} +- [[Recent + changes](https://open-iov.org/index.php/Special:RecentChanges "A list of recent changes in the wiki [r]"){accesskey="r"}]{#n-recentchanges} +- [[Random + page](index.php/Special:Random.html "Load a random page [x]"){accesskey="x"}]{#n-randompage} +- [[Help about + MediaWiki](https://www.mediawiki.org/wiki/Special:MyLanguage/Help:Contents)]{#n-help-mediawiki} +::: + +### Tools {#p-tb-label .vector-menu-heading} + +::: vector-menu-content +- [[What links + here](index.php/Special:WhatLinksHere/Virtual_I/O_Internals.html "A list of all wiki pages that link here [j]"){accesskey="j"}]{#t-whatlinkshere} +- [[Related + changes](https://open-iov.org/index.php/Special:RecentChangesLinked/Virtual_I/O_Internals "Recent changes in pages linked from this page [k]"){rel="nofollow" + accesskey="k"}]{#t-recentchangeslinked} +- [[Special + pages](index.php/Special:SpecialPages.html "A list of all special pages [q]"){accesskey="q"}]{#t-specialpages} +- [[Printable + version](javascript:print(); "Printable version of this page [p]"){rel="alternate" + accesskey="p"}]{#t-print} +- [[Permanent + link](index.php%3Ftitle=Virtual_I%252FO_Internals&oldid=23792.html "Permanent link to this revision of the page")]{#t-permalink} +- [[Page + information](index.php%3Ftitle=Virtual_I%252FO_Internals&action=info.html "More information about this page")]{#t-info} +::: +::: +::: + +- [This page was last edited on 19 June 2023, at + 17:54.]{#footer-info-lastmod} + +```{=html} +<!-- --> +``` +- [[Privacy + policy](https://open-iov.org/index.php/Open-IOV:Privacy_policy "Open-IOV:Privacy policy")]{#footer-places-privacy} +- [[About + Open-IOV](index.php/Open-IOV:About.html "Open-IOV:About")]{#footer-places-about} +- [[Disclaimers](https://open-iov.org/index.php/Open-IOV:General_disclaimer "Open-IOV:General disclaimer")]{#footer-places-disclaimer} + +```{=html} +<!-- --> +``` +- [[{srcset="resources/assets/poweredby_mediawiki_132x47.png 1.5x, resources/assets/poweredby_mediawiki_176x62.png 2x" + width="88" height="31" + loading="lazy"}](https://www.mediawiki.org/)]{#footer-poweredbyico} |