Initial commit: Open-IOV vGPU-IO complete stack

- Firmware decompilation: GSP 570.144W (28.5MB ELF, RV64IMAC), 132 decompiled C files
- Kernel modules: nvidia.ko (34MB, OpenRM), open-iov-vfio.ko (2.8MB, VFIO mdev), gsp-vgpu.ko (1.8MB, guest driver)
- Userspace: libiov.so/libvgpu.so (76KB), example_vgpu (28KB), rpc_emulator.c
- PCI config space emulation (256B with PM, MSI, MSI-X, PCIe caps)
- 12 RPC types fully mapped from firmware decompilation
- BAR trap, WPR, scheduling, notification, migration emulation
- Helper scripts: load_vgpu_stack, create_mdev, launch_qemu_vgpu, test_vgpu_stack, setup_guest_driver, unload_vgpu_stack
- Architecture documentation
- Wiki: 10 technical documents
- Git LFS for large binaries (.ko, .bin, .iso, .img, .o, ghidra repos)
- nvidia-open-gpu-kernel-modules as submodule

1 files changed, 1084 insertions, 0 deletions

diff --git a/wiki/GPU_Driver_Internals.md b/wiki/GPU_Driver_Internals.md
new file mode 100644
index 0000000..4c18fca
--- /dev/null
+++ b/wiki/GPU_Driver_Internals.md
@@ -0,0 +1,1084 @@
+::: {#mw-page-base .noprint}
+:::
+
+::: {#mw-head-base .noprint}
+:::
+
+::: {#content .mw-body role="main"}
+[]{#top}
+
+::: {#siteNotice}
+:::
+
+::: mw-indicators
+:::
+
+# GPU Driver Internals {#firstHeading .firstHeading}
+
+::: {#bodyContent .vector-body}
+::: {#siteSub .noprint}
+From Open-IOV
+:::
+
+::: {#contentSub}
+::: {.mw-revision .warningbox}
+::: {#mw-revision-info}
+Revision as of 17:53, 19 June 2023 by
+[Arthur](./index.php%3Ftitle=User:Arthur&action=edit&redlink=1.html "User:Arthur (page does not exist)"){.new
+.mw-userlink}
+[([talk](./index.php%3Ftitle=User_talk:Arthur&action=edit&redlink=1.html "User talk:Arthur (page does not exist)"){.new
+.mw-usertoollinks-talk} \|
+[contribs](index.php/Special:Contributions/Arthur.html "Special:Contributions/Arthur"){.mw-usertoollinks-contribs})]{.mw-usertoollinks}
+[([[[→‎References (Talks & Reading
+Material)](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#References_(Talks_&_Reading_Material))]{.autocomment}]{dir="auto"})]{.comment}
+:::
+
+::: {#mw-revision-nav}
+([diff](https://open-iov.org/index.php?title=GPU_Driver_Internals&diff=prev&oldid=23791 "GPU Driver Internals"))
+[← Older
+revision](https://open-iov.org/index.php?title=GPU_Driver_Internals&direction=prev&oldid=23791 "GPU Driver Internals")
+\| Latest revision (diff) \| Newer revision → (diff)
+:::
+:::
+:::
+
+::: {#contentSub2}
+:::
+
+::: {#jump-to-nav}
+:::
+
+[Jump to
+navigation](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#mw-head){.mw-jump-link}
+[Jump to
+search](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#searchInput){.mw-jump-link}
+
+::: {#mw-content-text .mw-body-content .mw-content-ltr lang="en" dir="ltr"}
+::: mw-parser-output
+This page will detail the internals of various GPU drivers for use with
+I/O Virtualization.
+
+> An absence of critical technical documentation has historically slowed
+> growth and adoption of developer ecosystems for GPU virtualization.
+>
+> This
+> [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/){.external
+> .text rel="nofollow"} licensed content can either be used with
+> attribution, or used as inspiration for new documentation, created by
+> GPU vendors for public commercial distribution as developer
+> documentation.
+>
+> Where possible, this documentation will clearly label dates and
+> versions of observed-but-not-guaranteed behaviour vs.
+> vendor-documented stable interfaces/behaviour with guarantees of
+> forward or backward compatibility.
+
+::: {#toc .toc role="navigation" aria-labelledby="mw-toc-heading"}
+::: {.toctitle lang="en" dir="ltr"}
+## Contents {#mw-toc-heading}
+
+[]{.toctogglespan}
+:::
+
+-   [[1]{.tocnumber} [Article
+    Structure]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Article_Structure)
+    -   [[1.1]{.tocnumber} [High Level
+        Architecture]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#High_Level_Architecture)
+    -   [[1.2]{.tocnumber}
+        [Initialization]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Initialization)
+    -   [[1.3]{.tocnumber}
+        [Scheduling]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Scheduling)
+        -   [[1.3.1]{.tocnumber} [In-VM
+            Scheduling]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#In-VM_Scheduling)
+        -   [[1.3.2]{.tocnumber} [Between-VM
+            Scheduling]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Between-VM_Scheduling)
+        -   [[1.3.3]{.tocnumber} [Firmware Scheduling (if
+            applicable)]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Firmware_Scheduling_(if_applicable))
+    -   [[1.4]{.tocnumber} [Memory
+        Management]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Memory_Management)
+    -   [[1.5]{.tocnumber} [Display Surface
+        Virtualization]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Display_Surface_Virtualization)
+        -   [[1.5.1]{.tocnumber} [Graphics Buffer
+            Sharing]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Graphics_Buffer_Sharing)
+-   [[2]{.tocnumber} [Vendor
+    Neutral]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Vendor_Neutral)
+    -   [[2.1]{.tocnumber} [Knowledge Resources
+        Used]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Knowledge_Resources_Used)
+    -   [[2.2]{.tocnumber} [Display Surface
+        Virtualization]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Display_Surface_Virtualization_2)
+        -   [[2.2.1]{.tocnumber} [Graphics Buffer
+            Sharing]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Graphics_Buffer_Sharing_2)
+            -   [[2.2.1.1]{.tocnumber} [Display Flow in
+                Linux]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Display_Flow_in_Linux)
+            -   [[2.2.1.2]{.tocnumber} [SR-IOV and
+                Mdev]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#SR-IOV_and_Mdev)
+            -   [[2.2.1.3]{.tocnumber} [VFs in QEMU Hypervisor
+                (VirtIO-GPU)]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#VFs_in_QEMU_Hypervisor_(VirtIO-GPU))
+            -   [[2.2.1.4]{.tocnumber}
+                [Transport]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Transport)
+                -   [[2.2.1.4.1]{.tocnumber} [QEMU Host-Side
+                    u-dma-buf]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#QEMU_Host-Side_u-dma-buf)
+                -   [[2.2.1.4.2]{.tocnumber} [QEMU Host-Side
+                    Modules]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#QEMU_Host-Side_Modules)
+            -   [[2.2.1.5]{.tocnumber} [Windows
+                VFs]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Windows_VFs)
+            -   [[2.2.1.6]{.tocnumber} [Windows Driver
+                Stack]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Windows_Driver_Stack)
+            -   [[2.2.1.7]{.tocnumber} [DVServer
+                KMD]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#DVServer_KMD)
+            -   [[2.2.1.8]{.tocnumber} [Android
+                VFs]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Android_VFs)
+-   [[3]{.tocnumber}
+    [i915]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#i915)
+    -   [[3.1]{.tocnumber} [Knowledge Resources
+        Used]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Knowledge_Resources_Used_2)
+    -   [[3.2]{.tocnumber} [High Level
+        Architecture]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#High_Level_Architecture_2)
+        -   [[3.2.1]{.tocnumber} [i915
+            Clients]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#i915_Clients)
+    -   [[3.3]{.tocnumber}
+        [Initialization]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Initialization_2)
+    -   [[3.4]{.tocnumber}
+        [Scheduling]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Scheduling_2)
+        -   [[3.4.1]{.tocnumber} [In-VM
+            Scheduling]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#In-VM_Scheduling_2)
+        -   [[3.4.2]{.tocnumber} [Guest kernel
+            (i915.ko)]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Guest_kernel_(i915.ko))
+            -   [[3.4.2.1]{.tocnumber}
+                [vExeclist]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#vExeclist)
+            -   [[3.4.2.2]{.tocnumber}
+                [vGuC]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#vGuC)
+        -   [[3.4.3]{.tocnumber} [Between-VM
+            Scheduling]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Between-VM_Scheduling_2)
+            -   [[3.4.3.1]{.tocnumber} [Host kernel
+                (i915.ko)]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Host_kernel_(i915.ko))
+                -   [[3.4.3.1.1]{.tocnumber}
+                    [Execlist]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Execlist)
+                -   [[3.4.3.1.2]{.tocnumber}
+                    [GuC]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#GuC)
+            -   [[3.4.3.2]{.tocnumber} [Device Firmware
+                (GuC)]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Device_Firmware_(GuC))
+    -   [[3.5]{.tocnumber} [Memory
+        Management]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Memory_Management_2)
+        -   [[3.5.1]{.tocnumber} [Translation
+            Tables]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Translation_Tables)
+            -   [[3.5.1.1]{.tocnumber} [GTT (Graphics Translation
+                Table)]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#GTT_(Graphics_Translation_Table))
+            -   [[3.5.1.2]{.tocnumber} [GGTT (Global Graphics
+                Translation
+                Table)]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#GGTT_(Global_Graphics_Translation_Table))
+            -   [[3.5.1.3]{.tocnumber} [PPGTT (Per Process Graphics
+                Translation
+                Table)]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#PPGTT_(Per_Process_Graphics_Translation_Table))
+            -   [[3.5.1.4]{.tocnumber} [Aliasing
+                PPGTT]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Aliasing_PPGTT)
+            -   [[3.5.1.5]{.tocnumber} [Real
+                PPGTT]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Real_PPGTT)
+    -   [[3.6]{.tocnumber} [Display Surface
+        Virtualization]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Display_Surface_Virtualization_3)
+        -   [[3.6.1]{.tocnumber}
+            [VirtIO-GPU]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#VirtIO-GPU)
+        -   [[3.6.2]{.tocnumber} [IDD (Indirect Display
+            Driver)]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#IDD_(Indirect_Display_Driver))
+        -   [[3.6.3]{.tocnumber} [Graphics Buffer
+            Sharing]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Graphics_Buffer_Sharing_3)
+-   [[4]{.tocnumber}
+    [OpenRM]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#OpenRM)
+    -   [[4.1]{.tocnumber} [Knowledge Resources
+        Used]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Knowledge_Resources_Used_3)
+    -   [[4.2]{.tocnumber} [High Level
+        Architecture]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#High_Level_Architecture_3)
+        -   [[4.2.1]{.tocnumber} [RM
+            Clients]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#RM_Clients)
+        -   [[4.2.2]{.tocnumber} [RM
+            Server]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#RM_Server)
+        -   [[4.2.3]{.tocnumber} [RM
+            API]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#RM_API)
+        -   [[4.2.4]{.tocnumber} [RM
+            Core]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#RM_Core)
+    -   [[4.3]{.tocnumber}
+        [Initialization]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Initialization_3)
+    -   [[4.4]{.tocnumber}
+        [Scheduling]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Scheduling_3)
+        -   [[4.4.1]{.tocnumber} [Command
+            Submission]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Command_Submission)
+            -   [[4.4.1.1]{.tocnumber}
+                [Runlist]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Runlist)
+        -   [[4.4.2]{.tocnumber} [In-VM
+            Scheduling]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#In-VM_Scheduling_3)
+            -   [[4.4.2.1]{.tocnumber} [Guest kernel
+                (nvidia.ko)]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Guest_kernel_(nvidia.ko))
+        -   [[4.4.3]{.tocnumber} [Between-VM
+            Scheduling]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Between-VM_Scheduling_3)
+            -   [[4.4.3.1]{.tocnumber} [Host kernel
+                (nvidia.ko)]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Host_kernel_(nvidia.ko))
+            -   [[4.4.3.2]{.tocnumber} [Host usermode (nvidia-vgpu-mgr /
+                libnvidia-vgpu.so)]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Host_usermode_(nvidia-vgpu-mgr_/_libnvidia-vgpu.so))
+            -   [[4.4.3.3]{.tocnumber}
+                [nvidia-vgpu-mgr]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#nvidia-vgpu-mgr)
+                -   [[4.4.3.3.1]{.tocnumber}
+                    [vmiop]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#vmiop)
+            -   [[4.4.3.4]{.tocnumber} [Device Firmware
+                (GSP)]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Device_Firmware_(GSP))
+    -   [[4.5]{.tocnumber} [Memory
+        Management]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Memory_Management_3)
+        -   [[4.5.1]{.tocnumber} [Programming the
+            MMU]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Programming_the_MMU)
+            -   [[4.5.1.1]{.tocnumber}
+                [AMAPLibrary]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#AMAPLibrary)
+            -   [[4.5.1.2]{.tocnumber} [Architecture
+                HALs]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Architecture_HALs)
+            -   [[4.5.1.3]{.tocnumber} [DMA from Falcon / NvRISC-V to
+                Frame Buffer Interface
+                (FBIF)]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#DMA_from_Falcon_/_NvRISC-V_to_Frame_Buffer_Interface_(FBIF))
+        -   [[4.5.2]{.tocnumber} [Translation
+            Tables]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Translation_Tables_2)
+            -   [[4.5.2.1]{.tocnumber}
+                [vmiop_gva]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#vmiop_gva)
+    -   [[4.6]{.tocnumber} [Display Surface
+        Virtualization]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Display_Surface_Virtualization_4)
+-   [[5]{.tocnumber}
+    [amdgpu]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#amdgpu)
+-   [[6]{.tocnumber} [References (Talks & Reading
+    Material)]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#References_(Talks_&_Reading_Material))
+:::
+
+## [Article Structure]{#Article_Structure .mw-headline}
+
+This article will aim to provide information about the following details
+of GPU drivers:
+
+### [High Level Architecture]{#High_Level_Architecture .mw-headline}
+
+This section will detail high level architectures of each driver.
+
+### [Initialization]{#Initialization .mw-headline}
+
+This section will cover how the GPU\'s embedded components, the GPU
+driver, and virtualization functions are initialized.
+
+### [Scheduling]{#Scheduling .mw-headline}
+
+This section will detail how the device schedules instructions for
+execution.
+
+This will attempt to provide a comprehensive view of scheduling from
+virtualized processes down to execution within the device.
+
+#### [In-VM Scheduling]{#In-VM_Scheduling .mw-headline}
+
+The In-VM Scheduling section will detail how instructions scheduled
+within the virtual machine\'s GPU device driver.
+
+#### [Between-VM Scheduling]{#Between-VM_Scheduling .mw-headline}
+
+The Between-VM Scheduling section will detail how the host kernel module
+and/or virtual GPU helper functions handle scheduling / context swaps
+between virtual machines on a computer system.
+
+#### []{#Firmware_Scheduling_.28if_applicable.29}[Firmware Scheduling (if applicable)]{#Firmware_Scheduling_(if_applicable) .mw-headline}
+
+The Firmware Scheduling section will cover the GPU\'s internal
+scheduling model if a deferred execution pathway is used (like i915\'s
+GuC or OpenRM\'s GSP).
+
+In the case an intermediate scheduling microcontroller is not used this
+section may be less applicable (ie: vExeclist scheduling under Intel
+vGPUs).
+
+### [Memory Management]{#Memory_Management .mw-headline}
+
+The Memory Management section will cover how the GPU driver manages
+memory for virtual GPUs and host processes.
+
+This will aim detail paging abstractions used for global memory
+translation, and per-process memory translation.
+
+### [Display Surface Virtualization]{#Display_Surface_Virtualization .mw-headline}
+
+The Display Surface Virtualization section will detail how virtual
+displays are provided to guests and/or graphics rendering buffers (pixel
+surfaces) are shared from guest to host if such functions are provided
+via the driver.
+
+#### [Graphics Buffer Sharing]{#Graphics_Buffer_Sharing .mw-headline}
+
+This section will cover methods provided by the GPU driver suitable for
+high performance graphics sharing.
+
+## [Vendor Neutral]{#Vendor_Neutral .mw-headline}
+
+This section will detail functions in common between GPU drivers.
+
+Also see [Display Virtualization for Windows
+OS](https://github.com/intel/Display-Virtualization-for-Windows-OS){.external
+.text rel="nofollow"}.
+
+### [Knowledge Resources Used]{#Knowledge_Resources_Used .mw-headline}
+
+This section is supported by significant contributions to documentation
+and open source by **Satyeshwar Singh**.
+
+### [Display Surface Virtualization]{#Display_Surface_Virtualization_2 .mw-headline}
+
+The Display Surface Virtualization section will detail how virtual
+displays are provided to guests and/or graphics rendering buffers (pixel
+surfaces) are shared from guest to host if such functions are provided
+via the driver.
+
+#### [Graphics Buffer Sharing]{#Graphics_Buffer_Sharing_2 .mw-headline}
+
+This section will deal with graphics buffer sharing for vendor neutral
+GPUs.
+
+##### [Display Flow in Linux]{#Display_Flow_in_Linux .mw-headline}
+
+Userspace applications provide their buffers to compositor.
+
+Compositor asks Mesa to create a framebuffer.
+
+Framebuffer is allocated through vendor driver.
+
+Vendor driver flips the framebuffer on the screen.
+
+##### [SR-IOV and Mdev]{#SR-IOV_and_Mdev .mw-headline}
+
+SR-IOV VFs don\'t have access to the display controller (only PF does).
+
+Known issue is displaying a VF\'s framebuffer on the screen.
+
+##### []{#VFs_in_QEMU_Hypervisor_.28VirtIO-GPU.29}[VFs in QEMU Hypervisor (VirtIO-GPU)]{#VFs_in_QEMU_Hypervisor_(VirtIO-GPU) .mw-headline}
+
+SR-IOV and Mdev VFs run in QEMU Hypervisor.
+
+QEMU has full access to all pages in VF\'s address space.
+
+VirtIO-GPU allows for allocating buffers (not API remoting in this
+case).
+
+Mesa\'s KMSRO allocates framebuffer via virtio-gpu.
+
+Mesa\'s KMSRO asks vendor driver to import framebuffer (no specific
+compositor dependancy).
+
+##### [Transport]{#Transport .mw-headline}
+
+A Scatter Gather List (SGL) of physical pages is constructed by
+VirtIO-GPU of VF.
+
+###### [QEMU Host-Side u-dma-buf]{#QEMU_Host-Side_u-dma-buf .mw-headline}
+
+Host QEMU uses u-dma-buf driver to reconstruct a virtual memory pointer
+from the SGL shared by VF.
+
+u-dma-buf driver allocates contiguous memory blocks in kernel as DMA
+buffers.
+
+###### [QEMU Host-Side Modules]{#QEMU_Host-Side_Modules .mw-headline}
+
+Once virtio-qemu has a DMA buffer, it shares it with the QEMU UI.
+
+QEMU UI has several toolkits to support like GTK, SDL, etc. with GTK.
+being the default.
+
+GTK uses EGL and passes it the DMABUF as a texture.
+
+##### [Windows VFs]{#Windows_VFs .mw-headline}
+
+Windows OS doesn\'t support DMABUF capability so we can\'t allocate
+buffers from VirtIO-GPU and share them with the Windows Kernel Mode
+Driver (KMD).
+
+Windows OS is different in the way that the FB is allocated via the OS
+rather than the miniport driver.
+
+Modified version of RedHat virtio-gpu-do (short for Display Only) driver
+used.
+
+##### [Windows Driver Stack]{#Windows_Driver_Stack .mw-headline}
+
+OS (DWM) allocates frame buffers and asks GPU to write to them through
+Miniport.
+
+DVServer UMD (IDD) asks the OS for the FB copy.
+
+DVServer UMD (IDD) passes the virtual memory address to DVServer KMD.
+
+##### [DVServer KMD]{#DVServer_KMD .mw-headline}
+
+DVServer KMD finds the SGL (Scatter Gather List) of physical pages for
+this virtual memory address.
+
+This SGL is passed via virt-queue from the VF to the host QEMU.
+
+##### [Android VFs]{#Android_VFs .mw-headline}
+
+Android uses Linux kernel underneath. VirtIO-GPU is present in
+Android\'s kernel.
+
+KMSRO patch to allocate framebuffer via VirtIO-GPU also ported over to
+Android.
+
+Need modifications in the minigbm lib of Android to connect with KMSRO
+part.
+
+Rest of stack looks identical to Linux VF/PF.
+
+## [i915]{#i915 .mw-headline}
+
+### [Knowledge Resources Used]{#Knowledge_Resources_Used_2 .mw-headline}
+
+This section is supported by significant contributions to documentation
+and open source by **Zhi Wang**, **Ben Widawsky**, and **Igor
+Bogdanov**.
+
+See references 2, 3, 4, 5, 6, 7, 8, and 9 in the [References (Talks &
+Reading
+Material)](index.php/GPU_Driver_Internals.html#References_(Talks_&_Reading_Material)){.external
+.text rel="nofollow"} section.
+
+### [High Level Architecture]{#High_Level_Architecture_2 .mw-headline}
+
+#### [i915 Clients]{#i915_Clients .mw-headline}
+
+Processes which make use of the Intel i915 driver receive an i915 Client
+ID.
+
+### [Initialization]{#Initialization_2 .mw-headline}
+
+During initialization of the i915 driver the GuC binary blob is
+offloaded into the Graphics Translation Table (GTT). This allows the GuC
+to read GTT-loaded binary blob from shared framebuffer memory so that it
+may boot.
+
+### [Scheduling]{#Scheduling_2 .mw-headline}
+
+::: {.thumb .tright}
+::: {.thumbinner style="width:302px;"}
+[![Figure 0: i915 vGPU
+Scheduling](images/thumb/c/c2/Figure_0-_i915_vGPU_Scheduling.png/300px-Figure_0-_i915_vGPU_Scheduling.png){.thumbimage
+decoding="async" width="300" height="227"
+srcset="images/thumb/c/c2/Figure_0-_i915_vGPU_Scheduling.png/450px-Figure_0-_i915_vGPU_Scheduling.png 1.5x, images/thumb/c/c2/Figure_0-_i915_vGPU_Scheduling.png/600px-Figure_0-_i915_vGPU_Scheduling.png 2x"}](index.php/File:Figure_0-_i915_vGPU_Scheduling.png.html){.image}
+
+::: thumbcaption
+::: magnify
+[](index.php/File:Figure_0-_i915_vGPU_Scheduling.png.html "Enlarge"){.internal}
+:::
+
+**Figure 0:** i915 vGPU high-level Scheduling. See slides from:
+[\[9\]](index.php/GPU_Driver_Internals.html#References_(Talks_&_Reading_Material)){.external
+.text rel="nofollow"}
+:::
+:::
+:::
+
+There are several abstractions for GPU virtual machine scheduling. Those
+are as follows:
+
+-   Guest kernel (i915.ko)
+-   Host kernel (i915.ko)
+-   Device Firmware (GuC)
+
+##### [In-VM Scheduling]{#In-VM_Scheduling_2 .mw-headline}
+
+##### []{#Guest_kernel_.28i915.ko.29}[Guest kernel (i915.ko)]{#Guest_kernel_(i915.ko) .mw-headline}
+
+###### [vExeclist]{#vExeclist .mw-headline}
+
+The vExeclist is a method to submit commands directly to the GPU without
+the use of an intermediate microcontroller.
+
+###### [vGuC]{#vGuC .mw-headline}
+
+vGuC is a command submission interface used to process commands to the
+Intel [Graphics Microcontroller
+(GuC)](index.php/GPU_Firmware.html#GuC){.external .text rel="nofollow"}.
+
+#### [Between-VM Scheduling]{#Between-VM_Scheduling_2 .mw-headline}
+
+::: {.thumb .tright}
+::: {.thumbinner style="width:302px;"}
+[![Figure 1: i915 vGPU Scheduling Requests &
+Events](images/thumb/c/c3/I915_Scheduling_Events_and_Requests.png/300px-I915_Scheduling_Events_and_Requests.png){.thumbimage
+decoding="async" width="300" height="232"
+srcset="images/thumb/c/c3/I915_Scheduling_Events_and_Requests.png/450px-I915_Scheduling_Events_and_Requests.png 1.5x, images/thumb/c/c3/I915_Scheduling_Events_and_Requests.png/600px-I915_Scheduling_Events_and_Requests.png 2x"}](index.php/File:I915_Scheduling_Events_and_Requests.png.html){.image}
+
+::: thumbcaption
+::: magnify
+[](index.php/File:I915_Scheduling_Events_and_Requests.png.html "Enlarge"){.internal}
+:::
+
+**Figure 1:** i915 vGPU Scheduling Requests & Events. See slides from:
+[\[9\]](index.php/GPU_Driver_Internals.html#References_(Talks_&_Reading_Material)){.external
+.text rel="nofollow"}
+:::
+:::
+:::
+
+##### []{#Host_kernel_.28i915.ko.29}[Host kernel (i915.ko)]{#Host_kernel_(i915.ko) .mw-headline}
+
+Submitting commands to the GPU under i915 can take several paths. One
+pathway makes use of direct command submission without an intermediate
+micro-controller whereas the other uses an intermediate
+micro-controller. The intermediate micro-controller approach increases
+the amount of binary-blob code used and abstracts the kernel module from
+the device\'s internal scheduling model.
+
+###### [Execlist]{#Execlist .mw-headline}
+
+Execlist executes commands synchronously on the device without an
+intermediate microcontroller. This is the preferred method of executing
+commands by some driver developers because of it\'s stability and
+transparency under current i915 development.
+
+###### [GuC]{#GuC .mw-headline}
+
+GuC provides an execution pathway with an intermediate microcontroller
+providing a scheduling abstraction for Intel\'s preferred internal
+scheduling model.
+
+::: {.thumb .tright}
+::: {.thumbinner style="width:302px;"}
+[![Figure 2: Intel vGPU Scheduler request
+flow.](images/thumb/e/e0/Figure_2-_Intel_vGPU_Scheduler.png/300px-Figure_2-_Intel_vGPU_Scheduler.png){.thumbimage
+decoding="async" width="300" height="233"
+srcset="images/thumb/e/e0/Figure_2-_Intel_vGPU_Scheduler.png/450px-Figure_2-_Intel_vGPU_Scheduler.png 1.5x, images/thumb/e/e0/Figure_2-_Intel_vGPU_Scheduler.png/600px-Figure_2-_Intel_vGPU_Scheduler.png 2x"}](index.php/File:Figure_2-_Intel_vGPU_Scheduler.png.html){.image}
+
+::: thumbcaption
+::: magnify
+[](index.php/File:Figure_2-_Intel_vGPU_Scheduler.png.html "Enlarge"){.internal}
+:::
+
+**Figure 2:** i915 vGPU Scheduler request flow. See slides from:
+[\[9\]](index.php/GPU_Driver_Internals.html#References_(Talks_&_Reading_Material)){.external
+.text rel="nofollow"}
+:::
+:::
+:::
+
+##### []{#Device_Firmware_.28GuC.29}[Device Firmware (GuC)]{#Device_Firmware_(GuC) .mw-headline}
+
+### [Memory Management]{#Memory_Management_2 .mw-headline}
+
+##### [Translation Tables]{#Translation_Tables .mw-headline}
+
+###### []{#GTT_.28Graphics_Translation_Table.29}[GTT (Graphics Translation Table)]{#GTT_(Graphics_Translation_Table) .mw-headline}
+
+GPU Memory on-device is a part of a GTT or Graphics Translation Table.
+This table stores information globally for all graphics processes within
+the system. Some processes access the Global Graphics Translation Table
+(GGTT) such as
+[DRI](https://en.wikipedia.org/wiki/Direct_Rendering_Infrastructure "wikipedia:Direct Rendering Infrastructure"){.extiw}
+while other\'s receive a Per Process Graphics Translation Table (PPGTT)
+buffer based on their i915 Client ID.
+
+###### []{#GGTT_.28Global_Graphics_Translation_Table.29}[GGTT (Global Graphics Translation Table)]{#GGTT_(Global_Graphics_Translation_Table) .mw-headline}
+
+###### []{#PPGTT_.28Per_Process_Graphics_Translation_Table.29}[PPGTT (Per Process Graphics Translation Table)]{#PPGTT_(Per_Process_Graphics_Translation_Table) .mw-headline}
+
+Process-specific memory buffers are stored inside a Per Process Graphics
+Translation Table or PPGTT. This is a [GPU
+MMU](index.php/Virtual_IO_Internals.html#VRAM_Isolation_(GPU_GMMU)){.external
+.text rel="nofollow"} translated subregion or IOVA of global GPU memory
+specific to a GPU process\'s client ID.
+
+###### [Aliasing PPGTT]{#Aliasing_PPGTT .mw-headline}
+
+Aliasing PPGTT (Per Process Graphics Translation Table) refers to
+partially separated GPU resources (process context).
+
+###### [Real PPGTT]{#Real_PPGTT .mw-headline}
+
+Real PPGTT (Per Process Graphics Translation Table) refers to fully
+separated GPU resources (process context).
+
+### [Display Surface Virtualization]{#Display_Surface_Virtualization_3 .mw-headline}
+
+The Intel vGPU makes use of two modes for Display Surface
+Virtualization.
+
+-   VirtIO-GPU (Linux)
+-   Indirect Display Driver (Windows)
+
+#### [VirtIO-GPU]{#VirtIO-GPU .mw-headline}
+
+#### []{#IDD_.28Indirect_Display_Driver.29}[IDD (Indirect Display Driver)]{#IDD_(Indirect_Display_Driver) .mw-headline}
+
+The IDD ([Indirect Display
+Driver](https://learn.microsoft.com/en-us/windows-hardware/drivers/display/indirect-display-driver-model-overview){.external
+.text rel="nofollow"}) provides virtual display functions with arbitrary
+resolutions to any rendering device virtual or physical.
+
+Intel\'s IDD can be found in their [Display Virtualization for Windows
+OS](https://github.com/intel/Display-Virtualization-for-Windows-OS/){.external
+.text rel="nofollow"} repository.
+
+#### [Graphics Buffer Sharing]{#Graphics_Buffer_Sharing_3 .mw-headline}
+
+Intel\'s i915 driver provides functionality to directly map display
+memory from a [guest vGPU Virtual Function (either SR-IOV or VFIO-Mdev)
+into the host GPU\'s Physical
+Function](index.php/Merged_Drivers.html){.external .text rel="nofollow"}
+without slow memory copies or graphics compression. For users running
+GPU virtualization on their local device this results in a significant
+performance uplift compared to traditional graphics sharing
+functionality built for remote access use-cases such as VDI.
+
+Intel\'s [0copy display virtualization
+tools](https://github.com/intel/Display-Virtualization-for-Windows-OS){.external
+.text rel="nofollow"} are simple to implement as sharing does not rely
+upon an added [IVSHMEM (Inter-VM Shared Memory
+device)](https://www.qemu.org/docs/master/system/devices/ivshmem.html){.external
+.text rel="nofollow"} - rather the host directly maps the guest\'s
+display memory via [udmabuf](https://lwn.net/Articles/758903/){.external
+.text rel="nofollow"} as the buffer sharing functions are provided
+within the i915 open source driver.
+
+## [OpenRM]{#OpenRM .mw-headline}
+
+::: {.thumb .tright}
+::: {.thumbinner style="width:302px;"}
+[![Figure 3: GPU BAR to Timeshared Syhededuling via Channel
+IO](images/thumb/f/fd/Figure_3-_GPU_BAR_to_Timeshared_Syhededuling_via_Channel_IO.png/300px-Figure_3-_GPU_BAR_to_Timeshared_Syhededuling_via_Channel_IO.png){.thumbimage
+decoding="async" width="300" height="226"
+srcset="images/thumb/f/fd/Figure_3-_GPU_BAR_to_Timeshared_Syhededuling_via_Channel_IO.png/450px-Figure_3-_GPU_BAR_to_Timeshared_Syhededuling_via_Channel_IO.png 1.5x, images/thumb/f/fd/Figure_3-_GPU_BAR_to_Timeshared_Syhededuling_via_Channel_IO.png/600px-Figure_3-_GPU_BAR_to_Timeshared_Syhededuling_via_Channel_IO.png 2x"}](index.php/File:Figure_3-_GPU_BAR_to_Timeshared_Syhededuling_via_Channel_IO.png.html){.image}
+
+::: thumbcaption
+::: magnify
+[](index.php/File:Figure_3-_GPU_BAR_to_Timeshared_Syhededuling_via_Channel_IO.png.html "Enlarge"){.internal}
+:::
+
+**Figure 3:** GPU BAR to Timeshared Scheduling via Channel IO.
+:::
+:::
+:::
+
+### [Knowledge Resources Used]{#Knowledge_Resources_Used_3 .mw-headline}
+
+This section is supported by significant contributions to documentation
+and open source by **Andy Currid**, **Neo Jia**, **John Fanelli**, and
+**Neha Joshi**.
+
+### [High Level Architecture]{#High_Level_Architecture_3 .mw-headline}
+
+The Open Resource Manager (RM driver) makes use of a highly object
+oriented paradigm comprised of multiple \"engines\" which act as
+micro-services for servicing driver requests.
+
+The Open Resource Manager driver (also known as
+[OpenRM](index.php/OpenRM.html){.external .text rel="nofollow"}) refers
+to Nvidia\'s
+[open-kernel-modules](https://github.com/NVIDIA/open-gpu-kernel-modules){.external
+.text rel="nofollow"}.
+
+Broadly speaking the OpenRM driver consists of two parts.
+
+-   The Platform RM (OpenRM)
+-   The Firmware RM (GSP RM / RM Core)
+
+The Platform RM is loaded into the Linux Kernel as nvidia.ko. This
+module communicates with the GSP RM for via [Remote Procedure Calls
+(RPCs)](https://en.wikipedia.org/wiki/Remote_procedure_call "wikipedia:Remote procedure call"){.extiw}
+to communicate with engines from the RM Core.
+
+##### [RM Clients]{#RM_Clients .mw-headline}
+
+Processes (local, remote, or virtualized) which make use of the RM
+driver receive an RM Client ID.
+
+#### [RM Server]{#RM_Server .mw-headline}
+
+The RM Server (or Resource Server) tracks RM Clients as well as the
+hardware and software resources they control, allocate, and free.
+
+#### [RM API]{#RM_API .mw-headline}
+
+API to control the Resource Manager Server.
+
+#### [RM Core]{#RM_Core .mw-headline}
+
+Core functions of the RM driver controlling resource locking, mapping,
+unmapping, control calls, constructors, and deconstructors. Under OpenRM
+the RM Core runs within the GPU System Processor (GSP micro-controller)
+while in pre-open source versions of the Resource Manager the RM Core
+ran within the nvidia.ko kernel module
+
+### [Initialization]{#Initialization_3 .mw-headline}
+
+During bring up of the hardware several binary blobs are loaded from
+embedded [Boot
+ROM](https://en.wikipedia.org/wiki/Boot_ROM "wikipedia:Boot ROM"){.extiw}
+memory to bootstrap embedded controller bring up from which point
+additional software is loaded from onboard
+[SPI](https://en.wikipedia.org/wiki/Serial_Peripheral_Interface "wikipedia:Serial Peripheral Interface"){.extiw}
+flash memory.
+
+Software loaded from SPI flash is necessary for the full initialization
+of the Falcon/NvRISC processor as well as a cached version of the
+software necessary to run the GPU System Processor (GSP).
+
+Once the platform is posted it is ready to communicate with the host
+platform\'s RM driver. The OpenRM driver offloads a binary blob
+containing the RM Core to the [GPU System Processor
+(GSP)](index.php/GPU_Firmware.html#GSP){.external .text rel="nofollow"}
+which is likely to contain a more recent version than the cached version
+contained in on-board SPI flash.
+
+### [Scheduling]{#Scheduling_3 .mw-headline}
+
+There are several abstractions for GPU virtual machine scheduling. Those
+are as follows:
+
+-   Guest kernel (nvidia.ko)
+-   Host kernel (nvidia.ko)
+-   Host usermode (gpu-mgr / libnvidiavgpu.so)
+-   Device Firmware (GSP)
+
+#### [Command Submission]{#Command_Submission .mw-headline}
+
+##### [Runlist]{#Runlist .mw-headline}
+
+#### [In-VM Scheduling]{#In-VM_Scheduling_3 .mw-headline}
+
+Virtual machines contain their own GPU scheduling within the Nvidia
+kernel module in the guest OS.
+
+##### []{#Guest_kernel_.28nvidia.ko.29}[Guest kernel (nvidia.ko)]{#Guest_kernel_(nvidia.ko) .mw-headline}
+
+Within a virtual machine running the Nvidia driver messages to the GPU
+are first sent to the guest nvidia.ko kernel module.
+
+The guest then determines whether a vRPC (virtual Remote Procedure Call)
+or a pRPC (physical Remote Procedure Call) should be sent. Both pRPCs
+and vRPCs are sent through the host RM driver (Resource Manager).
+
+> *Note: Unclear on execution pathway for pRPCs vs vRPCs. pRPCs may go
+> directly to device.*
+
+#### [Between-VM Scheduling]{#Between-VM_Scheduling_3 .mw-headline}
+
+In addition to scheduling which occurs within the virtual machine the
+Resource Manager driver also schedules messages to the GPU between
+GPU-accelerated virtual machines and host processes.
+
+##### []{#Host_kernel_.28nvidia.ko.29}[Host kernel (nvidia.ko)]{#Host_kernel_(nvidia.ko) .mw-headline}
+
+Messages sent by the guest (via vRPC or pRPC) are received by the host
+Nvidia.ko driver.
+
+Nvidia.ko contains a virtual GPU state machine which contains status
+information for the virtual GPU.
+
+Nvidia.ko also contains a virtual GPU kernel scheduler which interacts
+with virtual GPU objects.
+
+Nvidia.ko also contains an RM Call scheduler which schedules calls on an
+RM class.
+
+The Nividia.ko kernel module exits to userspace to execute the
+nvidia-vgpu-mgr and VMIOP (Virtual Machine Input Output Plugin).
+
+##### []{#Host_usermode_.28nvidia-vgpu-mgr_.2F_libnvidia-vgpu.so.29}[Host usermode (nvidia-vgpu-mgr / libnvidia-vgpu.so)]{#Host_usermode_(nvidia-vgpu-mgr_/_libnvidia-vgpu.so) .mw-headline}
+
+After exiting to userspace a daemon process (the nvidia-vgpu-mgr, and
+it\'s library libnvidia-vgpu.so) are executed to schedule VM-exits
+described below.
+
+##### [nvidia-vgpu-mgr]{#nvidia-vgpu-mgr .mw-headline}
+
+The nvidia-vgpu-mgr is a process which provides the spawning of virtual
+GPU stubs and population with capability information.
+
+This daemon process interacts with the libnvidia-vgpu.so, nvidia.ko, and
+nvidia-vgpu-vfio.ko components.
+
+This process and the libnvidia-vgpu.so contain, and execute the VMIOP.
+
+This process (and the VMIOP) schedules RPCs sent by the guest, receives
+VFIO BAR-exits, and relays requests to allocate, deallocate, pin, unpin,
+map, unmap to the [RM
+Core](index.php/GPU_Driver_Internals.html#RM_Core){.external .text
+rel="nofollow"}.
+
+###### [vmiop]{#vmiop .mw-headline}
+
+VMIOP (Virtual Machine Input Output Plugin) handles presenting
+virtualized functionality into the guest.
+
+The Virtual Machine Input Output Plugin software handles virtual
+displays, compute API offload, and most importantly [BAR (Base Address
+Register)
+quirks](index.php/Virtual_I/O_Internals.html#VFIO_Quirks_(region_traps)){.external
+.text rel="nofollow"}.
+
+The VMIOP is an [SDK (Software Development
+Kit)](https://en.wikipedia.org/wiki/Software_development_kit "wikipedia:Software development kit"){.extiw}
+provided in binary format split between the libnvidiavgpu.so and
+nvidia-vgpu-mgr which provides userland helper functions for GPU
+virtualization.
+
+Messages to the VMIOP are scheduled by Linux kernel
+[niceness](https://www.learnlinux.org.za/courses/build/internals/ch07s02.html){.external
+.text rel="nofollow"} (scheduling abstraction).
+
+##### []{#Device_Firmware_.28GSP.29}[Device Firmware (GSP)]{#Device_Firmware_(GSP) .mw-headline}
+
+Messages received by the host RM driver (Resource Manager) are then
+scheduled by the RM Core contained within the GSP (GPU System
+Processor). The GSP handles the device\'s internal scheduling model.
+
+### [Memory Management]{#Memory_Management_3 .mw-headline}
+
+Managing virtual machine memory is very important to the security of
+virtualization.
+
+This section will cover the method by which secure memory \"enclaves\"
+or [IO Virtual Addresses
+(IOVAs)](https://terenceli.github.io/%E6%8A%80%E6%9C%AF/2019/08/04/iommu-introduction){.external
+.text rel="nofollow"} may be provisioned and separation enforced by
+hardware constructs within the GPU.
+
+##### [Programming the MMU]{#Programming_the_MMU .mw-headline}
+
+In order to hardware enforce separation between memory allocated to
+Virtual Machines (VMs) virtualization software must program the GPU\'s
+MMU (GMMU controller) to create IO Virtual Addresses (IOVAs).
+
+In order to create such configurations several abstractions are used to
+translate high level representations of virtualization programmed via
+the vmiop and gpu-mgr into practical, architecture specific
+instructions.
+
+###### [AMAPLibrary]{#AMAPLibrary .mw-headline}
+
+The AMAPLibrary acts as a device abstraction framework for GPU driver
+software to program using high level representations of MMU
+configuration.
+
+The AMAPLibrary translates high level representations of GPU
+virtualization into graphics architecture-specific logic contained
+within architecture HALs (Hardware Abstraction Layers).
+
+###### [Architecture HALs]{#Architecture_HALs .mw-headline}
+
+GPU [Hardware Abstraction Layers
+(HALs)](https://archive.ll.mit.edu/HPEC/agendas/proc07/Day3/10_Hensley_Abstract.pdf){.external
+.text rel="nofollow"} contain logic specific to graphics architectures,
+for instance the precise method by which the GPU driver may interact
+with the Falcon to provision MMU protected memory.
+
+###### []{#DMA_from_Falcon_.2F_NvRISC-V_to_Frame_Buffer_Interface_.28FBIF.29}[DMA from Falcon / NvRISC-V to Frame Buffer Interface (FBIF)]{#DMA_from_Falcon_/_NvRISC-V_to_Frame_Buffer_Interface_(FBIF) .mw-headline}
+
+The Falcon (FAst Logic CONtroller) / NvRISC-V embedded controller emits
+DMAs to the Frame Buffer Interface (FBIF) in order to interact with the
+device\'s GMMU controller.
+
+User programs and VMs have their memory translated through the GMMU.
+
+Once created memory translations within a virtual machine\'s IO Virtual
+Address (IOVA) will be protected by the device\'s GMMU controller.
+
+##### [Translation Tables]{#Translation_Tables_2 .mw-headline}
+
+###### [vmiop_gva]{#vmiop_gva .mw-headline}
+
+### [Display Surface Virtualization]{#Display_Surface_Virtualization_4 .mw-headline}
+
+## [amdgpu]{#amdgpu .mw-headline}
+
+## []{#References_.28Talks_.26_Reading_Material.29}[References (Talks & Reading Material)]{#References_(Talks_&_Reading_Material) .mw-headline}
+
+1.  [Intel Graphics Programmer\'s Reference Manuals
+    (PRM)](https://01.org/linuxgraphics/documentation/hardware-specification-prms){.external
+    .text rel="nofollow"}
+2.  [i915: Hardware Contexts (and some bits about
+    batchbuffers)](https://bwidawsk.net/blog/2013/1/i915-hardware-contexts-and-some-bits-about-batchbuffers/){.external
+    .text rel="nofollow"}
+3.  [i915: The Global GTT Part
+    1](https://bwidawsk.net/blog/2014/6/the-global-gtt-part-1/){.external
+    .text rel="nofollow"}
+4.  [i915: Aliasing PPGTT Part
+    2](https://bwidawsk.net/blog/2014/6/aliasing-ppgtt-part-2/){.external
+    .text rel="nofollow"}
+5.  [i915: True PPGTT Part
+    3](https://bwidawsk.net/blog/2014/7/true-ppgtt-part-3/){.external
+    .text rel="nofollow"}
+6.  [i915: Future PPGTT Part 4 (Dynamic page table allocations, 64 bit
+    address space, GPU \"mirroring\", and yeah, something about relocs
+    too)](https://bwidawsk.net/blog/2014/7/future-ppgtt-part-4-dynamic-page-table-allocations-64-bit-address-space-gpu-mirroring-and-yeah-something-about-relocs-too/){.external
+    .text rel="nofollow"}
+7.  [i915: Security of the Intel Graphics Stack - Part 1 -
+    Introduction](https://igor-blue.github.io/2021/02/10/graphics-part1.html){.external
+    .text rel="nofollow"}
+8.  [i915: Security of the Intel Graphics Stack - Part 2 - FW \<-\>
+    GuC](https://igor-blue.github.io/2021/02/24/graphics-part2.html){.external
+    .text rel="nofollow"}
+9.  [i915: An Introduction to Intel GVT-g (with new
+    architecture)](https://01.org/sites/default/files/documentation/an_introduction_to_intel_gvt-g_for_external.pdf){.external
+    .text rel="nofollow"}
+10. [lwn.net: Add udmabuf misc
+    device](https://lwn.net/Articles/758903/){.external .text
+    rel="nofollow"}
+11. [Nvidia RISC-v
+    Story](https://riscv.org/wp-content/uploads/2016/07/Tue1100_Nvidia_RISCV_Story_V2.pdf){.external
+    .text rel="nofollow"}
+12. [IOMMU
+    Introduction](https://terenceli.github.io/%E6%8A%80%E6%9C%AF/2019/08/04/iommu-introduction){.external
+    .text rel="nofollow"}
+13. [Hardware and Compute Abstraction Layers For Accelerated Computing
+    Using Graphics Hardware and Conventional
+    CPUs](https://archive.ll.mit.edu/HPEC/agendas/proc07/Day3/10_Hensley_Abstract.pdf){.external
+    .text rel="nofollow"}
+14. [nVidia GPU Introduction
+    (envytools)](https://envytools.readthedocs.io/en/latest/hw/intro.html){.external
+    .text rel="nofollow"}
+15. [Delivering High Performance Remote Graphics With Nvidia GRID
+    Virtual
+    GPU](https://on-demand.gputechconf.com/gtc/2014/presentations/S4725-hi-perf-graphics-nvidia-grid-virtual-gpus.pdf){.external
+    .text rel="nofollow"}
+16. [NVIDIA Multi-Instance
+    GPU](https://nehajoshi.dev/post/nvidia_mig_feature/){.external .text
+    rel="nofollow"}
+17. [i915: Kaby Lake Intel Graphics Programmer\'s Reference Manual
+    \[Volume 5: Memory
+    Views\]](https://01.org/sites/default/files/documentation/intel-gfx-prm-osrc-kbl-vol05-memory_views.pdf){.external
+    .text rel="nofollow"}
+18. [Lifecycle of a Triangle - Nvidia\'s logical
+    pipeline](https://developer.nvidia.com/content/life-triangle-nvidias-logical-pipeline){.external
+    .text rel="nofollow"}
+19. [Nvidia Parallel Thread Execution
+    (PTX)](https://docs.nvidia.com/cuda/parallel-thread-execution/){.external
+    .text rel="nofollow"}
+20. [i915/GEM
+    Crashcourse](https://blog.ffwll.ch/2013/01/i915gem-crashcourse-overview.html){.external
+    .text rel="nofollow"}
+21. [drm: Add GEM (\"graphics execution manager\") to i915
+    driver.](https://lwn.net/Articles/292583/){.external .text
+    rel="nofollow"}
+:::
+
+::: printfooter
+Retrieved from
+\"[https://open-iov.org/index.php?title=GPU_Driver_Internals&oldid=23791](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html){dir="ltr"}\"
+:::
+:::
+
+::: {#catlinks .catlinks .catlinks-allhidden mw="interface"}
+:::
+:::
+:::
+
+::: {#mw-navigation}
+## Navigation menu
+
+::: {#mw-head}
+### Personal tools {#p-personal-label .vector-menu-heading}
+
+::: vector-menu-content
+-   [[Log
+    in](https://open-iov.org/index.php?title=Special:UserLogin&returnto=GPU+Driver+Internals&returntoquery=oldid%3D23791 "You are encouraged to log in; however, it is not mandatory [o]"){accesskey="o"}]{#pt-login}
+:::
+
+::: {#left-navigation}
+### Namespaces {#p-namespaces-label .vector-menu-heading}
+
+::: vector-menu-content
+-   [[Page](index.php/GPU_Driver_Internals.html "View the content page [c]"){accesskey="c"}]{#ca-nstab-main}
+-   [[Discussion](https://open-iov.org/index.php?title=Talk:GPU_Driver_Internals&action=edit&redlink=1 "Discussion about the content page (page does not exist) [t]"){rel="discussion"
+    accesskey="t"}]{#ca-talk}
+:::
+
+### Variants [expanded]{.vector-menu-checkbox-expanded} [collapsed]{.vector-menu-checkbox-collapsed} {#p-variants-label .vector-menu-heading}
+
+::: vector-menu-content
+:::
+:::
+
+::: {#right-navigation}
+### Views {#p-views-label .vector-menu-heading}
+
+::: vector-menu-content
+-   [[Read](index.php/GPU_Driver_Internals.html)]{#ca-view}
+-   [[View
+    source](index.php%3Ftitle=GPU_Driver_Internals&action=edit.html "This page is protected.
+    You can view its source [e]"){accesskey="e"}]{#ca-viewsource}
+-   [[View
+    history](index.php%3Ftitle=GPU_Driver_Internals&action=history.html "Past revisions of this page [h]"){accesskey="h"}]{#ca-history}
+:::
+
+### More [expanded]{.vector-menu-checkbox-expanded} [collapsed]{.vector-menu-checkbox-collapsed} {#p-cactions-label .vector-menu-heading}
+
+::: vector-menu-content
+:::
+
+::: {#p-search .vector-search-box role="search"}
+<div>
+
+### Search
+
+::: {#simpleSearch search-loc="header-navigation"}
+:::
+
+</div>
+:::
+:::
+:::
+
+::: {#mw-panel}
+::: {#p-logo role="banner"}
+[](images/archive/c/c3/20221214193624!I915_Scheduling_Events_and_Requests.png.html "Visit the main page"){.mw-wiki-logo}
+:::
+
+### Navigation {#p-navigation-label .vector-menu-heading}
+
+::: vector-menu-content
+-   [[Main
+    page](images/archive/c/c3/20221214193624!I915_Scheduling_Events_and_Requests.png.html "Visit the main page [z]"){accesskey="z"}]{#n-mainpage-description}
+-   [[Recent
+    changes](https://open-iov.org/index.php/Special:RecentChanges "A list of recent changes in the wiki [r]"){accesskey="r"}]{#n-recentchanges}
+-   [[Random
+    page](index.php/Special:Random.html "Load a random page [x]"){accesskey="x"}]{#n-randompage}
+-   [[Help about
+    MediaWiki](https://www.mediawiki.org/wiki/Special:MyLanguage/Help:Contents)]{#n-help-mediawiki}
+:::
+
+### Tools {#p-tb-label .vector-menu-heading}
+
+::: vector-menu-content
+-   [[What links
+    here](index.php/Special:WhatLinksHere/GPU_Driver_Internals.html "A list of all wiki pages that link here [j]"){accesskey="j"}]{#t-whatlinkshere}
+-   [[Related
+    changes](https://open-iov.org/index.php/Special:RecentChangesLinked/GPU_Driver_Internals "Recent changes in pages linked from this page [k]"){rel="nofollow"
+    accesskey="k"}]{#t-recentchangeslinked}
+-   [[Special
+    pages](index.php/Special:SpecialPages.html "A list of all special pages [q]"){accesskey="q"}]{#t-specialpages}
+-   [[Printable
+    version](javascript:print(); "Printable version of this page [p]"){rel="alternate"
+    accesskey="p"}]{#t-print}
+-   [[Permanent
+    link](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html "Permanent link to this revision of the page")]{#t-permalink}
+-   [[Page
+    information](index.php%3Ftitle=GPU_Driver_Internals&action=info.html "More information about this page")]{#t-info}
+:::
+:::
+:::
+
+-   [This page was last edited on 19 June 2023, at
+    17:53.]{#footer-info-lastmod}
+
+```{=html}
+<!-- -->
+```
+-   [[Privacy
+    policy](https://open-iov.org/index.php/Open-IOV:Privacy_policy "Open-IOV:Privacy policy")]{#footer-places-privacy}
+-   [[About
+    Open-IOV](index.php/Open-IOV:About.html "Open-IOV:About")]{#footer-places-about}
+-   [[Disclaimers](https://open-iov.org/index.php/Open-IOV:General_disclaimer "Open-IOV:General disclaimer")]{#footer-places-disclaimer}
+
+```{=html}
+<!-- -->
+```
+-   [[![Powered by
+    MediaWiki](resources/assets/poweredby_mediawiki_88x31.png){srcset="resources/assets/poweredby_mediawiki_132x47.png 1.5x, resources/assets/poweredby_mediawiki_176x62.png 2x"
+    width="88" height="31"
+    loading="lazy"}](https://www.mediawiki.org/)]{#footer-poweredbyico}