path: root/wiki/GPU_Driver_Internals.md

::: {#mw-page-base .noprint}
:::

::: {#mw-head-base .noprint}
:::

::: {#content .mw-body role="main"}
[]{#top}

::: {#siteNotice}
:::

::: mw-indicators
:::

# GPU Driver Internals {#firstHeading .firstHeading}

::: {#bodyContent .vector-body}
::: {#siteSub .noprint}
From Open-IOV
:::

::: {#contentSub}
::: {.mw-revision .warningbox}
::: {#mw-revision-info}
Revision as of 17:53, 19 June 2023 by
[Arthur](./index.php%3Ftitle=User:Arthur&action=edit&redlink=1.html "User:Arthur (page does not exist)"){.new
.mw-userlink}
[([talk](./index.php%3Ftitle=User_talk:Arthur&action=edit&redlink=1.html "User talk:Arthur (page does not exist)"){.new
.mw-usertoollinks-talk} \|
[contribs](index.php/Special:Contributions/Arthur.html "Special:Contributions/Arthur"){.mw-usertoollinks-contribs})]{.mw-usertoollinks}
[([[[→‎References (Talks & Reading
Material)](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#References_(Talks_&_Reading_Material))]{.autocomment}]{dir="auto"})]{.comment}
:::

::: {#mw-revision-nav}
([diff](https://open-iov.org/index.php?title=GPU_Driver_Internals&diff=prev&oldid=23791 "GPU Driver Internals"))
[← Older
revision](https://open-iov.org/index.php?title=GPU_Driver_Internals&direction=prev&oldid=23791 "GPU Driver Internals")
\| Latest revision (diff) \| Newer revision → (diff)
:::
:::
:::

::: {#contentSub2}
:::

::: {#jump-to-nav}
:::

[Jump to
navigation](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#mw-head){.mw-jump-link}
[Jump to
search](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#searchInput){.mw-jump-link}

::: {#mw-content-text .mw-body-content .mw-content-ltr lang="en" dir="ltr"}
::: mw-parser-output
This page will detail the internals of various GPU drivers for use with
I/O Virtualization.

> An absence of critical technical documentation has historically slowed
> growth and adoption of developer ecosystems for GPU virtualization.
>
> This
> [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/){.external
> .text rel="nofollow"} licensed content can either be used with
> attribution, or used as inspiration for new documentation, created by
> GPU vendors for public commercial distribution as developer
> documentation.
>
> Where possible, this documentation will clearly label dates and
> versions of observed-but-not-guaranteed behaviour vs.
> vendor-documented stable interfaces/behaviour with guarantees of
> forward or backward compatibility.

::: {#toc .toc role="navigation" aria-labelledby="mw-toc-heading"}
::: {.toctitle lang="en" dir="ltr"}
## Contents {#mw-toc-heading}

[]{.toctogglespan}
:::

-   [[1]{.tocnumber} [Article
    Structure]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Article_Structure)
    -   [[1.1]{.tocnumber} [High Level
        Architecture]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#High_Level_Architecture)
    -   [[1.2]{.tocnumber}
        [Initialization]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Initialization)
    -   [[1.3]{.tocnumber}
        [Scheduling]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Scheduling)
        -   [[1.3.1]{.tocnumber} [In-VM
            Scheduling]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#In-VM_Scheduling)
        -   [[1.3.2]{.tocnumber} [Between-VM
            Scheduling]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Between-VM_Scheduling)
        -   [[1.3.3]{.tocnumber} [Firmware Scheduling (if
            applicable)]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Firmware_Scheduling_(if_applicable))
    -   [[1.4]{.tocnumber} [Memory
        Management]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Memory_Management)
    -   [[1.5]{.tocnumber} [Display Surface
        Virtualization]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Display_Surface_Virtualization)
        -   [[1.5.1]{.tocnumber} [Graphics Buffer
            Sharing]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Graphics_Buffer_Sharing)
-   [[2]{.tocnumber} [Vendor
    Neutral]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Vendor_Neutral)
    -   [[2.1]{.tocnumber} [Knowledge Resources
        Used]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Knowledge_Resources_Used)
    -   [[2.2]{.tocnumber} [Display Surface
        Virtualization]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Display_Surface_Virtualization_2)
        -   [[2.2.1]{.tocnumber} [Graphics Buffer
            Sharing]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Graphics_Buffer_Sharing_2)
            -   [[2.2.1.1]{.tocnumber} [Display Flow in
                Linux]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Display_Flow_in_Linux)
            -   [[2.2.1.2]{.tocnumber} [SR-IOV and
                Mdev]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#SR-IOV_and_Mdev)
            -   [[2.2.1.3]{.tocnumber} [VFs in QEMU Hypervisor
                (VirtIO-GPU)]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#VFs_in_QEMU_Hypervisor_(VirtIO-GPU))
            -   [[2.2.1.4]{.tocnumber}
                [Transport]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Transport)
                -   [[2.2.1.4.1]{.tocnumber} [QEMU Host-Side
                    u-dma-buf]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#QEMU_Host-Side_u-dma-buf)
                -   [[2.2.1.4.2]{.tocnumber} [QEMU Host-Side
                    Modules]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#QEMU_Host-Side_Modules)
            -   [[2.2.1.5]{.tocnumber} [Windows
                VFs]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Windows_VFs)
            -   [[2.2.1.6]{.tocnumber} [Windows Driver
                Stack]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Windows_Driver_Stack)
            -   [[2.2.1.7]{.tocnumber} [DVServer
                KMD]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#DVServer_KMD)
            -   [[2.2.1.8]{.tocnumber} [Android
                VFs]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Android_VFs)
-   [[3]{.tocnumber}
    [i915]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#i915)
    -   [[3.1]{.tocnumber} [Knowledge Resources
        Used]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Knowledge_Resources_Used_2)
    -   [[3.2]{.tocnumber} [High Level
        Architecture]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#High_Level_Architecture_2)
        -   [[3.2.1]{.tocnumber} [i915
            Clients]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#i915_Clients)
    -   [[3.3]{.tocnumber}
        [Initialization]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Initialization_2)
    -   [[3.4]{.tocnumber}
        [Scheduling]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Scheduling_2)
        -   [[3.4.1]{.tocnumber} [In-VM
            Scheduling]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#In-VM_Scheduling_2)
        -   [[3.4.2]{.tocnumber} [Guest kernel
            (i915.ko)]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Guest_kernel_(i915.ko))
            -   [[3.4.2.1]{.tocnumber}
                [vExeclist]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#vExeclist)
            -   [[3.4.2.2]{.tocnumber}
                [vGuC]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#vGuC)
        -   [[3.4.3]{.tocnumber} [Between-VM
            Scheduling]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Between-VM_Scheduling_2)
            -   [[3.4.3.1]{.tocnumber} [Host kernel
                (i915.ko)]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Host_kernel_(i915.ko))
                -   [[3.4.3.1.1]{.tocnumber}
                    [Execlist]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Execlist)
                -   [[3.4.3.1.2]{.tocnumber}
                    [GuC]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#GuC)
            -   [[3.4.3.2]{.tocnumber} [Device Firmware
                (GuC)]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Device_Firmware_(GuC))
    -   [[3.5]{.tocnumber} [Memory
        Management]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Memory_Management_2)
        -   [[3.5.1]{.tocnumber} [Translation
            Tables]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Translation_Tables)
            -   [[3.5.1.1]{.tocnumber} [GTT (Graphics Translation
                Table)]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#GTT_(Graphics_Translation_Table))
            -   [[3.5.1.2]{.tocnumber} [GGTT (Global Graphics
                Translation
                Table)]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#GGTT_(Global_Graphics_Translation_Table))
            -   [[3.5.1.3]{.tocnumber} [PPGTT (Per Process Graphics
                Translation
                Table)]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#PPGTT_(Per_Process_Graphics_Translation_Table))
            -   [[3.5.1.4]{.tocnumber} [Aliasing
                PPGTT]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Aliasing_PPGTT)
            -   [[3.5.1.5]{.tocnumber} [Real
                PPGTT]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Real_PPGTT)
    -   [[3.6]{.tocnumber} [Display Surface
        Virtualization]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Display_Surface_Virtualization_3)
        -   [[3.6.1]{.tocnumber}
            [VirtIO-GPU]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#VirtIO-GPU)
        -   [[3.6.2]{.tocnumber} [IDD (Indirect Display
            Driver)]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#IDD_(Indirect_Display_Driver))
        -   [[3.6.3]{.tocnumber} [Graphics Buffer
            Sharing]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Graphics_Buffer_Sharing_3)
-   [[4]{.tocnumber}
    [OpenRM]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#OpenRM)
    -   [[4.1]{.tocnumber} [Knowledge Resources
        Used]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Knowledge_Resources_Used_3)
    -   [[4.2]{.tocnumber} [High Level
        Architecture]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#High_Level_Architecture_3)
        -   [[4.2.1]{.tocnumber} [RM
            Clients]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#RM_Clients)
        -   [[4.2.2]{.tocnumber} [RM
            Server]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#RM_Server)
        -   [[4.2.3]{.tocnumber} [RM
            API]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#RM_API)
        -   [[4.2.4]{.tocnumber} [RM
            Core]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#RM_Core)
    -   [[4.3]{.tocnumber}
        [Initialization]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Initialization_3)
    -   [[4.4]{.tocnumber}
        [Scheduling]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Scheduling_3)
        -   [[4.4.1]{.tocnumber} [Command
            Submission]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Command_Submission)
            -   [[4.4.1.1]{.tocnumber}
                [Runlist]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Runlist)
        -   [[4.4.2]{.tocnumber} [In-VM
            Scheduling]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#In-VM_Scheduling_3)
            -   [[4.4.2.1]{.tocnumber} [Guest kernel
                (nvidia.ko)]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Guest_kernel_(nvidia.ko))
        -   [[4.4.3]{.tocnumber} [Between-VM
            Scheduling]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Between-VM_Scheduling_3)
            -   [[4.4.3.1]{.tocnumber} [Host kernel
                (nvidia.ko)]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Host_kernel_(nvidia.ko))
            -   [[4.4.3.2]{.tocnumber} [Host usermode (nvidia-vgpu-mgr /
                libnvidia-vgpu.so)]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Host_usermode_(nvidia-vgpu-mgr_/_libnvidia-vgpu.so))
            -   [[4.4.3.3]{.tocnumber}
                [nvidia-vgpu-mgr]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#nvidia-vgpu-mgr)
                -   [[4.4.3.3.1]{.tocnumber}
                    [vmiop]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#vmiop)
            -   [[4.4.3.4]{.tocnumber} [Device Firmware
                (GSP)]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Device_Firmware_(GSP))
    -   [[4.5]{.tocnumber} [Memory
        Management]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Memory_Management_3)
        -   [[4.5.1]{.tocnumber} [Programming the
            MMU]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Programming_the_MMU)
            -   [[4.5.1.1]{.tocnumber}
                [AMAPLibrary]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#AMAPLibrary)
            -   [[4.5.1.2]{.tocnumber} [Architecture
                HALs]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Architecture_HALs)
            -   [[4.5.1.3]{.tocnumber} [DMA from Falcon / NvRISC-V to
                Frame Buffer Interface
                (FBIF)]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#DMA_from_Falcon_/_NvRISC-V_to_Frame_Buffer_Interface_(FBIF))
        -   [[4.5.2]{.tocnumber} [Translation
            Tables]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Translation_Tables_2)
            -   [[4.5.2.1]{.tocnumber}
                [vmiop_gva]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#vmiop_gva)
    -   [[4.6]{.tocnumber} [Display Surface
        Virtualization]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#Display_Surface_Virtualization_4)
-   [[5]{.tocnumber}
    [amdgpu]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#amdgpu)
-   [[6]{.tocnumber} [References (Talks & Reading
    Material)]{.toctext}](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html#References_(Talks_&_Reading_Material))
:::

## [Article Structure]{#Article_Structure .mw-headline}

This article will aim to provide information about the following details
of GPU drivers:

### [High Level Architecture]{#High_Level_Architecture .mw-headline}

This section will detail high level architectures of each driver.

### [Initialization]{#Initialization .mw-headline}

This section will cover how the GPU\'s embedded components, the GPU
driver, and virtualization functions are initialized.

### [Scheduling]{#Scheduling .mw-headline}

This section will detail how the device schedules instructions for
execution.

This will attempt to provide a comprehensive view of scheduling from
virtualized processes down to execution within the device.

#### [In-VM Scheduling]{#In-VM_Scheduling .mw-headline}

The In-VM Scheduling section will detail how instructions scheduled
within the virtual machine\'s GPU device driver.

#### [Between-VM Scheduling]{#Between-VM_Scheduling .mw-headline}

The Between-VM Scheduling section will detail how the host kernel module
and/or virtual GPU helper functions handle scheduling / context swaps
between virtual machines on a computer system.

#### []{#Firmware_Scheduling_.28if_applicable.29}[Firmware Scheduling (if applicable)]{#Firmware_Scheduling_(if_applicable) .mw-headline}

The Firmware Scheduling section will cover the GPU\'s internal
scheduling model if a deferred execution pathway is used (like i915\'s
GuC or OpenRM\'s GSP).

In the case an intermediate scheduling microcontroller is not used this
section may be less applicable (ie: vExeclist scheduling under Intel
vGPUs).

### [Memory Management]{#Memory_Management .mw-headline}

The Memory Management section will cover how the GPU driver manages
memory for virtual GPUs and host processes.

This will aim detail paging abstractions used for global memory
translation, and per-process memory translation.

### [Display Surface Virtualization]{#Display_Surface_Virtualization .mw-headline}

The Display Surface Virtualization section will detail how virtual
displays are provided to guests and/or graphics rendering buffers (pixel
surfaces) are shared from guest to host if such functions are provided
via the driver.

#### [Graphics Buffer Sharing]{#Graphics_Buffer_Sharing .mw-headline}

This section will cover methods provided by the GPU driver suitable for
high performance graphics sharing.

## [Vendor Neutral]{#Vendor_Neutral .mw-headline}

This section will detail functions in common between GPU drivers.

Also see [Display Virtualization for Windows
OS](https://github.com/intel/Display-Virtualization-for-Windows-OS){.external
.text rel="nofollow"}.

### [Knowledge Resources Used]{#Knowledge_Resources_Used .mw-headline}

This section is supported by significant contributions to documentation
and open source by **Satyeshwar Singh**.

### [Display Surface Virtualization]{#Display_Surface_Virtualization_2 .mw-headline}

The Display Surface Virtualization section will detail how virtual
displays are provided to guests and/or graphics rendering buffers (pixel
surfaces) are shared from guest to host if such functions are provided
via the driver.

#### [Graphics Buffer Sharing]{#Graphics_Buffer_Sharing_2 .mw-headline}

This section will deal with graphics buffer sharing for vendor neutral
GPUs.

##### [Display Flow in Linux]{#Display_Flow_in_Linux .mw-headline}

Userspace applications provide their buffers to compositor.

Compositor asks Mesa to create a framebuffer.

Framebuffer is allocated through vendor driver.

Vendor driver flips the framebuffer on the screen.

##### [SR-IOV and Mdev]{#SR-IOV_and_Mdev .mw-headline}

SR-IOV VFs don\'t have access to the display controller (only PF does).

Known issue is displaying a VF\'s framebuffer on the screen.

##### []{#VFs_in_QEMU_Hypervisor_.28VirtIO-GPU.29}[VFs in QEMU Hypervisor (VirtIO-GPU)]{#VFs_in_QEMU_Hypervisor_(VirtIO-GPU) .mw-headline}

SR-IOV and Mdev VFs run in QEMU Hypervisor.

QEMU has full access to all pages in VF\'s address space.

VirtIO-GPU allows for allocating buffers (not API remoting in this
case).

Mesa\'s KMSRO allocates framebuffer via virtio-gpu.

Mesa\'s KMSRO asks vendor driver to import framebuffer (no specific
compositor dependancy).

##### [Transport]{#Transport .mw-headline}

A Scatter Gather List (SGL) of physical pages is constructed by
VirtIO-GPU of VF.

###### [QEMU Host-Side u-dma-buf]{#QEMU_Host-Side_u-dma-buf .mw-headline}

Host QEMU uses u-dma-buf driver to reconstruct a virtual memory pointer
from the SGL shared by VF.

u-dma-buf driver allocates contiguous memory blocks in kernel as DMA
buffers.

###### [QEMU Host-Side Modules]{#QEMU_Host-Side_Modules .mw-headline}

Once virtio-qemu has a DMA buffer, it shares it with the QEMU UI.

QEMU UI has several toolkits to support like GTK, SDL, etc. with GTK.
being the default.

GTK uses EGL and passes it the DMABUF as a texture.

##### [Windows VFs]{#Windows_VFs .mw-headline}

Windows OS doesn\'t support DMABUF capability so we can\'t allocate
buffers from VirtIO-GPU and share them with the Windows Kernel Mode
Driver (KMD).

Windows OS is different in the way that the FB is allocated via the OS
rather than the miniport driver.

Modified version of RedHat virtio-gpu-do (short for Display Only) driver
used.

##### [Windows Driver Stack]{#Windows_Driver_Stack .mw-headline}

OS (DWM) allocates frame buffers and asks GPU to write to them through
Miniport.

DVServer UMD (IDD) asks the OS for the FB copy.

DVServer UMD (IDD) passes the virtual memory address to DVServer KMD.

##### [DVServer KMD]{#DVServer_KMD .mw-headline}

DVServer KMD finds the SGL (Scatter Gather List) of physical pages for
this virtual memory address.

This SGL is passed via virt-queue from the VF to the host QEMU.

##### [Android VFs]{#Android_VFs .mw-headline}

Android uses Linux kernel underneath. VirtIO-GPU is present in
Android\'s kernel.

KMSRO patch to allocate framebuffer via VirtIO-GPU also ported over to
Android.

Need modifications in the minigbm lib of Android to connect with KMSRO
part.

Rest of stack looks identical to Linux VF/PF.

## [i915]{#i915 .mw-headline}

### [Knowledge Resources Used]{#Knowledge_Resources_Used_2 .mw-headline}

This section is supported by significant contributions to documentation
and open source by **Zhi Wang**, **Ben Widawsky**, and **Igor
Bogdanov**.

See references 2, 3, 4, 5, 6, 7, 8, and 9 in the [References (Talks &
Reading
Material)](index.php/GPU_Driver_Internals.html#References_(Talks_&_Reading_Material)){.external
.text rel="nofollow"} section.

### [High Level Architecture]{#High_Level_Architecture_2 .mw-headline}

#### [i915 Clients]{#i915_Clients .mw-headline}

Processes which make use of the Intel i915 driver receive an i915 Client
ID.

### [Initialization]{#Initialization_2 .mw-headline}

During initialization of the i915 driver the GuC binary blob is
offloaded into the Graphics Translation Table (GTT). This allows the GuC
to read GTT-loaded binary blob from shared framebuffer memory so that it
may boot.

### [Scheduling]{#Scheduling_2 .mw-headline}

::: {.thumb .tright}
::: {.thumbinner style="width:302px;"}
[![Figure 0: i915 vGPU
Scheduling](images/thumb/c/c2/Figure_0-_i915_vGPU_Scheduling.png/300px-Figure_0-_i915_vGPU_Scheduling.png){.thumbimage
decoding="async" width="300" height="227"
srcset="images/thumb/c/c2/Figure_0-_i915_vGPU_Scheduling.png/450px-Figure_0-_i915_vGPU_Scheduling.png 1.5x, images/thumb/c/c2/Figure_0-_i915_vGPU_Scheduling.png/600px-Figure_0-_i915_vGPU_Scheduling.png 2x"}](index.php/File:Figure_0-_i915_vGPU_Scheduling.png.html){.image}

::: thumbcaption
::: magnify
[](index.php/File:Figure_0-_i915_vGPU_Scheduling.png.html "Enlarge"){.internal}
:::

**Figure 0:** i915 vGPU high-level Scheduling. See slides from:
[\[9\]](index.php/GPU_Driver_Internals.html#References_(Talks_&_Reading_Material)){.external
.text rel="nofollow"}
:::
:::
:::

There are several abstractions for GPU virtual machine scheduling. Those
are as follows:

-   Guest kernel (i915.ko)
-   Host kernel (i915.ko)
-   Device Firmware (GuC)

##### [In-VM Scheduling]{#In-VM_Scheduling_2 .mw-headline}

##### []{#Guest_kernel_.28i915.ko.29}[Guest kernel (i915.ko)]{#Guest_kernel_(i915.ko) .mw-headline}

###### [vExeclist]{#vExeclist .mw-headline}

The vExeclist is a method to submit commands directly to the GPU without
the use of an intermediate microcontroller.

###### [vGuC]{#vGuC .mw-headline}

vGuC is a command submission interface used to process commands to the
Intel [Graphics Microcontroller
(GuC)](index.php/GPU_Firmware.html#GuC){.external .text rel="nofollow"}.

#### [Between-VM Scheduling]{#Between-VM_Scheduling_2 .mw-headline}

::: {.thumb .tright}
::: {.thumbinner style="width:302px;"}
[![Figure 1: i915 vGPU Scheduling Requests &
Events](images/thumb/c/c3/I915_Scheduling_Events_and_Requests.png/300px-I915_Scheduling_Events_and_Requests.png){.thumbimage
decoding="async" width="300" height="232"
srcset="images/thumb/c/c3/I915_Scheduling_Events_and_Requests.png/450px-I915_Scheduling_Events_and_Requests.png 1.5x, images/thumb/c/c3/I915_Scheduling_Events_and_Requests.png/600px-I915_Scheduling_Events_and_Requests.png 2x"}](index.php/File:I915_Scheduling_Events_and_Requests.png.html){.image}

::: thumbcaption
::: magnify
[](index.php/File:I915_Scheduling_Events_and_Requests.png.html "Enlarge"){.internal}
:::

**Figure 1:** i915 vGPU Scheduling Requests & Events. See slides from:
[\[9\]](index.php/GPU_Driver_Internals.html#References_(Talks_&_Reading_Material)){.external
.text rel="nofollow"}
:::
:::
:::

##### []{#Host_kernel_.28i915.ko.29}[Host kernel (i915.ko)]{#Host_kernel_(i915.ko) .mw-headline}

Submitting commands to the GPU under i915 can take several paths. One
pathway makes use of direct command submission without an intermediate
micro-controller whereas the other uses an intermediate
micro-controller. The intermediate micro-controller approach increases
the amount of binary-blob code used and abstracts the kernel module from
the device\'s internal scheduling model.

###### [Execlist]{#Execlist .mw-headline}

Execlist executes commands synchronously on the device without an
intermediate microcontroller. This is the preferred method of executing
commands by some driver developers because of it\'s stability and
transparency under current i915 development.

###### [GuC]{#GuC .mw-headline}

GuC provides an execution pathway with an intermediate microcontroller
providing a scheduling abstraction for Intel\'s preferred internal
scheduling model.

::: {.thumb .tright}
::: {.thumbinner style="width:302px;"}
[![Figure 2: Intel vGPU Scheduler request
flow.](images/thumb/e/e0/Figure_2-_Intel_vGPU_Scheduler.png/300px-Figure_2-_Intel_vGPU_Scheduler.png){.thumbimage
decoding="async" width="300" height="233"
srcset="images/thumb/e/e0/Figure_2-_Intel_vGPU_Scheduler.png/450px-Figure_2-_Intel_vGPU_Scheduler.png 1.5x, images/thumb/e/e0/Figure_2-_Intel_vGPU_Scheduler.png/600px-Figure_2-_Intel_vGPU_Scheduler.png 2x"}](index.php/File:Figure_2-_Intel_vGPU_Scheduler.png.html){.image}

::: thumbcaption
::: magnify
[](index.php/File:Figure_2-_Intel_vGPU_Scheduler.png.html "Enlarge"){.internal}
:::

**Figure 2:** i915 vGPU Scheduler request flow. See slides from:
[\[9\]](index.php/GPU_Driver_Internals.html#References_(Talks_&_Reading_Material)){.external
.text rel="nofollow"}
:::
:::
:::

##### []{#Device_Firmware_.28GuC.29}[Device Firmware (GuC)]{#Device_Firmware_(GuC) .mw-headline}

### [Memory Management]{#Memory_Management_2 .mw-headline}

##### [Translation Tables]{#Translation_Tables .mw-headline}

###### []{#GTT_.28Graphics_Translation_Table.29}[GTT (Graphics Translation Table)]{#GTT_(Graphics_Translation_Table) .mw-headline}

GPU Memory on-device is a part of a GTT or Graphics Translation Table.
This table stores information globally for all graphics processes within
the system. Some processes access the Global Graphics Translation Table
(GGTT) such as
[DRI](https://en.wikipedia.org/wiki/Direct_Rendering_Infrastructure "wikipedia:Direct Rendering Infrastructure"){.extiw}
while other\'s receive a Per Process Graphics Translation Table (PPGTT)
buffer based on their i915 Client ID.

###### []{#GGTT_.28Global_Graphics_Translation_Table.29}[GGTT (Global Graphics Translation Table)]{#GGTT_(Global_Graphics_Translation_Table) .mw-headline}

###### []{#PPGTT_.28Per_Process_Graphics_Translation_Table.29}[PPGTT (Per Process Graphics Translation Table)]{#PPGTT_(Per_Process_Graphics_Translation_Table) .mw-headline}

Process-specific memory buffers are stored inside a Per Process Graphics
Translation Table or PPGTT. This is a [GPU
MMU](index.php/Virtual_IO_Internals.html#VRAM_Isolation_(GPU_GMMU)){.external
.text rel="nofollow"} translated subregion or IOVA of global GPU memory
specific to a GPU process\'s client ID.

###### [Aliasing PPGTT]{#Aliasing_PPGTT .mw-headline}

Aliasing PPGTT (Per Process Graphics Translation Table) refers to
partially separated GPU resources (process context).

###### [Real PPGTT]{#Real_PPGTT .mw-headline}

Real PPGTT (Per Process Graphics Translation Table) refers to fully
separated GPU resources (process context).

### [Display Surface Virtualization]{#Display_Surface_Virtualization_3 .mw-headline}

The Intel vGPU makes use of two modes for Display Surface
Virtualization.

-   VirtIO-GPU (Linux)
-   Indirect Display Driver (Windows)

#### [VirtIO-GPU]{#VirtIO-GPU .mw-headline}

#### []{#IDD_.28Indirect_Display_Driver.29}[IDD (Indirect Display Driver)]{#IDD_(Indirect_Display_Driver) .mw-headline}

The IDD ([Indirect Display
Driver](https://learn.microsoft.com/en-us/windows-hardware/drivers/display/indirect-display-driver-model-overview){.external
.text rel="nofollow"}) provides virtual display functions with arbitrary
resolutions to any rendering device virtual or physical.

Intel\'s IDD can be found in their [Display Virtualization for Windows
OS](https://github.com/intel/Display-Virtualization-for-Windows-OS/){.external
.text rel="nofollow"} repository.

#### [Graphics Buffer Sharing]{#Graphics_Buffer_Sharing_3 .mw-headline}

Intel\'s i915 driver provides functionality to directly map display
memory from a [guest vGPU Virtual Function (either SR-IOV or VFIO-Mdev)
into the host GPU\'s Physical
Function](index.php/Merged_Drivers.html){.external .text rel="nofollow"}
without slow memory copies or graphics compression. For users running
GPU virtualization on their local device this results in a significant
performance uplift compared to traditional graphics sharing
functionality built for remote access use-cases such as VDI.

Intel\'s [0copy display virtualization
tools](https://github.com/intel/Display-Virtualization-for-Windows-OS){.external
.text rel="nofollow"} are simple to implement as sharing does not rely
upon an added [IVSHMEM (Inter-VM Shared Memory
device)](https://www.qemu.org/docs/master/system/devices/ivshmem.html){.external
.text rel="nofollow"} - rather the host directly maps the guest\'s
display memory via [udmabuf](https://lwn.net/Articles/758903/){.external
.text rel="nofollow"} as the buffer sharing functions are provided
within the i915 open source driver.

## [OpenRM]{#OpenRM .mw-headline}

::: {.thumb .tright}
::: {.thumbinner style="width:302px;"}
[![Figure 3: GPU BAR to Timeshared Syhededuling via Channel
IO](images/thumb/f/fd/Figure_3-_GPU_BAR_to_Timeshared_Syhededuling_via_Channel_IO.png/300px-Figure_3-_GPU_BAR_to_Timeshared_Syhededuling_via_Channel_IO.png){.thumbimage
decoding="async" width="300" height="226"
srcset="images/thumb/f/fd/Figure_3-_GPU_BAR_to_Timeshared_Syhededuling_via_Channel_IO.png/450px-Figure_3-_GPU_BAR_to_Timeshared_Syhededuling_via_Channel_IO.png 1.5x, images/thumb/f/fd/Figure_3-_GPU_BAR_to_Timeshared_Syhededuling_via_Channel_IO.png/600px-Figure_3-_GPU_BAR_to_Timeshared_Syhededuling_via_Channel_IO.png 2x"}](index.php/File:Figure_3-_GPU_BAR_to_Timeshared_Syhededuling_via_Channel_IO.png.html){.image}

::: thumbcaption
::: magnify
[](index.php/File:Figure_3-_GPU_BAR_to_Timeshared_Syhededuling_via_Channel_IO.png.html "Enlarge"){.internal}
:::

**Figure 3:** GPU BAR to Timeshared Scheduling via Channel IO.
:::
:::
:::

### [Knowledge Resources Used]{#Knowledge_Resources_Used_3 .mw-headline}

This section is supported by significant contributions to documentation
and open source by **Andy Currid**, **Neo Jia**, **John Fanelli**, and
**Neha Joshi**.

### [High Level Architecture]{#High_Level_Architecture_3 .mw-headline}

The Open Resource Manager (RM driver) makes use of a highly object
oriented paradigm comprised of multiple \"engines\" which act as
micro-services for servicing driver requests.

The Open Resource Manager driver (also known as
[OpenRM](index.php/OpenRM.html){.external .text rel="nofollow"}) refers
to Nvidia\'s
[open-kernel-modules](https://github.com/NVIDIA/open-gpu-kernel-modules){.external
.text rel="nofollow"}.

Broadly speaking the OpenRM driver consists of two parts.

-   The Platform RM (OpenRM)
-   The Firmware RM (GSP RM / RM Core)

The Platform RM is loaded into the Linux Kernel as nvidia.ko. This
module communicates with the GSP RM for via [Remote Procedure Calls
(RPCs)](https://en.wikipedia.org/wiki/Remote_procedure_call "wikipedia:Remote procedure call"){.extiw}
to communicate with engines from the RM Core.

##### [RM Clients]{#RM_Clients .mw-headline}

Processes (local, remote, or virtualized) which make use of the RM
driver receive an RM Client ID.

#### [RM Server]{#RM_Server .mw-headline}

The RM Server (or Resource Server) tracks RM Clients as well as the
hardware and software resources they control, allocate, and free.

#### [RM API]{#RM_API .mw-headline}

API to control the Resource Manager Server.

#### [RM Core]{#RM_Core .mw-headline}

Core functions of the RM driver controlling resource locking, mapping,
unmapping, control calls, constructors, and deconstructors. Under OpenRM
the RM Core runs within the GPU System Processor (GSP micro-controller)
while in pre-open source versions of the Resource Manager the RM Core
ran within the nvidia.ko kernel module

### [Initialization]{#Initialization_3 .mw-headline}

During bring up of the hardware several binary blobs are loaded from
embedded [Boot
ROM](https://en.wikipedia.org/wiki/Boot_ROM "wikipedia:Boot ROM"){.extiw}
memory to bootstrap embedded controller bring up from which point
additional software is loaded from onboard
[SPI](https://en.wikipedia.org/wiki/Serial_Peripheral_Interface "wikipedia:Serial Peripheral Interface"){.extiw}
flash memory.

Software loaded from SPI flash is necessary for the full initialization
of the Falcon/NvRISC processor as well as a cached version of the
software necessary to run the GPU System Processor (GSP).

Once the platform is posted it is ready to communicate with the host
platform\'s RM driver. The OpenRM driver offloads a binary blob
containing the RM Core to the [GPU System Processor
(GSP)](index.php/GPU_Firmware.html#GSP){.external .text rel="nofollow"}
which is likely to contain a more recent version than the cached version
contained in on-board SPI flash.

### [Scheduling]{#Scheduling_3 .mw-headline}

There are several abstractions for GPU virtual machine scheduling. Those
are as follows:

-   Guest kernel (nvidia.ko)
-   Host kernel (nvidia.ko)
-   Host usermode (gpu-mgr / libnvidiavgpu.so)
-   Device Firmware (GSP)

#### [Command Submission]{#Command_Submission .mw-headline}

##### [Runlist]{#Runlist .mw-headline}

#### [In-VM Scheduling]{#In-VM_Scheduling_3 .mw-headline}

Virtual machines contain their own GPU scheduling within the Nvidia
kernel module in the guest OS.

##### []{#Guest_kernel_.28nvidia.ko.29}[Guest kernel (nvidia.ko)]{#Guest_kernel_(nvidia.ko) .mw-headline}

Within a virtual machine running the Nvidia driver messages to the GPU
are first sent to the guest nvidia.ko kernel module.

The guest then determines whether a vRPC (virtual Remote Procedure Call)
or a pRPC (physical Remote Procedure Call) should be sent. Both pRPCs
and vRPCs are sent through the host RM driver (Resource Manager).

> *Note: Unclear on execution pathway for pRPCs vs vRPCs. pRPCs may go
> directly to device.*

#### [Between-VM Scheduling]{#Between-VM_Scheduling_3 .mw-headline}

In addition to scheduling which occurs within the virtual machine the
Resource Manager driver also schedules messages to the GPU between
GPU-accelerated virtual machines and host processes.

##### []{#Host_kernel_.28nvidia.ko.29}[Host kernel (nvidia.ko)]{#Host_kernel_(nvidia.ko) .mw-headline}

Messages sent by the guest (via vRPC or pRPC) are received by the host
Nvidia.ko driver.

Nvidia.ko contains a virtual GPU state machine which contains status
information for the virtual GPU.

Nvidia.ko also contains a virtual GPU kernel scheduler which interacts
with virtual GPU objects.

Nvidia.ko also contains an RM Call scheduler which schedules calls on an
RM class.

The Nividia.ko kernel module exits to userspace to execute the
nvidia-vgpu-mgr and VMIOP (Virtual Machine Input Output Plugin).

##### []{#Host_usermode_.28nvidia-vgpu-mgr_.2F_libnvidia-vgpu.so.29}[Host usermode (nvidia-vgpu-mgr / libnvidia-vgpu.so)]{#Host_usermode_(nvidia-vgpu-mgr_/_libnvidia-vgpu.so) .mw-headline}

After exiting to userspace a daemon process (the nvidia-vgpu-mgr, and
it\'s library libnvidia-vgpu.so) are executed to schedule VM-exits
described below.

##### [nvidia-vgpu-mgr]{#nvidia-vgpu-mgr .mw-headline}

The nvidia-vgpu-mgr is a process which provides the spawning of virtual
GPU stubs and population with capability information.

This daemon process interacts with the libnvidia-vgpu.so, nvidia.ko, and
nvidia-vgpu-vfio.ko components.

This process and the libnvidia-vgpu.so contain, and execute the VMIOP.

This process (and the VMIOP) schedules RPCs sent by the guest, receives
VFIO BAR-exits, and relays requests to allocate, deallocate, pin, unpin,
map, unmap to the [RM
Core](index.php/GPU_Driver_Internals.html#RM_Core){.external .text
rel="nofollow"}.

###### [vmiop]{#vmiop .mw-headline}

VMIOP (Virtual Machine Input Output Plugin) handles presenting
virtualized functionality into the guest.

The Virtual Machine Input Output Plugin software handles virtual
displays, compute API offload, and most importantly [BAR (Base Address
Register)
quirks](index.php/Virtual_I/O_Internals.html#VFIO_Quirks_(region_traps)){.external
.text rel="nofollow"}.

The VMIOP is an [SDK (Software Development
Kit)](https://en.wikipedia.org/wiki/Software_development_kit "wikipedia:Software development kit"){.extiw}
provided in binary format split between the libnvidiavgpu.so and
nvidia-vgpu-mgr which provides userland helper functions for GPU
virtualization.

Messages to the VMIOP are scheduled by Linux kernel
[niceness](https://www.learnlinux.org.za/courses/build/internals/ch07s02.html){.external
.text rel="nofollow"} (scheduling abstraction).

##### []{#Device_Firmware_.28GSP.29}[Device Firmware (GSP)]{#Device_Firmware_(GSP) .mw-headline}

Messages received by the host RM driver (Resource Manager) are then
scheduled by the RM Core contained within the GSP (GPU System
Processor). The GSP handles the device\'s internal scheduling model.

### [Memory Management]{#Memory_Management_3 .mw-headline}

Managing virtual machine memory is very important to the security of
virtualization.

This section will cover the method by which secure memory \"enclaves\"
or [IO Virtual Addresses
(IOVAs)](https://terenceli.github.io/%E6%8A%80%E6%9C%AF/2019/08/04/iommu-introduction){.external
.text rel="nofollow"} may be provisioned and separation enforced by
hardware constructs within the GPU.

##### [Programming the MMU]{#Programming_the_MMU .mw-headline}

In order to hardware enforce separation between memory allocated to
Virtual Machines (VMs) virtualization software must program the GPU\'s
MMU (GMMU controller) to create IO Virtual Addresses (IOVAs).

In order to create such configurations several abstractions are used to
translate high level representations of virtualization programmed via
the vmiop and gpu-mgr into practical, architecture specific
instructions.

###### [AMAPLibrary]{#AMAPLibrary .mw-headline}

The AMAPLibrary acts as a device abstraction framework for GPU driver
software to program using high level representations of MMU
configuration.

The AMAPLibrary translates high level representations of GPU
virtualization into graphics architecture-specific logic contained
within architecture HALs (Hardware Abstraction Layers).

###### [Architecture HALs]{#Architecture_HALs .mw-headline}

GPU [Hardware Abstraction Layers
(HALs)](https://archive.ll.mit.edu/HPEC/agendas/proc07/Day3/10_Hensley_Abstract.pdf){.external
.text rel="nofollow"} contain logic specific to graphics architectures,
for instance the precise method by which the GPU driver may interact
with the Falcon to provision MMU protected memory.

###### []{#DMA_from_Falcon_.2F_NvRISC-V_to_Frame_Buffer_Interface_.28FBIF.29}[DMA from Falcon / NvRISC-V to Frame Buffer Interface (FBIF)]{#DMA_from_Falcon_/_NvRISC-V_to_Frame_Buffer_Interface_(FBIF) .mw-headline}

The Falcon (FAst Logic CONtroller) / NvRISC-V embedded controller emits
DMAs to the Frame Buffer Interface (FBIF) in order to interact with the
device\'s GMMU controller.

User programs and VMs have their memory translated through the GMMU.

Once created memory translations within a virtual machine\'s IO Virtual
Address (IOVA) will be protected by the device\'s GMMU controller.

##### [Translation Tables]{#Translation_Tables_2 .mw-headline}

###### [vmiop_gva]{#vmiop_gva .mw-headline}

### [Display Surface Virtualization]{#Display_Surface_Virtualization_4 .mw-headline}

## [amdgpu]{#amdgpu .mw-headline}

## []{#References_.28Talks_.26_Reading_Material.29}[References (Talks & Reading Material)]{#References_(Talks_&_Reading_Material) .mw-headline}

1.  [Intel Graphics Programmer\'s Reference Manuals
    (PRM)](https://01.org/linuxgraphics/documentation/hardware-specification-prms){.external
    .text rel="nofollow"}
2.  [i915: Hardware Contexts (and some bits about
    batchbuffers)](https://bwidawsk.net/blog/2013/1/i915-hardware-contexts-and-some-bits-about-batchbuffers/){.external
    .text rel="nofollow"}
3.  [i915: The Global GTT Part
    1](https://bwidawsk.net/blog/2014/6/the-global-gtt-part-1/){.external
    .text rel="nofollow"}
4.  [i915: Aliasing PPGTT Part
    2](https://bwidawsk.net/blog/2014/6/aliasing-ppgtt-part-2/){.external
    .text rel="nofollow"}
5.  [i915: True PPGTT Part
    3](https://bwidawsk.net/blog/2014/7/true-ppgtt-part-3/){.external
    .text rel="nofollow"}
6.  [i915: Future PPGTT Part 4 (Dynamic page table allocations, 64 bit
    address space, GPU \"mirroring\", and yeah, something about relocs
    too)](https://bwidawsk.net/blog/2014/7/future-ppgtt-part-4-dynamic-page-table-allocations-64-bit-address-space-gpu-mirroring-and-yeah-something-about-relocs-too/){.external
    .text rel="nofollow"}
7.  [i915: Security of the Intel Graphics Stack - Part 1 -
    Introduction](https://igor-blue.github.io/2021/02/10/graphics-part1.html){.external
    .text rel="nofollow"}
8.  [i915: Security of the Intel Graphics Stack - Part 2 - FW \<-\>
    GuC](https://igor-blue.github.io/2021/02/24/graphics-part2.html){.external
    .text rel="nofollow"}
9.  [i915: An Introduction to Intel GVT-g (with new
    architecture)](https://01.org/sites/default/files/documentation/an_introduction_to_intel_gvt-g_for_external.pdf){.external
    .text rel="nofollow"}
10. [lwn.net: Add udmabuf misc
    device](https://lwn.net/Articles/758903/){.external .text
    rel="nofollow"}
11. [Nvidia RISC-v
    Story](https://riscv.org/wp-content/uploads/2016/07/Tue1100_Nvidia_RISCV_Story_V2.pdf){.external
    .text rel="nofollow"}
12. [IOMMU
    Introduction](https://terenceli.github.io/%E6%8A%80%E6%9C%AF/2019/08/04/iommu-introduction){.external
    .text rel="nofollow"}
13. [Hardware and Compute Abstraction Layers For Accelerated Computing
    Using Graphics Hardware and Conventional
    CPUs](https://archive.ll.mit.edu/HPEC/agendas/proc07/Day3/10_Hensley_Abstract.pdf){.external
    .text rel="nofollow"}
14. [nVidia GPU Introduction
    (envytools)](https://envytools.readthedocs.io/en/latest/hw/intro.html){.external
    .text rel="nofollow"}
15. [Delivering High Performance Remote Graphics With Nvidia GRID
    Virtual
    GPU](https://on-demand.gputechconf.com/gtc/2014/presentations/S4725-hi-perf-graphics-nvidia-grid-virtual-gpus.pdf){.external
    .text rel="nofollow"}
16. [NVIDIA Multi-Instance
    GPU](https://nehajoshi.dev/post/nvidia_mig_feature/){.external .text
    rel="nofollow"}
17. [i915: Kaby Lake Intel Graphics Programmer\'s Reference Manual
    \[Volume 5: Memory
    Views\]](https://01.org/sites/default/files/documentation/intel-gfx-prm-osrc-kbl-vol05-memory_views.pdf){.external
    .text rel="nofollow"}
18. [Lifecycle of a Triangle - Nvidia\'s logical
    pipeline](https://developer.nvidia.com/content/life-triangle-nvidias-logical-pipeline){.external
    .text rel="nofollow"}
19. [Nvidia Parallel Thread Execution
    (PTX)](https://docs.nvidia.com/cuda/parallel-thread-execution/){.external
    .text rel="nofollow"}
20. [i915/GEM
    Crashcourse](https://blog.ffwll.ch/2013/01/i915gem-crashcourse-overview.html){.external
    .text rel="nofollow"}
21. [drm: Add GEM (\"graphics execution manager\") to i915
    driver.](https://lwn.net/Articles/292583/){.external .text
    rel="nofollow"}
:::

::: printfooter
Retrieved from
\"[https://open-iov.org/index.php?title=GPU_Driver_Internals&oldid=23791](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html){dir="ltr"}\"
:::
:::

::: {#catlinks .catlinks .catlinks-allhidden mw="interface"}
:::
:::
:::

::: {#mw-navigation}
## Navigation menu

::: {#mw-head}
### Personal tools {#p-personal-label .vector-menu-heading}

::: vector-menu-content
-   [[Log
    in](https://open-iov.org/index.php?title=Special:UserLogin&returnto=GPU+Driver+Internals&returntoquery=oldid%3D23791 "You are encouraged to log in; however, it is not mandatory [o]"){accesskey="o"}]{#pt-login}
:::

::: {#left-navigation}
### Namespaces {#p-namespaces-label .vector-menu-heading}

::: vector-menu-content
-   [[Page](index.php/GPU_Driver_Internals.html "View the content page [c]"){accesskey="c"}]{#ca-nstab-main}
-   [[Discussion](https://open-iov.org/index.php?title=Talk:GPU_Driver_Internals&action=edit&redlink=1 "Discussion about the content page (page does not exist) [t]"){rel="discussion"
    accesskey="t"}]{#ca-talk}
:::

### Variants [expanded]{.vector-menu-checkbox-expanded} [collapsed]{.vector-menu-checkbox-collapsed} {#p-variants-label .vector-menu-heading}

::: vector-menu-content
:::
:::

::: {#right-navigation}
### Views {#p-views-label .vector-menu-heading}

::: vector-menu-content
-   [[Read](index.php/GPU_Driver_Internals.html)]{#ca-view}
-   [[View
    source](index.php%3Ftitle=GPU_Driver_Internals&action=edit.html "This page is protected.
    You can view its source [e]"){accesskey="e"}]{#ca-viewsource}
-   [[View
    history](index.php%3Ftitle=GPU_Driver_Internals&action=history.html "Past revisions of this page [h]"){accesskey="h"}]{#ca-history}
:::

### More [expanded]{.vector-menu-checkbox-expanded} [collapsed]{.vector-menu-checkbox-collapsed} {#p-cactions-label .vector-menu-heading}

::: vector-menu-content
:::

::: {#p-search .vector-search-box role="search"}
<div>

### Search

::: {#simpleSearch search-loc="header-navigation"}
:::

</div>
:::
:::
:::

::: {#mw-panel}
::: {#p-logo role="banner"}
[](images/archive/c/c3/20221214193624!I915_Scheduling_Events_and_Requests.png.html "Visit the main page"){.mw-wiki-logo}
:::

### Navigation {#p-navigation-label .vector-menu-heading}

::: vector-menu-content
-   [[Main
    page](images/archive/c/c3/20221214193624!I915_Scheduling_Events_and_Requests.png.html "Visit the main page [z]"){accesskey="z"}]{#n-mainpage-description}
-   [[Recent
    changes](https://open-iov.org/index.php/Special:RecentChanges "A list of recent changes in the wiki [r]"){accesskey="r"}]{#n-recentchanges}
-   [[Random
    page](index.php/Special:Random.html "Load a random page [x]"){accesskey="x"}]{#n-randompage}
-   [[Help about
    MediaWiki](https://www.mediawiki.org/wiki/Special:MyLanguage/Help:Contents)]{#n-help-mediawiki}
:::

### Tools {#p-tb-label .vector-menu-heading}

::: vector-menu-content
-   [[What links
    here](index.php/Special:WhatLinksHere/GPU_Driver_Internals.html "A list of all wiki pages that link here [j]"){accesskey="j"}]{#t-whatlinkshere}
-   [[Related
    changes](https://open-iov.org/index.php/Special:RecentChangesLinked/GPU_Driver_Internals "Recent changes in pages linked from this page [k]"){rel="nofollow"
    accesskey="k"}]{#t-recentchangeslinked}
-   [[Special
    pages](index.php/Special:SpecialPages.html "A list of all special pages [q]"){accesskey="q"}]{#t-specialpages}
-   [[Printable
    version](javascript:print(); "Printable version of this page [p]"){rel="alternate"
    accesskey="p"}]{#t-print}
-   [[Permanent
    link](index.php%3Ftitle=GPU_Driver_Internals&oldid=23791.html "Permanent link to this revision of the page")]{#t-permalink}
-   [[Page
    information](index.php%3Ftitle=GPU_Driver_Internals&action=info.html "More information about this page")]{#t-info}
:::
:::
:::

-   [This page was last edited on 19 June 2023, at
    17:53.]{#footer-info-lastmod}

```{=html}
<!-- -->
```
-   [[Privacy
    policy](https://open-iov.org/index.php/Open-IOV:Privacy_policy "Open-IOV:Privacy policy")]{#footer-places-privacy}
-   [[About
    Open-IOV](index.php/Open-IOV:About.html "Open-IOV:About")]{#footer-places-about}
-   [[Disclaimers](https://open-iov.org/index.php/Open-IOV:General_disclaimer "Open-IOV:General disclaimer")]{#footer-places-disclaimer}

```{=html}
<!-- -->
```
-   [[![Powered by
    MediaWiki](resources/assets/poweredby_mediawiki_88x31.png){srcset="resources/assets/poweredby_mediawiki_132x47.png 1.5x, resources/assets/poweredby_mediawiki_176x62.png 2x"
    width="88" height="31"
    loading="lazy"}](https://www.mediawiki.org/)]{#footer-poweredbyico}