# GMMU, AMAPlib, HALs, Runlists, and GR

> The engine side: how addresses translate, how work is scheduled, and what
> "GR" actually is. Read this when you need to know what the guest's work
> *means*.

## 1. GMMU — the GPU memory management unit

The GMMU translates the GPU virtual addresses (VAs) used by engines into
physical addresses (VRAM offsets or system-memory addresses).

### 1.1 Page-table hierarchy

NVIDIA's GMMU uses a multi-level page table whose shape depends on the AS
generation. Turing (the reference) uses a four-level hierarchy with entry
coverage:

| Level | Entries | VA bits consumed | Coverage per entry |
|---|---|---|---|
| Root page directory (PDB) | 4 | top bits | large |
| PDE level 1 | 512 | 9 bits | e.g. 2^38 |
| PDE level 2 | 512 | 9 bits | e.g. 2^29 |
| PTE level | 512+ | 9+ bits | 4 KiB / 64 KiB / 2 MiB pages |

Verified properties (from the live guest):

* **PTE encoding: `PA[47:12] | flags`**, where `flags = 0xe` for a valid
  system-memory leaf entry in the format the client produces.
* Page sizes in use: 4 KiB (system memory objects) and larger (big pages) for
  large allocations.
* **Apertures**: during enumeration the guest uses `ADDR_SYSMEM = 1` and
  `ADDR_FBMEM = 2` to describe memory descriptors (e.g. channel instance
  memory in VRAM at offset `0xffed000`, method buffers in system memory at a
  physical address). The client hands the **physical** address in these
  descriptors — a fact we rely on for the memory model.
* The page-table geometry arrives over
  `NV90F1_CTRL_CMD_VASPACE_COPY_SERVER_RESERVED_PDES` (level physical
  addresses, page shifts, VA range) — see
  [`03-gsp-and-gsp-rm.md`](03-gsp-and-gsp-rm.md) §5.

### 1.2 What "aperture" means for us

Two address spaces matter:

* **System memory (sysmem)**: guest RAM. For a virtual device, the "GPU's view"
  of guest memory is established by the host (in real vGPU, by the vGPU manager
  via the platform IOMMU; here, by our module).
* **Video memory (vidmem / FB)**: the GPU's local memory. In the current
  emulation this is a host RAM buffer exposed as the BAR1 aperture; the
  forwarding phase replaces it with real GPU memory.

## 2. AMAPlib and the HALs

NVIDIA's driver abstracts hardware in layers:

```
driver logic (memmgr, FIFO, engines)
   │
   ├─ HAL layer: per-generation C functions that program registers and formats
   │     e.g. gmmu HAL (page-table formats, invalidations),
   │          engine HALs (CE/GR methods), host HAL (doorbells/PUT)
   │
   └─ AMAPlib: the "address mapping" library shared by RM and simulations
         (DRAM swizzling, bank/channel mapping, physical-to-aperture rules)
```

Developers study two component families to understand these layers:

* **AMAPlib** — the shared "address mapping" component used by the RM and by
  hardware simulations: DRAM swizzling, bank/channel mapping, aperture rules.
* **The HALs** — per-generation C code that programs registers and formats
  page-table entries and engine commands.

Our knowledge of them comes from reverse engineering and decompilation of
driver and firmware behaviour, plus measurement of live devices. What is
published here is our observations, numeric IDs and layouts — which is what
another engineer needs.

Why we care:

* When we **populate page tables ourselves** (server-owned ranges), we must
  produce entries the hardware and the guest both accept — AMAPlib/HAL code is
  the specification.
* When we **forward work**, page tables we write must match what real engines
  expect, including big-page encodings and invalidation semantics.

## 3. Channels, GPFIFO, USERD, doorbells

A **channel** is a hardware execution context. Its components:

| Piece | Where | Purpose |
|---|---|---|
| **GPFIFO** | GPU memory (sysmem or vidmem) | ring of entries; each points to a push buffer |
| **Push buffer** | GPU memory | the actual method stream the engine executes |
| **USERD** | vidmem (per channel) | channel control block: PUT/GET pointers, doorbell-related state |
| **Instance block** | vidmem or sysmem | the hardware context (engine state, page directory base, etc.) |
| **Runlist / TSG** | scheduled by the host | the list of channels currently eligible to run |

The submission flow (what a driver does, and what the guest therefore does):

```
write methods into a push buffer          (CPU, via a mapping)
write a GPFIFO entry pointing at the push (CPU)
update the channel's PUT pointer          (CPU → USERD or doorbell register)
   ... the scheduler eventually runs the channel; the engine fetches the push
   ... on completion the engine writes the GET pointer and/or a semaphore
```

Two details that repeatedly matter in practice:

* **The push buffer format** is a sequence of method/data pairs preceded by
  *incrementing-method headers* that encode the method offset (`address >> 2`),
  a subchannel and a count. A decoder for this format is the first tool you
  need to read a trace (our `scan_seq` module parameter found UVM's pushes by
  searching for the decoded data pattern).
* **Engine classes** are the vocabulary of pushes: `TURING_DMA_COPY_A`
  (0xc5b5) for copy engines, `TURING_CHANNEL_GPFIFO_A` (0xc46f) for channel
  objects, `GP100_UVM_SW` (0xc076) for UVM's software object, etc.

## 4. Runlist and scheduling

The GPU schedules channels through a **runlist**: a list of channel
descriptors the scheduler walks, giving each channel a time slice. A **TSG**
(Time-Slice Group) bundles channels that share a scheduling slot — this is the
mechanism for fair sharing and for interleaving contexts.

For virtualization this is a focal point:

* In real vGPU, the host vGPU manager owns the runlist and interleaves
  guest channels with host channels.
* In our target architecture, *we* own the schedule: guest channels must be
  mapped onto real GPU scheduling primitives with correct isolation (a guest
  must never observe or influence another guest's channels).
* The guest driver observes scheduling only indirectly: PUT/GET progress,
  semaphores, and fault timing. Faithful *semantics* (not exact timing) is the
  requirement.

## 5. GR — the graphics/compute engine

**GR** is the engine that executes shaders: both graphics pipelines and CUDA
compute kernels. It is the reason a GPU is a GPU.

What GR needs to run work:

* **A context** (the "golden context" + per-context state): register state,
  memory layout, page directory, shader code.
* **A channel** with a push buffer that issues *method* calls to GR:
  loading shader code, binding surfaces, launching grids (`QMD`s in modern
  architectures), and signaling completion.
* **Memory** containing the shader binary (SASS), kernel parameters,
  descriptors and data.
* **Interrupts/semaphores** to signal completion and errors.

For CUDA specifically:

```
cuLaunchKernel
   → CUDA driver builds a GR push (grid config, QMD, parameters)
   → submits on the process's GR channel
   → GR executes the shader on real hardware
   → completion semaphore / event; UVM handles memory effects
```

**Consequence for Open-IOV:** CUDA kernel execution is *real hardware
execution*. No emulation of GR can be correct and fast enough; the only viable
path is forwarding the guest's GR work to real hardware (see
[`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) and
the challenges document). This is also why "CUDA works" is a much stronger
statement than "`nvidia-smi` works".

## 6. Copy engines (CE) and other engines

* **CE (copy engines)** — DMA between memory regions; also used for UVM's
  internal work (page tables, memsets, semaphores). UVM cannot even initialize
  without working CE channels, which is why our CE capability reporting was
  the first big unblock (`CE_GET_PHYSICAL_CAPS`, `CE_GET_CE_PCE_MASK`).
* **SEC2** — secure copy engine used by UVM for protected operations.
* **NVENC/NVDEC/NVJPG/OFA** — video and image engines; roadmap for graphics
  and video phases.

Engine types are enumerated by the client during init (`FIFO_GET_DEVICE_INFO_TABLE`)
and reported in engine/capability queries. A guest's channel is bound to one
engine type (e.g. `COPY0 = 0x9`, `GR = 0x1`).

**Engine inventory is a correctness surface, not just data.** The userspace
driver builds its per-engine records from the device/engine information the
kernel driver reports, and filters them against the class list; a missing or
zeroed record is not ignored - it becomes an allocation with class `0` and the
whole context setup rolls back. This was exactly the CUDA bring-up blocker
until the engine inventory was completed: the async copy engine (COPY14,
engine descriptor `0x793ceb0e`, instance 14) was absent from the emulated
engine list, while a bare-metal oracle reports it as the seventh engine
alongside GR and CE0..4; it is now included. See
[`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md)
section 3.7 for the record layout the driver's engine walker consumes.

## 7. Practical reading list inside this repository

| Topic | Where |
|---|---|
| Page-table formats and HAL concepts | this tree's architecture notes plus the decompiled analysis notes |
| Class/method definitions (classes, methods, notifications) | our decompilation notes and reformatted tables in `wiki/` |
| Channel allocation parameter structures | reformatted in the field guide and in the reference wiki pages |
| GMMU controls and static info | `wiki/Address-Reference.md`, `wiki/RPC-Message-Format.md` |
| Notification formats | the reference wiki pages |
| Firmware structure and dispatchers | `wiki/GSP-570.144W-Firmware-Analysis.md` |

**Rule:** tracked documentation describes components and behaviour, with our
own observations, IDs and layouts. Exact analysis sources live in the
maintainers' study notes.