# UVM in the Guest

> **Scope.** What UVM (`nvidia_uvm.ko`) does inside the guest, why it is the
> most demanding consumer of the virtual GPU's semantics, and how it waits.
> Canonical source: [`../docs/03-architecture/04-uvm.md`](../docs/03-architecture/04-uvm.md);
> where this page and the docs tree disagree, the docs tree wins.

Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` observed with the
stock guest driver; `[wip]` partially working, software bridge in charge;
`[gated]` implemented default-off; `[target]` planned end state.

## 1. What UVM is

UVM provides unified memory and GPU-driven page-fault handling. In practice it
is a **second GPU driver layered on the RM**, owning GPU address spaces and
running its own memory-management work on the GPU.

| Component | Role |
|---|---|
| VA spaces (`uvm_va_space`) | per-process GPU address spaces |
| Channel managers | pools of GPU channels (CE, SEC2, GR as needed) for UVM's work |
| Push buffers / GPFIFO | UVM builds and submits its own pushes |
| Semaphores and trackers | UVM waits on GPU-written semaphores to track completion |
| Page trees | GPU page tables for its VA spaces |
| Fault handling | fault buffer plus notification; UVM resolves and replays/cancels faults |

For CUDA, UVM is entered early: `cuInit` → driver init → `UVM_REGISTER_GPU`.
If UVM cannot register the GPU, CUDA does not start — which is why "UVM
advances" is the project's primary correctness metric on the way to CUDA.

## 2. The registration path (traced end to end)

```text
libcuda: UVM_REGISTER_GPU ioctl (0x25)
  uvm_api_register_gpu -> uvm_va_space_register_gpu -> uvm_gpu_retain_by_uuid
    nvUvmInterfaceRegisterGpu / GetGpuInfo; alloc_parent_gpu / add_gpu
      nvUvmInterfaceDeviceCreate          (RM object creation)
      uvm_hal_init_gpu / get_gpu_caps     (CE caps)
      uvm_ats_add_gpu / uvm_gpu_init_isr  (fault/interrupt plumbing)
    init_channel_manager
      create semaphore pool (sysmem)
      create channels per CE (GPFIFO + pushbuffer + semaphores)
      channel_init() -> first push -> semaphore release -> wait
    uvm_page_tree_init                      (page tables for the VA space)
```

Each stage has a distinct failure signature — see
[`Driver-Lifecycle-Field-Guide.md`](Driver-Lifecycle-Field-Guide.md).

## 3. What UVM requires from the "GPU"

| Requirement | Detail | Status |
|---|---|---|
| GPU identity | must match the UUID from RM | `[live]` |
| CE capabilities | per-engine support bits; UVM requires `supported && !grce && sysmem` for CPU_TO_GPU / GPU_TO_CPU channels | `[live]` caps accepted; `[gated]` copies forwarded to the physical copy engine |
| Channel creation | RM channel objects with GPFIFO/userd/instance parameters | `[live]` |
| Pushes executed | first action on a new channel is an init push plus a semaphore-tracked wait | `[wip]` software bridge executes memsets, copies and releases; `[gated]` CE-class copies forwarded |
| Page trees | UVM initializes page trees with its own pushes; server-owned ranges must be populated by us | `[wip]` |
| Fault delivery | fault buffer plus notification; replay/cancel semantics | `[gated]` backend-side observation/relay; guest contract `[target]` |
| Completion semantics | tracker values must advance exactly | `[wip]` |

**Current frontier.** Context creation and the full CUDA driver-API smoke pass
(`ALL OK`, copies matched) run end to end. CE-class copies are forwarded to the
physical copy engine (default off) with engine-authored releases and a fault
relay. For GR, a forwarded guest launch reaches the front-end but the inline
launch-descriptor install has not been observed; context validity at dispatch
is ruled out ([`Execution-Frontier.md`](Execution-Frontier.md)). The remaining
UVM work is completion translation and the guest-side fault replay/cancel
contract.

## 4. The discovery that defines the work: how UVM waits

UVM's tracker is the single most visible symptom of an unfaithful GPU. It
spins, and after 30 seconds it dumps its state:

```text
uvm_tracker.c:285 Tracker entry for value 1 (sema VA 0x121000000)
uvm_channel.c:4184 pending push '' started at uvm_channel.c:2535 in channel_init()
    releasing value 1
```

Mechanically, UVM built a push ending in a **semaphore release** — a method
sequence (CE channel class, e.g. `TURING_DMA_COPY_A`) that sets an address and
a payload and launches a DMA operation to write the payload to memory — then
submitted it and waits for the payload to appear. On real hardware the copy
engine writes it; on our device nobody does unless the emulation does. The
push we found in guest RAM decodes as:

```text
20010000 0000c5b5   set-object (TURING_DMA_COPY_A)
2001a000 0000c076   set-object (GP100_UVM_SW)
20038090 00000001   incrementing-method header: count 3, addr 0x240
21000000 00000001   lower 32 bits of the semaphore VA; payload (1)
200180c0 0000000c   LAUNCH_DMA with SEMAPHORE_TYPE=RELEASE_ONE_WORD
```

So the emulation's job for this push is: **write `payload` to the memory at
the semaphore's GPU VA**, which requires a GPU-VA → guest-PA translation. The
host does this through the learned VA→PA map: while the channel's GPFIFO
has pending entries it reads each push, decodes the method stream and runs the
work (page-table memsets, copies, semaphore releases). When a push page cannot
be resolved — or a channel consumes entries without any release landing — it
falls back to content: guest RAM is scanned for release groups and pending
payloads are applied forward-only. A release is applied only when
`payload > current`, so recycled push buffers and stale copies are harmless.
One pitfall observed live: a semaphore-page VA can flap between two physical
pages across map rebuilds; pinning the validated translation fixes the stalled
wait.

## 5. The memory model UVM assumes

* UVM's semaphore pool is **system memory** (`UVM_APERTURE_SYS`), one 4 KiB
  page per channel manager, addressed as 4-byte slots (1024 per page).
* Each page has a **GPU VA** (chosen by the client's RM) and a **CPU VA** (a
  kernel mapping). UVM polls the CPU VA; the GPU writes the GPU VA.
* Both addresses are recorded in UVM's structures in guest memory — the key
  that makes translation possible without extra help from the guest.
* The CPU VA is a direct-map address, so `PA = cpu_va - page_offset_base`
  (verified live: writing the payload there completes the push).
* Values are per channel and monotonic (1, 2, 3 …); UVM's own check
  (`UVM_GPU_SEMAPHORE_MAX_JUMP`) detects and reports wrong values.
* The client's page tables for **server-owned ranges are empty chains**: the
  client hands us the physical addresses of its page-table levels and expects
  the server to own the lower levels ([`GSP-and-GSP-RM.md`](GSP-and-GSP-RM.md)).

## 6. Page trees and faults

`uvm_page_tree_init` initializes a VA space's page tables with pushes on UVM's
own channels (memops and page-table writes; the copy engine executes them on
real hardware). Fault handling is event-driven: UVM programs a fault buffer
(replayable and non-replayable; sizes from `INTERNAL_GMMU_GET_STATIC_INFO`,
answered by the emulation) and, on a fault, reads the entry, populates the
page table and replays or cancels the fault with a `MEM_OP` push. Reconciling
the guest's memory model with the real GPU's here is the deepest remaining work.

## 7. How to observe UVM

In the guest, UVM prints nothing by default:

```bash
echo 1 | sudo tee /sys/module/nvidia_uvm/parameters/uvm_debug_prints
```

Messages then appear with the `nvidia-uvm:` prefix. Known-good markers, in
order: `uvm_gpu_retain_by_uuid` → channel manager created →
`uvm_page_tree_init` pushes → `UVM_REGISTER_GPU` returns and CUDA proceeds.
Failure signatures characterized live:

| Message | Meaning |
|---|---|
| `Failed to initialize the channel manager: NV_ERR_NOT_SUPPORTED` | CE caps wrong or missing |
| `stuck waiting` plus `Tracker entry ... channel_init()` | the push's semaphore was never written |
| `Assert failed ... unexpected semaphore ... jump from 0x0 to 0x1010101` | a wrong payload was written; UVM caught it |

## 8. Why UVM is the perfect correctness oracle

UVM exercises in minutes what a driver test suite would test in hours: memory
allocation, page tables, channel setup, push generation, semaphore semantics,
fault handling and error recovery — all with continuous self-checks and loud
diagnostics. When UVM registers cleanly and CUDA workloads pass, the device is
behaving like a real GPU.

## 9. Related documentation

* [`../docs/03-architecture/04-uvm.md`](../docs/03-architecture/04-uvm.md) — canonical chapter (this page's source).
* [`Guest-Stack-and-CUDASW.md`](Guest-Stack-and-CUDASW.md), [`GSP-and-GSP-RM.md`](GSP-and-GSP-RM.md) — the guest stack and the RM objects UVM consumes.
* [`CUDA-Userspace-Driver.md`](CUDA-Userspace-Driver.md) — the userspace half of the contract; [`Execution-Plane.md`](Execution-Plane.md), [`Execution-Frontier.md`](Execution-Frontier.md) — how UVM's pushes execute, and the current wall.
* [`../docs/04-implementation/01-known-knowns-and-unknowns.md`](../docs/04-implementation/01-known-knowns-and-unknowns.md) — server-owned page tables and the fault contract.