# UVM — Unified Memory in the Guest > UVM (`nvidia_uvm.ko`) is the most demanding consumer of the virtual GPU's > semantics, and today the best measurement of how faithful our emulation is. > This document explains what UVM does, what it needs, and what we learned > from watching it run against our device. ## 1. What UVM is UVM provides unified memory and GPU-driven page-fault handling. In practice it is a second GPU driver, layered on the RM: * **VA spaces** (`uvm_va_space`) — per-process GPU address spaces. * **Channel managers** — pools of GPU channels (CE, SEC2, GR-as-needed) that UVM itself uses to execute its own memory-management work. * **Push buffers / GPFIFO** — UVM builds its own pushes and submits them. * **Semaphores and trackers** — UVM tracks GPU progress by waiting on semaphores that the GPU writes when work completes. * **Page trees** — UVM manages GPU page tables for its VA spaces. * **Fault handling** — GPU page faults are delivered to UVM (via a fault buffer and a notification channel), and UVM resolves them by populating page tables and replaying/cancelling faults. For CUDA, UVM is entered early: `cuInit` → driver init → `UVM_REGISTER_GPU`. If UVM cannot complete GPU registration, CUDA does not start. ## 2. The registration path (traced end to end) ``` libcuda: UVM_REGISTER_GPU ioctl (0x25) uvm_api_register_gpu uvm_va_space_register_gpu uvm_gpu_retain_by_uuid nvUvmInterfaceRegisterGpu / GetGpuInfo alloc_parent_gpu / add_gpu nvUvmInterfaceDeviceCreate (RM object creation) uvm_hal_init_gpu / get_gpu_caps (CE caps! see below) uvm_ats_add_gpu / uvm_gpu_init_isr (fault/intr plumbing) init_channel_manager create semaphore pool (sysmem) create channels per CE (each channel: GPFIFO + pushbuffer + semaphores) channel_init() → first push → semaphore release → wait uvm_page_tree_init (page tables for the VA space) ``` Each stage has a distinct failure signature; our debugging docs map them. ## 3. What UVM requires from the "GPU" | Requirement | Detail | Our status | |---|---|---| | GPU identity | must match the uuid from RM | ✅ | | CE capabilities | per-engine support bits; UVM *requires* `supported && !grce && sysmem` for CPU_TO_GPU/GPU_TO_CPU channels | ✅ (we answer CE_GET_PHYSICAL_CAPS with the correct bits) | | Channel creation | RM channel objects with GPFIFO/userd/instance parameters | ✅ objects created | | Pushes executed | UVM's first action on a new channel is an *init push* followed by a semaphore-tracked wait | 🟡 we can complete them (see below) | | Page trees | UVM initializes page trees with its own pushes; the *server-owned* ranges must be populated by us | 🟡 | | Fault delivery | fault buffer + notification; replay/cancel semantics | 🔴 | | Semantics of completions | tracker values must advance exactly | 🟡 | ## 4. The most important discovery: how UVM waits UVM's tracker (`uvm_tracker.c`) is the single most visible symptom of an unfaithful GPU. It spins, and after 30 seconds it dumps: ``` uvm_tracker.c:285 Tracker entry for value 1 (sema VA 0x121000000) channel ID 1:3 (0x1:0x3) CE 0 uvm_channel.c:4184 pending push '' started at uvm_channel.c:2535 in channel_init() releasing value 1 ``` What this means, mechanically: 1. UVM built a push in a push buffer and wrote a **semaphore release** into it: a method sequence (CE channel class, e.g. `TURING_DMA_COPY_A`) that sets an address and a payload and then launches a DMA operation that writes the payload to memory. 2. UVM submitted the push (GPFIFO entry + GPPUT/doorbell) and now waits for the semaphore payload to reach `value`. 3. On real hardware, the copy engine writes the payload into the semaphore's memory. On our device, *nobody does* unless we do it. The push we found in guest RAM (found by scanning for its distinctive byte sequence) decodes as: ``` 20010000 0000c5b5 set-object (TURING_DMA_COPY_A) 2001a000 0000c076 set-object (GP100_UVM_SW) 20038090 00000001 incrementing-method header: count 3, addr 0x240 + data: upper 32 bits of the semaphore VA 21000000 00000001 lower 32 bits of the VA; payload (1) 200180c0 0000000c LAUNCH_DMA with SEMAPHORE_TYPE=RELEASE_ONE_WORD ``` So: **the emulation's job for this push is to write `payload` to the memory at the semaphore's GPU VA.** Which requires a GPU-VA → guest-PA translation. ## 5. The memory model UVM assumes (and what we learned) * UVM's semaphore pool is **system memory** (`pool->aperture = UVM_APERTURE_SYS`), one 4 KiB page per channel manager, addressed as 4-byte slots (`UVM_SEMAPHORE_SIZE = 4`, 1024 slots per page). * Each semaphore page has a *GPU VA* (chosen by the client's RM) and a *CPU VA* (a kernel mapping). UVM polls the CPU VA; the GPU writes the GPU VA. * **Both addresses are recorded in UVM's structures in guest memory.** This is the key that makes translation possible without any extra help from the guest (see the host-module doc for the exact structures and offsets). * The CPU VA is a **direct-map** address, so `PA = cpu_va - page_offset_base`. Verified live: writing the payload to that physical address completes the push and UVM advances. * Values are *per channel* and *monotonic*: a channel's first push releases 1, the next 2, and so on. UVM has its own sanity check (`UVM_GPU_SEMAPHORE_MAX_JUMP`) — writing a wrong value is detected and reported, which is useful for validation. * The client's **GPU page tables for server-owned ranges are empty chains**: the client hands us the *physical addresses of its page-table levels* in `NV90F1_CTRL_CMD_VASPACE_COPY_SERVER_RESERVED_PDES` and expects the server to own the lower levels. (See the GSP-RM doc, §5.) ## 6. Page trees and faults * `uvm_page_tree_init` initializes the page tables for a VA space using pushes on UVM's own channels. These pushes are memops / page-table writes; on real hardware they are executed by the copy engine. * Fault handling is event-driven: UVM programs a **fault buffer** (a replayable and a non-replayable buffer, whose sizes come from `INTERNAL_GMMU_GET_STATIC_INFO` — answered by our emulation) and receives notifications when the GPU takes a fault. UVM then: 1. reads the fault entry, 2. populates the page table (possibly migrating memory), 3. **replays or cancels the fault** with a `MEM_OP` push. * For our forwarding design, faults are where "the guest's memory model" and "the real GPU's memory model" must be reconciled. This is the deepest part of the remaining work, and is tracked as a known-unknown in [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md). ## 7. How to observe UVM In the guest: ```bash # UVM prints nothing by default. Turn it on: echo 1 | sudo tee /sys/module/nvidia_uvm/parameters/uvm_debug_prints # Messages then appear with the "nvidia-uvm:" prefix. # Other knobs: uvm_release_asserts, uvm_release_asserts_dump_stack, # uvm_enable_debug_procfs, uvm_debug_enable_push_desc. ``` Known-good markers, in order: ``` uvm_gpu_retain_by_uuid → ... → channel manager created → uvm_page_tree_init pushes → (on success) UVM_REGISTER_GPU returns and CUDA proceeds ``` Failure signatures we have characterized: | Message | Meaning | |---|---| | `Failed to initialize the channel manager: NV_ERR_NOT_SUPPORTED` | CE caps were wrong/missing (we fixed this: see the field guide) | | `stuck waiting` + `Tracker entry ... channel_init()` | the push's semaphore was never written (the emulation must write it) | | `Assert failed ... unexpected semaphore ... jump from 0x0 to 0x1010101` | we wrote a wrong payload (and UVM caught us — useful!) | ## 8. Why UVM is the perfect correctness oracle UVM exercises in minutes what a driver test suite would test in hours: memory allocation, page tables, channel setup, push generation, semaphore semantics, fault handling, error recovery — all with continuous self-checks and loud diagnostics. When UVM registers cleanly and CUDA workloads pass, the device is behaving like a real GPU. That is why "UVM advances" is the project's primary correctness metric on the way to CUDA.