1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
|
# UVM in the Guest
> **Scope.** What UVM (`nvidia_uvm.ko`) does inside the guest, why it is the
> most demanding consumer of the virtual GPU's semantics, and how it waits.
> Canonical source: [`../docs/03-architecture/04-uvm.md`](../docs/03-architecture/04-uvm.md);
> where this page and the docs tree disagree, the docs tree wins.
Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` observed with the
stock guest driver; `[wip]` partially working, software bridge in charge;
`[gated]` implemented default-off; `[target]` planned end state.
## 1. What UVM is
UVM provides unified memory and GPU-driven page-fault handling. In practice it
is a **second GPU driver layered on the RM**, owning GPU address spaces and
running its own memory-management work on the GPU.
| Component | Role |
|---|---|
| VA spaces (`uvm_va_space`) | per-process GPU address spaces |
| Channel managers | pools of GPU channels (CE, SEC2, GR as needed) for UVM's work |
| Push buffers / GPFIFO | UVM builds and submits its own pushes |
| Semaphores and trackers | UVM waits on GPU-written semaphores to track completion |
| Page trees | GPU page tables for its VA spaces |
| Fault handling | fault buffer plus notification; UVM resolves and replays/cancels faults |
For CUDA, UVM is entered early: `cuInit` → driver init → `UVM_REGISTER_GPU`.
If UVM cannot register the GPU, CUDA does not start — which is why "UVM
advances" is the project's primary correctness metric on the way to CUDA.
## 2. The registration path (traced end to end)
```text
libcuda: UVM_REGISTER_GPU ioctl (0x25)
uvm_api_register_gpu -> uvm_va_space_register_gpu -> uvm_gpu_retain_by_uuid
nvUvmInterfaceRegisterGpu / GetGpuInfo; alloc_parent_gpu / add_gpu
nvUvmInterfaceDeviceCreate (RM object creation)
uvm_hal_init_gpu / get_gpu_caps (CE caps)
uvm_ats_add_gpu / uvm_gpu_init_isr (fault/interrupt plumbing)
init_channel_manager
create semaphore pool (sysmem)
create channels per CE (GPFIFO + pushbuffer + semaphores)
channel_init() -> first push -> semaphore release -> wait
uvm_page_tree_init (page tables for the VA space)
```
Each stage has a distinct failure signature — see
[`Driver-Lifecycle-Field-Guide.md`](Driver-Lifecycle-Field-Guide.md).
## 3. What UVM requires from the "GPU"
| Requirement | Detail | Status |
|---|---|---|
| GPU identity | must match the UUID from RM | `[live]` |
| CE capabilities | per-engine support bits; UVM requires `supported && !grce && sysmem` for CPU_TO_GPU / GPU_TO_CPU channels | `[live]` caps accepted; `[gated]` copies forwarded to the physical copy engine |
| Channel creation | RM channel objects with GPFIFO/userd/instance parameters | `[live]` |
| Pushes executed | first action on a new channel is an init push plus a semaphore-tracked wait | `[wip]` software bridge executes memsets, copies and releases; `[gated]` CE-class copies forwarded |
| Page trees | UVM initializes page trees with its own pushes; server-owned ranges must be populated by us | `[wip]` |
| Fault delivery | fault buffer plus notification; replay/cancel semantics | `[gated]` backend-side observation/relay; guest contract `[target]` |
| Completion semantics | tracker values must advance exactly | `[wip]` |
**Current frontier.** Context creation and the full CUDA driver-API smoke pass
(`ALL OK`, copies matched) run end to end. CE-class copies are forwarded to the
physical copy engine (default off) with engine-authored releases and a fault
relay. For GR, a forwarded guest launch reaches the front-end but the inline
launch-descriptor install has not been observed; context validity at dispatch
is ruled out ([`Execution-Frontier.md`](Execution-Frontier.md)). The remaining
UVM work is completion translation and the guest-side fault replay/cancel
contract.
## 4. The discovery that defines the work: how UVM waits
UVM's tracker is the single most visible symptom of an unfaithful GPU. It
spins, and after 30 seconds it dumps its state:
```text
uvm_tracker.c:285 Tracker entry for value 1 (sema VA 0x121000000)
uvm_channel.c:4184 pending push '' started at uvm_channel.c:2535 in channel_init()
releasing value 1
```
Mechanically, UVM built a push ending in a **semaphore release** — a method
sequence (CE channel class, e.g. `TURING_DMA_COPY_A`) that sets an address and
a payload and launches a DMA operation to write the payload to memory — then
submitted it and waits for the payload to appear. On real hardware the copy
engine writes it; on our device nobody does unless the emulation does. The
push we found in guest RAM decodes as:
```text
20010000 0000c5b5 set-object (TURING_DMA_COPY_A)
2001a000 0000c076 set-object (GP100_UVM_SW)
20038090 00000001 incrementing-method header: count 3, addr 0x240
21000000 00000001 lower 32 bits of the semaphore VA; payload (1)
200180c0 0000000c LAUNCH_DMA with SEMAPHORE_TYPE=RELEASE_ONE_WORD
```
So the emulation's job for this push is: **write `payload` to the memory at
the semaphore's GPU VA**, which requires a GPU-VA → guest-PA translation. The
host does this through the learned VA→PA map: while the channel's GPFIFO
has pending entries it reads each push, decodes the method stream and runs the
work (page-table memsets, copies, semaphore releases). When a push page cannot
be resolved — or a channel consumes entries without any release landing — it
falls back to content: guest RAM is scanned for release groups and pending
payloads are applied forward-only. A release is applied only when
`payload > current`, so recycled push buffers and stale copies are harmless.
One pitfall observed live: a semaphore-page VA can flap between two physical
pages across map rebuilds; pinning the validated translation fixes the stalled
wait.
## 5. The memory model UVM assumes
* UVM's semaphore pool is **system memory** (`UVM_APERTURE_SYS`), one 4 KiB
page per channel manager, addressed as 4-byte slots (1024 per page).
* Each page has a **GPU VA** (chosen by the client's RM) and a **CPU VA** (a
kernel mapping). UVM polls the CPU VA; the GPU writes the GPU VA.
* Both addresses are recorded in UVM's structures in guest memory — the key
that makes translation possible without extra help from the guest.
* The CPU VA is a direct-map address, so `PA = cpu_va - page_offset_base`
(verified live: writing the payload there completes the push).
* Values are per channel and monotonic (1, 2, 3 …); UVM's own check
(`UVM_GPU_SEMAPHORE_MAX_JUMP`) detects and reports wrong values.
* The client's page tables for **server-owned ranges are empty chains**: the
client hands us the physical addresses of its page-table levels and expects
the server to own the lower levels ([`GSP-and-GSP-RM.md`](GSP-and-GSP-RM.md)).
## 6. Page trees and faults
`uvm_page_tree_init` initializes a VA space's page tables with pushes on UVM's
own channels (memops and page-table writes; the copy engine executes them on
real hardware). Fault handling is event-driven: UVM programs a fault buffer
(replayable and non-replayable; sizes from `INTERNAL_GMMU_GET_STATIC_INFO`,
answered by the emulation) and, on a fault, reads the entry, populates the
page table and replays or cancels the fault with a `MEM_OP` push. Reconciling
the guest's memory model with the real GPU's here is the deepest remaining work.
## 7. How to observe UVM
In the guest, UVM prints nothing by default:
```bash
echo 1 | sudo tee /sys/module/nvidia_uvm/parameters/uvm_debug_prints
```
Messages then appear with the `nvidia-uvm:` prefix. Known-good markers, in
order: `uvm_gpu_retain_by_uuid` → channel manager created →
`uvm_page_tree_init` pushes → `UVM_REGISTER_GPU` returns and CUDA proceeds.
Failure signatures characterized live:
| Message | Meaning |
|---|---|
| `Failed to initialize the channel manager: NV_ERR_NOT_SUPPORTED` | CE caps wrong or missing |
| `stuck waiting` plus `Tracker entry ... channel_init()` | the push's semaphore was never written |
| `Assert failed ... unexpected semaphore ... jump from 0x0 to 0x1010101` | a wrong payload was written; UVM caught it |
## 8. Why UVM is the perfect correctness oracle
UVM exercises in minutes what a driver test suite would test in hours: memory
allocation, page tables, channel setup, push generation, semaphore semantics,
fault handling and error recovery — all with continuous self-checks and loud
diagnostics. When UVM registers cleanly and CUDA workloads pass, the device is
behaving like a real GPU.
## 9. Related documentation
* [`../docs/03-architecture/04-uvm.md`](../docs/03-architecture/04-uvm.md) — canonical chapter (this page's source).
* [`Guest-Stack-and-CUDASW.md`](Guest-Stack-and-CUDASW.md), [`GSP-and-GSP-RM.md`](GSP-and-GSP-RM.md) — the guest stack and the RM objects UVM consumes.
* [`CUDA-Userspace-Driver.md`](CUDA-Userspace-Driver.md) — the userspace half of the contract; [`Execution-Plane.md`](Execution-Plane.md), [`Execution-Frontier.md`](Execution-Frontier.md) — how UVM's pushes execute, and the current wall.
* [`../docs/04-implementation/01-known-knowns-and-unknowns.md`](../docs/04-implementation/01-known-knowns-and-unknowns.md) — server-owned page tables and the fault contract.
|
