1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
|
# Open-IOV Current State
> **Snapshot: 2026-09-30 05:54.** This is a dated public status page, not live
> state; it is refreshed on every progress check. Live working notes are
> maintained outside this repository; the public known-knowns/unknowns document
> is authoritative here:
> [`docs/04-implementation/01-known-knowns-and-unknowns.md`](docs/04-implementation/01-known-knowns-and-unknowns.md).
> The 2026-09-21 snapshot remains in [`STATE.md`](STATE.md) as a historical
> record.
## Mission
Run **unmodified NVIDIA guest drivers** in VMs with full GPU functionality
(CUDA, graphics, video, AI/ML) on consumer/data-center GPUs without
proprietary GRID licenses - via host-side GSP emulation in
`open-iov-vfio.ko`. The guest stays stock: no guest patches, no guest-side
Open-IOV code.
Architecture stance: *emulate the interface, pass the work through to real
hardware, never patch anyone.* The control plane is emulated; the work is
meant to execute on the physical GPU.
## Milestone ladder (this snapshot)
| Milestone | Status |
|---|---|
| GSP boot handshake, GSP-RM message queues/RPC, BAR/PCI/MSI-X emulation | ✅ |
| Stock 570.144 guest driver probe / `RmInitAdapter` | ✅ |
| `nvidia-smi -L` enumerates the emulated TU104 | ✅ |
| CUDA userspace: `cuInit`, `cuDeviceGetCount`, `cuDeviceGet` | ✅ |
| `cuCtxCreate` returns | ✅ (the whole smoke test now completes) |
| `cuMemAlloc` + `cuMemcpy` roundtrip (`ALL OK` in the smoke test) | 🟡 all calls succeed and the test prints `ALL OK`; the 64-byte data roundtrip is not yet correct |
| Full `nvidia-smi` output | ❌ (faults on the CUDA path) |
| CUDA kernel execution (`vectorAdd`) | ❌ not started (forwarding layer absent) |
| Graphics (3D), video | ❌ |
| Faults, multi-version, Windows, migration, multi-tenancy, performance | ❌ |
Test status is tracked with the CUDA driver-API smoke test
([`tools/test/cuda_drv_test.c`](tools/test/cuda_drv_test.c)): it exercises
`cuInit -> cuDeviceGetCount/Get -> cuCtxCreate -> cuMemAlloc -> cuMemcpyHtoD/
DtoH -> cuMemFree -> cuCtxDestroy` and prints `ALL OK` only when the whole
chain succeeds.
## Right now (frontier)
- **The smoke test now completes end-to-end**: `cuInit -> cuDeviceGet ->
cuCtxCreate -> cuMemAlloc -> cuMemcpyHtoD -> cuMemcpyDtoH -> cuMemFree ->
cuCtxDestroy` all succeed and the test prints `ALL OK`. The context-creation
error that stalled this for days is root-caused and fixed: the emulated GR
information table left the shader-pipe count and related clock fields zero,
which the client's clock-table validation rejected.
- **The remaining wall is the user payload**: the memory roundtrip returns
zeros - the 64-byte payload does not reach the emulated VRAM through the
path this allocation uses. The copies the host observes carry UVM
bookkeeping, not the payload; a full scan of the pinned copy-engine push
buffer and of the 256 MB VRAM shadow finds no trace of it. The next suspect
is the external-allocation path the client uses (RM-allocated memory mapped
into UVM), including a possible windowed CPU copy.
- Recap of the fixes behind this point: all 20/20 compute-channel releases
land (three decode paths routed to the exact counter slot); registration
delivers the incrementing tracking release for every consumed submission;
the copy-engine push buffer is pinned from guest truth on every boot; and
the copy-engine decoder consults pinned translations before the scanned
map, so every push-data operation executes (`memset=19473`, `ringops=10551`
live).
- Active work: find the payload's host-to-device path for the external
allocation - pattern-match the copy's source, audit the RM
allocation/descriptor path, and check where windowed writes land.
- Observability: a size-keyed copy watch, a VRAM shadow dump/search, and an
operation log are now module parameters, and the test's output survives
guest-agent stalls via the serial console.
- Guest health note: agent/SSH timeouts during long runs are the guest-side
trace reader flooding the ring; they recover on their own, so a first
timeout is not a reason to restart the VM.
- Execution backend (chosen, not started): the module will own the physical
GPU and run its real GSP firmware as the resource-manager server, so real
work executes with an unmodified host driver and no host daemon; see
[`docs/03-architecture/08-physical-gpu-integration.md`](docs/03-architecture/08-physical-gpu-integration.md).
- Host stability: the host hang class observed on 2026-09-28 (kernel stack
overflow in a channel snapshot path) is fixed and frame-audited; repeated
VM load/attach/teardown cycles run without panics or pstore records.
## Initialization, work and teardown path
Status tags: `[done]`, `[frontier]` (actively worked), `[partial]`,
`[not yet]`.
```
HOST SIDE GUEST (stock driver, unmodified)
========= ================================
[done] build + load open-iov-vfio.ko
| (module first: exports the symbols the NVIDIA open modules link against)
v
[done] mdev create ------------------------------------> /
| BAR/PCI/MSI-X emulation; GSP emulation thread
v
[done] VM attach (QEMU q35 + pcie-root-port) ----------> config space read; BAR mmap
|
v
[done] GSP boot handshake (emulated) ------------------> nvidia.ko probe / RmInitAdapter
|
v
[done] GSP-RM message queues / RPC / controls ---------> object + client setup
|
v
[done] engine + class lists ---------------------------> GPU_GET_ENGINES, class DB
| (two-GRCE fix; async copy engine)
v
[done] ------------------------------------------------> nvidia-smi -L enumerates TU104
|
v
[done] ------------------------------------------------> cuInit / DeviceGetCount / DeviceGet
|
v
[done] ------------------------------------------------> cuCtxCreate
| UVM register GPU; channel manager
| GR + CE channels; push buffers
| semaphore pool; completion counters
v
[partial] ---------------------------------------------> cuMemAlloc + cuMemcpy roundtrip
| (all calls succeed; payload data path pending)
[not yet] ---------------------------------------------> full nvidia-smi output
```
Work execution (host service vs client synchronization):
```
[partial] host decodes guest pushes (method streams): memsets, copies, releases
- CE pushes execute at scale (memset=19473, ringops=10551; misses at zero)
- the 64-byte test payload's HtoD path is not decoded yet
[partial] GPU VA -> guest PA translation and pinning; server-owned page tables in progress
[not yet] real work forwarded to the physical GPU (host-RM proxy or direct submission)
[not yet] correct results (vectorAdd)
```
Teardown and recovery:
```
[done] happy-path teardown exercised end-to-end (cuCtxDestroy OK)
[done] VM stop -> mdev remove -> rmmod, in order; never rmmod with a live VM attached
[done] crash recovery: pstore inspection; the host hang class is fixed and frame-audited
```
Recent chain of walls removed (2026-09-28 to 2026-09-30):
```
host hang (stack overflow) -> pump state persistence -> map collapse (merge rebuild)
-> rescue overhaul -> four-word semaphore releases -> counter injection (all 20)
-> stale semaphore-pool translation -> fake-only proven insufficient
-> push buffer found and validated -> per-page pins -> authoritative pin table
-> deterministic chain solved -> pin-drop and release-push fixes
-> copy-engine data path executes -> compute-window pinning
-> registration release delivery -> 20/20 counter slots -> GR-info clock fix
-> cuCtxCreate OK and the full smoke test ALL OK; payload data path remains
```
## Tools, code paths and fixes behind this progress
```
KERNEL MODULE (open-iov-vfio.ko) GUEST-SIDE TOOLS (tracked)
================================ ==========================
gsp_emulation.c boot, doorbells, MSI-X tools/test/cuda_drv_test.c smoke test
gsp_rm_emulation.c msgq, RPC, controls tools/guest-diag/vmrestart.sh
gsp_rm_tu104.h two-GRCE engine fix tools/guest-diag/cuda_cycle.sh
gsp_sema_service.c THE WORKHORSE tools/guest-diag/push_guest_tools.sh
channel pump, method decode, rescue, tools/guest-diag/inject_counters.sh
map rebuild, pins, counters, releases tools/guest-diag/semfix_watch.py
bar_emulation.c BAR shadows tools/guest-diag/track_waits.sh
main.c mdev/VFIO lifecycle tools/guest-diag/trkinfo.py
stack-frame audit (-Wframe-larger-than) tools/guest-diag/bp999.py / scanerr.py
dynamic_rpc_dispatcher.c firmware load fix tools/guest-diag/read_va.py
tools/guest-diag/guest_kprobes.sh
DATA STRUCTURES DEBUG PLANES
=============== ============
channel table (handle, GPFIFO, USERD, module stats: applied / miss / memset /
notifier, ring) ringops / nomap / skip / faked /
VA->PA map (merge rebuild, hot pages, cache) pins / ipins
pin table + authoritative injected pin table debug params: sema_probe, sema_counter,
completion counter table (per channel) sema_map_add, chan_dump, shadow_dump
semaphore pool page (GPU VA 0x121000000) QMP gva2gpa + guest gdbstub reads
push buffer allocation (GPU VA 0x120000000) guest kprobes; /proc/<pid>/pagemap
client trackers / RM error record chain marker-based /dev/kmsg parsing
```
| Wall | Fix |
|---|---|
| Host panic: oversized stack frame in a channel snapshot path | static buffers; frame-size audit on every build |
| Channel pump lost per-pass state (stale entries re-run; rate limit dead) | write-back of rescue/payload state; one-retry skip for compute channels |
| VA-to-PA map collapsed after client teardown | merge rebuild instead of wholesale replace |
| Rescue slow / false candidates | hot-page-first scan; full method-stream validation; retry limits |
| Four-word semaphore releases ignored | decode both release encodings |
| Only 4 of 20 completion counters ever moved | async injection queue; fast injector; per-boot pin, order fixed |
| UVM tracker wait never cleared | stale pool translation found; pool page pinned from guest truth each boot |
| Acknowledging releases without executing work | proven insufficient; pushes must be read and executed |
| Push buffers unreadable | located by CPU VA; per-page translation; authoritative injected pin table |
| Injected pins silently dropped once a pin-table cluster grew | authoritative injected-pin table checked first; larger table, longer probe |
| Real release pushes rejected as implausible (non-zero method start) | decode release-only when the translation came from injected pins |
| Completion release decoded but deferred (counter page untranslated) | pin the compute-channel mapping window from guest truth each boot |
## What "CUDA works" means (open gates)
Per [`docs/04-implementation/01-known-knowns-and-unknowns.md`](docs/04-implementation/01-known-knowns-and-unknowns.md)
section 4, a defensible claim needs **all** of:
1. every guest-to-device message for the CUDA path answered or forwarded,
with coherent replies for consumed fields;
2. every work VA translatable and server-owned page-table ranges populated;
3. launches of a grid of thread blocks (a CUDA kernel dispatch, e.g.
`vectorAdd`'s `<<<blocks, threads>>>`) actually executed on the physical
GPU's compute engine - not merely acknowledged - and copies executing on
real hardware with correct results;
4. completion (semaphores, events, faults) behaving as the guest expects;
5. at least two driver families passing;
6. tenant isolation;
7. measured performance within an agreed factor of native.
Until then the honest statement is: **initialization works; execution is in
progress.**
## Environment
- Host GPU: NVIDIA TU104 (RTX 2080 SUPER Mobile), PCI `10de:1ed3`.
- Host OS: Ubuntu, kernel 6.8.0-142; the distribution NVIDIA driver manages
the physical GPU; `open-iov-vfio.ko` loads as an independent module.
- Guest: Ubuntu 24.04.5 LTS, stock NVIDIA 570.144 driver, QEMU/KVM q35 with a
`pcie-root-port` for the vGPU, SeaBIOS, mdev passthrough.
- Firmware: TU104 GSP 570.144 (public release).
## Build and load order
Build: host module first (it generates the symbol versions the NVIDIA open
kernel modules link against), then the NVIDIA open kernel modules, then
userspace.
Load: VFIO prerequisites, then `open-iov-vfio.ko`, then create the mdev and
attach it to the VM. Unload in reverse; never `rmmod` while a VM holds the
device.
## See also
- [`docs/README.md`](docs/README.md) - documentation front door
- [`docs/04-implementation/01-known-knowns-and-unknowns.md`](docs/04-implementation/01-known-knowns-and-unknowns.md) - known-knowns, known-unknowns, definition of done
- [`userspace/cuda/READ-ME-FIRST.md`](userspace/cuda/READ-ME-FIRST.md) - CUDA analysis entry point
- [`docs/03-architecture/08-physical-gpu-integration.md`](docs/03-architecture/08-physical-gpu-integration.md) - chosen execution backend (module-owned GPU + real GSP session)
- [`docs/03-architecture/09-full-system-picture.md`](docs/03-architecture/09-full-system-picture.md) - visual end-to-end system map with status tags
- [`docs/03-architecture/10-interaction-and-scheduling.md`](docs/03-architecture/10-interaction-and-scheduling.md) - cross-component interaction, queues, interrupts, scheduling detail
- [`STATE.md`](STATE.md) - 2026-09-21 historical snapshot
---
*Refresh this page on every progress check (snapshot time in the header);
prefer moving detail into the `docs/` tree and keeping this page short.*
|
