1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
|
# Open-IOV Current State
> **Snapshot: 2026-09-29 23:57.** This is a dated public status page, not live
> state; it is refreshed on every progress check. Live working notes are
> maintained outside this repository; the public known-knowns/unknowns document
> is authoritative here:
> [`docs/04-implementation/01-known-knowns-and-unknowns.md`](docs/04-implementation/01-known-knowns-and-unknowns.md).
> The 2026-09-21 snapshot remains in [`STATE.md`](STATE.md) as a historical
> record.
## Mission
Run **unmodified NVIDIA guest drivers** in VMs with full GPU functionality
(CUDA, graphics, video, AI/ML) on consumer/data-center GPUs without
proprietary GRID licenses - via host-side GSP emulation in
`open-iov-vfio.ko`. The guest stays stock: no guest patches, no guest-side
Open-IOV code.
Architecture stance: *emulate the interface, pass the work through to real
hardware, never patch anyone.* The control plane is emulated; the work is
meant to execute on the physical GPU.
## Milestone ladder (this snapshot)
| Milestone | Status |
|---|---|
| GSP boot handshake, GSP-RM message queues/RPC, BAR/PCI/MSI-X emulation | ✅ |
| Stock 570.144 guest driver probe / `RmInitAdapter` | ✅ |
| `nvidia-smi -L` enumerates the emulated TU104 | ✅ |
| CUDA userspace: `cuInit`, `cuDeviceGetCount`, `cuDeviceGet` | ✅ |
| `cuCtxCreate` returns | ⏳ **frontier** |
| `cuMemAlloc` + `cuMemcpy` roundtrip (`ALL OK` in the smoke test) | ❌ not reached |
| Full `nvidia-smi` output | ❌ (faults on the CUDA path) |
| CUDA kernel execution (`vectorAdd`) | ❌ not started (forwarding layer absent) |
| Graphics (3D), video | ❌ |
| Faults, multi-version, Windows, migration, multi-tenancy, performance | ❌ |
Test status is tracked with the CUDA driver-API smoke test
([`tools/test/cuda_drv_test.c`](tools/test/cuda_drv_test.c)): it exercises
`cuInit -> cuDeviceGetCount/Get -> cuCtxCreate -> cuMemAlloc -> cuMemcpyHtoD/
DtoH -> cuMemFree -> cuCtxDestroy` and prints `ALL OK` only when the whole
chain succeeds.
## Right now (frontier)
- The stock guest driver initializes fully; the CUDA client reaches
`cuCtxCreate` within seconds and now fails with a forced `rc=999` a few
seconds after the compute-channel phase (a timeout path, not an RM error
record).
- The per-channel completion counters in guest memory are enumerated,
registered and published; the client's per-channel waits clear, and the
per-channel GPFIFO work on copy channels is consumed.
- Both engine push-buffer streams are reachable: the copy-engine buffer is
resolved deterministically, and the compute-engine mapping window plus
GPFIFO rings are pinned from guest page tables on every boot (a guest-side
scanner streams page-map deltas to the host).
- **The copy-engine data path executes**: decoded method streams run in
sustained fashion (memsets, copies, releases; thousands of push entries
consumed), and compute-channel releases decode.
- **Both counter walls are falling**: live raw dumps of the client's counter
page show all 20 compute-channel counter slots written at the correct
16-byte stride (host releases now land exactly where the client polls), and
auto-registration walks the full channel rank range.
- **The registration wall is root-caused and fixed**: the host consumed
channel submissions without delivering their tracking-semaphore release;
every consumed engine submission now delivers the incrementing release, and
the pool value advances within a second (previously it stalled until the
deadline). Registration no longer dominates the run.
- **New frontier: the guest's own GSP health check**: a second after the
compute-channel phase the guest driver runs a GSP health check while one of
its resource-manager replies is pending, then rolls the context back. Two
suspects under test: reply-path latency now that releases are applied at a
high rate, and a reply whose parameter size is not canonical (the guest
logs `NV_ERR_INVALID_PARAMETER` on its reply-cache path).
- Active work: an extended debug loop on the health-check failure - more
one-shot breakpoint tooling and fresh failure-instant guest/host snapshots
are being produced, with kernel and tool edits still in the tree awaiting
the next build/commit.
- Guest health note: agent/SSH timeouts during a run are usually guest-agent
starvation and recover when the client stops; one hard wedge occurred only
when pushed memsets were re-enabled, so bulk pushed operations stay disabled
while stale translations can exist.
- Execution backend (chosen, not started): the module will own the physical
GPU and run its real GSP firmware as the resource-manager server, so real
work executes with an unmodified host driver and no host daemon; see
[`docs/03-architecture/08-physical-gpu-integration.md`](docs/03-architecture/08-physical-gpu-integration.md).
- Host stability: the host hang class observed on 2026-09-28 (kernel stack
overflow in a channel snapshot path) is fixed and frame-audited; repeated
VM load/attach/teardown cycles run without panics or pstore records.
## Initialization, work and teardown path
Status tags: `[done]`, `[frontier]` (actively worked), `[partial]`,
`[not yet]`.
```
HOST SIDE GUEST (stock driver, unmodified)
========= ================================
[done] build + load open-iov-vfio.ko
| (module first: exports the symbols the NVIDIA open modules link against)
v
[done] mdev create ------------------------------------> /
| BAR/PCI/MSI-X emulation; GSP emulation thread
v
[done] VM attach (QEMU q35 + pcie-root-port) ----------> config space read; BAR mmap
|
v
[done] GSP boot handshake (emulated) ------------------> nvidia.ko probe / RmInitAdapter
|
v
[done] GSP-RM message queues / RPC / controls ---------> object + client setup
|
v
[done] engine + class lists ---------------------------> GPU_GET_ENGINES, class DB
| (two-GRCE fix; async copy engine)
v
[done] ------------------------------------------------> nvidia-smi -L enumerates TU104
|
v
[done] ------------------------------------------------> cuInit / DeviceGetCount / DeviceGet
|
v
[frontier] --------------------------------------------> cuCtxCreate
| UVM register GPU; channel manager
| GR + CE channels; push buffers
| semaphore pool; completion counters
v
[not yet] ---------------------------------------------> cuMemAlloc + cuMemcpy roundtrip
[not yet] ---------------------------------------------> full nvidia-smi output
```
Work execution (host service vs client synchronization):
```
[partial] host decodes guest pushes (method streams): memsets, copies, releases
- CE memop pushes now execute (memset/ringops climbing, misses at zero)
- GR/compute pushes not yet consumed
[partial] GPU VA -> guest PA translation and pinning; server-owned page tables in progress
[not yet] real work forwarded to the physical GPU (host-RM proxy or direct submission)
[not yet] correct results (vectorAdd)
```
Teardown and recovery:
```
[partial] failure teardown exercised (context rollback, channel stop, notifier status)
[done] VM stop -> mdev remove -> rmmod, in order; never rmmod with a live VM attached
[done] crash recovery: pstore inspection; the host hang class is fixed and frame-audited
```
Recent chain of walls removed (2026-09-28 to 2026-09-29):
```
host hang (stack overflow) -> pump state persistence -> map collapse (merge rebuild)
-> rescue overhaul -> four-word semaphore releases -> counter injection (all 20)
-> stale semaphore-pool translation -> fake-only proven insufficient
-> push buffer found and validated -> per-page pins -> authoritative pin table
-> deterministic chain solved -> pin-drop and release-push fixes
-> copy-engine data path executes -> compute-window pinning
```
## Tools, code paths and fixes behind this progress
```
KERNEL MODULE (open-iov-vfio.ko) GUEST-SIDE TOOLS (tracked)
================================ ==========================
gsp_emulation.c boot, doorbells, MSI-X tools/test/cuda_drv_test.c smoke test
gsp_rm_emulation.c msgq, RPC, controls tools/guest-diag/vmrestart.sh
gsp_rm_tu104.h two-GRCE engine fix tools/guest-diag/cuda_cycle.sh
gsp_sema_service.c THE WORKHORSE tools/guest-diag/push_guest_tools.sh
channel pump, method decode, rescue, tools/guest-diag/inject_counters.sh
map rebuild, pins, counters, releases tools/guest-diag/semfix_watch.py
bar_emulation.c BAR shadows tools/guest-diag/track_waits.sh
main.c mdev/VFIO lifecycle tools/guest-diag/trkinfo.py
stack-frame audit (-Wframe-larger-than) tools/guest-diag/bp999.py / scanerr.py
dynamic_rpc_dispatcher.c firmware load fix tools/guest-diag/read_va.py
tools/guest-diag/guest_kprobes.sh
DATA STRUCTURES DEBUG PLANES
=============== ============
channel table (handle, GPFIFO, USERD, module stats: applied / miss / memset /
notifier, ring) ringops / nomap / skip / faked /
VA->PA map (merge rebuild, hot pages, cache) pins / ipins
pin table + authoritative injected pin table debug params: sema_probe, sema_counter,
completion counter table (per channel) sema_map_add, chan_dump, shadow_dump
semaphore pool page (GPU VA 0x121000000) QMP gva2gpa + guest gdbstub reads
push buffer allocation (GPU VA 0x120000000) guest kprobes; /proc/<pid>/pagemap
client trackers / RM error record chain marker-based /dev/kmsg parsing
```
| Wall | Fix |
|---|---|
| Host panic: oversized stack frame in a channel snapshot path | static buffers; frame-size audit on every build |
| Channel pump lost per-pass state (stale entries re-run; rate limit dead) | write-back of rescue/payload state; one-retry skip for compute channels |
| VA-to-PA map collapsed after client teardown | merge rebuild instead of wholesale replace |
| Rescue slow / false candidates | hot-page-first scan; full method-stream validation; retry limits |
| Four-word semaphore releases ignored | decode both release encodings |
| Only 4 of 20 completion counters ever moved | async injection queue; fast injector; per-boot pin, order fixed |
| UVM tracker wait never cleared | stale pool translation found; pool page pinned from guest truth each boot |
| Acknowledging releases without executing work | proven insufficient; pushes must be read and executed |
| Push buffers unreadable | located by CPU VA; per-page translation; authoritative injected pin table |
| Injected pins silently dropped once a pin-table cluster grew | authoritative injected-pin table checked first; larger table, longer probe |
| Real release pushes rejected as implausible (non-zero method start) | decode release-only when the translation came from injected pins |
| Completion release decoded but deferred (counter page untranslated) | pin the compute-channel mapping window from guest truth each boot |
## What "CUDA works" means (open gates)
Per [`docs/04-implementation/01-known-knowns-and-unknowns.md`](docs/04-implementation/01-known-knowns-and-unknowns.md)
section 4, a defensible claim needs **all** of:
1. every guest-to-device message for the CUDA path answered or forwarded,
with coherent replies for consumed fields;
2. every work VA translatable and server-owned page-table ranges populated;
3. launches of a grid of thread blocks (a CUDA kernel dispatch, e.g.
`vectorAdd`'s `<<<blocks, threads>>>`) actually executed on the physical
GPU's compute engine - not merely acknowledged - and copies executing on
real hardware with correct results;
4. completion (semaphores, events, faults) behaving as the guest expects;
5. at least two driver families passing;
6. tenant isolation;
7. measured performance within an agreed factor of native.
Until then the honest statement is: **initialization works; execution is in
progress.**
## Environment
- Host GPU: NVIDIA TU104 (RTX 2080 SUPER Mobile), PCI `10de:1ed3`.
- Host OS: Ubuntu, kernel 6.8.0-142; the distribution NVIDIA driver manages
the physical GPU; `open-iov-vfio.ko` loads as an independent module.
- Guest: Ubuntu 24.04.5 LTS, stock NVIDIA 570.144 driver, QEMU/KVM q35 with a
`pcie-root-port` for the vGPU, SeaBIOS, mdev passthrough.
- Firmware: TU104 GSP 570.144 (public release).
## Build and load order
Build: host module first (it generates the symbol versions the NVIDIA open
kernel modules link against), then the NVIDIA open kernel modules, then
userspace.
Load: VFIO prerequisites, then `open-iov-vfio.ko`, then create the mdev and
attach it to the VM. Unload in reverse; never `rmmod` while a VM holds the
device.
## See also
- [`docs/README.md`](docs/README.md) - documentation front door
- [`docs/04-implementation/01-known-knowns-and-unknowns.md`](docs/04-implementation/01-known-knowns-and-unknowns.md) - known-knowns, known-unknowns, definition of done
- [`userspace/cuda/READ-ME-FIRST.md`](userspace/cuda/READ-ME-FIRST.md) - CUDA analysis entry point
- [`docs/03-architecture/08-physical-gpu-integration.md`](docs/03-architecture/08-physical-gpu-integration.md) - chosen execution backend (module-owned GPU + real GSP session)
- [`docs/03-architecture/09-full-system-picture.md`](docs/03-architecture/09-full-system-picture.md) - visual end-to-end system map with status tags
- [`docs/03-architecture/10-interaction-and-scheduling.md`](docs/03-architecture/10-interaction-and-scheduling.md) - cross-component interaction, queues, interrupts, scheduling detail
- [`STATE.md`](STATE.md) - 2026-09-21 historical snapshot
---
*Refresh this page on every progress check (snapshot time in the header);
prefer moving detail into the `docs/` tree and keeping this page short.*
|
