# Open-IOV Current State > **Snapshot: 2026-10-01 01:12.** This is a dated public status page, not live > state; it is refreshed on every progress check. Live working notes are > maintained outside this repository; the public known-knowns/unknowns document > is authoritative here: > [`docs/04-implementation/01-known-knowns-and-unknowns.md`](docs/04-implementation/01-known-knowns-and-unknowns.md). > The former 2026-09-21 snapshot (`STATE.md`) is retired; its content remains > in the git history. ## Mission Run **unmodified NVIDIA guest drivers** in VMs with full GPU functionality (CUDA, graphics, video, AI/ML) on consumer/data-center GPUs without proprietary GRID licenses - via host-side GSP emulation in `open-iov-vfio.ko`. The guest stays stock: no guest patches, no guest-side Open-IOV code. Architecture stance: *emulate the interface, pass the work through to real hardware, never patch anyone.* The control plane is emulated; the work is meant to execute on the physical GPU. ## Milestone ladder (this snapshot) | Milestone | Status | |---|---| | GSP boot handshake, GSP-RM message queues/RPC, BAR/PCI/MSI-X emulation | ✅ | | Stock 570.144 guest driver probe / `RmInitAdapter` | ✅ | | `nvidia-smi -L` enumerates the emulated TU104 | ✅ | | CUDA userspace: `cuInit`, `cuDeviceGetCount`, `cuDeviceGet` | ✅ | | `cuCtxCreate` returns | ✅ (the whole smoke test now completes) | | `cuMemAlloc` + `cuMemcpy` roundtrip (`ALL OK` in the smoke test) | ✅ payload roundtrip verified 64/64 in the guest | | Full `nvidia-smi` output / kernel launch | 🟡 the static runtime now initializes end-to-end with **real releases** (`cudaGetDeviceCount`/`cudaSetDevice`/3×`cudaMalloc`/2×`cudaMemcpyHtoD`/`kernel launch`/`cudaGetLastError` all `rc=0`) and queues a kernel launch; execution awaits the backend | | CUDA kernel execution (`vectorAdd`) | 🟡 execution backend: the device-claim stages (ownership, firmware staging, protected-region boot layout) were verified, but unbinding the distribution driver turns out to hard-freeze this host (dGPU power-gate); the claim path is gate-locked and the backend must be redesigned to keep the distribution driver bound | | Graphics (3D), video | ❌ | | Faults, multi-version, Windows, migration, multi-tenancy, performance | ❌ | Test status is tracked with the CUDA driver-API smoke test ([`tools/test/cuda_drv_test.c`](tools/test/cuda_drv_test.c)): it exercises `cuInit -> cuDeviceGetCount/Get -> cuCtxCreate -> cuMemAlloc -> cuMemcpyHtoD/ DtoH -> cuMemFree -> cuCtxDestroy` and prints `ALL OK` only when the whole chain succeeds. ## Right now (frontier) - **Driver-API baseline re-verified on the current build**: the smoke test passes end-to-end (`ALL OK`, payload 64/64) with real releases (`faked=0`, `miss=0`, `stale=0`), confirming the recent pump/parser and rewind changes did not regress the baseline. - **The full CUDA driver-API smoke test passes with correct data**: the client completes `cuCtxCreate -> cuMemAlloc -> cuMemcpyHtoD -> cuMemcpyDtoH -> cuMemFree -> cuCtxDestroy` and prints `ALL OK`, with the 64-byte payload roundtripping exactly (64/64 verified in the guest). - **The last two walls**: the GPU page-table walk's address decode (child pointers and leaves were 16x too low; fixed with the real entry encoding - walks now succeed at scale) and the device-to-host copy (a copy-engine class transfer bound on a compute channel whose work was denied by the engine-type gate; trusted host-injected pushes may now execute copy-engine work as well). - **Baseline green and quiet**: the payload roundtrip is back to 64/64 (`ALL OK`), the staging buffer's translation stays resolvable via pins streamed from the guest, and high-frequency diagnostics are now gated behind a default-off debug switch - a normal boot+run produces a ~520-line log instead of a flood (verified twice). The page-directory entry writes have since been captured live and validated (the earlier all-zero boots were operation-skipped/pre-pin boots); the remaining question is the device-address walk gap for the runtime's own buffers. - Recap of the fixes behind this point: all 20/20 compute-channel releases land (three decode paths routed to the exact counter slot); registration delivers the incrementing tracking release for every consumed submission; the copy-engine push buffer is pinned from guest truth on every boot; and the copy-engine decoder consults pinned translations before the scanned map, so every push-data operation executes (`memset=19473`, `ringops=10551` live). - **The runtime client now initializes and reaches kernel launch**: with a static-runtime `vectorAdd` client, `cudaGetDeviceCount`, `cudaSetDevice`, `cudaFree`, three `cudaMalloc`, the HtoD copies and `cudaGetLastError` all return success. The client then parks in `cudaDeviceSynchronize` - the launch is queued but never **executed**, because real execution requires the physical-GPU backend (the next project). - **What fixed it**: the failing call was never a kernel call - it was the clock-domain probe inside libcuda (two device attributes) whose three legacy clock controls were answered with zeros; they are now populated with exactly the fields the helper checks. - **Real-release mode now reaches the same rc=0 chain**: the earlier per-push 8 s stall was two pump bookkeeping bugs (release detection keyed on an unchanged semaphore VA; rewind replays re-entering the CE grace path) - both fixed, and the runtime init completes in ~1 min with real releases. The content rescue also learned the real method encoding (NON_INC/IMMD/ ONE_INC/END_PB headers). - Active work: the owed driver-API smoke-test rerun is now **verified 64/64 (`ALL OK`)** on the current commits, so the recent pump/parser changes did not regress the baseline. The device-claim bring-up verified ownership, firmware staging and the boot layout, but unbinding the distribution driver can power-gate this laptop's dGPU and hard-freeze the host (two occurrences, now root-caused), so that path is disabled behind a two-parameter hazard gate. A power-domain lock that keeps the board in D0 across the handoff is implemented and **verified across three supervised claim/release cycles on 2026-10-01** (zero power-domain cycles, zero failed resumes, distribution driver rebound cleanly each time, host healthy); the verified launch harness holds the GPU and root port in D0 before the distribution driver is stopped, and the root-port hold is the load-bearing invariant. The real GSP boot is the next stage. - Observability: a size-keyed copy watch, a VRAM shadow dump/search, and an operation log are now module parameters, and the test's output survives guest-agent stalls via the serial console. - Guest health note: agent/SSH timeouts during long runs are the guest-side trace reader flooding the ring; they recover on their own, so a first timeout is not a reason to restart the VM. Separately, one no-watcher boot crashed the guest kernel (a NULL callback in interrupt context) before any emulation channel existed - guest-side and untriaged; treat no-watcher runs as a known risk. The host stays stable throughout. - Execution backend (chosen; ownership hazard mitigated and verified): the module owns the physical GPU and will run its real GSP firmware as the resource-manager server, so real work executes with an unmodified host driver and no host daemon; see [`docs/03-architecture/08-physical-gpu-integration.md`](docs/03-architecture/08-physical-gpu-integration.md). The ownership path was verified live (release the distribution driver from the device, bind the module, map the BARs, sample the falcon state, then release and rebind the distribution driver); with the device owned, the matching GSP image is fetched and validated (ELF64 RISC-V, loadable image/version/signature sections identified) and the protected-region video-memory/boot layout is computed and logged - all without device writes. The unbind/rebind hazard is mitigated by the D0 power-domain lock, verified over three supervised cycles; the ownership handoff is now also hardened against the distribution driver's udev bind-event power-policy write, and a reusable supervised claim-cycle harness exists (`tools/guest-diag/phys_claim_cycle.sh`). The real-GSP-boot design is complete: exact secure-verification/booter/RISC-V/RM-init sequence and boot artifacts (booter/FMC/SEC2 BL bindata; VBIOS BIT FWSEC blob). The real FB size (8 GiB, decoded from the local-memory-range register - the BAR1-aperture approximation is retired) and the protected-region boot layout have been verified live under the supervised claim harness (three green claim/release cycles; the root-port D0 hold is the load-bearing invariant), and the secure-verification stage (VBIOS security-blob extraction, falcon reset and staging) is implemented build-only pending an adversarially-reviewed, supervised live boot; the booter and RISC-V start follow. - Host stability: the host hang class observed on 2026-09-28 (kernel stack overflow in a channel snapshot path) is fixed and frame-audited; repeated VM load/attach/teardown cycles run without panics or pstore records. A second, distinct freeze class was root-caused on 2026-09-30: the dGPU board's power rail is gated by an ACPI power resource behind its root port, released through an embedded-controller command. Unbinding the distribution driver (as the physical-GPU claim path does) lets the device runtime-suspend, the rail is cut, and the re-bind intermittently fails to power it back (`not ready ... after resume`), with the host hard-freezing about a minute later (no crash record; journal stops, pstore empty; both occurrences followed an unbind/rebind, and prior log history has no other instance). The claim path refuses to run unless both `gsp_phys_gpu=1` and `gsp_phys_hazard_ack=1` are passed, and the power-domain lock (hold the GPU and its board functions plus the root port in D0 across the handoff, aborting and holding the locks on any power-loss signature) is now **implemented and verified across three supervised claim/release cycles** (2026-10-01: zero power-domain cycles, zero failed resumes, clean rebind, host healthy). The verified launch harness holds the GPU and root port in D0 before the distribution driver is stopped; the root-port hold is the load-bearing invariant. The path must still not run unattended and every cycle is supervised. Normal module/VM work does not touch the driver binding and is unaffected. See [`docs/03-architecture/08-physical-gpu-integration.md`](docs/03-architecture/08-physical-gpu-integration.md). ## Initialization, work and teardown path Status tags: `[done]`, `[frontier]` (actively worked), `[partial]`, `[not yet]`. ``` HOST SIDE GUEST (stock driver, unmodified) ========= ================================ [done] build + load open-iov-vfio.ko | (module first: exports the symbols the NVIDIA open modules link against) v [done] mdev create ------------------------------------> / | BAR/PCI/MSI-X emulation; GSP emulation thread v [done] VM attach (QEMU q35 + pcie-root-port) ----------> config space read; BAR mmap | v [done] GSP boot handshake (emulated) ------------------> nvidia.ko probe / RmInitAdapter | v [done] GSP-RM message queues / RPC / controls ---------> object + client setup | v [done] engine + class lists ---------------------------> GPU_GET_ENGINES, class DB | (two-GRCE fix; async copy engine) v [done] ------------------------------------------------> nvidia-smi -L enumerates TU104 | v [done] ------------------------------------------------> cuInit / DeviceGetCount / DeviceGet | v [done] ------------------------------------------------> cuCtxCreate | UVM register GPU; channel manager | GR + CE channels; push buffers | semaphore pool; completion counters v [done] ------------------------------------------------> cuMemAlloc + cuMemcpy roundtrip | (passes end-to-end; payload 64/64 correct) [partial] ---------------------------------------------> full nvidia-smi output / kernel launch | (runtime init rc=0; launch queued; | execution awaits the backend) ``` Work execution (host service vs client synchronization): ``` [done] host decodes guest pushes (method streams): memsets, copies, releases - CE pushes execute at scale; the compute-class inline-DMA HtoD and the copy-engine-on-compute DtoH both execute; payload 64/64 - the runtime client now initializes (rc=0) and queues a kernel launch; the queued compute pushes decode but are not executed [partial] GPU VA -> guest PA translation: page-table walk fixed and succeeding; server-owned page-table population still in progress [not yet] real work forwarded to the physical GPU (host-RM proxy or direct submission) [not yet] correct results (vectorAdd) ``` Teardown and recovery: ``` [done] happy-path teardown exercised end-to-end (cuCtxDestroy OK) [done] VM stop -> mdev remove -> rmmod, in order; never rmmod with a live VM attached [done] crash recovery: pstore inspection; the host hang class is fixed and frame-audited ``` Recent chain of walls removed (2026-09-28 to 2026-09-30): ``` host hang (stack overflow) -> pump state persistence -> map collapse (merge rebuild) -> rescue overhaul -> four-word semaphore releases -> counter injection (all 20) -> stale semaphore-pool translation -> fake-only proven insufficient -> push buffer found and validated -> per-page pins -> authoritative pin table -> deterministic chain solved -> pin-drop and release-push fixes -> copy-engine data path executes -> compute-window pinning -> registration release delivery -> 20/20 counter slots -> GR-info clock fix -> cuCtxCreate OK; payload found (compute-class inline DMA) -> GMMU walk decode fixed (child/leaf encoding) -> inline HtoD lands -> DtoH copy-engine-on-compute allowed -> payload roundtrip 64/64 -> runtime client initializes rc=0 (legacy clock-domain fix) -> kernel launch queued -> real-release chain rc=0 (pump bookkeeping fixes) -> execution backend bring-up is the next wall ``` ## Tools, code paths and fixes behind this progress ``` KERNEL MODULE (open-iov-vfio.ko) GUEST-SIDE TOOLS (tracked) ================================ ========================== gsp_emulation.c boot, doorbells, MSI-X tools/test/cuda_drv_test.c smoke test gsp_rm_emulation.c msgq, RPC, controls tools/guest-diag/vmrestart.sh gsp_rm_tu104.h two-GRCE engine fix tools/guest-diag/cuda_cycle.sh gsp_sema_service.c THE WORKHORSE tools/guest-diag/push_guest_tools.sh channel pump, method decode, rescue, tools/guest-diag/inject_counters.sh map rebuild, pins, counters, releases tools/guest-diag/semfix_watch.py bar_emulation.c BAR shadows tools/guest-diag/track_waits.sh main.c mdev/VFIO lifecycle tools/guest-diag/trkinfo.py stack-frame audit (-Wframe-larger-than) tools/guest-diag/bp999.py / scanerr.py dynamic_rpc_dispatcher.c firmware load fix tools/guest-diag/read_va.py tools/guest-diag/guest_kprobes.sh DATA STRUCTURES DEBUG PLANES =============== ============ channel table (handle, GPFIFO, USERD, module stats: applied / miss / memset / notifier, ring) ringops / nomap / skip / faked / VA->PA map (merge rebuild, hot pages, cache) pins / ipins pin table + authoritative injected pin table debug params: sema_probe, sema_counter, completion counter table (per channel) sema_map_add, chan_dump, shadow_dump semaphore pool page (GPU VA 0x121000000) QMP gva2gpa + guest gdbstub reads push buffer allocation (GPU VA 0x120000000) guest kprobes; /proc//pagemap client trackers / RM error record chain marker-based /dev/kmsg parsing ``` | Wall | Fix | |---|---| | Host panic: oversized stack frame in a channel snapshot path | static buffers; frame-size audit on every build | | Channel pump lost per-pass state (stale entries re-run; rate limit dead) | write-back of rescue/payload state; one-retry skip for compute channels | | VA-to-PA map collapsed after client teardown | merge rebuild instead of wholesale replace | | Rescue slow / false candidates | hot-page-first scan; full method-stream validation; retry limits | | Four-word semaphore releases ignored | decode both release encodings | | Only 4 of 20 completion counters ever moved | async injection queue; fast injector; per-boot pin, order fixed | | UVM tracker wait never cleared | stale pool translation found; pool page pinned from guest truth each boot | | Acknowledging releases without executing work | proven insufficient; pushes must be read and executed | | Push buffers unreadable | located by CPU VA; per-page translation; authoritative injected pin table | | Injected pins silently dropped once a pin-table cluster grew | authoritative injected-pin table checked first; larger table, longer probe | | Real release pushes rejected as implausible (non-zero method start) | decode release-only when the translation came from injected pins | | Completion release decoded but deferred (counter page untranslated) | pin the compute-channel mapping window from guest truth each boot | ## What "CUDA works" means (open gates) Per [`docs/04-implementation/01-known-knowns-and-unknowns.md`](docs/04-implementation/01-known-knowns-and-unknowns.md) section 4, a defensible claim needs **all** of: 1. every guest-to-device message for the CUDA path answered or forwarded, with coherent replies for consumed fields; 2. every work VA translatable and server-owned page-table ranges populated; 3. launches of a grid of thread blocks (a CUDA kernel dispatch, e.g. `vectorAdd`'s `<<>>`) actually executed on the physical GPU's compute engine - not merely acknowledged - and copies executing on real hardware with correct results; 4. completion (semaphores, events, faults) behaving as the guest expects; 5. at least two driver families passing; 6. tenant isolation; 7. measured performance within an agreed factor of native. Until then the honest statement is: **initialization works; execution is in progress.** ## Environment - Host GPU: NVIDIA TU104 (RTX 2080 SUPER Mobile), PCI `10de:1ed3`. - Host OS: Ubuntu, kernel 6.8.0-142; the distribution NVIDIA driver manages the physical GPU; `open-iov-vfio.ko` loads as an independent module. - Guest: Ubuntu 24.04.5 LTS, stock NVIDIA 570.144 driver, QEMU/KVM q35 with a `pcie-root-port` for the vGPU, SeaBIOS, mdev passthrough. - Firmware: TU104 GSP 570.144 (public release). ## Build and load order Build: host module first (it generates the symbol versions the NVIDIA open kernel modules link against), then the NVIDIA open kernel modules, then userspace. Load: VFIO prerequisites, then `open-iov-vfio.ko`, then create the mdev and attach it to the VM. Unload in reverse; never `rmmod` while a VM holds the device. ## See also - [`docs/README.md`](docs/README.md) - documentation front door - [`docs/04-implementation/01-known-knowns-and-unknowns.md`](docs/04-implementation/01-known-knowns-and-unknowns.md) - known-knowns, known-unknowns, definition of done - [`userspace/cuda/READ-ME-FIRST.md`](userspace/cuda/READ-ME-FIRST.md) - CUDA analysis entry point - [`docs/03-architecture/08-physical-gpu-integration.md`](docs/03-architecture/08-physical-gpu-integration.md) - chosen execution backend (module-owned GPU + real GSP session) - [`docs/03-architecture/09-full-system-picture.md`](docs/03-architecture/09-full-system-picture.md) - visual end-to-end system map with status tags - [`docs/03-architecture/10-interaction-and-scheduling.md`](docs/03-architecture/10-interaction-and-scheduling.md) - cross-component interaction, queues, interrupts, scheduling detail --- *Refresh this page on every progress check (snapshot time in the header); prefer moving detail into the `docs/` tree and keeping this page short.*