# Open-IOV Current State > **Snapshot: 2026-09-29 15:25.** This is a dated public status page, not live > state; it is refreshed on every progress check. Live working notes are > maintained outside this repository; the public known-knowns/unknowns document > is authoritative here: > [`docs/04-implementation/01-known-knowns-and-unknowns.md`](docs/04-implementation/01-known-knowns-and-unknowns.md). > The 2026-09-21 snapshot remains in [`STATE.md`](STATE.md) as a historical > record. ## Mission Run **unmodified NVIDIA guest drivers** in VMs with full GPU functionality (CUDA, graphics, video, AI/ML) on consumer/data-center GPUs without proprietary GRID licenses - via host-side GSP emulation in `open-iov-vfio.ko`. The guest stays stock: no guest patches, no guest-side Open-IOV code. Architecture stance: *emulate the interface, pass the work through to real hardware, never patch anyone.* The control plane is emulated; the work is meant to execute on the physical GPU. ## Milestone ladder (this snapshot) | Milestone | Status | |---|---| | GSP boot handshake, GSP-RM message queues/RPC, BAR/PCI/MSI-X emulation | ✅ | | Stock 570.144 guest driver probe / `RmInitAdapter` | ✅ | | `nvidia-smi -L` enumerates the emulated TU104 | ✅ | | CUDA userspace: `cuInit`, `cuDeviceGetCount`, `cuDeviceGet` | ✅ | | `cuCtxCreate` returns | ⏳ **frontier** | | `cuMemAlloc` + `cuMemcpy` roundtrip (`ALL OK` in the smoke test) | ❌ not reached | | Full `nvidia-smi` output | ❌ (faults on the CUDA path) | | CUDA kernel execution (`vectorAdd`) | ❌ not started (forwarding layer absent) | | Graphics (3D), video | ❌ | | Faults, multi-version, Windows, migration, multi-tenancy, performance | ❌ | Test status is tracked with the CUDA driver-API smoke test ([`tools/test/cuda_drv_test.c`](tools/test/cuda_drv_test.c)): it exercises `cuInit -> cuDeviceGetCount/Get -> cuCtxCreate -> cuMemAlloc -> cuMemcpyHtoD/ DtoH -> cuMemFree -> cuCtxDestroy` and prints `ALL OK` only when the whole chain succeeds. ## Right now (frontier) - The stock guest driver initializes fully; the CUDA client reaches `cuCtxCreate` within seconds and runs there for minutes before failing with a forced `rc=999` (a timeout path, not an RM error record). - The per-channel completion counters in guest memory are enumerated, registered and published; the client's per-channel waits clear, and the per-channel GPFIFO work on copy channels is consumed. - Both engine push-buffer streams are now reachable: the copy-engine buffer is resolved deterministically (function-trace pointer chain, all pages pinned), and the compute-engine mapping window is being pinned from guest page tables on every boot. - **The copy-engine data path executes**: decoded method streams run in sustained fashion (memsets, copies, releases climbing; thousands of push entries consumed). - **Frontier: context creation still times out on one completion.** The final wait is a tracking semaphore whose release was decoded from a readable compute push but could not be applied in time because the counter page had no translation at that moment. The fix in progress is to pin the compute-channel mapping window from guest truth at boot so the release lands before the client's deadline. - Active work: the per-boot compute-window pin stream and re-running the full cycle. - Execution backend (chosen, not started): the module will own the physical GPU and run its real GSP firmware as the resource-manager server, so real work executes with an unmodified host driver and no host daemon; see [`docs/03-architecture/08-physical-gpu-integration.md`](docs/03-architecture/08-physical-gpu-integration.md). - Host stability: the host hang class observed on 2026-09-28 (kernel stack overflow in a channel snapshot path) is fixed and frame-audited; repeated VM load/attach/teardown cycles run without panics or pstore records. ## Initialization, work and teardown path Status tags: `[done]`, `[frontier]` (actively worked), `[partial]`, `[not yet]`. ``` HOST SIDE GUEST (stock driver, unmodified) ========= ================================ [done] build + load open-iov-vfio.ko | (module first: exports the symbols the NVIDIA open modules link against) v [done] mdev create ------------------------------------> / | BAR/PCI/MSI-X emulation; GSP emulation thread v [done] VM attach (QEMU q35 + pcie-root-port) ----------> config space read; BAR mmap | v [done] GSP boot handshake (emulated) ------------------> nvidia.ko probe / RmInitAdapter | v [done] GSP-RM message queues / RPC / controls ---------> object + client setup | v [done] engine + class lists ---------------------------> GPU_GET_ENGINES, class DB | (two-GRCE fix; async copy engine) v [done] ------------------------------------------------> nvidia-smi -L enumerates TU104 | v [done] ------------------------------------------------> cuInit / DeviceGetCount / DeviceGet | v [frontier] --------------------------------------------> cuCtxCreate | UVM register GPU; channel manager | GR + CE channels; push buffers | semaphore pool; completion counters v [not yet] ---------------------------------------------> cuMemAlloc + cuMemcpy roundtrip [not yet] ---------------------------------------------> full nvidia-smi output ``` Work execution (host service vs client synchronization): ``` [partial] host decodes guest pushes (method streams): memsets, copies, releases - CE memop pushes now execute (memset/ringops climbing, misses at zero) - GR/compute pushes not yet consumed [partial] GPU VA -> guest PA translation and pinning; server-owned page tables in progress [not yet] real work forwarded to the physical GPU (host-RM proxy or direct submission) [not yet] correct results (vectorAdd) ``` Teardown and recovery: ``` [partial] failure teardown exercised (context rollback, channel stop, notifier status) [done] VM stop -> mdev remove -> rmmod, in order; never rmmod with a live VM attached [done] crash recovery: pstore inspection; the host hang class is fixed and frame-audited ``` Recent chain of walls removed (2026-09-28 to 2026-09-29): ``` host hang (stack overflow) -> pump state persistence -> map collapse (merge rebuild) -> rescue overhaul -> four-word semaphore releases -> counter injection (all 20) -> stale semaphore-pool translation -> fake-only proven insufficient -> push buffer found and validated -> per-page pins -> authoritative pin table -> deterministic chain solved -> pin-drop and release-push fixes -> copy-engine data path executes -> compute-window pinning ``` ## Tools, code paths and fixes behind this progress ``` KERNEL MODULE (open-iov-vfio.ko) GUEST-SIDE TOOLS (tracked) ================================ ========================== gsp_emulation.c boot, doorbells, MSI-X tools/test/cuda_drv_test.c smoke test gsp_rm_emulation.c msgq, RPC, controls tools/guest-diag/vmrestart.sh gsp_rm_tu104.h two-GRCE engine fix tools/guest-diag/cuda_cycle.sh gsp_sema_service.c THE WORKHORSE tools/guest-diag/push_guest_tools.sh channel pump, method decode, rescue, tools/guest-diag/inject_counters.sh map rebuild, pins, counters, releases tools/guest-diag/semfix_watch.py bar_emulation.c BAR shadows tools/guest-diag/track_waits.sh main.c mdev/VFIO lifecycle tools/guest-diag/trkinfo.py stack-frame audit (-Wframe-larger-than) tools/guest-diag/bp999.py / scanerr.py dynamic_rpc_dispatcher.c firmware load fix tools/guest-diag/read_va.py tools/guest-diag/guest_kprobes.sh DATA STRUCTURES DEBUG PLANES =============== ============ channel table (handle, GPFIFO, USERD, module stats: applied / miss / memset / notifier, ring) ringops / nomap / skip / faked / VA->PA map (merge rebuild, hot pages, cache) pins / ipins pin table + authoritative injected pin table debug params: sema_probe, sema_counter, completion counter table (per channel) sema_map_add, chan_dump, shadow_dump semaphore pool page (GPU VA 0x121000000) QMP gva2gpa + guest gdbstub reads push buffer allocation (GPU VA 0x120000000) guest kprobes; /proc//pagemap client trackers / RM error record chain marker-based /dev/kmsg parsing ``` | Wall | Fix | |---|---| | Host panic: oversized stack frame in a channel snapshot path | static buffers; frame-size audit on every build | | Channel pump lost per-pass state (stale entries re-run; rate limit dead) | write-back of rescue/payload state; one-retry skip for compute channels | | VA-to-PA map collapsed after client teardown | merge rebuild instead of wholesale replace | | Rescue slow / false candidates | hot-page-first scan; full method-stream validation; retry limits | | Four-word semaphore releases ignored | decode both release encodings | | Only 4 of 20 completion counters ever moved | async injection queue; fast injector; per-boot pin, order fixed | | UVM tracker wait never cleared | stale pool translation found; pool page pinned from guest truth each boot | | Acknowledging releases without executing work | proven insufficient; pushes must be read and executed | | Push buffers unreadable | located by CPU VA; per-page translation; authoritative injected pin table | | Injected pins silently dropped once a pin-table cluster grew | authoritative injected-pin table checked first; larger table, longer probe | | Real release pushes rejected as implausible (non-zero method start) | decode release-only when the translation came from injected pins | | Completion release decoded but deferred (counter page untranslated) | pin the compute-channel mapping window from guest truth each boot | ## What "CUDA works" means (open gates) Per [`docs/04-implementation/01-known-knowns-and-unknowns.md`](docs/04-implementation/01-known-knowns-and-unknowns.md) section 4, a defensible claim needs **all** of: 1. every guest-to-device message for the CUDA path answered or forwarded, with coherent replies for consumed fields; 2. every work VA translatable and server-owned page-table ranges populated; 3. launches of a grid of thread blocks (a CUDA kernel dispatch, e.g. `vectorAdd`'s `<<>>`) actually executed on the physical GPU's compute engine - not merely acknowledged - and copies executing on real hardware with correct results; 4. completion (semaphores, events, faults) behaving as the guest expects; 5. at least two driver families passing; 6. tenant isolation; 7. measured performance within an agreed factor of native. Until then the honest statement is: **initialization works; execution is in progress.** ## Environment - Host GPU: NVIDIA TU104 (RTX 2080 SUPER Mobile), PCI `10de:1ed3`. - Host OS: Ubuntu, kernel 6.8.0-142; the distribution NVIDIA driver manages the physical GPU; `open-iov-vfio.ko` loads as an independent module. - Guest: Ubuntu 24.04.5 LTS, stock NVIDIA 570.144 driver, QEMU/KVM q35 with a `pcie-root-port` for the vGPU, SeaBIOS, mdev passthrough. - Firmware: TU104 GSP 570.144 (public release). ## Build and load order Build: host module first (it generates the symbol versions the NVIDIA open kernel modules link against), then the NVIDIA open kernel modules, then userspace. Load: VFIO prerequisites, then `open-iov-vfio.ko`, then create the mdev and attach it to the VM. Unload in reverse; never `rmmod` while a VM holds the device. ## See also - [`docs/README.md`](docs/README.md) - documentation front door - [`docs/04-implementation/01-known-knowns-and-unknowns.md`](docs/04-implementation/01-known-knowns-and-unknowns.md) - known-knowns, known-unknowns, definition of done - [`userspace/cuda/READ-ME-FIRST.md`](userspace/cuda/READ-ME-FIRST.md) - CUDA analysis entry point - [`docs/03-architecture/08-physical-gpu-integration.md`](docs/03-architecture/08-physical-gpu-integration.md) - chosen execution backend (module-owned GPU + real GSP session) - [`STATE.md`](STATE.md) - 2026-09-21 historical snapshot --- *Refresh this page on every progress check (snapshot time in the header); prefer moving detail into the `docs/` tree and keeping this page short.*