path: root/STATE-CURRENT.md

# Open-IOV Current State

> **Snapshot: 2026-10-01 00:12.** This is a dated public status page, not live
> state; it is refreshed on every progress check. Live working notes are
> maintained outside this repository; the public known-knowns/unknowns document
> is authoritative here:
> [`docs/04-implementation/01-known-knowns-and-unknowns.md`](docs/04-implementation/01-known-knowns-and-unknowns.md).
> The former 2026-09-21 snapshot (`STATE.md`) is retired; its content remains
> in the git history.

## Mission

Run **unmodified NVIDIA guest drivers** in VMs with full GPU functionality
(CUDA, graphics, video, AI/ML) on consumer/data-center GPUs without
proprietary GRID licenses - via host-side GSP emulation in
`open-iov-vfio.ko`. The guest stays stock: no guest patches, no guest-side
Open-IOV code.

Architecture stance: *emulate the interface, pass the work through to real
hardware, never patch anyone.* The control plane is emulated; the work is
meant to execute on the physical GPU.

## Milestone ladder (this snapshot)

| Milestone | Status |
|---|---|
| GSP boot handshake, GSP-RM message queues/RPC, BAR/PCI/MSI-X emulation | ✅ |
| Stock 570.144 guest driver probe / `RmInitAdapter` | ✅ |
| `nvidia-smi -L` enumerates the emulated TU104 | ✅ |
| CUDA userspace: `cuInit`, `cuDeviceGetCount`, `cuDeviceGet` | ✅ |
| `cuCtxCreate` returns | ✅ (the whole smoke test now completes) |
| `cuMemAlloc` + `cuMemcpy` roundtrip (`ALL OK` in the smoke test) | ✅ payload roundtrip verified 64/64 in the guest |
| Full `nvidia-smi` output / kernel launch | 🟡 the static runtime now initializes end-to-end with **real releases** (`cudaGetDeviceCount`/`cudaSetDevice`/3×`cudaMalloc`/2×`cudaMemcpyHtoD`/`kernel launch`/`cudaGetLastError` all `rc=0`) and queues a kernel launch; execution awaits the backend |
| CUDA kernel execution (`vectorAdd`) | 🟡 execution backend: the device-claim stages (ownership, firmware staging, protected-region boot layout) were verified, but unbinding the distribution driver turns out to hard-freeze this host (dGPU power-gate); the claim path is gate-locked and the backend must be redesigned to keep the distribution driver bound |
| Graphics (3D), video | ❌ |
| Faults, multi-version, Windows, migration, multi-tenancy, performance | ❌ |

Test status is tracked with the CUDA driver-API smoke test
([`tools/test/cuda_drv_test.c`](tools/test/cuda_drv_test.c)): it exercises
`cuInit -> cuDeviceGetCount/Get -> cuCtxCreate -> cuMemAlloc -> cuMemcpyHtoD/
DtoH -> cuMemFree -> cuCtxDestroy` and prints `ALL OK` only when the whole
chain succeeds.

## Right now (frontier)

- **Driver-API baseline re-verified on the current build**: the smoke test
  passes end-to-end (`ALL OK`, payload 64/64) with real releases
  (`faked=0`, `miss=0`, `stale=0`), confirming the recent pump/parser and
  rewind changes did not regress the baseline.
- **The full CUDA driver-API smoke test passes with correct data**: the client
  completes `cuCtxCreate -> cuMemAlloc -> cuMemcpyHtoD -> cuMemcpyDtoH ->
  cuMemFree -> cuCtxDestroy` and prints `ALL OK`, with the 64-byte payload
  roundtripping exactly (64/64 verified in the guest).
- **The last two walls**: the GPU page-table walk's address decode (child
  pointers and leaves were 16x too low; fixed with the real entry encoding -
  walks now succeed at scale) and the device-to-host copy (a copy-engine
  class transfer bound on a compute channel whose work was denied by the
  engine-type gate; trusted host-injected pushes may now execute copy-engine
  work as well).
- **Baseline green and quiet**: the payload roundtrip is back to 64/64
  (`ALL OK`), the staging buffer's translation stays resolvable via pins
  streamed from the guest, and high-frequency diagnostics are now gated
  behind a default-off debug switch - a normal boot+run produces a ~520-line
  log instead of a flood (verified twice). The page-directory entry writes
  have since been captured live and validated (the earlier all-zero boots
  were operation-skipped/pre-pin boots); the remaining question is the
  device-address walk gap for the runtime's own buffers.
- Recap of the fixes behind this point: all 20/20 compute-channel releases
  land (three decode paths routed to the exact counter slot); registration
  delivers the incrementing tracking release for every consumed submission;
  the copy-engine push buffer is pinned from guest truth on every boot; and
  the copy-engine decoder consults pinned translations before the scanned
  map, so every push-data operation executes (`memset=19473`, `ringops=10551`
  live).
- **The runtime client now initializes and reaches kernel launch**: with a
  static-runtime `vectorAdd` client, `cudaGetDeviceCount`, `cudaSetDevice`,
  `cudaFree`, three `cudaMalloc`, the HtoD copies and `cudaGetLastError` all
  return success. The client then parks in `cudaDeviceSynchronize` - the
  launch is queued but never **executed**, because real execution requires
  the physical-GPU backend (the next project).
- **What fixed it**: the failing call was never a kernel call - it was the
  clock-domain probe inside libcuda (two device attributes) whose three
  legacy clock controls were answered with zeros; they are now populated
  with exactly the fields the helper checks.
- **Real-release mode now reaches the same rc=0 chain**: the earlier
  per-push 8 s stall was two pump bookkeeping bugs (release detection keyed on
  an unchanged semaphore VA; rewind replays re-entering the CE grace path) -
  both fixed, and the runtime init completes in ~1 min with real releases.
  The content rescue also learned the real method encoding (NON_INC/IMMD/
  ONE_INC/END_PB headers).
- Active work: the owed driver-API smoke-test rerun is now **verified 64/64
  (`ALL OK`)** on the current commits, so the recent pump/parser changes did
  not regress the baseline. The device-claim bring-up verified ownership,
  firmware staging and the boot layout, but unbinding the distribution driver
  can power-gate this laptop's dGPU and hard-freeze the host (two occurrences,
  now root-caused), so that path is disabled behind a two-parameter hazard
  gate. A power-domain lock that keeps the board in D0 across the handoff is
  implemented and **verified across three supervised claim/release cycles on
  2026-10-01** (zero power-domain cycles, zero failed resumes, distribution
  driver rebound cleanly each time, host healthy); the verified launch
  harness holds the GPU and root port in D0 before the distribution driver is
  stopped, and the root-port hold is the load-bearing invariant. The real GSP
  boot is the next stage.
- Observability: a size-keyed copy watch, a VRAM shadow dump/search, and an
  operation log are now module parameters, and the test's output survives
  guest-agent stalls via the serial console.
- Guest health note: agent/SSH timeouts during long runs are the guest-side
  trace reader flooding the ring; they recover on their own, so a first
  timeout is not a reason to restart the VM. Separately, one no-watcher boot
  crashed the guest kernel (a NULL callback in interrupt context) before any
  emulation channel existed - guest-side and untriaged; treat no-watcher runs
  as a known risk. The host stays stable throughout.
- Execution backend (chosen; ownership hazard mitigated and verified): the
  module owns the physical GPU and will run its real GSP firmware as the
  resource-manager server, so real work executes with an unmodified host
  driver and no host daemon; see
  [`docs/03-architecture/08-physical-gpu-integration.md`](docs/03-architecture/08-physical-gpu-integration.md).
  The ownership path was verified live (release the distribution driver from
  the device, bind the module, map the BARs, sample the falcon state, then
  release and rebind the distribution driver); with the device owned, the
  matching GSP image is fetched and validated (ELF64 RISC-V, loadable
  image/version/signature sections identified) and the protected-region
  video-memory/boot layout is computed and logged - all without device writes.
  The unbind/rebind hazard is mitigated by the D0 power-domain lock, verified
  over three supervised cycles; the ownership handoff is now also hardened
  against the distribution driver's udev bind-event power-policy write, and a
  reusable supervised claim-cycle harness exists
  (`tools/guest-diag/phys_claim_cycle.sh`). The real-GSP-boot design is
  complete: exact secure-verification/booter/RISC-V/RM-init sequence, boot
  artifacts (booter/FMC/SEC2 BL bindata; VBIOS BIT FWSEC blob) and the real
  FB-size decode (local-memory-range register; the BAR1-aperture approximation
  is retired). Implementation starts with the real FB size and layout
  self-checks, then secure verification, then the booter and RISC-V start.
- Host stability: the host hang class observed on 2026-09-28 (kernel stack
  overflow in a channel snapshot path) is fixed and frame-audited; repeated
  VM load/attach/teardown cycles run without panics or pstore records. A
  second, distinct freeze class was root-caused on 2026-09-30: the dGPU
  board's power rail is gated by an ACPI power resource behind its root
  port, released through an embedded-controller command. Unbinding the
  distribution driver (as the physical-GPU claim path does) lets the device
  runtime-suspend, the rail is cut, and the re-bind intermittently fails to
  power it back (`not ready ... after resume`), with the host hard-freezing
  about a minute later (no crash record; journal stops, pstore empty; both
  occurrences followed an unbind/rebind, and prior log history has no other
  instance). The claim path refuses to run unless both `gsp_phys_gpu=1` and
  `gsp_phys_hazard_ack=1` are passed, and the power-domain lock (hold the GPU
  and its board functions plus the root port in D0 across the handoff,
  aborting and holding the locks on any power-loss signature) is now
  **implemented and verified across three supervised claim/release cycles**
  (2026-10-01: zero power-domain cycles, zero failed resumes, clean rebind,
  host healthy). The verified launch harness holds the GPU and root port in
  D0 before the distribution driver is stopped; the root-port hold is the
  load-bearing invariant. The path must still not run unattended and every
  cycle is supervised. Normal module/VM work does not touch the driver
  binding and is unaffected. See
  [`docs/03-architecture/08-physical-gpu-integration.md`](docs/03-architecture/08-physical-gpu-integration.md).

## Initialization, work and teardown path

Status tags: `[done]`, `[frontier]` (actively worked), `[partial]`,
`[not yet]`.

```
        HOST SIDE                                        GUEST (stock driver, unmodified)
        =========                                        ================================

 [done] build + load open-iov-vfio.ko
        |  (module first: exports the symbols the NVIDIA open modules link against)
        v
 [done] mdev create ------------------------------------>  /
        |  BAR/PCI/MSI-X emulation; GSP emulation thread
        v
 [done] VM attach (QEMU q35 + pcie-root-port) ---------->  config space read; BAR mmap
        |
        v
 [done] GSP boot handshake (emulated) ------------------>  nvidia.ko probe / RmInitAdapter
        |
        v
 [done] GSP-RM message queues / RPC / controls --------->  object + client setup
        |
        v
 [done] engine + class lists --------------------------->  GPU_GET_ENGINES, class DB
        |                                                  (two-GRCE fix; async copy engine)
        v
 [done] ------------------------------------------------>  nvidia-smi -L enumerates TU104
        |
        v
 [done] ------------------------------------------------>  cuInit / DeviceGetCount / DeviceGet
        |
        v
 [done] ------------------------------------------------>  cuCtxCreate
        |                                                  UVM register GPU; channel manager
        |                                                  GR + CE channels; push buffers
        |                                                  semaphore pool; completion counters
        v
 [done] ------------------------------------------------>  cuMemAlloc + cuMemcpy roundtrip
        |                                                  (passes end-to-end; payload 64/64 correct)
 [partial] --------------------------------------------->  full nvidia-smi output / kernel launch
        |                                                  (runtime init rc=0; launch queued;
        |                                                   execution awaits the backend)
```

Work execution (host service vs client synchronization):

```
 [done] host decodes guest pushes (method streams): memsets, copies, releases
           - CE pushes execute at scale; the compute-class inline-DMA HtoD and
             the copy-engine-on-compute DtoH both execute; payload 64/64
           - the runtime client now initializes (rc=0) and queues a kernel
             launch; the queued compute pushes decode but are not executed
 [partial] GPU VA -> guest PA translation: page-table walk fixed and succeeding;
           server-owned page-table population still in progress
 [not yet] real work forwarded to the physical GPU (host-RM proxy or direct submission)
 [not yet] correct results (vectorAdd)
```

Teardown and recovery:

```
 [done]    happy-path teardown exercised end-to-end (cuCtxDestroy OK)
 [done]    VM stop -> mdev remove -> rmmod, in order; never rmmod with a live VM attached
 [done]    crash recovery: pstore inspection; the host hang class is fixed and frame-audited
```

Recent chain of walls removed (2026-09-28 to 2026-09-30):

```
 host hang (stack overflow) -> pump state persistence -> map collapse (merge rebuild)
   -> rescue overhaul -> four-word semaphore releases -> counter injection (all 20)
   -> stale semaphore-pool translation -> fake-only proven insufficient
   -> push buffer found and validated -> per-page pins -> authoritative pin table
   -> deterministic chain solved -> pin-drop and release-push fixes
   -> copy-engine data path executes -> compute-window pinning
   -> registration release delivery -> 20/20 counter slots -> GR-info clock fix
   -> cuCtxCreate OK; payload found (compute-class inline DMA)
   -> GMMU walk decode fixed (child/leaf encoding) -> inline HtoD lands
   -> DtoH copy-engine-on-compute allowed -> payload roundtrip 64/64
   -> runtime client initializes rc=0 (legacy clock-domain fix) -> kernel launch
      queued -> real-release chain rc=0 (pump bookkeeping fixes) -> execution
      backend bring-up is the next wall
```

## Tools, code paths and fixes behind this progress

```
 KERNEL MODULE (open-iov-vfio.ko)                GUEST-SIDE TOOLS (tracked)
 ================================                ==========================
 gsp_emulation.c      boot, doorbells, MSI-X      tools/test/cuda_drv_test.c   smoke test
 gsp_rm_emulation.c   msgq, RPC, controls         tools/guest-diag/vmrestart.sh
   gsp_rm_tu104.h       two-GRCE engine fix       tools/guest-diag/cuda_cycle.sh
 gsp_sema_service.c   THE WORKHORSE               tools/guest-diag/push_guest_tools.sh
   channel pump, method decode, rescue,           tools/guest-diag/inject_counters.sh
   map rebuild, pins, counters, releases          tools/guest-diag/semfix_watch.py
 bar_emulation.c      BAR shadows                 tools/guest-diag/track_waits.sh
 main.c               mdev/VFIO lifecycle         tools/guest-diag/trkinfo.py
   stack-frame audit (-Wframe-larger-than)        tools/guest-diag/bp999.py / scanerr.py
 dynamic_rpc_dispatcher.c  firmware load fix      tools/guest-diag/read_va.py
                                                  tools/guest-diag/guest_kprobes.sh

 DATA STRUCTURES                                 DEBUG PLANES
 ===============                                 ============
 channel table (handle, GPFIFO, USERD,           module stats: applied / miss / memset /
   notifier, ring)                                 ringops / nomap / skip / faked /
 VA->PA map (merge rebuild, hot pages, cache)      pins / ipins
 pin table + authoritative injected pin table    debug params: sema_probe, sema_counter,
 completion counter table (per channel)            sema_map_add, chan_dump, shadow_dump
 semaphore pool page (GPU VA 0x121000000)        QMP gva2gpa + guest gdbstub reads
 push buffer allocation (GPU VA 0x120000000)     guest kprobes; /proc/<pid>/pagemap
 client trackers / RM error record chain         marker-based /dev/kmsg parsing
```

| Wall | Fix |
|---|---|
| Host panic: oversized stack frame in a channel snapshot path | static buffers; frame-size audit on every build |
| Channel pump lost per-pass state (stale entries re-run; rate limit dead) | write-back of rescue/payload state; one-retry skip for compute channels |
| VA-to-PA map collapsed after client teardown | merge rebuild instead of wholesale replace |
| Rescue slow / false candidates | hot-page-first scan; full method-stream validation; retry limits |
| Four-word semaphore releases ignored | decode both release encodings |
| Only 4 of 20 completion counters ever moved | async injection queue; fast injector; per-boot pin, order fixed |
| UVM tracker wait never cleared | stale pool translation found; pool page pinned from guest truth each boot |
| Acknowledging releases without executing work | proven insufficient; pushes must be read and executed |
| Push buffers unreadable | located by CPU VA; per-page translation; authoritative injected pin table |
| Injected pins silently dropped once a pin-table cluster grew | authoritative injected-pin table checked first; larger table, longer probe |
| Real release pushes rejected as implausible (non-zero method start) | decode release-only when the translation came from injected pins |
| Completion release decoded but deferred (counter page untranslated) | pin the compute-channel mapping window from guest truth each boot |

## What "CUDA works" means (open gates)

Per [`docs/04-implementation/01-known-knowns-and-unknowns.md`](docs/04-implementation/01-known-knowns-and-unknowns.md)
section 4, a defensible claim needs **all** of:

1. every guest-to-device message for the CUDA path answered or forwarded,
   with coherent replies for consumed fields;
2. every work VA translatable and server-owned page-table ranges populated;
3. launches of a grid of thread blocks (a CUDA kernel dispatch, e.g.
   `vectorAdd`'s `<<<blocks, threads>>>`) actually executed on the physical
   GPU's compute engine - not merely acknowledged - and copies executing on
   real hardware with correct results;
4. completion (semaphores, events, faults) behaving as the guest expects;
5. at least two driver families passing;
6. tenant isolation;
7. measured performance within an agreed factor of native.

Until then the honest statement is: **initialization works; execution is in
progress.**

## Environment

- Host GPU: NVIDIA TU104 (RTX 2080 SUPER Mobile), PCI `10de:1ed3`.
- Host OS: Ubuntu, kernel 6.8.0-142; the distribution NVIDIA driver manages
  the physical GPU; `open-iov-vfio.ko` loads as an independent module.
- Guest: Ubuntu 24.04.5 LTS, stock NVIDIA 570.144 driver, QEMU/KVM q35 with a
  `pcie-root-port` for the vGPU, SeaBIOS, mdev passthrough.
- Firmware: TU104 GSP 570.144 (public release).

## Build and load order

Build: host module first (it generates the symbol versions the NVIDIA open
kernel modules link against), then the NVIDIA open kernel modules, then
userspace.

Load: VFIO prerequisites, then `open-iov-vfio.ko`, then create the mdev and
attach it to the VM. Unload in reverse; never `rmmod` while a VM holds the
device.

## See also

- [`docs/README.md`](docs/README.md) - documentation front door
- [`docs/04-implementation/01-known-knowns-and-unknowns.md`](docs/04-implementation/01-known-knowns-and-unknowns.md) - known-knowns, known-unknowns, definition of done
- [`userspace/cuda/READ-ME-FIRST.md`](userspace/cuda/READ-ME-FIRST.md) - CUDA analysis entry point
- [`docs/03-architecture/08-physical-gpu-integration.md`](docs/03-architecture/08-physical-gpu-integration.md) - chosen execution backend (module-owned GPU + real GSP session)
- [`docs/03-architecture/09-full-system-picture.md`](docs/03-architecture/09-full-system-picture.md) - visual end-to-end system map with status tags
- [`docs/03-architecture/10-interaction-and-scheduling.md`](docs/03-architecture/10-interaction-and-scheduling.md) - cross-component interaction, queues, interrupts, scheduling detail

---

*Refresh this page on every progress check (snapshot time in the header);
prefer moving detail into the `docs/` tree and keeping this page short.*