path: root/STATE-CURRENT.md

# Open-IOV Current State

> **Snapshot: 2026-10-01 12:00.** This is a dated public status page, not live
> state; it is refreshed on every progress check. Live working notes are
> maintained outside this repository; the public known-knowns/unknowns document
> is authoritative here:
> [`docs/04-implementation/01-known-knowns-and-unknowns.md`](docs/04-implementation/01-known-knowns-and-unknowns.md).
> The former 2026-09-21 snapshot (`STATE.md`) is retired; its content remains
> in the git history.

## Mission

Run **unmodified NVIDIA guest drivers** in VMs with full GPU functionality
(CUDA, graphics, video, AI/ML) on consumer/data-center GPUs without
proprietary GRID licenses - via host-side GSP emulation in
`open-iov-vfio.ko`. The guest stays stock: no guest patches, no guest-side
Open-IOV code.

Architecture stance: *emulate the interface, pass the work through to real
hardware, never patch anyone.* The control plane is emulated; the work is
meant to execute on the physical GPU.

## Milestone ladder (this snapshot)

| Milestone | Status |
|---|---|
| GSP boot handshake, GSP-RM message queues/RPC, BAR/PCI/MSI-X emulation | ✅ |
| Stock 570.144 guest driver probe / `RmInitAdapter` | ✅ |
| `nvidia-smi -L` enumerates the emulated TU104 | ✅ |
| CUDA userspace: `cuInit`, `cuDeviceGetCount`, `cuDeviceGet` | ✅ |
| `cuCtxCreate` returns | ✅ (the whole smoke test now completes) |
| `cuMemAlloc` + `cuMemcpy` roundtrip (`ALL OK` in the smoke test) | ✅ payload roundtrip verified 64/64 in the guest |
| Full `nvidia-smi` output / kernel launch | 🟡 the static runtime now initializes end-to-end with **real releases** (`cudaGetDeviceCount`/`cudaSetDevice`/3×`cudaMalloc`/2×`cudaMemcpyHtoD`/`kernel launch`/`cudaGetLastError` all `rc=0`) and queues a kernel launch; execution awaits the backend |
| CUDA kernel execution (`vectorAdd`) | 🟡 execution backend: physical ownership + firmware staging verified; the unbind/rebind freeze hazard is mitigated by the supervised D0 power-domain lock (3/3 green claim cycles), and the real FB size (8 GiB) and boot layout are verified live. The supervised live secure-verification run raised the protected region (WPR2 up) for the first time, then proved a secure-verification-only carve is not host-recoverable on this fused part, so the fail-safe held the device and an approved reboot restored it (distribution driver rebound, GPU healthy). That one-way stage is now retired: the only device-write path is a single full boot lifecycle (secure verification -> bootloader -> RISC-V start -> resource-manager initialization), whose code is implemented in the module sources and has now run live: the first real boot raised WPR2, passed Booter Load and went RISC-V active, and the module received the first real GSP-RM message (event 0x1002); init-done (0x1001) and a clean release are the next steps (see the 2026-10-01 section). A second live boot on 2026-10-01 proved the full boot sequence again (protected region up, bootloader halt, RISC-V active, first GSP-RM message) and reached the resource-manager's inline CPU-sequencer request; the host executor rejected the real request because of an over-strict internal size cap, which was found with a raw read of the message in host memory, fixed, and rebuilt. The next supervised cycle is expected to run the sequencer, receive init-done and complete a clean release via the shutdown RPC. |
| Graphics (3D), video | ❌ |
| Faults, multi-version, Windows, migration, multi-tenancy, performance | ❌ |

Test status is tracked with the CUDA driver-API smoke test
([`tools/test/cuda_drv_test.c`](tools/test/cuda_drv_test.c)): it exercises
`cuInit -> cuDeviceGetCount/Get -> cuCtxCreate -> cuMemAlloc -> cuMemcpyHtoD/
DtoH -> cuMemFree -> cuCtxDestroy` and prints `ALL OK` only when the whole
chain succeeds.

## Right now (frontier)

- **Driver-API baseline re-verified on the current build**: the smoke test
  passes end-to-end (`ALL OK`, payload 64/64) with real releases
  (`faked=0`, `miss=0`, `stale=0`), confirming the recent pump/parser and
  rewind changes did not regress the baseline.
- **Physical-backend milestone (2026-10-01)**: the module-owned GPU path
  reached the first successful secure-verification (FWSEC) carve of the
  protected region on the real TU104 under the verified D0 power-domain
  harness. The teardown of a FWSEC-only carve is not host-recoverable on this
  fused part (both the secure-scratch and MMU privilege masks are fuse-locked,
  so the bootloader handoff cannot be seeded and WPR2 cannot be unlocked from
  the host), so the fail-safe deliberately held the device and an **approved
  reboot restored it** (distribution driver rebound, `nvidia-smi -L` green,
  host healthy, no crash record). The one-way stage is now retired: the only
  device-write path is a single supervised full boot lifecycle (secure
  verification -> bootloader -> RISC-V start -> resource-manager
  initialization). That lifecycle is now implemented in the module sources
  (bootloader runner + generated verification/boot image data) and has run
  live under the supervised harness: the first real boot succeeded (see the
  section below), and a second live cycle on the rung-4 build repeated the
  boot and reached the server's run-sequencer request; the executor's size
  cap rejected the real request value (16354), which was diagnosed from a
  raw host-memory read of the message, fixed, and rebuilt. The post-reboot
  recovery is now verified and the zero-timeout REG_POLL default fix
  (commit `98b861d`, build `87309e3a…`) was confirmed live. Cycles 2-3
  (2026-10-01 11:40 and 11:46) then reached **stage 3e on real hardware**:
  the real GSP-RM executed the full 165-command/490-dword sequencer and
  reported RM init failure `0x59` (`NV_ERR_OPERATING_SYSTEM`) on both
  firmware pairings (the default 570.211.01 image and a verified matching
  570.144 image - the image-pairing hypothesis is disproven). Each failed
  cycle now **releases cleanly back to the distribution driver without a
  reboot**: the server's own shutdown sets the remaining secure-scratch
  handoff bit, so all four handoff bits are set and a single `rmmod` clears
  the protected region, rebinds the distribution driver and releases the
  power-domain locks. The next evidence is the **GSP-RM NOCAT journal** (the
  matching-image cycle streamed ~851 journal records), whose capture is
  being built.
- **The full CUDA driver-API smoke test passes with correct data**: the client
  completes `cuCtxCreate -> cuMemAlloc -> cuMemcpyHtoD -> cuMemcpyDtoH ->
  cuMemFree -> cuCtxDestroy` and prints `ALL OK`, with the 64-byte payload
  roundtripping exactly (64/64 verified in the guest).
- **The last two walls**: the GPU page-table walk's address decode (child
  pointers and leaves were 16x too low; fixed with the real entry encoding -
  walks now succeed at scale) and the device-to-host copy (a copy-engine
  class transfer bound on a compute channel whose work was denied by the
  engine-type gate; trusted host-injected pushes may now execute copy-engine
  work as well).
- **Baseline green and quiet**: the payload roundtrip is back to 64/64
  (`ALL OK`), the staging buffer's translation stays resolvable via pins
  streamed from the guest, and high-frequency diagnostics are now gated
  behind a default-off debug switch - a normal boot+run produces a ~520-line
  log instead of a flood (verified twice). The page-directory entry writes
  have since been captured live and validated (the earlier all-zero boots
  were operation-skipped/pre-pin boots); the remaining question is the
  device-address walk gap for the runtime's own buffers.
- Recap of the fixes behind this point: all 20/20 compute-channel releases
  land (three decode paths routed to the exact counter slot); registration
  delivers the incrementing tracking release for every consumed submission;
  the copy-engine push buffer is pinned from guest truth on every boot; and
  the copy-engine decoder consults pinned translations before the scanned
  map, so every push-data operation executes (`memset=19473`, `ringops=10551`
  live).
- **The runtime client now initializes and reaches kernel launch**: with a
  static-runtime `vectorAdd` client, `cudaGetDeviceCount`, `cudaSetDevice`,
  `cudaFree`, three `cudaMalloc`, the HtoD copies and `cudaGetLastError` all
  return success. The client then parks in `cudaDeviceSynchronize` - the
  launch is queued but never **executed**, because real execution requires
  the physical-GPU backend (the next project).
- **What fixed it**: the failing call was never a kernel call - it was the
  clock-domain probe inside libcuda (two device attributes) whose three
  legacy clock controls were answered with zeros; they are now populated
  with exactly the fields the helper checks.
- **Real-release mode now reaches the same rc=0 chain**: the earlier
  per-push 8 s stall was two pump bookkeeping bugs (release detection keyed on
  an unchanged semaphore VA; rewind replays re-entering the CE grace path) -
  both fixed, and the runtime init completes in ~1 min with real releases.
  The content rescue also learned the real method encoding (NON_INC/IMMD/
  ONE_INC/END_PB headers).
- Active work: the owed driver-API smoke-test rerun is now **verified 64/64
  (`ALL OK`)** on the current commits, so the recent pump/parser changes did
  not regress the baseline. The device-claim bring-up verified ownership,
  firmware staging and the boot layout, but unbinding the distribution driver
  can power-gate this laptop's dGPU and hard-freeze the host (two occurrences,
  now root-caused), so that path is disabled behind a two-parameter hazard
  gate. A power-domain lock that keeps the board in D0 across the handoff is
  implemented and **verified across three supervised claim/release cycles on
  2026-10-01** (zero power-domain cycles, zero failed resumes, distribution
  driver rebound cleanly each time, host healthy); the verified launch
  harness holds the GPU and root port in D0 before the distribution driver is
  stopped, and the root-port hold is the load-bearing invariant. The real GSP
  boot has since run live on this path (see the 2026-10-01 section below).
- Observability: a size-keyed copy watch, a VRAM shadow dump/search, and an
  operation log are now module parameters, and the test's output survives
  guest-agent stalls via the serial console.
- Guest health note: agent/SSH timeouts during long runs are the guest-side
  trace reader flooding the ring; they recover on their own, so a first
  timeout is not a reason to restart the VM. Separately, one no-watcher boot
  crashed the guest kernel (a NULL callback in interrupt context) before any
  emulation channel existed - guest-side and untriaged; treat no-watcher runs
  as a known risk. The host stays stable throughout.
- Execution backend (chosen; ownership hazard mitigated and verified): the
  module owns the physical GPU and will run its real GSP firmware as the
  resource-manager server, so real work executes with an unmodified host
  driver and no host daemon; see
  [`docs/03-architecture/08-physical-gpu-integration.md`](docs/03-architecture/08-physical-gpu-integration.md).
  The ownership path was verified live (release the distribution driver from
  the device, bind the module, map the BARs, sample the falcon state, then
  release and rebind the distribution driver); with the device owned, the
  matching GSP image is fetched and validated (ELF64 RISC-V, loadable
  image/version/signature sections identified) and the protected-region
  video-memory/boot layout is computed and logged - all without device writes.
  The unbind/rebind hazard is mitigated by the D0 power-domain lock, verified
  over three supervised cycles; the ownership handoff is now also hardened
  against the distribution driver's udev bind-event power-policy write, and a
  reusable supervised claim-cycle harness exists
  (`tools/guest-diag/phys_claim_cycle.sh`). The real-GSP-boot design is
  complete: exact secure-verification/booter/RISC-V/RM-init sequence and boot
  artifacts (booter/FMC/SEC2 BL bindata; VBIOS BIT FWSEC blob). The real FB
  size (8 GiB, decoded from the local-memory-range register - the BAR1-aperture
  approximation is retired) and the protected-region boot layout have been
  verified live under the supervised claim harness (three green claim/release
  cycles; the root-port D0 hold is the load-bearing invariant), and the
  secure-verification stage (VBIOS security-blob extraction, falcon reset and
  staging) was validated offline and in a read-only dry-run, then run **live
  under the supervised claim harness**: it raised the protected region (WPR2
  up) for the first time on real hardware. Its release teardown, however,
  could not clear WPR2, and the fail-safe deliberately held the device
  (distribution driver unbound, power-domain locks held) rather than hand it
  back; a rescue path is being debugged (or, failing that, an approved
  reboot). A distribution-driver bus-master quirk on the claim path was also
  found and fixed in this run. The bootloader/RISC-V/resource-manager-init
  stages are now implemented in the module sources (bootloader runner plus
  generated boot and verification image data) and have now run live: the
  boot completed through an active RISC-V core and the module received the
  first real GSP-RM message (see the section below); a clean release is
  pending the server shutdown RPC.
- Host stability: the host hang class observed on 2026-09-28 (kernel stack
  overflow in a channel snapshot path) is fixed and frame-audited; repeated
  VM load/attach/teardown cycles run without panics or pstore records. A
  second, distinct freeze class was root-caused on 2026-09-30: the dGPU
  board's power rail is gated by an ACPI power resource behind its root
  port, released through an embedded-controller command. Unbinding the
  distribution driver (as the physical-GPU claim path does) lets the device
  runtime-suspend, the rail is cut, and the re-bind intermittently fails to
  power it back (`not ready ... after resume`), with the host hard-freezing
  about a minute later (no crash record; journal stops, pstore empty; both
  occurrences followed an unbind/rebind, and prior log history has no other
  instance). The claim path refuses to run unless both `gsp_phys_gpu=1` and
  `gsp_phys_hazard_ack=1` are passed, and the power-domain lock (hold the GPU
  and its board functions plus the root port in D0 across the handoff,
  aborting and holding the locks on any power-loss signature) is now
  **implemented and verified across three supervised claim/release cycles**
  (2026-10-01: zero power-domain cycles, zero failed resumes, clean rebind,
  host healthy). The verified launch harness holds the GPU and root port in
  D0 before the distribution driver is stopped; the root-port hold is the
  load-bearing invariant. The path must still not run unattended and every
  cycle is supervised. The device has since been restored by the approved
  reboot (distribution driver rebound, GPU healthy); the secure-verification
  stage is now folded into a single full boot lifecycle, and normal module/VM
  work does not touch the driver binding and is unaffected. See
  [`docs/03-architecture/08-physical-gpu-integration.md`](docs/03-architecture/08-physical-gpu-integration.md).

## First real GSP boot + live GSP-RM session (2026-10-01)

Cycle 09:53:27–09:54:59 on the supervised D0 harness (`--boot --keep`),
`open-iov-vfio.ko` sha256 `7815dacc…`, srcversion `2993AF5C…`. The boot order
observed on the physical TU104:

1. **FWSEC** — WPR2 came up; FRTS error code 0; `WPR2_LO=0x01ffe000`,
   `WPR2_HI=0x01ffee00`.
2. **Booter Load** on SEC2 — halted with `M0=0x0` after 146 ms.
3. **RISC-V active** — `RISCV_STATUS=0x00000035` (active bit 0),
   `FALCON_OS=0`; secure scratch14 raw `0x11000000` (booter-load handoff bit
   24 = 1, booter binary version 1).
4. **First real GSP-RM message** — the staged status queue linked and the
   server's first event was `function=0x1002`
   (`NV_VGPU_MSG_EVENT_GSP_RUN_CPU_SEQUENCER`, `rpc_result=0`), carrying a
   command buffer the host must execute; `GSP_INIT_DONE` (0x1001) follows only
   after that sequencer runs and was not observed in the 60 s poll.

This was the first host↔real-GSP-RM exchange on the physical path. At that
point a clean release was not yet possible: the Booter Unload gate requires
all four secure scratch14 handoff bits (24–27); a host-driven boot set only
bit 24, and bit 27 was expected from the server's shutdown path (host RPC
`UNLOADING_GUEST_DRIVER`, not yet implemented then). The host-side seed/unlock
alternative is fuse-locked on this part. The session was therefore
intentionally left claimed (`--keep`): the GPU was bound to
`open-iov-gsp-phys` and runtime-active; `nvidia` stayed unbound (host
headless; `nvidia-smi` fails by design); host healthy, pstore empty. A fresh
live cycle needed a power cycle at that point. **Status update (cycles 2-3,
same day):** this limitation is retired - the server's shutdown set the
remaining handoff bit and a single `rmmod` released the device without a
reboot (see the frontier section).

**Next steps:** (a) execute the 0x1002 sequencer command buffer in the module
and log 0x1001 as `stage 3e complete`; (b) send `UNLOADING_GUEST_DRIVER` on
the command queue so the server shuts down, sets the remaining handoff bit and
teardown can hand the GPU back without a reboot; (c) prove a request/reply
round trip with the server. All three were implemented, built and have now
run live: cycles 2-3 executed the sequencer and reached init-done, and the
server's own shutdown supplied the missing handoff bit, so each failed cycle
released without a reboot; a request/reply round trip remains to be
exercised. The module sources contain the sequencer executor (BAR0
register-write/modify/poll, delay, store and core-resume opcodes), a
generalized msgq producer (multi-element records,
checksum/sequence/free-space rules, doorbell) with the unload RPC, a
reply-matching status-queue reader, and a debugfs probe that requests the
server's static configuration after init-done; the rung 4a/4b/4c code passed
an adversarial wire-format review plus its fixes.

**Prepared for the next cycle (now history):** an early build (`f61308bd…`)
ran the boot green but the server's run-sequencer request was rejected by an
over-strict host-side cap on the request's own buffer size. The message was
read raw from host memory, the cap corrected (build `87309e3a…`), and the
sequencer then ran live in cycles 2-3 (see the frontier section).

Ending a claimed session: `sudo rmmod open_iov_vfio`, then `echo auto | sudo
tee /sys/bus/pci/devices/0000:01:00.0/power/control` and the same for
`0000:00:01.0`, then `sudo systemctl start nvidia-persistenced`. Since cycles
2-3 this passes the handoff gate (the server's shutdown sets the last handoff
bit) and completes without a power cycle; if the protected region survives
teardown the fail-safe keeps the locks and a power cycle is still required,
with approval.

## Initialization, work and teardown path

Status tags: `[done]`, `[frontier]` (actively worked), `[partial]`,
`[not yet]`.

```
        HOST SIDE                                        GUEST (stock driver, unmodified)
        =========                                        ================================

 [done] build + load open-iov-vfio.ko
        |  (module first: exports the symbols the NVIDIA open modules link against)
        v
 [done] mdev create ------------------------------------>  /
        |  BAR/PCI/MSI-X emulation; GSP emulation thread
        v
 [done] VM attach (QEMU q35 + pcie-root-port) ---------->  config space read; BAR mmap
        |
        v
 [done] GSP boot handshake (emulated) ------------------>  nvidia.ko probe / RmInitAdapter
        |
        v
 [done] GSP-RM message queues / RPC / controls --------->  object + client setup
        |
        v
 [done] engine + class lists --------------------------->  GPU_GET_ENGINES, class DB
        |                                                  (two-GRCE fix; async copy engine)
        v
 [done] ------------------------------------------------>  nvidia-smi -L enumerates TU104
        |
        v
 [done] ------------------------------------------------>  cuInit / DeviceGetCount / DeviceGet
        |
        v
 [done] ------------------------------------------------>  cuCtxCreate
        |                                                  UVM register GPU; channel manager
        |                                                  GR + CE channels; push buffers
        |                                                  semaphore pool; completion counters
        v
 [done] ------------------------------------------------>  cuMemAlloc + cuMemcpy roundtrip
        |                                                  (passes end-to-end; payload 64/64 correct)
 [partial] --------------------------------------------->  full nvidia-smi output / kernel launch
        |                                                  (runtime init rc=0; launch queued;
        |                                                   execution awaits the backend)
```

Work execution (host service vs client synchronization):

```
 [done] host decodes guest pushes (method streams): memsets, copies, releases
           - CE pushes execute at scale; the compute-class inline-DMA HtoD and
             the copy-engine-on-compute DtoH both execute; payload 64/64
           - the runtime client now initializes (rc=0) and queues a kernel
             launch; the queued compute pushes decode but are not executed
 [partial] GPU VA -> guest PA translation: page-table walk fixed and succeeding;
           server-owned page-table population still in progress
 [not yet] real work forwarded to the physical GPU (host-RM proxy or direct submission)
 [not yet] correct results (vectorAdd)
```

Teardown and recovery:

```
 [done]    happy-path teardown exercised end-to-end (cuCtxDestroy OK)
 [done]    VM stop -> mdev remove -> rmmod, in order; never rmmod with a live VM attached
 [done]    crash recovery: pstore inspection; the host hang class is fixed and frame-audited
```

Recent chain of walls removed (2026-09-28 to 2026-09-30):

```
 host hang (stack overflow) -> pump state persistence -> map collapse (merge rebuild)
   -> rescue overhaul -> four-word semaphore releases -> counter injection (all 20)
   -> stale semaphore-pool translation -> fake-only proven insufficient
   -> push buffer found and validated -> per-page pins -> authoritative pin table
   -> deterministic chain solved -> pin-drop and release-push fixes
   -> copy-engine data path executes -> compute-window pinning
   -> registration release delivery -> 20/20 counter slots -> GR-info clock fix
   -> cuCtxCreate OK; payload found (compute-class inline DMA)
   -> GMMU walk decode fixed (child/leaf encoding) -> inline HtoD lands
   -> DtoH copy-engine-on-compute allowed -> payload roundtrip 64/64
   -> runtime client initializes rc=0 (legacy clock-domain fix) -> kernel launch
      queued -> real-release chain rc=0 (pump bookkeeping fixes) -> execution
      backend bring-up is the next wall
```

## Tools, code paths and fixes behind this progress

```
 KERNEL MODULE (open-iov-vfio.ko)                GUEST-SIDE TOOLS (tracked)
 ================================                ==========================
 gsp_emulation.c      boot, doorbells, MSI-X      tools/test/cuda_drv_test.c   smoke test
 gsp_rm_emulation.c   msgq, RPC, controls         tools/guest-diag/vmrestart.sh
   gsp_rm_tu104.h       two-GRCE engine fix       tools/guest-diag/cuda_cycle.sh
 gsp_sema_service.c   THE WORKHORSE               tools/guest-diag/push_guest_tools.sh
   channel pump, method decode, rescue,           tools/guest-diag/inject_counters.sh
   map rebuild, pins, counters, releases          tools/guest-diag/semfix_watch.py
 bar_emulation.c      BAR shadows                 tools/guest-diag/track_waits.sh
 main.c               mdev/VFIO lifecycle         tools/guest-diag/trkinfo.py
   stack-frame audit (-Wframe-larger-than)        tools/guest-diag/bp999.py / scanerr.py
 dynamic_rpc_dispatcher.c  firmware load fix      tools/guest-diag/read_va.py
                                                  tools/guest-diag/guest_kprobes.sh

 DATA STRUCTURES                                 DEBUG PLANES
 ===============                                 ============
 channel table (handle, GPFIFO, USERD,           module stats: applied / miss / memset /
   notifier, ring)                                 ringops / nomap / skip / faked /
 VA->PA map (merge rebuild, hot pages, cache)      pins / ipins
 pin table + authoritative injected pin table    debug params: sema_probe, sema_counter,
 completion counter table (per channel)            sema_map_add, chan_dump, shadow_dump
 semaphore pool page (GPU VA 0x121000000)        QMP gva2gpa + guest gdbstub reads
 push buffer allocation (GPU VA 0x120000000)     guest kprobes; /proc/<pid>/pagemap
 client trackers / RM error record chain         marker-based /dev/kmsg parsing
```

| Wall | Fix |
|---|---|
| Host panic: oversized stack frame in a channel snapshot path | static buffers; frame-size audit on every build |
| Channel pump lost per-pass state (stale entries re-run; rate limit dead) | write-back of rescue/payload state; one-retry skip for compute channels |
| VA-to-PA map collapsed after client teardown | merge rebuild instead of wholesale replace |
| Rescue slow / false candidates | hot-page-first scan; full method-stream validation; retry limits |
| Four-word semaphore releases ignored | decode both release encodings |
| Only 4 of 20 completion counters ever moved | async injection queue; fast injector; per-boot pin, order fixed |
| UVM tracker wait never cleared | stale pool translation found; pool page pinned from guest truth each boot |
| Acknowledging releases without executing work | proven insufficient; pushes must be read and executed |
| Push buffers unreadable | located by CPU VA; per-page translation; authoritative injected pin table |
| Injected pins silently dropped once a pin-table cluster grew | authoritative injected-pin table checked first; larger table, longer probe |
| Real release pushes rejected as implausible (non-zero method start) | decode release-only when the translation came from injected pins |
| Completion release decoded but deferred (counter page untranslated) | pin the compute-channel mapping window from guest truth each boot |

## What "CUDA works" means (open gates)

Per [`docs/04-implementation/01-known-knowns-and-unknowns.md`](docs/04-implementation/01-known-knowns-and-unknowns.md)
section 4, a defensible claim needs **all** of:

1. every guest-to-device message for the CUDA path answered or forwarded,
   with coherent replies for consumed fields;
2. every work VA translatable and server-owned page-table ranges populated;
3. launches of a grid of thread blocks (a CUDA kernel dispatch, e.g.
   `vectorAdd`'s `<<<blocks, threads>>>`) actually executed on the physical
   GPU's compute engine - not merely acknowledged - and copies executing on
   real hardware with correct results;
4. completion (semaphores, events, faults) behaving as the guest expects;
5. at least two driver families passing;
6. tenant isolation;
7. measured performance within an agreed factor of native.

Until then the honest statement is: **initialization works; execution is in
progress.**

## Environment

- Host GPU: NVIDIA TU104 (RTX 2080 SUPER Mobile), PCI `10de:1ed3`.
- Host OS: Ubuntu, kernel 6.8.0-142; the distribution NVIDIA driver manages
  the physical GPU; `open-iov-vfio.ko` loads as an independent module.
- Guest: Ubuntu 24.04.5 LTS, stock NVIDIA 570.144 driver, QEMU/KVM q35 with a
  `pcie-root-port` for the vGPU, SeaBIOS, mdev passthrough.
- Firmware: TU104 GSP 570.144 (public release).

## Build and load order

Build: host module first (it generates the symbol versions the NVIDIA open
kernel modules link against), then the NVIDIA open kernel modules, then
userspace.

Load: VFIO prerequisites, then `open-iov-vfio.ko`, then create the mdev and
attach it to the VM. Unload in reverse; never `rmmod` while a VM holds the
device.

## See also

- [`docs/README.md`](docs/README.md) - documentation front door
- [`docs/04-implementation/01-known-knowns-and-unknowns.md`](docs/04-implementation/01-known-knowns-and-unknowns.md) - known-knowns, known-unknowns, definition of done
- [`userspace/cuda/READ-ME-FIRST.md`](userspace/cuda/READ-ME-FIRST.md) - CUDA analysis entry point
- [`docs/03-architecture/08-physical-gpu-integration.md`](docs/03-architecture/08-physical-gpu-integration.md) - chosen execution backend (module-owned GPU + real GSP session)
- [`docs/03-architecture/09-full-system-picture.md`](docs/03-architecture/09-full-system-picture.md) - visual end-to-end system map with status tags
- [`docs/03-architecture/10-interaction-and-scheduling.md`](docs/03-architecture/10-interaction-and-scheduling.md) - cross-component interaction, queues, interrupts, scheduling detail

---

*Refresh this page on every progress check (snapshot time in the header);
prefer moving detail into the `docs/` tree and keeping this page short.*