Open-IOV: initial public release

Open-IOV virtualizes NVIDIA GPUs so that unmodified guest drivers run with
real GPU functionality - CUDA first, 3D and video to follow - on unmodified
hosts, without a proprietary vGPU license and without patching any driver,
firmware or hypervisor.

Core principle: emulate the interface, pass through the work.

  kernel/     open-iov-vfio.ko - host VFIO mdev driver: virtual GPU device
              (PCI config space, BAR shadows, MSI-X), GSP boot emulation,
              GSP-RM message-queue/RPC emulation, memory model (GPU-VA to
              guest-PA discovery) and the semaphore-completion service that
              advances guest UVM progress.
  userspace/  libiov.so management API and examples.
  tools/      guest diagnostics, firmware analysis wrappers, RM probes.
  scripts/    stack manager, VM harness, build/load/test automation.
  config/     hardware/version profiles (the portability surface).
  docs/       documentation tree - start at docs/README.md.
  wiki/       long-form firmware and protocol deep dives.

Status: initialization works end to end for a stock 570.144 Linux guest on
Turing (TU104): driver probe, nvidia-smi, the full GSP-RM control battery,
object creation and the early CUDA/UVM start-up path. GPU work forwarding -
the step that makes CUDA kernels execute on real hardware - is the active
work item; see docs/04-implementation/01-known-knowns-and-unknowns.md.

License: BSD-3-Clause for the project; GPL-2.0 for kernel modules that link
GPL-only kernel interfaces. See LICENSE and docs/09-reference/legal-and-licensing.md.

1 files changed, 100 insertions, 0 deletions

diff --git a/STATE.md b/STATE.md
new file mode 100644
index 0000000..73a8654
--- /dev/null
+++ b/STATE.md
@@ -0,0 +1,100 @@
+# Open-IOV Project State
+
+*Last updated: 2026-09-21 (PM)*
+
+## Mission
+
+Run **unmodified NVIDIA guest drivers** in VMs with full GPU functionality
+(CUDA, graphics, video, AI/ML) on consumer/datacenter GPUs, without
+proprietary GRID licenses — via host-side GSP emulation in
+`open-iov-vfio.ko`.
+
+## Current milestone
+
+| Phase | Status |
+|-------|--------|
+| GSP 570.144 firmware decompilation (132 files, 12 RPC types) | ✅ Complete |
+| Refactor 1: GSP emulation engine (per-mdev thread, doorbell, shared mem, MSI-X) | ✅ Complete |
+| Refactor 2: PCI config space + BAR + MSI-X emulation | ✅ Complete |
+| Refactor 3: All 12 RPC types with emulation fallbacks | ✅ Complete |
+| Refactor 4: Portable architecture (fw analyzer, dynamic dispatcher, arch/version layers, YAML configs) | ✅ Complete |
+| Refactor 5: Guest driver integration (mdev creation, VFIO registration) | ✅ Unblocked, in test |
+| Refactor 6-11: CUDA / graphics / benchmarks / migration / Windows / hardening | ⏳ Pending |
+
+**Right now**: the **guest-probe hang root cause is found and fixed** (VFIO
+mmap garbage PFN → KVM EPT livelock — see below) plus a sweep of correctness
+fixes (mdev remove leaks, eventfd wiring, GSP shadow binding, RPC livelock).
+Guest driver **570.144 stock runfile installed** in the VM. The fixed module
+is built and pushed; it loads after the next host reboot (a leaked
+`gsp_emu` kthread from the pre-fix remove path pins the old module — rmmod
+would oops). Post-reboot: guest `nvidia.ko` probe → `nvidia-smi` → teardown
+regression → unit test → CUDA validation (`docs/DEBUGGING.md` §5 checklist).
+
+## Recently solved bugs
+
+| Bug | Root cause | Fix |
+|-----|-----------|-----|
+| `available_instances` stuck at 0, mdev create fails `-EUSERS` | mdev core's shared `parent->available_instances` counter leaked on crashed removes | `mdev_driver.get_available` callback computes availability from the driver's own device list (kernel/main.c) |
+| Module unload hang (`open_iov_vfio` stuck "Unloading") | Legacy `nv_vgpu_gsp_init()` booting physical GSP firmware, conflicting with host nvidia.ko; unload blocked in `cancel_work_sync` | Skip legacy GSP boot, use `gsp_emu_init()`; thread-stop via completion with 5s timeout; `cancel_work_sync` before `destroy_workqueue` |
+| mdev remove kernel panic (`vfio_device_group_unregister+0xa`) | NULL deref on VFIO group unregister | Defensive cleanup ordering in `nv_vgpu_mdev_remove()` |
+| mdev creation `EINVAL` (-22) | Double IOMMU group handling conflicted with `vfio_register_emulated_iommu_dev()` | Let VFIO create its own group; IOMMUFD emulated callbacks wired |
+| `Unknown symbol nvidia_vgpu_vfio_probe` | NVIDIA OpenRM built without `KBUILD_EXTRA_SYMBOLS` | Build order automated in `stack_manager.sh` (kernel → nvidia → userspace) |
+| `vfio 1: No such file or directory` | VFIO device registration is NOT automatic for mdev | `vfio_register_emulated_iommu_dev()` in probe, unregister in remove (Intel kvmgt pattern) |
+| Guest-visible PCI config space wrong | BAR types/offsets, subsystem ID offsets, MSI-X table size | Config space rewritten to match physical TU104 (BAR0 16MB 32-bit non-prefetch, BAR1 256MB 64-bit prefetch VRAM, BAR3 32MB; 32 MSI-X vectors; subsystem IDs at 0x2C/0x2E) |
+| **Guest driver probe hard-hangs whole VM (580 & 570.144)** | VFIO `mmap` used `virt_to_phys()` on `vzalloc`'d BAR0 shadow + ioremap'd BAR2/3 → garbage PFN → KVM `make_spte` reserved-bit WARN → EPT-violation livelock (diagnosed via QEMU GDB stub: one vCPU frozen on a single MMIO instruction) | BAR0 mapped per-page via `vmalloc_to_pfn()`; BAR2/3/4/5 map `bars[N].gpa` directly; `nv_vgpu_mmap_virt_to_pfn()` helper |
+| **GSP emulation thread never saw guest RPCs** | Engine polled the *host physical* BAR0 doorbell + a local `dma_alloc_wc` buffer; guest writes land in the vzalloc `bar0_shadow` | `gsp_emu_init` binds exclusively to `bar0_shadow` (doorbell/mailbox/PBA aliases); also stopped PBA writes corrupting the host GPU's physical BAR0+0x3000 |
+| **Guest never received MSI-X completions** | `SET_IRQS DATA_EVENTFD` handler discarded eventfds (`TODO` + `kfree`); `msix_trigger[]` all NULL; inject used an init-time (stale) copy | Full eventfd wiring: `eventfd_ctx_fdget` per vector, `-1` clears, close releases refs; inject looks up `vgpu->irqs[0].trigger` live |
+| **BAR_TRAP/MIGRATE RPC livelock** | dispatch → `gsp_rpc_submit_raw` rings the same doorbell the thread polls → self-triggered infinite loop + 2–30 s stalls | emulation-mode guards return `-ENOTSUPP` with `pr_warn_ratelimited` |
+| **mdev remove leaked everything** | remove never `list_del`'d (available_instances stuck), never ran cleanup (leaked `gsp_emu` kthread), never unregistered VFIO (leaked `/dev/vfio/N` + iommu group), never put pdev/freed struct | kvmgt-style remove: wait fds → `vfio_unregister_group_dev` → cleanup work → `list_del` → `pci_dev_put` → `vfio_put_device` |
+| `dma_map_page` IOVA corruption | 64-bit `dma_addr_t` stored in `int ret` (truncates ≥4 GB IOVAs under IOMMU) | dedicated `dma_addr_t iova` in `gsp_rpc_alloc_emul` |
+| config-space write returned `-1` (`SIZE_MAX`) | `while(count--)` wrap + bogus return expression; function was dead code while the live path did unfiltered byte copies | return fixed; VFIO write handler now routes through the per-register filter (guest can't rewrite vendor ID/BARs) |
+| Misleading `MODULE_SOFTDEP("pre: nvidia")` | module has zero nvidia imports (it *exports* `nvidia_vgpu_vfio_*`) | removed; load order is open-iov-vfio **before** nvidia.ko |
+| Kernel unit tests didn't build | written against a removed struct-based RPC API | rewritten to drive `gsp_emu_process_rpc()` with crafted mailbox messages; `depends: open-iov-vfio` |
+
+## Known open bugs / limitations
+
+- Guest `nvidia.ko` end-to-end probe not yet validated (fixed module loads
+  post-reboot; checklist in `docs/DEBUGGING.md` §5)
+- **Memory-model gap**: guest VRAM (BAR1) maps directly to physical BAR1 gpa,
+  but `gsp_rpc_alloc_emul` services ALLOC with host system pages — the two
+  are not unified; must be resolved before CUDA/DMA-heavy workloads
+  (`docs/GSP_EMULATION_ARCHITECTURE.md` §3)
+- BAR_TRAP / MIGRATE RPCs fail fast with `-ENOTSUPP` in emulation mode
+  (guest's normal BAR access is direct MMIO; real handlers TODO)
+- `gsp_emu_process_rpc()` doesn't bump stats counters (only the kthread
+  path does) — cosmetic
+- WPR callback registration lacks kernel eventfd support
+  (`TODO` in driver/wpr — legacy guest-driver tree)
+- Migration WPR state save/restore incomplete
+- BAR `force_access` path not fully wired (unused-function warning)
+- Kernel `SCHEDULE` emulation returns success without real work submission
+  (userspace emulation works)
+- Async `NOTIFY` events not emulated in kernel (no eventfd/poll yet)
+- Single GPU validated so far: TU104 (RTX 2080 SUPER Mobile), GSP 570.144,
+  kernel 6.8.0-139-generic
+- Live migration protocol implemented (4-phase) but not yet validated
+- Windows guest not yet validated (automation scaffolding present)
+- MSI-X emulation has dual stores (guest-visible `bar0_shadow` table vs
+  `bar_emul_ctx.msix_table[]` used by the currently-unused trap path) —
+  documented in `docs/GSP_EMULATION_ARCHITECTURE.md`; harmless with the
+  direct-mmap design
+
+## Reproducibility notes
+
+- NVIDIA base: `open-gpu-kernel-modules` tag **610.57.04** + patch series
+  in `patches/nvidia/` (re-brands version string to 570.144 to match the
+  GSP firmware ABI; adds the vGPU VFIO interface headers + symbol exports).
+- GSP firmware: `nvidia/tu102/gsp/gsp-570.144.bin` from
+  [linux-firmware.git](https://github.com/NVIDIA/linux-firmware)
+  (pin `2237b99e`), `.fwimage` section extracted with `objcopy`,
+  MD5 `13ce12a392a31eeb79a2e6ebd3a4a93a`. `install.sh` does this.
+- Firmware blobs, decompiled output, NVIDIA tree, VM images, Ghidra DBs and
+  local VM wrappers are intentionally **not** tracked in git.
+- Fresh-system bring-up: `./install.sh` (see README.md).
+
+## Hardware/software reference environment
+
+- Host GPU: NVIDIA RTX 2080 SUPER Mobile (TU104, PCI `10de:1ed3`)
+- Host OS: Ubuntu, kernel 6.8.0-139-generic
+- Guest OS: Ubuntu 24.04.5 LTS (QEMU/KVM, q35, OVMF)
+- GSP firmware: 570.144 (RV64IMAC, base `0x04000000`)