diff options
| author | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-09-21 11:09:36 -0400 |
|---|---|---|
| committer | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-09-21 11:09:36 -0400 |
| commit | ad1f7a6a3cd262cc084673ec5cf0c561328ec7c7 (patch) | |
| tree | e3369e40b8f36345911091b542021a4899bb474d /STATE.md | |
| parent | 0ca41da65d675b624fb6389da1f488a24ea416e1 (diff) | |
| download | open-iov-ad1f7a6a3cd262cc084673ec5cf0c561328ec7c7.tar.gz open-iov-ad1f7a6a3cd262cc084673ec5cf0c561328ec7c7.zip | |
Make repo fully path-portable and add reproducible installer
- install.sh: one-command fresh-Ubuntu reproduction (deps, NVIDIA clone
+ patch, GSP firmware fetch from linux-firmware.git with objcopy
.fwimage extraction + MD5 verify, ordered build, host config, load,
mdev creation). Wrapper-neutral: works with libvirt/raw QEMU.
- patches/nvidia: regenerated path-independent patch (KBUILD_EXTRA_SYMBOLS
resolved via $(src)/$(CURDIR) relative to the NVIDIA tree).
- scripts/stack_manager.sh and helper scripts auto-detect repo root
(OPEN_IOV_ROOT override); no hardcoded developer paths anywhere.
- firmware analysis scripts use script-relative paths (also fixes stale
firmware/570-GSP references); docs/wiki updated to $OPEN_IOV_ROOT.
- .github/workflows/ci.yml updated for standalone repo layout; firmware
fetched from linux-firmware in CI.
- Untrack remaining kbuild artifacts (Module.symvers, modules.order,
*.o.d); gitignore them.
- STATE.md: live project state, limitations, fixed/open bugs.
- README.md: rewritten for clone-and-run reproducibility.
Diffstat (limited to 'STATE.md')
| -rw-r--r-- | STATE.md | 77 |
1 files changed, 77 insertions, 0 deletions
diff --git a/STATE.md b/STATE.md new file mode 100644 index 0000000..2621bbd --- /dev/null +++ b/STATE.md @@ -0,0 +1,77 @@ +# Open-IOV Project State + +*Last updated: 2026-09-21* + +## Mission + +Run **unmodified NVIDIA guest drivers** in VMs with full GPU functionality +(CUDA, graphics, video, AI/ML) on consumer/datacenter GPUs, without +proprietary GRID licenses — via host-side GSP emulation in +`open-iov-vfio.ko`. + +## Current milestone + +| Phase | Status | +|-------|--------| +| GSP 570.144 firmware decompilation (132 files, 12 RPC types) | ✅ Complete | +| Refactor 1: GSP emulation engine (per-mdev thread, doorbell, shared mem, MSI-X) | ✅ Complete | +| Refactor 2: PCI config space + BAR + MSI-X emulation | ✅ Complete | +| Refactor 3: All 12 RPC types with emulation fallbacks | ✅ Complete | +| Refactor 4: Portable architecture (fw analyzer, dynamic dispatcher, arch/version layers, YAML configs) | ✅ Complete | +| Refactor 5: Guest driver integration (mdev creation, VFIO registration) | ✅ Unblocked, in test | +| Refactor 6-11: CUDA / graphics / benchmarks / migration / Windows / hardening | ⏳ Pending | + +**Right now**: host stack (nvidia.ko 610.57.04-base + open-iov-vfio.ko) loads +cleanly, mdev device creates and removes cleanly, IOMMU group assignment +works, `/dev/vfio/<n>` appears. Guest VM (Ubuntu 24.04) is installed; guest +NVIDIA driver installation is in progress. Next: guest `nvidia.ko` probe +against the emulated vGPU, then `nvidia-smi`, then CUDA validation. + +## Recently solved bugs + +| Bug | Root cause | Fix | +|-----|-----------|-----| +| Module unload hang (`open_iov_vfio` stuck "Unloading") | Legacy `nv_vgpu_gsp_init()` booting physical GSP firmware, conflicting with host nvidia.ko; unload blocked in `cancel_work_sync` | Skip legacy GSP boot, use `gsp_emu_init()`; thread-stop via completion with 5s timeout; `cancel_work_sync` before `destroy_workqueue` | +| mdev remove kernel panic (`vfio_device_group_unregister+0xa`) | NULL deref on VFIO group unregister | Defensive cleanup ordering in `nv_vgpu_mdev_remove()` | +| mdev creation `EINVAL` (-22) | Double IOMMU group handling conflicted with `vfio_register_emulated_iommu_dev()` | Let VFIO create its own group; IOMMUFD emulated callbacks wired | +| `Unknown symbol nvidia_vgpu_vfio_probe` | NVIDIA OpenRM built without `KBUILD_EXTRA_SYMBOLS` | Build order automated in `stack_manager.sh` (kernel → nvidia → userspace) | +| `vfio 1: No such file or directory` | VFIO device registration is NOT automatic for mdev | `vfio_register_emulated_iommu_dev()` in probe, unregister in remove (Intel kvmgt pattern) | +| Guest-visible PCI config space wrong | BAR types/offsets, subsystem ID offsets, MSI-X table size | Config space rewritten to match physical TU104 (BAR0 16MB 32-bit non-prefetch, BAR1 256MB 64-bit prefetch VRAM, BAR3 32MB; 32 MSI-X vectors; subsystem IDs at 0x2C/0x2E) | + +## Known open bugs / limitations + +- Guest `nvidia.ko` end-to-end probe not yet validated (in progress) +- Kernel test modules (`kernel/tests/`) need API updates to match final + driver headers (low priority) +- WPR callback registration lacks kernel eventfd support + (`TODO` in driver/wpr) +- Migration WPR state save/restore incomplete +- BAR `force_access` path not fully wired (unused-function warning) +- Kernel `SCHEDULE` emulation returns success without real work submission + (userspace emulation works) +- Async `NOTIFY` events not emulated in kernel (no eventfd/poll yet) +- `open-iov-vfio.ko` requires `nvidia.ko` loaded first (`MODULE_SOFTDEP`) +- Single GPU validated so far: TU104 (RTX 2080 SUPER Mobile), GSP 570.144, + kernel 6.8.0-139-generic +- Live migration protocol implemented (4-phase) but not yet validated +- Windows guest not yet validated (automation scaffolding present) + +## Reproducibility notes + +- NVIDIA base: `open-gpu-kernel-modules` tag **610.57.04** + patch series + in `patches/nvidia/` (re-brands version string to 570.144 to match the + GSP firmware ABI; adds the vGPU VFIO interface headers + symbol exports). +- GSP firmware: `nvidia/tu102/gsp/gsp-570.144.bin` from + [linux-firmware.git](https://github.com/NVIDIA/linux-firmware) + (pin `2237b99e`), `.fwimage` section extracted with `objcopy`, + MD5 `13ce12a392a31eeb79a2e6ebd3a4a93a`. `install.sh` does this. +- Firmware blobs, decompiled output, NVIDIA tree, VM images, Ghidra DBs and + local VM wrappers are intentionally **not** tracked in git. +- Fresh-system bring-up: `./install.sh` (see README.md). + +## Hardware/software reference environment + +- Host GPU: NVIDIA RTX 2080 SUPER Mobile (TU104, PCI `10de:1ed3`) +- Host OS: Ubuntu, kernel 6.8.0-139-generic +- Guest OS: Ubuntu 24.04.5 LTS (QEMU/KVM, q35, OVMF) +- GSP firmware: 570.144 (RV64IMAC, base `0x04000000`) |
