1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
|
# Open-IOV Project State
*Last updated: 2026-09-21*
## Mission
Run **unmodified NVIDIA guest drivers** in VMs with full GPU functionality
(CUDA, graphics, video, AI/ML) on consumer/datacenter GPUs, without
proprietary GRID licenses — via host-side GSP emulation in
`open-iov-vfio.ko`.
## Current milestone
| Phase | Status |
|-------|--------|
| GSP 570.144 firmware decompilation (132 files, 12 RPC types) | ✅ Complete |
| Refactor 1: GSP emulation engine (per-mdev thread, doorbell, shared mem, MSI-X) | ✅ Complete |
| Refactor 2: PCI config space + BAR + MSI-X emulation | ✅ Complete |
| Refactor 3: All 12 RPC types with emulation fallbacks | ✅ Complete |
| Refactor 4: Portable architecture (fw analyzer, dynamic dispatcher, arch/version layers, YAML configs) | ✅ Complete |
| Refactor 5: Guest driver integration (mdev creation, VFIO registration) | ✅ Unblocked, in test |
| Refactor 6-11: CUDA / graphics / benchmarks / migration / Windows / hardening | ⏳ Pending |
**Right now**: host stack (nvidia.ko 610.57.04-base + open-iov-vfio.ko) loads
cleanly, mdev device creates and removes cleanly, IOMMU group assignment
works, `/dev/vfio/<n>` appears. Guest VM (Ubuntu 24.04) is installed; guest
NVIDIA driver installation is in progress. Next: guest `nvidia.ko` probe
against the emulated vGPU, then `nvidia-smi`, then CUDA validation.
## Recently solved bugs
| Bug | Root cause | Fix |
|-----|-----------|-----|
| Module unload hang (`open_iov_vfio` stuck "Unloading") | Legacy `nv_vgpu_gsp_init()` booting physical GSP firmware, conflicting with host nvidia.ko; unload blocked in `cancel_work_sync` | Skip legacy GSP boot, use `gsp_emu_init()`; thread-stop via completion with 5s timeout; `cancel_work_sync` before `destroy_workqueue` |
| mdev remove kernel panic (`vfio_device_group_unregister+0xa`) | NULL deref on VFIO group unregister | Defensive cleanup ordering in `nv_vgpu_mdev_remove()` |
| mdev creation `EINVAL` (-22) | Double IOMMU group handling conflicted with `vfio_register_emulated_iommu_dev()` | Let VFIO create its own group; IOMMUFD emulated callbacks wired |
| `Unknown symbol nvidia_vgpu_vfio_probe` | NVIDIA OpenRM built without `KBUILD_EXTRA_SYMBOLS` | Build order automated in `stack_manager.sh` (kernel → nvidia → userspace) |
| `vfio 1: No such file or directory` | VFIO device registration is NOT automatic for mdev | `vfio_register_emulated_iommu_dev()` in probe, unregister in remove (Intel kvmgt pattern) |
| Guest-visible PCI config space wrong | BAR types/offsets, subsystem ID offsets, MSI-X table size | Config space rewritten to match physical TU104 (BAR0 16MB 32-bit non-prefetch, BAR1 256MB 64-bit prefetch VRAM, BAR3 32MB; 32 MSI-X vectors; subsystem IDs at 0x2C/0x2E) |
## Known open bugs / limitations
- Guest `nvidia.ko` end-to-end probe not yet validated (in progress)
- Kernel test modules (`kernel/tests/`) need API updates to match final
driver headers (low priority)
- WPR callback registration lacks kernel eventfd support
(`TODO` in driver/wpr)
- Migration WPR state save/restore incomplete
- BAR `force_access` path not fully wired (unused-function warning)
- Kernel `SCHEDULE` emulation returns success without real work submission
(userspace emulation works)
- Async `NOTIFY` events not emulated in kernel (no eventfd/poll yet)
- `open-iov-vfio.ko` requires `nvidia.ko` loaded first (`MODULE_SOFTDEP`)
- Single GPU validated so far: TU104 (RTX 2080 SUPER Mobile), GSP 570.144,
kernel 6.8.0-139-generic
- Live migration protocol implemented (4-phase) but not yet validated
- Windows guest not yet validated (automation scaffolding present)
## Reproducibility notes
- NVIDIA base: `open-gpu-kernel-modules` tag **610.57.04** + patch series
in `patches/nvidia/` (re-brands version string to 570.144 to match the
GSP firmware ABI; adds the vGPU VFIO interface headers + symbol exports).
- GSP firmware: `nvidia/tu102/gsp/gsp-570.144.bin` from
[linux-firmware.git](https://github.com/NVIDIA/linux-firmware)
(pin `2237b99e`), `.fwimage` section extracted with `objcopy`,
MD5 `13ce12a392a31eeb79a2e6ebd3a4a93a`. `install.sh` does this.
- Firmware blobs, decompiled output, NVIDIA tree, VM images, Ghidra DBs and
local VM wrappers are intentionally **not** tracked in git.
- Fresh-system bring-up: `./install.sh` (see README.md).
## Hardware/software reference environment
- Host GPU: NVIDIA RTX 2080 SUPER Mobile (TU104, PCI `10de:1ed3`)
- Host OS: Ubuntu, kernel 6.8.0-139-generic
- Guest OS: Ubuntu 24.04.5 LTS (QEMU/KVM, q35, OVMF)
- GSP firmware: 570.144 (RV64IMAC, base `0x04000000`)
|
