1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
|
# Open-IOV
**Open-IOV** is an open source reimplementation of NVIDIA vGPU (Virtual GPU)
technology. It lets an **unmodified NVIDIA guest driver** run inside a VM with
full GPU functionality, without a proprietary GRID license, by emulating the
GSP (GPU System Processor) RPC interface on the host.
- **open-iov-vfio.ko** — host VFIO mediated-device (mdev) kernel driver with a
host-side GSP emulation engine (all 12 RPC types: ALLOC, FREE, MAP, UNMAP,
PIN, UNPIN, SCHEDULE, YIELD, NOTIFY, BAR_TRAP, MIGRATE, DEBUG)
- **libiov.so** — userspace management library (`libvgpu.so` compat symlinks)
- **GSP firmware analysis** — complete decompilation of NVIDIA GSP 570.144
(RV64IMAC, TU102/TU104): RPC dispatchers, jump tables, WPR, BAR trap,
migration protocol, entry/boot sequence
The guest VM runs the **stock, unmodified** `nvidia.ko`. All Open-IOV code
lives on the host.
## Current state
See **[STATE.md](STATE.md)** for the live project state: what works, current
limitations, recently fixed bugs, and known open issues.
## Quick start (fresh Ubuntu 22.04/24.04 + supported NVIDIA GPU)
```bash
git clone https://git.vgpu.io/open-iov.git
cd open-iov
./install.sh
```
`install.sh` reproduces the entire setup:
| Stage | What it does |
|-----------|--------------|
| `deps` | apt build/runtime dependencies |
| `nvidia` | clone NVIDIA open-gpu-kernel-modules @ `610.57.04`, apply `patches/nvidia/` |
| `firmware`| fetch GSP 570.144 from linux-firmware.git, extract `.fwimage`, verify MD5 |
| `build` | `open-iov-vfio.ko` → `nvidia.ko` (KBUILD_EXTRA_SYMBOLS) → userspace |
| `system` | VFIO/mdev module persistence, hugepages, IOMMU/blacklist checks |
| `load` | load the driver stack (via `scripts/stack_manager.sh`) |
| `mdev` | create a vGPU mdev device |
Run individual stages with `./install.sh deps nvidia`, or skip the root-requiring
tail with `./install.sh --no-load`.
### Day-to-day operations
```bash
scripts/stack_manager.sh up # build all + load stack + create mdev
scripts/stack_manager.sh status # modules, mdevs, vfio, firmware, dmesg
scripts/stack_manager.sh qemu # launch reference raw-QEMU guest
scripts/stack_manager.sh unload # correct-order teardown
scripts/stack_manager.sh test # test suite
```
(`stack_manager.sh` auto-detects its location; override with `OPEN_IOV_ROOT`.)
## Using your own VM stack (libvirt, , ...)
Open-IOV is **hypervisor-management neutral**. It provides the vGPU itself
(an mdev device + VFIO group); how you attach it to a VM is up to you:
- **raw QEMU**: `scripts/launch_qemu_vgpu.sh` (reference launcher), or
`-device vfio-pci,sysfsdev=/sys/bus/mdev/devices/<UUID>`
- **libvirt**: `<hostdev mode='subsystem' type='mdev' model='vfio-pci'>`
with the UUID printed by `scripts/create_mdev.sh`
- **anything else** that can pass a VFIO mdev to QEMU
No Open-IOV component depends on a specific wrapper.
## Repository layout
```
kernel/ open-iov-vfio.ko host driver (mdev, VFIO, GSP emulation,
firmware analyzer, dynamic RPC dispatcher, arch/version layers)
userspace/ libiov.so management library, examples, tests
driver/ legacy guest driver (being removed in the host-only refactor)
firmware/ GSP analysis: scripts, docs, address maps (blobs NOT in git -
fetched by install.sh from linux-firmware.git)
patches/nvidia/ patch series applied to NVIDIA open-gpu-kernel-modules
(base tag 610.57.04; re-brands version to 570.144 to match
the GSP firmware ABI)
config/ declarative hardware (turing/ampere/blackwell) and
driver-version (550-580) YAML configs
scripts/ stack_manager.sh + helpers (wrapper-neutral)
tools/ debugging/profiling tools, Ghidra analysis scripts
docs/ architecture and implementation docs
wiki/ technical wiki (GSP firmware analysis, RPC format, ...)
```
**Not tracked in git** (by design): GSP firmware blobs, decompiled firmware
output, the NVIDIA kernel tree (cloned+patched by `install.sh`), VM disk
images/ISOs, Ghidra project databases, and any local VM-wrapper stacks.
## Requirements
- Linux kernel 6.8+ (tested on 6.8.0-139-generic), IOMMU enabled
(`intel_iommu=on` / `amd_iommu=on`)
- NVIDIA GPU: TU102/TU104 validated (RTX 2080/2080 Super); Ampere/Blackwell
config scaffolding in place
- No nvidia module blacklist on the kernel cmdline
## Documentation
- [STATE.md](STATE.md) — current state, limitations, bug tracker
- [Portable architecture](docs/PORTABLE_ARCHITECTURE.md)
- [Host-only GSP emulation design](docs/refactored-architecture.md)
- [Implementation TODO](docs/refactored-implementation-to-do.md)
- [Firmware analysis](wiki/GSP-570.144W-Firmware-Analysis.md) —
[RPC format](wiki/RPC-Message-Format.md) —
[BAR emulation](wiki/BAR-Emulation-Specification.md) —
[Migration protocol](wiki/Migration-Protocol.md) —
[Address reference](wiki/Address-Reference.md)
## License
GPL v2 (kernel components), LGPL (userspace library).
NVIDIA firmware remains property of NVIDIA and is not distributed here.
|
