1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
|
# Open-IOV
**Open-IOV** is an open source reimplementation of NVIDIA vGPU (Virtual GPU)
technology. It lets an **unmodified NVIDIA guest driver** run inside a VM with
full GPU functionality, without a proprietary GRID license, by emulating the
GSP (GPU System Processor) RPC interface on the host.
- **open-iov-vfio.ko** — host VFIO mediated-device (mdev) kernel driver with a
host-side GSP emulation engine (all 12 RPC types: ALLOC, FREE, MAP, UNMAP,
PIN, UNPIN, SCHEDULE, YIELD, NOTIFY, BAR_TRAP, MIGRATE, DEBUG)
- **libiov.so** — userspace management library (`libvgpu.so` compat symlinks)
- **GSP firmware analysis** — complete decompilation of NVIDIA GSP 570.144
(RV64IMAC, TU102/TU104): RPC dispatchers, jump tables, WPR, BAR trap,
migration protocol, entry/boot sequence
The guest VM runs the **stock, unmodified** `nvidia.ko`. All Open-IOV code
lives on the host.
## Current state
See **[STATE.md](STATE.md)** for the live project state: what works, current
limitations, recently fixed bugs, and known open issues.
## Quick start (fresh Ubuntu 22.04/24.04 + supported NVIDIA GPU)
```bash
git clone https://git.vgpu.io/open-iov.git
cd open-iov
./install.sh
```
`install.sh` reproduces the entire setup:
| Stage | What it does |
|-----------|--------------|
| `deps` | apt build/runtime dependencies |
| `nvidia` | clone NVIDIA open-gpu-kernel-modules @ `610.57.04`, apply `patches/nvidia/` |
| `firmware`| fetch GSP 570.144 from linux-firmware.git, extract `.fwimage`, verify MD5 |
| `build` | `open-iov-vfio.ko` → `nvidia.ko` (KBUILD_EXTRA_SYMBOLS) → userspace |
| `system` | VFIO/mdev module persistence, hugepages, IOMMU/blacklist checks |
| `load` | load the driver stack (via `scripts/stack_manager.sh`) |
| `mdev` | create a vGPU mdev device |
Run individual stages with `./install.sh deps nvidia`, or skip the root-requiring
tail with `./install.sh --no-load`.
### Day-to-day operations
```bash
scripts/stack_manager.sh up # build all + load stack + create mdev
scripts/stack_manager.sh status # modules, mdevs, vfio, firmware, dmesg
scripts/stack_manager.sh qemu # launch reference raw-QEMU guest
scripts/stack_manager.sh unload # correct-order teardown
scripts/stack_manager.sh test # test suite
```
(`stack_manager.sh` auto-detects its location; override with `OPEN_IOV_ROOT`.)
## Using your own VM stack (libvirt, , ...)
Open-IOV is **hypervisor-management neutral**. It provides the vGPU itself
(an mdev device + VFIO group); how you attach it to a VM is up to you:
- **raw QEMU**: `scripts/launch_qemu_vgpu.sh` (reference launcher), or
`-device vfio-pci,sysfsdev=/sys/bus/mdev/devices/<UUID>`
- **libvirt**: `<hostdev mode='subsystem' type='mdev' model='vfio-pci'>`
with the UUID printed by `scripts/create_mdev.sh`
- **anything else** that can pass a VFIO mdev to QEMU
No Open-IOV component depends on a specific wrapper.
## Repository layout
```
kernel/ open-iov-vfio.ko host driver (mdev, VFIO, GSP emulation,
firmware analyzer, dynamic RPC dispatcher, arch/version layers)
userspace/ libiov.so management library, examples, tests
legacy-driver-guest/ historical guest-side driver, preserved for reference
only - NOT built (see its README)
firmware/ GSP analysis: scripts, docs, address maps (blobs NOT in git -
fetched by install.sh from linux-firmware.git)
patches/nvidia/ patch series applied to NVIDIA open-gpu-kernel-modules
(base tag 610.57.04; re-brands version to 570.144 to match
the GSP firmware ABI)
config/ declarative hardware (turing/ampere/blackwell) and
driver-version (550-580) YAML configs
scripts/ stack_manager.sh + helpers (wrapper-neutral)
tools/ debugging/profiling tools, Ghidra analysis scripts
docs/ architecture and implementation docs
wiki/ technical wiki (GSP firmware analysis, RPC format, ...)
```
**Not tracked in git** (by design): GSP firmware blobs, decompiled firmware
output, the NVIDIA kernel tree (cloned+patched by `install.sh`), VM disk
images/ISOs, Ghidra project databases, and any local VM-wrapper stacks.
## Requirements
- Linux kernel 6.8+ (tested on 6.8.0-139-generic), IOMMU enabled
(`intel_iommu=on` / `amd_iommu=on`)
- NVIDIA GPU: TU102/TU104 validated (RTX 2080/2080 Super); Ampere/Blackwell
config scaffolding in place
- No nvidia module blacklist on the kernel cmdline
## Documentation
Start here: **[Developer onboarding](docs/ONBOARDING.md)** — repo tour,
coding standards, first build & test.
- [STATE.md](STATE.md) — current state, limitations, bug tracker
- [GSP emulation engine architecture](docs/GSP_EMULATION_ARCHITECTURE.md) —
how the host-side emulator works (shadow BAR0, doorbell protocol, dispatch,
MSI-X injection)
- [Debugging guide](docs/DEBUGGING.md) — QEMU GDB stub, serial console,
failure taxonomy, bring-up validation checklist
- [Version upgrade guide](docs/VERSION_UPGRADE.md) — supporting a new NVIDIA
driver/GSP version end-to-end (Ghidra workflow)
- [Performance testing](docs/PERFORMANCE.md) — bare-metal A/B methodology
- [Portable architecture](docs/PORTABLE_ARCHITECTURE.md) — multi-version,
multi-hardware design
- [Host-only GSP emulation design](docs/refactored-architecture.md) —
original refactor plan
- [Implementation TODO](docs/refactored-implementation-to-do.md)
- [Development workflow reference](docs/DEVELOPMENT_WORKFLOW.md) — exhaustive
command reference
- Firmware canon: [analysis](wiki/GSP-570.144W-Firmware-Analysis.md) —
[RPC format](wiki/RPC-Message-Format.md) —
[BAR emulation](wiki/BAR-Emulation-Specification.md) —
[migration protocol](wiki/Migration-Protocol.md) —
[address reference](wiki/Address-Reference.md)
## License
GPL v2 (kernel components), LGPL (userspace library).
NVIDIA firmware remains property of NVIDIA and is not distributed here.
|
