1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
|
# Install and First Run
> Goal: from a fresh Ubuntu 22.04/24.04 host with a supported NVIDIA GPU to a
> guest VM running stock `nvidia.ko` and `nvidia-smi`, in about half an hour.
## 1. Prerequisites
**Host**
* Ubuntu 22.04 or 24.04, x86_64 (aarch64 is a roadmap target;
see [`../06-portability/hardware-software-matrix.md`](../06-portability/hardware-software-matrix.md)).
* An NVIDIA GPU supported by the current emulation profile (Turing TU104 is the
reference; see the matrix document for the current support list).
* A recent NVIDIA **host** driver installed and working (`nvidia-smi` works on
the host). This is *not* modified by Open-IOV.
* IOMMU enabled (`intel_iommu=on` / `amd_iommu=on`), hugepages configured,
kernel headers for your running kernel.
* `qemu-system-x86_64`, `libvirt` optional. KVM enabled.
* Root (or sudo) for module load and mdev creation.
**Guest**
* Ubuntu 24.04 server ISO (the reference guest), or any Linux with the matching
stock NVIDIA driver available. Windows guests are a roadmap target.
* The guest needs no Open-IOV component at all — that is the point.
## 2. Get the code and build
```bash
git clone https://git.vgpu.io/open-iov.git
cd open-iov
./install.sh deps # build/runtime dependencies
./install.sh nvidia # optional: NVIDIA open kernel modules (submodule) + patches
./install.sh firmware # GSP firmware image (verified)
./install.sh build # kernel module + userspace
```
`install.sh` performs the same steps as the manual flow below and is safe to
re-run. To see what any step does, read `install.sh` — it is intentionally
short and comments the order.
### Manual build (equivalent, and what developers use day to day)
```bash
# 1. Kernel module first: it generates Module.symvers used by the next step.
make kernel
# 2. Optional: NVIDIA OpenRM modules (needed only for native vGPU-management
# workflows; see section 3). They must see our symbols:
# KBUILD_EXTRA_SYMBOLS=<repo>/kernel/Module.symvers
./scripts/build_nvidia_openrm.sh
# 3. Userspace library + examples
make userspace
```
**Build-order matters.** The NVIDIA modules resolve symbols that
`open-iov-vfio.ko` exports; building them in the wrong order produces
`Unknown symbol` failures at load time. See
[`../03-architecture/06-host-module-and-hybrid-model.md`](../03-architecture/06-host-module-and-hybrid-model.md).
## 3. Load the stack and create a virtual GPU
```bash
./scripts/stack_manager.sh status # check what is currently loaded
./scripts/stack_manager.sh load-stack # vfio/mdev prereqs -> open-iov -> nvidia (optional)
./scripts/stack_manager.sh mdev-create # create a vGPU mediated device
./scripts/stack_manager.sh mdev-list # inspect /sys/bus/mdev/devices/
```
**The `nvidia` (OpenRM) stage is optional.** It is needed only when the host
must run NVIDIA's OpenRM modules built against the symbols `open-iov-vfio.ko`
exports — that is, the native vGPU-management/RM-callback workflows. The
QEMU/emulated path uses the distribution driver that already manages the
physical GPU; `load-stack` skips the OpenRM submodule when a distribution
driver is present, and the build steps for it (`./install.sh nvidia`,
`./scripts/build_nvidia_openrm.sh`) can be skipped. When the OpenRM path *is*
used, the build and load order in
[`../03-architecture/06-host-module-and-hybrid-model.md`](../03-architecture/06-host-module-and-hybrid-model.md)
§7 applies.
The mdev device is the virtual GPU. Everything after this point treats it like
any other VFIO device.
## 4. Launch a reference guest
```bash
./scripts/stack_manager.sh qemu # reference raw-QEMU guest with the mdev
```
Or integrate with your own stack (libvirt, Xen, ...): attach the mdev
device to the guest exactly as you would any VFIO device:
```xml
<hostdev mode='subsystem' type='mdev' model='vfio-pci'>
<source>
<address uuid='<mdev-uuid>'/>
</source>
</hostdev>
```
**Important:** for correct PCIe link capabilities in the guest, attach the
device behind a root port rather than on the root bus (QEMU rewrites the
device's PCIe capability when it detects a root-complex-integrated endpoint).
This is documented with the actual register values in
[`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md).
## 5. Install the stock driver in the guest
Inside the guest (the reference guest has SSH and a guest agent configured):
```bash
sudo apt update
sudo apt install -y linux-headers-generic build-essential
# Install the stock NVIDIA driver matching the emulation profile.
# The validated combination is the stock 570.144 Linux runfile (same driver
# version as the emulation target). The distro 570.x package can install a
# different 570 build; if you use it, keep the version notes from the
# hardware/software matrix at hand.
sudo sh ./NVIDIA-Linux-x86_64-570.144.run
sudo reboot
```
Then verify:
```bash
nvidia-smi # must list the virtual GPU
lsmod | grep -c nvidia
```
If `nvidia.ko` fails to probe, capture the four standard artifacts described in
[`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md)
(host dmesg, guest dmesg, guest `lspci -vvv`, guest module parameters) and file
an issue.
## 6. First CUDA check
```bash
# in the guest
git clone https://github.com/NVIDIA/cuda-samples.git
cd cuda-samples/Samples/1_Utilities/deviceQuery && make && ./deviceQuery
```
**Expectation today:** driver probe, `nvidia-smi`, device enumeration, and the
full CUDA driver-API smoke (`ALL OK` 64/64, copies matched) work.
Engine execution is the remaining gap: CE-class copies are forwarded to the
physical copy engine (default off), while CUDA kernel execution is blocked on
GR engine-context validity. Before drawing conclusions from a CUDA failure,
read the honest status in
[`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md)
and the debugger's guide [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md).
## 7. Day-to-day operations
```bash
./scripts/stack_manager.sh up # build all + load + create mdev
./scripts/stack_manager.sh status # modules, mdev, vfio, firmware
./scripts/stack_manager.sh test # host test suite
./scripts/stack_manager.sh unload # correct-order teardown
./scripts/stack_manager.sh cleanup-crash
```
Load order is: prerequisites → kernel module → (optional) NVIDIA OpenRM →
firmware. Unload is the reverse. The manager enforces this; do not `rmmod`
by hand while a VM holds the mdev.
## 8. Watching the emulation work
Every meaningful decision the emulation makes is visible in `dmesg` with the
prefix **`gsp-resman-vfio:`** (module logs) and the per-RPC traces. Handy:
```bash
sudo dmesg -w | grep -iE 'gsp-resman-vfio|gsp_|rpc' # live
sudo dmesg -T | grep -iE 'gsp-resman-vfio' | tail -50 # recent
```
Developer debug knobs (module parameters) are documented in
[`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md).
## 9. Troubleshooting index
| Symptom | Start here |
|---|---|
| Module will not load | `dmesg`, `modinfo open-iov-vfio.ko`, stale module state → `cleanup-crash` |
| No mdev type for your GPU | [hardware matrix](../06-portability/hardware-software-matrix.md) |
| Guest sees wrong PCIe link speed/width | root-port attachment + config-space mirror (debugging doc) |
| `nvidia.ko` probe fails in guest | guest dmesg + `lspci -vvv` + [driver lifecycle field guide](../03-architecture/07-driver-lifecycle-field-guide.md) |
| `nvidia-smi` works, CUDA kernel execution does not | expected today: the driver-API smoke passes; CE copies are forwarded (default off) and GR execution is blocked on engine-context validity; see [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) and [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md) |
| VM boot fails or the guest hangs | `scripts/stack_manager.sh cleanup-crash`, then reboot |
## 10. Uninstall
```bash
./scripts/stack_manager.sh unload
sudo rmmod open_iov_vfio nvidia nvidia_modeset nvidia_uvm # if present
```
The project never modifies your host driver's files; removing the Open-IOV
module and its mdev devices returns the host to its previous state.
|
