1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
|
# Install and First Run
> **What this page is.** From a fresh Ubuntu host with a supported NVIDIA GPU
> to a guest VM running a stock NVIDIA driver and `nvidia-smi`, in about half
> an hour. Condensed from the maintained guide
> [`../docs/02-getting-started/01-install-and-first-run.md`](../docs/02-getting-started/01-install-and-first-run.md);
> the docs tree wins on disagreement.
Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` verified path;
`[target]` roadmap item.
## 1. Prerequisites
**Host**
* Ubuntu 22.04 or 24.04, x86_64 (aarch64 is a `[target]`);
* an NVIDIA GPU covered by the current emulation profile (Turing TU104 is the
reference — see [`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md));
* a working **host** NVIDIA driver (`nvidia-smi` runs on the host; Open-IOV
does not modify it);
* IOMMU enabled (`intel_iommu=on` / `amd_iommu=on`), hugepages configured,
kernel headers for the running kernel;
* `qemu-system-x86_64` with KVM; libvirt optional;
* root (or sudo) for module load and mdev creation.
**Guest**
* Ubuntu 24.04 server (the reference guest), or any Linux with the matching
stock NVIDIA driver available;
* **no Open-IOV component at all** — that is the point ([`System-Overview.md`](System-Overview.md)).
## 2. Get the code and build
```bash
git clone https://git.vgpu.io/open-iov.git
cd open-iov
./install.sh deps # build/runtime dependencies
./install.sh nvidia # optional: NVIDIA OpenRM modules + build glue
./install.sh firmware # GSP firmware image (verified)
./install.sh build # kernel module + userspace
```
`install.sh` is safe to re-run. The manual equivalent, and the order that
matters:
```bash
make kernel # first: generates Module.symvers
./scripts/build_nvidia_openrm.sh # optional (see below)
make userspace
```
**Build order matters.** NVIDIA's OpenRM modules resolve symbols that
`vgsp.ko` exports; building them in the wrong order produces `Unknown
symbol` failures at load time.
**The OpenRM stage is optional.** It is needed only for the native
vGPU-management workflows in which NVIDIA's modules are built against the
symbols `vgsp.ko` exports. The QEMU/emulated path uses the distribution
driver that already manages the physical GPU; `load-stack` skips the OpenRM
submodule when a distribution driver is present.
## 3. Load the stack and create a virtual GPU
```bash
./scripts/stack_manager.sh status # what is currently loaded
./scripts/stack_manager.sh load-stack # vfio/mdev prereqs -> vgsp.ko -> (optional) nvidia
./scripts/stack_manager.sh mdev-create # create a vGPU mediated device
./scripts/stack_manager.sh mdev-list # inspect /sys/bus/mdev/devices/
```
The mdev device *is* the virtual GPU; everything after this treats it like any
other VFIO device.
## 4. Launch a reference guest
```bash
./scripts/stack_manager.sh qemu # reference raw-QEMU guest with the mdev
```
To integrate with your own stack (libvirt, Xen, ...), attach the mdev exactly
as any VFIO device:
```xml
<hostdev mode='subsystem' type='mdev' model='vfio-pci'>
<source>
<address uuid='<mdev-uuid>'/>
</source>
</hostdev>
```
**Attach behind a root port, not on the root bus.** QEMU rewrites a
device's PCIe capability when it detects a root-complex-integrated endpoint,
which changes the link capabilities the guest driver sees. The reference
script does this for you.
## 5. Install the stock driver in the guest
Inside the guest:
```bash
sudo apt update
sudo apt install -y linux-headers-generic build-essential
# The validated combination is the stock 570.144 Linux runfile (the same
# driver version as the emulation target).
sudo sh ./NVIDIA-Linux-x86_64-570.144.run
sudo reboot
```
Verify:
```bash
nvidia-smi # must list the virtual GPU
lsmod | grep -c nvidia
```
If `nvidia.ko` fails to probe, capture host dmesg, guest dmesg, guest
`lspci -vvv` and the guest module parameters, and file an issue.
## 6. First CUDA check
```bash
# in the guest
git clone https://github.com/NVIDIA/cuda-samples.git
cd cuda-samples/Samples/1_Utilities/deviceQuery && make && ./deviceQuery
```
**Expectation today** (`[live]` unless noted): driver probe, `nvidia-smi`,
device enumeration and the full CUDA **driver-API** smoke complete end to end
(`ALL OK`, 64-byte payload round-trips). CUDA **kernel execution** is the
current frontier: a guest launch is forwarded and reaches the engine queue,
but engine execution is not yet observed (`[wall]`) — see
[`Project-Status.md`](Project-Status.md) and
[`Execution-Frontier.md`](Execution-Frontier.md) before drawing conclusions
from a kernel that does not run.
## 7. Day-to-day operations
```bash
./scripts/stack_manager.sh up # build all + load + create mdev
./scripts/stack_manager.sh status # modules, mdev, vfio, firmware
./scripts/stack_manager.sh test # host test suite
./scripts/stack_manager.sh unload # correct-order teardown
./scripts/stack_manager.sh cleanup-crash
```
Load order is prerequisites → kernel module → (optional) NVIDIA OpenRM →
firmware; unload is the reverse. Do not `rmmod` by hand while a VM holds the
mdev.
## 8. Watching the emulation work
The module logs with the prefix **`vgsp-<subsystem>:`** (for example
`vgsp-rpc:`, `vgsp-sema:`, `vgsp-work:`) plus per-RPC traces:
```bash
sudo dmesg -w | grep -i 'vgsp' # live
sudo dmesg -T | grep -i 'vgsp' | tail -50
```
Developer debug knobs (module parameters) are documented in the repository's
debugging workflow document.
## 9. Troubleshooting index
| Symptom | Start here |
|---|---|
| Module will not load | `dmesg`, `modinfo vgsp.ko`, stale module state → `cleanup-crash` |
| No mdev type for your GPU | [`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md) |
| Guest sees wrong PCIe link speed/width | attach behind a root port (section 4) |
| `nvidia.ko` probe fails in guest | guest dmesg + `lspci -vvv` + [`Driver-Lifecycle-Field-Guide.md`](Driver-Lifecycle-Field-Guide.md) |
| `nvidia-smi` works, a CUDA kernel does not run | expected today — the execution frontier, not your setup ([`Execution-Frontier.md`](Execution-Frontier.md)) |
| VM boot fails or the guest hangs | `scripts/stack_manager.sh cleanup-crash`, then reboot |
## 10. Uninstall
```bash
./scripts/stack_manager.sh unload
```
The project never modifies your host driver's files; removing the module and
its mdev devices returns the host to its previous state.
|
