# Install and First Run

> **What this page is.** From a fresh Ubuntu host with a supported NVIDIA GPU
> to a guest VM running a stock NVIDIA driver and `nvidia-smi`, in about half
> an hour. Condensed from the maintained guide
> [`../docs/02-getting-started/01-install-and-first-run.md`](../docs/02-getting-started/01-install-and-first-run.md);
> the docs tree wins on disagreement.

Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` verified path;
`[target]` roadmap item.

## 1. Prerequisites

**Host**

* Ubuntu 22.04 or 24.04, x86_64 (aarch64 is a `[target]`);
* an NVIDIA GPU covered by the current emulation profile (Turing TU104 is the
  reference — see [`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md));
* a working **host** NVIDIA driver (`nvidia-smi` runs on the host; Open-IOV
  does not modify it);
* IOMMU enabled (`intel_iommu=on` / `amd_iommu=on`), hugepages configured,
  kernel headers for the running kernel;
* `qemu-system-x86_64` with KVM; libvirt optional;
* root (or sudo) for module load and mdev creation.

**Guest**

* Ubuntu 24.04 server (the reference guest), or any Linux with the matching
  stock NVIDIA driver available;
* **no Open-IOV component at all** — that is the point ([`System-Overview.md`](System-Overview.md)).

## 2. Get the code and build

```bash
git clone https://git.vgpu.io/open-iov.git
cd open-iov
./install.sh deps        # build/runtime dependencies
./install.sh nvidia      # optional: NVIDIA OpenRM modules + build glue
./install.sh firmware    # GSP firmware image (verified)
./install.sh build       # kernel module + userspace
```

`install.sh` is safe to re-run. The manual equivalent, and the order that
matters:

```bash
make kernel                      # first: generates Module.symvers
./scripts/build_nvidia_openrm.sh # optional (see below)
make userspace
```

**Build order matters.** NVIDIA's OpenRM modules resolve symbols that
`vgsp.ko` exports; building them in the wrong order produces `Unknown
symbol` failures at load time.

**The OpenRM stage is optional.** It is needed only for the native
vGPU-management workflows in which NVIDIA's modules are built against the
symbols `vgsp.ko` exports. The QEMU/emulated path uses the distribution
driver that already manages the physical GPU; `load-stack` skips the OpenRM
submodule when a distribution driver is present.

## 3. Load the stack and create a virtual GPU

```bash
./scripts/stack_manager.sh status        # what is currently loaded
./scripts/stack_manager.sh load-stack    # vfio/mdev prereqs -> vgsp.ko -> (optional) nvidia
./scripts/stack_manager.sh mdev-create   # create a vGPU mediated device
./scripts/stack_manager.sh mdev-list     # inspect /sys/bus/mdev/devices/
```

The mdev device *is* the virtual GPU; everything after this treats it like any
other VFIO device.

## 4. Launch a reference guest

```bash
./scripts/stack_manager.sh qemu          # reference raw-QEMU guest with the mdev
```

To integrate with your own stack (libvirt, Xen, ...), attach the mdev exactly
as any VFIO device:

```xml
<hostdev mode='subsystem' type='mdev' model='vfio-pci'>
  <source>
    <address uuid='<mdev-uuid>'/>
  </source>
</hostdev>
```

**Attach behind a root port, not on the root bus.** QEMU rewrites a
device's PCIe capability when it detects a root-complex-integrated endpoint,
which changes the link capabilities the guest driver sees. The reference
script does this for you.

## 5. Install the stock driver in the guest

Inside the guest:

```bash
sudo apt update
sudo apt install -y linux-headers-generic build-essential
# The validated combination is the stock 570.144 Linux runfile (the same
# driver version as the emulation target).
sudo sh ./NVIDIA-Linux-x86_64-570.144.run
sudo reboot
```

Verify:

```bash
nvidia-smi                # must list the virtual GPU
lsmod | grep -c nvidia
```

If `nvidia.ko` fails to probe, capture host dmesg, guest dmesg, guest
`lspci -vvv` and the guest module parameters, and file an issue.

## 6. First CUDA check

```bash
# in the guest
git clone https://github.com/NVIDIA/cuda-samples.git
cd cuda-samples/Samples/1_Utilities/deviceQuery && make && ./deviceQuery
```

**Expectation today** (`[live]` unless noted): driver probe, `nvidia-smi`,
device enumeration and the full CUDA **driver-API** smoke complete end to end
(`ALL OK`, 64-byte payload round-trips). CUDA **kernel execution** is the
current frontier: a guest launch is forwarded and reaches the engine queue,
but engine execution is not yet observed (`[wall]`) — see
[`Project-Status.md`](Project-Status.md) and
[`Execution-Frontier.md`](Execution-Frontier.md) before drawing conclusions
from a kernel that does not run.

## 7. Day-to-day operations

```bash
./scripts/stack_manager.sh up          # build all + load + create mdev
./scripts/stack_manager.sh status      # modules, mdev, vfio, firmware
./scripts/stack_manager.sh test        # host test suite
./scripts/stack_manager.sh unload      # correct-order teardown
./scripts/stack_manager.sh cleanup-crash
```

Load order is prerequisites → kernel module → (optional) NVIDIA OpenRM →
firmware; unload is the reverse. Do not `rmmod` by hand while a VM holds the
mdev.

## 8. Watching the emulation work

The module logs with the prefix **`vgsp-<subsystem>:`** (for example
`vgsp-rpc:`, `vgsp-sema:`, `vgsp-work:`) plus per-RPC traces:

```bash
sudo dmesg -w | grep -i 'vgsp'          # live
sudo dmesg -T | grep -i 'vgsp' | tail -50
```

Developer debug knobs (module parameters) are documented in the repository's
debugging workflow document.

## 9. Troubleshooting index

| Symptom | Start here |
|---|---|
| Module will not load | `dmesg`, `modinfo vgsp.ko`, stale module state → `cleanup-crash` |
| No mdev type for your GPU | [`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md) |
| Guest sees wrong PCIe link speed/width | attach behind a root port (section 4) |
| `nvidia.ko` probe fails in guest | guest dmesg + `lspci -vvv` + [`Driver-Lifecycle-Field-Guide.md`](Driver-Lifecycle-Field-Guide.md) |
| `nvidia-smi` works, a CUDA kernel does not run | expected today — the execution frontier, not your setup ([`Execution-Frontier.md`](Execution-Frontier.md)) |
| VM boot fails or the guest hangs | `scripts/stack_manager.sh cleanup-crash`, then reboot |

## 10. Uninstall

```bash
./scripts/stack_manager.sh unload
```

The project never modifies your host driver's files; removing the module and
its mdev devices returns the host to its previous state.