# Install and First Run

> Goal: from a fresh Ubuntu 22.04/24.04 host with a supported NVIDIA GPU to a
> guest VM running stock `nvidia.ko` and `nvidia-smi`, in about half an hour.

## 1. Prerequisites

**Host**

* Ubuntu 22.04 or 24.04, x86_64 (aarch64 is a roadmap target;
  see [`../06-portability/hardware-software-matrix.md`](../06-portability/hardware-software-matrix.md)).
* An NVIDIA GPU supported by the current emulation profile (Turing TU104 is the
  reference; see the matrix document for the current support list).
* A recent NVIDIA **host** driver installed and working (`nvidia-smi` works on
  the host). This is *not* modified by Open-IOV.
* IOMMU enabled (`intel_iommu=on` / `amd_iommu=on`), hugepages configured,
  kernel headers for your running kernel.
* `qemu-system-x86_64`, `libvirt` optional. KVM enabled.
* Root (or sudo) for module load and mdev creation.

**Guest**

* Ubuntu 24.04 server ISO (the reference guest), or any Linux with the matching
  stock NVIDIA driver available. Windows guests are a roadmap target.
* The guest needs no Open-IOV component at all — that is the point.

## 2. Get the code and build

```bash
git clone https://git.vgpu.io/open-iov.git
cd open-iov
./install.sh deps        # build/runtime dependencies
./install.sh nvidia      # NVIDIA open kernel modules (submodule) + patches
./install.sh firmware    # GSP firmware image (verified)
./install.sh build       # kernel module + userspace
```

`install.sh` performs the same steps as the manual flow below and is safe to
re-run. To see what any step does, read `install.sh` — it is intentionally
short and comments the order.

### Manual build (equivalent, and what developers use day to day)

```bash
# 1. Kernel module first: it generates Module.symvers used by the next step.
make kernel

# 2. NVIDIA OpenRM modules (must see our symbols):
#    KBUILD_EXTRA_SYMBOLS=<repo>/kernel/Module.symvers
./scripts/build_nvidia_openrm.sh

# 3. Userspace library + examples
make userspace
```

**Build-order matters.** The NVIDIA modules resolve symbols that
`open-iov-vfio.ko` exports; building them in the wrong order produces
`Unknown symbol` failures at load time. See
[`../03-architecture/06-host-module-and-hybrid-model.md`](../03-architecture/06-host-module-and-hybrid-model.md).

## 3. Load the stack and create a virtual GPU

```bash
./scripts/stack_manager.sh status        # check what is currently loaded
./scripts/stack_manager.sh load-stack    # vfio/mdev prereqs -> open-iov -> nvidia (if used)
./scripts/stack_manager.sh mdev-create   # create a vGPU mediated device
./scripts/stack_manager.sh mdev-list     # inspect /sys/bus/mdev/devices/
```

The mdev device is the virtual GPU. Everything after this point treats it like
any other VFIO device.

## 4. Launch a reference guest

```bash
./scripts/stack_manager.sh qemu          # reference raw-QEMU guest with the mdev
```

Or integrate with your own stack (libvirt, ``, Xen, ...): attach the mdev
device to the guest exactly as you would any VFIO device:

```xml
<hostdev mode='subsystem' type='mdev' model='vfio-pci'>
  <source>
    <address uuid='<mdev-uuid>'/>
  </source>
</hostdev>
```

**Important:** for correct PCIe link capabilities in the guest, attach the
device behind a root port rather than on the root bus (QEMU rewrites the
device's PCIe capability when it detects a root-complex-integrated endpoint).
This is documented with the actual register values in
[`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md).

## 5. Install the stock driver in the guest

Inside the guest (the reference guest has SSH and a guest agent configured):

```bash
sudo apt update
sudo apt install -y linux-headers-generic build-essential
# Install the stock NVIDIA driver matching the firmware/driver profile
sudo apt install -y nvidia-driver-570-open
sudo reboot
```

Then verify:

```bash
nvidia-smi            # must list the virtual GPU
lsmod | grep -c nvidia
```

If `nvidia.ko` fails to probe, capture the four standard artifacts described in
[`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md)
(host dmesg, guest dmesg, guest `lspci -vvv`, guest module parameters) and file
an issue.

## 6. First CUDA check

```bash
# in the guest
git clone https://github.com/NVIDIA/cuda-samples.git
cd cuda-samples/Samples/1_Utilities/deviceQuery && make && ./deviceQuery
```

**Expectation today:** driver probe, `nvidia-smi`, device enumeration, and the
early CUDA start-up path work; full kernel execution is the active work item.
Read the honest status in
[`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md)
before drawing conclusions from a CUDA failure.

## 7. Day-to-day operations

```bash
./scripts/stack_manager.sh up          # build all + load + create mdev
./scripts/stack_manager.sh status      # modules, mdev, vfio, firmware
./scripts/stack_manager.sh test        # host test suite
./scripts/stack_manager.sh unload      # correct-order teardown
./scripts/stack_manager.sh cleanup-crash
```

Load order is: prerequisites → kernel module → (optional) NVIDIA OpenRM →
firmware. Unload is the reverse. The manager enforces this; do not `rmmod`
by hand while a VM holds the mdev.

## 8. Watching the emulation work

Every meaningful decision the emulation makes is visible in `dmesg` with the
prefix **`gsp-resman-vfio:`** (module logs) and the per-RPC traces. Handy:

```bash
sudo dmesg -w | grep -iE 'gsp-resman-vfio|gsp_|rpc'      # live
sudo dmesg -T | grep -iE 'gsp-resman-vfio' | tail -50    # recent
```

Developer debug knobs (module parameters) are documented in
[`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md).

## 9. Troubleshooting index

| Symptom | Start here |
|---|---|
| Module will not load | `dmesg`, `modinfo open-iov-vfio.ko`, stale module state → `cleanup-crash` |
| No mdev type for your GPU | [hardware matrix](../06-portability/hardware-software-matrix.md) |
| Guest sees wrong PCIe link speed/width | root-port attachment + config-space mirror (debugging doc) |
| `nvidia.ko` probe fails in guest | guest dmesg + `lspci -vvv` + handoff "driver lifecycle" |
| `nvidia-smi` works, CUDA does not | expected today; see the known-knowns doc |
| VM boot fails or the guest hangs | `scripts/stack_manager.sh cleanup-crash`, then reboot |

## 10. Uninstall

```bash
./scripts/stack_manager.sh unload
sudo rmmod open_iov_vfio nvidia nvidia_modeset nvidia_uvm  # if present
```

The project never modifies your host driver's files; removing the Open-IOV
module and its mdev devices returns the host to its previous state.