From 2302fc09080bad51c73f293294e19fd4db464eb0 Mon Sep 17 00:00:00 2001 From: Open-IOV Date: Wed, 23 Sep 2026 05:44:08 -0400 Subject: Open-IOV: initial public release Open-IOV virtualizes NVIDIA GPUs so that unmodified guest drivers run with real GPU functionality - CUDA first, 3D and video to follow - on unmodified hosts, without a proprietary vGPU license and without patching any driver, firmware or hypervisor. Core principle: emulate the interface, pass through the work. kernel/ open-iov-vfio.ko - host VFIO mdev driver: virtual GPU device (PCI config space, BAR shadows, MSI-X), GSP boot emulation, GSP-RM message-queue/RPC emulation, memory model (GPU-VA to guest-PA discovery) and the semaphore-completion service that advances guest UVM progress. userspace/ libiov.so management API and examples. tools/ guest diagnostics, firmware analysis wrappers, RM probes. scripts/ stack manager, VM harness, build/load/test automation. config/ hardware/version profiles (the portability surface). docs/ documentation tree - start at docs/README.md. wiki/ long-form firmware and protocol deep dives. Status: initialization works end to end for a stock 570.144 Linux guest on Turing (TU104): driver probe, nvidia-smi, the full GSP-RM control battery, object creation and the early CUDA/UVM start-up path. GPU work forwarding - the step that makes CUDA kernels execute on real hardware - is the active work item; see docs/04-implementation/01-known-knowns-and-unknowns.md. License: BSD-3-Clause for the project; GPL-2.0 for kernel modules that link GPL-only kernel interfaces. See LICENSE and docs/09-reference/legal-and-licensing.md. --- .../02-getting-started/01-install-and-first-run.md | 180 +++++++++++++++++++++ 1 file changed, 180 insertions(+) create mode 100644 docs/02-getting-started/01-install-and-first-run.md (limited to 'docs/02-getting-started/01-install-and-first-run.md') diff --git a/docs/02-getting-started/01-install-and-first-run.md b/docs/02-getting-started/01-install-and-first-run.md new file mode 100644 index 0000000..e0e71cb --- /dev/null +++ b/docs/02-getting-started/01-install-and-first-run.md @@ -0,0 +1,180 @@ +# Install and First Run + +> Goal: from a fresh Ubuntu 22.04/24.04 host with a supported NVIDIA GPU to a +> guest VM running stock `nvidia.ko` and `nvidia-smi`, in about half an hour. + +## 1. Prerequisites + +**Host** + +* Ubuntu 22.04 or 24.04, x86_64 (aarch64 is a roadmap target; + see [`../06-portability/hardware-software-matrix.md`](../06-portability/hardware-software-matrix.md)). +* An NVIDIA GPU supported by the current emulation profile (Turing TU104 is the + reference; see the matrix document for the current support list). +* A recent NVIDIA **host** driver installed and working (`nvidia-smi` works on + the host). This is *not* modified by Open-IOV. +* IOMMU enabled (`intel_iommu=on` / `amd_iommu=on`), hugepages configured, + kernel headers for your running kernel. +* `qemu-system-x86_64`, `libvirt` optional. KVM enabled. +* Root (or sudo) for module load and mdev creation. + +**Guest** + +* Ubuntu 24.04 server ISO (the reference guest), or any Linux with the matching + stock NVIDIA driver available. Windows guests are a roadmap target. +* The guest needs no Open-IOV component at all — that is the point. + +## 2. Get the code and build + +```bash +git clone https://git.vgpu.io/open-iov.git +cd open-iov +./install.sh deps # build/runtime dependencies +./install.sh nvidia # NVIDIA open kernel modules (submodule) + patches +./install.sh firmware # GSP firmware image (verified) +./install.sh build # kernel module + userspace +``` + +`install.sh` performs the same steps as the manual flow below and is safe to +re-run. To see what any step does, read `install.sh` — it is intentionally +short and comments the order. + +### Manual build (equivalent, and what developers use day to day) + +```bash +# 1. Kernel module first: it generates Module.symvers used by the next step. +make kernel + +# 2. NVIDIA OpenRM modules (must see our symbols): +# KBUILD_EXTRA_SYMBOLS=/kernel/Module.symvers +./scripts/build_nvidia_openrm.sh + +# 3. Userspace library + examples +make userspace +``` + +**Build-order matters.** The NVIDIA modules resolve symbols that +`open-iov-vfio.ko` exports; building them in the wrong order produces +`Unknown symbol` failures at load time. See +[`../03-architecture/06-host-module-and-hybrid-model.md`](../03-architecture/06-host-module-and-hybrid-model.md). + +## 3. Load the stack and create a virtual GPU + +```bash +./scripts/stack_manager.sh status # check what is currently loaded +./scripts/stack_manager.sh load-stack # vfio/mdev prereqs -> open-iov -> nvidia (if used) +./scripts/stack_manager.sh mdev-create # create a vGPU mediated device +./scripts/stack_manager.sh mdev-list # inspect /sys/bus/mdev/devices/ +``` + +The mdev device is the virtual GPU. Everything after this point treats it like +any other VFIO device. + +## 4. Launch a reference guest + +```bash +./scripts/stack_manager.sh qemu # reference raw-QEMU guest with the mdev +``` + +Or integrate with your own stack (libvirt, ``, Xen, ...): attach the mdev +device to the guest exactly as you would any VFIO device: + +```xml + + +
+ + +``` + +**Important:** for correct PCIe link capabilities in the guest, attach the +device behind a root port rather than on the root bus (QEMU rewrites the +device's PCIe capability when it detects a root-complex-integrated endpoint). +This is documented with the actual register values in +[`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md). + +## 5. Install the stock driver in the guest + +Inside the guest (the reference guest has SSH and a guest agent configured): + +```bash +sudo apt update +sudo apt install -y linux-headers-generic build-essential +# Install the stock NVIDIA driver matching the firmware/driver profile +sudo apt install -y nvidia-driver-570-open +sudo reboot +``` + +Then verify: + +```bash +nvidia-smi # must list the virtual GPU +lsmod | grep -c nvidia +``` + +If `nvidia.ko` fails to probe, capture the four standard artifacts described in +[`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md) +(host dmesg, guest dmesg, guest `lspci -vvv`, guest module parameters) and file +an issue. + +## 6. First CUDA check + +```bash +# in the guest +git clone https://github.com/NVIDIA/cuda-samples.git +cd cuda-samples/Samples/1_Utilities/deviceQuery && make && ./deviceQuery +``` + +**Expectation today:** driver probe, `nvidia-smi`, device enumeration, and the +early CUDA start-up path work; full kernel execution is the active work item. +Read the honest status in +[`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) +before drawing conclusions from a CUDA failure. + +## 7. Day-to-day operations + +```bash +./scripts/stack_manager.sh up # build all + load + create mdev +./scripts/stack_manager.sh status # modules, mdev, vfio, firmware +./scripts/stack_manager.sh test # host test suite +./scripts/stack_manager.sh unload # correct-order teardown +./scripts/stack_manager.sh cleanup-crash +``` + +Load order is: prerequisites → kernel module → (optional) NVIDIA OpenRM → +firmware. Unload is the reverse. The manager enforces this; do not `rmmod` +by hand while a VM holds the mdev. + +## 8. Watching the emulation work + +Every meaningful decision the emulation makes is visible in `dmesg` with the +prefix **`gsp-resman-vfio:`** (module logs) and the per-RPC traces. Handy: + +```bash +sudo dmesg -w | grep -iE 'gsp-resman-vfio|gsp_|rpc' # live +sudo dmesg -T | grep -iE 'gsp-resman-vfio' | tail -50 # recent +``` + +Developer debug knobs (module parameters) are documented in +[`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md). + +## 9. Troubleshooting index + +| Symptom | Start here | +|---|---| +| Module will not load | `dmesg`, `modinfo open-iov-vfio.ko`, stale module state → `cleanup-crash` | +| No mdev type for your GPU | [hardware matrix](../06-portability/hardware-software-matrix.md) | +| Guest sees wrong PCIe link speed/width | root-port attachment + config-space mirror (debugging doc) | +| `nvidia.ko` probe fails in guest | guest dmesg + `lspci -vvv` + handoff "driver lifecycle" | +| `nvidia-smi` works, CUDA does not | expected today; see the known-knowns doc | +| VM boot fails or the guest hangs | `scripts/stack_manager.sh cleanup-crash`, then reboot | + +## 10. Uninstall + +```bash +./scripts/stack_manager.sh unload +sudo rmmod open_iov_vfio nvidia nvidia_modeset nvidia_uvm # if present +``` + +The project never modifies your host driver's files; removing the Open-IOV +module and its mdev devices returns the host to its previous state. -- cgit v1.2.3