diff options
Diffstat (limited to 'wiki/Install-and-First-Run.md')
| -rw-r--r-- | wiki/Install-and-First-Run.md | 181 |
1 files changed, 181 insertions, 0 deletions
diff --git a/wiki/Install-and-First-Run.md b/wiki/Install-and-First-Run.md new file mode 100644 index 0000000..601e8ea --- /dev/null +++ b/wiki/Install-and-First-Run.md @@ -0,0 +1,181 @@ +# Install and First Run + +> **What this page is.** From a fresh Ubuntu host with a supported NVIDIA GPU +> to a guest VM running a stock NVIDIA driver and `nvidia-smi`, in about half +> an hour. Condensed from the maintained guide +> [`../docs/02-getting-started/01-install-and-first-run.md`](../docs/02-getting-started/01-install-and-first-run.md); +> the docs tree wins on disagreement. + +Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` verified path; +`[target]` roadmap item. + +## 1. Prerequisites + +**Host** + +* Ubuntu 22.04 or 24.04, x86_64 (aarch64 is a `[target]`); +* an NVIDIA GPU covered by the current emulation profile (Turing TU104 is the + reference — see [`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md)); +* a working **host** NVIDIA driver (`nvidia-smi` runs on the host; Open-IOV + does not modify it); +* IOMMU enabled (`intel_iommu=on` / `amd_iommu=on`), hugepages configured, + kernel headers for the running kernel; +* `qemu-system-x86_64` with KVM; libvirt optional; +* root (or sudo) for module load and mdev creation. + +**Guest** + +* Ubuntu 24.04 server (the reference guest), or any Linux with the matching + stock NVIDIA driver available; +* **no Open-IOV component at all** — that is the point ([`System-Overview.md`](System-Overview.md)). + +## 2. Get the code and build + +```bash +git clone https://git.vgpu.io/open-iov.git +cd open-iov +./install.sh deps # build/runtime dependencies +./install.sh nvidia # optional: NVIDIA OpenRM modules + build glue +./install.sh firmware # GSP firmware image (verified) +./install.sh build # kernel module + userspace +``` + +`install.sh` is safe to re-run. The manual equivalent, and the order that +matters: + +```bash +make kernel # first: generates Module.symvers +./scripts/build_nvidia_openrm.sh # optional (see below) +make userspace +``` + +**Build order matters.** NVIDIA's OpenRM modules resolve symbols that +`vgsp.ko` exports; building them in the wrong order produces `Unknown +symbol` failures at load time. + +**The OpenRM stage is optional.** It is needed only for the native +vGPU-management workflows in which NVIDIA's modules are built against the +symbols `vgsp.ko` exports. The QEMU/emulated path uses the distribution +driver that already manages the physical GPU; `load-stack` skips the OpenRM +submodule when a distribution driver is present. + +## 3. Load the stack and create a virtual GPU + +```bash +./scripts/stack_manager.sh status # what is currently loaded +./scripts/stack_manager.sh load-stack # vfio/mdev prereqs -> vgsp.ko -> (optional) nvidia +./scripts/stack_manager.sh mdev-create # create a vGPU mediated device +./scripts/stack_manager.sh mdev-list # inspect /sys/bus/mdev/devices/ +``` + +The mdev device *is* the virtual GPU; everything after this treats it like any +other VFIO device. + +## 4. Launch a reference guest + +```bash +./scripts/stack_manager.sh qemu # reference raw-QEMU guest with the mdev +``` + +To integrate with your own stack (libvirt, Xen, ...), attach the mdev exactly +as any VFIO device: + +```xml +<hostdev mode='subsystem' type='mdev' model='vfio-pci'> + <source> + <address uuid='<mdev-uuid>'/> + </source> +</hostdev> +``` + +**Attach behind a root port, not on the root bus.** QEMU rewrites a +device's PCIe capability when it detects a root-complex-integrated endpoint, +which changes the link capabilities the guest driver sees. The reference +script does this for you. + +## 5. Install the stock driver in the guest + +Inside the guest: + +```bash +sudo apt update +sudo apt install -y linux-headers-generic build-essential +# The validated combination is the stock 570.144 Linux runfile (the same +# driver version as the emulation target). +sudo sh ./NVIDIA-Linux-x86_64-570.144.run +sudo reboot +``` + +Verify: + +```bash +nvidia-smi # must list the virtual GPU +lsmod | grep -c nvidia +``` + +If `nvidia.ko` fails to probe, capture host dmesg, guest dmesg, guest +`lspci -vvv` and the guest module parameters, and file an issue. + +## 6. First CUDA check + +```bash +# in the guest +git clone https://github.com/NVIDIA/cuda-samples.git +cd cuda-samples/Samples/1_Utilities/deviceQuery && make && ./deviceQuery +``` + +**Expectation today** (`[live]` unless noted): driver probe, `nvidia-smi`, +device enumeration and the full CUDA **driver-API** smoke complete end to end +(`ALL OK`, 64-byte payload round-trips). CUDA **kernel execution** is the +current frontier: a guest launch is forwarded and reaches the engine queue, +but engine execution is not yet observed (`[wall]`) — see +[`Project-Status.md`](Project-Status.md) and +[`Execution-Frontier.md`](Execution-Frontier.md) before drawing conclusions +from a kernel that does not run. + +## 7. Day-to-day operations + +```bash +./scripts/stack_manager.sh up # build all + load + create mdev +./scripts/stack_manager.sh status # modules, mdev, vfio, firmware +./scripts/stack_manager.sh test # host test suite +./scripts/stack_manager.sh unload # correct-order teardown +./scripts/stack_manager.sh cleanup-crash +``` + +Load order is prerequisites → kernel module → (optional) NVIDIA OpenRM → +firmware; unload is the reverse. Do not `rmmod` by hand while a VM holds the +mdev. + +## 8. Watching the emulation work + +The module logs with the prefix **`vgsp-<subsystem>:`** (for example +`vgsp-rpc:`, `vgsp-sema:`, `vgsp-work:`) plus per-RPC traces: + +```bash +sudo dmesg -w | grep -i 'vgsp' # live +sudo dmesg -T | grep -i 'vgsp' | tail -50 +``` + +Developer debug knobs (module parameters) are documented in the repository's +debugging workflow document. + +## 9. Troubleshooting index + +| Symptom | Start here | +|---|---| +| Module will not load | `dmesg`, `modinfo vgsp.ko`, stale module state → `cleanup-crash` | +| No mdev type for your GPU | [`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md) | +| Guest sees wrong PCIe link speed/width | attach behind a root port (section 4) | +| `nvidia.ko` probe fails in guest | guest dmesg + `lspci -vvv` + [`Driver-Lifecycle-Field-Guide.md`](Driver-Lifecycle-Field-Guide.md) | +| `nvidia-smi` works, a CUDA kernel does not run | expected today — the execution frontier, not your setup ([`Execution-Frontier.md`](Execution-Frontier.md)) | +| VM boot fails or the guest hangs | `scripts/stack_manager.sh cleanup-crash`, then reboot | + +## 10. Uninstall + +```bash +./scripts/stack_manager.sh unload +``` + +The project never modifies your host driver's files; removing the module and +its mdev devices returns the host to its previous state. |