diff options
Diffstat (limited to 'docs')
34 files changed, 7441 insertions, 0 deletions
diff --git a/docs/01-orientation/01-purpose-and-scope.md b/docs/01-orientation/01-purpose-and-scope.md new file mode 100644 index 0000000..eab63bc --- /dev/null +++ b/docs/01-orientation/01-purpose-and-scope.md @@ -0,0 +1,157 @@ +# Purpose and Scope + +> Read this first. It defines what Open-IOV is for, what it is not for, and how +> we will judge whether we are succeeding. + +## 1. Mission + +**Make NVIDIA GPUs fully virtualizable, by anyone, for any hypervisor, without +modifying NVIDIA software, firmware or hardware.** + +Concretely, Open-IOV builds an independent, maintained IP block that: + +* presents a virtual NVIDIA GPU to a guest operating system, +* lets the guest run the **stock NVIDIA driver** (Linux and Windows), +* supports **compute (CUDA)** first, then **3D graphics** and **video**, +* runs on ordinary NVIDIA GPUs already deployed in the field, +* isolates guests from each other and from the host, +* and does not require a license key, a proprietary host service, a patched + guest driver, or a specific hypervisor. + +## 2. What problem are we solving? + +Today there are four ways to use an NVIDIA GPU with virtual machines, and all +of them have a hard limitation: + +| Approach | Limitation | +|---|---| +| Discrete device assignment (VFIO passthrough) | One GPU per VM. Complex, fragile for laptops and consumer PCs. No sharing. | +| NVIDIA proprietary vGPU | Requires a license. Requires a vendor host stack. Driver-version locked. Windows/Linux only where the vendor ships builds. | +| Software emulation of a GPU (e.g. QEMU's virtual GPUs, virgl) | No real CUDA. No real 3D performance. Not usable for AI. | +| Cloud-only inference | Needs an external service. Unacceptable for secure edge, offline, or private data. | + +Open-IOV occupies the gap: **real GPU acceleration, multiple VMs, stock +drivers, no license.** + +## 3. Scope + +### In scope + +* **Host-side virtualization of NVIDIA GPUs** for Linux hosts via VFIO/mdev + (KVM now; Xen and others later), as a self-contained kernel module plus a + small userspace library. +* **Full guest-visible device emulation**: PCI configuration space, BARs + (MMIO register space, VRAM aperture, instance memory), MSI-X, doorbells, + power management stubs. +* **GSP and GSP-RM interface emulation**: firmware boot handshake, message + queues, RPC transport, resource-manager controls, object model, event + channels, and the pieces of the hosted resource manager the guest depends on. +* **A memory model** that connects guest GPU address spaces to real GPU memory + and to guest system memory, including page tables and fault handling. +* **Compute forwarding**: taking the work the guest submits (channels, push + buffers, runlists, compute commands) and executing it on the physical GPU in + a way that is correct, isolated and fast. +* **Graphics and video forwarding** (later phase, same architecture). +* **Guest support**: Linux (x86_64 and aarch64) and Windows (x86_64). +* **Host support**: Linux (x86_64 now, aarch64 later), KVM today, Xen/Argo/ + OpenXT/Qubes OS Air on the roadmap. +* **Hardware portability**: Turing (TU104) today; Ampere, Hopper, + Grace-Hopper, Grace-Blackwell and GB10 targets; a configuration-driven + adaptation layer for new GPUs. + +### Out of scope (non-goals) + +* **Replacing bare-metal inference serving.** vLLM's continuous batching on + bare-metal hosts is how the industry scales multi-tenant inference in + datacenters, and it is better at that than any virtualization layer will be. + Open-IOV is for the cases where compute must live *inside* a guest. +* **Forking or patching the NVIDIA driver**, in guest or host. +* **Forking or patching GPU firmware.** +* **Forking or patching hypervisors** (QEMU, KVM, Xen, Argo, OpenXT, ...) to + make the device work. +* **Anti-cheat / DRM circumvention**, cloud-gaming-specifically defeating + vendor licensing checks for their own products. We interoperate; we do not + counterfeit. (We do not need a license for *our* device, because it is our + device.) +* **Emulating GPU instruction execution in software.** Software emulation of + CUDA kernels is 100–1000× too slow; work must run on real silicon. + +## 4. Value + +For **users**: GPU acceleration in VMs on hardware you own, with no per-VM fee, +on Windows and Linux, across generations of GPUs. + +For **integrators and hypervisor projects**: a GPU component that behaves like +any other virtual device, enabling security models that were previously +impossible (see HAAEEP in +[`../08-ecosystem/haaeep-and-collaboration.md`](../08-ecosystem/haaeep-and-collaboration.md)). + +For **the wider ecosystem**: an independent, permissively licensed +implementation of a previously single-vendor interface, with public +documentation of the protocol, the firmware and the architecture — a public +good that outlives any one vendor's product decisions. + +## 5. Model of multi-tenancy + +Open-IOV is built for **multiple independent tenants on one GPU**: + +* Each VM gets its own mediated device instance (mdev) and its own + guest-visible GPU. +* Memory is isolated by the platform IOMMU: a tenant's device can only reach + *that tenant's* memory. +* Interrupts are delivered per instance; no shared vectors. +* The host module holds no tenant-secret state; everything is per-VM. +* Scheduling between tenants is explicit policy (time-sliced now; MIG-backed + partitioning later where hardware supports it). +* The security posture is *fail closed*: if the isolation assumptions cannot be + verified at run time, the device refuses to start. + +How this is enforced and audited is documented in +[`../05-operations/01-performance-correctness-security.md`](../05-operations/01-performance-correctness-security.md). + +## 6. How we judge success + +We evaluate the project on five axes; each has a dedicated methodology +document: + +1. **Initialization** — the guest driver loads, probes, and initializes to the + same point it does on bare metal: `nvidia-smi`, CUDA runtime, `cuInit`, + context creation, module load. +2. **Runtime** — the guest *runs* real work: CUDA kernels with correct results, + OpenGL/Vulkan rendering, encode/decode; repeated, sustained, resilient. +3. **Interoperability** — no changes required anywhere: guest driver builds, + host driver builds, firmware, hypervisor, device model software. Both host + and guest firmware interfaces are honored. +4. **Stability** — crash-free under stress: start/stop cycles, VM kill, guest + driver reload, host driver reload, suspend/resume, migration. +5. **Performance and security** — within an agreed factor of native for the + supported cases, with isolation that survives adversarial review. + +## 7. Where we are, and where we need improvement + +* **Working today**: the virtual device, the GSP boot handshake, the RPC + transport, the resource-manager emulation surface the guest driver needs to + probe and enumerate, guest `nvidia.ko` and `nvidia-smi`, and the early UVM + initialization path. +* **Needs improvement**: GPU work forwarding (the reason CUDA kernels do not + yet run), the memory/page-table model, Windows guest support, non-KVM + hypervisors, and performance instrumentation. Each of these is tracked with + explicit "known-unknowns" in + [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md). + +## 8. Guiding principles + +1. **Own the interface, not a snapshot of it.** Every emulated interface is + described by a version-agnostic model plus per-version configuration; new + versions are added by measurement, not by rewriting. +2. **Emulate the interface, pass through the work.** Anything that must be fast + runs on real hardware; everything the guest can observe is virtual. +3. **No patches anywhere.** If a change is needed in someone else's software, + firmware or hardware, our design is wrong. +4. **Make the invisible visible.** Instrument everything; keep + `dmesg`-readable telemetry for every decision the emulation makes. +5. **Fail closed, fail loudly.** Isolation failures stop the device; protocol + surprises are logged with enough context to reproduce them. +6. **Document for the next person.** A feature is not done until a newcomer can + understand and maintain it from these docs alone. +7. **Permissive and portable by design** (BSD-3-Clause): adoption is a feature. diff --git a/docs/01-orientation/02-the-ai-edge-and-ecosystem.md b/docs/01-orientation/02-the-ai-edge-and-ecosystem.md new file mode 100644 index 0000000..58b1cee --- /dev/null +++ b/docs/01-orientation/02-the-ai-edge-and-ecosystem.md @@ -0,0 +1,173 @@ +# The AI Edge, Multi-Tenancy, and the Ecosystem Open-IOV Serves + +> This document explains *why now* — the industry context that makes a +> permissively licensed GPU virtualization block necessary, the kinds of +> multi-tenancy we do and do not serve, and the projects Open-IOV is designed +> to plug into. + +## 1. First, what we are not: vLLM on bare metal stays the datacenter answer + +Multi-tenant inference at datacenter scale — the kind measured in tokens per +second per dollar across an NVIDIA SuperPOD — is best served by **vLLM-style +continuous batching running on bare-metal GPU hosts**. That architecture gives +one model server the whole GPU, batches requests dynamically across tenants at +the *request* layer, and achieves utilization that no virtualization scheme can +match. Open-IOV will not replace it, and is not intended to. + +The correct layering is: + +``` +Datacenter / centralized inference → bare-metal vLLM (unchanged) +Edge / sovereign / compartmentalized → virtualized GPUs, in guests (Open-IOV) +Everything in between → whichever constraint is binding +``` + +Open-IOV exists for the cases that bare-metal serving *cannot* satisfy: + +* **Windows guest compute.** CUDA in Windows VMs, for applications that are + not Linux containers. +* **3D graphics and video** in guests, on ordinary GPUs. +* **Secure edge inference** where data must not leave the device, and where + agents, models, and infrastructure must be isolated from each other. +* **Shared GPUs on consumer/edge hardware**, where discrete passthrough is + impossible or wasteful (laptops, small servers, single-GPU workstations). + +## 2. The three needs that converge + +### 2.1 Compute inside guests (Linux and Windows) + +AI engineering and interactive agent work increasingly happen on endpoints and +edge servers. Those environments are not Kubernetes clusters; they are +workstations, laptops, kiosks, and small virtualized servers. Users of these +systems need the *guest* to have a real CUDA runtime, real 3D, and real +video — without a vendor license, a vendor host stack, or a version-locked +driver. + +### 2.2 A secure agent runtime at the edge + +An autonomous agent (Claude Code, OpenCode, a harness of your choosing) is a +**capability-constrained, untrusted-by-default workload**. It has network +access, it runs arbitrary generated code, and it should never be able to reach +the model weights, the training data, or another tenant's GPU state. Running +the agent in its own VM with a *virtualized* GPU — rather than sharing a +physical GPU through a passthrough device or through a process-level +abstraction — gives the isolation boundary the security model needs. + +Because this need is concrete and urgent, and because no permissively licensed +implementation exists, **now is the right time to reboot Open-IOV** as a +maintained project rather than a research artifact. + +### 2.3 Durability across hardware, firmware and software generations + +Every GPU virtualization deployment eventually faces: a new driver version, a +new firmware image, a new GPU, a new kernel, a new hypervisor. A stack that +cannot be updated by its users is a dead end. Open-IOV is designed as a +**config-driven, version-aware IP block** so those updates are routine (see +[`../06-portability/hardware-software-matrix.md`](../06-portability/hardware-software-matrix.md) +and [`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md)). + +## 3. HAAEEP: a High Assurance Agent Edge Environment Platform + +We propose a concrete, buildable target that ties all of the above together — +and we invite others to co-develop it. + +``` +┌───────────────────────────────────────────────────────────────────────┐ +│ EDGE DEVICE (single GPU, local inference only, fully offline) │ +│ │ +│ ┌──────────────────┐ capability-constrained, untrusted by default │ +│ │ Agent VM(s) │ (Claude Code / OpenCode / harness) │ +│ │ no GPU │ may hold credentials & code │ +│ └────────┬─────────┘ │ +│ │ API calls to model server (no external endpoint) │ +│ ┌────────┴─────────┐ model server VM: vLLM / SGLang / ... │ +│ │ Inference VM │ owns the GPU (via Open-IOV mdev or Xen fancy │ +│ │ GPU: virtual │ forwarding); isolated by the GPU hypervisor │ +│ └────────┬─────────┘ │ +│ │ │ +│ ┌────────┴─────────┐ infrastructure VMs: networking, storage, │ +│ │ Other VMs │ user desktop, ... — GPU-accelerated where │ +│ │ GPU: virtual │ needed, without touching the inference weights │ +│ └──────────────────┘ │ +│ │ +│ Hypervisor: KVM / Xen + Argo/OpenXT / Qubes OS Air — with │ +│ TrenchBoot measured boot and Flask-XSM/HMX mandatory access control │ +└───────────────────────────────────────────────────────────────────────┘ +``` + +Design points: + +1. **No external inference.** The model runs locally on the edge device inside + its own VM. No requests to an external OpenAI-style endpoint are required — + or permitted by the platform's network policy. +2. **GPU virtualization provides the separation.** A VM that owns the physical + GPU (e.g. a Xen domain with `xen-pci` passthrough) can serve GPU acceleration + to other domains through Open-IOV; an agent VM can never touch the GPU's + state, and the inference VM's weights are isolated from the agent VM. +3. **Layered isolation.** Agent VMs (capability-constrained) ≠ inference VM + (holds weights) ≠ adjacent functional VMs (desktop, network, storage) ≠ + hypervisor. Each boundary is enforced by a different mechanism (mandatory + access control, device virtualization, IOMMU). +4. **KV cache interoperability.** The inference VM's vLLM/SGLang deployment + keeps using its own plugins (including KV-cache management), unchanged. +5. **Attestable.** With TrenchBoot, the edge device can measure the + firmware → hypervisor → OS chain, so a remote party can verify the platform + before sending sensitive work to it. + +We call this the **High Assurance Agent Edge Environment Platform (HAAEEP)**. +It is a *use-case architecture*, not a product: Open-IOV provides the GPU +component, and each platform project provides the rest. See +[`../08-ecosystem/haaeep-and-collaboration.md`](../08-ecosystem/haaeep-and-collaboration.md) +for the full design, including the roadmap and how to help. + +## 4. The ecosystem Open-IOV plugs into + +Open-IOV is an *IP block*: it should be connectable to ecosystems without +those ecosystems having to modify themselves. + +| Project / platform | What it provides | What Open-IOV adds | +|---|---|---| +| **KVM + QEMU/VFIO** | Mainstream virtualization, VFIO/mdev transport | A GPU that works like any virtual device; multiple VMs per GPU | +| **Xen** | Strong isolation between domains, `xen-pci` forwarding | GPU virtualization *inside* domains; a domain can serve GPU to others | +| **Argo / OpenXT** | Hypervisor-level security, Hypervisor Mediated Exchange (HMX), Flask-XSM mandatory access control | A GPU path that respects MAC policy between VMs | +| **Qubes OS / Qubes OS Air** | Security-by-compartmentalization on modern hardware; administrative separation | GPU acceleration per compartment, especially AI workloads | +| **TrenchBoot** | Measured/attested boot (firmware → hypervisor → OS) | An attestable GPU service: the platform can prove what runs the model | +| **KVM/VFIO gaming setups** | Fast Windows VM + GPU passthrough | Solutions for users who need *multiple* VMs, or who cannot dedicate the whole GPU | +| **vLLM / SGLang / inference servers** | Continuous batching, KV-cache plugins, model serving | A way to run the server itself inside an isolated VM without losing GPU acceleration | +| **Windows driver ecosystems** | CUDA and 3D applications | Compute in Windows guests on consumer hardware | + +The unifying idea: **Open-IOV should be the component these projects reach for +when they need "an NVIDIA GPU, virtualized, on our terms."** + +## 5. Collaboration and IP models + +We explicitly invite *both* open source and commercial collaboration: + +* **Open source projects**: adopt BSD-3-Clause code directly; we will work with + you on interfaces (VFIO/mdev today, other transports on request) and accept + upstream requirements as design inputs. +* **Commercial vendors and integrators**: ship Open-IOV in products, add + proprietary value *on top*, fund specific hardware/driver-version support, + or co-develop with us under the same permissive terms. No license fees, no + contributor agreements that take rights away. +* **Hardware vendors**: nothing is required from you. Our goal is that new + hardware works through configuration and measurement, without firmware or + driver changes. If you want to help, test hardware and share traces. +* **Researchers and auditors**: the documentation in this tree — including the + protocol and firmware analysis — is written to be attackable. We want your + findings. + +The license split (BSD-3-Clause for the reusable project, GPL-2.0 for kernel +modules that link GPL-only kernel interfaces) is chosen to make all of the +above legally frictionless. See [`../09-reference/legal-and-licensing.md`](../09-reference/legal-and-licensing.md). + +## 6. What success looks like for this document + +* A security-focused platform project can integrate Open-IOV without asking us + to change the software. +* A user can run CUDA in a Windows guest on a laptop with one GPU, legally and + without a license server. +* An edge device can run an agent VM, an inference VM and a desktop VM on one + GPU, with the model never leaving the device. +* A new GPU generation is supported by a configuration update plus measured + verification — not a rewrite. diff --git a/docs/02-getting-started/01-install-and-first-run.md b/docs/02-getting-started/01-install-and-first-run.md new file mode 100644 index 0000000..e0e71cb --- /dev/null +++ b/docs/02-getting-started/01-install-and-first-run.md @@ -0,0 +1,180 @@ +# Install and First Run + +> Goal: from a fresh Ubuntu 22.04/24.04 host with a supported NVIDIA GPU to a +> guest VM running stock `nvidia.ko` and `nvidia-smi`, in about half an hour. + +## 1. Prerequisites + +**Host** + +* Ubuntu 22.04 or 24.04, x86_64 (aarch64 is a roadmap target; + see [`../06-portability/hardware-software-matrix.md`](../06-portability/hardware-software-matrix.md)). +* An NVIDIA GPU supported by the current emulation profile (Turing TU104 is the + reference; see the matrix document for the current support list). +* A recent NVIDIA **host** driver installed and working (`nvidia-smi` works on + the host). This is *not* modified by Open-IOV. +* IOMMU enabled (`intel_iommu=on` / `amd_iommu=on`), hugepages configured, + kernel headers for your running kernel. +* `qemu-system-x86_64`, `libvirt` optional. KVM enabled. +* Root (or sudo) for module load and mdev creation. + +**Guest** + +* Ubuntu 24.04 server ISO (the reference guest), or any Linux with the matching + stock NVIDIA driver available. Windows guests are a roadmap target. +* The guest needs no Open-IOV component at all — that is the point. + +## 2. Get the code and build + +```bash +git clone https://git.vgpu.io/open-iov.git +cd open-iov +./install.sh deps # build/runtime dependencies +./install.sh nvidia # NVIDIA open kernel modules (submodule) + patches +./install.sh firmware # GSP firmware image (verified) +./install.sh build # kernel module + userspace +``` + +`install.sh` performs the same steps as the manual flow below and is safe to +re-run. To see what any step does, read `install.sh` — it is intentionally +short and comments the order. + +### Manual build (equivalent, and what developers use day to day) + +```bash +# 1. Kernel module first: it generates Module.symvers used by the next step. +make kernel + +# 2. NVIDIA OpenRM modules (must see our symbols): +# KBUILD_EXTRA_SYMBOLS=<repo>/kernel/Module.symvers +./scripts/build_nvidia_openrm.sh + +# 3. Userspace library + examples +make userspace +``` + +**Build-order matters.** The NVIDIA modules resolve symbols that +`open-iov-vfio.ko` exports; building them in the wrong order produces +`Unknown symbol` failures at load time. See +[`../03-architecture/06-host-module-and-hybrid-model.md`](../03-architecture/06-host-module-and-hybrid-model.md). + +## 3. Load the stack and create a virtual GPU + +```bash +./scripts/stack_manager.sh status # check what is currently loaded +./scripts/stack_manager.sh load-stack # vfio/mdev prereqs -> open-iov -> nvidia (if used) +./scripts/stack_manager.sh mdev-create # create a vGPU mediated device +./scripts/stack_manager.sh mdev-list # inspect /sys/bus/mdev/devices/ +``` + +The mdev device is the virtual GPU. Everything after this point treats it like +any other VFIO device. + +## 4. Launch a reference guest + +```bash +./scripts/stack_manager.sh qemu # reference raw-QEMU guest with the mdev +``` + +Or integrate with your own stack (libvirt, ``, Xen, ...): attach the mdev +device to the guest exactly as you would any VFIO device: + +```xml +<hostdev mode='subsystem' type='mdev' model='vfio-pci'> + <source> + <address uuid='<mdev-uuid>'/> + </source> +</hostdev> +``` + +**Important:** for correct PCIe link capabilities in the guest, attach the +device behind a root port rather than on the root bus (QEMU rewrites the +device's PCIe capability when it detects a root-complex-integrated endpoint). +This is documented with the actual register values in +[`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md). + +## 5. Install the stock driver in the guest + +Inside the guest (the reference guest has SSH and a guest agent configured): + +```bash +sudo apt update +sudo apt install -y linux-headers-generic build-essential +# Install the stock NVIDIA driver matching the firmware/driver profile +sudo apt install -y nvidia-driver-570-open +sudo reboot +``` + +Then verify: + +```bash +nvidia-smi # must list the virtual GPU +lsmod | grep -c nvidia +``` + +If `nvidia.ko` fails to probe, capture the four standard artifacts described in +[`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md) +(host dmesg, guest dmesg, guest `lspci -vvv`, guest module parameters) and file +an issue. + +## 6. First CUDA check + +```bash +# in the guest +git clone https://github.com/NVIDIA/cuda-samples.git +cd cuda-samples/Samples/1_Utilities/deviceQuery && make && ./deviceQuery +``` + +**Expectation today:** driver probe, `nvidia-smi`, device enumeration, and the +early CUDA start-up path work; full kernel execution is the active work item. +Read the honest status in +[`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) +before drawing conclusions from a CUDA failure. + +## 7. Day-to-day operations + +```bash +./scripts/stack_manager.sh up # build all + load + create mdev +./scripts/stack_manager.sh status # modules, mdev, vfio, firmware +./scripts/stack_manager.sh test # host test suite +./scripts/stack_manager.sh unload # correct-order teardown +./scripts/stack_manager.sh cleanup-crash +``` + +Load order is: prerequisites → kernel module → (optional) NVIDIA OpenRM → +firmware. Unload is the reverse. The manager enforces this; do not `rmmod` +by hand while a VM holds the mdev. + +## 8. Watching the emulation work + +Every meaningful decision the emulation makes is visible in `dmesg` with the +prefix **`gsp-resman-vfio:`** (module logs) and the per-RPC traces. Handy: + +```bash +sudo dmesg -w | grep -iE 'gsp-resman-vfio|gsp_|rpc' # live +sudo dmesg -T | grep -iE 'gsp-resman-vfio' | tail -50 # recent +``` + +Developer debug knobs (module parameters) are documented in +[`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md). + +## 9. Troubleshooting index + +| Symptom | Start here | +|---|---| +| Module will not load | `dmesg`, `modinfo open-iov-vfio.ko`, stale module state → `cleanup-crash` | +| No mdev type for your GPU | [hardware matrix](../06-portability/hardware-software-matrix.md) | +| Guest sees wrong PCIe link speed/width | root-port attachment + config-space mirror (debugging doc) | +| `nvidia.ko` probe fails in guest | guest dmesg + `lspci -vvv` + handoff "driver lifecycle" | +| `nvidia-smi` works, CUDA does not | expected today; see the known-knowns doc | +| VM boot fails or the guest hangs | `scripts/stack_manager.sh cleanup-crash`, then reboot | + +## 10. Uninstall + +```bash +./scripts/stack_manager.sh unload +sudo rmmod open_iov_vfio nvidia nvidia_modeset nvidia_uvm # if present +``` + +The project never modifies your host driver's files; removing the Open-IOV +module and its mdev devices returns the host to its previous state. diff --git a/docs/03-architecture/01-system-overview.md b/docs/03-architecture/01-system-overview.md new file mode 100644 index 0000000..ed9b24b --- /dev/null +++ b/docs/03-architecture/01-system-overview.md @@ -0,0 +1,183 @@ +# System Overview + +> The one document to read if you want to understand *how Open-IOV works* before +> diving into any subsystem. + +## 1. The core architectural principle + +> **Emulate the interface. Pass through the work.** + +A virtual GPU is a *mediated passthrough* device. The guest must believe it +owns a real NVIDIA GPU with a real GPU System Processor (GSP) running NVIDIA's +firmware. The host must arrange for the guest's work to execute on real +hardware, because emulating compute in software is 100–1000× too slow to be +useful. + +Everything therefore falls into two buckets: + +``` +┌─────────────────────────────────────────────────────────────────────────┐ +│ EMULATE (software; guest expects virtual hardware) │ +│ • PCI configuration space, capability structures, power management │ +│ • BAR0 register interface (shadow, not the physical GPU's BAR0) │ +│ • BAR1 VRAM aperture (guest-visible video memory) │ +│ • Instance/PRAMIN window │ +│ • GSP firmware boot handshake (FWSEC → booter → RISCV → boot complete)│ +│ • GSP-RM message queues, RPC transport, resource-manager controls │ +│ • Event/notification channels │ +│ • MSI-X injection (via eventfd, from the hypervisor's perspective) │ +└─────────────────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────────────────┐ +│ PASS THROUGH (real hardware; for correctness and performance) │ +│ • GPU memory: guest VRAM objects backed by real VRAM (goal) │ +│ • Page tables / GMMU state used by the GPU (goal) │ +│ • Channel execution: the guest's push buffers run on real channels │ +│ • Compute (CUDA) and, later, 3D/video work │ +│ • DMA, completion and fault interrupts relayed back to the guest │ +└─────────────────────────────────────────────────────────────────────────┘ +``` + +The line between the two is the **interface boundary**, and it moves as the +project matures: early work emulates more (everything the guest can observe), +later work forwards more (everything that must be fast). + +## 2. Layer map + +``` +GUEST + application ── CUDA runtime (libcudart) ─┐ + application ── CUDA driver (libcuda) ─┤ userspace: CUDASW stack + OpenGL/Vulkan ── libGLX/libvulkan ──────┘ + │ ioctls / mmaps + nvidia.ko (unmodified) ── RM core, resource server, UVM, modeset/drm + │ MMIO + DMA + MSI-X (a virtual PCIe device) +════════╪════════════════════════════════════════════════════════════════ +HYPERVISOR (KVM/Xen/...) VFIO mdev transport + IOMMU isolation + │ +HOST MODULE open-iov-vfio.ko + ├─ virtual device: config space, BAR0/BAR1/BAR3 windows, MSI-X + ├─ emulation core: GSP boot + GSP-RM (RPC msgq, controls, objects, events) + ├─ memory model: guest RAM access, VA↔PA maps, VRAM backing + └─ work path: channel/push forwarding to the physical GPU [in progress] + │ +HOST DRIVER nvidia.ko (unmodified) and/or direct GPU interfaces + │ +PHYSICAL GPU +``` + +## 3. Host module architecture + +`open-iov-vfio.ko` is a **VFIO mediated-device (mdev) driver**. It registers a +parent on the physical GPU and exposes virtual GPU types; each `mdev` instance +is one virtual GPU for one VM. + +| Subsystem | Files (illustrative) | Responsibility | +|---|---|---| +| mdev/VFIO registration | `kernel/main.c` | parent/type registration, VFIO device, region and IRQ setup | +| Config space | `kernel/main.c`, `kernel/bar_init.c` | identify/command/status, BARs, PCIe caps, MSI-X caps, VF windows | +| BAR0 shadow | `kernel/bar_emulation.c` | 16 MB emulated register space; register reads/writes the guest driver depends on | +| VRAM aperture (BAR1) | `kernel/main.c`, `memory.c` | 256 MB guest-visible video memory (host RAM today; real VRAM later) | +| Instance memory (BAR3) | `kernel/bar_emulation.c` | instance/PRAMIN window behaviour | +| GSP emulation thread | `kernel/gsp_emulation.c` | boot stage completion, doorbell polling, MSI-X injection | +| GSP-RM emulation | `kernel/gsp_rm_emulation.c` | message queues, RPC dispatch, resource-manager controls, objects | +| RPC handlers (firmware-derived) | `kernel/gsp_rpc_client.c` | behaviours decompiled from the GSP firmware | +| Firmware analysis (runtime) | `kernel/firmware_analyzer.c` | CRC/SHA signatures, pattern matching, version identification | +| Version dispatch | `kernel/dynamic_rpc_dispatcher.c`, `kernel/version/` | version-aware handler selection | +| Hardware abstraction | `kernel/architecture/` | per-architecture constants and behaviours | +| Userspace library | `userspace/lib/libvgpu.c` | management API (`vgpu_*`), diagnostics | + +**Data flow for an MMIO access from the guest:** + +``` +guest nvidia.ko writes BAR0 offset X + → KVM exit → QEMU → VFIO region write into the mdev + → nv_vgpu_vfio_dev_write() (or directly into the BAR0 shadow if mmap'd) + → BAR0 shadow state machine (doorbells, VF registers, notify) + → may wake the GSP emulation thread +``` + +**Data flow for an RPC from the guest to "firmware":** + +``` +guest writes a command-queue element into a guest-memory msgq and rings the +GSP doorbell (BAR0) + → emulation thread reads the queue through the hypervisor's guest-RAM + accessor (vfio_dma_rw), parses the element, dispatches the RPC + → handler produces a reply element in the status queue + → emulation injects an MSI-X interrupt to the guest +``` + +## 4. Guest stack summary + +* `nvidia.ko` is the **unmodified** NVIDIA kernel driver. It contains the + resource manager (RM) core, the resource server (`resserv`), the memory + manager, the FIFO/runlist code, the GPU System Processor client path, and + UVM (the unified memory/virtual-memory module, `nvidia_uvm.ko`). +* User space consists of the CUDA driver (`libcuda.so`), the CUDA runtime + (`libcudart.so`), plus graphics/video libraries. The family of NVIDIA + *software* components that mediate between applications and the driver is + referred to in this project as **CUDASW**. +* The guest expects to program a real GPU: it writes registers, allocates + objects, creates channels, submits push buffers and waits on semaphores — + exactly as on bare metal. + +Detailed subsystem documents: + +* [`02-guest-stack-and-cudasw.md`](02-guest-stack-and-cudasw.md) +* [`03-gsp-and-gsp-rm.md`](03-gsp-and-gsp-rm.md) +* [`04-uvm.md`](04-uvm.md) +* [`05-gmmu-amap-hal-gr.md`](05-gmmu-amap-hal-gr.md) +* [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) +* [`07-driver-lifecycle-field-guide.md`](07-driver-lifecycle-field-guide.md) + +## 5. Why a GSP emulation is possible at all + +The guest driver never talks to GPU silicon directly for management: since the +Turing generation, NVIDIA GPUs boot a **RISC-V firmware (the GSP)** that hosts +the Resource Manager. The driver's normal path is to talk to that firmware over +shared memory queues using a small RPC protocol. + +That protocol is: + +* **small** (a few dozen message types at the levels we need), +* **stable** (message layouts are versioned and compiled into the driver), and +* **observable** (everything arrives in guest memory we can read). + +So instead of emulating a GPU, we emulate **the firmware's interface**. The +guest driver is satisfied because the protocol is satisfied; the guest's +*work* is then given to real hardware. + +The firmware itself we study directly: it is a 28 MB RV64IMAC ELF that we have +decompiled and mapped (see `wiki/GSP-570.144W-Firmware-Analysis.md` and +`wiki/Address-Reference.md`). + +## 6. What runs today + +1. Guest boots, enumerates the virtual PCIe device, binds `nvidia.ko`. +2. The driver performs the GSP boot handshake with our emulation and receives + plausible firmware identity/configuration. +3. The driver sets up the RPC message queues (guest sysmem), negotiates + versions, and performs the resource-manager initialization exchanges. +4. Object creation (clients, devices, subdevices), capability queries, + vaspaces, channels, CE caps, GMMU static info — every control the driver + probes — is answered with the correct shape and values. +5. `nvidia-smi` works; CUDA user-space starts; UVM begins channel-manager and + page-tree initialization. + +## 7. What is next (and why) + +CUDA kernel execution requires the guest's *work* to run on the physical GPU. +That means the host module must: + +1. know the memory the guest's GPU work refers to (the VA↔PA model), +2. reproduce or forward the channel/page-table state that the guest expects the + GSP to own, and +3. submit the guest's compute work to real engine(s) and relay completions, + faults and semaphores back. + +The design for this, including what is known and what remains to be learned, is +in: +[`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) +and +[`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md). diff --git a/docs/03-architecture/02-guest-stack-and-cudasw.md b/docs/03-architecture/02-guest-stack-and-cudasw.md new file mode 100644 index 0000000..4ec614b --- /dev/null +++ b/docs/03-architecture/02-guest-stack-and-cudasw.md @@ -0,0 +1,139 @@ +# The Guest Stack and CUDASW + +> What runs inside the guest, how it is layered, and what it expects from the +> device. This is the "what must keep working, unmodified" document. + +## 1. Vocabulary: what "CUDASW" means here + +**CUDASW** is the family of NVIDIA *software* components that sit between an +application and the GPU driver. In this project the term is used deliberately +broadly, because our compatibility target is the whole stack, not one library: + +``` +application + │ CUDA API (runtime) +libcudart.so ──────────────────────────────┐ CUDA runtime: device/stream/ + │ CUDA driver API │ event management, kernel launch +libcuda.so ────────────────────────────────┤ CUDA driver: contexts, modules, + │ ioctls │ memory, channel submission +/dev/nvidia* ──────────────────────────────┘ +nvidia.ko (RM + UVM + modeset) +``` + +Adjacent components in the same family: + +* **libnvidia-ptxjitcompiler** — JIT compilation of PTX to SASS. (No GPU + interaction beyond driver interfaces; usually "just works" once the driver + initializes.) +* **libnvidia-fatbinaryloader / nvrtc** — fatbinary loading. +* **UVM** (`nvidia_uvm.ko`) — unified memory and GPU page-fault-driven memory + management; see [`04-uvm.md`](04-uvm.md). +* **Graphics stack** — `libGLX_nvidia`, `libnvidia-glcore`, `libvulkan` ICD, + plus the kernel-side `nvidia_modeset`/`nvidia_drm` (roadmap). +* **Video stack** — NVENC/NVDEC userspace + kernel interfaces (roadmap). + +## 2. Layering rules the emulation must respect + +1. **The kernel driver is the contract.** Everything above it (libcuda, + libcudart, libnvidia-*) talks to `nvidia.ko` through stable ioctls, memory + mappings and `/dev/nvidia*` nodes. If the kernel driver initializes and + behaves, user space follows. +2. **User space is opaque to us.** We never intercept user-space APIs. All our + hooks are in the device model (host side) or in the *hardware interface* the + kernel driver sees (guest side). +3. **Version skew is normal.** The driver version, the CUDA version and the + firmware version are three independent axes. Our compatibility target is + "the driver family that matches the emulated firmware profile", with + configuration-driven adaptation for others. + +## 3. Kernel driver anatomy (what the guest brings) + +`nvidia.ko` is a large, self-contained stack. The parts that matter to +virtualization: + +| Component | Role | Why it matters to us | +|---|---|---| +| **RM core** | object model, handles, locks, device lifecycle | drives the GSP boot and the entire RPC/control surface | +| **Resource server (`resserv`)** | allocates and tracks resources (objects, memory, channels) | the client-side half of the client/server RM split (see the GSP-RM doc) | +| **Memory manager (`memmgr`)** | VRAM/sysmem objects, mappings, page tables | the guest expects the GSP to own page tables for its VA spaces | +| **FIFO/runlist** | channels, TSGs, GPFIFO, USERD, doorbells | where the guest's work is submitted | +| **GSP client** | message queues + RPC transport + event handling | the interface we emulate | +| **UVM** | unified memory, page-fault handling, channel manager | starts early in CUDA initialization; extremely stateful | +| **`nvidia_modeset` / `nvidia_drm`** | display/DRM | needed later for graphics | +| **CE / GR HALs** | engine programming | needed to *execute* work (forwarded) | + +## 4. Driver initialization vs. runtime + +Understanding the split between these two phases is the key to debugging: + +**Initialization (boot/probe)** — deterministic, mostly one-shot: + +``` +PCI probe → identify GPU from config space and BAR0 registers + → boot the GSP (our emulated handshake) + → read static configuration (device info tables, FB/BUS info, VBIOS, clocks) + → create root client/device/subdevice objects + → initialize RM subsystems (interrupts, FIFO, memory, GMMU, PMU) + → expose /dev/nvidia* and register with the kernel +``` + +**Runtime (per-process, per-context)** — driven by user-space workloads: + +``` +open /dev/nvidia* → cuInit → UVM registration (per process) + → cuCtxCreate → object/allocations → memory maps + → channel creation + push buffers + semaphores + → kernel launches (GR) / copies (CE) / memops + → faults, completions, events +``` + +The initialization path is where compatibility is won or lost; the runtime +path is where *execution* is won or lost. Open-IOV's current focus is the +runtime path (forwarding), having already reached the end of initialization. + +## 5. What the guest expects from the device, per phase + +| Phase | Guest action | Device/firmware expectation | +|---|---|---| +| Probe | config space reads, BAR sizing, MSI-X capability | a plausible TU104-family PCIe GPU | +| Boot | BAR0 register writes, WPR setup, firmware boot sequence | GSP boots, booter completes, "GSP init done" | +| Config | RPC `GET_GSP_STATIC_INFO`, device tables, VBIOS reads | consistent, self-describing tables | +| RM init | `GSP_RM_CONTROL` battery (FB/BUS/CE/INTR/GMMU/...) | correct shapes and plausible values | +| Objects | `GSP_RM_ALLOC` (root, device, subdevice, vaspace, channel, CE objects) | success + correct handles/params | +| Runtime | `GSP_RM_ALLOC`/`GSP_RM_CONTROL` for maps, channels; pushes | executed work, semaphores, faults | + +The field guide [`07-driver-lifecycle-field-guide.md`](07-driver-lifecycle-field-guide.md) +lists the actual message/control IDs observed at each step. + +## 6. Why user-space-only interception does not work + +Two hard-won facts from this project: + +1. **`libcuda` issues raw `syscall` instructions**, so an `LD_PRELOAD` shim + around `ioctl()` sees nothing. If you need to observe CUDA's kernel calls, + use a `ptrace`-based tracer (PTRACE_SYSCALL + PTRACE_GET_SYSCALL_INFO), as + our tools do. +2. **The CUDA driver talks to the *kernel driver*, not to the device.** All + device interaction happens in `nvidia.ko`. This is good news: it means the + emulation boundary is a small number of kernel-side interfaces (MMIO, DMA, + interrupts, and the GSP protocol) rather than a large user-space API + surface. + +## 7. Windows guests + +Windows brings its own `nvlddmkm.sys` driver, which uses the same underlying +device/firmware model (GSP) but different host interfaces (WDDM, MSI-X +semantics, ROM/VBIOS quirks). The emulation target is the same: satisfy the +firmware interface, forward the work. Windows support is a roadmap item (see +the portability matrix), and the design goal is that no Windows-specific code +lives in the guest. + +## 8. Compatibility notes for versions + +* The guest's driver and the emulated firmware profile must agree on the RPC + message versions. The guest negotiates these at start-up + (`SET_GUEST_SYSTEM_INFO` carries the VGX/RPC version); our emulation answers + with a profile that matches the driver family in use. +* CUDA versions ride on driver versions; the test matrix (see + [`../05-operations/01-performance-correctness-security.md`](../05-operations/01-performance-correctness-security.md)) + tracks (driver, CUDA, firmware) triples. diff --git a/docs/03-architecture/03-gsp-and-gsp-rm.md b/docs/03-architecture/03-gsp-and-gsp-rm.md new file mode 100644 index 0000000..20d335b --- /dev/null +++ b/docs/03-architecture/03-gsp-and-gsp-rm.md @@ -0,0 +1,204 @@ +# GSP and GSP-RM + +> The GPU System Processor hosts NVIDIA's Resource Manager in firmware. Since +> Turing, the guest driver's management path is *talking to that firmware*, not +> to silicon. This document explains that architecture and the interface we +> emulate. + +## 1. What the GSP is + +The **GSP** is a small embedded RISC-V processor (RV64IMAC) inside NVIDIA GPUs. +It runs a firmware image (for the 570.144 generation, a ~28 MB ELF) that +contains, among other things, a full port of NVIDIA's **Resource Manager (RM)** +— the component that owns GPU initialization, memory, engines, interrupts and +scheduling. + +Why this matters: a GPU driver as complex as NVIDIA's is not written to +program silicon directly. It is written to talk to an **RM**, and on modern +GPUs a *copy* of the RM runs on the GSP. The driver is then an **RM client**: +it issues requests, receives callbacks, and observes state through a +well-defined protocol. + +``` + ┌──────────────────────────── GUEST ────────────────────────────┐ + │ nvidia.ko │ + │ RM client path ── RPC ──┐ │ + │ GSP boot code │ │ + └────────────────────────────┼──────────────────────────────────┘ + │ shared memory queues + doorbells + ┌────────────────────────────┼──────────────────────────────────┐ + │ GSP (firmware, RISC-V) │ │ + │ RM core (RMCore) ◄─────┘ │ + │ RM API (object model, clients, handles) │ + │ Resource server (resserv): allocation & lifetime │ + │ Memory manager, GMMU, FIFO/runlist, interrupts, PMU │ + └───────────────────────────────────────────────────────────────┘ +``` + +## 2. GSP boot: the handshake we emulate + +The guest driver boots the GSP exactly as it would on bare metal: + +1. **WPR (Write-Protected Region) setup** — carve a protected region of VRAM + for firmware, write descriptors into BAR0 registers. +2. **FWSEC** — the driver verifies firmware signatures (our emulation answers + the microcode/falcon handshake plausibly). +3. **Booter** — the boot ROM/booter is released and consumes the firmware + image (we emulate completion). +4. **RISCV start** — the RISC-V core starts executing the firmware; on real + hardware the firmware then initializes. +5. **GSP init done** — the firmware signals readiness through a status mailbox; + the driver proceeds to initialization RPCs. + +Our emulation implements stage completion rather than the stages themselves — +which is legitimate because the *only observable* to the driver is the state +machine it polls. + +## 3. The transport: message queues and RPC + +GSP-RM's transport is a shared-memory message queue pair: + +``` + guest system memory (allocated by the guest driver, registered with us) + ┌───────────────────────────────┐ + │ command queue (guest→"GSP") │ 512 KB element ring, 4 KB elements + ├───────────────────────────────┤ + │ status queue ("GSP"→guest) │ 512 KB element ring + └───────────────────────────────┘ + + a page table describing the queue pages (the buffer is + NV_MEMORY_NONCONTIGUOUS: each 4 KB page is an independent allocation) +``` + +Each element begins with a transport header (`authTag`, `aad`, checksum, +sequence, element count) followed by an **RPC message**: + +``` +struct rpc_message_header_v { + u32 header_version; u32 signature; u32 length; u32 function; + u32 rpc_result; u32 rpc_result_private; u32 sequence; + union { u32 spare; u32 cpuRmGfid; }; +}; // 32 bytes; payload follows in a union of message structs +``` + +Observations worth knowing (all verified on live hardware): + +* `length` is `sizeof(header) + sizeof(fixed message struct)` — for messages + with variable payloads (e.g. `GSP_RM_ALLOC` with parameters) it does **not** + include the parameters. Do not use it to bound a payload. +* Function numbers are a flat enum shared by several protocols + (GSP-client RPCs, vGPU plugin RPCs). Examples observed from a stock 570.144 + guest: `SET_GUEST_SYSTEM_INFO` (1), `FREE` (10), `UNLOADING_GUEST_DRIVER` + (47), `GET_GSP_STATIC_INFO` (65), `UPDATE_BAR_PDE` (70), + `CONTINUATION_RECORD` (71), `GSP_SET_SYSTEM_INFO` (72), `SET_REGISTRY` (73), + `GSP_RM_CONTROL` (76), `GSP_RM_ALLOC` (103). +* Timing matters: a reply must be matched to the request sequence number; + the driver runs a reply-matching state machine, and a slow reply is + indistinguishable from a hang. + +## 4. RMCore, RM API and the resource server + +Three concepts from the RM show up constantly in traces and in the guest's +driver. Understanding them makes every log readable. + +### 4.1 RMCore + +RMCore is the object system. Everything in the RM is an **object** with: + +* a **class** (a 32-bit class ID, e.g. `0x2080` = `NV20_SUBDEVICE_0`, + `0x906f`-family = channels, `0xc46f` = `TURING_CHANNEL_GPFIFO_A`), +* a **handle** allocated in a per-client handle space (e.g. `0xc1d00002`), +* a **parent** object (forming a tree rooted at a client). + +Traces of `GSP_RM_ALLOC` are therefore traces of object creation, and traces +of `GSP_RM_CONTROL` are method calls on those objects. + +### 4.2 RM API + +The **RM API** is how in-kernel components (and user space via ioctls) allocate +objects and call controls. Two important variants exist in the client: + +* `Alloc` — allocates an object with an implicit handle, +* `AllocWithHandle` — the caller chooses the handle. + +For a GSP client, the *kernel-internal* API table is overridden so that some +calls become RPCs while others are handled locally. Knowing which is which is +the difference between "we see it in the trace" and "we must derive it another +way". Notably, memory objects are allocated **locally by the client RM** (the +pages are guest RAM), while *server-owned* ranges (see §5) are managed by the +GSP. + +### 4.3 Resource server (`resserv`) + +The resource server is the allocation authority: it owns the handle spaces, +enforces privileges, and tracks object lifetimes. Its directives include flags +that matter to us, for example `RMAPI_ALLOC_FLAGS_SERIALIZED` (the parameters +in an RPC are a serialized form that must be deserialized by the receiver) and +`RMAPI_ALLOC_FLAGS_SKIP_RPC` (the allocation is handled locally and never +reaches the server). + +## 5. The client/server split (and why page tables are ours) + +Modern RM has a **client/server** architecture: + +* The **client** (guest `nvidia.ko`) owns per-process state and submits work. +* The **server** (GSP-RM on the GPU, under our emulation) owns GPU-global + state: routing, page tables, scheduling domains, engine configuration. + +For virtualized/partitioned setups, the client tells the server which VA +ranges it wants the *server* to own, and the server manages the deeper page +table levels for those ranges. We receive this contract explicitly in the +`NV90F1_CTRL_CMD_VASPACE_COPY_SERVER_RESERVED_PDES` control, which carries the +**physical addresses of the client's page-table levels** plus the VA range and +page shifts they cover: + +``` +levels[0] phys=0x1297ac000 pageShift=47 (root page directory) +levels[1] phys=0x1297ad000 pageShift=38 +levels[2] phys=0x1297ae000 pageShift=29 +``` + +In a real system, the server then *fills in* the lower levels for +server-owned ranges; the client never writes them. This is a central fact for +the forwarding work: **when the guest's UVM asks the "GPU" to map memory, the +mapping is expected to materialize in page tables that only we can write.** + +## 6. Event channels and notifications + +Beyond request/reply, the guest needs asynchronous events: + +* **Event objects** (`NV01_EVENT_*` classes) are created by the client and + registered with a *notification* configuration: MSI-X vector, or a memory + "semaphore" the GPU can write. +* **Completion events** are delivered when objects change state (channels + finish work, errors occur). +* `cl2080_notification.h` documents the notification layout; the emulation + must deliver exactly the bit/handle/state triple the guest expects, on the + exact vector. + +Our MSI-X path is: emulation thread → `eventfd` → KVM → guest interrupt +vector. The PBA (pending bit array) is emulated so a masked vector's interrupt +is not lost. + +## 7. What we emulate today, precisely + +| Layer | Emulated | Notes | +|---|---|---| +| GSP boot state machine | ✅ | stages complete; the driver's poll loop is satisfied | +| Message queue transport | ✅ | including the non-contiguous page table for the queue buffer | +| RPC request/reply | ✅ | all functions the driver uses during init are answered | +| Static info tables | ✅ | device/FB/BUS/INTR/GMMU tables with plausible TU104 values | +| Controls during init | ✅ | ~160 distinct control IDs observed per CUDA run; the ones needed to proceed carry real values | +| Object creation | ✅ | correct handle/parameter shapes | +| Page tables for server-owned ranges | 🟡 | contract received and recorded; leaf population is the forwarding work | +| Event/notification delivery | 🟡 | MSI-X works; richer event semantics as forwarding lands | +| Firmware *behaviour* (routing, scheduling, real execution) | 🔴 | the forwarding work; see the implementation docs | + +## 8. Where the firmware analysis lives + +* `wiki/GSP-570.144W-Firmware-Analysis.md` — how the 28.5 MB image was + decompiled, RPC dispatchers located, jump tables mapped. +* `wiki/Address-Reference.md` — firmware addresses for structures, functions + and tables. +* `wiki/RPC-Message-Format.md`, `wiki/GSP-RPC-Emulation.md` — protocol details. +* `wiki/WPR-Management.md` — write-protected region semantics during boot. +* `kernel/gsp_rpc_client.c` — the firmware-derived handlers expressed as C. diff --git a/docs/03-architecture/04-uvm.md b/docs/03-architecture/04-uvm.md new file mode 100644 index 0000000..299f46b --- /dev/null +++ b/docs/03-architecture/04-uvm.md @@ -0,0 +1,172 @@ +# UVM — Unified Memory in the Guest + +> UVM (`nvidia_uvm.ko`) is the most demanding consumer of the virtual GPU's +> semantics, and today the best measurement of how faithful our emulation is. +> This document explains what UVM does, what it needs, and what we learned +> from watching it run against our device. + +## 1. What UVM is + +UVM provides unified memory and GPU-driven page-fault handling. In practice it +is a second GPU driver, layered on the RM: + +* **VA spaces** (`uvm_va_space`) — per-process GPU address spaces. +* **Channel managers** — pools of GPU channels (CE, SEC2, GR-as-needed) that + UVM itself uses to execute its own memory-management work. +* **Push buffers / GPFIFO** — UVM builds its own pushes and submits them. +* **Semaphores and trackers** — UVM tracks GPU progress by waiting on + semaphores that the GPU writes when work completes. +* **Page trees** — UVM manages GPU page tables for its VA spaces. +* **Fault handling** — GPU page faults are delivered to UVM (via a fault + buffer and a notification channel), and UVM resolves them by populating page + tables and replaying/cancelling faults. + +For CUDA, UVM is entered early: `cuInit` → driver init → `UVM_REGISTER_GPU`. +If UVM cannot complete GPU registration, CUDA does not start. + +## 2. The registration path (traced end to end) + +``` +libcuda: UVM_REGISTER_GPU ioctl (0x25) + uvm_api_register_gpu + uvm_va_space_register_gpu + uvm_gpu_retain_by_uuid + nvUvmInterfaceRegisterGpu / GetGpuInfo + alloc_parent_gpu / add_gpu + nvUvmInterfaceDeviceCreate (RM object creation) + uvm_hal_init_gpu / get_gpu_caps (CE caps! see below) + uvm_ats_add_gpu / uvm_gpu_init_isr (fault/intr plumbing) + init_channel_manager + create semaphore pool (sysmem) + create channels per CE (each channel: GPFIFO + pushbuffer + semaphores) + channel_init() → first push → semaphore release → wait + uvm_page_tree_init (page tables for the VA space) +``` + +Each stage has a distinct failure signature; our debugging docs map them. + +## 3. What UVM requires from the "GPU" + +| Requirement | Detail | Our status | +|---|---|---| +| GPU identity | must match the uuid from RM | ✅ | +| CE capabilities | per-engine support bits; UVM *requires* `supported && !grce && sysmem` for CPU_TO_GPU/GPU_TO_CPU channels | ✅ (we answer CE_GET_PHYSICAL_CAPS with the correct bits) | +| Channel creation | RM channel objects with GPFIFO/userd/instance parameters | ✅ objects created | +| Pushes executed | UVM's first action on a new channel is an *init push* followed by a semaphore-tracked wait | 🟡 we can complete them (see below) | +| Page trees | UVM initializes page trees with its own pushes; the *server-owned* ranges must be populated by us | 🟡 | +| Fault delivery | fault buffer + notification; replay/cancel semantics | 🔴 | +| Semantics of completions | tracker values must advance exactly | 🟡 | + +## 4. The most important discovery: how UVM waits + +UVM's tracker (`uvm_tracker.c`) is the single most visible symptom of an +unfaithful GPU. It spins, and after 30 seconds it dumps: + +``` +uvm_tracker.c:285 Tracker entry for value 1 (sema VA 0x121000000) + channel ID 1:3 (0x1:0x3) CE 0 +uvm_channel.c:4184 pending push '' started at uvm_channel.c:2535 in channel_init() + releasing value 1 +``` + +What this means, mechanically: + +1. UVM built a push in a push buffer and wrote a **semaphore release** into it: + a method sequence (CE channel class, e.g. `TURING_DMA_COPY_A`) that sets an + address and a payload and then launches a DMA operation that writes the + payload to memory. +2. UVM submitted the push (GPFIFO entry + GPPUT/doorbell) and now waits for the + semaphore payload to reach `value`. +3. On real hardware, the copy engine writes the payload into the semaphore's + memory. On our device, *nobody does* unless we do it. + +The push we found in guest RAM (found by scanning for its distinctive byte +sequence) decodes as: + +``` +20010000 0000c5b5 set-object (TURING_DMA_COPY_A) +2001a000 0000c076 set-object (GP100_UVM_SW) +20038090 00000001 incrementing-method header: count 3, addr 0x240 + + data: upper 32 bits of the semaphore VA +21000000 00000001 lower 32 bits of the VA; payload (1) +200180c0 0000000c LAUNCH_DMA with SEMAPHORE_TYPE=RELEASE_ONE_WORD +``` + +So: **the emulation's job for this push is to write `payload` to the memory at +the semaphore's GPU VA.** Which requires a GPU-VA → guest-PA translation. + +## 5. The memory model UVM assumes (and what we learned) + +* UVM's semaphore pool is **system memory** (`pool->aperture = UVM_APERTURE_SYS`), + one 4 KiB page per channel manager, addressed as 4-byte slots + (`UVM_SEMAPHORE_SIZE = 4`, 1024 slots per page). +* Each semaphore page has a *GPU VA* (chosen by the client's RM) and a *CPU VA* + (a kernel mapping). UVM polls the CPU VA; the GPU writes the GPU VA. +* **Both addresses are recorded in UVM's structures in guest memory.** This is + the key that makes translation possible without any extra help from the + guest (see the host-module doc for the exact structures and offsets). +* The CPU VA is a **direct-map** address, so + `PA = cpu_va - page_offset_base`. Verified live: writing the payload to that + physical address completes the push and UVM advances. +* Values are *per channel* and *monotonic*: a channel's first push releases 1, + the next 2, and so on. UVM has its own sanity check + (`UVM_GPU_SEMAPHORE_MAX_JUMP`) — writing a wrong value is detected and + reported, which is useful for validation. +* The client's **GPU page tables for server-owned ranges are empty chains**: + the client hands us the *physical addresses of its page-table levels* in + `NV90F1_CTRL_CMD_VASPACE_COPY_SERVER_RESERVED_PDES` and expects the server to + own the lower levels. (See the GSP-RM doc, §5.) + +## 6. Page trees and faults + +* `uvm_page_tree_init` initializes the page tables for a VA space using pushes + on UVM's own channels. These pushes are memops / page-table writes; on real + hardware they are executed by the copy engine. +* Fault handling is event-driven: UVM programs a **fault buffer** (a + replayable and a non-replayable buffer, whose sizes come from + `INTERNAL_GMMU_GET_STATIC_INFO` — answered by our emulation) and receives + notifications when the GPU takes a fault. UVM then: + 1. reads the fault entry, + 2. populates the page table (possibly migrating memory), + 3. **replays or cancels the fault** with a `MEM_OP` push. +* For our forwarding design, faults are where "the guest's memory model" and + "the real GPU's memory model" must be reconciled. This is the deepest part + of the remaining work, and is tracked as a known-unknown in + [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md). + +## 7. How to observe UVM + +In the guest: + +```bash +# UVM prints nothing by default. Turn it on: +echo 1 | sudo tee /sys/module/nvidia_uvm/parameters/uvm_debug_prints +# Messages then appear with the "nvidia-uvm:" prefix. +# Other knobs: uvm_release_asserts, uvm_release_asserts_dump_stack, +# uvm_enable_debug_procfs, uvm_debug_enable_push_desc. +``` + +Known-good markers, in order: + +``` +uvm_gpu_retain_by_uuid → ... → channel manager created +→ uvm_page_tree_init pushes +→ (on success) UVM_REGISTER_GPU returns and CUDA proceeds +``` + +Failure signatures we have characterized: + +| Message | Meaning | +|---|---| +| `Failed to initialize the channel manager: NV_ERR_NOT_SUPPORTED` | CE caps were wrong/missing (we fixed this: see the field guide) | +| `stuck waiting` + `Tracker entry ... channel_init()` | the push's semaphore was never written (the emulation must write it) | +| `Assert failed ... unexpected semaphore ... jump from 0x0 to 0x1010101` | we wrote a wrong payload (and UVM caught us — useful!) | + +## 8. Why UVM is the perfect correctness oracle + +UVM exercises in minutes what a driver test suite would test in hours: +memory allocation, page tables, channel setup, push generation, semaphore +semantics, fault handling, error recovery — all with continuous self-checks and +loud diagnostics. When UVM registers cleanly and CUDA workloads pass, the +device is behaving like a real GPU. That is why "UVM advances" is the project's +primary correctness metric on the way to CUDA. diff --git a/docs/03-architecture/05-gmmu-amap-hal-gr.md b/docs/03-architecture/05-gmmu-amap-hal-gr.md new file mode 100644 index 0000000..6ff9257 --- /dev/null +++ b/docs/03-architecture/05-gmmu-amap-hal-gr.md @@ -0,0 +1,198 @@ +# GMMU, AMAPlib, HALs, Runlists, and GR + +> The engine side: how addresses translate, how work is scheduled, and what +> "GR" actually is. Read this when you need to know what the guest's work +> *means*. + +## 1. GMMU — the GPU memory management unit + +The GMMU translates the GPU virtual addresses (VAs) used by engines into +physical addresses (VRAM offsets or system-memory addresses). + +### 1.1 Page-table hierarchy + +NVIDIA's GMMU uses a multi-level page table whose shape depends on the AS +generation. Turing (the reference) uses a four-level hierarchy with entry +coverage: + +| Level | Entries | VA bits consumed | Coverage per entry | +|---|---|---|---| +| Root page directory (PDB) | 4 | top bits | large | +| PDE level 1 | 512 | 9 bits | e.g. 2^38 | +| PDE level 2 | 512 | 9 bits | e.g. 2^29 | +| PTE level | 512+ | 9+ bits | 4 KiB / 64 KiB / 2 MiB pages | + +Verified properties (from the live guest): + +* **PTE encoding: `PA[47:12] | flags`**, where `flags = 0xe` for a valid + system-memory leaf entry in the format the client produces. +* Page sizes in use: 4 KiB (system memory objects) and larger (big pages) for + large allocations. +* **Apertures**: during enumeration the guest uses `ADDR_SYSMEM = 1` and + `ADDR_FBMEM = 2` to describe memory descriptors (e.g. channel instance + memory in VRAM at offset `0xffed000`, method buffers in system memory at a + physical address). The client hands the **physical** address in these + descriptors — a fact we rely on for the memory model. +* The page-table geometry arrives over + `NV90F1_CTRL_CMD_VASPACE_COPY_SERVER_RESERVED_PDES` (level physical + addresses, page shifts, VA range) — see + [`03-gsp-and-gsp-rm.md`](03-gsp-and-gsp-rm.md) §5. + +### 1.2 What "aperture" means for us + +Two address spaces matter: + +* **System memory (sysmem)**: guest RAM. For a virtual device, the "GPU's view" + of guest memory is established by the host (in real vGPU, by the vGPU manager + via the platform IOMMU; here, by our module). +* **Video memory (vidmem / FB)**: the GPU's local memory. In the current + emulation this is a host RAM buffer exposed as the BAR1 aperture; the + forwarding phase replaces it with real GPU memory. + +## 2. AMAPlib and the HALs + +NVIDIA's driver abstracts hardware in layers: + +``` +driver logic (memmgr, FIFO, engines) + │ + ├─ HAL layer: per-generation C functions that program registers and formats + │ e.g. gmmu HAL (page-table formats, invalidations), + │ engine HALs (CE/GR methods), host HAL (doorbells/PUT) + │ + └─ AMAPlib: the "address mapping" library shared by RM and simulations + (DRAM swizzling, bank/channel mapping, physical-to-aperture rules) +``` + +Developers study two component families to understand these layers: + +* **AMAPlib** — the shared "address mapping" component used by the RM and by + hardware simulations: DRAM swizzling, bank/channel mapping, aperture rules. +* **The HALs** — per-generation C code that programs registers and formats + page-table entries and engine commands. + +Our knowledge of them comes from reverse engineering and decompilation of +driver and firmware behaviour, plus measurement of live devices. What is +published here is our observations, numeric IDs and layouts — which is what +another engineer needs. + +Why we care: + +* When we **populate page tables ourselves** (server-owned ranges), we must + produce entries the hardware and the guest both accept — AMAPlib/HAL code is + the specification. +* When we **forward work**, page tables we write must match what real engines + expect, including big-page encodings and invalidation semantics. + +## 3. Channels, GPFIFO, USERD, doorbells + +A **channel** is a hardware execution context. Its components: + +| Piece | Where | Purpose | +|---|---|---| +| **GPFIFO** | GPU memory (sysmem or vidmem) | ring of entries; each points to a push buffer | +| **Push buffer** | GPU memory | the actual method stream the engine executes | +| **USERD** | vidmem (per channel) | channel control block: PUT/GET pointers, doorbell-related state | +| **Instance block** | vidmem or sysmem | the hardware context (engine state, page directory base, etc.) | +| **Runlist / TSG** | scheduled by the host | the list of channels currently eligible to run | + +The submission flow (what a driver does, and what the guest therefore does): + +``` +write methods into a push buffer (CPU, via a mapping) +write a GPFIFO entry pointing at the push (CPU) +update the channel's PUT pointer (CPU → USERD or doorbell register) + ... the scheduler eventually runs the channel; the engine fetches the push + ... on completion the engine writes the GET pointer and/or a semaphore +``` + +Two details that repeatedly matter in practice: + +* **The push buffer format** is a sequence of method/data pairs preceded by + *incrementing-method headers* that encode the method offset (`address >> 2`), + a subchannel and a count. A decoder for this format is the first tool you + need to read a trace (our `scan_seq` module parameter found UVM's pushes by + searching for the decoded data pattern). +* **Engine classes** are the vocabulary of pushes: `TURING_DMA_COPY_A` + (0xc5b5) for copy engines, `TURING_CHANNEL_GPFIFO_A` (0xc46f) for channel + objects, `GP100_UVM_SW` (0xc076) for UVM's software object, etc. + +## 4. Runlist and scheduling + +The GPU schedules channels through a **runlist**: a list of channel +descriptors the scheduler walks, giving each channel a time slice. A **TSG** +(Time-Slice Group) bundles channels that share a scheduling slot — this is the +mechanism for fair sharing and for interleaving contexts. + +For virtualization this is a focal point: + +* In real vGPU, the host vGPU manager owns the runlist and interleaves + guest channels with host channels. +* In our target architecture, *we* own the schedule: guest channels must be + mapped onto real GPU scheduling primitives with correct isolation (a guest + must never observe or influence another guest's channels). +* The guest driver observes scheduling only indirectly: PUT/GET progress, + semaphores, and fault timing. Faithful *semantics* (not exact timing) is the + requirement. + +## 5. GR — the graphics/compute engine + +**GR** is the engine that executes shaders: both graphics pipelines and CUDA +compute kernels. It is the reason a GPU is a GPU. + +What GR needs to run work: + +* **A context** (the "golden context" + per-context state): register state, + memory layout, page directory, shader code. +* **A channel** with a push buffer that issues *method* calls to GR: + loading shader code, binding surfaces, launching grids (`QMD`s in modern + architectures), and signaling completion. +* **Memory** containing the shader binary (SASS), kernel parameters, + descriptors and data. +* **Interrupts/semaphores** to signal completion and errors. + +For CUDA specifically: + +``` +cuLaunchKernel + → CUDA driver builds a GR push (grid config, QMD, parameters) + → submits on the process's GR channel + → GR executes the shader on real hardware + → completion semaphore / event; UVM handles memory effects +``` + +**Consequence for Open-IOV:** CUDA kernel execution is *real hardware +execution*. No emulation of GR can be correct and fast enough; the only viable +path is forwarding the guest's GR work to real hardware (see +[`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) and +the challenges document). This is also why "CUDA works" is a much stronger +statement than "`nvidia-smi` works". + +## 6. Copy engines (CE) and other engines + +* **CE (copy engines)** — DMA between memory regions; also used for UVM's + internal work (page tables, memsets, semaphores). UVM cannot even initialize + without working CE channels, which is why our CE capability reporting was + the first big unblock (`CE_GET_PHYSICAL_CAPS`, `CE_GET_CE_PCE_MASK`). +* **SEC2** — secure copy engine used by UVM for protected operations. +* **NVENC/NVDEC/NVJPG/OFA** — video and image engines; roadmap for graphics + and video phases. + +Engine types are enumerated by the client during init (`FIFO_GET_DEVICE_INFO_TABLE`) +and reported in engine/capability queries. A guest's channel is bound to one +engine type (e.g. `COPY0 = 0x9`, `GR = 0x1`). + +## 7. Practical reading list inside this repository + +| Topic | Where | +|---|---| +| Page-table formats and HAL concepts | this tree's architecture notes plus the decompiled analysis notes | +| Class/method definitions (classes, methods, notifications) | our decompilation notes and reformatted tables in `wiki/` | +| Channel allocation parameter structures | reformatted in the field guide and in the reference wiki pages | +| GMMU controls and static info | `wiki/Address-Reference.md`, `wiki/RPC-Message-Format.md` | +| Notification formats | the reference wiki pages | +| Firmware structure and dispatchers | `wiki/GSP-570.144W-Firmware-Analysis.md` | + +**Rule:** tracked documentation describes components and behaviour, with our +own observations, IDs and layouts. Exact analysis sources live in the +maintainers' study notes. diff --git a/docs/03-architecture/06-host-module-and-hybrid-model.md b/docs/03-architecture/06-host-module-and-hybrid-model.md new file mode 100644 index 0000000..fc4bb23 --- /dev/null +++ b/docs/03-architecture/06-host-module-and-hybrid-model.md @@ -0,0 +1,227 @@ +# The Host Module and the Hybrid Model + +> `open-iov-vfio.ko` in detail: the virtual device, the emulation engine, the +> memory model, and how work will be forwarded. This is the "where the code +> lives" document. + +## 1. Role and position + +``` + guest VM + │ PCIe MMIO / DMA / MSI-X + ┌────┴──────────────────────────────────────────────┐ + │ hypervisor (KVM/QEMU or Xen) │ + │ VFIO mdev transport, IOMMU (per-tenant memory) │ + └────┬──────────────────────────────────────────────┘ + │ + ┌────┴───────────────────────────────────────────────────────────────┐ + │ open-iov-vfio.ko │ + │ virtual device ── emulation core ── memory model ── work path │ + └────┬─────────────────────────────────────┬─────────────────────────┘ + │ host driver interfaces │ (future) vendor-neutral + ┌────┴────────────────────┐ │ + │ unmodified host nvidia.ko│ direct GPU interfaces / VF backends + └──────────────────────────┘ +``` + +The module registers itself as an **mdev parent** on the physical GPU and +exposes one device type per supported profile. Each `mdev` instance is an +independent virtual GPU with its own state, its own BAR windows, its own +interrupts and its own emulation context. + +## 2. Virtual device + +### 2.1 PCI configuration space + +The guest sees a plausible Turing-class GPU: vendor/device IDs, class code, +BARs, capability list (PCIe, MSI-X, power management), and the subsystem +identifiers the driver keys on. Several fields exist only because the driver +reads them for validation: + +* **PCIe link capabilities** — the guest's driver derives bus width/speed and + availability. The emulation must report them; the *hypervisor* can + accidentally rewrite them (see the debugging doc's root-port note). +* **MSI-X table/PBA** — emulated, with per-vector masking and pending bits. +* **VF (virtual function) windows** — registers the SR-IOV-aware driver paths + read during initialization; the emulation fills them with values consistent + with a non-SR-IOV or lightly-SR-IOV configuration as appropriate. + +### 2.2 BAR layout + +| BAR | Size | Backing | Notes | +|---|---|---|---| +| BAR0 | 16 MB | host `vzalloc` shadow | register interface: boot, doorbells, VF registers, PRAMIN window, VBIOS | +| BAR1 | 256 MB | host `vzalloc` vidmem buffer | guest-visible VRAM aperture; replaced by real VRAM backing in the forwarding phase | +| BAR3 | — | instance window | instance/PRAMIN behaviour | + +MMIO reads and writes are served either by the module's VFIO region callbacks +or directly from the shadow when the hypervisor has mapped it into the guest. + +### 2.3 VRAM backing + +Today BAR1 is host RAM. This is deliberate: correctness first, performance and +real VRAM semantics later. Consequences and their mitigations: + +* Guest reads/writes of "VRAM" are cheap and coherent. ✅ +* The real GPU cannot read it. The forwarding phase must either migrate these + objects to real VRAM or expose a DMA-mapped system-memory region to the GPU + (see §5). + +## 3. Emulation core + +### 3.1 GSP emulation thread + +A kernel thread drives the device's "firmware" behaviour: + +1. Watches BAR0 doorbells (and VF registers) for guest actions. +2. Reads message-queue elements from guest memory. +3. Dispatches RPCs to handlers. +4. Writes replies into the status queue. +5. Injects MSI-X interrupts to the guest. + +Guest memory is accessed through the hypervisor's DMA interface +(`vfio_dma_rw()`), which requires calls to originate from a kernel thread +context — a fact the debug tooling codifies. + +### 3.2 RPC handling + +Handlers are organized by function and, inside controls, by control ID. The +implementation philosophy: + +* **Correct shape first.** A reply must have the exact structure the driver + expects, or the driver walks off into uninitialized memory. +* **Plausible values second.** Values must be self-consistent (e.g. FB size, + BAR sizes, engine counts, page sizes) or later code paths compute nonsense. +* **Never echo garbage.** Queries that pass uninitialized input must receive + zeroed/valid output; echoing the request's stack garbage has caused real + failures. + +The handlers currently cover: the boot/static-info exchanges, the device/FB/ +BUS/INTR control tables, CE capability and PCE-mask queries, GMMU static info, +vaspace reserved-PDE contracts, channel/CE object allocation, and the general +"unknown control returns success with zeroed output" fallback. + +### 3.3 Event channels and interrupts + +Guest-visible interrupts (MSI-X) are delivered via `eventfd` to the hypervisor. +The emulation tracks vector masking and pending state so that an interrupt +raised while masked is delivered when unmasked, not lost. + +## 4. Memory model (the heart of the project) + +The guest's GPU work refers to *GPU virtual addresses*. To act on that work the +host module must translate them to memory it can touch. + +### 4.1 What the client gives us + +* **Channel memory descriptors** (`NV_MEMORY_DESC_PARAMS`) with *physical* + addresses: instance memory (VRAM offset), method buffer, userd, ramfc, + error notifiers. These are precise and trustworthy. +* **The page-table contract** for server-owned VA ranges + (`COPY_SERVER_RESERVED_PDES`): page-table level physical addresses, page + shifts, VA range — but with *empty* leaf levels, because the server owns the + leaves. +* **No mapping RPCs for the client's own system-memory objects.** The client + allocates sysmem locally and never tells the server its physical pages. + +### 4.2 What we derive instead + +Because the client does not hand us its sysmem mappings, the module derives +them from the guest's own in-memory bookkeeping: + +1. **VA↔PA pairs live in guest memory.** UVM and the RM keep structures that + record, for every mapped object, both the GPU VA and the kernel CPU VA. + Example (verified): `uvm_rm_mem_t.vas[CPU]` at +0x30 and `vas[GPU]` at + +0x38; channel tracking semaphores at `uvm_channel_t+0x80/0x88`. +2. **CPU VAs are direct-map addresses**, so + `PA = cpu_va - page_offset_base`. The direct-map base can be recovered from + guest memory itself (structures that store their own address), from + `/proc/kallsyms` during development, or from a debug poke session. +3. Therefore a translation table can be built **without any cooperation from + the guest driver** — a crucial property for adversarial interoperability, + and a general technique applicable to future versions. + +### 4.3 Acting on translations + +The module can read and write guest physical memory (via the hypervisor's DMA +interface). That is enough to: + +* complete semaphore releases (write the payload to the mapped slot), +* inspect push buffers and page tables, +* and later, create page tables for server-owned ranges. + +## 5. The forwarding path (design) + +``` +guest push submitted + → module detects submission (doorbell/GPPUT or push-buffer scan) + → decode the push (method stream) + → for each command decide: + (a) EMULATE: pure state/notification (semaphore release, membar, fence) + (b) FORWARD: real engine work (GR grid launch, CE copy, memop) + → for (b): submit equivalent work on a real channel bound to the physical GPU + → relay completions (semaphore/fault/interrupt) back into guest memory +``` + +Three forwarding backends are candidates, in increasing order of capability: + +1. **Host RM/GSP proxy** — use the unmodified host driver's RM to allocate real + memory and create real channels, then replay guest work through them. +2. **Direct engine submission** — build our own channel/push streams to real + engines using the GMMU/HAL definitions, bypassing the host driver. +3. **Hybrid** — proxy for management (memory, channels) and direct submission + for hot-path work (grid launches, copies). + +The decision between them is driven by the practical challenges document; the +short version is that (1) is safest for correctness, (2) is best for latency, +and both must be isolated to be acceptable. + +## 6. Debug tooling in the module + +The module exposes debug parameters (writes to +`/sys/module/open_iov_vfio/parameters/*`) that make the guest's memory and the +emulated device inspectable without rebuilding: + +| Parameter | Purpose | +|---|---| +| `scan_guest_ram=<v>` | find a 64-bit value in guest RAM (0–8 GB), report GPAs | +| `scan_vidmem=<v>` | same for the emulated VRAM shadow | +| `scan_seq=<v1,v2,...>` | find a 32-bit *sequence* (e.g. a method stream) | +| `scan_pt=<pa>` | find page-table entries pointing at a physical address | +| `scan_masked=<v>,<mask>` | find `(x & mask) == v` (e.g. a page-base VA) | +| `dump_vidmem=<off>,<len>` / `summary` | dump/scan the VRAM shadow | +| `poke_guest=<gpa>,<v>` | write a 32-bit value to guest physical memory | +| `poke_fill=<gpa>,<v>,<n>` | fill a range | +| `fill_vidmem=<v>` | fill the VRAM shadow | + +These are development-grade but production-useful: they are how the memory +model was reverse-engineered, and they remain the fastest way to diagnose a +mismatch. Their workflows are documented in +[`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md). + +## 7. Build and symbol contracts + +* `open-iov-vfio.ko` exports the `nvidia_vgpu_vfio_*` interface + (`nvidia_vgpu_vfio_probe/remove`, ops get/set, mdev parent registration) for + hosts that want to pair our device with the NVIDIA OpenRM host modules. +* When NVIDIA's OpenRM modules are built from source, they must be built with + `KBUILD_EXTRA_SYMBOLS=$(pwd)/kernel/Module.symvers`, and **after** the + Open-IOV module, or symbols will not resolve. +* On typical single-GPU developer systems the distribution's NVIDIA driver + manages the physical GPU and the Open-IOV module is used standalone; the + OpenRM submodule path is only needed for the native-vGPU-management + workflows. + +## 8. What is implemented vs. designed + +| Area | Status | +|---|---| +| mdev/VFIO device, config space, BARs, MSI-X | implemented | +| BAR0 register state machine, boot handshake | implemented | +| Message queues + RPC + all init controls observed from stock 570.144 | implemented | +| Guest sysmem access + VA↔PA discovery | implemented (debug-grade, to be productized) | +| Push decoding + semaphore completion | demonstrated live; being productized | +| Page-table population for server-owned ranges | designed, not implemented | +| GPU work forwarding (GR/CE execution) | designed, not implemented | +| Real VRAM backing | designed, not implemented | +| Windows guest, Xen transport, aarch64 hosts | designed, not implemented | diff --git a/docs/03-architecture/07-driver-lifecycle-field-guide.md b/docs/03-architecture/07-driver-lifecycle-field-guide.md new file mode 100644 index 0000000..8d80694 --- /dev/null +++ b/docs/03-architecture/07-driver-lifecycle-field-guide.md @@ -0,0 +1,188 @@ +# Driver Lifecycle Field Guide + +> A walking tour of what actually happens between "guest boots" and "CUDA +> initializes", with the real message and control IDs we observe from an +> unmodified 570.144 guest. Use this to interpret logs and to place new +> observations. + +## 0. How to use this document + +Each stage lists: what the guest does, what the device/firmware must answer, +what our emulation does today, and the log signature to look for +(`dmesg | grep gsp-resman-vfio` on the host; `dmesg` in the guest). + +The IDs are for the 570.144 generation on the Turing (TU104) profile. Other +generations will have additions and version differences; the *stages* are +stable. + +## 1. PCI probe and config space + +**Guest:** reads vendor/device/class IDs, walks capabilities, sizes BARs, +enables Memory Space and Bus Master, programs MSI-X. + +**Device must:** report a coherent TU104-family GPU with 16 MB BAR0, 256 MB +BAR1, an MSI-X capability, and a link capability the driver accepts. + +**Emulation:** config space is synthesized; BAR sizes are enforced; the +*hypervisor* is the usual source of surprises (a device attached to the root +bus can have its PCIe capability rewritten to "root complex integrated +endpoint", which removes link information the driver wants — attach behind a +root port). + +**Log signature:** none from us; guest-side `lspci -vvv` is the reference. + +## 2. GSP boot handshake + +**Guest:** writes WPR descriptors, releases FWSEC, then the booter, then the +RISC-V core; polls a status mailbox for "GSP init done". + +**Device must:** complete every stage in the order the driver polls, and leave +the mailbox in the state the driver expects. + +**Emulation:** `kernel/gsp_emulation.c` models the stage state machine and +auto-clears self-clearing triggers (e.g. MMU invalidate registers) so polls +terminate. + +**Log signature:** +``` +gsp-resman-vfio: GSP-BOOT: ... trigger auto-cleared ... +``` + +## 3. Static configuration + +**Guest:** `GET_GSP_STATIC_INFO` (65) → receives a large `GspStaticConfigInfo` +structure; reads the VBIOS image from the BAR0 window; queries device/FB/BUS +info. + +**Device must:** provide self-consistent sizes, engine inventories, memory +geometry, interrupt topology, and a VBIOS blob the driver can parse. + +**Emulation:** static structures are synthesized for TU104 (FB size, BAR +layout, engine list, vGPU-friendly VF values). + +**Log signature:** +``` +GSPRM: GET_GSP_STATIC_INFO → ... +``` + +## 4. Message queues + +**Guest:** sends `SET_GUEST_SYSTEM_INFO` (1) with libos regions, a queue +buffer description, and a page table (the queue buffer is +`NV_MEMORY_NONCONTIGUOUS`). It also sends `GSP_SET_SYSTEM_INFO` (72), +`SET_REGISTRY` (73) and `CONTINUATION_RECORD` (71) as init traffic. + +**Device must:** consume them silently (async) and start consuming the command +queue exactly in the order the guest writes elements. + +**Emulation:** parses the queue chain, builds the queue geometry, links the +status queue, and consumes elements one by one. **Element-based, not +byte-stream-based**: each message is one element and may span pages. + +**Log signature:** +``` +gsp-resman-vfio: libos region[i] id8=... pa=... size=... +gsp-resman-vfio: queue args sharedPA=... ptes=... cmdOff=... statOff=... +``` + +## 5. Resource-manager initialization (the control battery) + +**Guest:** sends `GSP_RM_CONTROL` (76) for every subsystem it initializes. +Representative controls (all observed, all answered with correct shapes): + +| Control | Meaning | What matters | +|---|---|---| +| `0x20800a40`-family | internal device info table | per-index values, entry count | +| `0x20801112` | FIFO device info table | engine inventory; count must match engine queries | +| `0x20800a5c` | interrupt kernel table | vector topology the driver maps | +| `0x20800a59` | GMMU static info | **fault buffer sizes** (replayable/legacy) — a wrong value breaks UVM early | +| `0x20800a01` | display static info | display count (headless values are fine) | +| `0x20801823/-a/b` | BUS/FB info V2 | sysmem connection type and PCIe link caps feed bandwidth calculations | +| `0x20802a07` | CE physical caps | per-engine support bits; UVM refuses to make channels without `sysmem` support | +| `0x20802a0a/0b` | all-CE caps | caps table for all engines | +| `0x20802a02` | CE PCE mask | one bit per engine | +| `0x20802a08` | CE fault method buffer size | method buffer sizing | +| `0x90f10106` | vaspace reserved-PDE contract | **carries page-table physical addresses** (see GSP-RM doc) | +| `0x20803039` | NVLink masks/VBIOS info | unpopulated but must be shaped | +| `0x20800a70` | BUS flush with sysmembar | UVM issues these frequently; success suffices | + +**Log signature:** +``` +GSPRM: GSP_RM_CONTROL cmd=0x... hClient=0x... hObject=0x... paramsSize=... +``` + +## 6. Object creation + +**Guest:** `GSP_RM_ALLOC` (103) creates the object tree. Typical order for a +CUDA process: + +``` +0x0 NV01_ROOT (client root, 120-byte params) +0x80 NV01_DEVICE_0 (56 bytes) +0x2080 NV20_SUBDEVICE_0 (4 bytes) +0x90f1 FERMI_VASPACE_A (48 bytes) ← the GPU address space +0xa06c KEPLER_CHANNEL_GROUP_A (20 bytes) ← CE channel group +0xc46f TURING_CHANNEL_GPFIFO_A(368 bytes) ← the channel (GPFIFO VA, instance/userd descriptors) +0xc076 GP100_UVM_SW (0 bytes) +0xc5b5 TURING_DMA_COPY_A (8 bytes) ← CE object +``` + +Channel parameters decode into: `gpFifoOffset` (a GPU VA), +`gpFifoEntries`, `engineType` (e.g. COPY0 = 0x9), and +`NV_MEMORY_DESC_PARAMS` for instance/userd/ramfc/method buffers with +**physical** addresses and apertures (`ADDR_SYSMEM = 1`, `ADDR_FBMEM = 2`). + +**Emulation:** handle management and parameter shapes are validated against +the reference structures; values are logged in a form that makes the object +tree reconstructible from `dmesg`. + +**Log signature:** +``` +GSPRM: GSP_RM_ALLOC class=0x... paramsSize=... hClient=... hParent=... hObject=... +``` + +## 7. UVM registration (the wall we climbed) + +**Guest:** `cuInit` → `UVM_REGISTER_GPU` → channel manager creation → +page-tree init. + +**Device must:** let UVM's channels initialize (each channel does a first push +and waits on a semaphore), and let page-tree initialization proceed. + +**Emulation (today):** we can decode the pushes UVM generates and complete +their semaphore releases by translating the semaphore VA to a guest physical +address (see the UVM and host-module docs). This is the live development front. + +**Log signatures (guest side):** +``` +nvidia-uvm: ... Tracker entry for value N (sema VA 0x...) channel ID ... CE ... +nvidia-uvm: ... pending push '' started at ... in channel_init() releasing value N +``` + +## 8. What comes after registration (roadmap) + +``` +UVM registers → cuInit returns → context creation → memory allocation +→ channel creation for the process → kernel launch → (real execution) +``` + +Each step will need the same treatment: satisfy the interface, forward the +work, relay the completions. The known-unknowns document lists what we expect +to be hard. + +## 9. Quick reference: message/control IDs seen from a stock 570.144 guest + +| Function | Name | Frequency per CUDA run (approx.) | +|---|---|---| +| 1 | SET_GUEST_SYSTEM_INFO | once | +| 10 | FREE | ~14 | +| 47 | UNLOADING_GUEST_DRIVER | on unload | +| 65 | GET_GSP_STATIC_INFO | once | +| 70 | UPDATE_BAR_PDE | a few | +| 71/72/73 | CONTINUATION_RECORD / GSP_SET_SYSTEM_INFO / SET_REGISTRY | once each | +| 76 | GSP_RM_CONTROL | ~160 | +| 103 | GSP_RM_ALLOC | ~60 | + +If a new driver version produces **new function numbers**, do not guess: add a +handler that logs the raw payload, then decode it against the decompiled +study notes for that driver version.
\ No newline at end of file diff --git a/docs/04-implementation/01-known-knowns-and-unknowns.md b/docs/04-implementation/01-known-knowns-and-unknowns.md new file mode 100644 index 0000000..0564ffc --- /dev/null +++ b/docs/04-implementation/01-known-knowns-and-unknowns.md @@ -0,0 +1,300 @@ +# Known-Knowns and Known-Unknowns + +> The most important document for planning work. It states, without +> flattery, what we know, what we know we do not know, and what we do not even +> know to ask yet — plus how each gap gets closed. + +## 0. How to read this + +| Category | Meaning | Our approach | +|---|---|---| +| **Known-knowns** | verified facts, with evidence in this repo | rely on them; keep them tested | +| **Known-unknowns** | gaps we have identified, with a concrete plan to close | schedule them, one by one | +| **Unknown-unknowns** | gaps we cannot yet name | discover them by systematic traversal + invariants + instrumentation | + +Anything listed here as a known-unknown has an owner-visible plan; anything +listed as an unknown-unknown has a *method* by which we will discover it. + +--- + +## 1. Known-knowns (verified) + +### 1.1 Device and platform + +* A stock 570.144 Linux guest driver probes and initializes against our + emulated TU104 device; `nvidia-smi` works. +* The same guest **boots unmodified**: no guest code, no guest module + parameters are required for the initialization path. +* The host is unmodified in the standalone configuration (distribution NVIDIA + driver manages the physical GPU); Open-IOV loads as an independent module. +* Multi-VM operation multiplies the same emulated device; each mdev is + independent. + +### 1.2 Protocol and firmware + +* The GSP boot state machine is emulatable by completing observable stages. +* The message queue transport is understood in detail: element header layout, + page-table translation of non-contiguous queue buffers, sequence matching, + lengths, and quirks (`length` excludes variable payloads). +* The RPC surface used during driver initialization is fully enumerated for + 570.144 (§9 of the field guide gives the ID table). +* Firmware images are analyzable: the 570.144 image is decompiled + (~130 functions of interest mapped to C) and its dispatcher/jump tables are + located. +* Version negotiation exists and is honored (the guest tells us its VGX/RPC + version; we answer with a matching profile). + +### 1.3 Memory and execution semantics + +* GPU VA → guest PA translation is recoverable **without guest cooperation** + by reading the guest's own bookkeeping (VA and CPU-VA pairs recorded in + UVM/RM structures; direct-map relationship for CPU VAs). +* Guest physical memory is readable and writable from the host module through + the hypervisor's DMA interface, from a kernel thread context. +* UVM's pushes are decodable (incrementing-method headers, engine classes, + `SET_SEMAPHORE_A/B/PAYLOAD`, `LAUNCH_DMA`), and completing their semaphore + releases advances UVM through channel manager creation and page-tree + initialization. +* The client's channel memory descriptors carry *physical* addresses we can + trust (instance/userd/ramfc/method buffers). +* The client's page-table contract for server-owned ranges is explicit + (`COPY_SERVER_RESERVED_PDES`) — and its leaf levels are *empty*, i.e. ours to + populate. + +### 1.4 Operational + +* Module reload, mdev create/destroy, VM start/stop, and guest reboot cycles + are stable when done in the documented order. +* Every layer of the emulation is instrumented and diagnosable from `dmesg` + plus a small set of module parameters. + +--- + +## 2. Known-unknowns (named gaps with plans) + +Each entry: **what we do not know**, **why it matters**, **how we will learn +it**, **rough cost**. + +### 2.1 Execution forwarding + +**Unknown:** the correct, isolated way to run a guest's engine work on the +physical GPU: which real scheduling primitive to use per engine, how to +translate guest pushes to real submissions, and how to relay completion and +fault semantics. + +**Why:** this is *the* blocker for CUDA kernel execution, and later for +graphics. + +**Plan:** evaluate the three backends (host RM/GSP proxy, direct engine +submission, hybrid) against a scorecard: correctness, isolation, latency, +maintenance cost across driver versions. Prototype the proxy first (it reuses +the unmodified host driver's knowledge), measure latency, then decide whether a +direct path is required. Deliverable: a forwarded `vectorAdd` with verified +results. + +**Cost:** weeks, not days; the largest single work item in the project. + +### 2.2 Server-owned page tables + +**Unknown:** the exact division of labour for page tables after +`COPY_SERVER_RESERVED_PDES`: which ranges the client will touch, when it +expects `RELEASE_ENTRIES`/`RESERVE_ENTRIES`, and how invalidations are +communicated. + +**Why:** mappings that only we can create must exist for the guest's own +memory to be reachable by engines. + +**Plan:** instrument all vaspace controls (add handlers that log their +parameters), then implement `RESERVE_ENTRIES`/`RELEASE_ENTRIES`/mirroring with +a test harness that maps memory and verifies the guest's expectations by +running its own validation paths (UVM page-tree init exercises this). + +**Cost:** days to weeks. + +### 2.3 Fault semantics + +**Unknown:** fault buffer formats and replay/cancel semantics as the *guest* +expects them in our virtualized setting (we know the buffer sizes and that +UVM programs them, but not yet the end-to-end contract under forwarding). + +**Why:** GPU page faults happen constantly in real CUDA workloads (UVM +oversubscription, migrations); correct handling is required for robustness. + +**Plan:** reproduce faults deliberately (map a VA to nothing, touch it from a +forwarded kernel), capture both sides, implement replay; validate with +`uvm_fault`-heavy tests (e.g. oversubscribed memory). + +**Cost:** weeks. + +### 2.4 Version matrix beyond 570.144 + +**Unknown:** which parts of the observed protocol drift between driver +families (550/560/580/610+), how much is negotiated, and how much must be +configuration. + +**Why:** durability is a core promise; users will run other versions +immediately. + +**Plan:** run the same init traces across 2–3 driver families in guests, +diff the message/control inventories, and refactor handlers to be +version-keyed where they differ. Deliverable: a capability matrix and a +documented "add a version" workflow. + +**Cost:** days per version, once the workflow exists. + +### 2.5 Windows guest specifics + +**Unknown:** the WDDM-facing differences (ROM/VBIOS expectations, MSI-X +semantics, power management, display topology) that the Windows driver +requires beyond the Linux path. + +**Why:** Windows CUDA/3D users are a first-class audience. + +**Plan:** bring up a Windows guest against the same device, capture the driver +logs (Windows ETW + our host traces), and add configuration for discovered +requirements. + +**Cost:** weeks. + +### 2.6 Non-KVM transports + +**Unknown:** what differs for Xen/Argo/OpenXT/Qubes OS Air: IRQ delivery +mechanisms, memory pinning, device-model integration points. + +**Why:** the security-focused platforms are a major part of the project's +reason for existing. + +**Plan:** abstract the transport boundary (IRQ injection, guest-memory access, +device registration) behind a small interface; implement a Xen backend against +it; test in a Xen guest. + +**Cost:** weeks. + +### 2.7 aarch64 hosts and Grace/GB10 + +**Unknown:** host-side differences (page sizes, cache attributes, SMMU +behaviour, power domains) and any firmware/ROM differences in the guest path. + +**Why:** Grace-Hopper/Grace-Blackwell/GB10 (DGX Spark, Surface-class laptops) +are explicit targets. + +**Plan:** make the emulation architecture-independent by construction +(no x86-specific assumptions in the memory model), then bring up on an +aarch64 host with the same test ladder. + +**Cost:** weeks per platform. + +### 2.8 Multi-tenant performance policy + +**Unknown:** best scheduling policy for sharing one GPU across VM tenants +(time slice, MIG partition, hybrid) and its fairness/isolation properties. + +**Why:** multi-tenancy is a core value proposition. + +**Plan:** build measurement first (per-tenant attribution), then implement +policy as configuration, then audit isolation. + +**Cost:** months (it is a research-y area). + +### 2.9 Tooling maturity + +**Unknown:** how much of the debug tooling (memory scanners, pokes, push +decoders) can be productized without exposing unsafe interfaces. + +**Why:** the tooling is currently how we make progress; it must not become a +security liability. + +**Plan:** move tooling behind a debug-only build flag/document its threat +model; keep safe, read-only diagnostics in production builds. + +**Cost:** days. + +--- + +## 3. Unknown-unknowns (and how we find them) + +Unknown-unknowns are discovered by *systematic traversal* and *invariants*, +not by luck. Our methods: + +### 3.1 Traversal by construction + +* **Enumerate every interface the guest touches.** For each: prove we answer + it or forward it. Any interface we have not seen is a discovery task, not a + bug. +* **Drive the guest further, always.** Each new guest-side milestone (probe → + nvidia-smi → cuInit → context → allocation → kernel) is a traversal step + that *forces* new interfaces to appear in the trace. +* **Differential testing across versions/hardware.** The same traversal on a + second driver version or GPU surfaces requirements that are invisible on + the first. + +### 3.2 Invariants that trip on unknowns + +We rely on the guest's own self-checks as an oracle: + +* UVM asserts on semaphore jumps, bad page tables, impossible capabilities. +* The RM validates structures and will fault on inconsistencies. +* Kernel drivers log "unexpected" states loudly. + +Every guest assert/log is a discovered unknown-unknown if we do not already +have it in this document. + +### 3.3 Structural instrumentation + +* **Trace-first policy:** no handler is written without logging the request + *and* the reply in a reconstructible form. +* **Reconstruct from dmesg:** the logs must be sufficient to rebuild the + object tree, memory map and push stream offline. This is how protocol drift + is detected early, before it becomes a crash. + +### 3.4 Fuzzing and negative tests + +* Feed the emulation malformed/partial guest actions (truncated elements, + bad sequence numbers, out-of-range handles) and require graceful + behaviour — this finds assumptions we did not know we had. +* Feed the *guest* plausible-but-wrong device behaviour in a lab profile and + verify the guest recovers (drivers do error recovery we must not break). + +### 3.5 Failure archaeology + +Every failure leaves artifacts. The discipline is: capture host `dmesg`, +guest `dmesg`, guest PCI config, and the emulation's per-RPC log; then write +the *decision tree* that leads to the failure into the debugging docs. The +tree is the artifact that survives personnel changes. + +--- + +## 4. What we need to know to declare "CUDA works" + +To make a defensible statement, we need all of: + +1. **Interfaces:** every guest→device message for the CUDA path is either + answered or forwarded, for the supported driver family. +2. **Memory:** every GPU VA used by work is translatable, and every server-owned + page-table range is populated correctly. +3. **Execution:** grid launches and copies run on real hardware with correct + results, repeatedly, with errors surfaced. +4. **Completion:** semaphores, events and faults behave as the guest expects + (values advance correctly, no lost interrupts, no stuck waits). +5. **Version:** at least two driver families pass, with a documented path for + more. +6. **Isolation:** a second tenant cannot observe or influence the first. +7. **Performance:** within an agreed factor of native for the supported cases, + measured with a published methodology. + +Until all seven hold, we say "initialization works; execution is in progress", +and we link here. + +--- + +## 5. Practical challenges (summary; details in the challenges doc) + +| Challenge | Why it is hard | Mitigation | +|---|---|---| +| Real execution | emulating GR is hopeless; forwarding must be correct and isolated | proxy-first, measure, iterate | +| Page tables both sides | guest's and hardware's expectations must agree | study AMAP/HAL definitions; build a page-table validator | +| Latency | RPC round-trips and push translation add overhead | batch, fast-path pure state in the emulation | +| Version drift | protocol and structures evolve | version-keyed handlers, config-driven profiles | +| Security | forwarding must not weaken isolation | IOMMU-only access, per-tenant contexts, audits | +| Debuggability | most bugs are in the guest's head, not ours | trace-first, reconstruct-from-logs policy | +| Test cost | real GPU time and guest boots are slow | scripted ladders, keep artifacts, parallelize |
\ No newline at end of file diff --git a/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md b/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md new file mode 100644 index 0000000..1deff02 --- /dev/null +++ b/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md @@ -0,0 +1,156 @@ +# Practical Challenges and Adversarial Interoperability + +> How we intend to implement compute/CUDA support *fast and correctly*, and the +> discipline that keeps the whole stack unmodified — ours included. + +## 1. The central engineering bet + +> Emulate the interface. Pass through the work. Never patch anyone. + +Everything below follows from taking that seriously. It forces hard choices +(no "just add a flag to the guest driver"), and it pays for itself by making +the project portable, maintainable and legally clean. + +## 2. Fast and correct: the tension + +### 2.1 What "correct" means here + +Correctness is not "no crash". It is: + +* the guest observes the same **values** (capabilities, handles, table + contents) it would on real hardware, +* the guest observes the same **ordering** (completions happen after the work + they signal; faults arrive before replays), +* the guest observes the same **failure semantics** (errors are surfaced, not + swallowed), +* and the work produces the same **results** (a CUDA kernel's output matches + native execution). + +### 2.2 What "fast" means here + +Two budgets matter, and they are different: + +* **Initialization latency** — how long from VM start to CUDA ready. Dominated + by RPC round-trips and guest driver's own work. Target: within 2× of physical + device initialization in the same guest. +* **Steady-state throughput/latency** — kernel launch overhead, copy + bandwidth, and completion latency. Target: published, stable, and within an + agreed factor of native for each supported operation class. + +We will publish the methodology and the numbers together; a performance claim +without a methodology is not accepted in this project. + +### 2.3 The fast paths + +* **Pure state operations** (semaphores, membar, fences, some memops) are + satisfied in the emulation directly — they are microseconds in hardware and + would otherwise cost a round trip to the physical GPU and back. +* **Management operations** (object creation, capability queries, mappings) + are answered from cached/derived state where the hardware has no side effects + the guest can observe. +* **Real work** (grid launches, copies) is forwarded. Nothing else is allowed + on the hot path. + +## 3. Forwarding: the three options and how we choose + +``` + (A) Host RM/GSP proxy + guest push → translate → host driver's RM API → real channel → GPU + + reuses a mature driver's knowledge + − extra hop, depends on host driver interfaces + + (B) Direct engine submission + guest push → translate → our own channel/push construction → GPU + + lowest latency, no host driver dependency + − we must own page tables, scheduling, invalidation, completion + + (C) Hybrid + management via (A), hot path via (B) +``` + +**Decision procedure:** prototype (A) first to get *correct* execution, with +measurement harnesses around it. Then measure the overhead. If the overhead +violates the budgets in §2.2 for the target workloads, implement (B) for the +hot path and keep (A) for management. The architecture must not preclude +either, so the work path is abstracted behind an interface from day one. + +## 4. Hard problems we must solve, and our stance on each + +| Problem | Stance | +|---|---| +| **Page tables on both sides** | We populate server-owned ranges using the layout the client hands us and the entry encodings from AMAP/HAL references; a validator compares our tables against what the guest expects before enabling a channel. | +| **Memory coherence** | Guest sysmem is coherent by construction (same physical pages); VRAM objects must be migrated to real VRAM or DMA-mapped. Phase 2 picks per-object policy with correctness first. | +| **Scheduling between tenants** | One real GPU, N guest channels — schedule explicitly (time-slice/partition), measure fairness, audit isolation. Never let the guest's own scheduler influence another tenant. | +| **Fault handling** | Reproduce faults under forwarding, implement replay/cancel on the guest's terms, and make fault latency measurable. | +| **State explosion** | Drivers create enormous state. We do not model all of it — we model what the guest *observes*, and forward what it *uses*. Anything unobserved stays unmodeled. | +| **Version drift** | Version-keyed handlers with a configuration-driven profile per (driver, firmware) family; new versions added by measurement, never by guessing. | +| **Security of the debug tooling** | Debug parameters are development-grade; the productized build keeps only read-only diagnostics and gates destructive ones behind explicit opt-in. | + +## 5. Adversarial interoperability + +### 5.1 The rule + +**No modification is permitted in any component we do not own**: guest driver, +host driver, GPU firmware, hypervisor, device model, guest OS. + +Why so absolute: every permitted exception becomes a maintenance liability that +scales with the number of versions in the field. A patch to the guest driver +means maintaining forks for every guest OS and driver version. A patch to the +hypervisor means every hypervisor project must carry it. A firmware change is +impossible in practice. + +### 5.2 How compliance is verified + +* **Contract tests**: CI builds run against *stock* drivers obtained from + distribution repositories (or vendor runfiles), and against *unmodified* + hypervisor packages. +* **No-patch checks**: our build must not depend on any patch file for + third-party components. Where patches exist in the tree for convenience + (e.g. build glue), the contract tests must pass without them; anything that + fails this is a bug. +* **Behavioral equivalence**: where we answer from emulation, we validate the + values against traces from real hardware (or from our decompiled study notes) — + not against our own previous answers. + +### 5.3 What we *can* change + +* Our own module, userspace library, tools, scripts and docs. +* Configuration: device profiles, version profiles, capability tables. +* Tests and harnesses. + +### 5.4 Where the line gets tested + +* **Firmware signatures**: the driver verifies firmware. We satisfy the + handshake by *emulating the interface*, not by patching the image or the + driver. +* **PCI IDs and VBIOS**: the guest driver decides how to treat the device from + what it reads. We present a coherent device; we never rely on the driver + skipping checks. +* **Licensing checks**: our device is our device; we do not impersonate a + licensed product. Interoperability is the goal, not circumvention of a + vendor's own licensing for their own products. + +## 6. Why this approach is durable + +Because our artifacts are: + +* **Interface definitions** (protocol tables, structures) — survive versions. +* **Measured contracts** (what the guest observes) — survive reimplementation. +* **Configuration** (profiles per version/hardware) — survive hardware. +* **Tests** (contract ladders) — catch drift immediately. + +A new driver version or GPU is therefore *data + verification*, not a rewrite — +the difference between a hobby project and infrastructure. + +## 7. The "thrilling" part (for developers) + +If you are reading this to decide whether to work on it: this project is one of +the few places where you can: + +* reverse engineer a 28 MB RISC-V firmware and turn it into maintainable C, +* design a memory model that makes two different address-space owners agree, +* build a virtualization path for the most commercially important accelerator + on Earth, under a permissive license, +* and have your work outlive the hardware generation you started on. + +Start with [`../07-developers/onboarding-and-standards.md`](../07-developers/onboarding-and-standards.md). diff --git a/docs/05-operations/01-performance-correctness-security.md b/docs/05-operations/01-performance-correctness-security.md new file mode 100644 index 0000000..c4d8153 --- /dev/null +++ b/docs/05-operations/01-performance-correctness-security.md @@ -0,0 +1,140 @@ +# Performance, Correctness, and Security Evaluation + +> How we measure whether Open-IOV is any good — and how auditors can verify the +> claims. Numbers without methodology are not accepted here. + +## 1. The evaluation ladder + +Every platform (hardware + driver + hypervisor + guest) is evaluated through +the same five-rung ladder. Each rung is a gate: you do not claim the next one +until the previous one is green and reproducible. + +``` +Rung 1 INITIALIZATION guest driver probes; nvidia-smi; device enumeration +Rung 2 RUNTIME START cuInit; context; allocation; UVM registration +Rung 3 EXECUTION kernels produce correct results; copies verified +Rung 4 SUSTAINED workloads run for hours; faults/errors recoverable +Rung 5 MULTI-TENANT N VMs, isolation + fairness + no cross-talk +``` + +## 2. Correctness + +### 2.1 Test corpus + +| Layer | Tests | +|---|---| +| Device | PCI config validation, BAR sizing, MSI-X semantics, power-management stubs | +| Driver init | guest `nvidia.ko` probe, `nvidia-smi` full query, `deviceQuery` | +| CUDA | `vectorAdd`, reduction, shared-memory kernels, memcpy patterns, streams/events, error injection | +| UVM | oversubscribed memory (fault path), unified memory migration, concurrency | +| Graphics (later) | GLXInfo/VulkanInfo, rendering correctness screenshots | +| Stability | start/stop cycles ×100, VM kill during work, guest reboot in flight | + +### 2.2 Verification methods + +1. **Golden results**: kernel outputs compared against native execution on the + same GPU, byte-for-byte, for deterministic kernels. +2. **Guest self-checks**: the guest's own validation paths are the strongest + signal (UVM asserts, RM validation, CUDA runtime self-tests). Every guest + diagnostic that fires is a test failure for us, even if the workload + continues. +3. **Differential runs**: same workload on (a) native, (b) our device — diff + observables, not just outputs: semaphore value sequences, PUT/GET progress, + interrupt counts. +4. **Negative testing**: inject malformed device behaviour in a lab profile and + require the guest driver to recover exactly as it would on hardware with a + transient error. + +### 2.3 Regression gates + +`make test` runs the host suite. The CI ladder runs the guest suite on a +scheduled cadence and on every release. A rung regression is a release blocker; +the handoff documents carry the current rung. + +## 3. Performance + +### 3.1 Metrics + +| Metric | Definition | Why it matters | +|---|---|---| +| T_init | VM boot → `cuInit` returns | user-visible startup | +| RPC latency p50/p99 | request write → reply visible in status queue | init and management overhead | +| Push overhead | guest submission → engine start (forwarded) | steady-state latency | +| Semaphore completion latency | release written → tracker observes | per-operation sync cost | +| Copy bandwidth | H2D/D2H/D2D, large and small | data-movement quality | +| Kernel throughput | kernels/second for a fixed workload | launch overhead | +| Multi-tenant fairness | per-tenant progress under contention | isolation quality | + +### 3.2 Method + +* **Same-guest baselines**: measure native GPU performance *in the same VM with + a passthrough device* where possible; otherwise on the same hardware with the + same driver. Comparisons across hosts are noise. +* **Warm and cold**: report both; report variance; report the number of runs. +* **Instrument everything**: the emulation timestamps requests and replies + (tracepoints, `trace_printk`, or counters) so overhead is attributable to a + component, not to a mystery. +* **Publish the harness**: the benchmark scripts live in `tools/`, and a + result is not comparable unless produced by them (or an equivalent, + documented harness). + +### 3.3 Targets + +* Initialization: within 2× of native, same guest. +* Synchronization-heavy operations: within an agreed constant factor, published + per operation class. +* Bulk copies and sustained compute: as close to native as the forwarding path + allows; the forwarding design (proxy vs direct) is decided by measurement. + +## 4. Security and isolation + +### 4.1 Threat model + +| Adversary | Capability | Boundary that must hold | +|---|---|---| +| Guest VM (tenant) | full control of its own driver, MMIO, DMA | cannot touch host or other tenants' memory/state | +| Malicious guest driver | arbitrary device programming | emulation must validate everything, fail closed | +| Host user (unprivileged) | can access `/dev/*` permissions given | no privilege escalation via our interfaces | +| Adjacent tenant | shares the GPU | cannot observe timing/state/interrupts beyond policy | +| Supply chain | modified driver/firmware | measured boot (TrenchBoot) verifies the platform | + +### 4.2 Enforcement + +* **Memory isolation is the platform IOMMU's job, and we never weaken it.** + The device performs DMA only through the hypervisor-provided DMA interface + for its own VM. No global mappings, no "identity for convenience". +* **Per-instance state**: every mdev has its own BAR shadows, queues, + interrupt state and emulation context. No shared mutable state between + tenants in the host module. +* **Fail closed**: if a required isolation property cannot be established + (IOMMU absent, DMA mapping failure, inconsistent state), the device refuses + to start and says why in `dmesg`. +* **Interrupts**: per-instance vectors; masked interrupts are queued in the + emulated PBA, not dropped into another tenant's vector. +* **Debug surfaces**: destructive debug parameters are opt-in, documented, and + excluded from hardened builds. Reads are side-effect-free. +* **No secrets in the module**: nothing about one tenant's workload is stored + where another tenant could reach it. + +### 4.3 Auditing + +* The security-relevant code paths are small and enumerated in this document + tree: device registration, DMA access, interrupt injection, memory + translation, and the debug surface. Auditors should be able to enumerate + them mechanically (`grep` for the gate points). +* Each release produces a checklist: threat model reviewed, fail-closed paths + exercised by tests, debug surface verified gated, dependency licenses + unchanged. +* We explicitly invite adversarial review — including of the emulation's + response to malformed guest input. + +## 5. Reporting + +Each release publishes: + +1. **Rung achieved** per platform, with logs. +2. **Performance table** (metrics above) with methodology and variance. +3. **Known issues**, linked to the known-unknowns document. +4. **Security checklist** status. + +This is the only accepted form of a claim in this project. diff --git a/docs/05-operations/02-debugging-and-maintenance-workflows.md b/docs/05-operations/02-debugging-and-maintenance-workflows.md new file mode 100644 index 0000000..bc38a4e --- /dev/null +++ b/docs/05-operations/02-debugging-and-maintenance-workflows.md @@ -0,0 +1,157 @@ +# Debugging and Maintenance Workflows + +> The practical book: how to see what is happening, how to find the cause, and +> how to add a new firmware, driver, hypervisor or hardware version without +> breaking anything. + +## 1. The observability model + +Three layers, in order of usefulness: + +1. **Guest diagnostics** — the guest driver is the best oracle. It prints + precisely what it expected and what it got (e.g. UVM's "Tracker entry for + value N … channel_init()"). Always start here. +2. **Host module trace** — every RPC and every emulation decision is logged + with the `gsp-resman-vfio:` prefix. A full CUDA run produces a few hundred + lines that reconstruct the object tree and the control battery. +3. **Memory observation** — when the problem is in memory state (mappings, + semaphores, pushes), the module's debug parameters read and write guest + physical memory directly. + +## 2. Standard captures for a bug report + +```bash +# Host +sudo dmesg -T | grep -iE 'gsp-resman-vfio|gsp_|nvidia|vfio|mdev' > host.log +./scripts/stack_manager.sh status > status.log +# Guest +sudo dmesg > guest-dmesg.log # include full dmesg +sudo lspci -vvv -d 10de: > guest-lspci.log # config space of the virtual GPU +cat /proc/cmdline > guest-cmdline.log +nvidia-smi -q > nvidia-smi-q.log +``` + +Plus: the emulated driver profile (GPU type, firmware image hash, driver +version) and the exact command sequence used. `tools/guest-diag/push_guest_tools.sh` +collects some of this automatically. + +## 3. Live debugging toolbox + +### 3.1 Kernel logs + +```bash +sudo dmesg -w | grep -iE 'gsp-resman-vfio' # live emulation trace +sudo dmesg -T | grep gsp-resman-vfio | tail -100 # recent +# capture to a file when the ring buffer is small and traffic is heavy: +sudo stdbuf -oL dmesg -w > run.log +``` + +### 3.2 Module debug parameters + +See the host-module doc for the full list. Canonical examples: + +```bash +# Find where a value lives in guest RAM (e.g. a semaphore VA): +echo 0x121000000 | sudo tee /sys/module/open_iov_vfio/parameters/scan_guest_ram + +# Find a decoded method sequence in guest RAM: +echo 0x1,0x21000000,0x1 | sudo tee /sys/module/open_iov_vfio/parameters/scan_seq + +# Write a value to a guest physical address (completing a semaphore release): +echo 0x120299010,0x1 | sudo tee /sys/module/open_iov_vfio/parameters/poke_guest +``` + +**Caution:** all guest-RAM operations run in a kernel thread context by design; +calling them from a normal process context fails with `-EFAULT` (the +hypervisor's DMA interface requires either the mapping's `mm` or no `mm` at +all). This is not a bug. + +### 3.3 QEMU gdbstub (guest virtual memory) + +The reference QEMU configuration exposes a gdbstub (`-gdb tcp::1234`). It can +read guest *virtual* memory through the vCPU's page tables and write to it — +the single most powerful trick we have when the guest's own kernel address +matters: + +```bash +gdb -batch -ex 'target remote :1234' \ + -ex 'x/8wx 0xffff...' -ex 'set *(unsigned int*)0xffff... = 1' +``` + +Workflow that made the memory model tractable: + +1. Find the guest's `page_offset_base` (direct-map base) from + `/proc/kallsyms` in the guest. +2. Convert a kernel CPU VA to a physical address: `PA = VA - page_offset_base`. +3. Use `poke_guest` to write the physical address from the host module, or use + the gdbstub to write the virtual address directly. + +### 3.4 Guest-side tracing + +* `ptrace` tracer (`tools/guest-diag/ptrace_ioctl.c`): libcuda issues raw + syscalls, so `LD_PRELOAD` shims see nothing; PTRACE_SYSCALL with + `PTRACE_GET_SYSCALL_INFO` does. +* UVM debug: `echo 1 > /sys/module/nvidia_uvm/parameters/uvm_debug_prints` + (messages appear as `nvidia-uvm:`). +* kprobes do not work on NVIDIA text (`notrace`); do not waste time on them. + +### 3.5 Failure decision trees + +When a failure is understood, write the decision tree into this document tree +— the tree is what makes the next failure cheap. The field guide's tables are +the result of that discipline so far. + +## 4. Workflows for new inputs + +### 4.1 New firmware version + +1. Add the image under `firmware/<SERIES>-GSP/<VERSION>/` (tracked by LFS). +2. Compute signatures (CRC32/SHA256) with the firmware analyzer. +3. Add a firmware profile entry (addresses/signatures) so runtime analysis can + identify it. +4. Run the driver against a guest using that firmware's matching driver; diff + the RPC/control inventory against the baseline. +5. Add/adjust handlers only where the diff requires it; re-run the ladder. + +### 4.2 New guest driver version + +1. Install the stock driver in a guest (no patches). +2. Capture the full init trace and diff function/control IDs against the + previous version. +3. For each difference: decode against our decompiled study notes for that version + (`g_rpc-structures.h`, `rpc_global_enums.h`, control headers). +4. Add a version-keyed handler or a capability table entry. +5. Re-run rungs 1–2; then the CUDA corpus. + +### 4.3 New hardware generation + +1. Add a hardware profile (PCI IDs, BAR layout, memory geometry, engine list). +2. Run the ladder on that hardware with the closest driver family. +3. Fix differences by configuration first; only add code when an interface + genuinely differs. +4. Publish the profile and the test evidence. + +### 4.4 New hypervisor + +1. Implement the transport boundary (device registration, guest-memory access, + interrupt injection) — the boundary is small by design. +2. Re-run the ladder; the emulation core must not need changes. +3. Document any hypervisor-specific quirk in the debugging docs. + +## 5. Crash and hang recovery + +| Situation | Action | +|---|---| +| Module stuck unloading | `stack_manager.sh cleanup-crash`; reboot if it persists (request approval per project policy) | +| VM wedged, guest filesystem dirty | stop the VM gracefully (`scripts/vgpu_vm.py stop`), fsck via `qemu-nbd` only if needed | +| Guest driver wedged (D-state processes) | reboot the guest; capture dmesg first | +| Kernel oops in the host module | capture the full oops + `dmesg` before reloading | + +## 6. Release engineering + +* Versions are tagged; each tag carries the rung/matrix table and the + performance table for the platforms tested. +* Configuration profiles are versioned with the code; a profile bump is a + reviewed change with evidence. +* The docs in this tree are part of the release: a release that changes + behaviour without updating them is incomplete. diff --git a/docs/06-portability/hardware-software-matrix.md b/docs/06-portability/hardware-software-matrix.md new file mode 100644 index 0000000..442e099 --- /dev/null +++ b/docs/06-portability/hardware-software-matrix.md @@ -0,0 +1,116 @@ +# Hardware and Software Matrix + +> What Open-IOV runs on, what it targets, and exactly how support for a new +> hardware or software combination is added. Support here means *tested and +> documented*, not "should probably work". + +## 1. Support ladder + +| Level | Meaning | +|---|---| +| **Reference** | continuously tested; bugs block releases | +| **Supported** | tested per release; documented workarounds if any | +| **Experimental** | known to boot/test partially; may need configuration | +| **Target** | architecture designed for it; bring-up in progress | +| **Out of scope** | not planned | + +## 2. Host platforms + +| Platform | Status | Notes | +|---|---|---| +| x86_64 + NVIDIA Turing (TU104) | **Reference** | the current development hardware (RTX 2080-class) | +| x86_64 + Ampere / Ada consumer | **Target** | profile work; expected to be configuration + measurement | +| x86_64 + Hopper / datacenter | **Target** | requires MIG-aware policy for multi-tenancy | +| aarch64 + Grace-Hopper (GH200) | **Target** | host-architecture work in the memory model | +| aarch64 + Grace-Blackwell | **Target** | as above, newer firmware/driver family | +| GB10 (DGX Spark, Surface-class RTX laptops) | **Target** | same architecture; firmware/ROM quirks to be measured | +| Other NVIDIA generations | **Experimental** | bring-up path documented in §5 | + +The defining property we are building toward: **each new host platform should +require configuration + verification, not new code.** + +## 3. Guest operating systems + +| Guest | Status | +|---|---| +| Linux x86_64 (stock `nvidia.ko`, CUDA) | **Reference** (initialization; execution in progress) | +| Linux aarch64 | **Target** | +| Windows x86_64 (WDDM driver, CUDA, 3D) | **Target** | +| Other Unix (FreeBSD, ...) | Out of scope until demand | + +**Guest rule:** the guest never contains Open-IOV code. No guest agent, no +driver patch, no kernel parameters beyond what the stock driver needs. + +## 4. Hypervisors and transports + +| Hypervisor | Status | Transport | +|---|---|---| +| KVM + QEMU (VFIO mdev) | **Reference** | VFIO mdev + eventfd IRQ + DMA interface | +| libvirt (mdev attach) | **Supported** | same, via libvirt XML | +| Zen / KVM on other device models | **Experimental** | as above | +| Xen (incl. Argo/OpenXT) | **Target** | Xen device model + event channels | +| Qubes OS / Qubes OS Air | **Target** | compartmentalized VMs; GPU per compartment | +| Cloud hypervisor / crosvm | **Experimental** | VFIO-based; needs testing | + +The transport boundary (how the device registers with the hypervisor, how +guest memory is accessed, how interrupts are injected) is deliberately narrow +so new transports are a small, isolated implementation. + +## 5. Adding a new combination + +### 5.1 New GPU / hardware profile + +1. Create a hardware profile: PCI IDs, BAR layout and sizes, memory geometry + (FB size, BAR windows), engine inventory, interrupt topology, and the + register values the guest driver validates during probe. +2. Run the evaluation ladder (see + [`../05-operations/01-performance-correctness-security.md`](../05-operations/01-performance-correctness-security.md)). +3. Record the differences; prefer configuration over code. +4. Publish the profile, the evidence, and any workarounds. + +### 5.2 New firmware version + +See [`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md) §4.1. +Firmware images are versioned under `firmware/<SERIES>-GSP/<VERSION>/` with +signatures and analysis artifacts. + +### 5.3 New driver version + +See the same document §4.2. The expected work per version is: run the trace, +diff the inventory, decode the differences against that version's reference +tree, add configuration/handlers, re-run the ladder. + +## 6. Version matrix (maintained in the repository) + +The authoritative matrix lives in `config/`: + +``` +config/ + hardware/<arch>.yaml # PCI IDs, BARs, memory geometry, engine lists + versions/<driver>.yaml # RPC/control inventories, capability tables + profiles/<name>.yaml # (hardware, driver, firmware) combinations we test +``` + +A profile is the unit of support: "hardware X + driver family Y + firmware Z = +profile P, tested at rung R on date D." Releases publish the profile table. + +## 7. Architecture properties that enable portability + +| Property | Why it matters | +|---|---| +| No architecture-specific assumptions in the memory model | x86_64 and aarch64 hosts share the same code | +| RPC/handler layer keyed by version | new driver families are additive | +| Hardware constants in profiles | new GPUs are data | +| Transport behind a narrow interface | KVM today, Xen/Argo/Qubes tomorrow | +| Guest-independent design | Windows and Linux guests share the same host code | +| Firmware analysis toolchain is version-generic | new firmware is analyzed with the same scripts | + +## 8. Known portability risks + +| Risk | Mitigation | +|---|---| +| Page-size/cache-attribute differences on aarch64 | keep memory model abstract; test on Grace early | +| Windows driver ROM/VBIOS expectations | explicit ROM/VBIOS provisioning in the device profile; measured bring-up | +| Xen interrupt/memory interfaces differ | narrow transport interface + Xen backend plan | +| Vendor ROMs and laptop firmware variants (GB10/Surface) | profile system supports per-device overrides | +| Driver-family drift (580/610+) | version-keyed handlers; matrix-driven testing | diff --git a/docs/07-developers/onboarding-and-standards.md b/docs/07-developers/onboarding-and-standards.md new file mode 100644 index 0000000..78503f4 --- /dev/null +++ b/docs/07-developers/onboarding-and-standards.md @@ -0,0 +1,163 @@ +# Developer Onboarding, Standards, and Tools + +> Everything a new contributor needs: what to read, how to build and test, how +> we write code, and how to land a change. + +## 1. Your first day + +### 1.1 Reading order (about 3 hours) + +1. [`../README.md`](../README.md) — the project front door (20 min). +2. [`../01-orientation/01-purpose-and-scope.md`](../01-orientation/01-purpose-and-scope.md) + and [`../01-orientation/02-the-ai-edge-and-ecosystem.md`](../01-orientation/02-the-ai-edge-and-ecosystem.md) + — why the project exists (30 min). +3. [`../03-architecture/01-system-overview.md`](../03-architecture/01-system-overview.md) + — the architecture in one document (30 min). +4. [`../03-architecture/06-host-module-and-hybrid-model.md`](../03-architecture/06-host-module-and-hybrid-model.md) + — where the code lives (30 min). +5. [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) + — what to work on and what is hard (30 min). +6. The subsystem doc for whatever you plan to touch. + +### 1.2 Environment + +```bash +git clone https://git.vgpu.io/open-iov.git +cd open-iov +./install.sh deps nvidia firmware build # full setup +./scripts/stack_manager.sh up # + load + create mdev +./scripts/stack_manager.sh test # host test suite +``` + +You will need: a supported NVIDIA GPU, IOMMU-enabled host, hugepages, kernel +headers, QEMU, and (for guest work) a Linux guest with the stock driver. + +### 1.3 The daily loop + +```bash +make kernel # kernel module (generates Module.symvers) +./scripts/stack_manager.sh build-kernel +./scripts/stack_manager.sh load-open-iov +./scripts/stack_manager.sh mdev-create +./scripts/vgpu_vm.py start vgpu-linux +# guest work ... +./scripts/stack_manager.sh dmesg # live trace +``` + +When you change kernel code, remember: reload requires removing the mdev and +the VM holding it. The stack manager enforces the order. + +## 2. Repository map + +``` +open-iov/ +├── kernel/ host module (GSP emulation, device model, memory model) +├── userspace/ management library + examples +├── tools/ debugging and analysis tools (guest-diag, rmq, +│ firmware analysis wrappers) +├── scripts/ build/load/test automation, VM harness +├── config/ hardware/version profiles (the portability surface) +├── firmware/ versioned firmware images + decompilation artifacts +├── (private) reverse-engineering study notes live untracked; +│ see the maintainers' study index +├── docs/ THIS documentation tree (start at docs/README.md) +└── wiki/ long-form firmware/protocol deep dives +``` + +## 3. Coding standards + +### 3.1 Kernel module (`kernel/`) + +* Linux kernel C style (tabs, 80-column-ish, `checkpatch` clean where + practical). +* **No comments explaining *what* the code does — comments explain *why*, + especially protocol quirks.** Our code encodes observations from hardware; + the observation must be recorded next to the code that depends on it. +* Every handler logs its request/reply in a reconstructible form (the + trace-first policy). A handler that silently succeeds is an anti-pattern. +* Never trust guest input: validate sizes, handles, indices, and sequences. + A malformed element must not be able to steer a write outside guest memory. +* Memory: `kmalloc`/`vzalloc` per the use; DMA access through the hypervisor + interface only. +* Error paths: fail closed, log once with context, and give up the device + cleanly. + +### 3.2 Userspace and tools (`userspace/`, `tools/`, `scripts/`) + +* C: same style as the kernel module where practical. +* Python/shell: `shellcheck` clean shell, `python3` with no exotic + dependencies; scripts must be re-runnable and idempotent. +* Tools must live **in-tree** (never in `/tmp`) so they survive reboots and + are reviewable. + +### 3.3 Documentation + +* Every behaviour change updates the doc that describes it. Docs are part of + the definition of done. +* Prefer a table or a decision tree over prose for anything an operator will + follow under pressure. +* Protocol facts (IDs, offsets, structures) must cite their source + (the decompiled study notes, or a live trace). + +## 4. Tooling + +| Tool | Purpose | +|---|---| +| `scripts/stack_manager.sh` | build/load/status/test/unload with correct ordering | +| `scripts/vgpu_vm.py` | VM lifecycle incl. guest-agent exec/shell | +| `tools/guest-diag/` | guest-side diagnostics (ptrace tracer, engines, probes, push helper) | +| `tools/firmware_analyzer.py` + Ghidra scripts | firmware analysis pipeline | +| module debug parameters | guest-memory scanning/poking, VRAM inspection | +| QEMU gdbstub workflow | guest virtual-memory inspection/writing | +| `wiki/` documents | protocol and firmware deep dives | + +## 5. Landing a change + +1. **Pick from the known-unknowns list** or a bug with a reproduction. +2. **Reproduce** with the standard captures; write the decision tree if new. +3. **Implement** with tests: host-side unit tests where possible; a scripted + guest scenario where not. +4. **Evidence**: attach the rung-ladder result (see the evaluation doc) and + the before/after trace excerpt. +5. **Docs**: update the affected documents (including the known-unknowns entry + if you closed one). +6. **Review**: a reviewer must be able to reproduce your evidence from the + PR description alone. + +## 6. Makefile targets + +```bash +make kernel # open-iov-vfio.ko (first!) +make nvidia # NVIDIA OpenRM modules with KBUILD_EXTRA_SYMBOLS +make userspace # libiov.so, examples +make all # in the correct order +make clean +``` + +## 7. Testing discipline + +* Kernel changes: at minimum, load/unload cycle ×3, mdev create/destroy, and a + guest boot + `nvidia-smi`. +* Protocol changes: diff the RPC inventory before/after; attach both. +* Memory-model changes: re-run the VA↔PA discovery tooling and a guest CUDA + start-up; attach the trace. +* Never land a change that makes a rung regress, even "temporarily". + +## 8. Security hygiene for contributors + +* Do not add code paths that map or copy memory outside the hypervisor-provided + DMA interface. +* Treat guest input as hostile; treat debug parameters as privileged. +* If you add a debug surface, document it in the debugging doc and gate it. +* If you discover a protocol fact that implies a security-relevant assumption + (e.g. "the guest can ask us to write anywhere"), stop and escalate: it is a + design issue, not a code issue. + +## 9. Getting unstuck + +* Start with the guest's own diagnostics (it is the oracle). +* Then the host trace; the answer is almost always in the last RPC before the + failure. +* Then memory observation (scan/poke) — but write down what you learn; the + next person needs it. +* Ask in the project channels with a *reproduction*, not a summary. diff --git a/docs/08-ecosystem/haaeep-and-collaboration.md b/docs/08-ecosystem/haaeep-and-collaboration.md new file mode 100644 index 0000000..c4fb352 --- /dev/null +++ b/docs/08-ecosystem/haaeep-and-collaboration.md @@ -0,0 +1,154 @@ +# HAAEEP and Collaboration + +> The High Assurance Agent Edge Environment Platform: a concrete, buildable +> use-case architecture that Open-IOV exists to enable — and how we want to +> build it with you. + +## 1. Motivation + +An AI agent harness (Claude Code, OpenCode, or any future equivalent) is an +untrusted, capability-constrained workload that runs generated code, holds +credentials, and talks to a model. In a *cloud* deployment, the agent's requests +leave the device. In a *secure edge* deployment — a workstation, a kiosk, a +vehicle, a medical or industrial system — they must not. + +We want an edge platform where: + +* inference runs **locally**, in its own VM, with the GPU; +* agents run in **their own VMs**, without the GPU and without the weights; +* other functions (desktop, networking, storage) run in **other VMs**, each + isolated; +* the whole stack is **attestable**, so remote parties can verify it; +* and nothing in this design requires the model server, the agent, or the GPU + driver to be modified. + +That platform is HAAEEP. Open-IOV is its GPU component. + +## 2. Architecture + +``` +┌──────────────────────────────────────────────────────────────────────────┐ +│ EDGE DEVICE — one GPU, local inference only │ +│ │ +│ ┌─────────────────┐ ┌──────────────────┐ ┌───────────────────┐ │ +│ │ Agent VM │ │ Inference VM │ │ Infrastructure │ │ +│ │ capability- │────▶│ vLLM / SGLang │ │ desktop, net, │ │ +│ │ constrained │ API │ owns GPU via │ │ storage VMs │ │ +│ │ no GPU │ │ Open-IOV mdev or │ │ optional GPU accel│ │ +│ │ │ │ xen-pci + relaying│ │ via Open-IOV │ │ +│ └─────────────────┘ └────────┬─────────┘ └───────────────────┘ │ +│ │ │ +│ ┌────────────────────────────────┴───────────────────────────────────┐ │ +│ │ Hypervisor: KVM / Xen + Argo/OpenXT / Qubes OS Air │ │ +│ │ • Flask-XSM / HMX mandatory access control between VMs │ │ +│ │ • TrenchBoot measured boot: firmware → hypervisor → OS │ │ +│ │ • Open-IOV: per-VM virtual GPUs, IOMMU-enforced isolation │ │ +│ └────────────────────────────────────────────────────────────────────┘ │ +└──────────────────────────────────────────────────────────────────────────┘ +``` + +### 2.1 Roles and trust + +| Component | Trust | Why | +|---|---|---| +| Agent VM | untrusted | runs generated code, may be compromised | +| Inference VM | high (holds weights) | isolated from agents by MAC and by device virtualization | +| GPU-owning VM (optional) | high | owns the physical device; can serve others through Open-IOV | +| Infrastructure VMs | medium | GPU acceleration optional, no access to weights | +| Hypervisor | TCB | enforced by the platform (Argo/OpenXT/Qubes) | +| Open-IOV device | TCB-adjacent | must fail closed; audited (see the security doc) | + +### 2.2 Data flow + +1. An agent VM issues an API call to the inference VM over an internal, + MAC-controlled channel. +2. The inference VM's vLLM/SGLang (unchanged) serves the request with the GPU, + using its **KV-cache plugins** (also unchanged). +3. No request ever leaves the device: the platform's network policy forbids + external inference endpoints by construction. + +### 2.3 Optional topology: GPU relay domain + +Where the platform prefers it (Xen with `xen-pci`), a single domain owns the +physical GPU and **relays GPU acceleration to other domains via Open-IOV**. This +gives a stronger separation: the process running the model is itself isolated +from the domain that owns the silicon, and agents are isolated from both. + +## 3. Why this needs Open-IOV (and not something else) + +* **Discrete passthrough** gives the GPU to exactly one VM — but HAAEEP needs + the GPU in the inference VM *and* optional acceleration elsewhere. +* **Process-level sharing** (host CUDA with containers) does not give + VM-level isolation, which is the whole point of the platform. +* **Proprietary vGPU** could do much of this — but you cannot build a + sovereign, auditable, redistributable edge platform on a licensed host stack + you cannot inspect or modify. +* **Software GPU emulation** cannot run models usefully. + +Open-IOV is the only component in this design that *must* be open, permissively +licensed and version-durable — so it is the component we are building. + +## 4. Roadmap for HAAEEP + +| Milestone | Depends on | +|---|---| +| CUDA execution in a guest (rung 3) | Open-IOV forwarding work | +| Two-VM coexistence (inference + agent) with isolation | Open-IOV multi-tenancy + platform MAC | +| Xen/OpenXT or Qubes Air variant | Open-IOV transport backends | +| Attested boot-to-inference chain | TrenchBoot integration | +| Published reference platform (scripts, docs) | all of the above | + +We will document each milestone as it lands, with the same evidence discipline +as the rest of the project. + +## 5. Collaboration models + +### 5.1 Open source projects + +Adopt the code under BSD-3-Clause (kernel modules are GPL-2.0). We will: + +* treat your interface requirements as design inputs (transports, IRQ models, + memory interfaces), +* provide the transport abstraction you need rather than asking you to fork, +* test against your platform in our matrix. + +### 5.2 Commercial partners + +Welcome, with no license fee: + +* ship Open-IOV in products; differentiate above it, +* sponsor support for specific hardware/driver versions, +* co-develop under the same permissive terms. + +### 5.3 Hardware vendors + +Nothing is required from you. Our goal is that new hardware works through +configuration and measurement. If you want to accelerate support, share +hardware and traces; we publish what we learn (within the bounds of what can be +published). + +### 5.4 Security researchers + +The emulation's attack surface is small and documented. Break it. We will +publish findings and fixes, and we will credit you. + +## 6. Cross-project notes + +| Project | Relationship | +|---|---| +| **Qubes OS Air** | provides compartmentalization on modern hardware; Open-IOV makes GPU acceleration usable per compartment (especially AI workloads) | +| **TrenchBoot** | provides measured/attested boot; combined, the platform can attest the inference path | +| **Argo / OpenXT** | provide hypervisor-mediated exchange (HMX) and Flask-XSM MAC; Open-IOV respects and complements that policy | +| **KVM/VFIO gaming** | established the fast-passthrough pattern; Open-IOV serves the users that need *multiple* VMs or cannot dedicate the GPU | +| **vLLM / SGLang** | remain the inference answer; Open-IOV isolates them without changing them; KV-cache plugins keep working | + +## 7. How to help + +* **Platform builders**: bring your hypervisor, tell us the transport you need. +* **Security engineers**: review the isolation model and the debug surface. +* **AI-infrastructure engineers**: define the reference HAAEEP deployment with + us (model server, KV cache, agent harness). +* **Everyone**: run the ladder on your hardware and send the evidence. + +Contact and participation channels are listed in the repository README. We +would rather have a hundred partners than a hundred forks. diff --git a/docs/09-reference/glossary-and-faq.md b/docs/09-reference/glossary-and-faq.md new file mode 100644 index 0000000..255d1a6 --- /dev/null +++ b/docs/09-reference/glossary-and-faq.md @@ -0,0 +1,103 @@ +# Glossary and FAQ + +## Glossary + +| Term | Meaning | +|---|---| +| **AMAPlib** | NVIDIA's shared "address mapping" library: swizzling, bank/channel mapping, aperture rules. Studied for page-table and memory-layout semantics. | +| **BAR0 / BAR1 / BAR3** | PCI BAR windows of the GPU: register interface (16 MB), VRAM aperture (256 MB), instance/PRAMIN window. | +| **CE** | Copy Engine: DMA engine; also used for UVM's internal work. | +| **CUDASW** | The family of NVIDIA user-space software that sits between applications and the driver: `libcuda`, `libcudart`, JIT compiler, fatbinary loader, and friends. | +| **Doorbell** | A register/memory location a driver writes to signal "new work is available" for a channel or queue. | +| **Engine type** | Numeric identifier for an execution engine (`GR = 0x1`, `COPY0 = 0x9`, ...). | +| **FWSEC** | Firmware security processor stage of the GSP boot sequence. | +| **GNND / GB10** | NVIDIA "Grace-Blackwell" systems-on-chip used in DGX Spark and Surface-class laptops; explicit portability targets. | +| **GPFIFO** | GPU FIFO: a ring of entries, each pointing to a push buffer for a channel. | +| **GR** | The graphics/compute engine that executes shaders (graphics pipelines and CUDA kernels). | +| **GSP** | GPU System Processor: the embedded RISC-V (RV64IMAC) that runs firmware containing the Resource Manager. | +| **GSP-RM** | The Resource Manager running on the GSP ("self-hosted RM"). | +| **GMMU** | GPU Memory Management Unit: page tables, TLB, fault handling. | +| **HAL** | Hardware Abstraction Layer: per-generation functions that program registers and formats. | +| **mdev** | Mediated device: the kernel mechanism that turns a physical device into virtual instances (VFIO). | +| **MSGQ** | The GSP message queue transport: shared-memory element rings with a page table, plus doorbells. | +| **PBDMA** | Push-buffer DMA engine in the host scheduler that fetches and executes push buffers. | +| **PDB** | Page Directory Base: root of a GPU page table. | +| **PDE / PTE** | Page-directory entry / page-table entry. | +| **Push buffer** | A stream of method calls (engine commands) submitted via a GPFIFO. | +| **RMCore** | The RM's object system: classes, handles, parents, methods. | +| **RM / RM API** | NVIDIA's Resource Manager and the API used to allocate objects and issue controls. | +| **resserv / resource server** | The RM component that owns allocation authority, privileges and lifetimes. | +| **Runlist** | The scheduler's list of channels eligible to run. | +| **SEC2** | Secure copy engine used by UVM for protected operations. | +| **Semaphore / tracker** | Memory location written by the GPU to signal progress; UVM's tracker waits on it. | +| **SRIOV / VF** | Single-root I/O virtualization / virtual functions (hardware partitioning). | +| **TSG** | Time-Slice Group: channels sharing a scheduling slot. | +| **USERD** | Per-channel control block in VRAM: PUT/GET pointers and control state. | +| **UVM** | NVIDIA's unified memory module: VA spaces, page trees, faults, channel manager. | +| **VA / PA / GPA / IOVA** | Virtual address / physical address / guest physical address / I/O virtual address. | +| **vidmem / sysmem / aperture** | GPU local memory / system memory / the way an address is qualified (`ADDR_FBMEM = 2`, `ADDR_SYSMEM = 1`). | +| **VGX version** | The version the guest announces during RPC negotiation; selects message layouts. | +| **vGPU** | Virtual GPU: this project's device. | +| **WPR** | Write-Protected Region: VRAM area reserved for firmware. | + +## FAQ + +**Is this a replacement for vLLM on bare metal?** +No. Continuous batching on bare-metal hosts is the right answer for centralized +inference. Open-IOV is for compute *inside guests*: Windows/Linux CUDA, 3D, +video, and secure edge isolation. See the orientation docs. + +**Does the guest driver need to be patched?** +No, and that is a hard project rule. No guest driver patches, no host driver +patches, no firmware patches, no hypervisor patches. + +**Does it work with the distribution's NVIDIA driver on the host?** +Yes — in the standalone configuration the distribution driver owns the physical +GPU and Open-IOV is an independent module. Building NVIDIA's OpenRM modules from +source is only needed for the native-vGPU-management workflows. + +**Why is `nvidia-smi` working but CUDA not?** +Because CUDA requires *execution*: kernels must run on real hardware. The +current state runs the entire initialization path (including UVM's start-up) +but stops short of executing engine work. See the known-knowns document for +the exact status and plan. + +**How is this legal / license-clean?** +Open-IOV is BSD-3-Clause (GPL-2.0 for kernel modules that link GPL-only kernel +exports); we interoperate with unmodified NVIDIA software rather than +relicensing it. See the legal document. + +**Will it support Windows guests?** +Yes, it is a target: same architecture, measured bring-up. See the portability +matrix. + +**Can I run several VMs on one GPU?** +Yes — that is a core design goal (per-VM mdev, IOMMU isolation, explicit +scheduling). Multi-tenant maturity depends on the scheduling/isolation work in +the known-unknowns list. + +**Does it support Xen / Qubes OS Air / Argo / OpenXT?** +Designed for them; the transport backends are on the roadmap. The emulation +core is hypervisor-independent by construction. + +**What about MIG?** +MIG is a hardware partitioning mechanism; Open-IOV's scheduling policy can use +it where available. It is part of the multi-tenancy work. + +**How fast is it?** +Initialization currently completes in the same order of magnitude as native; +execution forwarding is not implemented yet. Performance claims follow the +published methodology in the operations docs. + +**Where do I report bugs or ask for help?** +Open an issue with the standard captures listed in the debugging document. A +reproduction plus logs is worth more than a description. + +**Can my company use this commercially?** +Yes, under BSD-3-Clause (kernel modules GPL-2.0), with no fee. We also welcome +co-development; see the ecosystem document. + +**How do I contribute support for my GPU/driver/hypervisor?** +Follow the "adding a new combination" workflows in the portability and +maintenance docs; the deliverable is a profile plus evidence from the +evaluation ladder. diff --git a/docs/09-reference/legal-and-licensing.md b/docs/09-reference/legal-and-licensing.md new file mode 100644 index 0000000..74f1363 --- /dev/null +++ b/docs/09-reference/legal-and-licensing.md @@ -0,0 +1,81 @@ +# Legal, Licensing, and Third-Party Material + +> What is licensed how, what we may redistribute, and how we stay clean while +> doing adversarial interoperability. + +## 1. The project's licenses + +| Component | License | Rationale | +|---|---|---| +| `kernel/**` (e.g. `open-iov-vfio.ko`) | **GPL-2.0** (`MODULE_LICENSE("GPL v2")`) | The Linux kernel exports the interfaces we need (VFIO, mdev, IOMMU, DMA) under `EXPORT_SYMBOL_GPL`. GPL-2.0 is the legally correct choice. | +| `userspace/**`, `tools/**`, `scripts/**`, `config/**`, `docs/**` | **BSD-3-Clause** | Maximizes reuse and adoption — the project's mission. | +| Third-party submodules and imported trees | their own licenses | Not relicensed; used under their terms. | + +Per-file `SPDX-License-Identifier` headers always govern. + +The split is deliberate: the kernel side gets a clean legal footing; everything +vendor-facing, embeddable and reusable stays permissive. Commercial adoption is +welcome and requires no license fee. + +## 2. How we learn the interfaces + +Open-IOV is built by **reverse engineering and decompilation**: we study the +behaviour of vendor drivers and firmware — in VMs, on hardware, and by +disassembling published firmware images and analysed binaries — and we +reimplement the interfaces we discover. What this project publishes is our +own work: behaviour descriptions, numeric identifiers, layouts, emulation +code, and the evidence behind each observation. + +In this repository you will find: + +* **GSP firmware images** (`firmware/**/*.fwimage.bin`) — fetched from public + distribution channels (e.g. `linux-firmware`); used to understand and + emulate the firmware interface. +* **Decompiled firmware artifacts** (`firmware/**/decompiled/`) — our own + analysis output; kept for study and cited in the wiki. +* **Reformatted interface tables** — IDs, structures and message layouts that + we derived, published in `wiki/` and in this documentation tree. + +**Our rule:** publish *our work product* (facts, layouts, behaviours, code and +evidence). Do not copy third-party source or documentation into this project, +do not point readers at material we cannot redistribute, and do not +redistribute third-party binaries as project content beyond what their terms +allow. + +## 3. Adversarial interoperability and the law + +We implement the *interface contract* so that unmodified NVIDIA software can +talk to our device. We do not: + +* copy NVIDIA driver code into our kernel module, +* patch NVIDIA drivers or firmware, +* defeat licensing for NVIDIA's own products (our device is our device; no + impersonation of a licensed product identity). + +Where interoperability requires knowledge of structures and constants, we +obtain them from publicly distributed headers and from observation of the +unmodified guest in a lab, and we document the evidence. This is the same +discipline used by other interoperable implementations of closed interfaces. + +If you believe a specific artifact in this repository is improperly included, +open an issue: we would rather over-comply than argue. + +## 4. Contributions + +* Inbound = outbound: contributions to BSD-3-Clause files are BSD-3-Clause; + contributions to GPL-2.0 files are GPL-2.0. +* By contributing you confirm you have the right to do so and that the + contribution does not contain third-party material under incompatible terms. +* Write original code. Describe behaviour, cite the evidence, and reimplement + interfaces from observation rather than copying anyone's source. +* Do not commit firmware or ISO images unless they are explicitly permitted and + listed in `.gitattributes`/LFS policy. + +## 5. Export, safety and platform notes + +* GPU firmware and drivers may be subject to export regulations in your + jurisdiction; users are responsible for compliance. +* Security-sensitive deployments should follow the security evaluation doc and + the platform's attestation mechanisms (e.g. TrenchBoot). +* Nothing in this project's licensing changes the licenses of the host + distribution's NVIDIA driver, the guest's driver, or any hypervisor. diff --git a/docs/BUILD_NVIDIA_OPENRM.md b/docs/BUILD_NVIDIA_OPENRM.md new file mode 100644 index 0000000..5699036 --- /dev/null +++ b/docs/BUILD_NVIDIA_OPENRM.md @@ -0,0 +1,116 @@ +# Building NVIDIA OpenRM with Open-IOV vGPU VFIO Symbols + +## Overview + +The NVIDIA OpenRM `nvidia.ko` kernel module requires the vGPU VFIO interface symbols exported by `open-iov-vfio.ko`: +- `nvidia_vgpu_vfio_probe` +- `nvidia_vgpu_vfio_remove` +- `nvidia_vgpu_vfio_get_ops` +- `nvidia_vgpu_vfio_set_ops` +- `nv_vgpu_mdev_register_parent` +- `nv_vgpu_mdev_unregister_parent` +- And all GSP RPC integration symbols + +These symbols must be resolved at build time via `KBUILD_EXTRA_SYMBOLS` pointing to `open-iov-vfio.ko`'s `Module.symvers`. + +## Build Order (Critical) + +``` +1. Build open-iov-vfio.ko → Generates kernel/Module.symvers +2. Build NVIDIA OpenRM with KBUILD_EXTRA_SYMBOLS=kernel/Module.symvers +3. Load nvidia.ko first +4. Load open-iov-vfio.ko +``` + +## Automated Build (Recommended) + +Use the helper script: + +```bash +cd open-iov +./scripts/build_nvidia_openrm.sh +``` + +This script: +1. Verifies `open-iov-vfio.ko` has been built (checks for `Module.symvers`) +2. Cleans previous NVIDIA OpenRM build +3. Builds with `KBUILD_EXTRA_SYMBOLS` correctly set +4. Verifies vGPU symbols are resolved + +## Manual Build + +```bash +# Step 1: Build Open-IOV kernel module +cd $OPEN_IOV_ROOT +make kernel + +# Step 2: Build NVIDIA OpenRM with extra symbols +cd $OPEN_IOV_ROOT/nvidia-open-gpu-kernel-modules/kernel-open +KBUILD_EXTRA_SYMBOLS=$OPEN_IOV_ROOT/kernel/Module.symvers make -j$(nproc) modules + +# Step 3: Verify symbols resolved +grep -i vgpu Module.symvers +``` + +## Verification + +After building, verify the symbols are resolved: + +```bash +# Check Module.symvers has vGPU symbols +grep -i vgpu $OPEN_IOV_ROOT/nvidia-open-gpu-kernel-modules/kernel-open/Module.symvers + +# Check nvidia.ko modinfo +modinfo nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko | grep -i vgpu +``` + +## Common Issues + +| Issue | Cause | Fix | +|-------|-------|-----| +| `Unknown symbol nvidia_vgpu_vfio_probe` | `KBUILD_EXTRA_SYMBOLS` not passed or wrong path | Use helper script or verify path to `Module.symvers` | +| `Module.symvers not found` | open-iov-vfio.ko not built yet | Run `make kernel` first | +| Symbols in Module.symvers but not in nvidia.ko | Stale build cache | Run `make clean` in kernel-open before rebuild | +| `mdev_set_iommu_device` conftest fails | Kernel config missing IOMMU support | Enable `CONFIG_IOMMU_API` in kernel config | + +## Module Loading Order + +```bash +# 1. Load NVIDIA OpenRM (provides vGPU VFIO symbols) +sudo insmod nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko + +# 2. Load Open-IOV VFIO driver (registers mdev parent with nvidia.ko) +sudo insmod kernel/open-iov-vfio.ko + +# 3. Copy firmware +sudo cp firmware/57X-GSP/570.144/gsp-570.144.fwimage.bin /lib/firmware/ + +# 4. Create mdev device +./scripts/create_mdev.sh + +# 5. Launch QEMU with vGPU +./scripts/launch_qemu_vgpu.sh +``` + +## Key Files + +| File | Purpose | +|------|---------| +| `scripts/build_nvidia_openrm.sh` | Automated build script | +| `kernel/Module.symvers` | Exported symbols from open-iov-vfio.ko | +| `nvidia-open-gpu-kernel-modules/kernel-open/Kbuild` | Has `KBUILD_EXTRA_SYMBOLS` assignment | +| `nvidia-open-gpu-kernel-modules/kernel-open/Makefile` | Passes `KBUILD_EXTRA_SYMBOLS` to Kbuild | + +## Troubleshooting + +If symbols are still unresolved after rebuild: + +```bash +# Full clean rebuild +cd nvidia-open-gpu-kernel-modules/kernel-open +make clean +KBUILD_EXTRA_SYMBOLS=$OPEN_IOV_ROOT/kernel/Module.symvers make -j$(nproc) modules + +# Check modpost output for unresolved symbols +KBUILD_EXTRA_SYMBOLS=$OPEN_IOV_ROOT/kernel/Module.symvers make -j$(nproc) modules V=1 2>&1 | grep -i "modpost\|nvidia_vgpu" +```
\ No newline at end of file diff --git a/docs/DEBUGGING.md b/docs/DEBUGGING.md new file mode 100644 index 0000000..dc95ed7 --- /dev/null +++ b/docs/DEBUGGING.md @@ -0,0 +1,190 @@ +# Open-IOV Debugging Guide + +> Battle-tested procedures for bringing up the vGPU stack, diagnosing guest +> driver failures, and validating a fully working setup. Built from real +> debugging sessions (including the 2026-09-21 EPT-livelock hunt). + +--- + +## 1. Debug surfaces (know them all) + +| Surface | Where | What it shows | +|---------|-------|---------------| +| Host dmesg | `dmesg -T \| grep -iE 'gsp\|vgpu\|open-iov\|mdev\|vfio'` | mdev lifecycle, VFIO ioctls, guest BAR accesses, emulation thread, KVM warns | +| Guest serial | `/tmp/vgpu-guest-serial.log` ( yaml `serialLog`) | guest boot, kernel oops, driver prints — even when agent is dead | +| QEMU GDB stub | `tcp::1234` ( yaml `gdbPort`) | vCPU state, registers, disassembly — **works on a hard-wedged guest** | +| QMP | `/tmp/qemu-qmp.sock` | NMI injection, VM control, status | +| Guest agent | `/tmp/qemu-guest-agent.sock` (via `vgpu_vm.py exec`) | in-guest commands — **only while guest userspace is alive** | +| Guest SSH | `ssh -p 2222 <user>@localhost` | same, richer | +| GDB on host kernel | (optional) | host-side driver state | + +The VM config (`~/.local/libvf.io/vms/vgpu-linux.yaml`) should contain: + +```yaml +serialLog: "/tmp/vgpu-guest-serial.log" +gdbPort: 1234 +gdbFreeze: false # true = halt at reset vector (boot debugging) +``` + +--- + +## 2. The golden rule of failure triage + +**When the guest agent dies mid-test, the guest is wedged in kernel mode. +Do not keep retrying `exec` — go straight to the GDB stub and host dmesg.** + +Triage order: +1. `sudo dmesg -T | tail -50` — KVM warnings (`make_spte`, EPT), our driver's last messages +2. `tail -100 /tmp/vgpu-guest-serial.log` — guest kernel's last words +3. GDB stub vCPU sampling (below) — *where* is each vCPU stuck? + +--- + +## 3. QEMU GDB stub workflow + +Attach (no symbols needed for triage): + +```bash +gdb -q -ex 'set pagination off' -ex 'target remote localhost:1234' \ + -ex 'info threads' -ex 'thread 1' -ex 'p/x $rip' -ex 'x/8i $rip-8' \ + -ex 'thread 2' -ex 'p/x $rip' -ex 'x/8i $rip-8' -ex 'detach' -ex 'quit' +``` + +Reading the result: + +| Observation | Meaning | +|-------------|---------| +| vCPU RIP = `0xffffffff85......`, disasm shows `hlt` | CPU idle (kernel text range) — fine | +| vCPU RIP = `0xffffffffc0......` (module range), **same RIP+RSP across samples** | **hard spin in a kernel module** (e.g. nvidia.ko polling loop) — this is your hung driver | +| RIPs moving between samples | guest alive and progressing | + +For precise diagnosis, sample the stuck vCPU 3× a few seconds apart. Identical +`$rip` **and** `$rsp` every time = the *same instruction* faults/retries +(EPT-violation livelock) rather than a loop making progress. + +With guest symbols (`vmlinux` + module `.ko` with debug info), add +`add-symbol-file` and get real function names. + +### QMP NMI (soft alternative) + +```bash +(echo '{"execute":"qmp_capabilities"}'; sleep 0.3; \ + echo '{"execute":"human-monitor-command","arguments":{"command-line":"nmi 0"}}') \ + | sudo socat - UNIX-CONNECT:/tmp/qemu-qmp.sock +``` + +Watch the serial log afterwards for backtraces. Note: a truly hard-wedged vCPU +(spinning with a faulting MMIO instruction) may not print anything — use the +GDB stub instead. + +--- + +## 4. Case study: the 2026-09-21 EPT livelock + +Symptoms: guest `modprobe nvidia` hangs; guest agent dies; QEMU alive. + +1. **Host dmesg**: `WARNING: ... at arch/x86/kvm/mmu/spte.c:245 make_spte` with + `spte = 0x600404aa8001b07, rsvd bits = 0xfff8000000000` — a bogus PFN. +2. **GDB stub**: vCPU2 frozen at one address in module text + (`0xffffffffc0d968ef`), identical RIP/RSP across samples; other vCPUs in + `hlt`. The faulting instruction: `mov 0x4(%rax),%r13d` — a memory read that + can never complete. +3. **Root cause**: VFIO `mmap` used `virt_to_phys()` on the `vzalloc`'d BAR0 + shadow → garbage PFN in the EPT → every guest BAR0 read faulted → KVM + retried forever. +4. **Fix**: per-page `vmalloc_to_pfn()` mapping; see + `docs/GSP_EMULATION_ARCHITECTURE.md` §2. + +Lesson: **a KVM `make_spte` reserved-bits WARN + one frozen vCPU = bad PFN from +your VFIO mmap handler.** Always question `virt_to_phys()` on anything that +isn't lowmem. + +--- + +## 5. What a *successful* driver bring-up looks like (validation checklist) + +After `stack_manager.sh load-stack` on the host: + +- [ ] `lsmod | grep -E 'open_iov_vfio|nvidia '` — both present, no `~` (stale) markers +- [ ] `cat /sys/class/mdev_bus/0000:01:00.0/mdev_supported_types/nvidia-nvidia-vgpu-turing/available_instances` → **1** +- [ ] No `Unknown symbol`, `make_spte`, or `cut here` lines in dmesg + +After `vgpu_vm.py start vgpu-linux`: + +- [ ] `pgrep -af qemu-system` shows `-device vfio-pci,...,sysfsdev=/sys/bus/mdev/devices/<uuid>` +- [ ] Host dmesg: `gsp_emu_init: GSP emulation engine initialized successfully`, `GSP emulation thread started` +- [ ] Host dmesg: VFIO `SET_IRQS wired N eventfds` (eventfds actually stored) +- [ ] Host dmesg: `VFIO device mmap offset=0x0 size=0x1000000` (BAR0 shadow mapped) +- [ ] Guest serial reaches login prompt; `vgpu_vm.py exec "whoami"` → `root` + +When the guest nvidia driver probes (auto via udev, or `modprobe nvidia`): + +- [ ] Host dmesg shows **BAR0 doorbell writes being consumed** (no `Invalid seq_id` spam after the first) +- [ ] Host dmesg: GSP RPCs processed (dispatch logs; stats increment) +- [ ] **No** KVM `make_spte` WARN, **no** `pr_warn_ratelimited ... not supported in emulation mode` spam +- [ ] Guest: `nvidia-smi` lists the TU104 vGPU (name, memory, no "No devices found") +- [ ] Guest: `dmesg | grep -i nvidia` shows successful probe, GSP-RM init done +- [ ] Guest: `cat /proc/driver/nvidia/version` → **570.144** + +Teardown regression: + +- [ ] `vgpu_vm.py stop <uuid>`; remove mdev → `available_instances` returns to **1** +- [ ] `ps -e | grep gsp_emu` → **empty** (no leaked thread) +- [ ] `/dev/vfio/<N>` node gone; matching `/sys/kernel/iommu_groups/<N>` gone + +Kernel unit test: + +- [ ] `sudo insmod kernel/tests/gsp_emulation_test.ko` → dmesg `GSP_EMU_TEST: ALL TESTS PASSED` +- [ ] `sudo rmmod gsp_emulation_test` → clean + +--- + +## 6. Failure taxonomy (symptom → likely cause) + +| Symptom | Likely cause | First check | +|---------|--------------|-------------| +| `insmod open-iov-vfio.ko: Unknown symbol` | missing prereq module (`mdev`) | `modprobe mdev vfio vfio_iommu_type1 vfio_pci` | +| mdev create `-EUSERS` | `available_instances` = 0 (leak) | `cat .../available_instances` | +| Guest: no PCI device | mdev not attached | QEMU cmdline for `sysfsdev=` | +| Guest: sees device, probe hangs instantly | BAR0 mmap PFN bug (fixed) | host `make_spte` WARN | +| Guest: probe hangs, no KVM warn | doorbell/RPC never consumed | host dmesg for thread + doorbell writes | +| Guest: RPC processed but driver still waits | MSI-X never delivered | `SET_IRQS wired` line; `irqs[0].trigger` non-NULL | +| Guest agent dead, QEMU alive | guest kernel wedge | GDB stub vCPU sampling (§3) | +| `rmmod open_iov_vfio` hangs | leaked kthread / open fds | `ps -e \| grep gsp_emu`; mdevs still present? | +| Guest Code 43 / probe fail (Windows) | config space / MSI-X details | config read/write logs | +| `pr_warn_ratelimited(...not supported in emulation mode)` | guest used BAR_TRAP/MIGRATE RPC | expected fail-fast; wire real handler if hot | + +--- + +## 7. Host-guest version matrix discipline + +Probe failures caused by **version mismatch** look identical to emulation bugs. +Before deep debugging, verify the matrix: + +| Component | Must be | Check | +|-----------|---------|-------| +| Guest driver | 570.144 | `cat /proc/driver/nvidia/version` (guest) | +| Host OpenRM nvidia.ko | 570.144 | `cat /sys/module/nvidia/version` (host) | +| Host GSP firmware | 570.144 | `/lib/firmware/gsp-570.144.fwimage.bin` MD5 `13ce12a392a31eeb79a2e6ebd3a4a93a` | +| Emulation target | 570.144W | decompiled firmware basis | + +Rule: **match versions first, debug second.** (2026-09-21: 580.178 guest vs +570.144 emulation cost a full session before the real mmap bug was found — +the mismatch masked the signal.) + +--- + +## 8. Quick reference + +```bash +# Stack +./scripts/stack_manager.sh status|load-stack|unload|dmesg +# VM (harness ONLY - never raw /qemu) +./scripts/vgpu_vm.py start vgpu-linux|stop <uuid>|exec [--timeout N] <cmd>|shell +# Debug +tail -f /tmp/vgpu-guest-serial.log & +gdb -ex 'target remote localhost:1234' +sudo dmesg -Tw | grep -iE 'gsp|vgpu|open-iov' +# Recover wedged VM +./scripts/vgpu_vm.py stop <uuid> # or: sudo kill -9 <qemu pid> +``` diff --git a/docs/DEVELOPMENT_WORKFLOW.md b/docs/DEVELOPMENT_WORKFLOW.md new file mode 100644 index 0000000..9019ac1 --- /dev/null +++ b/docs/DEVELOPMENT_WORKFLOW.md @@ -0,0 +1,671 @@ +# Open-IOV vGPU-IO Development Workflow & Tools Guide + +## Overview + +This document describes the complete development workflow, tooling, and shell integration for working on the Open-IOV vGPU-IO project. It covers everything from firmware reverse engineering to QEMU guest debugging. + +--- + +## 1. Repository Structure & Navigation + +### Primary Workspace +```bash +$OPEN_IOV_ROOT/ # Main monorepo (git + LFS) +``` + +### Key Directories +| Directory | Purpose | +|-----------|---------| +| `kernel/` | Host VFIO mdev driver (`open-iov-vfio.ko`) | +| `legacy-driver-guest/` | Historical guest-side driver (reference only, **not built**) | +| `userspace/lib/` | `libiov.so` / `libvgpu.so` + `rpc_emulator.c` | +| `userspace/examples/` | `example_vgpu` test app | +| `firmware/57X-GSP/570.144/` | GSP firmware binary + 132 decompiled C files | +| `nvidia-open-gpu-kernel-modules/` | NVIDIA OpenRM source (submodule) | +| `scripts/` | Helper scripts (load, create_mdev, launch_qemu, test, debug) | +| `tools/` | Ghidra projects, VM tools, analysis scripts, guest agent | +| `docs/` | Architecture, implementation guides, TODO lists | +| `wiki/` | 10 technical wiki pages | + +--- + +## 2. Shell Integration (zshrc) + +### Required Aliases & Functions +Add to `~/.zshrc` and run `source ~/.zshrc`: + +```bash +# Project navigation +alias oiv="cd $OPEN_IOV_ROOT" +alias oivk="cd $OPEN_IOV_ROOT/kernel" +alias oivu="cd $OPEN_IOV_ROOT/userspace" +alias oivf="cd $OPEN_IOV_ROOT/firmware/57X-GSP/570.144" +alias oivd="cd $OPEN_IOV_ROOT/docs" +alias oivs="cd $OPEN_IOV_ROOT/scripts" +alias oivt="cd $OPEN_IOV_ROOT/tools" + +# Build commands +alias oiv-build="cd $OPEN_IOV_ROOT && make -j$(nproc)" +alias oiv-build-kernel="cd $OPEN_IOV_ROOT && make kernel" +alias oiv-build-userspace="cd $OPEN_IOV_ROOT && make userspace" +alias oiv-clean="cd $OPEN_IOV_ROOT && make clean" + +# Module loading (requires reboot first!) +alias oiv-load="$OPEN_IOV_ROOT/scripts/load_vgpu_stack.sh" +alias oiv-unload="$OPEN_IOV_ROOT/scripts/unload_vgpu_stack.sh" +alias oiv-test="$OPEN_IOV_ROOT/scripts/test_vgpu_stack.sh" +alias oiv-test-gsp="$OPEN_IOV_ROOT/scripts/test_gsp_emu.sh" +alias oiv-test-guest="$OPEN_IOV_ROOT/scripts/test_guest_driver.sh" +alias oiv-test-cuda="$OPEN_IOV_ROOT/scripts/test_cuda_workloads.sh" +alias oiv-test-graphics="$OPEN_IOV_ROOT/scripts/test_graphics_workloads.sh" +alias oiv-test-windows="$OPEN_IOV_ROOT/scripts/test_windows_guest.sh" + +# mdev management +alias oiv-mdev-create="$OPEN_IOV_ROOT/scripts/create_mdev.sh" +alias oiv-mdev-list="ls -la /sys/bus/mdev/devices/ 2>/dev/null || echo 'No mdev devices'" + +# QEMU +alias oiv-qemu="$OPEN_IOV_ROOT/scripts/launch_qemu_vgpu.sh" +alias oiv-qemu-debug="$OPEN_IOV_ROOT/scripts/debug_guest_init.sh" + +# Firmware analysis +alias oiv-ghidra="cd $OPEN_IOV_ROOT/tools/ghidra && ./ghidraRun" +alias oiv-fw-analyze="cd $OPEN_IOV_ROOT && python3 tools/firmware_analyzer.py" + +# Debugging tools (NEW) +alias oiv-guest-monitor="python3 $OPEN_IOV_ROOT/tools/guest_init_monitor.py" +alias oiv-failure-analyzer="python3 $OPEN_IOV_ROOT/tools/failure_analyzer.py" +alias oiv-benchmark="$OPEN_IOV_ROOT/tools/benchmark_infrastructure.sh" +alias oiv-windows-test="$OPEN_IOV_ROOT/scripts/windows_guest_automation.sh" + +# Kernel debugging +alias oiv-dmesg="dmesg -T | grep -i -e nvidia -e gsp -e vgpu -e open-iov -e mdev" +alias oiv-dmesg-follow="dmesg -T -w | grep -i -e nvidia -e gsp -e vgpu -e open-iov -e mdev" +alias oiv-dmesg-errors="dmesg -T | grep -i -e nvidia -e gsp -e vgpu -e open-iov -e mdev | grep -i -e error -e warn -e fail -e panic" +alias oiv-trace="sudo cat /sys/kernel/debug/tracing/trace_pipe | grep gsp_emu" + +# Git +alias oiv-git="cd $OPEN_IOV_ROOT && git status" +alias oiv-git-log="cd $OPEN_IOV_ROOT && git log --oneline -20" +alias oiv-git-diff="cd $OPEN_IOV_ROOT && git diff" + +# Restore point +alias oiv-restore="cd $OPEN_IOV_ROOT && git checkout f9ed5d8 -- . && git clean -fd" + +# Help +oiv-help() { + echo "╔═══════════════════════════════════════════════════════════════════════════════╗" + echo "║ Open-IOV vGPU-IO Development Commands ║" + echo "╠═══════════════════════════════════════════════════════════════════════════════╣" + echo "║ NAVIGATION ║" + echo "║ oiv - cd to open-iov root ║" + echo "║ oivk - cd to kernel/ ║" + echo "║ oivu - cd to userspace/ ║" + echo "║ oivf - cd to firmware/57X-GSP/570.144/ ║" + echo "║ oivd - cd to docs/ ║" + echo "║ oivs - cd to scripts/ ║" + echo "║ oivt - cd to tools/ ║" + echo "╠═══════════════════════════════════════════════════════════════════════════════╣" + echo "║ BUILD ║" + echo "║ oiv-build - Build everything (kernel + userspace + examples) ║" + echo "║ oiv-build-kernel - Build open-iov-vfio.ko only ║" + echo "║ oiv-build-userspace - Build libiov.so + example_vgpu ║" + echo "║ oiv-clean - Clean all build artifacts ║" + echo "╠═══════════════════════════════════════════════════════════════════════════════╣" + echo "║ MODULE OPERATIONS (require root) ║" + echo "║ oiv-load - Load nvidia.ko → open-iov-vfio.ko (refactored stack) ║" + echo "║ oiv-unload - Unload all modules + remove mdevs ║" + echo "╠═══════════════════════════════════════════════════════════════════════════════╣" + echo "║ TESTING ║" + echo "║ oiv-test - Run comprehensive test suite (test_vgpu_stack.sh) ║" + echo "║ oiv-test-gsp - Run GSP emulation unit tests (test_gsp_emu.sh) ║" + echo "║ oiv-test-guest - Test guest driver integration (test_guest_driver.sh)║" + echo "║ oiv-test-cuda - Test CUDA workloads (test_cuda_workloads.sh) ║" + echo "║ oiv-test-graphics - Test graphics workloads (test_graphics_workloads.sh)║" + echo "║ oiv-test-windows - Test Windows guest (test_windows_guest.sh) ║" + echo "╠═══════════════════════════════════════════════════════════════════════════════╣" + echo "║ MDEV MANAGEMENT ║" + echo "║ oiv-mdev-create - Create mdev device for QEMU ║" + echo "║ oiv-mdev-list - List existing mdev devices ║" + echo "╠═══════════════════════════════════════════════════════════════════════════════╣" + echo "║ QEMU ║" + echo "║ oiv-qemu - Launch QEMU with vGPU ║" + echo "║ oiv-qemu-debug - Launch QEMU with GDB + guest monitor ║" + echo "╠═══════════════════════════════════════════════════════════════════════════════╣" + echo "║ DEBUGGING ║" + echo "║ oiv-dmesg - Filtered dmesg for NVIDIA/GSP/vGPU/mdev ║" + echo "║ oiv-dmesg-follow - Live filtered dmesg (tail -f) ║" + echo "║ oiv-dmesg-errors - Show only errors/warnings ║" + echo "║ oiv-trace - Live kernel trace_pipe filtered for gsp_emu ║" + echo "║ oiv-guest-monitor - Monitor guest init (correlates host traces + serial) ║" + echo "║ oiv-failure-analyzer - Analyze/classify guest init failures ║" + echo "║ oiv-benchmark - Run VM vs bare metal benchmarks ║" + echo "║ oiv-windows-test - Windows guest test automation (WinRM) ║" + echo "╠═══════════════════════════════════════════════════════════════════════════════╣" + echo "║ GIT ║" + echo "║ oiv-git - Git status in open-iov ║" + echo "║ oiv-git-log - Git log (last 20 commits) ║" + echo "║ oiv-git-diff - Show uncommitted changes ║" + echo "╠═══════════════════════════════════════════════════════════════════════════════╣" + echo "║ FIRMWARE ANALYSIS ║" + echo "║ oiv-ghidra - Launch Ghidra GUI ║" + echo "║ oiv-fw-analyze - Run firmware analyzer (Python) ║" + echo "╠═══════════════════════════════════════════════════════════════════════════════╣" + echo "║ EMERGENCY ║" + echo "║ oiv-restore - Restore to commit f9ed5d8 (restore point) ║" + echo "╠═══════════════════════════════════════════════════════════════════════════════╣" + echo "║ oiv-help - This help ║" + echo "╚═══════════════════════════════════════════════════════════════════════════════╝" +} +``` + +### Usage +```bash +# After adding to ~/.zshrc: +source ~/.zshrc +oiv-help # Show all commands +oiv # Jump to project root +oiv-build # Build everything +oiv-load # Load kernel modules +oiv-mdev-create # Create mdev for QEMU +oiv-qemu # Launch QEMU guest +oiv-dmesg-follow # Live kernel logs +``` + +--- + +## 3. Build Workflow + +### Full Build +```bash +cd $OPEN_IOV_ROOT +make # Builds kernel + userspace + examples +# OR +oiv-build +``` + +### Incremental Builds +```bash +# Kernel only (open-iov-vfio.ko) +make kernel +# OR +cd kernel && make -j$(nproc) + +# Userspace only (libiov.so + libvgpu.so + example_vgpu) +make userspace +# OR +cd userspace && make -j$(nproc) + +# Examples only +make examples + +# Tests +make tests + +# Clean +make clean +# OR +oiv-clean +``` + +### Build Outputs +| Artifact | Location | Size | +|----------|----------|------| +| `nvidia.ko` | `nvidia-open-gpu-kernel-modules/kernel-open/` | 35MB | +| `open-iov-vfio.ko` | `kernel/` | ~3.8MB | +| `libiov.so` / `libvgpu.so` | `userspace/lib/` | 76KB | +| `example_vgpu` | `userspace/examples/` | 27KB | + +--- + +## 4. Kernel Module Loading + +### Prerequisites +- The **stock Ubuntu nvidia driver must not be loaded** when you want our + OpenRM nvidia.ko: both are named `nvidia`. The stock driver auto-loads on + desktop boots (udev) and is held open by the GUI. + `/etc/modprobe.d/blacklist-nvidia-open-iov.conf` (installed on this + machine) prevents that auto-load after reboot; `insmod` is unaffected. +- `stack_manager.sh load-nvidia` detects a foreign loaded nvidia + (compares `/sys/module/nvidia/version` with our build) and refuses with + guidance instead of silently skipping. + +### Load Order (Critical) +```bash +# 1. VFIO/mdev prerequisites +sudo modprobe vfio vfio_iommu_type1 vfio_pci mdev + +# 2. Open-IOV VFIO mdev driver FIRST - it is fully standalone (zero nvidia +# imports) and EXPORTS the nvidia_vgpu_vfio_* interface + registers the +# mdev parent on the physical GPU. +sudo insmod $OPEN_IOV_ROOT/kernel/open-iov-vfio.ko + +# 3. NVIDIA OpenRM host driver - imports our nvidia_vgpu_vfio_* symbols +# (must be built with KBUILD_EXTRA_SYMBOLS=kernel/Module.symvers) +sudo insmod $OPEN_IOV_ROOT/nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko + +# 4. GSP firmware for the host GPU +sudo cp firmware/57X-GSP/570.144/gsp-570.144.fwimage.bin /lib/firmware/ +``` + +### Using Helper Script +```bash +./scripts/stack_manager.sh load-stack # prereqs → open-iov-vfio → nvidia → firmware +``` + +### Verify Load +```bash +lsmod | grep -E "nvidia|open_iov" +dmesg -T | grep -i -e nvidia -e gsp -e vgpu -e open-iov +ls /sys/class/mdev_bus/0000:01:00.0/mdev_supported_types/ +cat /sys/class/mdev_bus/0000:01:00.0/mdev_supported_types/nvidia-nvidia-vgpu-turing/available_instances +``` + +### Unload +```bash +oiv-unload # Removes mdevs, unloads modules in reverse order +``` + +--- + +## 5. mdev Device Management + +### Create mdev (for QEMU) +```bash +oiv-mdev-create +# Interactive: selects GPU, vGPU type, generates UUID +# Creates: /sys/bus/mdev/devices/<UUID>/ +``` + +### Manual Creation +```bash +# List GPUs +lspci -D -d 10de: | grep -E "VGA|3D" + +# List supported vGPU types +ls /sys/bus/pci/devices/0000:XX:XX.X/mdev_supported_types/ + +# Create specific type +echo "<UUID>" | sudo tee /sys/bus/pci/devices/0000:XX:XX.X/mdev_supported_types/nvidia-vgpu-turing/create +``` + +### List mdevs +```bash +oiv-mdev-list +# OR +ls -la /sys/bus/mdev/devices/ +``` + +### Remove mdev +```bash +echo 1 | sudo tee /sys/bus/mdev/devices/<UUID>/remove +``` + +--- + +## 6. QEMU Guest Launch + +### Using Helper Script +```bash +oiv-qemu +# Interactive: selects mdev, guest image, launches QEMU +``` + +### Manual Launch +```bash +qemu-system-x86_64 \ + -enable-kvm \ + -machine q35,accel=kvm,kernel_irqchip=split \ + -cpu host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_vendor_id=NVIDIA42 \ + -smp 8,sockets=1,cores=8,threads=1 \ + -m 16G,slots=4,maxmem=64G \ + -mem-prealloc \ + -object memory-backend-file,id=mem,size=16G,mem-path=/dev/hugepages,share=on \ + -numa node,memdev=mem \ + -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/<UUID>,display=off,x-igd-opregion=on \ + -vga none -display none \ + -drive file=/var/lib/libvirt/images/ubuntu-22.04.qcow2,format=qcow2,if=virtio,cache=writeback,aio=threads \ + -netdev user,id=net0,hostfwd=tcp::2222-:22 \ + -device virtio-net-pci,netdev=net0 \ + -usb -device usb-tablet \ + -name "vGPU-Guest" \ + -pidfile /tmp/qemu-vgpu.pid \ + -daemonize +``` + +### Connect to Guest +```bash +# SSH (port 2222 forwarded) +ssh -p 2222 user@localhost + +# VNC (if enabled in launch script) +vncviewer localhost:1 +``` + +--- + +## 7. Guest Driver Setup + +### Inside Guest VM +```bash +# Run setup script (installs NVIDIA driver, firmware, loads module) +sudo $OPEN_IOV_ROOT/scripts/setup_guest_driver.sh + +# Or manual: +# 1. Install NVIDIA driver (OpenRM 570.144) +# 2. Copy firmware to /lib/firmware/nvidia/tu102/gsp/ +# 3. modprobe nvidia +# 4. nvidia-smi +``` + +### Verify Guest +```bash +nvidia-smi # Should show vGPU (T4/A100/etc.) +lspci -d 10de: # Should show 10de:1ed3 (TU104 vGPU) +dmesg -T | grep -i nvidia # Check probe success +``` + +--- + +## 8. Testing Workflow + +### Host Test Suite +```bash +oiv-test +# Runs: module loading, device nodes, library loading, example app, mdev creation, kernel logs, firmware verification +``` + +### Guest CUDA Tests +```bash +# Inside guest: +cd /tmp +git clone https://github.com/NVIDIA/cuda-samples.git +cd cuda-samples/Samples/1_Utilities/vectorAdd +make +./vectorAdd +``` + +### Guest Graphics Tests +```bash +# Inside guest: +vulkaninfo +vkcube +glxgears +glmark2 +``` + +### Guest Video Tests +```bash +# Inside guest: +ffmpeg -init_hw_device cuda=gpu -f lavfi -i testsrc=duration=10:size=1920x1080:rate=30 -c:v h264_nvenc -preset fast -y test.mp4 +ffmpeg -hwaccel cuda -i test.mp4 -f null - +``` + +### Windows Guest Tests +```powershell +# Inside Windows guest: +# 1. Install NVIDIA driver (570.144) +# 2. nvidia-smi.exe +# 3. dxdiag +# 4. Run CUDA-Z / GPU-Z +# 5. Run 3DMark / FurMark +``` + +--- + +## 9. Firmware Reverse Engineering Workflow + +### Analyze New Firmware +```bash +# 1. Download firmware from NVIDIA OpenRM release +# 2. Place in firmware dir +cp new-gsp.bin $OPEN_IOV_ROOT/firmware/570-GSP/gsp-XXX.fwimage.bin + +# 3. Run Ghidra headless analysis +cd $OPEN_IOV_ROOT/tools/ghidra +./support/analyzeHeadless /tmp/ghidra_project "gsp_XXX" \ + -import $OPEN_IOV_ROOT/firmware/570-GSP/gsp-XXX.fwimage.bin \ + -processor "RISCV:RV64:default" \ + -postScript DecompileFunctions.java \ + -deleteProject + +# 4. Run firmware analyzer (auto-discovers addresses) +cd $OPEN_IOV_ROOT +python3 tools/firmware_analyzer.py firmware/570-GSP/gsp-XXX.fwimage.bin > analysis_XXX.json + +# 5. Compare with previous version +python3 tools/compare_analysis.py analysis_570.json analysis_XXX.json + +# 6. Generate config if new version +python3 tools/generate_config.py analysis_XXX.json > config/versions/XXX.yaml +``` + +### Ghidra Scripts (in tools/ghidra/) +| Script | Purpose | +|--------|---------| +| `DecompileFunctions.java` | Batch decompile all functions | +| `FindRPCHandlers.java` | Locate RPC dispatcher + jump table | +| `ExtractStructLayouts.java` | Extract struct definitions from prologues | +| `ComputeFirmwareSignature.java` | SHA256 + CRC32 for version matching | + +### Key Firmware Addresses (570.144W) +| Component | Address | Description | +|-----------|---------|-------------| +| Primary Dispatcher | 0x057A1AC0 | 32 sub-commands | +| Secondary Dispatcher | 0x04ED39DC | Additional routing | +| BAR Trap Handler | 0x04CB8188 | 40KB | +| WPR Handler 1 | 0x04F00B0C | 62KB | +| WPR Handler 2 | 0x04F00C78 | 62KB | +| RPC Jump Table | 0x059BFEB0 | 12 RPC handlers | +| ALLOC Main | 0x057E94EC | 40KB allocator | +| SCHEDULE Main | 0x051EFBF4 | 1.8KB scheduler | +| Entry Point | 0x04009000 | Bootstrap | + +--- + +## 10. Debugging Workflows + +### Guest Driver Probe Issues +```bash +# Host: Watch live logs +oiv-dmesg-follow + +# Guest: Check PCI config +lspci -vvv -d 10de: +cat /proc/bus/pci/devices + +# Guest: Check BAR mapping +lspci -v -d 10de: | grep -A 10 "Memory at" + +# Guest: Check MSI-X +lspci -vvv -d 10de: | grep -A 20 "MSI-X" +``` + +### GSP Emulation Debugging +```bash +# Check emulation thread +ps aux | grep gsp_emu + +# Check RPC processing +dmesg -T | grep -i "gsp_emu\|rpc" + +# Debugfs stats +cat /sys/kernel/debug/open-iov-vfio/<mdev>/stats + +# Add trace to gsp_emulation.c: +trace_printk("RPC %d latency %lld ns\n", req->type, ktime_to_ns(ktime_sub(end, start))); +cat /sys/kernel/debug/tracing/trace +``` + +### QEMU with GDB (Guest Kernel Debugging) +```bash +# Launch QEMU with GDB server +qemu-system-x86_64 -s -S ... # -s = gdbserver on :1234, -S = freeze + +# In another terminal: +gdb ./vmlinux +(gdb) target remote :1234 +(gdb) break nvidia_probe +(gdb) continue +``` + +### NEW: Comprehensive Debug Session (One Command) +```bash +# Launch debug session (QEMU + GDB + guest monitor) +./scripts/debug_guest_init.sh --vmlinux /path/to/guest/vmlinux + +# Or manually: +# Terminal 1: QEMU with GDB +./scripts/launch_qemu_vgpu.sh --gdb --freeze + +# Terminal 2: Guest init monitor +python3 tools/guest_init_monitor.py --guest-serial /dev/ttyS0 --duration 300 + +# Terminal 3: GDB +gdb -x tools/gdb_scripts/nvidia_init.gdb /path/to/vmlinux +(gdb) target remote localhost:1234 +``` + +### NEW: Failure Analysis +```bash +# After a failed guest init, analyze logs +python3 tools/failure_analyzer.py \ + --guest-log /tmp/guest_serial.log \ + --host-trace /tmp/host_trace.log \ + --qemu-log /tmp/qemu.log \ + --output /tmp/failure_report.json + +# This classifies the failure and provides specific recommendations +``` + +### NEW: Benchmarking +```bash +# Run VM vs bare metal benchmarks +./tools/benchmark_infrastructure.sh all + +# Or specific categories +./tools/benchmark_infrastructure.sh cuda +./tools/benchmark_infrastructure.sh graphics +./tools/benchmark_infrastructure.sh pytorch +``` + +### Performance Profiling +```bash +# GSP RPC latency +# Add to gsp_emulation.c: +trace_printk("RPC %d latency %lld ns\n", req->type, ktime_to_ns(ktime_sub(end, start))) + +# View trace +cat /sys/kernel/debug/tracing/trace + +# CUDA kernel launch overhead (in guest) +nvprof --profile-from-start off ./vectorAdd + +# MSI-X injection latency +# Measure in gsp_emu_inject_completion_interrupt() +``` + +--- + +## 11. Git Workflow + +### Restore Point +```bash +oiv-restore # git checkout f9ed5d8 -- . && git clean -fd +``` + +### Branch for Refactor +```bash +cd $OPEN_IOV_ROOT +git checkout -b refactor/host-gsp-emulation +# Implement changes per docs/refactored-implementation-to-do.md +``` + +### Commit Changes +```bash +git add . +git commit -m "Description of changes" +``` + +### LFS for Large Files +```bash +git lfs ls-files # Shows tracked large files +git lfs track "*.ko" "*.bin" "*.iso" "*.img" "*.o" +``` + +--- + +## 12. Common Issues & Solutions + +| Issue | Solution | +|-------|----------| +| `nvidia.ko` won't load | Reboot to clear kernel cmdline blacklist | +| Guest `nvidia.ko` probe fails | Check PCI config space, BAR sizes, MSI-X caps | +| `nvidia-smi` shows "No devices" | BAR mapping failed - check VFIO region mmap | +| CUDA kernel hangs | GSP RPC timeout - check doorbell polling, shared memory | +| MSI-X not delivered | eventfd not signaled - check `gsp_emu_inject_completion_interrupt()` | +| Migration fails | State mismatch - checksum all migrated structures | +| Windows guest Code 43 | Driver signature/config - check PCI config space, MSI-X table | +| Build fails | `make clean && make` or check kernel headers match running kernel | + +--- + +## 13. Extending the Shell Environment + +### Adding New Helpers +When you create new scripts or workflows, add them to `~/.zshrc`: + +```bash +# Example: Add new test command +alias oiv-test-cuda="$OPEN_IOV_ROOT/scripts/test_cuda_workloads.sh" +alias oiv-test-gfx="$OPEN_IOV_ROOT/scripts/test_graphics_workloads.sh" + +# Example: Add firmware analysis shortcut +alias oiv-fw-new="cd $OPEN_IOV_ROOT && python3 tools/firmware_analyzer.py firmware/57X-GSP/570.144/gsp-XXX.fwimage.bin" + +# Reload +source ~/.zshrc +``` + +### Document New Helpers +Update `oiv-help()` function in `~/.zshrc` to include new commands. + +--- + +## 14. Key Files Reference + +| File | Purpose | +|------|---------| +| `Handoff-vGPU-IO.md` | Main handoff, current state, next steps | +| `AGENTS.md` | Agent brain - workflow, tools, architecture | +| `docs/PORTABLE_ARCHITECTURE.md` | Multi-version/hardware design | +| `docs/refactored-architecture.md` | Host-only emulation design | +| `docs/refactored-implementation-to-do.md` | Granular implementation TODO | +| `scripts/load_vgpu_stack.sh` | Load all kernel modules | +| `scripts/create_mdev.sh` | Create mdev device | +| `scripts/launch_qemu_vgpu.sh` | Launch QEMU with vGPU | +| `scripts/test_vgpu_stack.sh` | Comprehensive test suite | +| `scripts/debug_guest_init.sh` | NEW: One-command debug launch | +| `scripts/windows_guest_automation.sh` | NEW: Windows guest automation | +| `tools/guest_init_monitor.py` | NEW: Guest init monitor | +| `tools/failure_analyzer.py` | NEW: Failure analyzer | +| `tools/benchmark_infrastructure.sh` | NEW: Benchmark runner | +| `tools/guest_agent/guest_test_agent.py` | NEW: Guest test agent | +| `kernel/gsp_emulation.c/h` | GSP emulation engine | +| `kernel/firmware_analyzer.c/h` | Firmware analyzer | +| `kernel/dynamic_rpc_dispatcher.c/h` | Dynamic RPC dispatcher | +| `kernel/architecture/architecture.c/h` | Hardware abstraction | +| `kernel/version/version.c/h` | Version adaptation | +| `kernel/tests/gsp_emulation_test.c` | NEW: Kernel unit tests | +| `kernel/gsp_rpc_client.c` | Firmware-decompiled RPC handlers | +| `kernel/gsp_rpc_integration.c` | Integration layer | +| `kernel/main.c` | Module entry, mdev registration | +| `kernel/bar_emulation.c` | BAR + MSI-X emulation | +| `userspace/lib/libvgpu.c` | Userspace API | +| `userspace/lib/rpc_emulator.c` | Emulation fallbacks | + +--- + +*Development Workflow Guide Version: 1.0* +*Repository: $OPEN_IOV_ROOT* +*Last Updated: $(date)*
\ No newline at end of file diff --git a/docs/GSP_EMULATION_ARCHITECTURE.md b/docs/GSP_EMULATION_ARCHITECTURE.md new file mode 100644 index 0000000..8da8f49 --- /dev/null +++ b/docs/GSP_EMULATION_ARCHITECTURE.md @@ -0,0 +1,201 @@ +# Open-IOV GSP Emulation Engine — Architecture + +> **Audience**: developers working on `kernel/gsp_emulation.c`, `kernel/gsp_rpc_client.c`, +> `kernel/bar_emulation.c`, or debugging guest driver bring-up. +> +> **Scope**: how the host-side GSP emulator actually works today (post-2026-09-21 +> fixes). For multi-version/portability design see `PORTABLE_ARCHITECTURE.md`; +> for the original refactor plan see `refactored-architecture.md`. + +--- + +## 1. Purpose + +An NVIDIA guest driver (stock `nvidia.ko`, no Open-IOV code in the guest) expects +to talk to a **GSP (GPU System Processor)** running on the GPU: it writes RPC +messages to a mailbox in BAR0, rings a doorbell, and waits for a completion +interrupt plus a response written back into the mailbox. + +Open-IOV has no real GSP for the guest (the physical GPU's GSP belongs to the +host driver). Instead, `open-iov-vfio.ko` **emulates the entire GSP on the +host**: a per-mdev kernel thread watches the guest's emulated BAR0, consumes +RPCs, performs them in software (or declines them), writes responses back, and +injects MSI-X interrupts. + +``` +┌──────────────────────────── Guest VM ────────────────────────────┐ +│ stock nvidia.ko (570.144) │ +│ │ writel(doorbell), memcpy(RPC → BAR0+0x1000) │ +│ ▼ │ +│ vGPU BAR0 (guest phys) ── VFIO mmap ──► host bar0_shadow │ +└──────────────────────────────────────────────────────────────────┘ + │ (16 MB vzalloc'd shadow, uncached) +┌──────────────────────────── Host ────────────────────────────────┐ +│ gsp_emu thread (open-iov-vfio.ko) │ +│ poll shadow+0x110C00 (doorbell) │ +│ → read RPC msg from shadow+0x1000 │ +│ → dispatch (12 RPC types, emulation implementations) │ +│ → write response to shadow+0x1080 │ +│ → set PBA bit at shadow+0x3000 │ +│ → eventfd_signal(msix_trigger[0]) → guest MSI-X │ +└──────────────────────────────────────────────────────────────────┘ +``` + +--- + +## 2. The single source of truth: `bar0_shadow` + +The guest's BAR0 is **not** the physical GPU's BAR0. It is a 16 MB software +shadow allocated with `vzalloc()` in `nv_vgpu_bar_emul_init()` +(`bar_emulation.c`) and mapped into the guest via the VFIO `mmap` handler +(`main.c: nv_vgpu_vfio_dev_dev_mmap`, per-page `vmalloc_to_pfn`). + +Everything the guest driver does with BAR0 lands in this shadow: + +| BAR0 offset | Region | Guest writes go to | Emulation reads from | +|------------------|-----------------------------------|--------------------|----------------------| +| `0x0000-0x0FFF` | (shadow register area) | shadow | shadow | +| `0x1000` | RPC request mailbox (72 B msg) | shadow | `shared_mem_virt` | +| `0x1080` | RPC response mailbox (60 B) | shadow (written by emu) | guest reads | +| `0x2000` | MSI-X table (32 × 16 B) | shadow | (guest-managed) | +| `0x3000` | MSI-X PBA (pending bits) | shadow (written by emu) | guest reads | +| `0x110C00` | **GSP doorbell** | shadow | thread polls this | +| rest | emulated register space | shadow | shadow | + +> **Historical bug (fixed 2026-09-21)**: the emulation engine previously polled +> the *host physical* BAR0 (`vgpu->bars[0].vaddr`, an `ioremap` of the real +> GPU) for the doorbell and read RPCs from a separate `dma_alloc_wc` buffer — +> it could never see guest traffic and even wrote PBA bits into the real GPU's +> BAR0+0x3000. `gsp_emu_init()` now binds exclusively to `bar0_shadow`. +> `ctx->shared_mem_dma == 0` marks the shared buffer as a shadow alias so +> cleanup never `dma_free`s it. + +### 2.1 Transport protocol (as emulated) + +1. Guest writes the 72-byte `struct gsp_rpc_msg` to `BAR0+0x1000` + (`type`, `subcmd`, `payload[6]`, `seq_id`, `flags`). +2. Guest writes a non-zero value to the doorbell at `BAR0+0x110C00`. +3. The emulation thread sees the doorbell, ACKs it (writes 0), validates + `seq_id != 0`, and calls `gsp_emu_dispatch_rpc()`. +4. Dispatch fills a 60-byte `struct gsp_rpc_response` (`status`, `type`, + `result[6]`, `seq_id`) written to `BAR0+0x1080`. +5. Thread sets the vector-0 PBA bit at `BAR0+0x3000` and signals the wired + eventfd → QEMU delivers MSI-X to the guest. +6. Stats: the **thread path** increments `processed_rpcs`/`failed_rpcs` and + accumulates `total_latency_ns` (read via `gsp_emu_get_stats()`; note the + public `gsp_emu_process_rpc()` entry does not bump counters). + +--- + +## 3. RPC dispatch + +`gsp_emu_dispatch_rpc()` (`gsp_emulation.c`) switches on `msg->type` and calls +the `gsp_rpc_*()` API (`gsp_rpc_client.c`), which routes to an **emulation +implementation** when `gsp_rpc_emulation_is_enabled()` (set once per mdev by +`gsp_emu_init`). + +| Type (val) | Handler path in emulation | Notes | +|----------------|--------------------------------------------------------|-------| +| ALLOC (0x01) | `gsp_rpc_alloc_emul` — host pages + `dma_map_page` | tracks `gsp_vram_alloc` list | +| FREE (0x02) | `gsp_rpc_free_emul` | | +| MAP (0x03) | `gsp_rpc_map_emul` | | +| UNMAP (0x04) | `gsp_rpc_unmap_emul` | | +| PIN (0x05) | `gsp_rpc_pin_emul` | | +| UNPIN (0x06) | `gsp_rpc_unpin_emul` | | +| SCHEDULE (0x07)| local workqueue (`gsp_sched_wq`), completes immediately| no real work submission | +| YIELD (0x08) | no-op success | | +| NOTIFY (0x09) | log-only in dispatch | no eventfd delivery yet | +| BAR_TRAP (0x0A)| **`-ENOTSUPP`** (fail-fast) | guest BAR access is direct MMIO, no RPC needed; real shadow-register handler TODO | +| MIGRATE (0x0B) | **`-ENOTSUPP`** (fail-fast) | 4-phase host protocol lives in `migration.c`; RPC wiring TODO | +| DEBUG (0xFF) | log-only in dispatch | | +| unknown | `-ENOTSUPP` | | + +> **Livelock fixed (2026-09-21)**: BAR_TRAP/MIGRATE previously fell through to +> `gsp_rpc_submit_raw()`, which writes the request into the same shadow mailbox +> and **rings the same doorbell the thread polls** — a self-triggered infinite +> RPC loop plus multi-second stalls. Emulation mode now fails these fast with +> `pr_warn_ratelimited`. + +### Memory model (corrected 2026-09-22) + +Guest BAR1 (VRAM aperture) is backed by a **dedicated host-RAM vzalloc buffer** +(256 MB, `bar_init.c`), NOT the host GPU's physical BAR1. Guest VRAM accesses +hit our clean host-RAM backing. This is a placeholder — for real CUDA compute, +the memory model must evolve (see `refactored-architecture.md` Phase 2): + +- **Current (Phase 1)**: Guest VRAM = host RAM (safe, no host GPU corruption). + Guest driver probes but cannot compute. +- **Phase 2**: Guest VRAM backed by DMA-coherent host memory or physical GPU + VRAM partition (required for CUDA). +- **Phase 3**: Guest channel/pushbuffer work forwarded to physical GPU + channels (required for compute). + +Guest BAR3 (instance memory window) is also a **vzalloc'd shadow** (32 MB, +`bar_init.c`), NOT the physical GPU's BAR3. Guest BAR3 writes must not reach +the host GPU's instance memory (root cause of host GSP crash, fixed `85d0668`). + +--- + +## 4. Interrupt delivery (MSI-X) + +1. QEMU wires vectors with `VFIO_DEVICE_SET_IRQS` + `DATA_EVENTFD`. + Our ioctl handler takes `eventfd_ctx_fdget()` references into + `vgpu->irqs[vector].trigger` (fd `-1` clears; close releases all refs). +2. On RPC completion the thread calls `gsp_emu_inject_completion_interrupt()`: + sets the PBA bit in the shadow and `eventfd_signal()`s the **live** trigger + from `vgpu->irqs[0]` (looked up per injection — QEMU wires eventfds *after* + `gsp_emu_init`, so an init-time copy would be permanently NULL; this was a + real bug fixed 2026-09-21). + +INTx/MSI/ERR vectors are wired the same way (stored per-vector); the emulator +currently fires vector 0 for GSP completion. + +--- + +## 5. Lifecycle + +``` +mdev create (nv_vgpu_mdev_create) + └─ schedule init_work + └─ nv_vgpu_bar_init → nv_vgpu_bar_emul_init (bar0_shadow) + └─ nv_vgpu_irq_init / nv_vgpu_memory_init + └─ gsp_emu_init + ├─ bind ctx to bar0_shadow (doorbell/shared/mem aliases) + ├─ gsp_client_init_emulation (RPC client, emulation_mode=true) + ├─ gsp_rpc_emulation_set(true) + └─ kthread_run(gsp_emu_thread_fn) + +mdev remove (nv_vgpu_mdev_remove) + └─ cancel init_work, wait for VFIO fds to close + └─ vfio_unregister_group_dev + └─ cleanup work: migration → gsp_emu (kthread_stop w/ 5 s completion) + → memory → irq → BAR (frees bar0_shadow) + └─ list_del, pci_dev_put, vfio_put_device +``` + +--- + +## 6. Debugging the emulator + +- Host dmesg: `dmesg -T | grep -iE 'gsp|vgpu|open-iov'` +- Stats: `gsp_emu_get_stats()` (or add a debugfs readout; counters update on + the thread path only) +- Guest side: serial log `/tmp/vgpu-guest-serial.log`, QEMU GDB stub + `tcp::1234` — see `DEBUGGING.md` for the full workflow and the validation + checklist. +- Unit test: `kernel/tests/gsp_emulation_test.ko` drives + `gsp_emu_process_rpc()` with crafted mailbox messages (requires + `open-iov-vfio.ko` loaded; `depends:` is set accordingly). + +--- + +## 7. Key files + +| File | Role | +|------|------| +| `kernel/gsp_emulation.c/h` | thread, doorbell protocol, dispatch, MSI-X injection | +| `kernel/gsp_rpc_client.c/h` | RPC API + emulation implementations + message structs | +| `kernel/bar_emulation.c/h` | `bar0_shadow` allocation, MSI-X shadow state | +| `kernel/main.c` | VFIO ops: mmap (shadow mapping), SET_IRQS (eventfd wiring), config space | +| `kernel/memory.c` | simple mapping/pin tracking (host management API) | +| `kernel/migration.c` | 4-phase live-migration skeleton (not RPC-wired yet) | diff --git a/docs/ONBOARDING.md b/docs/ONBOARDING.md new file mode 100644 index 0000000..1d0a4f7 --- /dev/null +++ b/docs/ONBOARDING.md @@ -0,0 +1,166 @@ +# Open-IOV Developer Onboarding + +> Welcome. This guide gets you from zero to a productive contributor in ~30 +> minutes: what the project is, how the repo is laid out, how to build and +> test, and the standards we hold the code to. + +--- + +## 1. What this project is (60 seconds) + +**Open-IOV** reimplements NVIDIA vGPU so an **unmodified NVIDIA guest driver** +runs in a VM with full GPU functionality — no GRID license. The trick: the +host kernel module (`open-iov-vfio.ko`) emulates the GPU's **GSP** (the +firmware processor the driver talks to) in software, so the guest's stock +`nvidia.ko` can't tell it isn't running on real hardware. + +Everything guest-facing is **host-side only**. There is no Open-IOV code in +the guest. + +How we got the protocol: full reverse engineering of NVIDIA's GSP 570.144W +firmware (28.5 MB RV64IMAC ELF → 132 decompiled C files → 12 RPC types +mapped). See `wiki/` for the firmware analysis and `docs/GSP_EMULATION_ARCHITECTURE.md` +for how the emulator works. + +## 2. Repository tour + +``` +open-iov/ +├── Makefile # top-level: all/kernel/userspace/examples/tests/clean +├── install.sh # fresh-machine reproduction (deps→nvidia→firmware→build→load) +├── README.md # project intro + quick start +├── STATE.md # live project state (works/limitations/bugs) +├── kernel/ # ★ the host driver (open-iov-vfio.ko) +│ ├── main.c # module init, mdev parent/driver, VFIO ops, PCI config +│ ├── gsp_emulation.c/h # GSP emulation engine (thread, dispatch, MSI-X) +│ ├── gsp_rpc_client.c/h # RPC API + emulation implementations + msg structs +│ ├── bar_emulation.c/h # BAR0 shadow + MSI-X shadow state +│ ├── bar_init.c # physical BAR mapping (host side) +│ ├── memory.c # simple mapping/pin tracking +│ ├── irq.c # IRQ/eventfd plumbing +│ ├── ioctl.c # host management ioctls (libiov consumers) +│ ├── migration.c # 4-phase live-migration skeleton +│ ├── firmware_analyzer.c/h # runtime firmware signature/pattern analysis +│ ├── dynamic_rpc_dispatcher.c/h # version-aware RPC dispatch +│ ├── architecture/ # hw abstraction (Turing/Ampere/Blackwell) +│ ├── version/ # driver-version adaptation (550/560/570/580) +│ ├── include/ # kernel headers (nvidia-vgpu-vfio.h is the main one) +│ └── tests/ # kernel unit tests (gsp_emulation_test) +├── userspace/ +│ ├── lib/ # libiov.so (+ libvgpu.so compat) management API +│ ├── examples/ # example_vgpu test app +│ └── tests/ +├── legacy-driver-guest/ # historical guest-side driver (reference only, NOT built) +├── firmware/57X-GSP/570.144/ # GSP blob + analysis scripts (per-version dirs) +├── nvidia-open-gpu-kernel-modules/ # NVIDIA OpenRM source (patched, builds nvidia.ko) +├── patches/nvidia/ # our delta vs upstream OpenRM tag +├── scripts/ # stack_manager.sh (THE tool), vgpu_vm.py (THE harness), helpers +├── tools/ # guest monitor, failure analyzer, benchmarks, ghidra +├── config/ # declarative YAML: hardware/, versions/ +├── docs/ # ★ all developer documentation (start here) +└── wiki/ # firmware analysis pages (RPC format, addresses, ...) +``` + +## 3. Your first build & test + +```bash +cd open-iov +./scripts/stack_manager.sh build-all # kernel → nvidia → userspace (ORDER MATTERS) +sudo ./scripts/stack_manager.sh load-stack +sudo ./scripts/stack_manager.sh mdev-create +./scripts/stack_manager.sh status +``` + +Rules that will save your week (also enforced in `stack_manager.sh`): + +1. **Build order**: `build-kernel` → `build-nvidia` → `build-userspace`. + nvidia.ko resolves our symbols via `KBUILD_EXTRA_SYMBOLS=kernel/Module.symvers`. +2. **Load order**: prereqs → `open-iov-vfio.ko` → `nvidia.ko` → firmware. +3. **Unload order**: remove mdevs → `rmmod nvidia` → `rmmod open_iov_vfio`. +4. **Version matrix**: guest driver == host OpenRM == host GSP firmware == + emulation target (all **570.144** today). Mismatches look exactly like + emulation bugs — check versions first (`docs/DEBUGGING.md` §7). +5. **VM access**: ONLY through `scripts/vgpu_vm.py` (the QEMU harness). + Never raw `` or `qemu-system-*` commands. + +Run the unit tests (host, needs the module loaded): + +```bash +make tests +sudo insmod kernel/tests/gsp_emulation_test.ko # dmesg → ALL TESTS PASSED +sudo rmmod gsp_emulation_test +``` + +## 4. Coding standards + +We follow the **Linux kernel coding style** for kernel code and common-sense +C for userspace. The practical checklist: + +### Must +- **Tabs** for indentation in kernel code (8-wide). Userspace/scripts: match the + surrounding file. +- **Guard every allocation** (`kmalloc`/`vzalloc`/`kzalloc`) and propagate + `-ENOMEM` with clean unwinding (`goto err_...` in reverse init order). +- **Never `virt_to_phys()` on vmalloc/ioremap memory.** Use + `is_vmalloc_addr() ? vmalloc_to_pfn() : virt_to_phys()>>PAGE_SHIFT` + (see `nv_vgpu_mmap_virt_to_pfn`). This class of bug wedged our guest for a + whole session — it is called out in code review. +- **`dma_addr_t` for DMA addresses.** Never truncate into `int`/`u32` + (IOMMU IOVAs can exceed 32 bits). +- **Every `get` needs a `put`**: `pci_dev_get/pci_dev_put`, + `eventfd_ctx_fdget/eventfd_ctx_put`, `vfio_alloc_device/vfio_put_device`, + module refs. Unbalanced gets become leaks that block unload. +- **Remove paths must fully unwind init paths** (we had a remove that forgot + `list_del` + kthread stop + VFIO unregister — leaked everything). +- **Fail fast in emulation mode**: never `gsp_rpc_submit_raw()` from emulation + (it rings the doorbell our own thread polls → self-trigger livelock). + Return `-ENOTSUPP` with `pr_warn_ratelimited` until a real handler exists. +- **`pr_*`/`dev_*` logging** with the `nv_vgpu_*` wrappers (NULL-tolerant); + rate-limit anything a guest can spam. +- Comments explain **why**, not what — especially around hardware/firmware + behavior discovered by reverse engineering (cite the wiki/address). +- **Signed-off-by** style commit messages: short imperative summary line, + then the *why* (root cause), the *what*, and the user-visible effect. + Reference real dmesg/GDB evidence where relevant. + +### Must not +- No guest-side Open-IOV code. Ever. The guest is stock. +- No reboots/system-affecting commands from scripts without explicit operator + action (see the project's hard rule). +- No new `MODULE_SOFTDEP`/`depends` on `nvidia` — open-iov-vfio is standalone; + nvidia.ko imports *our* symbols, not vice versa. +- No direct ``/QEMU invocations in tooling — go through `vgpu_vm.py`. +- Don't widen an API's hardware assumptions silently: if a function needs + `vgpu->pdev`, say so in the comment (unit tests can't call those paths). + +### Kernel module specifics +- One module: `open-iov-vfio.ko` (`obj-m` in `kernel/Makefile`). +- Exported symbols: only what other components genuinely need + (`EXPORT_SYMBOL` right after the definition). +- New per-device resources belong in `struct nv_vgpu_device` with matching + init/cleanup in the `nv_vgpu_device_init_work` / cleanup-work chain. + +## 5. Documentation expectations + +- New subsystem → new doc in `docs/` + a link from `README.md` and this file. +- Changed behavior → update the affected doc in the **same commit**. +- Debugging discoveries → add to `docs/DEBUGGING.md` taxonomy table. +- Firmware addresses/structures → `wiki/Address-Reference.md` and friends. + +## 6. Where to read next (in order) + +1. `docs/GSP_EMULATION_ARCHITECTURE.md` — how the emulator works +2. `docs/DEBUGGING.md` — gdb stub/serial workflow + validation checklist +3. `docs/PORTABLE_ARCHITECTURE.md` — multi-version/multi-hardware design +4. `docs/VERSION_UPGRADE.md` — supporting a new driver version end-to-end +5. `docs/PERFORMANCE.md` — benchmarking and A/B vs bare metal +6. `docs/DEVELOPMENT_WORKFLOW.md` — exhaustive command reference +7. `wiki/` — firmware reverse-engineering canon + +## 7. Getting help / escalating + +- Stuck on firmware analysis? `tools/ghidra/` scripts + `wiki/`. +- Guest driver won't probe? `docs/DEBUGGING.md` §5-6 first. +- GSP emulation questions? `docs/GSP_EMULATION_ARCHITECTURE.md` + the unit test. +- Found a new failure mode? Debug it per the guide, then **write it down** in + the taxonomy table so the next person doesn't rediscover it. diff --git a/docs/PERFORMANCE.md b/docs/PERFORMANCE.md new file mode 100644 index 0000000..6077920 --- /dev/null +++ b/docs/PERFORMANCE.md @@ -0,0 +1,98 @@ +# Open-IOV Performance Testing & A/B Methodology + +> How we measure vGPU performance against bare metal, interpret the results, +> and keep overhead on the critical path honest. **Run correctness first** +> (`docs/DEBUGGING.md` §5) — numbers from a half-working stack are noise. + +--- + +## 1. What overhead is acceptable (targets) + +| Path | Target | Why | +|------|--------|-----| +| GSP RPC round-trip (doorbell→response→MSI-X) | < 50 µs (stretch: < 10 µs) | every driver op goes through here | +| MSI-X injection (eventfd → guest handler) | < 5 µs | completion latency dominates | +| Guest BAR0 MMIO read/write | < 2 µs (shadow RAM) | hot path for register polling | +| CUDA kernel launch (added overhead vs bare metal) | < 10% | user-visible | +| Memory alloc/map RPCs | < 100 µs | setup-time, amortized | +| CUDA compute throughput | ≥ 95% of bare metal | GPU-bound, should be ~free | +| Graphics (frame rate) | ≥ 90% of bare metal | swapchain + submission overhead | + +## 2. The A/B rule + +Every measurement runs **twice on the same machine, same boot if possible**: + +- **A (bare metal)**: workload on the physical GPU via the host's stock + driver (or our OpenRM nvidia.ko directly on the hardware). +- **B (vGPU)**: identical workload in the guest on the emulated vGPU. + +Same binary, same input sizes, same environment (governor, clocks, hugepages). +Report **B/A ratio %** — never absolute numbers alone. + +One variable at a time. If B regresses, you changed the stack, the workload, +or the machine — find which before optimizing. + +## 3. Microbenchmarks (build these first) + +1. **RPC round-trip**: from the guest, submit a DEBUG/NOOP-ish RPC in a loop; + time doorbell-write → response-visible. Host side: `gsp_emu_get_stats()` + gives `total_latency_ns` and counts — divide for per-RPC host cost. + (Add `trace_printk` in `gsp_emu_dispatch_rpc` entry/exit for distribution; + read via `/sys/kernel/debug/tracing/trace`.) +2. **MSI-X latency**: timestamp (guest TSC) at RPC submit → in the guest + driver's interrupt handler. Compare with the host eventfd_signal timestamp. +3. **BAR0 shadow read latency**: guest loops reading a shadow register; + expect ~memory speed (uncached, so ~100-300 ns) — anything ≥ 1 µs means + you're trapping somewhere you shouldn't. +4. **Doorbell polling cost**: host `top`/`perf` on `gsp_emu_N` — the adaptive + poll (busy → 50-100 µs sleep) should sit at ~0% CPU when idle, and react + within one sleep quantum under load. Tune `GSP_EMU_POLL_*` only with data. + +## 4. Workload benchmarks + +Runner: `tools/benchmark_infrastructure.sh` (extend it; keep the A/B pattern). +In-guest execution goes through the harness (`vgpu_vm.py exec`) or the guest +test agent (`tools/guest_agent/`). + +| Suite | What | Pass criterion | +|-------|------|----------------| +| `cuda-samples vectorAdd` | smoke + element-wise throughput | correct result; ≥ 95% B/A | +| `simpleCUBLAS` | SGEMM TFLOPS | ≥ 95% B/A | +| `nvbandwidth` (if available) | H2D/D2H/device copy GB/s | report; DMA path validation | +| `glxgears -info` / a real GL app | frame rate | ≥ 90% B/A | +| `nvenc`/`nvdec` sample | transcode fps | functional (perf later) | +| PyTorch CUDA op (matmul loop) | end-to-end framework | ≥ 90% B/A | + +Always log: guest/host driver versions, GPU clocks (`nvidia-smi -q -d CLOCK` +both sides — throttling skews everything), and the command lines. + +## 5. Where the overhead lives (profile in this order) + +1. **RPC dispatch** — the single biggest lever. Everything the guest driver + does funnels through the emulation thread. +2. **Doorbell polling cadence** — too slow = latency; too hot = host CPU burn. +3. **eventfd/MSI-X delivery** — measure, don't assume. +4. **BAR0 uncached semantics** — shadow is UC for the guest; that's + MMIO-correct but slow for large reads. If the driver bulk-reads shadow, + consider caching-safe sub-regions. +5. **Memory model gap** (ALLOC host-pages vs BAR1 direct VRAM) — see + `docs/GSP_EMULATION_ARCHITECTURE.md` §3. Until unified, DMA-heavy + workloads are not representative. + +## 6. Optimizing safely + +- Change one thing, re-run the A/B pair, keep the numbers with the commit. +- Never optimize off bare-metal-only data: the goal is B/A, not A. +- If a change makes B faster but breaks the §5 validation checklist in + `docs/DEBUGGING.md`, it doesn't count. +- Record results in `STATE.md` (date, versions, B/A ratios) so regressions + are visible across commits. + +## 7. Reporting template + +``` +Date / commit / versions (guest=host=firmware=emulation): +Machine / GPU / clocks locked?: +Test, unit, A (bare), B (vGPU), B/A %: +Notes (what changed vs last run): +``` diff --git a/docs/PORTABLE_ARCHITECTURE.md b/docs/PORTABLE_ARCHITECTURE.md new file mode 100644 index 0000000..47d0297 --- /dev/null +++ b/docs/PORTABLE_ARCHITECTURE.md @@ -0,0 +1,742 @@ +# Open-IOV vGPU-IO Portable Architecture Design + +## Design Goals + +### 1. Driver Version Portability +- **Support NVIDIA OpenRM 550+** without code changes per version +- **Automatic version detection** at runtime +- **Backward/forward compatibility** for RPC interfaces +- **Graceful degradation** when features unavailable + +### 2. Hardware Portability +- **Support multiple GPU architectures**: Turing (TU102/TU104), Ampere (GA100/GA102), Hopper (GH100), Blackwell (GB202/GB203/GB100), future +- **Support multiple form factors**: Consumer (RTX 3090/4090/5090), Data Center (A100/H100/B200), DGX Spark, DGX Station +- **Automatic hardware detection** via PCI device IDs and firmware signatures +- **Capability-based feature enablement** + +### 3. Zero Guest Modification +- **Guest runs stock NVIDIA driver** from NVIDIA website +- **No Open-IOV code in guest** kernel or userspace +- **Standard VFIO/mdev interface** only + +--- + +## Core Architecture: Three-Layer Abstraction + +``` +╔═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗ +║ PORTABLE ARCHITECTURE LAYERS ║ +╠═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╣ +║ ║ +║ ┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ +║ │ LAYER 1: HARDWARE ABSTRACTION LAYER (HAL) │ ║ +║ │ ┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ +║ │ │ GpuArchitecture (abstract base) │ │ ║ +║ │ │ ├── detect(pci_dev) → GpuArchitecture* │ │ ║ +║ │ │ ├── get_pci_device_ids() → vector<DeviceID> │ │ ║ +║ │ │ ├── get_bar_layout() → BarLayout │ │ ║ +║ │ │ ├── get_gsp_firmware_info() → FirmwareInfo │ │ ║ +║ │ │ ├── get_rpc_dispatcher_addresses() → RpcAddresses │ │ ║ +║ │ │ ├── get_capabilities() → GpuCapabilities │ │ ║ +║ │ │ ├── create_rpc_handler(rpc_type) → RpcHandler* │ │ ║ +║ │ │ └── create_bar_emulator() → BarEmulator* │ │ ║ +║ │ │ │ │ ║ +║ │ │ Concrete Implementations (per architecture): │ │ ║ +║ │ │ ├── TuringArchitecture (TU102, TU104, TU106, TU116, TU117) │ │ ║ +║ │ │ ├── AmpereArchitecture (GA100, GA102, GA103, GA104, GA106, GA107) │ │ ║ +║ │ │ ├── HopperArchitecture (GH100, GH200) │ │ ║ +║ │ │ ├── BlackwellArchitecture (GB202, GB203, GB205, GB206, GB100) │ │ ║ +║ │ │ └── FutureArchitecture (auto-generated from firmware analysis) │ │ ║ +║ │ └─────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ +║ └─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ +║ ║ +║ ┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ +║ │ LAYER 2: VERSION ADAPTATION LAYER (VAL) │ ║ +║ │ ┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ +║ │ │ DriverVersion (semantic: major.minor.patch) │ │ ║ +║ │ │ ├── parse(version_string) → DriverVersion │ │ ║ +║ │ │ ├── is_compatible(min_version, max_version) → bool │ │ ║ +║ │ │ ├── get_rpc_interface_version() → RpcInterfaceVersion │ │ ║ +║ │ │ ├── get_firmware_signature() → string (SHA256 of firmware) │ │ ║ +║ │ │ └── get_feature_flags() → FeatureFlags │ │ ║ +║ │ │ │ │ ║ +║ │ │ RpcInterfaceVersion (per driver version): │ │ ║ +║ │ │ ├── get_rpc_type_count() → int │ │ ║ +║ │ │ ├── get_rpc_struct_layout(rpc_type) → StructLayout │ │ ║ +║ │ │ ├── get_subcommand_mapping() → SubcommandMap │ │ ║ +║ │ │ ├── get_error_codes() → ErrorCodeMap │ │ ║ +║ │ │ └── get_capability_bits() → CapabilityBits │ │ ║ +║ │ │ │ │ ║ +║ │ │ Version Registry (loaded at runtime): │ │ ║ +║ │ │ ├── 550.x → RpcInterfaceV1 │ │ ║ +║ │ │ ├── 560.x → RpcInterfaceV2 (added MIGRATE subcommands) │ │ ║ +║ │ │ ├── 570.x → RpcInterfaceV3 (added DEBUG, NOTIFY enhancements) │ │ ║ +║ │ │ ├── 580.x → RpcInterfaceV4 (confidential computing, TEE) │ │ ║ +║ │ │ └── auto-detect from firmware signature │ │ ║ +║ │ └─────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ +║ └─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ +║ ║ +║ ┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ +║ │ LAYER 3: FIRMWARE-DRIVEN RPC ENGINE (FDRE) │ ║ +║ │ ┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ +║ │ │ FirmwareAnalyzer (runs at module load) │ │ ║ +║ │ │ ├── load_firmware(firmware_path) → FirmwareImage │ │ ║ +║ │ │ ├── find_rpc_dispatcher() → address │ │ ║ +║ │ │ ├── find_jump_table() → address │ │ ║ +║ │ │ ├── find_handler_addresses() → map<rpc_type, address> │ │ ║ +║ │ │ ├── extract_struct_layouts() → map<rpc_type, StructLayout> │ │ ║ +║ │ │ ├── compute_firmware_signature() → SHA256 │ │ ║ +║ │ │ └── validate_against_known_versions() → DriverVersion │ │ ║ +║ │ │ │ │ ║ +║ │ │ DynamicRpcDispatcher: │ │ ║ +║ │ │ ├── dispatch(rpc_msg) → uses discovered handler addresses │ │ ║ +║ │ │ ├── fallback_to_emulation(rpc_type) → if handler not found │ │ ║ +║ │ │ └── hot_patch_handler(rpc_type, new_address) → for live updates │ │ ║ +║ │ │ │ │ ║ +║ │ │ EmulationFallbackRegistry: │ │ ║ +║ │ │ ├── register_emulation(rpc_type, EmulationHandler*) │ │ ║ +║ │ │ ├── get_emulation(rpc_type) → EmulationHandler* │ │ ║ +║ │ │ └── supports_emulation(rpc_type) → bool │ │ ║ +║ │ └─────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ +║ └─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ +║ ║ +╚════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝ +``` + +--- + +## Configuration-Driven Design + +### 1. Hardware Configuration Files + +```yaml +# config/hardware/turing.yaml +architecture: "turing" +pci_device_ids: + - vendor: 0x10de + device: 0x1e07 # TU102 + name: "RTX 2080 Ti" + - vendor: 0x10de + device: 0x1e08 # TU104 + name: "RTX 2080 / T4" + - vendor: 0x10de + device: 0x1ed3 # TU104 mobile + name: "RTX 2080 Super Mobile" +bar_layout: + bar0_size: 16MB + bar1_size: 256MB + bar2_size: 32MB + bar3_size: 32MB + bar0_prefetchable: true + bar1_prefetchable: true +gsp: + firmware_base: 0x04000000 + doorbell_offset: 0x110C00 + shared_mem_offset: 0x1000 + shared_mem_size: 4KB + rpc_dispatcher_primary: 0x057A1AC0 + rpc_dispatcher_secondary: 0x04ED39DC + rpc_jump_table: 0x059BFEB0 + handlers: + alloc: 0x057E94EC + schedule: 0x051EFBF4 + bar_trap: 0x04CB8188 + wpr_1: 0x04F00B0C + wpr_2: 0x04F00C78 + migration_table_1: 0x04179ED8 + migration_table_2: 0x04179FBC +capabilities: + max_vfs: 7 + supports_mig: true + supports_wpr: true + supports_bar_trap: true + supports_scheduling: true + supports_notifications: true + msi_x_vectors: 32 +``` + +```yaml +# config/hardware/ampere.yaml +architecture: "ampere" +pci_device_ids: + - vendor: 0x10de + device: 0x20b0 # GA100 + name: "A100" + - vendor: 0x10de + device: 0x2204 # GA102 + name: "RTX 3090" + - vendor: 0x10de + device: 0x2206 # GA102 + name: "RTX 3080" +bar_layout: + bar0_size: 16MB + bar1_size: 512MB # Larger FB on Ampere + bar2_size: 64MB + bar3_size: 64MB +gsp: + firmware_base: 0x04000000 + doorbell_offset: 0x110C00 + shared_mem_offset: 0x1000 + # Addresses discovered at runtime via FirmwareAnalyzer +capabilities: + max_vfs: 7 + supports_mig: true + supports_mig_mig: true # MIG (Multi-Instance GPU) + supports_confidential: false + supports_wpr: true + supports_bar_trap: true + msi_x_vectors: 64 +``` + +```yaml +# config/hardware/blackwell.yaml +architecture: "blackwell" +pci_device_ids: + - vendor: 0x10de + device: 0x2480 # GB202 (RTX 5090) + name: "RTX 5090" + - vendor: 0x10de + device: 0x2380 # GB100 (B200) + name: "B200" + - vendor: 0x10de + device: 0x2384 # GB205 + name: "RTX 5070" +bar_layout: + bar0_size: 16MB + bar1_size: 1GB # Blackwell unified memory + bar2_size: 128MB + bar3_size: 128MB +gsp: + firmware_base: 0x04000000 + doorbell_offset: 0x110C00 + shared_mem_offset: 0x1000 +capabilities: + max_vfs: 8 + supports_mig: true + supports_mig_mig: true + supports_confidential: true # TEE support + supports_wpr: true + supports_bar_trap: true + supports_nvlink: true + msi_x_vectors: 128 +``` + +### 2. Driver Version Configuration + +```yaml +# config/versions/550.yaml +version: "550" +rpc_interface_version: 1 +rpc_types: + - ALLOC: 0x01 + - FREE: 0x02 + - MAP: 0x03 + - UNMAP: 0x04 + - PIN: 0x05 + - UNPIN: 0x06 + - SCHEDULE: 0x07 + - YIELD: 0x08 + - NOTIFY: 0x09 + - BAR_TRAP: 0x0A + - MIGRATE: 0x0B + - DEBUG: 0x0C +struct_layouts: + ALLOC: + size: 48 + fields: + - {name: size, offset: 0, type: u64} + - {name: alignment, offset: 8, type: u64} + - {name: mem_type, offset: 16, type: u32} + - {name: flags, offset: 20, type: u32} + - {name: addr_hint, offset: 24, type: u64} + - {name: pid, offset: 32, type: u32} +capabilities: + supports_migrate: true + migrate_subcommands: + - SAVE: 0x04 + - RESTORE: 0x05 + supports_notifications: basic +``` + +```yaml +# config/versions/570.yaml +version: "570" +rpc_interface_version: 3 +extends: "550" +rpc_types: + - NOTIFY: 0x09 # Enhanced with subcommands + - MIGRATE: 0x0B # Enhanced with 4-phase +struct_layouts: + NOTIFY: + size: 32 + fields: + - {name: type, offset: 0, type: u32} + - {name: subcmd, offset: 4, type: u32} + - {name: event_fd, offset: 8, type: u64} + - {name: flags, offset: 16, type: u32} +capabilities: + supports_notifications: enhanced + supports_migrate_4phase: true + migrate_phases: [QUERY, PREPARE, TRANSFER, COMMIT, CLEANUP] +``` + +```yaml +# config/versions/580.yaml +version: "580" +rpc_interface_version: 4 +extends: "570" +rpc_types: + - TEE_ATTEST: 0x0D # New in 580 for confidential computing + - TEE_SEAL: 0x0E + - TEE_UNSEAL: 0x0F +capabilities: + supports_confidential: true + supports_tee: true + tee_attestation_types: [CPU, GPU, MEMORY] +``` + +--- + +## Runtime Detection & Adaptation Flow + +``` +┌─────────────────────────────────────────────────────────────────────────────────────┐ +│ MODULE LOAD: RUNTIME DETECTION FLOW │ +├─────────────────────────────────────────────────────────────────────────────────────┤ +│ │ +│ 1. open-iov-vfio.ko loads │ +│ │ │ +│ ▼ │ +│ 2. Probe PCI device │ +│ │ │ +│ ▼ │ +│ 3. Match PCI Device ID → Hardware Config (turing.yaml, ampere.yaml, etc.) │ +│ │ │ +│ ▼ │ +│ 4. Load Hardware Config → Instantiate GpuArchitecture subclass │ +│ │ │ +│ ▼ │ +│ 5. Load GSP Firmware (from /lib/firmware or host nvidia.ko) │ +│ │ │ +│ ▼ │ +│ 6. FirmwareAnalyzer.analyze(firmware) │ +│ ├── Compute firmware SHA256 │ +│ ├── Find RPC dispatcher via pattern matching │ +│ ├── Find jump table via cross-references │ +│ ├── Extract handler addresses for all 12+ RPC types │ +│ ├── Extract struct layouts via pattern matching │ +│ └── Match firmware signature → DriverVersion (550, 560, 570, 580, etc.) │ +│ │ │ +│ ▼ │ +│ 7. Load Version Config → Instantiate RpcInterfaceVersion │ +│ │ │ +│ ▼ │ +│ 8. Validate: Hardware Config + Version Config compatible? │ +│ │ │ +│ ├── YES → Initialize DynamicRpcDispatcher with discovered addresses │ +│ ├── PARTIAL → Enable emulation for missing handlers │ +│ └── NO → Fail with clear error: "Unsupported GPU/driver combination" │ +│ │ │ +│ ▼ │ +│ 9. Register EmulationFallbackRegistry for all RPC types │ +│ │ │ +│ ▼ │ +│ 10. Start GSP Emulation Thread (per mdev) │ +│ │ +└─────────────────────────────────────────────────────────────────────────────────────┘ +``` + +--- + +## FirmwareAnalyzer - Automatic Address Discovery + +```c +// kernel/firmware_analyzer.c + +struct firmware_signature { + uint8_t sha256[32]; + char version_string[64]; + uint32_t crc32; +}; + +struct discovered_rpc_handlers { + uint64_t primary_dispatcher; + uint64_t secondary_dispatcher; + uint64_t jump_table; + struct { + uint32_t rpc_type; + uint64_t handler_address; + bool found; + } handlers[16]; // Support up to 16 RPC types + uint32_t handler_count; +}; + +struct firmware_analyzer { + const uint8_t *firmware; + size_t firmware_size; + uint64_t firmware_base; + struct discovered_rpc_handlers discovered; + struct firmware_signature signature; +}; + +int firmware_analyzer_init(struct firmware_analyzer *fa, const char *firmware_path) { + // Load firmware file + fa->firmware = load_firmware_file(firmware_path, &fa->firmware_size); + if (!fa->firmware) + return -ENOENT; + + // Compute signature + sha256(fa->firmware, fa->firmware_size, fa->signature.sha256); + fa->signature.crc32 = crc32(0, fa->firmware, fa->firmware_size); + + // Set base address (from hardware config) + fa->firmware_base = get_hardware_config()->gsp.firmware_base; + + return 0; +} + +int firmware_analyzer_discover(struct firmware_analyzer *fa) { + // 1. Find primary dispatcher: search for known pattern + // Pattern: "RPC_DISPATCH" string reference or known instruction sequence + fa->discovered.primary_dispatcher = find_pattern(fa->firmware, fa->firmware_size, + "RPC_DISPATCH", fa->firmware_base); + + // 2. Find jump table: cross-reference from dispatcher + fa->discovered.jump_table = find_jump_table(fa->firmware, fa->firmware_size, + fa->discovered.primary_dispatcher); + + // 3. Parse jump table entries (32 entries for 12 RPC types) + parse_jump_table(fa->firmware, fa->discovered.jump_table, &fa->discovered); + + // 4. Find secondary dispatcher + fa->discovered.secondary_dispatcher = find_secondary_dispatcher(fa->firmware, + fa->firmware_size, fa->discovered.primary_dispatcher); + + // 5. Find specialized handlers via string references + find_handler_by_string(fa, "BAR_TRAP", "BAR1", "ForceBarAccess", &fa->discovered.handlers[GSP_RPC_BAR_TRAP]); + find_handler_by_string(fa, "WPR", "completion", "timeout", &fa->discovered.handlers[GSP_RPC_NOTIFY]); + find_handler_by_string(fa, "ALLOC", "size", "alignment", &fa->discovered.handlers[GSP_RPC_ALLOC]); + find_handler_by_string(fa, "SCHEDULE", "work", "callback", &fa->discovered.handlers[GSP_RPC_SCHEDULE]); + find_handler_by_string(fa, "MIGRATE", "checkpoint", "progress", &fa->discovered.handlers[GSP_RPC_MIGRATE]); + + // 6. Extract struct layouts via pattern matching on handler prologues + extract_struct_layouts(fa); + + // 7. Match signature against known versions + match_firmware_version(fa); + + return 0; +} + +void firmware_analyzer_match_version(struct firmware_analyzer *fa) { + // Compare signature against known database + static const struct { + const char *version; + uint8_t sha256[32]; + uint32_t crc32; + } known_versions[] = { + {"550.163.01", {0x...}, 0x...}, + {"560.28.03", {0x...}, 0x...}, + {"570.144.00", {0x...}, 0x...}, + {"570.144.01", {0x...}, 0x...}, + {"580.00.00", {0x...}, 0x...}, + }; + + for (int i = 0; i < ARRAY_SIZE(known_versions); i++) { + if (memcmp(fa->signature.sha256, known_versions[i].sha256, 32) == 0) { + strncpy(fa->signature.version_string, known_versions[i].version, 63); + return; + } + } + + // Unknown version - use heuristic based on handler count and features + if (fa->discovered.handler_count >= 12) + strcpy(fa->signature.version_string, "570+ (unknown)"); + else if (fa->discovered.handler_count >= 10) + strcpy(fa->signature.version_string, "560+ (unknown)"); + else + strcpy(fa->signature.version_string, "550+ (unknown)"); +} +``` + +--- + +## Dynamic RPC Dispatcher + +```c +// kernel/dynamic_rpc_dispatcher.c + +struct dynamic_rpc_dispatcher { + struct discovered_rpc_handlers *discovered; + struct rpc_interface_version *interface; + struct emulation_fallback_registry *fallbacks; + spinlock_t lock; + uint64_t dispatch_stats[16]; +}; + +int dynamic_rpc_dispatcher_dispatch(struct dynamic_rpc_dispatcher *dr, + struct gsp_rpc_msg *req, + struct gsp_rpc_response *rsp) { + // 1. Validate RPC type + if (req->type >= dr->interface->rpc_type_count) + return -ENOTTY; + + // 2. Check if we have a native handler address + uint64_t handler_addr = 0; + for (int i = 0; i < dr->discovered->handler_count; i++) { + if (dr->discovered->handlers[i].rpc_type == req->type) { + handler_addr = dr->discovered->handlers[i].handler_address; + break; + } + } + + // 3. Dispatch to native handler if available + if (handler_addr) { + return dynamic_rpc_call_native(dr, handler_addr, req, rsp); + } + + // 4. Fallback to emulation + struct emulation_handler *emu = emulation_fallback_get(dr->fallbacks, req->type); + if (emu) { + dr->dispatch_stats[req->type]++; // Track emulation usage + return emu->handle(req, rsp); + } + + // 5. No handler, no emulation + rsp->status = -ENOTTY; + return -ENOTTY; +} + +int dynamic_rpc_call_native(struct dynamic_rpc_dispatcher *dr, + uint64_t handler_addr, + struct gsp_rpc_msg *req, + struct gsp_rpc_response *rsp) { + // Use firmware-decompiled calling convention + // This calls the actual GSP firmware handler via the discovered address + // In emulation mode, we call our C reimplementation instead + + if (gsp_emulation_mode_enabled()) { + // Call our C reimplementation (from gsp_rpc_client.c) + return gsp_rpc_client_call(req->type, req, rsp); + } else { + // Native mode: would call into GSP firmware + // Not used in our architecture (we always emulate in host) + return -ENOTSUPP; + } +} +``` + +--- + +## Capability Negotiation with Guest + +``` +┌─────────────────────────────────────────────────────────────────────────────────────┐ +│ GUEST DRIVER CAPABILITY NEGOTIATION │ +├─────────────────────────────────────────────────────────────────────────────────────┤ +│ │ +│ Guest nvidia.ko probes vGPU: │ +│ 1. Reads PCI Config Space → gets device ID, BAR sizes, capabilities │ +│ 2. Maps BAR0 → finds GSP doorbell + shared memory │ +│ 3. Sends RPC: QUERY_CAPABILITIES (new RPC type or subcommand) │ +│ │ +│ Host DynamicRpcDispatcher receives QUERY_CAPABILITIES: │ +│ 1. Constructs response from: │ +│ - Hardware Config capabilities │ +│ - Version Config capabilities │ +│ - EmulationFallbackRegistry (what we can emulate) │ +│ 2. Returns capability bitmap to guest │ +│ │ +│ Guest nvidia.ko enables features based on response: │ +│ - If MIG supported → enables MIG code paths │ +│ - If WPR supported → enables WPR code paths │ +│ - If CONFIDENTIAL supported → enables TEE code paths │ +│ - If NOTIFY enhanced → uses async eventfd │ +│ │ +│ This allows SAME guest driver to work across ALL hardware/versions! │ +│ │ +└─────────────────────────────────────────────────────────────────────────────────────┘ +``` + +--- + +## Update Strategy for New Driver Versions + +### 1. Adding New Driver Version (e.g., 590) + +```bash +# 1. Add version config +cp config/versions/580.yaml config/versions/590.yaml +# Edit: version, rpc_interface_version, new RPC types, capabilities + +# 2. Test with new firmware +# FirmwareAnalyzer auto-discovers addresses - NO CODE CHANGES NEEDED + +# 3. If new RPC types added: +# - Add to config/versions/590.yaml +# - Add emulation handler in kernel/emulation_fallbacks.c +# - Register in EmulationFallbackRegistry + +# 4. If struct layouts changed: +# - FirmwareAnalyzer extracts new layouts automatically +# - Version config specifies layout version +``` + +### 2. Adding New GPU Architecture (e.g., Rubin R100) + +```bash +# 1. Add hardware config +cp config/hardware/blackwell.yaml config/hardware/rubin.yaml +# Edit: PCI device IDs, BAR layout, capabilities + +# 2. Add GpuArchitecture subclass (if special handling needed) +# kernel/arch/rubin_architecture.c + +# 3. Register in architecture factory +# kernel/architecture_factory.c + +# 4. Test - FirmwareAnalyzer handles address discovery +``` + +### 3. Automated Firmware Analysis Pipeline + +```yaml +# .github/workflows/firmware-analysis.yml +name: Firmware Analysis +on: + schedule: + - cron: '0 0 * * 0' # Weekly + workflow_dispatch: + inputs: + firmware_url: + description: 'NVIDIA firmware download URL' + required: true + +jobs: + analyze: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Download firmware + run: wget -O firmware.bin "${{ inputs.firmware_url }}" + - name: Run FirmwareAnalyzer + run: | + ./tools/firmware_analyzer firmware.bin > analysis.json + - name: Compare with known versions + run: ./tools/compare_analysis.py analysis.json + - name: Generate config if new + run: | + if [ "$(cat analysis.json | jq -r '.is_new')" = "true" ]; then + ./tools/generate_config.py analysis.json > config/versions/NEW.yaml + git add config/versions/NEW.yaml + git commit -m "Add support for new firmware version" + git push + fi +``` + +--- + +## Testing Matrix for Portability + +| GPU Architecture | PCI Device IDs | Driver Versions | Test Status | +|------------------|----------------|-----------------|-------------| +| **Turing** | 10de:1e07, 10de:1e08, 10de:1ed3 | 550, 560, 570, 580 | ✅ Baseline | +| **Ampere** | 10de:20b0, 10de:2204, 10de:2206 | 550, 560, 570, 580 | 🔄 In Progress | +| **Hopper** | 10de:2330, 10de:2334 | 560, 570, 580 | 🔄 Planned | +| **Blackwell** | 10de:2480, 10de:2380, 10de:2384 | 570, 580 | 🔄 Planned | +| **Rubin (Future)** | TBD | 590+ | 📋 Designed | + +### Per-Combination Tests + +```bash +# Automated test matrix +for arch in turing ampere hopper blackwell; do + for version in 550 560 570 580; do + if [ -f "config/hardware/${arch}.yaml" ] && [ -f "config/versions/${version}.yaml" ]; then + echo "Testing ${arch} + ${version}..." + ./scripts/test_combination.sh ${arch} ${version} + fi + done +done +``` + +--- + +## Implementation File Structure (Portable) + +``` +open-iov/ +├── config/ +│ ├── hardware/ +│ │ ├── turing.yaml +│ │ ├── ampere.yaml +│ │ ├── hopper.yaml +│ │ ├── blackwell.yaml +│ │ └── rubin.yaml (future) +│ ├── versions/ +│ │ ├── 550.yaml +│ │ ├── 560.yaml +│ │ ├── 570.yaml +│ │ ├── 580.yaml +│ │ └── 590.yaml (future) +│ └── defaults.yaml +├── kernel/ +│ ├── architecture/ +│ │ ├── architecture.h # Abstract base class +│ │ ├── architecture_factory.c # Detection & instantiation +│ │ ├── turing_architecture.c +│ │ ├── ampere_architecture.c +│ │ ├── hopper_architecture.c +│ │ ├── blackwell_architecture.c +│ │ └── generic_architecture.c # Fallback +│ ├── version/ +│ │ ├── version.h # DriverVersion, RpcInterfaceVersion +│ │ ├── version_registry.c # Loads version configs +│ │ ├── interface_v1.c # 550 interface +│ │ ├── interface_v2.c # 560 interface +│ │ ├── interface_v3.c # 570 interface +│ │ └── interface_v4.c # 580 interface +│ ├── firmware/ +│ │ ├── firmware_analyzer.c # Auto-discovers addresses +│ │ ├── firmware_analyzer.h +│ │ ├── pattern_matcher.c # Binary pattern matching +│ │ └── struct_layout_extractor.c +│ ├── dispatch/ +│ │ ├── dynamic_rpc_dispatcher.c +│ │ ├── dynamic_rpc_dispatcher.h +│ │ ├── emulation_fallback_registry.c +│ │ └── emulation_fallback_registry.h +│ ├── gsp_emulation.c # Uses dynamic dispatcher +│ ├── main.c # Integrates all layers +│ └── Makefile +├── tools/ +│ ├── firmware_analyzer/ # Standalone analysis tool +│ ├── compare_analysis.py +│ ├── generate_config.py +│ └── test_combination.sh +└── docs/ + ├── PORTABLE_ARCHITECTURE.md + └── VERSION_COMPATIBILITY.md +``` + +--- + +## Key Design Principles Summary + +| Principle | Implementation | +|-----------|----------------| +| **No hardcoded addresses** | FirmwareAnalyzer discovers at runtime | +| **No hardcoded struct layouts** | Version config + FirmwareAnalyzer extraction | +| **No hardcoded device IDs** | Hardware config YAML files | +| **No hardcoded capabilities** | Capability negotiation via QUERY_CAPABILITIES RPC | +| **Graceful degradation** | EmulationFallbackRegistry for missing handlers | +| **Forward compatibility** | Version config `extends` mechanism | +| **Automated updates** | Firmware analysis pipeline + config generation | +| **Test matrix** | Automated CI across architecture × version combinations | + +--- + +*Document Version: 1.0* +*Repository: $OPEN_IOV_ROOT* +*Branch: refactor/host-gsp-emulation* +*Base Commit: f9ed5d8*
\ No newline at end of file diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..fff4f82 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,191 @@ +# Open-IOV Documentation + +> **One independent IP block for NVIDIA GPU virtualization.** +> Unmodified host drivers. Unmodified guest drivers. No firmware patches. +> No hypervisor forks. No licenses to buy. + +Welcome. This is the front door to the Open-IOV project. If you read only one +page, read this one — it tells you what Open-IOV is, why it exists, where it is +today, and exactly which document to read next for whatever you care about. + +--- + +## 1. What is Open-IOV? + +Open-IOV is an open source virtualization layer for NVIDIA GPUs. It runs +**stock, unmodified NVIDIA drivers inside virtual machines** — Linux and +Windows — with GPU compute (CUDA), and eventually 3D graphics and video, on +hardware you already own, without a proprietary vGPU license, without patching +the guest driver, without patching the host driver, and without patching the +GPU firmware. + +It does this by implementing, in a host kernel module, the *interface* the +guest driver expects to talk to — the GPU System Processor (GSP) and its +resource manager firmware, GSP-RM — and by arranging for the *work* the guest +submits to run on real GPU hardware. + +Think of Open-IOV as a **virtualization IP block**: a self-contained component +that can be attached to many ecosystems (KVM, Xen, Argo, OpenXT, Qubes OS +Air, ...), many guest operating systems (Linux, Windows, ...), many hardware +generations (Turing, Ampere, Hopper, Grace-Hopper, Grace-Blackwell, GB10, ...) +and many driver versions — without any of those ecosystems, operating systems, +drivers or devices having to change. + +``` + ┌─────────────────────────────────────────────┐ + │ GUEST VM │ + │ stock nvidia.ko + CUDA / OpenGL / Vulkan │ + └──────────────┬──────────────────────────────┘ + │ PCIe (emulated device) + ┌───────────────────────────────┴───────────────────────────────┐ + │ HYPERVISOR (KVM/Xen) │ + │ VFIO / mdev transport, IOMMU isolation │ + └───────────────────────────────┬───────────────────────────────┘ + │ + ┌───────────────────────────────┴───────────────────────────────┐ + │ HOST: open-iov-vfio.ko (this project) │ + │ • virtual GPU device (BARs, config space, MSI-X) │ + │ • GSP / GSP-RM interface emulation (RPC, msgq, controls) │ + │ • memory model + GPU work forwarding to the physical GPU │ + └───────────────────────────────┬───────────────────────────────┘ + │ + ┌──────────────┴──────────────┐ + │ UNMODIFIED host nvidia.ko │ + │ (or GPU passthrough path) │ + └──────────────┬──────────────┘ + │ + ┌───────┴────────┐ + │ NVIDIA GPU │ + └────────────────┘ +``` + +## 2. Why does this need to exist? + +Three real needs converge: + +1. **Compute in guests, on hardware people have.** Multi-tenant AI systems are + moving toward *bare-metal* vLLM continuous batching inside NVIDIA + SuperPOD-style clusters — and that is the right answer for centralized + datacenter scale. But there is a large class of systems that still need + *compute runtimes inside guests*: Windows and Linux desktops and servers that + need CUDA, 3D, and video; edge devices that must run inference locally; and + fleets where several VMs must share the GPU without discrete device + assignment. Today, the only way to do that is a proprietary, licensed, + driver-version-locked vGPU stack. +2. **Secure AI agents at the edge.** An agent runtime (Claude Code, OpenCode, + ...) should never share a trust boundary with the model server, with the + hypervisor's other tenants, or with your data. GPUs are the one resource + that makes this hard: they are big, stateful and not designed for + partitioning. Virtualizing them *without* a license and *without* closed + host components is what makes "local inference, full isolation, no cloud + dependency" practical. +3. **Long-term durability.** Any GPU virtualization stack must survive new + driver versions, new firmware, new hardware and new hypervisors. The only + sustainable way to do that is to own the *interface*, not a snapshot of it — + and to make the project a durable, independent IP block with a permissive + license (BSD-3-Clause) so anyone can carry it forward. + +See [`01-orientation/01-purpose-and-scope.md`](01-orientation/01-purpose-and-scope.md) +and [`01-orientation/02-the-ai-edge-and-ecosystem.md`](01-orientation/02-the-ai-edge-and-ecosystem.md). + +## 3. Where is the project today? + +**Honest status (2026-09-23):** + +| Area | State | +|------|-------| +| Emulated vGPU device (PCI, BARs, MSI-X, VFIO mdev) | ✅ Working | +| GSP boot emulation (FWSEC/booter/RISCV handshake) | ✅ Working | +| GSP-RM message queue + RPC transport | ✅ Working | +| GSP-RM controls: device/fb/bus tables, CE caps, vaspace, GMMU static info | ✅ Working | +| Guest `nvidia.ko` probe, `nvidia-smi` | ✅ Working (TU104) | +| Guest CUDA driver (`libcuda`) startup | 🟡 Partially — UVM advances through channel creation and page-tree init; see [`04-implementation/01-known-knowns-and-unknowns.md`](04-implementation/01-known-knowns-and-unknowns.md) | +| CUDA kernel execution | 🔴 Not yet — requires GPU work forwarding | +| OpenGL / Vulkan | 🔴 Not yet | +| Windows guest | 🔴 Not yet | +| Multi-tenant isolation (VFs, mdev-per-VM) | 🟡 Designed, partially implemented | + +We also know exactly *why* CUDA kernels do not run yet, and what the path is: +see the known-knowns/known-unknowns document. The short version: the emulation +is now faithful enough that the guest stack reaches UVM initialization; the +missing piece is forwarding the guest's *work* (push buffers, page trees, +execution) to the physical GPU under our control. + +## 4. How to read these docs (progressive disclosure) + +Read in order if you are new. Jump directly if you know what you need. + +| # | Document | Audience | Time | +|---|----------|----------|------| +| 1 | [Purpose and scope](01-orientation/01-purpose-and-scope.md) | everyone | 10 min | +| 2 | [The AI edge and the ecosystem](01-orientation/02-the-ai-edge-and-ecosystem.md) | everyone | 15 min | +| 3 | [Install and first run](02-getting-started/01-install-and-first-run.md) | users, evaluators | 20 min | +| 4 | [System overview](03-architecture/01-system-overview.md) | everyone technical | 20 min | +| 5 | [Guest stack and CUDASW](03-architecture/02-guest-stack-and-cudasw.md) | developers | 25 min | +| 6 | [GSP and GSP-RM](03-architecture/03-gsp-and-gsp-rm.md) | developers | 30 min | +| 7 | [UVM](03-architecture/04-uvm.md) | developers | 25 min | +| 8 | [GMMU, AMAPlib, HALs, runlist, GR](03-architecture/05-gmmu-amap-hal-gr.md) | developers | 25 min | +| 9 | [Host module and hybrid model](03-architecture/06-host-module-and-hybrid-model.md) | developers | 25 min | +| 10 | [Driver lifecycle field guide](03-architecture/07-driver-lifecycle-field-guide.md) | developers, debuggers | 30 min | +| 11 | [Known-knowns and known-unknowns](04-implementation/01-known-knowns-and-unknowns.md) | everyone technical | 25 min | +| 12 | [Practical challenges and adversarial interoperability](04-implementation/02-practical-challenges-and-adversarial-interop.md) | developers, stakeholders | 20 min | +| 13 | [Performance, correctness, security](05-operations/01-performance-correctness-security.md) | developers, auditors | 25 min | +| 14 | [Debugging and maintenance workflows](05-operations/02-debugging-and-maintenance-workflows.md) | developers, maintainers | 25 min | +| 15 | [Hardware and software matrix](06-portability/hardware-software-matrix.md) | developers, integrators | 20 min | +| 16 | [Developer onboarding and standards](07-developers/onboarding-and-standards.md) | contributors | 20 min | +| 17 | [HAAEEP and collaboration](08-ecosystem/haaeep-and-collaboration.md) | stakeholders, partners | 20 min | +| 18 | [Glossary and FAQ](09-reference/glossary-and-faq.md) | everyone | reference | + +Existing deep-dive material (kept, still valid, being folded into this tree): +the `wiki/` directory (firmware analysis, address reference, wire protocol, +RPC message format) and the `docs/` reference documents +(`GSP_EMULATION_ARCHITECTURE.md`, `DEBUGGING.md`, `PERFORMANCE.md`, +`BUILD_NVIDIA_OPENRM.md`, `VERSION_UPGRADE.md`). + +## 5. What makes Open-IOV different? + +| | Proprietary vGPU | Discrete passthrough (VFIO) | Emulated GPU (QEMU) | **Open-IOV** | +|---|---|---|---|---| +| Guest driver | vendor vGPU driver | stock driver | stock driver (limited) | **stock driver** | +| Guest CUDA | yes | yes | no/emulated | **yes (goal: real execution)** | +| Multiple VMs per GPU | yes (licensed, capped) | no (1 VM = 1 GPU) | yes (no accel) | **yes** | +| License | paid, per-VM | free | free | **BSD-3-Clause** | +| Host driver changes | vendor stack | none | none | **none** | +| Guest driver changes | vendor build | none | none | **none** | +| Survives driver updates | vendor-controlled | yes | n/a | **designed for it** | +| Windows + Linux guests | yes | yes | limited | **yes (goal)** | + +The distinguishing property is **adversarial interoperability**: we reimplement +the *behavior* of the NVIDIA interface — firmware, RPC protocol, memory model, +scheduling — from observation and public documentation, so that no component in +the stack has to be modified or licensed. + +## 6. How to get involved + +* **Use it.** Try the reference setup, file issues with logs (our debugging + docs tell you exactly what to capture). +* **Test it.** Hardware and driver matrix contributions are the single most + valuable thing you can send — especially non-Turing hardware, Windows + guests, and non-KVM hypervisors. +* **Extend it.** The architecture docs mark the interfaces that are expected to + grow (memory model, forwarding backends, config-driven portability). +* **Integrate it.** If you build a hypervisor, an edge platform or a security + stack, see [`08-ecosystem/haaeep-and-collaboration.md`](08-ecosystem/haaeep-and-collaboration.md). +* **Fund or co-develop it.** Commercial use is explicitly welcome under + BSD-3-Clause; see the same document for collaboration models. + +## 7. License + +Open-IOV is **BSD 3-Clause** ([`../LICENSE`](../LICENSE)) for everything +reusable — userspace libraries, tools, scripts, configuration and docs — +and **GPL-2.0** for kernel modules (`kernel/**`) that must legally link +against GPL-only kernel exports (VFIO, mdev, IOMMU). Per-file SPDX +identifiers govern. How we study vendor interfaces (reverse engineering and +decompilation) and what we publish is described in +[`09-reference/legal-and-licensing.md`](09-reference/legal-and-licensing.md). + +--- + +*This documentation tree is the authoritative entry point for the project. If +you find a document that contradicts it, the contradiction is a bug: please +report it.* diff --git a/docs/VERSION_UPGRADE.md b/docs/VERSION_UPGRADE.md new file mode 100644 index 0000000..94c7d68 --- /dev/null +++ b/docs/VERSION_UPGRADE.md @@ -0,0 +1,137 @@ +# Open-IOV Version Upgrade Guide + +> How to support a new NVIDIA driver version (e.g. moving from 570.144 to a +> 610-series release) end-to-end: firmware acquisition → reverse engineering → +> config → host driver → guest validation. The goal is that a version bump is +> **configuration + analysis**, not a rewrite. + +--- + +## 0. Version discipline (read first) + +Four components must stay in lock-step; a mismatch between any two produces +failures that are indistinguishable from emulation bugs: + +``` +guest nvidia driver == host OpenRM nvidia.ko == host GSP firmware == emulation target (decompiled GSP) +``` + +Today that anchor is **570.144** everywhere. When you bump one, you bump all +four. Verify with `docs/DEBUGGING.md` §7 before any deep debugging. + +--- + +## 1. Acquire the new firmware + matching OpenRM tag + +1. Download the Linux x86_64 runfile for the target version: + `https://us.download.nvidia.com/XFree86/Linux-x86_64/<VER>/NVIDIA-Linux-x86_64-<VER>.run` + (the `tesla/` path 404s for consumer releases). +2. Extract the GSP firmware (`gsp_tu10x.bin` / arch-specific) — or fetch the + versioned blob from `linux-firmware.git`: + `nvidia/<VER>/gsp_tu10x.bin`. +3. Extract the `.fwimage` section (host boot image) and record the MD5: + ```bash + objcopy --only-section=.fwimage gsp_tu10x.bin gsp-<VER>.fwimage.bin + md5sum gsp-<VER>.fwimage.bin + ``` +4. Note the matching OpenRM tag at + `github.com/NVIDIA/open-gpu-kernel-modules` (e.g. `570.144`). + +## 2. Lay down versioned directories (never overwrite a working version) + +``` +firmware/<SERIES>-GSP/<VER>/ # e.g. firmware/61X-GSP/610.XX/ + gsp-<VER>.fwimage.bin + decompiled/ # ghidra output lands here +config/versions/<MAJOR>.yaml # e.g. config/versions/610.yaml +``` + +Copy the analysis scripts from the previous version dir +(`firmware/57X-GSP/570.144/*.py`) into the new one — they are script-relative. + +## 3. Reverse-engineer the new firmware (Ghidra headless) + +```bash +cd tools/ghidra +python3 analyze_fwimage.py \ + ../../firmware/<SERIES>-GSP/<VER>/gsp-<VER>.fwimage.bin \ + /tmp/ghidra_project_<VER> "gsp_<VER>" +cd ../../firmware/<SERIES>-GSP/<VER> +python3 analyze_firmware.py gsp-<VER>.fwimage.bin +``` + +What to extract (compare against `wiki/Address-Reference.md` for 570.144W): + +| Artifact | Why it matters | +|----------|----------------| +| Primary RPC dispatcher + jump table | RPC types/subcommands may shift | +| BAR trap / WPR handlers | BAR emulation behavior | +| Entry/boot sequence | GSP boot RPC ordering the guest driver expects | +| Migration tables | migration protocol (if enabled) | +| RPC message layouts | struct changes break the emulator silently | +| Version string/signature | runtime firmware matching | + +Sanity checks: still RV64IMAC? still the same 12 top-level RPC types? new +subcommands? Confidential-computing/TEE additions? + +## 4. Produce the version config + +Update `config/versions/<MAJOR>.yaml` with: +- `version.name/major/minor` and `firmware.gsp_version` +- discovered dispatcher/handler addresses (if tracked in yaml) +- `features` toggles per what the firmware actually has +- `symbols` (the nvidia.ko vGPU VFIO interface — verify against the new + OpenRM tag; renamed exports break the nvidia build) + +## 5. Update the runtime signature database + +`kernel/firmware_analyzer.c` matches firmware by CRC32/SHA256 + patterns. +Add the new entry to `known_signatures[]` (version string, hashes, discovered +addresses). This is what lets one `open-iov-vfio.ko` serve multiple firmware +versions safely. + +## 6. Rebuild host + guest against the new anchor + +1. Point `patches/nvidia/` at the new OpenRM tag (rebase our delta; verify + `modinfo nvidia.ko | grep depends` still shows `open-iov-vfio`). +2. `./scripts/stack_manager.sh build-all` (kernel → nvidia → userspace). +3. Install the new host firmware to `/lib/firmware/` + (`stack_manager.sh install-firmware`). +4. **Guest**: install the *matching* stock driver in the VM (runfile for the + exact version), replacing the old one. Keep the guest able to boot without + nvidia auto-loading during the swap (temporarily move dkms-built modules + aside — see `docs/DEBUGGING.md`). + +## 7. Validate (same checklist every version) + +Run the full bring-up checklist in `docs/DEBUGGING.md` §5: +module load → mdev → VM boot → guest probe → `nvidia-smi` → teardown +regression → unit test. Then the workload tests (CUDA vectorAdd, graphics). +Only after the checklist is green do performance comparisons +(`docs/PERFORMANCE.md`). + +## 8. Rollback plan + +Keep the previous version's directory + config untouched. Rollback = +`git checkout` the repo tag/commit for the old anchor + reinstall the old +guest driver. Because versions live in parallel directories, the working +setup is never destroyed by a forward attempt. + +## 9. Multi-version testing matrix (target state) + +For each supported anchor version × each supported GPU family +(`config/hardware/*.yaml`), the CI matrix should run the §7 checklist. When +adding a version, extend the matrix rather than replacing the old column — +regressions across versions are the cheapest bugs to catch. + +## 10. Common pitfalls (learned the hard way) + +- **Runfile version drift**: Ubuntu's `nvidia-driver-XXX-open` metapackages + can resolve to a *newer* major (570-open pulled 580). Always check the + candidate version before installing; prefer the exact-version runfile. +- **GSP firmware filename**: guest driver requests `nvidia/<VER>/gsp_tu10x.bin`; + a wrong-version firmware file = probe failure that looks like an emulation bug. +- **Kernel text addresses differ per build**: re-derive Ghidra addresses per + firmware; never carry 570.144 addresses into a 610 config. +- **Symbol renames across OpenRM tags**: `nvidia_vgpu_vfio_*` renames break + the nvidia build with `Unknown symbol` — check `kernel-open` diffs. diff --git a/docs/old/ARCHITECTURE.md b/docs/old/ARCHITECTURE.md new file mode 100644 index 0000000..f9b7ae4 --- /dev/null +++ b/docs/old/ARCHITECTURE.md @@ -0,0 +1,476 @@ +# Open-IOV vGPU-IO Architecture Document + +## Overview + +Open-IOV vGPU-IO is a complete open-source implementation of NVIDIA vGPU (virtual GPU) technology for Turing architecture GPUs (TU102/TU104), enabling GPU virtualization without proprietary GRID host drivers. The project provides: + +- **Host-side VFIO mdev driver** (`open-iov-vfio.ko`) - Creates mediated devices for QEMU/KVM +- **Guest-side vGPU driver** (`gsp-vgpu.ko`) - Communicates with GSP firmware via RPC +- **Userspace library** (`libiov.so`/`libvgpu.so`) - C API for VF management and GPU operations +- **Firmware decompilation** - Complete reverse engineering of GSP 570.144W firmware + +## System Architecture + +``` +╔══════════════════════════════════════════════════════════════════════════════════════════════════════════════╗ +║ HOST SYSTEM ║ +║ ╔════════════════════════════════════════════════════════════════════════════════════════════════════════════╗ ║ +║ ║ ║ ║ +║ ║ ┌──────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ +║ ║ │ USERSPACE (Host) │ ║ ║ +║ ║ │ │ ║ ║ +║ ║ │ ┌──────────────┐ ┌─────────────────────┐ ┌──────────────────────────────────────────────┐ │ ║ ║ +║ ║ │ │ QEMU/KVM │ │ libvfio-user │ │ mdev management CLI / mdev management daemon │ │ ║ ║ +║ ║ │ │ │ │ │ │ (GPU Virtualization Management CLI/Manager) │ │ ║ ║ +║ ║ │ │ -device vfio-│ │ - VFIO userspace │ │ │ │ ║ ║ +║ ║ │ │ pci,sysfsdev│ │ library │ │ • mdev create/destroy │ │ ║ ║ +║ ║ │ │ │ │ - ioctl/eventfd │ │ • VF lifecycle management │ │ ║ ║ +║ ║ │ │ ┌──────────┐ │ │ - mmap support │ │ • Migration orchestration │ │ ║ ║ +║ ║ │ │ │ VFIO │ │ │ │ │ • Resource allocation │ │ ║ ║ +║ ║ │ │ │ Region │ │ └──────────┬──────────┘ └──────────────────────────────────────────────┘ │ ║ ║ +║ ║ │ │ │ (mdev) │ │ │ │ ║ ║ +║ ║ │ │ └────┬─────┘ │ │ │ ║ ║ +║ ║ │ │ │ │ ▼ │ ║ ║ +║ ║ │ │ │ │ ┌──────────────────────────────────────────────────────────────────────────┐ │ ║ ║ +║ ║ │ │ │ │ │ libvgpu.so / libiov.so (Host-side) │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ ║ ║ +║ ║ │ │ │ │ │ • vgpu_open/close/get_info │ │ ║ ║ +║ ║ │ │ │ │ │ • vgpu_vf_create/destroy/get_info │ │ ║ ║ +║ ║ │ │ │ │ │ • vgpu_mem_alloc/free/map/unmap/pin/unpin │ │ ║ ║ +║ ║ │ │ │ │ │ • vgpu_bar_config/force_access/save/restore │ │ ║ ║ +║ ║ │ │ │ │ │ • vgpu_wpr_protect_write/read/unprotect/set_timeout/register_callback │ │ ║ ║ +║ ║ │ │ │ │ │ • vgpu_schedule/yield │ │ ║ ║ +║ ║ │ │ │ │ │ • vgpu_register/unregister_event_callback, vgpu_process_events │ │ ║ ║ +║ ║ │ │ │ │ │ • vgpu_migrate_start/status/cancel │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ ║ ║ +║ ║ │ │ │ │ │ ┌──────────────────────────────────────────────────────────────────┐ │ │ ║ ║ +║ ║ │ │ │ │ │ │ rpc_emulator.c (Host-side Emulation) │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ • rpc_emu_bar_config/save/restore/get_state │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ • rpc_emu_wpr_protect/unprotect/set_timeout/check_access │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ • rpc_emu_schedule/yield │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ • rpc_emu_notify_register/unregister/send │ │ │ ║ ║ +║ ║ │ │ │ │ │ └──────────────────────────────────────────────────────────────────┘ │ │ ║ ║ +║ ║ │ │ │ │ └──────────────────────────────────────────────────────────────────────────┘ │ ║ ║ +║ ║ │ └──────┼────────┘ │ ║ ║ +║ ║ │ │ VFIO_DEVICE_GET_REGION_INFO / VFIO_DEVICE_GET_IRQ_INFO / VFIO_DEVICE_RESET │ ║ ║ +║ ║ │ │ mmap() / ioctl() / read() / write() / poll() │ ║ ║ +║ ║ │ ▼ │ ║ ║ +║ ║ └─────────┼──────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ +║ ║ │ ║ ║ +║ ║ ▼ ║ ║ +║ ║ ┌──────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ +║ ║ │ KERNEL SPACE (Host) │ ║ ║ +║ ║ │ │ ║ ║ +║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ +║ ║ │ │ nvidia.ko (Host RM Kernel Module) │ │ ║ ║ +║ ║ │ │ (NVIDIA OpenRM 570.144 - Host Driver) │ │ ║ ║ +║ ║ │ │ │ │ ║ ║ +║ ║ │ │ • GSP Firmware Boot & Management │ │ ║ ║ +║ ║ │ │ • Physical GPU Resource Management (VRAM, engines, etc.) │ │ ║ ║ +║ ║ │ │ • vGPU VFIO Symbol Exports: │ │ ║ ║ +║ ║ │ │ - nvidia_vgpu_vfio_get_ops() │ │ ║ ║ +║ ║ │ │ - nvidia_vgpu_vfio_set_ops() │ │ ║ ║ +║ ║ │ │ - nv_vgpu_mdev_register_parent() │ │ ║ ║ +║ ║ │ │ - nv_vgpu_mdev_unregister_parent() │ │ ║ ║ +║ ║ │ │ • GPC Callbacks (from open-iov-vfio.ko): │ │ ║ ║ +║ ║ │ │ - vgpu_create / vgpu_delete / vgpu_bar_info / vgpu_start │ │ ║ ║ +║ ║ │ │ - get_name / get_description / get_instances / get_sparse_mmap │ │ ║ ║ +║ ║ │ │ - update_request │ │ ║ ║ +║ ║ │ │ • Host IRQ/MSI-X Handler (owns physical GPU interrupt) │ │ ║ ║ +║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ +║ ║ │ │ ║ ║ +║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ +║ ║ │ │ open-iov-vfio.ko (Host VFIO mdev Driver) │ │ ║ ║ +║ ║ │ │ │ │ ║ ║ +║ ║ │ │ • mdev_parent registration with nvidia.ko │ │ ║ ║ +║ ║ │ │ • VFIO Device Ops: probe/remove/open/close/ioctl/mmap │ │ ║ ║ +║ ║ │ │ • PCI Config Space Emulation (256 bytes): │ │ ║ ║ +║ ║ │ │ - Vendor=0x10DE, Device=0x1ED3 (TU104 vGPU) │ │ ║ ║ +║ ║ │ │ - BAR0=16MB, BAR1=256MB, BAR2=32MB, BAR3=32MB (all 64-bit) │ │ ║ ║ +║ ║ │ │ - Capabilities: PM, MSI, MSI-X (Table@BAR0+0x2000), PCIe │ │ ║ ║ +║ ║ │ │ • BAR Trap Emulation (bar_emulation.c): │ │ ║ ║ +║ ║ │ │ - 64 trapped regions covering all BAR0 registers │ │ ║ ║ +║ ║ │ │ - Shadow memory for register state │ │ ║ ║ +║ ║ │ │ • GSP Communication (gsp_comm.c): │ │ ║ ║ +║ ║ │ │ - Polling-based doorbell (queue head @ 0x110c00) │ │ ║ ║ +║ ║ │ │ - Shared memory region for RPC messages │ │ ║ ║ +║ ║ │ │ • Firmware-Decompiled RPC Client (gsp_rpc_client.c + gsp_rpc_integration.c): │ │ ║ ║ +║ ║ │ │ - All 12 RPC types: ALLOC, FREE, MAP, UNMAP, PIN, UNPIN, │ │ ║ ║ +║ ║ │ │ SCHEDULE, YIELD, NOTIFY, BAR_TRAP, MIGRATE, DEBUG │ │ ║ ║ +║ ║ │ │ - Real firmware addresses from decompilation │ │ ║ ║ +║ ║ │ │ • Memory Management (memory.c): │ │ ║ ║ +║ ║ │ │ - DMA mapping, pin/unpin, map/unmap │ │ ║ ║ +║ ║ │ │ • IRQ Forwarding (irq.c): │ │ ║ ║ +║ ║ │ │ - MSI-X table/PBA emulation │ │ ║ ║ +║ ║ │ │ - eventfd trigger injection to guest │ │ ║ ║ +║ ║ │ │ • Live Migration (migration.c): │ │ ║ ║ +║ ║ │ │ - 4-phase: QUERY→PREPARE→TRANSFER→COMMIT→CLEANUP │ │ ║ ║ +║ ║ │ │ - Dual jump tables (0x04179ED8, 0x04179FBC) │ │ ║ ║ +║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ +║ ║ │ │ ║ ║ +║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ +║ ║ │ │ vfio-mdev.ko / vfio-pci.ko (Kernel VFIO Core) │ │ ║ ║ +║ ║ │ │ • Mediated device framework │ │ ║ ║ +║ ║ │ │ • VFIO PCI device emulation │ │ ║ ║ +║ ║ │ │ • IOMMU DMA mapping (DMA remapping) │ │ ║ ║ +║ ║ │ │ • Region/IRQ index mapping: │ │ ║ ║ +║ ║ │ │ - VFIO_PCI_BAR0_REGION_INDEX → BAR0 (16MB) │ │ ║ ║ +║ ║ │ │ - VFIO_PCI_BAR1_REGION_INDEX → BAR1 (256MB) │ │ ║ ║ +║ ║ │ │ - VFIO_PCI_BAR2_REGION_INDEX → BAR2 (32MB) │ │ ║ ║ +║ ║ │ │ - VFIO_PCI_BAR3_REGION_INDEX → BAR3 (32MB) │ │ ║ ║ +║ ║ │ │ - VFIO_PCI_CONFIG_REGION_INDEX → PCI Config Space (256B) │ │ ║ ║ +║ ║ │ │ - VFIO_PCI_MSIX_REGION_INDEX → MSI-X Table/PBA │ │ ║ ║ +║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ +║ ║ │ │ ║ ║ +║ ║ └──────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ +║ ║ ║ ║ +║ ║ ════════════════════════════════════════════════════════════════════════════════════════════════════════════ ║ ║ +║ ║ HARDWARE (Physical GPU) ║ ║ +║ ║ ┌──────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ +║ ║ │ NVIDIA TU102/TU104 GPU (RTX 2080 Ti / T4 / etc.) │ ║ ║ +║ ║ │ │ ║ ║ +║ ║ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────────┐ │ ║ ║ +║ ║ │ │ BAR0 │ │ BAR1 │ │ BAR2 │ │ BAR3 │ │ GSP FALCON │ │ ║ ║ +║ ║ │ │ (16 MB) │ │ (256 MB) │ │ (32 MB) │ │ (32 MB) │ │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ ┌─────────────────┐ │ │ ║ ║ +║ ║ │ │ Config Regs │ │ Framebuffer │ │ Reserved │ │ Reserved │ │ │ REAL GSP │ │ │ ║ ║ +║ ║ │ │ Doorbell │ │ (VRAM) │ │ │ │ │ │ │ FIRMWARE │ │ │ ║ ║ +║ ║ │ │ Queue Heads │ │ │ │ │ │ │ │ │ (570.144W) │ │ │ ║ ║ +║ ║ │ │ Shared Mem │ │ │ │ │ │ │ │ │ │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ RPC Dispatchers │ │ │ ║ ║ +║ ║ │ │ 0x110c00 │ │ │ │ │ │ │ │ │ • Primary │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ 0x057A1AC0 │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ • Secondary │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ 0x04ED39DC │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ Specialized │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ Handlers: │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ • BAR_TRAP │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ 0x04CB8188 │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ • WPR x2 │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ 0x04F00B0C │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ 0x04F00C78 │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ • ALLOC │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ 0x057E94EC │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ • SCHEDULE │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ 0x051EFBF4 │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ • Migration │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ 0x04179ED8 │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ 0x04179FBC │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ 12 RPC Types: │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ ALLOC, FREE, │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ MAP, UNMAP, │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ PIN, UNPIN, │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ SCHEDULE, │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ YIELD, NOTIFY, │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ BAR_TRAP, │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ │ MIGRATE, DEBUG │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ │ │ │ │ └─────────────────┘ │ │ ║ ║ +║ ║ │ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────────────────┘ │ ║ ║ +║ ║ │ │ ║ ║ +║ ║ │ Physical IRQ (MSI-X) ──────────────────────────────────────────────► nvidia.ko ISR │ ║ ║ +║ ║ └──────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ +║ ║ ║ ║ +║ ╚════════════════════════════════════════════════════════════════════════════════════════════════════════════╝ ║ +║ ║ +╚════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝ + + │ VFIO MEDIATED DEVICE (mdev) │ + │ sysfs: /sys/bus/mdev/devices/<UUID>/ │ + │ • PCI Config Space (256B) │ + │ • BAR0 Region (16MB) │ + │ • BAR1 Region (256MB) │ + │ • BAR2 Region (32MB) │ + │ • BAR3 Region (32MB) │ + │ • MSI-X Table/PBA Region │ + │ • IRQs (MSI-X vectors) │ + ▼ + +╔═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗ +║ GUEST VM ║ +║ ╔══════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗ ║ +║ ║ ║ ║ +║ ║ ┌──────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ +║ ║ │ USERSPACE (Guest) │ ║ ║ +║ ║ │ │ ║ ║ +║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ +║ ║ │ │ Guest Applications │ │ ║ ║ +║ ║ │ │ │ │ ║ ║ +║ ║ │ │ • CUDA Applications │ │ ║ ║ +║ ║ │ │ • OpenGL/Vulkan Applications │ │ ║ ║ +║ ║ │ │ • Video Encode/Decode (NVENC/NVDEC) │ │ ║ ║ +║ ║ │ │ • AI/ML Workloads (cuDNN, TensorRT) │ │ ║ ║ +║ ║ │ │ • Custom vGPU Management (via libvgpu.so) │ │ ║ ║ +║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ +║ ║ │ │ ║ ║ +║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ +║ ║ │ │ libvgpu.so / libiov.so (Guest-side Userspace Library) │ │ ║ ║ +║ ║ │ │ │ │ ║ ║ +║ ║ │ │ • vgpu_open/close/get_info │ │ ║ ║ +║ ║ │ │ • vgpu_vf_create/destroy/get_info │ │ ║ ║ +║ ║ │ │ • vgpu_mem_alloc/free/map/unmap/pin/unpin │ │ ║ ║ +║ ║ │ │ • vgpu_bar_config/force_access/save/restore │ │ ║ ║ +║ ║ │ │ • vgpu_wpr_protect_write/read/unprotect/set_timeout/register_callback │ │ ║ ║ +║ ║ │ │ • vgpu_schedule/yield │ │ ║ ║ +║ ║ │ │ • vgpu_register/unregister_event_callback, vgpu_process_events │ │ ║ ║ +║ ║ │ │ • vgpu_migrate_start/status/cancel │ │ ║ ║ +║ ║ │ │ │ │ ║ ║ +║ ║ │ │ ┌────────────────────────────────────────────────────────────────────────────────────────────┐ │ │ ║ ║ +║ ║ │ │ │ rpc_emulator.c (Guest-side Emulation Fallbacks) │ │ │ ║ ║ +║ ║ │ │ │ │ │ │ ║ ║ +║ ║ │ │ │ • rpc_emu_bar_config/save/restore/get_state │ │ │ ║ ║ +║ ║ │ │ │ • rpc_emu_wpr_protect/unprotect/set_timeout/check_access │ │ │ ║ ║ +║ ║ │ │ │ • rpc_emu_schedule/yield │ │ │ ║ ║ +║ ║ │ │ │ • rpc_emu_notify_register/unregister/send │ │ │ ║ ║ +║ ║ │ │ └────────────────────────────────────────────────────────────────────────────────────────────┘ │ │ ║ ║ +║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ +║ ║ │ │ ║ ║ +║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ +║ ║ │ │ libnvidia-ml.so / libnvidia-opencl.so / libcuda.so (NVIDIA Userspace Drivers) │ │ ║ ║ +║ ║ │ │ • Standard NVIDIA userspace driver stack (unchanged) │ │ ║ ║ +║ ║ │ │ • Communicates with guest nvidia.ko via ioctl on /dev/nvidia* │ │ ║ ║ +║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ +║ ║ │ │ ║ ║ +║ ║ └──────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ +║ ║ ║ ║ +║ ║ ═════════════════════════════════════════════════════════════════════════════════════════════════════════════ ║ ║ +║ ║ KERNEL SPACE (Guest) ║ ║ +║ ║ ┌──────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ +║ ║ │ │ ║ ║ +║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ +║ ║ │ │ nvidia.ko (Guest RM Kernel Module) │ │ ║ ║ +║ ║ │ │ (NVIDIA OpenRM 570.144 - Guest Driver, UNMODIFIED) │ │ ║ ║ +║ ║ │ │ │ │ ║ ║ +║ ║ │ │ • Detects vGPU via PCI Config Space (Vendor=0x10DE, Device=0x1ED3) │ │ ║ ║ +║ ║ │ │ • Maps BAR0 (config), BAR1 (framebuffer), BAR2, BAR3 │ │ ║ ║ +║ ║ │ │ • Sets up MSI-X interrupts (table @ BAR0+0x2000, PBA @ BAR0+0x3000) │ │ ║ ║ +║ ║ │ │ • GSP Client: │ │ ║ ║ +║ ║ │ │ - Writes to BAR0 doorbell (queue head @ 0x110c00) │ │ ║ ║ +║ ║ │ │ - Reads/writes shared memory for RPC messages │ │ ║ ║ +║ ║ │ │ - Expects GSP firmware with vGPU RPC handlers │ │ ║ ║ +║ ║ │ │ • RM (Resource Manager) for GPU resources │ │ ║ ║ +║ ║ │ │ • nvidia-modeset.ko / nvidia-uvm.ko / nvidia-peermem.ko │ │ ║ ║ +║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ +║ ║ │ │ ║ ║ +║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ +║ ║ │ │ gsp-vgpu.ko (Guest vGPU Driver - Open-IOV) │ │ ║ ║ +║ ║ │ │ │ │ ║ ║ +║ ║ │ │ • Char Device: /dev/gsp-vgpu (misc device) │ │ ║ ║ +║ ║ │ │ • IOCTL Interface (matches libvgpu.so API): │ │ ║ ║ +║ ║ │ │ - GSP_IOCTL_CREATE_VF / DESTROY_VF │ │ ║ ║ +║ ║ │ │ - GSP_IOCTL_ALLOC / FREE / MAP / UNMAP / PIN / UNPIN │ │ ║ ║ +║ ║ │ │ - GSP_IOCTL_BAR_CONFIG / FORCE / SAVE / RESTORE │ │ ║ ║ +║ ║ │ │ - GSP_IOCTL_WPR_PROTECT / UNPROTECT / TIMEOUT │ │ ║ ║ +║ ║ │ │ - GSP_IOCTL_SCHEDULE / YIELD │ │ ║ ║ +║ ║ │ │ - GSP_IOCTL_MIGRATE │ │ ║ ║ +║ ║ │ │ - GSP_IOCTL_GET_INFO │ │ ║ ║ +║ ║ │ │ • RPC Client (rpc/gsp_rpc.c): │ │ ║ ║ +║ ║ │ │ - Doorbell writes to queue head │ │ ║ ║ +║ ║ │ │ - Ring buffer management │ │ ║ ║ +║ ║ │ │ • Emulation Mode (AUTO-DETECTED): │ │ ║ ║ +║ ║ │ │ - Test BAR_TRAP RPC at load; if timeout → enable emulation │ │ ║ ║ +║ ║ │ │ - Module params: emulation_mode (ro), emulation_override (force) │ │ ║ ║ +║ ║ │ │ - In emulation: BAR_TRAP, WPR, ALLOC, PIN, MAP handled locally │ │ ║ ║ +║ ║ │ │ • Debugfs: /sys/kernel/debug/gsp-vgpu/ │ │ ║ ║ +║ ║ │ │ • VF Lifecycle (vf/gsp_vf.c) │ │ ║ ║ +║ ║ │ │ • BAR Emulation (bar/gsp_bar.c) │ │ ║ ║ +║ ║ │ │ • WPR Management (wpr/gsp_wpr.c) │ │ ║ ║ +║ ║ │ │ • Migration (migration/gsp_migration.c) │ │ ║ ║ +║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ +║ ║ │ │ ║ ║ +║ ║ │ ┌────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ║ ║ +║ ║ │ │ VFIO-PCI Core (Guest Kernel) │ │ ║ ║ +║ ║ │ │ • vfio-pci.ko: PCI device emulation for VFIO │ │ ║ ║ +║ ║ │ │ • Handles mmap of BAR regions │ │ ║ ║ +║ ║ │ │ • MSI-X interrupt handling │ │ ║ ║ +║ ║ │ │ • Config space read/write │ │ ║ ║ +║ ║ │ └────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ║ ║ +║ ║ │ │ ║ ║ +║ ║ └──────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ +║ ║ ║ ║ +║ ║ ══════════════════════════════════════════════════════════════════════════════════════════════════════════════ ║ ║ +║ ║ VIRTUAL GPU (mdev Device as seen by Guest) ║ ║ +║ ║ ┌──────────────────────────────────────────────────────────────────────────────────────────────────────────┐ ║ ║ +║ ║ │ PCI Config Space (256B) │ BAR0 (16MB) │ BAR1 (256MB) │ BAR2 (32MB) │ BAR3 (32MB) │ MSI-X │ ║ ║ +║ ║ │ • Vendor=0x10DE │ Config Regs │ Framebuffer │ Reserved │ Reserved │ Table │ ║ ║ +║ ║ │ • Device=0x1ED3 │ Doorbell │ (VRAM) │ │ │ @0x2000│ ║ ║ +║ ║ │ • Class=0x030000 │ Queue Heads │ │ │ │ PBA │ ║ ║ +║ ║ │ • BARs: 64-bit, prefetch │ Shared Mem │ │ │ │ @0x3000│ ║ ║ +║ ║ │ • Caps: PM, MSI, MSI-X, │ @0x110c00 │ │ │ │ │ ║ ║ +║ ║ │ PCIe │ ║ ║ +║ ║ └──────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ║ ║ +║ ║ ║ ║ +║ ╚═══════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝ ║ +╚══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝ +``` + +## Communication Pathways (Host ↔ Guest) + +### 1. **VFIO Region Access (Memory-Mapped I/O)** +``` +Guest App → libvgpu.so → ioctl(/dev/gsp-vgpu) → gsp-vgpu.ko + → Guest vfio-pci.ko → mmap(BAR0/BAR1/BAR2/BAR3) + → Host vfio-pci.ko → open-iov-vfio.ko BAR emulation + → Host nvidia.ko → Physical GPU BARs +``` +**Path**: Guest userspace → Guest kernel → Host kernel → Physical GPU +**Mechanism**: VFIO mmap with IOMMU DMA remapping + +### 2. **PCI Config Space Access** +``` +Guest nvidia.ko → pci_read_config_* / pci_write_config_* + → Guest vfio-pci.ko → VFIO_DEVICE_GET_REGION_INFO (CONFIG_REGION) + → Host open-iov-vfio.ko → nv_vgpu_config_read/write() + → Returns emulated 256-byte config space +``` +**Path**: Guest kernel → Host kernel (via VFIO ioctl) +**Mechanism**: VFIO config space region emulation + +### 3. **GSP RPC Communication (Native Path)** +``` +Guest nvidia.ko → Write BAR0 doorbell (0x110c00) + shared memory + → Physical GPU GSP firmware receives RPC + → GSP processes RPC (ALLOC, MAP, PIN, SCHEDULE, BAR_TRAP, etc.) + → GSP writes response to shared memory + → Guest nvidia.ko reads response +``` +**Path**: Guest kernel → Physical GPU (direct, no host kernel involvement) +**Mechanism**: Doorbell register + shared memory in BAR0 + +### 4. **GSP RPC Communication (Emulation Path - Host Mediated)** +``` +Guest nvidia.ko → Write BAR0 doorbell (0x110c00) + → Host open-iov-vfio.ko (polling) detects doorbell + → gsp_comm.c → gsp_rpc_integration.c → gsp_rpc_client.c + → Firmware-decompiled RPC handlers (emulated in host kernel) + → Response written to shared memory + → Guest nvidia.ko reads response +``` +**Path**: Guest kernel → Host kernel (open-iov-vfio.ko) → Emulated RPC +**Mechanism**: Host polls GSP doorbell, processes RPC in software + +### 5. **BAR Trap Emulation** +``` +Guest nvidia.ko → Write to trapped BAR register (page fault) + → Guest vfio-pci.ko → VFIO page fault handler + → Host open-iov-vfio.ko → nv_vgpu_bar_trap() + → If native: gsp_rpc_bar_trap() → GSP firmware (0x04CB8188) + → If emulated: rpc_emu_bar_trap() in host userspace (libvgpu.so) + → Result written back to guest BAR mapping +``` +**Path**: Guest kernel page fault → Host kernel → Host userspace (emulation) or GSP firmware +**Mechanism**: VFIO page fault + BAR trap emulation + +### 6. **Interrupt Forwarding (Host → Guest)** +``` +Physical GPU → MSI-X IRQ → Host nvidia.ko ISR + → nvidia.ko calls nv_vgpu_vfio_inject_interrupt() (VFIO ops) + → open-iov-vfio.ko → eventfd_signal() + → QEMU/KVM → Guest MSI-X interrupt injected + → Guest nvidia.ko ISR handles interrupt +``` +**Path**: Physical GPU → Host kernel → Host userspace (QEMU) → Guest kernel +**Mechanism**: VFIO interrupt injection via eventfd + +### 7. **Userspace Management (mdev management CLI/mdev management daemon)** +``` +mdev management CLI → libvgpu.so (host) → ioctl(/dev/vfio/<mdev>) + → open-iov-vfio.ko → mdev VFIO ops + → VF creation, BAR config, memory ops, migration +``` +**Path**: Host userspace → Host kernel (VFIO mdev) +**Mechanism**: Standard VFIO ioctl on mdev device + +### 8. **Guest vGPU Driver Communication** +``` +Guest App → libvgpu.so (guest) → ioctl(/dev/gsp-vgpu) + → gsp-vgpu.ko → Internal RPC client + → If emulation: Local handling in gsp-vgpu.ko + → If native: GSP RPC via BAR0 doorbell +``` +**Path**: Guest userspace → Guest kernel (gsp-vgpu.ko) → GSP or emulation + +## Key Component Summary + +| Component | Location | Type | Purpose | +|-----------|----------|------|---------| +| **nvidia.ko (Host)** | Host Kernel | RM Driver | Boots GSP, manages physical GPU, exports vGPU VFIO symbols | +| **open-iov-vfio.ko** | Host Kernel | VFIO mdev Driver | Creates mdev, PCI config space, BAR emulation, RPC integration, IRQ forwarding | +| **gsp-vgpu.ko** | Guest Kernel | vGPU Driver | Char device `/dev/gsp-vgpu`, ioctl API, emulation mode, RPC client | +| **nvidia.ko (Guest)** | Guest Kernel | RM Driver | Standard NVIDIA driver, detects vGPU, communicates with GSP | +| **libvgpu.so / libiov.so (Host)** | Host Userspace | Library | Management API (mdev management CLI/mdev management daemon), emulation fallbacks | +| **libvgpu.so / libiov.so (Guest)** | Guest Userspace | Library | Application API, emulation fallbacks | +| **libnvidia-*.so** | Guest Userspace | NVIDIA Drivers | Standard CUDA/OpenGL/OpenCL userspace stack | +| **mdev management CLI / mdev management daemon** | Host Userspace | Tools | mdev lifecycle, VF management, migration orchestration | +| **REAL GSP Firmware** | Physical GPU | Firmware | 570.144W, 12 RPC types, BAR_TRAP/WPR/SCHEDULE/MIGRATE handlers | +| **EMULATED GSP** | Host Kernel + Userspace | Software | rpc_emulator.c, gsp_rpc_client.c (when GSP lacks vGPU handlers) | + +## Emulation Mode Details + +### When Enabled +- **Auto-detect**: At gsp-vgpu.ko load, sends test BAR_TRAP RPC +- **Timeout/Error** → `emulation_mode = true` +- **Module params**: `emulation_override=1` (force on), `emulation_override=0` (force off) + +### What Gets Emulated +| Operation | Host Kernel (open-iov-vfio.ko) | Host Userspace (libvgpu.so) | Guest Kernel (gsp-vgpu.ko) | +|-----------|-------------------------------|----------------------------|---------------------------| +| BAR_TRAP | nv_vgpu_bar_trap() → rpc_emu_bar_trap() | rpc_emu_bar_config/save/restore | gsp_bar_force_access() | +| WPR | nv_vgpu_wpr_*() | rpc_emu_wpr_protect/unprotect | gsp_vf_wpr_protect/unprotect | +| ALLOC/MAP/PIN | gsp_rpc_client.c (local) | rpc_emu_* fallbacks | gsp_vf_mem_alloc/map/pin | +| SCHEDULE/YIELD | gsp_rpc_schedule/yield (workqueue) | rpc_emu_schedule/yield | gsp_vf_schedule/yield | +| NOTIFY | gsp_rpc_notify_register/unregister | rpc_emu_notify_register/unregister/send | gsp_vf_notify_* | +| MIGRATION | migration.c (4-phase) | rpc_emu_migrate_* | gsp_migration_start | + +## Build Artifacts + +``` +$OPEN_IOV_ROOT/ +├── nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko (34MB) Host RM Driver +├── kernel/open-iov-vfio.ko (2.8MB) Host VFIO mdev Driver +├── driver/gsp-vgpu.ko (1.8MB) Guest vGPU Driver +├── userspace/lib/libiov.so (76KB) Userspace Library (libvgpu.so symlink) +├── userspace/examples/example_vgpu (28KB) Test Application +├── firmware/570-GSP/gsp-570.144.fwimage.bin (28MB) GSP Firmware +├── scripts/load_vgpu_stack.sh Host module loader +├── scripts/create_mdev.sh mdev creation +├── scripts/launch_qemu_vgpu.sh QEMU launcher +├── scripts/test_vgpu_stack.sh Test suite +├── scripts/setup_guest_driver.sh Guest driver setup +├── scripts/unload_vgpu_stack.sh Cleanup +├── docs/ARCHITECTURE.md This document +└── wiki/ 10 technical docs +``` + +## Deployment Sequence + +```bash +# HOST (after reboot to clear module_blacklist) +# 1. Load NVIDIA OpenRM host driver (exports vGPU VFIO symbols) +sudo insmod nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko + +# 2. Load Open-IOV VFIO mdev driver (registers mdev parent with nvidia.ko) +sudo insmod kernel/open-iov-vfio.ko + +# 3. Create mediated device for QEMU +./scripts/create_mdev.sh # Creates /sys/bus/mdev/devices/<UUID>/ + +# 4. Launch QEMU with vGPU +./scripts/launch_qemu_vgpu.sh # -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/<UUID> + +# GUEST (inside VM) +# 5. Install NVIDIA driver (unmodified OpenRM) +./scripts/setup_guest_driver.sh # Installs nvidia.ko, firmware, loads driver + +# 6. Verify +nvidia-smi # Should show vGPU (T4 / A100 / etc.) +lspci -d 10de: # Should show 10de:1ed3 (TU104 vGPU) +``` + +--- + +*Architecture Version: 2.0 (with detailed host/guest communication pathways)* +*Last Updated: $(date)* +*Project: $OPEN_IOV_ROOT*
\ No newline at end of file diff --git a/docs/refactored-architecture.md b/docs/refactored-architecture.md new file mode 100644 index 0000000..9a2f401 --- /dev/null +++ b/docs/refactored-architecture.md @@ -0,0 +1,296 @@ +# Open-IOV vGPU-IO Architecture + +> **Status**: Living document. Updated 2026-09-22 post host-stability fix. +> **Supersedes**: Previous refactored-architecture.md (v1.0). + +--- + +## 1. Problem Statement + +**Goal**: Enable unmodified NVIDIA GPU drivers (OpenRM 570.144+) to run in guest VMs with **full GPU functionality** (CUDA, graphics, video, AI/ML) on consumer and data-center GPUs without proprietary GRID licenses. + +**Correctness requirement**: Guest must see a fully functional NVIDIA GPU. `nvidia-smi`, CUDA, OpenGL/Vulkan, and compute must all work. + +**Performance requirement**: GPU compute must execute on the physical GPU hardware, not in software. Target: < 5% overhead vs native for CUDA kernel launches. + +**Constraints**: +- Guest runs **stock** `nvidia.ko` + standard NVIDIA userspace (no Open-IOV code in guest) +- Host runs **apt** `nvidia-driver-570` for physical GPU management (570.211.01) +- Host runs **open-iov-vfio.ko** for vGPU emulation (mediated device, mdev) +- No SR-IOV required (works on consumer GPUs like RTX 2080 Super) + +--- + +## 2. Core Architectural Principle: Emulate the Interface, Pass Through the Work + +A vGPU is a **mediated passthrough** device. Some functions must be emulated (the guest expects virtual hardware), others must be passed through to the physical GPU (for performance). + +``` +┌──────────────────────────────────────────────────────────────────────┐ +│ EMULATE (software — guest expects virtual hardware) │ +│ • PCI config space (guest sees virtual TU104 [10de:1ed3]) │ +│ • BAR0 register interface (shadow, not physical GPU BAR0) │ +│ • GSP boot sequence (FWSEC/Booter/RISCV auto-complete) │ +│ • GSP-RM protocol layer (command queue, reply format) │ +│ • Interrupt routing (eventfd → guest MSI-X) │ +│ • Display output (headless / virtual display) │ +│ • BAR3 instance memory window (shadow, not physical GPU) │ +└──────────────────────────────────────────────────────────────────────┘ + +┌──────────────────────────────────────────────────────────────────────┐ +│ PASS-THROUGH (hardware — for performance) │ +│ • GPU compute (CUDA kernels) → execute on physical GPU │ +│ • GPU memory (guest VRAM) → backed by real GPU VRAM or DMA-mapped │ +│ • DMA operations → use real GPU DMA engines │ +│ • Channel/pushbuffer work → physical GPU channels │ +│ • Graphics rendering → physical GPU render pipeline │ +│ • Video encode/decode → physical GPU NVENC/NVDEC engines │ +└──────────────────────────────────────────────────────────────────────┘ +``` + +**Why not full emulation?** Software emulation of CUDA kernels would be 100-1000x slower than native. The physical GPU must do the actual compute work. + +**Why not full passthrough?** The guest driver expects to manage the GSP, boot firmware, and own the GPU. We can't give it direct physical GPU access (the host driver already owns it). The GSP emulation layer translates guest commands into host GPU operations. + +--- + +## 3. Host Module Architecture (Corrected 2026-09-22) + +``` +╔══════════════════════════════════════════════════════════════════════╗ +║ HOST KERNEL SPACE ║ +║ ║ +║ ┌────────────────────────────────────────────────────────────────┐ ║ +║ │ nvidia.ko (apt 570.211.01) │ ║ +║ │ • Manages physical RTX 2080 Super (GSP firmware booted) │ ║ +║ │ • Provides /dev/nvidia0 for nvidia-smi on host │ ║ +║ │ • Owns physical GPU interrupts, BARs, VRAM │ ║ +║ │ • NOT modified, NOT rebuilt — stock Ubuntu package │ ║ +║ └────────────────────────────────────────────────────────────────┘ ║ +║ ║ +║ ┌────────────────────────────────────────────────────────────────┐ ║ +║ │ open-iov-vfio.ko (Open-IOV vGPU VFIO driver) │ ║ +║ │ • Registers mdev parent on the physical GPU's PCI device │ ║ +║ │ • Creates mdev types (nvidia-vgpu-turing, etc.) │ ║ +║ │ • Per-mdev: emulated PCI config, BAR0 shadow, BAR1 vidmem, │ ║ +║ │ BAR3 shadow, MSI-X emulation, GSP emulation thread │ ║ +║ │ • Exports VFIO device ops (read/write/ioctl/mmap) │ ║ +║ │ • Self-contained: depends only on vfio, mdev │ ║ +║ │ • NO dependency on nvidia submodule (removed 2026-09-22) │ ║ +║ └────────────────────────────────────────────────────────────────┘ ║ +║ ║ +║ ┌────────────────────────────────────────────────────────────────┐ ║ +║ │ vfio / mdev (kernel prerequisites) │ ║ +║ │ • vfio, vfio_iommu_type1, vfio_pci, mdev │ ║ +║ │ • IOMMU domain management, device isolation │ ║ +║ └────────────────────────────────────────────────────────────────┘ ║ +║ ║ +║ PHYSICAL GPU: RTX 2080 Super (TU104, 10de:1ed3) ║ +║ • Managed by apt nvidia driver (host nvidia-smi works) ║ +║ • open-iov-vfio registers mdev parent but does NOT drive the GPU ║ +║ • Guest vGPU is a software construct — physical GPU unaffected ║ +╚══════════════════════════════════════════════════════════════════════╝ + │ VFIO mdev + │ /sys/bus/mdev/devices/<UUID>/ + ▼ +╔══════════════════════════════════════════════════════════════════════╗ +║ GUEST VM (QEMU/KVM) ║ +║ ┌────────────────────────────────────────────────────────────────┐ ║ +║ │ nvidia.ko (stock 570.144) │ ║ +║ │ • PCI probe: sees TU104 [10de:1ed3] via emulated config │ ║ +║ │ • BAR0: shadow (vzalloc'd, mapped via VFIO) │ ║ +║ │ • BAR1: emulated vidmem (vzalloc'd host RAM) │ ║ +║ │ • BAR3: shadow (vzalloc'd) │ ║ +║ │ • MSI-X: eventfd → QEMU → guest ISR │ ║ +║ │ • GSP client: msgq RPCs via emulated GSP-RM channel │ ║ +║ │ • NO Open-IOV code in guest │ ║ +║ └────────────────────────────────────────────────────────────────┘ ║ +╚══════════════════════════════════════════════════════════════════════╝ +``` + +### Why no nvidia submodule on host? + +The **nvidia-open-gpu-kernel-modules submodule (610.57.04-based)** was originally intended for NVIDIA's *native* vGPU management (RM callbacks like `vgpu_create_callback`). For our QEMU emulated-VFIO path, it's **unnecessary and harmful**: + +1. **Unnecessary**: open-iov-vfio.ko handles all VFIO emulation independently +2. **Harmful**: Loading the 610-based submodule alongside apt 570.211.01 causes driver conflicts that crash the host GSP (WPR2 stuck, `unexpected WPR2 already up`) + +**Root cause fixed 2026-09-22** (`85d0668`): stack_manager skips submodule when apt nvidia is present. + +--- + +## 4. Emulated Component Details + +### 4.1 BAR0 Shadow (16 MB vzalloc) + +The guest's BAR0 is a software shadow, NOT the physical GPU's BAR0. + +| BAR0 offset | Region | Who writes | Who reads | +|-------------------|-------------------------------------|------------|-----------| +| `0x000-0xFFF` | Emulated registers (PMC boot, etc.) | shadow | shadow | +| `0x1000` | RPC request mailbox | guest | emu thread| +| `0x1080` | RPC response mailbox | emu thread | guest | +| `0x2000` | MSI-X table (32 entries) | guest | guest | +| `0x3000` | MSI-X PBA (pending bits) | emu thread | guest | +| `0x110040` | GSP MAILBOX0 (boot args) | guest | emu thread| +| `0x110C00` | GSP doorbell | guest | emu thread| +| `0x300000` | VBIOS (1MB PROM window) | host init | guest | +| rest | Emulated register space | shadow | shadow | + +**Critical**: The GSP emulation thread polls the **shadow** exclusively. Touching the physical GPU's BAR0 (even via `ioremap`) caused the original GSP crash (fixed `f60ae7f`). + +### 4.2 BAR1 Emulated Vidmem (256 MB vzalloc) + +Guest VRAM aperture backed by host RAM. This is a **placeholder** — for real compute, this must be backed by physical GPU memory (see Phase 3 below). + +### 4.3 BAR3 Shadow (32 MB vzalloc) + +Instance memory window. **Must be shadowed** — mapping to the physical GPU's BAR3 would let guest writes corrupt host GPU instance memory (fixed `85d0668`). + +### 4.4 GSP Boot Emulation + +The guest driver boots "GSP" through falcon stages: + +1. **FWSEC FRTS** (GSP falcon CPUCTL) → auto-complete: HALTED + VBIOS_SCRATCH[0x0E]=0 + WPR2 ADDR +2. **Booter Load** (SEC2 falcon CPUCTL) → auto-complete: SEC2_MAILBOX0=0 + HALTED + RISCV_STATUS=1 +3. **GSP-RM msgq init** (libos args via MAILBOX0/1) → our emulation parses the queue chain and processes RPCs + +### 4.5 GSP-RM Emulation + +The guest's physical-path driver uses the **real GSP-RM msgq protocol** (not the old PV doorbell/BAR0-shared-mem). Our emulation: +- Parses `LibosMemoryRegionInitArgument` chain from GSP MAILBOX0/1 +- Finds the shared queue buffer in guest sysmem (129 non-contiguous 4K pages) +- Processes commands from the command queue +- Writes replies to the status queue +- Injects MSI-X interrupts + +Key RPC types handled: GSP_INIT_DONE, SET_SYSTEM_INFO, SET_REGISTRY, SET_GUEST_SYSTEM_INFO, GET_GSP_STATIC_INFO, GSP_RM_CONTROL (multiple sub-commands). + +--- + +## 5. Phased Implementation Roadmap + +### Phase 1: Correctness — GSP-RM Emulation Complete (IN PROGRESS) + +**Goal**: `nvidia-smi` works in the guest. Guest driver fully initializes. + +**What's needed**: Complete GSP-RM reply content for the RM init sequence: +- `INTERNAL_GET_DEVICE_INFO_TABLE` (0x20800a40) — 11 device entries +- `FIFO_GET_DEVICE_INFO_TABLE` (0x20801112) — 12 engine entries +- `INTERNAL_INTR_GET_KERNEL_TABLE` (0x20800a5c) — 8 entries + subtreeMap +- `GspStaticConfigInfo` — full TU104 config +- All remaining `GSP_RM_CONTROL` sub-commands + +**Oracle**: The host apt nvidia driver (570.211.01) has a **real GSP** managing the physical GPU. Use `tools/rmq/rmq` or `nvidia-smi` on host to dump ground-truth struct layouts. Compare against `kernel/gsp_rm_tu104.h`. + +**Current blocker**: `RmInitNvDevice` fails with `Cannot initialize the device (0x24:0x72:1099)`. The fn-10 FREE timeout during cleanup is characterized but not fully resolved. + +### Phase 2: Memory Model — Guest VRAM Backed by Real GPU Memory + +**Goal**: Guest CUDA memory allocations land in physical GPU VRAM (or DMA-coherent host memory accessible by the GPU). + +**What's needed**: +- Replace emulated vidmem (BAR1 host RAM) with DMA-mapped region the physical GPU can access +- Implement `SET_PAGE_DIRECTORY`, `UPDATE_PDE_2`, `ALLOC` handlers that map guest allocations to real GPU address space +- Ensure cache coherency between guest CPU and GPU DMA + +**Key challenge**: The guest's GPU VA space must map to physical GPU VRAM addresses. The host driver's GSP manages the physical GPU's memory. We'd need to either: +- (a) Cooperate with host GSP (share VRAM regions) — complex +- (b) Use a dedicated VRAM partition for the guest — requires host driver cooperation +- (c) Use system memory with DMA mapping (slower but simpler) — guest sees "unified memory" + +### Phase 3: Compute Forwarding — Guest Work Executes on Physical GPU + +**Goal**: CUDA kernels launched in the guest execute on the physical GPU. + +**What's needed**: +- Guest channel/pushbuffer writes are intercepted and forwarded to host GPU channels +- GPU context switching between host and guest contexts +- Completion interrupts relayed from host GPU to guest + +**Approach options**: +1. **GSP-RM translation**: Translate guest GSP-RM compute commands to host GSP-RM commands. The host GSP firmware already knows how to schedule work on the GPU. We'd be a "GSP proxy". +2. **Channel forwarding**: Intercept guest channel doorbell writes, translate guest pushbuffer contents, submit to host GPU channels. Requires deep understanding of GPU channel format. +3. **UVM integration**: Use the host's Unified Virtual Memory (UVM) to share CUDA memory between host and guest. Launch kernels via host CUDA APIs on behalf of guest. + +**Most promising**: Option 1 (GSP-RM translation). The guest driver sends GSP-RM commands for compute (ALLOC, MAP, SCHEDULE). Our emulation translates these to equivalent host GSP-RM commands via the host driver. The host GSP firmware executes them on the physical GPU. Results are relayed back. + +### Phase 4: Performance Optimization + +**Goal**: < 5% overhead vs native for CUDA workloads. + +**What's needed**: +- Optimize GSP-RM translation latency (batch commands, avoid unnecessary copies) +- Direct DMA mapping for large memory transfers +- Interrupt coalescing for high-frequency completions +- Profile and eliminate hot-path bottlenecks (doorbell polling, shared memory access) + +### Phase 5: Graphics & Video + +**Goal**: OpenGL/Vulkan rendering, NVENC/NVDEC video. + +**What's needed**: +- Display surface management (virtual display or forwarded framebuffer) +- Graphics pipeline forwarding (similar to compute forwarding) +- Video engine access (NVENC/NVDEC on physical GPU) + +--- + +## 6. Testing Strategy + +### Correctness Testing (Phase 1) +- [x] Host nvidia-smi works with open-iov-vfio loaded +- [x] mdev create/remove lifecycle clean (no hang, no host GPU corruption) +- [x] QEMU guest boots with vGPU attached +- [x] Guest PCI probe detects vGPU (TU104 [10de:1ed3]) +- [x] Guest nvidia driver loads and creates /dev/nvidia0 +- [ ] Guest RmInitAdapter completes (RmInitNvDevice succeeds) +- [ ] Guest nvidia-smi shows GPU +- [ ] Guest CUDA context creation + +### Performance Testing (Phases 2-4) +- [ ] GSP-RM round-trip latency < 10μs +- [ ] CUDA kernel launch overhead < 5% vs native +- [ ] Memory bandwidth within 10% of native +- [ ] Graphics frame time within 5% of native + +--- + +## 7. Key Files + +| File | Role | +|------|------| +| `kernel/gsp_emulation.c/h` | GSP emulation thread, falcon boot auto-complete, MSI-X injection | +| `kernel/gsp_rm_emulation.c/h` | GSP-RM msgq protocol emulation (guest sysmem queues) | +| `kernel/gsp_rm_tu104.h` | TU104 hardware tables (engines, interrupts, falcons, device info) | +| `kernel/gsp_rpc_client.c/h` | RPC API + emulation implementations | +| `kernel/bar_emulation.c/h` | BAR0 shadow allocation, MSI-X shadow state | +| `kernel/bar_init.c` | BAR init (shadow vidmem, BAR3 shadow, physical ioremap for reference) | +| `kernel/main.c` | VFIO ops, mdev create/remove, PCI config space, VFIO mmap | +| `kernel/memory.c` | DMA mapping, VRAM management | +| `kernel/irq.c` | MSI-X/MSI setup and injection | +| `tools/rmq/rmq.c` | Host RM API client (oracle for GSP-RM ground truth) | + +--- + +## 8. Host Stability Guarantees (2026-09-22) + +After the host-stability fix (`85d0668`), the following are guaranteed: + +1. **open-iov-vfio.ko loads/unloads cleanly** with apt nvidia driver present +2. **Host nvidia-smi works throughout** the full lifecycle (load, mdev, QEMU, unload) +3. **No host GPU corruption** — no WPR2 stuck, no GSP crash, no module hang +4. **Guest driver probe is safe** — even when RmInitNvDevice fails, host is unaffected +5. **mdev create/remove is clean** — gsp_emu thread starts/stops properly + +**Root causes that were fixed**: +- Submodule nvidia.ko conflict with apt nvidia → skip submodule +- BAR3 physical mapping to guest → vzalloc'd shadow +- GSP-RM teardown garbage declaring GSP death → quiet stop polling + +--- + +*Document Version: 2.0 (Emulate vs Pass-Through Architecture)* +*Updated: 2026-09-22 (post host-stability fix)* +*Repository: ~/Git/integdev_gpu_drv/open-iov* +*Git Commit: 85d0668 (host stability fix)* diff --git a/docs/refactored-implementation-to-do.md b/docs/refactored-implementation-to-do.md new file mode 100644 index 0000000..f7e0097 --- /dev/null +++ b/docs/refactored-implementation-to-do.md @@ -0,0 +1,509 @@ +# Open-IOV vGPU-IO Refactored Implementation TODO List + +## Overview +This document tracks the granular implementation steps to refactor Open-IOV from guest-side emulation to **host-side GSP emulation**, enabling unmodified guest NVIDIA drivers to run with full functionality. + +**Restore Point**: Git commit `f9ed5d8` - complete current stack before refactoring +**Target**: Host-only GSP emulation in `open-iov-vfio.ko`, guest runs pure unmodified `nvidia.ko` + +--- + +## Phase 0: Repository & Infrastructure Setup + +### 0.1 Git Branching +- [ ] `git checkout -b refactor/host-gsp-emulation` from `f9ed5d8` +- [ ] Create `docs/old/` with previous ARCHITECTURE.md (DONE) +- [ ] Create `docs/refactored-architecture.md` (DONE) +- [ ] Create this TODO document + +### 0.2 Build System Verification +- [ ] Verify current build works: `make clean && make` in open-iov root +- [ ] Verify all 3 kernel modules build: `nvidia.ko`, `open-iov-vfio.ko`, `gsp-vgpu.ko` +- [ ] Verify userspace builds: `libiov.so`, `example_vgpu` +- [ ] Document current build commands for regression testing + +### 0.3 Test Infrastructure +- [x] Create `scripts/test_gsp_emu.sh` - automated GSP emulation unit tests +- [x] Create `scripts/test_guest_driver.sh` - guest driver validation suite +- [x] Create `scripts/test_cuda_workloads.sh` - CUDA workload validation +- [x] Create `scripts/test_graphics_workloads.sh` - graphics validation +- [x] Create `scripts/test_windows_guest.sh` - Windows guest validation +- [x] Create `scripts/test_vgpu_stack.sh` - comprehensive test runner +- [ ] Set up CI pipeline for incremental testing + +--- + +## Phase 1: Host Kernel - GSP Emulation Engine (Core) + +### 1.1 New File: `kernel/gsp_emulation.h` +- [x] Define `struct gsp_emu_context` with: + - [x] `struct nv_vgpu_device *vgpu` + - [x] `struct task_struct *thread` + - [x] `void __iomem *bar0_vaddr`, `*doorbell_reg`, `*shared_mem` + - [x] `struct gsp_client *rpc_client` + - [x] `spinlock_t lock` + - [x] `bool running` + - [x] Statistics: `u64 processed_rpcs`, `failed_rpcs`, `total_latency_ns` +- [x] Function prototypes: + - [x] `int gsp_emu_init(struct nv_vgpu_device *vgpu)` + - [x] `void gsp_emu_cleanup(struct nv_vgpu_device *vgpu)` + - [x] `int gsp_emu_process_rpc(struct gsp_emu_context *ctx)` + - [x] `void gsp_emu_inject_completion_interrupt(struct gsp_emu_context *ctx, u32 seq_id)` + +### 1.2 New File: `kernel/gsp_emulation.c` +- [x] Implement `gsp_emu_thread_fn()` - per-mdev polling thread + - [x] Read doorbell at `bar0_vaddr + 0x110C00` + - [x] ACK doorbell immediately (write 0) + - [x] Call `gsp_emu_process_rpc()` + - [x] Adaptive polling: busy-wait first 100μs, then `usleep_range(50, 100)` + - [x] Handle `kthread_should_stop()` for clean shutdown +- [x] Implement `gsp_emu_init()` + - [x] Allocate `gsp_emu_context` + - [x] Map BAR0, locate doorbell reg + shared mem offset (0x1000) + - [x] Allocate shared memory (4KB, page-aligned, WC mapping) + - [x] Map shared memory into guest BAR0 at 0x1000 + - [x] Initialize `gsp_client` via `gsp_client_init()` (reuse existing) + - [x] Start kthread `gsp_emu_thread_fn` +- [x] Implement `gsp_emu_cleanup()` + - [x] Stop kthread + - [x] Free shared memory + - [x] Call `gsp_client_fini()` + - [x] Free context +- [x] Implement `gsp_emu_process_rpc()` - dispatch all 12 RPC types + - [x] Copy request from shared memory (offset 0x0000) + - [x] Validate seq_id != 0 + - [x] Switch on `req.type` → call appropriate `gsp_rpc_*` function + - [x] Write response to shared memory (offset 0x0080) + - [x] Call `gsp_emu_inject_completion_interrupt()` +- [x] Implement `gsp_emu_inject_completion_interrupt()` + - [x] Write to MSI-X PBA at `bar0_vaddr + 0x3000` (vector 0) + - [x] Signal eventfd: `eventfd_signal(vgpu->msix_trigger[0])` + +### 1.3 Modify `kernel/main.c` +- [x] Add `struct gsp_emu_context *gsp_emu` to `struct nv_vgpu_device` (reused `gsp_comm_ctx`) +- [x] In `nv_vgpu_device_init_work()`: call `gsp_emu_init(vgpu)` after `nv_vgpu_gsp_init()` +- [x] In `nv_vgpu_device_cleanup_work()`: call `gsp_emu_cleanup(vgpu)` before `nv_vgpu_gsp_cleanup()` +- [x] Include `gsp_emulation.h` + +### 1.4 Modify `kernel/Makefile` +- [x] Add `gsp_emulation.o` to `open-iov-vfio-y` or module objects +- [x] Verify build: `make clean && make` + +### 1.5 Unit Tests for GSP Emulation (kernel) +- [x] Create `kernel/tests/gsp_emulation_test.c` +- [x] Create `kernel/tests/Makefile` +- [x] Test ALLOC/FREE round-trip +- [x] Test MAP/UNMAP with various sizes/alignments +- [x] Test PIN/UNPIN returns valid DMA addresses +- [x] Test SCHEDULE/YIELD completes +- [x] Test BAR_TRAP read/write emulation +- [x] Test NOTIFY register/unregister +- [x] Test MIGRATE 4-phase state machine +- [x] Test concurrent RPCs from multiple threads +- [x] Test error handling (invalid seq_id, bad payload) +- [x] Add to `kernel/tests/Makefile` + +--- + +## Phase 2: PCI Config Space & BAR Enhancements + +### 2.1 Enhance PCI Config Space (`kernel/main.c`) +- [x] Verify MSI-X capability at 0x70 points to Table@BAR0+0x2000, PBA@BAR0+0x3000 +- [x] Ensure `VFIO_PCI_CONFIG_REGION_INDEX` returns full 256B config space +- [x] Add config space write handlers for: + - [x] Command register (0x04): Memory/IO/BusMaster enable + - [x] MSI-X Message Control (0x72): Enable/disable, table size + - [x] MSI-X Table Offset (0x74-0x77): Verify BAR0+0x2000 + - [x] MSI-X PBA Offset (0x78-0x7B): Verify BAR0+0x3000 + +### 2.2 BAR Region VFIO Mapping (`kernel/bar_emulation.c`) +- [x] Implement `VFIO_PCI_BAR0_REGION_INDEX` → 16MB emulated config space +- [x] Implement `VFIO_PCI_BAR1_REGION_INDEX` → 256MB VRAM (backed by host allocation) +- [x] Implement `VFIO_PCI_BAR2_REGION_INDEX` → 32MB reserved +- [x] Implement `VFIO_PCI_BAR3_REGION_INDEX` → 32MB reserved +- [x] Implement `VFIO_PCI_MSIX_REGION_INDEX` → MSI-X Table (0x2000) + PBA (0x3000) +- [x] Ensure BAR0 mmap includes doorbell @ 0x110C00 and shared mem @ 0x1000 + +### 2.3 MSI-X Table/PBA Emulation +- [x] Allocate MSI-X table (32 entries × 16 bytes = 512 bytes) in BAR0 at 0x2000 +- [x] Allocate PBA (32 bits = 4 bytes, aligned to 64 bytes) in BAR0 at 0x3000 +- [x] Guest writes to MSI-X table entries → host stores for interrupt injection +- [x] Guest reads PBA → host returns pending bits + +--- + +## Phase 3: GSP RPC Handler Completion + +### 3.1 Verify All 12 RPC Handlers in `gsp_rpc_client.c` +- [x] ALLOC (0x01): VRAM allocation via host nvidia.ko +- [x] FREE (0x02): Free allocation +- [x] MAP (0x03): Map to GPU VA +- [x] UNMAP (0x04): Unmap GPU VA +- [x] PIN (0x05): Pin for DMA, return DMA address +- [x] UNPIN (0x06): Unpin pages +- [x] SCHEDULE (0x07): Submit work to local workqueue +- [x] YIELD (0x08): No-op success +- [x] NOTIFY (0x09): Async notification registration +- [x] BAR_TRAP (0x0A): Shadow register read/write +- [x] MIGRATE (0x0B): 4-phase protocol +- [x] DEBUG (0xFF): Firmware introspection + +### 3.2 Enhance Handler Implementations +- [x] ALLOC: Use host page allocation + DMA mapping +- [x] MAP: GPU VA mapping management +- [x] PIN: Return DMA address from mapping +- [x] BAR_TRAP: Use bar_emulation.c shadow memory +- [x] MIGRATE: Use migration.c 4-phase state machine +- [x] NOTIFY: Use gsp_rpc_notify_register() + eventfd for guest delivery + +### 3.3 Add RPC Validation & Error Handling +- [x] Sequence ID tracking (detect duplicates, gaps) +- [x] Payload size validation per RPC type +- [x] Timeout handling (guest watchdog) +- [x] Proper error codes in response.status + +--- + +## Phase 4: Portable Architecture Implementation + +### 4.1 Firmware Analyzer (`kernel/firmware_analyzer.c`) +- [x] Compute firmware CRC32 and SHA256 signatures +- [x] Pattern matching for RPC dispatcher discovery +- [x] Pattern matching for jump table discovery +- [x] Pattern matching for ALLOC handler discovery +- [x] Known firmware signature database +- [x] Auto-discovery fallback for unknown firmware +- [x] `fw_analyzer_analyze()` - analyze firmware blob +- [x] `fw_analyzer_load_and_analyze()` - load and analyze from filesystem +- [x] `fw_analyzer_print_result()` - debug output + +### 4.2 Dynamic RPC Dispatcher (`kernel/dynamic_rpc_dispatcher.c`) +- [x] Version-specific interface registry (550, 560, 570, 580) +- [x] Native firmware handler function pointers +- [x] Emulation fallback interface +- [x] Automatic interface selection based on firmware analysis +- [x] Per-RPC-type emulation toggle +- [x] Dispatcher API for all 12 RPC types +- [x] `dynamic_dispatcher_init()` - initialize with firmware path +- [x] `dynamic_dispatcher_get_interface()` - get current interface +- [x] `dynamic_dispatcher_use_emulation()` - check if emulation used for type + +### 4.3 Hardware Architecture Layer (`kernel/architecture/`) +- [x] `architecture.h` - hardware abstraction API +- [x] `architecture.c` - architecture implementations +- [x] Turing architecture (TU102/TU104) config +- [x] Ampere architecture (GA100/GA102/GA104) config +- [x] Blackwell architecture (GB100/GB202) config +- [x] Architecture lookup by name and PCI ID +- [x] BAR configuration queries +- [x] Doorbell/shared memory offset queries +- [x] MSI-X vector count queries +- [x] Feature support queries (MIG, SR-IOV) + +### 4.4 Driver Version Adaptation Layer (`kernel/version/`) +- [x] `version.h` - version adaptation API +- [x] `version.c` - version implementations +- [x] Version 550 interface +- [x] Version 560 interface +- [x] Version 570 interface +- [x] Version 580 interface +- [x] Feature detection per version +- [x] Interface lookup by version string + +### 4.5 Declarative Configuration Files +- [x] `config/hardware/turing.yaml` - Turing hardware config +- [x] `config/hardware/ampere.yaml` - Ampere hardware config +- [x] `config/hardware/blackwell.yaml` - Blackwell hardware config +- [x] `config/versions/550.yaml` - Driver 550 config +- [x] `config/versions/560.yaml` - Driver 560 config +- [x] `config/versions/570.yaml` - Driver 570 config +- [x] `config/versions/580.yaml` - Driver 580 config + +--- + +## Phase 5: Guest Driver Integration Testing + +### 4.1 Host Module Loading Sequence +- [ ] `sudo insmod nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko` +- [ ] `sudo insmod kernel/open-iov-vfio.ko` +- [ ] Verify mdev types appear: `ls /sys/bus/pci/devices/0000:XX:XX.X/mdev_supported_types/` +- [ ] Create mdev: `echo <UUID> | sudo tee /sys/bus/pci/devices/.../mdev_supported_types/nvidia-vgpu-turing/create` + +### 4.2 QEMU Launch with mdev +- [ ] `qemu-system-x86_64 -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/<UUID>,display=off` +- [ ] Verify guest sees PCI device: `lspci -d 10de:` → should show 10de:1ed3 +- [ ] Verify BARs: `lspci -v -d 10de:` → BAR0=16MB, BAR1=256MB, BAR2=32MB, BAR3=32MB +- [ ] Verify MSI-X: `lspci -v -d 10de:` → MSI-X capability present + +### 4.3 Guest NVIDIA Driver Installation +- [ ] Install OpenRM 570.144 in guest: `./NVIDIA-Linux-x86_64-570.144.run --silent --dkms` +- [ ] Install firmware: Copy `bootloader`, `booter_load`, `gsp` binaries to `/lib/firmware/nvidia/tu102/gsp/` +- [ ] Load driver: `sudo modprobe nvidia` +- [ ] Verify: `nvidia-smi` shows vGPU (T4 or equivalent) +- [ ] Verify: `dmesg | grep -i nvidia` shows successful probe + +### 4.4 Basic Functionality Tests +- [ ] `nvidia-smi -q` shows all fields populated +- [ ] `nvidia-smi dmon` shows utilization counters +- [ ] `nvidia-smi mig` shows MIG not supported (expected for vGPU) +- [ ] `cat /proc/driver/nvidia/gpus/0000:00:00.0/information` shows correct info + +--- + +## Phase 5: CUDA Workload Validation + +### 5.1 CUDA SDK Tests +- [ ] Compile and run `vectorAdd` sample +- [ ] Compile and run `matrixMul` sample +- [ ] Compile and run `simpleCUBLAS` sample +- [ ] Compile and run `simpleCUFFT` sample +- [ ] Verify numerical correctness vs CPU reference + +### 5.2 cuBLAS / cuDNN Tests +- [ ] SGEMM/DGEMM various sizes (64x64 to 4096x4096) +- [ ] Batched GEMM +- [ ] cuDNN convolution forward/backward +- [ ] cuDNN RNN/LSTM +- [ ] Performance within 10% of native + +### 5.3 PyTorch / TensorFlow Integration +- [ ] PyTorch: `torch.cuda.is_available()`, tensor ops, autograd +- [ ] PyTorch: Training loop (ResNet-50, BERT-small) +- [ ] TensorFlow: `tf.config.list_physical_devices('GPU')` +- [ ] TensorFlow: Keras model training +- [ ] Mixed precision (FP16/BF16) works + +### 5.4 Multi-GPU / NCCL Tests +- [ ] Create 2+ vGPUs on same physical GPU +- [ ] NCCL all-reduce across vGPUs +- [ ] NCCL broadcast/reduce +- [ ] Multi-process CUDA (MPS) if supported + +### 5.5 CUDA Graphs & Streams +- [ ] CUDA Graph capture/replay +- [ ] Multiple streams concurrent execution +- [ ] Stream priorities +- [ ] Event synchronization + +--- + +## Phase 6: Graphics Workload Validation + +### 6.1 Vulkan Tests +- [ ] `vulkaninfo` shows GPU properties +- [ ] `vkcube` renders correctly +- [ ] `vkcubepp` (Vulkan 1.1+) renders +- [ ] Sascha Willems Vulkan samples (deferred, compute, ray tracing if supported) +- [ ] VK_KHR_swapchain, VK_KHR_dedicated_allocation work + +### 6.2 OpenGL Tests +- [ ] `glxinfo | grep -i nvidia` shows correct renderer +- [ ] `glxgears` runs at expected FPS +- [ ] `glmark2` benchmark completes +- [ ] Piglit test suite (subset) passes + +### 6.3 Video Encode/Decode +- [ ] FFmpeg + NVENC: H.264 encode 1080p60 +- [ ] FFmpeg + NVENC: HEVC encode 4K30 +- [ ] FFmpeg + NVDEC: H.264 decode 1080p60 +- [ ] FFmpeg + NVDEC: HEVC decode 4K30 +- [ ] GStreamer nvh264enc/nvh265enc elements +- [ ] GStreamer nvh264dec/nvh265dec elements + +### 6.4 Windows Guest Graphics (Separate Track) +- [ ] Windows 11 guest with NVIDIA 570.144 driver +- [ ] Device Manager: "NVIDIA T4" (or equivalent) with no error codes +- [ ] GPU-Z / CUDA-Z detect vGPU correctly +- [ ] D3D11: `D3D11CreateDevice` succeeds, render triangle +- [ ] D3D12: `D3D12CreateDevice` succeeds, render triangle +- [ ] Vulkan on Windows: `vkCreateInstance`, render +- [ ] DXGI swapchain present works +- [ ] Video encode via Media Foundation (MFT) +- [ ] Video decode via Media Foundation (MFT) + +--- + +## Phase 7: Performance Benchmarking + +### 7.1 Microbenchmarks +- [ ] GSP RPC round-trip latency (target < 10μs) +- [ ] BAR register access latency (target < 2μs) +- [ ] VRAM alloc/free latency (target < 50μs) +- [ ] VRAM map/unmap latency (target < 20μs) +- [ ] DMA pin/unpin latency (target < 30μs) +- [ ] MSI-X injection latency (target < 5μs) + +### 7.2 Macrobenchmarks +- [ ] CUDA kernel launch overhead (target < 5% vs native) +- [ ] cuBLAS SGEMM 4096x4096 (target > 90% native GFLOPS) +- [ ] cuDNN ResNet-50 training step (target > 85% native) +- [ ] Vulkan frame time (target < 2ms @ 1080p) +- [ ] NVENC 1080p60 encode latency (target < 5ms) +- [ ] NVDEC 1080p60 decode latency (target < 3ms) + +### 7.3 Stress Tests +- [ ] 24-hour CUDA compute stress (no errors, no memory leaks) +- [ ] 24-hour graphics stress (Unigine Heaven/Valley loop) +- [ ] 1000x VF create/destroy cycles +- [ ] Concurrent CUDA + graphics workloads +- [ ] Memory pressure test (allocate 90% VRAM, verify OOM handling) + +--- + +## Phase 8: Live Migration Validation + +### 8.1 Migration Protocol Tests +- [ ] QUERY phase: source reports capabilities +- [ ] PREPARE phase: destination allocates resources +- [ ] TRANSFER phase: iterative VRAM + GSP state copy +- [ ] COMMIT phase: cutover, guest resumes on destination +- [ ] CLEANUP phase: source releases resources + +### 8.2 Migration Under Load +- [ ] Migrate idle vGPU (downtime < 50ms) +- [ ] Migrate running CUDA kernel (downtime < 100ms) +- [ ] Migrate running graphics (no visual artifacts) +- [ ] Migrate with pinned DMA buffers +- [ ] Verify post-migration CUDA/graphics work correctly + +### 8.3 Migration State Verification +- [ ] GSP context (registers, PC, stack) restored +- [ ] Page tables identical +- [ ] VRAM contents identical (checksum) +- [ ] BAR register state identical +- [ ] MSI-X table/PBA state identical + +--- + +## Phase 9: Windows Guest Validation + +### 9.1 Windows Guest Setup +- [ ] Windows 10 22H2 / Windows 11 23H2 guest +- [ ] VirtIO drivers for network/storage +- [ ] QEMU guest agent +- [ ] UEFI firmware (OVMF with TPM) + +### 9.2 NVIDIA Driver Installation +- [ ] Download 570.144 Windows driver +- [ ] Install with `setup.exe -s -noreboot` (silent) +- [ ] Reboot guest +- [ ] Verify in Device Manager: "NVIDIA T4" (Code 0 = working) + +### 9.3 Windows Compute Tests +- [ ] `nvidia-smi.exe` shows vGPU +- [ ] CUDA-Z detects GPU, runs device query +- [ ] `vectorAdd.exe` (compiled in guest) executes +- [ ] PyTorch/TensorFlow in Windows (WSL2 or native) + +### 9.4 Windows Graphics Tests +- [ ] `dxdiag` shows correct GPU info +- [ ] D3D11 tutorial samples render +- [ ] D3D12 tutorial samples render +- [ ] Vulkan SDK samples run +- [ ] FurMark / 3DMark stress test + +### 9.5 Windows Video Tests +- [ ] Media Foundation H.264/HEVC encode +- [ ] Media Foundation H.264/HEVC decode +- [ ] NVENC API direct usage +- [ ] NVDEC API direct usage + +--- + +## Phase 10: Reliability & Production Hardening + +### 10.1 Error Handling & Recovery +- [ ] Guest driver crash → host cleans up mdev state +- [ ] Host kernel oops in GSP emulation → watchdog restarts thread +- [ ] Physical GPU reset → host reinitializes GSP emulation +- [ ] IOMMU fault → proper error to guest +- [ ] VRAM OOM → proper CUDA_OUT_OF_MEMORY to guest + +### 10.2 Monitoring & Observability +- [ ] Debugfs interface: `/sys/kernel/debug/open-iov-vfio/` + - [ ] Per-mdev RPC statistics + - [ ] Latency histograms + - [ ] Error counters + - [ ] GSP emulation thread CPU usage +- [ ] Prometheus metrics exporter (optional) +- [ ] Tracepoints for RPC processing + +### 10.3 Security Hardening +- [ ] Validate all guest inputs (seq_id, payload sizes, pointers) +- [ ] Isolate mdev memory (no cross-VF access) +- [ ] Rate-limit doorbell writes (prevent DoS) +- [ ] Audit VFIO ioctl permissions + +### 10.4 Documentation & Handoff +- [ ] Update `docs/refactored-architecture.md` with as-built details +- [ ] Create `docs/OPERATIONS.md` for deployment +- [ ] Create `docs/TROUBLESHOOTING.md` for common issues +- [ ] Record all firmware addresses used +- [ ] Document test results for each phase + +--- + +## Success Criteria Checklist + +### Minimum Viable Product (Phases 0-4) +- [ ] Host loads `open-iov-vfio.ko` with GSP emulation thread +- [ ] mdev creates successfully with correct PCI config space +- [ ] Guest `nvidia.ko` loads, probes vGPU successfully +- [ ] `nvidia-smi` in guest shows vGPU +- [ ] Simple CUDA kernel (`vectorAdd`) executes correctly + +### Compute Ready (Phases 5-6) +- [ ] cuBLAS/cuDNN workloads pass numerical verification +- [ ] PyTorch/TensorFlow training works +- [ ] Vulkan/OpenGL rendering works +- [ ] Video encode/decode works +- [ ] Performance > 85% native for compute, > 90% for graphics + +### Production Ready (Phases 7-10) +- [ ] 24-hour stress tests pass +- [ ] Live migration works under load +- [ ] Windows guest fully functional +- [ ] Monitoring/observability in place +- [ ] Security audit passed +- [ ] Documentation complete + +--- + +## Dependencies & Prerequisites + +| Component | Version | Source | +|-----------|---------|--------| +| Linux Kernel | 6.8+ | Host distro | +| NVIDIA OpenRM | 570.144 | `nvidia-open-gpu-kernel-modules` submodule | +| QEMU | 8.0+ | Host distro / source | +| libvfio-user | Latest | `libvfio-user` repo | +| CUDA Toolkit | 12.4+ | Guest (for compilation) | +| Vulkan SDK | 1.3+ | Guest | +| Windows 10/11 | 22H2/23H2 | Guest ISO | + +--- + +## Tracking + +| Phase | Status | Start Date | End Date | Notes | +|-------|--------|------------|----------|-------| +| 0: Setup | ✅ DONE | - | - | Git repo, docs, restore point | +| 1: GSP Emulation Engine | ✅ DONE | | | Core implementation complete | +| 2: PCI/BAR Enhancements | ✅ DONE | | | Config space, MSI-X, BAR regions | +| 3: RPC Handler Completion | ✅ DONE | | | All 12 RPC types with emulation | +| 4: Portable Architecture | ✅ DONE | | | Firmware analyzer, dispatcher, arch/version layers, configs | +| 5: Guest Driver Integration | 🔄 IN PROGRESS | | | Requires reboot to test | +| 6: CUDA Validation | ⏳ PENDING | | | | +| 7: Graphics Validation | ⏳ PENDING | | | | +| 8: Performance Benchmarks | ⏳ PENDING | | | | +| 9: Migration Validation | ⏳ PENDING | | | | +| 10: Windows Guest | ⏳ PENDING | | | | +| 11: Production Hardening | ⏳ PENDING | | | | +| 10: Production Hardening | ⏳ PENDING | | | | + +--- + +*TODO List Version: 1.0* +*Repository: $OPEN_IOV_ROOT* +*Branch: refactor/host-gsp-emulation* +*Base Commit: f9ed5d8*
\ No newline at end of file diff --git a/docs/tools/README.md b/docs/tools/README.md new file mode 100644 index 0000000..232fbe5 --- /dev/null +++ b/docs/tools/README.md @@ -0,0 +1,80 @@ +# Open-IOV Tools Documentation + +## Overview +This directory contains documentation for userspace tools built on top of the +Open-IOV vGPU-IO stack. The RM-API tooling reuses behaviour and interfaces +from the driver components we studied by reverse engineering and decompilation +(tracked docs describe components and behaviour). + +## Tools + +### Debugging & Observability (Implemented) +| Tool | Location | Description | +|------|----------|-------------| +| `guest_init_monitor.py` | `tools/` | Correlates host dmesg with guest serial output | +| `failure_analyzer.py` | `tools/` | Classifies guest init failures with recommendations | +| `benchmark_infrastructure.sh` | `tools/` | VM vs bare metal benchmark runner | +| `guest_test_agent.py` | `tools/guest_agent/` | Guest-side test agent (vsock/TCP) | +| `debug_guest_init.sh` | `scripts/` | One-command debug launch (QEMU + GDB + monitor) | +| `windows_guest_automation.sh` | `scripts/` | Windows guest automation via WinRM | + +### RM API Tooling (Planned) +| Tool | Documentation | Description | +|------|---------------|-------------| +| `rmtool` | [rmtool.md](rmtool.md) | RM API Swiss Army Knife - CLI for state enumeration, engine control, memory ops, NV2080 direct control | +| `rmtop` | [rmtop.md](rmtop.md) | GPU htop equivalent - ncurses real-time dashboard | + +## Code Reuse Sources (component level) + +All planned tools reuse the driver components we studied. Tracked +documentation refers to them by component only: + +| Component | Use Case | +|-----------|----------| +| NVML / nvidia-smi component | CLI framework, GPU enumeration, NV2080 ioctls | +| gpumon / perfmon component | Performance sampling, perfmon counters | +| NV2080 control component | RM control interfaces (classes, methods, events) | +| Resource server component | Resource Server (resserve) API | +| CUDA software (CUDASW) component | CUDA internal layer behaviours | +| Resource manager component | Resman API | + +## Build Integration +Tools will be built under `open-iov/tools/`: +``` +tools/ +├── rmtool/ +│ ├── src/ +│ ├── lib/ +│ └── Makefile +├── rmtop/ +│ ├── src/ +│ ├── lib/ +│ └── Makefile +└── Makefile # Builds all tools +``` + +## Dependencies +- `libnvidia-rmapi.so` - reused RM API layer (component level) +- `libgpumon.so` - reused perfmon layer (component level) +- `libpciaccess` - PCI configuration access +- `libnuma` - NUMA topology +- `ncursesw` - rmtop UI (wide character support) +- `libjson-c` - JSON output formatting +- `libcurl` - Remote API (future) + +## Integration with Open-IOV Stack +Tools will integrate with the host VFIO driver via: +- `/dev/nvidia*` character devices (existing nvidia.ko) +- `open-iov-vfio.ko` debugfs interface (`/sys/kernel/debug/open-iov-vfio/`) +- VFIO ioctls for guest state introspection +- GSP-RM telemetry via firmware analyzer / dynamic dispatcher + +## Future Tool Ideas +| Tool | Purpose | +|------|---------| +| `rmmigrate` | Live migration coordinator via resserve | +| `rmprofile` | Workload profiler (CUDA/Video/AI correlation) | +| `rmpolicy` | Policy engine for automated engine allocation | +| `rmtelemetry` | Telemetry exporter (Prometheus/OTel/InfluxDB) | +| `rmguest` | Guest introspection via vGPU VFIO interface | +| `rmfirmware` | GSP firmware analysis / update utility |
\ No newline at end of file diff --git a/docs/tools/rmtool.md b/docs/tools/rmtool.md new file mode 100644 index 0000000..a7ec2c6 --- /dev/null +++ b/docs/tools/rmtool.md @@ -0,0 +1,150 @@ +# rmtool - RM API Swiss Army Knife + +## Overview +`rmtool` is a comprehensive userspace utility for interacting with the NVIDIA Resource Manager (RM) API. It provides direct access to RM internal state, engine control, memory management, and performance telemetry. Built on top of driver components studied through reverse engineering and +decompilation (referenced by component). + +## Source Code Reuse +| Component | Source Location | Purpose | +|-----------|-----------------|---------| +| nvidia-smi component | CLI framework, NV2080 ioctls, GPU enumeration | +| gpumon / perfmon component | Performance monitoring, sampling infrastructure | +| NV2080 control component | RM control interfaces (classes, methods, events) | +| Resource server component | Resource Server (resserve) API | +| CUDA software (CUDASW) component | CUDA internal layer | +| Resource manager component | Resman API | + +## Core Commands + +### GPU Enumeration & Topology +```bash +rmtool gpu list # List all GPUs with PCI info +rmtool gpu topology # Show GPU topology (NVLink, PCIe, PCI IDs) +rmtool gpu info -d 0000:65:00.0 # Detailed GPU info (like nvidia-smi -q) +``` + +### Engine Management +```bash +rmtool engine list # List all engines (GR, CE, NVDEC, NVENC, NVJPG, OFA) +rmtool engine info -d 0 -e gr # Engine details (registers, capabilities) +rmtool engine allocate -d 0 -e gr -c 4 # Allocate 4 GR channels +rmtool engine start -d 0 -e gr -c 0 # Start engine channel +rmtool engine stop -d 0 -e gr -c 0 # Stop engine channel +rmtool engine config -d 0 -e nvenc --preset=h264 --bitrate=10M +``` + +### Channel & Context Control +```bash +rmtool channel list -d 0 # List all channels on GPU +rmtool channel create -d 0 -e gr -t compute # Create compute channel +rmtool channel submit -d 0 -c 0 --pushbuf=work.bin # Submit work +rmtool channel status -d 0 -c 0 # Channel status (busy, pending, error) +rmtool context list -d 0 # List all contexts +rmtool context info -d 0 -c 0 # Context details (page tables, memory) +``` + +### Memory Operations +```bash +rmtool mem alloc -d 0 --size=1G --type=video # Allocate VRAM +rmtool mem alloc -d 0 --size=512M --type=system # Allocate sysmem +rmtool mem map -d 0 -h 0x1234 --gpu-va=0x100000 # Map to GPU VA +rmtool mem unmap -d 0 -h 0x1234 # Unmap +rmtool mem pin -d 0 -h 0x1234 # Pin for DMA +rmtool mem info -d 0 -h 0x1234 # Allocation details +rmtool mem stats -d 0 # VRAM/sysmem usage breakdown +``` + +### Performance Sampling +```bash +rmtool sample engine -d 0 -e gr --interval=100ms --count=100 +rmtool sample clocks -d 0 --interval=1s +rmtool sample power -d 0 --interval=1s +rmtool sample thermal -d 0 --interval=1s +rmtool sample pcie -d 0 --interval=1s +``` + +### NV2080 Direct Control +```bash +rmtool nv2080 class list -d 0 # List all NV2080 classes +rmtool nv2080 method call -d 0 -c 0x906D -m 0x100 --params=... +rmtool nv2080 event subscribe -d 0 -c 0x906D -e 0x100 +rmtool nv2080 event wait -d 0 -c 0x906D -e 0x100 --timeout=5000 +``` + +### Resource Server (resserve) Integration +```bash +rmtool resserve status # Resource Server status +rmtool resserve reservation list # List all reservations +rmtool resserve reservation create --gpu=0 --engine=gr --size=4 +rmtool resserve reservation release --id=123 +``` + +### CUDA Internal Layer (CUDASW) +```bash +rmtool cudasw context list # CUDA contexts +rmtool cudasw module list # Loaded CUDA modules +rmtool cudasw stream list -c 0 # Streams in context +rmtool cudasw event list -c 0 # Events in context +``` + +### Resource Manager (Resman) +```bash +rmtool resman client list # Resman clients +rmtool resman resource list -c 0 # Resources owned by client +rmtool resman heap info -d 0 # Heap state +``` + +### GSP-RM Telemetry +```bash +rmtool gsp rpc-stats -d 0 # RPC queue depths, latency +rmtool gsp doorbell -d 0 # Doorbell rates +rmtool gsp firmware -d 0 # Firmware version, threads +rmtool gsp wpr -d 0 # WPR state +``` + +## Output Formats +```bash +rmtool gpu list --format=json +rmtool engine list --format=csv +rmtool sample engine --format=prometheus +rmtool mem stats --format=influx +``` + +## Build +```bash +cd open-iov/tools/rmtool +make +# Links against: libnvidia-rmapi.so (reused RM API component), libpciaccess, libnuma +``` + +## Architecture +``` +rmtool/ +├── src/ +│ ├── main.c # CLI parsing, command dispatch +│ ├── gpu.c # GPU enumeration, topology +│ ├── engine.c # Engine allocation/control +│ ├── channel.c # Channel/context management +│ ├── memory.c # Memory alloc/map/pin +│ ├── sample.c # Performance sampling +│ ├── nv2080.c # NV2080 direct control +│ ├── resserve.c # Resource Server client +│ ├── cudasw.c # CUDA Internal Layer +│ ├── resman.c # Resource Manager +│ ├── gsp.c # GSP-RM telemetry +│ └── output.c # JSON/CSV/Prometheus formatters +├── include/ +│ └── rmtool.h # Internal headers +├── lib/ +│ ├── librmapi.a # Recycled RM API code +│ └── libgpumon.a # Recycled perfmon code +└── Makefile +``` + +## Future Extensions +- **Remote gRPC/REST API** for fleet management +- **Policy Engine** for automated engine allocation +- **Migration Coordination** via resserve +- **Telemetry Export** (Prometheus, OpenTelemetry, InfluxDB) +- **Guest Introspection** via vGPU VFIO interface +- **Workload Profiling** with CUDA/GR engine correlation
\ No newline at end of file diff --git a/docs/tools/rmtop.md b/docs/tools/rmtop.md new file mode 100644 index 0000000..6408268 --- /dev/null +++ b/docs/tools/rmtop.md @@ -0,0 +1,227 @@ +# rmtop - GPU htop Equivalent + +## Overview +`rmtop` is an interactive, ncurses-based real-time GPU monitoring dashboard. Like `htop` but for NVIDIA GPUs, providing live visualization of engine utilization, memory, channels, power, thermals, and GSP-RM telemetry. + +## Source Code Reuse +Same recycled components as `rmtool`: +- `gpumon_perfmon` - sampling infrastructure, perfmon counters +- `nvidia-smi` - GPU enumeration, NVML-style queries +- `nv2080` - direct RM control for real-time state +- `resserve` - resource server state +- `resman` - resource manager heaps/clients + +## UI Layout + +``` +┌─────────────────────────────────────────────────────────────────────────────┐ +│ rmtop 1.0 │ GPU 0: RTX 5090 (GB202) │ Driver: 610.XX │ 14:32:15 │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ ENGINES │ UTIL │ FREQ │ POWER │ TEMP │ ACTIVE CHANNELS │ +│ ├─ GR (Graphics) │ ████████░░ 82% │ 2850MHz │ 320W │ 78°C │ ch: 0,3,7,12 │ +│ ├─ CE0 (Copy) │ ████░░░░░░ 41% │ 2100MHz │ │ │ ch: 1,4 │ +│ ├─ CE1 (Copy) │ ░░░░░░░░░░ 0% │ 300MHz │ │ │ - │ +│ ├─ NVDEC0 │ ██████████ 100% │ 1800MHz │ │ │ ch: 2 │ +│ ├─ NVDEC1 │ ░░░░░░░░░░ 0% │ 300MHz │ │ │ - │ +│ ├─ NVENC0 │ ████████░░ 82% │ 1800MHz │ │ │ ch: 5 │ +│ ├─ NVENC1 │ ░░░░░░░░░░ 0% │ 300MHz │ │ │ - │ +│ ├─ NVJPG │ ░░░░░░░░░░ 0% │ 300MHz │ │ │ - │ +│ └─ OFA │ ░░░░░░░░░░ 0% │ 300MHz │ │ │ - │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ MEMORY │ USED │ TOTAL │ FREE │ FRAG │ BAR1 │ +│ ├─ VRAM │ 18.2G │ 24.0G │ 5.8G │ 12% │ 256M / 256M │ +│ ├─ System RAM │ 4.1G │ 64.0G │ 59.9G │ 3% │ │ +│ └─ BAR1 (CPU) │ 12M │ 256M │ 244M │ 1% │ │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ CHANNELS / CONTEXTS (PID │ ENG │ CTX │ VRAM │ SYSMEM │ UTIL │ STATE) │ +│ ├─ 1247 │ GR │ 0x3 │ 2.1G │ 512M │ 82% │ RUNNING │ +│ ├─ 1247 │ CE0 │ 0x3 │ 128M │ 64M │ 41% │ RUNNING │ +│ ├─ 1247 │ DEC │ 0x3 │ 4.0G │ 256M │ 100% │ RUNNING │ +│ ├─ 1247 │ ENC │ 0x3 │ 1.5G │ 128M │ 82% │ RUNNING │ +│ ├─ 3891 │ GR │ 0x7 │ 512M │ 128M │ 12% │ WAITING │ +│ └─ 3891 │ CE0 │ 0x7 │ 64M │ 32M │ 5% │ WAITING │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ POWER / CLOCKS / THERMALS │ +│ ├─ Power Draw: 320W / 450W (71%) │ GPU Clock: 2850 MHz (max) │ +│ ├─ Mem Clock: 1313 MHz (max) │ SM Clock: 2850 MHz │ +│ ├─ Temp: 78°C (slowdown: 83°C) │ Fan: 65% (auto) │ +│ └─ PState: P0 (Performance) │ Perf Cap: None │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ GSP-RM TELEMETRY │ +│ ├─ RPC Queue: GR=12 CE=3 DEC=0 ENC=1 (depth / max=64) │ +│ ├─ Doorbell Rate: 45K/s │ Firmware Threads: 8/16 active │ +│ ├─ WPR Regions: 3 active │ Migration: IDLE │ +│ └─ Errors: 0 (CRC) 0 (Timeout) 0 (Page Fault) │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ PCIe / NVLink │ +│ ├─ Link: Gen4 x16 │ Throughput: 12.5 GB/s / 31.5 GB/s (40%) │ +│ ├─ Replay Timer: 0 │ NAK: 0 │ L0→L1: 12 L1→L0: 12 │ +│ └─ NVLink: N/A (single GPU) │ +└─────────────────────────────────────────────────────────────────────────────┘ + [F1] Help [F2] GPU Select [F3] Engine Detail [F4] Memory Detail [F5] Processes [F6] GSP [F10] Quit +``` + +## Key Bindings +| Key | Action | +|-----|--------| +| `↑/↓` | Navigate process list | +| `←/→` | Switch GPU (multi-GPU) | +| `F1` | Help | +| `F2` | GPU Select menu | +| `F3` | Engine detail view | +| `F4` | Memory detail view | +| `F5` | Process list (expanded) | +| `F6` | GSP-RM telemetry | +| `F7` | PCIe/NVLink detail | +| `F8` | Power/thermal graphs | +| `F9` | Configuration | +| `F10` / `q` | Quit | +| `Space` | Pause/resume updates | +| `s` | Change sample interval (100ms-10s) | +| `u` | Sort by GPU util / Memory / PID | +| `k` | Kill process (send SIGTERM to guest) | +| `e` | Export snapshot (JSON) | + +## Engine Detail View (F3) +``` +┌─ GR Engine Detail ────────────────────────────────────────────────────────┐ +│ Channel 0 (PID 1247, CTX 0x3) │ Util: 82% │ Submitted: 1.2M ops │ +│ ├─ Pushbuffer: 0x7f8a... │ Fence: 0x123456 │ Pending: 47 │ +│ ├─ Context: 0x3 │ Page Tables: 0x100000 │ VRAM: 2.1G │ +│ └─ Perf: SM=82% TEX=45% L2=67% ROPS=34% │ +│ Channel 3 (PID 1247, CTX 0x3) │ Util: 12% │ Submitted: 45K ops │ +│ └─ ... │ +│ Channel 7 (PID 3891, CTX 0x7) │ Util: 5% │ Submitted: 12K ops │ +│ └─ ... │ +│ Channel 12 (PID 3891, CTX 0x7) │ Util: 0% │ Submitted: 0 ops │ +│ └─ IDLE │ +└────────────────────────────────────────────────────────────────────────────────┘ +``` + +## Memory Detail View (F4) +``` +┌─ Memory Detail ────────────────────────────────────────────────────────────┐ +│ VRAM Allocations (by PID) │ +│ ├─ 1247: 18.2G ████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 76% │ +│ ├─ 3891: 0.6G ████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 2% │ +│ └─ System: 0.1G ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 0% │ +│ │ +│ Fragmentation: 12% │ Largest Free Block: 2.1G │ Allocation Count: 347 │ +│ │ +│ BAR1 Mappings (CPU→GPU) │ +│ ├─ 0x7f8a00000000 → GPU VA 0x100000 (256M) RW PID 1247 │ +│ └─ 0x7f8b00000000 → GPU VA 0x200000 (128M) RO PID 3891 │ +└────────────────────────────────────────────────────────────────────────────────┘ +``` + +## GSP-RM Telemetry View (F6) +``` +┌─ GSP-RM Telemetry ────────────────────────────────────────────────────────┐ +│ RPC Queues (depth / max=64) │ +│ ├─ GR: ████████████░░░░░░░░░░ 12/64 │ Avg Latency: 2.3μs │ +│ ├─ CE0: ███░░░░░░░░░░░░░░░░░░░░░ 3/64 │ Avg Latency: 1.1μs │ +│ ├─ CE1: ░░░░░░░░░░░░░░░░░░░░░░░░ 0/64 │ - │ +│ ├─ NVDEC: ██████████████████████ 64/64 │ Avg Latency: 4.7μs ⚠ SATURATED│ +│ ├─ NVENC: ████████████░░░░░░░░░░░░░ 1/64 │ Avg Latency: 3.2μs │ +│ └─ OFA: ░░░░░░░░░░░░░░░░░░░░░░░░ 0/64 │ - │ +│ │ +│ Doorbell Statistics │ +│ ├─ Total Rate: 45,231/s │ Peak: 120,451/s │ +│ ├─ GR: 28,123/s │ CE: 8,456/s │ DEC: 5,234/s │ ENC: 3,418/s │ +│ └─ Errors: 0 (spurious) 0 (invalid) │ +│ │ +│ Firmware Threads (16 total) │ +│ ├─ Active: 8 │ Idle: 6 │ Blocked: 2 │ +│ ├─ Thread 0: GR Scheduler │ CPU: 45% │ RPC: 12,341/s │ +│ ├─ Thread 1: CE Manager │ CPU: 12% │ RPC: 3,456/s │ +│ ├─ Thread 2: DEC Manager │ CPU: 78% │ RPC: 5,234/s ⚠ HIGH │ +│ └─ ... │ +│ │ +│ WPR State │ +│ ├─ Region 0: 0x100000-0x200000 (1M) READ-ONLY PID 1247 │ +│ ├─ Region 1: 0x300000-0x400000 (1M) NO-ACCESS PID 1247 │ +│ └─ Region 2: 0x500000-0x600000 (1M) READ-WRITE PID 3891 │ +└────────────────────────────────────────────────────────────────────────────────┘ +``` + +## Multi-GPU Support +``` +rmtop -g 0 # Start on GPU 0 +rmtop -g all # Cycle through all GPUs (tab) +rmtop --layout=grid # Show all GPUs in grid (requires wide terminal) +``` + +## Configuration (`~/.config/rmtop/config.toml`) +```toml +[general] +sample_interval_ms = 500 +theme = "dark" # dark, light, mono +show_gsp = true +show_pcie = true +show_nvlink = false + +[colors] +engine_high = "red" +engine_medium = "yellow" +engine_low = "green" +warning = "red" +critical = "bright_red" + +[columns] +# Show/hide columns in process list +pid = true +name = true +engine = true +context = true +vram = true +sysmem = true +util = true +state = true + +[alerts] +gsp_queue_warning = 48 # 75% of max +gsp_queue_critical = 56 # 87.5% of max +temp_warning = 80 +temp_critical = 85 +power_warning = 400 +``` + +## Build +```bash +cd open-iov/tools/rmtop +make +# Requires: ncursesw, libnvidia-rmapi.so, libgpumon.so, libpciaccess +``` + +## Architecture +``` +rmtop/ +├── src/ +│ ├── main.c # ncurses init, main loop +│ ├── ui.c # Layout, rendering, input handling +│ ├── gpu_panel.c # GPU summary panel +│ ├── engine_panel.c # Engine utilization bars +│ ├── memory_panel.c # Memory breakdown +│ ├── process_panel.c # Process/channel list +│ ├── power_panel.c # Power/clocks/thermals +│ ├── gsp_panel.c # GSP-RM telemetry +│ ├── pcie_panel.c # PCIe/NVLink +│ ├── detail_views.c # F3-F8 detail panels +│ ├── sampler.c # Background sampling thread +│ └── config.c # Config file parsing +├── include/ +│ └── rmtop.h +├── lib/ +│ ├── librmapi.a +│ └── libgpumon.a +└── Makefile +``` + +## Future Extensions +- **Remote Dashboard**: WebSocket server for browser-based monitoring +- **Historical Graphs**: In-terminal sparklines for utilization history +- **Guest Correlation**: Map host channels to guest PIDs via vGPU VFIO +- **Workload Classification**: Auto-detect CUDA/Video/AI/Graphic workloads +- **Policy Visualization**: Show resserve reservations, quotas +- **Migration Tracking**: Live migration progress bars +- **Telemetry Export**: Background Prometheus/OpenTelemetry push
\ No newline at end of file |