Open-IOV: initial public release

Open-IOV virtualizes NVIDIA GPUs so that unmodified guest drivers run with
real GPU functionality - CUDA first, 3D and video to follow - on unmodified
hosts, without a proprietary vGPU license and without patching any driver,
firmware or hypervisor.

Core principle: emulate the interface, pass through the work.

  kernel/     open-iov-vfio.ko - host VFIO mdev driver: virtual GPU device
              (PCI config space, BAR shadows, MSI-X), GSP boot emulation,
              GSP-RM message-queue/RPC emulation, memory model (GPU-VA to
              guest-PA discovery) and the semaphore-completion service that
              advances guest UVM progress.
  userspace/  libiov.so management API and examples.
  tools/      guest diagnostics, firmware analysis wrappers, RM probes.
  scripts/    stack manager, VM harness, build/load/test automation.
  config/     hardware/version profiles (the portability surface).
  docs/       documentation tree - start at docs/README.md.
  wiki/       long-form firmware and protocol deep dives.

Status: initialization works end to end for a stock 570.144 Linux guest on
Turing (TU104): driver probe, nvidia-smi, the full GSP-RM control battery,
object creation and the early CUDA/UVM start-up path. GPU work forwarding -
the step that makes CUDA kernels execute on real hardware - is the active
work item; see docs/04-implementation/01-known-knowns-and-unknowns.md.

License: BSD-3-Clause for the project; GPL-2.0 for kernel modules that link
GPL-only kernel interfaces. See LICENSE and docs/09-reference/legal-and-licensing.md.

1 files changed, 139 insertions, 0 deletions

diff --git a/docs/03-architecture/02-guest-stack-and-cudasw.md b/docs/03-architecture/02-guest-stack-and-cudasw.md
new file mode 100644
index 0000000..4ec614b
--- /dev/null
+++ b/docs/03-architecture/02-guest-stack-and-cudasw.md
@@ -0,0 +1,139 @@
+# The Guest Stack and CUDASW
+
+> What runs inside the guest, how it is layered, and what it expects from the
+> device. This is the "what must keep working, unmodified" document.
+
+## 1. Vocabulary: what "CUDASW" means here
+
+**CUDASW** is the family of NVIDIA *software* components that sit between an
+application and the GPU driver. In this project the term is used deliberately
+broadly, because our compatibility target is the whole stack, not one library:
+
+```
+application
+   │  CUDA API (runtime)
+libcudart.so ──────────────────────────────┐   CUDA runtime: device/stream/
+   │  CUDA driver API                     │   event management, kernel launch
+libcuda.so ────────────────────────────────┤   CUDA driver: contexts, modules,
+   │  ioctls                             │   memory, channel submission
+/dev/nvidia* ──────────────────────────────┘
+nvidia.ko (RM + UVM + modeset)
+```
+
+Adjacent components in the same family:
+
+* **libnvidia-ptxjitcompiler** — JIT compilation of PTX to SASS. (No GPU
+  interaction beyond driver interfaces; usually "just works" once the driver
+  initializes.)
+* **libnvidia-fatbinaryloader / nvrtc** — fatbinary loading.
+* **UVM** (`nvidia_uvm.ko`) — unified memory and GPU page-fault-driven memory
+  management; see [`04-uvm.md`](04-uvm.md).
+* **Graphics stack** — `libGLX_nvidia`, `libnvidia-glcore`, `libvulkan` ICD,
+  plus the kernel-side `nvidia_modeset`/`nvidia_drm` (roadmap).
+* **Video stack** — NVENC/NVDEC userspace + kernel interfaces (roadmap).
+
+## 2. Layering rules the emulation must respect
+
+1. **The kernel driver is the contract.** Everything above it (libcuda,
+   libcudart, libnvidia-*) talks to `nvidia.ko` through stable ioctls, memory
+   mappings and `/dev/nvidia*` nodes. If the kernel driver initializes and
+   behaves, user space follows.
+2. **User space is opaque to us.** We never intercept user-space APIs. All our
+   hooks are in the device model (host side) or in the *hardware interface* the
+   kernel driver sees (guest side).
+3. **Version skew is normal.** The driver version, the CUDA version and the
+   firmware version are three independent axes. Our compatibility target is
+   "the driver family that matches the emulated firmware profile", with
+   configuration-driven adaptation for others.
+
+## 3. Kernel driver anatomy (what the guest brings)
+
+`nvidia.ko` is a large, self-contained stack. The parts that matter to
+virtualization:
+
+| Component | Role | Why it matters to us |
+|---|---|---|
+| **RM core** | object model, handles, locks, device lifecycle | drives the GSP boot and the entire RPC/control surface |
+| **Resource server (`resserv`)** | allocates and tracks resources (objects, memory, channels) | the client-side half of the client/server RM split (see the GSP-RM doc) |
+| **Memory manager (`memmgr`)** | VRAM/sysmem objects, mappings, page tables | the guest expects the GSP to own page tables for its VA spaces |
+| **FIFO/runlist** | channels, TSGs, GPFIFO, USERD, doorbells | where the guest's work is submitted |
+| **GSP client** | message queues + RPC transport + event handling | the interface we emulate |
+| **UVM** | unified memory, page-fault handling, channel manager | starts early in CUDA initialization; extremely stateful |
+| **`nvidia_modeset` / `nvidia_drm`** | display/DRM | needed later for graphics |
+| **CE / GR HALs** | engine programming | needed to *execute* work (forwarded) |
+
+## 4. Driver initialization vs. runtime
+
+Understanding the split between these two phases is the key to debugging:
+
+**Initialization (boot/probe)** — deterministic, mostly one-shot:
+
+```
+PCI probe → identify GPU from config space and BAR0 registers
+   → boot the GSP (our emulated handshake)
+   → read static configuration (device info tables, FB/BUS info, VBIOS, clocks)
+   → create root client/device/subdevice objects
+   → initialize RM subsystems (interrupts, FIFO, memory, GMMU, PMU)
+   → expose /dev/nvidia* and register with the kernel
+```
+
+**Runtime (per-process, per-context)** — driven by user-space workloads:
+
+```
+open /dev/nvidia* → cuInit → UVM registration (per process)
+   → cuCtxCreate → object/allocations → memory maps
+   → channel creation + push buffers + semaphores
+   → kernel launches (GR) / copies (CE) / memops
+   → faults, completions, events
+```
+
+The initialization path is where compatibility is won or lost; the runtime
+path is where *execution* is won or lost. Open-IOV's current focus is the
+runtime path (forwarding), having already reached the end of initialization.
+
+## 5. What the guest expects from the device, per phase
+
+| Phase | Guest action | Device/firmware expectation |
+|---|---|---|
+| Probe | config space reads, BAR sizing, MSI-X capability | a plausible TU104-family PCIe GPU |
+| Boot | BAR0 register writes, WPR setup, firmware boot sequence | GSP boots, booter completes, "GSP init done" |
+| Config | RPC `GET_GSP_STATIC_INFO`, device tables, VBIOS reads | consistent, self-describing tables |
+| RM init | `GSP_RM_CONTROL` battery (FB/BUS/CE/INTR/GMMU/...) | correct shapes and plausible values |
+| Objects | `GSP_RM_ALLOC` (root, device, subdevice, vaspace, channel, CE objects) | success + correct handles/params |
+| Runtime | `GSP_RM_ALLOC`/`GSP_RM_CONTROL` for maps, channels; pushes | executed work, semaphores, faults |
+
+The field guide [`07-driver-lifecycle-field-guide.md`](07-driver-lifecycle-field-guide.md)
+lists the actual message/control IDs observed at each step.
+
+## 6. Why user-space-only interception does not work
+
+Two hard-won facts from this project:
+
+1. **`libcuda` issues raw `syscall` instructions**, so an `LD_PRELOAD` shim
+   around `ioctl()` sees nothing. If you need to observe CUDA's kernel calls,
+   use a `ptrace`-based tracer (PTRACE_SYSCALL + PTRACE_GET_SYSCALL_INFO), as
+   our tools do.
+2. **The CUDA driver talks to the *kernel driver*, not to the device.** All
+   device interaction happens in `nvidia.ko`. This is good news: it means the
+   emulation boundary is a small number of kernel-side interfaces (MMIO, DMA,
+   interrupts, and the GSP protocol) rather than a large user-space API
+   surface.
+
+## 7. Windows guests
+
+Windows brings its own `nvlddmkm.sys` driver, which uses the same underlying
+device/firmware model (GSP) but different host interfaces (WDDM, MSI-X
+semantics, ROM/VBIOS quirks). The emulation target is the same: satisfy the
+firmware interface, forward the work. Windows support is a roadmap item (see
+the portability matrix), and the design goal is that no Windows-specific code
+lives in the guest.
+
+## 8. Compatibility notes for versions
+
+* The guest's driver and the emulated firmware profile must agree on the RPC
+  message versions. The guest negotiates these at start-up
+  (`SET_GUEST_SYSTEM_INFO` carries the VGX/RPC version); our emulation answers
+  with a profile that matches the driver family in use.
+* CUDA versions ride on driver versions; the test matrix (see
+  [`../05-operations/01-performance-correctness-security.md`](../05-operations/01-performance-correctness-security.md))
+  tracks (driver, CUDA, firmware) triples.