Open-IOV: initial public release

Open-IOV virtualizes NVIDIA GPUs so that unmodified guest drivers run with
real GPU functionality - CUDA first, 3D and video to follow - on unmodified
hosts, without a proprietary vGPU license and without patching any driver,
firmware or hypervisor.

Core principle: emulate the interface, pass through the work.

  kernel/     open-iov-vfio.ko - host VFIO mdev driver: virtual GPU device
              (PCI config space, BAR shadows, MSI-X), GSP boot emulation,
              GSP-RM message-queue/RPC emulation, memory model (GPU-VA to
              guest-PA discovery) and the semaphore-completion service that
              advances guest UVM progress.
  userspace/  libiov.so management API and examples.
  tools/      guest diagnostics, firmware analysis wrappers, RM probes.
  scripts/    stack manager, VM harness, build/load/test automation.
  config/     hardware/version profiles (the portability surface).
  docs/       documentation tree - start at docs/README.md.
  wiki/       long-form firmware and protocol deep dives.

Status: initialization works end to end for a stock 570.144 Linux guest on
Turing (TU104): driver probe, nvidia-smi, the full GSP-RM control battery,
object creation and the early CUDA/UVM start-up path. GPU work forwarding -
the step that makes CUDA kernels execute on real hardware - is the active
work item; see docs/04-implementation/01-known-knowns-and-unknowns.md.

License: BSD-3-Clause for the project; GPL-2.0 for kernel modules that link
GPL-only kernel interfaces. See LICENSE and docs/09-reference/legal-and-licensing.md.

1 files changed, 183 insertions, 0 deletions

diff --git a/docs/03-architecture/01-system-overview.md b/docs/03-architecture/01-system-overview.md
new file mode 100644
index 0000000..ed9b24b
--- /dev/null
+++ b/docs/03-architecture/01-system-overview.md
@@ -0,0 +1,183 @@
+# System Overview
+
+> The one document to read if you want to understand *how Open-IOV works* before
+> diving into any subsystem.
+
+## 1. The core architectural principle
+
+> **Emulate the interface. Pass through the work.**
+
+A virtual GPU is a *mediated passthrough* device. The guest must believe it
+owns a real NVIDIA GPU with a real GPU System Processor (GSP) running NVIDIA's
+firmware. The host must arrange for the guest's work to execute on real
+hardware, because emulating compute in software is 100–1000× too slow to be
+useful.
+
+Everything therefore falls into two buckets:
+
+```
+┌─────────────────────────────────────────────────────────────────────────┐
+│  EMULATE  (software; guest expects virtual hardware)                    │
+│    • PCI configuration space, capability structures, power management   │
+│    • BAR0 register interface (shadow, not the physical GPU's BAR0)      │
+│    • BAR1 VRAM aperture (guest-visible video memory)                    │
+│    • Instance/PRAMIN window                                             │
+│    • GSP firmware boot handshake (FWSEC → booter → RISCV → boot complete)│
+│    • GSP-RM message queues, RPC transport, resource-manager controls    │
+│    • Event/notification channels                                        │
+│    • MSI-X injection (via eventfd, from the hypervisor's perspective)   │
+└─────────────────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────────────────┐
+│  PASS THROUGH  (real hardware; for correctness and performance)         │
+│    • GPU memory: guest VRAM objects backed by real VRAM (goal)          │
+│    • Page tables / GMMU state used by the GPU (goal)                    │
+│    • Channel execution: the guest's push buffers run on real channels   │
+│    • Compute (CUDA) and, later, 3D/video work                           │
+│    • DMA, completion and fault interrupts relayed back to the guest     │
+└─────────────────────────────────────────────────────────────────────────┘
+```
+
+The line between the two is the **interface boundary**, and it moves as the
+project matures: early work emulates more (everything the guest can observe),
+later work forwards more (everything that must be fast).
+
+## 2. Layer map
+
+```
+GUEST
+  application ── CUDA runtime (libcudart) ─┐
+  application ── CUDA driver  (libcuda)   ─┤  userspace: CUDASW stack
+  OpenGL/Vulkan ── libGLX/libvulkan ──────┘
+        │  ioctls / mmaps
+  nvidia.ko (unmodified)  ── RM core, resource server, UVM, modeset/drm
+        │  MMIO + DMA + MSI-X   (a virtual PCIe device)
+════════╪════════════════════════════════════════════════════════════════
+HYPERVISOR (KVM/Xen/...)  VFIO mdev transport + IOMMU isolation
+        │
+HOST MODULE  open-iov-vfio.ko
+  ├─ virtual device: config space, BAR0/BAR1/BAR3 windows, MSI-X
+  ├─ emulation core: GSP boot + GSP-RM (RPC msgq, controls, objects, events)
+  ├─ memory model: guest RAM access, VA↔PA maps, VRAM backing
+  └─ work path: channel/push forwarding to the physical GPU      [in progress]
+        │
+HOST DRIVER  nvidia.ko (unmodified) and/or direct GPU interfaces
+        │
+PHYSICAL GPU
+```
+
+## 3. Host module architecture
+
+`open-iov-vfio.ko` is a **VFIO mediated-device (mdev) driver**. It registers a
+parent on the physical GPU and exposes virtual GPU types; each `mdev` instance
+is one virtual GPU for one VM.
+
+| Subsystem | Files (illustrative) | Responsibility |
+|---|---|---|
+| mdev/VFIO registration | `kernel/main.c` | parent/type registration, VFIO device, region and IRQ setup |
+| Config space | `kernel/main.c`, `kernel/bar_init.c` | identify/command/status, BARs, PCIe caps, MSI-X caps, VF windows |
+| BAR0 shadow | `kernel/bar_emulation.c` | 16 MB emulated register space; register reads/writes the guest driver depends on |
+| VRAM aperture (BAR1) | `kernel/main.c`, `memory.c` | 256 MB guest-visible video memory (host RAM today; real VRAM later) |
+| Instance memory (BAR3) | `kernel/bar_emulation.c` | instance/PRAMIN window behaviour |
+| GSP emulation thread | `kernel/gsp_emulation.c` | boot stage completion, doorbell polling, MSI-X injection |
+| GSP-RM emulation | `kernel/gsp_rm_emulation.c` | message queues, RPC dispatch, resource-manager controls, objects |
+| RPC handlers (firmware-derived) | `kernel/gsp_rpc_client.c` | behaviours decompiled from the GSP firmware |
+| Firmware analysis (runtime) | `kernel/firmware_analyzer.c` | CRC/SHA signatures, pattern matching, version identification |
+| Version dispatch | `kernel/dynamic_rpc_dispatcher.c`, `kernel/version/` | version-aware handler selection |
+| Hardware abstraction | `kernel/architecture/` | per-architecture constants and behaviours |
+| Userspace library | `userspace/lib/libvgpu.c` | management API (`vgpu_*`), diagnostics |
+
+**Data flow for an MMIO access from the guest:**
+
+```
+guest nvidia.ko writes BAR0 offset X
+   → KVM exit → QEMU → VFIO region write into the mdev
+   → nv_vgpu_vfio_dev_write()  (or directly into the BAR0 shadow if mmap'd)
+   → BAR0 shadow state machine (doorbells, VF registers, notify)
+   → may wake the GSP emulation thread
+```
+
+**Data flow for an RPC from the guest to "firmware":**
+
+```
+guest writes a command-queue element into a guest-memory msgq and rings the
+GSP doorbell (BAR0)
+   → emulation thread reads the queue through the hypervisor's guest-RAM
+     accessor (vfio_dma_rw), parses the element, dispatches the RPC
+   → handler produces a reply element in the status queue
+   → emulation injects an MSI-X interrupt to the guest
+```
+
+## 4. Guest stack summary
+
+* `nvidia.ko` is the **unmodified** NVIDIA kernel driver. It contains the
+  resource manager (RM) core, the resource server (`resserv`), the memory
+  manager, the FIFO/runlist code, the GPU System Processor client path, and
+  UVM (the unified memory/virtual-memory module, `nvidia_uvm.ko`).
+* User space consists of the CUDA driver (`libcuda.so`), the CUDA runtime
+  (`libcudart.so`), plus graphics/video libraries. The family of NVIDIA
+  *software* components that mediate between applications and the driver is
+  referred to in this project as **CUDASW**.
+* The guest expects to program a real GPU: it writes registers, allocates
+  objects, creates channels, submits push buffers and waits on semaphores —
+  exactly as on bare metal.
+
+Detailed subsystem documents:
+
+* [`02-guest-stack-and-cudasw.md`](02-guest-stack-and-cudasw.md)
+* [`03-gsp-and-gsp-rm.md`](03-gsp-and-gsp-rm.md)
+* [`04-uvm.md`](04-uvm.md)
+* [`05-gmmu-amap-hal-gr.md`](05-gmmu-amap-hal-gr.md)
+* [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md)
+* [`07-driver-lifecycle-field-guide.md`](07-driver-lifecycle-field-guide.md)
+
+## 5. Why a GSP emulation is possible at all
+
+The guest driver never talks to GPU silicon directly for management: since the
+Turing generation, NVIDIA GPUs boot a **RISC-V firmware (the GSP)** that hosts
+the Resource Manager. The driver's normal path is to talk to that firmware over
+shared memory queues using a small RPC protocol.
+
+That protocol is:
+
+* **small** (a few dozen message types at the levels we need),
+* **stable** (message layouts are versioned and compiled into the driver), and
+* **observable** (everything arrives in guest memory we can read).
+
+So instead of emulating a GPU, we emulate **the firmware's interface**. The
+guest driver is satisfied because the protocol is satisfied; the guest's
+*work* is then given to real hardware.
+
+The firmware itself we study directly: it is a 28 MB RV64IMAC ELF that we have
+decompiled and mapped (see `wiki/GSP-570.144W-Firmware-Analysis.md` and
+`wiki/Address-Reference.md`).
+
+## 6. What runs today
+
+1. Guest boots, enumerates the virtual PCIe device, binds `nvidia.ko`.
+2. The driver performs the GSP boot handshake with our emulation and receives
+   plausible firmware identity/configuration.
+3. The driver sets up the RPC message queues (guest sysmem), negotiates
+   versions, and performs the resource-manager initialization exchanges.
+4. Object creation (clients, devices, subdevices), capability queries,
+   vaspaces, channels, CE caps, GMMU static info — every control the driver
+   probes — is answered with the correct shape and values.
+5. `nvidia-smi` works; CUDA user-space starts; UVM begins channel-manager and
+   page-tree initialization.
+
+## 7. What is next (and why)
+
+CUDA kernel execution requires the guest's *work* to run on the physical GPU.
+That means the host module must:
+
+1. know the memory the guest's GPU work refers to (the VA↔PA model),
+2. reproduce or forward the channel/page-table state that the guest expects the
+   GSP to own, and
+3. submit the guest's compute work to real engine(s) and relay completions,
+   faults and semaphores back.
+
+The design for this, including what is known and what remains to be learned, is
+in:
+[`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md)
+and
+[`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md).