Open-IOV: initial public release

Open-IOV virtualizes NVIDIA GPUs so that unmodified guest drivers run with
real GPU functionality - CUDA first, 3D and video to follow - on unmodified
hosts, without a proprietary vGPU license and without patching any driver,
firmware or hypervisor.

Core principle: emulate the interface, pass through the work.

  kernel/     open-iov-vfio.ko - host VFIO mdev driver: virtual GPU device
              (PCI config space, BAR shadows, MSI-X), GSP boot emulation,
              GSP-RM message-queue/RPC emulation, memory model (GPU-VA to
              guest-PA discovery) and the semaphore-completion service that
              advances guest UVM progress.
  userspace/  libiov.so management API and examples.
  tools/      guest diagnostics, firmware analysis wrappers, RM probes.
  scripts/    stack manager, VM harness, build/load/test automation.
  config/     hardware/version profiles (the portability surface).
  docs/       documentation tree - start at docs/README.md.
  wiki/       long-form firmware and protocol deep dives.

Status: initialization works end to end for a stock 570.144 Linux guest on
Turing (TU104): driver probe, nvidia-smi, the full GSP-RM control battery,
object creation and the early CUDA/UVM start-up path. GPU work forwarding -
the step that makes CUDA kernels execute on real hardware - is the active
work item; see docs/04-implementation/01-known-knowns-and-unknowns.md.

License: BSD-3-Clause for the project; GPL-2.0 for kernel modules that link
GPL-only kernel interfaces. See LICENSE and docs/09-reference/legal-and-licensing.md.

2 files changed, 297 insertions, 0 deletions

diff --git a/docs/05-operations/01-performance-correctness-security.md b/docs/05-operations/01-performance-correctness-security.md
new file mode 100644
index 0000000..c4d8153
--- /dev/null
+++ b/docs/05-operations/01-performance-correctness-security.md
@@ -0,0 +1,140 @@
+# Performance, Correctness, and Security Evaluation
+
+> How we measure whether Open-IOV is any good — and how auditors can verify the
+> claims. Numbers without methodology are not accepted here.
+
+## 1. The evaluation ladder
+
+Every platform (hardware + driver + hypervisor + guest) is evaluated through
+the same five-rung ladder. Each rung is a gate: you do not claim the next one
+until the previous one is green and reproducible.
+
+```
+Rung 1  INITIALIZATION    guest driver probes; nvidia-smi; device enumeration
+Rung 2  RUNTIME START     cuInit; context; allocation; UVM registration
+Rung 3  EXECUTION         kernels produce correct results; copies verified
+Rung 4  SUSTAINED         workloads run for hours; faults/errors recoverable
+Rung 5  MULTI-TENANT      N VMs, isolation + fairness + no cross-talk
+```
+
+## 2. Correctness
+
+### 2.1 Test corpus
+
+| Layer | Tests |
+|---|---|
+| Device | PCI config validation, BAR sizing, MSI-X semantics, power-management stubs |
+| Driver init | guest `nvidia.ko` probe, `nvidia-smi` full query, `deviceQuery` |
+| CUDA | `vectorAdd`, reduction, shared-memory kernels, memcpy patterns, streams/events, error injection |
+| UVM | oversubscribed memory (fault path), unified memory migration, concurrency |
+| Graphics (later) | GLXInfo/VulkanInfo, rendering correctness screenshots |
+| Stability | start/stop cycles ×100, VM kill during work, guest reboot in flight |
+
+### 2.2 Verification methods
+
+1. **Golden results**: kernel outputs compared against native execution on the
+   same GPU, byte-for-byte, for deterministic kernels.
+2. **Guest self-checks**: the guest's own validation paths are the strongest
+   signal (UVM asserts, RM validation, CUDA runtime self-tests). Every guest
+   diagnostic that fires is a test failure for us, even if the workload
+   continues.
+3. **Differential runs**: same workload on (a) native, (b) our device — diff
+   observables, not just outputs: semaphore value sequences, PUT/GET progress,
+   interrupt counts.
+4. **Negative testing**: inject malformed device behaviour in a lab profile and
+   require the guest driver to recover exactly as it would on hardware with a
+   transient error.
+
+### 2.3 Regression gates
+
+`make test` runs the host suite. The CI ladder runs the guest suite on a
+scheduled cadence and on every release. A rung regression is a release blocker;
+the handoff documents carry the current rung.
+
+## 3. Performance
+
+### 3.1 Metrics
+
+| Metric | Definition | Why it matters |
+|---|---|---|
+| T_init | VM boot → `cuInit` returns | user-visible startup |
+| RPC latency p50/p99 | request write → reply visible in status queue | init and management overhead |
+| Push overhead | guest submission → engine start (forwarded) | steady-state latency |
+| Semaphore completion latency | release written → tracker observes | per-operation sync cost |
+| Copy bandwidth | H2D/D2H/D2D, large and small | data-movement quality |
+| Kernel throughput | kernels/second for a fixed workload | launch overhead |
+| Multi-tenant fairness | per-tenant progress under contention | isolation quality |
+
+### 3.2 Method
+
+* **Same-guest baselines**: measure native GPU performance *in the same VM with
+  a passthrough device* where possible; otherwise on the same hardware with the
+  same driver. Comparisons across hosts are noise.
+* **Warm and cold**: report both; report variance; report the number of runs.
+* **Instrument everything**: the emulation timestamps requests and replies
+  (tracepoints, `trace_printk`, or counters) so overhead is attributable to a
+  component, not to a mystery.
+* **Publish the harness**: the benchmark scripts live in `tools/`, and a
+  result is not comparable unless produced by them (or an equivalent,
+  documented harness).
+
+### 3.3 Targets
+
+* Initialization: within 2× of native, same guest.
+* Synchronization-heavy operations: within an agreed constant factor, published
+  per operation class.
+* Bulk copies and sustained compute: as close to native as the forwarding path
+  allows; the forwarding design (proxy vs direct) is decided by measurement.
+
+## 4. Security and isolation
+
+### 4.1 Threat model
+
+| Adversary | Capability | Boundary that must hold |
+|---|---|---|
+| Guest VM (tenant) | full control of its own driver, MMIO, DMA | cannot touch host or other tenants' memory/state |
+| Malicious guest driver | arbitrary device programming | emulation must validate everything, fail closed |
+| Host user (unprivileged) | can access `/dev/*` permissions given | no privilege escalation via our interfaces |
+| Adjacent tenant | shares the GPU | cannot observe timing/state/interrupts beyond policy |
+| Supply chain | modified driver/firmware | measured boot (TrenchBoot) verifies the platform |
+
+### 4.2 Enforcement
+
+* **Memory isolation is the platform IOMMU's job, and we never weaken it.**
+  The device performs DMA only through the hypervisor-provided DMA interface
+  for its own VM. No global mappings, no "identity for convenience".
+* **Per-instance state**: every mdev has its own BAR shadows, queues,
+  interrupt state and emulation context. No shared mutable state between
+  tenants in the host module.
+* **Fail closed**: if a required isolation property cannot be established
+  (IOMMU absent, DMA mapping failure, inconsistent state), the device refuses
+  to start and says why in `dmesg`.
+* **Interrupts**: per-instance vectors; masked interrupts are queued in the
+  emulated PBA, not dropped into another tenant's vector.
+* **Debug surfaces**: destructive debug parameters are opt-in, documented, and
+  excluded from hardened builds. Reads are side-effect-free.
+* **No secrets in the module**: nothing about one tenant's workload is stored
+  where another tenant could reach it.
+
+### 4.3 Auditing
+
+* The security-relevant code paths are small and enumerated in this document
+  tree: device registration, DMA access, interrupt injection, memory
+  translation, and the debug surface. Auditors should be able to enumerate
+  them mechanically (`grep` for the gate points).
+* Each release produces a checklist: threat model reviewed, fail-closed paths
+  exercised by tests, debug surface verified gated, dependency licenses
+  unchanged.
+* We explicitly invite adversarial review — including of the emulation's
+  response to malformed guest input.
+
+## 5. Reporting
+
+Each release publishes:
+
+1. **Rung achieved** per platform, with logs.
+2. **Performance table** (metrics above) with methodology and variance.
+3. **Known issues**, linked to the known-unknowns document.
+4. **Security checklist** status.
+
+This is the only accepted form of a claim in this project.
diff --git a/docs/05-operations/02-debugging-and-maintenance-workflows.md b/docs/05-operations/02-debugging-and-maintenance-workflows.md
new file mode 100644
index 0000000..bc38a4e
--- /dev/null
+++ b/docs/05-operations/02-debugging-and-maintenance-workflows.md
@@ -0,0 +1,157 @@
+# Debugging and Maintenance Workflows
+
+> The practical book: how to see what is happening, how to find the cause, and
+> how to add a new firmware, driver, hypervisor or hardware version without
+> breaking anything.
+
+## 1. The observability model
+
+Three layers, in order of usefulness:
+
+1. **Guest diagnostics** — the guest driver is the best oracle. It prints
+   precisely what it expected and what it got (e.g. UVM's "Tracker entry for
+   value N … channel_init()"). Always start here.
+2. **Host module trace** — every RPC and every emulation decision is logged
+   with the `gsp-resman-vfio:` prefix. A full CUDA run produces a few hundred
+   lines that reconstruct the object tree and the control battery.
+3. **Memory observation** — when the problem is in memory state (mappings,
+   semaphores, pushes), the module's debug parameters read and write guest
+   physical memory directly.
+
+## 2. Standard captures for a bug report
+
+```bash
+# Host
+sudo dmesg -T | grep -iE 'gsp-resman-vfio|gsp_|nvidia|vfio|mdev' > host.log
+./scripts/stack_manager.sh status > status.log
+# Guest
+sudo dmesg > guest-dmesg.log                 # include full dmesg
+sudo lspci -vvv -d 10de: > guest-lspci.log   # config space of the virtual GPU
+cat /proc/cmdline > guest-cmdline.log
+nvidia-smi -q > nvidia-smi-q.log
+```
+
+Plus: the emulated driver profile (GPU type, firmware image hash, driver
+version) and the exact command sequence used. `tools/guest-diag/push_guest_tools.sh`
+collects some of this automatically.
+
+## 3. Live debugging toolbox
+
+### 3.1 Kernel logs
+
+```bash
+sudo dmesg -w | grep -iE 'gsp-resman-vfio'          # live emulation trace
+sudo dmesg -T | grep gsp-resman-vfio | tail -100    # recent
+# capture to a file when the ring buffer is small and traffic is heavy:
+sudo stdbuf -oL dmesg -w > run.log
+```
+
+### 3.2 Module debug parameters
+
+See the host-module doc for the full list. Canonical examples:
+
+```bash
+# Find where a value lives in guest RAM (e.g. a semaphore VA):
+echo 0x121000000 | sudo tee /sys/module/open_iov_vfio/parameters/scan_guest_ram
+
+# Find a decoded method sequence in guest RAM:
+echo 0x1,0x21000000,0x1 | sudo tee /sys/module/open_iov_vfio/parameters/scan_seq
+
+# Write a value to a guest physical address (completing a semaphore release):
+echo 0x120299010,0x1 | sudo tee /sys/module/open_iov_vfio/parameters/poke_guest
+```
+
+**Caution:** all guest-RAM operations run in a kernel thread context by design;
+calling them from a normal process context fails with `-EFAULT` (the
+hypervisor's DMA interface requires either the mapping's `mm` or no `mm` at
+all). This is not a bug.
+
+### 3.3 QEMU gdbstub (guest virtual memory)
+
+The reference QEMU configuration exposes a gdbstub (`-gdb tcp::1234`). It can
+read guest *virtual* memory through the vCPU's page tables and write to it —
+the single most powerful trick we have when the guest's own kernel address
+matters:
+
+```bash
+gdb -batch -ex 'target remote :1234' \
+    -ex 'x/8wx 0xffff...' -ex 'set *(unsigned int*)0xffff... = 1'
+```
+
+Workflow that made the memory model tractable:
+
+1. Find the guest's `page_offset_base` (direct-map base) from
+   `/proc/kallsyms` in the guest.
+2. Convert a kernel CPU VA to a physical address: `PA = VA - page_offset_base`.
+3. Use `poke_guest` to write the physical address from the host module, or use
+   the gdbstub to write the virtual address directly.
+
+### 3.4 Guest-side tracing
+
+* `ptrace` tracer (`tools/guest-diag/ptrace_ioctl.c`): libcuda issues raw
+  syscalls, so `LD_PRELOAD` shims see nothing; PTRACE_SYSCALL with
+  `PTRACE_GET_SYSCALL_INFO` does.
+* UVM debug: `echo 1 > /sys/module/nvidia_uvm/parameters/uvm_debug_prints`
+  (messages appear as `nvidia-uvm:`).
+* kprobes do not work on NVIDIA text (`notrace`); do not waste time on them.
+
+### 3.5 Failure decision trees
+
+When a failure is understood, write the decision tree into this document tree
+— the tree is what makes the next failure cheap. The field guide's tables are
+the result of that discipline so far.
+
+## 4. Workflows for new inputs
+
+### 4.1 New firmware version
+
+1. Add the image under `firmware/<SERIES>-GSP/<VERSION>/` (tracked by LFS).
+2. Compute signatures (CRC32/SHA256) with the firmware analyzer.
+3. Add a firmware profile entry (addresses/signatures) so runtime analysis can
+   identify it.
+4. Run the driver against a guest using that firmware's matching driver; diff
+   the RPC/control inventory against the baseline.
+5. Add/adjust handlers only where the diff requires it; re-run the ladder.
+
+### 4.2 New guest driver version
+
+1. Install the stock driver in a guest (no patches).
+2. Capture the full init trace and diff function/control IDs against the
+   previous version.
+3. For each difference: decode against our decompiled study notes for that version
+   (`g_rpc-structures.h`, `rpc_global_enums.h`, control headers).
+4. Add a version-keyed handler or a capability table entry.
+5. Re-run rungs 1–2; then the CUDA corpus.
+
+### 4.3 New hardware generation
+
+1. Add a hardware profile (PCI IDs, BAR layout, memory geometry, engine list).
+2. Run the ladder on that hardware with the closest driver family.
+3. Fix differences by configuration first; only add code when an interface
+   genuinely differs.
+4. Publish the profile and the test evidence.
+
+### 4.4 New hypervisor
+
+1. Implement the transport boundary (device registration, guest-memory access,
+   interrupt injection) — the boundary is small by design.
+2. Re-run the ladder; the emulation core must not need changes.
+3. Document any hypervisor-specific quirk in the debugging docs.
+
+## 5. Crash and hang recovery
+
+| Situation | Action |
+|---|---|
+| Module stuck unloading | `stack_manager.sh cleanup-crash`; reboot if it persists (request approval per project policy) |
+| VM wedged, guest filesystem dirty | stop the VM gracefully (`scripts/vgpu_vm.py stop`), fsck via `qemu-nbd` only if needed |
+| Guest driver wedged (D-state processes) | reboot the guest; capture dmesg first |
+| Kernel oops in the host module | capture the full oops + `dmesg` before reloading |
+
+## 6. Release engineering
+
+* Versions are tagged; each tag carries the rung/matrix table and the
+  performance table for the platforms tested.
+* Configuration profiles are versioned with the code; a profile bump is a
+  reviewed change with evidence.
+* The docs in this tree are part of the release: a release that changes
+  behaviour without updating them is incomplete.