Open-IOV: initial public release

Open-IOV virtualizes NVIDIA GPUs so that unmodified guest drivers run with
real GPU functionality - CUDA first, 3D and video to follow - on unmodified
hosts, without a proprietary vGPU license and without patching any driver,
firmware or hypervisor.

Core principle: emulate the interface, pass through the work.

  kernel/     open-iov-vfio.ko - host VFIO mdev driver: virtual GPU device
              (PCI config space, BAR shadows, MSI-X), GSP boot emulation,
              GSP-RM message-queue/RPC emulation, memory model (GPU-VA to
              guest-PA discovery) and the semaphore-completion service that
              advances guest UVM progress.
  userspace/  libiov.so management API and examples.
  tools/      guest diagnostics, firmware analysis wrappers, RM probes.
  scripts/    stack manager, VM harness, build/load/test automation.
  config/     hardware/version profiles (the portability surface).
  docs/       documentation tree - start at docs/README.md.
  wiki/       long-form firmware and protocol deep dives.

Status: initialization works end to end for a stock 570.144 Linux guest on
Turing (TU104): driver probe, nvidia-smi, the full GSP-RM control battery,
object creation and the early CUDA/UVM start-up path. GPU work forwarding -
the step that makes CUDA kernels execute on real hardware - is the active
work item; see docs/04-implementation/01-known-knowns-and-unknowns.md.

License: BSD-3-Clause for the project; GPL-2.0 for kernel modules that link
GPL-only kernel interfaces. See LICENSE and docs/09-reference/legal-and-licensing.md.

2 files changed, 456 insertions, 0 deletions

diff --git a/docs/04-implementation/01-known-knowns-and-unknowns.md b/docs/04-implementation/01-known-knowns-and-unknowns.md
new file mode 100644
index 0000000..0564ffc
--- /dev/null
+++ b/docs/04-implementation/01-known-knowns-and-unknowns.md
@@ -0,0 +1,300 @@
+# Known-Knowns and Known-Unknowns
+
+> The most important document for planning work. It states, without
+> flattery, what we know, what we know we do not know, and what we do not even
+> know to ask yet — plus how each gap gets closed.
+
+## 0. How to read this
+
+| Category | Meaning | Our approach |
+|---|---|---|
+| **Known-knowns** | verified facts, with evidence in this repo | rely on them; keep them tested |
+| **Known-unknowns** | gaps we have identified, with a concrete plan to close | schedule them, one by one |
+| **Unknown-unknowns** | gaps we cannot yet name | discover them by systematic traversal + invariants + instrumentation |
+
+Anything listed here as a known-unknown has an owner-visible plan; anything
+listed as an unknown-unknown has a *method* by which we will discover it.
+
+---
+
+## 1. Known-knowns (verified)
+
+### 1.1 Device and platform
+
+* A stock 570.144 Linux guest driver probes and initializes against our
+  emulated TU104 device; `nvidia-smi` works.
+* The same guest **boots unmodified**: no guest code, no guest module
+  parameters are required for the initialization path.
+* The host is unmodified in the standalone configuration (distribution NVIDIA
+  driver manages the physical GPU); Open-IOV loads as an independent module.
+* Multi-VM operation multiplies the same emulated device; each mdev is
+  independent.
+
+### 1.2 Protocol and firmware
+
+* The GSP boot state machine is emulatable by completing observable stages.
+* The message queue transport is understood in detail: element header layout,
+  page-table translation of non-contiguous queue buffers, sequence matching,
+  lengths, and quirks (`length` excludes variable payloads).
+* The RPC surface used during driver initialization is fully enumerated for
+  570.144 (§9 of the field guide gives the ID table).
+* Firmware images are analyzable: the 570.144 image is decompiled
+  (~130 functions of interest mapped to C) and its dispatcher/jump tables are
+  located.
+* Version negotiation exists and is honored (the guest tells us its VGX/RPC
+  version; we answer with a matching profile).
+
+### 1.3 Memory and execution semantics
+
+* GPU VA → guest PA translation is recoverable **without guest cooperation**
+  by reading the guest's own bookkeeping (VA and CPU-VA pairs recorded in
+  UVM/RM structures; direct-map relationship for CPU VAs).
+* Guest physical memory is readable and writable from the host module through
+  the hypervisor's DMA interface, from a kernel thread context.
+* UVM's pushes are decodable (incrementing-method headers, engine classes,
+  `SET_SEMAPHORE_A/B/PAYLOAD`, `LAUNCH_DMA`), and completing their semaphore
+  releases advances UVM through channel manager creation and page-tree
+  initialization.
+* The client's channel memory descriptors carry *physical* addresses we can
+  trust (instance/userd/ramfc/method buffers).
+* The client's page-table contract for server-owned ranges is explicit
+  (`COPY_SERVER_RESERVED_PDES`) — and its leaf levels are *empty*, i.e. ours to
+  populate.
+
+### 1.4 Operational
+
+* Module reload, mdev create/destroy, VM start/stop, and guest reboot cycles
+  are stable when done in the documented order.
+* Every layer of the emulation is instrumented and diagnosable from `dmesg`
+  plus a small set of module parameters.
+
+---
+
+## 2. Known-unknowns (named gaps with plans)
+
+Each entry: **what we do not know**, **why it matters**, **how we will learn
+it**, **rough cost**.
+
+### 2.1 Execution forwarding
+
+**Unknown:** the correct, isolated way to run a guest's engine work on the
+physical GPU: which real scheduling primitive to use per engine, how to
+translate guest pushes to real submissions, and how to relay completion and
+fault semantics.
+
+**Why:** this is *the* blocker for CUDA kernel execution, and later for
+graphics.
+
+**Plan:** evaluate the three backends (host RM/GSP proxy, direct engine
+submission, hybrid) against a scorecard: correctness, isolation, latency,
+maintenance cost across driver versions. Prototype the proxy first (it reuses
+the unmodified host driver's knowledge), measure latency, then decide whether a
+direct path is required. Deliverable: a forwarded `vectorAdd` with verified
+results.
+
+**Cost:** weeks, not days; the largest single work item in the project.
+
+### 2.2 Server-owned page tables
+
+**Unknown:** the exact division of labour for page tables after
+`COPY_SERVER_RESERVED_PDES`: which ranges the client will touch, when it
+expects `RELEASE_ENTRIES`/`RESERVE_ENTRIES`, and how invalidations are
+communicated.
+
+**Why:** mappings that only we can create must exist for the guest's own
+memory to be reachable by engines.
+
+**Plan:** instrument all vaspace controls (add handlers that log their
+parameters), then implement `RESERVE_ENTRIES`/`RELEASE_ENTRIES`/mirroring with
+a test harness that maps memory and verifies the guest's expectations by
+running its own validation paths (UVM page-tree init exercises this).
+
+**Cost:** days to weeks.
+
+### 2.3 Fault semantics
+
+**Unknown:** fault buffer formats and replay/cancel semantics as the *guest*
+expects them in our virtualized setting (we know the buffer sizes and that
+UVM programs them, but not yet the end-to-end contract under forwarding).
+
+**Why:** GPU page faults happen constantly in real CUDA workloads (UVM
+oversubscription, migrations); correct handling is required for robustness.
+
+**Plan:** reproduce faults deliberately (map a VA to nothing, touch it from a
+forwarded kernel), capture both sides, implement replay; validate with
+`uvm_fault`-heavy tests (e.g. oversubscribed memory).
+
+**Cost:** weeks.
+
+### 2.4 Version matrix beyond 570.144
+
+**Unknown:** which parts of the observed protocol drift between driver
+families (550/560/580/610+), how much is negotiated, and how much must be
+configuration.
+
+**Why:** durability is a core promise; users will run other versions
+immediately.
+
+**Plan:** run the same init traces across 2–3 driver families in guests,
+diff the message/control inventories, and refactor handlers to be
+version-keyed where they differ. Deliverable: a capability matrix and a
+documented "add a version" workflow.
+
+**Cost:** days per version, once the workflow exists.
+
+### 2.5 Windows guest specifics
+
+**Unknown:** the WDDM-facing differences (ROM/VBIOS expectations, MSI-X
+semantics, power management, display topology) that the Windows driver
+requires beyond the Linux path.
+
+**Why:** Windows CUDA/3D users are a first-class audience.
+
+**Plan:** bring up a Windows guest against the same device, capture the driver
+logs (Windows ETW + our host traces), and add configuration for discovered
+requirements.
+
+**Cost:** weeks.
+
+### 2.6 Non-KVM transports
+
+**Unknown:** what differs for Xen/Argo/OpenXT/Qubes OS Air: IRQ delivery
+mechanisms, memory pinning, device-model integration points.
+
+**Why:** the security-focused platforms are a major part of the project's
+reason for existing.
+
+**Plan:** abstract the transport boundary (IRQ injection, guest-memory access,
+device registration) behind a small interface; implement a Xen backend against
+it; test in a Xen guest.
+
+**Cost:** weeks.
+
+### 2.7 aarch64 hosts and Grace/GB10
+
+**Unknown:** host-side differences (page sizes, cache attributes, SMMU
+behaviour, power domains) and any firmware/ROM differences in the guest path.
+
+**Why:** Grace-Hopper/Grace-Blackwell/GB10 (DGX Spark, Surface-class laptops)
+are explicit targets.
+
+**Plan:** make the emulation architecture-independent by construction
+(no x86-specific assumptions in the memory model), then bring up on an
+aarch64 host with the same test ladder.
+
+**Cost:** weeks per platform.
+
+### 2.8 Multi-tenant performance policy
+
+**Unknown:** best scheduling policy for sharing one GPU across VM tenants
+(time slice, MIG partition, hybrid) and its fairness/isolation properties.
+
+**Why:** multi-tenancy is a core value proposition.
+
+**Plan:** build measurement first (per-tenant attribution), then implement
+policy as configuration, then audit isolation.
+
+**Cost:** months (it is a research-y area).
+
+### 2.9 Tooling maturity
+
+**Unknown:** how much of the debug tooling (memory scanners, pokes, push
+decoders) can be productized without exposing unsafe interfaces.
+
+**Why:** the tooling is currently how we make progress; it must not become a
+security liability.
+
+**Plan:** move tooling behind a debug-only build flag/document its threat
+model; keep safe, read-only diagnostics in production builds.
+
+**Cost:** days.
+
+---
+
+## 3. Unknown-unknowns (and how we find them)
+
+Unknown-unknowns are discovered by *systematic traversal* and *invariants*,
+not by luck. Our methods:
+
+### 3.1 Traversal by construction
+
+* **Enumerate every interface the guest touches.** For each: prove we answer
+  it or forward it. Any interface we have not seen is a discovery task, not a
+  bug.
+* **Drive the guest further, always.** Each new guest-side milestone (probe →
+  nvidia-smi → cuInit → context → allocation → kernel) is a traversal step
+  that *forces* new interfaces to appear in the trace.
+* **Differential testing across versions/hardware.** The same traversal on a
+  second driver version or GPU surfaces requirements that are invisible on
+  the first.
+
+### 3.2 Invariants that trip on unknowns
+
+We rely on the guest's own self-checks as an oracle:
+
+* UVM asserts on semaphore jumps, bad page tables, impossible capabilities.
+* The RM validates structures and will fault on inconsistencies.
+* Kernel drivers log "unexpected" states loudly.
+
+Every guest assert/log is a discovered unknown-unknown if we do not already
+have it in this document.
+
+### 3.3 Structural instrumentation
+
+* **Trace-first policy:** no handler is written without logging the request
+  *and* the reply in a reconstructible form.
+* **Reconstruct from dmesg:** the logs must be sufficient to rebuild the
+  object tree, memory map and push stream offline. This is how protocol drift
+  is detected early, before it becomes a crash.
+
+### 3.4 Fuzzing and negative tests
+
+* Feed the emulation malformed/partial guest actions (truncated elements,
+  bad sequence numbers, out-of-range handles) and require graceful
+  behaviour — this finds assumptions we did not know we had.
+* Feed the *guest* plausible-but-wrong device behaviour in a lab profile and
+  verify the guest recovers (drivers do error recovery we must not break).
+
+### 3.5 Failure archaeology
+
+Every failure leaves artifacts. The discipline is: capture host `dmesg`,
+guest `dmesg`, guest PCI config, and the emulation's per-RPC log; then write
+the *decision tree* that leads to the failure into the debugging docs. The
+tree is the artifact that survives personnel changes.
+
+---
+
+## 4. What we need to know to declare "CUDA works"
+
+To make a defensible statement, we need all of:
+
+1. **Interfaces:** every guest→device message for the CUDA path is either
+   answered or forwarded, for the supported driver family.
+2. **Memory:** every GPU VA used by work is translatable, and every server-owned
+   page-table range is populated correctly.
+3. **Execution:** grid launches and copies run on real hardware with correct
+   results, repeatedly, with errors surfaced.
+4. **Completion:** semaphores, events and faults behave as the guest expects
+   (values advance correctly, no lost interrupts, no stuck waits).
+5. **Version:** at least two driver families pass, with a documented path for
+   more.
+6. **Isolation:** a second tenant cannot observe or influence the first.
+7. **Performance:** within an agreed factor of native for the supported cases,
+   measured with a published methodology.
+
+Until all seven hold, we say "initialization works; execution is in progress",
+and we link here.
+
+---
+
+## 5. Practical challenges (summary; details in the challenges doc)
+
+| Challenge | Why it is hard | Mitigation |
+|---|---|---|
+| Real execution | emulating GR is hopeless; forwarding must be correct and isolated | proxy-first, measure, iterate |
+| Page tables both sides | guest's and hardware's expectations must agree | study AMAP/HAL definitions; build a page-table validator |
+| Latency | RPC round-trips and push translation add overhead | batch, fast-path pure state in the emulation |
+| Version drift | protocol and structures evolve | version-keyed handlers, config-driven profiles |
+| Security | forwarding must not weaken isolation | IOMMU-only access, per-tenant contexts, audits |
+| Debuggability | most bugs are in the guest's head, not ours | trace-first, reconstruct-from-logs policy |
+| Test cost | real GPU time and guest boots are slow | scripted ladders, keep artifacts, parallelize |
\ No newline at end of file
diff --git a/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md b/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md
new file mode 100644
index 0000000..1deff02
--- /dev/null
+++ b/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md
@@ -0,0 +1,156 @@
+# Practical Challenges and Adversarial Interoperability
+
+> How we intend to implement compute/CUDA support *fast and correctly*, and the
+> discipline that keeps the whole stack unmodified — ours included.
+
+## 1. The central engineering bet
+
+> Emulate the interface. Pass through the work. Never patch anyone.
+
+Everything below follows from taking that seriously. It forces hard choices
+(no "just add a flag to the guest driver"), and it pays for itself by making
+the project portable, maintainable and legally clean.
+
+## 2. Fast and correct: the tension
+
+### 2.1 What "correct" means here
+
+Correctness is not "no crash". It is:
+
+* the guest observes the same **values** (capabilities, handles, table
+  contents) it would on real hardware,
+* the guest observes the same **ordering** (completions happen after the work
+  they signal; faults arrive before replays),
+* the guest observes the same **failure semantics** (errors are surfaced, not
+  swallowed),
+* and the work produces the same **results** (a CUDA kernel's output matches
+  native execution).
+
+### 2.2 What "fast" means here
+
+Two budgets matter, and they are different:
+
+* **Initialization latency** — how long from VM start to CUDA ready. Dominated
+  by RPC round-trips and guest driver's own work. Target: within 2× of physical
+  device initialization in the same guest.
+* **Steady-state throughput/latency** — kernel launch overhead, copy
+  bandwidth, and completion latency. Target: published, stable, and within an
+  agreed factor of native for each supported operation class.
+
+We will publish the methodology and the numbers together; a performance claim
+without a methodology is not accepted in this project.
+
+### 2.3 The fast paths
+
+* **Pure state operations** (semaphores, membar, fences, some memops) are
+  satisfied in the emulation directly — they are microseconds in hardware and
+  would otherwise cost a round trip to the physical GPU and back.
+* **Management operations** (object creation, capability queries, mappings)
+  are answered from cached/derived state where the hardware has no side effects
+  the guest can observe.
+* **Real work** (grid launches, copies) is forwarded. Nothing else is allowed
+  on the hot path.
+
+## 3. Forwarding: the three options and how we choose
+
+```
+   (A) Host RM/GSP proxy
+       guest push → translate → host driver's RM API → real channel → GPU
+       + reuses a mature driver's knowledge
+       − extra hop, depends on host driver interfaces
+
+   (B) Direct engine submission
+       guest push → translate → our own channel/push construction → GPU
+       + lowest latency, no host driver dependency
+       − we must own page tables, scheduling, invalidation, completion
+
+   (C) Hybrid
+       management via (A), hot path via (B)
+```
+
+**Decision procedure:** prototype (A) first to get *correct* execution, with
+measurement harnesses around it. Then measure the overhead. If the overhead
+violates the budgets in §2.2 for the target workloads, implement (B) for the
+hot path and keep (A) for management. The architecture must not preclude
+either, so the work path is abstracted behind an interface from day one.
+
+## 4. Hard problems we must solve, and our stance on each
+
+| Problem | Stance |
+|---|---|
+| **Page tables on both sides** | We populate server-owned ranges using the layout the client hands us and the entry encodings from AMAP/HAL references; a validator compares our tables against what the guest expects before enabling a channel. |
+| **Memory coherence** | Guest sysmem is coherent by construction (same physical pages); VRAM objects must be migrated to real VRAM or DMA-mapped. Phase 2 picks per-object policy with correctness first. |
+| **Scheduling between tenants** | One real GPU, N guest channels — schedule explicitly (time-slice/partition), measure fairness, audit isolation. Never let the guest's own scheduler influence another tenant. |
+| **Fault handling** | Reproduce faults under forwarding, implement replay/cancel on the guest's terms, and make fault latency measurable. |
+| **State explosion** | Drivers create enormous state. We do not model all of it — we model what the guest *observes*, and forward what it *uses*. Anything unobserved stays unmodeled. |
+| **Version drift** | Version-keyed handlers with a configuration-driven profile per (driver, firmware) family; new versions added by measurement, never by guessing. |
+| **Security of the debug tooling** | Debug parameters are development-grade; the productized build keeps only read-only diagnostics and gates destructive ones behind explicit opt-in. |
+
+## 5. Adversarial interoperability
+
+### 5.1 The rule
+
+**No modification is permitted in any component we do not own**: guest driver,
+host driver, GPU firmware, hypervisor, device model, guest OS.
+
+Why so absolute: every permitted exception becomes a maintenance liability that
+scales with the number of versions in the field. A patch to the guest driver
+means maintaining forks for every guest OS and driver version. A patch to the
+hypervisor means every hypervisor project must carry it. A firmware change is
+impossible in practice.
+
+### 5.2 How compliance is verified
+
+* **Contract tests**: CI builds run against *stock* drivers obtained from
+  distribution repositories (or vendor runfiles), and against *unmodified*
+  hypervisor packages.
+* **No-patch checks**: our build must not depend on any patch file for
+  third-party components. Where patches exist in the tree for convenience
+  (e.g. build glue), the contract tests must pass without them; anything that
+  fails this is a bug.
+* **Behavioral equivalence**: where we answer from emulation, we validate the
+  values against traces from real hardware (or from our decompiled study notes) —
+  not against our own previous answers.
+
+### 5.3 What we *can* change
+
+* Our own module, userspace library, tools, scripts and docs.
+* Configuration: device profiles, version profiles, capability tables.
+* Tests and harnesses.
+
+### 5.4 Where the line gets tested
+
+* **Firmware signatures**: the driver verifies firmware. We satisfy the
+  handshake by *emulating the interface*, not by patching the image or the
+  driver.
+* **PCI IDs and VBIOS**: the guest driver decides how to treat the device from
+  what it reads. We present a coherent device; we never rely on the driver
+  skipping checks.
+* **Licensing checks**: our device is our device; we do not impersonate a
+  licensed product. Interoperability is the goal, not circumvention of a
+  vendor's own licensing for their own products.
+
+## 6. Why this approach is durable
+
+Because our artifacts are:
+
+* **Interface definitions** (protocol tables, structures) — survive versions.
+* **Measured contracts** (what the guest observes) — survive reimplementation.
+* **Configuration** (profiles per version/hardware) — survive hardware.
+* **Tests** (contract ladders) — catch drift immediately.
+
+A new driver version or GPU is therefore *data + verification*, not a rewrite —
+the difference between a hobby project and infrastructure.
+
+## 7. The "thrilling" part (for developers)
+
+If you are reading this to decide whether to work on it: this project is one of
+the few places where you can:
+
+* reverse engineer a 28 MB RISC-V firmware and turn it into maintainable C,
+* design a memory model that makes two different address-space owners agree,
+* build a virtualization path for the most commercially important accelerator
+  on Earth, under a permissive license,
+* and have your work outlive the hardware generation you started on.
+
+Start with [`../07-developers/onboarding-and-standards.md`](../07-developers/onboarding-and-standards.md).