Open-IOV: initial public release

Open-IOV virtualizes NVIDIA GPUs so that unmodified guest drivers run with
real GPU functionality - CUDA first, 3D and video to follow - on unmodified
hosts, without a proprietary vGPU license and without patching any driver,
firmware or hypervisor.

Core principle: emulate the interface, pass through the work.

  kernel/     open-iov-vfio.ko - host VFIO mdev driver: virtual GPU device
              (PCI config space, BAR shadows, MSI-X), GSP boot emulation,
              GSP-RM message-queue/RPC emulation, memory model (GPU-VA to
              guest-PA discovery) and the semaphore-completion service that
              advances guest UVM progress.
  userspace/  libiov.so management API and examples.
  tools/      guest diagnostics, firmware analysis wrappers, RM probes.
  scripts/    stack manager, VM harness, build/load/test automation.
  config/     hardware/version profiles (the portability surface).
  docs/       documentation tree - start at docs/README.md.
  wiki/       long-form firmware and protocol deep dives.

Status: initialization works end to end for a stock 570.144 Linux guest on
Turing (TU104): driver probe, nvidia-smi, the full GSP-RM control battery,
object creation and the early CUDA/UVM start-up path. GPU work forwarding -
the step that makes CUDA kernels execute on real hardware - is the active
work item; see docs/04-implementation/01-known-knowns-and-unknowns.md.

License: BSD-3-Clause for the project; GPL-2.0 for kernel modules that link
GPL-only kernel interfaces. See LICENSE and docs/09-reference/legal-and-licensing.md.

1 files changed, 300 insertions, 0 deletions

diff --git a/docs/04-implementation/01-known-knowns-and-unknowns.md b/docs/04-implementation/01-known-knowns-and-unknowns.md
new file mode 100644
index 0000000..0564ffc
--- /dev/null
+++ b/docs/04-implementation/01-known-knowns-and-unknowns.md
@@ -0,0 +1,300 @@
+# Known-Knowns and Known-Unknowns
+
+> The most important document for planning work. It states, without
+> flattery, what we know, what we know we do not know, and what we do not even
+> know to ask yet — plus how each gap gets closed.
+
+## 0. How to read this
+
+| Category | Meaning | Our approach |
+|---|---|---|
+| **Known-knowns** | verified facts, with evidence in this repo | rely on them; keep them tested |
+| **Known-unknowns** | gaps we have identified, with a concrete plan to close | schedule them, one by one |
+| **Unknown-unknowns** | gaps we cannot yet name | discover them by systematic traversal + invariants + instrumentation |
+
+Anything listed here as a known-unknown has an owner-visible plan; anything
+listed as an unknown-unknown has a *method* by which we will discover it.
+
+---
+
+## 1. Known-knowns (verified)
+
+### 1.1 Device and platform
+
+* A stock 570.144 Linux guest driver probes and initializes against our
+  emulated TU104 device; `nvidia-smi` works.
+* The same guest **boots unmodified**: no guest code, no guest module
+  parameters are required for the initialization path.
+* The host is unmodified in the standalone configuration (distribution NVIDIA
+  driver manages the physical GPU); Open-IOV loads as an independent module.
+* Multi-VM operation multiplies the same emulated device; each mdev is
+  independent.
+
+### 1.2 Protocol and firmware
+
+* The GSP boot state machine is emulatable by completing observable stages.
+* The message queue transport is understood in detail: element header layout,
+  page-table translation of non-contiguous queue buffers, sequence matching,
+  lengths, and quirks (`length` excludes variable payloads).
+* The RPC surface used during driver initialization is fully enumerated for
+  570.144 (§9 of the field guide gives the ID table).
+* Firmware images are analyzable: the 570.144 image is decompiled
+  (~130 functions of interest mapped to C) and its dispatcher/jump tables are
+  located.
+* Version negotiation exists and is honored (the guest tells us its VGX/RPC
+  version; we answer with a matching profile).
+
+### 1.3 Memory and execution semantics
+
+* GPU VA → guest PA translation is recoverable **without guest cooperation**
+  by reading the guest's own bookkeeping (VA and CPU-VA pairs recorded in
+  UVM/RM structures; direct-map relationship for CPU VAs).
+* Guest physical memory is readable and writable from the host module through
+  the hypervisor's DMA interface, from a kernel thread context.
+* UVM's pushes are decodable (incrementing-method headers, engine classes,
+  `SET_SEMAPHORE_A/B/PAYLOAD`, `LAUNCH_DMA`), and completing their semaphore
+  releases advances UVM through channel manager creation and page-tree
+  initialization.
+* The client's channel memory descriptors carry *physical* addresses we can
+  trust (instance/userd/ramfc/method buffers).
+* The client's page-table contract for server-owned ranges is explicit
+  (`COPY_SERVER_RESERVED_PDES`) — and its leaf levels are *empty*, i.e. ours to
+  populate.
+
+### 1.4 Operational
+
+* Module reload, mdev create/destroy, VM start/stop, and guest reboot cycles
+  are stable when done in the documented order.
+* Every layer of the emulation is instrumented and diagnosable from `dmesg`
+  plus a small set of module parameters.
+
+---
+
+## 2. Known-unknowns (named gaps with plans)
+
+Each entry: **what we do not know**, **why it matters**, **how we will learn
+it**, **rough cost**.
+
+### 2.1 Execution forwarding
+
+**Unknown:** the correct, isolated way to run a guest's engine work on the
+physical GPU: which real scheduling primitive to use per engine, how to
+translate guest pushes to real submissions, and how to relay completion and
+fault semantics.
+
+**Why:** this is *the* blocker for CUDA kernel execution, and later for
+graphics.
+
+**Plan:** evaluate the three backends (host RM/GSP proxy, direct engine
+submission, hybrid) against a scorecard: correctness, isolation, latency,
+maintenance cost across driver versions. Prototype the proxy first (it reuses
+the unmodified host driver's knowledge), measure latency, then decide whether a
+direct path is required. Deliverable: a forwarded `vectorAdd` with verified
+results.
+
+**Cost:** weeks, not days; the largest single work item in the project.
+
+### 2.2 Server-owned page tables
+
+**Unknown:** the exact division of labour for page tables after
+`COPY_SERVER_RESERVED_PDES`: which ranges the client will touch, when it
+expects `RELEASE_ENTRIES`/`RESERVE_ENTRIES`, and how invalidations are
+communicated.
+
+**Why:** mappings that only we can create must exist for the guest's own
+memory to be reachable by engines.
+
+**Plan:** instrument all vaspace controls (add handlers that log their
+parameters), then implement `RESERVE_ENTRIES`/`RELEASE_ENTRIES`/mirroring with
+a test harness that maps memory and verifies the guest's expectations by
+running its own validation paths (UVM page-tree init exercises this).
+
+**Cost:** days to weeks.
+
+### 2.3 Fault semantics
+
+**Unknown:** fault buffer formats and replay/cancel semantics as the *guest*
+expects them in our virtualized setting (we know the buffer sizes and that
+UVM programs them, but not yet the end-to-end contract under forwarding).
+
+**Why:** GPU page faults happen constantly in real CUDA workloads (UVM
+oversubscription, migrations); correct handling is required for robustness.
+
+**Plan:** reproduce faults deliberately (map a VA to nothing, touch it from a
+forwarded kernel), capture both sides, implement replay; validate with
+`uvm_fault`-heavy tests (e.g. oversubscribed memory).
+
+**Cost:** weeks.
+
+### 2.4 Version matrix beyond 570.144
+
+**Unknown:** which parts of the observed protocol drift between driver
+families (550/560/580/610+), how much is negotiated, and how much must be
+configuration.
+
+**Why:** durability is a core promise; users will run other versions
+immediately.
+
+**Plan:** run the same init traces across 2–3 driver families in guests,
+diff the message/control inventories, and refactor handlers to be
+version-keyed where they differ. Deliverable: a capability matrix and a
+documented "add a version" workflow.
+
+**Cost:** days per version, once the workflow exists.
+
+### 2.5 Windows guest specifics
+
+**Unknown:** the WDDM-facing differences (ROM/VBIOS expectations, MSI-X
+semantics, power management, display topology) that the Windows driver
+requires beyond the Linux path.
+
+**Why:** Windows CUDA/3D users are a first-class audience.
+
+**Plan:** bring up a Windows guest against the same device, capture the driver
+logs (Windows ETW + our host traces), and add configuration for discovered
+requirements.
+
+**Cost:** weeks.
+
+### 2.6 Non-KVM transports
+
+**Unknown:** what differs for Xen/Argo/OpenXT/Qubes OS Air: IRQ delivery
+mechanisms, memory pinning, device-model integration points.
+
+**Why:** the security-focused platforms are a major part of the project's
+reason for existing.
+
+**Plan:** abstract the transport boundary (IRQ injection, guest-memory access,
+device registration) behind a small interface; implement a Xen backend against
+it; test in a Xen guest.
+
+**Cost:** weeks.
+
+### 2.7 aarch64 hosts and Grace/GB10
+
+**Unknown:** host-side differences (page sizes, cache attributes, SMMU
+behaviour, power domains) and any firmware/ROM differences in the guest path.
+
+**Why:** Grace-Hopper/Grace-Blackwell/GB10 (DGX Spark, Surface-class laptops)
+are explicit targets.
+
+**Plan:** make the emulation architecture-independent by construction
+(no x86-specific assumptions in the memory model), then bring up on an
+aarch64 host with the same test ladder.
+
+**Cost:** weeks per platform.
+
+### 2.8 Multi-tenant performance policy
+
+**Unknown:** best scheduling policy for sharing one GPU across VM tenants
+(time slice, MIG partition, hybrid) and its fairness/isolation properties.
+
+**Why:** multi-tenancy is a core value proposition.
+
+**Plan:** build measurement first (per-tenant attribution), then implement
+policy as configuration, then audit isolation.
+
+**Cost:** months (it is a research-y area).
+
+### 2.9 Tooling maturity
+
+**Unknown:** how much of the debug tooling (memory scanners, pokes, push
+decoders) can be productized without exposing unsafe interfaces.
+
+**Why:** the tooling is currently how we make progress; it must not become a
+security liability.
+
+**Plan:** move tooling behind a debug-only build flag/document its threat
+model; keep safe, read-only diagnostics in production builds.
+
+**Cost:** days.
+
+---
+
+## 3. Unknown-unknowns (and how we find them)
+
+Unknown-unknowns are discovered by *systematic traversal* and *invariants*,
+not by luck. Our methods:
+
+### 3.1 Traversal by construction
+
+* **Enumerate every interface the guest touches.** For each: prove we answer
+  it or forward it. Any interface we have not seen is a discovery task, not a
+  bug.
+* **Drive the guest further, always.** Each new guest-side milestone (probe →
+  nvidia-smi → cuInit → context → allocation → kernel) is a traversal step
+  that *forces* new interfaces to appear in the trace.
+* **Differential testing across versions/hardware.** The same traversal on a
+  second driver version or GPU surfaces requirements that are invisible on
+  the first.
+
+### 3.2 Invariants that trip on unknowns
+
+We rely on the guest's own self-checks as an oracle:
+
+* UVM asserts on semaphore jumps, bad page tables, impossible capabilities.
+* The RM validates structures and will fault on inconsistencies.
+* Kernel drivers log "unexpected" states loudly.
+
+Every guest assert/log is a discovered unknown-unknown if we do not already
+have it in this document.
+
+### 3.3 Structural instrumentation
+
+* **Trace-first policy:** no handler is written without logging the request
+  *and* the reply in a reconstructible form.
+* **Reconstruct from dmesg:** the logs must be sufficient to rebuild the
+  object tree, memory map and push stream offline. This is how protocol drift
+  is detected early, before it becomes a crash.
+
+### 3.4 Fuzzing and negative tests
+
+* Feed the emulation malformed/partial guest actions (truncated elements,
+  bad sequence numbers, out-of-range handles) and require graceful
+  behaviour — this finds assumptions we did not know we had.
+* Feed the *guest* plausible-but-wrong device behaviour in a lab profile and
+  verify the guest recovers (drivers do error recovery we must not break).
+
+### 3.5 Failure archaeology
+
+Every failure leaves artifacts. The discipline is: capture host `dmesg`,
+guest `dmesg`, guest PCI config, and the emulation's per-RPC log; then write
+the *decision tree* that leads to the failure into the debugging docs. The
+tree is the artifact that survives personnel changes.
+
+---
+
+## 4. What we need to know to declare "CUDA works"
+
+To make a defensible statement, we need all of:
+
+1. **Interfaces:** every guest→device message for the CUDA path is either
+   answered or forwarded, for the supported driver family.
+2. **Memory:** every GPU VA used by work is translatable, and every server-owned
+   page-table range is populated correctly.
+3. **Execution:** grid launches and copies run on real hardware with correct
+   results, repeatedly, with errors surfaced.
+4. **Completion:** semaphores, events and faults behave as the guest expects
+   (values advance correctly, no lost interrupts, no stuck waits).
+5. **Version:** at least two driver families pass, with a documented path for
+   more.
+6. **Isolation:** a second tenant cannot observe or influence the first.
+7. **Performance:** within an agreed factor of native for the supported cases,
+   measured with a published methodology.
+
+Until all seven hold, we say "initialization works; execution is in progress",
+and we link here.
+
+---
+
+## 5. Practical challenges (summary; details in the challenges doc)
+
+| Challenge | Why it is hard | Mitigation |
+|---|---|---|
+| Real execution | emulating GR is hopeless; forwarding must be correct and isolated | proxy-first, measure, iterate |
+| Page tables both sides | guest's and hardware's expectations must agree | study AMAP/HAL definitions; build a page-table validator |
+| Latency | RPC round-trips and push translation add overhead | batch, fast-path pure state in the emulation |
+| Version drift | protocol and structures evolve | version-keyed handlers, config-driven profiles |
+| Security | forwarding must not weaken isolation | IOMMU-only access, per-tenant contexts, audits |
+| Debuggability | most bugs are in the guest's head, not ours | trace-first, reconstruct-from-logs policy |
+| Test cost | real GPU time and guest boots are slow | scripted ladders, keep artifacts, parallelize |
\ No newline at end of file