diff options
Diffstat (limited to 'docs/04-implementation/01-known-knowns-and-unknowns.md')
| -rw-r--r-- | docs/04-implementation/01-known-knowns-and-unknowns.md | 300 |
1 files changed, 300 insertions, 0 deletions
diff --git a/docs/04-implementation/01-known-knowns-and-unknowns.md b/docs/04-implementation/01-known-knowns-and-unknowns.md new file mode 100644 index 0000000..0564ffc --- /dev/null +++ b/docs/04-implementation/01-known-knowns-and-unknowns.md @@ -0,0 +1,300 @@ +# Known-Knowns and Known-Unknowns + +> The most important document for planning work. It states, without +> flattery, what we know, what we know we do not know, and what we do not even +> know to ask yet — plus how each gap gets closed. + +## 0. How to read this + +| Category | Meaning | Our approach | +|---|---|---| +| **Known-knowns** | verified facts, with evidence in this repo | rely on them; keep them tested | +| **Known-unknowns** | gaps we have identified, with a concrete plan to close | schedule them, one by one | +| **Unknown-unknowns** | gaps we cannot yet name | discover them by systematic traversal + invariants + instrumentation | + +Anything listed here as a known-unknown has an owner-visible plan; anything +listed as an unknown-unknown has a *method* by which we will discover it. + +--- + +## 1. Known-knowns (verified) + +### 1.1 Device and platform + +* A stock 570.144 Linux guest driver probes and initializes against our + emulated TU104 device; `nvidia-smi` works. +* The same guest **boots unmodified**: no guest code, no guest module + parameters are required for the initialization path. +* The host is unmodified in the standalone configuration (distribution NVIDIA + driver manages the physical GPU); Open-IOV loads as an independent module. +* Multi-VM operation multiplies the same emulated device; each mdev is + independent. + +### 1.2 Protocol and firmware + +* The GSP boot state machine is emulatable by completing observable stages. +* The message queue transport is understood in detail: element header layout, + page-table translation of non-contiguous queue buffers, sequence matching, + lengths, and quirks (`length` excludes variable payloads). +* The RPC surface used during driver initialization is fully enumerated for + 570.144 (§9 of the field guide gives the ID table). +* Firmware images are analyzable: the 570.144 image is decompiled + (~130 functions of interest mapped to C) and its dispatcher/jump tables are + located. +* Version negotiation exists and is honored (the guest tells us its VGX/RPC + version; we answer with a matching profile). + +### 1.3 Memory and execution semantics + +* GPU VA → guest PA translation is recoverable **without guest cooperation** + by reading the guest's own bookkeeping (VA and CPU-VA pairs recorded in + UVM/RM structures; direct-map relationship for CPU VAs). +* Guest physical memory is readable and writable from the host module through + the hypervisor's DMA interface, from a kernel thread context. +* UVM's pushes are decodable (incrementing-method headers, engine classes, + `SET_SEMAPHORE_A/B/PAYLOAD`, `LAUNCH_DMA`), and completing their semaphore + releases advances UVM through channel manager creation and page-tree + initialization. +* The client's channel memory descriptors carry *physical* addresses we can + trust (instance/userd/ramfc/method buffers). +* The client's page-table contract for server-owned ranges is explicit + (`COPY_SERVER_RESERVED_PDES`) — and its leaf levels are *empty*, i.e. ours to + populate. + +### 1.4 Operational + +* Module reload, mdev create/destroy, VM start/stop, and guest reboot cycles + are stable when done in the documented order. +* Every layer of the emulation is instrumented and diagnosable from `dmesg` + plus a small set of module parameters. + +--- + +## 2. Known-unknowns (named gaps with plans) + +Each entry: **what we do not know**, **why it matters**, **how we will learn +it**, **rough cost**. + +### 2.1 Execution forwarding + +**Unknown:** the correct, isolated way to run a guest's engine work on the +physical GPU: which real scheduling primitive to use per engine, how to +translate guest pushes to real submissions, and how to relay completion and +fault semantics. + +**Why:** this is *the* blocker for CUDA kernel execution, and later for +graphics. + +**Plan:** evaluate the three backends (host RM/GSP proxy, direct engine +submission, hybrid) against a scorecard: correctness, isolation, latency, +maintenance cost across driver versions. Prototype the proxy first (it reuses +the unmodified host driver's knowledge), measure latency, then decide whether a +direct path is required. Deliverable: a forwarded `vectorAdd` with verified +results. + +**Cost:** weeks, not days; the largest single work item in the project. + +### 2.2 Server-owned page tables + +**Unknown:** the exact division of labour for page tables after +`COPY_SERVER_RESERVED_PDES`: which ranges the client will touch, when it +expects `RELEASE_ENTRIES`/`RESERVE_ENTRIES`, and how invalidations are +communicated. + +**Why:** mappings that only we can create must exist for the guest's own +memory to be reachable by engines. + +**Plan:** instrument all vaspace controls (add handlers that log their +parameters), then implement `RESERVE_ENTRIES`/`RELEASE_ENTRIES`/mirroring with +a test harness that maps memory and verifies the guest's expectations by +running its own validation paths (UVM page-tree init exercises this). + +**Cost:** days to weeks. + +### 2.3 Fault semantics + +**Unknown:** fault buffer formats and replay/cancel semantics as the *guest* +expects them in our virtualized setting (we know the buffer sizes and that +UVM programs them, but not yet the end-to-end contract under forwarding). + +**Why:** GPU page faults happen constantly in real CUDA workloads (UVM +oversubscription, migrations); correct handling is required for robustness. + +**Plan:** reproduce faults deliberately (map a VA to nothing, touch it from a +forwarded kernel), capture both sides, implement replay; validate with +`uvm_fault`-heavy tests (e.g. oversubscribed memory). + +**Cost:** weeks. + +### 2.4 Version matrix beyond 570.144 + +**Unknown:** which parts of the observed protocol drift between driver +families (550/560/580/610+), how much is negotiated, and how much must be +configuration. + +**Why:** durability is a core promise; users will run other versions +immediately. + +**Plan:** run the same init traces across 2–3 driver families in guests, +diff the message/control inventories, and refactor handlers to be +version-keyed where they differ. Deliverable: a capability matrix and a +documented "add a version" workflow. + +**Cost:** days per version, once the workflow exists. + +### 2.5 Windows guest specifics + +**Unknown:** the WDDM-facing differences (ROM/VBIOS expectations, MSI-X +semantics, power management, display topology) that the Windows driver +requires beyond the Linux path. + +**Why:** Windows CUDA/3D users are a first-class audience. + +**Plan:** bring up a Windows guest against the same device, capture the driver +logs (Windows ETW + our host traces), and add configuration for discovered +requirements. + +**Cost:** weeks. + +### 2.6 Non-KVM transports + +**Unknown:** what differs for Xen/Argo/OpenXT/Qubes OS Air: IRQ delivery +mechanisms, memory pinning, device-model integration points. + +**Why:** the security-focused platforms are a major part of the project's +reason for existing. + +**Plan:** abstract the transport boundary (IRQ injection, guest-memory access, +device registration) behind a small interface; implement a Xen backend against +it; test in a Xen guest. + +**Cost:** weeks. + +### 2.7 aarch64 hosts and Grace/GB10 + +**Unknown:** host-side differences (page sizes, cache attributes, SMMU +behaviour, power domains) and any firmware/ROM differences in the guest path. + +**Why:** Grace-Hopper/Grace-Blackwell/GB10 (DGX Spark, Surface-class laptops) +are explicit targets. + +**Plan:** make the emulation architecture-independent by construction +(no x86-specific assumptions in the memory model), then bring up on an +aarch64 host with the same test ladder. + +**Cost:** weeks per platform. + +### 2.8 Multi-tenant performance policy + +**Unknown:** best scheduling policy for sharing one GPU across VM tenants +(time slice, MIG partition, hybrid) and its fairness/isolation properties. + +**Why:** multi-tenancy is a core value proposition. + +**Plan:** build measurement first (per-tenant attribution), then implement +policy as configuration, then audit isolation. + +**Cost:** months (it is a research-y area). + +### 2.9 Tooling maturity + +**Unknown:** how much of the debug tooling (memory scanners, pokes, push +decoders) can be productized without exposing unsafe interfaces. + +**Why:** the tooling is currently how we make progress; it must not become a +security liability. + +**Plan:** move tooling behind a debug-only build flag/document its threat +model; keep safe, read-only diagnostics in production builds. + +**Cost:** days. + +--- + +## 3. Unknown-unknowns (and how we find them) + +Unknown-unknowns are discovered by *systematic traversal* and *invariants*, +not by luck. Our methods: + +### 3.1 Traversal by construction + +* **Enumerate every interface the guest touches.** For each: prove we answer + it or forward it. Any interface we have not seen is a discovery task, not a + bug. +* **Drive the guest further, always.** Each new guest-side milestone (probe → + nvidia-smi → cuInit → context → allocation → kernel) is a traversal step + that *forces* new interfaces to appear in the trace. +* **Differential testing across versions/hardware.** The same traversal on a + second driver version or GPU surfaces requirements that are invisible on + the first. + +### 3.2 Invariants that trip on unknowns + +We rely on the guest's own self-checks as an oracle: + +* UVM asserts on semaphore jumps, bad page tables, impossible capabilities. +* The RM validates structures and will fault on inconsistencies. +* Kernel drivers log "unexpected" states loudly. + +Every guest assert/log is a discovered unknown-unknown if we do not already +have it in this document. + +### 3.3 Structural instrumentation + +* **Trace-first policy:** no handler is written without logging the request + *and* the reply in a reconstructible form. +* **Reconstruct from dmesg:** the logs must be sufficient to rebuild the + object tree, memory map and push stream offline. This is how protocol drift + is detected early, before it becomes a crash. + +### 3.4 Fuzzing and negative tests + +* Feed the emulation malformed/partial guest actions (truncated elements, + bad sequence numbers, out-of-range handles) and require graceful + behaviour — this finds assumptions we did not know we had. +* Feed the *guest* plausible-but-wrong device behaviour in a lab profile and + verify the guest recovers (drivers do error recovery we must not break). + +### 3.5 Failure archaeology + +Every failure leaves artifacts. The discipline is: capture host `dmesg`, +guest `dmesg`, guest PCI config, and the emulation's per-RPC log; then write +the *decision tree* that leads to the failure into the debugging docs. The +tree is the artifact that survives personnel changes. + +--- + +## 4. What we need to know to declare "CUDA works" + +To make a defensible statement, we need all of: + +1. **Interfaces:** every guest→device message for the CUDA path is either + answered or forwarded, for the supported driver family. +2. **Memory:** every GPU VA used by work is translatable, and every server-owned + page-table range is populated correctly. +3. **Execution:** grid launches and copies run on real hardware with correct + results, repeatedly, with errors surfaced. +4. **Completion:** semaphores, events and faults behave as the guest expects + (values advance correctly, no lost interrupts, no stuck waits). +5. **Version:** at least two driver families pass, with a documented path for + more. +6. **Isolation:** a second tenant cannot observe or influence the first. +7. **Performance:** within an agreed factor of native for the supported cases, + measured with a published methodology. + +Until all seven hold, we say "initialization works; execution is in progress", +and we link here. + +--- + +## 5. Practical challenges (summary; details in the challenges doc) + +| Challenge | Why it is hard | Mitigation | +|---|---|---| +| Real execution | emulating GR is hopeless; forwarding must be correct and isolated | proxy-first, measure, iterate | +| Page tables both sides | guest's and hardware's expectations must agree | study AMAP/HAL definitions; build a page-table validator | +| Latency | RPC round-trips and push translation add overhead | batch, fast-path pure state in the emulation | +| Version drift | protocol and structures evolve | version-keyed handlers, config-driven profiles | +| Security | forwarding must not weaken isolation | IOMMU-only access, per-tenant contexts, audits | +| Debuggability | most bugs are in the guest's head, not ours | trace-first, reconstruct-from-logs policy | +| Test cost | real GPU time and guest boots are slow | scripted ladders, keep artifacts, parallelize |
\ No newline at end of file |