path: root/docs/04-implementation/01-known-knowns-and-unknowns.md

# Known-Knowns and Known-Unknowns

> The most important document for planning work. It states, without
> flattery, what we know, what we know we do not know, and what we do not even
> know to ask yet — plus how each gap gets closed.

## 0. How to read this

| Category | Meaning | Our approach |
|---|---|---|
| **Known-knowns** | verified facts, with evidence in this repo | rely on them; keep them tested |
| **Known-unknowns** | gaps we have identified, with a concrete plan to close | schedule them, one by one |
| **Unknown-unknowns** | gaps we cannot yet name | discover them by systematic traversal + invariants + instrumentation |

Anything listed here as a known-unknown has an owner-visible plan; anything
listed as an unknown-unknown has a *method* by which we will discover it.

---

## 1. Known-knowns (verified)

### 1.1 Device and platform

* A stock 570.144 Linux guest driver probes and initializes against our
  emulated TU104 device; `nvidia-smi -L` works and CUDA user-space starts
  (`cuInit`, `cuDeviceGetCount`, `cuDeviceGet`). Context creation is in
  progress: the client reaches `cuCtxCreate` and currently ends in a forced
  `rc=999` on a completion wait; the engine inventory is complete (the async
  copy engine is included) and the remaining work is in the completion and
  work-execution path (see §2.1).
* The same guest **boots unmodified**: no guest code, no guest module
  parameters are required for the initialization path.
* The host is unmodified in the standalone configuration (distribution NVIDIA
  driver manages the physical GPU); Open-IOV loads as an independent module.
* Multi-VM operation multiplies the same emulated device; each mdev is
  independent.

### 1.2 Protocol and firmware

* The GSP boot state machine is emulatable by completing observable stages.
* The message queue transport is understood in detail: element header layout,
  page-table translation of non-contiguous queue buffers, sequence matching,
  lengths, and quirks (`length` excludes variable payloads).
* The RPC surface used during driver initialization is fully enumerated for
  570.144 (§9 of the field guide gives the ID table).
* Firmware images are analyzable: the 570.144 image is decompiled
  (~130 functions of interest mapped to C) and its dispatcher/jump tables are
  located.
* Version negotiation exists and is honored (the guest tells us its VGX/RPC
  version; we answer with a matching profile).

### 1.3 Memory and execution semantics

* GPU VA → guest PA translation is recoverable **without guest cooperation**
  by reading the guest's own bookkeeping (VA and CPU-VA pairs recorded in
  UVM/RM structures; direct-map relationship for CPU VAs).
* Guest physical memory is readable and writable from the host module through
  the hypervisor's DMA interface, from a kernel thread context.
* UVM's pushes are decodable (incrementing-method headers, engine classes,
  `SET_SEMAPHORE_A/B/PAYLOAD`, `LAUNCH_DMA`), and completing their semaphore
  releases advances UVM through channel manager creation and page-tree
  initialization.
* The client's channel memory descriptors carry *physical* addresses we can
  trust (instance/userd/ramfc/method buffers).
* The client's page-table contract for server-owned ranges is explicit
  (`COPY_SERVER_RESERVED_PDES`) — and its leaf levels are *empty*, i.e. ours to
  populate.

### 1.4 Operational

* Module reload, mdev create/destroy, VM start/stop, and guest reboot cycles
  are stable when done in the documented order.
* Every layer of the emulation is instrumented and diagnosable from `dmesg`
  plus a small set of module parameters.

---

## 2. Known-unknowns (named gaps with plans)

Each entry: **what we do not know**, **why it matters**, **how we will learn
it**, **rough cost**.

### 2.1 Execution forwarding

**Engine topology (solved):** `GPU_GET_ENGINES_V2` now reports the full engine
inventory including the async copy engine, and the copy-engine capabilities
match the physical device (two GRCEs). The earlier zeroed secondary engine
record that made `libcuda` rewrite the channel alloc class and roll back is
gone; context creation now progresses past engine selection and stalls later,
on completion of decoded work (see §1.1).

**Unknown (resolved by decision, not yet implemented):** the correct,
isolated way to run a guest's engine work on the physical GPU. The interface
analysis in
[`../03-architecture/08-physical-gpu-integration.md`](../03-architecture/08-physical-gpu-integration.md)
shows that proxying through the distribution host driver has no callable entry
point from kernel context, so the **chosen mechanism** is for the module to own
the physical GPU and run its real GSP firmware as the resource-manager server
(the module becomes the host-side RM client). The emulation path remains the
control plane.

**Why:** this is *the* blocker for CUDA kernel execution, and later for
graphics.

**Plan:** build the chosen mechanism in stages: device ownership + real GSP
bring-up; host-side RM client for real objects and channels; translation of
guest work and real submission; completion and fault relay. Measure each stage
against a scorecard: correctness, isolation, latency, maintenance cost across
driver versions. Deliverable: a forwarded `vectorAdd` with verified results.
Alternatives (own scheduling without GSP; vendor vGPU or SR-IOV paths) are
analyzed and rejected or deployment-gated in the integration doc.

**Cost:** weeks, not days; the largest single work item in the project.

### 2.2 Server-owned page tables

**Unknown:** the exact division of labour for page tables after
`COPY_SERVER_RESERVED_PDES`: which ranges the client will touch, when it
expects `RELEASE_ENTRIES`/`RESERVE_ENTRIES`, and how invalidations are
communicated.

**Why:** mappings that only we can create must exist for the guest's own
memory to be reachable by engines.

**Plan:** instrument all vaspace controls (add handlers that log their
parameters), then implement `RESERVE_ENTRIES`/`RELEASE_ENTRIES`/mirroring with
a test harness that maps memory and verifies the guest's expectations by
running its own validation paths (UVM page-tree init exercises this).

**Cost:** days to weeks.

**Reachability resolution note (2026-10-02):** the related IOMMU/DMA
reachability question is closed on the reference host. The physical GPU's
IOMMU group is an identity domain (all-RAM identity) and the mediated device
registers as an emulated-IOMMU VFIO device, so its type-1 container holds
guest DMA mappings privately and no kernel `iommu_domain` exists for it:
`iommu_iova_to_phys` cannot translate guest GPAs. Guest RAM is backed by a
shared file instead, and the module translates GPA->HPA through that file's
page cache. A channel page-table line the GMMU has already walked cannot be
updated at runtime (no TLB-invalidate path exists), so channel page-table
slots for the guest window are pre-built before any engine access. Live proof:
the physical copy engine copied 4 KiB from a real guest page into a module
buffer and back, semaphore released, with the guest process observing the
write and the driver-API smoke green (`ALL OK` 64/64). Wrong-page-table writes
remain the top risk; the division-of-labour question above is unchanged.

### 2.3 Fault semantics

**Unknown:** fault buffer formats and replay/cancel semantics as the *guest*
expects them in our virtualized setting (we know the buffer sizes and that
UVM programs them, but not yet the end-to-end contract under forwarding).

**Why:** GPU page faults happen constantly in real CUDA workloads (UVM
oversubscription, migrations); correct handling is required for robustness.

**Plan:** reproduce faults deliberately (map a VA to nothing, touch it from a
forwarded kernel), capture both sides, implement replay; validate with
`uvm_fault`-heavy tests (e.g. oversubscribed memory).

**Cost:** weeks.

### 2.4 Version matrix beyond 570.144

**Unknown:** which parts of the observed protocol drift between driver
families (550/560/570/580 — the families with entries under
`config/versions/`), how much is negotiated, and how much must be
configuration.

**Why:** durability is a core promise; users will run other versions
immediately.

**Plan:** run the same init traces across 2–3 driver families in guests,
diff the message/control inventories, and refactor handlers to be
version-keyed where they differ. Deliverable: a capability matrix and a
documented "add a version" workflow.

**Cost:** days per version, once the workflow exists.

### 2.5 Windows guest specifics

**Unknown:** the WDDM-facing differences (ROM/VBIOS expectations, MSI-X
semantics, power management, display topology) that the Windows driver
requires beyond the Linux path.

**Why:** Windows CUDA/3D users are a first-class audience.

**Plan:** bring up a Windows guest against the same device, capture the driver
logs (Windows ETW + our host traces), and add configuration for discovered
requirements.

**Cost:** weeks.

### 2.6 Non-KVM transports

**Unknown:** what differs for Xen/Argo/OpenXT/Qubes OS Air: IRQ delivery
mechanisms, memory pinning, device-model integration points.

**Why:** the security-focused platforms are a major part of the project's
reason for existing.

**Plan:** abstract the transport boundary (IRQ injection, guest-memory access,
device registration) behind a small interface; implement a Xen backend against
it; test in a Xen guest.

**Cost:** weeks.

### 2.7 aarch64 hosts and Grace/GB10

**Unknown:** host-side differences (page sizes, cache attributes, SMMU
behaviour, power domains) and any firmware/ROM differences in the guest path.

**Why:** Grace-Hopper/Grace-Blackwell/GB10 (DGX Spark, Surface-class laptops)
are explicit targets.

**Plan:** make the emulation architecture-independent by construction
(no x86-specific assumptions in the memory model), then bring up on an
aarch64 host with the same test ladder.

**Cost:** weeks per platform.

### 2.8 Multi-tenant performance policy

**Unknown:** best scheduling policy for sharing one GPU across VM tenants
(time slice, MIG partition, hybrid) and its fairness/isolation properties.

**Why:** multi-tenancy is a core value proposition.

**Plan:** build measurement first (per-tenant attribution), then implement
policy as configuration, then audit isolation.

**Cost:** months (it is a research-y area).

### 2.9 Tooling maturity

**Unknown:** how much of the debug tooling (memory scanners, pokes, push
decoders) can be productized without exposing unsafe interfaces.

**Why:** the tooling is currently how we make progress; it must not become a
security liability.

**Plan:** move tooling behind a debug-only build flag/document its threat
model; keep safe, read-only diagnostics in production builds.

**Cost:** days.

---

## 3. Unknown-unknowns (and how we find them)

Unknown-unknowns are discovered by *systematic traversal* and *invariants*,
not by luck. Our methods:

### 3.1 Traversal by construction

* **Enumerate every interface the guest touches.** For each: prove we answer
  it or forward it. Any interface we have not seen is a discovery task, not a
  bug.
* **Drive the guest further, always.** Each new guest-side milestone (probe →
  nvidia-smi → cuInit → context → allocation → kernel) is a traversal step
  that *forces* new interfaces to appear in the trace.
* **Differential testing across versions/hardware.** The same traversal on a
  second driver version or GPU surfaces requirements that are invisible on
  the first.

### 3.2 Invariants that trip on unknowns

We rely on the guest's own self-checks as an oracle:

* UVM asserts on semaphore jumps, bad page tables, impossible capabilities.
* The RM validates structures and will fault on inconsistencies.
* Kernel drivers log "unexpected" states loudly.

Every guest assert/log is a discovered unknown-unknown if we do not already
have it in this document.

### 3.3 Structural instrumentation

* **Trace-first policy:** no handler is written without logging the request
  *and* the reply in a reconstructible form.
* **Reconstruct from dmesg:** the logs must be sufficient to rebuild the
  object tree, memory map and push stream offline. This is how protocol drift
  is detected early, before it becomes a crash.

### 3.4 Fuzzing and negative tests

* Feed the emulation malformed/partial guest actions (truncated elements,
  bad sequence numbers, out-of-range handles) and require graceful
  behaviour — this finds assumptions we did not know we had.
* Feed the *guest* plausible-but-wrong device behaviour in a lab profile and
  verify the guest recovers (drivers do error recovery we must not break).

### 3.5 Failure archaeology

Every failure leaves artifacts. The discipline is: capture host `dmesg`,
guest `dmesg`, guest PCI config, and the emulation's per-RPC log; then write
the *decision tree* that leads to the failure into the debugging docs. The
tree is the artifact that survives personnel changes.

---

## 4. What we need to know to declare "CUDA works"

To make a defensible statement, we need all of:

1. **Interfaces:** every guest→device message for the CUDA path is either
   answered or forwarded, for the supported driver family — and every reply
   whose fields the driver consumes is coherent, not just `SUCCESS`-shaped
   (the reply-fidelity ladder: status-only → coherent zeroed params → coherent
   values for consumed fields → exact values; see
   [`../../userspace/cuda/docs/04-virtualization-and-emulation.md` §6](../../userspace/cuda/docs/04-virtualization-and-emulation.md)).
2. **Memory:** every GPU VA used by work is translatable, and every server-owned
   page-table range is populated correctly.
3. **Execution:** grid launches and copies run on real hardware with correct
   results, repeatedly, with errors surfaced.
4. **Completion:** semaphores, events and faults behave as the guest expects
   (values advance correctly, no lost interrupts, no stuck waits).
5. **Version:** at least two driver families pass, with a documented path for
   more.
6. **Isolation:** a second tenant cannot observe or influence the first.
7. **Performance:** within an agreed factor of native for the supported cases,
   measured with a published methodology.

Until all seven hold, we say "initialization works; execution is in progress",
and we link here. The CUDA-side checklist behind these gates and the current
blocker are in
[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md` §9](../../userspace/cuda/docs/03-driver-and-gpu-interface.md);
start at
[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md).

---

## 5. Practical challenges (summary; details in the challenges doc)

| Challenge | Why it is hard | Mitigation |
|---|---|---|
| Real execution | emulating GR is hopeless; forwarding must be correct and isolated | proxy-first, measure, iterate |
| Page tables both sides | guest's and hardware's expectations must agree | study AMAP/HAL definitions; build a page-table validator |
| Latency | RPC round-trips and push translation add overhead | batch, fast-path pure state in the emulation |
| Version drift | protocol and structures evolve | version-keyed handlers, config-driven profiles |
| Security | forwarding must not weaken isolation | IOMMU-only access, per-tenant contexts, audits |
| Debuggability | most bugs are in the guest's head, not ours | trace-first, reconstruct-from-logs policy |
| Test cost | real GPU time and guest boots are slow | scripted ladders, keep artifacts, parallelize |