# Known-Knowns and Known-Unknowns

> The most important document for planning work. It states, without
> flattery, what we know, what we know we do not know, and what we do not even
> know to ask yet — plus how each gap gets closed.

## 0. How to read this

| Category | Meaning | Our approach |
|---|---|---|
| **Known-knowns** | verified facts, with evidence in this repo | rely on them; keep them tested |
| **Known-unknowns** | gaps we have identified, with a concrete plan to close | schedule them, one by one |
| **Unknown-unknowns** | gaps we cannot yet name | discover them by systematic traversal + invariants + instrumentation |

Anything listed here as a known-unknown has an owner-visible plan; anything
listed as an unknown-unknown has a *method* by which we will discover it.

---

## 1. Known-knowns (verified)

### 1.1 Device and platform

* A stock 570.144 Linux guest driver probes and initializes against our
  emulated TU104 device; `nvidia-smi -L` works and the full CUDA driver-API
  smoke passes (`ALL OK` 64/64, copies matched) — `cuInit`, device queries,
  context creation, allocation, copies, free and destroy complete, and the
  copies are verified byte-for-byte. The execution backend is live: CE-class copies are
  forwarded to the physical copy engine (default off) with engine-authored
  semaphore releases and a fault relay; the remaining frontier is engine
  execution — a forwarded guest launch now reaches the front-end (entry
  fetched, methods run) but the inline launch-descriptor install has not been
  observed and no grid has dispatched; context validity at dispatch is ruled
  out (see §2.1 and [`03-execution-frontier.md`](03-execution-frontier.md)).
* The same guest **boots unmodified**: no guest code, no guest module
  parameters are required for the initialization path.
* The host is unmodified in the standalone configuration (distribution NVIDIA
  driver manages the physical GPU); Open-IOV loads as an independent module.
* Multi-VM operation multiplies the same emulated device; each mdev is
  independent.

### 1.2 Protocol and firmware

* The GSP boot state machine is emulatable by completing observable stages.
* The message queue transport is understood in detail: element header layout,
  page-table translation of non-contiguous queue buffers, sequence matching,
  lengths, and quirks (`length` excludes variable payloads).
* The RPC surface used during driver initialization is fully enumerated for
  570.144 (§9 of the field guide gives the ID table).
* Firmware images are analyzable: the 570.144 image is decompiled
  (~130 functions of interest mapped to C) and its dispatcher/jump tables are
  located.
* Version negotiation exists and is honored (the guest tells us its VGX/RPC
  version; we answer with a matching profile).

### 1.3 Memory and execution semantics

* GPU VA → guest PA translation is recoverable **without guest cooperation**
  by reading the guest's own bookkeeping (VA and CPU-VA pairs recorded in
  UVM/RM structures; direct-map relationship for CPU VAs).
* Guest physical memory is readable and writable from the host module through
  the hypervisor's DMA interface, from a kernel thread context.
* UVM's pushes are decodable (incrementing-method headers, engine classes,
  `SET_SEMAPHORE_A/B/PAYLOAD`, `LAUNCH_DMA`), and completing their semaphore
  releases advances UVM through channel manager creation and page-tree
  initialization.
* The client's channel memory descriptors carry *physical* addresses we can
  trust (instance/userd/ramfc/method buffers).
* The client's page-table contract for server-owned ranges is explicit
  (`COPY_SERVER_RESERVED_PDES`) — and its leaf levels are *empty*, i.e. ours to
  populate.
* **End-to-end CUDA works in emulation mode.** A stock CUDA program in the
  guest (device properties, `cudaMalloc`, a GPU-authored string kernel, two
  4 MB host-to-device copies, a 1 M-element `vectorAdd`, and the
  device-to-host copy-back) completes with `bad=0` against the host-computed
  reference and the expected checksum. The host-side bridge executes the
  client's kernels on the physical GPU, and the emulated CE path stages the
  data in and out. Two correctness invariants are required and verified:
  a decoded copy executes at most once per GPFIFO entry identity, and every
  virtual operand is translated per 4 KB page (see
  [`../03-architecture/06-host-module-and-hybrid-model.md`](../03-architecture/06-host-module-and-hybrid-model.md) §4.4).
  The acceptance is deterministic across consecutive supervised cycles
  (identical input/output hashes and checksum).

### 1.4 Operational

* Module reload, mdev create/destroy, VM start/stop, and guest reboot cycles
  are stable when done in the documented order.
* Every layer of the emulation is instrumented and diagnosable from `dmesg`
  plus a small set of module parameters.

---

## 2. Known-unknowns (named gaps with plans)

Each entry: **what we do not know**, **why it matters**, **how we will learn
it**, **rough cost**.

### 2.1 Execution forwarding

**Engine topology (solved):** `GPU_GET_ENGINES_V2` now reports the full engine
inventory including the async copy engine, and the copy-engine capabilities
match the physical device (two GRCEs). The earlier zeroed secondary engine
record that made `libcuda` rewrite the channel alloc class and roll back is
gone; context creation completes and the driver-API smoke passes (see §1.1).

**Partially delivered:** the correct, isolated way to run a guest's engine
work on the physical GPU. The interface analysis in
[`../03-architecture/08-physical-gpu-integration.md`](../03-architecture/08-physical-gpu-integration.md)
shows that proxying through the distribution host driver has no callable entry
point from kernel context, so the **chosen mechanism** is for the module to own
the physical GPU and run its real GSP firmware as the resource-manager server
(the module becomes the host-side RM client). That backend is now live: CE-class
copies and their releases are forwarded to the physical copy engine (default
off) with a fault relay, while the emulation path remains the control plane.
What is still open is **GR (kernel) execution**: a forwarded guest launch now
reaches the front-end (ring entry fetched, methods run) but the inline
launch-descriptor install has not been observed and no grid has dispatched —
context validity at dispatch is ruled out. The chain, evidence and live
investigation lines are in
[`03-execution-frontier.md`](03-execution-frontier.md).

**Why:** GR execution is *the* blocker for CUDA kernels, and later for
graphics.

**Plan:** finish the backend in stages: the launch-descriptor install path and
GR kernel submission (see
[`03-execution-frontier.md`](03-execution-frontier.md)); translation of the
remaining guest work; completion and fault relay for every engine class. Measure each stage against a scorecard: correctness,
isolation, latency, maintenance cost across driver versions. Deliverable: a
forwarded `vectorAdd` with verified results. Alternatives (own scheduling
without GSP; vendor vGPU or SR-IOV paths) are analyzed and rejected or
deployment-gated in the integration doc.

**Cost:** weeks, not days; the largest single work item in the project.

### 2.2 Server-owned page tables

**Unknown:** the exact division of labour for page tables after
`COPY_SERVER_RESERVED_PDES`: which ranges the client will touch, when it
expects `RELEASE_ENTRIES`/`RESERVE_ENTRIES`, and how invalidations are
communicated.

**Why:** mappings that only we can create must exist for the guest's own
memory to be reachable by engines.

**Plan:** instrument all vaspace controls (add handlers that log their
parameters), then implement `RESERVE_ENTRIES`/`RELEASE_ENTRIES`/mirroring with
a test harness that maps memory and verifies the guest's expectations by
running its own validation paths (UVM page-tree init exercises this).

**Cost:** days to weeks.

**Reachability resolution note (2026-10-02):** the related IOMMU/DMA
reachability question is closed on the reference host. The physical GPU's
IOMMU group is an identity domain (all-RAM identity) and the mediated device
registers as an emulated-IOMMU VFIO device, so its type-1 container holds
guest DMA mappings privately and no kernel `iommu_domain` exists for it:
`iommu_iova_to_phys` cannot translate guest GPAs. Guest RAM is backed by a
shared file instead, and the module translates GPA->HPA through that file's
page cache. A channel page-table line the GMMU has already walked cannot be
updated at runtime (no TLB-invalidate path exists), so channel page-table
slots for the guest window are pre-built before any engine access. Live proof:
the physical copy engine copied 4 KiB from a real guest page into a module
buffer and back, semaphore released, with the guest process observing the
write and the driver-API smoke green (`ALL OK` 64/64). Wrong-page-table writes
remain the top risk; the division-of-labour question above is unchanged.

### 2.3 Fault semantics

**Partly known:** the backend-side fault observation and relay now exists
(default off — a notifier write plus an event post; the auxiliary event post
can fail with `no_event`). **Still unknown:** the replay/cancel contract as the
*guest* expects it in our virtualized setting (we know the buffer sizes and
that UVM programs them, but not the end-to-end guest-side contract under
forwarding).

**Why:** GPU page faults happen constantly in real CUDA workloads (UVM
oversubscription, migrations); correct handling is required for robustness.

**Plan:** reproduce faults deliberately (map a VA to nothing, touch it from a
forwarded kernel), capture both sides, implement the guest-side replay/cancel
contract; validate with `uvm_fault`-heavy tests (e.g. oversubscribed memory).

**Cost:** weeks.

### 2.4 Version matrix beyond 570.144

**Unknown:** which parts of the observed protocol drift between driver
families (550/560/570/580 — the families with entries under
`config/versions/`), how much is negotiated, and how much must be
configuration.

**Why:** durability is a core promise; users will run other versions
immediately.

**Plan:** run the same init traces across 2–3 driver families in guests,
diff the message/control inventories, and refactor handlers to be
version-keyed where they differ. Deliverable: a capability matrix and a
documented "add a version" workflow.

**Cost:** days per version, once the workflow exists.

### 2.5 Windows guest specifics

**Unknown:** the WDDM-facing differences (ROM/VBIOS expectations, MSI-X
semantics, power management, display topology) that the Windows driver
requires beyond the Linux path.

**Why:** Windows CUDA/3D users are a first-class audience.

**Plan:** bring up a Windows guest against the same device, capture the driver
logs (Windows ETW + our host traces), and add configuration for discovered
requirements.

**Cost:** weeks.

### 2.6 Non-KVM transports

**Unknown:** what differs for Xen/Argo/OpenXT/Qubes OS Air: IRQ delivery
mechanisms, memory pinning, device-model integration points.

**Why:** the security-focused platforms are a major part of the project's
reason for existing.

**Plan:** abstract the transport boundary (IRQ injection, guest-memory access,
device registration) behind a small interface; implement a Xen backend against
it; test in a Xen guest.

**Cost:** weeks.

### 2.7 aarch64 hosts and Grace/GB10

**Unknown:** host-side differences (page sizes, cache attributes, SMMU
behaviour, power domains) and any firmware/ROM differences in the guest path.

**Why:** Grace-Hopper/Grace-Blackwell/GB10 (DGX Spark, Surface-class laptops)
are explicit targets.

**Plan:** make the emulation architecture-independent by construction
(no x86-specific assumptions in the memory model), then bring up on an
aarch64 host with the same test ladder.

**Cost:** weeks per platform.

### 2.8 Multi-tenant performance policy

**Unknown:** best scheduling policy for sharing one GPU across VM tenants
(time slice, MIG partition, hybrid) and its fairness/isolation properties.

**Why:** multi-tenancy is a core value proposition.

**Plan:** build measurement first (per-tenant attribution), then implement
policy as configuration, then audit isolation.

**Cost:** months (it is a research-y area).

### 2.9 Tooling maturity

**Unknown:** how much of the debug tooling (memory scanners, pokes, push
decoders) can be productized without exposing unsafe interfaces.

**Why:** the tooling is currently how we make progress; it must not become a
security liability.

**Plan:** move tooling behind a debug-only build flag/document its threat
model; keep safe, read-only diagnostics in production builds.

**Cost:** days.

---

## 3. Unknown-unknowns (and how we find them)

Unknown-unknowns are discovered by *systematic traversal* and *invariants*,
not by luck. Our methods:

### 3.1 Traversal by construction

* **Enumerate every interface the guest touches.** For each: prove we answer
  it or forward it. Any interface we have not seen is a discovery task, not a
  bug.
* **Drive the guest further, always.** Each new guest-side milestone (probe →
  nvidia-smi → cuInit → context → allocation → kernel) is a traversal step
  that *forces* new interfaces to appear in the trace.
* **Differential testing across versions/hardware.** The same traversal on a
  second driver version or GPU surfaces requirements that are invisible on
  the first.

### 3.2 Invariants that trip on unknowns

We rely on the guest's own self-checks as an oracle:

* UVM asserts on semaphore jumps, bad page tables, impossible capabilities.
* The RM validates structures and will fault on inconsistencies.
* Kernel drivers log "unexpected" states loudly.

Every guest assert/log is a discovered unknown-unknown if we do not already
have it in this document.

### 3.3 Structural instrumentation

* **Trace-first policy:** no handler is written without logging the request
  *and* the reply in a reconstructible form.
* **Reconstruct from dmesg:** the logs must be sufficient to rebuild the
  object tree, memory map and push stream offline. This is how protocol drift
  is detected early, before it becomes a crash.

### 3.4 Fuzzing and negative tests

* Feed the emulation malformed/partial guest actions (truncated elements,
  bad sequence numbers, out-of-range handles) and require graceful
  behaviour — this finds assumptions we did not know we had.
* Feed the *guest* plausible-but-wrong device behaviour in a lab profile and
  verify the guest recovers (drivers do error recovery we must not break).

### 3.5 Failure archaeology

Every failure leaves artifacts. The discipline is: capture host `dmesg`,
guest `dmesg`, guest PCI config, and the emulation's per-RPC log; then write
the *decision tree* that leads to the failure into the debugging docs. The
tree is the artifact that survives personnel changes.

---

## 4. What we need to know to declare "CUDA works"

To make a defensible statement, we need all of:

1. **Interfaces:** every guest→device message for the CUDA path is either
   answered or forwarded, for the supported driver family — and every reply
   whose fields the driver consumes is coherent, not just `SUCCESS`-shaped
   (the reply-fidelity ladder: status-only → coherent zeroed params → coherent
   values for consumed fields → exact values; see
   [`../../userspace/cuda/docs/04-virtualization-and-emulation.md` §6](../../userspace/cuda/docs/04-virtualization-and-emulation.md)).
2. **Memory:** every GPU VA used by work is translatable, and every server-owned
   page-table range is populated correctly.
3. **Execution:** grid launches and copies run on real hardware with correct
   results, repeatedly, with errors surfaced.
4. **Completion:** semaphores, events and faults behave as the guest expects
   (values advance correctly, no lost interrupts, no stuck waits).
5. **Version:** at least two driver families pass, with a documented path for
   more.
6. **Isolation:** a second tenant cannot observe or influence the first.
7. **Performance:** within an agreed factor of native for the supported cases,
   measured with a published methodology.

Until all seven hold, we say "initialization and the driver-API path work;
engine execution is in progress", and we link here. The CUDA-side checklist
behind these gates and the current execution frontier are in
[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md` §9](../../userspace/cuda/docs/03-driver-and-gpu-interface.md);
start at
[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md).

---

## 5. Practical challenges (summary; details in the challenges doc)

| Challenge | Why it is hard | Mitigation |
|---|---|---|
| Real execution | emulating GR is hopeless; forwarding must be correct and isolated | proxy-first, measure, iterate |
| Page tables both sides | guest's and hardware's expectations must agree | study AMAP/HAL definitions; build a page-table validator |
| Latency | RPC round-trips and push translation add overhead | batch, fast-path pure state in the emulation |
| Version drift | protocol and structures evolve | version-keyed handlers, config-driven profiles |
| Security | forwarding must not weaken isolation | IOMMU-only access, per-tenant contexts, audits |
| Debuggability | most bugs are in the guest's head, not ours | trace-first, reconstruct-from-logs policy |
| Test cost | real GPU time and guest boots are slow | scripted ladders, keep artifacts, parallelize |