From 929646875d80408a21c642ca88ef70efbc0d736a Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Tue, 6 Oct 2026 15:18:39 -0400 Subject: docs: add the GR compute-launch frontier page and align execution status across the tree New page docs/04-implementation/03-execution-frontier.md describes the chain that must fire for a guest CUDA launch (guest push -> GPFIFO/front-end -> inline launch-descriptor install -> engine read/dispatch -> completion), where the leading edge is (the descriptor install path), how we know (front-end vs engine-authored observables and their limits), what is exonerated (context validity at dispatch, destination window mapping, fetch alignment, shared-buffer rule, parenting), the ranked issues (verbatim prelude targets, missing descriptor-virtualization control, sentinel hygiene), live investigation lines and labelled theories. Align every page that still said GR execution is blocked on engine-context validity to the current frontier, and add the new page to the README reading order (row 15, later rows renumbered). Docs only; no code. --- docs/01-orientation/01-purpose-and-scope.md | 7 +- .../02-getting-started/01-install-and-first-run.md | 7 +- docs/03-architecture/01-system-overview.md | 7 +- docs/03-architecture/02-guest-stack-and-cudasw.md | 6 +- docs/03-architecture/03-gsp-and-gsp-rm.md | 8 +- docs/03-architecture/04-uvm.md | 13 +- .../07-driver-lifecycle-field-guide.md | 7 + .../03-architecture/08-physical-gpu-integration.md | 4 +- .../10-interaction-and-scheduling.md | 2 +- .../11-physical-backend-guest-wiring.md | 8 + .../01-known-knowns-and-unknowns.md | 21 +- ...practical-challenges-and-adversarial-interop.md | 7 +- docs/04-implementation/03-execution-frontier.md | 220 +++++++++++++++++++++ .../01-performance-correctness-security.md | 8 +- docs/06-portability/hardware-software-matrix.md | 2 +- docs/08-ecosystem/haaeep-and-collaboration.md | 2 +- docs/09-reference/glossary-and-faq.md | 5 +- docs/README.md | 48 +++-- 18 files changed, 327 insertions(+), 55 deletions(-) create mode 100644 docs/04-implementation/03-execution-frontier.md diff --git a/docs/01-orientation/01-purpose-and-scope.md b/docs/01-orientation/01-purpose-and-scope.md index 278ad0d..ef4b258 100644 --- a/docs/01-orientation/01-purpose-and-scope.md +++ b/docs/01-orientation/01-purpose-and-scope.md @@ -136,8 +136,11 @@ document: (the driver-API smoke passes). * **Needs improvement**: engine execution first — the backend is live and CE-class copies and their releases are forwarded to the physical copy engine - (default off) with a fault relay, but CUDA kernel execution is blocked on GR - engine-context validity — then the remaining reply fidelity (some consumed + (default off) with a fault relay, but for CUDA kernel execution the guest's + launch now reaches the front-end and the inline launch-descriptor install has + not been observed (context validity at dispatch is ruled out; see + [`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md)) + — then the remaining reply fidelity (some consumed fields are still generic fallbacks), the memory/page-table model, Windows guest support, non-KVM hypervisors, and performance instrumentation. Each of these is tracked with explicit "known-unknowns" in diff --git a/docs/02-getting-started/01-install-and-first-run.md b/docs/02-getting-started/01-install-and-first-run.md index 7bc12bc..ef50522 100644 --- a/docs/02-getting-started/01-install-and-first-run.md +++ b/docs/02-getting-started/01-install-and-first-run.md @@ -144,8 +144,9 @@ cd cuda-samples/Samples/1_Utilities/deviceQuery && make && ./deviceQuery **Expectation today:** driver probe, `nvidia-smi`, device enumeration, and the full CUDA driver-API smoke (`ALL OK` 64/64, copies matched) work. Engine execution is the remaining gap: CE-class copies are forwarded to the -physical copy engine (default off), while CUDA kernel execution is blocked on -GR engine-context validity. Before drawing conclusions from a CUDA failure, +physical copy engine (default off), while for CUDA kernel execution the +guest's launch now reaches the front-end but the inline launch-descriptor +install has not been observed (context validity at dispatch is ruled out). Before drawing conclusions from a CUDA failure, read the honest status in [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) and the debugger's guide [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md). @@ -185,7 +186,7 @@ Developer debug knobs (module parameters) are documented in | No mdev type for your GPU | [hardware matrix](../06-portability/hardware-software-matrix.md) | | Guest sees wrong PCIe link speed/width | root-port attachment + config-space mirror (debugging doc) | | `nvidia.ko` probe fails in guest | guest dmesg + `lspci -vvv` + [driver lifecycle field guide](../03-architecture/07-driver-lifecycle-field-guide.md) | -| `nvidia-smi` works, CUDA kernel execution does not | expected today: the driver-API smoke passes; CE copies are forwarded (default off) and GR execution is blocked on engine-context validity; see [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) and [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md) | +| `nvidia-smi` works, CUDA kernel execution does not | expected today: the driver-API smoke passes; CE copies are forwarded (default off); the guest's GR launch reaches the front-end but the inline launch-descriptor install has not been observed (context validity at dispatch is ruled out) — see [`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md) and [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) | | VM boot fails or the guest hangs | `scripts/stack_manager.sh cleanup-crash`, then reboot | ## 10. Uninstall diff --git a/docs/03-architecture/01-system-overview.md b/docs/03-architecture/01-system-overview.md index 8205154..71f52f2 100644 --- a/docs/03-architecture/01-system-overview.md +++ b/docs/03-architecture/01-system-overview.md @@ -174,9 +174,10 @@ decompiled and mapped (see `wiki/GSP-570.144W-Firmware-Analysis.md` and completes and the full CUDA driver-API smoke passes (`ALL OK` 64/64, copies matched). The execution backend is live: CE-class copies are forwarded to the physical copy engine (default off) with engine-authored - semaphore releases and a fault relay; kernel execution is blocked on GR - engine-context validity (see - [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)). + semaphore releases and a fault relay; for GR, a forwarded guest launch + reaches the front-end but the inline launch-descriptor install has not been + observed (no grid dispatched; context validity at dispatch is ruled out — + see [`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md)). The CUDA-side contract and the reply-fidelity rules are documented in [`../../userspace/cuda/docs/03-driver-and-gpu-interface.md` §9](../../userspace/cuda/docs/03-driver-and-gpu-interface.md) diff --git a/docs/03-architecture/02-guest-stack-and-cudasw.md b/docs/03-architecture/02-guest-stack-and-cudasw.md index f9fe249..845e00f 100644 --- a/docs/03-architecture/02-guest-stack-and-cudasw.md +++ b/docs/03-architecture/02-guest-stack-and-cudasw.md @@ -95,8 +95,10 @@ module-owned GPU running its real GSP firmware as the resource-manager server (see [`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md) and [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)) — with -CE-class copies forwarded to the physical copy engine (default off) and GR -kernel execution blocked on GR engine-context validity. +CE-class copies forwarded to the physical copy engine (default off); for GR, a +forwarded guest launch reaches the front-end but the inline launch-descriptor +install has not been observed (context validity at dispatch is ruled out; see +[`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md)). For the CUDA userspace side of this stack — its architecture, how it is built and executes, and the exact kernel/GPU interface contract we must satisfy — diff --git a/docs/03-architecture/03-gsp-and-gsp-rm.md b/docs/03-architecture/03-gsp-and-gsp-rm.md index 10e6a73..56e77be 100644 --- a/docs/03-architecture/03-gsp-and-gsp-rm.md +++ b/docs/03-architecture/03-gsp-and-gsp-rm.md @@ -232,9 +232,11 @@ The engine-inventory row was the CUDA blocker at the time of writing: `libcuda` filters the engine list it receives against the class list and expects the async copy engine to be present, because the secondary (async) engine record is otherwise zeroed. That inventory is now complete and context -creation passes; the current frontier is engine execution, where CE-class -copies are forwarded to the physical copy engine (default off) and GR kernel -execution is blocked on GR engine-context validity. See +creation passes; the current frontier is engine execution: CE-class copies are +forwarded to the physical copy engine (default off), and for GR a forwarded +guest launch reaches the front-end but the inline launch-descriptor install has +not been observed (context validity at dispatch is ruled out; see +[`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md)). See [`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md) section 3.7 for the offsets and the consumed fields, and [`../../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../../userspace/cuda/docs/03-driver-and-gpu-interface.md) diff --git a/docs/03-architecture/04-uvm.md b/docs/03-architecture/04-uvm.md index 9396e6a..d4232e0 100644 --- a/docs/03-architecture/04-uvm.md +++ b/docs/03-architecture/04-uvm.md @@ -52,17 +52,20 @@ Each stage has a distinct failure signature; our debugging docs map them. | GPU identity | must match the uuid from RM | ✅ | | CE capabilities | per-engine support bits; UVM *requires* `supported && !grce && sysmem` for CPU_TO_GPU/GPU_TO_CPU channels | 🟡 the caps we answer are accepted, the inventory now includes the async copy engine, and CE copies are forwarded to the physical copy engine (default off) with engine-authored releases | | Channel creation | RM channel objects with GPFIFO/userd/instance parameters | ✅ objects created | -| Pushes executed | UVM's first action on a new channel is an *init push* followed by a semaphore-tracked wait | 🟡 the host decodes method streams and executes memsets, copies and semaphore releases in software (a bridge); CE-class copies and releases are additionally forwarded to the physical copy engine (default off); no GR kernel is forwarded yet (blocked on GR engine-context validity) | +| Pushes executed | UVM's first action on a new channel is an *init push* followed by a semaphore-tracked wait | 🟡 the host decodes method streams and executes memsets, copies and semaphore releases in software (a bridge); CE-class copies and releases are additionally forwarded to the physical copy engine (default off); for GR, a forwarded launch reaches the front-end but the inline launch-descriptor install has not been observed (see [`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md)) | | Page trees | UVM initializes page trees with its own pushes; the *server-owned* ranges must be populated by us | 🟡 | | Fault delivery | fault buffer + notification; replay/cancel semantics | 🟡 backend-side fault observation/relay exists (default off: notifier write + event post); the guest replay/cancel contract is open | | Semantics of completions | tracker values must advance exactly | 🟡 | -**Current frontier (2026-10-02):** context creation and the full CUDA +**Current frontier (2026-10-06):** context creation and the full CUDA driver-API smoke pass (`ALL OK`, copies matched); the execution backend is live and CE-class copies are forwarded to the physical copy engine (default -off) with engine-authored releases and a fault relay. GR kernel execution is -blocked on GR engine-context validity; the remaining UVM work is completion -translation and the guest-side fault replay/cancel contract +off) with engine-authored releases and a fault relay. For GR, a forwarded guest +launch reaches the front-end but the inline launch-descriptor install has not +been observed (context validity at dispatch is ruled out; see +[`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md)). +The remaining UVM work is completion translation and the guest-side fault +replay/cancel contract ([`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)). Failed-context teardown has produced guest kernel oopses in `uvm_pmm_devmem_page_to_va_space` in earlier runs. The CUDA-side diff --git a/docs/03-architecture/07-driver-lifecycle-field-guide.md b/docs/03-architecture/07-driver-lifecycle-field-guide.md index 3a85da6..ff886c6 100644 --- a/docs/03-architecture/07-driver-lifecycle-field-guide.md +++ b/docs/03-architecture/07-driver-lifecycle-field-guide.md @@ -185,6 +185,13 @@ push faults with an uncommitted GR context (`CTXNOTVALID`), and the the designed locked fail-safe (see [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md) §7/§8). +**Update (2026-10-06):** the GR frontier has moved past the context-commit +stage. A forwarded guest CUDA launch now reaches the front-end (ring entry +fetched, methods executed) but the inline launch-descriptor install has not +been observed and no grid has dispatched; context validity at dispatch is ruled +out. See +[`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md). + **Log signatures (guest side):** ``` nvidia-uvm: ... Tracker entry for value N (sema VA 0x...) channel ID ... CE ... diff --git a/docs/03-architecture/08-physical-gpu-integration.md b/docs/03-architecture/08-physical-gpu-integration.md index 6e13e7b..c8005c6 100644 --- a/docs/03-architecture/08-physical-gpu-integration.md +++ b/docs/03-architecture/08-physical-gpu-integration.md @@ -287,9 +287,9 @@ per-context scheduling. Tenant policy and fairness are a later layer on top. | Power-domain lock across the claim window | verified — 3/3 supervised claim/release cycles (2026-10-01; §4.2, §8); latched by default since the G0 bundle (`gsp_phys_repower=0`) | | Real GSP boot by the module | done — full boot through init-done (`0x1001`, `rpc_result=0`) and a clean release with no reboot on every supervised cycle since 2026-10-01 | | Host-side resource-manager client for real objects/channels | done — full static-config reply served; real memory, vaspace, channel-group and channel objects allocated; real CE copies executed on the physical engine | -| Work translation and real submission | CE forwarding and engine-authored release relay implemented default-off and verified live; GR execution open | +| Work translation and real submission | CE forwarding and engine-authored release relay implemented default-off and verified live; for GR, a forwarded guest launch reaches the front-end but the inline launch-descriptor install has not been observed (see [`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md)) | | Completion relay through real hardware | implemented default-off — engine-authored release verified live and forwarded-engine fault observation/relay behind `gsp_phys_fault_relay`, fail-closed to software on any miss; one relay data mismatch under triage | -| CUDA kernel execution (`vectorAdd`) | blocked on GR context validity/execution — the host GR ladder's first push faulted `CTXNOTVALID` and the promote attempt wedged GSP-RM | +| CUDA kernel execution (`vectorAdd`) | open — context validity at dispatch is ruled out; the frontier is the launch descriptor install path (stage 6–7 of [`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md)) | ## 7. Open questions diff --git a/docs/03-architecture/10-interaction-and-scheduling.md b/docs/03-architecture/10-interaction-and-scheduling.md index ffc02d7..617d219 100644 --- a/docs/03-architecture/10-interaction-and-scheduling.md +++ b/docs/03-architecture/10-interaction-and-scheduling.md @@ -22,7 +22,7 @@ to, and where it stands. | Guest doorbell signal (boot, mailbox, channel) | write into BAR0 shadow, observed by the polling service | steering only | [done] | | Guest RPC request | element placed in a guest-sysmem message queue, read by the module over VFIO DMA | 10s of us per round trip | [done] | | RPC reply + interrupt | module writes reply into the queue, signals eventfd, QEMU raises irqfd into KVM, MSI-X lands in the guest | us | [done] | -| Push-buffer work | guest appends GPFIFO entries; the per-channel pump reads, decodes and applies them | per entry, us to low ms | [wip] (CE copies forward default-off; GR execution open) | +| Push-buffer work | guest appends GPFIFO entries; the per-channel pump reads, decodes and applies them | per entry, us to low ms | [wip] (CE copies forward default-off; GR entry fetch live, launch-descriptor install is the frontier — [`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md)) | | Semaphore release to guest memory | module writes the payload word into the resolved guest page; a forwarded CE copy can have the engine author it | us | [done] (relay default off) | | VA-to-PA translation discovery | in-module structure walk, content scan, rescue, pins | us to seconds (scan bounded) | [wip] | | Translation internalization from protocol | derive mappings from RM/UVM objects as they are created | us | [planned] | diff --git a/docs/03-architecture/11-physical-backend-guest-wiring.md b/docs/03-architecture/11-physical-backend-guest-wiring.md index 943dd44..22d06ed 100644 --- a/docs/03-architecture/11-physical-backend-guest-wiring.md +++ b/docs/03-architecture/11-physical-backend-guest-wiring.md @@ -27,6 +27,14 @@ > forwarded-engine fault relay is live green, default off. The P6a trigger > residual is fixed: triggers 6/7 re-translate at trigger time and use fresh > scratch lines. +> +> **Update (2026-10-06): the frontier has moved.** A forwarded guest CUDA +> launch now reaches the front-end (ring entry fetched, methods executed) but +> the inline launch-descriptor (QMD) install has not been observed and no grid +> has dispatched; context validity at dispatch is ruled out. The current wall +> is the form of the forwarded launch method stream — the chain, evidence and +> live investigation lines are in +> [`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md). > The physical backend itself is live: the module-owned > GSP-RM completes initialization on the real TU104, serves the full > static-config reply, and shuts down gracefully. The host resource-manager diff --git a/docs/04-implementation/01-known-knowns-and-unknowns.md b/docs/04-implementation/01-known-knowns-and-unknowns.md index 084afc8..881d608 100644 --- a/docs/04-implementation/01-known-knowns-and-unknowns.md +++ b/docs/04-implementation/01-known-knowns-and-unknowns.md @@ -28,8 +28,10 @@ listed as an unknown-unknown has a *method* by which we will discover it. copies are verified byte-for-byte. The execution backend is live: CE-class copies are forwarded to the physical copy engine (default off) with engine-authored semaphore releases and a fault relay; the remaining frontier is engine - execution — GR kernel execution is blocked on GR engine-context validity - (see §2.1). + execution — a forwarded guest launch now reaches the front-end (entry + fetched, methods run) but the inline launch-descriptor install has not been + observed and no grid has dispatched; context validity at dispatch is ruled + out (see §2.1 and [`03-execution-frontier.md`](03-execution-frontier.md)). * The same guest **boots unmodified**: no guest code, no guest module parameters are required for the initialization path. * The host is unmodified in the standalone configuration (distribution NVIDIA @@ -111,15 +113,20 @@ the physical GPU and run its real GSP firmware as the resource-manager server (the module becomes the host-side RM client). That backend is now live: CE-class copies and their releases are forwarded to the physical copy engine (default off) with a fault relay, while the emulation path remains the control plane. -What is still open is **GR (kernel) execution**, blocked on GR engine-context -validity. +What is still open is **GR (kernel) execution**: a forwarded guest launch now +reaches the front-end (ring entry fetched, methods run) but the inline +launch-descriptor install has not been observed and no grid has dispatched — +context validity at dispatch is ruled out. The chain, evidence and live +investigation lines are in +[`03-execution-frontier.md`](03-execution-frontier.md). **Why:** GR execution is *the* blocker for CUDA kernels, and later for graphics. -**Plan:** finish the backend in stages: GR engine-context validity and kernel -submission; translation of the remaining guest work; completion and fault relay -for every engine class. Measure each stage against a scorecard: correctness, +**Plan:** finish the backend in stages: the launch-descriptor install path and +GR kernel submission (see +[`03-execution-frontier.md`](03-execution-frontier.md)); translation of the +remaining guest work; completion and fault relay for every engine class. Measure each stage against a scorecard: correctness, isolation, latency, maintenance cost across driver versions. Deliverable: a forwarded `vectorAdd` with verified results. Alternatives (own scheduling without GSP; vendor vGPU or SR-IOV paths) are analyzed and rejected or diff --git a/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md b/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md index 4b886b3..93245c4 100644 --- a/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md +++ b/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md @@ -56,8 +56,11 @@ without a methodology is not accepted in this project. physical GPU (see §3). CE-class copies and their semaphore releases are now forwarded to the physical copy engine (default off) with a host-side fault relay, and memsets/copies/releases the host decoder understands still run in - software as a fail-closed correctness bridge. No CUDA kernel is forwarded - yet: GR kernel execution is blocked on GR engine-context validity. + software as a fail-closed correctness bridge. GR (kernel) execution is the + open frontier: a forwarded guest launch reaches the front-end but the inline + launch-descriptor install has not been observed and no grid has dispatched + (context validity at dispatch is ruled out — see + [`03-execution-frontier.md`](03-execution-frontier.md)). ## 3. Forwarding: the options and the chosen mechanism diff --git a/docs/04-implementation/03-execution-frontier.md b/docs/04-implementation/03-execution-frontier.md new file mode 100644 index 0000000..f5546ca --- /dev/null +++ b/docs/04-implementation/03-execution-frontier.md @@ -0,0 +1,220 @@ +# The Execution Frontier: The GR Compute-Launch Chain + +> **Status: 2026-10-06.** This page describes the current leading edge of +> guest-facing execution: the chain that must fire for a stock guest CUDA +> kernel to run on the physical GPU, what we have proven about it, where it +> currently stops, how we know, and the open lines of investigation and +> theories. It is evidence-led: every statement carries either a live +> observation, a decompilation finding, or a clearly labelled theory. +> +> Evidence sources for this page: supervised physical cycles (module-owned +> GSP-RM on the reference TU104), offline firmware emulation/probing, the host +> client's driver sources for the supported version, and decompilation of the +> shipped firmware. Fidelity rules apply: emulator output is +> instruction/memory-level (L0/L1); no timing or device-environment claims are +> made from it. + +--- + +## 1. The chain that should fire + +A guest CUDA launch becomes a **method stream** in a push buffer, referenced by +a **GPFIFO ring entry**, announced by a **doorbell**. The host emulation decodes +it and forwards an equivalent method stream to a real channel owned by the +module's resource-manager session. On the hardware side the **front-end** +(PBDMA) fetches and executes the methods, the launch descriptor (**QMD**) is +materialised and read by the **GR engine**, the grid is dispatched to the SMs, +and completion is published as **semaphores** the guest waits on. + +```mermaid +flowchart TB + A["Guest: CUDA builds launch
program, parameters, QMD, method stream
GPFIFO entry + doorbell"] --> B + B["Host: doorbell trap, decode,
rebuild forward stream,
prewrite QMD, ring/USERD, doorbell"] --> C + C["Front-end: fetch GPFIFO entry
(ring cursor advances)"] --> D + D["Front-end: execute methods
object bind, cache ops, report sem"] --> E + E["Front-end: install inline QMD
into the descriptor buffer"] --> F + F["GR engine: read QMD through
the channel's page tables"] --> G + G["Context switch loads channel context
(FECS/ucode)"] --> H + H["GR: dispatch grid to SMs"] --> I + I["Engine authors release semaphore
+ channel semaphore"] --> J + J["Host polls release, relays to guest;
CUDA wakes"] +``` + +| # | Stage | State (2026-10-06) | +|---|---|---| +| 1 | Guest builds launch + method stream | ✅ observed (stock driver, unmodified) | +| 2 | GPFIFO entry + doorbell | ✅ observed | +| 3 | Host decode + forward stream + QMD prewrite + ring/USERD | ✅ observed (forwarding path) | +| 4 | Front-end fetches the ring entry | ✅ observed (ring cursor advances) | +| 5 | Front-end executes methods | ✅ observed on the ladder channel (object bind, cache ops, report semaphore) | +| 6 | **Inline-QMD install** (descriptor materialised into the QMD buffer) | ✗ **not observed** | +| 7 | **Engine reads the QMD** | ✗ **not observed** | +| 8 | Context switch loads the channel context | ✅ context validity at dispatch is ruled in (instance flag set) — see §4 | +| 9 | GR dispatches the grid to the SMs | ✗ never observed on this stack | +| 10 | Engine authors release + channel semaphore | ✗ absent for forwarded launches | +| 11 | Host relays completion; guest wakes | ✗ guest waits and times out | + +**One-line summary:** the guest, the emulation, the forwarding path and the +front-end are live; the **launch-descriptor install path** (stages 6–7) is the +leading edge, and with it stages 9–11. + +--- + +## 2. Where the leading edge is + +A stock guest program now reaches its kernel launch and submits it through the +forwarding path. On the wire we observe the ring entry consumed and the +front-end methods run — but the launch descriptor is never materialised, the +engine never reads a QMD, no engine-authored completion appears, and the guest +blocks in its synchronisation wait. + +This is **not** a context-validity wall (that was the previous frontier and is +now ruled out at dispatch, §4), and it is **not** a window-mapping wall for the +descriptor destination (that install is accepted). The wall is the **form of +the launch method stream**: what the front-end requires in order to install the +inline descriptor, and what the engine requires in order to read and dispatch +it. + +--- + +## 3. How we know (observables and their limits) + +| Observable | What it proves | What it does **not** prove | +|---|---|---| +| Ring entry consumed; ring cursor advances | The front-end is alive and fetching | Nothing about the engine | +| Method execution on the ladder channel (report semaphore, channel semaphore) | The front-end can execute the method vocabulary | **Not** engine execution — these signals can be front-end-authored | +| Descriptor buffer prewritten by the host with a detectable sentinel | Whether the descriptor was materialised/modified after submission | A read does not modify memory; the sentinel detects installs, not reads | +| Install sentinel unchanged after submit | The inline-install method did not run | It does not say why (mapping, required control method, or ordering) | +| Engine-authored release word absent; completion poll times out | The engine did not complete the launch | It does not say whether the engine started | +| No GR fault recorded | The engine did not raise an error | Silence is not health: a skipped or never-reached launch is silent by construction | +| Channel instance context flag set | The channel is context-valid at dispatch | Nothing about the descriptor path | + +**Consequence:** progress on this frontier must be measured by an +**engine-authored** signal (the release word advancing) — not by front-end +signals, which we have already learned to distinguish. + +--- + +## 4. What is exonerated (do not relitigate) + +* **Context validity at dispatch** — the channel's context is valid when the + launch is submitted (instance flag set). The earlier "engine context never + committed" wall is a historical stage. +* **The descriptor destination window mapping** — the destination window is + installed in the engine's table view at submit (`status=0`). +* **Front-end fetch and control-block alignment** — the entry is consumed with + the host control block at offset 0; index semantics are not applied to a + host-backed userd descriptor. +* **The shared context-buffer one-shot rule** — the refusal seen on one + channel family was traced to a shared per-GPU global buffer already + initialised in the same cycle; it is not the dispatch delta. +* **Parenting/TSG for the fetch** — both parenting variants fetch identically. +* **The copy-engine path** — CE-class work, its releases and its fault relay + are live (default off); the GR path is a separate problem. + +--- + +## 5. The clear issues (ranked) + +**Issue A — the forwarded stream carries a verbatim guest prelude that cannot +complete.** +The forwarded stream includes a verbatim copy of the guest's prelude whose +inline-data writes target addresses that are not present in the engine's view +of the channel's page tables, and the referenced data is not relocated. The +shorter, proven stream shape fires its tail (channel semaphore) but still does +not schedule the descriptor. +*How we know:* the prelude stalls in the forwarded form; the shorter shape's +tail fires; the descriptor-install sentinel never changes in either case. +*If fixed:* the install path is reached and we can observe stage 6 directly. + +**Issue B — the descriptor-virtualization control method is absent.** +The method that arms the inline-descriptor install path (the QMD +virtualization control) is not present in the forwarded stream. +*How we know:* the method does not appear in the submitted stream, and the +install sentinel never changes. +*If fixed:* the front-end should materialise the descriptor — falsifiable in +one cycle by watching the sentinel and the engine-read signal. + +**Issue C — diagnostic hygiene.** +The descriptor sentinel placed at word 0 overlaps real descriptor fields; it +must be moved to an unused word so diagnostics do not perturb what they +measure. + +--- + +## 6. Lines of investigation (live) + +1. **Restore the proven launch shape** as the baseline (the shorter stream that + fires its tail) and **re-add the virtualization control method** — one + variable at a time, with the install sentinel and the engine-read signal as + the readout. +2. **Relocate or drop the verbatim prelude** so the inline-data writes target + addresses present in the engine's view, or replace it with the equivalent + host-side staging the proven shape already uses. +3. **Move the sentinel** and re-run to confirm the readout is clean. +4. **Diff against a known-good reference.** Two ways to obtain one: (a) make + our own ladder dispatch a descriptor built from the same source material + (in-house reference), or (b) capture a working launch of the stock stack on + the same GPU (real method stream + descriptor + engine-visible channel + state) and compare field by field. Either reference converts the remaining + work from inference to differential. +5. **Offline probing** of any rule the front-end or engine is believed to + require, using the firmware emulator/probe tooling before spending a + supervised cycle. + +--- + +## 7. Theories (labelled; not yet proven) + +* **T1 — the virtualization control is the gate.** The front-end will not + materialise an inline descriptor until the virtualization-control method + enables the install path; its absence alone explains the inert descriptor. +* **T2 — materialise-then-read.** The engine only schedules a descriptor that + exists in its own view; an install targeting unmapped destinations aborts + silently rather than faulting. +* **T3 — the verbatim prelude poisons the launch.** Even where the tail shape + works, the prelude's refused writes leave the launch path in a state that + prevents the descriptor from being scheduled. +* **T4 — relocation is the missing transform.** The guest's inline descriptor + is valid, but it is built for the guest's view of memory; the forwarding path + must relocate referenced data (and any embedded pointers) into the channel's + view before install. + +Each theory has a one-cycle falsifier: the install sentinel, the engine-read +signal, and the engine-authored release, in that order. + +--- + +## 8. Claim discipline + +* **Engine-authored release is the only proof of execution.** Front-end + signals (ring cursor, method execution, semaphores that the front-end can + author) are necessary but not sufficient. +* **Silence is not health.** No fault recorded does not mean no fault + occurred; the current GR path runs without an error-reporting object, which + is itself a known observability gap. +* **Emulator verdicts are L0/L1.** They bound behaviour, never timing or + device environment. +* **A wall is closed only by an acceptance signal**, and the acceptance signal + for this frontier is named before the experiment. + +--- + +## 9. Where the rest of the knowledge lives + +| Topic | Document | +|---|---| +| Honest known-knowns / known-unknowns (the execution gate) | [`01-known-knowns-and-unknowns.md`](01-known-knowns-and-unknowns.md) | +| The physical backend lifecycle and guest wiring (init → work → completion → teardown) | [`../03-architecture/11-physical-backend-guest-wiring.md`](../03-architecture/11-physical-backend-guest-wiring.md) | +| Channels, GPFIFO, USERD, runlist, GR and QMD concepts | [`../03-architecture/05-gmmu-amap-hal-gr.md`](../03-architecture/05-gmmu-amap-hal-gr.md) | +| Interaction and scheduling model (guest ↔ host ↔ engine) | [`../03-architecture/10-interaction-and-scheduling.md`](../03-architecture/10-interaction-and-scheduling.md) | +| Supervised-cycle mechanics and hazard discipline | [`../eng-ops.md`](../eng-ops.md), [`../RUNBOOK.md`](../RUNBOOK.md) | +| Offline firmware emulation/debugger | [`../05-operations/03-risc-v-gsp-rm-vm.md`](../05-operations/03-risc-v-gsp-rm-vm.md), [`../05-operations/04-risc-v-gsp-rm-debugger-user-guide.md`](../05-operations/04-risc-v-gsp-rm-debugger-user-guide.md) | + +--- + +## 10. Change log + +| Date | Change | +|---|---| +| 2026-10-06 | Created: the GR compute-launch chain, the leading edge (inline-descriptor install), evidence limits, exonerated walls, ranked issues, live investigation lines and theories. | diff --git a/docs/05-operations/01-performance-correctness-security.md b/docs/05-operations/01-performance-correctness-security.md index 21afd06..2f10b35 100644 --- a/docs/05-operations/01-performance-correctness-security.md +++ b/docs/05-operations/01-performance-correctness-security.md @@ -22,8 +22,12 @@ Rung 5 MULTI-TENANT N VMs, isolation + fairness + no cross-talk (`ALL OK` 64/64, copies matched) — `cuInit`, device queries, context creation, allocation, copies, free and destroy complete, and the copies are verified byte-for-byte. **Rung 3 is partial:** CE-class copies execute on the physical -copy engine (default off) with engine-authored releases and a fault relay, but -CUDA kernel execution is blocked on GR engine-context validity. Before Rung 3 +copy engine (default off) with engine-authored releases and a fault relay; for +GR kernel execution a forwarded guest launch now reaches the front-end but the +inline launch-descriptor install has not been observed (no grid dispatched; +context validity at dispatch is ruled out — +[`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md)). +Before Rung 3 is claimed, the **reply-fidelity gate** applies: a control whose reply fields the driver consumes is not implemented until the reply is coherent (status-only → diff --git a/docs/06-portability/hardware-software-matrix.md b/docs/06-portability/hardware-software-matrix.md index 857d752..0573647 100644 --- a/docs/06-portability/hardware-software-matrix.md +++ b/docs/06-portability/hardware-software-matrix.md @@ -35,7 +35,7 @@ yet and selection is compile-time (see §6). | Guest | Status | |---|---| -| Linux x86_64 (stock `nvidia.ko`, CUDA) | **Experimental** (bring-up; probe, `nvidia-smi` and the driver-API smoke pass, and CE copies are forwarded default-off, but kernel execution is blocked on GR engine-context validity and there is no continuous guest testing — see §6.1) | +| Linux x86_64 (stock `nvidia.ko`, CUDA) | **Experimental** (bring-up; probe, `nvidia-smi` and the driver-API smoke pass, and CE copies are forwarded default-off; for GR, a forwarded guest launch reaches the front-end but the inline launch-descriptor install has not been observed — see [`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md); no continuous guest testing yet — see §6.1) | | Linux aarch64 | **Target** | | Windows x86_64 (WDDM driver, CUDA, 3D) | **Target** | | Other Unix (FreeBSD, ...) | Out of scope until demand | diff --git a/docs/08-ecosystem/haaeep-and-collaboration.md b/docs/08-ecosystem/haaeep-and-collaboration.md index ee847d4..c76a36c 100644 --- a/docs/08-ecosystem/haaeep-and-collaboration.md +++ b/docs/08-ecosystem/haaeep-and-collaboration.md @@ -92,7 +92,7 @@ licensed and version-durable — so it is the component we are building. | Milestone | Depends on | |---|---| -| CUDA execution in a guest (rung 3) | Rung 2 (runtime start) is green — the driver-API smoke test completes end to end (`ALL OK`, payload 64/64). Copy-engine-class work and its releases are forwarded to real hardware with release and fault relay in place (parameters default off); graphics/compute (GR) execution is the open frontier. The interface contract is documented under [`../../userspace/cuda/README.md`](../../userspace/cuda/README.md) (start at [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md)) | +| CUDA execution in a guest (rung 3) | Rung 2 (runtime start) is green — the driver-API smoke test completes end to end (`ALL OK`, payload 64/64). Copy-engine-class work and its releases are forwarded to real hardware with release and fault relay in place (parameters default off); for graphics/compute (GR), a forwarded guest launch reaches the front-end but the inline launch-descriptor install has not been observed (no grid dispatched). The interface contract is documented under [`../../userspace/cuda/README.md`](../../userspace/cuda/README.md) (start at [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md)); the frontier is tracked in [`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md) | | Two-VM coexistence (inference + agent) with isolation | Open-IOV multi-tenancy + platform MAC | | Xen/OpenXT or Qubes Air variant | Open-IOV transport backends | | Attested boot-to-inference chain | TrenchBoot integration | diff --git a/docs/09-reference/glossary-and-faq.md b/docs/09-reference/glossary-and-faq.md index 1668e35..248bf83 100644 --- a/docs/09-reference/glossary-and-faq.md +++ b/docs/09-reference/glossary-and-faq.md @@ -99,7 +99,10 @@ it where available. It is part of the multi-tenancy work. **How fast is it?** No published numbers yet: copy-engine-class work forwarding and engine-authored release + fault relay are implemented but default off, and GR/kernel execution -is still the open frontier; even the initialization path has not been measured +is still the open frontier (a forwarded guest launch reaches the front-end but +the inline launch-descriptor install has not been observed — see +[`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md)); +even the initialization path has not been measured under the project's methodology. Any performance claim will come with the methodology and the measurements together (see [`../05-operations/01-performance-correctness-security.md`](../05-operations/01-performance-correctness-security.md)). diff --git a/docs/README.md b/docs/README.md index 05ee0d6..3955b6b 100644 --- a/docs/README.md +++ b/docs/README.md @@ -111,7 +111,7 @@ A dated snapshot of this status, refreshed on every progress check, is kept at | GSP-RM controls: device/fb/bus tables, CE caps, vaspace, GMMU static info | 🟡 Mostly — the engine inventory now includes the async copy engine and the CE capabilities match the device; reply fidelity for the remaining consumed fields is still being tightened | | Guest `nvidia.ko` probe, `nvidia-smi` | ✅ Working (TU104) | | Guest CUDA driver (`libcuda`) startup | ✅ Driver-API baseline — `cuInit`, device enumeration, `cuCtxCreate`, `cuMemAlloc`, copies, free and destroy complete end to end; the smoke test prints `ALL OK` and the 64-byte payload round-trips exactly (64/64, re-verified 2026-10-02 on the current build). See [`04-implementation/01-known-knowns-and-unknowns.md`](04-implementation/01-known-knowns-and-unknowns.md) and [`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md) | -| CUDA kernel execution | 🟡 Copy-engine-class guest copies and their releases are forwarded to the real hardware with release and fault relay in place (parameters default off); GR (graphics/compute) execution is the open wall — the host GR ladder brings up a GR0 channel with a compute-class object, but its first push faults on a context that was never committed. The device is currently held in the designed locked fail-safe awaiting a supervised reboot. See [`03-architecture/11-physical-backend-guest-wiring.md`](03-architecture/11-physical-backend-guest-wiring.md), [`03-architecture/08-physical-gpu-integration.md`](03-architecture/08-physical-gpu-integration.md) and [`../STATE-CURRENT.md`](../STATE-CURRENT.md) | +| CUDA kernel execution | 🟡 A stock guest program now reaches its kernel launch and submits it through the forwarding path; the front-end fetches the ring entry and executes methods, but the inline launch-descriptor (QMD) install has not been observed and no grid has dispatched yet. Context validity at dispatch is ruled out; the frontier is the launch method-stream form. Copy-engine-class work, its releases and its fault relay remain forwarded (parameters default off). See [`04-implementation/03-execution-frontier.md`](04-implementation/03-execution-frontier.md) and [`03-architecture/11-physical-backend-guest-wiring.md`](03-architecture/11-physical-backend-guest-wiring.md) | | OpenGL / Vulkan | 🔴 Not yet | | Windows guest | 🔴 Not yet | | Multi-tenant isolation (VFs, mdev-per-VM) | 🟡 Designed, partially implemented | @@ -124,12 +124,15 @@ returns and the 64-byte payload round-trips exactly (64/64, re-verified 2026-10-02 on the current build). The full mechanism, with offsets and the decompiled call path, is in [`../userspace/cuda/docs/05-570.144-decompilation.md`](../userspace/cuda/docs/05-570.144-decompilation.md). -The remaining frontier is guest-facing execution: copy-engine-class copies and +The remaining frontier is guest-facing execution. Copy-engine-class copies and their releases are forwarded to real hardware with release relay and fault -relay in place (parameters default off); GR execution is the open wall — the -host GR ladder brings up a GR0 channel with a compute-class object, but its -first push faults on an engine context that was never committed. The device is -currently held in the designed locked fail-safe awaiting a supervised reboot. +relay in place (parameters default off). For GR (graphics/compute), the guest +now reaches and submits a real kernel launch: the ring entry is fetched and +front-end methods execute, but the inline launch-descriptor install has not +been observed and no grid has dispatched yet. The wall is the form of the +forwarded launch method stream — not context validity, which is ruled out at +dispatch. The chain, the evidence and the live investigation lines are in +[`04-implementation/03-execution-frontier.md`](04-implementation/03-execution-frontier.md). The backend, a module-owned GPU running its real GSP firmware as the resource-manager server, is specified in [`03-architecture/08-physical-gpu-integration.md`](03-architecture/08-physical-gpu-integration.md); @@ -169,19 +172,20 @@ Read in order if you are new. Jump directly if you know what you need. | 12 | [Driver lifecycle field guide](03-architecture/07-driver-lifecycle-field-guide.md) | developers, debuggers | 30 min | | 13 | [Physical GPU integration](03-architecture/08-physical-gpu-integration.md) | developers, architects | 20 min | | 14 | [Physical backend guest wiring](03-architecture/11-physical-backend-guest-wiring.md) | developers, architects | 25 min | -| 15 | [Host VF acceleration — future goal (deferred)](03-architecture/12-host-vf-acceleration-future-goal.md) | architects, planners | 10 min | -| 16 | [Known-knowns and known-unknowns](04-implementation/01-known-knowns-and-unknowns.md) | everyone technical | 25 min | -| 17 | [Practical challenges and adversarial interoperability](04-implementation/02-practical-challenges-and-adversarial-interop.md) | developers, stakeholders | 20 min | -| 18 | [Performance, correctness, security](05-operations/01-performance-correctness-security.md) | developers, auditors | 25 min | -| 19 | [Debugging and maintenance workflows](05-operations/02-debugging-and-maintenance-workflows.md) | developers, maintainers | 25 min | -| 20 | [GSP in a RISC-V VM (risc-v-gsp-rm-debugger, experimental)](05-operations/03-risc-v-gsp-rm-vm.md) — implemented, documented and tested ([audit](../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md); [value demos](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md)) | developers, analysts | 15 min | -| 21 | [risc-v-gsp-rm-debugger user guide (experimental)](05-operations/04-risc-v-gsp-rm-debugger-user-guide.md) — implemented, documented and tested ([audit](../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md); [value demos](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md)) | developers, analysts | 15 min | -| 22 | [Defensive Operations Runbook](RUNBOOK.md) | maintainers, operators | 20 min | -| 23 | [Engineering operations](eng-ops.md) — tool admission and workflows (Part I) plus live physical-backend operations (Part II) | maintainers, developers | 20 min | -| 24 | [Hardware and software matrix](06-portability/hardware-software-matrix.md) | developers, integrators | 20 min | -| 25 | [Developer onboarding and standards](07-developers/onboarding-and-standards.md) | contributors | 20 min | -| 26 | [HAAEEP and collaboration](08-ecosystem/haaeep-and-collaboration.md) | stakeholders, partners | 20 min | -| 27 | [Glossary and FAQ](09-reference/glossary-and-faq.md) | everyone | reference | +| 15 | [The execution frontier: the GR compute-launch chain](04-implementation/03-execution-frontier.md) | developers, debuggers | 15 min | +| 16 | [Host VF acceleration — future goal (deferred)](03-architecture/12-host-vf-acceleration-future-goal.md) | architects, planners | 10 min | +| 17 | [Known-knowns and known-unknowns](04-implementation/01-known-knowns-and-unknowns.md) | everyone technical | 25 min | +| 18 | [Practical challenges and adversarial interoperability](04-implementation/02-practical-challenges-and-adversarial-interop.md) | developers, stakeholders | 20 min | +| 19 | [Performance, correctness, security](05-operations/01-performance-correctness-security.md) | developers, auditors | 25 min | +| 20 | [Debugging and maintenance workflows](05-operations/02-debugging-and-maintenance-workflows.md) | developers, maintainers | 25 min | +| 21 | [GSP in a RISC-V VM (risc-v-gsp-rm-debugger, experimental)](05-operations/03-risc-v-gsp-rm-vm.md) — implemented, documented and tested ([audit](../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md); [value demos](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md)) | developers, analysts | 15 min | +| 22 | [risc-v-gsp-rm-debugger user guide (experimental)](05-operations/04-risc-v-gsp-rm-debugger-user-guide.md) — implemented, documented and tested ([audit](../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md); [value demos](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md)) | developers, analysts | 15 min | +| 23 | [Defensive Operations Runbook](RUNBOOK.md) | maintainers, operators | 20 min | +| 24 | [Engineering operations](eng-ops.md) — tool admission and workflows (Part I) plus live physical-backend operations (Part II) | maintainers, developers | 20 min | +| 25 | [Hardware and software matrix](06-portability/hardware-software-matrix.md) | developers, integrators | 20 min | +| 26 | [Developer onboarding and standards](07-developers/onboarding-and-standards.md) | contributors | 20 min | +| 27 | [HAAEEP and collaboration](08-ecosystem/haaeep-and-collaboration.md) | stakeholders, partners | 20 min | +| 28 | [Glossary and FAQ](09-reference/glossary-and-faq.md) | everyone | reference | -Row 15 is a **deferred future goal**: host NVIDIA-driver acceleration over a + + +Row 16 is a **deferred future goal**: host NVIDIA-driver acceleration over a virtual function. The feasibility verdict and a first design sketch are recorded there for planning; it is not on the active critical path (guest CUDA execution, multi-VM, 3D, performance come first). -- cgit v1.2.3