# The Execution Frontier > **What this page is.** The state of guest-facing execution: the chain that > must fire for a stock guest CUDA kernel to run on the physical GPU, what has > been proven about it, how we know, and where the leading edge is now. The > chain fires end to end: verified guest CUDA clients execute on the physical > GPU and wake up correctly; the leading edge is now breadth and hardening. It > is evidence-led: every statement carries a live observation, a > decompilation finding, or a clearly labelled theory. Condensed from > [`../docs/04-implementation/03-execution-frontier.md`](../docs/04-implementation/03-execution-frontier.md); > the docs tree wins on disagreement. Status: 2026-10-20, tree `master @ 16291c72`. Tags: `[live]` observed; `[wip]` partial; `[wall]` current blocker; theories are labelled as such. ## 1. The chain that should fire A guest CUDA launch becomes a **method stream** in a push buffer, referenced by a **GPFIFO ring entry**, announced by a **doorbell**. The host emulation decodes it and forwards an equivalent method stream to a real channel owned by the module's resource-manager session. The hardware **front-end** fetches and executes the methods, the launch descriptor (**QMD**) is materialised and read by the **GR engine**, the grid is dispatched to the SMs, and completion is published as **semaphores** the guest waits on. ```text guest: builds launch + method stream GPFIFO entry + doorbell -> host: doorbell trap, decode, rebuild forward stream, prewrite QMD, ring/USERD, doorbell -> front-end: fetch GPFIFO entry (ring cursor advances) -> front-end: execute methods (object bind, cache ops, report sem) -> front-end: install inline QMD into the descriptor buffer -> GR engine: read QMD through the channel's page tables -> context switch loads the channel context (FECS/ucode) -> GR: dispatch grid to the SMs -> engine authors release semaphore + channel semaphore -> host polls release, relays to guest; CUDA wakes ``` | # | Stage | State | |---|---|---| | 1 | Guest builds launch + method stream | `[live]` observed (stock driver, unmodified) | | 2 | GPFIFO entry + doorbell | `[live]` observed | | 3 | Host decode + forward stream + QMD prewrite + ring/USERD | `[live]` observed | | 4 | Front-end fetches the ring entry | `[live]` observed (ring cursor advances) | | 5 | Front-end executes methods | `[live]` observed (object bind, cache ops, report semaphore) | | 6 | **Inline-QMD install** (descriptor materialised) | `[live]` observed | | 7 | **Engine reads the QMD** | `[live]` observed | | 8 | Context switch loads the channel context | `[live]` context validity at dispatch is ruled in | | 9 | GR dispatches the grid to the SMs | `[live]` observed (the guest's results verify) | | 10 | Engine authors release + channel semaphore | `[live]` engine-authored releases relay to the guest | | 11 | Host relays completion; guest wakes | `[live]` the guest's synchronisation waits resolve | **One-line summary:** the whole chain is live; the leading edge is now breadth and hardening — the large-buffer staged-transfer class, reproducibility across boots, and per-guest execution isolation. ## 2. Where the leading edge is A stock, unmodified guest CUDA program executes its kernels on the physical GPU and verifies its results. On fresh boots the vector-add client (`victory_mail`) prints `VADD n=32768 bad=0` + `STATUS OK`, the matrix-multiply client (`victory_matmul`) prints `MATMUL n=256 bad=0` + `STATUS OK`, and a transformer kernel prints `dh=256 seq=64 bad=0`, `FFN4 layers=4 bad=0` + `STATUS OK`. The executing engine is the physical GPU under the module's own real GSP-RM session; the host CUDA bridge is not part of this path. Engine-authored completions relay to the guest and its synchronisation waits resolve. The leading edge has moved from the launch chain itself — it fires — to breadth and hardening around it: * **The large-buffer staged-transfer class.** Multi-megabyte staged transfers between guest memory and device memory are under repair: some large multi-chunk clients still mis-verify. * **Reproducibility across boots.** A boot-poisoning defect was root-caused and fixed: under the driver's status-queue mode the emulation read the wrong queue cursor, so under load the reply ring could desynchronise and the first CUDA client would fail (the user-visible `STATUS FAIL`-then-freeze report). Re-measurement across fresh boots is in progress. * **Per-guest execution isolation.** Multiple mediated devices with different emulated VRAM sizes and multiple guests already coexist at the device level; per-guest execution isolation for multiple simultaneous CUDA guests is in progress. No breadth beyond these is claimed: OpenGL/Vulkan, Windows guests, migration and performance numbers remain planned. ## 3. How we know (observables and their limits) | Observable | What it proves | What it does not prove | |---|---|---| | Client verification output (`bad=0` + `STATUS OK`) | the guest executed its kernel and verified its results | by itself, which engine produced the results | | Engine-authored release relayed to the guest | the engine executed the launch and the relay works | nothing about other launches | | The guest's synchronisation wait resolving | completion reached the guest through the relay | nothing about other launches | | Ring entry consumed; ring cursor advances | the front-end is alive and fetching | **not** engine execution — front-end signals are necessary but not sufficient | | Reproduction across fresh boots | the path is not a lucky boot | no "every boot" claim yet while re-measurement is in progress | | No GR fault recorded | the engine did not raise this error | silence is not health: a skipped launch is silent by construction | **Consequence:** acceptance is verified guest results plus the engine-authored release; a green run counts as progress only when it reproduces across fresh boots. ## 4. What is exonerated (do not relitigate) * Context validity at dispatch — the channel's context is valid when the launch is submitted (instance flag set). * The descriptor destination window mapping — installed in the engine's table view at submit. * Front-end fetch and control-block alignment — the entry is consumed with the host control block at offset 0. * The shared context-buffer one-shot rule — traced to a shared per-GPU buffer already initialised in the same cycle; not the dispatch delta. * Parenting/TSG for the fetch — both variants fetch identically. * The copy-engine path — CE-class work, releases and fault relay are live (default off); the GR path is separate. * The descriptor-gate line of investigation (the old T1–T4 theories) — the inline-descriptor install, the engine read, the dispatch, the engine-authored release and the guest wake-up are all observed; closed by observation. * The old boot-poisoning defect — root-caused to the status-queue consumption cursor and fixed. ## 5. The clear issues (ranked) **Issue A — the large-buffer staged-transfer class.** Multi-megabyte staged transfers between guest memory and device memory are under repair: some large multi-chunk clients still mis-verify. **Issue B — reproducibility across boots (re-measurement).** The boot-poisoning defect (wrong status-queue consumption cursor under the driver's status-queue mode; the reply ring could desynchronise under load, the first client would fail) was root-caused and fixed. Re-measurement across fresh boots is in progress; no "every boot" claim is made until it completes. **Issue C — per-guest execution isolation.** Multiple mediated devices with different emulated VRAM sizes and multiple guests coexist at the device level; per-guest execution isolation for simultaneous CUDA guests is in progress. ## 6. Lines of investigation (live) 1. **Repair the large-buffer staged-transfer class** — the three green clients stay the regression battery and must remain green throughout. 2. **Complete the fresh-boot re-measurement** — demonstrate the fixed boot path across fresh boots before claiming boot-to-boot reproducibility. 3. **Per-guest execution isolation** — build from the device-level coexistence to isolation for simultaneous CUDA guests. 4. **Hold the acceptance battery** — vector-add, matrix-multiply and the transformer kernel; each entry is accepted only on its own verification output (`bad=0` + `STATUS OK`). ## 7. Theories, resolved The descriptor-gate theories (T1–T4) are resolved and are not kept as open: the inline-descriptor install, the engine's QMD read, the grid dispatch, the engine-authored release and the guest wake-up are all directly observed, so the launch chain they questioned is proven end to end. The remaining work is engineering (large-buffer transfers, boot-to-boot reproducibility, per-guest isolation), each with an acceptance signal named before the work. ## 8. Claim discipline * **Engine-authored release is the acceptance signal for execution** — now observed end to end; the guest's own verification (`bad=0` + `STATUS OK`) is the guest-side acceptance signal. * **Front-end signals are necessary but not sufficient.** Ring cursor and front-end method execution can be front-end-authored; they never substitute for the engine-authored release. * **Silence is not health.** No fault recorded does not mean no fault occurred. * **Emulator verdicts are instruction/memory-level.** They bound behaviour, never timing or device environment. * **A green run counts only when it reproduces across fresh boots.** * **A wall is closed only by an acceptance signal**, named before the experiment. ## 9. Related pages * [`Project-Status.md`](Project-Status.md) — the project-level ladder. * [`CUDA-Userspace-Driver.md`](CUDA-Userspace-Driver.md) — the guest-side client path. * [`Physical-GPU-Backend.md`](Physical-GPU-Backend.md) — the backend the work is forwarded to. * [`Execution-Plane.md`](Execution-Plane.md) — the execution layer's design. * [`Introduction-to-GMMU.md`](Introduction-to-GMMU.md) — page tables, QMDs and the engines. * [`../docs/04-implementation/03-execution-frontier.md`](../docs/04-implementation/03-execution-frontier.md) — the canonical, continuously-updated version with the full evidence tables.