From ae534205925529fc06e0539218f0ddf9fe5f835a Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Tue, 20 Oct 2026 05:22:19 -0400 Subject: site: 2026-10-20 status refresh - guest CUDA kernel execution working --- wiki/Execution-Frontier.md | 177 ++++++++++++++++++++++++--------------------- 1 file changed, 95 insertions(+), 82 deletions(-) (limited to 'wiki/Execution-Frontier.md') diff --git a/wiki/Execution-Frontier.md b/wiki/Execution-Frontier.md index fe21fda..2d9d80c 100644 --- a/wiki/Execution-Frontier.md +++ b/wiki/Execution-Frontier.md @@ -1,15 +1,16 @@ # The Execution Frontier -> **What this page is.** The current leading edge of guest-facing execution: -> the chain that must fire for a stock guest CUDA kernel to run on the -> physical GPU, what has been proven about it, where it stops, how we know, -> and the open lines of investigation and theories. It is evidence-led: every -> statement carries a live observation, a decompilation finding, or a clearly -> labelled theory. Condensed from +> **What this page is.** The state of guest-facing execution: the chain that +> must fire for a stock guest CUDA kernel to run on the physical GPU, what has +> been proven about it, how we know, and where the leading edge is now. The +> chain fires end to end: verified guest CUDA clients execute on the physical +> GPU and wake up correctly; the leading edge is now breadth and hardening. It +> is evidence-led: every statement carries a live observation, a +> decompilation finding, or a clearly labelled theory. Condensed from > [`../docs/04-implementation/03-execution-frontier.md`](../docs/04-implementation/03-execution-frontier.md); > the docs tree wins on disagreement. -Status: 2026-10-12 (source page dated 2026-10-06), tree `master @ 2f066727`. +Status: 2026-10-20, tree `master @ 16291c72`. Tags: `[live]` observed; `[wip]` partial; `[wall]` current blocker; theories are labelled as such. @@ -45,46 +46,62 @@ guest: builds launch + method stream | 3 | Host decode + forward stream + QMD prewrite + ring/USERD | `[live]` observed | | 4 | Front-end fetches the ring entry | `[live]` observed (ring cursor advances) | | 5 | Front-end executes methods | `[live]` observed (object bind, cache ops, report semaphore) | -| 6 | **Inline-QMD install** (descriptor materialised) | `[wall]` not observed | -| 7 | **Engine reads the QMD** | `[wall]` not observed | +| 6 | **Inline-QMD install** (descriptor materialised) | `[live]` observed | +| 7 | **Engine reads the QMD** | `[live]` observed | | 8 | Context switch loads the channel context | `[live]` context validity at dispatch is ruled in | -| 9 | GR dispatches the grid to the SMs | never observed on this stack | -| 10 | Engine authors release + channel semaphore | absent for forwarded launches | -| 11 | Host relays completion; guest wakes | the guest waits and times out | +| 9 | GR dispatches the grid to the SMs | `[live]` observed (the guest's results verify) | +| 10 | Engine authors release + channel semaphore | `[live]` engine-authored releases relay to the guest | +| 11 | Host relays completion; guest wakes | `[live]` the guest's synchronisation waits resolve | -**One-line summary:** the guest, the emulation, the forwarding path and the -front-end are live; the **launch-descriptor install path** (stages 6–7) is the -leading edge, and with it stages 9–11. +**One-line summary:** the whole chain is live; the leading edge is now breadth +and hardening — the large-buffer staged-transfer class, reproducibility across +boots, and per-guest execution isolation. ## 2. Where the leading edge is -A stock guest program now reaches its kernel launch and submits it through the -forwarding path. On the wire the ring entry is consumed and the front-end -methods run — but the launch descriptor is never materialised, the engine -never reads a QMD, no engine-authored completion appears, and the guest blocks -in its synchronisation wait. - -This is **not** a context-validity wall (ruled out at dispatch) and **not** a -window-mapping wall for the descriptor destination (that install is accepted). -The wall is the **form of the launch method stream**: what the front-end -requires in order to install the inline descriptor, and what the engine -requires in order to read and dispatch it. +A stock, unmodified guest CUDA program executes its kernels on the physical +GPU and verifies its results. On fresh boots the vector-add client +(`victory_mail`) prints `VADD n=32768 bad=0` + `STATUS OK`, the +matrix-multiply client (`victory_matmul`) prints `MATMUL n=256 bad=0` + +`STATUS OK`, and a transformer kernel prints `dh=256 seq=64 bad=0`, +`FFN4 layers=4 bad=0` + `STATUS OK`. The executing engine is the physical GPU +under the module's own real GSP-RM session; the host CUDA bridge is not part +of this path. Engine-authored completions relay to the guest and its +synchronisation waits resolve. + +The leading edge has moved from the launch chain itself — it fires — to +breadth and hardening around it: + +* **The large-buffer staged-transfer class.** Multi-megabyte staged transfers + between guest memory and device memory are under repair: some large + multi-chunk clients still mis-verify. +* **Reproducibility across boots.** A boot-poisoning defect was root-caused + and fixed: under the driver's status-queue mode the emulation read the wrong + queue cursor, so under load the reply ring could desynchronise and the first + CUDA client would fail (the user-visible `STATUS FAIL`-then-freeze report). + Re-measurement across fresh boots is in progress. +* **Per-guest execution isolation.** Multiple mediated devices with different + emulated VRAM sizes and multiple guests already coexist at the device level; + per-guest execution isolation for multiple simultaneous CUDA guests is in + progress. + +No breadth beyond these is claimed: OpenGL/Vulkan, Windows guests, migration +and performance numbers remain planned. ## 3. How we know (observables and their limits) | Observable | What it proves | What it does not prove | |---|---|---| -| Ring entry consumed; ring cursor advances | the front-end is alive and fetching | nothing about the engine | -| Method execution on the ladder channel | the front-end can execute the method vocabulary | **not** engine execution — these signals can be front-end-authored | -| Descriptor buffer prewritten with a detectable sentinel | whether the descriptor was materialised after submission | a read does not modify memory; the sentinel detects installs, not reads | -| Install sentinel unchanged after submit | the inline-install method did not run | it does not say why | -| Engine-authored release word absent; poll times out | the engine did not complete the launch | it does not say whether the engine started | +| Client verification output (`bad=0` + `STATUS OK`) | the guest executed its kernel and verified its results | by itself, which engine produced the results | +| Engine-authored release relayed to the guest | the engine executed the launch and the relay works | nothing about other launches | +| The guest's synchronisation wait resolving | completion reached the guest through the relay | nothing about other launches | +| Ring entry consumed; ring cursor advances | the front-end is alive and fetching | **not** engine execution — front-end signals are necessary but not sufficient | +| Reproduction across fresh boots | the path is not a lucky boot | no "every boot" claim yet while re-measurement is in progress | | No GR fault recorded | the engine did not raise this error | silence is not health: a skipped launch is silent by construction | -| Channel instance context flag set | the channel is context-valid at dispatch | nothing about the descriptor path | -**Consequence:** progress on this frontier must be measured by an -**engine-authored** signal (the release word advancing) — not by front-end -signals, which we have already learned to distinguish. +**Consequence:** acceptance is verified guest results plus the engine-authored +release; a green run counts as progress only when it reproduces across fresh +boots. ## 4. What is exonerated (do not relitigate) @@ -98,68 +115,64 @@ signals, which we have already learned to distinguish. already initialised in the same cycle; not the dispatch delta. * Parenting/TSG for the fetch — both variants fetch identically. * The copy-engine path — CE-class work, releases and fault relay are live - (default off); the GR path is a separate problem. + (default off); the GR path is separate. +* The descriptor-gate line of investigation (the old T1–T4 theories) — the + inline-descriptor install, the engine read, the dispatch, the + engine-authored release and the guest wake-up are all observed; closed by + observation. +* The old boot-poisoning defect — root-caused to the status-queue consumption + cursor and fixed. ## 5. The clear issues (ranked) -**Issue A — the forwarded stream carries a verbatim guest prelude that cannot -complete.** The forwarded stream includes a verbatim copy of the guest's -prelude whose inline-data writes target addresses not present in the engine's -view of the channel's page tables, and the referenced data is not relocated. -The shorter, proven stream shape fires its tail (channel semaphore) but still -does not schedule the descriptor. +**Issue A — the large-buffer staged-transfer class.** Multi-megabyte staged +transfers between guest memory and device memory are under repair: some large +multi-chunk clients still mis-verify. -**Issue B — the descriptor-virtualization control method is absent.** The -method that arms the inline-descriptor install path is not present in the -forwarded stream. Fixing it should materialise the descriptor — falsifiable in -one cycle by watching the sentinel and the engine-read signal. +**Issue B — reproducibility across boots (re-measurement).** The +boot-poisoning defect (wrong status-queue consumption cursor under the +driver's status-queue mode; the reply ring could desynchronise under load, the +first client would fail) was root-caused and fixed. Re-measurement across +fresh boots is in progress; no "every boot" claim is made until it completes. -**Issue C — diagnostic hygiene.** The sentinel placed at word 0 overlaps real -descriptor fields; it must be moved to an unused word so diagnostics do not -perturb what they measure. +**Issue C — per-guest execution isolation.** Multiple mediated devices with +different emulated VRAM sizes and multiple guests coexist at the device level; +per-guest execution isolation for simultaneous CUDA guests is in progress. ## 6. Lines of investigation (live) -1. **Restore the proven launch shape** as the baseline and **re-add the - virtualization control method** — one variable at a time, with the install - sentinel and the engine-read signal as the readout. -2. **Relocate or drop the verbatim prelude** so inline-data writes target - addresses present in the engine's view, or replace it with the host-side - staging the proven shape already uses. -3. **Move the sentinel** and re-run to confirm the readout is clean. -4. **Diff against a known-good reference** — our own ladder dispatching a - descriptor built from the same source material, or a captured working - launch of the stock stack on the same GPU — converting the remaining work - from inference to differential. -5. **Offline probing** of any rule the front-end or engine is believed to - require, using the firmware debugger before spending a supervised cycle - ([`Riscv-GSP-RM-Debugger.md`](Riscv-GSP-RM-Debugger.md)). - -## 7. Theories (labelled; not yet proven) - -* **T1 — the virtualization control is the gate.** The front-end will not - materialise an inline descriptor until the control method enables the - install path. -* **T2 — materialise-then-read.** The engine only schedules a descriptor that - exists in its own view; an install targeting unmapped destinations aborts - silently rather than faulting. -* **T3 — the verbatim prelude poisons the launch.** Its refused writes leave - the launch path unable to schedule the descriptor. -* **T4 — relocation is the missing transform.** The guest's descriptor is - valid for the guest's view of memory; the forwarding path must relocate - referenced data into the channel's view before install. - -Each theory has a one-cycle falsifier: the install sentinel, the engine-read -signal, and the engine-authored release, in that order. +1. **Repair the large-buffer staged-transfer class** — the three green clients + stay the regression battery and must remain green throughout. +2. **Complete the fresh-boot re-measurement** — demonstrate the fixed boot + path across fresh boots before claiming boot-to-boot reproducibility. +3. **Per-guest execution isolation** — build from the device-level coexistence + to isolation for simultaneous CUDA guests. +4. **Hold the acceptance battery** — vector-add, matrix-multiply and the + transformer kernel; each entry is accepted only on its own verification + output (`bad=0` + `STATUS OK`). + +## 7. Theories, resolved + +The descriptor-gate theories (T1–T4) are resolved and are not kept as open: +the inline-descriptor install, the engine's QMD read, the grid dispatch, the +engine-authored release and the guest wake-up are all directly observed, so +the launch chain they questioned is proven end to end. The remaining work is +engineering (large-buffer transfers, boot-to-boot reproducibility, per-guest +isolation), each with an acceptance signal named before the work. ## 8. Claim discipline -* **Engine-authored release is the only proof of execution.** Front-end - signals are necessary but not sufficient. +* **Engine-authored release is the acceptance signal for execution** — now + observed end to end; the guest's own verification (`bad=0` + `STATUS OK`) + is the guest-side acceptance signal. +* **Front-end signals are necessary but not sufficient.** Ring cursor and + front-end method execution can be front-end-authored; they never substitute + for the engine-authored release. * **Silence is not health.** No fault recorded does not mean no fault occurred. * **Emulator verdicts are instruction/memory-level.** They bound behaviour, never timing or device environment. +* **A green run counts only when it reproduces across fresh boots.** * **A wall is closed only by an acceptance signal**, named before the experiment. -- cgit v1.2.3