site: 2026-10-20 status refresh - guest CUDA kernel execution working

1 files changed, 131 insertions, 101 deletions

diff --git a/docs/04-implementation/03-execution-frontier.md b/docs/04-implementation/03-execution-frontier.md
index f5546ca..a8e03ee 100644
--- a/docs/04-implementation/03-execution-frontier.md
+++ b/docs/04-implementation/03-execution-frontier.md
@@ -1,11 +1,17 @@
 # The Execution Frontier: The GR Compute-Launch Chain
 
-> **Status: 2026-10-06.** This page describes the current leading edge of
+> **Status: 2026-10-20.** This page describes the current state of
 > guest-facing execution: the chain that must fire for a stock guest CUDA
-> kernel to run on the physical GPU, what we have proven about it, where it
-> currently stops, how we know, and the open lines of investigation and
-> theories. It is evidence-led: every statement carries either a live
-> observation, a decompilation finding, or a clearly labelled theory.
+> kernel to run on the physical GPU, what we have proven about it, how we
+> know, and where the leading edge is now. The chain fires end to end:
+> verified guest CUDA clients (vector-add, matrix-multiply and a transformer
+> kernel) execute on the physical GPU, their results verify, and the
+> engine-authored completions relay back to the guest. The leading edge has
+> moved from "can a forwarded launch execute at all?" to breadth and
+> hardening — the large-buffer staged-transfer class, reproducibility across
+> boots, and per-guest execution isolation. It is evidence-led: every
+> statement carries either a live observation, a decompilation finding, or a
+> clearly labelled theory.
 >
 > Evidence sources for this page: supervised physical cycles (module-owned
 > GSP-RM on the reference TU104), offline firmware emulation/probing, the host
@@ -40,40 +46,59 @@ flowchart TB
   J["Host polls release, relays to guest;<br/>CUDA wakes"]
 ```
 
-| # | Stage | State (2026-10-06) |
+| # | Stage | State (2026-10-20) |
 |---|---|---|
 | 1 | Guest builds launch + method stream | ✅ observed (stock driver, unmodified) |
 | 2 | GPFIFO entry + doorbell | ✅ observed |
 | 3 | Host decode + forward stream + QMD prewrite + ring/USERD | ✅ observed (forwarding path) |
 | 4 | Front-end fetches the ring entry | ✅ observed (ring cursor advances) |
-| 5 | Front-end executes methods | ✅ observed on the ladder channel (object bind, cache ops, report semaphore) |
-| 6 | **Inline-QMD install** (descriptor materialised into the QMD buffer) | ✗ **not observed** |
-| 7 | **Engine reads the QMD** | ✗ **not observed** |
-| 8 | Context switch loads the channel context | ✅ context validity at dispatch is ruled in (instance flag set) — see §4 |
-| 9 | GR dispatches the grid to the SMs | ✗ never observed on this stack |
-| 10 | Engine authors release + channel semaphore | ✗ absent for forwarded launches |
-| 11 | Host relays completion; guest wakes | ✗ guest waits and times out |
-
-**One-line summary:** the guest, the emulation, the forwarding path and the
-front-end are live; the **launch-descriptor install path** (stages 6–7) is the
-leading edge, and with it stages 9–11.
+| 5 | Front-end executes methods | ✅ observed (object bind, cache ops, report semaphore) |
+| 6 | Inline-QMD install (descriptor materialised into the QMD buffer) | ✅ observed |
+| 7 | Engine reads the QMD | ✅ observed |
+| 8 | Context switch loads the channel context | ✅ context validity at dispatch is ruled in — see §4 |
+| 9 | GR dispatches the grid to the SMs | ✅ observed (the guest's results verify) |
+| 10 | Engine authors release + channel semaphore | ✅ observed — engine-authored releases relay to the guest |
+| 11 | Host relays completion; guest wakes | ✅ observed — the guest's synchronisation waits resolve |
+
+**One-line summary:** the whole chain is live — verified guest CUDA clients
+execute on the physical GPU and wake up correctly. The leading edge is no
+longer the chain itself but its breadth and hardening: the large-buffer
+staged-transfer class, reproducibility across boots, and per-guest execution
+isolation.
 
 ---
 
 ## 2. Where the leading edge is
 
-A stock guest program now reaches its kernel launch and submits it through the
-forwarding path. On the wire we observe the ring entry consumed and the
-front-end methods run — but the launch descriptor is never materialised, the
-engine never reads a QMD, no engine-authored completion appears, and the guest
-blocks in its synchronisation wait.
-
-This is **not** a context-validity wall (that was the previous frontier and is
-now ruled out at dispatch, §4), and it is **not** a window-mapping wall for the
-descriptor destination (that install is accepted). The wall is the **form of
-the launch method stream**: what the front-end requires in order to install the
-inline descriptor, and what the engine requires in order to read and dispatch
-it.
+A stock, unmodified guest CUDA program now executes its kernels on the
+physical GPU and verifies its results. On fresh boots the vector-add client
+(`victory_mail`) prints `VADD n=32768 bad=0` + `STATUS OK`, the
+matrix-multiply client (`victory_matmul`) prints `MATMUL n=256 bad=0` +
+`STATUS OK`, and a transformer kernel prints `dh=256 seq=64 bad=0`,
+`FFN4 layers=4 bad=0` + `STATUS OK`. The executing engine is the physical GPU
+under the module's own real GSP-RM session; the host CUDA bridge is not part
+of this path. The engine authors the completions, the host relays them to the
+guest, and the guest's synchronisation waits resolve.
+
+The leading edge has therefore moved from the launch chain itself — it fires —
+to breadth and hardening around it:
+
+* **The large-buffer staged-transfer class.** Multi-megabyte staged transfers
+  between guest memory and device memory are under repair: some large
+  multi-chunk clients still mis-verify.
+* **Reproducibility across boots.** A boot-poisoning defect was root-caused
+  and fixed: under the driver's status-queue mode the emulation read the wrong
+  queue cursor, so under load the reply ring could desynchronise, the first
+  CUDA client would fail, and the rest of that boot was poisoned (the
+  user-visible `STATUS FAIL`-then-freeze report). Fixed; re-measurement
+  across fresh boots is in progress.
+* **Per-guest execution isolation.** Multiple mediated devices with different
+  emulated VRAM sizes and multiple guests already coexist at the device level;
+  per-guest execution isolation for multiple simultaneous CUDA guests is in
+  progress.
+
+No breadth beyond these is claimed: OpenGL/Vulkan, Windows guests, migration
+and performance numbers remain planned.
 
 ---
 
@@ -81,17 +106,17 @@ it.
 
 | Observable | What it proves | What it does **not** prove |
 |---|---|---|
-| Ring entry consumed; ring cursor advances | The front-end is alive and fetching | Nothing about the engine |
-| Method execution on the ladder channel (report semaphore, channel semaphore) | The front-end can execute the method vocabulary | **Not** engine execution — these signals can be front-end-authored |
-| Descriptor buffer prewritten by the host with a detectable sentinel | Whether the descriptor was materialised/modified after submission | A read does not modify memory; the sentinel detects installs, not reads |
-| Install sentinel unchanged after submit | The inline-install method did not run | It does not say why (mapping, required control method, or ordering) |
-| Engine-authored release word absent; completion poll times out | The engine did not complete the launch | It does not say whether the engine started |
-| No GR fault recorded | The engine did not raise an error | Silence is not health: a skipped or never-reached launch is silent by construction |
-| Channel instance context flag set | The channel is context-valid at dispatch | Nothing about the descriptor path |
-
-**Consequence:** progress on this frontier must be measured by an
-**engine-authored** signal (the release word advancing) — not by front-end
-signals, which we have already learned to distinguish.
+| Client verification output (`bad=0` + `STATUS OK`) | The guest executed its kernel, read its results back and verified them | by itself, which engine produced the results |
+| Engine-authored release word advancing, relayed to the guest | The GR engine executed the launch and the completion relay path works | nothing about other launches |
+| The guest's synchronisation wait resolving | Completion reached the guest through the relay | nothing about other launches |
+| Ring entry consumed; ring cursor advances | The front-end is alive and fetching | **not** engine execution — these signals can be front-end-authored |
+| Reproduction across fresh boots | The path is not a lucky boot | not yet an "every boot" claim while re-measurement is in progress |
+| No GR fault recorded | The engine did not raise this error | silence is not health: a skipped or never-reached launch is silent by construction |
+
+**Consequence:** acceptance is now measured by verified guest results plus the
+engine-authored release; front-end signals remain necessary but not
+sufficient. A green run counts as progress only when the same acceptance
+reproduces across fresh boots.
 
 ---
 
@@ -111,90 +136,94 @@ signals, which we have already learned to distinguish.
 * **Parenting/TSG for the fetch** — both parenting variants fetch identically.
 * **The copy-engine path** — CE-class work, its releases and its fault relay
   are live (default off); the GR path is a separate problem.
+* **The descriptor-gate line of investigation (the old T1–T4 theories).** The
+  inline-descriptor install, the engine's QMD read, the grid dispatch, the
+  engine-authored release and the guest wake-up are all now observed; the
+  gate they questioned is closed by observation (§1, §7).
+* **The old boot-poisoning defect.** Root-caused to the status-queue
+  consumption cursor and fixed; it is no longer the explanation for a failing
+  first client (§5, issue B).
 
 ---
 
 ## 5. The clear issues (ranked)
 
-**Issue A — the forwarded stream carries a verbatim guest prelude that cannot
-complete.**
-The forwarded stream includes a verbatim copy of the guest's prelude whose
-inline-data writes target addresses that are not present in the engine's view
-of the channel's page tables, and the referenced data is not relocated. The
-shorter, proven stream shape fires its tail (channel semaphore) but still does
-not schedule the descriptor.
-*How we know:* the prelude stalls in the forwarded form; the shorter shape's
-tail fires; the descriptor-install sentinel never changes in either case.
-*If fixed:* the install path is reached and we can observe stage 6 directly.
-
-**Issue B — the descriptor-virtualization control method is absent.**
-The method that arms the inline-descriptor install path (the QMD
-virtualization control) is not present in the forwarded stream.
-*How we know:* the method does not appear in the submitted stream, and the
-install sentinel never changes.
-*If fixed:* the front-end should materialise the descriptor — falsifiable in
-one cycle by watching the sentinel and the engine-read signal.
-
-**Issue C — diagnostic hygiene.**
-The descriptor sentinel placed at word 0 overlaps real descriptor fields; it
-must be moved to an unused word so diagnostics do not perturb what they
-measure.
+**Issue A — the large-buffer staged-transfer class.**
+Multi-megabyte staged transfers between guest memory and device memory are
+under repair: some large multi-chunk clients still mis-verify.
+*How we know:* the small and medium clients (vector-add, matrix-multiply, the
+transformer kernel) verify green while large multi-chunk transfers mis-compare
+or stall in the same configurations.
+*If fixed:* the client battery should accept the large-buffer class with the
+same verification discipline as the green clients.
+
+**Issue B — reproducibility across boots (re-measurement).**
+A boot-poisoning defect was root-caused and fixed: under the driver's
+status-queue mode the emulation read the wrong queue cursor, so under load the
+reply ring could desynchronise, the first CUDA client would fail, and the rest
+of that boot was poisoned (the user-visible `STATUS FAIL`-then-freeze report).
+*How we know:* the failure was reproduced, the cursor rule was corrected, and
+per-client unregister and session flush work again.
+*If fixed:* green on every fresh boot — re-measurement is in progress, and no
+"every boot" claim is made until it completes.
+
+**Issue C — per-guest execution isolation.**
+Multiple mediated devices with different emulated VRAM sizes and multiple
+guests already coexist at the device level; per-guest execution isolation for
+multiple simultaneous CUDA guests is in progress.
+*How we know:* the device-level coexistence is live; execution isolation is
+not complete.
+*If fixed:* multiple CUDA guests execute simultaneously without interference.
 
 ---
 
 ## 6. Lines of investigation (live)
 
-1. **Restore the proven launch shape** as the baseline (the shorter stream that
-   fires its tail) and **re-add the virtualization control method** — one
-   variable at a time, with the install sentinel and the engine-read signal as
-   the readout.
-2. **Relocate or drop the verbatim prelude** so the inline-data writes target
-   addresses present in the engine's view, or replace it with the equivalent
-   host-side staging the proven shape already uses.
-3. **Move the sentinel** and re-run to confirm the readout is clean.
-4. **Diff against a known-good reference.** Two ways to obtain one: (a) make
-   our own ladder dispatch a descriptor built from the same source material
-   (in-house reference), or (b) capture a working launch of the stock stack on
-   the same GPU (real method stream + descriptor + engine-visible channel
-   state) and compare field by field. Either reference converts the remaining
-   work from inference to differential.
-5. **Offline probing** of any rule the front-end or engine is believed to
-   require, using the firmware emulator/probe tooling before spending a
-   supervised cycle.
+1. **Repair the large-buffer staged-transfer class** — the multi-megabyte
+   guest↔device transfers that still mis-verify. The three green clients stay
+   the regression battery and must remain green throughout.
+2. **Complete the fresh-boot re-measurement** — demonstrate the fixed boot
+   path across fresh boots before claiming boot-to-boot reproducibility, and
+   keep the re-measured battery as the regression gate.
+3. **Per-guest execution isolation** — build from the device-level
+   coexistence of multiple mediated devices and guests to per-guest execution
+   isolation.
+4. **Hold the acceptance battery** — vector-add, matrix-multiply and the
+   transformer kernel run as a battery on every re-measurement; each entry is
+   accepted only on its own verification output (`bad=0` + `STATUS OK`).
 
 ---
 
-## 7. Theories (labelled; not yet proven)
-
-* **T1 — the virtualization control is the gate.** The front-end will not
-  materialise an inline descriptor until the virtualization-control method
-  enables the install path; its absence alone explains the inert descriptor.
-* **T2 — materialise-then-read.** The engine only schedules a descriptor that
-  exists in its own view; an install targeting unmapped destinations aborts
-  silently rather than faulting.
-* **T3 — the verbatim prelude poisons the launch.** Even where the tail shape
-  works, the prelude's refused writes leave the launch path in a state that
-  prevents the descriptor from being scheduled.
-* **T4 — relocation is the missing transform.** The guest's inline descriptor
-  is valid, but it is built for the guest's view of memory; the forwarding path
-  must relocate referenced data (and any embedded pointers) into the channel's
-  view before install.
-
-Each theory has a one-cycle falsifier: the install sentinel, the engine-read
-signal, and the engine-authored release, in that order.
+## 7. Theories, resolved
+
+The descriptor-gate theories (T1–T4) are resolved and are not kept as open:
+the inline-descriptor install, the engine's QMD read, the grid dispatch, the
+engine-authored release and the guest wake-up are all directly observed (§1),
+so the launch chain they questioned is now proven end to end.
+
+The remaining leading-edge work is engineering rather than theory: the
+large-buffer transfer class, boot-to-boot reproducibility, and per-guest
+execution isolation. Each has an acceptance signal named in §5 before the
+work.
 
 ---
 
 ## 8. Claim discipline
 
-* **Engine-authored release is the only proof of execution.** Front-end
-  signals (ring cursor, method execution, semaphores that the front-end can
-  author) are necessary but not sufficient.
+* **Engine-authored release is the acceptance signal for execution** — now
+  observed end to end for the verified clients; the guest's own verification
+  (`bad=0` + `STATUS OK`) is the guest-side acceptance signal.
+* **Front-end signals are necessary but not sufficient.** Ring cursor and
+  front-end method execution can be front-end-authored; they never substitute
+  for the engine-authored release.
 * **Silence is not health.** No fault recorded does not mean no fault
   occurred; the current GR path runs without an error-reporting object, which
   is itself a known observability gap.
 * **Emulator verdicts are L0/L1.** They bound behaviour, never timing or
   device environment.
+* **A green run counts only when it reproduces across fresh boots.** The
+  boot-poisoning defect is the worked example: one green boot is not
+  reproducibility.
 * **A wall is closed only by an acceptance signal**, and the acceptance signal
   for this frontier is named before the experiment.
 
@@ -218,3 +247,4 @@ signal, and the engine-authored release, in that order.
 | Date | Change |
 |---|---|
 | 2026-10-06 | Created: the GR compute-launch chain, the leading edge (inline-descriptor install), evidence limits, exonerated walls, ranked issues, live investigation lines and theories. |
+| 2026-10-20 | Chain complete: verified guest CUDA clients execute on the physical GPU (vector-add, matrix-multiply, transformer; engine-authored completions relay and guest waits resolve). Leading edge moved to the large-buffer staged-transfer class, fresh-boot reproducibility (boot-poisoning defect fixed) and per-guest execution isolation; the descriptor-gate theories are resolved. |