1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
|
# The Execution Frontier
> **What this page is.** The state of guest-facing execution: the chain that
> must fire for a stock guest CUDA kernel to run on the physical GPU, what has
> been proven about it, how we know, and where the leading edge is now. The
> chain fires end to end: verified guest CUDA clients execute on the physical
> GPU and wake up correctly; the leading edge is now breadth and hardening. It
> is evidence-led: every statement carries a live observation, a
> decompilation finding, or a clearly labelled theory. Condensed from
> [`../docs/04-implementation/03-execution-frontier.md`](../docs/04-implementation/03-execution-frontier.md);
> the docs tree wins on disagreement.
Status: 2026-10-20, tree `master @ 16291c72`.
Tags: `[live]` observed; `[wip]` partial; `[wall]` current blocker; theories
are labelled as such.
## 1. The chain that should fire
A guest CUDA launch becomes a **method stream** in a push buffer, referenced
by a **GPFIFO ring entry**, announced by a **doorbell**. The host emulation
decodes it and forwards an equivalent method stream to a real channel owned by
the module's resource-manager session. The hardware **front-end** fetches and
executes the methods, the launch descriptor (**QMD**) is materialised and read
by the **GR engine**, the grid is dispatched to the SMs, and completion is
published as **semaphores** the guest waits on.
```text
guest: builds launch + method stream
GPFIFO entry + doorbell
-> host: doorbell trap, decode, rebuild forward stream,
prewrite QMD, ring/USERD, doorbell
-> front-end: fetch GPFIFO entry (ring cursor advances)
-> front-end: execute methods (object bind, cache ops, report sem)
-> front-end: install inline QMD into the descriptor buffer
-> GR engine: read QMD through the channel's page tables
-> context switch loads the channel context (FECS/ucode)
-> GR: dispatch grid to the SMs
-> engine authors release semaphore + channel semaphore
-> host polls release, relays to guest; CUDA wakes
```
| # | Stage | State |
|---|---|---|
| 1 | Guest builds launch + method stream | `[live]` observed (stock driver, unmodified) |
| 2 | GPFIFO entry + doorbell | `[live]` observed |
| 3 | Host decode + forward stream + QMD prewrite + ring/USERD | `[live]` observed |
| 4 | Front-end fetches the ring entry | `[live]` observed (ring cursor advances) |
| 5 | Front-end executes methods | `[live]` observed (object bind, cache ops, report semaphore) |
| 6 | **Inline-QMD install** (descriptor materialised) | `[live]` observed |
| 7 | **Engine reads the QMD** | `[live]` observed |
| 8 | Context switch loads the channel context | `[live]` context validity at dispatch is ruled in |
| 9 | GR dispatches the grid to the SMs | `[live]` observed (the guest's results verify) |
| 10 | Engine authors release + channel semaphore | `[live]` engine-authored releases relay to the guest |
| 11 | Host relays completion; guest wakes | `[live]` the guest's synchronisation waits resolve |
**One-line summary:** the whole chain is live; the leading edge is now breadth
and hardening — the large-buffer staged-transfer class, reproducibility across
boots, and per-guest execution isolation.
## 2. Where the leading edge is
A stock, unmodified guest CUDA program executes its kernels on the physical
GPU and verifies its results. On fresh boots the vector-add client
(`victory_mail`) prints `VADD n=32768 bad=0` + `STATUS OK`, the
matrix-multiply client (`victory_matmul`) prints `MATMUL n=256 bad=0` +
`STATUS OK`, and a transformer kernel prints `dh=256 seq=64 bad=0`,
`FFN4 layers=4 bad=0` + `STATUS OK`. The executing engine is the physical GPU
under the module's own real GSP-RM session; the host CUDA bridge is not part
of this path. Engine-authored completions relay to the guest and its
synchronisation waits resolve.
The leading edge has moved from the launch chain itself — it fires — to
breadth and hardening around it:
* **The large-buffer staged-transfer class.** Multi-megabyte staged transfers
between guest memory and device memory are under repair: some large
multi-chunk clients still mis-verify.
* **Reproducibility across boots.** A boot-poisoning defect was root-caused
and fixed: under the driver's status-queue mode the emulation read the wrong
queue cursor, so under load the reply ring could desynchronise and the first
CUDA client would fail (the user-visible `STATUS FAIL`-then-freeze report).
Re-measurement across fresh boots is in progress.
* **Per-guest execution isolation.** Multiple mediated devices with different
emulated VRAM sizes and multiple guests already coexist at the device level;
per-guest execution isolation for multiple simultaneous CUDA guests is in
progress.
No breadth beyond these is claimed: OpenGL/Vulkan, Windows guests, migration
and performance numbers remain planned.
## 3. How we know (observables and their limits)
| Observable | What it proves | What it does not prove |
|---|---|---|
| Client verification output (`bad=0` + `STATUS OK`) | the guest executed its kernel and verified its results | by itself, which engine produced the results |
| Engine-authored release relayed to the guest | the engine executed the launch and the relay works | nothing about other launches |
| The guest's synchronisation wait resolving | completion reached the guest through the relay | nothing about other launches |
| Ring entry consumed; ring cursor advances | the front-end is alive and fetching | **not** engine execution — front-end signals are necessary but not sufficient |
| Reproduction across fresh boots | the path is not a lucky boot | no "every boot" claim yet while re-measurement is in progress |
| No GR fault recorded | the engine did not raise this error | silence is not health: a skipped launch is silent by construction |
**Consequence:** acceptance is verified guest results plus the engine-authored
release; a green run counts as progress only when it reproduces across fresh
boots.
## 4. What is exonerated (do not relitigate)
* Context validity at dispatch — the channel's context is valid when the
launch is submitted (instance flag set).
* The descriptor destination window mapping — installed in the engine's table
view at submit.
* Front-end fetch and control-block alignment — the entry is consumed with the
host control block at offset 0.
* The shared context-buffer one-shot rule — traced to a shared per-GPU buffer
already initialised in the same cycle; not the dispatch delta.
* Parenting/TSG for the fetch — both variants fetch identically.
* The copy-engine path — CE-class work, releases and fault relay are live
(default off); the GR path is separate.
* The descriptor-gate line of investigation (the old T1–T4 theories) — the
inline-descriptor install, the engine read, the dispatch, the
engine-authored release and the guest wake-up are all observed; closed by
observation.
* The old boot-poisoning defect — root-caused to the status-queue consumption
cursor and fixed.
## 5. The clear issues (ranked)
**Issue A — the large-buffer staged-transfer class.** Multi-megabyte staged
transfers between guest memory and device memory are under repair: some large
multi-chunk clients still mis-verify.
**Issue B — reproducibility across boots (re-measurement).** The
boot-poisoning defect (wrong status-queue consumption cursor under the
driver's status-queue mode; the reply ring could desynchronise under load, the
first client would fail) was root-caused and fixed. Re-measurement across
fresh boots is in progress; no "every boot" claim is made until it completes.
**Issue C — per-guest execution isolation.** Multiple mediated devices with
different emulated VRAM sizes and multiple guests coexist at the device level;
per-guest execution isolation for simultaneous CUDA guests is in progress.
## 6. Lines of investigation (live)
1. **Repair the large-buffer staged-transfer class** — the three green clients
stay the regression battery and must remain green throughout.
2. **Complete the fresh-boot re-measurement** — demonstrate the fixed boot
path across fresh boots before claiming boot-to-boot reproducibility.
3. **Per-guest execution isolation** — build from the device-level coexistence
to isolation for simultaneous CUDA guests.
4. **Hold the acceptance battery** — vector-add, matrix-multiply and the
transformer kernel; each entry is accepted only on its own verification
output (`bad=0` + `STATUS OK`).
## 7. Theories, resolved
The descriptor-gate theories (T1–T4) are resolved and are not kept as open:
the inline-descriptor install, the engine's QMD read, the grid dispatch, the
engine-authored release and the guest wake-up are all directly observed, so
the launch chain they questioned is proven end to end. The remaining work is
engineering (large-buffer transfers, boot-to-boot reproducibility, per-guest
isolation), each with an acceptance signal named before the work.
## 8. Claim discipline
* **Engine-authored release is the acceptance signal for execution** — now
observed end to end; the guest's own verification (`bad=0` + `STATUS OK`)
is the guest-side acceptance signal.
* **Front-end signals are necessary but not sufficient.** Ring cursor and
front-end method execution can be front-end-authored; they never substitute
for the engine-authored release.
* **Silence is not health.** No fault recorded does not mean no fault
occurred.
* **Emulator verdicts are instruction/memory-level.** They bound behaviour,
never timing or device environment.
* **A green run counts only when it reproduces across fresh boots.**
* **A wall is closed only by an acceptance signal**, named before the
experiment.
## 9. Related pages
* [`Project-Status.md`](Project-Status.md) — the project-level ladder.
* [`CUDA-Userspace-Driver.md`](CUDA-Userspace-Driver.md) — the guest-side
client path.
* [`Physical-GPU-Backend.md`](Physical-GPU-Backend.md) — the backend the work
is forwarded to.
* [`Execution-Plane.md`](Execution-Plane.md) — the execution layer's design.
* [`Introduction-to-GMMU.md`](Introduction-to-GMMU.md) — page tables, QMDs and
the engines.
* [`../docs/04-implementation/03-execution-frontier.md`](../docs/04-implementation/03-execution-frontier.md) —
the canonical, continuously-updated version with the full evidence tables.
|
