1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
|
# The Execution Frontier: The GR Compute-Launch Chain
> **Status: 2026-10-20.** This page describes the current state of
> guest-facing execution: the chain that must fire for a stock guest CUDA
> kernel to run on the physical GPU, what we have proven about it, how we
> know, and where the leading edge is now. The chain fires end to end:
> verified guest CUDA clients (vector-add, matrix-multiply and a transformer
> kernel) execute on the physical GPU, their results verify, and the
> engine-authored completions relay back to the guest. The leading edge has
> moved from "can a forwarded launch execute at all?" to breadth and
> hardening — the large-buffer staged-transfer class, reproducibility across
> boots, and per-guest execution isolation. It is evidence-led: every
> statement carries either a live observation, a decompilation finding, or a
> clearly labelled theory.
>
> Evidence sources for this page: supervised physical cycles (module-owned
> GSP-RM on the reference TU104), offline firmware emulation/probing, the host
> client's driver sources for the supported version, and decompilation of the
> shipped firmware. Fidelity rules apply: emulator output is
> instruction/memory-level (L0/L1); no timing or device-environment claims are
> made from it.
---
## 1. The chain that should fire
A guest CUDA launch becomes a **method stream** in a push buffer, referenced by
a **GPFIFO ring entry**, announced by a **doorbell**. The host emulation decodes
it and forwards an equivalent method stream to a real channel owned by the
module's resource-manager session. On the hardware side the **front-end**
(PBDMA) fetches and executes the methods, the launch descriptor (**QMD**) is
materialised and read by the **GR engine**, the grid is dispatched to the SMs,
and completion is published as **semaphores** the guest waits on.
```mermaid
flowchart TB
A["Guest: CUDA builds launch<br/>program, parameters, QMD, method stream<br/>GPFIFO entry + doorbell"] --> B
B["Host: doorbell trap, decode,<br/>rebuild forward stream,<br/>prewrite QMD, ring/USERD, doorbell"] --> C
C["Front-end: fetch GPFIFO entry<br/>(ring cursor advances)"] --> D
D["Front-end: execute methods<br/>object bind, cache ops, report sem"] --> E
E["Front-end: install inline QMD<br/>into the descriptor buffer"] --> F
F["GR engine: read QMD through<br/>the channel's page tables"] --> G
G["Context switch loads channel context<br/>(FECS/ucode)"] --> H
H["GR: dispatch grid to SMs"] --> I
I["Engine authors release semaphore<br/>+ channel semaphore"] --> J
J["Host polls release, relays to guest;<br/>CUDA wakes"]
```
| # | Stage | State (2026-10-20) |
|---|---|---|
| 1 | Guest builds launch + method stream | ✅ observed (stock driver, unmodified) |
| 2 | GPFIFO entry + doorbell | ✅ observed |
| 3 | Host decode + forward stream + QMD prewrite + ring/USERD | ✅ observed (forwarding path) |
| 4 | Front-end fetches the ring entry | ✅ observed (ring cursor advances) |
| 5 | Front-end executes methods | ✅ observed (object bind, cache ops, report semaphore) |
| 6 | Inline-QMD install (descriptor materialised into the QMD buffer) | ✅ observed |
| 7 | Engine reads the QMD | ✅ observed |
| 8 | Context switch loads the channel context | ✅ context validity at dispatch is ruled in — see §4 |
| 9 | GR dispatches the grid to the SMs | ✅ observed (the guest's results verify) |
| 10 | Engine authors release + channel semaphore | ✅ observed — engine-authored releases relay to the guest |
| 11 | Host relays completion; guest wakes | ✅ observed — the guest's synchronisation waits resolve |
**One-line summary:** the whole chain is live — verified guest CUDA clients
execute on the physical GPU and wake up correctly. The leading edge is no
longer the chain itself but its breadth and hardening: the large-buffer
staged-transfer class, reproducibility across boots, and per-guest execution
isolation.
---
## 2. Where the leading edge is
A stock, unmodified guest CUDA program now executes its kernels on the
physical GPU and verifies its results. On fresh boots the vector-add client
(`victory_mail`) prints `VADD n=32768 bad=0` + `STATUS OK`, the
matrix-multiply client (`victory_matmul`) prints `MATMUL n=256 bad=0` +
`STATUS OK`, and a transformer kernel prints `dh=256 seq=64 bad=0`,
`FFN4 layers=4 bad=0` + `STATUS OK`. The executing engine is the physical GPU
under the module's own real GSP-RM session; the host CUDA bridge is not part
of this path. The engine authors the completions, the host relays them to the
guest, and the guest's synchronisation waits resolve.
The leading edge has therefore moved from the launch chain itself — it fires —
to breadth and hardening around it:
* **The large-buffer staged-transfer class.** Multi-megabyte staged transfers
between guest memory and device memory are under repair: some large
multi-chunk clients still mis-verify.
* **Reproducibility across boots.** A boot-poisoning defect was root-caused
and fixed: under the driver's status-queue mode the emulation read the wrong
queue cursor, so under load the reply ring could desynchronise, the first
CUDA client would fail, and the rest of that boot was poisoned (the
user-visible `STATUS FAIL`-then-freeze report). Fixed; re-measurement
across fresh boots is in progress.
* **Per-guest execution isolation.** Multiple mediated devices with different
emulated VRAM sizes and multiple guests already coexist at the device level;
per-guest execution isolation for multiple simultaneous CUDA guests is in
progress.
No breadth beyond these is claimed: OpenGL/Vulkan, Windows guests, migration
and performance numbers remain planned.
---
## 3. How we know (observables and their limits)
| Observable | What it proves | What it does **not** prove |
|---|---|---|
| Client verification output (`bad=0` + `STATUS OK`) | The guest executed its kernel, read its results back and verified them | by itself, which engine produced the results |
| Engine-authored release word advancing, relayed to the guest | The GR engine executed the launch and the completion relay path works | nothing about other launches |
| The guest's synchronisation wait resolving | Completion reached the guest through the relay | nothing about other launches |
| Ring entry consumed; ring cursor advances | The front-end is alive and fetching | **not** engine execution — these signals can be front-end-authored |
| Reproduction across fresh boots | The path is not a lucky boot | not yet an "every boot" claim while re-measurement is in progress |
| No GR fault recorded | The engine did not raise this error | silence is not health: a skipped or never-reached launch is silent by construction |
**Consequence:** acceptance is now measured by verified guest results plus the
engine-authored release; front-end signals remain necessary but not
sufficient. A green run counts as progress only when the same acceptance
reproduces across fresh boots.
---
## 4. What is exonerated (do not relitigate)
* **Context validity at dispatch** — the channel's context is valid when the
launch is submitted (instance flag set). The earlier "engine context never
committed" wall is a historical stage.
* **The descriptor destination window mapping** — the destination window is
installed in the engine's table view at submit (`status=0`).
* **Front-end fetch and control-block alignment** — the entry is consumed with
the host control block at offset 0; index semantics are not applied to a
host-backed userd descriptor.
* **The shared context-buffer one-shot rule** — the refusal seen on one
channel family was traced to a shared per-GPU global buffer already
initialised in the same cycle; it is not the dispatch delta.
* **Parenting/TSG for the fetch** — both parenting variants fetch identically.
* **The copy-engine path** — CE-class work, its releases and its fault relay
are live (default off); the GR path is a separate problem.
* **The descriptor-gate line of investigation (the old T1–T4 theories).** The
inline-descriptor install, the engine's QMD read, the grid dispatch, the
engine-authored release and the guest wake-up are all now observed; the
gate they questioned is closed by observation (§1, §7).
* **The old boot-poisoning defect.** Root-caused to the status-queue
consumption cursor and fixed; it is no longer the explanation for a failing
first client (§5, issue B).
---
## 5. The clear issues (ranked)
**Issue A — the large-buffer staged-transfer class.**
Multi-megabyte staged transfers between guest memory and device memory are
under repair: some large multi-chunk clients still mis-verify.
*How we know:* the small and medium clients (vector-add, matrix-multiply, the
transformer kernel) verify green while large multi-chunk transfers mis-compare
or stall in the same configurations.
*If fixed:* the client battery should accept the large-buffer class with the
same verification discipline as the green clients.
**Issue B — reproducibility across boots (re-measurement).**
A boot-poisoning defect was root-caused and fixed: under the driver's
status-queue mode the emulation read the wrong queue cursor, so under load the
reply ring could desynchronise, the first CUDA client would fail, and the rest
of that boot was poisoned (the user-visible `STATUS FAIL`-then-freeze report).
*How we know:* the failure was reproduced, the cursor rule was corrected, and
per-client unregister and session flush work again.
*If fixed:* green on every fresh boot — re-measurement is in progress, and no
"every boot" claim is made until it completes.
**Issue C — per-guest execution isolation.**
Multiple mediated devices with different emulated VRAM sizes and multiple
guests already coexist at the device level; per-guest execution isolation for
multiple simultaneous CUDA guests is in progress.
*How we know:* the device-level coexistence is live; execution isolation is
not complete.
*If fixed:* multiple CUDA guests execute simultaneously without interference.
---
## 6. Lines of investigation (live)
1. **Repair the large-buffer staged-transfer class** — the multi-megabyte
guest↔device transfers that still mis-verify. The three green clients stay
the regression battery and must remain green throughout.
2. **Complete the fresh-boot re-measurement** — demonstrate the fixed boot
path across fresh boots before claiming boot-to-boot reproducibility, and
keep the re-measured battery as the regression gate.
3. **Per-guest execution isolation** — build from the device-level
coexistence of multiple mediated devices and guests to per-guest execution
isolation.
4. **Hold the acceptance battery** — vector-add, matrix-multiply and the
transformer kernel run as a battery on every re-measurement; each entry is
accepted only on its own verification output (`bad=0` + `STATUS OK`).
---
## 7. Theories, resolved
The descriptor-gate theories (T1–T4) are resolved and are not kept as open:
the inline-descriptor install, the engine's QMD read, the grid dispatch, the
engine-authored release and the guest wake-up are all directly observed (§1),
so the launch chain they questioned is now proven end to end.
The remaining leading-edge work is engineering rather than theory: the
large-buffer transfer class, boot-to-boot reproducibility, and per-guest
execution isolation. Each has an acceptance signal named in §5 before the
work.
---
## 8. Claim discipline
* **Engine-authored release is the acceptance signal for execution** — now
observed end to end for the verified clients; the guest's own verification
(`bad=0` + `STATUS OK`) is the guest-side acceptance signal.
* **Front-end signals are necessary but not sufficient.** Ring cursor and
front-end method execution can be front-end-authored; they never substitute
for the engine-authored release.
* **Silence is not health.** No fault recorded does not mean no fault
occurred; the current GR path runs without an error-reporting object, which
is itself a known observability gap.
* **Emulator verdicts are L0/L1.** They bound behaviour, never timing or
device environment.
* **A green run counts only when it reproduces across fresh boots.** The
boot-poisoning defect is the worked example: one green boot is not
reproducibility.
* **A wall is closed only by an acceptance signal**, and the acceptance signal
for this frontier is named before the experiment.
---
## 9. Where the rest of the knowledge lives
| Topic | Document |
|---|---|
| Honest known-knowns / known-unknowns (the execution gate) | [`01-known-knowns-and-unknowns.md`](01-known-knowns-and-unknowns.md) |
| The physical backend lifecycle and guest wiring (init → work → completion → teardown) | [`../03-architecture/11-physical-backend-guest-wiring.md`](../03-architecture/11-physical-backend-guest-wiring.md) |
| Channels, GPFIFO, USERD, runlist, GR and QMD concepts | [`../03-architecture/05-gmmu-amap-hal-gr.md`](../03-architecture/05-gmmu-amap-hal-gr.md) |
| Interaction and scheduling model (guest ↔ host ↔ engine) | [`../03-architecture/10-interaction-and-scheduling.md`](../03-architecture/10-interaction-and-scheduling.md) |
| Supervised-cycle mechanics and hazard discipline | [`../eng-ops.md`](../eng-ops.md), [`../RUNBOOK.md`](../RUNBOOK.md) |
| Offline firmware emulation/debugger | [`../05-operations/03-risc-v-gsp-rm-vm.md`](../05-operations/03-risc-v-gsp-rm-vm.md), [`../05-operations/04-risc-v-gsp-rm-debugger-user-guide.md`](../05-operations/04-risc-v-gsp-rm-debugger-user-guide.md) |
---
## 10. Change log
| Date | Change |
|---|---|
| 2026-10-06 | Created: the GR compute-launch chain, the leading edge (inline-descriptor install), evidence limits, exonerated walls, ranked issues, live investigation lines and theories. |
| 2026-10-20 | Chain complete: verified guest CUDA clients execute on the physical GPU (vector-add, matrix-multiply, transformer; engine-authored completions relay and guest waits resolve). Leading edge moved to the large-buffer staged-transfer class, fresh-boot reproducibility (boot-poisoning defect fixed) and per-guest execution isolation; the descriptor-gate theories are resolved. |
|
