summaryrefslogtreecommitdiffstats
path: root/LOTTERY-REPORT.md
blob: 379444062165492d3450bbc61fbf2114d53f1f33 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
# LOTTERY-REPORT — per-boot engine lottery (`ecfb1278` baseline, r155 live)

* Offline analysis + implementation attempt. Worktree: `/home/rmcore/Git/integdev_gpu_drv/open-iov-lottery`
  (branch `lottery`, based on `ecfb1278`). No module load, no QEMU/mdev/GPU/guest writes, no reboots.
  Build verified: `cd kernel && make -j8` (kernel 6.8.0-147; note: this worktree has no
  `nvidia-open-gpu-kernel-modules` submodule, so the build needs `KBUILD_EXTRA_SYMBOLS=` on the command
  line to reach modpost; the plain `make -j8` failed only on that missing file).
* Primary live evidence: `/tmp/opencode/dmesg-follow-r155.log` (read-only) and its client outputs in
  `/tmp/opencode/soak-20261015-172722/` + `/tmp/opencode/r155-*.log`.
* Prior reports: `analysis-2026-10-14b/AGENT-C-engine-allwrong-analysis.md`,
  `analysis-2026-10-15/AGENT-bigcopy-analysis.md`.
* **Branch state: no kernel fix committed.** An F1-style fail-closed release gate was implemented, then
  reverted after its decisive unsoundness was proven against the live log (finding F3). The two commits
  are preserved on local branch `lottery-m156-gate-attempt` (`de91d640`, `eee92231`); `lottery` carries
  only this report (`99e5a61e`) on top of the clean `ecfb1278` baseline. Kernel builds green from the
  baseline (`make -j8`).

---

## 1. Top findings (ranked)

### F1 — PROVEN: the engine's completion ack (`qmd_rel`) is a front-end retirement, not an execution
witness; the host returns success on it and the client reads wrong data.

Live, this boot, single client, end to end (`dmesg-follow-r155.log`):

| line | event |
|---|---|
| 27847 | `launch captured h=0x5c00001f push=0x2008002cc … rel=0x204027c7c/6` (streams2 launch) |
| 28163 | `GR LAUNCH submitted … qmdwin=0x107000` (1044.568) |
| 28216 | `output retirement wait: 0/205 pages changed after 720 ms (no writes)` |
| 28440 | `0/205 …` after outcome retry 1 |
| 28493 | `0/205 …` after outcome retry 2 |
| 28564 | `0/205 …` after outcome retry 3 |
| 28576 | **`GUEST GR LAUNCH OK … qmd_rel=0x00000006 ch_rel=0xc0ffee02 elapsed_ms=3370 ok=8`** |
| soak `out-r1-streams2.txt` | **`STREAMS n=32768 bad=32768` / `STATUS FAIL`** (all-zero output) |

The release word at `0x204027c7c` could not have been host-authored before 28576 (the hold's software
release runs only after the work item returns at 28576; `svc_gr_sw_release` is at
`gsp_sema_service.c:11388`, called from the hold resolution `:11754`/`:11768`), so the `qmd_rel=6`
that broke the poll was **hardware-authored with zero SM writes**. Same signature in r152 longevity
(matmul `bad=65536`, `0/205` after 725 ms) and r150 (`AGENT-bigcopy` M1). Code path:
`gsp_phys_booter.c:18564` returns 0; `gsp_sema_service.c:12031-12050` (`hw_rc=0`), `:11768` applies
the (no-op) release.

### F2 — PROVEN: three distinct host paths author client-visible releases for launches nothing executed.

1. **Acked-no-write** — `gsp_phys_booter.c:18388-18454` retries, then accepts at `:18564` (`ret=0`)
   with the retirement wait observed `wrote==0`; the hold's success edge (`gsp_sema_service.c:11767-11768`)
   then applies the completion release. Client proceeds with stale/zero output. (F1's path.)
2. **Timeout / no ack** — `gsp_phys_booter.c:18005-18072` returns `-ETIMEDOUT`; the sema hold's failure
   branch (`gsp_sema_service.c:11736-11758` → `svc_fwd_hold_gr_release` :11409 → `svc_gr_sw_release`
   :11388) authors the completion in software. This is AGENT-bigcopy M1 (r150 `memcpy_roundtrip`
   `bad=10485760 csum=0`).
3. **Refused capture (never even attempted)** — `gsp_sema_service.c:14321-14375`: when a GR capture
   cannot queue (`svc_gr_request` refuses while a hold is active, `:12958-12963`), the entry loop
   *consumes* the push and soft-authors its completion at **`:14369-14374`**. Live: during ok=7's
   14.8 s backend saga the vecadd_loop burst produced **98 such releases with zero execution attempts**
   (`work_guard … direct=99 gr_sw_rel=98`, log:22599; `rel-hold direct … (no work)` +
   `GR launch release software-applied` pairs at log:20589-21255+). Those results were correct only
   because vecadd_loop repeats an idempotent kernel; for a non-idempotent multi-launch client this is a
   silent-zero factory.

### F3 — PROVEN: the zero-write witness cannot distinguish no-execute from an identical rewrite, so
F1's proposed fail-closed release is **unsound** and was reverted.

`vecadd_loop` launches 2..100 are the *same* `c = a + b` over the same buffers. At ok=7 (launch 2) the
post-run window dump shows the output already holding the result **before** the copy-out:

```
log:22613  window[64] reloc=0x150000 first=00000000 3f400000 3fc00000 40100000 changed=0
log:20100  window[64] reloc=0x150000 first=00000000 3f400000 3fc00000 40100000 changed=1  (launch 1)
```
`0x3f400000/0x3fc00000/0x40100000` = 0.75/1.5/2.25 = `a+b` for the first elements. An executed
identical rewrite writes the same bytes → `0/205 changed`; a no-execute also gives `0/205`. Both are
indistinguishable, and for the identical case the data is **already correct**. A gate that fails a
launch on `0/205` would have converted currently-green clients (`vecadd_loop` repeats, smiley passes
2..4 with identical tuples per AGENT-C §3, repeated matmul runs) into client-visible errors on correct
data. `smiley`/`matmul` repeats have the same property. This was not considered by AGENT-bigcopy F1
(which reasoned about read-only kernels only); it is the reason no fail-closed release was landed.

### F4 — PROVEN: the no-execute state starts at one launch and then persists module-wide for the boot.

r155 sequence (all lines in the live log):

```
20280  908.833  GR LAUNCH submitted h=0x5c000019 push=0x20043c19c slot=0     (vecadd launch 2)
20281  908.886  sked rec +5ms cwd0=00012480 act=a0000005 grfifo=10100001 fecs_cur=b0143808  (enqueued, ctx loaded)
20338  913.840  GR LAUNCH FAILED timeout after 5000 ms (qmd_rel=0x5 …)        (no retirement)
20389  914.004  sked rec +435ms cwd0=00002080 act=0 fecs_cur=0               (context saved/unloaded after the timeout)
20537  919.115  retry 1/3: fresh ring entry slot=1 published + rescheduled
20538  919.115  sked rec cwd0=00012480 … fecs_cur=b0143808 act=0             (re-enqueued + ctx reloaded, still no dispatch)
20543  919.836  output retirement wait: 0/205 pages changed (no writes)
…      (outcome retries 2,3 with the m155 lane-tail rearm each time: #1059/#1061 accepted, #1063 0x1f;
        firmware asserts 0x5294444 + FECS_A/B + 0x52c1504 accompany the tail: log:919.03-920.69)
22621  923.503  GUEST GR LAUNCH OK … qmd_rel=6 wrote=0
…      every later launch (including streams2 on a different channel h=0x5c00001f, 2 min later) never executes.
```

* The FECS save/unload at 914.0 is a **consequence** of the 5 s timeout + rearm, not the trigger:
  ok=7 was enqueued with the context loaded at +5 ms and sat un-dispatched for 5 s first.
* **m155's lane-enable rearm does not revive dispatch** (first live test — it fires here; the context
  reloads at 919.115, `act=0`, still no writes; r154's conclusion therefore still stands).
* Post-OK `CWD_T0=0x12480` is **not** a discriminator: it also appears for the good launches ok=1,
  ok=2, ok=3, ok=5, ok=6 (log:7903, 8126, 12229, 16380, 20106). No proven register signature yet
  (seen only 5 `sked rec` rows/boot).

### F5 — HYPOTHESIS (ranked) for the ok=7 transition; each is a one-variable experiment.

1. **Single-buffered QMD window.** `GSP_PHYS_RMCALL_VA_GR_QMDWIN` (`gsp_phys_booter.c:1063`) is *one*
   page (0x107000, planted `:9565-9595`, prewritten every launch `:17711-17721`, published `:17802`).
   The two-bank win0 rotation (754bbbe8) protects the win0 pool but not this page, which is rewritten
   every launch while the previous launch's QMD-unit/look-ahead reads of the same VA may still be in
   flight (the exact rationale the m151 comment gives for the win0 banks). **Ok=7 is the first launch
   that reuses a win0 bank too** (`rmcall_gr_shadow_win0_used = (epoch&1)*8`, `:17038-17042`), i.e. the
   first launch after the transition point where any pool VA is re-used.
2. **2-bank win0 is too shallow** — bank reuse happens every 2 launches; the handoff's lead (b) (4-bank)
   mitigates and can be extended cheaply (pool VA space 0x1e0000..0x1effff).
3. **Engine-side stale const/PTE line for the win0 data path** (AGENT-C H1, m132/m133 precedent): the
   M151 pre/post hashes show host pages identical, so any such staleness is engine-side; the two-bank
   rotation is the only current mitigation.
4. Not supported by this log: m152 scratch-pool exhaustion (0 `launch refused` this boot), alias-cache
   line reuse (the cache is HPA-keyed and never rewrites), constant-bank fill gaps (pre/post hashes
   equal), PTE-line caching for the win0 line (planted pre-boot; m132/m133 unchanged).

### F6 — PROVEN (ops): the soak's green is not an execution witness for repeated launches.

`vecadd_loop`'s `VERIFY bad=0` passes because `c` was already correct after launch 1, even though
launches 2..100 were never executed (F2.3: 98 soft releases). Do not use vecadd_loop/soak green as
evidence that the forward path ran; use `output retirement wait` + `window[…] changed=` and the client's
own first-touch buffers (streams2/matmul).

---

## 2. Q1 — What a good boot has that a bad boot lacks

**It is not a boot property; it is a transition at one launch, after which every GR launch fails for the
rest of the boot (F4).** The checked candidates:

| Candidate | Evidence | Verdict |
|---|---|---|
| SKED CWD allocation | Post-state `CWD_T0=0x12480` is identical for good ok=1/2/3/5/6 and bad ok=7/8 (log:7903…28576) | not discriminating |
| FECS context load | bad attempt at 908.885: `fecs_cur=b0143808` (VALID=1, loaded) for 5 s while `act=0`; the save at 914.0 follows the timeout | not the trigger |
| Channel scheduling state after m155 lane-tail | m155 replays #1059/#1061/#1063 + schedules + doorbell; context reloads (919.115) but `act=0`, 0 writes on all 4 attempts | re-arm insufficient |
| win0 bank alternating | 2 banks ⇒ reuse every 2 launches; ok=7 is the first reuse (epoch parity) | suspect (F5.1/2) |
| PTE fetch-line caching | win0/bounce lines planted at stage 3, never rewritten (`pdb_build` :9565-9700); m132/m133 fixed the only observed line issue | no new evidence |
| alias-cache line reuse (m152) | HPA-keyed, `line_out=~0` prevents extension; 0 `launch refused`, pool capacity 2048 (`guest scratch PT built … capacity=2048` at boot) | refuted this boot |
| constant-bank fills | `win0 param hash pre==post` and `guest-param hash pre==post` (e.g. log:22619-22620) | host-side verified |
| m152 scratch pool state | 0 refusals in r154/r155 (`launch refused`=0 in both handoff counts and this log) | refuted |

---

## 3. Q2 — The exact retry chain and its client-visible outcome

Attempt flow in `gsp_phys_backend_guest_gr_submit` (`gsp_phys_booter.c:16924`):

1. `release0` selection `:17053-17104` — either a Tier-2 shadow alias of the guest release page
   (engine writes the client's counter directly) or the module sem.
2. Copy-in `:17778-17800` (pre-doorbell, m142), publish `:17802`, poll `:17948`.
3. **Timeout branch `:18005`**: log `GR LAUNCH FAILED: timeout after 5000 ms (qmd_rel=… want=…)`;
   - one re-arm `:18015-18033` (`gsp_phys_rmcall_gr_resched_locked` `:16737-16784` — m155 adds the
     mode-35 lane-enable tail at `:16756`, then GPFIFO_SCHEDULE + TSG + doorbell), re-poll 5 s;
   - up to `GSP_PHYS_RMCALL_GR_ATTEMPTS`=3 fresh-entry retries (`:18044-18072`, 2 s poll each);
   - then the post-mortem block (`:18095-18353`) and `ret=-ETIMEDOUT` (`:18355`).
4. **Success branch**: `gsp_phys_gr_output_retire_wait` (`:16824-16888`) — whole resolved-page window,
   adaptive exits (all changed / settled 3 polls / no writes 600 ms / 1.5 s cap). On `wrote==0`,
   `gsp_phys_rmcall_gr_nowrite_retry` (`:18388-18454`) refills the bounce from the guest (`:16897-16922`),
   re-publishes a fresh ring entry, reschedules and re-waits ≤3 times (shared attempt budget).
   Then copy-out (`:18461-18538`, writes the window back regardless of writes) and **`ret=0`,
   `GUEST GR LAUNCH OK` at `:18564`**.

Client-visible outcomes:

* **After 3 outcome retries with `wrote==0`**: still success. The hold's success edge
  (`gsp_sema_service.c:11767-11768`) applies the (no-op if the engine acked) completion; the client
  proceeds and reads whatever the window holds — zeros/stale for never-written outputs (F1), correct
  for idempotent repeats (F3).
* **After the timeout path exhausts attempts**: `-ETIMEDOUT` → `svc_gr_req_bridge_fallback`
  (`:12902-12920`, no-op when `cuda_bridge=0`) → the hold failure branch (`:11736-11758`) applies the
  recorded releases **and** `svc_fwd_hold_gr_release` (`:11409`) authors the launch completion in
  software (`svc_gr_sw_release` `:11388`). Client proceeds with stale/zero data (r150 `bad=10485760`).
* **Hold expiry backstop**: `SVC_FWD_GR_HOLD_TIMEOUT_MS` = 12 s (`:10909`), checked at `:11419-11426`
  from `svc_fwd_hold_expire_check` (`:11610-11668`). Note this is *shorter* than the backend's worst
  case (5+5+3×2 s + waits ≈ 26 s): an expired hold resolves mid-flight with the software release
  (`:11649`), another fail-open edge.
* **Never-queued captures**: soft release at `:14369-14374` (F2.3).

---

## 4. Q3 — Ranked fix proposals (with soundness assessment)

### R1 (recommended first): defer "backend busy" refusals instead of soft-releasing them.

* **Where**: `gsp_sema_service.c:14274-14293` (the m150 bounded retry-hold) and `:14321-14375`.
  When `svc_gr_request` refused because `svc_fwd_hold.active`/INFLIGHT (i.e. the launch *could* run
  later), leave the entry unconsumed (no GP_GET write) under the same bounded window as the m150
  `svc_gr_data_stalled` path, instead of consuming it and authoring the release.
* **Why sound**: the capture is not executed by anything today; deferring gives it a real attempt when
  the backend frees, and the release stops being a lie. No false positives (only client latency,
  bounded by the window). This closes F2.3 — the 98-release burst.
* **Risk/needs**: pump re-capture cost and interaction with the defer/release machinery need a live
  cycle; `svc_gr_data_stalled`'s one-shot-per-entry window is the proven shape.

### R2: win0 4-bank + QMD-window double-buffer (isolation hardening; two commits).

* win0: grow `GSP_PHYS_RMCALL_GR_SHADOW0_SLOTS` 16→32 (VA 0x1e0000..0x1effff is free; static asserts at
  `:2330-2346` must be extended) and select `(epoch & 3)` banks in `:17035-17042`.
* qmdwin: plant a second page at the freed old-win0 slot 0x108 (same 128 B PT line as 0x107; the line
  is written once at stage 3, `:9565-9595`) and alternate `qmd_dst` by epoch in `:17602-17615`.
  Rationale: the qmdwin page is prewritten every launch while the previous launch's reads of the same
  VA may be in flight (F5.1).
* **Why**: either pool being re-used while reads are in flight is the best-supported remaining
  host-side mechanism (F5); both changes are strictly additive isolation and cannot make a dispatch
  worse. Correctness is conditional on the mechanism existing — validate with the F4 timeline.

### R3: try the `cuda_bridge=1` fallback operationally while R1/R2 land.

* `svc_gr_req_bridge_fallback` (`gsp_sema_service.c:12902-12920`) exists and is the designed order
  (`:11888-11890`); with the bridge on, a lost forward is executed on the host GPU instead of being
  soft-released (`:14366-14367` documents the bridge owning the release). This is not a kernel change
  (one insmod flag) and directly converts the lottery into correct results for the affected launches.

### R4: warm sem push in the outcome-retry path (experiment; the handoff's own lead).

* **Where**: `gsp_phys_booter.c:18388-18454`, call `gsp_phys_rmcall_gr_sem_push_locked`
  (`:14801-14877`) once before the first re-publish of a zero-write retry (bounded by its 5 s poll).
* **Why**: if the GR engine can still execute the known-good stage-4 semaphore block, the channel's
  dispatch state is alive and the launch retry has a chance; if the sem push times out, the engine is
  wedged and the launch is unsalvageable (fail fast instead of 4 futile attempts).
  Efficacy unknown — it is a diagnostic *and* a possible revive lever; needs one cycle.

### R5: fail-closed release — REJECTED as specified; only sound with write-provenance.

AGENT-bigcopy F1 (withhold the completion release for unwitnessed data-bearing launches) was
implemented and then reverted (F3). A sound variant needs per-page write provenance: mark a guest data
page "engine-authored" when a retirement wait observes a change on it, invalidate on host-visible CE
software copies, and fail-close only launches whose *entire* data set is unprovenanced. Even that has
corners (guest CPU writes to sysmem operands, idempotent after CE rewrite); it is a design, not a
minimal fix, and must be evaluated on its own. The disabled attempt lives on
`lottery-m156-gate-attempt` for reference.

### R6: find a *sound* execution discriminator, then gate on it.

Before any fail-closed ships, establish a signature that separates executed-but-idempotent from
not-executed. Candidates to sample during the retire wait (1-2 ms cadence, `gr_skedrec`-style):
`PGRAPH_STATUS` (0x400500-class), `SKED_ACTIVITY`, `GRFIFO` GET/PUT, `CWD_T0`, `FECS_CTXSW_STATUS`,
plus the TPC exception latches. If a signature exists, it enables both a targeted re-arm and a sound
fail-closed (fail only the provably-not-executed non-idempotent launches... still ambiguous for
idempotent, but the *provably wedged* state lets the host avoid pretending success at the sema layer
where the client's data may be wrong).

---

## 5. Q4 — Stalls

### Userspace-spin class (r153 matmul 50-min spin; fixed by m154)

* **Mechanism (proven, `e07c62b1`)**: the guest scanner's merged CE window covered a live CE channel's
  GPFIFO ring VA (0x121010000); injected pins take precedence in the ring read path, so the ring read
  the wrong PA (all-zero entry), the entry was silently consumed and the pool release was never
  authored → client spins in userspace forever.
* **Fix**: `svc_va_hits_chan_ring` (`gsp_sema_service.c:15424-15451`) drops injected pins inside any
  live channel's ring range in `svc_handle_map_add` (`:15492-15501`), counted + logged (first 8).
* **Coverage**: live in r155 — one drop fired (`log:23300 injected pin va=0x121011000 DROPPED`),
  0 in r154. The class is armed and observed; keep it.
* **r154 r2 smiley 400 s timeout (`rc=124`, un-diagnosed)**: ran *with* m154 and 0 `DROPPED` lines, so
  it is not that collision. Residual userspace-spin families that remain: (a) a different translation
  shadowing a release slot (demand pins vs scanner races), (b) a hold window that expires without an
  author (the 12 s GR hold vs 26 s backend worst case, §3), (c) CE resolution latency (the scanner's
  kprobe-gated chain discovery 35-90 s class — r147 recovered after ~90 s; a >400 s instance would
  time out the client). Detection: `rc=124` + no `STATUS` line + frozen push counters. Forensics:
  guest `wchan`/syscall of the spinning thread, `pushes`/`crel` counters in the sema summary,
  `stalled VA` list, `DROPPED` count, and the hold lines (`rel-hold direct`, `GR launch release
  software-applied`) around the client's start. No fix exists yet; capture the next occurrence early.

### Cold-context first-launch timeout

* **Mechanism**: the first launch after the engine context is cold (client start or after a save)
  enqueues to SKED but `qmd_rel` sticks at 5 (schedule/fetch ack; no retirement). r155 ok=4 is the
  textbook recovery: `GR LAUNCH FAILED` at +5 s → m155 rearm → `GR LAUNCH FAILED` at +10 s →
  `retry 1/3: fresh ring entry` → executed within 18 ms (`log:15999/16124/16175/16177/16189`,
  `elapsed_ms=10385`). The good retry state shows `cwd0=00002480 grfifo=02020001 fecs_cur=30143808`
  (`log:16176`) vs the wedged `cwd0=00012480 grfifo=00000001 act=0` (F4) — a promising discriminator
  to sample deliberately (R6).
* **Coverage**: the transient variant is recovered by m142 (re-arm) + 6b090f70 (fresh-entry retry) +
  m155 (lane tail) — cost ~10 s per cold first launch. The **persistent variant** (F4: the same
  channel/boot never dispatches again) is *not* recovered by any current mechanism; it is the same
  state as the "engine lottery" no-execute and is what produces F1's silent zeros.

---

## 6. Proposed live experiments (one variable each) + dmesg lines to watch

All runs on a fresh module load per the campaign's normal cycle; keep `cuda_bridge=0` unless the
experiment says otherwise. Always `sudo dmesg -w` follower.

**E1 — F4 trigger isolation (no code change).** Run the exact r155 order
(`mail → vecadd_loop → streams2`) plus `smiley`. Watch, per launch:
`GR LAUNCH submitted`, `sked rec +Xms cwd0=… act=… grfifo=… fecs_cur=…`,
`output retirement wait: N/205 pages changed after T ms…`, `window[N] reloc=… changed=`,
`GUEST GR LAUNCH OK …`, `GR LAUNCH FAILED: timeout…`, `retry N/3: fresh ring entry`.
Decision: if the first `0/205` launch is again the first *bank-reuse* launch (check the
`qmd_word … page copied to window-0 shadow VA=0x1eX000` lines' slot sequence), F5.1/2 rises.

**E2 — 4-bank win0 only (one commit).** §4 R2a. Watch: the first `0/205` should move to the first
reuse depth ≥4 (or disappear); `win0 param hash pre==post`, `window[…] changed=` for the executed
first-touch launch (`matmul C`/`smiley`), and `GUEST GR LAUNCH OK` with `changed>0`.

**E3 — QMD-window double-buffer (one commit, after E2).** §4 R2b. Watch the same lines plus
`qmdwin: relocating inline-QMD dest 0x…` (should alternate 0x107000/0x108000 per launch).

**E4 — defer busy refusals (one commit).** §4 R1. Watch:
`GR launch release software-applied` and `rel-hold direct … (no work)` counts must collapse;
`GR LAUNCH submitted` count for a 100-launch client must approach 100; the client's completion latency
must stay bounded. Falsifier: a capture whose entry is re-captured forever (GP_GET not advancing) or a
client exceeding its timeout.

**E5 — warm sem push on zero-write retry (one commit).** §4 R4. Watch on the first `0/205`:
`REAL GR SEM OK sem=0xc0ffee01` (engine alive) vs `REAL GR SEM FAILED: no release within 5000 ms`
(wedged); then whether the following `outcome retry N done: engine wrote U page(s)` becomes `U>0`.

**E6 — register-signature hunt (instrumentation only).** Sample the F4/R6 register set at ~2 ms during
every retire wait and log on change, for a full boot. Compare `wrote==0` (and the F3-verified
idempotent launches) against `wrote>0`. Only if a signature separates executed-idempotent from
wedged does fail-closed (or a targeted re-arm) become sound.

**E7 — bridge fallback A/B (flag only).** `cuda_bridge=1` for one cycle: all F2 outcomes should become
correct results (bridge authors data), at the cost of host-GPU execution. Watch `fwd_gr_bridge`,
`bridge` launch lines, and the client verifications (streams2/matmul/smiley first-touch clients).

---

## 7. Proven vs hypothesis

**Proven (evidence cited):**
1. `qmd_rel`/`ch_rel` are hardware-authored without SM execution in the failing class; the backend
   returns `ret=0` (F1).
2. Three host paths author releases without an execution witness (F2); 98 releases in the r155 burst.
3. Zero-write fail-closed is unsound for idempotent repeats (F3, `window[64]` proof).
4. The bad state starts at one launch and persists module-wide for the boot; the m155 lane-tail rearm
   re-enqueues + reloads the context but does not revive dispatch (F4).
5. `CWD_T0=0x12480` is not a discriminator (good ok=1/2/3/5/6 share it).
6. m154 ring-pin drop is live (1 drop in r155) and covers its class.
7. Host-side launch state (win0 param, guest param, fills) verified identical pre/post by the m151
   hashes on the failing launch.

**Hypothesis (needs E1/E2/E3/E6):**
1. 2-bank win0 / single-buffered qmdwin reuse-while-in-flight is the no-execute trigger (F5.1/2).
2. Engine-side stale const/PTE read is the stale/mixed-tuple class; the fix is more bank depth or the
   ctx-window/chunked data path (AGENT-C R3/R4).
3. Warm sem push revives dispatch (E5).
4. A register signature exists that separates wedged from executed-idempotent (E6).

**Recommended first action for the next live session:** E1 + E7 (both no-code/one-flag) on a fresh
cycle, then land E2 (4-bank) as the first code change — it is the only change whose mechanism is
supported by this boot's transition timing and cannot regress a working launch.