path: root/docs/04-implementation/05-graphics-execution-plan.md

# Graphics execution plan: real GL pixels through the emulated vGPU

> **Status: OPEN (2026-10-28).** This is the implementation-ready plan for the
> **graphics execution layer**: the component that must execute the guest
> NVIDIA vGPU driver's OpenGL/3D command streams on the physical GPU so that
> glxgears and GZDoom's OpenGL renderer produce real pixels through the
> emulated mdev. Nothing here is implemented yet beyond the pieces cited;
> every statement carries a code location or a measured session finding.
>
> Evidence base: `docs/04-implementation/03-execution-frontier.md` (the CUDA
> compute chain, proven), `docs/04-implementation/04-real-render-walls.md`
> (§8-§21, all measured graphics walls), the SESSION 149 R12-R18 round log
> (`session-149-BRIEF.md`: the M1 C597 recognizer, M2 materialization, M3
> SKEDCHECK clear, R17 real-SASS execution), and SESSION 144 addenda 9/10/16
> (the X/GL wait anatomy). The vendored 570.144 wire protocol and the public
> class headers are the interface authority; no guest driver change is
> allowed.
>
> Hard rules carried from the project: **no other GPU** (virtio-gpu/virgl,
> VirtualGL, llvmpipe, any second device are banned for this work), **no
> fake/labelled software rendering** presented as GL, **host-only emulation**
> (guest runs stock `nvidia.ko`), and **no host reboot** (AGENTS.md hard
> rule; a reboot needs the handoff update plus explicit human approval).

---

## 0. What "real pixels" means here

The acceptance target is: a stock guest GL application's draw commands are
executed by the physical GPU's **fixed-function 3D pipeline** (vertex fetch,
primitive assembly, rasterization, fragment work, ROP), its render target is
readable by the guest, and that image reaches the capture path through the
emulated mdev. The pixels must not come from Mesa, llvmpipe, virgl, a
software rasterizer, or a host display round-trip.

Two consequences drive every milestone below:

1. **A compute launch is not rasterization.** The SESSION 149 M1/M3 work
   proved the engine will execute a *synthesized* C5C0 compute launch built
   from the GL stream and will retire it
   (`poll +500ms qmd_rel=0x600df00d`, R17), but the GL client's draws need
   the 3D pipeline and the client's own per-draw state; rasterization is
   fixed-function GR (`session-149-BRIEF.md:288-291`).
2. **The GL client's own channel is the source of truth.** The C597 section
   the emulation can capture today comes from a readable channel (the X
   server side); the GL client's GPFIFO ring (`0xbeef0100` family,
   `0xbfef0002` observed in SESSION 153) is where its context-init uploads
   and per-frame draws live. Until that ring is readable, no approach can
   derive a correct draw (`session-149-BRIEF.md:438-451`).

---

## A. Gap list, ordered by dependency

Each gap lists what exists (with the code location that implements it), what
is missing, and the measurable gate that closes it. A gap is closed only when
its gate has been met on a fresh cycle with evidence kept.

### G-A Readability: the GL client's ring and pushes must read

**Exists.** The pump resolves rings through an autopin ladder (notifier-
derived, USERD-derived, trusted walk, shadow scan, USERD-shape scan) with
dead-entry rejection: `kernel/gsp_sema_service.c:16534-16602`
(`svc_ring_entry_dead`, `svc_ring_walk_live`, `svc_gr_ring_candidate`),
`:17891`, `:17972`, `:18013`, `:18075`, `:18123`, `:18242`, `:18298`. The
failure is witnessed at `:17633` (`sema-chan: GR ring h=... RAM content scan
found no candidate in N slice(s)`). Push reads report through `sema-rdw`
(`:28622`) and `sema-pushdec` (`:28868-28882`). The guest pin stream's client
selection and windows are `tools/guest-diag/guest_pin_stream.py:1429-1475`,
`tools/guest-diag/pin_gr_stream.py:134-160`, and the supervisor already
defaults to the widened GL prefix set
(`tools/guest-diag/pin_supervisor.sh:70-83`:
`cdt,va,victory,Xorg,X,glxgears,gzdoom`).

**Missing.** For the GL client channel the ring page is not found by content
at the scan instants ("GR ring h=0xbfef0002 rank=2 ... no candidate in 12
slice(s) (va=0x1203cfff0)", SESSION 153 item 3). The client's allocations are
just above the old 64 MB capture bound in the Vulkan case
(`docs/04-implementation/04-real-render-walls.md` §14-§15), and the pump's
ring candidates are shape-based; a client ring written late (after the DMA
test window) is missed. The guest pagemap stream (`#pid` + `va,pa,n` lines,
`guest_pin_stream.py:8-12`) is the authoritative VA->PA source but was not
covering the GL client until SESSION 153; the supervisor default now does.

**Gate G-A.** On a run with glxgears alive for 60 s and the standard X
recipe: (a) zero new `sema-chan: GR ring h=0x... found no candidate` rows for
the GL client's handle; (b) every client push logs `sema-pushdec ... src>0
unreadable=0`; (c) at least one entry is consumed from the client ring
(pump advances `last_get`; `sema-chan: entry[...] h=0x... eng=0` with a sane
`va/len`); (d) the X channel (`0x1000013`) stays `plausible=1` (SESSION 153
item 1).

### G-B Capture: the client's C597 sections and per-frame draws must decode

**Exists.** A gated recognizer (`sema_gl_xlate`, default 0,
`kernel/gsp_sema_service.c:6812-6815`) opens a section at a `SET_OBJECT`
class bind of `0xC597` (TURING_A) on the channel subchannel and collects
observed VA operands: program (`0x2014`/`0x2018` runs),
tex header/sampler pools (`0x1574`/`0x155c`), vertex substitute (`0x0f84`),
upload source/destination (`0x180`), and the LAUNCH_DMA/LOAD_INLINE_DATA
payload (`0x1b0`) — `:6945-7049`, `:11993-12032`. It synthesizes a 64-word
QMD and captures it through the unchanged GR candidate ring
(`:7060-7287`, `:6686`). The client's memory-writing uploads are applied by
a standalone subchannel-0 pair tracker (`:12045-12084`) and the section
walker (`:7194-7217`) through `svc_grce_inline_write` (`:8979-9060`).
Completion of the capture is gated by content trust: a plausible stream
admitted by `cap_trust` (`:28995`, `:29069-29080`), with hash-verified
rewind replays admitted (`:28513-28517`, `:29106-29119`); the flush guard
requires `seen/prog/param/rel/trusted` (`:7082-7084`). Witness lines:
`sema-gl: section captured ...` (`:7248`), `sema-gl: uploads applied=...`
(`:7213`), `sema-gl-lid: ...` (`:12073`), `sema-gl: M3 code src=...`
(`:7261`).

**Missing.**
1. The recognizer's method vocabulary covers init-shaped operands only. The
   per-draw 3D state and draw-issuing methods (vertex streams/index, render
   target and ZS, viewport/scissor, blend/depth/ROP state, draw calls) are
   not decoded (`svc_gl_field` handles six addresses,
   `:6945-7049`; the C597 stream carries far more).
2. The GL program/constant delivery is a context-init upload on the client
   channel and is lost today: R17 measured that no readable push ever writes
   `prog=0x121201b00` (`session-149-BRIEF.md:276-285`). The synthesized
   launch therefore only runs a 12-28 B fallback blob or an operator hex
   program (`sema_gl_prog_hex`, `:6877-6913`) — a development crutch, not
   pixels.
3. Capture must survive the "one-shot setup push rewritten in place"
   class: re-captures come from hash-verified replays now
   (`:29069-29080`), but the per-frame draws are separate entries and need
   the same treatment per entry.

**Gate G-B.** For ≥3 consecutive frames of a live GL client: `sema-gl:
section captured` on the **client's** handle with nonzero `prog/param/rel`,
one captured section per frame (bounded replays suppressed), and a decoded
draw record (new `sema-gl-draw` witness) whose vertex/primitive count is
consistent with the app (e.g. glxgears gear geometry). Program source
priority shows `trusted-prog` (real bytes) for the context-init upload.

### G-C Execution semantics: the GL work must run on the real 3D engine

This is section **B**; the gap gate is M-GL1/M-GL2/M-GL3.

### G-D Memory and resource mapping: every operand must be reachable by the engine and readable by the guest

**Exists.** The physical backend builds engine-side page tables and shadow
windows for launch data: Tier-1 identity install or Tier-2 shadow copy
(`kernel/gsp_phys_booter.c:19654-19724`, `:19734-19785`), a two-bank
window-0 shadow pool of 8 pages per bank (`:1174`, `:19406-19410`) with a
24-page shadow chunk (`:1317`), QMD-window relocation
(`gsp_phys_rmcall_gr_qmdwin`), and a verbatim-prelude replay path that
relocates the inline-QMD address words and data words
(`:16878-16921`). Data pages are copied in before the doorbell and copied
back after the retire wait (`:17693`, `:18243`, `:21966`, `:22242`).
`svc_grce_inline_write` writes upload bytes to vidmem shadow, guest RAM, or
the VA shadow with a once-only key (`kernel/gsp_sema_service.c:8979-9060`).

**Missing.**
1. Address-word relocation is QMD-only (words 32-49 and the indirect-page
   pointer fixup at `booter:19765-19779`). A 3D method stream names VAs at
   class-specific offsets; each must be resolved to a shadowed page and the
   word rewritten (or the page mapped at its guest VA).
2. The engine-visible page budget is small: 16 pages per request
   (`kernel/gsp_phys_booter.h:355`), 8 inline records (`:356`), 8 pages per
   win0 bank (`booter:1174`). A GL frame names program + constants + texture
   pools + vertex stream + render target + ZS; the shadow pool and the
   per-request page list must grow or pages must be grouped per buffer.
3. Render-target writeback: the copy-out tracks only the launch's own data
   pages; a 3D draw writes the RT, which must be mapped into guest-visible
   memory (BAR1 shadow or pinned guest RAM) and copied back on the
   completion edge, exactly like the current bounce/copy-out but for the RT
   and depth surfaces.
4. Program/constant bytes: R17's measurement stands — the program page is
   empty in every host-visible source until the CE/LID delivery executes
   (`docs/04-implementation/04-real-render-walls.md` §20-§21).

**Gate G-D.** Engine fetch of the 3D program/params/vertex/RT pages with
`FAULT-PDE count=0` (`vgsp-m91: FAULT-CAP ... code_src=...`,
`booter:18997-18999`), and a measurable post-launch change in the emulated
vidmem at the RT VA (>0 changed dwords in the `dump_vidmem map` class,
`docs/04-implementation/04-real-render-walls.md` §11).

### G-E Completion and sync: the guest must be released by real completions

**Exists.** Engine completion is polled (`qmd_rel/ch_rel`,
`booter:20814-20825`), the launch's release0 can be redirected to the module
semaphore and relayed (`booter:19499-19520`), the hold edge and the
backend-outcome edge author the guest word (`kernel/gsp_sema_service.c:
21329-21345`, `:21298-21322`), and failures synthesize a client-visible
channel error (`SYNTH-FAULT`, `:21400-21448`, `GR-COMPLETE-SW`, `:21504`).
The pre-built ring continuation beyond GP_PUT is swept for releases
(`sema_gr_ahead`, `:26564-26566`, `svc_gr_ahead_push`, `:26809`).

**Missing.** The guest GL client's waits are still satisfied by the give-up
ladder or by host pokes: the X server's present wait spins on the X channel
member/counter pair (S144 addendum 12; the mirror thread `sema_xw_mirror`
`kernel/main.c:3103-3129`), and the GL client's `PhaseWait` compares
`obj+0x558` against a word in the shared `memfd:/.nvidia_drv` (+0x8) that the
emulation never advances (SESSION 149 R3, `session-149-BRIEF.md:115-126`:
"one u32 write 3 → 128.8 FPS", but only via `/proc/<pid>/mem`). The correct
long-term fix is the one named in addendum 16b §C.4: **the emulation
completes the real syncs so the equality is satisfied, with no lying pokes**.

**Gate G-E.** M-GL4 (below): ≥60 fps sustained with no forcing daemon, no
`sema_xw_mirror` target written, and the guest's waits exiting on a natural
completion edge.

### G-F Present/frame path: the rendered image must land where the guest and the capture can read it

**Exists.** Guest-side capture tools: `tools/guest-diag/sdl_cap.c` (SDL swap
hook → `glReadPixels` → GZF2 record stream, `:1-18`, `:188-229`),
`tools/guest-diag/fbcopy_pub.c` (BAR1 scanout copy → kvmfr publisher),
`tools/guest-diag/qga_fetch.py` (QGA file fetch), host decoder
`tools/lg/frame_reader.py` and the resample/encode scripts.

**Missing.** On the emulated-vGPU stack the X screen is black for GL windows
(DRI3-present path; SESSION 152) and `xwd` hangs (R4/R5; no rendered pixels
exist anywhere). The present path (the C5B5 copy-class and 902D 2D parts of
the GL push, R13) is decoded for releases but not executed as a display
path, and the capture must come from the GL client's swap, not from X.

**Gate G-F.** M-GL2 (glxgears non-black) and M-GL3 (GZDoom).

### G-G Concurrency: two VMs, two mdevs

**Exists (designed in).** The service thread iterates every open device each
pass (`kernel/gsp_sema_service.c:32119-32236`), with the explicit note that
"with two, both guests get a pump turn per iteration (hardware forwarding
stays serialized by the single-inflight slot)" (`:32161-32167`). Per-guest
sessions exist (`SVC_MAX_GUESTS = 8`, `:728-736`), and the guest-RAM ranges
are registered per device at boot (`/tmp/opencode/start_vm_keep.sh:210-213`).

**Missing / to verify.** Single-inflight forwarding and holds
(`:19878`, `:25528-25545`); VA-keyed global tables that assume one guest's VAs
do not co-reside: the write/read ledger is keyed by VA page only and its own
comment says collisions "only occur ... across clients whose pages do not
co-reside in time" (`:7547-7559`), plus `svc_gl` (`:6863`),
`svc_gr_cand_ring` (`:6493`), `svc_gr_seen` (`:6547`), `svc_grpark` (`:7429`);
the win0 banks are module-global per submit epoch (`booter:19406-19410`); the
server-managed VAS relocation windows are a finite global set
(`SVC_GR_DATA_WINS_MAX 31`, `:6441`). SESSION 151 avoided a second mdev "to
avoid a second fragile RM init", so a second RM bootstrap on the same module
is unproven.

**Gate G-G.** M-GL5 (below).

---

## B. Execution-semantics approaches

### (i) 3D method-stream replay on the real engine

Forward the client's captured C597 method stream (class bind + 3D state +
upload/draw methods + in-stream releases) to the module's real GR channel
and let the fixed-function pipeline execute it.

What must be mirrored/provided:

| Requirement | Where it stands |
|---|---|
| Real execution channel/context | Mode 35 "real-client user lane" replays the 570.144 client's compute-context sequence and executes on V7 (`booter:924`, `:9022-9033`); the promote ladder builds MAIN/PM/PATCH and the golden image. |
| 3D class object (0xC597) | `GSP_PHYS_RMCALL_CLASS_GR_3D 0xc597` is defined (`booter:396`) and the alloc helper is generic (`gsp_phys_rmcall_alloc_gr_object_class`, `:8936-8964`); it is constructed on the golden lane in several modes (`:9035-9043`, `:9665`), but **not on V7** (mode 35 logs "the capture carries no 0xc597 on the user lane", `:9029-9033`) — a class object must be added on V7 and validated. |
| 3D context global buffers | Already promoted: BUNDLE_CB / PAGEPOOL / ATTRIBUTE_CB / RTV_CB_GLOBAL (`booter:531-534`, `:605-621`, golden9 sized buffers `:985-1008`). The engine context image therefore contains the 3D subcontext data. |
| Method block | `gsp_phys_rmcall_build_gr_launch` (`:16834-16947`) emits compute init + inline QMD + PCAS + report-sem; the verbatim path (`:16878-16921`) already replays a captured raw method window and rewrites its address/QMD words. A 3D mode adds `SET_OBJECT 0xC597` + the captured state/upload/draw methods instead of the compute prelude. Note the channel's GR subchannel is 1 (`:442`) and the compute init binds `0xC5C0` (`:16693-16698`); the 3D bind replaces it per section. |
| Operand pages | The page list is built from QMD words 32-49 (`:19678`, `:6673-6682`); it must be generalized to the recognized 3D operand VAs, with the shadow/alias machinery (`:19654-19785`) reused. |
| Completion/sync | Keep the stream's own releases and the module release0 redirect (`:19499-19520`); the hold/relay path already authors the guest word on the engine edge (`gsp_sema_service.c:21329-21345`). |

Pros: authentic rasterization is done by the hardware pipeline; no state is
invented; the engine has already proven it fetches and executes forwarded
work on this lane (R17 `qmd_rel=0x600df00d`); the completion machinery is
unchanged.

Cons/risks: the promoted context may still be compute-shaped (class object
construction on a wired golden channel deadlocked in earlier experiments,
`booter:816-822`; V7 is the tested path, so construct there); the 3D method
vocabulary is large (state validation can silently fault); address-word
relocation must cover every class encoding; ~6-12 operand pages per frame
exceed today's 8-page win0 bank, requiring grouping/expansion.

Effort: L. Depends on G-A, G-B.

### (ii) Synthesize a "graphics-shaped" launch from the captured C597 stream

On Turing, a QMD is a *compute* launch descriptor; there is no QMD that
rasterizes. "Graphics-shaped" therefore means: derive the draw inputs and
author the state + a DRAW method ourselves — program (`0x2014/0x2018`,
already decoded), constant buffers (bind + LID uploads, partly decoded),
vertex buffers (SET_VERTEX_STREAM_SUBSTITUTE is decoded only as a pointer,
`gsp_sema_service.c:6979-6983`; the real stream/index methods are not),
render targets/ZS (not decoded; the only observed target is the release
target `0x120001000`, R13), viewport/scissor/blend/depth state (not decoded).
The vertex count for the draw would be inferred, not read.

Pros: fewer pages to mirror than a full replay if only draw-relevant state is
bound; still uses the hardware rasterizer.

Cons: we author state the driver owns — any wrong field silently
mis-renders; the per-draw state lives in the client ring, so G-A/G-B are
still prerequisites; the program bytes are still lost without G-B; it is a
subset of (i)'s work plus extra invention.

Effort: L. Risk: high. Not first.

### (iii) Bounded compute translation for the subset GZDoom uses

Replace each GL draw with a compute kernel (or the existing synthesized
C5C0 launch carrying the client's real SASS) that rasterizes triangles into
the RT.

What breaks correctness, concretely:
- **Fixed-function raster and ROP**: depth test, stencil, alpha blending,
  culling, clipping, perspective-correct interpolation, MSAA — none exist in
  a compute launch; reimplementing them is writing a GPU.
- **Sampling**: mipmaps, LOD selection, anisotropic filtering, wrap modes
  and sRGB conversion are texture-unit behaviour.
- **Shader semantics**: the client's GLSL is already compiled to SASS by the
  guest driver; executing that SASS as a whole-SM compute program does not
  give it vertex inputs, varying interpolation, or per-fragment coverage —
  the very things the 3D pipeline supplies.
- **The claim itself**: even GZDoom's first frame uses FBOs
  (scene → post-process), depth testing, alpha blending and mipmapped
  textures; output would be a lookalike, not the renderer's pixels, and
  could not be verified against a reference.

Verdict: acceptable only as a **diagnostic probe** (e.g. a known-pattern
kernel proving the lane), never as the milestone path, and it must be
labelled as such. Not first.

### Recommendation

**Implement (i) first.** It is the only approach that yields authentic GL
pixels; its difficult prerequisites (G-A ring readability, G-B stream
capture) are shared by every alternative; and the backend already provides
the three pieces that make it tractable: a real execution lane with a
validated context (mode 35), a generic 0xC597 class-object allocator, and a
verbatim method-window replay with address/QMD rewriting plus page shadowing
and completion relay. (ii) is (i)'s draw-issuing step taken prematurely;
(iii) trades correctness for nothing.

Order of implementation inside (i): **G-A** → **G-B** → replay the captured
*context-init* section (M-GL1, proves class bind + uploads + completion on
the 3D lane) → replay the client's per-frame *draw* sections (M-GL2/3).

---

## C. Milestone ladder

All gates are measured with the artifacts listed in section D; a milestone is
met only when its gate reproduces on **two consecutive fresh cycles** (three
for M-GL1) with the module hash recorded.

### M-GL1 — Client context-init work decodes and executes; the client leaves its retry loop

Gate (all in one run):
1. GL client ring resolved: no `sema-chan: GR ring h=<client> ... found no
   candidate` row after the first minute; an autopin row exists
   (`sema-chan: auto-pinned GR ring ...`).
2. Client pushes read: `sema-pushdec` rows for the client handle with
   `src>0 unreadable=0 plausible=1`; no `sema-rdw ... rc=-2` for them.
3. Section captured from the client: `sema-gl: section captured h=<client>`
   with `prog/param/rel` nonzero and `replay=0` (or the hash-verified replay
   admission), at least once per client lifetime.
4. Forwarded and executed: `vgsp-gr: GR LAUNCH submitted h=<client>
   ... pages>=2 shadow=1`, followed by engine evidence (`vgsp-gr: poll +...
   qmd_rel=0x600df00d`) and either a natural release or `GR-COMPLETE-SW`.
5. Client behaviour: glxgears no longer loops on context creation (no
   repeated init-failure retry; the X server stays alive; `xdpyinfo` OK,
   `DMAFAIL=0`), and the app reaches its first swap.
6. Bounded failure hygiene: no SYNTH-FAULT storm for the accepted client
   (≤ a handful in the window), no guest oops.

Expected failure signatures (test one at a time):
`GR ring ... found no candidate` (G-A);
`unreadable=1 first8=00000000` (G-A, the S147 addendum 9 zero-page class);
`GR request without candidate` / `cand_valid=0`
(`gsp_sema_service.c:25479-25485`; G-B ordering: capture must precede the
request);
`GR submit refused: degenerate data set n_data=N (min M)`
(`booter:19352`; operand list, G-D);
`FAULT-PDE` at the program VA + `CODEHASH h0=0`
(`booter:18806-18883`, `:18997`; G-B/G-D);
`GR LAUNCH FAILED: timeout ... qmd_rel=...` (`booter:20825`; engine did not
retire — class/sched fields or completion wiring).

### M-GL2 — glxgears renders non-black frames

Gate:
1. A GZF2 record stream is captured from the **GL client's swap** (see D for
   the GLX shim; the repo's `sdl_cap.c` is SDL-only — glxgears does not use
   SDL).
2. ≥1 frame with ≥5% of pixels above luminance 8/255 and ≥2 distinct
   dominant colours; two frames ≥500 ms apart differ in ≥1% of pixels
   (motion, not a static noise tile).
3. The same time window contains the per-frame GR launch + completion in
   dmesg, and the guest's `glxinfo` renderer is the NVIDIA driver (not Mesa).
4. The first frames are reached without any sync poke. If a diagnostic poke
   is used to unblock the run, that run is labelled diagnostics and does not
   satisfy the gate.

Failure signatures: swap hook fires but frames all zero → no rasterization
(G-C); `xwd`/X path black → expected, do not use; frames show only clear
colour → draws executed but state/vertex data wrong (G-D);
`FAULT-CAP ... code_src=guest-hpa code64=000...` → program source lost
(G-B).

### M-GL3 — GZDoom GL opens a window and presents non-black frames through the mdev

Gate:
1. GZDoom (`vid_rendermode=4`, `vid_preferbackend=0`) runs on `DISPLAY=:0`
   backed by the emulated vGPU (no Xvfb, no Mesa overrides).
2. ≥300 consecutive non-black GZF2 frames from the SDL swap hook, including
   gameplay (not only the menu); the in-frame FPS counter ≥30.
3. dmesg shows one GR launch per rendered frame (bounded), no SYNTH-FAULT
   loop; the X server stays alive for the whole run.
4. Capture fetched via QGA and decoded offline; a screenshot
   (`+screenshot`) matches the captured frame within a small pixel delta.

Failure signatures: window opens, swap fires, all frames black → draws not
executed; `GR submit refused ... n_data` → operand pages; app crashes with
`Illegal Instruction Encoding` (ESR 0x504730=0x9) → program bytes are not
SASS (G-B; cf. R18's single-page kernel lesson); `SYNTH-FAULT` per frame →
forward failed or refused.

### M-GL4 — 60 fps sustained in ONE VM, no forcing

Gate:
1. glxgears: "N frames in T seconds = X FPS" with X ≥ 60 over a 60 s
   measurement (`vblank_mode=0 __GL_SYNC_TO_VBLANK=0` for the measurement
   itself), and GZDoom's counter ≈60.
2. **No** poke/force/mirror daemon is running and no runtime module param is
   used to fake progress: `sema_xw_mirror` is `0,0` (stopped,
   `kernel/main.c:3103-3129`), no `/proc/<pid>/mem` write helper, no
   strike/deadline forcing.
3. GZF2 inter-frame deltas: p95 ≤ 17 ms over 10 s (capture path keeps up).
4. The X present waits exit with member advancing naturally (Xorg log WAIT
   lines absent or resolving without the kickfix daemon), and the glcore
   equality word no longer requires the bump.
5. CUDA regression: `victory_mail` `VADD n=32768 bad=0` / `STATUS OK` on the
   same boot after the GL run (battery-first ordering).

Failure signature: the two ~11 s glcore waits return (the app's own strike
ladder; S144 addendum 16b) → G-E; X WAIT with static member → present
completion not authored; `GR-STALL-backend` per frame → completion is
software-only and the app's ladder is what "ran".

### M-GL5 — Two VMs, two mdevs concurrently

Verification checklist before the gate:
1. Second mdev creates and its guest RM bootstraps (the S151 open question:
   "a second fragile RM init"); both devices appear in the service loop
   (`gsp_sema_service.c:32119-32236`).
2. Per-guest isolation of the shared tables: `svc_ledger` VA-only key
   (`:7547-7559`), `svc_gl` (`:6863`), `svc_gr_cand_ring` (`:6493`),
   `svc_gr_seen` (`:6547`), `svc_grpark` (`:7429`); either prove guest VAs do
   not collide in practice or key them by `(vgpu, va)`.
3. Guest RAM: both VMs' memory-backend ranges registered and resolving
   (the cycle registers one file per guest; verify the per-device map).
4. Server VAS relocation: no two guests' data windows alias in the managed
   region (`SVC_GR_DATA_WINS_MAX 31`, `:6441`; comment `:6436-6439`).
5. Win0 banks/epochs (`booter:19406-19410`) are safe under serialized
   submissions; single-inflight holds (`:19878`) do not starve one guest.

Gate: both VMs run glxgears and GZDoom for ≥10 minutes; both present
non-black frames; each ≥30 fps (60 target); `victory_mail` green in both
afterwards; no cross-guest data anomalies (`sema-ledger: DIVERGE` /
`LEDGER-RESET` absent), no host oops, no guest wedge.

Failure signatures: one VM starves the other's forward slot (`GR request
busy: hold active`, `:25509`); second guest's launches resolve the first
guest's pages (wrong results / ledger rewrite storms, `:7584-7594`); second
RM init fails in the boot ladder.

### M-GL6 — Flawless 60 fps split-screen video

Gate: 600+ frames at 60/1 fps, ≥95% unique frames, both halves show real
GZDoom GL gameplay through the two mdevs, encoded with the existing
resample/compose pipeline, uploaded with the files-upload skill and
pull-back-md5 verified. Encode with `libx264` (the R18 path did:
`/tmp/opencode/r18_final.sh`); NVENC is unavailable while the module owns
the physical GPU (the host NVIDIA driver is displaced).

---

## D. Experiment recipes

### D.0 Standard cycle (all milestones)

Host, one controlled attempt per cycle (script leaves the VM running):

```
nohup /tmp/opencode/start_vm_keep.sh rNNN "<EXTRA_ARGS>" \
  > /tmp/opencode/rNNN-cycle.log 2>&1 &
# poll until: DONE (no teardown; VM left running)   (~70-90 s)
```

`EXTRA_ARGS` = the SESSION 149 standard profile
(`session-149-BRIEF.md:65-67`), verbatim:

```
gsp_phys_gr_nowrite_retries=0 gsp_phys_gr_retry_fast=1
gsp_phys_gr_first_poll_ms=2500 gsp_sema_winfill=1 gsp_sema_gr_swrel_gate=1
gsp_phys_rmcall_gr_promote=35 gsp_phys_rmcall_fwclient_mask=0xE3F
gsp_phys_rmcall_srv_ctxcommit=0 gsp_phys_rmcall_gr_qmdwin=1
gsp_phys_rmcall_gr_shadow_win0=1 gsp_phys_rmcall_gr_lmem_en=1
gsp_phys_rmcall_gr_skedrec=1 gsp_phys_rmcall_gr_inst_sys=1
gsp_phys_rmcall_gr_r1_first=1 gsp_phys_gr_strict_reloc=0
gsp_phys_fault_relay=1 sema_bigcopy_log=512
```

GL overlay (M-GL1..M-GL3 runs): append

```
sema_gl_xlate=1 sema_push_raw=4 gsp_phys_gr_min_data=0
sema_gr_util_scan=0 sema_dbg=Y
```

(`sema_gl_xlate` is off by default in-tree, `gsp_sema_service.c:6812`;
`min_data=0` is the explicit no-external-data bypass validated in S143 and
used by R15-R17; `util_scan>0` is a measured regression source, keep 0;
`sema_dbg=Y` raises the bounded witnesses). The base script's ladder already
passes `gr_promote=25`; the later `=35` wins at insmod and is snapshotted by
the boot ladder — verify the boot log names mode 35
(`booter:8612-8613`).

Post-load, before Xorg:

```
sudo sh -c 'echo 1000 > /sys/module/vgsp/parameters/sema_vas_cand_ms'
sudo nohup dmesg -w > /tmp/opencode/dmesg_live_rNNN.log 2>&1 &
```

Guest Xorg (one per boot; ~50 % of boots fail DMA and need a re-cycle):

```
echo ubuntu | sudo -S setsid bash -c \
 'nohup Xorg :0 -ac -noreset -logfile /tmp/xorg.log >/tmp/xorg.out 2>&1 </dev/null &'
# require: DISPLAY=:0 xdpyinfo -> 1280x720; grep -c "Failed to initialize DMA" /tmp/xorg.log == 0
```

After Xorg is up:

```
sudo sh -c 'echo 1 > /sys/module/vgsp/parameters/sema_force_flush'
```

Evidence per run, always kept: module md5 (`md5sum kernel/vgsp.ko`), the
echoed insmod args, the full dmesg capture, guest `/tmp/xorg.log`,
`/tmp/glx*.log`, the GZF2 file and its md5, the decoded frame PPMs, and the
acceptance-grep output.

### D.1 M-GL1 recipe

Guest: start glxgears and watch the client channel:

```
DISPLAY=:0 setsid nohup stdbuf -oL glxgears -geometry 800x600+0+0 \
  > /tmp/glx1.log 2>&1 </dev/null &
```

Host acceptance greps on the dmesg capture (client handle filled from the
first `sema-chan` row):

```
grep -E "auto-pinned GR ring h=0x<client>"            # G-A pass
grep -E "sema-pushdec: h=0x<client>.*unreadable=0"    # G-A pass
grep -E "sema-gl: section captured h=0x<client>"      # G-B pass
grep -E "GR LAUNCH submitted h=0x<client>"            # capture->forward
grep -E "qmd_rel=0x600df00d|GR-COMPLETE-SW"           # engine executed/completed
```

If G-A fails: confirm the guest scanner is running the widened prefixes
(`/tmp/guest_pin_stream.log`; supervisor env at
`tools/guest-diag/pin_supervisor.sh:70-83`), force a demand resolve via
`sema_trace_h=<client handle>` (S148 addendum 10) to see which source fails,
and use `echo <va> >> /tmp/guest_want_vas.txt` for a targeted pin. If G-B
fails with `entry_replay=1` keeping `cap_trust=0`, the pass is an un-hashed
replay: wait one fresh push or use the hash-admitted path
(`gsp_sema_service.c:29069-29080`) — do not add a bypass.

Expected time-to-first-signal: minutes. Do not proceed to M-GL2 until 3
cycles pass.

### D.2 M-GL2 recipe (glxgears pixels)

The repo shim is SDL-only (`sdl_cap.c:188-229` hooks `SDL_GL_SwapWindow`,
`SDL_RenderPresent`, `SDL_UpdateWindowSurface`); glxgears is GLX and calls
neither. Add a GLX mode to the same tool (hooks `glXSwapBuffers` →
`glReadPixels`, same GZF2 record layout), build:

```
gcc -shared -fPIC -O2 -o /tmp/glx_cap.so glx_cap.c -ldl
```

Guest:

```
DISPLAY=:0 GLXCAP=/dev/shm/glxframes.bin GZCAP_MAXF=0 \
  LD_PRELOAD=/tmp/glx_cap.so setsid nohup stdbuf -oL \
  glxgears -geometry 800x600+0+0 > /tmp/glx2.log 2>&1 </dev/null &
```

Fetch + decode on the host:

```
python3 tools/guest-diag/qga_fetch.py /dev/shm/glxframes.bin /tmp/opencode/glxframes.bin
# parse GZF2 records (tools/lg/frame_reader.py / gz_final_gpu.py's reader),
# then measure: % pixels > 8/255, colour count, motion between frames.
```

Do not use `vblank_mode=0` for the first pixel check (sync-on is fine); keep
the shim's swap hook ordered as "capture then forward" (same as sdl_cap).

### D.3 M-GL3 recipe (GZDoom GL)

Guest (xorg.conf untouched, the emulated GPU as the only GPU):

```
VK_ICD_FILENAMES=/nonexistent.json VK_DRIVER_FILES=/nonexistent.json \
GZCAP=/dev/shm/gzframes.bin GZCAP_MAXF=0 LD_PRELOAD=/tmp/sdl_cap.so \
DISPLAY=:0 setsid nohup ./gzdoom -iwad freedoom2.wad \
  +vid_rendermode 4 +vid_preferbackend 0 +vid_activeinbackground true \
  +sv_cheats 1 +noclip > /tmp/gz.log 2>&1 </dev/null &
# window management with tools/guest-diag/{xresize,xfocus,xtest_move}.c as in
# SESSION 151; XTEST alone is ignored by SDL here - use the XSendEvent path.
```

Host: `qga_fetch.py /dev/shm/gzframes.bin /tmp/opencode/gz.bin`, decode, then
`/tmp/opencode/gz_final_gpu.py` (resample to natural-speed 60 fps) for the
video. **Do not** set `LIBGL_ALWAYS_SOFTWARE` or the Mesa EGL vendor list:
that is the S151 software path and would invalidate the milestone.

### D.4 M-GL4 recipe (60 fps, no forcing)

Same as D.1/D.3, plus:

- ensure `sema_xw_mirror` is stopped (`echo 0,0 > .../sema_xw_mirror`) and no
  forcing/kickfix daemon is running (`pgrep -af` exact names first);
- measure 60 s windows only: glxgears FPS line, GZDoom counter, GZF2
  timestamps;
- keep `sema_gr_grace_ms`/`sema_ce_grace_ms` at their defaults; the R1 A/B
  proved they are not the pacer (`session-149-BRIEF.md:96-99`);
- battery-first CUDA regression in the same boot.

### D.5 M-GL5 recipe (two VMs)

Second mdev + second VM (own disk copy, own sockets, own SSH/QGA ports), both
with the vGPU device this time. Verify in order: second mdev appears; second
guest's `nvidia-smi`/`glxinfo` sees the vGPU; service log shows both device
handles per pump pass (`sema-chan` rows for both); then run the D.1/D.3 apps
in both. Keep per-VM captures separate (two GZF2 files, two QGA sockets).

### D.6 M-GL6 recipe (video)

Per-VM SDL capture → `qga_fetch.py` → `/tmp/opencode/gz_final_gpu.py`
(timestamp resample + split-screen compose) → `ffmpeg -c:v libx264` (R18
pattern) → files-upload skill → pull-back md5 verification. 1280x440
split (640x440 per guest) matches the two delivered videos.

---

## E. What must NOT be done

1. **No other GPUs or virtual GPUs.** No virtio-gpu/virgl, no VirtualGL, no
   llvmpipe/Mesa software GL, no second physical GPU, no host display
   round-trip. The SESSION 152 video is explicitly *not* evidence for this
   plan.
2. **No fake or mislabelled rendering.** No software rasterizer or compute
   lookalike presented as GZDoom's OpenGL renderer. A compute-translation
   probe (approach iii) may exist only as a labelled diagnostic and is never
   an acceptance artifact.
3. **No lying sync pokes in accepted runs.** The `/proc/<pid>/mem` counter
   bump, strike forcing, kickfix and `sema_xw_mirror` are diagnostics; any
   milestone evidence must state they were absent.
4. **No host reboot.** AGENTS.md hard rule: handoff update first, explicit
   human approval, never automatic. Guest VM cycles and Xorg restarts on the
   test VM are the allowed recovery paths.
5. **No guest modifications.** Stock guest `nvidia.ko` and stock apps; no
   injected code, patched firmware or modified driver in the guest; capture
   shims are LD_PRELOAD observation only and must not alter GL semantics.
6. **No time/timing claims from unsupported sources.** The physical backend
   is the execution authority; simulator (L0/L1) findings never substitute
   for a live engine result.
7. **No unguarded experiments on the physical GPU.** Every cycle is
   supervised, single-attempt, evidence-captured (the cycle script's hard
   abort signatures are not bypassed).

---

## F. Open questions and note-vs-code discrepancies

1. **`sdl_cap.c` is SDL-only.** The milestone recipe in the SESSION 149 brief
   names it for glxgears, but glxgears is GLX and never calls
   `SDL_GL_SwapWindow` (`tools/guest-diag/sdl_cap.c:6-9`, `:188`). A GLX
   variant is required (D.2); `sdl_cap.c` remains the right shim for GZDoom
   (M-GL3).
2. **The GL-client pin prefix fix already looks landed.**
   `tools/guest-diag/pin_supervisor.sh:73-83` defaults
   `CLIENT_PREFIXES=cdt,va,victory,Xorg,X,glxgears,gzdoom` and the cycle
   starts the supervisor (`start_vm_keep.sh:242`), while SESSION 153 still
   lists the "permanent fix" as pending. Verify the running guest scanner
   actually received the widened env (`/tmp/guest_pin_stream.log`) before
   treating G-A as closed.
3. **The logged failure strings differ slightly from the briefs.** The code
   prints `vgsp-work: GR request without candidate`
   (`gsp_sema_service.c:25481`), not bare "GR request without candidate"; the
   ring failure is `sema-chan: GR ring h=... found no candidate`
   (`:17633`). Grep acceptance scripts for the exact prefixes.
4. **`n_data` threshold history.** The default is 16
   (`booter:17709-17717`); S143+ cycles used `min_data=0` as an explicit
   bypass for the no-external-data launch class, enabled by the r800 guard.
   The brief's "min 1"/"min 16" variants refer to different experiments; the
   GL profile below uses 0 and must say so in every evidence record.
5. **Stale project status row.** `docs/README.md:114` still lists CUDA
   execution as "no grid has dispatched yet", which predates the proven
   physical forward and the R18 CUDA video. This plan relies on the newer
   code and `docs/04-implementation/03-execution-frontier.md`, not that row.
6. **The 3D context on the execution lane is the main unknown.** Mode 35's
   capture "carries no 0xc597 on the user lane"
   (`booter:9022-9033`); golden-lane 3D objects exist (`:9035-9043`) but the
   execution channel is V7. The first implementation experiment must prove a
   0xC597 class object can be constructed on V7 after the accepted promote
   (same helper, `:8936`) without the wired-golden deadlock class
   (`:816-822`), before any stream replay work.