# Graphics execution plan: real GL pixels through the emulated vGPU > **Status: OPEN (2026-10-28).** This is the implementation-ready plan for the > **graphics execution layer**: the component that must execute the guest > NVIDIA vGPU driver's OpenGL/3D command streams on the physical GPU so that > glxgears and GZDoom's OpenGL renderer produce real pixels through the > emulated mdev. Nothing here is implemented yet beyond the pieces cited; > every statement carries a code location or a measured session finding. > > Evidence base: `docs/04-implementation/03-execution-frontier.md` (the CUDA > compute chain, proven), `docs/04-implementation/04-real-render-walls.md` > (§8-§21, all measured graphics walls), the SESSION 149 R12-R18 round log > (`session-149-BRIEF.md`: the M1 C597 recognizer, M2 materialization, M3 > SKEDCHECK clear, R17 real-SASS execution), and SESSION 144 addenda 9/10/16 > (the X/GL wait anatomy). The vendored 570.144 wire protocol and the public > class headers are the interface authority; no guest driver change is > allowed. > > Hard rules carried from the project: **no other GPU** (virtio-gpu/virgl, > VirtualGL, llvmpipe, any second device are banned for this work), **no > fake/labelled software rendering** presented as GL, **host-only emulation** > (guest runs stock `nvidia.ko`), and **no host reboot** (AGENTS.md hard > rule; a reboot needs the handoff update plus explicit human approval). --- ## 0. What "real pixels" means here The acceptance target is: a stock guest GL application's draw commands are executed by the physical GPU's **fixed-function 3D pipeline** (vertex fetch, primitive assembly, rasterization, fragment work, ROP), its render target is readable by the guest, and that image reaches the capture path through the emulated mdev. The pixels must not come from Mesa, llvmpipe, virgl, a software rasterizer, or a host display round-trip. Two consequences drive every milestone below: 1. **A compute launch is not rasterization.** The SESSION 149 M1/M3 work proved the engine will execute a *synthesized* C5C0 compute launch built from the GL stream and will retire it (`poll +500ms qmd_rel=0x600df00d`, R17), but the GL client's draws need the 3D pipeline and the client's own per-draw state; rasterization is fixed-function GR (`session-149-BRIEF.md:288-291`). 2. **The GL client's own channel is the source of truth.** The C597 section the emulation can capture today comes from a readable channel (the X server side); the GL client's GPFIFO ring (`0xbeef0100` family, `0xbfef0002` observed in SESSION 153) is where its context-init uploads and per-frame draws live. Until that ring is readable, no approach can derive a correct draw (`session-149-BRIEF.md:438-451`). --- ## A. Gap list, ordered by dependency Each gap lists what exists (with the code location that implements it), what is missing, and the measurable gate that closes it. A gap is closed only when its gate has been met on a fresh cycle with evidence kept. ### G-A Readability: the GL client's ring and pushes must read **Exists.** The pump resolves rings through an autopin ladder (notifier- derived, USERD-derived, trusted walk, shadow scan, USERD-shape scan) with dead-entry rejection: `kernel/gsp_sema_service.c:16534-16602` (`svc_ring_entry_dead`, `svc_ring_walk_live`, `svc_gr_ring_candidate`), `:17891`, `:17972`, `:18013`, `:18075`, `:18123`, `:18242`, `:18298`. The failure is witnessed at `:17633` (`sema-chan: GR ring h=... RAM content scan found no candidate in N slice(s)`). Push reads report through `sema-rdw` (`:28622`) and `sema-pushdec` (`:28868-28882`). The guest pin stream's client selection and windows are `tools/guest-diag/guest_pin_stream.py:1429-1475`, `tools/guest-diag/pin_gr_stream.py:134-160`, and the supervisor already defaults to the widened GL prefix set (`tools/guest-diag/pin_supervisor.sh:70-83`: `cdt,va,victory,Xorg,X,glxgears,gzdoom`). **Missing.** For the GL client channel the ring page is not found by content at the scan instants ("GR ring h=0xbfef0002 rank=2 ... no candidate in 12 slice(s) (va=0x1203cfff0)", SESSION 153 item 3). The client's allocations are just above the old 64 MB capture bound in the Vulkan case (`docs/04-implementation/04-real-render-walls.md` §14-§15), and the pump's ring candidates are shape-based; a client ring written late (after the DMA test window) is missed. The guest pagemap stream (`#pid` + `va,pa,n` lines, `guest_pin_stream.py:8-12`) is the authoritative VA->PA source but was not covering the GL client until SESSION 153; the supervisor default now does. **Gate G-A.** On a run with glxgears alive for 60 s and the standard X recipe: (a) zero new `sema-chan: GR ring h=0x... found no candidate` rows for the GL client's handle; (b) every client push logs `sema-pushdec ... src>0 unreadable=0`; (c) at least one entry is consumed from the client ring (pump advances `last_get`; `sema-chan: entry[...] h=0x... eng=0` with a sane `va/len`); (d) the X channel (`0x1000013`) stays `plausible=1` (SESSION 153 item 1). ### G-B Capture: the client's C597 sections and per-frame draws must decode **Exists.** A gated recognizer (`sema_gl_xlate`, default 0, `kernel/gsp_sema_service.c:6812-6815`) opens a section at a `SET_OBJECT` class bind of `0xC597` (TURING_A) on the channel subchannel and collects observed VA operands: program (`0x2014`/`0x2018` runs), tex header/sampler pools (`0x1574`/`0x155c`), vertex substitute (`0x0f84`), upload source/destination (`0x180`), and the LAUNCH_DMA/LOAD_INLINE_DATA payload (`0x1b0`) — `:6945-7049`, `:11993-12032`. It synthesizes a 64-word QMD and captures it through the unchanged GR candidate ring (`:7060-7287`, `:6686`). The client's memory-writing uploads are applied by a standalone subchannel-0 pair tracker (`:12045-12084`) and the section walker (`:7194-7217`) through `svc_grce_inline_write` (`:8979-9060`). Completion of the capture is gated by content trust: a plausible stream admitted by `cap_trust` (`:28995`, `:29069-29080`), with hash-verified rewind replays admitted (`:28513-28517`, `:29106-29119`); the flush guard requires `seen/prog/param/rel/trusted` (`:7082-7084`). Witness lines: `sema-gl: section captured ...` (`:7248`), `sema-gl: uploads applied=...` (`:7213`), `sema-gl-lid: ...` (`:12073`), `sema-gl: M3 code src=...` (`:7261`). **Missing.** 1. The recognizer's method vocabulary covers init-shaped operands only. The per-draw 3D state and draw-issuing methods (vertex streams/index, render target and ZS, viewport/scissor, blend/depth/ROP state, draw calls) are not decoded (`svc_gl_field` handles six addresses, `:6945-7049`; the C597 stream carries far more). 2. The GL program/constant delivery is a context-init upload on the client channel and is lost today: R17 measured that no readable push ever writes `prog=0x121201b00` (`session-149-BRIEF.md:276-285`). The synthesized launch therefore only runs a 12-28 B fallback blob or an operator hex program (`sema_gl_prog_hex`, `:6877-6913`) — a development crutch, not pixels. 3. Capture must survive the "one-shot setup push rewritten in place" class: re-captures come from hash-verified replays now (`:29069-29080`), but the per-frame draws are separate entries and need the same treatment per entry. **Gate G-B.** For ≥3 consecutive frames of a live GL client: `sema-gl: section captured` on the **client's** handle with nonzero `prog/param/rel`, one captured section per frame (bounded replays suppressed), and a decoded draw record (new `sema-gl-draw` witness) whose vertex/primitive count is consistent with the app (e.g. glxgears gear geometry). Program source priority shows `trusted-prog` (real bytes) for the context-init upload. ### G-C Execution semantics: the GL work must run on the real 3D engine This is section **B**; the gap gate is M-GL1/M-GL2/M-GL3. ### G-D Memory and resource mapping: every operand must be reachable by the engine and readable by the guest **Exists.** The physical backend builds engine-side page tables and shadow windows for launch data: Tier-1 identity install or Tier-2 shadow copy (`kernel/gsp_phys_booter.c:19654-19724`, `:19734-19785`), a two-bank window-0 shadow pool of 8 pages per bank (`:1174`, `:19406-19410`) with a 24-page shadow chunk (`:1317`), QMD-window relocation (`gsp_phys_rmcall_gr_qmdwin`), and a verbatim-prelude replay path that relocates the inline-QMD address words and data words (`:16878-16921`). Data pages are copied in before the doorbell and copied back after the retire wait (`:17693`, `:18243`, `:21966`, `:22242`). `svc_grce_inline_write` writes upload bytes to vidmem shadow, guest RAM, or the VA shadow with a once-only key (`kernel/gsp_sema_service.c:8979-9060`). **Missing.** 1. Address-word relocation is QMD-only (words 32-49 and the indirect-page pointer fixup at `booter:19765-19779`). A 3D method stream names VAs at class-specific offsets; each must be resolved to a shadowed page and the word rewritten (or the page mapped at its guest VA). 2. The engine-visible page budget is small: 16 pages per request (`kernel/gsp_phys_booter.h:355`), 8 inline records (`:356`), 8 pages per win0 bank (`booter:1174`). A GL frame names program + constants + texture pools + vertex stream + render target + ZS; the shadow pool and the per-request page list must grow or pages must be grouped per buffer. 3. Render-target writeback: the copy-out tracks only the launch's own data pages; a 3D draw writes the RT, which must be mapped into guest-visible memory (BAR1 shadow or pinned guest RAM) and copied back on the completion edge, exactly like the current bounce/copy-out but for the RT and depth surfaces. 4. Program/constant bytes: R17's measurement stands — the program page is empty in every host-visible source until the CE/LID delivery executes (`docs/04-implementation/04-real-render-walls.md` §20-§21). **Gate G-D.** Engine fetch of the 3D program/params/vertex/RT pages with `FAULT-PDE count=0` (`vgsp-m91: FAULT-CAP ... code_src=...`, `booter:18997-18999`), and a measurable post-launch change in the emulated vidmem at the RT VA (>0 changed dwords in the `dump_vidmem map` class, `docs/04-implementation/04-real-render-walls.md` §11). ### G-E Completion and sync: the guest must be released by real completions **Exists.** Engine completion is polled (`qmd_rel/ch_rel`, `booter:20814-20825`), the launch's release0 can be redirected to the module semaphore and relayed (`booter:19499-19520`), the hold edge and the backend-outcome edge author the guest word (`kernel/gsp_sema_service.c: 21329-21345`, `:21298-21322`), and failures synthesize a client-visible channel error (`SYNTH-FAULT`, `:21400-21448`, `GR-COMPLETE-SW`, `:21504`). The pre-built ring continuation beyond GP_PUT is swept for releases (`sema_gr_ahead`, `:26564-26566`, `svc_gr_ahead_push`, `:26809`). **Missing.** The guest GL client's waits are still satisfied by the give-up ladder or by host pokes: the X server's present wait spins on the X channel member/counter pair (S144 addendum 12; the mirror thread `sema_xw_mirror` `kernel/main.c:3103-3129`), and the GL client's `PhaseWait` compares `obj+0x558` against a word in the shared `memfd:/.nvidia_drv` (+0x8) that the emulation never advances (SESSION 149 R3, `session-149-BRIEF.md:115-126`: "one u32 write 3 → 128.8 FPS", but only via `/proc//mem`). The correct long-term fix is the one named in addendum 16b §C.4: **the emulation completes the real syncs so the equality is satisfied, with no lying pokes**. **Gate G-E.** M-GL4 (below): ≥60 fps sustained with no forcing daemon, no `sema_xw_mirror` target written, and the guest's waits exiting on a natural completion edge. ### G-F Present/frame path: the rendered image must land where the guest and the capture can read it **Exists.** Guest-side capture tools: `tools/guest-diag/sdl_cap.c` (SDL swap hook → `glReadPixels` → GZF2 record stream, `:1-18`, `:188-229`), `tools/guest-diag/fbcopy_pub.c` (BAR1 scanout copy → kvmfr publisher), `tools/guest-diag/qga_fetch.py` (QGA file fetch), host decoder `tools/lg/frame_reader.py` and the resample/encode scripts. **Missing.** On the emulated-vGPU stack the X screen is black for GL windows (DRI3-present path; SESSION 152) and `xwd` hangs (R4/R5; no rendered pixels exist anywhere). The present path (the C5B5 copy-class and 902D 2D parts of the GL push, R13) is decoded for releases but not executed as a display path, and the capture must come from the GL client's swap, not from X. **Gate G-F.** M-GL2 (glxgears non-black) and M-GL3 (GZDoom). ### G-G Concurrency: two VMs, two mdevs **Exists (designed in).** The service thread iterates every open device each pass (`kernel/gsp_sema_service.c:32119-32236`), with the explicit note that "with two, both guests get a pump turn per iteration (hardware forwarding stays serialized by the single-inflight slot)" (`:32161-32167`). Per-guest sessions exist (`SVC_MAX_GUESTS = 8`, `:728-736`), and the guest-RAM ranges are registered per device at boot (`/tmp/opencode/start_vm_keep.sh:210-213`). **Missing / to verify.** Single-inflight forwarding and holds (`:19878`, `:25528-25545`); VA-keyed global tables that assume one guest's VAs do not co-reside: the write/read ledger is keyed by VA page only and its own comment says collisions "only occur ... across clients whose pages do not co-reside in time" (`:7547-7559`), plus `svc_gl` (`:6863`), `svc_gr_cand_ring` (`:6493`), `svc_gr_seen` (`:6547`), `svc_grpark` (`:7429`); the win0 banks are module-global per submit epoch (`booter:19406-19410`); the server-managed VAS relocation windows are a finite global set (`SVC_GR_DATA_WINS_MAX 31`, `:6441`). SESSION 151 avoided a second mdev "to avoid a second fragile RM init", so a second RM bootstrap on the same module is unproven. **Gate G-G.** M-GL5 (below). --- ## B. Execution-semantics approaches ### (i) 3D method-stream replay on the real engine Forward the client's captured C597 method stream (class bind + 3D state + upload/draw methods + in-stream releases) to the module's real GR channel and let the fixed-function pipeline execute it. What must be mirrored/provided: | Requirement | Where it stands | |---|---| | Real execution channel/context | Mode 35 "real-client user lane" replays the 570.144 client's compute-context sequence and executes on V7 (`booter:924`, `:9022-9033`); the promote ladder builds MAIN/PM/PATCH and the golden image. | | 3D class object (0xC597) | `GSP_PHYS_RMCALL_CLASS_GR_3D 0xc597` is defined (`booter:396`) and the alloc helper is generic (`gsp_phys_rmcall_alloc_gr_object_class`, `:8936-8964`); it is constructed on the golden lane in several modes (`:9035-9043`, `:9665`), but **not on V7** (mode 35 logs "the capture carries no 0xc597 on the user lane", `:9029-9033`) — a class object must be added on V7 and validated. | | 3D context global buffers | Already promoted: BUNDLE_CB / PAGEPOOL / ATTRIBUTE_CB / RTV_CB_GLOBAL (`booter:531-534`, `:605-621`, golden9 sized buffers `:985-1008`). The engine context image therefore contains the 3D subcontext data. | | Method block | `gsp_phys_rmcall_build_gr_launch` (`:16834-16947`) emits compute init + inline QMD + PCAS + report-sem; the verbatim path (`:16878-16921`) already replays a captured raw method window and rewrites its address/QMD words. A 3D mode adds `SET_OBJECT 0xC597` + the captured state/upload/draw methods instead of the compute prelude. Note the channel's GR subchannel is 1 (`:442`) and the compute init binds `0xC5C0` (`:16693-16698`); the 3D bind replaces it per section. | | Operand pages | The page list is built from QMD words 32-49 (`:19678`, `:6673-6682`); it must be generalized to the recognized 3D operand VAs, with the shadow/alias machinery (`:19654-19785`) reused. | | Completion/sync | Keep the stream's own releases and the module release0 redirect (`:19499-19520`); the hold/relay path already authors the guest word on the engine edge (`gsp_sema_service.c:21329-21345`). | Pros: authentic rasterization is done by the hardware pipeline; no state is invented; the engine has already proven it fetches and executes forwarded work on this lane (R17 `qmd_rel=0x600df00d`); the completion machinery is unchanged. Cons/risks: the promoted context may still be compute-shaped (class object construction on a wired golden channel deadlocked in earlier experiments, `booter:816-822`; V7 is the tested path, so construct there); the 3D method vocabulary is large (state validation can silently fault); address-word relocation must cover every class encoding; ~6-12 operand pages per frame exceed today's 8-page win0 bank, requiring grouping/expansion. Effort: L. Depends on G-A, G-B. ### (ii) Synthesize a "graphics-shaped" launch from the captured C597 stream On Turing, a QMD is a *compute* launch descriptor; there is no QMD that rasterizes. "Graphics-shaped" therefore means: derive the draw inputs and author the state + a DRAW method ourselves — program (`0x2014/0x2018`, already decoded), constant buffers (bind + LID uploads, partly decoded), vertex buffers (SET_VERTEX_STREAM_SUBSTITUTE is decoded only as a pointer, `gsp_sema_service.c:6979-6983`; the real stream/index methods are not), render targets/ZS (not decoded; the only observed target is the release target `0x120001000`, R13), viewport/scissor/blend/depth state (not decoded). The vertex count for the draw would be inferred, not read. Pros: fewer pages to mirror than a full replay if only draw-relevant state is bound; still uses the hardware rasterizer. Cons: we author state the driver owns — any wrong field silently mis-renders; the per-draw state lives in the client ring, so G-A/G-B are still prerequisites; the program bytes are still lost without G-B; it is a subset of (i)'s work plus extra invention. Effort: L. Risk: high. Not first. ### (iii) Bounded compute translation for the subset GZDoom uses Replace each GL draw with a compute kernel (or the existing synthesized C5C0 launch carrying the client's real SASS) that rasterizes triangles into the RT. What breaks correctness, concretely: - **Fixed-function raster and ROP**: depth test, stencil, alpha blending, culling, clipping, perspective-correct interpolation, MSAA — none exist in a compute launch; reimplementing them is writing a GPU. - **Sampling**: mipmaps, LOD selection, anisotropic filtering, wrap modes and sRGB conversion are texture-unit behaviour. - **Shader semantics**: the client's GLSL is already compiled to SASS by the guest driver; executing that SASS as a whole-SM compute program does not give it vertex inputs, varying interpolation, or per-fragment coverage — the very things the 3D pipeline supplies. - **The claim itself**: even GZDoom's first frame uses FBOs (scene → post-process), depth testing, alpha blending and mipmapped textures; output would be a lookalike, not the renderer's pixels, and could not be verified against a reference. Verdict: acceptable only as a **diagnostic probe** (e.g. a known-pattern kernel proving the lane), never as the milestone path, and it must be labelled as such. Not first. ### Recommendation **Implement (i) first.** It is the only approach that yields authentic GL pixels; its difficult prerequisites (G-A ring readability, G-B stream capture) are shared by every alternative; and the backend already provides the three pieces that make it tractable: a real execution lane with a validated context (mode 35), a generic 0xC597 class-object allocator, and a verbatim method-window replay with address/QMD rewriting plus page shadowing and completion relay. (ii) is (i)'s draw-issuing step taken prematurely; (iii) trades correctness for nothing. Order of implementation inside (i): **G-A** → **G-B** → replay the captured *context-init* section (M-GL1, proves class bind + uploads + completion on the 3D lane) → replay the client's per-frame *draw* sections (M-GL2/3). --- ## C. Milestone ladder All gates are measured with the artifacts listed in section D; a milestone is met only when its gate reproduces on **two consecutive fresh cycles** (three for M-GL1) with the module hash recorded. ### M-GL1 — Client context-init work decodes and executes; the client leaves its retry loop Gate (all in one run): 1. GL client ring resolved: no `sema-chan: GR ring h= ... found no candidate` row after the first minute; an autopin row exists (`sema-chan: auto-pinned GR ring ...`). 2. Client pushes read: `sema-pushdec` rows for the client handle with `src>0 unreadable=0 plausible=1`; no `sema-rdw ... rc=-2` for them. 3. Section captured from the client: `sema-gl: section captured h=` with `prog/param/rel` nonzero and `replay=0` (or the hash-verified replay admission), at least once per client lifetime. 4. Forwarded and executed: `vgsp-gr: GR LAUNCH submitted h= ... pages>=2 shadow=1`, followed by engine evidence (`vgsp-gr: poll +... qmd_rel=0x600df00d`) and either a natural release or `GR-COMPLETE-SW`. 5. Client behaviour: glxgears no longer loops on context creation (no repeated init-failure retry; the X server stays alive; `xdpyinfo` OK, `DMAFAIL=0`), and the app reaches its first swap. 6. Bounded failure hygiene: no SYNTH-FAULT storm for the accepted client (≤ a handful in the window), no guest oops. Expected failure signatures (test one at a time): `GR ring ... found no candidate` (G-A); `unreadable=1 first8=00000000` (G-A, the S147 addendum 9 zero-page class); `GR request without candidate` / `cand_valid=0` (`gsp_sema_service.c:25479-25485`; G-B ordering: capture must precede the request); `GR submit refused: degenerate data set n_data=N (min M)` (`booter:19352`; operand list, G-D); `FAULT-PDE` at the program VA + `CODEHASH h0=0` (`booter:18806-18883`, `:18997`; G-B/G-D); `GR LAUNCH FAILED: timeout ... qmd_rel=...` (`booter:20825`; engine did not retire — class/sched fields or completion wiring). ### M-GL2 — glxgears renders non-black frames Gate: 1. A GZF2 record stream is captured from the **GL client's swap** (see D for the GLX shim; the repo's `sdl_cap.c` is SDL-only — glxgears does not use SDL). 2. ≥1 frame with ≥5% of pixels above luminance 8/255 and ≥2 distinct dominant colours; two frames ≥500 ms apart differ in ≥1% of pixels (motion, not a static noise tile). 3. The same time window contains the per-frame GR launch + completion in dmesg, and the guest's `glxinfo` renderer is the NVIDIA driver (not Mesa). 4. The first frames are reached without any sync poke. If a diagnostic poke is used to unblock the run, that run is labelled diagnostics and does not satisfy the gate. Failure signatures: swap hook fires but frames all zero → no rasterization (G-C); `xwd`/X path black → expected, do not use; frames show only clear colour → draws executed but state/vertex data wrong (G-D); `FAULT-CAP ... code_src=guest-hpa code64=000...` → program source lost (G-B). ### M-GL3 — GZDoom GL opens a window and presents non-black frames through the mdev Gate: 1. GZDoom (`vid_rendermode=4`, `vid_preferbackend=0`) runs on `DISPLAY=:0` backed by the emulated vGPU (no Xvfb, no Mesa overrides). 2. ≥300 consecutive non-black GZF2 frames from the SDL swap hook, including gameplay (not only the menu); the in-frame FPS counter ≥30. 3. dmesg shows one GR launch per rendered frame (bounded), no SYNTH-FAULT loop; the X server stays alive for the whole run. 4. Capture fetched via QGA and decoded offline; a screenshot (`+screenshot`) matches the captured frame within a small pixel delta. Failure signatures: window opens, swap fires, all frames black → draws not executed; `GR submit refused ... n_data` → operand pages; app crashes with `Illegal Instruction Encoding` (ESR 0x504730=0x9) → program bytes are not SASS (G-B; cf. R18's single-page kernel lesson); `SYNTH-FAULT` per frame → forward failed or refused. ### M-GL4 — 60 fps sustained in ONE VM, no forcing Gate: 1. glxgears: "N frames in T seconds = X FPS" with X ≥ 60 over a 60 s measurement (`vblank_mode=0 __GL_SYNC_TO_VBLANK=0` for the measurement itself), and GZDoom's counter ≈60. 2. **No** poke/force/mirror daemon is running and no runtime module param is used to fake progress: `sema_xw_mirror` is `0,0` (stopped, `kernel/main.c:3103-3129`), no `/proc//mem` write helper, no strike/deadline forcing. 3. GZF2 inter-frame deltas: p95 ≤ 17 ms over 10 s (capture path keeps up). 4. The X present waits exit with member advancing naturally (Xorg log WAIT lines absent or resolving without the kickfix daemon), and the glcore equality word no longer requires the bump. 5. CUDA regression: `victory_mail` `VADD n=32768 bad=0` / `STATUS OK` on the same boot after the GL run (battery-first ordering). Failure signature: the two ~11 s glcore waits return (the app's own strike ladder; S144 addendum 16b) → G-E; X WAIT with static member → present completion not authored; `GR-STALL-backend` per frame → completion is software-only and the app's ladder is what "ran". ### M-GL5 — Two VMs, two mdevs concurrently Verification checklist before the gate: 1. Second mdev creates and its guest RM bootstraps (the S151 open question: "a second fragile RM init"); both devices appear in the service loop (`gsp_sema_service.c:32119-32236`). 2. Per-guest isolation of the shared tables: `svc_ledger` VA-only key (`:7547-7559`), `svc_gl` (`:6863`), `svc_gr_cand_ring` (`:6493`), `svc_gr_seen` (`:6547`), `svc_grpark` (`:7429`); either prove guest VAs do not collide in practice or key them by `(vgpu, va)`. 3. Guest RAM: both VMs' memory-backend ranges registered and resolving (the cycle registers one file per guest; verify the per-device map). 4. Server VAS relocation: no two guests' data windows alias in the managed region (`SVC_GR_DATA_WINS_MAX 31`, `:6441`; comment `:6436-6439`). 5. Win0 banks/epochs (`booter:19406-19410`) are safe under serialized submissions; single-inflight holds (`:19878`) do not starve one guest. Gate: both VMs run glxgears and GZDoom for ≥10 minutes; both present non-black frames; each ≥30 fps (60 target); `victory_mail` green in both afterwards; no cross-guest data anomalies (`sema-ledger: DIVERGE` / `LEDGER-RESET` absent), no host oops, no guest wedge. Failure signatures: one VM starves the other's forward slot (`GR request busy: hold active`, `:25509`); second guest's launches resolve the first guest's pages (wrong results / ledger rewrite storms, `:7584-7594`); second RM init fails in the boot ladder. ### M-GL6 — Flawless 60 fps split-screen video Gate: 600+ frames at 60/1 fps, ≥95% unique frames, both halves show real GZDoom GL gameplay through the two mdevs, encoded with the existing resample/compose pipeline, uploaded with the files-upload skill and pull-back-md5 verified. Encode with `libx264` (the R18 path did: `/tmp/opencode/r18_final.sh`); NVENC is unavailable while the module owns the physical GPU (the host NVIDIA driver is displaced). --- ## D. Experiment recipes ### D.0 Standard cycle (all milestones) Host, one controlled attempt per cycle (script leaves the VM running): ``` nohup /tmp/opencode/start_vm_keep.sh rNNN "" \ > /tmp/opencode/rNNN-cycle.log 2>&1 & # poll until: DONE (no teardown; VM left running) (~70-90 s) ``` `EXTRA_ARGS` = the SESSION 149 standard profile (`session-149-BRIEF.md:65-67`), verbatim: ``` gsp_phys_gr_nowrite_retries=0 gsp_phys_gr_retry_fast=1 gsp_phys_gr_first_poll_ms=2500 gsp_sema_winfill=1 gsp_sema_gr_swrel_gate=1 gsp_phys_rmcall_gr_promote=35 gsp_phys_rmcall_fwclient_mask=0xE3F gsp_phys_rmcall_srv_ctxcommit=0 gsp_phys_rmcall_gr_qmdwin=1 gsp_phys_rmcall_gr_shadow_win0=1 gsp_phys_rmcall_gr_lmem_en=1 gsp_phys_rmcall_gr_skedrec=1 gsp_phys_rmcall_gr_inst_sys=1 gsp_phys_rmcall_gr_r1_first=1 gsp_phys_gr_strict_reloc=0 gsp_phys_fault_relay=1 sema_bigcopy_log=512 ``` GL overlay (M-GL1..M-GL3 runs): append ``` sema_gl_xlate=1 sema_push_raw=4 gsp_phys_gr_min_data=0 sema_gr_util_scan=0 sema_dbg=Y ``` (`sema_gl_xlate` is off by default in-tree, `gsp_sema_service.c:6812`; `min_data=0` is the explicit no-external-data bypass validated in S143 and used by R15-R17; `util_scan>0` is a measured regression source, keep 0; `sema_dbg=Y` raises the bounded witnesses). The base script's ladder already passes `gr_promote=25`; the later `=35` wins at insmod and is snapshotted by the boot ladder — verify the boot log names mode 35 (`booter:8612-8613`). Post-load, before Xorg: ``` sudo sh -c 'echo 1000 > /sys/module/vgsp/parameters/sema_vas_cand_ms' sudo nohup dmesg -w > /tmp/opencode/dmesg_live_rNNN.log 2>&1 & ``` Guest Xorg (one per boot; ~50 % of boots fail DMA and need a re-cycle): ``` echo ubuntu | sudo -S setsid bash -c \ 'nohup Xorg :0 -ac -noreset -logfile /tmp/xorg.log >/tmp/xorg.out 2>&1 1280x720; grep -c "Failed to initialize DMA" /tmp/xorg.log == 0 ``` After Xorg is up: ``` sudo sh -c 'echo 1 > /sys/module/vgsp/parameters/sema_force_flush' ``` Evidence per run, always kept: module md5 (`md5sum kernel/vgsp.ko`), the echoed insmod args, the full dmesg capture, guest `/tmp/xorg.log`, `/tmp/glx*.log`, the GZF2 file and its md5, the decoded frame PPMs, and the acceptance-grep output. ### D.1 M-GL1 recipe Guest: start glxgears and watch the client channel: ``` DISPLAY=:0 setsid nohup stdbuf -oL glxgears -geometry 800x600+0+0 \ > /tmp/glx1.log 2>&1 " # G-A pass grep -E "sema-pushdec: h=0x.*unreadable=0" # G-A pass grep -E "sema-gl: section captured h=0x" # G-B pass grep -E "GR LAUNCH submitted h=0x" # capture->forward grep -E "qmd_rel=0x600df00d|GR-COMPLETE-SW" # engine executed/completed ``` If G-A fails: confirm the guest scanner is running the widened prefixes (`/tmp/guest_pin_stream.log`; supervisor env at `tools/guest-diag/pin_supervisor.sh:70-83`), force a demand resolve via `sema_trace_h=` (S148 addendum 10) to see which source fails, and use `echo >> /tmp/guest_want_vas.txt` for a targeted pin. If G-B fails with `entry_replay=1` keeping `cap_trust=0`, the pass is an un-hashed replay: wait one fresh push or use the hash-admitted path (`gsp_sema_service.c:29069-29080`) — do not add a bypass. Expected time-to-first-signal: minutes. Do not proceed to M-GL2 until 3 cycles pass. ### D.2 M-GL2 recipe (glxgears pixels) The repo shim is SDL-only (`sdl_cap.c:188-229` hooks `SDL_GL_SwapWindow`, `SDL_RenderPresent`, `SDL_UpdateWindowSurface`); glxgears is GLX and calls neither. Add a GLX mode to the same tool (hooks `glXSwapBuffers` → `glReadPixels`, same GZF2 record layout), build: ``` gcc -shared -fPIC -O2 -o /tmp/glx_cap.so glx_cap.c -ldl ``` Guest: ``` DISPLAY=:0 GLXCAP=/dev/shm/glxframes.bin GZCAP_MAXF=0 \ LD_PRELOAD=/tmp/glx_cap.so setsid nohup stdbuf -oL \ glxgears -geometry 800x600+0+0 > /tmp/glx2.log 2>&1 8/255, colour count, motion between frames. ``` Do not use `vblank_mode=0` for the first pixel check (sync-on is fine); keep the shim's swap hook ordered as "capture then forward" (same as sdl_cap). ### D.3 M-GL3 recipe (GZDoom GL) Guest (xorg.conf untouched, the emulated GPU as the only GPU): ``` VK_ICD_FILENAMES=/nonexistent.json VK_DRIVER_FILES=/nonexistent.json \ GZCAP=/dev/shm/gzframes.bin GZCAP_MAXF=0 LD_PRELOAD=/tmp/sdl_cap.so \ DISPLAY=:0 setsid nohup ./gzdoom -iwad freedoom2.wad \ +vid_rendermode 4 +vid_preferbackend 0 +vid_activeinbackground true \ +sv_cheats 1 +noclip > /tmp/gz.log 2>&1 .../sema_xw_mirror`) and no forcing/kickfix daemon is running (`pgrep -af` exact names first); - measure 60 s windows only: glxgears FPS line, GZDoom counter, GZF2 timestamps; - keep `sema_gr_grace_ms`/`sema_ce_grace_ms` at their defaults; the R1 A/B proved they are not the pacer (`session-149-BRIEF.md:96-99`); - battery-first CUDA regression in the same boot. ### D.5 M-GL5 recipe (two VMs) Second mdev + second VM (own disk copy, own sockets, own SSH/QGA ports), both with the vGPU device this time. Verify in order: second mdev appears; second guest's `nvidia-smi`/`glxinfo` sees the vGPU; service log shows both device handles per pump pass (`sema-chan` rows for both); then run the D.1/D.3 apps in both. Keep per-VM captures separate (two GZF2 files, two QGA sockets). ### D.6 M-GL6 recipe (video) Per-VM SDL capture → `qga_fetch.py` → `/tmp/opencode/gz_final_gpu.py` (timestamp resample + split-screen compose) → `ffmpeg -c:v libx264` (R18 pattern) → files-upload skill → pull-back md5 verification. 1280x440 split (640x440 per guest) matches the two delivered videos. --- ## E. What must NOT be done 1. **No other GPUs or virtual GPUs.** No virtio-gpu/virgl, no VirtualGL, no llvmpipe/Mesa software GL, no second physical GPU, no host display round-trip. The SESSION 152 video is explicitly *not* evidence for this plan. 2. **No fake or mislabelled rendering.** No software rasterizer or compute lookalike presented as GZDoom's OpenGL renderer. A compute-translation probe (approach iii) may exist only as a labelled diagnostic and is never an acceptance artifact. 3. **No lying sync pokes in accepted runs.** The `/proc//mem` counter bump, strike forcing, kickfix and `sema_xw_mirror` are diagnostics; any milestone evidence must state they were absent. 4. **No host reboot.** AGENTS.md hard rule: handoff update first, explicit human approval, never automatic. Guest VM cycles and Xorg restarts on the test VM are the allowed recovery paths. 5. **No guest modifications.** Stock guest `nvidia.ko` and stock apps; no injected code, patched firmware or modified driver in the guest; capture shims are LD_PRELOAD observation only and must not alter GL semantics. 6. **No time/timing claims from unsupported sources.** The physical backend is the execution authority; simulator (L0/L1) findings never substitute for a live engine result. 7. **No unguarded experiments on the physical GPU.** Every cycle is supervised, single-attempt, evidence-captured (the cycle script's hard abort signatures are not bypassed). --- ## F. Open questions and note-vs-code discrepancies 1. **`sdl_cap.c` is SDL-only.** The milestone recipe in the SESSION 149 brief names it for glxgears, but glxgears is GLX and never calls `SDL_GL_SwapWindow` (`tools/guest-diag/sdl_cap.c:6-9`, `:188`). A GLX variant is required (D.2); `sdl_cap.c` remains the right shim for GZDoom (M-GL3). 2. **The GL-client pin prefix fix already looks landed.** `tools/guest-diag/pin_supervisor.sh:73-83` defaults `CLIENT_PREFIXES=cdt,va,victory,Xorg,X,glxgears,gzdoom` and the cycle starts the supervisor (`start_vm_keep.sh:242`), while SESSION 153 still lists the "permanent fix" as pending. Verify the running guest scanner actually received the widened env (`/tmp/guest_pin_stream.log`) before treating G-A as closed. 3. **The logged failure strings differ slightly from the briefs.** The code prints `vgsp-work: GR request without candidate` (`gsp_sema_service.c:25481`), not bare "GR request without candidate"; the ring failure is `sema-chan: GR ring h=... found no candidate` (`:17633`). Grep acceptance scripts for the exact prefixes. 4. **`n_data` threshold history.** The default is 16 (`booter:17709-17717`); S143+ cycles used `min_data=0` as an explicit bypass for the no-external-data launch class, enabled by the r800 guard. The brief's "min 1"/"min 16" variants refer to different experiments; the GL profile below uses 0 and must say so in every evidence record. 5. **Stale project status row.** `docs/README.md:114` still lists CUDA execution as "no grid has dispatched yet", which predates the proven physical forward and the R18 CUDA video. This plan relies on the newer code and `docs/04-implementation/03-execution-frontier.md`, not that row. 6. **The 3D context on the execution lane is the main unknown.** Mode 35's capture "carries no 0xc597 on the user lane" (`booter:9022-9033`); golden-lane 3D objects exist (`:9035-9043`) but the execution channel is V7. The first implementation experiment must prove a 0xC597 class object can be constructed on V7 after the accepted promote (same helper, `:8936`) without the wired-golden deadlock class (`:816-822`), before any stream replay work.