diff options
Diffstat (limited to 'docs/04-implementation')
| -rw-r--r-- | docs/04-implementation/05-graphics-execution-plan.md | 721 |
1 files changed, 721 insertions, 0 deletions
diff --git a/docs/04-implementation/05-graphics-execution-plan.md b/docs/04-implementation/05-graphics-execution-plan.md new file mode 100644 index 0000000..6089548 --- /dev/null +++ b/docs/04-implementation/05-graphics-execution-plan.md @@ -0,0 +1,721 @@ +# Graphics execution plan: real GL pixels through the emulated vGPU + +> **Status: OPEN (2026-10-28).** This is the implementation-ready plan for the +> **graphics execution layer**: the component that must execute the guest +> NVIDIA vGPU driver's OpenGL/3D command streams on the physical GPU so that +> glxgears and GZDoom's OpenGL renderer produce real pixels through the +> emulated mdev. Nothing here is implemented yet beyond the pieces cited; +> every statement carries a code location or a measured session finding. +> +> Evidence base: `docs/04-implementation/03-execution-frontier.md` (the CUDA +> compute chain, proven), `docs/04-implementation/04-real-render-walls.md` +> (§8-§21, all measured graphics walls), the SESSION 149 R12-R18 round log +> (`session-149-BRIEF.md`: the M1 C597 recognizer, M2 materialization, M3 +> SKEDCHECK clear, R17 real-SASS execution), and SESSION 144 addenda 9/10/16 +> (the X/GL wait anatomy). The vendored 570.144 wire protocol and the public +> class headers are the interface authority; no guest driver change is +> allowed. +> +> Hard rules carried from the project: **no other GPU** (virtio-gpu/virgl, +> VirtualGL, llvmpipe, any second device are banned for this work), **no +> fake/labelled software rendering** presented as GL, **host-only emulation** +> (guest runs stock `nvidia.ko`), and **no host reboot** (AGENTS.md hard +> rule; a reboot needs the handoff update plus explicit human approval). + +--- + +## 0. What "real pixels" means here + +The acceptance target is: a stock guest GL application's draw commands are +executed by the physical GPU's **fixed-function 3D pipeline** (vertex fetch, +primitive assembly, rasterization, fragment work, ROP), its render target is +readable by the guest, and that image reaches the capture path through the +emulated mdev. The pixels must not come from Mesa, llvmpipe, virgl, a +software rasterizer, or a host display round-trip. + +Two consequences drive every milestone below: + +1. **A compute launch is not rasterization.** The SESSION 149 M1/M3 work + proved the engine will execute a *synthesized* C5C0 compute launch built + from the GL stream and will retire it + (`poll +500ms qmd_rel=0x600df00d`, R17), but the GL client's draws need + the 3D pipeline and the client's own per-draw state; rasterization is + fixed-function GR (`session-149-BRIEF.md:288-291`). +2. **The GL client's own channel is the source of truth.** The C597 section + the emulation can capture today comes from a readable channel (the X + server side); the GL client's GPFIFO ring (`0xbeef0100` family, + `0xbfef0002` observed in SESSION 153) is where its context-init uploads + and per-frame draws live. Until that ring is readable, no approach can + derive a correct draw (`session-149-BRIEF.md:438-451`). + +--- + +## A. Gap list, ordered by dependency + +Each gap lists what exists (with the code location that implements it), what +is missing, and the measurable gate that closes it. A gap is closed only when +its gate has been met on a fresh cycle with evidence kept. + +### G-A Readability: the GL client's ring and pushes must read + +**Exists.** The pump resolves rings through an autopin ladder (notifier- +derived, USERD-derived, trusted walk, shadow scan, USERD-shape scan) with +dead-entry rejection: `kernel/gsp_sema_service.c:16534-16602` +(`svc_ring_entry_dead`, `svc_ring_walk_live`, `svc_gr_ring_candidate`), +`:17891`, `:17972`, `:18013`, `:18075`, `:18123`, `:18242`, `:18298`. The +failure is witnessed at `:17633` (`sema-chan: GR ring h=... RAM content scan +found no candidate in N slice(s)`). Push reads report through `sema-rdw` +(`:28622`) and `sema-pushdec` (`:28868-28882`). The guest pin stream's client +selection and windows are `tools/guest-diag/guest_pin_stream.py:1429-1475`, +`tools/guest-diag/pin_gr_stream.py:134-160`, and the supervisor already +defaults to the widened GL prefix set +(`tools/guest-diag/pin_supervisor.sh:70-83`: +`cdt,va,victory,Xorg,X,glxgears,gzdoom`). + +**Missing.** For the GL client channel the ring page is not found by content +at the scan instants ("GR ring h=0xbfef0002 rank=2 ... no candidate in 12 +slice(s) (va=0x1203cfff0)", SESSION 153 item 3). The client's allocations are +just above the old 64 MB capture bound in the Vulkan case +(`docs/04-implementation/04-real-render-walls.md` §14-§15), and the pump's +ring candidates are shape-based; a client ring written late (after the DMA +test window) is missed. The guest pagemap stream (`#pid` + `va,pa,n` lines, +`guest_pin_stream.py:8-12`) is the authoritative VA->PA source but was not +covering the GL client until SESSION 153; the supervisor default now does. + +**Gate G-A.** On a run with glxgears alive for 60 s and the standard X +recipe: (a) zero new `sema-chan: GR ring h=0x... found no candidate` rows for +the GL client's handle; (b) every client push logs `sema-pushdec ... src>0 +unreadable=0`; (c) at least one entry is consumed from the client ring +(pump advances `last_get`; `sema-chan: entry[...] h=0x... eng=0` with a sane +`va/len`); (d) the X channel (`0x1000013`) stays `plausible=1` (SESSION 153 +item 1). + +### G-B Capture: the client's C597 sections and per-frame draws must decode + +**Exists.** A gated recognizer (`sema_gl_xlate`, default 0, +`kernel/gsp_sema_service.c:6812-6815`) opens a section at a `SET_OBJECT` +class bind of `0xC597` (TURING_A) on the channel subchannel and collects +observed VA operands: program (`0x2014`/`0x2018` runs), +tex header/sampler pools (`0x1574`/`0x155c`), vertex substitute (`0x0f84`), +upload source/destination (`0x180`), and the LAUNCH_DMA/LOAD_INLINE_DATA +payload (`0x1b0`) — `:6945-7049`, `:11993-12032`. It synthesizes a 64-word +QMD and captures it through the unchanged GR candidate ring +(`:7060-7287`, `:6686`). The client's memory-writing uploads are applied by +a standalone subchannel-0 pair tracker (`:12045-12084`) and the section +walker (`:7194-7217`) through `svc_grce_inline_write` (`:8979-9060`). +Completion of the capture is gated by content trust: a plausible stream +admitted by `cap_trust` (`:28995`, `:29069-29080`), with hash-verified +rewind replays admitted (`:28513-28517`, `:29106-29119`); the flush guard +requires `seen/prog/param/rel/trusted` (`:7082-7084`). Witness lines: +`sema-gl: section captured ...` (`:7248`), `sema-gl: uploads applied=...` +(`:7213`), `sema-gl-lid: ...` (`:12073`), `sema-gl: M3 code src=...` +(`:7261`). + +**Missing.** +1. The recognizer's method vocabulary covers init-shaped operands only. The + per-draw 3D state and draw-issuing methods (vertex streams/index, render + target and ZS, viewport/scissor, blend/depth/ROP state, draw calls) are + not decoded (`svc_gl_field` handles six addresses, + `:6945-7049`; the C597 stream carries far more). +2. The GL program/constant delivery is a context-init upload on the client + channel and is lost today: R17 measured that no readable push ever writes + `prog=0x121201b00` (`session-149-BRIEF.md:276-285`). The synthesized + launch therefore only runs a 12-28 B fallback blob or an operator hex + program (`sema_gl_prog_hex`, `:6877-6913`) — a development crutch, not + pixels. +3. Capture must survive the "one-shot setup push rewritten in place" + class: re-captures come from hash-verified replays now + (`:29069-29080`), but the per-frame draws are separate entries and need + the same treatment per entry. + +**Gate G-B.** For ≥3 consecutive frames of a live GL client: `sema-gl: +section captured` on the **client's** handle with nonzero `prog/param/rel`, +one captured section per frame (bounded replays suppressed), and a decoded +draw record (new `sema-gl-draw` witness) whose vertex/primitive count is +consistent with the app (e.g. glxgears gear geometry). Program source +priority shows `trusted-prog` (real bytes) for the context-init upload. + +### G-C Execution semantics: the GL work must run on the real 3D engine + +This is section **B**; the gap gate is M-GL1/M-GL2/M-GL3. + +### G-D Memory and resource mapping: every operand must be reachable by the engine and readable by the guest + +**Exists.** The physical backend builds engine-side page tables and shadow +windows for launch data: Tier-1 identity install or Tier-2 shadow copy +(`kernel/gsp_phys_booter.c:19654-19724`, `:19734-19785`), a two-bank +window-0 shadow pool of 8 pages per bank (`:1174`, `:19406-19410`) with a +24-page shadow chunk (`:1317`), QMD-window relocation +(`gsp_phys_rmcall_gr_qmdwin`), and a verbatim-prelude replay path that +relocates the inline-QMD address words and data words +(`:16878-16921`). Data pages are copied in before the doorbell and copied +back after the retire wait (`:17693`, `:18243`, `:21966`, `:22242`). +`svc_grce_inline_write` writes upload bytes to vidmem shadow, guest RAM, or +the VA shadow with a once-only key (`kernel/gsp_sema_service.c:8979-9060`). + +**Missing.** +1. Address-word relocation is QMD-only (words 32-49 and the indirect-page + pointer fixup at `booter:19765-19779`). A 3D method stream names VAs at + class-specific offsets; each must be resolved to a shadowed page and the + word rewritten (or the page mapped at its guest VA). +2. The engine-visible page budget is small: 16 pages per request + (`kernel/gsp_phys_booter.h:355`), 8 inline records (`:356`), 8 pages per + win0 bank (`booter:1174`). A GL frame names program + constants + texture + pools + vertex stream + render target + ZS; the shadow pool and the + per-request page list must grow or pages must be grouped per buffer. +3. Render-target writeback: the copy-out tracks only the launch's own data + pages; a 3D draw writes the RT, which must be mapped into guest-visible + memory (BAR1 shadow or pinned guest RAM) and copied back on the + completion edge, exactly like the current bounce/copy-out but for the RT + and depth surfaces. +4. Program/constant bytes: R17's measurement stands — the program page is + empty in every host-visible source until the CE/LID delivery executes + (`docs/04-implementation/04-real-render-walls.md` §20-§21). + +**Gate G-D.** Engine fetch of the 3D program/params/vertex/RT pages with +`FAULT-PDE count=0` (`vgsp-m91: FAULT-CAP ... code_src=...`, +`booter:18997-18999`), and a measurable post-launch change in the emulated +vidmem at the RT VA (>0 changed dwords in the `dump_vidmem map` class, +`docs/04-implementation/04-real-render-walls.md` §11). + +### G-E Completion and sync: the guest must be released by real completions + +**Exists.** Engine completion is polled (`qmd_rel/ch_rel`, +`booter:20814-20825`), the launch's release0 can be redirected to the module +semaphore and relayed (`booter:19499-19520`), the hold edge and the +backend-outcome edge author the guest word (`kernel/gsp_sema_service.c: +21329-21345`, `:21298-21322`), and failures synthesize a client-visible +channel error (`SYNTH-FAULT`, `:21400-21448`, `GR-COMPLETE-SW`, `:21504`). +The pre-built ring continuation beyond GP_PUT is swept for releases +(`sema_gr_ahead`, `:26564-26566`, `svc_gr_ahead_push`, `:26809`). + +**Missing.** The guest GL client's waits are still satisfied by the give-up +ladder or by host pokes: the X server's present wait spins on the X channel +member/counter pair (S144 addendum 12; the mirror thread `sema_xw_mirror` +`kernel/main.c:3103-3129`), and the GL client's `PhaseWait` compares +`obj+0x558` against a word in the shared `memfd:/.nvidia_drv` (+0x8) that the +emulation never advances (SESSION 149 R3, `session-149-BRIEF.md:115-126`: +"one u32 write 3 → 128.8 FPS", but only via `/proc/<pid>/mem`). The correct +long-term fix is the one named in addendum 16b §C.4: **the emulation +completes the real syncs so the equality is satisfied, with no lying pokes**. + +**Gate G-E.** M-GL4 (below): ≥60 fps sustained with no forcing daemon, no +`sema_xw_mirror` target written, and the guest's waits exiting on a natural +completion edge. + +### G-F Present/frame path: the rendered image must land where the guest and the capture can read it + +**Exists.** Guest-side capture tools: `tools/guest-diag/sdl_cap.c` (SDL swap +hook → `glReadPixels` → GZF2 record stream, `:1-18`, `:188-229`), +`tools/guest-diag/fbcopy_pub.c` (BAR1 scanout copy → kvmfr publisher), +`tools/guest-diag/qga_fetch.py` (QGA file fetch), host decoder +`tools/lg/frame_reader.py` and the resample/encode scripts. + +**Missing.** On the emulated-vGPU stack the X screen is black for GL windows +(DRI3-present path; SESSION 152) and `xwd` hangs (R4/R5; no rendered pixels +exist anywhere). The present path (the C5B5 copy-class and 902D 2D parts of +the GL push, R13) is decoded for releases but not executed as a display +path, and the capture must come from the GL client's swap, not from X. + +**Gate G-F.** M-GL2 (glxgears non-black) and M-GL3 (GZDoom). + +### G-G Concurrency: two VMs, two mdevs + +**Exists (designed in).** The service thread iterates every open device each +pass (`kernel/gsp_sema_service.c:32119-32236`), with the explicit note that +"with two, both guests get a pump turn per iteration (hardware forwarding +stays serialized by the single-inflight slot)" (`:32161-32167`). Per-guest +sessions exist (`SVC_MAX_GUESTS = 8`, `:728-736`), and the guest-RAM ranges +are registered per device at boot (`/tmp/opencode/start_vm_keep.sh:210-213`). + +**Missing / to verify.** Single-inflight forwarding and holds +(`:19878`, `:25528-25545`); VA-keyed global tables that assume one guest's VAs +do not co-reside: the write/read ledger is keyed by VA page only and its own +comment says collisions "only occur ... across clients whose pages do not +co-reside in time" (`:7547-7559`), plus `svc_gl` (`:6863`), +`svc_gr_cand_ring` (`:6493`), `svc_gr_seen` (`:6547`), `svc_grpark` (`:7429`); +the win0 banks are module-global per submit epoch (`booter:19406-19410`); the +server-managed VAS relocation windows are a finite global set +(`SVC_GR_DATA_WINS_MAX 31`, `:6441`). SESSION 151 avoided a second mdev "to +avoid a second fragile RM init", so a second RM bootstrap on the same module +is unproven. + +**Gate G-G.** M-GL5 (below). + +--- + +## B. Execution-semantics approaches + +### (i) 3D method-stream replay on the real engine + +Forward the client's captured C597 method stream (class bind + 3D state + +upload/draw methods + in-stream releases) to the module's real GR channel +and let the fixed-function pipeline execute it. + +What must be mirrored/provided: + +| Requirement | Where it stands | +|---|---| +| Real execution channel/context | Mode 35 "real-client user lane" replays the 570.144 client's compute-context sequence and executes on V7 (`booter:924`, `:9022-9033`); the promote ladder builds MAIN/PM/PATCH and the golden image. | +| 3D class object (0xC597) | `GSP_PHYS_RMCALL_CLASS_GR_3D 0xc597` is defined (`booter:396`) and the alloc helper is generic (`gsp_phys_rmcall_alloc_gr_object_class`, `:8936-8964`); it is constructed on the golden lane in several modes (`:9035-9043`, `:9665`), but **not on V7** (mode 35 logs "the capture carries no 0xc597 on the user lane", `:9029-9033`) — a class object must be added on V7 and validated. | +| 3D context global buffers | Already promoted: BUNDLE_CB / PAGEPOOL / ATTRIBUTE_CB / RTV_CB_GLOBAL (`booter:531-534`, `:605-621`, golden9 sized buffers `:985-1008`). The engine context image therefore contains the 3D subcontext data. | +| Method block | `gsp_phys_rmcall_build_gr_launch` (`:16834-16947`) emits compute init + inline QMD + PCAS + report-sem; the verbatim path (`:16878-16921`) already replays a captured raw method window and rewrites its address/QMD words. A 3D mode adds `SET_OBJECT 0xC597` + the captured state/upload/draw methods instead of the compute prelude. Note the channel's GR subchannel is 1 (`:442`) and the compute init binds `0xC5C0` (`:16693-16698`); the 3D bind replaces it per section. | +| Operand pages | The page list is built from QMD words 32-49 (`:19678`, `:6673-6682`); it must be generalized to the recognized 3D operand VAs, with the shadow/alias machinery (`:19654-19785`) reused. | +| Completion/sync | Keep the stream's own releases and the module release0 redirect (`:19499-19520`); the hold/relay path already authors the guest word on the engine edge (`gsp_sema_service.c:21329-21345`). | + +Pros: authentic rasterization is done by the hardware pipeline; no state is +invented; the engine has already proven it fetches and executes forwarded +work on this lane (R17 `qmd_rel=0x600df00d`); the completion machinery is +unchanged. + +Cons/risks: the promoted context may still be compute-shaped (class object +construction on a wired golden channel deadlocked in earlier experiments, +`booter:816-822`; V7 is the tested path, so construct there); the 3D method +vocabulary is large (state validation can silently fault); address-word +relocation must cover every class encoding; ~6-12 operand pages per frame +exceed today's 8-page win0 bank, requiring grouping/expansion. + +Effort: L. Depends on G-A, G-B. + +### (ii) Synthesize a "graphics-shaped" launch from the captured C597 stream + +On Turing, a QMD is a *compute* launch descriptor; there is no QMD that +rasterizes. "Graphics-shaped" therefore means: derive the draw inputs and +author the state + a DRAW method ourselves — program (`0x2014/0x2018`, +already decoded), constant buffers (bind + LID uploads, partly decoded), +vertex buffers (SET_VERTEX_STREAM_SUBSTITUTE is decoded only as a pointer, +`gsp_sema_service.c:6979-6983`; the real stream/index methods are not), +render targets/ZS (not decoded; the only observed target is the release +target `0x120001000`, R13), viewport/scissor/blend/depth state (not decoded). +The vertex count for the draw would be inferred, not read. + +Pros: fewer pages to mirror than a full replay if only draw-relevant state is +bound; still uses the hardware rasterizer. + +Cons: we author state the driver owns — any wrong field silently +mis-renders; the per-draw state lives in the client ring, so G-A/G-B are +still prerequisites; the program bytes are still lost without G-B; it is a +subset of (i)'s work plus extra invention. + +Effort: L. Risk: high. Not first. + +### (iii) Bounded compute translation for the subset GZDoom uses + +Replace each GL draw with a compute kernel (or the existing synthesized +C5C0 launch carrying the client's real SASS) that rasterizes triangles into +the RT. + +What breaks correctness, concretely: +- **Fixed-function raster and ROP**: depth test, stencil, alpha blending, + culling, clipping, perspective-correct interpolation, MSAA — none exist in + a compute launch; reimplementing them is writing a GPU. +- **Sampling**: mipmaps, LOD selection, anisotropic filtering, wrap modes + and sRGB conversion are texture-unit behaviour. +- **Shader semantics**: the client's GLSL is already compiled to SASS by the + guest driver; executing that SASS as a whole-SM compute program does not + give it vertex inputs, varying interpolation, or per-fragment coverage — + the very things the 3D pipeline supplies. +- **The claim itself**: even GZDoom's first frame uses FBOs + (scene → post-process), depth testing, alpha blending and mipmapped + textures; output would be a lookalike, not the renderer's pixels, and + could not be verified against a reference. + +Verdict: acceptable only as a **diagnostic probe** (e.g. a known-pattern +kernel proving the lane), never as the milestone path, and it must be +labelled as such. Not first. + +### Recommendation + +**Implement (i) first.** It is the only approach that yields authentic GL +pixels; its difficult prerequisites (G-A ring readability, G-B stream +capture) are shared by every alternative; and the backend already provides +the three pieces that make it tractable: a real execution lane with a +validated context (mode 35), a generic 0xC597 class-object allocator, and a +verbatim method-window replay with address/QMD rewriting plus page shadowing +and completion relay. (ii) is (i)'s draw-issuing step taken prematurely; +(iii) trades correctness for nothing. + +Order of implementation inside (i): **G-A** → **G-B** → replay the captured +*context-init* section (M-GL1, proves class bind + uploads + completion on +the 3D lane) → replay the client's per-frame *draw* sections (M-GL2/3). + +--- + +## C. Milestone ladder + +All gates are measured with the artifacts listed in section D; a milestone is +met only when its gate reproduces on **two consecutive fresh cycles** (three +for M-GL1) with the module hash recorded. + +### M-GL1 — Client context-init work decodes and executes; the client leaves its retry loop + +Gate (all in one run): +1. GL client ring resolved: no `sema-chan: GR ring h=<client> ... found no + candidate` row after the first minute; an autopin row exists + (`sema-chan: auto-pinned GR ring ...`). +2. Client pushes read: `sema-pushdec` rows for the client handle with + `src>0 unreadable=0 plausible=1`; no `sema-rdw ... rc=-2` for them. +3. Section captured from the client: `sema-gl: section captured h=<client>` + with `prog/param/rel` nonzero and `replay=0` (or the hash-verified replay + admission), at least once per client lifetime. +4. Forwarded and executed: `vgsp-gr: GR LAUNCH submitted h=<client> + ... pages>=2 shadow=1`, followed by engine evidence (`vgsp-gr: poll +... + qmd_rel=0x600df00d`) and either a natural release or `GR-COMPLETE-SW`. +5. Client behaviour: glxgears no longer loops on context creation (no + repeated init-failure retry; the X server stays alive; `xdpyinfo` OK, + `DMAFAIL=0`), and the app reaches its first swap. +6. Bounded failure hygiene: no SYNTH-FAULT storm for the accepted client + (≤ a handful in the window), no guest oops. + +Expected failure signatures (test one at a time): +`GR ring ... found no candidate` (G-A); +`unreadable=1 first8=00000000` (G-A, the S147 addendum 9 zero-page class); +`GR request without candidate` / `cand_valid=0` +(`gsp_sema_service.c:25479-25485`; G-B ordering: capture must precede the +request); +`GR submit refused: degenerate data set n_data=N (min M)` +(`booter:19352`; operand list, G-D); +`FAULT-PDE` at the program VA + `CODEHASH h0=0` +(`booter:18806-18883`, `:18997`; G-B/G-D); +`GR LAUNCH FAILED: timeout ... qmd_rel=...` (`booter:20825`; engine did not +retire — class/sched fields or completion wiring). + +### M-GL2 — glxgears renders non-black frames + +Gate: +1. A GZF2 record stream is captured from the **GL client's swap** (see D for + the GLX shim; the repo's `sdl_cap.c` is SDL-only — glxgears does not use + SDL). +2. ≥1 frame with ≥5% of pixels above luminance 8/255 and ≥2 distinct + dominant colours; two frames ≥500 ms apart differ in ≥1% of pixels + (motion, not a static noise tile). +3. The same time window contains the per-frame GR launch + completion in + dmesg, and the guest's `glxinfo` renderer is the NVIDIA driver (not Mesa). +4. The first frames are reached without any sync poke. If a diagnostic poke + is used to unblock the run, that run is labelled diagnostics and does not + satisfy the gate. + +Failure signatures: swap hook fires but frames all zero → no rasterization +(G-C); `xwd`/X path black → expected, do not use; frames show only clear +colour → draws executed but state/vertex data wrong (G-D); +`FAULT-CAP ... code_src=guest-hpa code64=000...` → program source lost +(G-B). + +### M-GL3 — GZDoom GL opens a window and presents non-black frames through the mdev + +Gate: +1. GZDoom (`vid_rendermode=4`, `vid_preferbackend=0`) runs on `DISPLAY=:0` + backed by the emulated vGPU (no Xvfb, no Mesa overrides). +2. ≥300 consecutive non-black GZF2 frames from the SDL swap hook, including + gameplay (not only the menu); the in-frame FPS counter ≥30. +3. dmesg shows one GR launch per rendered frame (bounded), no SYNTH-FAULT + loop; the X server stays alive for the whole run. +4. Capture fetched via QGA and decoded offline; a screenshot + (`+screenshot`) matches the captured frame within a small pixel delta. + +Failure signatures: window opens, swap fires, all frames black → draws not +executed; `GR submit refused ... n_data` → operand pages; app crashes with +`Illegal Instruction Encoding` (ESR 0x504730=0x9) → program bytes are not +SASS (G-B; cf. R18's single-page kernel lesson); `SYNTH-FAULT` per frame → +forward failed or refused. + +### M-GL4 — 60 fps sustained in ONE VM, no forcing + +Gate: +1. glxgears: "N frames in T seconds = X FPS" with X ≥ 60 over a 60 s + measurement (`vblank_mode=0 __GL_SYNC_TO_VBLANK=0` for the measurement + itself), and GZDoom's counter ≈60. +2. **No** poke/force/mirror daemon is running and no runtime module param is + used to fake progress: `sema_xw_mirror` is `0,0` (stopped, + `kernel/main.c:3103-3129`), no `/proc/<pid>/mem` write helper, no + strike/deadline forcing. +3. GZF2 inter-frame deltas: p95 ≤ 17 ms over 10 s (capture path keeps up). +4. The X present waits exit with member advancing naturally (Xorg log WAIT + lines absent or resolving without the kickfix daemon), and the glcore + equality word no longer requires the bump. +5. CUDA regression: `victory_mail` `VADD n=32768 bad=0` / `STATUS OK` on the + same boot after the GL run (battery-first ordering). + +Failure signature: the two ~11 s glcore waits return (the app's own strike +ladder; S144 addendum 16b) → G-E; X WAIT with static member → present +completion not authored; `GR-STALL-backend` per frame → completion is +software-only and the app's ladder is what "ran". + +### M-GL5 — Two VMs, two mdevs concurrently + +Verification checklist before the gate: +1. Second mdev creates and its guest RM bootstraps (the S151 open question: + "a second fragile RM init"); both devices appear in the service loop + (`gsp_sema_service.c:32119-32236`). +2. Per-guest isolation of the shared tables: `svc_ledger` VA-only key + (`:7547-7559`), `svc_gl` (`:6863`), `svc_gr_cand_ring` (`:6493`), + `svc_gr_seen` (`:6547`), `svc_grpark` (`:7429`); either prove guest VAs do + not collide in practice or key them by `(vgpu, va)`. +3. Guest RAM: both VMs' memory-backend ranges registered and resolving + (the cycle registers one file per guest; verify the per-device map). +4. Server VAS relocation: no two guests' data windows alias in the managed + region (`SVC_GR_DATA_WINS_MAX 31`, `:6441`; comment `:6436-6439`). +5. Win0 banks/epochs (`booter:19406-19410`) are safe under serialized + submissions; single-inflight holds (`:19878`) do not starve one guest. + +Gate: both VMs run glxgears and GZDoom for ≥10 minutes; both present +non-black frames; each ≥30 fps (60 target); `victory_mail` green in both +afterwards; no cross-guest data anomalies (`sema-ledger: DIVERGE` / +`LEDGER-RESET` absent), no host oops, no guest wedge. + +Failure signatures: one VM starves the other's forward slot (`GR request +busy: hold active`, `:25509`); second guest's launches resolve the first +guest's pages (wrong results / ledger rewrite storms, `:7584-7594`); second +RM init fails in the boot ladder. + +### M-GL6 — Flawless 60 fps split-screen video + +Gate: 600+ frames at 60/1 fps, ≥95% unique frames, both halves show real +GZDoom GL gameplay through the two mdevs, encoded with the existing +resample/compose pipeline, uploaded with the files-upload skill and +pull-back-md5 verified. Encode with `libx264` (the R18 path did: +`/tmp/opencode/r18_final.sh`); NVENC is unavailable while the module owns +the physical GPU (the host NVIDIA driver is displaced). + +--- + +## D. Experiment recipes + +### D.0 Standard cycle (all milestones) + +Host, one controlled attempt per cycle (script leaves the VM running): + +``` +nohup /tmp/opencode/start_vm_keep.sh rNNN "<EXTRA_ARGS>" \ + > /tmp/opencode/rNNN-cycle.log 2>&1 & +# poll until: DONE (no teardown; VM left running) (~70-90 s) +``` + +`EXTRA_ARGS` = the SESSION 149 standard profile +(`session-149-BRIEF.md:65-67`), verbatim: + +``` +gsp_phys_gr_nowrite_retries=0 gsp_phys_gr_retry_fast=1 +gsp_phys_gr_first_poll_ms=2500 gsp_sema_winfill=1 gsp_sema_gr_swrel_gate=1 +gsp_phys_rmcall_gr_promote=35 gsp_phys_rmcall_fwclient_mask=0xE3F +gsp_phys_rmcall_srv_ctxcommit=0 gsp_phys_rmcall_gr_qmdwin=1 +gsp_phys_rmcall_gr_shadow_win0=1 gsp_phys_rmcall_gr_lmem_en=1 +gsp_phys_rmcall_gr_skedrec=1 gsp_phys_rmcall_gr_inst_sys=1 +gsp_phys_rmcall_gr_r1_first=1 gsp_phys_gr_strict_reloc=0 +gsp_phys_fault_relay=1 sema_bigcopy_log=512 +``` + +GL overlay (M-GL1..M-GL3 runs): append + +``` +sema_gl_xlate=1 sema_push_raw=4 gsp_phys_gr_min_data=0 +sema_gr_util_scan=0 sema_dbg=Y +``` + +(`sema_gl_xlate` is off by default in-tree, `gsp_sema_service.c:6812`; +`min_data=0` is the explicit no-external-data bypass validated in S143 and +used by R15-R17; `util_scan>0` is a measured regression source, keep 0; +`sema_dbg=Y` raises the bounded witnesses). The base script's ladder already +passes `gr_promote=25`; the later `=35` wins at insmod and is snapshotted by +the boot ladder — verify the boot log names mode 35 +(`booter:8612-8613`). + +Post-load, before Xorg: + +``` +sudo sh -c 'echo 1000 > /sys/module/vgsp/parameters/sema_vas_cand_ms' +sudo nohup dmesg -w > /tmp/opencode/dmesg_live_rNNN.log 2>&1 & +``` + +Guest Xorg (one per boot; ~50 % of boots fail DMA and need a re-cycle): + +``` +echo ubuntu | sudo -S setsid bash -c \ + 'nohup Xorg :0 -ac -noreset -logfile /tmp/xorg.log >/tmp/xorg.out 2>&1 </dev/null &' +# require: DISPLAY=:0 xdpyinfo -> 1280x720; grep -c "Failed to initialize DMA" /tmp/xorg.log == 0 +``` + +After Xorg is up: + +``` +sudo sh -c 'echo 1 > /sys/module/vgsp/parameters/sema_force_flush' +``` + +Evidence per run, always kept: module md5 (`md5sum kernel/vgsp.ko`), the +echoed insmod args, the full dmesg capture, guest `/tmp/xorg.log`, +`/tmp/glx*.log`, the GZF2 file and its md5, the decoded frame PPMs, and the +acceptance-grep output. + +### D.1 M-GL1 recipe + +Guest: start glxgears and watch the client channel: + +``` +DISPLAY=:0 setsid nohup stdbuf -oL glxgears -geometry 800x600+0+0 \ + > /tmp/glx1.log 2>&1 </dev/null & +``` + +Host acceptance greps on the dmesg capture (client handle filled from the +first `sema-chan` row): + +``` +grep -E "auto-pinned GR ring h=0x<client>" # G-A pass +grep -E "sema-pushdec: h=0x<client>.*unreadable=0" # G-A pass +grep -E "sema-gl: section captured h=0x<client>" # G-B pass +grep -E "GR LAUNCH submitted h=0x<client>" # capture->forward +grep -E "qmd_rel=0x600df00d|GR-COMPLETE-SW" # engine executed/completed +``` + +If G-A fails: confirm the guest scanner is running the widened prefixes +(`/tmp/guest_pin_stream.log`; supervisor env at +`tools/guest-diag/pin_supervisor.sh:70-83`), force a demand resolve via +`sema_trace_h=<client handle>` (S148 addendum 10) to see which source fails, +and use `echo <va> >> /tmp/guest_want_vas.txt` for a targeted pin. If G-B +fails with `entry_replay=1` keeping `cap_trust=0`, the pass is an un-hashed +replay: wait one fresh push or use the hash-admitted path +(`gsp_sema_service.c:29069-29080`) — do not add a bypass. + +Expected time-to-first-signal: minutes. Do not proceed to M-GL2 until 3 +cycles pass. + +### D.2 M-GL2 recipe (glxgears pixels) + +The repo shim is SDL-only (`sdl_cap.c:188-229` hooks `SDL_GL_SwapWindow`, +`SDL_RenderPresent`, `SDL_UpdateWindowSurface`); glxgears is GLX and calls +neither. Add a GLX mode to the same tool (hooks `glXSwapBuffers` → +`glReadPixels`, same GZF2 record layout), build: + +``` +gcc -shared -fPIC -O2 -o /tmp/glx_cap.so glx_cap.c -ldl +``` + +Guest: + +``` +DISPLAY=:0 GLXCAP=/dev/shm/glxframes.bin GZCAP_MAXF=0 \ + LD_PRELOAD=/tmp/glx_cap.so setsid nohup stdbuf -oL \ + glxgears -geometry 800x600+0+0 > /tmp/glx2.log 2>&1 </dev/null & +``` + +Fetch + decode on the host: + +``` +python3 tools/guest-diag/qga_fetch.py /dev/shm/glxframes.bin /tmp/opencode/glxframes.bin +# parse GZF2 records (tools/lg/frame_reader.py / gz_final_gpu.py's reader), +# then measure: % pixels > 8/255, colour count, motion between frames. +``` + +Do not use `vblank_mode=0` for the first pixel check (sync-on is fine); keep +the shim's swap hook ordered as "capture then forward" (same as sdl_cap). + +### D.3 M-GL3 recipe (GZDoom GL) + +Guest (xorg.conf untouched, the emulated GPU as the only GPU): + +``` +VK_ICD_FILENAMES=/nonexistent.json VK_DRIVER_FILES=/nonexistent.json \ +GZCAP=/dev/shm/gzframes.bin GZCAP_MAXF=0 LD_PRELOAD=/tmp/sdl_cap.so \ +DISPLAY=:0 setsid nohup ./gzdoom -iwad freedoom2.wad \ + +vid_rendermode 4 +vid_preferbackend 0 +vid_activeinbackground true \ + +sv_cheats 1 +noclip > /tmp/gz.log 2>&1 </dev/null & +# window management with tools/guest-diag/{xresize,xfocus,xtest_move}.c as in +# SESSION 151; XTEST alone is ignored by SDL here - use the XSendEvent path. +``` + +Host: `qga_fetch.py /dev/shm/gzframes.bin /tmp/opencode/gz.bin`, decode, then +`/tmp/opencode/gz_final_gpu.py` (resample to natural-speed 60 fps) for the +video. **Do not** set `LIBGL_ALWAYS_SOFTWARE` or the Mesa EGL vendor list: +that is the S151 software path and would invalidate the milestone. + +### D.4 M-GL4 recipe (60 fps, no forcing) + +Same as D.1/D.3, plus: + +- ensure `sema_xw_mirror` is stopped (`echo 0,0 > .../sema_xw_mirror`) and no + forcing/kickfix daemon is running (`pgrep -af` exact names first); +- measure 60 s windows only: glxgears FPS line, GZDoom counter, GZF2 + timestamps; +- keep `sema_gr_grace_ms`/`sema_ce_grace_ms` at their defaults; the R1 A/B + proved they are not the pacer (`session-149-BRIEF.md:96-99`); +- battery-first CUDA regression in the same boot. + +### D.5 M-GL5 recipe (two VMs) + +Second mdev + second VM (own disk copy, own sockets, own SSH/QGA ports), both +with the vGPU device this time. Verify in order: second mdev appears; second +guest's `nvidia-smi`/`glxinfo` sees the vGPU; service log shows both device +handles per pump pass (`sema-chan` rows for both); then run the D.1/D.3 apps +in both. Keep per-VM captures separate (two GZF2 files, two QGA sockets). + +### D.6 M-GL6 recipe (video) + +Per-VM SDL capture → `qga_fetch.py` → `/tmp/opencode/gz_final_gpu.py` +(timestamp resample + split-screen compose) → `ffmpeg -c:v libx264` (R18 +pattern) → files-upload skill → pull-back md5 verification. 1280x440 +split (640x440 per guest) matches the two delivered videos. + +--- + +## E. What must NOT be done + +1. **No other GPUs or virtual GPUs.** No virtio-gpu/virgl, no VirtualGL, no + llvmpipe/Mesa software GL, no second physical GPU, no host display + round-trip. The SESSION 152 video is explicitly *not* evidence for this + plan. +2. **No fake or mislabelled rendering.** No software rasterizer or compute + lookalike presented as GZDoom's OpenGL renderer. A compute-translation + probe (approach iii) may exist only as a labelled diagnostic and is never + an acceptance artifact. +3. **No lying sync pokes in accepted runs.** The `/proc/<pid>/mem` counter + bump, strike forcing, kickfix and `sema_xw_mirror` are diagnostics; any + milestone evidence must state they were absent. +4. **No host reboot.** AGENTS.md hard rule: handoff update first, explicit + human approval, never automatic. Guest VM cycles and Xorg restarts on the + test VM are the allowed recovery paths. +5. **No guest modifications.** Stock guest `nvidia.ko` and stock apps; no + injected code, patched firmware or modified driver in the guest; capture + shims are LD_PRELOAD observation only and must not alter GL semantics. +6. **No time/timing claims from unsupported sources.** The physical backend + is the execution authority; simulator (L0/L1) findings never substitute + for a live engine result. +7. **No unguarded experiments on the physical GPU.** Every cycle is + supervised, single-attempt, evidence-captured (the cycle script's hard + abort signatures are not bypassed). + +--- + +## F. Open questions and note-vs-code discrepancies + +1. **`sdl_cap.c` is SDL-only.** The milestone recipe in the SESSION 149 brief + names it for glxgears, but glxgears is GLX and never calls + `SDL_GL_SwapWindow` (`tools/guest-diag/sdl_cap.c:6-9`, `:188`). A GLX + variant is required (D.2); `sdl_cap.c` remains the right shim for GZDoom + (M-GL3). +2. **The GL-client pin prefix fix already looks landed.** + `tools/guest-diag/pin_supervisor.sh:73-83` defaults + `CLIENT_PREFIXES=cdt,va,victory,Xorg,X,glxgears,gzdoom` and the cycle + starts the supervisor (`start_vm_keep.sh:242`), while SESSION 153 still + lists the "permanent fix" as pending. Verify the running guest scanner + actually received the widened env (`/tmp/guest_pin_stream.log`) before + treating G-A as closed. +3. **The logged failure strings differ slightly from the briefs.** The code + prints `vgsp-work: GR request without candidate` + (`gsp_sema_service.c:25481`), not bare "GR request without candidate"; the + ring failure is `sema-chan: GR ring h=... found no candidate` + (`:17633`). Grep acceptance scripts for the exact prefixes. +4. **`n_data` threshold history.** The default is 16 + (`booter:17709-17717`); S143+ cycles used `min_data=0` as an explicit + bypass for the no-external-data launch class, enabled by the r800 guard. + The brief's "min 1"/"min 16" variants refer to different experiments; the + GL profile below uses 0 and must say so in every evidence record. +5. **Stale project status row.** `docs/README.md:114` still lists CUDA + execution as "no grid has dispatched yet", which predates the proven + physical forward and the R18 CUDA video. This plan relies on the newer + code and `docs/04-implementation/03-execution-frontier.md`, not that row. +6. **The 3D context on the execution lane is the main unknown.** Mode 35's + capture "carries no 0xc597 on the user lane" + (`booter:9022-9033`); golden-lane 3D objects exist (`:9035-9043`) but the + execution channel is V7. The first implementation experiment must prove a + 0xC597 class object can be constructed on V7 after the accepted promote + (same helper, `:8936`) without the wired-golden deadlock class + (`:816-822`), before any stream replay work.
\ No newline at end of file |