diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/01-orientation/01-purpose-and-scope.md | 32 | ||||
| -rw-r--r-- | docs/03-architecture/13-vexecution-plane.md | 81 | ||||
| -rw-r--r-- | docs/04-implementation/03-execution-frontier.md | 232 | ||||
| -rw-r--r-- | docs/04-implementation/04-real-render-walls.md | 909 | ||||
| -rw-r--r-- | docs/04-implementation/05-graphics-execution-plan.md | 721 | ||||
| -rw-r--r-- | docs/05-operations/01-performance-correctness-security.md | 42 | ||||
| -rw-r--r-- | docs/05-operations/05-gpubreach-audit.md | 95 | ||||
| -rw-r--r-- | docs/05-operations/06-guest-frame-path-and-looking-glass.md | 102 | ||||
| -rw-r--r-- | docs/README.md | 48 |
9 files changed, 2108 insertions, 154 deletions
diff --git a/docs/01-orientation/01-purpose-and-scope.md b/docs/01-orientation/01-purpose-and-scope.md index ef4b258..242f470 100644 --- a/docs/01-orientation/01-purpose-and-scope.md +++ b/docs/01-orientation/01-purpose-and-scope.md @@ -131,20 +131,26 @@ document: * **Working today**: the virtual device, the GSP boot handshake, the RPC transport, the resource-manager emulation surface the guest driver needs to - probe and enumerate, guest `nvidia.ko` and `nvidia-smi`, and the full CUDA - driver-API path through context creation, allocation and verified copies - (the driver-API smoke passes). -* **Needs improvement**: engine execution first — the backend is live and - CE-class copies and their releases are forwarded to the physical copy engine - (default off) with a fault relay, but for CUDA kernel execution the guest's - launch now reaches the front-end and the inline launch-descriptor install has - not been observed (context validity at dispatch is ruled out; see - [`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md)) - — then the remaining reply fidelity (some consumed - fields are still generic fallbacks), the memory/page-table model, Windows - guest support, non-KVM hypervisors, and performance instrumentation. Each of - these is tracked with explicit "known-unknowns" in + probe and enumerate, guest `nvidia.ko` and `nvidia-smi`, the full CUDA + driver-API path, and **CUDA kernel execution on the physical GPU** — + verified guest clients (vector-add, matrix-multiply, transformer) execute on + the engine with engine-authored completions relayed to the guest. On the + graphics side, the X server and GLX run on the emulated vGPU, the engine + executes GL method streams (verbatim 3D replay and synthesized launches), + and the guest Vulkan pipeline runs end to end. +* **Needs improvement**: the staged-transfer class (some large multi-chunk + CUDA clients still mis-verify; failure modes proven), the every-boot + acceptance campaign, per-guest execution isolation for simultaneous guests, + the rendered-pixel frontier for OpenGL/Vulkan (program delivery plus + program-VA mapping), the remaining reply fidelity (some consumed fields are + still generic fallbacks), Windows guest support, non-KVM hypervisors, and + performance instrumentation. Each of these is tracked with explicit + "known-unknowns" in [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md); + execution detail: + [`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md) + and + [`../04-implementation/05-graphics-execution-plan.md`](../04-implementation/05-graphics-execution-plan.md); for the CUDA analysis area, start at [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md). diff --git a/docs/03-architecture/13-vexecution-plane.md b/docs/03-architecture/13-vexecution-plane.md index 733da13..3d97d07 100644 --- a/docs/03-architecture/13-vexecution-plane.md +++ b/docs/03-architecture/13-vexecution-plane.md @@ -568,6 +568,39 @@ forwards (local evidence and the prepared harness: forwarding-service cost, not engine time — the number to drive down is the 7.9 ms and the per-window round trips. +### 3.5a Copy-out change detection: fingerprint rule + +A forwarded launch stages every referenced data page into the data-bounce +chunk, and after the retire wait the copy-out ships back only the pages the +engine actually modified. That decision used to be a 32-bit additive +checksum of the page's words, compared against a baseline captured at +copy-in. The additive form has a structural blind spot: doubling every word +of a 4 KB page raises each float exponent by one (+2^23 per word), and 1024 +words contribute `1024 * 2^23 = 2^33`, which is exactly `0 (mod 2^32)` — an +in-place power-of-two scale of a whole page (`c[i] *= 2.0f`) left the +checksum bit-identical, the page was classified "engine did not touch", +the write-back was skipped, and the client read the pre-scale values. The +same blindness applies to `*= 0.5f`, `* 4`, `* 8`, etc. Since `r452` +(commit `19399f4f`) every change decision in the copy path — the retire +wait's settle scan, the tracked and gap copy-outs, the window witness, the +late-store witness and the idempotent-launch fingerprint — uses a 64-bit +FNV-1a hash over the page instead; the additive sum survives only as the +all-zero filter. Policy rule for the plane: **a change-detection +fingerprint must not be an algebraic invariant of any plausible kernel +operation** — use a hash, never a sum. + +The reader side of the same decision is the landing table: a D2H copy of a +device VA resolves through the last recorded write of that VA. Since `r453` +the forwarded QMD's release0 is redirected to the module semaphore by +default (`gsp_phys_rmcall_gr_rel_mod=1`): the guest-visible release word is +authored by the host only on the completion edge, after the copy-out has +shipped the engine's stores, so a client's stream-ordered readback can no +longer run ahead of the write-back (the historical shape: the engine's own +release satisfied the guest while the copy-out was still pending). The +engine-completion observable remains the module semaphore, and the +per-outcome authoring path guarantees the guest word is applied on every +outcome (timeout, wedge, no-write). + ### 3.6 Citation map for this section | Claim | Authority | @@ -1038,6 +1071,53 @@ technical barrier between the project's architecture and its promise. --- +## m91-class I-fetch faults: capture and icache coherence (2026-10-18) + +The physical submit's remaining transient failure class ("m91 class" after the +trap-decode campaign that first named the SM instruction-fetch signature) has +two observable faces. In one, the first attempt of a mid-sequence launch +raises SM warp exceptions ("Illegal Instruction Encoding", "Out Of Range +Register") on garbage instruction bytes while the engine-authored release +stays at its failure value, and a retry of the *identical* QMD and shadow +content succeeds. In the other, the launch reports fetch success but writes +nothing (dispatch loss). The submit retry that recovers the first face is the +fast re-arm: the same method block is re-published on a fresh GPFIFO ring +entry and the mode-35 lane-enable tail plus channel/TSG schedules and the +doorbell are re-issued — no shadow re-copy, no QMD re-read, no content +change. Identical content flipping outcome across attempts is the signature +of stale engine-side fetch state, not a content or relocation error. + +Two code changes follow from that: + +* **FAULT-CAP (diagnostic).** When a forwarded launch's fault epoch advanced + across its window, the submit emits one `vgsp-m91: FAULT-CAP` line per + launch (capped at 20 per module-load cycle, then one suppression notice): + fault-epoch delta, the forwarded QMD address and first 64 B, the live + program pair (word 48/49 read) with its resolved engine-visible VA/PA, the + first 64 B at that code address, the retry decision (fast re-arm / + resched/fresh-entry / none) and outcome (ok vs timeout), and the newest + relay-decoded fault note (exceptType, chid, errString). It is inert on a + fault-free launch: one epoch read and one compare. +* **icache coherence (fix, default on).** The window-0 shadow pool alternates + two 8-page banks by submit epoch, so the bytes at a program VA change every + two launches, while the captured guest QMDs carry the QMD v2.3 + cache-control word (word 5) with the + `INVALIDATE_INSTRUCTION_CACHE` bit (bit 30) clear — the field the client + driver arms only under its forced-invalidate mode. The submit now sets + that bit in the forwarded QMD when the QMD carries the driver's standard + cache-control signature (word 5 bits 26..29 set), forcing an instruction + fetch at dispatch and removing the stale-line window; an unknown QMD + revision is never touched, and `gsp_phys_gr_icache_inv=0` restores the + exact pre-change QMD for a one-flip A/B. Setting the bit cannot alter + instruction bytes or data — it only invalidates cached fetch state — so + green launches keep their semantics. + +The next physical cycle's evidence is the FAULT-CAP lines (or their absence) +against the four-client class: zero FAULT-CAP lines with the four clients +green confirms the stale-fetch mechanism; FAULT-CAP content policies the +remaining candidates (shadow-VA reuse distance, cross-cycle context state +left by killed clients). + ## See also * [`01-system-overview.md`](01-system-overview.md) — principles and layer map @@ -1057,3 +1137,4 @@ technical barrier between the project's architecture and its promise. |---|---| | 2026-10-08 | Created: canonical name (vGSP Execution Plane, vEP), four domains, component/data-flow/mode-matrix diagrams, discovery/truth/release/completion/submit reality, GR chain vocabulary, arbitrary-CUDA program, challenges, iteration plan, portability paths, CUDA/architecture mapping, future gaps, 3D interaction, framing. Tree `66388560`. | | 2026-10-08 | Review pass: status hash refreshed to `master @ 1f571dda`; drifted line anchors updated for the post-`66388560` code (`svc_thread`, `svc_gr_request`, `svc_pending_add`, H3 guard, readiness gate); GR completion §3.4/§6.1 reconciled with FORENSICS-grv5 (wake-up fires; the counter publication is the leading reading); discovery cadence wording corrected (2-second paced passes, not "milliseconds"); mode-matrix source note corrected. | +| 2026-10-18 | m91-class I-fetch section: FAULT-CAP first-fault capture (`vgsp-m91: FAULT-CAP`, 20/cycle) and the default-on forwarded-QMD instruction-cache invalidate (`gsp_phys_gr_icache_inv`) with the retry-state analysis (identical content, fresh ring entry + lane-enable replay; stale engine-side fetch state at reused shadow VAs). | diff --git a/docs/04-implementation/03-execution-frontier.md b/docs/04-implementation/03-execution-frontier.md index f5546ca..a8e03ee 100644 --- a/docs/04-implementation/03-execution-frontier.md +++ b/docs/04-implementation/03-execution-frontier.md @@ -1,11 +1,17 @@ # The Execution Frontier: The GR Compute-Launch Chain -> **Status: 2026-10-06.** This page describes the current leading edge of +> **Status: 2026-10-20.** This page describes the current state of > guest-facing execution: the chain that must fire for a stock guest CUDA -> kernel to run on the physical GPU, what we have proven about it, where it -> currently stops, how we know, and the open lines of investigation and -> theories. It is evidence-led: every statement carries either a live -> observation, a decompilation finding, or a clearly labelled theory. +> kernel to run on the physical GPU, what we have proven about it, how we +> know, and where the leading edge is now. The chain fires end to end: +> verified guest CUDA clients (vector-add, matrix-multiply and a transformer +> kernel) execute on the physical GPU, their results verify, and the +> engine-authored completions relay back to the guest. The leading edge has +> moved from "can a forwarded launch execute at all?" to breadth and +> hardening — the large-buffer staged-transfer class, reproducibility across +> boots, and per-guest execution isolation. It is evidence-led: every +> statement carries either a live observation, a decompilation finding, or a +> clearly labelled theory. > > Evidence sources for this page: supervised physical cycles (module-owned > GSP-RM on the reference TU104), offline firmware emulation/probing, the host @@ -40,40 +46,59 @@ flowchart TB J["Host polls release, relays to guest;<br/>CUDA wakes"] ``` -| # | Stage | State (2026-10-06) | +| # | Stage | State (2026-10-20) | |---|---|---| | 1 | Guest builds launch + method stream | ✅ observed (stock driver, unmodified) | | 2 | GPFIFO entry + doorbell | ✅ observed | | 3 | Host decode + forward stream + QMD prewrite + ring/USERD | ✅ observed (forwarding path) | | 4 | Front-end fetches the ring entry | ✅ observed (ring cursor advances) | -| 5 | Front-end executes methods | ✅ observed on the ladder channel (object bind, cache ops, report semaphore) | -| 6 | **Inline-QMD install** (descriptor materialised into the QMD buffer) | ✗ **not observed** | -| 7 | **Engine reads the QMD** | ✗ **not observed** | -| 8 | Context switch loads the channel context | ✅ context validity at dispatch is ruled in (instance flag set) — see §4 | -| 9 | GR dispatches the grid to the SMs | ✗ never observed on this stack | -| 10 | Engine authors release + channel semaphore | ✗ absent for forwarded launches | -| 11 | Host relays completion; guest wakes | ✗ guest waits and times out | - -**One-line summary:** the guest, the emulation, the forwarding path and the -front-end are live; the **launch-descriptor install path** (stages 6–7) is the -leading edge, and with it stages 9–11. +| 5 | Front-end executes methods | ✅ observed (object bind, cache ops, report semaphore) | +| 6 | Inline-QMD install (descriptor materialised into the QMD buffer) | ✅ observed | +| 7 | Engine reads the QMD | ✅ observed | +| 8 | Context switch loads the channel context | ✅ context validity at dispatch is ruled in — see §4 | +| 9 | GR dispatches the grid to the SMs | ✅ observed (the guest's results verify) | +| 10 | Engine authors release + channel semaphore | ✅ observed — engine-authored releases relay to the guest | +| 11 | Host relays completion; guest wakes | ✅ observed — the guest's synchronisation waits resolve | + +**One-line summary:** the whole chain is live — verified guest CUDA clients +execute on the physical GPU and wake up correctly. The leading edge is no +longer the chain itself but its breadth and hardening: the large-buffer +staged-transfer class, reproducibility across boots, and per-guest execution +isolation. --- ## 2. Where the leading edge is -A stock guest program now reaches its kernel launch and submits it through the -forwarding path. On the wire we observe the ring entry consumed and the -front-end methods run — but the launch descriptor is never materialised, the -engine never reads a QMD, no engine-authored completion appears, and the guest -blocks in its synchronisation wait. - -This is **not** a context-validity wall (that was the previous frontier and is -now ruled out at dispatch, §4), and it is **not** a window-mapping wall for the -descriptor destination (that install is accepted). The wall is the **form of -the launch method stream**: what the front-end requires in order to install the -inline descriptor, and what the engine requires in order to read and dispatch -it. +A stock, unmodified guest CUDA program now executes its kernels on the +physical GPU and verifies its results. On fresh boots the vector-add client +(`victory_mail`) prints `VADD n=32768 bad=0` + `STATUS OK`, the +matrix-multiply client (`victory_matmul`) prints `MATMUL n=256 bad=0` + +`STATUS OK`, and a transformer kernel prints `dh=256 seq=64 bad=0`, +`FFN4 layers=4 bad=0` + `STATUS OK`. The executing engine is the physical GPU +under the module's own real GSP-RM session; the host CUDA bridge is not part +of this path. The engine authors the completions, the host relays them to the +guest, and the guest's synchronisation waits resolve. + +The leading edge has therefore moved from the launch chain itself — it fires — +to breadth and hardening around it: + +* **The large-buffer staged-transfer class.** Multi-megabyte staged transfers + between guest memory and device memory are under repair: some large + multi-chunk clients still mis-verify. +* **Reproducibility across boots.** A boot-poisoning defect was root-caused + and fixed: under the driver's status-queue mode the emulation read the wrong + queue cursor, so under load the reply ring could desynchronise, the first + CUDA client would fail, and the rest of that boot was poisoned (the + user-visible `STATUS FAIL`-then-freeze report). Fixed; re-measurement + across fresh boots is in progress. +* **Per-guest execution isolation.** Multiple mediated devices with different + emulated VRAM sizes and multiple guests already coexist at the device level; + per-guest execution isolation for multiple simultaneous CUDA guests is in + progress. + +No breadth beyond these is claimed: OpenGL/Vulkan, Windows guests, migration +and performance numbers remain planned. --- @@ -81,17 +106,17 @@ it. | Observable | What it proves | What it does **not** prove | |---|---|---| -| Ring entry consumed; ring cursor advances | The front-end is alive and fetching | Nothing about the engine | -| Method execution on the ladder channel (report semaphore, channel semaphore) | The front-end can execute the method vocabulary | **Not** engine execution — these signals can be front-end-authored | -| Descriptor buffer prewritten by the host with a detectable sentinel | Whether the descriptor was materialised/modified after submission | A read does not modify memory; the sentinel detects installs, not reads | -| Install sentinel unchanged after submit | The inline-install method did not run | It does not say why (mapping, required control method, or ordering) | -| Engine-authored release word absent; completion poll times out | The engine did not complete the launch | It does not say whether the engine started | -| No GR fault recorded | The engine did not raise an error | Silence is not health: a skipped or never-reached launch is silent by construction | -| Channel instance context flag set | The channel is context-valid at dispatch | Nothing about the descriptor path | - -**Consequence:** progress on this frontier must be measured by an -**engine-authored** signal (the release word advancing) — not by front-end -signals, which we have already learned to distinguish. +| Client verification output (`bad=0` + `STATUS OK`) | The guest executed its kernel, read its results back and verified them | by itself, which engine produced the results | +| Engine-authored release word advancing, relayed to the guest | The GR engine executed the launch and the completion relay path works | nothing about other launches | +| The guest's synchronisation wait resolving | Completion reached the guest through the relay | nothing about other launches | +| Ring entry consumed; ring cursor advances | The front-end is alive and fetching | **not** engine execution — these signals can be front-end-authored | +| Reproduction across fresh boots | The path is not a lucky boot | not yet an "every boot" claim while re-measurement is in progress | +| No GR fault recorded | The engine did not raise this error | silence is not health: a skipped or never-reached launch is silent by construction | + +**Consequence:** acceptance is now measured by verified guest results plus the +engine-authored release; front-end signals remain necessary but not +sufficient. A green run counts as progress only when the same acceptance +reproduces across fresh boots. --- @@ -111,90 +136,94 @@ signals, which we have already learned to distinguish. * **Parenting/TSG for the fetch** — both parenting variants fetch identically. * **The copy-engine path** — CE-class work, its releases and its fault relay are live (default off); the GR path is a separate problem. +* **The descriptor-gate line of investigation (the old T1–T4 theories).** The + inline-descriptor install, the engine's QMD read, the grid dispatch, the + engine-authored release and the guest wake-up are all now observed; the + gate they questioned is closed by observation (§1, §7). +* **The old boot-poisoning defect.** Root-caused to the status-queue + consumption cursor and fixed; it is no longer the explanation for a failing + first client (§5, issue B). --- ## 5. The clear issues (ranked) -**Issue A — the forwarded stream carries a verbatim guest prelude that cannot -complete.** -The forwarded stream includes a verbatim copy of the guest's prelude whose -inline-data writes target addresses that are not present in the engine's view -of the channel's page tables, and the referenced data is not relocated. The -shorter, proven stream shape fires its tail (channel semaphore) but still does -not schedule the descriptor. -*How we know:* the prelude stalls in the forwarded form; the shorter shape's -tail fires; the descriptor-install sentinel never changes in either case. -*If fixed:* the install path is reached and we can observe stage 6 directly. - -**Issue B — the descriptor-virtualization control method is absent.** -The method that arms the inline-descriptor install path (the QMD -virtualization control) is not present in the forwarded stream. -*How we know:* the method does not appear in the submitted stream, and the -install sentinel never changes. -*If fixed:* the front-end should materialise the descriptor — falsifiable in -one cycle by watching the sentinel and the engine-read signal. - -**Issue C — diagnostic hygiene.** -The descriptor sentinel placed at word 0 overlaps real descriptor fields; it -must be moved to an unused word so diagnostics do not perturb what they -measure. +**Issue A — the large-buffer staged-transfer class.** +Multi-megabyte staged transfers between guest memory and device memory are +under repair: some large multi-chunk clients still mis-verify. +*How we know:* the small and medium clients (vector-add, matrix-multiply, the +transformer kernel) verify green while large multi-chunk transfers mis-compare +or stall in the same configurations. +*If fixed:* the client battery should accept the large-buffer class with the +same verification discipline as the green clients. + +**Issue B — reproducibility across boots (re-measurement).** +A boot-poisoning defect was root-caused and fixed: under the driver's +status-queue mode the emulation read the wrong queue cursor, so under load the +reply ring could desynchronise, the first CUDA client would fail, and the rest +of that boot was poisoned (the user-visible `STATUS FAIL`-then-freeze report). +*How we know:* the failure was reproduced, the cursor rule was corrected, and +per-client unregister and session flush work again. +*If fixed:* green on every fresh boot — re-measurement is in progress, and no +"every boot" claim is made until it completes. + +**Issue C — per-guest execution isolation.** +Multiple mediated devices with different emulated VRAM sizes and multiple +guests already coexist at the device level; per-guest execution isolation for +multiple simultaneous CUDA guests is in progress. +*How we know:* the device-level coexistence is live; execution isolation is +not complete. +*If fixed:* multiple CUDA guests execute simultaneously without interference. --- ## 6. Lines of investigation (live) -1. **Restore the proven launch shape** as the baseline (the shorter stream that - fires its tail) and **re-add the virtualization control method** — one - variable at a time, with the install sentinel and the engine-read signal as - the readout. -2. **Relocate or drop the verbatim prelude** so the inline-data writes target - addresses present in the engine's view, or replace it with the equivalent - host-side staging the proven shape already uses. -3. **Move the sentinel** and re-run to confirm the readout is clean. -4. **Diff against a known-good reference.** Two ways to obtain one: (a) make - our own ladder dispatch a descriptor built from the same source material - (in-house reference), or (b) capture a working launch of the stock stack on - the same GPU (real method stream + descriptor + engine-visible channel - state) and compare field by field. Either reference converts the remaining - work from inference to differential. -5. **Offline probing** of any rule the front-end or engine is believed to - require, using the firmware emulator/probe tooling before spending a - supervised cycle. +1. **Repair the large-buffer staged-transfer class** — the multi-megabyte + guest↔device transfers that still mis-verify. The three green clients stay + the regression battery and must remain green throughout. +2. **Complete the fresh-boot re-measurement** — demonstrate the fixed boot + path across fresh boots before claiming boot-to-boot reproducibility, and + keep the re-measured battery as the regression gate. +3. **Per-guest execution isolation** — build from the device-level + coexistence of multiple mediated devices and guests to per-guest execution + isolation. +4. **Hold the acceptance battery** — vector-add, matrix-multiply and the + transformer kernel run as a battery on every re-measurement; each entry is + accepted only on its own verification output (`bad=0` + `STATUS OK`). --- -## 7. Theories (labelled; not yet proven) - -* **T1 — the virtualization control is the gate.** The front-end will not - materialise an inline descriptor until the virtualization-control method - enables the install path; its absence alone explains the inert descriptor. -* **T2 — materialise-then-read.** The engine only schedules a descriptor that - exists in its own view; an install targeting unmapped destinations aborts - silently rather than faulting. -* **T3 — the verbatim prelude poisons the launch.** Even where the tail shape - works, the prelude's refused writes leave the launch path in a state that - prevents the descriptor from being scheduled. -* **T4 — relocation is the missing transform.** The guest's inline descriptor - is valid, but it is built for the guest's view of memory; the forwarding path - must relocate referenced data (and any embedded pointers) into the channel's - view before install. - -Each theory has a one-cycle falsifier: the install sentinel, the engine-read -signal, and the engine-authored release, in that order. +## 7. Theories, resolved + +The descriptor-gate theories (T1–T4) are resolved and are not kept as open: +the inline-descriptor install, the engine's QMD read, the grid dispatch, the +engine-authored release and the guest wake-up are all directly observed (§1), +so the launch chain they questioned is now proven end to end. + +The remaining leading-edge work is engineering rather than theory: the +large-buffer transfer class, boot-to-boot reproducibility, and per-guest +execution isolation. Each has an acceptance signal named in §5 before the +work. --- ## 8. Claim discipline -* **Engine-authored release is the only proof of execution.** Front-end - signals (ring cursor, method execution, semaphores that the front-end can - author) are necessary but not sufficient. +* **Engine-authored release is the acceptance signal for execution** — now + observed end to end for the verified clients; the guest's own verification + (`bad=0` + `STATUS OK`) is the guest-side acceptance signal. +* **Front-end signals are necessary but not sufficient.** Ring cursor and + front-end method execution can be front-end-authored; they never substitute + for the engine-authored release. * **Silence is not health.** No fault recorded does not mean no fault occurred; the current GR path runs without an error-reporting object, which is itself a known observability gap. * **Emulator verdicts are L0/L1.** They bound behaviour, never timing or device environment. +* **A green run counts only when it reproduces across fresh boots.** The + boot-poisoning defect is the worked example: one green boot is not + reproducibility. * **A wall is closed only by an acceptance signal**, and the acceptance signal for this frontier is named before the experiment. @@ -218,3 +247,4 @@ signal, and the engine-authored release, in that order. | Date | Change | |---|---| | 2026-10-06 | Created: the GR compute-launch chain, the leading edge (inline-descriptor install), evidence limits, exonerated walls, ranked issues, live investigation lines and theories. | +| 2026-10-20 | Chain complete: verified guest CUDA clients execute on the physical GPU (vector-add, matrix-multiply, transformer; engine-authored completions relay and guest waits resolve). Leading edge moved to the large-buffer staged-transfer class, fresh-boot reproducibility (boot-poisoning defect fixed) and per-guest execution isolation; the descriptor-gate theories are resolved. | diff --git a/docs/04-implementation/04-real-render-walls.md b/docs/04-implementation/04-real-render-walls.md new file mode 100644 index 0000000..f2ba0ab --- /dev/null +++ b/docs/04-implementation/04-real-render-walls.md @@ -0,0 +1,909 @@ +# Real-render walls: Vulkan and Xorg on the emulated vGPU + +Status: OPEN (2026-10-29, SESSION 64). This page records the measured state +of the guest's *display/graphics* consumers - the Vulkan ICD and the Xorg +server - on the emulated vGPU, with the exact evidence and the next +subsystem to implement. It complements +[05-operations/06-guest-frame-path-and-looking-glass.md](../05-operations/06-guest-frame-path-and-looking-glass.md) +(which covers the CUDA -> ivshmem -> host frame transport that is already +proven). + +## 1. What works today + +* The guest's shipped 570.144 Vulkan userspace **loads and enumerates**: + `vkCreateInstance` succeeds and two physical devices are reported - + `NVIDIA GeForce RTX 2080 SUPER` (discrete, Vulkan 1.4.303) and the + software `llvmpipe` - from `/etc/vulkan/icd.d/nvidia_icd.json`. Probes: + `tools/vk/vk_probe.py`, `tools/vk/vk_probe2.py`. +* `eglInitialize` succeeds with `vendor=NVIDIA` on the guest EGL device + platform (the EGL probe `tools/guest-diag/egl_retry.c` and the older + `egl_frame`). +* CUDA is unaffected (its clients run through the same emulated GSP; see + the battery evidence in the handoff). + +## 2. Wall 1 - `vkCreateDevice` fails in nvidia-modeset's ALLOC_DEVICE + +`vkCreateDevice` returns `VK_ERROR_INITIALIZATION_FAILED`. An +`LD_PRELOAD` ioctl logger (`tools/guest-diag/nvkmstrace.c`) shows the +ICD's first nvKms ioctl failing: + +``` +NVKMS_IOCTL cmd=0 size=1488 addr=0x... rc=-1 # NVKMS_IOCTL_ALLOC_DEVICE +``` + +i.e. the kernel's `nvKmsAllocDevice` (in `nvidia-modeset`) returned a +non-success status; the userspace frees everything and reports the +generic initialization error. Reading the driver's own modeset source +(`nvkms-rm.c`) gives the alloc order and the two gates we hit and fixed: + +1. `ProbeHeadCountAndWindowAssignment()` -> `NV0073_CTRL_CMD_SYSTEM_GET_NUM_HEADS` + (`0x730102`): a zero reply logs `No heads found on board!` and fails the + alloc. The emulated GSP RSP answered all NV0073 controls through the + generic zero-reply path. +2. After the heads: `NoValidDisplays()` -> `NV0073_CTRL_CMD_SYSTEM_GET_SUPPORTED` + (`0x730107`) must report at least one display id, and the connector + path (`SPECIFIC_OR_GET_INFO` `0x73028b`, `SPECIFIC_GET_CONNECTOR_DATA` + `0x730250`, `DFP_GET_INFO` `0x731140`) must describe a connector. + +The host GSP-RM emulation now implements a minimal headless display: +one head (`0x730102`/`0x730287`), flexible window assignment (`0x7302ad`), +one static HDMI connector on display id 0 (SOR + single-link TMDS, which +avoids the DP-AUX library), plus DDC support. With this shim the alloc +gets past heads and `NoValidDisplays` and proceeds through the connector +queries and the display-object allocations (classes `0xc570`, `0xc57d`, +`0xc372`, controls `0x50700709`, `0x73010c`, `0xc3700201`) before failing +later in the display-core/modeset-channel path. That remaining subsystem +is the wall: the modeset core channel and the per-display dpy objects +must be emulated (a coupled object + channel + memory set whose exact +wire forms should be captured from a real-GPU session). + +A related experiment ruled out one shortcut: booting the guest with +`nvidia-modeset`/`nvidia-drm` blacklisted (no `/dev/nvidia-modeset`) does +not avoid the wall - the ICD's nvKms client still issues ALLOC_DEVICE and +fails, so the display path cannot be sidestepped by disabling KMS. + +## 3. Wall 2 - Xorg ScreenInit fails in nvidia-push's channel test + +Xorg starts (the GPU is detected, `Virtual screen 1280x1024`) and then: + +``` +(EE) NVIDIA(GPU-0): Failed to initialize DMA. +(EE) NVIDIA(0): Failed to allocate push buffer +``` + +`nvidia-push` logs `Failed to initialize DMA.` from +`__nvPushTestPushBuffer()`: right after allocating a push buffer the +driver writes a NOP and waits (10 s) for the channel to consume it. Host +evidence shows why it never does: + +* the X driver's channel registers (`class 0xc46f` GPFIFO channel, + `gpfifo=0x120090000`, `userd=0x350000` sysmem) and lives with + **`submissions=0`** - its published GP_PUT never advances in the host's + view, because the host needs the channel's VA->PA translations to read + its USERD/GPFIFO and that resolution comes from the guest-side scanner + that currently follows *CUDA clients only* (`targeted resolve + va=0x20400f000: full RAM scan found nothing`); +* on a busy boot (Xorg started while a CUDA battery ran) the channel's + completion-counter slot is rejected: `counter auto h=0x1000013 rank=0 + pa=0x11c827ff0 rejected (slot owned by another channel)` - two + concurrent clients both derive rank 0's slot from the per-device + structural formula. + +Fixing Xorg therefore needs (1) a platform-wide (non-CUDA) VA-resolution +path for the scanner or the host, and (2) counter-slot ownership keyed by +client identity (or a per-client slot domain), not just rank. + +## 4. Wall 3 - the EGL/glsi engine-init gate (unchanged) + +`eglMakeCurrent` returns `EGL_BAD_ALLOC (0x3003)`; the glsi engine-init +entry returns a "busy" status once, sets a state bit, and the eglcore +caller does not retry. Ten in-process retries (`egl_retry`) all fail, so +the state machine fails on a real input - the return-value trace of its +callees remains the way in (see +[userspace/cuda/570.144/docs/ANALYSIS-egl-engine-selection.md](../../userspace/cuda/570.144/docs/ANALYSIS-egl-engine-selection.md)). + +## 5. Why this matters for the frame path + +Everything downstream of a working GL/Vulkan context is already in place: +a guest publisher writes frames into the kvmfr ivshmem BAR +(`tools/vk/vk_frame.c`, `tools/guest-diag/xshm_pub.c` use the same layout +as the proven CUDA `frame_producer`), the host reader decodes them +(`tools/lg/frame_reader.py`) and ffmpeg assembles video. The missing +piece is a *working* graphics API on the guest GPU - the two subsystems +above. +## 6. UPDATE 2026-10-25 (RESUME #56): the wall is now fully bisected + +Three new instruments settled the remaining questions: + +* **An `LD_PRELOAD` nvKms ioctl logger** (`tools/guest-diag/nvkmstrace.c`) + decodes the failing `NVKMS_IOCTL_ALLOC_DEVICE` reply: status **5 = + `NVKMS_ALLOC_DEVICE_STATUS_CORE_CHANNEL_ALLOC_FAILED`**. +* **The QEMU gdb stub (`-gdb tcp::1234`) as a guest-kernel microscope** + (the guest kernel's `nvidia_modeset` symbols are in `/proc/kallsyms`): + breaking inside `nvAllocCoreChannelEvo` (the blob's offsets were mapped + from a disassembly of the shipped `nv-modeset-kernel.o_binary`) dumped + the driver's class list (42 classes; all `NVC57x` display classes are + present; **`NV927C_BASE_CHANNEL_DMA` (0x927c) is not** - it exists in no + HAL class list, so the "base channel" fallback path can never succeed). + Breaking at the capability parse showed the **capability notifier buffer + is all zeros** (nothing emulates the display engine that would fill it). +* **The caps-fill experiment**: with gdb writing `SYS_CAP = HEAD0_EXISTS` + and `SYS_CAPB = WINDOW0..7_EXISTS` into the caps buffer right before the + parse, the driver flips from the dead base path to the **window path** + (`nvRMAllocateWindowChannels`) - the hypothesis that `numWindows > 0` + unblocks the flow is confirmed. + +The window path then fails one step later: `nvRmApiAlloc(display, +class=0xc57e NVC57E_WINDOW_CHANNEL_DMA)` returns **`NV_ERR_NOT_SUPPORTED` +(0x56) locally in the guest kernel** (it never reaches the emulated +GSP - no ALLOC RPC is logged). This matches the shipped driver's vGPU +architecture: the guest runs in vGPU mode (`IS_VIRTUAL`), where the +display/modeset channel classes are rejected (the reference source's +`_rpcAllocObjectPrologue` documents "Display classes are added for DCE, +vGPU Guest does not support those") - a vGPU guest's display is meant to +be emulated by the **host-side vGPU display service** (the vGPU RPC +display path), which the emulation does not implement. + +**Consequence for the roadmap.** Guest Xorg/GL/Vulkan on this emulated +vGPU needs (a) the display capability fill on the host side (mechanics +proven by the gdb experiment) and (b) a vGPU display service: the +display-engine MMIO/RPC backend the guest expects from its host (the +equivalent of NVIDIA's vGPU display plugin), or the specific vGPU +configuration that makes the kernel accept the display channel class. +Until (b) exists the guest's NVIDIA display path is closed by design, and +GZDOOM-class workloads cannot render on the guest GPU. +## 7. UPDATE 2026-10-25 #2 (SESSION 66): the 0x56 pinned to the RS alloc path + +A new instrument closed the gap: the guest's `nvidia.ko` (DKMS build) is +**not stripped** - copy it out (`/lib/modules/<kver>/updates/dkms/nvidia.ko`), +`objdump -t`/`-dr` it, compute the module's `.text` runtime base from any +`/proc/kallsyms` symbol, and every guest-kernel function/offset becomes +addressable for live gdb breakpoints on the QEMU stub. With it: + +* `rpcAllocObject_v29_06`/`_rpcAllocObjectPrologue` are **exonerated** for + 0xc57e (their reject list is 0x73/0xc67*/0xc770/0xc77d only; the register + mapping `hClass = r8d` was verified live). +* The failing chain is pinned exactly: `rmapiAllocWithSecInfo` **enters** + with class 0xc57e, calls `serverAllocResourceUnderLock`, and that + function returns **0x56** - *before* the GSP-RM alloc RPC + (`rpcRmApiAlloc_GSP`, called from under-lock+0x447; the emulation never + sees an ALLOC for 0xc57e). The same probe shows 0xc57d/0xc570/0xc573 + return 0. +* So the last blocker for the window path is a **local RS-layer validation + inside `serverAllocResourceUnderLock`** (candidates: the inlined + privilege/class-support checks, `rmapiFixupAllocParams`' per-class fixup, + or `clientAllocResource`'s construct dispatch). The 0x56-producing + instruction is not yet pinned - a `stepi` loop over the QEMU stub is far + too slow and wedged the guest (recovered by a fresh cycle); the next + session should use conditional breakpoints on the function's call-return + sites (the approach in `gdbdiag19` works; the earlier run had truncated + addresses and needs the `0xffffffff00000000` base restored) or a + watchpoint on the status word. + +**gdb-stub field notes**: the stub is single-client; killing an attached +gdb leaves the VM stopped (recover with a detach-only connect); +`stepi` loops over the stub freeze the guest's userspace for minutes - +prefer breakpoints; every probe run with the caps-fill breakpoint changes +the driver's path, so keep the fill in each experiment. + +## 8. UPDATE 2026-10-25 #3 (SESSION 126): the fence fold landed - the service authors the Vulkan fence releases from the driver's own push bytes; the compute-dispatch shape decoded (direct C5C0 method stream, no inline QMD) + +**The fence wall is closed.** The client's per-frame fence wait polls a +16 KB BO word that a submit-push INC-5 semaphore group targets +(`[2005c017][0x45ba090][0][payload 1][0][0x02100001]`). The group sits in a +GPFIFO entry beyond the driver's kick boundary (GP_PUT=11 while the ring +already carries the entries it pre-built for the next submits, idx12..46), +so the GP_PUT-bounded pump walk never decoded it and a host-side assist had +to write the word. The service now sweeps that pre-built continuation +itself: `sema_gr_ahead` (default 64) walks entries beyond GP_PUT through the +ring's validated translation, accepts only pushes that pass the strict +method-stream gate, extracts the addr-0x5c RELEASE groups and applies them +forward-only through the standard pending ledger. Live evidence: the sweep +finds the groups and the ledger applies the fence word once per frame; +`VKFRAME seq=1..3/DONE` completes with no assist. The frames are still +black - the shader work itself is unexecuted. + +**The compute-dispatch shape (why nothing executes).** The workload is a +single compute dispatch (640x480 storage image). Its push is a DIRECT +method-stream programming of the Turing compute class: `SET_OBJECT 0xC5C0` +on subchannel 1, then the context windows (`0x7b4` local-memory window B, +`0x2a4` shared-memory window B, `0x270`), an immediate-config block, and +the code/const data runs (`0x3800`/`0x39C8` pairs plus a subchannel-0 data +port stream of `0x11c`/`0x114` runs in a separate 17 KB push). There is NO +inline-QMD write (no `0x318/0x31c/0x320..0x41c` run), so the launch capture +that feeds the physical-engine forward path never fires - and the CE-shaped +ops path is not even consulted for GR-class channels by construction. Real +pixels therefore require a translator that recognizes the C5C0 section, +builds the vendor launch the physical channel expects (the same prelude/QMD +shape the CUDA path already executes) from the section's fields and data +runs, and forwards it with the existing operand/relocation machinery. The +method vocabulary overlaps the physical prelude's existing knowledge +(`0x7b0/0x7b4`, `0x2a4`), so the landmarks for that translator are known. + +## 9. UPDATE 2026-10-26 (SESSION 130): the utility-channel LOAD_INLINE_DATA arm fires; the payloads are the driver's zero buffer; the SASS delivery is the remaining non-black gate + +**The arm was dead by branch order.** The utility channel's engine-side +uploads ride subchannel 2 (the A140/KEPLER_INLINE_TO_MEMORY_B lane), and +their operand group (`0x188/0x18c` destination, `0x180` line length, +`0x184` line count) was parsed only by an `else if (subch == 1 || subch == +2)` branch that could never run: the preceding `if (xlate && subch >= 2)` +TURING-copy parser also matches subchannel 2 and swallowed the group +unread. With `have_out`/`line_len` unset, the `0x1b4 +LOAD_INLINE_DATA` arm never opened (`saw=0`). The 0x180..0x18c group is now +parsed first for subchannel 1|2; the arm reaches its write decision and +authors (`writes=2` on r744: 256 B -> vidmem 0x1e0000, 12288 B -> vidmem +0x1c0000). + +**The payloads are stably zero - and that is the driver's zero buffer.** +Four independent reads (the sweep's channel path, the direct +registered-VAS walk - 22k attempts, never non-zero - the guest's own +`/proc/<pid>/mem` view of the client's mapping, and the guest's +`/proc/kcore` physical read) plus a whole-run 1 MB staging-BO poller and a +gdb hardware write-watchpoint that never fires agree: the driver never +writes the payload chunks for these entries. The 256 B size equals +`GF100_ZERO_BUFFER_SIZE`, and the driver's `gf100DownloadZeroBuffer()` +uploads exactly 256 B of zeros through this path. The arm therefore +applies zero payloads under a variant once-key (`h | 1`) - idempotent over +an empty destination - while a later non-zero payload for the same entry +still lands under its own key. + +**The non-black gate, precisely.** The launch's `prog=0xdfd8feb900` +resolves to vidmem `0x62b000`, which is empty; the compute shader SASS is +not delivered by any decoded entry. The utility channel's ring holds +exactly 14 entries (`GP_PUT=14`), of which two carry LOAD_INLINE_DATA - +both the zero objects above. The Vulkan (NVN) download path uploads via a +separate download state and points its destination view at the pitch-pool +allocation, so the SASS/constants i2m shape is a different destination +family than the observed zero-buffer uploads. Next steps: find the SASS +upload shape (pitch-pool destination, possibly on another channel or a +non-LID copy) and then the `n_data` descriptor/operand wiring +(section 8); with both, the physical forward can execute the dispatch. + +## 10. UPDATE 2026-10-26 (SESSION 131): the utility-ring translation is the live problem - the decoded LID entries are from a STALE ring page; the live ring is in vidmem + +**The ring is in vidmem, and its content varies per client.** The +utility channel's ring VA (0x4020000) resolves through the service's +learned map to a **vidmem** offset (PA 0xa0000, next to the channel's +USERD at 0x8fe00), and the pump's own entry read shows the live ring's +content: on the r750-era clients `entry[0] va=0x120100000 len=0x28` +(`sema-chan: entry[0] h=0xbeef0100 eng=0 raw=0000280120100000 +va=0x120100000 len=40`). A dump of the vidmem ring (dump_vidmem 0xa0000) +confirms exactly one entry there. + +**The decoded LID entries (the 14-entry ring at 0x4000000/0x4250000) are +from a STALE page.** The sweep's ring read (`svc_read_va` of the ring VA) +can resolve a **guest-RAM** page (e.g. PA 0x13003d000) that holds an +EARLIER client's ring - a coherent 14-entry ring whose LID uploads target +the Vulkan client's own allocations (dst 0x423f000 -> vidmem 0x1e0000, +dst 0x4448000 -> vidmem 0x1c0000, 256 B = GF100_ZERO_BUFFER_SIZE and +12288 B). Those entries are real, but they belong to a dead session: the +payload chunks they point at (VA 0x4250000/0x4250110 -> the session's +1 MB staging BO) read zero through EVERY path - the sweep's channel read, +the direct registered-VAS walk, the learned map, the guest's own +`/proc/<pid>/mem` view of the BO, `/proc/kcore`, a whole-run +high-frequency poller of every 1 MB BO the client maps, and a gdb +hardware write-watchpoint that never fires - because the session ended +and the BO's pages were freed/reused. **The payload is not missing from a +live upload; it is a stale-session artifact.** + +**The CUDA contrast proves the mechanism.** A CUDA client's LID rows on +the same boot show the arm working with real payloads: +`h=0x5c000019 i=2 va=0x2004002c0 len=0xb8 ... tail_va=0x200400378 +tail=0x1c t8=200125a6 00000001 ... saw=1 fired=1 zero=0 +dst=0xd2e00000` - the tail (the next GPFIFO entry) is CONTIGUOUS with the +push and holds a real method stream, the arm materializes it into the +high vidmem family, and the CUDA battery passes. + +**Next steps (ranked).** +1. Read the LIVE client's ring through the map (vidmem 0xa0000) during a + client run and decode its entries - that is the live upload path; the + sweep's `svc_read_va` source preference must not answer with a stale + RAM page (verify the ring page by content: the entries must lie in the + client's current VA allocations, and the ring must be the same page the + pump reads). +2. With the live ring decoded, the SASS upload shape follows; the + descriptor/operand wiring (`n_data=0`) remains the second gate. +3. `sema_util_dec_log` (96/boot) and `svc_gr_util_log` (24/boot) budgets + were exhausted early in multi-client boots, hiding later rows; make + them per-session for diagnosis. + +## 11. UPDATE 2026-10-26 (SESSION 131b): the A/B that names the frontier - CUDA's uploads land in the emulated vidmem, the Vulkan client's do not + +`dump_vidmem map` (new: one line per 1 MB block with non-zero content, +`nz` = non-zero dwords) on a fresh boot, same module: + +* **CUDA client (`victory_mail`, STATUS OK)** - the shadow fills up: + `blk 0x200000 nz=40811`, `blk 0x1400000 nz=178868`, + `blk 0x1500000 nz=96346`, `blk 0x2a00000 nz=22880`, + `blk 0x7800000 nz=98257`, ... (~450k dwords total). The CUDA driver's + uploads (LID i2m writes with real payloads + CE copies) materialize in + the emulated vidmem. +* **Vulkan client (`vk_frame120`)** - the shadow stays at the 14 + baseline dwords (the fence/counter words at 0xfee88, the GR param bank + at 0x385e08, one word at 0x4d0b10). **Nothing the Vulkan client uploads + reaches the emulated vidmem at all.** + +That is the frontier in one measurement: the Vulkan (NVN/GL) driver's +upload path is not among the writes the emulation can see. Combined with +section 10 (the utility ring lives in vidmem via the learned map but the +sweep can read a stale RAM page for it; the live ring holds a single +`0x120100000/40` entry, not the 14-entry LID ring), the next session must: +(1) capture the LIVE ring through the same source the pump uses while the +client is up and decode its entries (the 14-entry LID ring is a +stale-session page); (2) trace where the live client's upload writes go +(the map shows they are NOT in the first 256 MB of the shadow - check the +guest's own view of the client's staging BOs during the upload window); +(3) the descriptor/operand wiring (`n_data=0`) after that. + +## 12. UPDATE 2026-10-26 (SESSION 132): THE LIVE RING IS PROVEN (guest RAM, walk-confirmed); the vidmem "ring" was a false-positive pin whose dead entry the pump skipped 39 times; the two LID uploads are the driver's zero buffers by design; the code VA's mapping rotates and the shadow's lat cache can go stale + +**The live ring is in guest RAM and the sweep's decode was right.** +`walk_probe 0x4020000` (the client's own page tables, vas[0] +`h=0xfade0002 pdb=0x114039000`) resolves the utility channel's GPFIFO VA +to **guest RAM** (PA `0x11623d000` on r759; the page moves per run: +`0x104b55000` on r761) and its content is the 14-entry ring the sweep +decodes: entry0 -> push `0x4000000` len `0x8bc` (the methods), entry1 -> +the 256 B payload slot `0x4250000`, entry8 -> the 12288 B payload slot +`0x4250110`, ... - all pushes inside THIS client's own allocation window +(`PREV149-OUT ... off=0x4000000/0x4020000/0x4060000` in the client's own +VIDHEAP log). The decoded ring is live content, not a stale page. + +**The vidmem "ring" (PA 0xa0000) is a false-positive autopin.** On r759 +the VRAM shape scan found the channel's REAL USERD at vidmem `0xb0000`; +the userd-derived candidate loop then tried `0xb0000 - 0x10000 = +0xa0000`, whose entry0 (`push 0x120100000/40` - the CE-pool push +pattern) passed the plausibility gate and whose USERD probe hit the real +0xb0000 block, so the pin landed. The pinned entry's push reads all-zero +through every source; the pump retried 39 times and logged `skipping +stale entry h=0xbeef0100 va=0x120100000 len=40 after 39 failed rescues`, +while `sema_probe 0x4020000` then answered the wrong page +(`pa=0xa0000 aperture=vidmem`) and poisoned every later read (the sweep +included). Section 10's "stale session page" conclusion was this pin, not +the sweep. + +**Fixes landed (commit `72de0bed`).** (1) `svc_ring_entry_dead` + +`svc_ring_walk_live`: a candidate whose entry0 push provably reads +all-zero through the client's page tables is rejected when the VAS walk +resolves the ring VA to a page with a live entry (applied at the +userd-derived, FB-convention and existing-pin paths; the walk cannot vote +for itself); (2) `svc_gr_ring_candidate` rejects dead-entry candidates +and accepts the channel's known USERD cross-aperture (the driver places +GPFIFO and USERD independently: ring in guest RAM, USERD in vidmem); (3) +the util sweep arbitrates a dead channel-path ring entry against the +direct VAS walk per slot; (4) the util-dec/util-sweep witness budgets and +their dedup keys are per-client now (reset in `svc_session_flush`); (5) +`sema-copyent` logs once per copy OP, not per row - one 480-row frame +copy had exhausted the 128/session budget and hidden every later copy. +Verified on r760/r761: `sema-chan: h=0xbeef0100 ... ring rc=0 +pa=0x108cba000 ap=1` (the live RAM page), `sema-rdw: h=0xbeef0100 +va=0x4000000 len=2236 rc=0`, the sweep decodes the live 14 entries, the +fence fold releases fire, and the client completes (`VKFRAME_DONE +frames=3 640x480 saved=3`, `VKFRAME_IDLE ok`). + +**The two utility-channel LID uploads are the driver's zero buffers by +design, not the SASS.** Live decode (r760): i=0 -> `dst=0x423f000 ll=256 +lc=1 zero=1` (256 B = `GF100_ZERO_BUFFER_SIZE`) and i=8 -> +`dst=0x4448000 ll=12288 lc=1 zero=1` (12288 B = the CE fault method +buffer size). The payload slots (`0x4250000` -> RAM `0x116219000`; +`0x4250110`) read zero for the whole client run - the driver never +writes them. The GR channel's own LID entry DOES carry a real payload +(`dst=0x4b86020 ll=1568 zero=0` - the launch param bank at `param= +0x4b86000 size=0x1900`), which proves the arm works with real bytes on +this boot too. + +**Where the code page stands.** `prog=0xdfd8feb900` (the captured +launch): `sema_probe` answered vidmem `0x62b900` early in the run while +`walk_probe` answered vidmem `0xc8b000` later - the driver's code-buffer +mapping ROTATES (a code pool), and the shadow's learned lat entry for the +VA can go stale. Both pages read zero in the shadow: the driver's code +upload has not been observed on any decoded path (no LID, no CE copy - +the only `sema-copyent` op of the whole run is the 480-row frame +copy-out `dst=0x50b0000 src=0xdfd8d60000`, which executes `rc=0 +bytes=0x12c000` and correctly copies the still-empty image). The +remaining question is the SASS delivery shape for the C5C0 compute path +(section 6's `0x3800/0x39C8` code-run/data-port semantics) and the stale +lat preference for the rotating code VA. + +**Next steps (ranked).** +1. Resolve the code VA through the VAS walk at launch time (the walk is + the live truth; the lat entry goes stale as the pool rotates) and + watch the code page the QMD actually names. +2. Find the SASS delivery: dump the C5C0 section's `0x3800/0x39C8` runs + with a bounded methods witness while the client is live (the params + arrive by LID; the code must arrive by a sibling path). +3. Then the `n_data=0` descriptor/operand wiring (section 8). + +## 13. UPDATE 2026-10-26 (SESSION 133): THE SASS DELIVERY SHAPE IS FOUND - the driver embeds the program bytes INLINE in a CE LAUNCH_DMA push and copies them to vidmem; the CE channel's ring/USERD resolution is the boot-lottery gate that keeps the copies from running + +**The delivery shape (decoded from the CE channel).** A CE-class channel +(eng=11) carries, as entry[4] of its ring, a push whose method stream is a +LAUNCH_DMA copy with the SASS bytes INLINE: + +``` +sema-chan: entry[4] h=0xcaf00012 eng=11 raw=0021340120a00000 va=0x120a00000 len=8500 +sema-bigrun: h=0xcaf00012 push=0x120a00000 off=0x0 hdr=0x64000002 op=3 cnt=1024 subch=0 addr=0x8 +sema-lda: h=0xcaf00012 eng=11 push=0x120a00000 dma=0x00002181 xfer=1 dstphys=1 srcphys=0 out=0x205000 in=0x120a00004 line=4096 have_out=1 have_in=1 +sema-opadd: h=0xcaf00012 type=1 dst=0x205000 src=0x120a00004 bytes=4096 ap=0 src_ap=0 phys=1 src_phys=0 +``` + +The source is the push itself (`in=0x120a00004`, right after the method +header) and the destination is a vidmem VA (`out=0x205000`, then +`0x206000`, ... 4096 B per run) - the engine-side "download program +range" path. The Vulkan client's CE channels carry the SAME shaped +entries (r763: `sema-utildec: h=0xbeef0101 i=4 va=0x120a00000 len=0x2134`, +19 entries swept) - so the SASS delivery is a CE inline copy, not the +utility-channel LID entries (those are the driver's zero buffers, +section 12). + +**Why the copies never ran (three gates).** (1) The CE channel's ring and +USERD live in VIDMEM and resolve only by boot-lottery: the FB convention +matched on r763 (`auto-pinned ring h=0xbeef0101 rank=0 va=0x4100000 -> +fb 0x80000 (entry0 va=0x120000000 len=40)`; entries then decoded and the +frame path ran) but read zeros on r765/r767/r768 - the USERD shape scan +instead finds the channel blocks at 0xfee00/0xff000/0xff200 (r767/r768), +so the rings sit at unaligned offsets like 0xefe00. (2) The CE push VAs +(0x120000000+) need the client's VAS registered: the parked +COPY_SERVER_PDES candidate only registered at +15 s (age-out, +`sema_vas_cand_ms`) while the entry hold grace (`sema_ce_grace_ms`, +8000 ms from the cycle) expired first - r763 logged `skipping stale +entry h=0xbeef0101 va=0x120000000 len=40 after 8000 ms grace` and +`sema-pushfail: va=0x120a00000 ... trusted_rc=-2 ipin=0` for the copy +pushes. Setting `sema_vas_cand_ms=5000` registers the VAS at +5 s +(verified: `VAS 0xfade0002 ... registered UNVERIFIED after 5000 ms +parked`) but the ring gate still blocks. (3) The shadow ring scan +fallback (commits `52ef3f9d` + `2735180a`: 0x100 stepping, rate-limited +retry, entry0 shape + USERD probe at +0x10000/+0x2000/+0x1000) found no +candidate on r768: the ring/USERD content is volatile at scan instants +(a `find_userd` read shows `off=0xfee00 get=11 put=11` while a +`dump_vidmem 0xfee00` one second later reads zero), so a periodic scan +misses the window. + +**Next steps (ranked).** 1. Derive the CE ring from a USERD found by its +own shape (put at +0x8c, get at +0x88 - the `find_userd` scan already +does this) instead of scanning for ring entries: for every USERD-shaped +block, try ring = userd - {0x10000, 0x2000, 0x1000} and validate entry0 ++ the probe. 2. Or latch the ring at the first CE submission (the ring +content appears exactly when the driver writes entry0). 3. With the ring +latched and the VAS at +5 s, verify the inline copies execute +(`sema-lda`/`sema-opadd`/`sema-copyent` with the code dsts) and the code +page gets content; then the `n_data` descriptor wiring (section 8). + +## 14. UPDATE 2026-10-26 (SESSION 134/135): the CE first-submission capture landed and its rank-blind first-hit caused a CUDA-client stall regression (fixed); the Vulkan client's CE rings are allocated ABOVE the capture scan's 64 MB bound and the frame chain reduces to that un-latched ring + +**SESSION 134 (commits `36642718`, `928c0730`).** A new instrument fires in +the notifier-completion path for CE channels *before* the clear write - the +one instant the ring content and the USERD are both live - reads entry0 +through the channel translation and the trusted walk, then scans the shadow +(0x100 step, first 64 MB) for a ring-shaped page (entry0 naming a 1 MB +aligned push VA in `[0x120000000, 0x121000000)` with a sane length) and runs +the standard USERD gate at `+0x10000/+0x2000/+0x1000/+0x0`; a validating +candidate is pinned/latched exactly like the regular CE autopin. `928c0730` +fixed the entry decode: the push VA is 40 bits, the scan had declared it +`u32`, so `scan_hits` was zero by construction. + +**SESSION 135 (commit `a674c21c`) - the rank-blind first-hit regression.** +The scan started at offset 0 and accepted the first ring-shaped page for +*every* CE channel. On r779 both `caf00012` (rank 0) and `caf0001d` +(rank 1) latched `off=0x80000 upa=0x90000`: rank 1 read rank 0's +ring/page, its first release decoded to the wrong semaphore word (+0 +instead of +4) and was dropped as stale (`cur=1`, already written by +rank 0), so the client's rank-1 tracker wait never completed. victory_mail +then spun at 100% CPU with `pub_calls=0` on three consecutive boots +(r777/r778/r779: battery pass=0 fail=7). A control run of the r746-era +module (`25523f84`) on a fresh boot passed the same client (VADD +`bad=0 checksum=12204516.000000`, `STATUS OK`), isolating this to the +SESSION 133/134 code. The fix tries the rank's convention offset +(`0x80000 + 0x20000 * rank`) first, scans the rest bounded as before, and +never accepts an offset another CE rank already latched this session; an +unlatched rank stays for the regular per-pass latch. Verified on r781: +ranks 0..5 latched correctly (0x80000/0xa0000/0xc0000/0xe0000/0x100000/ +0x120000, USERD +0x10000 each), rank 1's own semaphore word applied +(`pool-wr va=0x121000004 cur=0 -> val=1 ok`), and the client progressed +(`pub_calls=185132+`, no spin). + +**The Vulkan CE ring on the same boots.** r781 (classic autopin, rank 0): +`auto-pinned ring h=0xbeef0101 rank=0 va=0x4100000 -> fb 0x80000 (entry0 +va=0x120000000 len=40)` - a live entry - but the client's pushes decoded +`unread=1 injected=0`: the guest pin scanner covered CUDA clients +(`cdt,va,victory`) and never `vk_frame`. Extending the scanner to `vk` +engaged the tools (demand pins active) but r782/r783's captures reported +`scan_hits=0 latch=0`: no ring-shaped page anywhere in the shadow's first +64 MB, and no classic autopin either. + +**Where the Vulkan rings actually are.** The client's own vidheap ioctl +log (`PREV149-OUT` lines, r783) names the CE/GR channel allocations: +`off=0x4020000/0x4100000/0x4160000` - matching the registered gpfifos +(`0x4020000/0x4100000/0x4160000`) and lying just ABOVE the capture's +`lim = min(shadow, 0x4000000)` (64 MB). Direct dumps of those offsets (and +of `0x80000`) read zero, and a byte search of the full 4 GB guest-RAM image +for the entry pattern (`00 00 00 20 01 28 00 00` / the `0x20100000` +sibling) finds nothing - the ring content reaches neither the shadow at its +allocation offset nor guest RAM. + +**The frame chain (r783, live).** The Vulkan client completes three frames +(`VKFRAME_DONE frames=3`, `VKFRAME_IDLE ok`) but all PPMs are pure black. +The live GR channel (`h=0xbeec0100`) shows the dependency explicitly: +`sema-gr: launch captured ... prog=0xdff0feb900 ... inline=8` together +with `sema-gr: copy pend h=0xbeec0100 push=0x4f4580c src=0xdff0d60000 +dst=0x50b0000 row=0xa00 rows=480 ...` repeated ~20/s. The trusted walk +maps the copy source to VIDMEM (`walk_probe 0xdff0d60000 -> rc=0 +pa=0x570000 aperture=vidmem`) and the destination `0x50b0000` is the +framebuffer. The shadow at `0x570000` reads zero: the data pages are the +ones the CE inline copies are supposed to fill, and the GR launch is held +on their absence (the designed `dstpend` dependency). So the whole Vulkan +chain reduces once more to the un-latched CE ring: no ring -> no inline +i2m copies -> no data/code pages -> held launches -> black frame. + +**Next steps (ranked).** 1. Cover the client's real allocation in the ring +capture: raise the capture scan bound above the recorded channel +allocations (e.g. `0x6000000`) and/or seed candidates from the client's +vidheap offsets, then re-check whether the entry arrives at the +submission instant (a live `sema_vidmem_dump` at `0x4100000` while the +client is streaming would settle it; the post-run dumps read zero). 2. If +the shadow still reads zero at the allocation offset during live +submissions, the client-side write watch is the remaining forensic: find +the CPU pointer the driver writes GPFIFO entries through, its PA via the +guest pagemap, and why the GPU PTE/walk does not name that PA. 3. With a +latched ring, verify the inline copies execute (`sema-lda`/`sema-opadd` +with the code dsts), the code/data pages fill, then the `n_data` +descriptor wiring (section 8) → non-black frames. 4. Re-run the CUDA +battery on the fixed module (the r781 boot was a slow-crawl boot; the +spin stall class is gone) and revisit `sema_gr_xlate`'s multikernel +forward timeout. + +## 15. UPDATE 2026-10-26 (SESSION 135b): every "Vulkan CE ring latch" on record was a STALE CUDA BLOCK - the Vulkan CE ring content has never been observed; the write path is not BAR1 and the content is in no shadow, no RAM, and not at the PTE target + +**Forensics on four fresh boots (r782-r786; commits `a674c21c` in tree, `c3cdcf4b` new).** + +*The write path works; the driver does not use it for the ring.* A direct +BAR1 test - the guest writing a known pattern at `BAR1+0x4100000` through +`/sys/bus/pci/devices/0000:01:00.0/resource1` - lands in the emulated +vidmem shadow verbatim (`RINGTEST` + `AAAA...` read back by both the guest +and `sema_vidmem_dump`), so the mdev's BAR1 region path (mmapped, direct +into `bars[1].vaddr`) is sound at those offsets and at `0x80000`. A +high-frequency watcher (12 candidate offsets, ~2.5 Hz, including +`0x80000/0xa0000/.../0x4020000/0x4100000/0x4160000` and the +0x10000 +USERDs) recorded ZERO nonzero dumps across the whole live streaming window +of a clean Vulkan run (r786; three frames, `VKFRAME_RATE fps=0.05`, +`VKFRAME_DONE frames=3`, black PPMs). The `shadow_scan` over all three BAR +shadows (0/1/3) for the CE push header `{0x120,0xb5c50000,...}` and the +ring-entry pair `{0x20000000,0x2801}` finds nothing, and a byte search of +the full 4 GB guest-RAM image for both patterns (little-endian, corrected +from an earlier reversed search) finds nothing either. The walk and the +learned map resolve the CE ring VA (`0x4100000`) to a guest-RAM page +(per-boot PA, e.g. `0x135726000`) that reads all zero, and the vidheap +ioctl log (`PREV149-OUT`) names the channel allocations +`0x4020000/0x4100000/0x4160000` on every boot - so the ring content is +written neither to the shadow at its allocation offsets nor anywhere in +RAM reachable through the guest's own page tables. + +*A stale-block trap explains every historical "latch".* On CUDA-first +boots the capture latched `fb 0x80000 / userd 0x90000` for the Vulkan +channel `beef0101` with `put=1 get=1` - the dead CUDA client's `caf00012` +block surviving in VRAM after its session. The decoded entry +(`va=0x120000000 len=40`) is the old session's (both clients' first CE +pushes are shape-identical), the expectations seed from the stale pool +words (`want=2/3` vs `pay=1`; the word page read 3-6), and the Vulkan +client stalls into its fence timeout (`VKFAIL vkWaitForFences -> 2`) on +every run. r763's and r781's "auto-pinned ring h=0xbeef0101 -> fb 0x80000" +rows are the same artefact: both were CUDA-first boots. All genuine +first-submission latches on record read `put=1 get=0` at the capture +instant. `c3cdcf4b` rejects a consumed-state candidate (`get == put > 0`) +in the first-submission capture - a just-submitted entry cannot be +consumed at the notifier-clear instant. + +*The frontier is now precise.* The Vulkan client's CE submissions complete +only through the fabricated completion machinery (the driver never reads +its ring back), its CE ring content is invisible to every host source, and +the GR launch for each frame is held on the unexecuted CE i2m copies +(`copy pend src=0xdff0d60000 dst=0x50b0000`; walk: src -> vidmem +`0x570000`, zero). Next: the client-side write watch - find the CPU +pointer the guest driver writes GPFIFO entries through, its PA via the +guest pagemap, and why neither the GPU page table target nor any shadow +holds the result (guest-kernel gdb via the QEMU gdbstub or a kprobe on +the driver's ring-write helper). + +## 16. UPDATE 2026-10-26 (SESSION 137): THE CE-RING HYPOTHESIS FOR THE VULKAN CLIENT IS DISPROVEN BY DIRECT WRITE WATCHES - no kernel and no client writer ever touches the ring page + +**The write-watch toolchain (all three instrument classes).** (1) kprobes: +`channelFillGpFifo` (the kernel's GPFIFO-entry writer, called from the CE +and SEC2 submission paths per the open driver sources) is marked +`notrace` - `trace_kprobe: Could not probe notrace function +channelFillGpFifo [nvidia]` - so kprobes and ftrace can never attach +(this is why the named helpers "never fired" in earlier sessions: they +could never register). (2) QEMU-gdbstub hardware breakpoints work: a +verification `hbreak *__x64_sys_ioctl` fired on guest ioctls +(`IOCTL#1 pc=0xffffffffbc316160`). An `hbreak` on the exact entry store +(`channelFillGpFifo+0xb5`, `mov %edi,(%rdx)`, where `%rdx = +pbCpuVA + channelPbSize + lastSubmittedEntry*8` per the disassembly and +the source) stayed silent across a complete Vulkan run - even though that +same run produced the CE channel's notifier clear +(`sema-chan: notifier cleared h=0xbeef0101 info32=0x5`) and the +first-submission capture (`sema-cefirst: h=0xbeef0101 rank=1 ... +scan_hits=0 latch=0`). (3) A client-side hardware watchpoint: the ring +VA resolves through the guest page tables to a guest-RAM page that is +mapped into the client process (`/dev/nvidiactl` mapping; e.g. PA +`0x101021000` -> client VA `0x7fffee69b000`; a later run PA +`0x10087b000` -> VA `0x7fffee7d8000`), and the walk and the emulation's +own read agree the page is ZERO. An in-client gdb (the t122 recipe plus +a file-fed watchpoint) armed `watch *(unsigned long*)0x7fffee7d8000` +from the client's FIRST ioctl (`Hardware watchpoint 2` + `RGWATCH-SET` +logged before `GLBASE`) caught ZERO writes across the whole run +(3 frames, `VKFRAME_DONE`, IDLE). + +**Consequences.** On vk-first boots the client creates its CE channel but +never submits CE work: no GPFIFO entry is written by the kernel +(`channelFillGpFifo` never executes), nor by the client (hardware watch, +zero hits), nor is any content found in any BAR shadow or in RAM. The +historical "Vulkan CE ring latches" were the dead CUDA client's stale +blocks (section 15). The image/upload path for the Vulkan client rides +its GR channels (`launch captured ... inline=8`; the frame copy held as +`sema-gr: copy pend ... src=0xdff0d60000 dst=0x50b0000 rows=480`), and +the non-black gate must be attacked on that path - the data/code pages +those launches depend on are never filled because no CE i2m copies are +ever queued, and the launch is held on their absence (dstpend/n_data). +Next: instrument the GR launch's dependency resolution (which pages the +dstpend gate waits for, and whether the inline records the capture holds +are sufficient to fill them) instead of hunting a CE ring that nothing +writes. + +## 17. UPDATE 2026-10-26 (SESSION 138): THE GR LAUNCH'S WIRING IS PRINTED - one QMD-window page carries prog+pointer data (deduped and never scanned), its only fill candidate is an unresolvable inline word, and the backend refuses `n_data=0 (min 16)` + +**The witness (commit `10b09217`, sema_dbg-gated).** Per launch, bounded: +the candidate handle, `param_va`, `prog_va`, `n_pointer` + QMD pointer +pairs, inline count, and the resolved `gr_pages` list; then the fill's +built candidate set with the arg flag. + +**First capture (r788, Vulkan `h=0xbeec0100`, push=0x4f4509c).** +``` +sema-gr: cand h=0xbeec0100 param=0x4b86000 prog=0xdff0feb900 n_ptr=1 inline=8 n_pages=2 push=0x4f4509c +sema-gr: ptr[0] va=0xdff0febc00 word=34 +sema-gr: page[0] va=0x4b86000 pa=0x386000 ap=2 qmd_word=32 +sema-gr: page[1] va=0xdff0feb000 pa=0x56b000 ap=2 qmd_word=48 +sema-gr: fill cand n=1 pages=2 push=0x4f4509c +sema-gr: win[0] va=0x43f00000 arg=1 +``` +Reading: the launch's page list is the param page and the +prog/QMD-window page (both resolving as `ap=2`, i.e. vidmem). +`ptr[0]` (QMD word 34, `0xdff0febc00`) shares its 4K page with the program +page, so `svc_gr_page_note` dedupes it into the `qmd_word=48` entry - and +the fill's scan passes cover only `qmd_word==32` and `32..47 even`, so +that page's content is never scanned for data pointers. The fill's single +candidate comes from the inline bytes instead (`0x43f00000`, arg=1) and +resolves in **no** vaspace (`svc-gr arg-miss base=0x43f00000 probe=none +... vas[0..3] rc=-2`), so `n_data` stays 0 and the backend refuses: +`vgsp-gr: GR submit refused: degenerate data set n_data=0 (min 16) +launch_va=0x4f4509c`. + +**Content state.** The shadow at the param page (`0x386000`), the arg +region (`0x386150`), the program area (`0x56b900`) and the pointer/data +area (`0x56bc00/0x56bd00`) reads all zero on this boot - the launch's +data and code bytes exist in no host-visible source (consistent with +section 16: nothing queues the CE i2m copies for the Vulkan client). + +**Next.** (1) Scan the deduped QMD-window page (or scan a page once under +every qmd_word that names it) so the pointer-region data words enter the +candidate set; (2) resolve the inline/dst candidates through the vidmem +serving lane (the pages ARE resolved to vidmem PAs - the missing piece is +the candidate list, not the resolver); (3) drive `n_data` toward the +launch's `min 16` (the window's expected page count) - and where the +window content must come from is the open question the byte-state above +poses. + +## 18. UPDATE 2026-10-26 (SESSION 139): THE FILL'S CANDIDATE SOURCE IS DISPROVEN - `0x43f00000` IS THE FLOAT 480.0f; after the float-shape filter the candidate set is EMPTY and the wall is content, not wiring + +**The false candidate (commit `add22329`).** A full dump of the launch's +param page (vidmem `0x386000`, r789) holds exactly two nonzero spots: +`+0x30: 08 00 00 00` and `+0x630: cd cc 4c 3e | 00 00 20 44 | 00 00 f0 +43 | 00 00 00 00` - i.e. the f32 scalars **0.2f, 640.0f, 480.0f** (the +640x480 viewport!). `0x43f00000` is 480.0f: the fill's only candidate +("arg base=0x43f00000 resolved 0 pages") has been a **float scalar +misread as a device VA** by the scan's shape filter (`>= 0x10000000`, +4-aligned). The GRCE inline write that does fire (`dst=0x4b86020 -> +pa=0x386020 bytes=1568 data8=0000000000000000`) writes the driver's +zero stub (the push carries only an 8-byte nonzero inline fragment at +`+0xb0`; the rest reads zero - the SESSION 130 zero-buffer finding). + +**The fix and verification.** All four scan sites (the three page scans +plus `svc_gr_scan_cand`) now reject float-shaped words (f32 exponent byte +in ~`[0x6c..0x8f]` = 2^-19..2^16; device VAs as f32 land in subnormal +exponents), and the bank-page scan covers word 48 (the deduped program +page). r790: **`sema-gr: fill cand n=0`** - no false candidates, no false +demand-loop traffic; the launch still refuses (`GR submit refused: +degenerate data set n_data=0 (min 16)`). + +**The wall, precisely.** The launch's real data pointers are not in any +scanned source: the param page holds scalars; the QMD pointer word +(`0xdff0febc00`) is one window buffer whose content is never written; the +inline payload is the zero stub. The content could only arrive via the +driver's i2m/LOAD_INLINE_DATA copies (which the Vulkan client never +queues - SESSION 137) or from the real GPU (physical forward, which the +n_data gate refuses first). This is now a pure content-source problem: +either find/queue the i2m that fills the QMD window, or break the +chicken-and-egg by forwarding a launch with the window zero (letting the +real engine read it) behind an explicit parameter. + +## 19. UPDATE 2026-10-26 (SESSION 140): THE n_data WIRING - the QMD's own pointer values ARE the launch's data set; the fill now reaches them (candidates appear, the page is correctly launch-internal), and the degenerate-set guard's bypass reaches the backend only to oops in the empty-set walk + +**Three wiring corrections (commit `5c0fcc21`).** (1) The QMD page itself +is noted into the request page list (word 34) - the pointer values live +inside the QMD (r790: word 34 = `0xdff0febc00`) and no scan ever covered +it. (2) The fill seeds candidates directly from the launch's own QMD +pointer pairs (`gr_qmd` words 34..47, decoded from the push's inline-QMD +run): r793 `sema-gr: qmd-ptr seed va=0xdff0feb000 (word 34)` - the values +need no page scan (the page shadows read empty; the +invisible-vidmem-write class). (3) The scans read pin-aware +(`svc_read_va` first) - the staging copy the decode reads is a pinned +page; a raw mapping of the vidmem pa sees zeros. + +**Result.** `fill cand n=1` with the QMD window page seeded; +`sema_probe 0xdff0feb000 -> pa=0x56b000 aperture=vidmem` resolves - but +the R3 hygiene filter classifies the page launch-internal, so `n_data` +stays 0: **this launch's QMD declares exactly one pointer pair and it +names its own QMD window - a launch with no external data, which the +"degenerate set" refusal class wrongly covers.** + +**The egg-break test + the oops.** With the r564 guard bypassed +(`gsp_phys_gr_min_data=0` - and the value clamp removed for the test -) +the refusal disappears and the forward ENTERS the backend: +`Oops: 0000 [#2] PREEMPT SMP NOPTI`, `Workqueue: gsp-phys-work +svc_fwd_work_fn [vgsp]`, `RIP: gsp_phys_backend_guest_gr_submit+0x4394` +(NULL-base table walk: CR2=0x1e93c, R14=0, RAX=0x1e000, RSI=0xdff0feb900 +- the program VA). The empty-data walk needs that guard before +`min_data=0` can be used; the clamp revert was withdrawn (crash +enabler). The copy pipeline is unaffected: `sema-gr: copy done +h=0xbeec0100 push=0x4f4580c dst=0x50b0000 bytes=0x12c000 rows=480 rc=0` +runs cleanly in the same runs. + +**Next.** Guard the empty-set walk at `gsp_phys_backend_guest_gr_submit` +(the struct walk reading 0x04/0x10/0x38/0x3c off a NULL base with the +program VA in RSI), then re-test `min_data=0` for the no-external-data +launch class: the forward should reach the real engine, and the copy-out +(and the frame copy that already executes) can carry rendered pixels +back. + +## 20. UPDATE 2026-10-26 (SESSION 143): THE r800 GUARD HOLDS AND THE `min_data=0` BYPASS REACHES THE REAL ENGINE - IT FETCHES THE PROGRAM AND MMU-FAULTS WITH `FAULT_PDE` AT THE PROGRAM VA; THE WALL IS NOW THE PROGRAM PAGE'S MAPPING AND CONTENT, NOT THE DATA-SET GATE + +**The test (r801b).** `gsp_phys_gr_min_data=0` is now an explicit bypass +(commit `34e611f4`; 0 previously clamped to 1). On a clean cycle +(module from `34e611f4`, full params, `sema_gr_xlate=1`, +`sema_gr_util_scan=0`, `sema_vas_cand_ms=5000`, `sema_ce_grace_ms=20000`, +`sema_ce_hold_ms=0`, `sema_ops_log=8`, `sema_dbg=Y`, `min_data=0`) the +Vulkan client ran once: + +* **No oops.** `grep -c Oops` = 0 for the whole run; the r794 + `gsp_phys_backend_guest_gr_submit+0x4394` crash class is gone. +* **The refusal is gone.** Zero `GR submit refused: degenerate data set` + lines; the launch is submitted: + `GR LAUNCH submitted h=0xbeec0100 ... pages=3 shadow=1 + qmd_va=0x4b85f00 slot=3 words=239 ... qmdwin=0x108000`. +* **The engine ran and fetched.** After ~2.5 s: + `GR LAUNCH FAILED: timeout after 2500 ms (qmd_rel=0x00000000 + want=0x600df00d ...)`, and the real firmware reported: + `vgsp-fault: func=0x1006 ... errString="Ch 00000001, intr 00000000. + MMU Fault: ENGINE GRAPHICS GPC2 GPCCLIENT_GCC faulted @ 0xdf_d8feb000. + Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ"`. +* **The copy-out still works:** `sema-gr: copy done h=0xbeec0100 + push=0x4f4580c dst=0x50b0000 bytes=0x12c000 rows=480 rc=0`. +* **The client now fails earlier than the black-frame runs:** it reports + `VKDEVICE ok family=0` then `VKFAIL vkWaitForFences ... -> 2` and no + frame is saved (the service synthesizes a client-visible fault for the + failed forward, where the refusal path used to software-complete and + let three black frames through). + +**The launch wiring measured on this boot.** +``` +page[0] va=0x4b86000 pa=0x386000 ap=2 qmd_word=32 (param) +page[1] va=0xdfd8feb000 pa=0x62b000 ap=2 qmd_word=48 (program) +page[2] va=0x4b85000 pa=0x385000 ap=2 qmd_word=34 (QMD window bank) +``` +The program page resolves in the emulated vidmem at `0x62b000`, whose +backing host page (`0x5e0504000`) reads ZERO; `CODEHASH ... h0=0 ... +SUSPECT (first 16B)` and `FAULT-CAP ... prog_w48=0xd8feb900 +prog_pa=0x5e0504000 code_src=guest-hpa off=0x900 code64=000...` confirm +the source is empty. The QMD deposit keeps the guest program VA +(`w48/49=d8feb900 000000df`); in the retry only the `qmd_word 34` page is +copied to the window-0 shadow (`qmd_word 34 page copied to window-0 +shadow VA=0x1e8000`), so the program pointer is left at the guest VA and +the engine's page tables have no PDE for it. The post-mortem program +target reads `slot=30 ... w0..15 = 0`. + +**Reading.** The data-set gate and the r800 guard are settled: the +no-external-data launch is legitimately forwarded and the engine's +channel/context are alive (it fetched the QMD from the host deposit and +issued a real MMU fault). The remaining wall is exactly two coupled +facts: +1. **Mapping** - the engine's page tables do not map the guest program VA + (`FAULT_PDE`), and the host keeps that VA because the code source is + an empty page (the shadow rewrite is only used for sources it can + trust). +2. **Content** - the program page is empty in every host-visible source. + The driver's program-upload path stages the ucode and copies it to the + code heap (the observed CE `LAUNCH_DMA`/inline shape); that copy has + not been observed executing for the Vulkan client. + +**Next.** (1) Find and execute the Vulkan client's program-upload copy +(the staging->code-heap CE copy) so the program page becomes non-zero - +the utility/GR sweeps are the instruments, and the code page becoming +non-zero is the milestone. (2) Then map/relocate the program VA for the +engine (or install the PDE) so the fetch succeeds. (3) Then non-black +frames -> Xorg/GZDOOM -> KVMFR/LG -> CUDA 15/15 + soak -> the 1.0.0 +video -> Windows. Until (1)/(2) land, keep `min_data` at its default 16: +the bypass forwards a launch the engine cannot complete, which is worse +for the client than the software-completion fallback. + +## 21. UPDATE 2026-10-26 (SESSION 143b): THE COPY-CLASS CHANNELS ARE REGISTERED, SWEPT AND NEVER PUSHED; NO CHANNEL CARRIES PROGRAM BYTES OR A LAUNCH_DMA; THE CODE PAGE NEVER FILLS - AND THE REFUSAL PATH NO LONGER PRODUCES THE BLACK-FRAME BASELINE + +**The instrument (commit `c5510744`).** The util sweep now logs one-shot +`sema-utilsweep: begin|skip|read=none` for every channel plus a 128-row +per-pass cap, so "absent from the log" can no longer be confused with +"absent from the channel table". + +**Channel visibility (r803, full params, `util_scan=64`, `min_data=16`).** +Ten channels in the table, all with gpfifo/entries, **all USERD +unresolved** at first sight: +``` +begin h=0xbeef0101 eng=11 gpfifo=0x4100000 entries=32768 userd=0 +begin h=0xbeef0100 eng=0 gpfifo=0x4020000 entries=32768 userd=0 +begin h=0xbeef0102 eng=12 gpfifo=0x4160000 entries=32768 userd=0 +begin h=0xbeee0101 eng=11 gpfifo=0x4500000 ... +begin h=0xbeee0102 eng=12 gpfifo=0x4560000 ... +begin h=0xbeed0101 eng=12 gpfifo=0x49c0000 ... +begin h=0xbeee0100 eng=0 gpfifo=0x44a0000 ... +begin h=0xbeec0104 eng=0 gpfifo=0x4d20000 ... +begin h=0xbeec0100 eng=0 gpfifo=0x4cc0000 ... +``` +The Vulkan client's copy-class channels **0xbeef0101 (eng 11) and +0xbeef0102 (eng 12) are in the table and swept** - and their rings carry +nothing all boot: `read=none` on first sight, `swept=0` on every pass, no +`utildec`, no LID, no copy. The client posted one event on 0xbeef0101 at +registration (`post_event hSrc=0xbeef0101`) and never pushed. + +**What did decode.** All boot: 3 LID writes only - the driver's two zero +buffers (256 B -> vidmem 0x423f000, 12288 B -> 0x4448000, both +`zero=1`) and the launch's own param bank (1568 B -> 0x4b86020, +`zero=0`). No `sema-lda`/`sema-opadd` rows at all, no `copy-in` rows, no +`utildec` on the two CE channels. Global swept=1652, writes=3. + +**The code page never fills.** `0x62b000` (the launch's program page for +`prog=0xdfd8feb900`) and the rotating-pool page `0xc8b000` read ALL-ZERO +before the client, after it, and at a final re-dump; the whole 256 MB +aperture holds only 12 non-zero dwords boot-wide (the viewport scalars +`0x44200000`/`0x43f00000` at 0x386634/0x386638, the releases, the +baseline). Nothing anywhere writes program bytes. + +**Client outcome.** `VKDEVICE ok family=0` then +`VKFAIL vkWaitForFences ... -> 2`; no frames, no PPMs. The service +answered `GR-COMPLETE-SW` 10 347 times for the same refused push while +the client kept waiting. This is the same outcome as the r801b forward +test, and it differs from the earlier black-frame baseline +(`VKFRAME seq=1..3`, three black PPMs): on a refused/failed forward the +service authors a client-visible `SYNTH-FAULT` (commit `c6c152cc`, +`sema_refusal_fault=1` since r618), and the current completion/release +path does not get the client past its fence. `util_scan` is excluded as +the cause (r801b failed with it off). + +**Reading.** The Vulkan client *believes* it uploaded its compute shader +(it dispatches with a valid program VA), but no CPU write, no CE LID, no +CE copy and no GR i2m in any observed channel ever delivers the program +bytes to the emulated vidmem. The driver's upload path is a +`staging ring -> code heap` copy (CPU memcpy into a staging BO, then a +GPU copy to the code heap); the copy is not among the decoded pushes and +the staging BO is not visible in the emulated vidmem. Two separate +questions now stand: +1. **Delivery** - where the driver queues that copy (a CE channel it + never pushes, a GR subchannel/method shape the parser does not + decode, or a sysmem staging BO that a different read path would + expose). +2. **Completion regression** - restore the refusal-path software + completion (or gate the synthetic fault) so the client reaches the + fence-released black-frame baseline again; this is needed for every + downstream observation. + +**Next.** (a) Dump and decode the client's push method stream around the +upload (the `staging ring -> code heap` copy) rather than more ring +sweeps; the staging BO's CPU write is the most likely host-visible copy +of the SASS. (b) Instrument the completion path for one refused Vulkan +launch (release ordering vs the synth-fault) to restore the black-frame +baseline. (c) Only then resume the mapping work (the r801b `FAULT_PDE` at +the program VA). diff --git a/docs/04-implementation/05-graphics-execution-plan.md b/docs/04-implementation/05-graphics-execution-plan.md new file mode 100644 index 0000000..6089548 --- /dev/null +++ b/docs/04-implementation/05-graphics-execution-plan.md @@ -0,0 +1,721 @@ +# Graphics execution plan: real GL pixels through the emulated vGPU + +> **Status: OPEN (2026-10-28).** This is the implementation-ready plan for the +> **graphics execution layer**: the component that must execute the guest +> NVIDIA vGPU driver's OpenGL/3D command streams on the physical GPU so that +> glxgears and GZDoom's OpenGL renderer produce real pixels through the +> emulated mdev. Nothing here is implemented yet beyond the pieces cited; +> every statement carries a code location or a measured session finding. +> +> Evidence base: `docs/04-implementation/03-execution-frontier.md` (the CUDA +> compute chain, proven), `docs/04-implementation/04-real-render-walls.md` +> (§8-§21, all measured graphics walls), the SESSION 149 R12-R18 round log +> (`session-149-BRIEF.md`: the M1 C597 recognizer, M2 materialization, M3 +> SKEDCHECK clear, R17 real-SASS execution), and SESSION 144 addenda 9/10/16 +> (the X/GL wait anatomy). The vendored 570.144 wire protocol and the public +> class headers are the interface authority; no guest driver change is +> allowed. +> +> Hard rules carried from the project: **no other GPU** (virtio-gpu/virgl, +> VirtualGL, llvmpipe, any second device are banned for this work), **no +> fake/labelled software rendering** presented as GL, **host-only emulation** +> (guest runs stock `nvidia.ko`), and **no host reboot** (AGENTS.md hard +> rule; a reboot needs the handoff update plus explicit human approval). + +--- + +## 0. What "real pixels" means here + +The acceptance target is: a stock guest GL application's draw commands are +executed by the physical GPU's **fixed-function 3D pipeline** (vertex fetch, +primitive assembly, rasterization, fragment work, ROP), its render target is +readable by the guest, and that image reaches the capture path through the +emulated mdev. The pixels must not come from Mesa, llvmpipe, virgl, a +software rasterizer, or a host display round-trip. + +Two consequences drive every milestone below: + +1. **A compute launch is not rasterization.** The SESSION 149 M1/M3 work + proved the engine will execute a *synthesized* C5C0 compute launch built + from the GL stream and will retire it + (`poll +500ms qmd_rel=0x600df00d`, R17), but the GL client's draws need + the 3D pipeline and the client's own per-draw state; rasterization is + fixed-function GR (`session-149-BRIEF.md:288-291`). +2. **The GL client's own channel is the source of truth.** The C597 section + the emulation can capture today comes from a readable channel (the X + server side); the GL client's GPFIFO ring (`0xbeef0100` family, + `0xbfef0002` observed in SESSION 153) is where its context-init uploads + and per-frame draws live. Until that ring is readable, no approach can + derive a correct draw (`session-149-BRIEF.md:438-451`). + +--- + +## A. Gap list, ordered by dependency + +Each gap lists what exists (with the code location that implements it), what +is missing, and the measurable gate that closes it. A gap is closed only when +its gate has been met on a fresh cycle with evidence kept. + +### G-A Readability: the GL client's ring and pushes must read + +**Exists.** The pump resolves rings through an autopin ladder (notifier- +derived, USERD-derived, trusted walk, shadow scan, USERD-shape scan) with +dead-entry rejection: `kernel/gsp_sema_service.c:16534-16602` +(`svc_ring_entry_dead`, `svc_ring_walk_live`, `svc_gr_ring_candidate`), +`:17891`, `:17972`, `:18013`, `:18075`, `:18123`, `:18242`, `:18298`. The +failure is witnessed at `:17633` (`sema-chan: GR ring h=... RAM content scan +found no candidate in N slice(s)`). Push reads report through `sema-rdw` +(`:28622`) and `sema-pushdec` (`:28868-28882`). The guest pin stream's client +selection and windows are `tools/guest-diag/guest_pin_stream.py:1429-1475`, +`tools/guest-diag/pin_gr_stream.py:134-160`, and the supervisor already +defaults to the widened GL prefix set +(`tools/guest-diag/pin_supervisor.sh:70-83`: +`cdt,va,victory,Xorg,X,glxgears,gzdoom`). + +**Missing.** For the GL client channel the ring page is not found by content +at the scan instants ("GR ring h=0xbfef0002 rank=2 ... no candidate in 12 +slice(s) (va=0x1203cfff0)", SESSION 153 item 3). The client's allocations are +just above the old 64 MB capture bound in the Vulkan case +(`docs/04-implementation/04-real-render-walls.md` §14-§15), and the pump's +ring candidates are shape-based; a client ring written late (after the DMA +test window) is missed. The guest pagemap stream (`#pid` + `va,pa,n` lines, +`guest_pin_stream.py:8-12`) is the authoritative VA->PA source but was not +covering the GL client until SESSION 153; the supervisor default now does. + +**Gate G-A.** On a run with glxgears alive for 60 s and the standard X +recipe: (a) zero new `sema-chan: GR ring h=0x... found no candidate` rows for +the GL client's handle; (b) every client push logs `sema-pushdec ... src>0 +unreadable=0`; (c) at least one entry is consumed from the client ring +(pump advances `last_get`; `sema-chan: entry[...] h=0x... eng=0` with a sane +`va/len`); (d) the X channel (`0x1000013`) stays `plausible=1` (SESSION 153 +item 1). + +### G-B Capture: the client's C597 sections and per-frame draws must decode + +**Exists.** A gated recognizer (`sema_gl_xlate`, default 0, +`kernel/gsp_sema_service.c:6812-6815`) opens a section at a `SET_OBJECT` +class bind of `0xC597` (TURING_A) on the channel subchannel and collects +observed VA operands: program (`0x2014`/`0x2018` runs), +tex header/sampler pools (`0x1574`/`0x155c`), vertex substitute (`0x0f84`), +upload source/destination (`0x180`), and the LAUNCH_DMA/LOAD_INLINE_DATA +payload (`0x1b0`) — `:6945-7049`, `:11993-12032`. It synthesizes a 64-word +QMD and captures it through the unchanged GR candidate ring +(`:7060-7287`, `:6686`). The client's memory-writing uploads are applied by +a standalone subchannel-0 pair tracker (`:12045-12084`) and the section +walker (`:7194-7217`) through `svc_grce_inline_write` (`:8979-9060`). +Completion of the capture is gated by content trust: a plausible stream +admitted by `cap_trust` (`:28995`, `:29069-29080`), with hash-verified +rewind replays admitted (`:28513-28517`, `:29106-29119`); the flush guard +requires `seen/prog/param/rel/trusted` (`:7082-7084`). Witness lines: +`sema-gl: section captured ...` (`:7248`), `sema-gl: uploads applied=...` +(`:7213`), `sema-gl-lid: ...` (`:12073`), `sema-gl: M3 code src=...` +(`:7261`). + +**Missing.** +1. The recognizer's method vocabulary covers init-shaped operands only. The + per-draw 3D state and draw-issuing methods (vertex streams/index, render + target and ZS, viewport/scissor, blend/depth/ROP state, draw calls) are + not decoded (`svc_gl_field` handles six addresses, + `:6945-7049`; the C597 stream carries far more). +2. The GL program/constant delivery is a context-init upload on the client + channel and is lost today: R17 measured that no readable push ever writes + `prog=0x121201b00` (`session-149-BRIEF.md:276-285`). The synthesized + launch therefore only runs a 12-28 B fallback blob or an operator hex + program (`sema_gl_prog_hex`, `:6877-6913`) — a development crutch, not + pixels. +3. Capture must survive the "one-shot setup push rewritten in place" + class: re-captures come from hash-verified replays now + (`:29069-29080`), but the per-frame draws are separate entries and need + the same treatment per entry. + +**Gate G-B.** For ≥3 consecutive frames of a live GL client: `sema-gl: +section captured` on the **client's** handle with nonzero `prog/param/rel`, +one captured section per frame (bounded replays suppressed), and a decoded +draw record (new `sema-gl-draw` witness) whose vertex/primitive count is +consistent with the app (e.g. glxgears gear geometry). Program source +priority shows `trusted-prog` (real bytes) for the context-init upload. + +### G-C Execution semantics: the GL work must run on the real 3D engine + +This is section **B**; the gap gate is M-GL1/M-GL2/M-GL3. + +### G-D Memory and resource mapping: every operand must be reachable by the engine and readable by the guest + +**Exists.** The physical backend builds engine-side page tables and shadow +windows for launch data: Tier-1 identity install or Tier-2 shadow copy +(`kernel/gsp_phys_booter.c:19654-19724`, `:19734-19785`), a two-bank +window-0 shadow pool of 8 pages per bank (`:1174`, `:19406-19410`) with a +24-page shadow chunk (`:1317`), QMD-window relocation +(`gsp_phys_rmcall_gr_qmdwin`), and a verbatim-prelude replay path that +relocates the inline-QMD address words and data words +(`:16878-16921`). Data pages are copied in before the doorbell and copied +back after the retire wait (`:17693`, `:18243`, `:21966`, `:22242`). +`svc_grce_inline_write` writes upload bytes to vidmem shadow, guest RAM, or +the VA shadow with a once-only key (`kernel/gsp_sema_service.c:8979-9060`). + +**Missing.** +1. Address-word relocation is QMD-only (words 32-49 and the indirect-page + pointer fixup at `booter:19765-19779`). A 3D method stream names VAs at + class-specific offsets; each must be resolved to a shadowed page and the + word rewritten (or the page mapped at its guest VA). +2. The engine-visible page budget is small: 16 pages per request + (`kernel/gsp_phys_booter.h:355`), 8 inline records (`:356`), 8 pages per + win0 bank (`booter:1174`). A GL frame names program + constants + texture + pools + vertex stream + render target + ZS; the shadow pool and the + per-request page list must grow or pages must be grouped per buffer. +3. Render-target writeback: the copy-out tracks only the launch's own data + pages; a 3D draw writes the RT, which must be mapped into guest-visible + memory (BAR1 shadow or pinned guest RAM) and copied back on the + completion edge, exactly like the current bounce/copy-out but for the RT + and depth surfaces. +4. Program/constant bytes: R17's measurement stands — the program page is + empty in every host-visible source until the CE/LID delivery executes + (`docs/04-implementation/04-real-render-walls.md` §20-§21). + +**Gate G-D.** Engine fetch of the 3D program/params/vertex/RT pages with +`FAULT-PDE count=0` (`vgsp-m91: FAULT-CAP ... code_src=...`, +`booter:18997-18999`), and a measurable post-launch change in the emulated +vidmem at the RT VA (>0 changed dwords in the `dump_vidmem map` class, +`docs/04-implementation/04-real-render-walls.md` §11). + +### G-E Completion and sync: the guest must be released by real completions + +**Exists.** Engine completion is polled (`qmd_rel/ch_rel`, +`booter:20814-20825`), the launch's release0 can be redirected to the module +semaphore and relayed (`booter:19499-19520`), the hold edge and the +backend-outcome edge author the guest word (`kernel/gsp_sema_service.c: +21329-21345`, `:21298-21322`), and failures synthesize a client-visible +channel error (`SYNTH-FAULT`, `:21400-21448`, `GR-COMPLETE-SW`, `:21504`). +The pre-built ring continuation beyond GP_PUT is swept for releases +(`sema_gr_ahead`, `:26564-26566`, `svc_gr_ahead_push`, `:26809`). + +**Missing.** The guest GL client's waits are still satisfied by the give-up +ladder or by host pokes: the X server's present wait spins on the X channel +member/counter pair (S144 addendum 12; the mirror thread `sema_xw_mirror` +`kernel/main.c:3103-3129`), and the GL client's `PhaseWait` compares +`obj+0x558` against a word in the shared `memfd:/.nvidia_drv` (+0x8) that the +emulation never advances (SESSION 149 R3, `session-149-BRIEF.md:115-126`: +"one u32 write 3 → 128.8 FPS", but only via `/proc/<pid>/mem`). The correct +long-term fix is the one named in addendum 16b §C.4: **the emulation +completes the real syncs so the equality is satisfied, with no lying pokes**. + +**Gate G-E.** M-GL4 (below): ≥60 fps sustained with no forcing daemon, no +`sema_xw_mirror` target written, and the guest's waits exiting on a natural +completion edge. + +### G-F Present/frame path: the rendered image must land where the guest and the capture can read it + +**Exists.** Guest-side capture tools: `tools/guest-diag/sdl_cap.c` (SDL swap +hook → `glReadPixels` → GZF2 record stream, `:1-18`, `:188-229`), +`tools/guest-diag/fbcopy_pub.c` (BAR1 scanout copy → kvmfr publisher), +`tools/guest-diag/qga_fetch.py` (QGA file fetch), host decoder +`tools/lg/frame_reader.py` and the resample/encode scripts. + +**Missing.** On the emulated-vGPU stack the X screen is black for GL windows +(DRI3-present path; SESSION 152) and `xwd` hangs (R4/R5; no rendered pixels +exist anywhere). The present path (the C5B5 copy-class and 902D 2D parts of +the GL push, R13) is decoded for releases but not executed as a display +path, and the capture must come from the GL client's swap, not from X. + +**Gate G-F.** M-GL2 (glxgears non-black) and M-GL3 (GZDoom). + +### G-G Concurrency: two VMs, two mdevs + +**Exists (designed in).** The service thread iterates every open device each +pass (`kernel/gsp_sema_service.c:32119-32236`), with the explicit note that +"with two, both guests get a pump turn per iteration (hardware forwarding +stays serialized by the single-inflight slot)" (`:32161-32167`). Per-guest +sessions exist (`SVC_MAX_GUESTS = 8`, `:728-736`), and the guest-RAM ranges +are registered per device at boot (`/tmp/opencode/start_vm_keep.sh:210-213`). + +**Missing / to verify.** Single-inflight forwarding and holds +(`:19878`, `:25528-25545`); VA-keyed global tables that assume one guest's VAs +do not co-reside: the write/read ledger is keyed by VA page only and its own +comment says collisions "only occur ... across clients whose pages do not +co-reside in time" (`:7547-7559`), plus `svc_gl` (`:6863`), +`svc_gr_cand_ring` (`:6493`), `svc_gr_seen` (`:6547`), `svc_grpark` (`:7429`); +the win0 banks are module-global per submit epoch (`booter:19406-19410`); the +server-managed VAS relocation windows are a finite global set +(`SVC_GR_DATA_WINS_MAX 31`, `:6441`). SESSION 151 avoided a second mdev "to +avoid a second fragile RM init", so a second RM bootstrap on the same module +is unproven. + +**Gate G-G.** M-GL5 (below). + +--- + +## B. Execution-semantics approaches + +### (i) 3D method-stream replay on the real engine + +Forward the client's captured C597 method stream (class bind + 3D state + +upload/draw methods + in-stream releases) to the module's real GR channel +and let the fixed-function pipeline execute it. + +What must be mirrored/provided: + +| Requirement | Where it stands | +|---|---| +| Real execution channel/context | Mode 35 "real-client user lane" replays the 570.144 client's compute-context sequence and executes on V7 (`booter:924`, `:9022-9033`); the promote ladder builds MAIN/PM/PATCH and the golden image. | +| 3D class object (0xC597) | `GSP_PHYS_RMCALL_CLASS_GR_3D 0xc597` is defined (`booter:396`) and the alloc helper is generic (`gsp_phys_rmcall_alloc_gr_object_class`, `:8936-8964`); it is constructed on the golden lane in several modes (`:9035-9043`, `:9665`), but **not on V7** (mode 35 logs "the capture carries no 0xc597 on the user lane", `:9029-9033`) — a class object must be added on V7 and validated. | +| 3D context global buffers | Already promoted: BUNDLE_CB / PAGEPOOL / ATTRIBUTE_CB / RTV_CB_GLOBAL (`booter:531-534`, `:605-621`, golden9 sized buffers `:985-1008`). The engine context image therefore contains the 3D subcontext data. | +| Method block | `gsp_phys_rmcall_build_gr_launch` (`:16834-16947`) emits compute init + inline QMD + PCAS + report-sem; the verbatim path (`:16878-16921`) already replays a captured raw method window and rewrites its address/QMD words. A 3D mode adds `SET_OBJECT 0xC597` + the captured state/upload/draw methods instead of the compute prelude. Note the channel's GR subchannel is 1 (`:442`) and the compute init binds `0xC5C0` (`:16693-16698`); the 3D bind replaces it per section. | +| Operand pages | The page list is built from QMD words 32-49 (`:19678`, `:6673-6682`); it must be generalized to the recognized 3D operand VAs, with the shadow/alias machinery (`:19654-19785`) reused. | +| Completion/sync | Keep the stream's own releases and the module release0 redirect (`:19499-19520`); the hold/relay path already authors the guest word on the engine edge (`gsp_sema_service.c:21329-21345`). | + +Pros: authentic rasterization is done by the hardware pipeline; no state is +invented; the engine has already proven it fetches and executes forwarded +work on this lane (R17 `qmd_rel=0x600df00d`); the completion machinery is +unchanged. + +Cons/risks: the promoted context may still be compute-shaped (class object +construction on a wired golden channel deadlocked in earlier experiments, +`booter:816-822`; V7 is the tested path, so construct there); the 3D method +vocabulary is large (state validation can silently fault); address-word +relocation must cover every class encoding; ~6-12 operand pages per frame +exceed today's 8-page win0 bank, requiring grouping/expansion. + +Effort: L. Depends on G-A, G-B. + +### (ii) Synthesize a "graphics-shaped" launch from the captured C597 stream + +On Turing, a QMD is a *compute* launch descriptor; there is no QMD that +rasterizes. "Graphics-shaped" therefore means: derive the draw inputs and +author the state + a DRAW method ourselves — program (`0x2014/0x2018`, +already decoded), constant buffers (bind + LID uploads, partly decoded), +vertex buffers (SET_VERTEX_STREAM_SUBSTITUTE is decoded only as a pointer, +`gsp_sema_service.c:6979-6983`; the real stream/index methods are not), +render targets/ZS (not decoded; the only observed target is the release +target `0x120001000`, R13), viewport/scissor/blend/depth state (not decoded). +The vertex count for the draw would be inferred, not read. + +Pros: fewer pages to mirror than a full replay if only draw-relevant state is +bound; still uses the hardware rasterizer. + +Cons: we author state the driver owns — any wrong field silently +mis-renders; the per-draw state lives in the client ring, so G-A/G-B are +still prerequisites; the program bytes are still lost without G-B; it is a +subset of (i)'s work plus extra invention. + +Effort: L. Risk: high. Not first. + +### (iii) Bounded compute translation for the subset GZDoom uses + +Replace each GL draw with a compute kernel (or the existing synthesized +C5C0 launch carrying the client's real SASS) that rasterizes triangles into +the RT. + +What breaks correctness, concretely: +- **Fixed-function raster and ROP**: depth test, stencil, alpha blending, + culling, clipping, perspective-correct interpolation, MSAA — none exist in + a compute launch; reimplementing them is writing a GPU. +- **Sampling**: mipmaps, LOD selection, anisotropic filtering, wrap modes + and sRGB conversion are texture-unit behaviour. +- **Shader semantics**: the client's GLSL is already compiled to SASS by the + guest driver; executing that SASS as a whole-SM compute program does not + give it vertex inputs, varying interpolation, or per-fragment coverage — + the very things the 3D pipeline supplies. +- **The claim itself**: even GZDoom's first frame uses FBOs + (scene → post-process), depth testing, alpha blending and mipmapped + textures; output would be a lookalike, not the renderer's pixels, and + could not be verified against a reference. + +Verdict: acceptable only as a **diagnostic probe** (e.g. a known-pattern +kernel proving the lane), never as the milestone path, and it must be +labelled as such. Not first. + +### Recommendation + +**Implement (i) first.** It is the only approach that yields authentic GL +pixels; its difficult prerequisites (G-A ring readability, G-B stream +capture) are shared by every alternative; and the backend already provides +the three pieces that make it tractable: a real execution lane with a +validated context (mode 35), a generic 0xC597 class-object allocator, and a +verbatim method-window replay with address/QMD rewriting plus page shadowing +and completion relay. (ii) is (i)'s draw-issuing step taken prematurely; +(iii) trades correctness for nothing. + +Order of implementation inside (i): **G-A** → **G-B** → replay the captured +*context-init* section (M-GL1, proves class bind + uploads + completion on +the 3D lane) → replay the client's per-frame *draw* sections (M-GL2/3). + +--- + +## C. Milestone ladder + +All gates are measured with the artifacts listed in section D; a milestone is +met only when its gate reproduces on **two consecutive fresh cycles** (three +for M-GL1) with the module hash recorded. + +### M-GL1 — Client context-init work decodes and executes; the client leaves its retry loop + +Gate (all in one run): +1. GL client ring resolved: no `sema-chan: GR ring h=<client> ... found no + candidate` row after the first minute; an autopin row exists + (`sema-chan: auto-pinned GR ring ...`). +2. Client pushes read: `sema-pushdec` rows for the client handle with + `src>0 unreadable=0 plausible=1`; no `sema-rdw ... rc=-2` for them. +3. Section captured from the client: `sema-gl: section captured h=<client>` + with `prog/param/rel` nonzero and `replay=0` (or the hash-verified replay + admission), at least once per client lifetime. +4. Forwarded and executed: `vgsp-gr: GR LAUNCH submitted h=<client> + ... pages>=2 shadow=1`, followed by engine evidence (`vgsp-gr: poll +... + qmd_rel=0x600df00d`) and either a natural release or `GR-COMPLETE-SW`. +5. Client behaviour: glxgears no longer loops on context creation (no + repeated init-failure retry; the X server stays alive; `xdpyinfo` OK, + `DMAFAIL=0`), and the app reaches its first swap. +6. Bounded failure hygiene: no SYNTH-FAULT storm for the accepted client + (≤ a handful in the window), no guest oops. + +Expected failure signatures (test one at a time): +`GR ring ... found no candidate` (G-A); +`unreadable=1 first8=00000000` (G-A, the S147 addendum 9 zero-page class); +`GR request without candidate` / `cand_valid=0` +(`gsp_sema_service.c:25479-25485`; G-B ordering: capture must precede the +request); +`GR submit refused: degenerate data set n_data=N (min M)` +(`booter:19352`; operand list, G-D); +`FAULT-PDE` at the program VA + `CODEHASH h0=0` +(`booter:18806-18883`, `:18997`; G-B/G-D); +`GR LAUNCH FAILED: timeout ... qmd_rel=...` (`booter:20825`; engine did not +retire — class/sched fields or completion wiring). + +### M-GL2 — glxgears renders non-black frames + +Gate: +1. A GZF2 record stream is captured from the **GL client's swap** (see D for + the GLX shim; the repo's `sdl_cap.c` is SDL-only — glxgears does not use + SDL). +2. ≥1 frame with ≥5% of pixels above luminance 8/255 and ≥2 distinct + dominant colours; two frames ≥500 ms apart differ in ≥1% of pixels + (motion, not a static noise tile). +3. The same time window contains the per-frame GR launch + completion in + dmesg, and the guest's `glxinfo` renderer is the NVIDIA driver (not Mesa). +4. The first frames are reached without any sync poke. If a diagnostic poke + is used to unblock the run, that run is labelled diagnostics and does not + satisfy the gate. + +Failure signatures: swap hook fires but frames all zero → no rasterization +(G-C); `xwd`/X path black → expected, do not use; frames show only clear +colour → draws executed but state/vertex data wrong (G-D); +`FAULT-CAP ... code_src=guest-hpa code64=000...` → program source lost +(G-B). + +### M-GL3 — GZDoom GL opens a window and presents non-black frames through the mdev + +Gate: +1. GZDoom (`vid_rendermode=4`, `vid_preferbackend=0`) runs on `DISPLAY=:0` + backed by the emulated vGPU (no Xvfb, no Mesa overrides). +2. ≥300 consecutive non-black GZF2 frames from the SDL swap hook, including + gameplay (not only the menu); the in-frame FPS counter ≥30. +3. dmesg shows one GR launch per rendered frame (bounded), no SYNTH-FAULT + loop; the X server stays alive for the whole run. +4. Capture fetched via QGA and decoded offline; a screenshot + (`+screenshot`) matches the captured frame within a small pixel delta. + +Failure signatures: window opens, swap fires, all frames black → draws not +executed; `GR submit refused ... n_data` → operand pages; app crashes with +`Illegal Instruction Encoding` (ESR 0x504730=0x9) → program bytes are not +SASS (G-B; cf. R18's single-page kernel lesson); `SYNTH-FAULT` per frame → +forward failed or refused. + +### M-GL4 — 60 fps sustained in ONE VM, no forcing + +Gate: +1. glxgears: "N frames in T seconds = X FPS" with X ≥ 60 over a 60 s + measurement (`vblank_mode=0 __GL_SYNC_TO_VBLANK=0` for the measurement + itself), and GZDoom's counter ≈60. +2. **No** poke/force/mirror daemon is running and no runtime module param is + used to fake progress: `sema_xw_mirror` is `0,0` (stopped, + `kernel/main.c:3103-3129`), no `/proc/<pid>/mem` write helper, no + strike/deadline forcing. +3. GZF2 inter-frame deltas: p95 ≤ 17 ms over 10 s (capture path keeps up). +4. The X present waits exit with member advancing naturally (Xorg log WAIT + lines absent or resolving without the kickfix daemon), and the glcore + equality word no longer requires the bump. +5. CUDA regression: `victory_mail` `VADD n=32768 bad=0` / `STATUS OK` on the + same boot after the GL run (battery-first ordering). + +Failure signature: the two ~11 s glcore waits return (the app's own strike +ladder; S144 addendum 16b) → G-E; X WAIT with static member → present +completion not authored; `GR-STALL-backend` per frame → completion is +software-only and the app's ladder is what "ran". + +### M-GL5 — Two VMs, two mdevs concurrently + +Verification checklist before the gate: +1. Second mdev creates and its guest RM bootstraps (the S151 open question: + "a second fragile RM init"); both devices appear in the service loop + (`gsp_sema_service.c:32119-32236`). +2. Per-guest isolation of the shared tables: `svc_ledger` VA-only key + (`:7547-7559`), `svc_gl` (`:6863`), `svc_gr_cand_ring` (`:6493`), + `svc_gr_seen` (`:6547`), `svc_grpark` (`:7429`); either prove guest VAs do + not collide in practice or key them by `(vgpu, va)`. +3. Guest RAM: both VMs' memory-backend ranges registered and resolving + (the cycle registers one file per guest; verify the per-device map). +4. Server VAS relocation: no two guests' data windows alias in the managed + region (`SVC_GR_DATA_WINS_MAX 31`, `:6441`; comment `:6436-6439`). +5. Win0 banks/epochs (`booter:19406-19410`) are safe under serialized + submissions; single-inflight holds (`:19878`) do not starve one guest. + +Gate: both VMs run glxgears and GZDoom for ≥10 minutes; both present +non-black frames; each ≥30 fps (60 target); `victory_mail` green in both +afterwards; no cross-guest data anomalies (`sema-ledger: DIVERGE` / +`LEDGER-RESET` absent), no host oops, no guest wedge. + +Failure signatures: one VM starves the other's forward slot (`GR request +busy: hold active`, `:25509`); second guest's launches resolve the first +guest's pages (wrong results / ledger rewrite storms, `:7584-7594`); second +RM init fails in the boot ladder. + +### M-GL6 — Flawless 60 fps split-screen video + +Gate: 600+ frames at 60/1 fps, ≥95% unique frames, both halves show real +GZDoom GL gameplay through the two mdevs, encoded with the existing +resample/compose pipeline, uploaded with the files-upload skill and +pull-back-md5 verified. Encode with `libx264` (the R18 path did: +`/tmp/opencode/r18_final.sh`); NVENC is unavailable while the module owns +the physical GPU (the host NVIDIA driver is displaced). + +--- + +## D. Experiment recipes + +### D.0 Standard cycle (all milestones) + +Host, one controlled attempt per cycle (script leaves the VM running): + +``` +nohup /tmp/opencode/start_vm_keep.sh rNNN "<EXTRA_ARGS>" \ + > /tmp/opencode/rNNN-cycle.log 2>&1 & +# poll until: DONE (no teardown; VM left running) (~70-90 s) +``` + +`EXTRA_ARGS` = the SESSION 149 standard profile +(`session-149-BRIEF.md:65-67`), verbatim: + +``` +gsp_phys_gr_nowrite_retries=0 gsp_phys_gr_retry_fast=1 +gsp_phys_gr_first_poll_ms=2500 gsp_sema_winfill=1 gsp_sema_gr_swrel_gate=1 +gsp_phys_rmcall_gr_promote=35 gsp_phys_rmcall_fwclient_mask=0xE3F +gsp_phys_rmcall_srv_ctxcommit=0 gsp_phys_rmcall_gr_qmdwin=1 +gsp_phys_rmcall_gr_shadow_win0=1 gsp_phys_rmcall_gr_lmem_en=1 +gsp_phys_rmcall_gr_skedrec=1 gsp_phys_rmcall_gr_inst_sys=1 +gsp_phys_rmcall_gr_r1_first=1 gsp_phys_gr_strict_reloc=0 +gsp_phys_fault_relay=1 sema_bigcopy_log=512 +``` + +GL overlay (M-GL1..M-GL3 runs): append + +``` +sema_gl_xlate=1 sema_push_raw=4 gsp_phys_gr_min_data=0 +sema_gr_util_scan=0 sema_dbg=Y +``` + +(`sema_gl_xlate` is off by default in-tree, `gsp_sema_service.c:6812`; +`min_data=0` is the explicit no-external-data bypass validated in S143 and +used by R15-R17; `util_scan>0` is a measured regression source, keep 0; +`sema_dbg=Y` raises the bounded witnesses). The base script's ladder already +passes `gr_promote=25`; the later `=35` wins at insmod and is snapshotted by +the boot ladder — verify the boot log names mode 35 +(`booter:8612-8613`). + +Post-load, before Xorg: + +``` +sudo sh -c 'echo 1000 > /sys/module/vgsp/parameters/sema_vas_cand_ms' +sudo nohup dmesg -w > /tmp/opencode/dmesg_live_rNNN.log 2>&1 & +``` + +Guest Xorg (one per boot; ~50 % of boots fail DMA and need a re-cycle): + +``` +echo ubuntu | sudo -S setsid bash -c \ + 'nohup Xorg :0 -ac -noreset -logfile /tmp/xorg.log >/tmp/xorg.out 2>&1 </dev/null &' +# require: DISPLAY=:0 xdpyinfo -> 1280x720; grep -c "Failed to initialize DMA" /tmp/xorg.log == 0 +``` + +After Xorg is up: + +``` +sudo sh -c 'echo 1 > /sys/module/vgsp/parameters/sema_force_flush' +``` + +Evidence per run, always kept: module md5 (`md5sum kernel/vgsp.ko`), the +echoed insmod args, the full dmesg capture, guest `/tmp/xorg.log`, +`/tmp/glx*.log`, the GZF2 file and its md5, the decoded frame PPMs, and the +acceptance-grep output. + +### D.1 M-GL1 recipe + +Guest: start glxgears and watch the client channel: + +``` +DISPLAY=:0 setsid nohup stdbuf -oL glxgears -geometry 800x600+0+0 \ + > /tmp/glx1.log 2>&1 </dev/null & +``` + +Host acceptance greps on the dmesg capture (client handle filled from the +first `sema-chan` row): + +``` +grep -E "auto-pinned GR ring h=0x<client>" # G-A pass +grep -E "sema-pushdec: h=0x<client>.*unreadable=0" # G-A pass +grep -E "sema-gl: section captured h=0x<client>" # G-B pass +grep -E "GR LAUNCH submitted h=0x<client>" # capture->forward +grep -E "qmd_rel=0x600df00d|GR-COMPLETE-SW" # engine executed/completed +``` + +If G-A fails: confirm the guest scanner is running the widened prefixes +(`/tmp/guest_pin_stream.log`; supervisor env at +`tools/guest-diag/pin_supervisor.sh:70-83`), force a demand resolve via +`sema_trace_h=<client handle>` (S148 addendum 10) to see which source fails, +and use `echo <va> >> /tmp/guest_want_vas.txt` for a targeted pin. If G-B +fails with `entry_replay=1` keeping `cap_trust=0`, the pass is an un-hashed +replay: wait one fresh push or use the hash-admitted path +(`gsp_sema_service.c:29069-29080`) — do not add a bypass. + +Expected time-to-first-signal: minutes. Do not proceed to M-GL2 until 3 +cycles pass. + +### D.2 M-GL2 recipe (glxgears pixels) + +The repo shim is SDL-only (`sdl_cap.c:188-229` hooks `SDL_GL_SwapWindow`, +`SDL_RenderPresent`, `SDL_UpdateWindowSurface`); glxgears is GLX and calls +neither. Add a GLX mode to the same tool (hooks `glXSwapBuffers` → +`glReadPixels`, same GZF2 record layout), build: + +``` +gcc -shared -fPIC -O2 -o /tmp/glx_cap.so glx_cap.c -ldl +``` + +Guest: + +``` +DISPLAY=:0 GLXCAP=/dev/shm/glxframes.bin GZCAP_MAXF=0 \ + LD_PRELOAD=/tmp/glx_cap.so setsid nohup stdbuf -oL \ + glxgears -geometry 800x600+0+0 > /tmp/glx2.log 2>&1 </dev/null & +``` + +Fetch + decode on the host: + +``` +python3 tools/guest-diag/qga_fetch.py /dev/shm/glxframes.bin /tmp/opencode/glxframes.bin +# parse GZF2 records (tools/lg/frame_reader.py / gz_final_gpu.py's reader), +# then measure: % pixels > 8/255, colour count, motion between frames. +``` + +Do not use `vblank_mode=0` for the first pixel check (sync-on is fine); keep +the shim's swap hook ordered as "capture then forward" (same as sdl_cap). + +### D.3 M-GL3 recipe (GZDoom GL) + +Guest (xorg.conf untouched, the emulated GPU as the only GPU): + +``` +VK_ICD_FILENAMES=/nonexistent.json VK_DRIVER_FILES=/nonexistent.json \ +GZCAP=/dev/shm/gzframes.bin GZCAP_MAXF=0 LD_PRELOAD=/tmp/sdl_cap.so \ +DISPLAY=:0 setsid nohup ./gzdoom -iwad freedoom2.wad \ + +vid_rendermode 4 +vid_preferbackend 0 +vid_activeinbackground true \ + +sv_cheats 1 +noclip > /tmp/gz.log 2>&1 </dev/null & +# window management with tools/guest-diag/{xresize,xfocus,xtest_move}.c as in +# SESSION 151; XTEST alone is ignored by SDL here - use the XSendEvent path. +``` + +Host: `qga_fetch.py /dev/shm/gzframes.bin /tmp/opencode/gz.bin`, decode, then +`/tmp/opencode/gz_final_gpu.py` (resample to natural-speed 60 fps) for the +video. **Do not** set `LIBGL_ALWAYS_SOFTWARE` or the Mesa EGL vendor list: +that is the S151 software path and would invalidate the milestone. + +### D.4 M-GL4 recipe (60 fps, no forcing) + +Same as D.1/D.3, plus: + +- ensure `sema_xw_mirror` is stopped (`echo 0,0 > .../sema_xw_mirror`) and no + forcing/kickfix daemon is running (`pgrep -af` exact names first); +- measure 60 s windows only: glxgears FPS line, GZDoom counter, GZF2 + timestamps; +- keep `sema_gr_grace_ms`/`sema_ce_grace_ms` at their defaults; the R1 A/B + proved they are not the pacer (`session-149-BRIEF.md:96-99`); +- battery-first CUDA regression in the same boot. + +### D.5 M-GL5 recipe (two VMs) + +Second mdev + second VM (own disk copy, own sockets, own SSH/QGA ports), both +with the vGPU device this time. Verify in order: second mdev appears; second +guest's `nvidia-smi`/`glxinfo` sees the vGPU; service log shows both device +handles per pump pass (`sema-chan` rows for both); then run the D.1/D.3 apps +in both. Keep per-VM captures separate (two GZF2 files, two QGA sockets). + +### D.6 M-GL6 recipe (video) + +Per-VM SDL capture → `qga_fetch.py` → `/tmp/opencode/gz_final_gpu.py` +(timestamp resample + split-screen compose) → `ffmpeg -c:v libx264` (R18 +pattern) → files-upload skill → pull-back md5 verification. 1280x440 +split (640x440 per guest) matches the two delivered videos. + +--- + +## E. What must NOT be done + +1. **No other GPUs or virtual GPUs.** No virtio-gpu/virgl, no VirtualGL, no + llvmpipe/Mesa software GL, no second physical GPU, no host display + round-trip. The SESSION 152 video is explicitly *not* evidence for this + plan. +2. **No fake or mislabelled rendering.** No software rasterizer or compute + lookalike presented as GZDoom's OpenGL renderer. A compute-translation + probe (approach iii) may exist only as a labelled diagnostic and is never + an acceptance artifact. +3. **No lying sync pokes in accepted runs.** The `/proc/<pid>/mem` counter + bump, strike forcing, kickfix and `sema_xw_mirror` are diagnostics; any + milestone evidence must state they were absent. +4. **No host reboot.** AGENTS.md hard rule: handoff update first, explicit + human approval, never automatic. Guest VM cycles and Xorg restarts on the + test VM are the allowed recovery paths. +5. **No guest modifications.** Stock guest `nvidia.ko` and stock apps; no + injected code, patched firmware or modified driver in the guest; capture + shims are LD_PRELOAD observation only and must not alter GL semantics. +6. **No time/timing claims from unsupported sources.** The physical backend + is the execution authority; simulator (L0/L1) findings never substitute + for a live engine result. +7. **No unguarded experiments on the physical GPU.** Every cycle is + supervised, single-attempt, evidence-captured (the cycle script's hard + abort signatures are not bypassed). + +--- + +## F. Open questions and note-vs-code discrepancies + +1. **`sdl_cap.c` is SDL-only.** The milestone recipe in the SESSION 149 brief + names it for glxgears, but glxgears is GLX and never calls + `SDL_GL_SwapWindow` (`tools/guest-diag/sdl_cap.c:6-9`, `:188`). A GLX + variant is required (D.2); `sdl_cap.c` remains the right shim for GZDoom + (M-GL3). +2. **The GL-client pin prefix fix already looks landed.** + `tools/guest-diag/pin_supervisor.sh:73-83` defaults + `CLIENT_PREFIXES=cdt,va,victory,Xorg,X,glxgears,gzdoom` and the cycle + starts the supervisor (`start_vm_keep.sh:242`), while SESSION 153 still + lists the "permanent fix" as pending. Verify the running guest scanner + actually received the widened env (`/tmp/guest_pin_stream.log`) before + treating G-A as closed. +3. **The logged failure strings differ slightly from the briefs.** The code + prints `vgsp-work: GR request without candidate` + (`gsp_sema_service.c:25481`), not bare "GR request without candidate"; the + ring failure is `sema-chan: GR ring h=... found no candidate` + (`:17633`). Grep acceptance scripts for the exact prefixes. +4. **`n_data` threshold history.** The default is 16 + (`booter:17709-17717`); S143+ cycles used `min_data=0` as an explicit + bypass for the no-external-data launch class, enabled by the r800 guard. + The brief's "min 1"/"min 16" variants refer to different experiments; the + GL profile below uses 0 and must say so in every evidence record. +5. **Stale project status row.** `docs/README.md:114` still lists CUDA + execution as "no grid has dispatched yet", which predates the proven + physical forward and the R18 CUDA video. This plan relies on the newer + code and `docs/04-implementation/03-execution-frontier.md`, not that row. +6. **The 3D context on the execution lane is the main unknown.** Mode 35's + capture "carries no 0xc597 on the user lane" + (`booter:9022-9033`); golden-lane 3D objects exist (`:9035-9043`) but the + execution channel is V7. The first implementation experiment must prove a + 0xC597 class object can be constructed on V7 after the accepted promote + (same helper, `:8936`) without the wired-golden deadlock class + (`:816-822`), before any stream replay work.
\ No newline at end of file diff --git a/docs/05-operations/01-performance-correctness-security.md b/docs/05-operations/01-performance-correctness-security.md index 2f10b35..1391426 100644 --- a/docs/05-operations/01-performance-correctness-security.md +++ b/docs/05-operations/01-performance-correctness-security.md @@ -17,26 +17,32 @@ Rung 4 SUSTAINED workloads run for hours; faults/errors recoverable Rung 5 MULTI-TENANT N VMs, isolation + fairness + no cross-talk ``` -**Current rung (2026-10-02):** Rungs 1–2 are green: the stock driver probes, -`nvidia-smi` enumerates, and the full CUDA driver-API smoke passes -(`ALL OK` 64/64, copies matched) — `cuInit`, device queries, context creation, -allocation, copies, free and destroy complete, and the copies are verified -byte-for-byte. **Rung 3 is partial:** CE-class copies execute on the physical -copy engine (default off) with engine-authored releases and a fault relay; for -GR kernel execution a forwarded guest launch now reaches the front-end but the -inline launch-descriptor install has not been observed (no grid dispatched; -context validity at dispatch is ruled out — -[`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md)). -Before Rung 3 -is claimed, the -**reply-fidelity gate** applies: a control whose reply fields the driver -consumes is not implemented until the reply is coherent (status-only → -coherent zeroed params → coherent values for consumed fields → exact values) — -see -[`../../userspace/cuda/docs/04-virtualization-and-emulation.md` §6](../../userspace/cuda/docs/04-virtualization-and-emulation.md). -New to CUDA analysis here: start at +**Current rung (2026-10-30):** Rungs 1–3 are green for the verified CUDA +clients: the stock driver probes, `nvidia-smi` enumerates, the full CUDA +driver-API smoke passes (`ALL OK` 64/64), and guest kernels **execute on the +physical engine** — vector-add, matrix-multiply and transformer clients verify +on fresh boots with engine-authored completions relayed to the guest. What +remains before Rung 3 is claimed outright: the staged-transfer class (some +large multi-chunk clients still mis-verify; failure modes proven) and the +every-boot acceptance campaign — plus the **reply-fidelity gate** below. Rung 4 +(sustained) has partial evidence (a 100-launch loop, long soak pending); rung 5 +(multi-tenant) is partial: two mediated devices and two guests coexist, and +per-guest execution isolation plus fairness measurement are in progress. On the +graphics side, the X server/GLX run on the emulated vGPU, the engine executes +GL method streams, and the guest Vulkan pipeline runs end to end; rendered +(non-black) frames are the frontier +([`../04-implementation/05-graphics-execution-plan.md`](../04-implementation/05-graphics-execution-plan.md)). +**No performance numbers are published yet**; the A/B programme in §3 is the +next measurement step, and the first scheduled public measurement is the +graphics 60 fps target (no forcing). New to CUDA analysis here: start at [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md). +The **reply-fidelity gate** applies to every emulated control: a control whose +reply fields the driver consumes is not implemented until the reply is +coherent (status-only → coherent zeroed params → coherent values for consumed +fields → exact values) — see +[`../../userspace/cuda/docs/04-virtualization-and-emulation.md` §6](../../userspace/cuda/docs/04-virtualization-and-emulation.md). + ## 2. Correctness ### 2.1 Test corpus diff --git a/docs/05-operations/05-gpubreach-audit.md b/docs/05-operations/05-gpubreach-audit.md new file mode 100644 index 0000000..3c60385 --- /dev/null +++ b/docs/05-operations/05-gpubreach-audit.md @@ -0,0 +1,95 @@ +# GPUBreach-class audit and threat model (vGPU host module) + +Status: audit performed 2026-10-21 against the host module at `229bec2c` +(`vgsp.ko`, `kernel/`), with the write-path guards landed in the same window +(`7d4919dd`, `7a1f4805`). Scope: every host-side write whose target address +derives from guest-supplied data. Companion reading: +[`01-performance-correctness-security.md`](01-performance-correctness-security.md). + +## 1. What the GPUBreach threat is + +GPUBreach (IEEE S&P 2026, U. Toronto) is a GPU Rowhammer chain: the attacker +flips bits in the physical GPU's page tables (PTEs), which turns the GPU's +MMU into an arbitrary *device-memory* read/write primitive; the paper then +escalates to CPU root by exploiting memory-safety bugs in the *driver* code +that consumes those corrupted translations. It works with the IOMMU on, +because the corruption happens inside the GPU, after translation. + +The relevant question for this project is therefore not "can a guest read +device memory" (device memory is the guest's own resource here) but: + +1. can guest-influenced data reach a *host* write without validation, and +2. what bounds a write that lands on a *wrong* target. + +## 2. Structural mitigations (why the host is not on that path) + +* **Guest GPU page tables are host-emulated.** The guest's GMMU page tables + live in the emulated vidmem shadow (BAR1 backing) or in guest RAM; the + host resolves guest GPU VAs through its own walk of the guest's published + page directories (`SET_PAGE_DIRECTORY`) or through content-verified pins. + There is no physical GPU MMU walk of guest-authored tables. +* **The physical GPU's PTEs are host-owned.** Backend jobs are submitted by + the host module on host-authored PTE lines; the guest never supplies a + descriptor that the real hardware MMU consumes for host memory. +* **Every write is aperture-bounded.** The one write primitive, + `svc_write_ap()`, sends SYS writes through `vfio_dma_rw()` on the mdev's + own VFIO container - the IOMMU/vfio layer only maps the guest's RAM - and + FB writes are bounds-checked against the emulated shadow size. A + guest-influenced target can therefore only ever land inside the guest's + own RAM or the vidmem shadow; a wrong target is a correctness bug, not a + host-memory write. +* **No guest DMA to host buffers.** The guest's drivers talk to the + emulated device; they cannot address host kernel buffers. IOMMU is on + (vfio), and the mdev exposes only guest RAM plus the emulated windows. +* **Guest descriptors are not trusted by construction.** Every path that + turns a guest VA into a host PA goes through the resolution chain + (guest-verified injected pins -> write/read landing ledger -> published + page-directory walk -> targeted record scan), and the chain's outputs are + re-validated per page (see §3). + +## 3. Write-path inventory and validation state + +| Write path | Target derivation | Validation today | +|---|---|---| +| CE fill / memset (`svc_fill`) | guest VA or physical FB/GPA from the push | per-page destination resolution (pin -> ledger -> walk), page-boundary clamped; a page with no resolution truncates loudly and demands the VA through the pin channel (`FILL-TRUNC`, `stalled VA`); zero fills onto page-table-shaped guest pages are refused (`FILL-PT-REFUSED`); below-floor window fills that resolve outside vidmem are refused (`LOWFILL-REFUSED`) | +| CE copy (`svc_copy`) | dst/src VAs from the push | per-page resolution with pin-agreement (a fresh pin beats a stale ledger entry and repairs it); device-dst copies without a pin-resolved translation are held (`sema_copy_dst_pin_guard`) | +| GRCE inline writes (`LOAD_INLINE_DATA`) | destination VA from the push | once-only identity gate keyed by (push VA, push content hash, dst, bytes), session-scoped and 8192 slots wide | +| GR launch operand staging (`svc_gr_data_fill`) | argument pointers scanned from the QMD | per-page pin/ledger resolution; the resolution is stored in the ledger as the landing authority; dropped pages are named (`GRFILL-DROP`) and demanded | +| Launch copy-out (engine results back to the shadow/guest page) | the launch's own landing records | per-page destination resolution; every write is witnessed (`OUT-PAGE-WIT`/`OUT-PAGE-POST`); staged-identity rule (only the page staged into a slot may receive the slot's content) | +| Release/counter writes (semaphores, completion counters) | guest semaphore VAs from pushes | pool-range writes go through a pin/record-verified cache (`pool-pa`), re-validated against the current pin before the 4-byte write; exact-slot fallback for the client's tracker; forward-only apply | +| GPFIFO/USERD writes (put/get bookkeeping) | channel registration data | the channel's own USERD/instance blocks from the alloc params, aperture-tagged | +| Window/PTE installs (forward PDE install) | the launch's windows | installs are refused loudly when the backend rejects the PDE; the regular path leaves guest VAs in place (no host-side table mutation) | + +## 4. Residual risks and findings + +* **Wrong-but-valid writes remain possible** when every resolver agrees on a + stale page (e.g. a guest freed and reallocated a VA). The r391/r398/r401 + fixes removed the known instances (contiguous fill past the first page, + replayed inline writes, capture-window eviction), but the class is bounded, + not eliminated: the guards convert silent corruption into loud + truncation/refusal where a page is unresolved, and the ledger is + session-scoped and pin-repaired where a fresh pin exists. +* **The boot-corruption lottery** (a zero fill landing on guest page-table + pages) is structurally mitigated by the page-table-shape refusal and the + per-page clamping; no instance has been observed since r391. +* **Diagnostics stay on**: `DST-DIVERGE` (copy-dst ledger vs probe), + `FILL-TRUNC`, `GRFILL-DROP`, `NOWRITE-*` and the stalled-VA demand rows + exist so a future divergence is visible in the boot log rather than + silent. +* **What this audit does not claim**: no isolation or side-channel claim. + GPU Rowhammer itself is a *physical* fault-injection threat that this + software emulation cannot mitigate; the mitigation argument here is only + that the host write paths cannot be steered outside the guest's own + memory, and that wrong targets are bounded by IOMMU-mapped guest RAM or + the emulated shadow. + +## 5. Recommended follow-ups + +1. Keep the copy-destination ledger under the `DST-DIVERGE` witness and + promote it to a bounded hold only if a failing run names it. +2. Extend the pin-demand channel to client device VAs (today the in-guest + scanner resolves only its own push/table windows), so an unresolved + operand page becomes a wait on real data instead of a truncated fill. +3. Re-run this audit when the write-path guards change (any new + `svc_write_ap()` caller with a guest-derived target is in scope by + default). diff --git a/docs/05-operations/06-guest-frame-path-and-looking-glass.md b/docs/05-operations/06-guest-frame-path-and-looking-glass.md new file mode 100644 index 0000000..bc5447a --- /dev/null +++ b/docs/05-operations/06-guest-frame-path-and-looking-glass.md @@ -0,0 +1,102 @@ +# Guest frame path and Looking Glass status (2026-10-21) + +This page records the state of the "show the guest's GPU output on the host" +goal: what is proven, what is built, and exactly which pieces remain. It is +written from live evidence on the vGPU-IO stack (guest CUDA green, ivshmem +devices present, kvmfr module loaded on the host). + +## 1. What is proven + +* **Guest GPU rendering works through CUDA.** `tools/cuda-ab/frame_producer.cu` + renders an animated Julia set on the guest GPU (800x600 RGBA, ~10 fps), + copies the frame to host memory and publishes it into the ivshmem BAR + (`/sys/bus/pci/devices/0000:00:03.0/resource2`, the 128 MiB kvmfr device + QEMU backs with `/dev/shm/kvmfr-<uuid>`), with a small header + (`magic/version/width/height/stride/format/seq/timestamp`) at offset 1 MiB. +* **The host can read those frames.** `tools/lg/frame_reader.py` maps the same + shm file and writes each new sequence number out as a PPM, i.e. a real + GPU-rendered guest frame lands on the host. This is the full transport path + (guest GPU -> guest sysmem -> ivshmem -> host), independent of any display + protocol. +* **The ivshmem devices are visible in the guest** (`lspci`: two + "Inter-VM shared memory" functions; `resource2` of `0000:00:03.0` is the + 128 MiB window, `0000:00:04.0` the 2 MiB cursor window), and the guest can + mmap them (root required for the sysfs resource files). +* **EGL enumerates the NVIDIA device in the guest.** With + `EGL_EXT_platform_device`/`EGL_EXT_device_enumeration` present, + `eglQueryDevicesEXT` returns 3 devices and `eglInitialize` on the NVIDIA + device succeeds (`vendor=NVIDIA version=1.5`). + +## 2. What is blocked, and why + +* **Guest GL contexts do not allocate.** `eglMakeCurrent` on an offscreen + pbuffer fails with `EGL_BAD_ALLOC` (0x3003) on every platform tried + (device, surfaceless, default). Xorg's own failure is the same class: + `(EE) NVIDIA(GPU-0): Failed to allocate VASpace object` then + `Failing initialization of X screen`. The driver cannot allocate GPU virtual + address space for a GL context/surface through the vGPU, so both the X + display path and the EGL offscreen path stop at the same wall. This is the + single remaining blocker for guest OpenGL (and therefore for GZDOOM / + doomsday rendering in the guest). +* **The Looking Glass client (B4) cannot consume our frames directly.** + `looking-glass-client` (installed at `/usr/local/bin/looking-glass-client`, + runs under Xvfb with the `libbfd` shim in `/tmp/opencode/lgcompat`) supports + two transports, LGMP and SPICE. Its `app:shmFile` option still expects an + **LGMP session** (the `KVMFR` session header plus the shared-memory message + queues), not a raw framebuffer. Upstream's frame *producer* (`host/`) is the + Windows DXGI/IDD path. So a Linux guest-side producer must either implement + the LGMP session protocol or we keep the frame reader as the host-side + substitute (which is what `frame_reader.py` is). +* **GZDOOM is not packaged for Ubuntu noble** (`apt-cache policy gzdoom` has + no candidate); `doomsday` + `freedoom` are installed as the fallback, but + doomsday cannot start a renderer without a working GL context (blocker + above). + +## 3. Exact remaining pieces (in order of leverage) + +1. **Fix the guest GL VASpace allocation.** This is an RM/vGPU-side gap: the + driver's `nv_vgpu` VASpace allocation for a GL context fails. Fixing it + unlocks Xorg, EGL offscreen GL, doomsday/GZDOOM, and the standard Looking + Glass guest path (the guest's GL capture driver would then work as + upstream expects). +2. **Implement the LGMP session producer** (or a small Linux capture client) + that writes frames into the ivshmem region with the KVMFR session header + and the frame queue protocol the client expects. The transport half is + already proven; this is protocol work in the guest (or a host-side + substitute that speaks LGMP). +3. **Packaging**: GZDOOM would need a source build or a flatpak/snap; the + doomsday fallback is installed and only blocked by (1). + +## 4. Reproduce + +```bash +# guest (root, because the sysfs resource is root-only): +sudo /tmp/frame_producer 60 800 600 10 # 60 frames, 800x600, 10 fps + +# host: +python3 tools/lg/frame_reader.py --frames 5 --out /tmp/opencode/frames +# -> SAVED seq=N 800x600 fmt=0 -> /tmp/opencode/frames/frame-NNNN.ppm + +# guest EGL probe (shows the enumeration success and the MakeCurrent wall): +gcc -O2 -o egl_frame tools/guest-diag/egl_frame.c -lEGL -lGLESv2 +./egl_frame +``` + +## 5. Session-13 update (2026-10-21, r432→r441) + +* The single-frame path is unchanged and still proven: guest `frame_producer` + (CUDA Julia) -> kvmfr ivshmem BAR -> host `frame_reader.py` -> + `/tmp/opencode/frames/frame-0001.png` (320x240, 1431 colours). +* `tools/lg/frame_sequence.sh` captures an **animated** sequence end to end + (reader first, then N frames from the guest). It is blocked by the + GR no-execute wedge: when the lane is wedged the render launches write zero + pages and the producer republishes stale identical frames (observed: 6/6 + identical MD5s on r438; r439 published 5 identical plus one frame left in + the BAR by the previous boot). A multi-frame animation needs a healthy + lane; the frame pipeline itself is not the blocker. +* The GL wall is untouched: `Xorg :0` -> `(EE) NVIDIA(GPU-0): Failed to + allocate VASpace object`, guest dmesg `NVRM: vaspaceapiConstruct_IMPL: + Could not construct VA space. Status ffff`, i.e. `vmmCreateVaspace` inside + the guest's own RM. Next step: reproduce with the guest RM's + `ResmanDebugLevel` raised (a guest module reload, the service's m177 path) + to capture which inner call returns the generic error. diff --git a/docs/README.md b/docs/README.md index 13f0ae5..95ef85c 100644 --- a/docs/README.md +++ b/docs/README.md @@ -98,7 +98,7 @@ and [`01-orientation/02-the-ai-edge-and-ecosystem.md`](01-orientation/02-the-ai- ## 3. Where is the project today? -**Honest status (2026-10-02):** +**Honest status (2026-10-30):** A dated snapshot of this status, refreshed on every progress check, is kept at [`../STATE-CURRENT.md`](../STATE-CURRENT.md). @@ -111,28 +111,30 @@ A dated snapshot of this status, refreshed on every progress check, is kept at | GSP-RM controls: device/fb/bus tables, CE caps, vaspace, GMMU static info | 🟡 Mostly — the engine inventory now includes the async copy engine and the CE capabilities match the device; reply fidelity for the remaining consumed fields is still being tightened | | Guest `nvidia.ko` probe, `nvidia-smi` | ✅ Working (TU104) | | Guest CUDA driver (`libcuda`) startup | ✅ Driver-API baseline — `cuInit`, device enumeration, `cuCtxCreate`, `cuMemAlloc`, copies, free and destroy complete end to end; the smoke test prints `ALL OK` and the 64-byte payload round-trips exactly (64/64, re-verified 2026-10-02 on the current build). See [`04-implementation/01-known-knowns-and-unknowns.md`](04-implementation/01-known-knowns-and-unknowns.md) and [`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md) | -| CUDA kernel execution | 🟡 A stock guest program now reaches its kernel launch and submits it through the forwarding path; the front-end fetches the ring entry and executes methods, but the inline launch-descriptor (QMD) install has not been observed and no grid has dispatched yet. Context validity at dispatch is ruled out; the frontier is the launch method-stream form. Copy-engine-class work, its releases and its fault relay remain forwarded (parameters default off). See [`04-implementation/03-execution-frontier.md`](04-implementation/03-execution-frontier.md) and [`03-architecture/11-physical-backend-guest-wiring.md`](03-architecture/11-physical-backend-guest-wiring.md) | -| OpenGL / Vulkan | 🔴 Not yet | -| Windows guest | 🔴 Not yet | -| Multi-tenant isolation (VFs, mdev-per-VM) | 🟡 Designed, partially implemented | - -We also know exactly where the CUDA bring-up stands and what the next step is: -the userspace `libcuda` segfault was fixed (an RM allocation reply now echoes -the request parameters instead of returning zeros), the engine inventory is -complete, and the driver-API smoke test now completes end to end — `cuCtxCreate` -returns and the 64-byte payload round-trips exactly (64/64, re-verified -2026-10-02 on the current build). The full -mechanism, with offsets and the decompiled call path, is in +| CUDA kernel execution | ✅ Working — verified guest clients execute kernels on the physical GPU with engine-authored completions relayed to the guest: the vector-add (`VADD n=32768 bad=0`), matrix-multiply (`MATMUL n=256 bad=0`) and transformer (`dh=256 seq=64 bad=0`, `FFN4 layers=4 bad=0`) clients print `STATUS OK` on fresh boots, and `vecadd_loop` passes 100 launches. Remaining: the staged-transfer class (some large multi-chunk clients still mis-verify; failure modes proven), the every-boot acceptance campaign, and per-guest execution isolation. See [`04-implementation/03-execution-frontier.md`](04-implementation/03-execution-frontier.md) and [`03-architecture/11-physical-backend-guest-wiring.md`](03-architecture/11-physical-backend-guest-wiring.md) | +| OpenGL (3D graphics) | 🟡 In progress — the X server and GLX run on the emulated vGPU with the stock driver; the engine executes real GL method streams (verbatim 3D replay and synthesized launches); a direct GL client creates its channels and its readback path returns clean. Rendered (non-black) frames are the frontier. See [`04-implementation/05-graphics-execution-plan.md`](04-implementation/05-graphics-execution-plan.md) | +| Vulkan | 🟡 In progress — the guest Vulkan pipeline runs end to end on the emulated vGPU (device, allocation, pipeline, command buffer, fence, queue submit, readback); frames are still black — program delivery and program-VA mapping are the frontier | +| Windows guest | 🔴 Not yet (target) | +| Multi-tenant isolation (mdev per VM) | 🟡 Two mediated devices with different emulated VRAM sizes and two guests coexist; per-guest execution isolation for simultaneous CUDA guests is in progress | + +Guest CUDA now executes on the physical GPU: the driver-API path completes end +to end, forwarded launches reach the engine, the grid dispatches, the engine +authors the completion and the host relays it, and the guest's synchronisation +waits resolve (verified with vector-add, matrix-multiply and transformer +clients on fresh boots). The mechanism, with offsets and the decompiled call +path, is in [`../userspace/cuda/docs/05-570.144-decompilation.md`](../userspace/cuda/docs/05-570.144-decompilation.md). -The remaining frontier is guest-facing execution. Copy-engine-class copies and -their releases are forwarded to real hardware with release relay and fault -relay in place (parameters default off). For GR (graphics/compute), the guest -now reaches and submits a real kernel launch: the ring entry is fetched and -front-end methods execute, but the inline launch-descriptor install has not -been observed and no grid has dispatched yet. The wall is the form of the -forwarded launch method stream — not context validity, which is ruled out at -dispatch. The chain, the evidence and the live investigation lines are in -[`04-implementation/03-execution-frontier.md`](04-implementation/03-execution-frontier.md). +The remaining CUDA work is breadth and hardening: the staged-transfer class +(some large multi-chunk clients still mis-verify; the failure modes are +proven), the every-boot acceptance campaign, and per-guest execution isolation +for simultaneous guests. Graphics shares the same submission machinery and is +the other active frontier: the X server and GLX run on the emulated vGPU, the +engine executes GL method streams, and the guest Vulkan pipeline runs end to +end — rendered (non-black) pixels are the wall. The chain, the evidence and the +live investigation lines are in +[`04-implementation/03-execution-frontier.md`](04-implementation/03-execution-frontier.md) +and +[`04-implementation/05-graphics-execution-plan.md`](04-implementation/05-graphics-execution-plan.md). The backend, a module-owned GPU running its real GSP firmware as the resource-manager server, is specified in [`03-architecture/08-physical-gpu-integration.md`](03-architecture/08-physical-gpu-integration.md); @@ -183,6 +185,8 @@ Read in order if you are new. Jump directly if you know what you need. | 23 | [GSP in a RISC-V VM (risc-v-gsp-rm-debugger, experimental)](05-operations/03-risc-v-gsp-rm-vm.md) — implemented, documented and tested ([audit](../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md); [value demos](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md)) | developers, analysts | 15 min | | 24 | [risc-v-gsp-rm-debugger user guide (experimental)](05-operations/04-risc-v-gsp-rm-debugger-user-guide.md) — implemented, documented and tested ([audit](../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md); [value demos](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md)) | developers, analysts | 15 min | | 25 | [Defensive Operations Runbook](RUNBOOK.md) | maintainers, operators | 20 min | +| 25a | [GPUBreach-class audit and threat model](05-operations/05-gpubreach-audit.md) | developers, auditors | 15 min | +| 25b | [Real-render walls: Vulkan and Xorg on the emulated vGPU](04-implementation/04-real-render-walls.md) | developers, debuggers | 10 min | | 26 | [Engineering operations](eng-ops.md) — tool admission and workflows (Part I) plus live physical-backend operations (Part II) | maintainers, developers | 20 min | | 27 | [Hardware and software matrix](06-portability/hardware-software-matrix.md) | developers, integrators | 20 min | | 28 | [Developer onboarding and standards](07-developers/onboarding-and-standards.md) | contributors | 20 min | |