diff options
Diffstat (limited to 'docs/04-implementation')
| -rw-r--r-- | docs/04-implementation/03-execution-frontier.md | 232 | ||||
| -rw-r--r-- | docs/04-implementation/04-real-render-walls.md | 113 |
2 files changed, 244 insertions, 101 deletions
diff --git a/docs/04-implementation/03-execution-frontier.md b/docs/04-implementation/03-execution-frontier.md index f5546ca..a8e03ee 100644 --- a/docs/04-implementation/03-execution-frontier.md +++ b/docs/04-implementation/03-execution-frontier.md @@ -1,11 +1,17 @@ # The Execution Frontier: The GR Compute-Launch Chain -> **Status: 2026-10-06.** This page describes the current leading edge of +> **Status: 2026-10-20.** This page describes the current state of > guest-facing execution: the chain that must fire for a stock guest CUDA -> kernel to run on the physical GPU, what we have proven about it, where it -> currently stops, how we know, and the open lines of investigation and -> theories. It is evidence-led: every statement carries either a live -> observation, a decompilation finding, or a clearly labelled theory. +> kernel to run on the physical GPU, what we have proven about it, how we +> know, and where the leading edge is now. The chain fires end to end: +> verified guest CUDA clients (vector-add, matrix-multiply and a transformer +> kernel) execute on the physical GPU, their results verify, and the +> engine-authored completions relay back to the guest. The leading edge has +> moved from "can a forwarded launch execute at all?" to breadth and +> hardening — the large-buffer staged-transfer class, reproducibility across +> boots, and per-guest execution isolation. It is evidence-led: every +> statement carries either a live observation, a decompilation finding, or a +> clearly labelled theory. > > Evidence sources for this page: supervised physical cycles (module-owned > GSP-RM on the reference TU104), offline firmware emulation/probing, the host @@ -40,40 +46,59 @@ flowchart TB J["Host polls release, relays to guest;<br/>CUDA wakes"] ``` -| # | Stage | State (2026-10-06) | +| # | Stage | State (2026-10-20) | |---|---|---| | 1 | Guest builds launch + method stream | ✅ observed (stock driver, unmodified) | | 2 | GPFIFO entry + doorbell | ✅ observed | | 3 | Host decode + forward stream + QMD prewrite + ring/USERD | ✅ observed (forwarding path) | | 4 | Front-end fetches the ring entry | ✅ observed (ring cursor advances) | -| 5 | Front-end executes methods | ✅ observed on the ladder channel (object bind, cache ops, report semaphore) | -| 6 | **Inline-QMD install** (descriptor materialised into the QMD buffer) | ✗ **not observed** | -| 7 | **Engine reads the QMD** | ✗ **not observed** | -| 8 | Context switch loads the channel context | ✅ context validity at dispatch is ruled in (instance flag set) — see §4 | -| 9 | GR dispatches the grid to the SMs | ✗ never observed on this stack | -| 10 | Engine authors release + channel semaphore | ✗ absent for forwarded launches | -| 11 | Host relays completion; guest wakes | ✗ guest waits and times out | - -**One-line summary:** the guest, the emulation, the forwarding path and the -front-end are live; the **launch-descriptor install path** (stages 6–7) is the -leading edge, and with it stages 9–11. +| 5 | Front-end executes methods | ✅ observed (object bind, cache ops, report semaphore) | +| 6 | Inline-QMD install (descriptor materialised into the QMD buffer) | ✅ observed | +| 7 | Engine reads the QMD | ✅ observed | +| 8 | Context switch loads the channel context | ✅ context validity at dispatch is ruled in — see §4 | +| 9 | GR dispatches the grid to the SMs | ✅ observed (the guest's results verify) | +| 10 | Engine authors release + channel semaphore | ✅ observed — engine-authored releases relay to the guest | +| 11 | Host relays completion; guest wakes | ✅ observed — the guest's synchronisation waits resolve | + +**One-line summary:** the whole chain is live — verified guest CUDA clients +execute on the physical GPU and wake up correctly. The leading edge is no +longer the chain itself but its breadth and hardening: the large-buffer +staged-transfer class, reproducibility across boots, and per-guest execution +isolation. --- ## 2. Where the leading edge is -A stock guest program now reaches its kernel launch and submits it through the -forwarding path. On the wire we observe the ring entry consumed and the -front-end methods run — but the launch descriptor is never materialised, the -engine never reads a QMD, no engine-authored completion appears, and the guest -blocks in its synchronisation wait. - -This is **not** a context-validity wall (that was the previous frontier and is -now ruled out at dispatch, §4), and it is **not** a window-mapping wall for the -descriptor destination (that install is accepted). The wall is the **form of -the launch method stream**: what the front-end requires in order to install the -inline descriptor, and what the engine requires in order to read and dispatch -it. +A stock, unmodified guest CUDA program now executes its kernels on the +physical GPU and verifies its results. On fresh boots the vector-add client +(`victory_mail`) prints `VADD n=32768 bad=0` + `STATUS OK`, the +matrix-multiply client (`victory_matmul`) prints `MATMUL n=256 bad=0` + +`STATUS OK`, and a transformer kernel prints `dh=256 seq=64 bad=0`, +`FFN4 layers=4 bad=0` + `STATUS OK`. The executing engine is the physical GPU +under the module's own real GSP-RM session; the host CUDA bridge is not part +of this path. The engine authors the completions, the host relays them to the +guest, and the guest's synchronisation waits resolve. + +The leading edge has therefore moved from the launch chain itself — it fires — +to breadth and hardening around it: + +* **The large-buffer staged-transfer class.** Multi-megabyte staged transfers + between guest memory and device memory are under repair: some large + multi-chunk clients still mis-verify. +* **Reproducibility across boots.** A boot-poisoning defect was root-caused + and fixed: under the driver's status-queue mode the emulation read the wrong + queue cursor, so under load the reply ring could desynchronise, the first + CUDA client would fail, and the rest of that boot was poisoned (the + user-visible `STATUS FAIL`-then-freeze report). Fixed; re-measurement + across fresh boots is in progress. +* **Per-guest execution isolation.** Multiple mediated devices with different + emulated VRAM sizes and multiple guests already coexist at the device level; + per-guest execution isolation for multiple simultaneous CUDA guests is in + progress. + +No breadth beyond these is claimed: OpenGL/Vulkan, Windows guests, migration +and performance numbers remain planned. --- @@ -81,17 +106,17 @@ it. | Observable | What it proves | What it does **not** prove | |---|---|---| -| Ring entry consumed; ring cursor advances | The front-end is alive and fetching | Nothing about the engine | -| Method execution on the ladder channel (report semaphore, channel semaphore) | The front-end can execute the method vocabulary | **Not** engine execution — these signals can be front-end-authored | -| Descriptor buffer prewritten by the host with a detectable sentinel | Whether the descriptor was materialised/modified after submission | A read does not modify memory; the sentinel detects installs, not reads | -| Install sentinel unchanged after submit | The inline-install method did not run | It does not say why (mapping, required control method, or ordering) | -| Engine-authored release word absent; completion poll times out | The engine did not complete the launch | It does not say whether the engine started | -| No GR fault recorded | The engine did not raise an error | Silence is not health: a skipped or never-reached launch is silent by construction | -| Channel instance context flag set | The channel is context-valid at dispatch | Nothing about the descriptor path | - -**Consequence:** progress on this frontier must be measured by an -**engine-authored** signal (the release word advancing) — not by front-end -signals, which we have already learned to distinguish. +| Client verification output (`bad=0` + `STATUS OK`) | The guest executed its kernel, read its results back and verified them | by itself, which engine produced the results | +| Engine-authored release word advancing, relayed to the guest | The GR engine executed the launch and the completion relay path works | nothing about other launches | +| The guest's synchronisation wait resolving | Completion reached the guest through the relay | nothing about other launches | +| Ring entry consumed; ring cursor advances | The front-end is alive and fetching | **not** engine execution — these signals can be front-end-authored | +| Reproduction across fresh boots | The path is not a lucky boot | not yet an "every boot" claim while re-measurement is in progress | +| No GR fault recorded | The engine did not raise this error | silence is not health: a skipped or never-reached launch is silent by construction | + +**Consequence:** acceptance is now measured by verified guest results plus the +engine-authored release; front-end signals remain necessary but not +sufficient. A green run counts as progress only when the same acceptance +reproduces across fresh boots. --- @@ -111,90 +136,94 @@ signals, which we have already learned to distinguish. * **Parenting/TSG for the fetch** — both parenting variants fetch identically. * **The copy-engine path** — CE-class work, its releases and its fault relay are live (default off); the GR path is a separate problem. +* **The descriptor-gate line of investigation (the old T1–T4 theories).** The + inline-descriptor install, the engine's QMD read, the grid dispatch, the + engine-authored release and the guest wake-up are all now observed; the + gate they questioned is closed by observation (§1, §7). +* **The old boot-poisoning defect.** Root-caused to the status-queue + consumption cursor and fixed; it is no longer the explanation for a failing + first client (§5, issue B). --- ## 5. The clear issues (ranked) -**Issue A — the forwarded stream carries a verbatim guest prelude that cannot -complete.** -The forwarded stream includes a verbatim copy of the guest's prelude whose -inline-data writes target addresses that are not present in the engine's view -of the channel's page tables, and the referenced data is not relocated. The -shorter, proven stream shape fires its tail (channel semaphore) but still does -not schedule the descriptor. -*How we know:* the prelude stalls in the forwarded form; the shorter shape's -tail fires; the descriptor-install sentinel never changes in either case. -*If fixed:* the install path is reached and we can observe stage 6 directly. - -**Issue B — the descriptor-virtualization control method is absent.** -The method that arms the inline-descriptor install path (the QMD -virtualization control) is not present in the forwarded stream. -*How we know:* the method does not appear in the submitted stream, and the -install sentinel never changes. -*If fixed:* the front-end should materialise the descriptor — falsifiable in -one cycle by watching the sentinel and the engine-read signal. - -**Issue C — diagnostic hygiene.** -The descriptor sentinel placed at word 0 overlaps real descriptor fields; it -must be moved to an unused word so diagnostics do not perturb what they -measure. +**Issue A — the large-buffer staged-transfer class.** +Multi-megabyte staged transfers between guest memory and device memory are +under repair: some large multi-chunk clients still mis-verify. +*How we know:* the small and medium clients (vector-add, matrix-multiply, the +transformer kernel) verify green while large multi-chunk transfers mis-compare +or stall in the same configurations. +*If fixed:* the client battery should accept the large-buffer class with the +same verification discipline as the green clients. + +**Issue B — reproducibility across boots (re-measurement).** +A boot-poisoning defect was root-caused and fixed: under the driver's +status-queue mode the emulation read the wrong queue cursor, so under load the +reply ring could desynchronise, the first CUDA client would fail, and the rest +of that boot was poisoned (the user-visible `STATUS FAIL`-then-freeze report). +*How we know:* the failure was reproduced, the cursor rule was corrected, and +per-client unregister and session flush work again. +*If fixed:* green on every fresh boot — re-measurement is in progress, and no +"every boot" claim is made until it completes. + +**Issue C — per-guest execution isolation.** +Multiple mediated devices with different emulated VRAM sizes and multiple +guests already coexist at the device level; per-guest execution isolation for +multiple simultaneous CUDA guests is in progress. +*How we know:* the device-level coexistence is live; execution isolation is +not complete. +*If fixed:* multiple CUDA guests execute simultaneously without interference. --- ## 6. Lines of investigation (live) -1. **Restore the proven launch shape** as the baseline (the shorter stream that - fires its tail) and **re-add the virtualization control method** — one - variable at a time, with the install sentinel and the engine-read signal as - the readout. -2. **Relocate or drop the verbatim prelude** so the inline-data writes target - addresses present in the engine's view, or replace it with the equivalent - host-side staging the proven shape already uses. -3. **Move the sentinel** and re-run to confirm the readout is clean. -4. **Diff against a known-good reference.** Two ways to obtain one: (a) make - our own ladder dispatch a descriptor built from the same source material - (in-house reference), or (b) capture a working launch of the stock stack on - the same GPU (real method stream + descriptor + engine-visible channel - state) and compare field by field. Either reference converts the remaining - work from inference to differential. -5. **Offline probing** of any rule the front-end or engine is believed to - require, using the firmware emulator/probe tooling before spending a - supervised cycle. +1. **Repair the large-buffer staged-transfer class** — the multi-megabyte + guest↔device transfers that still mis-verify. The three green clients stay + the regression battery and must remain green throughout. +2. **Complete the fresh-boot re-measurement** — demonstrate the fixed boot + path across fresh boots before claiming boot-to-boot reproducibility, and + keep the re-measured battery as the regression gate. +3. **Per-guest execution isolation** — build from the device-level + coexistence of multiple mediated devices and guests to per-guest execution + isolation. +4. **Hold the acceptance battery** — vector-add, matrix-multiply and the + transformer kernel run as a battery on every re-measurement; each entry is + accepted only on its own verification output (`bad=0` + `STATUS OK`). --- -## 7. Theories (labelled; not yet proven) - -* **T1 — the virtualization control is the gate.** The front-end will not - materialise an inline descriptor until the virtualization-control method - enables the install path; its absence alone explains the inert descriptor. -* **T2 — materialise-then-read.** The engine only schedules a descriptor that - exists in its own view; an install targeting unmapped destinations aborts - silently rather than faulting. -* **T3 — the verbatim prelude poisons the launch.** Even where the tail shape - works, the prelude's refused writes leave the launch path in a state that - prevents the descriptor from being scheduled. -* **T4 — relocation is the missing transform.** The guest's inline descriptor - is valid, but it is built for the guest's view of memory; the forwarding path - must relocate referenced data (and any embedded pointers) into the channel's - view before install. - -Each theory has a one-cycle falsifier: the install sentinel, the engine-read -signal, and the engine-authored release, in that order. +## 7. Theories, resolved + +The descriptor-gate theories (T1–T4) are resolved and are not kept as open: +the inline-descriptor install, the engine's QMD read, the grid dispatch, the +engine-authored release and the guest wake-up are all directly observed (§1), +so the launch chain they questioned is now proven end to end. + +The remaining leading-edge work is engineering rather than theory: the +large-buffer transfer class, boot-to-boot reproducibility, and per-guest +execution isolation. Each has an acceptance signal named in §5 before the +work. --- ## 8. Claim discipline -* **Engine-authored release is the only proof of execution.** Front-end - signals (ring cursor, method execution, semaphores that the front-end can - author) are necessary but not sufficient. +* **Engine-authored release is the acceptance signal for execution** — now + observed end to end for the verified clients; the guest's own verification + (`bad=0` + `STATUS OK`) is the guest-side acceptance signal. +* **Front-end signals are necessary but not sufficient.** Ring cursor and + front-end method execution can be front-end-authored; they never substitute + for the engine-authored release. * **Silence is not health.** No fault recorded does not mean no fault occurred; the current GR path runs without an error-reporting object, which is itself a known observability gap. * **Emulator verdicts are L0/L1.** They bound behaviour, never timing or device environment. +* **A green run counts only when it reproduces across fresh boots.** The + boot-poisoning defect is the worked example: one green boot is not + reproducibility. * **A wall is closed only by an acceptance signal**, and the acceptance signal for this frontier is named before the experiment. @@ -218,3 +247,4 @@ signal, and the engine-authored release, in that order. | Date | Change | |---|---| | 2026-10-06 | Created: the GR compute-launch chain, the leading edge (inline-descriptor install), evidence limits, exonerated walls, ranked issues, live investigation lines and theories. | +| 2026-10-20 | Chain complete: verified guest CUDA clients execute on the physical GPU (vector-add, matrix-multiply, transformer; engine-authored completions relay and guest waits resolve). Leading edge moved to the large-buffer staged-transfer class, fresh-boot reproducibility (boot-poisoning defect fixed) and per-guest execution isolation; the descriptor-gate theories are resolved. | diff --git a/docs/04-implementation/04-real-render-walls.md b/docs/04-implementation/04-real-render-walls.md new file mode 100644 index 0000000..3570769 --- /dev/null +++ b/docs/04-implementation/04-real-render-walls.md @@ -0,0 +1,113 @@ +# Real-render walls: Vulkan and Xorg on the emulated vGPU + +Status: OPEN (2026-10-29, SESSION 64). This page records the measured state +of the guest's *display/graphics* consumers - the Vulkan ICD and the Xorg +server - on the emulated vGPU, with the exact evidence and the next +subsystem to implement. It complements +[05-operations/06-guest-frame-path-and-looking-glass.md](../05-operations/06-guest-frame-path-and-looking-glass.md) +(which covers the CUDA -> ivshmem -> host frame transport that is already +proven). + +## 1. What works today + +* The guest's shipped 570.144 Vulkan userspace **loads and enumerates**: + `vkCreateInstance` succeeds and two physical devices are reported - + `NVIDIA GeForce RTX 2080 SUPER` (discrete, Vulkan 1.4.303) and the + software `llvmpipe` - from `/etc/vulkan/icd.d/nvidia_icd.json`. Probes: + `tools/vk/vk_probe.py`, `tools/vk/vk_probe2.py`. +* `eglInitialize` succeeds with `vendor=NVIDIA` on the guest EGL device + platform (the EGL probe `tools/guest-diag/egl_retry.c` and the older + `egl_frame`). +* CUDA is unaffected (its clients run through the same emulated GSP; see + the battery evidence in the handoff). + +## 2. Wall 1 - `vkCreateDevice` fails in nvidia-modeset's ALLOC_DEVICE + +`vkCreateDevice` returns `VK_ERROR_INITIALIZATION_FAILED`. An +`LD_PRELOAD` ioctl logger (`tools/guest-diag/nvkmstrace.c`) shows the +ICD's first nvKms ioctl failing: + +``` +NVKMS_IOCTL cmd=0 size=1488 addr=0x... rc=-1 # NVKMS_IOCTL_ALLOC_DEVICE +``` + +i.e. the kernel's `nvKmsAllocDevice` (in `nvidia-modeset`) returned a +non-success status; the userspace frees everything and reports the +generic initialization error. Reading the driver's own modeset source +(`nvkms-rm.c`) gives the alloc order and the two gates we hit and fixed: + +1. `ProbeHeadCountAndWindowAssignment()` -> `NV0073_CTRL_CMD_SYSTEM_GET_NUM_HEADS` + (`0x730102`): a zero reply logs `No heads found on board!` and fails the + alloc. The emulated GSP RSP answered all NV0073 controls through the + generic zero-reply path. +2. After the heads: `NoValidDisplays()` -> `NV0073_CTRL_CMD_SYSTEM_GET_SUPPORTED` + (`0x730107`) must report at least one display id, and the connector + path (`SPECIFIC_OR_GET_INFO` `0x73028b`, `SPECIFIC_GET_CONNECTOR_DATA` + `0x730250`, `DFP_GET_INFO` `0x731140`) must describe a connector. + +The host GSP-RM emulation now implements a minimal headless display: +one head (`0x730102`/`0x730287`), flexible window assignment (`0x7302ad`), +one static HDMI connector on display id 0 (SOR + single-link TMDS, which +avoids the DP-AUX library), plus DDC support. With this shim the alloc +gets past heads and `NoValidDisplays` and proceeds through the connector +queries and the display-object allocations (classes `0xc570`, `0xc57d`, +`0xc372`, controls `0x50700709`, `0x73010c`, `0xc3700201`) before failing +later in the display-core/modeset-channel path. That remaining subsystem +is the wall: the modeset core channel and the per-display dpy objects +must be emulated (a coupled object + channel + memory set whose exact +wire forms should be captured from a real-GPU session). + +A related experiment ruled out one shortcut: booting the guest with +`nvidia-modeset`/`nvidia-drm` blacklisted (no `/dev/nvidia-modeset`) does +not avoid the wall - the ICD's nvKms client still issues ALLOC_DEVICE and +fails, so the display path cannot be sidestepped by disabling KMS. + +## 3. Wall 2 - Xorg ScreenInit fails in nvidia-push's channel test + +Xorg starts (the GPU is detected, `Virtual screen 1280x1024`) and then: + +``` +(EE) NVIDIA(GPU-0): Failed to initialize DMA. +(EE) NVIDIA(0): Failed to allocate push buffer +``` + +`nvidia-push` logs `Failed to initialize DMA.` from +`__nvPushTestPushBuffer()`: right after allocating a push buffer the +driver writes a NOP and waits (10 s) for the channel to consume it. Host +evidence shows why it never does: + +* the X driver's channel registers (`class 0xc46f` GPFIFO channel, + `gpfifo=0x120090000`, `userd=0x350000` sysmem) and lives with + **`submissions=0`** - its published GP_PUT never advances in the host's + view, because the host needs the channel's VA->PA translations to read + its USERD/GPFIFO and that resolution comes from the guest-side scanner + that currently follows *CUDA clients only* (`targeted resolve + va=0x20400f000: full RAM scan found nothing`); +* on a busy boot (Xorg started while a CUDA battery ran) the channel's + completion-counter slot is rejected: `counter auto h=0x1000013 rank=0 + pa=0x11c827ff0 rejected (slot owned by another channel)` - two + concurrent clients both derive rank 0's slot from the per-device + structural formula. + +Fixing Xorg therefore needs (1) a platform-wide (non-CUDA) VA-resolution +path for the scanner or the host, and (2) counter-slot ownership keyed by +client identity (or a per-client slot domain), not just rank. + +## 4. Wall 3 - the EGL/glsi engine-init gate (unchanged) + +`eglMakeCurrent` returns `EGL_BAD_ALLOC (0x3003)`; the glsi engine-init +entry returns a "busy" status once, sets a state bit, and the eglcore +caller does not retry. Ten in-process retries (`egl_retry`) all fail, so +the state machine fails on a real input - the return-value trace of its +callees remains the way in (see +[userspace/cuda/570.144/docs/ANALYSIS-egl-engine-selection.md](../../userspace/cuda/570.144/docs/ANALYSIS-egl-engine-selection.md)). + +## 5. Why this matters for the frame path + +Everything downstream of a working GL/Vulkan context is already in place: +a guest publisher writes frames into the kvmfr ivshmem BAR +(`tools/vk/vk_frame.c`, `tools/guest-diag/xshm_pub.c` use the same layout +as the proven CUDA `frame_producer`), the host reader decodes them +(`tools/lg/frame_reader.py`) and ffmpeg assembles video. The missing +piece is a *working* graphics API on the guest GPU - the two subsystems +above.
\ No newline at end of file |