diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/03-architecture/13-vexecution-plane.md | 33 | ||||
| -rw-r--r-- | docs/04-implementation/04-real-render-walls.md | 187 | ||||
| -rw-r--r-- | docs/05-operations/06-guest-frame-path-and-looking-glass.md | 19 | ||||
| -rw-r--r-- | docs/README.md | 1 |
4 files changed, 240 insertions, 0 deletions
diff --git a/docs/03-architecture/13-vexecution-plane.md b/docs/03-architecture/13-vexecution-plane.md index caaad01..3d97d07 100644 --- a/docs/03-architecture/13-vexecution-plane.md +++ b/docs/03-architecture/13-vexecution-plane.md @@ -568,6 +568,39 @@ forwards (local evidence and the prepared harness: forwarding-service cost, not engine time — the number to drive down is the 7.9 ms and the per-window round trips. +### 3.5a Copy-out change detection: fingerprint rule + +A forwarded launch stages every referenced data page into the data-bounce +chunk, and after the retire wait the copy-out ships back only the pages the +engine actually modified. That decision used to be a 32-bit additive +checksum of the page's words, compared against a baseline captured at +copy-in. The additive form has a structural blind spot: doubling every word +of a 4 KB page raises each float exponent by one (+2^23 per word), and 1024 +words contribute `1024 * 2^23 = 2^33`, which is exactly `0 (mod 2^32)` — an +in-place power-of-two scale of a whole page (`c[i] *= 2.0f`) left the +checksum bit-identical, the page was classified "engine did not touch", +the write-back was skipped, and the client read the pre-scale values. The +same blindness applies to `*= 0.5f`, `* 4`, `* 8`, etc. Since `r452` +(commit `19399f4f`) every change decision in the copy path — the retire +wait's settle scan, the tracked and gap copy-outs, the window witness, the +late-store witness and the idempotent-launch fingerprint — uses a 64-bit +FNV-1a hash over the page instead; the additive sum survives only as the +all-zero filter. Policy rule for the plane: **a change-detection +fingerprint must not be an algebraic invariant of any plausible kernel +operation** — use a hash, never a sum. + +The reader side of the same decision is the landing table: a D2H copy of a +device VA resolves through the last recorded write of that VA. Since `r453` +the forwarded QMD's release0 is redirected to the module semaphore by +default (`gsp_phys_rmcall_gr_rel_mod=1`): the guest-visible release word is +authored by the host only on the completion edge, after the copy-out has +shipped the engine's stores, so a client's stream-ordered readback can no +longer run ahead of the write-back (the historical shape: the engine's own +release satisfied the guest while the copy-out was still pending). The +engine-completion observable remains the module semaphore, and the +per-outcome authoring path guarantees the guest word is applied on every +outcome (timeout, wedge, no-write). + ### 3.6 Citation map for this section | Claim | Authority | diff --git a/docs/04-implementation/04-real-render-walls.md b/docs/04-implementation/04-real-render-walls.md new file mode 100644 index 0000000..f6ea39d --- /dev/null +++ b/docs/04-implementation/04-real-render-walls.md @@ -0,0 +1,187 @@ +# Real-render walls: Vulkan and Xorg on the emulated vGPU + +Status: OPEN (2026-10-29, SESSION 64). This page records the measured state +of the guest's *display/graphics* consumers - the Vulkan ICD and the Xorg +server - on the emulated vGPU, with the exact evidence and the next +subsystem to implement. It complements +[05-operations/06-guest-frame-path-and-looking-glass.md](../05-operations/06-guest-frame-path-and-looking-glass.md) +(which covers the CUDA -> ivshmem -> host frame transport that is already +proven). + +## 1. What works today + +* The guest's shipped 570.144 Vulkan userspace **loads and enumerates**: + `vkCreateInstance` succeeds and two physical devices are reported - + `NVIDIA GeForce RTX 2080 SUPER` (discrete, Vulkan 1.4.303) and the + software `llvmpipe` - from `/etc/vulkan/icd.d/nvidia_icd.json`. Probes: + `tools/vk/vk_probe.py`, `tools/vk/vk_probe2.py`. +* `eglInitialize` succeeds with `vendor=NVIDIA` on the guest EGL device + platform (the EGL probe `tools/guest-diag/egl_retry.c` and the older + `egl_frame`). +* CUDA is unaffected (its clients run through the same emulated GSP; see + the battery evidence in the handoff). + +## 2. Wall 1 - `vkCreateDevice` fails in nvidia-modeset's ALLOC_DEVICE + +`vkCreateDevice` returns `VK_ERROR_INITIALIZATION_FAILED`. An +`LD_PRELOAD` ioctl logger (`tools/guest-diag/nvkmstrace.c`) shows the +ICD's first nvKms ioctl failing: + +``` +NVKMS_IOCTL cmd=0 size=1488 addr=0x... rc=-1 # NVKMS_IOCTL_ALLOC_DEVICE +``` + +i.e. the kernel's `nvKmsAllocDevice` (in `nvidia-modeset`) returned a +non-success status; the userspace frees everything and reports the +generic initialization error. Reading the driver's own modeset source +(`nvkms-rm.c`) gives the alloc order and the two gates we hit and fixed: + +1. `ProbeHeadCountAndWindowAssignment()` -> `NV0073_CTRL_CMD_SYSTEM_GET_NUM_HEADS` + (`0x730102`): a zero reply logs `No heads found on board!` and fails the + alloc. The emulated GSP RSP answered all NV0073 controls through the + generic zero-reply path. +2. After the heads: `NoValidDisplays()` -> `NV0073_CTRL_CMD_SYSTEM_GET_SUPPORTED` + (`0x730107`) must report at least one display id, and the connector + path (`SPECIFIC_OR_GET_INFO` `0x73028b`, `SPECIFIC_GET_CONNECTOR_DATA` + `0x730250`, `DFP_GET_INFO` `0x731140`) must describe a connector. + +The host GSP-RM emulation now implements a minimal headless display: +one head (`0x730102`/`0x730287`), flexible window assignment (`0x7302ad`), +one static HDMI connector on display id 0 (SOR + single-link TMDS, which +avoids the DP-AUX library), plus DDC support. With this shim the alloc +gets past heads and `NoValidDisplays` and proceeds through the connector +queries and the display-object allocations (classes `0xc570`, `0xc57d`, +`0xc372`, controls `0x50700709`, `0x73010c`, `0xc3700201`) before failing +later in the display-core/modeset-channel path. That remaining subsystem +is the wall: the modeset core channel and the per-display dpy objects +must be emulated (a coupled object + channel + memory set whose exact +wire forms should be captured from a real-GPU session). + +A related experiment ruled out one shortcut: booting the guest with +`nvidia-modeset`/`nvidia-drm` blacklisted (no `/dev/nvidia-modeset`) does +not avoid the wall - the ICD's nvKms client still issues ALLOC_DEVICE and +fails, so the display path cannot be sidestepped by disabling KMS. + +## 3. Wall 2 - Xorg ScreenInit fails in nvidia-push's channel test + +Xorg starts (the GPU is detected, `Virtual screen 1280x1024`) and then: + +``` +(EE) NVIDIA(GPU-0): Failed to initialize DMA. +(EE) NVIDIA(0): Failed to allocate push buffer +``` + +`nvidia-push` logs `Failed to initialize DMA.` from +`__nvPushTestPushBuffer()`: right after allocating a push buffer the +driver writes a NOP and waits (10 s) for the channel to consume it. Host +evidence shows why it never does: + +* the X driver's channel registers (`class 0xc46f` GPFIFO channel, + `gpfifo=0x120090000`, `userd=0x350000` sysmem) and lives with + **`submissions=0`** - its published GP_PUT never advances in the host's + view, because the host needs the channel's VA->PA translations to read + its USERD/GPFIFO and that resolution comes from the guest-side scanner + that currently follows *CUDA clients only* (`targeted resolve + va=0x20400f000: full RAM scan found nothing`); +* on a busy boot (Xorg started while a CUDA battery ran) the channel's + completion-counter slot is rejected: `counter auto h=0x1000013 rank=0 + pa=0x11c827ff0 rejected (slot owned by another channel)` - two + concurrent clients both derive rank 0's slot from the per-device + structural formula. + +Fixing Xorg therefore needs (1) a platform-wide (non-CUDA) VA-resolution +path for the scanner or the host, and (2) counter-slot ownership keyed by +client identity (or a per-client slot domain), not just rank. + +## 4. Wall 3 - the EGL/glsi engine-init gate (unchanged) + +`eglMakeCurrent` returns `EGL_BAD_ALLOC (0x3003)`; the glsi engine-init +entry returns a "busy" status once, sets a state bit, and the eglcore +caller does not retry. Ten in-process retries (`egl_retry`) all fail, so +the state machine fails on a real input - the return-value trace of its +callees remains the way in (see +[userspace/cuda/570.144/docs/ANALYSIS-egl-engine-selection.md](../../userspace/cuda/570.144/docs/ANALYSIS-egl-engine-selection.md)). + +## 5. Why this matters for the frame path + +Everything downstream of a working GL/Vulkan context is already in place: +a guest publisher writes frames into the kvmfr ivshmem BAR +(`tools/vk/vk_frame.c`, `tools/guest-diag/xshm_pub.c` use the same layout +as the proven CUDA `frame_producer`), the host reader decodes them +(`tools/lg/frame_reader.py`) and ffmpeg assembles video. The missing +piece is a *working* graphics API on the guest GPU - the two subsystems +above. +## 6. UPDATE 2026-10-25 (RESUME #56): the wall is now fully bisected + +Three new instruments settled the remaining questions: + +* **An `LD_PRELOAD` nvKms ioctl logger** (`tools/guest-diag/nvkmstrace.c`) + decodes the failing `NVKMS_IOCTL_ALLOC_DEVICE` reply: status **5 = + `NVKMS_ALLOC_DEVICE_STATUS_CORE_CHANNEL_ALLOC_FAILED`**. +* **The QEMU gdb stub (`-gdb tcp::1234`) as a guest-kernel microscope** + (the guest kernel's `nvidia_modeset` symbols are in `/proc/kallsyms`): + breaking inside `nvAllocCoreChannelEvo` (the blob's offsets were mapped + from a disassembly of the shipped `nv-modeset-kernel.o_binary`) dumped + the driver's class list (42 classes; all `NVC57x` display classes are + present; **`NV927C_BASE_CHANNEL_DMA` (0x927c) is not** - it exists in no + HAL class list, so the "base channel" fallback path can never succeed). + Breaking at the capability parse showed the **capability notifier buffer + is all zeros** (nothing emulates the display engine that would fill it). +* **The caps-fill experiment**: with gdb writing `SYS_CAP = HEAD0_EXISTS` + and `SYS_CAPB = WINDOW0..7_EXISTS` into the caps buffer right before the + parse, the driver flips from the dead base path to the **window path** + (`nvRMAllocateWindowChannels`) - the hypothesis that `numWindows > 0` + unblocks the flow is confirmed. + +The window path then fails one step later: `nvRmApiAlloc(display, +class=0xc57e NVC57E_WINDOW_CHANNEL_DMA)` returns **`NV_ERR_NOT_SUPPORTED` +(0x56) locally in the guest kernel** (it never reaches the emulated +GSP - no ALLOC RPC is logged). This matches the shipped driver's vGPU +architecture: the guest runs in vGPU mode (`IS_VIRTUAL`), where the +display/modeset channel classes are rejected (the reference source's +`_rpcAllocObjectPrologue` documents "Display classes are added for DCE, +vGPU Guest does not support those") - a vGPU guest's display is meant to +be emulated by the **host-side vGPU display service** (the vGPU RPC +display path), which the emulation does not implement. + +**Consequence for the roadmap.** Guest Xorg/GL/Vulkan on this emulated +vGPU needs (a) the display capability fill on the host side (mechanics +proven by the gdb experiment) and (b) a vGPU display service: the +display-engine MMIO/RPC backend the guest expects from its host (the +equivalent of NVIDIA's vGPU display plugin), or the specific vGPU +configuration that makes the kernel accept the display channel class. +Until (b) exists the guest's NVIDIA display path is closed by design, and +GZDOOM-class workloads cannot render on the guest GPU. +## 7. UPDATE 2026-10-25 #2 (SESSION 66): the 0x56 pinned to the RS alloc path + +A new instrument closed the gap: the guest's `nvidia.ko` (DKMS build) is +**not stripped** - copy it out (`/lib/modules/<kver>/updates/dkms/nvidia.ko`), +`objdump -t`/`-dr` it, compute the module's `.text` runtime base from any +`/proc/kallsyms` symbol, and every guest-kernel function/offset becomes +addressable for live gdb breakpoints on the QEMU stub. With it: + +* `rpcAllocObject_v29_06`/`_rpcAllocObjectPrologue` are **exonerated** for + 0xc57e (their reject list is 0x73/0xc67*/0xc770/0xc77d only; the register + mapping `hClass = r8d` was verified live). +* The failing chain is pinned exactly: `rmapiAllocWithSecInfo` **enters** + with class 0xc57e, calls `serverAllocResourceUnderLock`, and that + function returns **0x56** - *before* the GSP-RM alloc RPC + (`rpcRmApiAlloc_GSP`, called from under-lock+0x447; the emulation never + sees an ALLOC for 0xc57e). The same probe shows 0xc57d/0xc570/0xc573 + return 0. +* So the last blocker for the window path is a **local RS-layer validation + inside `serverAllocResourceUnderLock`** (candidates: the inlined + privilege/class-support checks, `rmapiFixupAllocParams`' per-class fixup, + or `clientAllocResource`'s construct dispatch). The 0x56-producing + instruction is not yet pinned - a `stepi` loop over the QEMU stub is far + too slow and wedged the guest (recovered by a fresh cycle); the next + session should use conditional breakpoints on the function's call-return + sites (the approach in `gdbdiag19` works; the earlier run had truncated + addresses and needs the `0xffffffff00000000` base restored) or a + watchpoint on the status word. + +**gdb-stub field notes**: the stub is single-client; killing an attached +gdb leaves the VM stopped (recover with a detach-only connect); +`stepi` loops over the stub freeze the guest's userspace for minutes - +prefer breakpoints; every probe run with the caps-fill breakpoint changes +the driver's path, so keep the fill in each experiment. diff --git a/docs/05-operations/06-guest-frame-path-and-looking-glass.md b/docs/05-operations/06-guest-frame-path-and-looking-glass.md index 3dda4f1..bc5447a 100644 --- a/docs/05-operations/06-guest-frame-path-and-looking-glass.md +++ b/docs/05-operations/06-guest-frame-path-and-looking-glass.md @@ -81,3 +81,22 @@ python3 tools/lg/frame_reader.py --frames 5 --out /tmp/opencode/frames gcc -O2 -o egl_frame tools/guest-diag/egl_frame.c -lEGL -lGLESv2 ./egl_frame ``` + +## 5. Session-13 update (2026-10-21, r432→r441) + +* The single-frame path is unchanged and still proven: guest `frame_producer` + (CUDA Julia) -> kvmfr ivshmem BAR -> host `frame_reader.py` -> + `/tmp/opencode/frames/frame-0001.png` (320x240, 1431 colours). +* `tools/lg/frame_sequence.sh` captures an **animated** sequence end to end + (reader first, then N frames from the guest). It is blocked by the + GR no-execute wedge: when the lane is wedged the render launches write zero + pages and the producer republishes stale identical frames (observed: 6/6 + identical MD5s on r438; r439 published 5 identical plus one frame left in + the BAR by the previous boot). A multi-frame animation needs a healthy + lane; the frame pipeline itself is not the blocker. +* The GL wall is untouched: `Xorg :0` -> `(EE) NVIDIA(GPU-0): Failed to + allocate VASpace object`, guest dmesg `NVRM: vaspaceapiConstruct_IMPL: + Could not construct VA space. Status ffff`, i.e. `vmmCreateVaspace` inside + the guest's own RM. Next step: reproduce with the guest RM's + `ResmanDebugLevel` raised (a guest module reload, the service's m177 path) + to capture which inner call returns the generic error. diff --git a/docs/README.md b/docs/README.md index 27e04e9..b423762 100644 --- a/docs/README.md +++ b/docs/README.md @@ -184,6 +184,7 @@ Read in order if you are new. Jump directly if you know what you need. | 24 | [risc-v-gsp-rm-debugger user guide (experimental)](05-operations/04-risc-v-gsp-rm-debugger-user-guide.md) — implemented, documented and tested ([audit](../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md); [value demos](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md)) | developers, analysts | 15 min | | 25 | [Defensive Operations Runbook](RUNBOOK.md) | maintainers, operators | 20 min | | 25a | [GPUBreach-class audit and threat model](05-operations/05-gpubreach-audit.md) | developers, auditors | 15 min | +| 25b | [Real-render walls: Vulkan and Xorg on the emulated vGPU](04-implementation/04-real-render-walls.md) | developers, debuggers | 10 min | | 26 | [Engineering operations](eng-ops.md) — tool admission and workflows (Part I) plus live physical-backend operations (Part II) | maintainers, developers | 20 min | | 27 | [Hardware and software matrix](06-portability/hardware-software-matrix.md) | developers, integrators | 20 min | | 28 | [Developer onboarding and standards](07-developers/onboarding-and-standards.md) | contributors | 20 min | |