path: root/docs/04-implementation/04-real-render-walls.md

# Real-render walls: Vulkan and Xorg on the emulated vGPU

Status: OPEN (2026-10-29, SESSION 64). This page records the measured state
of the guest's *display/graphics* consumers - the Vulkan ICD and the Xorg
server - on the emulated vGPU, with the exact evidence and the next
subsystem to implement. It complements
[05-operations/06-guest-frame-path-and-looking-glass.md](../05-operations/06-guest-frame-path-and-looking-glass.md)
(which covers the CUDA -> ivshmem -> host frame transport that is already
proven).

## 1. What works today

* The guest's shipped 570.144 Vulkan userspace **loads and enumerates**:
  `vkCreateInstance` succeeds and two physical devices are reported -
  `NVIDIA GeForce RTX 2080 SUPER` (discrete, Vulkan 1.4.303) and the
  software `llvmpipe` - from `/etc/vulkan/icd.d/nvidia_icd.json`. Probes:
  `tools/vk/vk_probe.py`, `tools/vk/vk_probe2.py`.
* `eglInitialize` succeeds with `vendor=NVIDIA` on the guest EGL device
  platform (the EGL probe `tools/guest-diag/egl_retry.c` and the older
  `egl_frame`).
* CUDA is unaffected (its clients run through the same emulated GSP; see
  the battery evidence in the handoff).

## 2. Wall 1 - `vkCreateDevice` fails in nvidia-modeset's ALLOC_DEVICE

`vkCreateDevice` returns `VK_ERROR_INITIALIZATION_FAILED`. An
`LD_PRELOAD` ioctl logger (`tools/guest-diag/nvkmstrace.c`) shows the
ICD's first nvKms ioctl failing:

```
NVKMS_IOCTL cmd=0 size=1488 addr=0x... rc=-1   # NVKMS_IOCTL_ALLOC_DEVICE
```

i.e. the kernel's `nvKmsAllocDevice` (in `nvidia-modeset`) returned a
non-success status; the userspace frees everything and reports the
generic initialization error. Reading the driver's own modeset source
(`nvkms-rm.c`) gives the alloc order and the two gates we hit and fixed:

1. `ProbeHeadCountAndWindowAssignment()` -> `NV0073_CTRL_CMD_SYSTEM_GET_NUM_HEADS`
   (`0x730102`): a zero reply logs `No heads found on board!` and fails the
   alloc. The emulated GSP RSP answered all NV0073 controls through the
   generic zero-reply path.
2. After the heads: `NoValidDisplays()` -> `NV0073_CTRL_CMD_SYSTEM_GET_SUPPORTED`
   (`0x730107`) must report at least one display id, and the connector
   path (`SPECIFIC_OR_GET_INFO` `0x73028b`, `SPECIFIC_GET_CONNECTOR_DATA`
   `0x730250`, `DFP_GET_INFO` `0x731140`) must describe a connector.

The host GSP-RM emulation now implements a minimal headless display:
one head (`0x730102`/`0x730287`), flexible window assignment (`0x7302ad`),
one static HDMI connector on display id 0 (SOR + single-link TMDS, which
avoids the DP-AUX library), plus DDC support. With this shim the alloc
gets past heads and `NoValidDisplays` and proceeds through the connector
queries and the display-object allocations (classes `0xc570`, `0xc57d`,
`0xc372`, controls `0x50700709`, `0x73010c`, `0xc3700201`) before failing
later in the display-core/modeset-channel path. That remaining subsystem
is the wall: the modeset core channel and the per-display dpy objects
must be emulated (a coupled object + channel + memory set whose exact
wire forms should be captured from a real-GPU session).

A related experiment ruled out one shortcut: booting the guest with
`nvidia-modeset`/`nvidia-drm` blacklisted (no `/dev/nvidia-modeset`) does
not avoid the wall - the ICD's nvKms client still issues ALLOC_DEVICE and
fails, so the display path cannot be sidestepped by disabling KMS.

## 3. Wall 2 - Xorg ScreenInit fails in nvidia-push's channel test

Xorg starts (the GPU is detected, `Virtual screen 1280x1024`) and then:

```
(EE) NVIDIA(GPU-0): Failed to initialize DMA.
(EE) NVIDIA(0): Failed to allocate push buffer
```

`nvidia-push` logs `Failed to initialize DMA.` from
`__nvPushTestPushBuffer()`: right after allocating a push buffer the
driver writes a NOP and waits (10 s) for the channel to consume it. Host
evidence shows why it never does:

* the X driver's channel registers (`class 0xc46f` GPFIFO channel,
  `gpfifo=0x120090000`, `userd=0x350000` sysmem) and lives with
  **`submissions=0`** - its published GP_PUT never advances in the host's
  view, because the host needs the channel's VA->PA translations to read
  its USERD/GPFIFO and that resolution comes from the guest-side scanner
  that currently follows *CUDA clients only* (`targeted resolve
  va=0x20400f000: full RAM scan found nothing`);
* on a busy boot (Xorg started while a CUDA battery ran) the channel's
  completion-counter slot is rejected: `counter auto h=0x1000013 rank=0
  pa=0x11c827ff0 rejected (slot owned by another channel)` - two
  concurrent clients both derive rank 0's slot from the per-device
  structural formula.

Fixing Xorg therefore needs (1) a platform-wide (non-CUDA) VA-resolution
path for the scanner or the host, and (2) counter-slot ownership keyed by
client identity (or a per-client slot domain), not just rank.

## 4. Wall 3 - the EGL/glsi engine-init gate (unchanged)

`eglMakeCurrent` returns `EGL_BAD_ALLOC (0x3003)`; the glsi engine-init
entry returns a "busy" status once, sets a state bit, and the eglcore
caller does not retry. Ten in-process retries (`egl_retry`) all fail, so
the state machine fails on a real input - the return-value trace of its
callees remains the way in (see
[userspace/cuda/570.144/docs/ANALYSIS-egl-engine-selection.md](../../userspace/cuda/570.144/docs/ANALYSIS-egl-engine-selection.md)).

## 5. Why this matters for the frame path

Everything downstream of a working GL/Vulkan context is already in place:
a guest publisher writes frames into the kvmfr ivshmem BAR
(`tools/vk/vk_frame.c`, `tools/guest-diag/xshm_pub.c` use the same layout
as the proven CUDA `frame_producer`), the host reader decodes them
(`tools/lg/frame_reader.py`) and ffmpeg assembles video. The missing
piece is a *working* graphics API on the guest GPU - the two subsystems
above.
## 6. UPDATE 2026-10-25 (RESUME #56): the wall is now fully bisected

Three new instruments settled the remaining questions:

* **An `LD_PRELOAD` nvKms ioctl logger** (`tools/guest-diag/nvkmstrace.c`)
  decodes the failing `NVKMS_IOCTL_ALLOC_DEVICE` reply: status **5 =
  `NVKMS_ALLOC_DEVICE_STATUS_CORE_CHANNEL_ALLOC_FAILED`**.
* **The QEMU gdb stub (`-gdb tcp::1234`) as a guest-kernel microscope**
  (the guest kernel's `nvidia_modeset` symbols are in `/proc/kallsyms`):
  breaking inside `nvAllocCoreChannelEvo` (the blob's offsets were mapped
  from a disassembly of the shipped `nv-modeset-kernel.o_binary`) dumped
  the driver's class list (42 classes; all `NVC57x` display classes are
  present; **`NV927C_BASE_CHANNEL_DMA` (0x927c) is not** - it exists in no
  HAL class list, so the "base channel" fallback path can never succeed).
  Breaking at the capability parse showed the **capability notifier buffer
  is all zeros** (nothing emulates the display engine that would fill it).
* **The caps-fill experiment**: with gdb writing `SYS_CAP = HEAD0_EXISTS`
  and `SYS_CAPB = WINDOW0..7_EXISTS` into the caps buffer right before the
  parse, the driver flips from the dead base path to the **window path**
  (`nvRMAllocateWindowChannels`) - the hypothesis that `numWindows > 0`
  unblocks the flow is confirmed.

The window path then fails one step later: `nvRmApiAlloc(display,
class=0xc57e NVC57E_WINDOW_CHANNEL_DMA)` returns **`NV_ERR_NOT_SUPPORTED`
(0x56) locally in the guest kernel** (it never reaches the emulated
GSP - no ALLOC RPC is logged). This matches the shipped driver's vGPU
architecture: the guest runs in vGPU mode (`IS_VIRTUAL`), where the
display/modeset channel classes are rejected (the reference source's
`_rpcAllocObjectPrologue` documents "Display classes are added for DCE,
vGPU Guest does not support those") - a vGPU guest's display is meant to
be emulated by the **host-side vGPU display service** (the vGPU RPC
display path), which the emulation does not implement.

**Consequence for the roadmap.** Guest Xorg/GL/Vulkan on this emulated
vGPU needs (a) the display capability fill on the host side (mechanics
proven by the gdb experiment) and (b) a vGPU display service: the
display-engine MMIO/RPC backend the guest expects from its host (the
equivalent of NVIDIA's vGPU display plugin), or the specific vGPU
configuration that makes the kernel accept the display channel class.
Until (b) exists the guest's NVIDIA display path is closed by design, and
GZDOOM-class workloads cannot render on the guest GPU.
## 7. UPDATE 2026-10-25 #2 (SESSION 66): the 0x56 pinned to the RS alloc path

A new instrument closed the gap: the guest's `nvidia.ko` (DKMS build) is
**not stripped** - copy it out (`/lib/modules/<kver>/updates/dkms/nvidia.ko`),
`objdump -t`/`-dr` it, compute the module's `.text` runtime base from any
`/proc/kallsyms` symbol, and every guest-kernel function/offset becomes
addressable for live gdb breakpoints on the QEMU stub. With it:

* `rpcAllocObject_v29_06`/`_rpcAllocObjectPrologue` are **exonerated** for
  0xc57e (their reject list is 0x73/0xc67*/0xc770/0xc77d only; the register
  mapping `hClass = r8d` was verified live).
* The failing chain is pinned exactly: `rmapiAllocWithSecInfo` **enters**
  with class 0xc57e, calls `serverAllocResourceUnderLock`, and that
  function returns **0x56** - *before* the GSP-RM alloc RPC
  (`rpcRmApiAlloc_GSP`, called from under-lock+0x447; the emulation never
  sees an ALLOC for 0xc57e). The same probe shows 0xc57d/0xc570/0xc573
  return 0.
* So the last blocker for the window path is a **local RS-layer validation
  inside `serverAllocResourceUnderLock`** (candidates: the inlined
  privilege/class-support checks, `rmapiFixupAllocParams`' per-class fixup,
  or `clientAllocResource`'s construct dispatch). The 0x56-producing
  instruction is not yet pinned - a `stepi` loop over the QEMU stub is far
  too slow and wedged the guest (recovered by a fresh cycle); the next
  session should use conditional breakpoints on the function's call-return
  sites (the approach in `gdbdiag19` works; the earlier run had truncated
  addresses and needs the `0xffffffff00000000` base restored) or a
  watchpoint on the status word.

**gdb-stub field notes**: the stub is single-client; killing an attached
gdb leaves the VM stopped (recover with a detach-only connect);
`stepi` loops over the stub freeze the guest's userspace for minutes -
prefer breakpoints; every probe run with the caps-fill breakpoint changes
the driver's path, so keep the fill in each experiment.

## 8. UPDATE 2026-10-25 #3 (SESSION 126): the fence fold landed - the service authors the Vulkan fence releases from the driver's own push bytes; the compute-dispatch shape decoded (direct C5C0 method stream, no inline QMD)

**The fence wall is closed.** The client's per-frame fence wait polls a
16 KB BO word that a submit-push INC-5 semaphore group targets
(`[2005c017][0x45ba090][0][payload 1][0][0x02100001]`). The group sits in a
GPFIFO entry beyond the driver's kick boundary (GP_PUT=11 while the ring
already carries the entries it pre-built for the next submits, idx12..46),
so the GP_PUT-bounded pump walk never decoded it and a host-side assist had
to write the word. The service now sweeps that pre-built continuation
itself: `sema_gr_ahead` (default 64) walks entries beyond GP_PUT through the
ring's validated translation, accepts only pushes that pass the strict
method-stream gate, extracts the addr-0x5c RELEASE groups and applies them
forward-only through the standard pending ledger. Live evidence: the sweep
finds the groups and the ledger applies the fence word once per frame;
`VKFRAME seq=1..3/DONE` completes with no assist. The frames are still
black - the shader work itself is unexecuted.

**The compute-dispatch shape (why nothing executes).** The workload is a
single compute dispatch (640x480 storage image). Its push is a DIRECT
method-stream programming of the Turing compute class: `SET_OBJECT 0xC5C0`
on subchannel 1, then the context windows (`0x7b4` local-memory window B,
`0x2a4` shared-memory window B, `0x270`), an immediate-config block, and
the code/const data runs (`0x3800`/`0x39C8` pairs plus a subchannel-0 data
port stream of `0x11c`/`0x114` runs in a separate 17 KB push). There is NO
inline-QMD write (no `0x318/0x31c/0x320..0x41c` run), so the launch capture
that feeds the physical-engine forward path never fires - and the CE-shaped
ops path is not even consulted for GR-class channels by construction. Real
pixels therefore require a translator that recognizes the C5C0 section,
builds the vendor launch the physical channel expects (the same prelude/QMD
shape the CUDA path already executes) from the section's fields and data
runs, and forwards it with the existing operand/relocation machinery. The
method vocabulary overlaps the physical prelude's existing knowledge
(`0x7b0/0x7b4`, `0x2a4`), so the landmarks for that translator are known.

## 9. UPDATE 2026-10-26 (SESSION 130): the utility-channel LOAD_INLINE_DATA arm fires; the payloads are the driver's zero buffer; the SASS delivery is the remaining non-black gate

**The arm was dead by branch order.** The utility channel's engine-side
uploads ride subchannel 2 (the A140/KEPLER_INLINE_TO_MEMORY_B lane), and
their operand group (`0x188/0x18c` destination, `0x180` line length,
`0x184` line count) was parsed only by an `else if (subch == 1 || subch ==
2)` branch that could never run: the preceding `if (xlate && subch >= 2)`
TURING-copy parser also matches subchannel 2 and swallowed the group
unread. With `have_out`/`line_len` unset, the `0x1b4
LOAD_INLINE_DATA` arm never opened (`saw=0`). The 0x180..0x18c group is now
parsed first for subchannel 1|2; the arm reaches its write decision and
authors (`writes=2` on r744: 256 B -> vidmem 0x1e0000, 12288 B -> vidmem
0x1c0000).

**The payloads are stably zero - and that is the driver's zero buffer.**
Four independent reads (the sweep's channel path, the direct
registered-VAS walk - 22k attempts, never non-zero - the guest's own
`/proc/<pid>/mem` view of the client's mapping, and the guest's
`/proc/kcore` physical read) plus a whole-run 1 MB staging-BO poller and a
gdb hardware write-watchpoint that never fires agree: the driver never
writes the payload chunks for these entries. The 256 B size equals
`GF100_ZERO_BUFFER_SIZE`, and the driver's `gf100DownloadZeroBuffer()`
uploads exactly 256 B of zeros through this path. The arm therefore
applies zero payloads under a variant once-key (`h | 1`) - idempotent over
an empty destination - while a later non-zero payload for the same entry
still lands under its own key.

**The non-black gate, precisely.** The launch's `prog=0xdfd8feb900`
resolves to vidmem `0x62b000`, which is empty; the compute shader SASS is
not delivered by any decoded entry. The utility channel's ring holds
exactly 14 entries (`GP_PUT=14`), of which two carry LOAD_INLINE_DATA -
both the zero objects above. The Vulkan (NVN) download path uploads via a
separate download state and points its destination view at the pitch-pool
allocation, so the SASS/constants i2m shape is a different destination
family than the observed zero-buffer uploads. Next steps: find the SASS
upload shape (pitch-pool destination, possibly on another channel or a
non-LID copy) and then the `n_data` descriptor/operand wiring
(section 8); with both, the physical forward can execute the dispatch.

## 10. UPDATE 2026-10-26 (SESSION 131): the utility-ring translation is the live problem - the decoded LID entries are from a STALE ring page; the live ring is in vidmem

**The ring is in vidmem, and its content varies per client.** The
utility channel's ring VA (0x4020000) resolves through the service's
learned map to a **vidmem** offset (PA 0xa0000, next to the channel's
USERD at 0x8fe00), and the pump's own entry read shows the live ring's
content: on the r750-era clients `entry[0] va=0x120100000 len=0x28`
(`sema-chan: entry[0] h=0xbeef0100 eng=0 raw=0000280120100000
va=0x120100000 len=40`). A dump of the vidmem ring (dump_vidmem 0xa0000)
confirms exactly one entry there.

**The decoded LID entries (the 14-entry ring at 0x4000000/0x4250000) are
from a STALE page.** The sweep's ring read (`svc_read_va` of the ring VA)
can resolve a **guest-RAM** page (e.g. PA 0x13003d000) that holds an
EARLIER client's ring - a coherent 14-entry ring whose LID uploads target
the Vulkan client's own allocations (dst 0x423f000 -> vidmem 0x1e0000,
dst 0x4448000 -> vidmem 0x1c0000, 256 B = GF100_ZERO_BUFFER_SIZE and
12288 B). Those entries are real, but they belong to a dead session: the
payload chunks they point at (VA 0x4250000/0x4250110 -> the session's
1 MB staging BO) read zero through EVERY path - the sweep's channel read,
the direct registered-VAS walk, the learned map, the guest's own
`/proc/<pid>/mem` view of the BO, `/proc/kcore`, a whole-run
high-frequency poller of every 1 MB BO the client maps, and a gdb
hardware write-watchpoint that never fires - because the session ended
and the BO's pages were freed/reused. **The payload is not missing from a
live upload; it is a stale-session artifact.**

**The CUDA contrast proves the mechanism.** A CUDA client's LID rows on
the same boot show the arm working with real payloads:
`h=0x5c000019 i=2 va=0x2004002c0 len=0xb8 ... tail_va=0x200400378
tail=0x1c t8=200125a6 00000001 ... saw=1 fired=1 zero=0
dst=0xd2e00000` - the tail (the next GPFIFO entry) is CONTIGUOUS with the
push and holds a real method stream, the arm materializes it into the
high vidmem family, and the CUDA battery passes.

**Next steps (ranked).**
1. Read the LIVE client's ring through the map (vidmem 0xa0000) during a
   client run and decode its entries - that is the live upload path; the
   sweep's `svc_read_va` source preference must not answer with a stale
   RAM page (verify the ring page by content: the entries must lie in the
   client's current VA allocations, and the ring must be the same page the
   pump reads).
2. With the live ring decoded, the SASS upload shape follows; the
   descriptor/operand wiring (`n_data=0`) remains the second gate.
3. `sema_util_dec_log` (96/boot) and `svc_gr_util_log` (24/boot) budgets
   were exhausted early in multi-client boots, hiding later rows; make
   them per-session for diagnosis.

## 11. UPDATE 2026-10-26 (SESSION 131b): the A/B that names the frontier - CUDA's uploads land in the emulated vidmem, the Vulkan client's do not

`dump_vidmem map` (new: one line per 1 MB block with non-zero content,
`nz` = non-zero dwords) on a fresh boot, same module:

* **CUDA client (`victory_mail`, STATUS OK)** - the shadow fills up:
  `blk 0x200000 nz=40811`, `blk 0x1400000 nz=178868`,
  `blk 0x1500000 nz=96346`, `blk 0x2a00000 nz=22880`,
  `blk 0x7800000 nz=98257`, ... (~450k dwords total). The CUDA driver's
  uploads (LID i2m writes with real payloads + CE copies) materialize in
  the emulated vidmem.
* **Vulkan client (`vk_frame120`)** - the shadow stays at the 14
  baseline dwords (the fence/counter words at 0xfee88, the GR param bank
  at 0x385e08, one word at 0x4d0b10). **Nothing the Vulkan client uploads
  reaches the emulated vidmem at all.**

That is the frontier in one measurement: the Vulkan (NVN/GL) driver's
upload path is not among the writes the emulation can see. Combined with
section 10 (the utility ring lives in vidmem via the learned map but the
sweep can read a stale RAM page for it; the live ring holds a single
`0x120100000/40` entry, not the 14-entry LID ring), the next session must:
(1) capture the LIVE ring through the same source the pump uses while the
client is up and decode its entries (the 14-entry LID ring is a
stale-session page); (2) trace where the live client's upload writes go
(the map shows they are NOT in the first 256 MB of the shadow - check the
guest's own view of the client's staging BOs during the upload window);
(3) the descriptor/operand wiring (`n_data=0`) after that.

## 12. UPDATE 2026-10-26 (SESSION 132): THE LIVE RING IS PROVEN (guest RAM, walk-confirmed); the vidmem "ring" was a false-positive pin whose dead entry the pump skipped 39 times; the two LID uploads are the driver's zero buffers by design; the code VA's mapping rotates and the shadow's lat cache can go stale

**The live ring is in guest RAM and the sweep's decode was right.**
`walk_probe 0x4020000` (the client's own page tables, vas[0]
`h=0xfade0002 pdb=0x114039000`) resolves the utility channel's GPFIFO VA
to **guest RAM** (PA `0x11623d000` on r759; the page moves per run:
`0x104b55000` on r761) and its content is the 14-entry ring the sweep
decodes: entry0 -> push `0x4000000` len `0x8bc` (the methods), entry1 ->
the 256 B payload slot `0x4250000`, entry8 -> the 12288 B payload slot
`0x4250110`, ... - all pushes inside THIS client's own allocation window
(`PREV149-OUT ... off=0x4000000/0x4020000/0x4060000` in the client's own
VIDHEAP log). The decoded ring is live content, not a stale page.

**The vidmem "ring" (PA 0xa0000) is a false-positive autopin.** On r759
the VRAM shape scan found the channel's REAL USERD at vidmem `0xb0000`;
the userd-derived candidate loop then tried `0xb0000 - 0x10000 =
0xa0000`, whose entry0 (`push 0x120100000/40` - the CE-pool push
pattern) passed the plausibility gate and whose USERD probe hit the real
0xb0000 block, so the pin landed. The pinned entry's push reads all-zero
through every source; the pump retried 39 times and logged `skipping
stale entry h=0xbeef0100 va=0x120100000 len=40 after 39 failed rescues`,
while `sema_probe 0x4020000` then answered the wrong page
(`pa=0xa0000 aperture=vidmem`) and poisoned every later read (the sweep
included). Section 10's "stale session page" conclusion was this pin, not
the sweep.

**Fixes landed (commit `72de0bed`).** (1) `svc_ring_entry_dead` +
`svc_ring_walk_live`: a candidate whose entry0 push provably reads
all-zero through the client's page tables is rejected when the VAS walk
resolves the ring VA to a page with a live entry (applied at the
userd-derived, FB-convention and existing-pin paths; the walk cannot vote
for itself); (2) `svc_gr_ring_candidate` rejects dead-entry candidates
and accepts the channel's known USERD cross-aperture (the driver places
GPFIFO and USERD independently: ring in guest RAM, USERD in vidmem); (3)
the util sweep arbitrates a dead channel-path ring entry against the
direct VAS walk per slot; (4) the util-dec/util-sweep witness budgets and
their dedup keys are per-client now (reset in `svc_session_flush`); (5)
`sema-copyent` logs once per copy OP, not per row - one 480-row frame
copy had exhausted the 128/session budget and hidden every later copy.
Verified on r760/r761: `sema-chan: h=0xbeef0100 ... ring rc=0
pa=0x108cba000 ap=1` (the live RAM page), `sema-rdw: h=0xbeef0100
va=0x4000000 len=2236 rc=0`, the sweep decodes the live 14 entries, the
fence fold releases fire, and the client completes (`VKFRAME_DONE
frames=3 640x480 saved=3`, `VKFRAME_IDLE ok`).

**The two utility-channel LID uploads are the driver's zero buffers by
design, not the SASS.** Live decode (r760): i=0 -> `dst=0x423f000 ll=256
lc=1 zero=1` (256 B = `GF100_ZERO_BUFFER_SIZE`) and i=8 ->
`dst=0x4448000 ll=12288 lc=1 zero=1` (12288 B = the CE fault method
buffer size). The payload slots (`0x4250000` -> RAM `0x116219000`;
`0x4250110`) read zero for the whole client run - the driver never
writes them. The GR channel's own LID entry DOES carry a real payload
(`dst=0x4b86020 ll=1568 zero=0` - the launch param bank at `param=
0x4b86000 size=0x1900`), which proves the arm works with real bytes on
this boot too.

**Where the code page stands.** `prog=0xdfd8feb900` (the captured
launch): `sema_probe` answered vidmem `0x62b900` early in the run while
`walk_probe` answered vidmem `0xc8b000` later - the driver's code-buffer
mapping ROTATES (a code pool), and the shadow's learned lat entry for the
VA can go stale. Both pages read zero in the shadow: the driver's code
upload has not been observed on any decoded path (no LID, no CE copy -
the only `sema-copyent` op of the whole run is the 480-row frame
copy-out `dst=0x50b0000 src=0xdfd8d60000`, which executes `rc=0
bytes=0x12c000` and correctly copies the still-empty image). The
remaining question is the SASS delivery shape for the C5C0 compute path
(section 6's `0x3800/0x39C8` code-run/data-port semantics) and the stale
lat preference for the rotating code VA.

**Next steps (ranked).**
1. Resolve the code VA through the VAS walk at launch time (the walk is
   the live truth; the lat entry goes stale as the pool rotates) and
   watch the code page the QMD actually names.
2. Find the SASS delivery: dump the C5C0 section's `0x3800/0x39C8` runs
   with a bounded methods witness while the client is live (the params
   arrive by LID; the code must arrive by a sibling path).
3. Then the `n_data=0` descriptor/operand wiring (section 8).

## 13. UPDATE 2026-10-26 (SESSION 133): THE SASS DELIVERY SHAPE IS FOUND - the driver embeds the program bytes INLINE in a CE LAUNCH_DMA push and copies them to vidmem; the CE channel's ring/USERD resolution is the boot-lottery gate that keeps the copies from running

**The delivery shape (decoded from the CE channel).** A CE-class channel
(eng=11) carries, as entry[4] of its ring, a push whose method stream is a
LAUNCH_DMA copy with the SASS bytes INLINE:

```
sema-chan: entry[4] h=0xcaf00012 eng=11 raw=0021340120a00000 va=0x120a00000 len=8500
sema-bigrun: h=0xcaf00012 push=0x120a00000 off=0x0 hdr=0x64000002 op=3 cnt=1024 subch=0 addr=0x8
sema-lda: h=0xcaf00012 eng=11 push=0x120a00000 dma=0x00002181 xfer=1 dstphys=1 srcphys=0 out=0x205000 in=0x120a00004 line=4096 have_out=1 have_in=1
sema-opadd: h=0xcaf00012 type=1 dst=0x205000 src=0x120a00004 bytes=4096 ap=0 src_ap=0 phys=1 src_phys=0
```

The source is the push itself (`in=0x120a00004`, right after the method
header) and the destination is a vidmem VA (`out=0x205000`, then
`0x206000`, ... 4096 B per run) - the engine-side "download program
range" path. The Vulkan client's CE channels carry the SAME shaped
entries (r763: `sema-utildec: h=0xbeef0101 i=4 va=0x120a00000 len=0x2134`,
19 entries swept) - so the SASS delivery is a CE inline copy, not the
utility-channel LID entries (those are the driver's zero buffers,
section 12).

**Why the copies never ran (three gates).** (1) The CE channel's ring and
USERD live in VIDMEM and resolve only by boot-lottery: the FB convention
matched on r763 (`auto-pinned ring h=0xbeef0101 rank=0 va=0x4100000 ->
fb 0x80000 (entry0 va=0x120000000 len=40)`; entries then decoded and the
frame path ran) but read zeros on r765/r767/r768 - the USERD shape scan
instead finds the channel blocks at 0xfee00/0xff000/0xff200 (r767/r768),
so the rings sit at unaligned offsets like 0xefe00. (2) The CE push VAs
(0x120000000+) need the client's VAS registered: the parked
COPY_SERVER_PDES candidate only registered at +15 s (age-out,
`sema_vas_cand_ms`) while the entry hold grace (`sema_ce_grace_ms`,
8000 ms from the cycle) expired first - r763 logged `skipping stale
entry h=0xbeef0101 va=0x120000000 len=40 after 8000 ms grace` and
`sema-pushfail: va=0x120a00000 ... trusted_rc=-2 ipin=0` for the copy
pushes. Setting `sema_vas_cand_ms=5000` registers the VAS at +5 s
(verified: `VAS 0xfade0002 ... registered UNVERIFIED after 5000 ms
parked`) but the ring gate still blocks. (3) The shadow ring scan
fallback (commits `52ef3f9d` + `2735180a`: 0x100 stepping, rate-limited
retry, entry0 shape + USERD probe at +0x10000/+0x2000/+0x1000) found no
candidate on r768: the ring/USERD content is volatile at scan instants
(a `find_userd` read shows `off=0xfee00 get=11 put=11` while a
`dump_vidmem 0xfee00` one second later reads zero), so a periodic scan
misses the window.

**Next steps (ranked).** 1. Derive the CE ring from a USERD found by its
own shape (put at +0x8c, get at +0x88 - the `find_userd` scan already
does this) instead of scanning for ring entries: for every USERD-shaped
block, try ring = userd - {0x10000, 0x2000, 0x1000} and validate entry0
+ the probe. 2. Or latch the ring at the first CE submission (the ring
content appears exactly when the driver writes entry0). 3. With the ring
latched and the VAS at +5 s, verify the inline copies execute
(`sema-lda`/`sema-opadd`/`sema-copyent` with the code dsts) and the code
page gets content; then the `n_data` descriptor wiring (section 8).

## 14. UPDATE 2026-10-26 (SESSION 134/135): the CE first-submission capture landed and its rank-blind first-hit caused a CUDA-client stall regression (fixed); the Vulkan client's CE rings are allocated ABOVE the capture scan's 64 MB bound and the frame chain reduces to that un-latched ring

**SESSION 134 (commits `36642718`, `928c0730`).** A new instrument fires in
the notifier-completion path for CE channels *before* the clear write - the
one instant the ring content and the USERD are both live - reads entry0
through the channel translation and the trusted walk, then scans the shadow
(0x100 step, first 64 MB) for a ring-shaped page (entry0 naming a 1 MB
aligned push VA in `[0x120000000, 0x121000000)` with a sane length) and runs
the standard USERD gate at `+0x10000/+0x2000/+0x1000/+0x0`; a validating
candidate is pinned/latched exactly like the regular CE autopin. `928c0730`
fixed the entry decode: the push VA is 40 bits, the scan had declared it
`u32`, so `scan_hits` was zero by construction.

**SESSION 135 (commit `a674c21c`) - the rank-blind first-hit regression.**
The scan started at offset 0 and accepted the first ring-shaped page for
*every* CE channel. On r779 both `caf00012` (rank 0) and `caf0001d`
(rank 1) latched `off=0x80000 upa=0x90000`: rank 1 read rank 0's
ring/page, its first release decoded to the wrong semaphore word (+0
instead of +4) and was dropped as stale (`cur=1`, already written by
rank 0), so the client's rank-1 tracker wait never completed. victory_mail
then spun at 100% CPU with `pub_calls=0` on three consecutive boots
(r777/r778/r779: battery pass=0 fail=7). A control run of the r746-era
module (`25523f84`) on a fresh boot passed the same client (VADD
`bad=0 checksum=12204516.000000`, `STATUS OK`), isolating this to the
SESSION 133/134 code. The fix tries the rank's convention offset
(`0x80000 + 0x20000 * rank`) first, scans the rest bounded as before, and
never accepts an offset another CE rank already latched this session; an
unlatched rank stays for the regular per-pass latch. Verified on r781:
ranks 0..5 latched correctly (0x80000/0xa0000/0xc0000/0xe0000/0x100000/
0x120000, USERD +0x10000 each), rank 1's own semaphore word applied
(`pool-wr va=0x121000004 cur=0 -> val=1 ok`), and the client progressed
(`pub_calls=185132+`, no spin).

**The Vulkan CE ring on the same boots.** r781 (classic autopin, rank 0):
`auto-pinned ring h=0xbeef0101 rank=0 va=0x4100000 -> fb 0x80000 (entry0
va=0x120000000 len=40)` - a live entry - but the client's pushes decoded
`unread=1 injected=0`: the guest pin scanner covered CUDA clients
(`cdt,va,victory`) and never `vk_frame`. Extending the scanner to `vk`
engaged the tools (demand pins active) but r782/r783's captures reported
`scan_hits=0 latch=0`: no ring-shaped page anywhere in the shadow's first
64 MB, and no classic autopin either.

**Where the Vulkan rings actually are.** The client's own vidheap ioctl
log (`PREV149-OUT` lines, r783) names the CE/GR channel allocations:
`off=0x4020000/0x4100000/0x4160000` - matching the registered gpfifos
(`0x4020000/0x4100000/0x4160000`) and lying just ABOVE the capture's
`lim = min(shadow, 0x4000000)` (64 MB). Direct dumps of those offsets (and
of `0x80000`) read zero, and a byte search of the full 4 GB guest-RAM image
for the entry pattern (`00 00 00 20 01 28 00 00` / the `0x20100000`
sibling) finds nothing - the ring content reaches neither the shadow at its
allocation offset nor guest RAM.

**The frame chain (r783, live).** The Vulkan client completes three frames
(`VKFRAME_DONE frames=3`, `VKFRAME_IDLE ok`) but all PPMs are pure black.
The live GR channel (`h=0xbeec0100`) shows the dependency explicitly:
`sema-gr: launch captured ... prog=0xdff0feb900 ... inline=8` together
with `sema-gr: copy pend h=0xbeec0100 push=0x4f4580c src=0xdff0d60000
dst=0x50b0000 row=0xa00 rows=480 ...` repeated ~20/s. The trusted walk
maps the copy source to VIDMEM (`walk_probe 0xdff0d60000 -> rc=0
pa=0x570000 aperture=vidmem`) and the destination `0x50b0000` is the
framebuffer. The shadow at `0x570000` reads zero: the data pages are the
ones the CE inline copies are supposed to fill, and the GR launch is held
on their absence (the designed `dstpend` dependency). So the whole Vulkan
chain reduces once more to the un-latched CE ring: no ring -> no inline
i2m copies -> no data/code pages -> held launches -> black frame.

**Next steps (ranked).** 1. Cover the client's real allocation in the ring
capture: raise the capture scan bound above the recorded channel
allocations (e.g. `0x6000000`) and/or seed candidates from the client's
vidheap offsets, then re-check whether the entry arrives at the
submission instant (a live `sema_vidmem_dump` at `0x4100000` while the
client is streaming would settle it; the post-run dumps read zero). 2. If
the shadow still reads zero at the allocation offset during live
submissions, the client-side write watch is the remaining forensic: find
the CPU pointer the driver writes GPFIFO entries through, its PA via the
guest pagemap, and why the GPU PTE/walk does not name that PA. 3. With a
latched ring, verify the inline copies execute (`sema-lda`/`sema-opadd`
with the code dsts), the code/data pages fill, then the `n_data`
descriptor wiring (section 8) → non-black frames. 4. Re-run the CUDA
battery on the fixed module (the r781 boot was a slow-crawl boot; the
spin stall class is gone) and revisit `sema_gr_xlate`'s multikernel
forward timeout.

## 15. UPDATE 2026-10-26 (SESSION 135b): every "Vulkan CE ring latch" on record was a STALE CUDA BLOCK - the Vulkan CE ring content has never been observed; the write path is not BAR1 and the content is in no shadow, no RAM, and not at the PTE target

**Forensics on four fresh boots (r782-r786; commits `a674c21c` in tree, `c3cdcf4b` new).**

*The write path works; the driver does not use it for the ring.* A direct
BAR1 test - the guest writing a known pattern at `BAR1+0x4100000` through
`/sys/bus/pci/devices/0000:01:00.0/resource1` - lands in the emulated
vidmem shadow verbatim (`RINGTEST` + `AAAA...` read back by both the guest
and `sema_vidmem_dump`), so the mdev's BAR1 region path (mmapped, direct
into `bars[1].vaddr`) is sound at those offsets and at `0x80000`. A
high-frequency watcher (12 candidate offsets, ~2.5 Hz, including
`0x80000/0xa0000/.../0x4020000/0x4100000/0x4160000` and the +0x10000
USERDs) recorded ZERO nonzero dumps across the whole live streaming window
of a clean Vulkan run (r786; three frames, `VKFRAME_RATE fps=0.05`,
`VKFRAME_DONE frames=3`, black PPMs). The `shadow_scan` over all three BAR
shadows (0/1/3) for the CE push header `{0x120,0xb5c50000,...}` and the
ring-entry pair `{0x20000000,0x2801}` finds nothing, and a byte search of
the full 4 GB guest-RAM image for both patterns (little-endian, corrected
from an earlier reversed search) finds nothing either. The walk and the
learned map resolve the CE ring VA (`0x4100000`) to a guest-RAM page
(per-boot PA, e.g. `0x135726000`) that reads all zero, and the vidheap
ioctl log (`PREV149-OUT`) names the channel allocations
`0x4020000/0x4100000/0x4160000` on every boot - so the ring content is
written neither to the shadow at its allocation offsets nor anywhere in
RAM reachable through the guest's own page tables.

*A stale-block trap explains every historical "latch".* On CUDA-first
boots the capture latched `fb 0x80000 / userd 0x90000` for the Vulkan
channel `beef0101` with `put=1 get=1` - the dead CUDA client's `caf00012`
block surviving in VRAM after its session. The decoded entry
(`va=0x120000000 len=40`) is the old session's (both clients' first CE
pushes are shape-identical), the expectations seed from the stale pool
words (`want=2/3` vs `pay=1`; the word page read 3-6), and the Vulkan
client stalls into its fence timeout (`VKFAIL vkWaitForFences -> 2`) on
every run. r763's and r781's "auto-pinned ring h=0xbeef0101 -> fb 0x80000"
rows are the same artefact: both were CUDA-first boots. All genuine
first-submission latches on record read `put=1 get=0` at the capture
instant. `c3cdcf4b` rejects a consumed-state candidate (`get == put > 0`)
in the first-submission capture - a just-submitted entry cannot be
consumed at the notifier-clear instant.

*The frontier is now precise.* The Vulkan client's CE submissions complete
only through the fabricated completion machinery (the driver never reads
its ring back), its CE ring content is invisible to every host source, and
the GR launch for each frame is held on the unexecuted CE i2m copies
(`copy pend src=0xdff0d60000 dst=0x50b0000`; walk: src -> vidmem
`0x570000`, zero). Next: the client-side write watch - find the CPU
pointer the guest driver writes GPFIFO entries through, its PA via the
guest pagemap, and why neither the GPU page table target nor any shadow
holds the result (guest-kernel gdb via the QEMU gdbstub or a kprobe on
the driver's ring-write helper).

## 16. UPDATE 2026-10-26 (SESSION 137): THE CE-RING HYPOTHESIS FOR THE VULKAN CLIENT IS DISPROVEN BY DIRECT WRITE WATCHES - no kernel and no client writer ever touches the ring page

**The write-watch toolchain (all three instrument classes).** (1) kprobes:
`channelFillGpFifo` (the kernel's GPFIFO-entry writer, called from the CE
and SEC2 submission paths per the open driver sources) is marked
`notrace` - `trace_kprobe: Could not probe notrace function
channelFillGpFifo [nvidia]` - so kprobes and ftrace can never attach
(this is why the named helpers "never fired" in earlier sessions: they
could never register). (2) QEMU-gdbstub hardware breakpoints work: a
verification `hbreak *__x64_sys_ioctl` fired on guest ioctls
(`IOCTL#1 pc=0xffffffffbc316160`). An `hbreak` on the exact entry store
(`channelFillGpFifo+0xb5`, `mov %edi,(%rdx)`, where `%rdx =
pbCpuVA + channelPbSize + lastSubmittedEntry*8` per the disassembly and
the source) stayed silent across a complete Vulkan run - even though that
same run produced the CE channel's notifier clear
(`sema-chan: notifier cleared h=0xbeef0101 info32=0x5`) and the
first-submission capture (`sema-cefirst: h=0xbeef0101 rank=1 ...
scan_hits=0 latch=0`). (3) A client-side hardware watchpoint: the ring
VA resolves through the guest page tables to a guest-RAM page that is
mapped into the client process (`/dev/nvidiactl` mapping; e.g. PA
`0x101021000` -> client VA `0x7fffee69b000`; a later run PA
`0x10087b000` -> VA `0x7fffee7d8000`), and the walk and the emulation's
own read agree the page is ZERO. An in-client gdb (the t122 recipe plus
a file-fed watchpoint) armed `watch *(unsigned long*)0x7fffee7d8000`
from the client's FIRST ioctl (`Hardware watchpoint 2` + `RGWATCH-SET`
logged before `GLBASE`) caught ZERO writes across the whole run
(3 frames, `VKFRAME_DONE`, IDLE).

**Consequences.** On vk-first boots the client creates its CE channel but
never submits CE work: no GPFIFO entry is written by the kernel
(`channelFillGpFifo` never executes), nor by the client (hardware watch,
zero hits), nor is any content found in any BAR shadow or in RAM. The
historical "Vulkan CE ring latches" were the dead CUDA client's stale
blocks (section 15). The image/upload path for the Vulkan client rides
its GR channels (`launch captured ... inline=8`; the frame copy held as
`sema-gr: copy pend ... src=0xdff0d60000 dst=0x50b0000 rows=480`), and
the non-black gate must be attacked on that path - the data/code pages
those launches depend on are never filled because no CE i2m copies are
ever queued, and the launch is held on their absence (dstpend/n_data).
Next: instrument the GR launch's dependency resolution (which pages the
dstpend gate waits for, and whether the inline records the capture holds
are sufficient to fill them) instead of hunting a CE ring that nothing
writes.