path: root/docs/04-implementation/04-real-render-walls.md

# Real-render walls: Vulkan and Xorg on the emulated vGPU

Status: OPEN (2026-10-29, SESSION 64). This page records the measured state
of the guest's *display/graphics* consumers - the Vulkan ICD and the Xorg
server - on the emulated vGPU, with the exact evidence and the next
subsystem to implement. It complements
[05-operations/06-guest-frame-path-and-looking-glass.md](../05-operations/06-guest-frame-path-and-looking-glass.md)
(which covers the CUDA -> ivshmem -> host frame transport that is already
proven).

## 1. What works today

* The guest's shipped 570.144 Vulkan userspace **loads and enumerates**:
  `vkCreateInstance` succeeds and two physical devices are reported -
  `NVIDIA GeForce RTX 2080 SUPER` (discrete, Vulkan 1.4.303) and the
  software `llvmpipe` - from `/etc/vulkan/icd.d/nvidia_icd.json`. Probes:
  `tools/vk/vk_probe.py`, `tools/vk/vk_probe2.py`.
* `eglInitialize` succeeds with `vendor=NVIDIA` on the guest EGL device
  platform (the EGL probe `tools/guest-diag/egl_retry.c` and the older
  `egl_frame`).
* CUDA is unaffected (its clients run through the same emulated GSP; see
  the battery evidence in the handoff).

## 2. Wall 1 - `vkCreateDevice` fails in nvidia-modeset's ALLOC_DEVICE

`vkCreateDevice` returns `VK_ERROR_INITIALIZATION_FAILED`. An
`LD_PRELOAD` ioctl logger (`tools/guest-diag/nvkmstrace.c`) shows the
ICD's first nvKms ioctl failing:

```
NVKMS_IOCTL cmd=0 size=1488 addr=0x... rc=-1   # NVKMS_IOCTL_ALLOC_DEVICE
```

i.e. the kernel's `nvKmsAllocDevice` (in `nvidia-modeset`) returned a
non-success status; the userspace frees everything and reports the
generic initialization error. Reading the driver's own modeset source
(`nvkms-rm.c`) gives the alloc order and the two gates we hit and fixed:

1. `ProbeHeadCountAndWindowAssignment()` -> `NV0073_CTRL_CMD_SYSTEM_GET_NUM_HEADS`
   (`0x730102`): a zero reply logs `No heads found on board!` and fails the
   alloc. The emulated GSP RSP answered all NV0073 controls through the
   generic zero-reply path.
2. After the heads: `NoValidDisplays()` -> `NV0073_CTRL_CMD_SYSTEM_GET_SUPPORTED`
   (`0x730107`) must report at least one display id, and the connector
   path (`SPECIFIC_OR_GET_INFO` `0x73028b`, `SPECIFIC_GET_CONNECTOR_DATA`
   `0x730250`, `DFP_GET_INFO` `0x731140`) must describe a connector.

The host GSP-RM emulation now implements a minimal headless display:
one head (`0x730102`/`0x730287`), flexible window assignment (`0x7302ad`),
one static HDMI connector on display id 0 (SOR + single-link TMDS, which
avoids the DP-AUX library), plus DDC support. With this shim the alloc
gets past heads and `NoValidDisplays` and proceeds through the connector
queries and the display-object allocations (classes `0xc570`, `0xc57d`,
`0xc372`, controls `0x50700709`, `0x73010c`, `0xc3700201`) before failing
later in the display-core/modeset-channel path. That remaining subsystem
is the wall: the modeset core channel and the per-display dpy objects
must be emulated (a coupled object + channel + memory set whose exact
wire forms should be captured from a real-GPU session).

A related experiment ruled out one shortcut: booting the guest with
`nvidia-modeset`/`nvidia-drm` blacklisted (no `/dev/nvidia-modeset`) does
not avoid the wall - the ICD's nvKms client still issues ALLOC_DEVICE and
fails, so the display path cannot be sidestepped by disabling KMS.

## 3. Wall 2 - Xorg ScreenInit fails in nvidia-push's channel test

Xorg starts (the GPU is detected, `Virtual screen 1280x1024`) and then:

```
(EE) NVIDIA(GPU-0): Failed to initialize DMA.
(EE) NVIDIA(0): Failed to allocate push buffer
```

`nvidia-push` logs `Failed to initialize DMA.` from
`__nvPushTestPushBuffer()`: right after allocating a push buffer the
driver writes a NOP and waits (10 s) for the channel to consume it. Host
evidence shows why it never does:

* the X driver's channel registers (`class 0xc46f` GPFIFO channel,
  `gpfifo=0x120090000`, `userd=0x350000` sysmem) and lives with
  **`submissions=0`** - its published GP_PUT never advances in the host's
  view, because the host needs the channel's VA->PA translations to read
  its USERD/GPFIFO and that resolution comes from the guest-side scanner
  that currently follows *CUDA clients only* (`targeted resolve
  va=0x20400f000: full RAM scan found nothing`);
* on a busy boot (Xorg started while a CUDA battery ran) the channel's
  completion-counter slot is rejected: `counter auto h=0x1000013 rank=0
  pa=0x11c827ff0 rejected (slot owned by another channel)` - two
  concurrent clients both derive rank 0's slot from the per-device
  structural formula.

Fixing Xorg therefore needs (1) a platform-wide (non-CUDA) VA-resolution
path for the scanner or the host, and (2) counter-slot ownership keyed by
client identity (or a per-client slot domain), not just rank.

## 4. Wall 3 - the EGL/glsi engine-init gate (unchanged)

`eglMakeCurrent` returns `EGL_BAD_ALLOC (0x3003)`; the glsi engine-init
entry returns a "busy" status once, sets a state bit, and the eglcore
caller does not retry. Ten in-process retries (`egl_retry`) all fail, so
the state machine fails on a real input - the return-value trace of its
callees remains the way in (see
[userspace/cuda/570.144/docs/ANALYSIS-egl-engine-selection.md](../../userspace/cuda/570.144/docs/ANALYSIS-egl-engine-selection.md)).

## 5. Why this matters for the frame path

Everything downstream of a working GL/Vulkan context is already in place:
a guest publisher writes frames into the kvmfr ivshmem BAR
(`tools/vk/vk_frame.c`, `tools/guest-diag/xshm_pub.c` use the same layout
as the proven CUDA `frame_producer`), the host reader decodes them
(`tools/lg/frame_reader.py`) and ffmpeg assembles video. The missing
piece is a *working* graphics API on the guest GPU - the two subsystems
above.
## 6. UPDATE 2026-10-25 (RESUME #56): the wall is now fully bisected

Three new instruments settled the remaining questions:

* **An `LD_PRELOAD` nvKms ioctl logger** (`tools/guest-diag/nvkmstrace.c`)
  decodes the failing `NVKMS_IOCTL_ALLOC_DEVICE` reply: status **5 =
  `NVKMS_ALLOC_DEVICE_STATUS_CORE_CHANNEL_ALLOC_FAILED`**.
* **The QEMU gdb stub (`-gdb tcp::1234`) as a guest-kernel microscope**
  (the guest kernel's `nvidia_modeset` symbols are in `/proc/kallsyms`):
  breaking inside `nvAllocCoreChannelEvo` (the blob's offsets were mapped
  from a disassembly of the shipped `nv-modeset-kernel.o_binary`) dumped
  the driver's class list (42 classes; all `NVC57x` display classes are
  present; **`NV927C_BASE_CHANNEL_DMA` (0x927c) is not** - it exists in no
  HAL class list, so the "base channel" fallback path can never succeed).
  Breaking at the capability parse showed the **capability notifier buffer
  is all zeros** (nothing emulates the display engine that would fill it).
* **The caps-fill experiment**: with gdb writing `SYS_CAP = HEAD0_EXISTS`
  and `SYS_CAPB = WINDOW0..7_EXISTS` into the caps buffer right before the
  parse, the driver flips from the dead base path to the **window path**
  (`nvRMAllocateWindowChannels`) - the hypothesis that `numWindows > 0`
  unblocks the flow is confirmed.

The window path then fails one step later: `nvRmApiAlloc(display,
class=0xc57e NVC57E_WINDOW_CHANNEL_DMA)` returns **`NV_ERR_NOT_SUPPORTED`
(0x56) locally in the guest kernel** (it never reaches the emulated
GSP - no ALLOC RPC is logged). This matches the shipped driver's vGPU
architecture: the guest runs in vGPU mode (`IS_VIRTUAL`), where the
display/modeset channel classes are rejected (the reference source's
`_rpcAllocObjectPrologue` documents "Display classes are added for DCE,
vGPU Guest does not support those") - a vGPU guest's display is meant to
be emulated by the **host-side vGPU display service** (the vGPU RPC
display path), which the emulation does not implement.

**Consequence for the roadmap.** Guest Xorg/GL/Vulkan on this emulated
vGPU needs (a) the display capability fill on the host side (mechanics
proven by the gdb experiment) and (b) a vGPU display service: the
display-engine MMIO/RPC backend the guest expects from its host (the
equivalent of NVIDIA's vGPU display plugin), or the specific vGPU
configuration that makes the kernel accept the display channel class.
Until (b) exists the guest's NVIDIA display path is closed by design, and
GZDOOM-class workloads cannot render on the guest GPU.
## 7. UPDATE 2026-10-25 #2 (SESSION 66): the 0x56 pinned to the RS alloc path

A new instrument closed the gap: the guest's `nvidia.ko` (DKMS build) is
**not stripped** - copy it out (`/lib/modules/<kver>/updates/dkms/nvidia.ko`),
`objdump -t`/`-dr` it, compute the module's `.text` runtime base from any
`/proc/kallsyms` symbol, and every guest-kernel function/offset becomes
addressable for live gdb breakpoints on the QEMU stub. With it:

* `rpcAllocObject_v29_06`/`_rpcAllocObjectPrologue` are **exonerated** for
  0xc57e (their reject list is 0x73/0xc67*/0xc770/0xc77d only; the register
  mapping `hClass = r8d` was verified live).
* The failing chain is pinned exactly: `rmapiAllocWithSecInfo` **enters**
  with class 0xc57e, calls `serverAllocResourceUnderLock`, and that
  function returns **0x56** - *before* the GSP-RM alloc RPC
  (`rpcRmApiAlloc_GSP`, called from under-lock+0x447; the emulation never
  sees an ALLOC for 0xc57e). The same probe shows 0xc57d/0xc570/0xc573
  return 0.
* So the last blocker for the window path is a **local RS-layer validation
  inside `serverAllocResourceUnderLock`** (candidates: the inlined
  privilege/class-support checks, `rmapiFixupAllocParams`' per-class fixup,
  or `clientAllocResource`'s construct dispatch). The 0x56-producing
  instruction is not yet pinned - a `stepi` loop over the QEMU stub is far
  too slow and wedged the guest (recovered by a fresh cycle); the next
  session should use conditional breakpoints on the function's call-return
  sites (the approach in `gdbdiag19` works; the earlier run had truncated
  addresses and needs the `0xffffffff00000000` base restored) or a
  watchpoint on the status word.

**gdb-stub field notes**: the stub is single-client; killing an attached
gdb leaves the VM stopped (recover with a detach-only connect);
`stepi` loops over the stub freeze the guest's userspace for minutes -
prefer breakpoints; every probe run with the caps-fill breakpoint changes
the driver's path, so keep the fill in each experiment.

## 8. UPDATE 2026-10-25 #3 (SESSION 126): the fence fold landed - the service authors the Vulkan fence releases from the driver's own push bytes; the compute-dispatch shape decoded (direct C5C0 method stream, no inline QMD)

**The fence wall is closed.** The client's per-frame fence wait polls a
16 KB BO word that a submit-push INC-5 semaphore group targets
(`[2005c017][0x45ba090][0][payload 1][0][0x02100001]`). The group sits in a
GPFIFO entry beyond the driver's kick boundary (GP_PUT=11 while the ring
already carries the entries it pre-built for the next submits, idx12..46),
so the GP_PUT-bounded pump walk never decoded it and a host-side assist had
to write the word. The service now sweeps that pre-built continuation
itself: `sema_gr_ahead` (default 64) walks entries beyond GP_PUT through the
ring's validated translation, accepts only pushes that pass the strict
method-stream gate, extracts the addr-0x5c RELEASE groups and applies them
forward-only through the standard pending ledger. Live evidence: the sweep
finds the groups and the ledger applies the fence word once per frame;
`VKFRAME seq=1..3/DONE` completes with no assist. The frames are still
black - the shader work itself is unexecuted.

**The compute-dispatch shape (why nothing executes).** The workload is a
single compute dispatch (640x480 storage image). Its push is a DIRECT
method-stream programming of the Turing compute class: `SET_OBJECT 0xC5C0`
on subchannel 1, then the context windows (`0x7b4` local-memory window B,
`0x2a4` shared-memory window B, `0x270`), an immediate-config block, and
the code/const data runs (`0x3800`/`0x39C8` pairs plus a subchannel-0 data
port stream of `0x11c`/`0x114` runs in a separate 17 KB push). There is NO
inline-QMD write (no `0x318/0x31c/0x320..0x41c` run), so the launch capture
that feeds the physical-engine forward path never fires - and the CE-shaped
ops path is not even consulted for GR-class channels by construction. Real
pixels therefore require a translator that recognizes the C5C0 section,
builds the vendor launch the physical channel expects (the same prelude/QMD
shape the CUDA path already executes) from the section's fields and data
runs, and forwards it with the existing operand/relocation machinery. The
method vocabulary overlaps the physical prelude's existing knowledge
(`0x7b0/0x7b4`, `0x2a4`), so the landmarks for that translator are known.