1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
|
# Real-render walls: Vulkan and Xorg on the emulated vGPU
Status: OPEN (2026-10-29, SESSION 64). This page records the measured state
of the guest's *display/graphics* consumers - the Vulkan ICD and the Xorg
server - on the emulated vGPU, with the exact evidence and the next
subsystem to implement. It complements
[05-operations/06-guest-frame-path-and-looking-glass.md](../05-operations/06-guest-frame-path-and-looking-glass.md)
(which covers the CUDA -> ivshmem -> host frame transport that is already
proven).
## 1. What works today
* The guest's shipped 570.144 Vulkan userspace **loads and enumerates**:
`vkCreateInstance` succeeds and two physical devices are reported -
`NVIDIA GeForce RTX 2080 SUPER` (discrete, Vulkan 1.4.303) and the
software `llvmpipe` - from `/etc/vulkan/icd.d/nvidia_icd.json`. Probes:
`tools/vk/vk_probe.py`, `tools/vk/vk_probe2.py`.
* `eglInitialize` succeeds with `vendor=NVIDIA` on the guest EGL device
platform (the EGL probe `tools/guest-diag/egl_retry.c` and the older
`egl_frame`).
* CUDA is unaffected (its clients run through the same emulated GSP; see
the battery evidence in the handoff).
## 2. Wall 1 - `vkCreateDevice` fails in nvidia-modeset's ALLOC_DEVICE
`vkCreateDevice` returns `VK_ERROR_INITIALIZATION_FAILED`. An
`LD_PRELOAD` ioctl logger (`tools/guest-diag/nvkmstrace.c`) shows the
ICD's first nvKms ioctl failing:
```
NVKMS_IOCTL cmd=0 size=1488 addr=0x... rc=-1 # NVKMS_IOCTL_ALLOC_DEVICE
```
i.e. the kernel's `nvKmsAllocDevice` (in `nvidia-modeset`) returned a
non-success status; the userspace frees everything and reports the
generic initialization error. Reading the driver's own modeset source
(`nvkms-rm.c`) gives the alloc order and the two gates we hit and fixed:
1. `ProbeHeadCountAndWindowAssignment()` -> `NV0073_CTRL_CMD_SYSTEM_GET_NUM_HEADS`
(`0x730102`): a zero reply logs `No heads found on board!` and fails the
alloc. The emulated GSP RSP answered all NV0073 controls through the
generic zero-reply path.
2. After the heads: `NoValidDisplays()` -> `NV0073_CTRL_CMD_SYSTEM_GET_SUPPORTED`
(`0x730107`) must report at least one display id, and the connector
path (`SPECIFIC_OR_GET_INFO` `0x73028b`, `SPECIFIC_GET_CONNECTOR_DATA`
`0x730250`, `DFP_GET_INFO` `0x731140`) must describe a connector.
The host GSP-RM emulation now implements a minimal headless display:
one head (`0x730102`/`0x730287`), flexible window assignment (`0x7302ad`),
one static HDMI connector on display id 0 (SOR + single-link TMDS, which
avoids the DP-AUX library), plus DDC support. With this shim the alloc
gets past heads and `NoValidDisplays` and proceeds through the connector
queries and the display-object allocations (classes `0xc570`, `0xc57d`,
`0xc372`, controls `0x50700709`, `0x73010c`, `0xc3700201`) before failing
later in the display-core/modeset-channel path. That remaining subsystem
is the wall: the modeset core channel and the per-display dpy objects
must be emulated (a coupled object + channel + memory set whose exact
wire forms should be captured from a real-GPU session).
A related experiment ruled out one shortcut: booting the guest with
`nvidia-modeset`/`nvidia-drm` blacklisted (no `/dev/nvidia-modeset`) does
not avoid the wall - the ICD's nvKms client still issues ALLOC_DEVICE and
fails, so the display path cannot be sidestepped by disabling KMS.
## 3. Wall 2 - Xorg ScreenInit fails in nvidia-push's channel test
Xorg starts (the GPU is detected, `Virtual screen 1280x1024`) and then:
```
(EE) NVIDIA(GPU-0): Failed to initialize DMA.
(EE) NVIDIA(0): Failed to allocate push buffer
```
`nvidia-push` logs `Failed to initialize DMA.` from
`__nvPushTestPushBuffer()`: right after allocating a push buffer the
driver writes a NOP and waits (10 s) for the channel to consume it. Host
evidence shows why it never does:
* the X driver's channel registers (`class 0xc46f` GPFIFO channel,
`gpfifo=0x120090000`, `userd=0x350000` sysmem) and lives with
**`submissions=0`** - its published GP_PUT never advances in the host's
view, because the host needs the channel's VA->PA translations to read
its USERD/GPFIFO and that resolution comes from the guest-side scanner
that currently follows *CUDA clients only* (`targeted resolve
va=0x20400f000: full RAM scan found nothing`);
* on a busy boot (Xorg started while a CUDA battery ran) the channel's
completion-counter slot is rejected: `counter auto h=0x1000013 rank=0
pa=0x11c827ff0 rejected (slot owned by another channel)` - two
concurrent clients both derive rank 0's slot from the per-device
structural formula.
Fixing Xorg therefore needs (1) a platform-wide (non-CUDA) VA-resolution
path for the scanner or the host, and (2) counter-slot ownership keyed by
client identity (or a per-client slot domain), not just rank.
## 4. Wall 3 - the EGL/glsi engine-init gate (unchanged)
`eglMakeCurrent` returns `EGL_BAD_ALLOC (0x3003)`; the glsi engine-init
entry returns a "busy" status once, sets a state bit, and the eglcore
caller does not retry. Ten in-process retries (`egl_retry`) all fail, so
the state machine fails on a real input - the return-value trace of its
callees remains the way in (see
[userspace/cuda/570.144/docs/ANALYSIS-egl-engine-selection.md](../../userspace/cuda/570.144/docs/ANALYSIS-egl-engine-selection.md)).
## 5. Why this matters for the frame path
Everything downstream of a working GL/Vulkan context is already in place:
a guest publisher writes frames into the kvmfr ivshmem BAR
(`tools/vk/vk_frame.c`, `tools/guest-diag/xshm_pub.c` use the same layout
as the proven CUDA `frame_producer`), the host reader decodes them
(`tools/lg/frame_reader.py`) and ffmpeg assembles video. The missing
piece is a *working* graphics API on the guest GPU - the two subsystems
above.
## 6. UPDATE 2026-10-25 (RESUME #56): the wall is now fully bisected
Three new instruments settled the remaining questions:
* **An `LD_PRELOAD` nvKms ioctl logger** (`tools/guest-diag/nvkmstrace.c`)
decodes the failing `NVKMS_IOCTL_ALLOC_DEVICE` reply: status **5 =
`NVKMS_ALLOC_DEVICE_STATUS_CORE_CHANNEL_ALLOC_FAILED`**.
* **The QEMU gdb stub (`-gdb tcp::1234`) as a guest-kernel microscope**
(the guest kernel's `nvidia_modeset` symbols are in `/proc/kallsyms`):
breaking inside `nvAllocCoreChannelEvo` (the blob's offsets were mapped
from a disassembly of the shipped `nv-modeset-kernel.o_binary`) dumped
the driver's class list (42 classes; all `NVC57x` display classes are
present; **`NV927C_BASE_CHANNEL_DMA` (0x927c) is not** - it exists in no
HAL class list, so the "base channel" fallback path can never succeed).
Breaking at the capability parse showed the **capability notifier buffer
is all zeros** (nothing emulates the display engine that would fill it).
* **The caps-fill experiment**: with gdb writing `SYS_CAP = HEAD0_EXISTS`
and `SYS_CAPB = WINDOW0..7_EXISTS` into the caps buffer right before the
parse, the driver flips from the dead base path to the **window path**
(`nvRMAllocateWindowChannels`) - the hypothesis that `numWindows > 0`
unblocks the flow is confirmed.
The window path then fails one step later: `nvRmApiAlloc(display,
class=0xc57e NVC57E_WINDOW_CHANNEL_DMA)` returns **`NV_ERR_NOT_SUPPORTED`
(0x56) locally in the guest kernel** (it never reaches the emulated
GSP - no ALLOC RPC is logged). This matches the shipped driver's vGPU
architecture: the guest runs in vGPU mode (`IS_VIRTUAL`), where the
display/modeset channel classes are rejected (the reference source's
`_rpcAllocObjectPrologue` documents "Display classes are added for DCE,
vGPU Guest does not support those") - a vGPU guest's display is meant to
be emulated by the **host-side vGPU display service** (the vGPU RPC
display path), which the emulation does not implement.
**Consequence for the roadmap.** Guest Xorg/GL/Vulkan on this emulated
vGPU needs (a) the display capability fill on the host side (mechanics
proven by the gdb experiment) and (b) a vGPU display service: the
display-engine MMIO/RPC backend the guest expects from its host (the
equivalent of NVIDIA's vGPU display plugin), or the specific vGPU
configuration that makes the kernel accept the display channel class.
Until (b) exists the guest's NVIDIA display path is closed by design, and
GZDOOM-class workloads cannot render on the guest GPU.
## 7. UPDATE 2026-10-25 #2 (SESSION 66): the 0x56 pinned to the RS alloc path
A new instrument closed the gap: the guest's `nvidia.ko` (DKMS build) is
**not stripped** - copy it out (`/lib/modules/<kver>/updates/dkms/nvidia.ko`),
`objdump -t`/`-dr` it, compute the module's `.text` runtime base from any
`/proc/kallsyms` symbol, and every guest-kernel function/offset becomes
addressable for live gdb breakpoints on the QEMU stub. With it:
* `rpcAllocObject_v29_06`/`_rpcAllocObjectPrologue` are **exonerated** for
0xc57e (their reject list is 0x73/0xc67*/0xc770/0xc77d only; the register
mapping `hClass = r8d` was verified live).
* The failing chain is pinned exactly: `rmapiAllocWithSecInfo` **enters**
with class 0xc57e, calls `serverAllocResourceUnderLock`, and that
function returns **0x56** - *before* the GSP-RM alloc RPC
(`rpcRmApiAlloc_GSP`, called from under-lock+0x447; the emulation never
sees an ALLOC for 0xc57e). The same probe shows 0xc57d/0xc570/0xc573
return 0.
* So the last blocker for the window path is a **local RS-layer validation
inside `serverAllocResourceUnderLock`** (candidates: the inlined
privilege/class-support checks, `rmapiFixupAllocParams`' per-class fixup,
or `clientAllocResource`'s construct dispatch). The 0x56-producing
instruction is not yet pinned - a `stepi` loop over the QEMU stub is far
too slow and wedged the guest (recovered by a fresh cycle); the next
session should use conditional breakpoints on the function's call-return
sites (the approach in `gdbdiag19` works; the earlier run had truncated
addresses and needs the `0xffffffff00000000` base restored) or a
watchpoint on the status word.
**gdb-stub field notes**: the stub is single-client; killing an attached
gdb leaves the VM stopped (recover with a detach-only connect);
`stepi` loops over the stub freeze the guest's userspace for minutes -
prefer breakpoints; every probe run with the caps-fill breakpoint changes
the driver's path, so keep the fill in each experiment.
|
