summaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-24 23:00:39 -0400
committerArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-24 23:00:39 -0400
commit148e65754832ee29e8d3ae13ded219402e2d0fbd (patch)
treec4926ffbb90d16f70cb68784abab193723a00682
parentb43291c5420964424bfcf0aab0f1454c2c3b0613 (diff)
downloadopen-iov-148e65754832ee29e8d3ae13ded219402e2d0fbd.tar.gz
open-iov-148e65754832ee29e8d3ae13ded219402e2d0fbd.zip
docs: record the real-render walls measured this session (Vulkan ICD enumerates on the emulated vGPU; vkCreateDevice fails in nvidia-modeset's ALLOC_DEVICE past the new NV0073 head/connector shim, residual wall = the display core/modeset channel objects; Xorg ScreenInit fails in nvidia-push's 10 s channel test with submissions=0 and unresolved X-channel VAs - the guest scanner serves CUDA clients only - plus a rank-0 counter-slot collision with a live CUDA client; EGL/glsi unchanged) and link the page from the README
-rw-r--r--docs/04-implementation/04-real-render-walls.md113
-rw-r--r--docs/README.md1
2 files changed, 114 insertions, 0 deletions
diff --git a/docs/04-implementation/04-real-render-walls.md b/docs/04-implementation/04-real-render-walls.md
new file mode 100644
index 0000000..3570769
--- /dev/null
+++ b/docs/04-implementation/04-real-render-walls.md
@@ -0,0 +1,113 @@
+# Real-render walls: Vulkan and Xorg on the emulated vGPU
+
+Status: OPEN (2026-10-29, SESSION 64). This page records the measured state
+of the guest's *display/graphics* consumers - the Vulkan ICD and the Xorg
+server - on the emulated vGPU, with the exact evidence and the next
+subsystem to implement. It complements
+[05-operations/06-guest-frame-path-and-looking-glass.md](../05-operations/06-guest-frame-path-and-looking-glass.md)
+(which covers the CUDA -> ivshmem -> host frame transport that is already
+proven).
+
+## 1. What works today
+
+* The guest's shipped 570.144 Vulkan userspace **loads and enumerates**:
+ `vkCreateInstance` succeeds and two physical devices are reported -
+ `NVIDIA GeForce RTX 2080 SUPER` (discrete, Vulkan 1.4.303) and the
+ software `llvmpipe` - from `/etc/vulkan/icd.d/nvidia_icd.json`. Probes:
+ `tools/vk/vk_probe.py`, `tools/vk/vk_probe2.py`.
+* `eglInitialize` succeeds with `vendor=NVIDIA` on the guest EGL device
+ platform (the EGL probe `tools/guest-diag/egl_retry.c` and the older
+ `egl_frame`).
+* CUDA is unaffected (its clients run through the same emulated GSP; see
+ the battery evidence in the handoff).
+
+## 2. Wall 1 - `vkCreateDevice` fails in nvidia-modeset's ALLOC_DEVICE
+
+`vkCreateDevice` returns `VK_ERROR_INITIALIZATION_FAILED`. An
+`LD_PRELOAD` ioctl logger (`tools/guest-diag/nvkmstrace.c`) shows the
+ICD's first nvKms ioctl failing:
+
+```
+NVKMS_IOCTL cmd=0 size=1488 addr=0x... rc=-1 # NVKMS_IOCTL_ALLOC_DEVICE
+```
+
+i.e. the kernel's `nvKmsAllocDevice` (in `nvidia-modeset`) returned a
+non-success status; the userspace frees everything and reports the
+generic initialization error. Reading the driver's own modeset source
+(`nvkms-rm.c`) gives the alloc order and the two gates we hit and fixed:
+
+1. `ProbeHeadCountAndWindowAssignment()` -> `NV0073_CTRL_CMD_SYSTEM_GET_NUM_HEADS`
+ (`0x730102`): a zero reply logs `No heads found on board!` and fails the
+ alloc. The emulated GSP RSP answered all NV0073 controls through the
+ generic zero-reply path.
+2. After the heads: `NoValidDisplays()` -> `NV0073_CTRL_CMD_SYSTEM_GET_SUPPORTED`
+ (`0x730107`) must report at least one display id, and the connector
+ path (`SPECIFIC_OR_GET_INFO` `0x73028b`, `SPECIFIC_GET_CONNECTOR_DATA`
+ `0x730250`, `DFP_GET_INFO` `0x731140`) must describe a connector.
+
+The host GSP-RM emulation now implements a minimal headless display:
+one head (`0x730102`/`0x730287`), flexible window assignment (`0x7302ad`),
+one static HDMI connector on display id 0 (SOR + single-link TMDS, which
+avoids the DP-AUX library), plus DDC support. With this shim the alloc
+gets past heads and `NoValidDisplays` and proceeds through the connector
+queries and the display-object allocations (classes `0xc570`, `0xc57d`,
+`0xc372`, controls `0x50700709`, `0x73010c`, `0xc3700201`) before failing
+later in the display-core/modeset-channel path. That remaining subsystem
+is the wall: the modeset core channel and the per-display dpy objects
+must be emulated (a coupled object + channel + memory set whose exact
+wire forms should be captured from a real-GPU session).
+
+A related experiment ruled out one shortcut: booting the guest with
+`nvidia-modeset`/`nvidia-drm` blacklisted (no `/dev/nvidia-modeset`) does
+not avoid the wall - the ICD's nvKms client still issues ALLOC_DEVICE and
+fails, so the display path cannot be sidestepped by disabling KMS.
+
+## 3. Wall 2 - Xorg ScreenInit fails in nvidia-push's channel test
+
+Xorg starts (the GPU is detected, `Virtual screen 1280x1024`) and then:
+
+```
+(EE) NVIDIA(GPU-0): Failed to initialize DMA.
+(EE) NVIDIA(0): Failed to allocate push buffer
+```
+
+`nvidia-push` logs `Failed to initialize DMA.` from
+`__nvPushTestPushBuffer()`: right after allocating a push buffer the
+driver writes a NOP and waits (10 s) for the channel to consume it. Host
+evidence shows why it never does:
+
+* the X driver's channel registers (`class 0xc46f` GPFIFO channel,
+ `gpfifo=0x120090000`, `userd=0x350000` sysmem) and lives with
+ **`submissions=0`** - its published GP_PUT never advances in the host's
+ view, because the host needs the channel's VA->PA translations to read
+ its USERD/GPFIFO and that resolution comes from the guest-side scanner
+ that currently follows *CUDA clients only* (`targeted resolve
+ va=0x20400f000: full RAM scan found nothing`);
+* on a busy boot (Xorg started while a CUDA battery ran) the channel's
+ completion-counter slot is rejected: `counter auto h=0x1000013 rank=0
+ pa=0x11c827ff0 rejected (slot owned by another channel)` - two
+ concurrent clients both derive rank 0's slot from the per-device
+ structural formula.
+
+Fixing Xorg therefore needs (1) a platform-wide (non-CUDA) VA-resolution
+path for the scanner or the host, and (2) counter-slot ownership keyed by
+client identity (or a per-client slot domain), not just rank.
+
+## 4. Wall 3 - the EGL/glsi engine-init gate (unchanged)
+
+`eglMakeCurrent` returns `EGL_BAD_ALLOC (0x3003)`; the glsi engine-init
+entry returns a "busy" status once, sets a state bit, and the eglcore
+caller does not retry. Ten in-process retries (`egl_retry`) all fail, so
+the state machine fails on a real input - the return-value trace of its
+callees remains the way in (see
+[userspace/cuda/570.144/docs/ANALYSIS-egl-engine-selection.md](../../userspace/cuda/570.144/docs/ANALYSIS-egl-engine-selection.md)).
+
+## 5. Why this matters for the frame path
+
+Everything downstream of a working GL/Vulkan context is already in place:
+a guest publisher writes frames into the kvmfr ivshmem BAR
+(`tools/vk/vk_frame.c`, `tools/guest-diag/xshm_pub.c` use the same layout
+as the proven CUDA `frame_producer`), the host reader decodes them
+(`tools/lg/frame_reader.py`) and ffmpeg assembles video. The missing
+piece is a *working* graphics API on the guest GPU - the two subsystems
+above. \ No newline at end of file
diff --git a/docs/README.md b/docs/README.md
index 27e04e9..b423762 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -184,6 +184,7 @@ Read in order if you are new. Jump directly if you know what you need.
| 24 | [risc-v-gsp-rm-debugger user guide (experimental)](05-operations/04-risc-v-gsp-rm-debugger-user-guide.md) — implemented, documented and tested ([audit](../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md); [value demos](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md)) | developers, analysts | 15 min |
| 25 | [Defensive Operations Runbook](RUNBOOK.md) | maintainers, operators | 20 min |
| 25a | [GPUBreach-class audit and threat model](05-operations/05-gpubreach-audit.md) | developers, auditors | 15 min |
+| 25b | [Real-render walls: Vulkan and Xorg on the emulated vGPU](04-implementation/04-real-render-walls.md) | developers, debuggers | 10 min |
| 26 | [Engineering operations](eng-ops.md) — tool admission and workflows (Part I) plus live physical-backend operations (Part II) | maintainers, developers | 20 min |
| 27 | [Hardware and software matrix](06-portability/hardware-software-matrix.md) | developers, integrators | 20 min |
| 28 | [Developer onboarding and standards](07-developers/onboarding-and-standards.md) | contributors | 20 min |