1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
|
# Vulkan
> **Scope.** How a stock guest Vulkan application runs on the emulated vGPU:
> the guest-side stack, the submit/completion path, and the honest state of
> the work — including the one thing every Vulkan run on the emulated vGPU
> currently has in common: **the pipeline completes, and the frames are
> black**. The shared host machinery is on
> [`Graphics-Execution.md`](Graphics-Execution.md); the CUDA compute chain
> (further along) is on [`Execution-Frontier.md`](Execution-Frontier.md).
> Canonical sources:
> [`../docs/04-implementation/04-real-render-walls.md`](../docs/04-implementation/04-real-render-walls.md)
> and
> [`../docs/04-implementation/05-graphics-execution-plan.md`](../docs/04-implementation/05-graphics-execution-plan.md);
> the docs tree wins on disagreement.
Status: 2026-10-30, tree `master @ b4930168`. Tags: `[live]` observed with the
stock guest driver; `[wip]` partial; `[wall]` current blocker; `[target]`
defined success state.
## 1. The guest Vulkan stack
A Vulkan application loads the **loader** (`libvulkan.so`), which dispatches to
an **installable client driver (ICD)** — for NVIDIA GPUs, the NVIDIA Vulkan
driver. On Linux, Vulkan compute does not require an X server; the display
path does (through `nvidia-modeset`/DRM for presentation). The kernel-side
requirements are the same as CUDA's: `nvidia.ko`, the GSP client, channels,
memory objects and fences.
At device creation the driver also walks the **display/modeset device**
allocation path (the `ALLOC_DEVICE` class family) — on a vGPU-style device the
driver expects the host to provide those display classes, which the emulation
must answer for `vkCreateDevice` to succeed. Past that gate, the driver
behaves much like the compute stack: memory types through its VidHeap classes,
queues as channels, and work as method streams.
## 2. The submit and completion path (observed)
```mermaid
sequenceDiagram
participant App as Guest app
participant ICD as NVIDIA Vulkan driver
participant KMD as nvidia.ko + GSP client
participant VG as vgsp.ko emulation
participant GPU as Physical GPU (GR/CE)
App->>ICD: record command buffer, then vkQueueSubmit(fence)
ICD->>KMD: push buffer + GPFIFO entry + doorbell
KMD-->>VG: method stream (compute C5C0 shape + copy shapes)
VG->>VG: capture the section, translate to a vendor launch,<br/>relocate operands into reachable shadow pages
VG->>GPU: submit on a real channel (launch descriptor install)
GPU-->>VG: engine-authored release
VG-->>KMD: completion relay (MSI-X + poller)
KMD-->>App: fence released, vkWaitForFences returns
App->>App: reuse buffers, read back the frame
```
```text
app: vkQueueSubmit(fence)
-> NVIDIA driver: push buffer + GPFIFO entry + doorbell
-> vgsp.ko: capture section, translate to a vendor launch,
relocate operands into shadow pages, submit on a real channel
-> physical GPU: front-end fetch, engine executes, authors release
-> vgsp.ko: completion relay (MSI-X + poller) back to the guest
-> app: fence releases; frame readback proceeds
ASCII fallback above; the sequence shows one queue submit end to end.
```
Two mechanics make this work today: the emulation decodes the driver's
compute-dispatch shape — a **direct method stream** (object bind, window runs,
code runs, a subchannel data-port stream) with **no inline launch descriptor**
— and *translates* it into a launch the real engine accepts, and it derives
the submit's release groups from the driver's own push words so the fence
value is released when the engine completes (the "fence fold": the pre-built
GPFIFO continuation beyond the put pointer is swept and the submit's release
groups are applied forward-only).
## 3. Status on the emulated vGPU
| Item | State |
|---|---|
| `vkCreateDevice` on the emulated vGPU (display/modeset allocation path answered) | `[live]` |
| Full pipeline: instance, device, allocation, image/view/buffer/shader/descriptor, pipeline, command buffer, fence | `[live]` |
| Queue submit, engine-side execution of the translated launch, fence release, frame readback | `[live]` |
| The client saves frames end to end (`VKFRAME` evidence) | `[live]` |
| Non-black frames | `[wall]` |
| The driver's program (SASS) upload materialised by the emulation | `[wall]` |
| The engine's page tables mapping the program VA | `[wall]` |
## 4. Where the wall is, precisely
The client *believes* it uploaded its shader and dispatches with a valid
program address, but the bytes never appear in any source the emulation can
see: the driver delivers programmed SASS and descriptors through an **MME
program + inline-to-memory stream** on its copy/utility channel, and the
inline-data writes are not yet materialised for the graphics clients. Two
coupled facts define the remaining work:
1. **Delivery** — decode and execute the driver's inline upload so the
program page becomes non-zero (the same machinery already works for CUDA's
uploads; the graphics clients' shapes are the extension).
2. **Mapping** — the engine's page tables must map the program VA. With the
data-set gate bypassed, the engine fetches the launch and raises a real
MMU fault (`FAULT_PDE`) at the program VA — proof the channel, context and
fetch path are alive, and that mapping plus content is the missing pair.
When both land, the emulated vGPU should carry its first non-black Vulkan
frame — the milestone the whole graphics effort is converging on
([`Graphics-Execution.md`](Graphics-Execution.md) §6).
## 5. What this page is not claiming
* No rendered pixels yet — every frame saved so far is black.
* No performance numbers.
* `vkCreateDevice` success does not imply rendering; it means the device
model answers the driver's device-view correctly.
## 6. Related pages
* [`Graphics-Execution.md`](Graphics-Execution.md) — the shared lanes, the
`0xC597` graphics stream and the ranked walls.
* [`OpenGL.md`](OpenGL.md) — the GL path through the X server.
* [`DirectX.md`](DirectX.md) — the Windows/DirectX target.
* [`CUDA-Userspace-Driver.md`](CUDA-Userspace-Driver.md) — the userspace
driver model the Vulkan ICD mirrors in structure.
* [`Introduction-to-GMMU.md`](Introduction-to-GMMU.md) — translation and the
engines.
* [`../docs/04-implementation/04-real-render-walls.md`](../docs/04-implementation/04-real-render-walls.md) —
the measured wall ledger, with every experiment and its verdict.
|
