1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
|
# The vGSP Execution Plane (vEP)
> **Canonical, maintained version of this page:**
> [`../docs/03-architecture/13-vexecution-plane.md`](../docs/03-architecture/13-vexecution-plane.md).
> This wiki page is the condensed public deep-dive: what the layer is, how a
> guest CUDA launch flows through it, where it stands today, and where it is
> going. It is condensed from the canonical document and its cited evidence;
> when the two disagree, the docs tree wins. Front door:
> [`../docs/README.md`](../docs/README.md).
Status: 2026-10-08, tree `master @ 1f571dda`. Tags: `[live]` proven with the
stock guest driver; `[gated]` implemented, default-off or parameter-gated;
`[wip]` partial; `[wall]` current blocker; `[target]` named success latch;
`[planned]` designed, not built.
## 1. What it is
**vGSP (virtual GSP)** is our host-side reimplementation of the GPU System
Processor interface: the virtual device plus the emulated GSP-RM service the
stock guest driver talks to, implemented by `vgsp.ko`. The **vGSP Execution
Plane (vEP)** is the half of vGSP that turns guest-authored GPU work into
physical execution and guest-visible completion: channel ring and USERD
discovery, GPU VA-to-PA truth, semaphore-release application, completion
counters and notifiers, QMD handling for forwarded launches, and submission to
the physical GPU's engines under the module-owned backend session.
The other half — the control plane (boot handshake, message queues, RPC, RM
objects, BAR shadows) — makes the guest driver *run*. The vEP makes the
guest's work *run*. The canonical document defines the name and rejects the
alternates ("work plane", "submission plane", "forwarding plane", ...) that
would collide with existing project vocabulary.
**Four work domains** organize the subsystem:
| Domain | Question | Implemented by |
|---|---|---|
| Scheduler | which channel's work runs, at what pace | doorbell steering, the per-channel pump, discovery cadence, one ordered workqueue, one in-flight request |
| Allocator | where rings, USERDs, counters, notifiers and scratch windows live | ring/USERD autopin and pins, content-keyed pairing, counter derive, ladder scratch/PT pools, the BAR1 vidmem shadow |
| Control | which RM objects/controls shape execution | channel and event registration, `SET_PAGE_DIRECTORY`, `PROMOTE_CTX`, engine topology, the mode matrix |
| Share | how clients/mdevs share the one GPU | mdev backend refcount, one real session under a single lock, one workqueue; per-mdev clients and fairness are planned |
## 2. The shape of the plane
```mermaid
flowchart TB
subgraph GUEST["Guest VM - stock software"]
APP["CUDA application -> libcuda -> nvidia.ko<br/>push + QMD + GPFIFO + doorbell"]
end
subgraph VEP["vGSP Execution Plane inside vgsp.ko"]
DB["doorbell steering"] --> PUMP["channel pump"]
PUMP --> DISC["ring/USERD discovery + pins"]
DISC --> TRUTH["VA-to-PA truth<br/>walk, rescue, pins"]
TRUTH --> DEC["push decode + QMD capture"]
DEC --> REL["release apply + counters + notifiers"]
DEC --> WQ["ordered workqueue<br/>one in-flight request"]
WQ --> SUB["ladder submission<br/>mode matrix"]
end
subgraph PHY["Physical backend"]
LAD["module-owned ladder channel"] --> GSP["real GSP-RM server"]
GSP --> ENG["GR / CE engines"]
end
APP -->|"MMIO + DMA"| DB
REL -->|"release, counter, event"| APP
SUB --> LAD
ENG -->|"engine-authored release"| SUB
```
ASCII fallback:
```text
guest: push + QMD + GPFIFO entry + doorbell (BAR0 0x88704)
-> pump (per-channel) -> ring/USERD discovery -> VA-to-PA truth -> decode
-> release apply + counter publish + notifier/event -> MSI-X -> guest wakes
-> (for real work) ordered workqueue -> ladder submission -> real GSP/engines
-> engine-authored release -> poll -> completion publication
```
## 3. One CUDA kernel launch, end to end
1. **Launch build (guest).** `libcuda`/`nvidia.ko` build the GR push: context
methods, the inline launch descriptor (QMD), SASS and parameter pointers,
and a release target. They append a GPFIFO entry and ring the BAR0
doorbell. `[live]` observed with the stock driver.
2. **Steering and truth (host).** The doorbell wakes the service thread; the
pump resolves the channel's ring VA and USERD, reads `GP_PUT`, fetches the
GPFIFO entry and decodes the method stream. Translations are read-back
proven from driver-published page directories, pins, or content-rescue.
`[live]` for CE; `[wip]` for GR (discovery now succeeds, see §4).
3. **Capture.** The decoder accumulates the QMD window and captures the launch
candidate; the sema service queues one request on the single ordered
workqueue. `[live]` observed (ring consumed, methods execute).
4. **Physical submit.** The backend relocates the QMD, copies program/param
pages into module shadow windows whose PTEs were planted before the channel
commit (the engine never sees a late PTE write — it caches 128-byte fetch
lines with no invalidate), builds the launch method block, publishes it to
the ladder channel's GPFIFO and rings the real doorbell. `[gated]`
default-off; latched on a proven real GR submission.
5. **Engine.** The front-end fetches the entry and executes methods; then the
**inline-QMD install** must materialise the descriptor for the engine to
read, currently **`[wall]`** — not observed. Context validity, destination
mapping and fetch/alignment are exonerated; the recovered-method-stream
form is the investigation.
6. **Completion.** The engine authors the QMD release word and the channel
release; the host applies the deferred release exactly once, publishes the
completion counter, clears the notifier and posts the channel event; MSI-X
wakes the guest. CE work is live in this shape `[gated]`; for GR the
completion-counter publication is the current second face of the wall.
## 4. How work is found: discovery, truth, completion
* **Ring/USERD discovery** has three cooperating sources: injected pins, a
kernel-native trusted walk of driver-published page directories, and
content-keyed discovery when neither names the ring. Content discovery
scans **both apertures** (emulated vidmem and guest RAM), runs the FB scan
first on a 2-second cadence, pairs a ring with the rank's staged ctx-init
push, and validates the pair before pinning. Since the 2026-10-08 re-arm
work (`7abd8f87`, `66388560`) the discovery no longer retires permanently
at a pass cap.
* **The semaphore service** decodes releases, translates their target pages
through the trusted resolver plus a driver-published-VAS pool resolver,
defers untranslatable releases with retry telemetry, and applies each
release exactly once — in software, or by letting the engine author it under
the relay (default off).
* **Completion counters** use a fixed 20-slot pool (a partial live channel
count poisoned slots before the fix); each decoded release lands in the
channel's exact slot. **Notifiers** are serviced by the pump, and family
events are delivered as RM events + MSI-X through the guest's registered
event objects.
## 5. Challenges, honestly
* **GR dispatch wall.** The forward path runs front-end methods, the context
is valid and the destination window is accepted — and the descriptor
install is still not observed. The mode matrix carries dozens of
single-variable falsifiers (promote shapes, prelude/PCAS form, local-memory
block, lane controls, shadow windows, pointer relocation).
* **Completion publication.** A boot where GR rings were discovered and
releases applied still saw the guest re-queue the same push and give up. A
later `sema_dbg=Y` run settled it: the release was written to the exact
slot, the notifier cleared and the event posted, while the completion
counter publication never wrote because the fixed-total autoreg mapping
makes it a structural no-op. The leading reading is the counter-slot
publication; the GP_GET write-back is the secondary, still-open candidate.
Both are debug-gated today.
* **Reproducibility (ISSUE-0001).** The green/stall split is per-boot. The
first fix wave landed and the earlier CE freeze is gone; the stall moved to
GR ring discovery, which the latest discovery work addresses. Acceptance
is 10 cold boots green, pending.
* **Multiple clients (ISSUE-0002).** A second CUDA process wedges in guest
UVM because host per-client sema/GR state is not reset when the guest
re-runs its per-client boot handshake. Per-client state containers and
reset are the designed path.
## 6. Portability
The **architecture** ports: observe pushes, resolve truth, decode,
execute/forward, publish completion. So does the **RPC/RM object model** and
the mode-matrix practice. What does **not** port is address-level fact: ring
conventions, VA windows, QMD offsets, context-image layout, PTE encodings,
engine inventories and scratch layouts must be re-derived per driver version
and GPU architecture by measurement, never guessed. A new version enters as a
profile plus a measured delta; driver 610 additionally sits behind an explicit
human approval gate.
## 7. Future
The planned next steps are reproducibility acceptance, the descriptor-install
and completion-publication investigations (offline-first with the RISC-V GSP-RM debugger
where possible), the arbitrary-kernel acceptance suite, then the near-native
performance program. The unsolved future classes — multi-tenant fairness,
fault replay/cancel, context switching between tenants, hostile-guest
hardening, and version drift — each have a proactive design direction in the
canonical document. 3D graphics reuses this pipeline once compute dispatch is
proven; it adds engine breadth (video/image), more channels and synchronisation
volume, and a much larger method vocabulary, but not a new plane.
---
*Canonical document: [`docs/03-architecture/13-vexecution-plane.md`](../docs/03-architecture/13-vexecution-plane.md).
See also: [full system picture](../docs/03-architecture/09-full-system-picture.md),
[physical backend guest wiring](../docs/03-architecture/11-physical-backend-guest-wiring.md),
[the execution frontier](../docs/04-implementation/03-execution-frontier.md).*
|
