# The vGSP Execution Plane (vEP)
> **Canonical, maintained version of this page:**
> [`../docs/03-architecture/13-vexecution-plane.md`](../docs/03-architecture/13-vexecution-plane.md).
> This wiki page is the condensed public deep-dive: what the layer is, how a
> guest CUDA launch flows through it, where it stands today, and where it is
> going. It is condensed from the canonical document and its cited evidence;
> when the two disagree, the docs tree wins. Front door:
> [`../docs/README.md`](../docs/README.md).
Status: 2026-10-08, tree `master @ 1f571dda`. Tags: `[live]` proven with the
stock guest driver; `[gated]` implemented, default-off or parameter-gated;
`[wip]` partial; `[wall]` current blocker; `[target]` named success latch;
`[planned]` designed, not built.
## 1. What it is
**vGSP (virtual GSP)** is our host-side reimplementation of the GPU System
Processor interface: the virtual device plus the emulated GSP-RM service the
stock guest driver talks to, implemented by `vgsp.ko`. The **vGSP Execution
Plane (vEP)** is the half of vGSP that turns guest-authored GPU work into
physical execution and guest-visible completion: channel ring and USERD
discovery, GPU VA-to-PA truth, semaphore-release application, completion
counters and notifiers, QMD handling for forwarded launches, and submission to
the physical GPU's engines under the module-owned backend session.
The other half — the control plane (boot handshake, message queues, RPC, RM
objects, BAR shadows) — makes the guest driver *run*. The vEP makes the
guest's work *run*. The canonical document defines the name and rejects the
alternates ("work plane", "submission plane", "forwarding plane", ...) that
would collide with existing project vocabulary.
**Four work domains** organize the subsystem:
| Domain | Question | Implemented by |
|---|---|---|
| Scheduler | which channel's work runs, at what pace | doorbell steering, the per-channel pump, discovery cadence, one ordered workqueue, one in-flight request |
| Allocator | where rings, USERDs, counters, notifiers and scratch windows live | ring/USERD autopin and pins, content-keyed pairing, counter derive, ladder scratch/PT pools, the BAR1 vidmem shadow |
| Control | which RM objects/controls shape execution | channel and event registration, `SET_PAGE_DIRECTORY`, `PROMOTE_CTX`, engine topology, the mode matrix |
| Share | how clients/mdevs share the one GPU | mdev backend refcount, one real session under a single lock, one workqueue; per-mdev clients and fairness are planned |
## 2. The shape of the plane
```mermaid
flowchart TB
subgraph GUEST["Guest VM - stock software"]
APP["CUDA application -> libcuda -> nvidia.ko
push + QMD + GPFIFO + doorbell"]
end
subgraph VEP["vGSP Execution Plane inside vgsp.ko"]
DB["doorbell steering"] --> PUMP["channel pump"]
PUMP --> DISC["ring/USERD discovery + pins"]
DISC --> TRUTH["VA-to-PA truth
walk, rescue, pins"]
TRUTH --> DEC["push decode + QMD capture"]
DEC --> REL["release apply + counters + notifiers"]
DEC --> WQ["ordered workqueue
one in-flight request"]
WQ --> SUB["ladder submission
mode matrix"]
end
subgraph PHY["Physical backend"]
LAD["module-owned ladder channel"] --> GSP["real GSP-RM server"]
GSP --> ENG["GR / CE engines"]
end
APP -->|"MMIO + DMA"| DB
REL -->|"release, counter, event"| APP
SUB --> LAD
ENG -->|"engine-authored release"| SUB
```
ASCII fallback:
```text
guest: push + QMD + GPFIFO entry + doorbell (BAR0 0x88704)
-> pump (per-channel) -> ring/USERD discovery -> VA-to-PA truth -> decode
-> release apply + counter publish + notifier/event -> MSI-X -> guest wakes
-> (for real work) ordered workqueue -> ladder submission -> real GSP/engines
-> engine-authored release -> poll -> completion publication
```
## 3. One CUDA kernel launch, end to end
1. **Launch build (guest).** `libcuda`/`nvidia.ko` build the GR push: context
methods, the inline launch descriptor (QMD), SASS and parameter pointers,
and a release target. They append a GPFIFO entry and ring the BAR0
doorbell. `[live]` observed with the stock driver.
2. **Steering and truth (host).** The doorbell wakes the service thread; the
pump resolves the channel's ring VA and USERD, reads `GP_PUT`, fetches the
GPFIFO entry and decodes the method stream. Translations are read-back
proven from driver-published page directories, pins, or content-rescue.
`[live]` for CE; `[wip]` for GR (discovery now succeeds, see §4).
3. **Capture.** The decoder accumulates the QMD window and captures the launch
candidate; the sema service queues one request on the single ordered
workqueue. `[live]` observed (ring consumed, methods execute).
4. **Physical submit.** The backend relocates the QMD, copies program/param
pages into module shadow windows whose PTEs were planted before the channel
commit (the engine never sees a late PTE write — it caches 128-byte fetch
lines with no invalidate), builds the launch method block, publishes it to
the ladder channel's GPFIFO and rings the real doorbell. `[gated]`
default-off; latched on a proven real GR submission.
5. **Engine.** The front-end fetches the entry and executes methods; then the
**inline-QMD install** must materialise the descriptor for the engine to
read, currently **`[wall]`** — not observed. Context validity, destination
mapping and fetch/alignment are exonerated; the recovered-method-stream
form is the investigation.
6. **Completion.** The engine authors the QMD release word and the channel
release; the host applies the deferred release exactly once, publishes the
completion counter, clears the notifier and posts the channel event; MSI-X
wakes the guest. CE work is live in this shape `[gated]`; for GR the
completion-counter publication is the current second face of the wall.
## 4. How work is found: discovery, truth, completion
* **Ring/USERD discovery** has three cooperating sources: injected pins, a
kernel-native trusted walk of driver-published page directories, and
content-keyed discovery when neither names the ring. Content discovery
scans **both apertures** (emulated vidmem and guest RAM), runs the FB scan
first on a 2-second cadence, pairs a ring with the rank's staged ctx-init
push, and validates the pair before pinning. Since the 2026-10-08 re-arm
work (`7abd8f87`, `66388560`) the discovery no longer retires permanently
at a pass cap.
* **The semaphore service** decodes releases, translates their target pages
through the trusted resolver plus a driver-published-VAS pool resolver,
defers untranslatable releases with retry telemetry, and applies each
release exactly once — in software, or by letting the engine author it under
the relay (default off).
* **Completion counters** use a fixed 20-slot pool (a partial live channel
count poisoned slots before the fix); each decoded release lands in the
channel's exact slot. **Notifiers** are serviced by the pump, and family
events are delivered as RM events + MSI-X through the guest's registered
event objects.
## 5. Challenges, honestly
* **GR dispatch wall.** The forward path runs front-end methods, the context
is valid and the destination window is accepted — and the descriptor
install is still not observed. The mode matrix carries dozens of
single-variable falsifiers (promote shapes, prelude/PCAS form, local-memory
block, lane controls, shadow windows, pointer relocation).
* **Completion publication.** A boot where GR rings were discovered and
releases applied still saw the guest re-queue the same push and give up. A
later `sema_dbg=Y` run settled it: the release was written to the exact
slot, the notifier cleared and the event posted, while the completion
counter publication never wrote because the fixed-total autoreg mapping
makes it a structural no-op. The leading reading is the counter-slot
publication; the GP_GET write-back is the secondary, still-open candidate.
Both are debug-gated today.
* **Reproducibility (ISSUE-0001).** The green/stall split is per-boot. The
first fix wave landed and the earlier CE freeze is gone; the stall moved to
GR ring discovery, which the latest discovery work addresses. Acceptance
is 10 cold boots green, pending.
* **Multiple clients (ISSUE-0002).** A second CUDA process wedges in guest
UVM because host per-client sema/GR state is not reset when the guest
re-runs its per-client boot handshake. Per-client state containers and
reset are the designed path.
## 6. Portability
The **architecture** ports: observe pushes, resolve truth, decode,
execute/forward, publish completion. So does the **RPC/RM object model** and
the mode-matrix practice. What does **not** port is address-level fact: ring
conventions, VA windows, QMD offsets, context-image layout, PTE encodings,
engine inventories and scratch layouts must be re-derived per driver version
and GPU architecture by measurement, never guessed. A new version enters as a
profile plus a measured delta; driver 610 additionally sits behind an explicit
human approval gate.
## 7. Future
The planned next steps are reproducibility acceptance, the descriptor-install
and completion-publication investigations (offline-first with the RISC-V GSP-RM debugger
where possible), the arbitrary-kernel acceptance suite, then the near-native
performance program. The unsolved future classes — multi-tenant fairness,
fault replay/cancel, context switching between tenants, hostile-guest
hardening, and version drift — each have a proactive design direction in the
canonical document. 3D graphics reuses this pipeline once compute dispatch is
proven; it adds engine breadth (video/image), more channels and synchronisation
volume, and a much larger method vocabulary, but not a new plane.
---
*Canonical document: [`docs/03-architecture/13-vexecution-plane.md`](../docs/03-architecture/13-vexecution-plane.md).
See also: [full system picture](../docs/03-architecture/09-full-system-picture.md),
[physical backend guest wiring](../docs/03-architecture/11-physical-backend-guest-wiring.md),
[the execution frontier](../docs/04-implementation/03-execution-frontier.md).*