# The vGSP Execution Plane (vEP)
> **Canonical, maintained version of this page:**
> [`../docs/03-architecture/13-vexecution-plane.md`](../docs/03-architecture/13-vexecution-plane.md).
> This wiki page is the condensed public deep-dive: what the layer is, how a
> guest CUDA launch flows through it, where it stands today, and where it is
> going. It is condensed from the canonical document and its cited evidence;
> when the two disagree, the docs tree wins. Front door:
> [`../docs/README.md`](../docs/README.md).
Status: 2026-10-30, tree `master @ b4930168`. Tags: `[live]` proven with the
stock guest driver; `[gated]` implemented, default-off or parameter-gated;
`[wip]` partial; `[wall]` current blocker; `[target]` named success latch;
`[planned]` designed, not built.
## 1. What it is
**vGSP (virtual GSP)** is our host-side reimplementation of the GPU System
Processor interface: the virtual device plus the emulated GSP-RM service the
stock guest driver talks to, implemented by `vgsp.ko`. The **vGSP Execution
Plane (vEP)** is the half of vGSP that turns guest-authored GPU work into
physical execution and guest-visible completion: channel ring and USERD
discovery, GPU VA-to-PA truth, semaphore-release application, completion
counters and notifiers, QMD handling for forwarded launches, and submission to
the physical GPU's engines under the module-owned backend session.
The other half — the control plane (boot handshake, message queues, RPC, RM
objects, BAR shadows) — makes the guest driver *run*. The vEP makes the
guest's work *run*. The canonical document defines the name and rejects the
alternates ("work plane", "submission plane", "forwarding plane", ...) that
would collide with existing project vocabulary.
**Four work domains** organize the subsystem:
| Domain | Question | Implemented by |
|---|---|---|
| Scheduler | which channel's work runs, at what pace | doorbell steering, the per-channel pump, discovery cadence, one ordered workqueue, one in-flight request |
| Allocator | where rings, USERDs, counters, notifiers and scratch windows live | ring/USERD autopin and pins, content-keyed pairing, counter derive, ladder scratch/PT pools, the BAR1 vidmem shadow |
| Control | which RM objects/controls shape execution | channel and event registration, `SET_PAGE_DIRECTORY`, `PROMOTE_CTX`, engine topology, the mode matrix |
| Share | how clients/mdevs share the one GPU | mdev backend refcount, one real session under a single lock, one workqueue; two guests with different emulated VRAM sizes coexist at the device level, per-guest execution isolation is in progress |
## 2. The shape of the plane
```mermaid
flowchart TB
subgraph GUEST["Guest VM - stock software"]
APP["CUDA application -> libcuda -> nvidia.ko
push + QMD + GPFIFO + doorbell"]
end
subgraph VEP["vGSP Execution Plane inside vgsp.ko"]
DB["doorbell steering"] --> PUMP["channel pump"]
PUMP --> DISC["ring/USERD discovery + pins"]
DISC --> TRUTH["VA-to-PA truth
walk, rescue, pins"]
TRUTH --> DEC["push decode + QMD capture"]
DEC --> REL["release apply + counters + notifiers"]
DEC --> WQ["ordered workqueue
one in-flight request"]
WQ --> SUB["ladder submission
mode matrix"]
end
subgraph PHY["Physical backend"]
LAD["module-owned ladder channel"] --> GSP["real GSP-RM server"]
GSP --> ENG["GR / CE engines"]
end
APP -->|"MMIO + DMA"| DB
REL -->|"release, counter, event"| APP
SUB --> LAD
ENG -->|"engine-authored release"| SUB
```
ASCII fallback:
```text
guest: push + QMD + GPFIFO entry + doorbell (BAR0 0x88704)
-> pump (per-channel) -> ring/USERD discovery -> VA-to-PA truth -> decode
-> release apply + counter publish + notifier/event -> MSI-X -> guest wakes
-> (for real work) ordered workqueue -> ladder submission -> real GSP/engines
-> engine-authored release -> poll -> completion publication
```
## 3. One CUDA kernel launch, end to end
1. **Launch build (guest).** `libcuda`/`nvidia.ko` build the GR push: context
methods, the inline launch descriptor (QMD), SASS and parameter pointers,
and a release target. They append a GPFIFO entry and ring the BAR0
doorbell. `[live]` observed with the stock driver.
2. **Steering and truth (host).** The doorbell wakes the service thread; the
pump resolves the channel's ring VA and USERD, reads `GP_PUT`, fetches the
GPFIFO entry and decodes the method stream. Translations are read-back
proven from driver-published page directories, pins, or content-rescue.
`[live]` for CE; `[wip]` for GR (discovery now succeeds, see §4).
3. **Capture.** The decoder accumulates the QMD window and captures the launch
candidate; the sema service queues one request on the single ordered
workqueue. `[live]` observed (ring consumed, methods execute).
4. **Physical submit.** The backend relocates the QMD, copies program/param
pages into module shadow windows whose PTEs were planted before the channel
commit (the engine never sees a late PTE write — it caches 128-byte fetch
lines with no invalidate), builds the launch method block, publishes it to
the ladder channel's GPFIFO and rings the real doorbell. `[live]` —
exercised on every verified guest launch (default-off while the seams are
validated).
5. **Engine.** The front-end fetches the entry and executes methods; the
inline launch descriptor installs and the engine reads it, the grid
dispatches to the SMs, and the engine authors the release. `[live]` —
context validity, destination mapping and fetch/alignment were exonerated
on the way.
6. **Completion.** The engine authors the QMD release word and the channel
release; the host applies the deferred release exactly once, publishes the
completion counter, clears the notifier and posts the channel event; MSI-X
wakes the guest. `[live]` for CE and GR: engine-authored completions relay
to the guest and its synchronisation waits resolve.
## 4. How work is found: discovery, truth, completion
* **Ring/USERD discovery** has three cooperating sources: injected pins, a
kernel-native trusted walk of driver-published page directories, and
content-keyed discovery when neither names the ring. Content discovery
scans **both apertures** (emulated vidmem and guest RAM), runs the FB scan
first on a 2-second cadence, pairs a ring with the rank's staged ctx-init
push, and validates the pair before pinning. Since the 2026-10-08 re-arm
work (`7abd8f87`, `66388560`) the discovery no longer retires permanently
at a pass cap.
* **The semaphore service** decodes releases, translates their target pages
through the trusted resolver plus a driver-published-VAS pool resolver,
defers untranslatable releases with retry telemetry, and applies each
release exactly once — in software, or by letting the engine author it under
the relay (default off).
* **Completion counters** use a fixed 20-slot pool (a partial live channel
count poisoned slots before the fix); each decoded release lands in the
channel's exact slot. **Notifiers** are serviced by the pump, and family
events are delivered as RM events + MSI-X through the guest's registered
event objects.
## 5. Challenges, honestly
* **The staged-transfer class (compute).** Some large multi-chunk CUDA clients
still mis-verify; the failure modes are proven (a release applied before
the op it owes, a ledger-vs-writer page divergence, a readback racing a
writeback) and the fixes are in flight.
* **Every-boot reproducibility.** The boot-poisoning defect (wrong
status-queue consumption cursor under the driver's swap mode) and the
post-reboot ring-block ownership defect were root-caused and fixed; green
clients reproduce across fresh boots, and the formal every-boot acceptance
campaign is the remaining work.
* **Multiple clients and per-guest isolation.** Two guests with different
emulated VRAM sizes coexist at the device level; per-guest execution
isolation for simultaneous CUDA guests is in progress.
* **The graphics pixel frontier.** Graphics work now rides this plane in two
lanes (verbatim 3D method-stream replay and synthesized launches) and the
engine retires real graphics streams — but no rendered (non-black) frame
has been produced yet. Program delivery and program-VA mapping are the
named blockers; the lanes and the ladder are on
[`Graphics-Execution.md`](Graphics-Execution.md).
## 6. Portability
The **architecture** ports: observe pushes, resolve truth, decode,
execute/forward, publish completion. So does the **RPC/RM object model** and
the mode-matrix practice. What does **not** port is address-level fact: ring
conventions, VA windows, QMD offsets, context-image layout, PTE encodings,
engine inventories and scratch layouts must be re-derived per driver version
and GPU architecture by measurement, never guessed. A new version enters as a
profile plus a measured delta; driver 610 additionally sits behind an explicit
human approval gate.
## 7. Future
The planned next steps are the staged-transfer fixes, the every-boot
acceptance campaign, per-guest isolation, and the graphics pixel frontier
(offline-first with the RISC-V GSP-RM debugger where possible), then the
near-native performance program. The unsolved future classes — multi-tenant
fairness, fault replay/cancel, context switching between tenants, hostile-guest
hardening, and version drift — each have a proactive design direction in the
canonical document. 3D graphics reuses this pipeline and now executes across
two lanes ([`Graphics-Execution.md`](Graphics-Execution.md)); it adds engine
breadth (video/image), more channels and synchronisation volume, and a much
larger method vocabulary, but not a new plane.
---
*Canonical document: [`docs/03-architecture/13-vexecution-plane.md`](../docs/03-architecture/13-vexecution-plane.md).
See also: [full system picture](../docs/03-architecture/09-full-system-picture.md),
[physical backend guest wiring](../docs/03-architecture/11-physical-backend-guest-wiring.md),
[the execution frontier](../docs/04-implementation/03-execution-frontier.md).*