diff options
Diffstat (limited to 'wiki/Execution-Plane.md')
| -rw-r--r-- | wiki/Execution-Plane.md | 179 |
1 files changed, 179 insertions, 0 deletions
diff --git a/wiki/Execution-Plane.md b/wiki/Execution-Plane.md new file mode 100644 index 0000000..0f6eafe --- /dev/null +++ b/wiki/Execution-Plane.md @@ -0,0 +1,179 @@ +# The vGSP Execution Plane (vEP) + +> **Canonical, maintained version of this page:** +> [`../docs/03-architecture/13-vexecution-plane.md`](../docs/03-architecture/13-vexecution-plane.md). +> This wiki page is the condensed public deep-dive: what the layer is, how a +> guest CUDA launch flows through it, where it stands today, and where it is +> going. It is condensed from the canonical document and its cited evidence; +> when the two disagree, the docs tree wins. Front door: +> [`../docs/README.md`](../docs/README.md). + +Status: 2026-10-08, tree `master @ 66388560`. Tags: `[live]` proven with the +stock guest driver; `[gated]` implemented, default-off or parameter-gated; +`[wip]` partial; `[wall]` current blocker; `[target]` named success latch; +`[planned]` designed, not built. + +## 1. What it is + +**vGSP (virtual GSP)** is our host-side reimplementation of the GPU System +Processor interface: the virtual device plus the emulated GSP-RM service the +stock guest driver talks to, implemented by `vgsp.ko`. The **vGSP Execution +Plane (vEP)** is the half of vGSP that turns guest-authored GPU work into +physical execution and guest-visible completion: channel ring and USERD +discovery, GPU VA-to-PA truth, semaphore-release application, completion +counters and notifiers, QMD handling for forwarded launches, and submission to +the physical GPU's engines under the module-owned backend session. + +The other half — the control plane (boot handshake, message queues, RPC, RM +objects, BAR shadows) — makes the guest driver *run*. The vEP makes the +guest's work *run*. The canonical document defines the name and rejects the +alternates ("work plane", "submission plane", "forwarding plane", ...) that +would collide with existing project vocabulary. + +**Four work domains** organize the subsystem: + +| Domain | Question | Implemented by | +|---|---|---| +| Scheduler | which channel's work runs, at what pace | doorbell steering, the per-channel pump, discovery cadence, one ordered workqueue, one in-flight request | +| Allocator | where rings, USERDs, counters, notifiers and scratch windows live | ring/USERD autopin and pins, content-keyed pairing, counter derive, ladder scratch/PT pools, the BAR1 vidmem shadow | +| Control | which RM objects/controls shape execution | channel and event registration, `SET_PAGE_DIRECTORY`, `PROMOTE_CTX`, engine topology, the mode matrix | +| Share | how clients/mdevs share the one GPU | mdev backend refcount, one real session under a single lock, one workqueue; per-mdev clients and fairness are planned | + +## 2. The shape of the plane + +```mermaid +flowchart TB + subgraph GUEST["Guest VM - stock software"] + APP["CUDA application -> libcuda -> nvidia.ko<br/>push + QMD + GPFIFO + doorbell"] + end + subgraph VEP["vGSP Execution Plane inside vgsp.ko"] + DB["doorbell steering"] --> PUMP["channel pump"] + PUMP --> DISC["ring/USERD discovery + pins"] + DISC --> TRUTH["VA-to-PA truth<br/>walk, rescue, pins"] + TRUTH --> DEC["push decode + QMD capture"] + DEC --> REL["release apply + counters + notifiers"] + DEC --> WQ["ordered workqueue<br/>one in-flight request"] + WQ --> SUB["ladder submission<br/>mode matrix"] + end + subgraph PHY["Physical backend"] + LAD["module-owned ladder channel"] --> GSP["real GSP-RM server"] + GSP --> ENG["GR / CE engines"] + end + APP -->|"MMIO + DMA"| DB + REL -->|"release, counter, event"| APP + SUB --> LAD + ENG -->|"engine-authored release"| SUB +``` + +ASCII fallback: + +```text +guest: push + QMD + GPFIFO entry + doorbell (BAR0 0x88704) + -> pump (per-channel) -> ring/USERD discovery -> VA-to-PA truth -> decode + -> release apply + counter publish + notifier/event -> MSI-X -> guest wakes + -> (for real work) ordered workqueue -> ladder submission -> real GSP/engines + -> engine-authored release -> poll -> completion publication +``` + +## 3. One CUDA kernel launch, end to end + +1. **Launch build (guest).** `libcuda`/`nvidia.ko` build the GR push: context + methods, the inline launch descriptor (QMD), SASS and parameter pointers, + and a release target. They append a GPFIFO entry and ring the BAR0 + doorbell. `[live]` observed with the stock driver. +2. **Steering and truth (host).** The doorbell wakes the service thread; the + pump resolves the channel's ring VA and USERD, reads `GP_PUT`, fetches the + GPFIFO entry and decodes the method stream. Translations are read-back + proven from driver-published page directories, pins, or content-rescue. + `[live]` for CE; `[wip]` for GR (discovery now succeeds, see §4). +3. **Capture.** The decoder accumulates the QMD window and captures the launch + candidate; the sema service queues one request on the single ordered + workqueue. `[live]` observed (ring consumed, methods execute). +4. **Physical submit.** The backend relocates the QMD, copies program/param + pages into module shadow windows whose PTEs were planted before the channel + commit (the engine never sees a late PTE write — it caches 128-byte fetch + lines with no invalidate), builds the launch method block, publishes it to + the ladder channel's GPFIFO and rings the real doorbell. `[gated]` + default-off; latched on a proven real GR submission. +5. **Engine.** The front-end fetches the entry and executes methods; then the + **inline-QMD install** must materialise the descriptor for the engine to + read, currently **`[wall]`** — not observed. Context validity, destination + mapping and fetch/alignment are exonerated; the recovered-method-stream + form is the investigation. +6. **Completion.** The engine authors the QMD release word and the channel + release; the host applies the deferred release exactly once, publishes the + completion counter, clears the notifier and posts the channel event; MSI-X + wakes the guest. CE work is live in this shape `[gated]`; for GR the + wake-up step is the current second face of the wall. + +## 4. How work is found: discovery, truth, completion + +* **Ring/USERD discovery** has three cooperating sources: injected pins, a + kernel-native trusted walk of driver-published page directories, and + content-keyed discovery when neither names the ring. Content discovery + scans **both apertures** (emulated vidmem and guest RAM), runs the FB scan + first on a 2-second cadence, pairs a ring with the rank's staged ctx-init + push, and validates the pair before pinning. Since the 2026-10-08 re-arm + work (`7abd8f87`, `66388560`) the discovery no longer retires permanently + at a pass cap. +* **The semaphore service** decodes releases, translates their target pages + through the trusted resolver plus a driver-published-VAS pool resolver, + defers untranslatable releases with retry telemetry, and applies each + release exactly once — in software, or by letting the engine author it under + the relay (default off). +* **Completion counters** use a fixed 20-slot pool (a partial live channel + count poisoned slots before the fix); each decoded release lands in the + channel's exact slot. **Notifiers** are serviced by the pump, and family + events are delivered as RM events + MSI-X through the guest's registered + event objects. + +## 5. Challenges, honestly + +* **GR dispatch wall.** The forward path runs front-end methods, the context + is valid and the destination window is accepted — and the descriptor + install is still not observed. The mode matrix carries dozens of + single-variable falsifiers (promote shapes, prelude/PCAS form, local-memory + block, lane controls, shadow windows, pointer relocation). +* **Completion wake-up.** A boot where GR rings were discovered and releases + applied still saw the guest re-queue the same push and give up; the + best-supported reading is that the notifier/event wake-up never arrived. + The named one-variable experiment is a `sema_dbg` cycle to un-gate the + notifier/publish/event traces. +* **Reproducibility (ISSUE-0001).** The green/stall split is per-boot. The + first fix wave landed and the earlier CE freeze is gone; the stall moved to + GR ring discovery, which the latest discovery work addresses. Acceptance + is 10 cold boots green, pending. +* **Multiple clients (ISSUE-0002).** A second CUDA process wedges in guest + UVM because host per-client sema/GR state is not reset when the guest + re-runs its per-client boot handshake. Per-client state containers and + reset are the designed path. + +## 6. Portability + +The **architecture** ports: observe pushes, resolve truth, decode, +execute/forward, publish completion. So does the **RPC/RM object model** and +the mode-matrix practice. What does **not** port is address-level fact: ring +conventions, VA windows, QMD offsets, context-image layout, PTE encodings, +engine inventories and scratch layouts must be re-derived per driver version +and GPU architecture by measurement, never guessed. A new version enters as a +profile plus a measured delta; driver 610 additionally sits behind an explicit +human approval gate. + +## 7. Future + +The planned next steps are reproducibility acceptance, the descriptor-install +and wake-up investigations (offline-first with the RISC-V GSP-RM debugger +where possible), the arbitrary-kernel acceptance suite, then the near-native +performance program. The unsolved future classes — multi-tenant fairness, +fault replay/cancel, context switching between tenants, hostile-guest +hardening, and version drift — each have a proactive design direction in the +canonical document. 3D graphics reuses this pipeline once compute dispatch is +proven; it adds engine breadth (video/image), more channels and synchronisation +volume, and a much larger method vocabulary, but not a new plane. + +--- + +*Canonical document: [`docs/03-architecture/13-vexecution-plane.md`](../docs/03-architecture/13-vexecution-plane.md). +See also: [full system picture](../docs/03-architecture/09-full-system-picture.md), +[physical backend guest wiring](../docs/03-architecture/11-physical-backend-guest-wiring.md), +[the execution frontier](../docs/04-implementation/03-execution-frontier.md).*
\ No newline at end of file |