summaryrefslogtreecommitdiffstats
path: root/wiki
diff options
context:
space:
mode:
authorArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-08 13:41:31 -0400
committerArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-08 13:41:31 -0400
commit9d1e0cf87df63a2f81c7bb471481336384874861 (patch)
tree8e036e19e062673907ff9cb133ec4d0bfacbf548 /wiki
parentd0213b5a7ca33a0b5f763e0f256ff48bf4f19dc8 (diff)
downloadopen-iov-9d1e0cf87df63a2f81c7bb471481336384874861.tar.gz
open-iov-9d1e0cf87df63a2f81c7bb471481336384874861.zip
docs(vexecution-plane): add the vGSP Execution Plane architecture doc, wiki deep dive and indexes
Diffstat (limited to 'wiki')
-rw-r--r--wiki/Execution-Plane.md179
-rw-r--r--wiki/README.md1
2 files changed, 180 insertions, 0 deletions
diff --git a/wiki/Execution-Plane.md b/wiki/Execution-Plane.md
new file mode 100644
index 0000000..0f6eafe
--- /dev/null
+++ b/wiki/Execution-Plane.md
@@ -0,0 +1,179 @@
+# The vGSP Execution Plane (vEP)
+
+> **Canonical, maintained version of this page:**
+> [`../docs/03-architecture/13-vexecution-plane.md`](../docs/03-architecture/13-vexecution-plane.md).
+> This wiki page is the condensed public deep-dive: what the layer is, how a
+> guest CUDA launch flows through it, where it stands today, and where it is
+> going. It is condensed from the canonical document and its cited evidence;
+> when the two disagree, the docs tree wins. Front door:
+> [`../docs/README.md`](../docs/README.md).
+
+Status: 2026-10-08, tree `master @ 66388560`. Tags: `[live]` proven with the
+stock guest driver; `[gated]` implemented, default-off or parameter-gated;
+`[wip]` partial; `[wall]` current blocker; `[target]` named success latch;
+`[planned]` designed, not built.
+
+## 1. What it is
+
+**vGSP (virtual GSP)** is our host-side reimplementation of the GPU System
+Processor interface: the virtual device plus the emulated GSP-RM service the
+stock guest driver talks to, implemented by `vgsp.ko`. The **vGSP Execution
+Plane (vEP)** is the half of vGSP that turns guest-authored GPU work into
+physical execution and guest-visible completion: channel ring and USERD
+discovery, GPU VA-to-PA truth, semaphore-release application, completion
+counters and notifiers, QMD handling for forwarded launches, and submission to
+the physical GPU's engines under the module-owned backend session.
+
+The other half — the control plane (boot handshake, message queues, RPC, RM
+objects, BAR shadows) — makes the guest driver *run*. The vEP makes the
+guest's work *run*. The canonical document defines the name and rejects the
+alternates ("work plane", "submission plane", "forwarding plane", ...) that
+would collide with existing project vocabulary.
+
+**Four work domains** organize the subsystem:
+
+| Domain | Question | Implemented by |
+|---|---|---|
+| Scheduler | which channel's work runs, at what pace | doorbell steering, the per-channel pump, discovery cadence, one ordered workqueue, one in-flight request |
+| Allocator | where rings, USERDs, counters, notifiers and scratch windows live | ring/USERD autopin and pins, content-keyed pairing, counter derive, ladder scratch/PT pools, the BAR1 vidmem shadow |
+| Control | which RM objects/controls shape execution | channel and event registration, `SET_PAGE_DIRECTORY`, `PROMOTE_CTX`, engine topology, the mode matrix |
+| Share | how clients/mdevs share the one GPU | mdev backend refcount, one real session under a single lock, one workqueue; per-mdev clients and fairness are planned |
+
+## 2. The shape of the plane
+
+```mermaid
+flowchart TB
+ subgraph GUEST["Guest VM - stock software"]
+ APP["CUDA application -> libcuda -> nvidia.ko<br/>push + QMD + GPFIFO + doorbell"]
+ end
+ subgraph VEP["vGSP Execution Plane inside vgsp.ko"]
+ DB["doorbell steering"] --> PUMP["channel pump"]
+ PUMP --> DISC["ring/USERD discovery + pins"]
+ DISC --> TRUTH["VA-to-PA truth<br/>walk, rescue, pins"]
+ TRUTH --> DEC["push decode + QMD capture"]
+ DEC --> REL["release apply + counters + notifiers"]
+ DEC --> WQ["ordered workqueue<br/>one in-flight request"]
+ WQ --> SUB["ladder submission<br/>mode matrix"]
+ end
+ subgraph PHY["Physical backend"]
+ LAD["module-owned ladder channel"] --> GSP["real GSP-RM server"]
+ GSP --> ENG["GR / CE engines"]
+ end
+ APP -->|"MMIO + DMA"| DB
+ REL -->|"release, counter, event"| APP
+ SUB --> LAD
+ ENG -->|"engine-authored release"| SUB
+```
+
+ASCII fallback:
+
+```text
+guest: push + QMD + GPFIFO entry + doorbell (BAR0 0x88704)
+ -> pump (per-channel) -> ring/USERD discovery -> VA-to-PA truth -> decode
+ -> release apply + counter publish + notifier/event -> MSI-X -> guest wakes
+ -> (for real work) ordered workqueue -> ladder submission -> real GSP/engines
+ -> engine-authored release -> poll -> completion publication
+```
+
+## 3. One CUDA kernel launch, end to end
+
+1. **Launch build (guest).** `libcuda`/`nvidia.ko` build the GR push: context
+ methods, the inline launch descriptor (QMD), SASS and parameter pointers,
+ and a release target. They append a GPFIFO entry and ring the BAR0
+ doorbell. `[live]` observed with the stock driver.
+2. **Steering and truth (host).** The doorbell wakes the service thread; the
+ pump resolves the channel's ring VA and USERD, reads `GP_PUT`, fetches the
+ GPFIFO entry and decodes the method stream. Translations are read-back
+ proven from driver-published page directories, pins, or content-rescue.
+ `[live]` for CE; `[wip]` for GR (discovery now succeeds, see §4).
+3. **Capture.** The decoder accumulates the QMD window and captures the launch
+ candidate; the sema service queues one request on the single ordered
+ workqueue. `[live]` observed (ring consumed, methods execute).
+4. **Physical submit.** The backend relocates the QMD, copies program/param
+ pages into module shadow windows whose PTEs were planted before the channel
+ commit (the engine never sees a late PTE write — it caches 128-byte fetch
+ lines with no invalidate), builds the launch method block, publishes it to
+ the ladder channel's GPFIFO and rings the real doorbell. `[gated]`
+ default-off; latched on a proven real GR submission.
+5. **Engine.** The front-end fetches the entry and executes methods; then the
+ **inline-QMD install** must materialise the descriptor for the engine to
+ read, currently **`[wall]`** — not observed. Context validity, destination
+ mapping and fetch/alignment are exonerated; the recovered-method-stream
+ form is the investigation.
+6. **Completion.** The engine authors the QMD release word and the channel
+ release; the host applies the deferred release exactly once, publishes the
+ completion counter, clears the notifier and posts the channel event; MSI-X
+ wakes the guest. CE work is live in this shape `[gated]`; for GR the
+ wake-up step is the current second face of the wall.
+
+## 4. How work is found: discovery, truth, completion
+
+* **Ring/USERD discovery** has three cooperating sources: injected pins, a
+ kernel-native trusted walk of driver-published page directories, and
+ content-keyed discovery when neither names the ring. Content discovery
+ scans **both apertures** (emulated vidmem and guest RAM), runs the FB scan
+ first on a 2-second cadence, pairs a ring with the rank's staged ctx-init
+ push, and validates the pair before pinning. Since the 2026-10-08 re-arm
+ work (`7abd8f87`, `66388560`) the discovery no longer retires permanently
+ at a pass cap.
+* **The semaphore service** decodes releases, translates their target pages
+ through the trusted resolver plus a driver-published-VAS pool resolver,
+ defers untranslatable releases with retry telemetry, and applies each
+ release exactly once — in software, or by letting the engine author it under
+ the relay (default off).
+* **Completion counters** use a fixed 20-slot pool (a partial live channel
+ count poisoned slots before the fix); each decoded release lands in the
+ channel's exact slot. **Notifiers** are serviced by the pump, and family
+ events are delivered as RM events + MSI-X through the guest's registered
+ event objects.
+
+## 5. Challenges, honestly
+
+* **GR dispatch wall.** The forward path runs front-end methods, the context
+ is valid and the destination window is accepted — and the descriptor
+ install is still not observed. The mode matrix carries dozens of
+ single-variable falsifiers (promote shapes, prelude/PCAS form, local-memory
+ block, lane controls, shadow windows, pointer relocation).
+* **Completion wake-up.** A boot where GR rings were discovered and releases
+ applied still saw the guest re-queue the same push and give up; the
+ best-supported reading is that the notifier/event wake-up never arrived.
+ The named one-variable experiment is a `sema_dbg` cycle to un-gate the
+ notifier/publish/event traces.
+* **Reproducibility (ISSUE-0001).** The green/stall split is per-boot. The
+ first fix wave landed and the earlier CE freeze is gone; the stall moved to
+ GR ring discovery, which the latest discovery work addresses. Acceptance
+ is 10 cold boots green, pending.
+* **Multiple clients (ISSUE-0002).** A second CUDA process wedges in guest
+ UVM because host per-client sema/GR state is not reset when the guest
+ re-runs its per-client boot handshake. Per-client state containers and
+ reset are the designed path.
+
+## 6. Portability
+
+The **architecture** ports: observe pushes, resolve truth, decode,
+execute/forward, publish completion. So does the **RPC/RM object model** and
+the mode-matrix practice. What does **not** port is address-level fact: ring
+conventions, VA windows, QMD offsets, context-image layout, PTE encodings,
+engine inventories and scratch layouts must be re-derived per driver version
+and GPU architecture by measurement, never guessed. A new version enters as a
+profile plus a measured delta; driver 610 additionally sits behind an explicit
+human approval gate.
+
+## 7. Future
+
+The planned next steps are reproducibility acceptance, the descriptor-install
+and wake-up investigations (offline-first with the RISC-V GSP-RM debugger
+where possible), the arbitrary-kernel acceptance suite, then the near-native
+performance program. The unsolved future classes — multi-tenant fairness,
+fault replay/cancel, context switching between tenants, hostile-guest
+hardening, and version drift — each have a proactive design direction in the
+canonical document. 3D graphics reuses this pipeline once compute dispatch is
+proven; it adds engine breadth (video/image), more channels and synchronisation
+volume, and a much larger method vocabulary, but not a new plane.
+
+---
+
+*Canonical document: [`docs/03-architecture/13-vexecution-plane.md`](../docs/03-architecture/13-vexecution-plane.md).
+See also: [full system picture](../docs/03-architecture/09-full-system-picture.md),
+[physical backend guest wiring](../docs/03-architecture/11-physical-backend-guest-wiring.md),
+[the execution frontier](../docs/04-implementation/03-execution-frontier.md).* \ No newline at end of file
diff --git a/wiki/README.md b/wiki/README.md
index 4c90b66..378ea31 100644
--- a/wiki/README.md
+++ b/wiki/README.md
@@ -9,6 +9,7 @@
## Documentation Index
### Core Analysis
+- [The vGSP Execution Plane (vEP)](Execution-Plane.md) - The work/execution layer of `vgsp.ko`: guest pushes to physical execution and guest-visible completion (condensed; canonical doc [`../docs/03-architecture/13-vexecution-plane.md`](../docs/03-architecture/13-vexecution-plane.md))
- [GSP 570.144W Firmware Analysis](GSP-570.144W-Firmware-Analysis.md) - Firmware decompilation and address record
- [Address Reference](Address-Reference.md) - Firmware-side address tables
- [CUDA Userspace Driver](CUDA-Userspace-Driver.md) - `libcuda.so` deep dive: layer model, guest interface, forwarding status, crash-path lesson and lookup tooling