diff options
Diffstat (limited to 'wiki')
| -rw-r--r-- | wiki/Execution-Plane.md | 179 | ||||
| -rw-r--r-- | wiki/GMMU-Multi-Tenant-Protection.md | 110 | ||||
| -rw-r--r-- | wiki/README.md | 1 |
3 files changed, 290 insertions, 0 deletions
diff --git a/wiki/Execution-Plane.md b/wiki/Execution-Plane.md new file mode 100644 index 0000000..0f6eafe --- /dev/null +++ b/wiki/Execution-Plane.md @@ -0,0 +1,179 @@ +# The vGSP Execution Plane (vEP) + +> **Canonical, maintained version of this page:** +> [`../docs/03-architecture/13-vexecution-plane.md`](../docs/03-architecture/13-vexecution-plane.md). +> This wiki page is the condensed public deep-dive: what the layer is, how a +> guest CUDA launch flows through it, where it stands today, and where it is +> going. It is condensed from the canonical document and its cited evidence; +> when the two disagree, the docs tree wins. Front door: +> [`../docs/README.md`](../docs/README.md). + +Status: 2026-10-08, tree `master @ 66388560`. Tags: `[live]` proven with the +stock guest driver; `[gated]` implemented, default-off or parameter-gated; +`[wip]` partial; `[wall]` current blocker; `[target]` named success latch; +`[planned]` designed, not built. + +## 1. What it is + +**vGSP (virtual GSP)** is our host-side reimplementation of the GPU System +Processor interface: the virtual device plus the emulated GSP-RM service the +stock guest driver talks to, implemented by `vgsp.ko`. The **vGSP Execution +Plane (vEP)** is the half of vGSP that turns guest-authored GPU work into +physical execution and guest-visible completion: channel ring and USERD +discovery, GPU VA-to-PA truth, semaphore-release application, completion +counters and notifiers, QMD handling for forwarded launches, and submission to +the physical GPU's engines under the module-owned backend session. + +The other half — the control plane (boot handshake, message queues, RPC, RM +objects, BAR shadows) — makes the guest driver *run*. The vEP makes the +guest's work *run*. The canonical document defines the name and rejects the +alternates ("work plane", "submission plane", "forwarding plane", ...) that +would collide with existing project vocabulary. + +**Four work domains** organize the subsystem: + +| Domain | Question | Implemented by | +|---|---|---| +| Scheduler | which channel's work runs, at what pace | doorbell steering, the per-channel pump, discovery cadence, one ordered workqueue, one in-flight request | +| Allocator | where rings, USERDs, counters, notifiers and scratch windows live | ring/USERD autopin and pins, content-keyed pairing, counter derive, ladder scratch/PT pools, the BAR1 vidmem shadow | +| Control | which RM objects/controls shape execution | channel and event registration, `SET_PAGE_DIRECTORY`, `PROMOTE_CTX`, engine topology, the mode matrix | +| Share | how clients/mdevs share the one GPU | mdev backend refcount, one real session under a single lock, one workqueue; per-mdev clients and fairness are planned | + +## 2. The shape of the plane + +```mermaid +flowchart TB + subgraph GUEST["Guest VM - stock software"] + APP["CUDA application -> libcuda -> nvidia.ko<br/>push + QMD + GPFIFO + doorbell"] + end + subgraph VEP["vGSP Execution Plane inside vgsp.ko"] + DB["doorbell steering"] --> PUMP["channel pump"] + PUMP --> DISC["ring/USERD discovery + pins"] + DISC --> TRUTH["VA-to-PA truth<br/>walk, rescue, pins"] + TRUTH --> DEC["push decode + QMD capture"] + DEC --> REL["release apply + counters + notifiers"] + DEC --> WQ["ordered workqueue<br/>one in-flight request"] + WQ --> SUB["ladder submission<br/>mode matrix"] + end + subgraph PHY["Physical backend"] + LAD["module-owned ladder channel"] --> GSP["real GSP-RM server"] + GSP --> ENG["GR / CE engines"] + end + APP -->|"MMIO + DMA"| DB + REL -->|"release, counter, event"| APP + SUB --> LAD + ENG -->|"engine-authored release"| SUB +``` + +ASCII fallback: + +```text +guest: push + QMD + GPFIFO entry + doorbell (BAR0 0x88704) + -> pump (per-channel) -> ring/USERD discovery -> VA-to-PA truth -> decode + -> release apply + counter publish + notifier/event -> MSI-X -> guest wakes + -> (for real work) ordered workqueue -> ladder submission -> real GSP/engines + -> engine-authored release -> poll -> completion publication +``` + +## 3. One CUDA kernel launch, end to end + +1. **Launch build (guest).** `libcuda`/`nvidia.ko` build the GR push: context + methods, the inline launch descriptor (QMD), SASS and parameter pointers, + and a release target. They append a GPFIFO entry and ring the BAR0 + doorbell. `[live]` observed with the stock driver. +2. **Steering and truth (host).** The doorbell wakes the service thread; the + pump resolves the channel's ring VA and USERD, reads `GP_PUT`, fetches the + GPFIFO entry and decodes the method stream. Translations are read-back + proven from driver-published page directories, pins, or content-rescue. + `[live]` for CE; `[wip]` for GR (discovery now succeeds, see §4). +3. **Capture.** The decoder accumulates the QMD window and captures the launch + candidate; the sema service queues one request on the single ordered + workqueue. `[live]` observed (ring consumed, methods execute). +4. **Physical submit.** The backend relocates the QMD, copies program/param + pages into module shadow windows whose PTEs were planted before the channel + commit (the engine never sees a late PTE write — it caches 128-byte fetch + lines with no invalidate), builds the launch method block, publishes it to + the ladder channel's GPFIFO and rings the real doorbell. `[gated]` + default-off; latched on a proven real GR submission. +5. **Engine.** The front-end fetches the entry and executes methods; then the + **inline-QMD install** must materialise the descriptor for the engine to + read, currently **`[wall]`** — not observed. Context validity, destination + mapping and fetch/alignment are exonerated; the recovered-method-stream + form is the investigation. +6. **Completion.** The engine authors the QMD release word and the channel + release; the host applies the deferred release exactly once, publishes the + completion counter, clears the notifier and posts the channel event; MSI-X + wakes the guest. CE work is live in this shape `[gated]`; for GR the + wake-up step is the current second face of the wall. + +## 4. How work is found: discovery, truth, completion + +* **Ring/USERD discovery** has three cooperating sources: injected pins, a + kernel-native trusted walk of driver-published page directories, and + content-keyed discovery when neither names the ring. Content discovery + scans **both apertures** (emulated vidmem and guest RAM), runs the FB scan + first on a 2-second cadence, pairs a ring with the rank's staged ctx-init + push, and validates the pair before pinning. Since the 2026-10-08 re-arm + work (`7abd8f87`, `66388560`) the discovery no longer retires permanently + at a pass cap. +* **The semaphore service** decodes releases, translates their target pages + through the trusted resolver plus a driver-published-VAS pool resolver, + defers untranslatable releases with retry telemetry, and applies each + release exactly once — in software, or by letting the engine author it under + the relay (default off). +* **Completion counters** use a fixed 20-slot pool (a partial live channel + count poisoned slots before the fix); each decoded release lands in the + channel's exact slot. **Notifiers** are serviced by the pump, and family + events are delivered as RM events + MSI-X through the guest's registered + event objects. + +## 5. Challenges, honestly + +* **GR dispatch wall.** The forward path runs front-end methods, the context + is valid and the destination window is accepted — and the descriptor + install is still not observed. The mode matrix carries dozens of + single-variable falsifiers (promote shapes, prelude/PCAS form, local-memory + block, lane controls, shadow windows, pointer relocation). +* **Completion wake-up.** A boot where GR rings were discovered and releases + applied still saw the guest re-queue the same push and give up; the + best-supported reading is that the notifier/event wake-up never arrived. + The named one-variable experiment is a `sema_dbg` cycle to un-gate the + notifier/publish/event traces. +* **Reproducibility (ISSUE-0001).** The green/stall split is per-boot. The + first fix wave landed and the earlier CE freeze is gone; the stall moved to + GR ring discovery, which the latest discovery work addresses. Acceptance + is 10 cold boots green, pending. +* **Multiple clients (ISSUE-0002).** A second CUDA process wedges in guest + UVM because host per-client sema/GR state is not reset when the guest + re-runs its per-client boot handshake. Per-client state containers and + reset are the designed path. + +## 6. Portability + +The **architecture** ports: observe pushes, resolve truth, decode, +execute/forward, publish completion. So does the **RPC/RM object model** and +the mode-matrix practice. What does **not** port is address-level fact: ring +conventions, VA windows, QMD offsets, context-image layout, PTE encodings, +engine inventories and scratch layouts must be re-derived per driver version +and GPU architecture by measurement, never guessed. A new version enters as a +profile plus a measured delta; driver 610 additionally sits behind an explicit +human approval gate. + +## 7. Future + +The planned next steps are reproducibility acceptance, the descriptor-install +and wake-up investigations (offline-first with the RISC-V GSP-RM debugger +where possible), the arbitrary-kernel acceptance suite, then the near-native +performance program. The unsolved future classes — multi-tenant fairness, +fault replay/cancel, context switching between tenants, hostile-guest +hardening, and version drift — each have a proactive design direction in the +canonical document. 3D graphics reuses this pipeline once compute dispatch is +proven; it adds engine breadth (video/image), more channels and synchronisation +volume, and a much larger method vocabulary, but not a new plane. + +--- + +*Canonical document: [`docs/03-architecture/13-vexecution-plane.md`](../docs/03-architecture/13-vexecution-plane.md). +See also: [full system picture](../docs/03-architecture/09-full-system-picture.md), +[physical backend guest wiring](../docs/03-architecture/11-physical-backend-guest-wiring.md), +[the execution frontier](../docs/04-implementation/03-execution-frontier.md).*
\ No newline at end of file diff --git a/wiki/GMMU-Multi-Tenant-Protection.md b/wiki/GMMU-Multi-Tenant-Protection.md new file mode 100644 index 0000000..83b00ba --- /dev/null +++ b/wiki/GMMU-Multi-Tenant-Protection.md @@ -0,0 +1,110 @@ +# GMMU Multi-Tenant Protection + +> **Current — companion summary.** This page condenses +> [`../docs/03-architecture/14-multi-tenant-gmmu-protection.md`](../docs/03-architecture/14-multi-tenant-gmmu-protection.md) +> (the canonical document: problem statement, full design, RM struct sketches, +> walked example, portability, M1–M8 roadmap). Read the canonical document for +> anything you intend to build against; this page is the quick map. + +## Overview + +Open-IOV must isolate **multiple mdevs (VMs)** and, for multi-tenant +inference, **multiple tenants inside one guest** at the GMMU. The design has +two halves: + +1. **Service-state ownership (baseline 1.0).** The work/completion service + (`kernel/gsp_sema_service.c`) is one global context (`static struct + sema_svc svc`, `:676`), the pump services the first open mdev + (`:13296-13308`), and channels/VA spaces/translations are keyed by + guest-chosen values alone (`:847`, `:988`, `:113-130`). Two guests can + pick the same handles and VAs (live: channel `0xcaf000d4`; + `../debug/issue/issue-number-2.md`), so owner-keying is a prerequisite + for any GMMU policy. +2. **GMMU policy for paged attention (research).** A tenant/agent-aware KV + cache protection design across vLLM, a guest utility, RM controls, the + vEP, and page tables — post-1.0, dual-gated; never a baseline dependency. + +## Isolation layers + +| Layer | Mechanism | Status | +|---|---|---| +| L0 platform | IOMMU/SMMU per-VM DMA, KVM memory/vCPU boundary | live | +| L1 transport | VFIO mdev per tenant, config/BAR/IRQ mediation | live | +| L2 vEP state | per-mdev emulation context + (needed) per-mdev work service | live / design | +| L3 GMMU | per-context page directory (VA space) | live (guest-owned) / design (host-owned) | +| L4 arbitration | channel ownership, TSG/runlist policy | design | +| L5 completion | owner-keyed release resolution; no cross-tenant apply | wip | +| L6 fault | fault attributed to the owning channel/tenant only | wip | +| L7 application | per-tenant/per-agent VA pooling inside a guest | research | + +## Hardware primitives: proven vs hypothesis + +| Primitive | Status | +|---|---| +| 4-level tree, 49-bit VA, 4K/64K/2M pages, dual PDE0 | **proven** (`docs/43-GMMU-PTE-DECODER`; `kernel/gsp_sema_service.c:1058-1072`) | +| VER2 PTE fields incl. `read_only` (6), `privilege` (5), `encrypted` (4), `kind` | **proven format** (`docs/43` §3) | +| Separate page directory per VA space (`SET_PAGE_DIRECTORY` with `hVASpace`/`chId`) | **proven** (`kernel/gsp_rm_emulation.c:3310-3314`) | +| No tenant/ACL/PASID field in the VER2 PTE/PDE | **proven absence** in the field catalogue | +| RO-leaf write enforcement | **hypothesis** (needs a supervised probe; M7) | +| Encryption-domain (CCS) as a tenant boundary | **hypothesis** | + +## The design in brief + +* **Design A (stock-compatible).** Extended vLLM reserves a VA range per + agent; a new guest utility (`oiv-agentd`) publishes tag records to a + device-memory **tag mailbox** the vEP can read (emulation mode: guest + vidmem is host RAM, `kernel/bar_init.c:68-77`). The vEP enforces + owner↔VA-subtree match at map time and at submit time. No guest KMD + change. Research (M6). +* **Design B (explicit controls).** New vGPU-private RM controls — + `TENANT_BIND` (0x90f10120), `TAG_RANGE` (…21), `UNTAG_RANGE` (…22), + `TENANT_SHARE` (…23), `AGENT_CHANNEL_BIND` (…24), `TAG_QUERY` (…25) — + plus struct sketches in the canonical doc §4.5. Requires a guest KMD + extension: post-1.0, gated (M8). +* **Granularities.** Per-KV-block tagging (finest; software+PTE policy, + sharing needs RO enforcement); per-tenant separate page directories + (the only proven hardware lever); per-agent separate channels (fault/QoS + containment; optional per-agent sub-directories). Degradation ladder and + per-granularity cost/failure analysis in canonical §4.7. + +## Walked example (two tenants, paged_attention_v2) + +Two agents A1/A2 each own a VA pool and channels. A malicious T1 kernel +naming an A2 VA hits a missing PTE (GMMU fault) and/or is refused by vEP +submit-time policy; the fault is attributed to A1 only (T2 untouched); a +periodic owner-vs-PTE audit with canaries catches escapes. If only +per-tenant separation exists, the deployable contract is **one agent per +tenant, one tenant per context** (canonical §5.4). + +## Key code anchors (master `66388560`) + +| What | Where | +|---|---| +| per-mdev emulation context | `kernel/include/nvidia-vgpu-vfio.h:133`, `kernel/gsp_emulation.h:130` | +| global service state | `kernel/gsp_sema_service.c:676`; pump `:13296`; doorbell `:13456` | +| channel/vaspace registries | `kernel/gsp_sema_service.c:847`, `:988` | +| GMMU walk (VER2) | `kernel/gsp_sema_service.c:1174`; decode `:1093` | +| page directory record | `kernel/gsp_rm_emulation.c:3320`; reserved PDEs `:2237-2262` | +| per-mdev FB sizing | `kernel/bar_init.c:60-77`; `kernel/include/ioctl.h:39-46` | +| L2 invalidate acked | `kernel/gsp_rm_emulation.c:2519-2532` | + +## Roadmap (acceptance criteria in canonical §7.3) + +M1 per-mdev service state — M2 two-guest isolation gate — M3 per-mdev real +GSP clients + fairness — M4 independent VRAM sizes/sparse backing — M5 +invalidate + eviction/reclaim (I1–I6 invariants) — M6 Design A prototype — +M7 RO/permission probe — M8 Design B / CCS / 610 (gated). M1–M5 are the +1.0 programme; M6–M8 are post-1.0 research, dual-gated, duplicate-not- +replace, never required by the baseline. RM 610 stays behind its own human +gate (`docs/23-RM610-READINESS`). + +## Related documentation + +* [`../docs/03-architecture/14-multi-tenant-gmmu-protection.md`](../docs/03-architecture/14-multi-tenant-gmmu-protection.md) — canonical +* [`../docs/03-architecture/13-vexecution-plane.md`](../docs/03-architecture/13-vexecution-plane.md) — vEP vocabulary (parallel) +* [`../docs/03-architecture/05-gmmu-amap-hal-gr.md`](../docs/03-architecture/05-gmmu-amap-hal-gr.md) — GMMU background +* [`../docs/03-architecture/09-full-system-picture.md`](../docs/03-architecture/09-full-system-picture.md) — multi-tenancy and isolation map +* [`../docs/03-architecture/11-physical-backend-guest-wiring.md`](../docs/03-architecture/11-physical-backend-guest-wiring.md) — forwarding and page-table seams +* [`../docs/05-operations/01-performance-correctness-security.md`](../docs/05-operations/01-performance-correctness-security.md) — threat model, rung 5 +* [`../debug/issue/issue-number-2.md`](../debug/issue/issue-number-2.md) — per-client state wedge +* [`../debug/risc-v-gsp-rm-vm/docs/43-GMMU-PTE-DECODER.md`](../debug/risc-v-gsp-rm-vm/docs/43-GMMU-PTE-DECODER.md) — GMMU field catalogue diff --git a/wiki/README.md b/wiki/README.md index 4c90b66..378ea31 100644 --- a/wiki/README.md +++ b/wiki/README.md @@ -9,6 +9,7 @@ ## Documentation Index ### Core Analysis +- [The vGSP Execution Plane (vEP)](Execution-Plane.md) - The work/execution layer of `vgsp.ko`: guest pushes to physical execution and guest-visible completion (condensed; canonical doc [`../docs/03-architecture/13-vexecution-plane.md`](../docs/03-architecture/13-vexecution-plane.md)) - [GSP 570.144W Firmware Analysis](GSP-570.144W-Firmware-Analysis.md) - Firmware decompilation and address record - [Address Reference](Address-Reference.md) - Firmware-side address tables - [CUDA Userspace Driver](CUDA-Userspace-Driver.md) - `libcuda.so` deep dive: layer model, guest interface, forwarding status, crash-path lesson and lookup tooling |