From b7c5212f7691f71202a7c3661993354132429712 Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Thu, 8 Oct 2026 13:41:31 -0400 Subject: docs(vexecution-plane): add the vGSP Execution Plane architecture doc, wiki deep dive and indexes --- wiki/Execution-Plane.md | 179 ++++++++++++++++++++++++++++++++++++++++++++++++ wiki/README.md | 1 + 2 files changed, 180 insertions(+) create mode 100644 wiki/Execution-Plane.md (limited to 'wiki') diff --git a/wiki/Execution-Plane.md b/wiki/Execution-Plane.md new file mode 100644 index 0000000..0f6eafe --- /dev/null +++ b/wiki/Execution-Plane.md @@ -0,0 +1,179 @@ +# The vGSP Execution Plane (vEP) + +> **Canonical, maintained version of this page:** +> [`../docs/03-architecture/13-vexecution-plane.md`](../docs/03-architecture/13-vexecution-plane.md). +> This wiki page is the condensed public deep-dive: what the layer is, how a +> guest CUDA launch flows through it, where it stands today, and where it is +> going. It is condensed from the canonical document and its cited evidence; +> when the two disagree, the docs tree wins. Front door: +> [`../docs/README.md`](../docs/README.md). + +Status: 2026-10-08, tree `master @ 66388560`. Tags: `[live]` proven with the +stock guest driver; `[gated]` implemented, default-off or parameter-gated; +`[wip]` partial; `[wall]` current blocker; `[target]` named success latch; +`[planned]` designed, not built. + +## 1. What it is + +**vGSP (virtual GSP)** is our host-side reimplementation of the GPU System +Processor interface: the virtual device plus the emulated GSP-RM service the +stock guest driver talks to, implemented by `vgsp.ko`. The **vGSP Execution +Plane (vEP)** is the half of vGSP that turns guest-authored GPU work into +physical execution and guest-visible completion: channel ring and USERD +discovery, GPU VA-to-PA truth, semaphore-release application, completion +counters and notifiers, QMD handling for forwarded launches, and submission to +the physical GPU's engines under the module-owned backend session. + +The other half — the control plane (boot handshake, message queues, RPC, RM +objects, BAR shadows) — makes the guest driver *run*. The vEP makes the +guest's work *run*. The canonical document defines the name and rejects the +alternates ("work plane", "submission plane", "forwarding plane", ...) that +would collide with existing project vocabulary. + +**Four work domains** organize the subsystem: + +| Domain | Question | Implemented by | +|---|---|---| +| Scheduler | which channel's work runs, at what pace | doorbell steering, the per-channel pump, discovery cadence, one ordered workqueue, one in-flight request | +| Allocator | where rings, USERDs, counters, notifiers and scratch windows live | ring/USERD autopin and pins, content-keyed pairing, counter derive, ladder scratch/PT pools, the BAR1 vidmem shadow | +| Control | which RM objects/controls shape execution | channel and event registration, `SET_PAGE_DIRECTORY`, `PROMOTE_CTX`, engine topology, the mode matrix | +| Share | how clients/mdevs share the one GPU | mdev backend refcount, one real session under a single lock, one workqueue; per-mdev clients and fairness are planned | + +## 2. The shape of the plane + +```mermaid +flowchart TB + subgraph GUEST["Guest VM - stock software"] + APP["CUDA application -> libcuda -> nvidia.ko
push + QMD + GPFIFO + doorbell"] + end + subgraph VEP["vGSP Execution Plane inside vgsp.ko"] + DB["doorbell steering"] --> PUMP["channel pump"] + PUMP --> DISC["ring/USERD discovery + pins"] + DISC --> TRUTH["VA-to-PA truth
walk, rescue, pins"] + TRUTH --> DEC["push decode + QMD capture"] + DEC --> REL["release apply + counters + notifiers"] + DEC --> WQ["ordered workqueue
one in-flight request"] + WQ --> SUB["ladder submission
mode matrix"] + end + subgraph PHY["Physical backend"] + LAD["module-owned ladder channel"] --> GSP["real GSP-RM server"] + GSP --> ENG["GR / CE engines"] + end + APP -->|"MMIO + DMA"| DB + REL -->|"release, counter, event"| APP + SUB --> LAD + ENG -->|"engine-authored release"| SUB +``` + +ASCII fallback: + +```text +guest: push + QMD + GPFIFO entry + doorbell (BAR0 0x88704) + -> pump (per-channel) -> ring/USERD discovery -> VA-to-PA truth -> decode + -> release apply + counter publish + notifier/event -> MSI-X -> guest wakes + -> (for real work) ordered workqueue -> ladder submission -> real GSP/engines + -> engine-authored release -> poll -> completion publication +``` + +## 3. One CUDA kernel launch, end to end + +1. **Launch build (guest).** `libcuda`/`nvidia.ko` build the GR push: context + methods, the inline launch descriptor (QMD), SASS and parameter pointers, + and a release target. They append a GPFIFO entry and ring the BAR0 + doorbell. `[live]` observed with the stock driver. +2. **Steering and truth (host).** The doorbell wakes the service thread; the + pump resolves the channel's ring VA and USERD, reads `GP_PUT`, fetches the + GPFIFO entry and decodes the method stream. Translations are read-back + proven from driver-published page directories, pins, or content-rescue. + `[live]` for CE; `[wip]` for GR (discovery now succeeds, see §4). +3. **Capture.** The decoder accumulates the QMD window and captures the launch + candidate; the sema service queues one request on the single ordered + workqueue. `[live]` observed (ring consumed, methods execute). +4. **Physical submit.** The backend relocates the QMD, copies program/param + pages into module shadow windows whose PTEs were planted before the channel + commit (the engine never sees a late PTE write — it caches 128-byte fetch + lines with no invalidate), builds the launch method block, publishes it to + the ladder channel's GPFIFO and rings the real doorbell. `[gated]` + default-off; latched on a proven real GR submission. +5. **Engine.** The front-end fetches the entry and executes methods; then the + **inline-QMD install** must materialise the descriptor for the engine to + read, currently **`[wall]`** — not observed. Context validity, destination + mapping and fetch/alignment are exonerated; the recovered-method-stream + form is the investigation. +6. **Completion.** The engine authors the QMD release word and the channel + release; the host applies the deferred release exactly once, publishes the + completion counter, clears the notifier and posts the channel event; MSI-X + wakes the guest. CE work is live in this shape `[gated]`; for GR the + wake-up step is the current second face of the wall. + +## 4. How work is found: discovery, truth, completion + +* **Ring/USERD discovery** has three cooperating sources: injected pins, a + kernel-native trusted walk of driver-published page directories, and + content-keyed discovery when neither names the ring. Content discovery + scans **both apertures** (emulated vidmem and guest RAM), runs the FB scan + first on a 2-second cadence, pairs a ring with the rank's staged ctx-init + push, and validates the pair before pinning. Since the 2026-10-08 re-arm + work (`7abd8f87`, `66388560`) the discovery no longer retires permanently + at a pass cap. +* **The semaphore service** decodes releases, translates their target pages + through the trusted resolver plus a driver-published-VAS pool resolver, + defers untranslatable releases with retry telemetry, and applies each + release exactly once — in software, or by letting the engine author it under + the relay (default off). +* **Completion counters** use a fixed 20-slot pool (a partial live channel + count poisoned slots before the fix); each decoded release lands in the + channel's exact slot. **Notifiers** are serviced by the pump, and family + events are delivered as RM events + MSI-X through the guest's registered + event objects. + +## 5. Challenges, honestly + +* **GR dispatch wall.** The forward path runs front-end methods, the context + is valid and the destination window is accepted — and the descriptor + install is still not observed. The mode matrix carries dozens of + single-variable falsifiers (promote shapes, prelude/PCAS form, local-memory + block, lane controls, shadow windows, pointer relocation). +* **Completion wake-up.** A boot where GR rings were discovered and releases + applied still saw the guest re-queue the same push and give up; the + best-supported reading is that the notifier/event wake-up never arrived. + The named one-variable experiment is a `sema_dbg` cycle to un-gate the + notifier/publish/event traces. +* **Reproducibility (ISSUE-0001).** The green/stall split is per-boot. The + first fix wave landed and the earlier CE freeze is gone; the stall moved to + GR ring discovery, which the latest discovery work addresses. Acceptance + is 10 cold boots green, pending. +* **Multiple clients (ISSUE-0002).** A second CUDA process wedges in guest + UVM because host per-client sema/GR state is not reset when the guest + re-runs its per-client boot handshake. Per-client state containers and + reset are the designed path. + +## 6. Portability + +The **architecture** ports: observe pushes, resolve truth, decode, +execute/forward, publish completion. So does the **RPC/RM object model** and +the mode-matrix practice. What does **not** port is address-level fact: ring +conventions, VA windows, QMD offsets, context-image layout, PTE encodings, +engine inventories and scratch layouts must be re-derived per driver version +and GPU architecture by measurement, never guessed. A new version enters as a +profile plus a measured delta; driver 610 additionally sits behind an explicit +human approval gate. + +## 7. Future + +The planned next steps are reproducibility acceptance, the descriptor-install +and wake-up investigations (offline-first with the RISC-V GSP-RM debugger +where possible), the arbitrary-kernel acceptance suite, then the near-native +performance program. The unsolved future classes — multi-tenant fairness, +fault replay/cancel, context switching between tenants, hostile-guest +hardening, and version drift — each have a proactive design direction in the +canonical document. 3D graphics reuses this pipeline once compute dispatch is +proven; it adds engine breadth (video/image), more channels and synchronisation +volume, and a much larger method vocabulary, but not a new plane. + +--- + +*Canonical document: [`docs/03-architecture/13-vexecution-plane.md`](../docs/03-architecture/13-vexecution-plane.md). +See also: [full system picture](../docs/03-architecture/09-full-system-picture.md), +[physical backend guest wiring](../docs/03-architecture/11-physical-backend-guest-wiring.md), +[the execution frontier](../docs/04-implementation/03-execution-frontier.md).* \ No newline at end of file diff --git a/wiki/README.md b/wiki/README.md index 4c90b66..378ea31 100644 --- a/wiki/README.md +++ b/wiki/README.md @@ -9,6 +9,7 @@ ## Documentation Index ### Core Analysis +- [The vGSP Execution Plane (vEP)](Execution-Plane.md) - The work/execution layer of `vgsp.ko`: guest pushes to physical execution and guest-visible completion (condensed; canonical doc [`../docs/03-architecture/13-vexecution-plane.md`](../docs/03-architecture/13-vexecution-plane.md)) - [GSP 570.144W Firmware Analysis](GSP-570.144W-Firmware-Analysis.md) - Firmware decompilation and address record - [Address Reference](Address-Reference.md) - Firmware-side address tables - [CUDA Userspace Driver](CUDA-Userspace-Driver.md) - `libcuda.so` deep dive: layer model, guest interface, forwarding status, crash-path lesson and lookup tooling -- cgit v1.2.3 From dc551a6bf1458f6a9d621ce97bd86607e38b1080 Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Thu, 8 Oct 2026 13:51:22 -0400 Subject: docs(multi-tenant-gmmu): multi-mdev isolation design, GMMU tenant/agent paged-attention protection, VRAM oversubscription, portability and M1-M8 roadmap (wiki summary; local provenance in adjacent LOCAL notes) --- .../14-multi-tenant-gmmu-protection.md | 1095 ++++++++++++++++++++ wiki/GMMU-Multi-Tenant-Protection.md | 110 ++ 2 files changed, 1205 insertions(+) create mode 100644 docs/03-architecture/14-multi-tenant-gmmu-protection.md create mode 100644 wiki/GMMU-Multi-Tenant-Protection.md (limited to 'wiki') diff --git a/docs/03-architecture/14-multi-tenant-gmmu-protection.md b/docs/03-architecture/14-multi-tenant-gmmu-protection.md new file mode 100644 index 0000000..c3f58c7 --- /dev/null +++ b/docs/03-architecture/14-multi-tenant-gmmu-protection.md @@ -0,0 +1,1095 @@ +# Multi-Tenant GMMU Protection: Isolation Design and Paged-Attention Controls + +> **Status: design (2026-10-08, master `66388560`).** This document specifies +> how Open-IOV isolates multiple tenants — multiple mdevs *and* multiple +> tenants inside one guest — at the GPU memory-management unit (GMMU), and how +> a paged-attention inference stack (vLLM-style) can use those protections. +> Sections 1–3 are **baseline 1.0 design**; sections 4.5+ and roadmap stages +> M6–M8 are **post-1.0 research**, explicitly gated (see §7.4 and +> `AGENTS.md` index entry 11). No part of the research design may become a +> design input for baseline 1.0 code: no forward-looking parameter, hook or +> abstraction may be added to a product path for it. +> +> **Method.** Every empirical claim about our system cites a code line or +> tracked document; hardware-format claims are marked **proven** (observed +> live, or in a tracked field catalogue) or **hypothesis** (needs a +> supervised experiment). No timing/cycle/silicon-accuracy claim is made. +> +> +> **Companion document.** [`13-vexecution-plane.md`](13-vexecution-plane.md) +> (written in parallel) defines the vEP/instance/agent vocabulary used here; +> where the two disagree, 13 is the vocabulary authority and this document +> is the GMMU-policy authority. + +Tags: `[live]` exercised; `[wip]` default-off; `[design]` not built; `[hypothesis]` unexercised; `[research]` post-1.0, dual-gated. + +--- + +## 1. Problem statement: 1.0 needs N mdevs, N guests, and flawless isolation + +### 1.1 The 1.0 requirement + +The project's stated 1.0 gate is multiple mediated devices, two or more +guests, multiple VRAM sizes, and isolation that survives adversarial review. +The tracked status is explicit that this area is **designed, partially +implemented** ([`../README.md` §3](../README.md), row "Multi-tenant isolation +(VFs, mdev-per-VM)"), and the multi-tenancy model is stated as a first-class +value: one mdev per VM, per-instance interrupts, no shared host state, fail +closed ([`../01-orientation/01-purpose-and-scope.md` §5](../01-orientation/01-purpose-and-scope.md)). +The evaluation ladder puts "N VMs, isolation + fairness + no cross-talk" on +rung 5 ([`../05-operations/01-performance-correctness-security.md` §1](../05-operations/01-performance-correctness-security.md)). + +Three 1.0 requirements drive this document: + +1. **Two guests on one GPU.** Two VMs, two mdevs, both running the stock + driver, concurrently and repeatedly. Today the second *client in one + guest* already wedges (`../../debug/issue/issue-number-2.md`), and the + two-guest harness does not exist. +2. **Independent VRAM sizes.** Each mdev exposes an emulated framebuffer + size that need not relate to the physical framebuffer or to another + mdev's size: `nv_vgpu_type_info.vram_mb` + (`kernel/include/ioctl.h:39-46`), BAR1 sized from it + (`kernel/bar_init.c:60-77`), and GSCI/PFB reports derived from the same + value (`kernel/gsp_rm_emulation.c:1034,1106-1110`, + `kernel/bar_emulation.c:271-310`, `kernel/gsp_emulation.c:743-753`). + The mechanism exists per mdev; the *capacity* story does not (§3). +3. **Multi-tenant inference inside one guest.** A vLLM-style server + multiplexes many *logical tenants* (API users, agents, sequences) inside + one VM that owns one mdev. VM-level isolation (IOMMU, mdev) is blind to + them. This is the workload §4 targets. + +### 1.2 Where isolation can exist + +Isolation is not one mechanism; it is a stack of layers, each with a +different owner and a different failure mode. The design must say, for each +layer, what holds and what does not. + +``` +L0 PLATFORM IOMMU/SMMU: per-VM DMA address spaces [live] + │ KVM memory + vCPU boundary [live] +L1 TRANSPORT VFIO mdev: one mediated device per tenant [live] + │ config/BAR/IRQ mediation, per-device eventfd +L2 vEP STATE per-mdev emulation context (BAR shadows, [live/wip] + │ queues, msgq, interrupt state) §2.2 + │ per-mdev work-service state [design] +L3 GMMU per-context page directory (VA space) [live//design] + │ PTE-level permissions (RO, privilege) [proven fmt] +L4 ARBITRATION channel ownership, runlist/TSG scheduling [design] +L5 COMPLETION semaphore resolution per owner; no cross- [wip] + │ tenant release apply §2.5 +L6 FAULT fault attribution to the owning tenant; [wip] + notifier only to that tenant §2.6 +L7 APPLICATION per-tenant/per-agent VA partitioning inside [research] + a guest (vLLM + agent utility) §4 +``` + +The honest summary from the tracked docs applies: **memory isolation between +VMs is hardware-plus-hypervisor enforced today; engine-level context +separation and performance isolation are what the project must still build** +([`09-full-system-picture.md` §6](09-full-system-picture.md)). + +### 1.3 Trust boundaries and the principals + +| Boundary | Attacker | Capability | Must hold | +|---|---|---|---| +| Guest ↔ guest | whole VM | arbitrary device programming, arbitrary guest kernel | no memory, state, interrupt or completion observation across mdevs | +| Guest ↔ host | whole VM | arbitrary DMA/MMIO against its own device | no host memory access, no module state escape, fail closed | +| Host user ↔ device | unprivileged host user | access to `/dev/*` as granted | no privilege escalation via our interfaces | +| Tenant ↔ tenant (in one guest) | guest userspace tenant | arbitrary CUDA kernels, arbitrary block-table edits | a tenant cannot read/write another tenant's KV blocks | +| Agent ↔ agent (in one tenant) | guest userspace agent | arbitrary CUDA kernels | an agent cannot read/write a sibling agent's KV blocks | + +The first three are baseline 1.0 (they follow the threat model in +[`../05-operations/01-performance-correctness-security.md` §4.1](../05-operations/01-performance-correctness-security.md)). +The last two are the *multi-tenant inference* boundaries. Note that they +assume the **guest kernel is trusted**: everything below the application +runs the stock driver, and the tenant/agent separation is enforced at the +application's use of the device, not against a malicious guest KMD. A +malicious guest KMD is the guest↔host boundary, already covered by L0/L1. + +### 1.4 Where the code actually stands (the gap this document fills) + +The guest-facing plane is per mdev: `nv_vgpu_device.gsp_comm_ctx` +(`kernel/include/nvidia-vgpu-vfio.h:133`) wraps a per-instance +`struct gsp_emu_context` (`kernel/gsp_emulation.h:130`) created by +`gsp_emu_init()` (`kernel/gsp_emulation.c:724`) from +`nv_vgpu_device_init_work()` (`kernel/main.c:1657`). BAR shadows, message +queues and MSI-X state are per instance. + +The **work/completion service is not**. `gsp_sema_service.c` holds one +global `static struct sema_svc svc;` (`kernel/gsp_sema_service.c:676`), and: + +* the service thread pumps **the first open mdev it finds** + (`kernel/gsp_sema_service.c:13296-13308`); +* the work-submission doorbell carries no owner — `gsp_sema_service_doorbell(u32 token)` + (`kernel/gsp_sema_service.c:13456`), called from the VFIO write path with + no mdev argument (`kernel/main.c:865`); +* channel registrations are keyed by the guest-chosen `h_object` alone + (`kernel/gsp_sema_service.c:847`, table bounded by `SVC_MAX_CHANNELS 128` + at `:122`), and VA spaces by the guest-chosen `h_vaspace` alone + (`:988`, `:1019`, bounded by `SVC_MAX_VASPACES 8` at `:309`); +* the VA→PA learned map and pins are keyed by GPU VA only + (`kernel/gsp_sema_service.c:113-130`, `:1045`), with no mdev key; +* completion counter pools and rank latches are global boot state, as the + second-client wedge demonstrates: the old client's channel/rank/ring state + makes the host's completion path go quiet for the new client + (`../../debug/issue/issue-number-2.md` §Root cause). + +Two guests can trivially choose the *same* handles and GPU VAs (live: +channel `0xcaf000d4`, `../../debug/issue/issue-number-2.md` §Evidence); keyed by +value alone, those registrations and translations collide — the exact class +that already breaks the second client in one guest. **Multi-mdev isolation +is therefore first a service-state-ownership problem, and only second a GMMU +problem.** §2 fixes the ownership; §4 builds the GMMU policy on it. + +### 1.5 Vocabulary + +**Tenant** — isolation principal (`tenantId`): a VM/mdev at L0–L2, a logical +serving principal inside a guest in §4. **Agent** — finest principal inside +a tenant (`agentId`): a vLLM engine/worker, a CUDA context, a process. +**vEP** — virtual execution plane: the per-instance mediator (guest +emulation + work execution) for one mdev; see +[`13-vexecution-plane.md`](13-vexecution-plane.md). **KV block** — +paged-attention cache page (vLLM `paged_attention_v2`: fixed-size blocks, +one `block_tables` entry per logical block per sequence); a GPU allocation +range, not a hardware object. + +--- + +## 2. Security and isolation design for multi-mdev (vEP terms) + +### 2.1 Address-space separation + +**Rule S1.** Every host-side table that describes guest state is keyed by +`(mdev, guest-name)` — handle, GPU VA, doorbell token, counter slot — never +by the guest-chosen value alone. + +What exists and what changes: + +| State | Today | Required | +|---|---|---| +| emulation context | per mdev `[live]` (`nvidia-vgpu-vfio.h:133`, `gsp_emulation.h:130`) | unchanged | +| `svc.chan[]` | keyed `h_object` (`gsp_sema_service.c:847`) | key `(vgpu, h_object)`; per-mdev lifetime reset | +| `svc.vas[]` | keyed `h_vaspace` (`:988`) | key `(vgpu, h_vaspace)` | +| learned VA→PA map / pins | keyed GPU VA (`:113-130`) | key `(vgpu, VA)`; per-mdev epoch | +| doorbell token | ownerless (`:13456`, `main.c:865`) | `(vgpu, token)`; route to the owner's pump | +| completion counters | global boot state | per mdev, reset on session/rebind | +| pump | first open mdev (`:13296-13308`) | per-mdev pump (or round-robin over all open mdevs) | + +The GMMU half of address-space separation is the page directory: each VA +space has one, published by the guest through `SET_PAGE_DIRECTORY` with its +`hVASpace` and `chId` (`kernel/gsp_rm_emulation.c:3310-3314,3366-3372` for +the recording path; the format is documented at `:3307-3319`). The vEP +records the directory per owner and must translate only within the owning +mdev's tables. `[design]` + +### 2.2 Per-mdev channel ownership + +**Rule S2.** A channel belongs to exactly one mdev for its whole life, and +its engine work, doorbell token, error notifier and completion counters are +resolved through the owner only. + +The channel descriptor today carries GPFIFO VA, USERD, instance and +notifier pages (`kernel/gsp_sema_service.h:22-33`); the registration arrives +from the guest's `GSP_RM_ALLOC` handling while the emulation context is +known (`ctx->vgpu`), so the owner is available at registration time — it is +simply not stored. The fix is structural (add the owner to `svc_channel` +and to every lookup), not algorithmic. + +Engine arbitration is where ownership becomes policy: channels of the same +engine from different mdevs must be scheduled by our policy on top of the +real GSP scheduler ([`10-interaction-and-scheduling.md` §7](10-interaction-and-scheduling.md)). +Today the single real GSP session and its global `gsp_phys_rm_lock` serialize +all real firmware traffic (`11-physical-backend-guest-wiring.md` §1.3), so +one tenant's slow RPC blocks every other tenant's work — head-of-line +blocking, addressed in §2.7. + +### 2.3 DMA / IOMMU interplay + +Ground rules, unchanged by multi-tenancy: + +* Guest memory is reached only through `vfio_dma_rw(&vgpu->vfio_dev, ...)` + (`kernel/gsp_sema_service.c:754-762`, `kernel/gsp_rm_emulation.c:27-55`), + i.e. through the hypervisor's mapping for **that** mdev's VFIO container. + A GPA from guest A cannot be resolved through guest B's device — the + boundary is enforced below us, and we never weaken it + ([`../05-operations/01-performance-correctness-security.md` §4.2](../05-operations/01-performance-correctness-security.md)). +* The emulated vidmem (BAR1) is a private host-RAM buffer per mdev + (`kernel/bar_init.c:60-77`); FB-aperture accesses stay inside the owning + mdev's buffer (`svc_read_ap`/`svc_write_ap` bounds-check + `vgpu->bars[1]`, `kernel/gsp_sema_service.c:786-814`). +* Physical-backend caveat (real VRAM path): on the reference host the + physical GPU's IOMMU group is an identity domain and the mediated device + registers as an emulated-IOMMU VFIO device, so guest GPAs are translated + through the QEMU shared-RAM file rather than a kernel `iommu_domain` + (`11-physical-backend-guest-wiring.md` §P6a). This translation is + per-mdev-gated but is **not** an IOMMU guarantee; with multiple mdevs on + one backend it must be audited per mdev, and the active-mdev gate must + never select another mdev's file. `[design]` +* When the backend owns the GPU, any host-translated address that reaches a + real engine must come from the owning mdev's resolved set, never from a + global table (`11-physical-backend-guest-wiring.md` §6.4). + +### 2.4 Data-path separation + +* BAR0 shadow, BAR1 vidmem shadow, BAR3/instance window are per mdev + (`kernel/bar_init.c:59-77`; create at `kernel/main.c:554`; cited in + `11-physical-backend-guest-wiring.md` §0.1 `[live]`). Two mdevs' + shadows are separate allocations; the invariant to test is that no + handler ever mixes the two base pointers (handlers take + `struct gsp_emu_context *ctx` and dereference `ctx->vgpu`, e.g. + `kernel/gsp_rm_emulation.c:2829-2834`; the tests in §2.8 prove it). +* Real-VRAM forwarding already isolates *engines* from guest memory by + submitting through module-built scratch page tables that map + already-resolved host pages and never reuse a fetch line within a boot + (`11-physical-backend-guest-wiring.md` §P6b). With multiple mdevs, scratch + pools must be per mdev: capacity is a per-boot resource (512 pairs at + 16 PTEs/line in the current design) and shares a hardware resource with + every tenant. +* Guest page tables are **guest-owned** (the driver builds them; we walk + them — `kernel/gsp_sema_service.c:1052-1072`), so a malicious guest can + retarget its own PTEs at any time: fine for guest↔host (the IOMMU caps the + damage to the guest's own memory), but the fundamental limit of intra-guest + enforcement in §4 — *policy can be only as strong as page-table ownership*. + +### 2.5 Completion isolation + +**Rule S3.** A decoded semaphore release is applied only in the owning +mdev's address space, and a completion event is posted only to the owning +mdev's event object. + +Today the release path resolves the target VA through *global* tables +(`svc_resolve_va`, `svc_sem_write`; `kernel/gsp_sema_service.c:2098,1514`) +and posts through `gsp_rm_post_channel_event()` on the emulation context +(`kernel/gsp_rm_emulation.c:602`). The first is the cross-tenant hazard: a +release record decoded from guest A's push buffer could resolve through a +pin learned from guest B if the VA collides. The exactly-once rules must be +preserved per owner (a copy runs at most once per GPFIFO entry; a virtual +operand is translated per 4 KB page; a consumed slot never runs later — +[`06-host-module-and-hybrid-model.md` §4.4](06-host-module-and-hybrid-model.md)). + +### 2.6 Fault containment + +**Rule S4.** A fault (engine fault, malformed RPC, translation failure, +timeout) is attributed to its owner, delivered only to the owner's notifier +and event object, and must not stall or corrupt another tenant. + +The machinery exists per channel (error notifier slots in the channel +descriptor, `kernel/gsp_sema_service.h:31`) and for forwarded engines +(fault observation/relay default-off, +`11-physical-backend-guest-wiring.md` §P6c C3), and the malformed-RPC path +is fail-closed. What is missing is the owner key at the relay decision, and +a per-tenant fault-injection test (§2.8). A wedged server is a +*module-global* fail-safe today (`08-physical-gpu-integration.md` §7), which +is correct for one backend session but must never be reachable from a +guest-triggered path: guest input may abort the guest, never the service. + +### 2.7 Time-slicing risks + +With one physical GPU, tenants share engines by time slicing. Known risks: + +* **Head-of-line blocking:** the single real GSP session + `gsp_phys_rm_lock` + serializes backend RPCs; a tenant that floods controls starves the others. + Mitigation design: per-mdev backend clients (already planned, + [`09-full-system-picture.md` §5](09-full-system-picture.md)), bounded + per-tenant RPC rates in the vEP, and per-tenant accounting. +* **Slice inversion:** guest A's long-running engine work delays B's + completion relay; the relay path must stay bounded and never poll a + hardware queue while holding a lock a second tenant needs + (`11-physical-backend-guest-wiring.md` §1.3 lock order). +* **Starvation:** without a policy, first-come submission can starve a + tenant. The policy is planned + ([`10-interaction-and-scheduling.md` §7](10-interaction-and-scheduling.md)); + until it exists, the honest statement is "fairness is unmeasured". + +### 2.8 Side channels (public knowledge only) + +We make no original side-channel claims and have no measurements. The +classes that apply to any shared-GPU time-slicing design, from public +literature, are: memory-bandwidth contention observable through timing; +L2/LLC state effects; scheduler/slice timing; and interrupt-timing +observation. The project's stated bar is that an adjacent tenant must not +observe timing/state/interrupts *beyond policy* +([`../05-operations/01-performance-correctness-security.md` §4.1](../05-operations/01-performance-correctness-security.md)): +interrupts are already per-instance, timing policy must be explicit and +documented, and measurement is a 1.0-era task (rung 5). Until measured, +treat side channels as **unmitigated but bounded by per-instance state +separation**, and never claim isolation stronger than the ladder proves. + +### 2.9 Validation and test plan for the 1.0 "two guests" gate + +The gate: two QEMU guests, two mdevs, concurrent, stock driver; every test +captures evidence (dmesg, counters) and passes at least 3 consecutive runs. + +| # | Test | Pass criterion | +|---|---|---| +| 2G-01 | Boot both guests concurrently on two mdevs; `nvidia-smi`; CUDA driver-API smoke | both green, payload round-trip exact; no cross errors in dmesg | +| 2G-02 | Distinct VRAM types (e.g. 512M and 1G) on the two mdevs | each guest's `nvidia-smi`/GSCI reports its own size; BAR1 sizes differ and match | +| 2G-03 | Handle/VA collision: force both guests to allocate the same RM handles and GPU VAs (the `0xcaf000d4` class) | both clients progress; no release lands in the wrong guest (verify by canary words around each guest's semaphores) | +| 2G-04 | Doorbell cross-talk: ring guest A's tokens while B is idle | B's channels/state unchanged; B's poll loop untouched; A completes | +| 2G-05 | Completion ownership: craft a release VA that exists in both guests' address spaces | resolves only in the owner's memory; the other guest's canary unchanged | +| 2G-06 | Fault injection: malformed RPC, bad queue link, engine-fault relay from A | A fails closed (documented error); B unaffected; no module-global stall | +| 2G-07 | Teardown: remove mdev B while A runs; rmmod after both stop | A continues; teardown order preserved; module unload clean | +| 2G-08 | Stress: N≥20 concurrent start/stop cycles; guest reboots; module reloads | no leaked per-mdev state; no cross-tenant counters; clean dmesg | +| 2G-09 | Sequencing: the ISSUE-0002 acceptance (3 sequential clients in one boot) plus 2 concurrent processes | green; per-client reset proven (issue-number-2 §Validation plan) | + +Known blockers before the gate can run: the per-client/per-channel reset +(`../../debug/issue/issue-number-2.md` §Fix plan items 1–2) and the per-mdev +service-state split (§2.1 Rule S1). The test harness (two VMs; the current +tools assume one, `../../scripts/vgpu_vm.py`) is itself a deliverable. + +### 2.10 Fail-closed rules + +Isolation properties that cannot be established (missing IOMMU container, +DMA mapping failure, inconsistent owner state, unkeyed table entry) stop the +affected mdev and say why; nothing silently degrades +([`../05-operations/01-performance-correctness-security.md` §4.2](../05-operations/01-performance-correctness-security.md)). +Malformed or stale queue elements, PTE chains and release records are +rejected without disturbing the queue, with strict sequence matching +([`10-interaction-and-scheduling.md` §3](10-interaction-and-scheduling.md)). +No forwarding seam programs guest-derived page tables into a real MMU until +the population path is proven end to end +(`11-physical-backend-guest-wiring.md` §6.4). A cross-owner table lookup is +a programming error: log once and refuse, never "resolve anyway". + +--- + +## 3. VRAM larger than physical, by scheduling and virtualization + +### 3.1 What "VRAM size" means per mdev today + +Each mdev derives a consistent set of numbers from +`nv_vgpu_fb_emul_size(vgpu)` (`kernel/include/nvidia-vgpu-vfio.h:232`): +BAR1 size (`kernel/bar_init.c:68`), the `LOCAL_MEMORY_RANGE` register +encoding (`kernel/bar_emulation.c:271-310`), the GSCI `fb_length` and +`frtsOffset` (`kernel/gsp_rm_emulation.c:1106-1110,1177`) and the WPR2 +address (`kernel/gsp_emulation.c:743-753`). The mdev type advertises the +same size to management tools (`framebuffer=%uM`, +`kernel/main.c:116-132`). The sizes available are dictated by the types the +host driver registers (`kernel/include/ioctl.h:39-46`, +`kernel/nv_vgpu_register_type_info` callers). + +Two consequences: **presenting a size is solved** (two mdevs can present +512 MiB and 2 GiB simultaneously; nothing in the guest path forces them to +match the physical framebuffer), but **backing that size is not** — in +emulation mode each mdev's "VRAM" is a private host-RAM `vzalloc` +(`kernel/bar_init.c:70-77`), so the aggregate is bounded by host RAM; with +real VRAM backing (`08-physical-gpu-integration.md` §4.6) it is bounded by +physical VRAM. Either way `sum(vram_mb)` may exceed capacity — the +oversubscription problem. + +### 3.2 Design: present size independent of physical size + +**Principle.** Guest-visible VRAM size is a *contract*; the resident set is +an *implementation detail* — decouple them with tiered, revocable backing: + +``` + guest-visible VRAM (mdev A, e.g. 2 GiB) guest-visible (mdev B, 1 GiB) + │ │ + ┌──────┴──────────────────────────────┐ ┌──────┴────────────┐ + │ vEP per-mdev residency manager │ │ ... │ + │ pinned set | evictable set │ │ │ + └──────┬──────────────────────────────┘ └───────────────────┘ + │ page-in/out (4 KiB-2 MiB) + ┌────────────┴───────────┐ ┌───────────────────────────────┐ + │ real VRAM (hot blocks) │ │ sysmem staging (cold blocks) │ + └────────────────────────┘ └───────────────────────────────┘ +``` + +Techniques, each with its current status: + +1. **Sparse allocation (allocate-on-touch).** The guest-visible range exists + before it is backed; backing is populated on first write (emulation: RAM + page; backend: real VRAM object). Today the whole BAR1 is + `vzalloc`'d up front (`kernel/bar_init.c:70-77`) `[wip]`; sparse backing + is `[design]`. +2. **VRAM ↔ sysmem paging.** Cold blocks move to guest sysmem (or host + staging) and their PTEs are rewritten to the sysmem aperture; hot blocks + move back. The GMMU already distinguishes apertures + (`svc_pte_aperture`, `kernel/gsp_sema_service.c:1101-1108`) and the guest + uses `ADDR_SYSMEM=1`/`ADDR_FBMEM=2` descriptors + ([`05-gmmu-amap-hal-gr.md` §1.1](05-gmmu-amap-hal-gr.md)) `[design]`. +3. **Replayable pushes.** A push whose instruction stream is re-executed + must be idempotent or suppressed; the host already has the machinery: + per-entry identity checks, per-operation done bitmaps, consumed-slot + marks, and the deferred-release hold that waits for hardware completion + before applying releases exactly once + ([`06-host-module-and-hybrid-model.md` §4.4](06-host-module-and-hybrid-model.md), + `11-physical-backend-guest-wiring.md` §P6b wave 1.5) `[wip]`. + Paging reuses this primitive: an operation that references an evicted + block is held until the block is resident. +4. **GMMU invalidations.** Rewriting a PTE is not enough: the engine caches + PTE fetches (observed 128-byte granularity) and there is no invalidate + path in the forwarding code today; a runtime PTE write into an + already-walked line was accepted but never completed + (`11-physical-backend-guest-wiring.md` §P6b "PTE fetch-line rule"). + The guest's own `MEMSYS_L2_INVALIDATE_EVICT` control is acknowledged + without effect in emulation (`kernel/gsp_rm_emulation.c:2519-2532`). + A real invalidate path is a prerequisite for paging `[design]`. +5. **UVM fault replay.** Guest UVM already handles memory faults and + re-registration; the host can signal "not resident" and let the guest + replay the access. The guest-visible mechanism is the channel + event/notifier path (`gsp_rm_post_channel_event`, + `kernel/gsp_rm_emulation.c:602`); whether every access class can be + replayed this way is `[hypothesis]` — it needs a live experiment. +6. **Migration hooks.** The module already has save/restore state sections + (`kernel/migration.c:85+`); residency metadata (which blocks are + resident, tags, ownership) must be part of the migrated state for + correctness across live migration `[design]`. + +### 3.3 Tradeoffs + +| Effect | Cause | Control | +|---|---|---| +| Latency spike on page-in | block fetch from sysmem to VRAM | bound resident-set misses per slice; prefetch on sequential access (paged attention accesses KV blocks sequentially per sequence) | +| Throughput collapse under thrash | working set > capacity | admission control: refuse the allocation that would guarantee thrash; per-tenant quotas | +| Completion jitter | held pushes waiting for residency | bound the hold; on expiry fail closed per operation (existing bounded-hold pattern, `gsp_sema_service.h:220-236`) | +| Fairness loss | one tenant's evictions victimize another | per-tenant residency targets, self-eviction first (see §3.5) | + +These are unmeasured; the numbers must come from the benchmark harness +([`../05-operations/01-performance-correctness-security.md` §3](../05-operations/01-performance-correctness-security.md)). +Do not quote an overhead figure from this section. + +### 3.4 Front-running risks and correctness invariants + +**Front-running.** A tenant that can force reclaim of *another tenant's* +pages gains two advantages: it can degrade a competitor, and it can race to +observe reused memory if reclaim does not scrub. Rule: reclaim only within +the requesting tenant's own residency budget unless a global-capacity policy +explicitly and observably selects a victim (accounted, rate-limited, and +scrubbed before reuse). + +**Invariants (must hold for any paging implementation).** + +* **I1 — No live PTE to reclaimed memory.** A block may be reallocated only + after every PTE that references it has been revoked and the revocation is + visible to all engines. +* **I2 — Revoke implies quiesce.** No engine work may be in flight on a + block at revoke; the deferred-release hold + channel drain is the + primitive (`11-physical-backend-guest-wiring.md` §P6b). +* **I3 — Exactly-once completion.** Paging must not duplicate or lose a + release; the existing replay-suppression rules apply per owner + ([`06-host-module-and-hybrid-model.md` §4.4](06-host-module-and-hybrid-model.md)). +* **I4 — Owner-stable residency.** A block's backing is owned by one mdev + (or one intra-guest tenant, under §4); no shared physical page between + owners without an explicit, tagged share record. +* **I5 — Scrub before reuse.** Reclaimed memory is zeroed (or poison-filled) + before it can be mapped into any other owner. +* **I6 — Migration carries residency.** Save/restore includes the residency + map and tags, or the restored device must cold-start the cache. + +### 3.5 What vEP must add + +* **Per-mdev accounting:** resident/pinned bytes, fault-in counts, hold + timeouts — the counters in `struct sema_svc` + (`kernel/gsp_sema_service.c:439-676`) made per owner. +* **Reclaim signal:** a guest→vEP hint that a range is evictable (vLLM knows + when a KV block is freed): an allocation/free observed at RM-object level + in the stock-compatible design, or an explicit control in the tagged + design (§4.5). +* **QoS policy:** per-mdev weights/slices on top of the GSP scheduler + ([`10-interaction-and-scheduling.md` §7](10-interaction-and-scheduling.md)), + with measured properties. `[design]` +* **Capacity admission:** enforce the oversubscription ratio at create and + large-allocation time; refuse (fail closed) rather than promise capacity. + +--- + +## 4. Custom GMMU protections for paged-attention multi-tenancy + +> This is the core ask: given a tenant ID and an agent ID, protect a +> paged-attention KV cache shared by many tenants inside one guest. The +> design below is complete on paper across the full stack. Stages M6–M8 +> are **post-1.0 research**: dual-gated, duplicate-not-replace, never +> required by the baseline. Stages M1–M5 belong to the 1.0 programme. + +### 4.1 The workload, precisely + +Public knowledge of `paged_attention_v2`: the KV cache is a pool of +fixed-size **blocks**; each sequence has a **block table** (a GPU tensor of +block indices) mapping its logical positions to physical blocks. Decode +kernels gather K/V from the blocks named by the table. A vLLM server +multiplexes many sequences (many logical tenants) in one continuous batch +inside one process and, typically, one CUDA context. Nothing in this +pipeline is hardware-protected: any kernel in the context can dereference +any block table entry, and a buggy or malicious sequence implementation can +read every other tenant's KV cache. With prefix caching, blocks are even +*shared* between sequences (copy-on-write), which is a feature to preserve, +not eliminate. + +The assumed identities: `tenantId` identifies the serving principal; +`agentId` identifies the finest unit inside a tenant (a worker/context; in +a single-process vLLM, an engine or an attention backend instance). Both are +provided by the orchestration layer; provisioning them is out of scope. + +### 4.2 Hardware GMMU primitives: proven vs hypothesis + +What the Turing (TU104) VER2 format and our live code establish (`docs/NN` +below resolves to `../../debug/risc-v-gsp-rm-vm/docs/NN-*.md`): + +| Primitive | Status | Evidence | +|---|---|---| +| 4-level page tree; 49-bit VA; 4 KiB/64 KiB/2 MiB pages; dual 16-byte PDE0 | **proven** | tracked decoder `../../debug/risc-v-gsp-rm-vm/docs/43-GMMU-PTE-DECODER.md` §6; `kernel/gsp_sema_service.c:1058-1072` | +| VER2 leaf PTE fields incl. `read_only` (bit 6), `privilege` (5), `encrypted` (4), `atomic_disable` (7), `address_sys/vid`, `kind` | **proven format** | `docs/43` §3 | +| Separate page directory per VA space, published per channel (`SET_PAGE_DIRECTORY` with `hVASpace`/`chId`) | **proven** | `kernel/gsp_rm_emulation.c:3310-3314,3366-3372`; live walks `kernel/gsp_sema_service.c:1174-1312` | +| Directory-level apertures and big-page leaves | **proven** | `docs/43` §3–§6; `svc_pde_aperture`, `kernel/gsp_sema_service.c:1110-1117` | +| **No tenant/ACL/PASID field in the VER2 PTE or PDE** | **proven absence in the format** | the full field list in `docs/43` §3–§5 has no such field; `peer_id` is a fabric peer address selector, not a security domain | +| Read-only PTE *enforcement* (the engine refuses a write through an RO leaf) | **hypothesis** | the format defines the bit; we have never exercised an RO-protected leaf on hardware | +| Encryption-domain isolation (CCS/confidential computing) usable as a tenant boundary | **hypothesis** | format has an `encrypted` bit; CCS object model and semantics are unstudied for us | +| MMU invalidate semantics fine-grained enough for per-block revocation | **hypothesis** | no invalidate path exercised; see `11-physical-backend-guest-wiring.md` §P6b | + +**The decisive design fact:** on this generation, **the only proven +hardware isolation primitive is the page directory** (per-context page +tables). Per-PTE ACLs are research-grade and cannot be assumed. Therefore +the design must place every security-relevant decision in the *topology* +(which directory a page appears in, which channel may name it) plus +*software validation and audit* in the vEP. The `read_only` bit is a proven +format field and the right mechanism for *sharing without write access* +(e.g., shared prefix blocks), but its enforcement needs the M7 experiment +before we depend on it. + +### 4.3 The full stack + +The requested chain, end to end: + +``` +┌─────────────────────────────────────────────────────────────────────────────┐ +│ GUEST │ +│ │ +│ extended vLLM (serving engine) │ +│ ├─ per-tenant sequences + block tables (unchanged algorithms) │ +│ ├─ agent-aware KV allocator: blocks come from per-agent VA pools │ +│ └─ calls oiv-agentd for identity + sharing decisions │ +│ │ Unix socket (guest-local) │ +│ oiv-agentd (NEW guest utility; application-level, no driver patch in A) │ +│ ├─ resolves (tenantId, agentId) -> VA pool + tag records │ +│ ├─ Design A: publishes tag records to a mailbox in device memory │ +│ └─ Design B: issues new RM controls through the guest KMD │ +│ │ CUDA driver API (stock) │ RM control (B) │ +│ guest KMD (stock nvidia.ko; unmodified in A, extended in B [research]) │ +│ │ GSP-RM RPC / MMIO │ +├───────────┼─────────────────────────────────────────────────────────────────┤ +│ vEP HOST (vgsp.ko) │ +│ GSP-RM emulation: controls, objects, page-directory records │ +│ tag registry (per mdev): tenant/agent -> VA ranges, access, shares │ +│ page-table policy: owner checks at SET_PAGE_DIRECTORY / map / submit │ +│ channel binding: channel -> (tenant, agent); arbitration + completion │ +│ fault attribution: fault -> channel -> tenant; owner-only notifier │ +│ │ real page tables / forwarded work │ +│ GPU: GMMU page directories (topology) + PTE bits (read_only, kind) │ +└─────────────────────────────────────────────────────────────────────────────┘ +``` + +### 4.4 Design A — VA-subtree partitioning (no guest KMD change) + +This is the **stock-compatible** design and the one that can be prototyped +inside the 1.0 programme. It relies on one idea: **the VA space is the +security domain, and a VA is self-describing because its prefix encodes its +owner.** + +* Extended vLLM reserves a disjoint VA range per agent: `cuMemAddressReserve` + (stock CUDA VMM API) at a base derived from `(tenantId, agentId)`, e.g. + a deterministic partition of the GPU VA space into tenant subtrees and + agent subtrees. KV blocks are `cuMemCreate`d and mapped only inside the + owner's range. +* `oiv-agentd` publishes the binding records (which `(tenantId, agentId)` + owns which VA range, with what access) to a **tag mailbox**: a device + allocation whose content the vEP can read. In emulation mode this works + because guest vidmem *is* host RAM (`kernel/bar_init.c:68-77`; FB + aperture reads are direct, `svc_read_ap` `kernel/gsp_sema_service.c:786-799`), + so the vEP can scan the mailbox page through the owner's shadow. A fixed + mailbox offset (requested from the extension at init) keeps discovery + trivial. +* vEP policy, enforced when walking/populating page tables and at + work-submission time: + 1. a page directory is owned by the mdev (L3) and, under this design, by + a `(tenantId, agentId)` (L7); + 2. a PTE whose VA is in tenant T's subtree may only point at memory + tagged T (or shared with T); + 3. a channel bound to agent A may only execute work referencing VAs in + A's subtree (validation at decode/submit, using the existing + owner-aware claim of §2.2); + 4. violations are refused and attributed to the owning channel. +* Sharing for prefix caching: a shared block lives in the *owner's* range + and is exposed to the borrower through a new PTE in the borrower's + directory with `read_only` set (proven format field) — or, until M7 + proves RO enforcement, not exposed at all and prefix sharing is + downgraded to per-agent copies. + +Limits, stated honestly: + +* It does not protect against a malicious guest KMD (the KMD builds the + page tables; it can lie about prefixes). It protects tenants from each + other *within* the tenant's trust model, which is the vLLM threat model. +* Emulation-mode mailbox discovery does not survive real VRAM backing + (the host cannot read real VRAM as RAM); the real-VRAM version needs a + protocol path (Design B) or a host-readable staging mailbox. +* It requires extended vLLM (not stock vLLM) — an application change, not + a driver change. + +### 4.5 Design B — explicit tenant/agent RM controls `[research]` + +The explicit design: new vGPU-private RM controls flow through the guest +KMD to the vEP; the vEP stores tags, enforces policy, and can build real +page tables from them. This requires a guest KMD that knows the controls; +therefore it is **post-1.0** (it modifies the guest, which the baseline +guest rule forbids, [`../06-portability/hardware-software-matrix.md` §3](../06-portability/hardware-software-matrix.md)), +dual-gated, and never a 1.0 dependency. + +Proposed controls (vGPU-private control range; the proven vGPU-specific +example is `0x90f10106`, `COPY_SERVER_RESERVED_PDES`, +`kernel/gsp_rm_emulation.c:2266`): + +| Control | ID (proposed) | Semantics | +|---|---|---| +| `NV90F1_CTRL_CMD_VGPU_TENANT_BIND` | `0x90f10120` | bind the calling client to a `(tenantId, agentId)`; idempotent join; optional exclusive | +| `NV90F1_CTRL_CMD_VGPU_TENANT_TAG_RANGE` | `0x90f10121` | tag a GPU VA range with owner + access + volatility | +| `NV90F1_CTRL_CMD_VGPU_TENANT_UNTAG_RANGE` | `0x90f10122` | remove a tag; range becomes untouchable by non-owners | +| `NV90F1_CTRL_CMD_VGPU_TENANT_SHARE` | `0x90f10123` | grant a borrower a (revocable) view of an owner range | +| `NV90F1_CTRL_CMD_VGPU_AGENT_CHANNEL_BIND` | `0x90f10124` | bind an RM channel to an agent for arbitration/completion attribution | +| `NV90F1_CTRL_CMD_VGPU_TAG_QUERY` | `0x90f10125` | read-only: effective identity, enforcement mode, range tags | + +Struct sketches (wire types follow the driver's `NvU32`/`NvU64`/`NvHandle` +conventions; all little-endian, 8-byte aligned): + +```c +/* 0x90f10120 */ +typedef struct NV_VGPU_TENANT_BIND_PARAMS { + NvU64 tenantId; /* stable, VM-scoped; 0 invalid */ + NvU64 agentId; /* 0 = tenant's default agent */ + NvU32 flags; /* CREATE 0x1 | JOIN 0x2 | EXCLUSIVE 0x4 */ + NvU32 abiVersion; /* 1 */ + NvU8 name[32]; /* diagnostic only */ +} NV_VGPU_TENANT_BIND_PARAMS; + +/* 0x90f10121 / 0x90f10122 */ +typedef struct NV_VGPU_TENANT_TAG_RANGE_PARAMS { + NvU64 tenantId; + NvU64 agentId; + NvU64 gpuVa; /* page aligned */ + NvU64 size; + NvU32 access; /* READ 0x1 | WRITE 0x2 (0 invalid) */ + NvU32 flags; /* SHARED 0x1 | EVICTABLE 0x2 | PINNED 0x4 | + KV_BLOCK 0x8 | BLOCK_TABLE 0x10 */ + NvU64 tagCookie; /* opaque allocator cookie; echoed in faults */ +} NV_VGPU_TENANT_TAG_RANGE_PARAMS; + +/* 0x90f10123 */ +typedef struct NV_VGPU_TENANT_SHARE_PARAMS { + NvU64 ownerTenantId, ownerAgentId; + NvU64 borrowerTenantId, borrowerAgentId; + NvU64 gpuVa; /* range start as tagged by the owner */ + NvU64 size; + NvU32 access; /* borrower view: READ | READ|WRITE */ + NvU32 flags; /* REVOCABLE 0x1 (default) | COW 0x2 */ +} NV_VGPU_TENANT_SHARE_PARAMS; + +/* 0x90f10124 */ +typedef struct NV_VGPU_AGENT_CHANNEL_BIND_PARAMS { + NvHandle hChannel; + NvU64 tenantId, agentId; + NvU32 flags; /* EXCLUSIVE 0x1 */ + NvU32 abiVersion; /* 1 */ +} NV_VGPU_AGENT_CHANNEL_BIND_PARAMS; +``` + +Design A's mailbox record (the stock-compatible carrier): + +```c +typedef struct NV_VGPU_TAG_MAILBOX_RECORD { + NvU32 magic; /* 'OIV1' = 0x4f495631 */ + NvU32 recordType; /* 1 bind, 2 tag, 3 untag, 4 share */ + NvU64 seq; /* producer sequence; consumed strictly in order */ + NvU64 tenantId, agentId; + NvU64 gpuVa, size; + NvU32 access, flags; + NvU64 tagCookie; + NvU32 crc32; /* over the record with crc32 field = 0 */ +} NV_VGPU_TAG_MAILBOX_RECORD; +``` + +Message-semantics rules: controls are synchronous (reply = status/params); +tag records are monotonic (later record for a range wins; untag terminal); +shares are revocable, revocation delivered to the borrower's event object; +every mutating control is forwarded to the vEP, whose reply is authoritative +— not the echo/zeroed-success fallback used for unknown IDs today +(`kernel/gsp_rm_emulation.c:2829`; `11-physical-backend-guest-wiring.md` +§2.1 #7, #13). + +### 4.6 Where enforcement lives + +| Layer | Mechanism | Strength | +|---|---|---| +| Guest application (vLLM + agentd) | VA-pool discipline, block-table construction | advisory; defeated by a buggy kernel | +| Guest KMD (Design B) | control validation, identity binding, forwarding | local; guest-trusted | +| vEP tag registry + policy | owner checks at directory record / map / submit / completion | **the enforcement point** | +| GMMU topology | separate page directory (+ optional sub-directory per agent) | hardware-enforced for context separation | +| GMMU PTE bits | `read_only` for shares; `kind` for memory type | hardware-enforced *if* M7 confirms | +| Completion/fault paths | owner-keyed resolution and attribution | software, load-bearing | +| Audit | periodic owner-vs-PTE audit; canary pages | detection; catches policy escapes | + +### 4.7 The three granularities + +#### 4.7.1 Per-KV-block (page/PTE level) + +*Tag every block; enforce at map time; share with read-only where hardware +allows.* + +| Aspect | Assessment | +|---|---| +| Benefits | finest sharing (prefix cache reuse across agents); per-block revoke for eviction; fault granularity is the block | +| Drawbacks | per-block metadata and per-map validation cost; vLLM allocates blocks in bursts — validation must be batched or the map path becomes the bottleneck; sharing needs RO enforcement (M7) or degrades to copies | +| Performance | map-time cost per block (currently an emulated RPC is tens of µs — `10-interaction-and-scheduling.md` §6 — and tagging must not add a round trip per block); no extra TLB misses if topology is unchanged, but per-block VA fragmentation reduces big-page opportunities, which *does* cost TLB reach | +| Implementation cost | high: vLLM allocator change, agentd, mailbox/controls, vEP registry, audit | +| Portability | the *concept* is portable; the enforcement is software+topology, because no PTE tenant field exists (proven absence, §4.2); `read_only` mapping must be re-derived per generation | +| Failure modes | stale tag after reallocation (block freed, VA reused); share revoked but a borrower PTE survives (no invalidate, I1 violation); tag/PTE divergence after migration; audit misses a crafted PTE | + +#### 4.7.2 Per-tenant (sub-address-space: separate channel + page directory) + +*One tenant = one page directory + at least one channel; agents inside a +tenant share the directory (and are separated only by VA-subtree convention +unless per-agent directories are added).* + +| Aspect | Assessment | +|---|---| +| Benefits | **the only proven hardware lever** (§4.2); low steady-state overhead (matches the driver's own per-context model); naturally portable; fault and completion attribution are trivial (directory owner = tenant) | +| Drawbacks | no fine-grained sharing; a tenant is a scheduling unit (context switches); TLB working set per tenant; intra-tenant agents are NOT isolated | +| Performance | launch path unchanged; extra contexts cost context-switch and TLB refill; no per-block map overhead | +| Implementation cost | medium: owner keys in the vEP (§2.1–2.6) plus per-mdev backend clients; Design A needs the allocator/agentd work only for the VA pools | +| Portability | high: object model and directory-per-context survive driver-version and (probably) generation changes; exact PTE encoding does not matter | +| Failure modes | directory reuse on handle collision (today's bug class); a page accidentally mapped into two directories; one tenant's RPC wedge stalls others (single backend session); eviction invalidating a sibling's PTE without quiesce | + +#### 4.7.3 Per-agent (finest grouping inside a tenant) + +*One agent = one channel; optionally one sub-directory per agent within the +tenant directory.* + +| Aspect | Assessment | +|---|---| +| Benefits | fault containment per agent (a crashed agent context does not kill siblings); QoS per agent; completion attributed per agent; the natural mapping of CUDA contexts/streams | +| Drawbacks | more channels/contexts = more scheduler state and context switches; per-agent directories multiply TLB pressure and page-table memory; vLLM must be structured so each agent is its own context (a serving-engine change) | +| Performance | overhead grows with agent count; launch unchanged; arbitration and completion bookkeeping per channel | +| Implementation cost | medium–high: channel binding, arbitration policy, per-agent counter pools; vLLM work | +| Portability | high (channels/streams are stock); optional per-agent directories depend on the guest building them (trusted guest) or the host owning page tables (Design B/E1) | +| Failure modes | misbinding a channel to the wrong agent; handle collisions across agents (the ISSUE-0002 class); a shared channel silently breaks agent isolation; completion counters reused across agents | + +**Degradation ladder** (what you lose when you can only do a coarser +grain): + +``` +per-KV-block + RO shares best sharing, software-enforced identity, needs M7 + │ (no RO hw -> drop sharing, keep tagging) +per-tenant directories real hardware context separation; agents inside + │ a tenant share a directory -> intra-tenant + │ isolation is software-only +per-agent channels real fault/QoS separation at channel granularity, + │ no per-block sharing +per-mdev (1.0 baseline) real VM isolation; intra-guest tenants are + application-trusted only +``` + +### 4.8 Failure modes across all granularities + +* **TOCTOU on guest-owned tables.** The guest can rewrite its PTEs between + validation and execution. Mitigation: host-owned page tables for protected + ranges (server-owned contract exists: `COPY_SERVER_RESERVED_PDES`, + `kernel/gsp_rm_emulation.c:2237-2262`; real population is P6c, + `11-physical-backend-guest-wiring.md` §2.1 #9/#10) or validate at + submission and hold the entry (deferred-release hold). +* **Identity laundering.** An agent claims another's `(tenantId, agentId)`. + The identity source is provisioning (guest-local Unix socket in A; KMD + control validation in B); the vEP binds identity to the *client/context* + at bind time, and a second bind for the same range from a different + client is refused. +* **Stale-share escape.** Revoke without invalidate leaves a borrower PTE. + Invariant I1/I2; until invalidate exists, shares must be configured as + copy-on-write or non-revocable-at-runtime (drained between batches). +* **Handle collision.** Two guests/agents pick the same RM handles — + observed. Owner-keyed tables + per-client reset (issue-number-2). +* **Migration inconsistency.** Saved state without tags/residency can + reattach pages to the wrong owner. Invariant I6. + +--- + +## 5. End-to-end walked example: two tenants, `paged_attention_v2`, one guest + +### 5.1 Setup + +One guest, one mdev. Two tenants: **T1** with agent **A1** (model X), **T2** +with agent **A2** (model Y). Each runs its own serving engine process; the +extended vLLM allocates KV blocks from the agent's VA pool. `oiv-agentd` +binds `(T1,A1)` and `(T2,A2)` and publishes tag records for their pools +into the mailbox (Design A) or issues `TENANT_BIND`/`TAG_RANGE` (Design B). +Channels: A1 and A2 each own a compute channel and a copy channel; both are +bound to their agent (`AGENT_CHANNEL_BIND`). + +``` +GUEST VA SPACE (one directory per tenant under Design B, or per agent) + T1 subtree / pool T2 subtree / pool + ┌───────────────┐ ┌───────────────┐ + │ block B11 ... │ │ block B21 ... │ + └───────────────┘ └───────────────┘ + block tables (int32) block tables (int32) + ├─ seq s1: [B11,B12,..] ├─ seq s3: [B21,B22,..] + └─ seq s2: [B13,..] └─ seq s4: [B21,...] (prefix share? B21 RO) +``` + +### 5.2 Flow (happy path) + +```mermaid +sequenceDiagram + autonumber + participant V1 as vLLM engine T1 + participant V2 as vLLM engine T2 + participant AG as oiv-agentd + participant KMD as guest KMD (stock in A) + participant EP as vEP host (vgsp.ko) + participant MMU as GMMU page tables + participant GPU as engines + + V1->>AG: register (T1, A1); request VA pool + V2->>AG: register (T2, A2); request VA pool + AG->>EP: tag records: T1/A1 -> VA range R1, T2/A2 -> R2 (mailbox or RM ctrl) + EP->>EP: registry: owner(R1)=T1/A1, owner(R2)=T2/A2 + V1->>KMD: cuMemCreate/cuMemMap KV blocks inside R1 + V2->>KMD: cuMemCreate/cuMemMap KV blocks inside R2 + KMD->>EP: SET_PAGE_DIRECTORY (recorded, owner = mdev) + EP->>MMU: (backend) PTEs for R1 only in T1 dir, R2 only in T2 dir + V1->>KMD: launch decode kernel on A1 channel (block table in R1) + V2->>KMD: launch decode kernel on A2 channel (block table in R2) + EP->>EP: submit-time check: channel A1 -> VA in R1 only; A2 -> R2 only + EP->>GPU: forward work + GPU-->>EP: completions + EP-->>V1: release -> A1's semaphores only + EP-->>V2: release -> A2's semaphores only +``` + +### 5.3 Malicious attempt, and which layer stops it + +Scenario: tenant T1 crafts a decode kernel whose block table contains a VA +inside R2, or directly dereferences an R2 VA. + +1. **Application layer (none).** vLLM's own bookkeeping does not stop it — + this is the gap being fixed. +2. **GMMU topology.** In T1's page directory there is **no PTE** for an R2 + VA. The engine's access faults (invalid leaf / unmapped). Under + per-tenant directories this is the primary stop; the fault carries the + faulting VA and the channel. +3. **vEP submit-time policy.** Before forwarding, the decoded work is + checked against the channel's allowed VA set (owner map + tags). The + submission is refused and counted *before* any engine access is + programmed; this also covers accesses that would otherwise appear legal + because of a stale PTE. +4. **vEP map-time policy.** If T1 instead tries to *map* an R2-owned block + into R1 (e.g., by getting the guest KMD to place a PTE pointing at + T2's memory through a shared directory), the tag registry rejects the + mapping: owner(R2 physical page) ≠ T1. Refused at population time. +5. **Fault attribution.** The fault (from step 2/3) is attributed to + `(mdev, channel)` → `(T1, A1)`; the notifier and event go only to A1's + event object; T2's channels and completions are untouched. A1's context + is the only thing torn down (policy: kill the offending agent, not the + server). +6. **Audit.** A periodic owner-vs-PTE audit (canary pages at pool + boundaries, checksummed) detects any escape that steps 2–4 missed and + fails closed for the owning tenant. + +What this stop costs: a fault is ms-class (engine fault → status queue → +vEP relay → guest notifier), and the refused submission costs one +validation pass at decode time (class: µs, same order as the existing +per-entry decode, `10-interaction-and-scheduling.md` §6). Neither number +is measured multi-tenant yet. + +### 5.4 Degradation: if only per-tenant separation is available + +Assume the hardware/format experiments (M6–M7) do not land and we ship +only per-tenant directories plus owner-keyed software: + +* T1 and T2 remain separated **if and only if** they are in different + tenants **and** each tenant's agents do not share a directory (the + deployable unit is one agent per tenant, or per-agent contexts). +* If two agents inside one tenant share a directory (the common vLLM + single-process case), the cross-agent guarantee downgrades to + **app-level**: block tables are validated at submit time against the + agent's VA pool, but a kernel dereferencing a sibling's VA resolves (the + PTE exists). The stop becomes detection (audit) + kill, not prevention. +* Prefix-cache sharing across agents degrades to copies (no proven RO + enforcement), costing VRAM and copy bandwidth. +* Honest recommendation for the degraded mode: **one agent per tenant, one + tenant per context** (vLLM tenant count = context count) — a + configuration contract, not a hardware guarantee. + +--- + +## 6. Portability + +### 6.1 What is portable vs not + +| Artifact | Portable? | Why / what to re-derive | +|---|---|---| +| Object model (client, device, vaspace, channel, events) | **yes** | stable across 550–580 as a family; verified live only on 570.144 | +| Page-directory contract (`SET_PAGE_DIRECTORY`/`UNSET`, `COPY_SERVER_RESERVED_PDES`) | **yes, versioned** | controls carry `hVASpace`/`chId`/geometry; re-verify field offsets per version | +| Our GMMU walker/decoder approach | **yes** | decode captured tables; field catalogue per generation (Turing VER2 proven; others re-derive) | +| PTE/PDE bit encodings | **no** | generation-specific (VER2 on Turing; newer formats expected on Blackwell); re-derive from captures + format docs per device | +| Invalidate semantics / fetch-line behavior | **no** | measured per implementation; current 128-byte fetch-line, no-invalidate result is Turing+our-path-specific | +| Tag controls / mailbox (this design) | **concept yes; wire no** | new controls are per-version additions; mailbox depends on guest vidmem being host-readable (emulation only) | +| Per-tenant/per-agent policy | **yes** | policy is ours; mechanism depends on the above | +| mdev type/VRAM sizes | **yes** | per-type metadata registered by the host driver; sizes are policy, not hardware | + +### 6.2 Driver 610 + +RM 610 is under a **hard gate**: no acquisition, analysis, emulation edit, +physical work or support claim before explicit named-human approval +(`../../debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md`). Nothing here +authorizes 610 work; the port plan for this design is M8, gated. Expected +to change and need re-derivation when approved: control inventories and +object model extensions, engine caps, the accelerator/CCS object family, +and possibly the GMMU entry format +(`../../debug/risc-v-gsp-rm-vm/docs/34-BLACKWELL-SOURCES.md`). + +### 6.3 Grace-Hopper / Grace-Blackwell / GB10 + +Targets per the matrix +([`../06-portability/hardware-software-matrix.md` §2](../06-portability/hardware-software-matrix.md)); +what matters here: + +* **aarch64 host.** Page-size/cache-attribute differences are a known risk + ([`hardware-software-matrix.md` §8](../06-portability/hardware-software-matrix.md)); + the VA↔PA discovery technique (guest direct-map arithmetic, + [`06-host-module-and-hybrid-model.md` §4.2](06-host-module-and-hybrid-model.md)) + must be re-verified. +* **Memory hierarchy.** Coherent NVLink-C2C blurs sysmem/vidmem; the + aperture model and the §3 oversubscription design must be re-derived + around C2C coherence. `[hypothesis]` +* **GMMU formats.** Entry formats/page sizes are expected to differ; + anything per-PTE (including a possible ACL-like field) must be re-derived + before being relied on. `[hypothesis]` +* **Confidential computing (CCS).** CoCo changes the trust boundary in our + favour (hardware memory encryption) and against us: the host cannot read + guest memory content, which breaks the content-scan discovery techniques + in the current memory model ([`06-host-module-and-hybrid-model.md` + §4.4](06-host-module-and-hybrid-model.md)). The `encrypted` PTE bit + exists (proven format); semantics are unstudied. `[hypothesis]` +* **MIG** on datacenter parts: an alternative hardware partitioning + substrate the policy layer can use where available + ([`09-reference/glossary-and-faq.md`](../09-reference/glossary-and-faq.md), + "What about MIG?"). `[design]` + +### 6.4 Re-derivation checklist (per new device/version) + +Capture page tables and decode them; confirm the encoding and write down +explicitly whether any owner/ACL-like field exists. Confirm the directory +is still per-context and the controls still carry the VA space and channel. +Confirm invalidate semantics with a controlled experiment (no-op until +proven; M5/M7). Re-run the owner-keying review against the new control/RPC +inventory, then the two-guest gate and the §5 walkthrough, attaching +sessions and M1–M6-style tables per the portability discipline +(`../../debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md`). + +--- + +## 7. Current and future challenges, tooling gaps, roadmap + +### 7.1 Current challenges + +1. **The service-state split is the immediate blocker.** `gsp_sema_service.c` + is global-boot-state code (single `svc`, global counters, value-keyed + tables); splitting it per mdev is review-heavy, with latent assumptions + like the ISSUE-0002 root cause. +2. **GR execution is upstream of every engine-isolation claim.** Until guest + compute executes on real engines (`../04-implementation/03-execution-frontier.md`), + tenant engine isolation can only be validated on the emulation plane. +3. **No per-tenant fault injection.** The backend's `gsp_phys_fault_inject` + (`06-host-module-and-hybrid-model.md` §6) is tied to one session; there is + no "inject as tenant B, prove A unaffected". +4. **No GMMU permission-probe methodology.** The `read_only` bit is a proven + format field but unexercised, and we have no owner/ACL search protocol + beyond decode-and-eyeball. +5. **No two-guest harness.** Tools assume one VM/one mdev + (`../../scripts/vgpu_vm.py`); the 1.0 gate needs a two-VM driver. +6. **No side-channel measurement** — no methodology, no data; explicitly + deferred to measurement by + [`../05-operations/01-performance-correctness-security.md` §4](../05-operations/01-performance-correctness-security.md). +7. **Real VRAM backing is not implemented** (`06-host-module-and-hybrid-model.md` + §8): today isolation-by-address is over emulated RAM; real DMA, real page + tables and shared scratch pools arrive with the backend + (`11-physical-backend-guest-wiring.md` §P6b). +8. **vLLM integration cost.** The extension touches allocator/block + internals; budget a pinned fork plus a thin plugin, not an upstream bet. + +### 7.2 What our tools do not address today + +`docs/NN` below = `../../debug/risc-v-gsp-rm-vm/docs/NN-*.md`. + +| Existing tool (public) | Useful for | Gap | +|---|---|---| +| GMMU PTE decoder (`../../debug/risc-v-gsp-rm-vm/docs/43`) | decode captured entries; field catalogue | decode-only; no permission semantics, no walker dereference | +| GMMU oracle workflow (`docs/44`) | compare simulated vs observed register/value traces | no enforcement experiment; oracle capture is supervised-human | +| Validation predicate probe (`docs/47`) / surgical rehost (`docs/46`) | run one firmware validation branch offline with crafted state | firmware-side; not a GMMU permission probe (candidate substrate, needs a design) | +| Control discovery (`docs/19`) / reach (`docs/19`) | hidden control IDs; symbol reachability | simulator surface only; not real-RM | +| Host debug parameters (`06-host-module-and-hybrid-model.md` §6) | inspect/poke memory, channels, semaphores | single-device oriented; scans are device-selected by first-open | +| Two-guest harness / tenant fault injector / side-channel harness | — | **none of the three exists** | + +### 7.3 Roadmap + +Each stage has an observable acceptance criterion; no stage advances on +intention. **[1.0]** = baseline programme; **[research]** = post-1.0, +dual-gated, duplicate-not-replace, never required by the baseline. + +| Stage | Scope | Acceptance (measurable) | Class | +|---|---|---|---| +| **M1** | Owner-key every vEP table; split `svc` into per-mdev contexts (or keyed tables); per-mdev pump/service; per-client reset | two concurrent mdevs both get service (no first-open starvation); crafted handle/VA collision test fails closed; ISSUE-0002 sequential acceptance stays green; one-guest regression suite green | [1.0] | +| **M2** | Two-guest gate harness + the 2G-01…2G-09 matrix (§2.9) | all tests pass 3× consecutive with evidence; ladder rung 5 "isolation" checklist partially met (fairness still open) | [1.0] | +| **M3** | Per-mdev real GSP clients; per-tenant RPC accounting and bounded rates; head-of-line measurements | two guests' forwarded CE work both completes; p99 cross-tenant delay measured and within the published bound; no cross-tenant completion (canaries) | [1.0, after GR/backend] | +| **M4** | Independent VRAM sizes end to end + sparse backing v1 + capacity admission | two mdevs with different types; sum(declared) > capacity with refusal-only admission; per-mdev accounting counters; no leak under 20 start/stop cycles | [1.0-adjacent] | +| **M5** | Real invalidate path + eviction/reclaim for KV blocks; residency invariants I1–I6 tested | controlled page-out/page-in cycle with hardware work verified byte-exact; no PTE outlives its block; hold timeouts bounded | [1.0-adjacent, needs backend] | +| **M6** | Design A prototype: extended vLLM + `oiv-agentd` + mailbox; VA-subtree policy in the vEP | cross-tenant VA access faults (T2 block from T1 channel) and is attributed to T1; differential test shows no foreign block readable/writable; overhead vs unextended vLLM measured and published | [research] | +| **M7** | GMMU permission/predicate probe: supervised experiment for `read_only` (and any ACL-like field, per generation) | a controlled write through an RO leaf behaves as documented (or the bit is declared unusable); result recorded with sessions and the fidelity label | [research] | +| **M8** | Design B (new RM controls + guest KMD extension) and/or CCS/610 port, only when approved | controls round-trip with enforcement at map and submit; fault attribution per agent; 610/CCS work follows its own gate (`docs/23`) | [research, gated] | + +Kill criteria (so research can end): M6 dies if tagging overhead exceeds +the project's sync-heavy factor without a security gain over M1–M5 (which +deliver VM/tenant separation with stock drivers); M7 dies on a family with +no RO-equivalent and no owner field if the capture shows no enforcement +(per-PTE protection impossible; the design then rests on topology, which +M6 must state); M8 dies if M6 meets the same guarantee within the +baseline's envelope and no CCS requirement materialises. + +### 7.4 Baseline discipline (mandatory) + +Per `AGENTS.md` index entry 11 and +`../10-post-1.0/experimental-host-acceleration.md`, **post-1.0 goals are +not design inputs**: nothing in sections 4.5–4.8, 5.3's research stops, or +stages M6–M8 may shape baseline code (no "just in case" fields, hooks or +parameters). Research stages are dual-gated (compile-time + module-load +flags), duplicate rather than replace, and are never required by the +baseline build or 1.0 acceptance. The 1.0 programme owns M1–M5 only; M6+ +starts after 1.0 goals are met, on an explicit human decision. + +--- + +## 8. Cross-links + +* [`13-vexecution-plane.md`](13-vexecution-plane.md) — vEP vocabulary and + the execution plane this document polices (written in parallel). +* [`05-gmmu-amap-hal-gr.md`](05-gmmu-amap-hal-gr.md) — GMMU, page tables, + channels, scheduling; the hardware background for §4. +* [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md) — + backend ownership, isolation target (§4.8), open questions; + [`11-physical-backend-guest-wiring.md`](11-physical-backend-guest-wiring.md) — + the guest-wiring seams (P6a/P6b/P6c) and risk guards this design needs. +* [`10-interaction-and-scheduling.md`](10-interaction-and-scheduling.md) — + concurrency, ownership rules, multi-tenant scheduling model (§7); + [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) — + memory model, VA↔PA discovery, debug parameters, exactly-once rules. +* [`../05-operations/01-performance-correctness-security.md`](../05-operations/01-performance-correctness-security.md) — + evaluation ladder (rung 5), threat model, measurement discipline; + [`../06-portability/hardware-software-matrix.md`](../06-portability/hardware-software-matrix.md) — + support ladder, new-device/version procedures, guest rule; + [`../01-orientation/01-purpose-and-scope.md` §5](../01-orientation/01-purpose-and-scope.md) — + multi-tenancy model. +* `../../debug/issue/issue-number-1.md`, `../../debug/issue/issue-number-2.md` — the + CUDA reproducibility and per-client state issues M1 resolves; + `../../debug/risc-v-gsp-rm-vm/docs/{43-GMMU-PTE-DECODER,44-GMMU-ORACLE-AND-WORKFLOW,23-RM610-READINESS}.md` — + tooling and gates cited above. +* Local analysis: `LOCAL-MULTI-TENANT-GMMU.md` (adjacent; part of the + local analysis) — provenance behind §4.2 and the unredacted design + reasoning. diff --git a/wiki/GMMU-Multi-Tenant-Protection.md b/wiki/GMMU-Multi-Tenant-Protection.md new file mode 100644 index 0000000..83b00ba --- /dev/null +++ b/wiki/GMMU-Multi-Tenant-Protection.md @@ -0,0 +1,110 @@ +# GMMU Multi-Tenant Protection + +> **Current — companion summary.** This page condenses +> [`../docs/03-architecture/14-multi-tenant-gmmu-protection.md`](../docs/03-architecture/14-multi-tenant-gmmu-protection.md) +> (the canonical document: problem statement, full design, RM struct sketches, +> walked example, portability, M1–M8 roadmap). Read the canonical document for +> anything you intend to build against; this page is the quick map. + +## Overview + +Open-IOV must isolate **multiple mdevs (VMs)** and, for multi-tenant +inference, **multiple tenants inside one guest** at the GMMU. The design has +two halves: + +1. **Service-state ownership (baseline 1.0).** The work/completion service + (`kernel/gsp_sema_service.c`) is one global context (`static struct + sema_svc svc`, `:676`), the pump services the first open mdev + (`:13296-13308`), and channels/VA spaces/translations are keyed by + guest-chosen values alone (`:847`, `:988`, `:113-130`). Two guests can + pick the same handles and VAs (live: channel `0xcaf000d4`; + `../debug/issue/issue-number-2.md`), so owner-keying is a prerequisite + for any GMMU policy. +2. **GMMU policy for paged attention (research).** A tenant/agent-aware KV + cache protection design across vLLM, a guest utility, RM controls, the + vEP, and page tables — post-1.0, dual-gated; never a baseline dependency. + +## Isolation layers + +| Layer | Mechanism | Status | +|---|---|---| +| L0 platform | IOMMU/SMMU per-VM DMA, KVM memory/vCPU boundary | live | +| L1 transport | VFIO mdev per tenant, config/BAR/IRQ mediation | live | +| L2 vEP state | per-mdev emulation context + (needed) per-mdev work service | live / design | +| L3 GMMU | per-context page directory (VA space) | live (guest-owned) / design (host-owned) | +| L4 arbitration | channel ownership, TSG/runlist policy | design | +| L5 completion | owner-keyed release resolution; no cross-tenant apply | wip | +| L6 fault | fault attributed to the owning channel/tenant only | wip | +| L7 application | per-tenant/per-agent VA pooling inside a guest | research | + +## Hardware primitives: proven vs hypothesis + +| Primitive | Status | +|---|---| +| 4-level tree, 49-bit VA, 4K/64K/2M pages, dual PDE0 | **proven** (`docs/43-GMMU-PTE-DECODER`; `kernel/gsp_sema_service.c:1058-1072`) | +| VER2 PTE fields incl. `read_only` (6), `privilege` (5), `encrypted` (4), `kind` | **proven format** (`docs/43` §3) | +| Separate page directory per VA space (`SET_PAGE_DIRECTORY` with `hVASpace`/`chId`) | **proven** (`kernel/gsp_rm_emulation.c:3310-3314`) | +| No tenant/ACL/PASID field in the VER2 PTE/PDE | **proven absence** in the field catalogue | +| RO-leaf write enforcement | **hypothesis** (needs a supervised probe; M7) | +| Encryption-domain (CCS) as a tenant boundary | **hypothesis** | + +## The design in brief + +* **Design A (stock-compatible).** Extended vLLM reserves a VA range per + agent; a new guest utility (`oiv-agentd`) publishes tag records to a + device-memory **tag mailbox** the vEP can read (emulation mode: guest + vidmem is host RAM, `kernel/bar_init.c:68-77`). The vEP enforces + owner↔VA-subtree match at map time and at submit time. No guest KMD + change. Research (M6). +* **Design B (explicit controls).** New vGPU-private RM controls — + `TENANT_BIND` (0x90f10120), `TAG_RANGE` (…21), `UNTAG_RANGE` (…22), + `TENANT_SHARE` (…23), `AGENT_CHANNEL_BIND` (…24), `TAG_QUERY` (…25) — + plus struct sketches in the canonical doc §4.5. Requires a guest KMD + extension: post-1.0, gated (M8). +* **Granularities.** Per-KV-block tagging (finest; software+PTE policy, + sharing needs RO enforcement); per-tenant separate page directories + (the only proven hardware lever); per-agent separate channels (fault/QoS + containment; optional per-agent sub-directories). Degradation ladder and + per-granularity cost/failure analysis in canonical §4.7. + +## Walked example (two tenants, paged_attention_v2) + +Two agents A1/A2 each own a VA pool and channels. A malicious T1 kernel +naming an A2 VA hits a missing PTE (GMMU fault) and/or is refused by vEP +submit-time policy; the fault is attributed to A1 only (T2 untouched); a +periodic owner-vs-PTE audit with canaries catches escapes. If only +per-tenant separation exists, the deployable contract is **one agent per +tenant, one tenant per context** (canonical §5.4). + +## Key code anchors (master `66388560`) + +| What | Where | +|---|---| +| per-mdev emulation context | `kernel/include/nvidia-vgpu-vfio.h:133`, `kernel/gsp_emulation.h:130` | +| global service state | `kernel/gsp_sema_service.c:676`; pump `:13296`; doorbell `:13456` | +| channel/vaspace registries | `kernel/gsp_sema_service.c:847`, `:988` | +| GMMU walk (VER2) | `kernel/gsp_sema_service.c:1174`; decode `:1093` | +| page directory record | `kernel/gsp_rm_emulation.c:3320`; reserved PDEs `:2237-2262` | +| per-mdev FB sizing | `kernel/bar_init.c:60-77`; `kernel/include/ioctl.h:39-46` | +| L2 invalidate acked | `kernel/gsp_rm_emulation.c:2519-2532` | + +## Roadmap (acceptance criteria in canonical §7.3) + +M1 per-mdev service state — M2 two-guest isolation gate — M3 per-mdev real +GSP clients + fairness — M4 independent VRAM sizes/sparse backing — M5 +invalidate + eviction/reclaim (I1–I6 invariants) — M6 Design A prototype — +M7 RO/permission probe — M8 Design B / CCS / 610 (gated). M1–M5 are the +1.0 programme; M6–M8 are post-1.0 research, dual-gated, duplicate-not- +replace, never required by the baseline. RM 610 stays behind its own human +gate (`docs/23-RM610-READINESS`). + +## Related documentation + +* [`../docs/03-architecture/14-multi-tenant-gmmu-protection.md`](../docs/03-architecture/14-multi-tenant-gmmu-protection.md) — canonical +* [`../docs/03-architecture/13-vexecution-plane.md`](../docs/03-architecture/13-vexecution-plane.md) — vEP vocabulary (parallel) +* [`../docs/03-architecture/05-gmmu-amap-hal-gr.md`](../docs/03-architecture/05-gmmu-amap-hal-gr.md) — GMMU background +* [`../docs/03-architecture/09-full-system-picture.md`](../docs/03-architecture/09-full-system-picture.md) — multi-tenancy and isolation map +* [`../docs/03-architecture/11-physical-backend-guest-wiring.md`](../docs/03-architecture/11-physical-backend-guest-wiring.md) — forwarding and page-table seams +* [`../docs/05-operations/01-performance-correctness-security.md`](../docs/05-operations/01-performance-correctness-security.md) — threat model, rung 5 +* [`../debug/issue/issue-number-2.md`](../debug/issue/issue-number-2.md) — per-client state wedge +* [`../debug/risc-v-gsp-rm-vm/docs/43-GMMU-PTE-DECODER.md`](../debug/risc-v-gsp-rm-vm/docs/43-GMMU-PTE-DECODER.md) — GMMU field catalogue -- cgit v1.2.3