diff options
Diffstat (limited to 'wiki')
| -rw-r--r-- | wiki/GMMU-Multi-Tenant-Protection.md | 110 |
1 files changed, 110 insertions, 0 deletions
diff --git a/wiki/GMMU-Multi-Tenant-Protection.md b/wiki/GMMU-Multi-Tenant-Protection.md new file mode 100644 index 0000000..83b00ba --- /dev/null +++ b/wiki/GMMU-Multi-Tenant-Protection.md @@ -0,0 +1,110 @@ +# GMMU Multi-Tenant Protection + +> **Current — companion summary.** This page condenses +> [`../docs/03-architecture/14-multi-tenant-gmmu-protection.md`](../docs/03-architecture/14-multi-tenant-gmmu-protection.md) +> (the canonical document: problem statement, full design, RM struct sketches, +> walked example, portability, M1–M8 roadmap). Read the canonical document for +> anything you intend to build against; this page is the quick map. + +## Overview + +Open-IOV must isolate **multiple mdevs (VMs)** and, for multi-tenant +inference, **multiple tenants inside one guest** at the GMMU. The design has +two halves: + +1. **Service-state ownership (baseline 1.0).** The work/completion service + (`kernel/gsp_sema_service.c`) is one global context (`static struct + sema_svc svc`, `:676`), the pump services the first open mdev + (`:13296-13308`), and channels/VA spaces/translations are keyed by + guest-chosen values alone (`:847`, `:988`, `:113-130`). Two guests can + pick the same handles and VAs (live: channel `0xcaf000d4`; + `../debug/issue/issue-number-2.md`), so owner-keying is a prerequisite + for any GMMU policy. +2. **GMMU policy for paged attention (research).** A tenant/agent-aware KV + cache protection design across vLLM, a guest utility, RM controls, the + vEP, and page tables — post-1.0, dual-gated; never a baseline dependency. + +## Isolation layers + +| Layer | Mechanism | Status | +|---|---|---| +| L0 platform | IOMMU/SMMU per-VM DMA, KVM memory/vCPU boundary | live | +| L1 transport | VFIO mdev per tenant, config/BAR/IRQ mediation | live | +| L2 vEP state | per-mdev emulation context + (needed) per-mdev work service | live / design | +| L3 GMMU | per-context page directory (VA space) | live (guest-owned) / design (host-owned) | +| L4 arbitration | channel ownership, TSG/runlist policy | design | +| L5 completion | owner-keyed release resolution; no cross-tenant apply | wip | +| L6 fault | fault attributed to the owning channel/tenant only | wip | +| L7 application | per-tenant/per-agent VA pooling inside a guest | research | + +## Hardware primitives: proven vs hypothesis + +| Primitive | Status | +|---|---| +| 4-level tree, 49-bit VA, 4K/64K/2M pages, dual PDE0 | **proven** (`docs/43-GMMU-PTE-DECODER`; `kernel/gsp_sema_service.c:1058-1072`) | +| VER2 PTE fields incl. `read_only` (6), `privilege` (5), `encrypted` (4), `kind` | **proven format** (`docs/43` §3) | +| Separate page directory per VA space (`SET_PAGE_DIRECTORY` with `hVASpace`/`chId`) | **proven** (`kernel/gsp_rm_emulation.c:3310-3314`) | +| No tenant/ACL/PASID field in the VER2 PTE/PDE | **proven absence** in the field catalogue | +| RO-leaf write enforcement | **hypothesis** (needs a supervised probe; M7) | +| Encryption-domain (CCS) as a tenant boundary | **hypothesis** | + +## The design in brief + +* **Design A (stock-compatible).** Extended vLLM reserves a VA range per + agent; a new guest utility (`oiv-agentd`) publishes tag records to a + device-memory **tag mailbox** the vEP can read (emulation mode: guest + vidmem is host RAM, `kernel/bar_init.c:68-77`). The vEP enforces + owner↔VA-subtree match at map time and at submit time. No guest KMD + change. Research (M6). +* **Design B (explicit controls).** New vGPU-private RM controls — + `TENANT_BIND` (0x90f10120), `TAG_RANGE` (…21), `UNTAG_RANGE` (…22), + `TENANT_SHARE` (…23), `AGENT_CHANNEL_BIND` (…24), `TAG_QUERY` (…25) — + plus struct sketches in the canonical doc §4.5. Requires a guest KMD + extension: post-1.0, gated (M8). +* **Granularities.** Per-KV-block tagging (finest; software+PTE policy, + sharing needs RO enforcement); per-tenant separate page directories + (the only proven hardware lever); per-agent separate channels (fault/QoS + containment; optional per-agent sub-directories). Degradation ladder and + per-granularity cost/failure analysis in canonical §4.7. + +## Walked example (two tenants, paged_attention_v2) + +Two agents A1/A2 each own a VA pool and channels. A malicious T1 kernel +naming an A2 VA hits a missing PTE (GMMU fault) and/or is refused by vEP +submit-time policy; the fault is attributed to A1 only (T2 untouched); a +periodic owner-vs-PTE audit with canaries catches escapes. If only +per-tenant separation exists, the deployable contract is **one agent per +tenant, one tenant per context** (canonical §5.4). + +## Key code anchors (master `66388560`) + +| What | Where | +|---|---| +| per-mdev emulation context | `kernel/include/nvidia-vgpu-vfio.h:133`, `kernel/gsp_emulation.h:130` | +| global service state | `kernel/gsp_sema_service.c:676`; pump `:13296`; doorbell `:13456` | +| channel/vaspace registries | `kernel/gsp_sema_service.c:847`, `:988` | +| GMMU walk (VER2) | `kernel/gsp_sema_service.c:1174`; decode `:1093` | +| page directory record | `kernel/gsp_rm_emulation.c:3320`; reserved PDEs `:2237-2262` | +| per-mdev FB sizing | `kernel/bar_init.c:60-77`; `kernel/include/ioctl.h:39-46` | +| L2 invalidate acked | `kernel/gsp_rm_emulation.c:2519-2532` | + +## Roadmap (acceptance criteria in canonical §7.3) + +M1 per-mdev service state — M2 two-guest isolation gate — M3 per-mdev real +GSP clients + fairness — M4 independent VRAM sizes/sparse backing — M5 +invalidate + eviction/reclaim (I1–I6 invariants) — M6 Design A prototype — +M7 RO/permission probe — M8 Design B / CCS / 610 (gated). M1–M5 are the +1.0 programme; M6–M8 are post-1.0 research, dual-gated, duplicate-not- +replace, never required by the baseline. RM 610 stays behind its own human +gate (`docs/23-RM610-READINESS`). + +## Related documentation + +* [`../docs/03-architecture/14-multi-tenant-gmmu-protection.md`](../docs/03-architecture/14-multi-tenant-gmmu-protection.md) — canonical +* [`../docs/03-architecture/13-vexecution-plane.md`](../docs/03-architecture/13-vexecution-plane.md) — vEP vocabulary (parallel) +* [`../docs/03-architecture/05-gmmu-amap-hal-gr.md`](../docs/03-architecture/05-gmmu-amap-hal-gr.md) — GMMU background +* [`../docs/03-architecture/09-full-system-picture.md`](../docs/03-architecture/09-full-system-picture.md) — multi-tenancy and isolation map +* [`../docs/03-architecture/11-physical-backend-guest-wiring.md`](../docs/03-architecture/11-physical-backend-guest-wiring.md) — forwarding and page-table seams +* [`../docs/05-operations/01-performance-correctness-security.md`](../docs/05-operations/01-performance-correctness-security.md) — threat model, rung 5 +* [`../debug/issue/issue-number-2.md`](../debug/issue/issue-number-2.md) — per-client state wedge +* [`../debug/risc-v-gsp-rm-vm/docs/43-GMMU-PTE-DECODER.md`](../debug/risc-v-gsp-rm-vm/docs/43-GMMU-PTE-DECODER.md) — GMMU field catalogue |