# GMMU Multi-Tenant Protection

> **Current — companion summary.** This page condenses
> [`../docs/03-architecture/14-multi-tenant-gmmu-protection.md`](../docs/03-architecture/14-multi-tenant-gmmu-protection.md)
> (the canonical document: problem statement, full design, RM struct sketches,
> walked example, portability, M1–M8 roadmap). Read the canonical document for
> anything you intend to build against; this page is the quick map.

## Overview

Open-IOV must isolate **multiple mdevs (VMs)** and, for multi-tenant
inference, **multiple tenants inside one guest** at the GMMU. The design has
two halves:

1. **Service-state ownership (baseline 1.0).** The work/completion service
   (`kernel/gsp_sema_service.c`) is one global context (`static struct
   sema_svc svc`, `:676`), the pump services the first open mdev
   (`:13352-13364`), and channels/VA spaces/translations are keyed by
   guest-chosen values alone (`:847`, `:988`, `:113-130`). Two guests can
   pick the same handles and VAs (live: channel `0xcaf000d4`;
   `../debug/issue/issue-number-2.md`), so owner-keying is a prerequisite
   for any GMMU policy.
2. **GMMU policy for paged attention (research).** A tenant/agent-aware KV
   cache protection design across vLLM, a guest utility, RM controls, the
   vEP, and page tables — post-1.0, dual-gated; never a baseline dependency.

## Isolation layers

| Layer | Mechanism | Status |
|---|---|---|
| L0 platform | IOMMU/SMMU per-VM DMA, KVM memory/vCPU boundary | live |
| L1 transport | VFIO mdev per tenant, config/BAR/IRQ mediation | live |
| L2 vEP state | per-mdev emulation context + (needed) per-mdev work service | live / design |
| L3 GMMU | per-context page directory (VA space) | live (guest-owned) / design (host-owned) |
| L4 arbitration | channel ownership, TSG/runlist policy | design |
| L5 completion | owner-keyed release resolution; no cross-tenant apply | wip |
| L6 fault | fault attributed to the owning channel/tenant only | wip |
| L7 application | per-tenant/per-agent VA pooling inside a guest | research |

## Hardware primitives: proven vs hypothesis

| Primitive | Status |
|---|---|
| 4-level tree, 49-bit VA, 4K/64K/2M pages, dual PDE0 | **proven** (`docs/43-GMMU-PTE-DECODER`; `kernel/gsp_sema_service.c:1058-1072`) |
| VER2 PTE fields incl. `read_only` (6), `privilege` (5), `encrypted` (4), `kind` | **proven format** (`docs/43` §3) |
| Separate page directory per VA space (`SET_PAGE_DIRECTORY` with `hVASpace`/`chId`) | **proven** (`kernel/gsp_rm_emulation.c:3310-3314`) |
| No tenant/ACL/PASID field in the VER2 PTE/PDE | **proven absence** in the field catalogue |
| RO-leaf write enforcement | **hypothesis** (needs a supervised probe; M7) |
| Encryption-domain (CCS) as a tenant boundary | **hypothesis** |

## The design in brief

* **Design A (stock-compatible).** Extended vLLM reserves a VA range per
  agent; a new guest utility (`oiv-agentd`) publishes tag records to a
  device-memory **tag mailbox** the vEP can read (emulation mode: guest
  vidmem is host RAM, `kernel/bar_init.c:68-77`). The vEP enforces
  owner↔VA-subtree match at map time and at submit time. No guest KMD
  change. Research (M6).
* **Design B (explicit controls).** New vGPU-private RM controls —
  `TENANT_BIND` (0x90f10120), `TAG_RANGE` (…21), `UNTAG_RANGE` (…22),
  `TENANT_SHARE` (…23), `AGENT_CHANNEL_BIND` (…24), `TAG_QUERY` (…25) —
  plus struct sketches in the canonical doc §4.5. Requires a guest KMD
  extension: post-1.0, gated (M8).
* **Granularities.** Per-KV-block tagging (finest; software+PTE policy,
  sharing needs RO enforcement); per-tenant separate page directories
  (the only proven hardware lever); per-agent separate channels (fault/QoS
  containment; optional per-agent sub-directories). Degradation ladder and
  per-granularity cost/failure analysis in canonical §4.7.

## Walked example (two tenants, paged_attention_v2)

Two agents A1/A2 each own a VA pool and channels. A malicious T1 kernel
naming an A2 VA hits a missing PTE (GMMU fault) and/or is refused by vEP
submit-time policy; the fault is attributed to A1 only (T2 untouched); a
periodic owner-vs-PTE audit with canaries catches escapes. If only
per-tenant separation exists, the deployable contract is **one agent per
tenant, one tenant per context** (canonical §5.4).

## Key code anchors (master `1f571dda`)

| What | Where |
|---|---|
| per-mdev emulation context | `kernel/include/nvidia-vgpu-vfio.h:133`, `kernel/gsp_emulation.h:130` |
| global service state | `kernel/gsp_sema_service.c:676`; pump `:13352`; doorbell `:13512` |
| channel/vaspace registries | `kernel/gsp_sema_service.c:847`, `:988` |
| GMMU walk (VER2) | `kernel/gsp_sema_service.c:1174`; decode `:1093` |
| page directory record | `kernel/gsp_rm_emulation.c:3320`; reserved PDEs `:2237-2262` |
| per-mdev FB sizing | `kernel/bar_init.c:60-77`; `kernel/include/ioctl.h:39-46` |
| L2 invalidate acked | `kernel/gsp_rm_emulation.c:2519-2532` |

## Roadmap (acceptance criteria in canonical §7.3)

M1 per-mdev service state — M2 two-guest isolation gate — M3 per-mdev real
GSP clients + fairness — M4 independent VRAM sizes/sparse backing — M5
invalidate + eviction/reclaim (I1–I6 invariants) — M6 Design A prototype —
M7 RO/permission probe — M8 Design B / CCS / 610 (gated). M1–M5 are the
1.0 programme; M6–M8 are post-1.0 research, dual-gated, duplicate-not-
replace, never required by the baseline. RM 610 stays behind its own human
gate (`docs/23-RM610-READINESS`).

## Related documentation

* [`Introduction-to-GMMU.md`](Introduction-to-GMMU.md) — from-first-principles GMMU introduction (what it is, page tables, IOMMU/MMU/SMMU relationships)
* [`../docs/03-architecture/14-multi-tenant-gmmu-protection.md`](../docs/03-architecture/14-multi-tenant-gmmu-protection.md) — canonical
* [`../docs/03-architecture/13-vexecution-plane.md`](../docs/03-architecture/13-vexecution-plane.md) — vEP vocabulary (parallel)
* [`../docs/03-architecture/05-gmmu-amap-hal-gr.md`](../docs/03-architecture/05-gmmu-amap-hal-gr.md) — GMMU background
* [`../docs/03-architecture/09-full-system-picture.md`](../docs/03-architecture/09-full-system-picture.md) — multi-tenancy and isolation map
* [`../docs/03-architecture/11-physical-backend-guest-wiring.md`](../docs/03-architecture/11-physical-backend-guest-wiring.md) — forwarding and page-table seams
* [`../docs/05-operations/01-performance-correctness-security.md`](../docs/05-operations/01-performance-correctness-security.md) — threat model, rung 5
* [`../debug/issue/issue-number-2.md`](../debug/issue/issue-number-2.md) — per-client state wedge
* [`../debug/risc-v-gsp-rm-vm/docs/43-GMMU-PTE-DECODER.md`](../debug/risc-v-gsp-rm-vm/docs/43-GMMU-PTE-DECODER.md) — GMMU field catalogue