1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
|
# GMMU Multi-Tenant Protection
> **Current — companion summary.** This page condenses
> [`../docs/03-architecture/14-multi-tenant-gmmu-protection.md`](../docs/03-architecture/14-multi-tenant-gmmu-protection.md)
> (the canonical document: problem statement, full design, RM struct sketches,
> walked example, portability, M1–M8 roadmap). Read the canonical document for
> anything you intend to build against; this page is the quick map.
## Overview
Open-IOV must isolate **multiple mdevs (VMs)** and, for multi-tenant
inference, **multiple tenants inside one guest** at the GMMU. The design has
two halves:
1. **Service-state ownership (baseline 1.0).** The work/completion service
(`kernel/gsp_sema_service.c`) is one global context (`static struct
sema_svc svc`, `:676`), the pump services the first open mdev
(`:13296-13308`), and channels/VA spaces/translations are keyed by
guest-chosen values alone (`:847`, `:988`, `:113-130`). Two guests can
pick the same handles and VAs (live: channel `0xcaf000d4`;
`../debug/issue/issue-number-2.md`), so owner-keying is a prerequisite
for any GMMU policy.
2. **GMMU policy for paged attention (research).** A tenant/agent-aware KV
cache protection design across vLLM, a guest utility, RM controls, the
vEP, and page tables — post-1.0, dual-gated; never a baseline dependency.
## Isolation layers
| Layer | Mechanism | Status |
|---|---|---|
| L0 platform | IOMMU/SMMU per-VM DMA, KVM memory/vCPU boundary | live |
| L1 transport | VFIO mdev per tenant, config/BAR/IRQ mediation | live |
| L2 vEP state | per-mdev emulation context + (needed) per-mdev work service | live / design |
| L3 GMMU | per-context page directory (VA space) | live (guest-owned) / design (host-owned) |
| L4 arbitration | channel ownership, TSG/runlist policy | design |
| L5 completion | owner-keyed release resolution; no cross-tenant apply | wip |
| L6 fault | fault attributed to the owning channel/tenant only | wip |
| L7 application | per-tenant/per-agent VA pooling inside a guest | research |
## Hardware primitives: proven vs hypothesis
| Primitive | Status |
|---|---|
| 4-level tree, 49-bit VA, 4K/64K/2M pages, dual PDE0 | **proven** (`docs/43-GMMU-PTE-DECODER`; `kernel/gsp_sema_service.c:1058-1072`) |
| VER2 PTE fields incl. `read_only` (6), `privilege` (5), `encrypted` (4), `kind` | **proven format** (`docs/43` §3) |
| Separate page directory per VA space (`SET_PAGE_DIRECTORY` with `hVASpace`/`chId`) | **proven** (`kernel/gsp_rm_emulation.c:3310-3314`) |
| No tenant/ACL/PASID field in the VER2 PTE/PDE | **proven absence** in the field catalogue |
| RO-leaf write enforcement | **hypothesis** (needs a supervised probe; M7) |
| Encryption-domain (CCS) as a tenant boundary | **hypothesis** |
## The design in brief
* **Design A (stock-compatible).** Extended vLLM reserves a VA range per
agent; a new guest utility (`oiv-agentd`) publishes tag records to a
device-memory **tag mailbox** the vEP can read (emulation mode: guest
vidmem is host RAM, `kernel/bar_init.c:68-77`). The vEP enforces
owner↔VA-subtree match at map time and at submit time. No guest KMD
change. Research (M6).
* **Design B (explicit controls).** New vGPU-private RM controls —
`TENANT_BIND` (0x90f10120), `TAG_RANGE` (…21), `UNTAG_RANGE` (…22),
`TENANT_SHARE` (…23), `AGENT_CHANNEL_BIND` (…24), `TAG_QUERY` (…25) —
plus struct sketches in the canonical doc §4.5. Requires a guest KMD
extension: post-1.0, gated (M8).
* **Granularities.** Per-KV-block tagging (finest; software+PTE policy,
sharing needs RO enforcement); per-tenant separate page directories
(the only proven hardware lever); per-agent separate channels (fault/QoS
containment; optional per-agent sub-directories). Degradation ladder and
per-granularity cost/failure analysis in canonical §4.7.
## Walked example (two tenants, paged_attention_v2)
Two agents A1/A2 each own a VA pool and channels. A malicious T1 kernel
naming an A2 VA hits a missing PTE (GMMU fault) and/or is refused by vEP
submit-time policy; the fault is attributed to A1 only (T2 untouched); a
periodic owner-vs-PTE audit with canaries catches escapes. If only
per-tenant separation exists, the deployable contract is **one agent per
tenant, one tenant per context** (canonical §5.4).
## Key code anchors (master `66388560`)
| What | Where |
|---|---|
| per-mdev emulation context | `kernel/include/nvidia-vgpu-vfio.h:133`, `kernel/gsp_emulation.h:130` |
| global service state | `kernel/gsp_sema_service.c:676`; pump `:13296`; doorbell `:13456` |
| channel/vaspace registries | `kernel/gsp_sema_service.c:847`, `:988` |
| GMMU walk (VER2) | `kernel/gsp_sema_service.c:1174`; decode `:1093` |
| page directory record | `kernel/gsp_rm_emulation.c:3320`; reserved PDEs `:2237-2262` |
| per-mdev FB sizing | `kernel/bar_init.c:60-77`; `kernel/include/ioctl.h:39-46` |
| L2 invalidate acked | `kernel/gsp_rm_emulation.c:2519-2532` |
## Roadmap (acceptance criteria in canonical §7.3)
M1 per-mdev service state — M2 two-guest isolation gate — M3 per-mdev real
GSP clients + fairness — M4 independent VRAM sizes/sparse backing — M5
invalidate + eviction/reclaim (I1–I6 invariants) — M6 Design A prototype —
M7 RO/permission probe — M8 Design B / CCS / 610 (gated). M1–M5 are the
1.0 programme; M6–M8 are post-1.0 research, dual-gated, duplicate-not-
replace, never required by the baseline. RM 610 stays behind its own human
gate (`docs/23-RM610-READINESS`).
## Related documentation
* [`../docs/03-architecture/14-multi-tenant-gmmu-protection.md`](../docs/03-architecture/14-multi-tenant-gmmu-protection.md) — canonical
* [`../docs/03-architecture/13-vexecution-plane.md`](../docs/03-architecture/13-vexecution-plane.md) — vEP vocabulary (parallel)
* [`../docs/03-architecture/05-gmmu-amap-hal-gr.md`](../docs/03-architecture/05-gmmu-amap-hal-gr.md) — GMMU background
* [`../docs/03-architecture/09-full-system-picture.md`](../docs/03-architecture/09-full-system-picture.md) — multi-tenancy and isolation map
* [`../docs/03-architecture/11-physical-backend-guest-wiring.md`](../docs/03-architecture/11-physical-backend-guest-wiring.md) — forwarding and page-table seams
* [`../docs/05-operations/01-performance-correctness-security.md`](../docs/05-operations/01-performance-correctness-security.md) — threat model, rung 5
* [`../debug/issue/issue-number-2.md`](../debug/issue/issue-number-2.md) — per-client state wedge
* [`../debug/risc-v-gsp-rm-vm/docs/43-GMMU-PTE-DECODER.md`](../debug/risc-v-gsp-rm-vm/docs/43-GMMU-PTE-DECODER.md) — GMMU field catalogue
|
