docs(multi-tenant-gmmu): multi-mdev isolation design, GMMU tenant/agent paged-attention protection, VRAM oversubscription, portability and M1-M8 roadmap (wiki summary; local provenance in adjacent LOCAL notes)

1 files changed, 1095 insertions, 0 deletions

diff --git a/docs/03-architecture/14-multi-tenant-gmmu-protection.md b/docs/03-architecture/14-multi-tenant-gmmu-protection.md
new file mode 100644
index 0000000..c3f58c7
--- /dev/null
+++ b/docs/03-architecture/14-multi-tenant-gmmu-protection.md
@@ -0,0 +1,1095 @@
+# Multi-Tenant GMMU Protection: Isolation Design and Paged-Attention Controls
+
+> **Status: design (2026-10-08, master `66388560`).** This document specifies
+> how Open-IOV isolates multiple tenants — multiple mdevs *and* multiple
+> tenants inside one guest — at the GPU memory-management unit (GMMU), and how
+> a paged-attention inference stack (vLLM-style) can use those protections.
+> Sections 1–3 are **baseline 1.0 design**; sections 4.5+ and roadmap stages
+> M6–M8 are **post-1.0 research**, explicitly gated (see §7.4 and
+> `AGENTS.md` index entry 11). No part of the research design may become a
+> design input for baseline 1.0 code: no forward-looking parameter, hook or
+> abstraction may be added to a product path for it.
+>
+> **Method.** Every empirical claim about our system cites a code line or
+> tracked document; hardware-format claims are marked **proven** (observed
+> live, or in a tracked field catalogue) or **hypothesis** (needs a
+> supervised experiment). No timing/cycle/silicon-accuracy claim is made.
+>
+>
+> **Companion document.** [`13-vexecution-plane.md`](13-vexecution-plane.md)
+> (written in parallel) defines the vEP/instance/agent vocabulary used here;
+> where the two disagree, 13 is the vocabulary authority and this document
+> is the GMMU-policy authority.
+
+Tags: `[live]` exercised; `[wip]` default-off; `[design]` not built; `[hypothesis]` unexercised; `[research]` post-1.0, dual-gated.
+
+---
+
+## 1. Problem statement: 1.0 needs N mdevs, N guests, and flawless isolation
+
+### 1.1 The 1.0 requirement
+
+The project's stated 1.0 gate is multiple mediated devices, two or more
+guests, multiple VRAM sizes, and isolation that survives adversarial review.
+The tracked status is explicit that this area is **designed, partially
+implemented** ([`../README.md` §3](../README.md), row "Multi-tenant isolation
+(VFs, mdev-per-VM)"), and the multi-tenancy model is stated as a first-class
+value: one mdev per VM, per-instance interrupts, no shared host state, fail
+closed ([`../01-orientation/01-purpose-and-scope.md` §5](../01-orientation/01-purpose-and-scope.md)).
+The evaluation ladder puts "N VMs, isolation + fairness + no cross-talk" on
+rung 5 ([`../05-operations/01-performance-correctness-security.md` §1](../05-operations/01-performance-correctness-security.md)).
+
+Three 1.0 requirements drive this document:
+
+1. **Two guests on one GPU.** Two VMs, two mdevs, both running the stock
+   driver, concurrently and repeatedly. Today the second *client in one
+   guest* already wedges (`../../debug/issue/issue-number-2.md`), and the
+   two-guest harness does not exist.
+2. **Independent VRAM sizes.** Each mdev exposes an emulated framebuffer
+   size that need not relate to the physical framebuffer or to another
+   mdev's size: `nv_vgpu_type_info.vram_mb`
+   (`kernel/include/ioctl.h:39-46`), BAR1 sized from it
+   (`kernel/bar_init.c:60-77`), and GSCI/PFB reports derived from the same
+   value (`kernel/gsp_rm_emulation.c:1034,1106-1110`,
+   `kernel/bar_emulation.c:271-310`, `kernel/gsp_emulation.c:743-753`).
+   The mechanism exists per mdev; the *capacity* story does not (§3).
+3. **Multi-tenant inference inside one guest.** A vLLM-style server
+   multiplexes many *logical tenants* (API users, agents, sequences) inside
+   one VM that owns one mdev. VM-level isolation (IOMMU, mdev) is blind to
+   them. This is the workload §4 targets.
+
+### 1.2 Where isolation can exist
+
+Isolation is not one mechanism; it is a stack of layers, each with a
+different owner and a different failure mode. The design must say, for each
+layer, what holds and what does not.
+
+```
+L0 PLATFORM          IOMMU/SMMU: per-VM DMA address spaces        [live]
+   │                 KVM memory + vCPU boundary                    [live]
+L1 TRANSPORT         VFIO mdev: one mediated device per tenant     [live]
+   │                 config/BAR/IRQ mediation, per-device eventfd
+L2 vEP STATE         per-mdev emulation context (BAR shadows,      [live/wip]
+   │                 queues, msgq, interrupt state)                 §2.2
+   │                 per-mdev work-service state                    [design]
+L3 GMMU              per-context page directory (VA space)         [live//design]
+   │                 PTE-level permissions (RO, privilege)          [proven fmt]
+L4 ARBITRATION       channel ownership, runlist/TSG scheduling     [design]
+L5 COMPLETION        semaphore resolution per owner; no cross-      [wip]
+   │                 tenant release apply                           §2.5
+L6 FAULT             fault attribution to the owning tenant;       [wip]
+                      notifier only to that tenant                 §2.6
+L7 APPLICATION       per-tenant/per-agent VA partitioning inside    [research]
+                      a guest (vLLM + agent utility)               §4
+```
+
+The honest summary from the tracked docs applies: **memory isolation between
+VMs is hardware-plus-hypervisor enforced today; engine-level context
+separation and performance isolation are what the project must still build**
+([`09-full-system-picture.md` §6](09-full-system-picture.md)).
+
+### 1.3 Trust boundaries and the principals
+
+| Boundary | Attacker | Capability | Must hold |
+|---|---|---|---|
+| Guest ↔ guest | whole VM | arbitrary device programming, arbitrary guest kernel | no memory, state, interrupt or completion observation across mdevs |
+| Guest ↔ host | whole VM | arbitrary DMA/MMIO against its own device | no host memory access, no module state escape, fail closed |
+| Host user ↔ device | unprivileged host user | access to `/dev/*` as granted | no privilege escalation via our interfaces |
+| Tenant ↔ tenant (in one guest) | guest userspace tenant | arbitrary CUDA kernels, arbitrary block-table edits | a tenant cannot read/write another tenant's KV blocks |
+| Agent ↔ agent (in one tenant) | guest userspace agent | arbitrary CUDA kernels | an agent cannot read/write a sibling agent's KV blocks |
+
+The first three are baseline 1.0 (they follow the threat model in
+[`../05-operations/01-performance-correctness-security.md` §4.1](../05-operations/01-performance-correctness-security.md)).
+The last two are the *multi-tenant inference* boundaries. Note that they
+assume the **guest kernel is trusted**: everything below the application
+runs the stock driver, and the tenant/agent separation is enforced at the
+application's use of the device, not against a malicious guest KMD. A
+malicious guest KMD is the guest↔host boundary, already covered by L0/L1.
+
+### 1.4 Where the code actually stands (the gap this document fills)
+
+The guest-facing plane is per mdev: `nv_vgpu_device.gsp_comm_ctx`
+(`kernel/include/nvidia-vgpu-vfio.h:133`) wraps a per-instance
+`struct gsp_emu_context` (`kernel/gsp_emulation.h:130`) created by
+`gsp_emu_init()` (`kernel/gsp_emulation.c:724`) from
+`nv_vgpu_device_init_work()` (`kernel/main.c:1657`). BAR shadows, message
+queues and MSI-X state are per instance.
+
+The **work/completion service is not**. `gsp_sema_service.c` holds one
+global `static struct sema_svc svc;` (`kernel/gsp_sema_service.c:676`), and:
+
+* the service thread pumps **the first open mdev it finds**
+  (`kernel/gsp_sema_service.c:13296-13308`);
+* the work-submission doorbell carries no owner — `gsp_sema_service_doorbell(u32 token)`
+  (`kernel/gsp_sema_service.c:13456`), called from the VFIO write path with
+  no mdev argument (`kernel/main.c:865`);
+* channel registrations are keyed by the guest-chosen `h_object` alone
+  (`kernel/gsp_sema_service.c:847`, table bounded by `SVC_MAX_CHANNELS 128`
+  at `:122`), and VA spaces by the guest-chosen `h_vaspace` alone
+  (`:988`, `:1019`, bounded by `SVC_MAX_VASPACES 8` at `:309`);
+* the VA→PA learned map and pins are keyed by GPU VA only
+  (`kernel/gsp_sema_service.c:113-130`, `:1045`), with no mdev key;
+* completion counter pools and rank latches are global boot state, as the
+  second-client wedge demonstrates: the old client's channel/rank/ring state
+  makes the host's completion path go quiet for the new client
+  (`../../debug/issue/issue-number-2.md` §Root cause).
+
+Two guests can trivially choose the *same* handles and GPU VAs (live:
+channel `0xcaf000d4`, `../../debug/issue/issue-number-2.md` §Evidence); keyed by
+value alone, those registrations and translations collide — the exact class
+that already breaks the second client in one guest. **Multi-mdev isolation
+is therefore first a service-state-ownership problem, and only second a GMMU
+problem.** §2 fixes the ownership; §4 builds the GMMU policy on it.
+
+### 1.5 Vocabulary
+
+**Tenant** — isolation principal (`tenantId`): a VM/mdev at L0–L2, a logical
+serving principal inside a guest in §4. **Agent** — finest principal inside
+a tenant (`agentId`): a vLLM engine/worker, a CUDA context, a process.
+**vEP** — virtual execution plane: the per-instance mediator (guest
+emulation + work execution) for one mdev; see
+[`13-vexecution-plane.md`](13-vexecution-plane.md). **KV block** —
+paged-attention cache page (vLLM `paged_attention_v2`: fixed-size blocks,
+one `block_tables` entry per logical block per sequence); a GPU allocation
+range, not a hardware object.
+
+---
+
+## 2. Security and isolation design for multi-mdev (vEP terms)
+
+### 2.1 Address-space separation
+
+**Rule S1.** Every host-side table that describes guest state is keyed by
+`(mdev, guest-name)` — handle, GPU VA, doorbell token, counter slot — never
+by the guest-chosen value alone.
+
+What exists and what changes:
+
+| State | Today | Required |
+|---|---|---|
+| emulation context | per mdev `[live]` (`nvidia-vgpu-vfio.h:133`, `gsp_emulation.h:130`) | unchanged |
+| `svc.chan[]` | keyed `h_object` (`gsp_sema_service.c:847`) | key `(vgpu, h_object)`; per-mdev lifetime reset |
+| `svc.vas[]` | keyed `h_vaspace` (`:988`) | key `(vgpu, h_vaspace)` |
+| learned VA→PA map / pins | keyed GPU VA (`:113-130`) | key `(vgpu, VA)`; per-mdev epoch |
+| doorbell token | ownerless (`:13456`, `main.c:865`) | `(vgpu, token)`; route to the owner's pump |
+| completion counters | global boot state | per mdev, reset on session/rebind |
+| pump | first open mdev (`:13296-13308`) | per-mdev pump (or round-robin over all open mdevs) |
+
+The GMMU half of address-space separation is the page directory: each VA
+space has one, published by the guest through `SET_PAGE_DIRECTORY` with its
+`hVASpace` and `chId` (`kernel/gsp_rm_emulation.c:3310-3314,3366-3372` for
+the recording path; the format is documented at `:3307-3319`). The vEP
+records the directory per owner and must translate only within the owning
+mdev's tables. `[design]`
+
+### 2.2 Per-mdev channel ownership
+
+**Rule S2.** A channel belongs to exactly one mdev for its whole life, and
+its engine work, doorbell token, error notifier and completion counters are
+resolved through the owner only.
+
+The channel descriptor today carries GPFIFO VA, USERD, instance and
+notifier pages (`kernel/gsp_sema_service.h:22-33`); the registration arrives
+from the guest's `GSP_RM_ALLOC` handling while the emulation context is
+known (`ctx->vgpu`), so the owner is available at registration time — it is
+simply not stored. The fix is structural (add the owner to `svc_channel`
+and to every lookup), not algorithmic.
+
+Engine arbitration is where ownership becomes policy: channels of the same
+engine from different mdevs must be scheduled by our policy on top of the
+real GSP scheduler ([`10-interaction-and-scheduling.md` §7](10-interaction-and-scheduling.md)).
+Today the single real GSP session and its global `gsp_phys_rm_lock` serialize
+all real firmware traffic (`11-physical-backend-guest-wiring.md` §1.3), so
+one tenant's slow RPC blocks every other tenant's work — head-of-line
+blocking, addressed in §2.7.
+
+### 2.3 DMA / IOMMU interplay
+
+Ground rules, unchanged by multi-tenancy:
+
+* Guest memory is reached only through `vfio_dma_rw(&vgpu->vfio_dev, ...)`
+  (`kernel/gsp_sema_service.c:754-762`, `kernel/gsp_rm_emulation.c:27-55`),
+  i.e. through the hypervisor's mapping for **that** mdev's VFIO container.
+  A GPA from guest A cannot be resolved through guest B's device — the
+  boundary is enforced below us, and we never weaken it
+  ([`../05-operations/01-performance-correctness-security.md` §4.2](../05-operations/01-performance-correctness-security.md)).
+* The emulated vidmem (BAR1) is a private host-RAM buffer per mdev
+  (`kernel/bar_init.c:60-77`); FB-aperture accesses stay inside the owning
+  mdev's buffer (`svc_read_ap`/`svc_write_ap` bounds-check
+  `vgpu->bars[1]`, `kernel/gsp_sema_service.c:786-814`).
+* Physical-backend caveat (real VRAM path): on the reference host the
+  physical GPU's IOMMU group is an identity domain and the mediated device
+  registers as an emulated-IOMMU VFIO device, so guest GPAs are translated
+  through the QEMU shared-RAM file rather than a kernel `iommu_domain`
+  (`11-physical-backend-guest-wiring.md` §P6a). This translation is
+  per-mdev-gated but is **not** an IOMMU guarantee; with multiple mdevs on
+  one backend it must be audited per mdev, and the active-mdev gate must
+  never select another mdev's file. `[design]`
+* When the backend owns the GPU, any host-translated address that reaches a
+  real engine must come from the owning mdev's resolved set, never from a
+  global table (`11-physical-backend-guest-wiring.md` §6.4).
+
+### 2.4 Data-path separation
+
+* BAR0 shadow, BAR1 vidmem shadow, BAR3/instance window are per mdev
+  (`kernel/bar_init.c:59-77`; create at `kernel/main.c:554`; cited in
+  `11-physical-backend-guest-wiring.md` §0.1 `[live]`). Two mdevs'
+  shadows are separate allocations; the invariant to test is that no
+  handler ever mixes the two base pointers (handlers take
+  `struct gsp_emu_context *ctx` and dereference `ctx->vgpu`, e.g.
+  `kernel/gsp_rm_emulation.c:2829-2834`; the tests in §2.8 prove it).
+* Real-VRAM forwarding already isolates *engines* from guest memory by
+  submitting through module-built scratch page tables that map
+  already-resolved host pages and never reuse a fetch line within a boot
+  (`11-physical-backend-guest-wiring.md` §P6b). With multiple mdevs, scratch
+  pools must be per mdev: capacity is a per-boot resource (512 pairs at
+  16 PTEs/line in the current design) and shares a hardware resource with
+  every tenant.
+* Guest page tables are **guest-owned** (the driver builds them; we walk
+  them — `kernel/gsp_sema_service.c:1052-1072`), so a malicious guest can
+  retarget its own PTEs at any time: fine for guest↔host (the IOMMU caps the
+  damage to the guest's own memory), but the fundamental limit of intra-guest
+  enforcement in §4 — *policy can be only as strong as page-table ownership*.
+
+### 2.5 Completion isolation
+
+**Rule S3.** A decoded semaphore release is applied only in the owning
+mdev's address space, and a completion event is posted only to the owning
+mdev's event object.
+
+Today the release path resolves the target VA through *global* tables
+(`svc_resolve_va`, `svc_sem_write`; `kernel/gsp_sema_service.c:2098,1514`)
+and posts through `gsp_rm_post_channel_event()` on the emulation context
+(`kernel/gsp_rm_emulation.c:602`). The first is the cross-tenant hazard: a
+release record decoded from guest A's push buffer could resolve through a
+pin learned from guest B if the VA collides. The exactly-once rules must be
+preserved per owner (a copy runs at most once per GPFIFO entry; a virtual
+operand is translated per 4 KB page; a consumed slot never runs later —
+[`06-host-module-and-hybrid-model.md` §4.4](06-host-module-and-hybrid-model.md)).
+
+### 2.6 Fault containment
+
+**Rule S4.** A fault (engine fault, malformed RPC, translation failure,
+timeout) is attributed to its owner, delivered only to the owner's notifier
+and event object, and must not stall or corrupt another tenant.
+
+The machinery exists per channel (error notifier slots in the channel
+descriptor, `kernel/gsp_sema_service.h:31`) and for forwarded engines
+(fault observation/relay default-off,
+`11-physical-backend-guest-wiring.md` §P6c C3), and the malformed-RPC path
+is fail-closed. What is missing is the owner key at the relay decision, and
+a per-tenant fault-injection test (§2.8). A wedged server is a
+*module-global* fail-safe today (`08-physical-gpu-integration.md` §7), which
+is correct for one backend session but must never be reachable from a
+guest-triggered path: guest input may abort the guest, never the service.
+
+### 2.7 Time-slicing risks
+
+With one physical GPU, tenants share engines by time slicing. Known risks:
+
+* **Head-of-line blocking:** the single real GSP session + `gsp_phys_rm_lock`
+  serializes backend RPCs; a tenant that floods controls starves the others.
+  Mitigation design: per-mdev backend clients (already planned,
+  [`09-full-system-picture.md` §5](09-full-system-picture.md)), bounded
+  per-tenant RPC rates in the vEP, and per-tenant accounting.
+* **Slice inversion:** guest A's long-running engine work delays B's
+  completion relay; the relay path must stay bounded and never poll a
+  hardware queue while holding a lock a second tenant needs
+  (`11-physical-backend-guest-wiring.md` §1.3 lock order).
+* **Starvation:** without a policy, first-come submission can starve a
+  tenant. The policy is planned
+  ([`10-interaction-and-scheduling.md` §7](10-interaction-and-scheduling.md));
+  until it exists, the honest statement is "fairness is unmeasured".
+
+### 2.8 Side channels (public knowledge only)
+
+We make no original side-channel claims and have no measurements. The
+classes that apply to any shared-GPU time-slicing design, from public
+literature, are: memory-bandwidth contention observable through timing;
+L2/LLC state effects; scheduler/slice timing; and interrupt-timing
+observation. The project's stated bar is that an adjacent tenant must not
+observe timing/state/interrupts *beyond policy*
+([`../05-operations/01-performance-correctness-security.md` §4.1](../05-operations/01-performance-correctness-security.md)):
+interrupts are already per-instance, timing policy must be explicit and
+documented, and measurement is a 1.0-era task (rung 5). Until measured,
+treat side channels as **unmitigated but bounded by per-instance state
+separation**, and never claim isolation stronger than the ladder proves.
+
+### 2.9 Validation and test plan for the 1.0 "two guests" gate
+
+The gate: two QEMU guests, two mdevs, concurrent, stock driver; every test
+captures evidence (dmesg, counters) and passes at least 3 consecutive runs.
+
+| # | Test | Pass criterion |
+|---|---|---|
+| 2G-01 | Boot both guests concurrently on two mdevs; `nvidia-smi`; CUDA driver-API smoke | both green, payload round-trip exact; no cross errors in dmesg |
+| 2G-02 | Distinct VRAM types (e.g. 512M and 1G) on the two mdevs | each guest's `nvidia-smi`/GSCI reports its own size; BAR1 sizes differ and match |
+| 2G-03 | Handle/VA collision: force both guests to allocate the same RM handles and GPU VAs (the `0xcaf000d4` class) | both clients progress; no release lands in the wrong guest (verify by canary words around each guest's semaphores) |
+| 2G-04 | Doorbell cross-talk: ring guest A's tokens while B is idle | B's channels/state unchanged; B's poll loop untouched; A completes |
+| 2G-05 | Completion ownership: craft a release VA that exists in both guests' address spaces | resolves only in the owner's memory; the other guest's canary unchanged |
+| 2G-06 | Fault injection: malformed RPC, bad queue link, engine-fault relay from A | A fails closed (documented error); B unaffected; no module-global stall |
+| 2G-07 | Teardown: remove mdev B while A runs; rmmod after both stop | A continues; teardown order preserved; module unload clean |
+| 2G-08 | Stress: N≥20 concurrent start/stop cycles; guest reboots; module reloads | no leaked per-mdev state; no cross-tenant counters; clean dmesg |
+| 2G-09 | Sequencing: the ISSUE-0002 acceptance (3 sequential clients in one boot) plus 2 concurrent processes | green; per-client reset proven (issue-number-2 §Validation plan) |
+
+Known blockers before the gate can run: the per-client/per-channel reset
+(`../../debug/issue/issue-number-2.md` §Fix plan items 1–2) and the per-mdev
+service-state split (§2.1 Rule S1). The test harness (two VMs; the current
+tools assume one, `../../scripts/vgpu_vm.py`) is itself a deliverable.
+
+### 2.10 Fail-closed rules
+
+Isolation properties that cannot be established (missing IOMMU container,
+DMA mapping failure, inconsistent owner state, unkeyed table entry) stop the
+affected mdev and say why; nothing silently degrades
+([`../05-operations/01-performance-correctness-security.md` §4.2](../05-operations/01-performance-correctness-security.md)).
+Malformed or stale queue elements, PTE chains and release records are
+rejected without disturbing the queue, with strict sequence matching
+([`10-interaction-and-scheduling.md` §3](10-interaction-and-scheduling.md)).
+No forwarding seam programs guest-derived page tables into a real MMU until
+the population path is proven end to end
+(`11-physical-backend-guest-wiring.md` §6.4). A cross-owner table lookup is
+a programming error: log once and refuse, never "resolve anyway".
+
+---
+
+## 3. VRAM larger than physical, by scheduling and virtualization
+
+### 3.1 What "VRAM size" means per mdev today
+
+Each mdev derives a consistent set of numbers from
+`nv_vgpu_fb_emul_size(vgpu)` (`kernel/include/nvidia-vgpu-vfio.h:232`):
+BAR1 size (`kernel/bar_init.c:68`), the `LOCAL_MEMORY_RANGE` register
+encoding (`kernel/bar_emulation.c:271-310`), the GSCI `fb_length` and
+`frtsOffset` (`kernel/gsp_rm_emulation.c:1106-1110,1177`) and the WPR2
+address (`kernel/gsp_emulation.c:743-753`). The mdev type advertises the
+same size to management tools (`framebuffer=%uM`,
+`kernel/main.c:116-132`). The sizes available are dictated by the types the
+host driver registers (`kernel/include/ioctl.h:39-46`,
+`kernel/nv_vgpu_register_type_info` callers).
+
+Two consequences: **presenting a size is solved** (two mdevs can present
+512 MiB and 2 GiB simultaneously; nothing in the guest path forces them to
+match the physical framebuffer), but **backing that size is not** — in
+emulation mode each mdev's "VRAM" is a private host-RAM `vzalloc`
+(`kernel/bar_init.c:70-77`), so the aggregate is bounded by host RAM; with
+real VRAM backing (`08-physical-gpu-integration.md` §4.6) it is bounded by
+physical VRAM. Either way `sum(vram_mb)` may exceed capacity — the
+oversubscription problem.
+
+### 3.2 Design: present size independent of physical size
+
+**Principle.** Guest-visible VRAM size is a *contract*; the resident set is
+an *implementation detail* — decouple them with tiered, revocable backing:
+
+```
+            guest-visible VRAM (mdev A, e.g. 2 GiB)      guest-visible (mdev B, 1 GiB)
+                     │                                        │
+              ┌──────┴──────────────────────────────┐  ┌──────┴────────────┐
+              │  vEP per-mdev residency manager     │  │   ...             │
+              │  pinned set | evictable set         │  │                   │
+              └──────┬──────────────────────────────┘  └───────────────────┘
+                     │ page-in/out (4 KiB-2 MiB)
+        ┌────────────┴───────────┐        ┌───────────────────────────────┐
+        │ real VRAM (hot blocks) │        │ sysmem staging (cold blocks)  │
+        └────────────────────────┘        └───────────────────────────────┘
+```
+
+Techniques, each with its current status:
+
+1. **Sparse allocation (allocate-on-touch).** The guest-visible range exists
+   before it is backed; backing is populated on first write (emulation: RAM
+   page; backend: real VRAM object). Today the whole BAR1 is
+   `vzalloc`'d up front (`kernel/bar_init.c:70-77`) `[wip]`; sparse backing
+   is `[design]`.
+2. **VRAM ↔ sysmem paging.** Cold blocks move to guest sysmem (or host
+   staging) and their PTEs are rewritten to the sysmem aperture; hot blocks
+   move back. The GMMU already distinguishes apertures
+   (`svc_pte_aperture`, `kernel/gsp_sema_service.c:1101-1108`) and the guest
+   uses `ADDR_SYSMEM=1`/`ADDR_FBMEM=2` descriptors
+   ([`05-gmmu-amap-hal-gr.md` §1.1](05-gmmu-amap-hal-gr.md)) `[design]`.
+3. **Replayable pushes.** A push whose instruction stream is re-executed
+   must be idempotent or suppressed; the host already has the machinery:
+   per-entry identity checks, per-operation done bitmaps, consumed-slot
+   marks, and the deferred-release hold that waits for hardware completion
+   before applying releases exactly once
+   ([`06-host-module-and-hybrid-model.md` §4.4](06-host-module-and-hybrid-model.md),
+   `11-physical-backend-guest-wiring.md` §P6b wave 1.5) `[wip]`.
+   Paging reuses this primitive: an operation that references an evicted
+   block is held until the block is resident.
+4. **GMMU invalidations.** Rewriting a PTE is not enough: the engine caches
+   PTE fetches (observed 128-byte granularity) and there is no invalidate
+   path in the forwarding code today; a runtime PTE write into an
+   already-walked line was accepted but never completed
+   (`11-physical-backend-guest-wiring.md` §P6b "PTE fetch-line rule").
+   The guest's own `MEMSYS_L2_INVALIDATE_EVICT` control is acknowledged
+   without effect in emulation (`kernel/gsp_rm_emulation.c:2519-2532`).
+   A real invalidate path is a prerequisite for paging `[design]`.
+5. **UVM fault replay.** Guest UVM already handles memory faults and
+   re-registration; the host can signal "not resident" and let the guest
+   replay the access. The guest-visible mechanism is the channel
+   event/notifier path (`gsp_rm_post_channel_event`,
+   `kernel/gsp_rm_emulation.c:602`); whether every access class can be
+   replayed this way is `[hypothesis]` — it needs a live experiment.
+6. **Migration hooks.** The module already has save/restore state sections
+   (`kernel/migration.c:85+`); residency metadata (which blocks are
+   resident, tags, ownership) must be part of the migrated state for
+   correctness across live migration `[design]`.
+
+### 3.3 Tradeoffs
+
+| Effect | Cause | Control |
+|---|---|---|
+| Latency spike on page-in | block fetch from sysmem to VRAM | bound resident-set misses per slice; prefetch on sequential access (paged attention accesses KV blocks sequentially per sequence) |
+| Throughput collapse under thrash | working set > capacity | admission control: refuse the allocation that would guarantee thrash; per-tenant quotas |
+| Completion jitter | held pushes waiting for residency | bound the hold; on expiry fail closed per operation (existing bounded-hold pattern, `gsp_sema_service.h:220-236`) |
+| Fairness loss | one tenant's evictions victimize another | per-tenant residency targets, self-eviction first (see §3.5) |
+
+These are unmeasured; the numbers must come from the benchmark harness
+([`../05-operations/01-performance-correctness-security.md` §3](../05-operations/01-performance-correctness-security.md)).
+Do not quote an overhead figure from this section.
+
+### 3.4 Front-running risks and correctness invariants
+
+**Front-running.** A tenant that can force reclaim of *another tenant's*
+pages gains two advantages: it can degrade a competitor, and it can race to
+observe reused memory if reclaim does not scrub. Rule: reclaim only within
+the requesting tenant's own residency budget unless a global-capacity policy
+explicitly and observably selects a victim (accounted, rate-limited, and
+scrubbed before reuse).
+
+**Invariants (must hold for any paging implementation).**
+
+* **I1 — No live PTE to reclaimed memory.** A block may be reallocated only
+  after every PTE that references it has been revoked and the revocation is
+  visible to all engines.
+* **I2 — Revoke implies quiesce.** No engine work may be in flight on a
+  block at revoke; the deferred-release hold + channel drain is the
+  primitive (`11-physical-backend-guest-wiring.md` §P6b).
+* **I3 — Exactly-once completion.** Paging must not duplicate or lose a
+  release; the existing replay-suppression rules apply per owner
+  ([`06-host-module-and-hybrid-model.md` §4.4](06-host-module-and-hybrid-model.md)).
+* **I4 — Owner-stable residency.** A block's backing is owned by one mdev
+  (or one intra-guest tenant, under §4); no shared physical page between
+  owners without an explicit, tagged share record.
+* **I5 — Scrub before reuse.** Reclaimed memory is zeroed (or poison-filled)
+  before it can be mapped into any other owner.
+* **I6 — Migration carries residency.** Save/restore includes the residency
+  map and tags, or the restored device must cold-start the cache.
+
+### 3.5 What vEP must add
+
+* **Per-mdev accounting:** resident/pinned bytes, fault-in counts, hold
+  timeouts — the counters in `struct sema_svc`
+  (`kernel/gsp_sema_service.c:439-676`) made per owner.
+* **Reclaim signal:** a guest→vEP hint that a range is evictable (vLLM knows
+  when a KV block is freed): an allocation/free observed at RM-object level
+  in the stock-compatible design, or an explicit control in the tagged
+  design (§4.5).
+* **QoS policy:** per-mdev weights/slices on top of the GSP scheduler
+  ([`10-interaction-and-scheduling.md` §7](10-interaction-and-scheduling.md)),
+  with measured properties. `[design]`
+* **Capacity admission:** enforce the oversubscription ratio at create and
+  large-allocation time; refuse (fail closed) rather than promise capacity.
+
+---
+
+## 4. Custom GMMU protections for paged-attention multi-tenancy
+
+> This is the core ask: given a tenant ID and an agent ID, protect a
+> paged-attention KV cache shared by many tenants inside one guest. The
+> design below is complete on paper across the full stack. Stages M6–M8
+> are **post-1.0 research**: dual-gated, duplicate-not-replace, never
+> required by the baseline. Stages M1–M5 belong to the 1.0 programme.
+
+### 4.1 The workload, precisely
+
+Public knowledge of `paged_attention_v2`: the KV cache is a pool of
+fixed-size **blocks**; each sequence has a **block table** (a GPU tensor of
+block indices) mapping its logical positions to physical blocks. Decode
+kernels gather K/V from the blocks named by the table. A vLLM server
+multiplexes many sequences (many logical tenants) in one continuous batch
+inside one process and, typically, one CUDA context. Nothing in this
+pipeline is hardware-protected: any kernel in the context can dereference
+any block table entry, and a buggy or malicious sequence implementation can
+read every other tenant's KV cache. With prefix caching, blocks are even
+*shared* between sequences (copy-on-write), which is a feature to preserve,
+not eliminate.
+
+The assumed identities: `tenantId` identifies the serving principal;
+`agentId` identifies the finest unit inside a tenant (a worker/context; in
+a single-process vLLM, an engine or an attention backend instance). Both are
+provided by the orchestration layer; provisioning them is out of scope.
+
+### 4.2 Hardware GMMU primitives: proven vs hypothesis
+
+What the Turing (TU104) VER2 format and our live code establish (`docs/NN`
+below resolves to `../../debug/risc-v-gsp-rm-vm/docs/NN-*.md`):
+
+| Primitive | Status | Evidence |
+|---|---|---|
+| 4-level page tree; 49-bit VA; 4 KiB/64 KiB/2 MiB pages; dual 16-byte PDE0 | **proven** | tracked decoder `../../debug/risc-v-gsp-rm-vm/docs/43-GMMU-PTE-DECODER.md` §6; `kernel/gsp_sema_service.c:1058-1072` |
+| VER2 leaf PTE fields incl. `read_only` (bit 6), `privilege` (5), `encrypted` (4), `atomic_disable` (7), `address_sys/vid`, `kind` | **proven format** | `docs/43` §3 |
+| Separate page directory per VA space, published per channel (`SET_PAGE_DIRECTORY` with `hVASpace`/`chId`) | **proven** | `kernel/gsp_rm_emulation.c:3310-3314,3366-3372`; live walks `kernel/gsp_sema_service.c:1174-1312` |
+| Directory-level apertures and big-page leaves | **proven** | `docs/43` §3–§6; `svc_pde_aperture`, `kernel/gsp_sema_service.c:1110-1117` |
+| **No tenant/ACL/PASID field in the VER2 PTE or PDE** | **proven absence in the format** | the full field list in `docs/43` §3–§5 has no such field; `peer_id` is a fabric peer address selector, not a security domain |
+| Read-only PTE *enforcement* (the engine refuses a write through an RO leaf) | **hypothesis** | the format defines the bit; we have never exercised an RO-protected leaf on hardware |
+| Encryption-domain isolation (CCS/confidential computing) usable as a tenant boundary | **hypothesis** | format has an `encrypted` bit; CCS object model and semantics are unstudied for us |
+| MMU invalidate semantics fine-grained enough for per-block revocation | **hypothesis** | no invalidate path exercised; see `11-physical-backend-guest-wiring.md` §P6b |
+
+**The decisive design fact:** on this generation, **the only proven
+hardware isolation primitive is the page directory** (per-context page
+tables). Per-PTE ACLs are research-grade and cannot be assumed. Therefore
+the design must place every security-relevant decision in the *topology*
+(which directory a page appears in, which channel may name it) plus
+*software validation and audit* in the vEP. The `read_only` bit is a proven
+format field and the right mechanism for *sharing without write access*
+(e.g., shared prefix blocks), but its enforcement needs the M7 experiment
+before we depend on it.
+
+### 4.3 The full stack
+
+The requested chain, end to end:
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│ GUEST                                                                       │
+│                                                                             │
+│  extended vLLM (serving engine)                                             │
+│    ├─ per-tenant sequences + block tables (unchanged algorithms)            │
+│    ├─ agent-aware KV allocator: blocks come from per-agent VA pools         │
+│    └─ calls oiv-agentd for identity + sharing decisions                     │
+│           │ Unix socket (guest-local)                                       │
+│  oiv-agentd (NEW guest utility; application-level, no driver patch in A)    │
+│    ├─ resolves (tenantId, agentId) -> VA pool + tag records                 │
+│    ├─ Design A: publishes tag records to a mailbox in device memory         │
+│    └─ Design B: issues new RM controls through the guest KMD               │
+│           │ CUDA driver API (stock)      │ RM control (B)                   │
+│  guest KMD (stock nvidia.ko; unmodified in A, extended in B [research])     │
+│           │ GSP-RM RPC / MMIO                                               │
+├───────────┼─────────────────────────────────────────────────────────────────┤
+│ vEP HOST (vgsp.ko)                                                          │
+│    GSP-RM emulation: controls, objects, page-directory records              │
+│    tag registry (per mdev): tenant/agent -> VA ranges, access, shares       │
+│    page-table policy: owner checks at SET_PAGE_DIRECTORY / map / submit     │
+│    channel binding: channel -> (tenant, agent); arbitration + completion    │
+│    fault attribution: fault -> channel -> tenant; owner-only notifier       │
+│           │ real page tables / forwarded work                               │
+│ GPU: GMMU page directories (topology) + PTE bits (read_only, kind)          │
+└─────────────────────────────────────────────────────────────────────────────┘
+```
+
+### 4.4 Design A — VA-subtree partitioning (no guest KMD change)
+
+This is the **stock-compatible** design and the one that can be prototyped
+inside the 1.0 programme. It relies on one idea: **the VA space is the
+security domain, and a VA is self-describing because its prefix encodes its
+owner.**
+
+* Extended vLLM reserves a disjoint VA range per agent: `cuMemAddressReserve`
+  (stock CUDA VMM API) at a base derived from `(tenantId, agentId)`, e.g.
+  a deterministic partition of the GPU VA space into tenant subtrees and
+  agent subtrees. KV blocks are `cuMemCreate`d and mapped only inside the
+  owner's range.
+* `oiv-agentd` publishes the binding records (which `(tenantId, agentId)`
+  owns which VA range, with what access) to a **tag mailbox**: a device
+  allocation whose content the vEP can read. In emulation mode this works
+  because guest vidmem *is* host RAM (`kernel/bar_init.c:68-77`; FB
+  aperture reads are direct, `svc_read_ap` `kernel/gsp_sema_service.c:786-799`),
+  so the vEP can scan the mailbox page through the owner's shadow. A fixed
+  mailbox offset (requested from the extension at init) keeps discovery
+  trivial.
+* vEP policy, enforced when walking/populating page tables and at
+  work-submission time:
+  1. a page directory is owned by the mdev (L3) and, under this design, by
+     a `(tenantId, agentId)` (L7);
+  2. a PTE whose VA is in tenant T's subtree may only point at memory
+     tagged T (or shared with T);
+  3. a channel bound to agent A may only execute work referencing VAs in
+     A's subtree (validation at decode/submit, using the existing
+     owner-aware claim of §2.2);
+  4. violations are refused and attributed to the owning channel.
+* Sharing for prefix caching: a shared block lives in the *owner's* range
+  and is exposed to the borrower through a new PTE in the borrower's
+  directory with `read_only` set (proven format field) — or, until M7
+  proves RO enforcement, not exposed at all and prefix sharing is
+  downgraded to per-agent copies.
+
+Limits, stated honestly:
+
+* It does not protect against a malicious guest KMD (the KMD builds the
+  page tables; it can lie about prefixes). It protects tenants from each
+  other *within* the tenant's trust model, which is the vLLM threat model.
+* Emulation-mode mailbox discovery does not survive real VRAM backing
+  (the host cannot read real VRAM as RAM); the real-VRAM version needs a
+  protocol path (Design B) or a host-readable staging mailbox.
+* It requires extended vLLM (not stock vLLM) — an application change, not
+  a driver change.
+
+### 4.5 Design B — explicit tenant/agent RM controls `[research]`
+
+The explicit design: new vGPU-private RM controls flow through the guest
+KMD to the vEP; the vEP stores tags, enforces policy, and can build real
+page tables from them. This requires a guest KMD that knows the controls;
+therefore it is **post-1.0** (it modifies the guest, which the baseline
+guest rule forbids, [`../06-portability/hardware-software-matrix.md` §3](../06-portability/hardware-software-matrix.md)),
+dual-gated, and never a 1.0 dependency.
+
+Proposed controls (vGPU-private control range; the proven vGPU-specific
+example is `0x90f10106`, `COPY_SERVER_RESERVED_PDES`,
+`kernel/gsp_rm_emulation.c:2266`):
+
+| Control | ID (proposed) | Semantics |
+|---|---|---|
+| `NV90F1_CTRL_CMD_VGPU_TENANT_BIND` | `0x90f10120` | bind the calling client to a `(tenantId, agentId)`; idempotent join; optional exclusive |
+| `NV90F1_CTRL_CMD_VGPU_TENANT_TAG_RANGE` | `0x90f10121` | tag a GPU VA range with owner + access + volatility |
+| `NV90F1_CTRL_CMD_VGPU_TENANT_UNTAG_RANGE` | `0x90f10122` | remove a tag; range becomes untouchable by non-owners |
+| `NV90F1_CTRL_CMD_VGPU_TENANT_SHARE` | `0x90f10123` | grant a borrower a (revocable) view of an owner range |
+| `NV90F1_CTRL_CMD_VGPU_AGENT_CHANNEL_BIND` | `0x90f10124` | bind an RM channel to an agent for arbitration/completion attribution |
+| `NV90F1_CTRL_CMD_VGPU_TAG_QUERY` | `0x90f10125` | read-only: effective identity, enforcement mode, range tags |
+
+Struct sketches (wire types follow the driver's `NvU32`/`NvU64`/`NvHandle`
+conventions; all little-endian, 8-byte aligned):
+
+```c
+/* 0x90f10120 */
+typedef struct NV_VGPU_TENANT_BIND_PARAMS {
+    NvU64 tenantId;      /* stable, VM-scoped; 0 invalid */
+    NvU64 agentId;       /* 0 = tenant's default agent */
+    NvU32 flags;         /* CREATE 0x1 | JOIN 0x2 | EXCLUSIVE 0x4 */
+    NvU32 abiVersion;    /* 1 */
+    NvU8  name[32];      /* diagnostic only */
+} NV_VGPU_TENANT_BIND_PARAMS;
+
+/* 0x90f10121 / 0x90f10122 */
+typedef struct NV_VGPU_TENANT_TAG_RANGE_PARAMS {
+    NvU64 tenantId;
+    NvU64 agentId;
+    NvU64 gpuVa;         /* page aligned */
+    NvU64 size;
+    NvU32 access;        /* READ 0x1 | WRITE 0x2 (0 invalid) */
+    NvU32 flags;         /* SHARED 0x1 | EVICTABLE 0x2 | PINNED 0x4 |
+                            KV_BLOCK 0x8 | BLOCK_TABLE 0x10 */
+    NvU64 tagCookie;     /* opaque allocator cookie; echoed in faults */
+} NV_VGPU_TENANT_TAG_RANGE_PARAMS;
+
+/* 0x90f10123 */
+typedef struct NV_VGPU_TENANT_SHARE_PARAMS {
+    NvU64 ownerTenantId, ownerAgentId;
+    NvU64 borrowerTenantId, borrowerAgentId;
+    NvU64 gpuVa;         /* range start as tagged by the owner */
+    NvU64 size;
+    NvU32 access;        /* borrower view: READ | READ|WRITE */
+    NvU32 flags;         /* REVOCABLE 0x1 (default) | COW 0x2 */
+} NV_VGPU_TENANT_SHARE_PARAMS;
+
+/* 0x90f10124 */
+typedef struct NV_VGPU_AGENT_CHANNEL_BIND_PARAMS {
+    NvHandle hChannel;
+    NvU64 tenantId, agentId;
+    NvU32 flags;         /* EXCLUSIVE 0x1 */
+    NvU32 abiVersion;    /* 1 */
+} NV_VGPU_AGENT_CHANNEL_BIND_PARAMS;
+```
+
+Design A's mailbox record (the stock-compatible carrier):
+
+```c
+typedef struct NV_VGPU_TAG_MAILBOX_RECORD {
+    NvU32 magic;         /* 'OIV1' = 0x4f495631 */
+    NvU32 recordType;    /* 1 bind, 2 tag, 3 untag, 4 share */
+    NvU64 seq;           /* producer sequence; consumed strictly in order */
+    NvU64 tenantId, agentId;
+    NvU64 gpuVa, size;
+    NvU32 access, flags;
+    NvU64 tagCookie;
+    NvU32 crc32;         /* over the record with crc32 field = 0 */
+} NV_VGPU_TAG_MAILBOX_RECORD;
+```
+
+Message-semantics rules: controls are synchronous (reply = status/params);
+tag records are monotonic (later record for a range wins; untag terminal);
+shares are revocable, revocation delivered to the borrower's event object;
+every mutating control is forwarded to the vEP, whose reply is authoritative
+— not the echo/zeroed-success fallback used for unknown IDs today
+(`kernel/gsp_rm_emulation.c:2829`; `11-physical-backend-guest-wiring.md`
+§2.1 #7, #13).
+
+### 4.6 Where enforcement lives
+
+| Layer | Mechanism | Strength |
+|---|---|---|
+| Guest application (vLLM + agentd) | VA-pool discipline, block-table construction | advisory; defeated by a buggy kernel |
+| Guest KMD (Design B) | control validation, identity binding, forwarding | local; guest-trusted |
+| vEP tag registry + policy | owner checks at directory record / map / submit / completion | **the enforcement point** |
+| GMMU topology | separate page directory (+ optional sub-directory per agent) | hardware-enforced for context separation |
+| GMMU PTE bits | `read_only` for shares; `kind` for memory type | hardware-enforced *if* M7 confirms |
+| Completion/fault paths | owner-keyed resolution and attribution | software, load-bearing |
+| Audit | periodic owner-vs-PTE audit; canary pages | detection; catches policy escapes |
+
+### 4.7 The three granularities
+
+#### 4.7.1 Per-KV-block (page/PTE level)
+
+*Tag every block; enforce at map time; share with read-only where hardware
+allows.*
+
+| Aspect | Assessment |
+|---|---|
+| Benefits | finest sharing (prefix cache reuse across agents); per-block revoke for eviction; fault granularity is the block |
+| Drawbacks | per-block metadata and per-map validation cost; vLLM allocates blocks in bursts — validation must be batched or the map path becomes the bottleneck; sharing needs RO enforcement (M7) or degrades to copies |
+| Performance | map-time cost per block (currently an emulated RPC is tens of µs — `10-interaction-and-scheduling.md` §6 — and tagging must not add a round trip per block); no extra TLB misses if topology is unchanged, but per-block VA fragmentation reduces big-page opportunities, which *does* cost TLB reach |
+| Implementation cost | high: vLLM allocator change, agentd, mailbox/controls, vEP registry, audit |
+| Portability | the *concept* is portable; the enforcement is software+topology, because no PTE tenant field exists (proven absence, §4.2); `read_only` mapping must be re-derived per generation |
+| Failure modes | stale tag after reallocation (block freed, VA reused); share revoked but a borrower PTE survives (no invalidate, I1 violation); tag/PTE divergence after migration; audit misses a crafted PTE |
+
+#### 4.7.2 Per-tenant (sub-address-space: separate channel + page directory)
+
+*One tenant = one page directory + at least one channel; agents inside a
+tenant share the directory (and are separated only by VA-subtree convention
+unless per-agent directories are added).*
+
+| Aspect | Assessment |
+|---|---|
+| Benefits | **the only proven hardware lever** (§4.2); low steady-state overhead (matches the driver's own per-context model); naturally portable; fault and completion attribution are trivial (directory owner = tenant) |
+| Drawbacks | no fine-grained sharing; a tenant is a scheduling unit (context switches); TLB working set per tenant; intra-tenant agents are NOT isolated |
+| Performance | launch path unchanged; extra contexts cost context-switch and TLB refill; no per-block map overhead |
+| Implementation cost | medium: owner keys in the vEP (§2.1–2.6) plus per-mdev backend clients; Design A needs the allocator/agentd work only for the VA pools |
+| Portability | high: object model and directory-per-context survive driver-version and (probably) generation changes; exact PTE encoding does not matter |
+| Failure modes | directory reuse on handle collision (today's bug class); a page accidentally mapped into two directories; one tenant's RPC wedge stalls others (single backend session); eviction invalidating a sibling's PTE without quiesce |
+
+#### 4.7.3 Per-agent (finest grouping inside a tenant)
+
+*One agent = one channel; optionally one sub-directory per agent within the
+tenant directory.*
+
+| Aspect | Assessment |
+|---|---|
+| Benefits | fault containment per agent (a crashed agent context does not kill siblings); QoS per agent; completion attributed per agent; the natural mapping of CUDA contexts/streams |
+| Drawbacks | more channels/contexts = more scheduler state and context switches; per-agent directories multiply TLB pressure and page-table memory; vLLM must be structured so each agent is its own context (a serving-engine change) |
+| Performance | overhead grows with agent count; launch unchanged; arbitration and completion bookkeeping per channel |
+| Implementation cost | medium–high: channel binding, arbitration policy, per-agent counter pools; vLLM work |
+| Portability | high (channels/streams are stock); optional per-agent directories depend on the guest building them (trusted guest) or the host owning page tables (Design B/E1) |
+| Failure modes | misbinding a channel to the wrong agent; handle collisions across agents (the ISSUE-0002 class); a shared channel silently breaks agent isolation; completion counters reused across agents |
+
+**Degradation ladder** (what you lose when you can only do a coarser
+grain):
+
+```
+per-KV-block + RO shares   best sharing, software-enforced identity, needs M7
+        │  (no RO hw -> drop sharing, keep tagging)
+per-tenant directories     real hardware context separation; agents inside
+        │                  a tenant share a directory -> intra-tenant
+        │                  isolation is software-only
+per-agent channels         real fault/QoS separation at channel granularity,
+        │                  no per-block sharing
+per-mdev (1.0 baseline)    real VM isolation; intra-guest tenants are
+                           application-trusted only
+```
+
+### 4.8 Failure modes across all granularities
+
+* **TOCTOU on guest-owned tables.** The guest can rewrite its PTEs between
+  validation and execution. Mitigation: host-owned page tables for protected
+  ranges (server-owned contract exists: `COPY_SERVER_RESERVED_PDES`,
+  `kernel/gsp_rm_emulation.c:2237-2262`; real population is P6c,
+  `11-physical-backend-guest-wiring.md` §2.1 #9/#10) or validate at
+  submission and hold the entry (deferred-release hold).
+* **Identity laundering.** An agent claims another's `(tenantId, agentId)`.
+  The identity source is provisioning (guest-local Unix socket in A; KMD
+  control validation in B); the vEP binds identity to the *client/context*
+  at bind time, and a second bind for the same range from a different
+  client is refused.
+* **Stale-share escape.** Revoke without invalidate leaves a borrower PTE.
+  Invariant I1/I2; until invalidate exists, shares must be configured as
+  copy-on-write or non-revocable-at-runtime (drained between batches).
+* **Handle collision.** Two guests/agents pick the same RM handles —
+  observed. Owner-keyed tables + per-client reset (issue-number-2).
+* **Migration inconsistency.** Saved state without tags/residency can
+  reattach pages to the wrong owner. Invariant I6.
+
+---
+
+## 5. End-to-end walked example: two tenants, `paged_attention_v2`, one guest
+
+### 5.1 Setup
+
+One guest, one mdev. Two tenants: **T1** with agent **A1** (model X), **T2**
+with agent **A2** (model Y). Each runs its own serving engine process; the
+extended vLLM allocates KV blocks from the agent's VA pool. `oiv-agentd`
+binds `(T1,A1)` and `(T2,A2)` and publishes tag records for their pools
+into the mailbox (Design A) or issues `TENANT_BIND`/`TAG_RANGE` (Design B).
+Channels: A1 and A2 each own a compute channel and a copy channel; both are
+bound to their agent (`AGENT_CHANNEL_BIND`).
+
+```
+GUEST VA SPACE (one directory per tenant under Design B, or per agent)
+  T1 subtree / pool        T2 subtree / pool
+  ┌───────────────┐        ┌───────────────┐
+  │ block B11 ... │        │ block B21 ... │
+  └───────────────┘        └───────────────┘
+  block tables (int32)     block tables (int32)
+  ├─ seq s1: [B11,B12,..]  ├─ seq s3: [B21,B22,..]
+  └─ seq s2: [B13,..]      └─ seq s4: [B21,...]  (prefix share? B21 RO)
+```
+
+### 5.2 Flow (happy path)
+
+```mermaid
+sequenceDiagram
+  autonumber
+  participant V1 as vLLM engine T1
+  participant V2 as vLLM engine T2
+  participant AG as oiv-agentd
+  participant KMD as guest KMD (stock in A)
+  participant EP as vEP host (vgsp.ko)
+  participant MMU as GMMU page tables
+  participant GPU as engines
+
+  V1->>AG: register (T1, A1); request VA pool
+  V2->>AG: register (T2, A2); request VA pool
+  AG->>EP: tag records: T1/A1 -> VA range R1, T2/A2 -> R2 (mailbox or RM ctrl)
+  EP->>EP: registry: owner(R1)=T1/A1, owner(R2)=T2/A2
+  V1->>KMD: cuMemCreate/cuMemMap KV blocks inside R1
+  V2->>KMD: cuMemCreate/cuMemMap KV blocks inside R2
+  KMD->>EP: SET_PAGE_DIRECTORY (recorded, owner = mdev)
+  EP->>MMU: (backend) PTEs for R1 only in T1 dir, R2 only in T2 dir
+  V1->>KMD: launch decode kernel on A1 channel (block table in R1)
+  V2->>KMD: launch decode kernel on A2 channel (block table in R2)
+  EP->>EP: submit-time check: channel A1 -> VA in R1 only; A2 -> R2 only
+  EP->>GPU: forward work
+  GPU-->>EP: completions
+  EP-->>V1: release -> A1's semaphores only
+  EP-->>V2: release -> A2's semaphores only
+```
+
+### 5.3 Malicious attempt, and which layer stops it
+
+Scenario: tenant T1 crafts a decode kernel whose block table contains a VA
+inside R2, or directly dereferences an R2 VA.
+
+1. **Application layer (none).** vLLM's own bookkeeping does not stop it —
+   this is the gap being fixed.
+2. **GMMU topology.** In T1's page directory there is **no PTE** for an R2
+   VA. The engine's access faults (invalid leaf / unmapped). Under
+   per-tenant directories this is the primary stop; the fault carries the
+   faulting VA and the channel.
+3. **vEP submit-time policy.** Before forwarding, the decoded work is
+   checked against the channel's allowed VA set (owner map + tags). The
+   submission is refused and counted *before* any engine access is
+   programmed; this also covers accesses that would otherwise appear legal
+   because of a stale PTE.
+4. **vEP map-time policy.** If T1 instead tries to *map* an R2-owned block
+   into R1 (e.g., by getting the guest KMD to place a PTE pointing at
+   T2's memory through a shared directory), the tag registry rejects the
+   mapping: owner(R2 physical page) ≠ T1. Refused at population time.
+5. **Fault attribution.** The fault (from step 2/3) is attributed to
+   `(mdev, channel)` → `(T1, A1)`; the notifier and event go only to A1's
+   event object; T2's channels and completions are untouched. A1's context
+   is the only thing torn down (policy: kill the offending agent, not the
+   server).
+6. **Audit.** A periodic owner-vs-PTE audit (canary pages at pool
+   boundaries, checksummed) detects any escape that steps 2–4 missed and
+   fails closed for the owning tenant.
+
+What this stop costs: a fault is ms-class (engine fault → status queue →
+vEP relay → guest notifier), and the refused submission costs one
+validation pass at decode time (class: µs, same order as the existing
+per-entry decode, `10-interaction-and-scheduling.md` §6). Neither number
+is measured multi-tenant yet.
+
+### 5.4 Degradation: if only per-tenant separation is available
+
+Assume the hardware/format experiments (M6–M7) do not land and we ship
+only per-tenant directories plus owner-keyed software:
+
+* T1 and T2 remain separated **if and only if** they are in different
+  tenants **and** each tenant's agents do not share a directory (the
+  deployable unit is one agent per tenant, or per-agent contexts).
+* If two agents inside one tenant share a directory (the common vLLM
+  single-process case), the cross-agent guarantee downgrades to
+  **app-level**: block tables are validated at submit time against the
+  agent's VA pool, but a kernel dereferencing a sibling's VA resolves (the
+  PTE exists). The stop becomes detection (audit) + kill, not prevention.
+* Prefix-cache sharing across agents degrades to copies (no proven RO
+  enforcement), costing VRAM and copy bandwidth.
+* Honest recommendation for the degraded mode: **one agent per tenant, one
+  tenant per context** (vLLM tenant count = context count) — a
+  configuration contract, not a hardware guarantee.
+
+---
+
+## 6. Portability
+
+### 6.1 What is portable vs not
+
+| Artifact | Portable? | Why / what to re-derive |
+|---|---|---|
+| Object model (client, device, vaspace, channel, events) | **yes** | stable across 550–580 as a family; verified live only on 570.144 |
+| Page-directory contract (`SET_PAGE_DIRECTORY`/`UNSET`, `COPY_SERVER_RESERVED_PDES`) | **yes, versioned** | controls carry `hVASpace`/`chId`/geometry; re-verify field offsets per version |
+| Our GMMU walker/decoder approach | **yes** | decode captured tables; field catalogue per generation (Turing VER2 proven; others re-derive) |
+| PTE/PDE bit encodings | **no** | generation-specific (VER2 on Turing; newer formats expected on Blackwell); re-derive from captures + format docs per device |
+| Invalidate semantics / fetch-line behavior | **no** | measured per implementation; current 128-byte fetch-line, no-invalidate result is Turing+our-path-specific |
+| Tag controls / mailbox (this design) | **concept yes; wire no** | new controls are per-version additions; mailbox depends on guest vidmem being host-readable (emulation only) |
+| Per-tenant/per-agent policy | **yes** | policy is ours; mechanism depends on the above |
+| mdev type/VRAM sizes | **yes** | per-type metadata registered by the host driver; sizes are policy, not hardware |
+
+### 6.2 Driver 610
+
+RM 610 is under a **hard gate**: no acquisition, analysis, emulation edit,
+physical work or support claim before explicit named-human approval
+(`../../debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md`). Nothing here
+authorizes 610 work; the port plan for this design is M8, gated. Expected
+to change and need re-derivation when approved: control inventories and
+object model extensions, engine caps, the accelerator/CCS object family,
+and possibly the GMMU entry format
+(`../../debug/risc-v-gsp-rm-vm/docs/34-BLACKWELL-SOURCES.md`).
+
+### 6.3 Grace-Hopper / Grace-Blackwell / GB10
+
+Targets per the matrix
+([`../06-portability/hardware-software-matrix.md` §2](../06-portability/hardware-software-matrix.md));
+what matters here:
+
+* **aarch64 host.** Page-size/cache-attribute differences are a known risk
+  ([`hardware-software-matrix.md` §8](../06-portability/hardware-software-matrix.md));
+  the VA↔PA discovery technique (guest direct-map arithmetic,
+  [`06-host-module-and-hybrid-model.md` §4.2](06-host-module-and-hybrid-model.md))
+  must be re-verified.
+* **Memory hierarchy.** Coherent NVLink-C2C blurs sysmem/vidmem; the
+  aperture model and the §3 oversubscription design must be re-derived
+  around C2C coherence. `[hypothesis]`
+* **GMMU formats.** Entry formats/page sizes are expected to differ;
+  anything per-PTE (including a possible ACL-like field) must be re-derived
+  before being relied on. `[hypothesis]`
+* **Confidential computing (CCS).** CoCo changes the trust boundary in our
+  favour (hardware memory encryption) and against us: the host cannot read
+  guest memory content, which breaks the content-scan discovery techniques
+  in the current memory model ([`06-host-module-and-hybrid-model.md`
+  §4.4](06-host-module-and-hybrid-model.md)). The `encrypted` PTE bit
+  exists (proven format); semantics are unstudied. `[hypothesis]`
+* **MIG** on datacenter parts: an alternative hardware partitioning
+  substrate the policy layer can use where available
+  ([`09-reference/glossary-and-faq.md`](../09-reference/glossary-and-faq.md),
+  "What about MIG?"). `[design]`
+
+### 6.4 Re-derivation checklist (per new device/version)
+
+Capture page tables and decode them; confirm the encoding and write down
+explicitly whether any owner/ACL-like field exists. Confirm the directory
+is still per-context and the controls still carry the VA space and channel.
+Confirm invalidate semantics with a controlled experiment (no-op until
+proven; M5/M7). Re-run the owner-keying review against the new control/RPC
+inventory, then the two-guest gate and the §5 walkthrough, attaching
+sessions and M1–M6-style tables per the portability discipline
+(`../../debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md`).
+
+---
+
+## 7. Current and future challenges, tooling gaps, roadmap
+
+### 7.1 Current challenges
+
+1. **The service-state split is the immediate blocker.** `gsp_sema_service.c`
+   is global-boot-state code (single `svc`, global counters, value-keyed
+   tables); splitting it per mdev is review-heavy, with latent assumptions
+   like the ISSUE-0002 root cause.
+2. **GR execution is upstream of every engine-isolation claim.** Until guest
+   compute executes on real engines (`../04-implementation/03-execution-frontier.md`),
+   tenant engine isolation can only be validated on the emulation plane.
+3. **No per-tenant fault injection.** The backend's `gsp_phys_fault_inject`
+   (`06-host-module-and-hybrid-model.md` §6) is tied to one session; there is
+   no "inject as tenant B, prove A unaffected".
+4. **No GMMU permission-probe methodology.** The `read_only` bit is a proven
+   format field but unexercised, and we have no owner/ACL search protocol
+   beyond decode-and-eyeball.
+5. **No two-guest harness.** Tools assume one VM/one mdev
+   (`../../scripts/vgpu_vm.py`); the 1.0 gate needs a two-VM driver.
+6. **No side-channel measurement** — no methodology, no data; explicitly
+   deferred to measurement by
+   [`../05-operations/01-performance-correctness-security.md` §4](../05-operations/01-performance-correctness-security.md).
+7. **Real VRAM backing is not implemented** (`06-host-module-and-hybrid-model.md`
+   §8): today isolation-by-address is over emulated RAM; real DMA, real page
+   tables and shared scratch pools arrive with the backend
+   (`11-physical-backend-guest-wiring.md` §P6b).
+8. **vLLM integration cost.** The extension touches allocator/block
+   internals; budget a pinned fork plus a thin plugin, not an upstream bet.
+
+### 7.2 What our tools do not address today
+
+`docs/NN` below = `../../debug/risc-v-gsp-rm-vm/docs/NN-*.md`.
+
+| Existing tool (public) | Useful for | Gap |
+|---|---|---|
+| GMMU PTE decoder (`../../debug/risc-v-gsp-rm-vm/docs/43`) | decode captured entries; field catalogue | decode-only; no permission semantics, no walker dereference |
+| GMMU oracle workflow (`docs/44`) | compare simulated vs observed register/value traces | no enforcement experiment; oracle capture is supervised-human |
+| Validation predicate probe (`docs/47`) / surgical rehost (`docs/46`) | run one firmware validation branch offline with crafted state | firmware-side; not a GMMU permission probe (candidate substrate, needs a design) |
+| Control discovery (`docs/19`) / reach (`docs/19`) | hidden control IDs; symbol reachability | simulator surface only; not real-RM |
+| Host debug parameters (`06-host-module-and-hybrid-model.md` §6) | inspect/poke memory, channels, semaphores | single-device oriented; scans are device-selected by first-open |
+| Two-guest harness / tenant fault injector / side-channel harness | — | **none of the three exists** |
+
+### 7.3 Roadmap
+
+Each stage has an observable acceptance criterion; no stage advances on
+intention. **[1.0]** = baseline programme; **[research]** = post-1.0,
+dual-gated, duplicate-not-replace, never required by the baseline.
+
+| Stage | Scope | Acceptance (measurable) | Class |
+|---|---|---|---|
+| **M1** | Owner-key every vEP table; split `svc` into per-mdev contexts (or keyed tables); per-mdev pump/service; per-client reset | two concurrent mdevs both get service (no first-open starvation); crafted handle/VA collision test fails closed; ISSUE-0002 sequential acceptance stays green; one-guest regression suite green | [1.0] |
+| **M2** | Two-guest gate harness + the 2G-01…2G-09 matrix (§2.9) | all tests pass 3× consecutive with evidence; ladder rung 5 "isolation" checklist partially met (fairness still open) | [1.0] |
+| **M3** | Per-mdev real GSP clients; per-tenant RPC accounting and bounded rates; head-of-line measurements | two guests' forwarded CE work both completes; p99 cross-tenant delay measured and within the published bound; no cross-tenant completion (canaries) | [1.0, after GR/backend] |
+| **M4** | Independent VRAM sizes end to end + sparse backing v1 + capacity admission | two mdevs with different types; sum(declared) > capacity with refusal-only admission; per-mdev accounting counters; no leak under 20 start/stop cycles | [1.0-adjacent] |
+| **M5** | Real invalidate path + eviction/reclaim for KV blocks; residency invariants I1–I6 tested | controlled page-out/page-in cycle with hardware work verified byte-exact; no PTE outlives its block; hold timeouts bounded | [1.0-adjacent, needs backend] |
+| **M6** | Design A prototype: extended vLLM + `oiv-agentd` + mailbox; VA-subtree policy in the vEP | cross-tenant VA access faults (T2 block from T1 channel) and is attributed to T1; differential test shows no foreign block readable/writable; overhead vs unextended vLLM measured and published | [research] |
+| **M7** | GMMU permission/predicate probe: supervised experiment for `read_only` (and any ACL-like field, per generation) | a controlled write through an RO leaf behaves as documented (or the bit is declared unusable); result recorded with sessions and the fidelity label | [research] |
+| **M8** | Design B (new RM controls + guest KMD extension) and/or CCS/610 port, only when approved | controls round-trip with enforcement at map and submit; fault attribution per agent; 610/CCS work follows its own gate (`docs/23`) | [research, gated] |
+
+Kill criteria (so research can end): M6 dies if tagging overhead exceeds
+the project's sync-heavy factor without a security gain over M1–M5 (which
+deliver VM/tenant separation with stock drivers); M7 dies on a family with
+no RO-equivalent and no owner field if the capture shows no enforcement
+(per-PTE protection impossible; the design then rests on topology, which
+M6 must state); M8 dies if M6 meets the same guarantee within the
+baseline's envelope and no CCS requirement materialises.
+
+### 7.4 Baseline discipline (mandatory)
+
+Per `AGENTS.md` index entry 11 and
+`../10-post-1.0/experimental-host-acceleration.md`, **post-1.0 goals are
+not design inputs**: nothing in sections 4.5–4.8, 5.3's research stops, or
+stages M6–M8 may shape baseline code (no "just in case" fields, hooks or
+parameters). Research stages are dual-gated (compile-time + module-load
+flags), duplicate rather than replace, and are never required by the
+baseline build or 1.0 acceptance. The 1.0 programme owns M1–M5 only; M6+
+starts after 1.0 goals are met, on an explicit human decision.
+
+---
+
+## 8. Cross-links
+
+* [`13-vexecution-plane.md`](13-vexecution-plane.md) — vEP vocabulary and
+  the execution plane this document polices (written in parallel).
+* [`05-gmmu-amap-hal-gr.md`](05-gmmu-amap-hal-gr.md) — GMMU, page tables,
+  channels, scheduling; the hardware background for §4.
+* [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md) —
+  backend ownership, isolation target (§4.8), open questions;
+  [`11-physical-backend-guest-wiring.md`](11-physical-backend-guest-wiring.md) —
+  the guest-wiring seams (P6a/P6b/P6c) and risk guards this design needs.
+* [`10-interaction-and-scheduling.md`](10-interaction-and-scheduling.md) —
+  concurrency, ownership rules, multi-tenant scheduling model (§7);
+  [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) —
+  memory model, VA↔PA discovery, debug parameters, exactly-once rules.
+* [`../05-operations/01-performance-correctness-security.md`](../05-operations/01-performance-correctness-security.md) —
+  evaluation ladder (rung 5), threat model, measurement discipline;
+  [`../06-portability/hardware-software-matrix.md`](../06-portability/hardware-software-matrix.md) —
+  support ladder, new-device/version procedures, guest rule;
+  [`../01-orientation/01-purpose-and-scope.md` §5](../01-orientation/01-purpose-and-scope.md) —
+  multi-tenancy model.
+* `../../debug/issue/issue-number-1.md`, `../../debug/issue/issue-number-2.md` — the
+  CUDA reproducibility and per-client state issues M1 resolves;
+  `../../debug/risc-v-gsp-rm-vm/docs/{43-GMMU-PTE-DECODER,44-GMMU-ORACLE-AND-WORKFLOW,23-RM610-READINESS}.md` —
+  tooling and gates cited above.
+* Local analysis: `LOCAL-MULTI-TENANT-GMMU.md` (adjacent; part of the
+  local analysis) — provenance behind §4.2 and the unredacted design
+  reasoning.