path: root/docs/03-architecture/14-multi-tenant-gmmu-protection.md

# Multi-Tenant GMMU Protection: Isolation Design and Paged-Attention Controls

> **Status: design (2026-10-08, master `1f571dda`).** This document specifies
> how Open-IOV isolates multiple tenants — multiple mdevs *and* multiple
> tenants inside one guest — at the GPU memory-management unit (GMMU), and how
> a paged-attention inference stack (vLLM-style) can use those protections.
> Sections 1–3 are **baseline 1.0 design**; sections 4.5+ and roadmap stages
> M6–M8 are **post-1.0 research**, explicitly gated (see §7.4 and
> `AGENTS.md` index entry 11). No part of the research design may become a
> design input for baseline 1.0 code: no forward-looking parameter, hook or
> abstraction may be added to a product path for it.
>
> **Method.** Every empirical claim about our system cites a code line or
> tracked document; hardware-format claims are marked **proven** (observed
> live, or in a tracked field catalogue) or **hypothesis** (needs a
> supervised experiment). No timing/cycle/silicon-accuracy claim is made.
>
>
> **Companion document.** [`13-vexecution-plane.md`](13-vexecution-plane.md)
> (written in parallel) defines the vEP/instance/agent vocabulary used here;
> where the two disagree, 13 is the vocabulary authority and this document
> is the GMMU-policy authority.

Tags: `[live]` exercised; `[wip]` default-off; `[design]` not built; `[hypothesis]` unexercised; `[research]` post-1.0, dual-gated.

---

## 1. Problem statement: 1.0 needs N mdevs, N guests, and flawless isolation

### 1.1 The 1.0 requirement

The project's stated 1.0 gate is multiple mediated devices, two or more
guests, multiple VRAM sizes, and isolation that survives adversarial review.
The tracked status is explicit that this area is **designed, partially
implemented** ([`../README.md` §3](../README.md), row "Multi-tenant isolation
(VFs, mdev-per-VM)"), and the multi-tenancy model is stated as a first-class
value: one mdev per VM, per-instance interrupts, no shared host state, fail
closed ([`../01-orientation/01-purpose-and-scope.md` §5](../01-orientation/01-purpose-and-scope.md)).
The evaluation ladder puts "N VMs, isolation + fairness + no cross-talk" on
rung 5 ([`../05-operations/01-performance-correctness-security.md` §1](../05-operations/01-performance-correctness-security.md)).

Three 1.0 requirements drive this document:

1. **Two guests on one GPU.** Two VMs, two mdevs, both running the stock
   driver, concurrently and repeatedly. Today the second *client in one
   guest* already wedges (`../../debug/issue/issue-number-2.md`), and the
   two-guest harness does not exist.
2. **Independent VRAM sizes.** Each mdev exposes an emulated framebuffer
   size that need not relate to the physical framebuffer or to another
   mdev's size: `nv_vgpu_type_info.vram_mb`
   (`kernel/include/ioctl.h:39-46`), BAR1 sized from it
   (`kernel/bar_init.c:60-77`), and GSCI/PFB reports derived from the same
   value (`kernel/gsp_rm_emulation.c:1034,1106-1110`,
   `kernel/bar_emulation.c:271-310`, `kernel/gsp_emulation.c:743-753`).
   The mechanism exists per mdev; the *capacity* story does not (§3).
3. **Multi-tenant inference inside one guest.** A vLLM-style server
   multiplexes many *logical tenants* (API users, agents, sequences) inside
   one VM that owns one mdev. VM-level isolation (IOMMU, mdev) is blind to
   them. This is the workload §4 targets.

### 1.2 Where isolation can exist

Isolation is not one mechanism; it is a stack of layers, each with a
different owner and a different failure mode. The design must say, for each
layer, what holds and what does not.

```
L0 PLATFORM          IOMMU/SMMU: per-VM DMA address spaces        [live]
   │                 KVM memory + vCPU boundary                    [live]
L1 TRANSPORT         VFIO mdev: one mediated device per tenant     [live]
   │                 config/BAR/IRQ mediation, per-device eventfd
L2 vEP STATE         per-mdev emulation context (BAR shadows,      [live/wip]
   │                 queues, msgq, interrupt state)                 §2.2
   │                 per-mdev work-service state                    [design]
L3 GMMU              per-context page directory (VA space)         [live//design]
   │                 PTE-level permissions (RO, privilege)          [proven fmt]
L4 ARBITRATION       channel ownership, runlist/TSG scheduling     [design]
L5 COMPLETION        semaphore resolution per owner; no cross-      [wip]
   │                 tenant release apply                           §2.5
L6 FAULT             fault attribution to the owning tenant;       [wip]
                      notifier only to that tenant                 §2.6
L7 APPLICATION       per-tenant/per-agent VA partitioning inside    [research]
                      a guest (vLLM + agent utility)               §4
```

The honest summary from the tracked docs applies: **memory isolation between
VMs is hardware-plus-hypervisor enforced today; engine-level context
separation and performance isolation are what the project must still build**
([`09-full-system-picture.md` §6](09-full-system-picture.md)).

### 1.3 Trust boundaries and the principals

| Boundary | Attacker | Capability | Must hold |
|---|---|---|---|
| Guest ↔ guest | whole VM | arbitrary device programming, arbitrary guest kernel | no memory, state, interrupt or completion observation across mdevs |
| Guest ↔ host | whole VM | arbitrary DMA/MMIO against its own device | no host memory access, no module state escape, fail closed |
| Host user ↔ device | unprivileged host user | access to `/dev/*` as granted | no privilege escalation via our interfaces |
| Tenant ↔ tenant (in one guest) | guest userspace tenant | arbitrary CUDA kernels, arbitrary block-table edits | a tenant cannot read/write another tenant's KV blocks |
| Agent ↔ agent (in one tenant) | guest userspace agent | arbitrary CUDA kernels | an agent cannot read/write a sibling agent's KV blocks |

The first three are baseline 1.0 (they follow the threat model in
[`../05-operations/01-performance-correctness-security.md` §4.1](../05-operations/01-performance-correctness-security.md)).
The last two are the *multi-tenant inference* boundaries. Note that they
assume the **guest kernel is trusted**: everything below the application
runs the stock driver, and the tenant/agent separation is enforced at the
application's use of the device, not against a malicious guest KMD. A
malicious guest KMD is the guest↔host boundary, already covered by L0/L1.

### 1.4 Where the code actually stands (the gap this document fills)

The guest-facing plane is per mdev: `nv_vgpu_device.gsp_comm_ctx`
(`kernel/include/nvidia-vgpu-vfio.h:133`) wraps a per-instance
`struct gsp_emu_context` (`kernel/gsp_emulation.h:130`) created by
`gsp_emu_init()` (`kernel/gsp_emulation.c:724`) from
`nv_vgpu_device_init_work()` (`kernel/main.c:1657`). BAR shadows, message
queues and MSI-X state are per instance.

The **work/completion service is not**. `gsp_sema_service.c` holds one
global `static struct sema_svc svc;` (`kernel/gsp_sema_service.c:676`), and:

* the service thread pumps **the first open mdev it finds**
  (`kernel/gsp_sema_service.c:13352-13364`);
* the work-submission doorbell carries no owner — `gsp_sema_service_doorbell(u32 token)`
  (`kernel/gsp_sema_service.c:13512`), called from the VFIO write path with
  no mdev argument (`kernel/main.c:865`);
* channel registrations are keyed by the guest-chosen `h_object` alone
  (`kernel/gsp_sema_service.c:847`, table bounded by `SVC_MAX_CHANNELS 128`
  at `:122`), and VA spaces by the guest-chosen `h_vaspace` alone
  (`:988`, `:1019`, bounded by `SVC_MAX_VASPACES 8` at `:309`);
* the VA→PA learned map and pins are keyed by GPU VA only
  (`kernel/gsp_sema_service.c:113-130`, `:1045`), with no mdev key;
* completion counter pools and rank latches are global boot state, as the
  second-client wedge demonstrates: the old client's channel/rank/ring state
  makes the host's completion path go quiet for the new client
  (`../../debug/issue/issue-number-2.md` §Root cause).

Two guests can trivially choose the *same* handles and GPU VAs (live:
channel `0xcaf000d4`, `../../debug/issue/issue-number-2.md` §Evidence); keyed by
value alone, those registrations and translations collide — the exact class
that already breaks the second client in one guest. **Multi-mdev isolation
is therefore first a service-state-ownership problem, and only second a GMMU
problem.** §2 fixes the ownership; §4 builds the GMMU policy on it.

### 1.5 Vocabulary

**Tenant** — isolation principal (`tenantId`): a VM/mdev at L0–L2, a logical
serving principal inside a guest in §4. **Agent** — finest principal inside
a tenant (`agentId`): a vLLM engine/worker, a CUDA context, a process.
**vEP** — the vGSP Execution Plane: the execution half of vGSP (guest work →
physical execution → guest-visible completion), as defined in
[`13-vexecution-plane.md`](13-vexecution-plane.md). This document's problem
is that the vEP's work-service state is currently one global context rather
than per-mdev. **KV block** —
paged-attention cache page (vLLM `paged_attention_v2`: fixed-size blocks,
one `block_tables` entry per logical block per sequence); a GPU allocation
range, not a hardware object.

---

## 2. Security and isolation design for multi-mdev (vEP terms)

### 2.1 Address-space separation

**Rule S1.** Every host-side table that describes guest state is keyed by
`(mdev, guest-name)` — handle, GPU VA, doorbell token, counter slot — never
by the guest-chosen value alone.

What exists and what changes:

| State | Today | Required |
|---|---|---|
| emulation context | per mdev `[live]` (`nvidia-vgpu-vfio.h:133`, `gsp_emulation.h:130`) | unchanged |
| `svc.chan[]` | keyed `h_object` (`gsp_sema_service.c:847`) | key `(vgpu, h_object)`; per-mdev lifetime reset |
| `svc.vas[]` | keyed `h_vaspace` (`:988`) | key `(vgpu, h_vaspace)` |
| learned VA→PA map / pins | keyed GPU VA (`:113-130`) | key `(vgpu, VA)`; per-mdev epoch |
| doorbell token | ownerless (`:13512`, `main.c:865`) | `(vgpu, token)`; route to the owner's pump |
| completion counters | global boot state | per mdev, reset on session/rebind |
| pump | first open mdev (`:13352-13364`) | per-mdev pump (or round-robin over all open mdevs) |

The GMMU half of address-space separation is the page directory: each VA
space has one, published by the guest through `SET_PAGE_DIRECTORY` with its
`hVASpace` and `chId` (`kernel/gsp_rm_emulation.c:3310-3314,3366-3372` for
the recording path; the format is documented at `:3307-3319`). The vEP
records the directory per owner and must translate only within the owning
mdev's tables. `[design]`

### 2.2 Per-mdev channel ownership

**Rule S2.** A channel belongs to exactly one mdev for its whole life, and
its engine work, doorbell token, error notifier and completion counters are
resolved through the owner only.

The channel descriptor today carries GPFIFO VA, USERD, instance and
notifier pages (`kernel/gsp_sema_service.h:22-33`); the registration arrives
from the guest's `GSP_RM_ALLOC` handling while the emulation context is
known (`ctx->vgpu`), so the owner is available at registration time — it is
simply not stored. The fix is structural (add the owner to `svc_channel`
and to every lookup), not algorithmic.

Engine arbitration is where ownership becomes policy: channels of the same
engine from different mdevs must be scheduled by our policy on top of the
real GSP scheduler ([`10-interaction-and-scheduling.md` §7](10-interaction-and-scheduling.md)).
Today the single real GSP session and its global `gsp_phys_rm_lock` serialize
all real firmware traffic (`11-physical-backend-guest-wiring.md` §1.3), so
one tenant's slow RPC blocks every other tenant's work — head-of-line
blocking, addressed in §2.7.

### 2.3 DMA / IOMMU interplay

Ground rules, unchanged by multi-tenancy:

* Guest memory is reached only through `vfio_dma_rw(&vgpu->vfio_dev, ...)`
  (`kernel/gsp_sema_service.c:754-762`, `kernel/gsp_rm_emulation.c:27-55`),
  i.e. through the hypervisor's mapping for **that** mdev's VFIO container.
  A GPA from guest A cannot be resolved through guest B's device — the
  boundary is enforced below us, and we never weaken it
  ([`../05-operations/01-performance-correctness-security.md` §4.2](../05-operations/01-performance-correctness-security.md)).
* The emulated vidmem (BAR1) is a private host-RAM buffer per mdev
  (`kernel/bar_init.c:60-77`); FB-aperture accesses stay inside the owning
  mdev's buffer (`svc_read_ap`/`svc_write_ap` bounds-check
  `vgpu->bars[1]`, `kernel/gsp_sema_service.c:786-814`).
* Physical-backend caveat (real VRAM path): on the reference host the
  physical GPU's IOMMU group is an identity domain and the mediated device
  registers as an emulated-IOMMU VFIO device, so guest GPAs are translated
  through the QEMU shared-RAM file rather than a kernel `iommu_domain`
  (`11-physical-backend-guest-wiring.md` §P6a). This translation is
  per-mdev-gated but is **not** an IOMMU guarantee; with multiple mdevs on
  one backend it must be audited per mdev, and the active-mdev gate must
  never select another mdev's file. `[design]`
* When the backend owns the GPU, any host-translated address that reaches a
  real engine must come from the owning mdev's resolved set, never from a
  global table (`11-physical-backend-guest-wiring.md` §6.4).

### 2.4 Data-path separation

* BAR0 shadow, BAR1 vidmem shadow, BAR3/instance window are per mdev
  (`kernel/bar_init.c:59-77`; create at `kernel/main.c:554`; cited in
  `11-physical-backend-guest-wiring.md` §0.1 `[live]`). Two mdevs'
  shadows are separate allocations; the invariant to test is that no
  handler ever mixes the two base pointers (handlers take
  `struct gsp_emu_context *ctx` and dereference `ctx->vgpu`, e.g.
  `kernel/gsp_rm_emulation.c:2829-2834`; the tests in §2.8 prove it).
* Real-VRAM forwarding already isolates *engines* from guest memory by
  submitting through module-built scratch page tables that map
  already-resolved host pages and never reuse a fetch line within a boot
  (`11-physical-backend-guest-wiring.md` §P6b). With multiple mdevs, scratch
  pools must be per mdev: capacity is a per-boot resource (512 pairs at
  16 PTEs/line in the current design) and shares a hardware resource with
  every tenant.
* Guest page tables are **guest-owned** (the driver builds them; we walk
  them — `kernel/gsp_sema_service.c:1052-1072`), so a malicious guest can
  retarget its own PTEs at any time: fine for guest↔host (the IOMMU caps the
  damage to the guest's own memory), but the fundamental limit of intra-guest
  enforcement in §4 — *policy can be only as strong as page-table ownership*.

### 2.5 Completion isolation

**Rule S3.** A decoded semaphore release is applied only in the owning
mdev's address space, and a completion event is posted only to the owning
mdev's event object.

Today the release path resolves the target VA through *global* tables
(`svc_resolve_va`, `svc_sem_write`; `kernel/gsp_sema_service.c:2098,2474`)
and posts through `gsp_rm_post_channel_event()` on the emulation context
(`kernel/gsp_rm_emulation.c:602`). The first is the cross-tenant hazard: a
release record decoded from guest A's push buffer could resolve through a
pin learned from guest B if the VA collides. The exactly-once rules must be
preserved per owner (a copy runs at most once per GPFIFO entry; a virtual
operand is translated per 4 KB page; a consumed slot never runs later —
[`06-host-module-and-hybrid-model.md` §4.4](06-host-module-and-hybrid-model.md)).

### 2.6 Fault containment

**Rule S4.** A fault (engine fault, malformed RPC, translation failure,
timeout) is attributed to its owner, delivered only to the owner's notifier
and event object, and must not stall or corrupt another tenant.

The machinery exists per channel (error notifier slots in the channel
descriptor, `kernel/gsp_sema_service.h:31`) and for forwarded engines
(fault observation/relay default-off,
`11-physical-backend-guest-wiring.md` §P6c C3), and the malformed-RPC path
is fail-closed. What is missing is the owner key at the relay decision, and
a per-tenant fault-injection test (§2.8). A wedged server is a
*module-global* fail-safe today (`08-physical-gpu-integration.md` §7), which
is correct for one backend session but must never be reachable from a
guest-triggered path: guest input may abort the guest, never the service.

### 2.7 Time-slicing risks

With one physical GPU, tenants share engines by time slicing. Known risks:

* **Head-of-line blocking:** the single real GSP session + `gsp_phys_rm_lock`
  serializes backend RPCs; a tenant that floods controls starves the others.
  Mitigation design: per-mdev backend clients (already planned,
  [`09-full-system-picture.md` §5](09-full-system-picture.md)), bounded
  per-tenant RPC rates in the vEP, and per-tenant accounting.
* **Slice inversion:** guest A's long-running engine work delays B's
  completion relay; the relay path must stay bounded and never poll a
  hardware queue while holding a lock a second tenant needs
  (`11-physical-backend-guest-wiring.md` §1.3 lock order).
* **Starvation:** without a policy, first-come submission can starve a
  tenant. The policy is planned
  ([`10-interaction-and-scheduling.md` §7](10-interaction-and-scheduling.md));
  until it exists, the honest statement is "fairness is unmeasured".

### 2.8 Side channels (public knowledge only)

We make no original side-channel claims and have no measurements. The
classes that apply to any shared-GPU time-slicing design, from public
literature, are: memory-bandwidth contention observable through timing;
L2/LLC state effects; scheduler/slice timing; and interrupt-timing
observation. The project's stated bar is that an adjacent tenant must not
observe timing/state/interrupts *beyond policy*
([`../05-operations/01-performance-correctness-security.md` §4.1](../05-operations/01-performance-correctness-security.md)):
interrupts are already per-instance, timing policy must be explicit and
documented, and measurement is a 1.0-era task (rung 5). Until measured,
treat side channels as **unmitigated but bounded by per-instance state
separation**, and never claim isolation stronger than the ladder proves.

### 2.9 Validation and test plan for the 1.0 "two guests" gate

The gate: two QEMU guests, two mdevs, concurrent, stock driver; every test
captures evidence (dmesg, counters) and passes at least 3 consecutive runs.

| # | Test | Pass criterion |
|---|---|---|
| 2G-01 | Boot both guests concurrently on two mdevs; `nvidia-smi`; CUDA driver-API smoke | both green, payload round-trip exact; no cross errors in dmesg |
| 2G-02 | Distinct VRAM types (e.g. 512M and 1G) on the two mdevs | each guest's `nvidia-smi`/GSCI reports its own size; BAR1 sizes differ and match |
| 2G-03 | Handle/VA collision: force both guests to allocate the same RM handles and GPU VAs (the `0xcaf000d4` class) | both clients progress; no release lands in the wrong guest (verify by canary words around each guest's semaphores) |
| 2G-04 | Doorbell cross-talk: ring guest A's tokens while B is idle | B's channels/state unchanged; B's poll loop untouched; A completes |
| 2G-05 | Completion ownership: craft a release VA that exists in both guests' address spaces | resolves only in the owner's memory; the other guest's canary unchanged |
| 2G-06 | Fault injection: malformed RPC, bad queue link, engine-fault relay from A | A fails closed (documented error); B unaffected; no module-global stall |
| 2G-07 | Teardown: remove mdev B while A runs; rmmod after both stop | A continues; teardown order preserved; module unload clean |
| 2G-08 | Stress: N≥20 concurrent start/stop cycles; guest reboots; module reloads | no leaked per-mdev state; no cross-tenant counters; clean dmesg |
| 2G-09 | Sequencing: the ISSUE-0002 acceptance (3 sequential clients in one boot) plus 2 concurrent processes | green; per-client reset proven (issue-number-2 §Validation plan) |

Known blockers before the gate can run: the per-client/per-channel reset
(`../../debug/issue/issue-number-2.md` §Fix plan items 1–2) and the per-mdev
service-state split (§2.1 Rule S1). The test harness (two VMs; the current
tools assume one, `../../scripts/vgpu_vm.py`) is itself a deliverable.

### 2.10 Fail-closed rules

Isolation properties that cannot be established (missing IOMMU container,
DMA mapping failure, inconsistent owner state, unkeyed table entry) stop the
affected mdev and say why; nothing silently degrades
([`../05-operations/01-performance-correctness-security.md` §4.2](../05-operations/01-performance-correctness-security.md)).
Malformed or stale queue elements, PTE chains and release records are
rejected without disturbing the queue, with strict sequence matching
([`10-interaction-and-scheduling.md` §3](10-interaction-and-scheduling.md)).
No forwarding seam programs guest-derived page tables into a real MMU until
the population path is proven end to end
(`11-physical-backend-guest-wiring.md` §6.4). A cross-owner table lookup is
a programming error: log once and refuse, never "resolve anyway".

---

## 3. VRAM larger than physical, by scheduling and virtualization

### 3.1 What "VRAM size" means per mdev today

Each mdev derives a consistent set of numbers from
`nv_vgpu_fb_emul_size(vgpu)` (`kernel/include/nvidia-vgpu-vfio.h:232`):
BAR1 size (`kernel/bar_init.c:68`), the `LOCAL_MEMORY_RANGE` register
encoding (`kernel/bar_emulation.c:271-310`), the GSCI `fb_length` and
`frtsOffset` (`kernel/gsp_rm_emulation.c:1106-1110,1177`) and the WPR2
address (`kernel/gsp_emulation.c:743-753`). The mdev type advertises the
same size to management tools (`framebuffer=%uM`,
`kernel/main.c:116-132`). The sizes available are dictated by the types the
host driver registers (`kernel/include/ioctl.h:39-46`,
`kernel/nv_vgpu_register_type_info` callers).

Two consequences: **presenting a size is solved** (two mdevs can present
512 MiB and 2 GiB simultaneously; nothing in the guest path forces them to
match the physical framebuffer), but **backing that size is not** — in
emulation mode each mdev's "VRAM" is a private host-RAM `vzalloc`
(`kernel/bar_init.c:70-77`), so the aggregate is bounded by host RAM; with
real VRAM backing (`08-physical-gpu-integration.md` §4.6) it is bounded by
physical VRAM. Either way `sum(vram_mb)` may exceed capacity — the
oversubscription problem.

### 3.2 Design: present size independent of physical size

**Principle.** Guest-visible VRAM size is a *contract*; the resident set is
an *implementation detail* — decouple them with tiered, revocable backing:

```
            guest-visible VRAM (mdev A, e.g. 2 GiB)      guest-visible (mdev B, 1 GiB)
                     │                                        │
              ┌──────┴──────────────────────────────┐  ┌──────┴────────────┐
              │  vEP per-mdev residency manager     │  │   ...             │
              │  pinned set | evictable set         │  │                   │
              └──────┬──────────────────────────────┘  └───────────────────┘
                     │ page-in/out (4 KiB-2 MiB)
        ┌────────────┴───────────┐        ┌───────────────────────────────┐
        │ real VRAM (hot blocks) │        │ sysmem staging (cold blocks)  │
        └────────────────────────┘        └───────────────────────────────┘
```

Techniques, each with its current status:

1. **Sparse allocation (allocate-on-touch).** The guest-visible range exists
   before it is backed; backing is populated on first write (emulation: RAM
   page; backend: real VRAM object). Today the whole BAR1 is
   `vzalloc`'d up front (`kernel/bar_init.c:70-77`) `[wip]`; sparse backing
   is `[design]`.
2. **VRAM ↔ sysmem paging.** Cold blocks move to guest sysmem (or host
   staging) and their PTEs are rewritten to the sysmem aperture; hot blocks
   move back. The GMMU already distinguishes apertures
   (`svc_pte_aperture`, `kernel/gsp_sema_service.c:1101-1108`) and the guest
   uses `ADDR_SYSMEM=1`/`ADDR_FBMEM=2` descriptors
   ([`05-gmmu-amap-hal-gr.md` §1.1](05-gmmu-amap-hal-gr.md)) `[design]`.
3. **Replayable pushes.** A push whose instruction stream is re-executed
   must be idempotent or suppressed; the host already has the machinery:
   per-entry identity checks, per-operation done bitmaps, consumed-slot
   marks, and the deferred-release hold that waits for hardware completion
   before applying releases exactly once
   ([`06-host-module-and-hybrid-model.md` §4.4](06-host-module-and-hybrid-model.md),
   `11-physical-backend-guest-wiring.md` §P6b wave 1.5) `[wip]`.
   Paging reuses this primitive: an operation that references an evicted
   block is held until the block is resident.
4. **GMMU invalidations.** Rewriting a PTE is not enough: the engine caches
   PTE fetches (observed 128-byte granularity) and there is no invalidate
   path in the forwarding code today; a runtime PTE write into an
   already-walked line was accepted but never completed
   (`11-physical-backend-guest-wiring.md` §P6b "PTE fetch-line rule").
   The guest's own `MEMSYS_L2_INVALIDATE_EVICT` control is acknowledged
   without effect in emulation (`kernel/gsp_rm_emulation.c:2519-2532`).
   A real invalidate path is a prerequisite for paging `[design]`.
5. **UVM fault replay.** Guest UVM already handles memory faults and
   re-registration; the host can signal "not resident" and let the guest
   replay the access. The guest-visible mechanism is the channel
   event/notifier path (`gsp_rm_post_channel_event`,
   `kernel/gsp_rm_emulation.c:602`); whether every access class can be
   replayed this way is `[hypothesis]` — it needs a live experiment.
6. **Migration hooks.** The module already has save/restore state sections
   (`kernel/migration.c:85+`); residency metadata (which blocks are
   resident, tags, ownership) must be part of the migrated state for
   correctness across live migration `[design]`.

### 3.3 Tradeoffs

| Effect | Cause | Control |
|---|---|---|
| Latency spike on page-in | block fetch from sysmem to VRAM | bound resident-set misses per slice; prefetch on sequential access (paged attention accesses KV blocks sequentially per sequence) |
| Throughput collapse under thrash | working set > capacity | admission control: refuse the allocation that would guarantee thrash; per-tenant quotas |
| Completion jitter | held pushes waiting for residency | bound the hold; on expiry fail closed per operation (existing bounded-hold pattern, `gsp_sema_service.h:220-236`) |
| Fairness loss | one tenant's evictions victimize another | per-tenant residency targets, self-eviction first (see §3.5) |

These are unmeasured; the numbers must come from the benchmark harness
([`../05-operations/01-performance-correctness-security.md` §3](../05-operations/01-performance-correctness-security.md)).
Do not quote an overhead figure from this section.

### 3.4 Front-running risks and correctness invariants

**Front-running.** A tenant that can force reclaim of *another tenant's*
pages gains two advantages: it can degrade a competitor, and it can race to
observe reused memory if reclaim does not scrub. Rule: reclaim only within
the requesting tenant's own residency budget unless a global-capacity policy
explicitly and observably selects a victim (accounted, rate-limited, and
scrubbed before reuse).

**Invariants (must hold for any paging implementation).**

* **I1 — No live PTE to reclaimed memory.** A block may be reallocated only
  after every PTE that references it has been revoked and the revocation is
  visible to all engines.
* **I2 — Revoke implies quiesce.** No engine work may be in flight on a
  block at revoke; the deferred-release hold + channel drain is the
  primitive (`11-physical-backend-guest-wiring.md` §P6b).
* **I3 — Exactly-once completion.** Paging must not duplicate or lose a
  release; the existing replay-suppression rules apply per owner
  ([`06-host-module-and-hybrid-model.md` §4.4](06-host-module-and-hybrid-model.md)).
* **I4 — Owner-stable residency.** A block's backing is owned by one mdev
  (or one intra-guest tenant, under §4); no shared physical page between
  owners without an explicit, tagged share record.
* **I5 — Scrub before reuse.** Reclaimed memory is zeroed (or poison-filled)
  before it can be mapped into any other owner.
* **I6 — Migration carries residency.** Save/restore includes the residency
  map and tags, or the restored device must cold-start the cache.

### 3.5 What vEP must add

* **Per-mdev accounting:** resident/pinned bytes, fault-in counts, hold
  timeouts — the counters in `struct sema_svc`
  (`kernel/gsp_sema_service.c:439-676`) made per owner.
* **Reclaim signal:** a guest→vEP hint that a range is evictable (vLLM knows
  when a KV block is freed): an allocation/free observed at RM-object level
  in the stock-compatible design, or an explicit control in the tagged
  design (§4.5).
* **QoS policy:** per-mdev weights/slices on top of the GSP scheduler
  ([`10-interaction-and-scheduling.md` §7](10-interaction-and-scheduling.md)),
  with measured properties. `[design]`
* **Capacity admission:** enforce the oversubscription ratio at create and
  large-allocation time; refuse (fail closed) rather than promise capacity.

---

## 4. Custom GMMU protections for paged-attention multi-tenancy

> This is the core ask: given a tenant ID and an agent ID, protect a
> paged-attention KV cache shared by many tenants inside one guest. The
> design below is complete on paper across the full stack. Stages M6–M8
> are **post-1.0 research**: dual-gated, duplicate-not-replace, never
> required by the baseline. Stages M1–M5 belong to the 1.0 programme.

### 4.1 The workload, precisely

Public knowledge of `paged_attention_v2`: the KV cache is a pool of
fixed-size **blocks**; each sequence has a **block table** (a GPU tensor of
block indices) mapping its logical positions to physical blocks. Decode
kernels gather K/V from the blocks named by the table. A vLLM server
multiplexes many sequences (many logical tenants) in one continuous batch
inside one process and, typically, one CUDA context. Nothing in this
pipeline is hardware-protected: any kernel in the context can dereference
any block table entry, and a buggy or malicious sequence implementation can
read every other tenant's KV cache. With prefix caching, blocks are even
*shared* between sequences (copy-on-write), which is a feature to preserve,
not eliminate.

The assumed identities: `tenantId` identifies the serving principal;
`agentId` identifies the finest unit inside a tenant (a worker/context; in
a single-process vLLM, an engine or an attention backend instance). Both are
provided by the orchestration layer; provisioning them is out of scope.

### 4.2 Hardware GMMU primitives: proven vs hypothesis

What the Turing (TU104) VER2 format and our live code establish (`docs/NN`
below resolves to `../../debug/risc-v-gsp-rm-vm/docs/NN-*.md`):

| Primitive | Status | Evidence |
|---|---|---|
| 4-level page tree; 49-bit VA; 4 KiB/64 KiB/2 MiB pages; dual 16-byte PDE0 | **proven** | tracked decoder `../../debug/risc-v-gsp-rm-vm/docs/43-GMMU-PTE-DECODER.md` §6; `kernel/gsp_sema_service.c:1058-1072` |
| VER2 leaf PTE fields incl. `read_only` (bit 6), `privilege` (5), `encrypted` (4), `atomic_disable` (7), `address_sys/vid`, `kind` | **proven format** | `docs/43` §3 |
| Separate page directory per VA space, published per channel (`SET_PAGE_DIRECTORY` with `hVASpace`/`chId`) | **proven** | `kernel/gsp_rm_emulation.c:3310-3314,3366-3372`; live walks `kernel/gsp_sema_service.c:1174-1312` |
| Directory-level apertures and big-page leaves | **proven** | `docs/43` §3–§6; `svc_pde_aperture`, `kernel/gsp_sema_service.c:1110-1117` |
| **No tenant/ACL/PASID field in the VER2 PTE or PDE** | **proven absence in the format** | the full field list in `docs/43` §3–§5 has no such field; `peer_id` is a fabric peer address selector, not a security domain |
| Read-only PTE *enforcement* (the engine refuses a write through an RO leaf) | **hypothesis** | the format defines the bit; we have never exercised an RO-protected leaf on hardware |
| Encryption-domain isolation (CCS/confidential computing) usable as a tenant boundary | **hypothesis** | format has an `encrypted` bit; CCS object model and semantics are unstudied for us |
| MMU invalidate semantics fine-grained enough for per-block revocation | **hypothesis** | no invalidate path exercised; see `11-physical-backend-guest-wiring.md` §P6b |

**The decisive design fact:** on this generation, **the only proven
hardware isolation primitive is the page directory** (per-context page
tables). Per-PTE ACLs are research-grade and cannot be assumed. Therefore
the design must place every security-relevant decision in the *topology*
(which directory a page appears in, which channel may name it) plus
*software validation and audit* in the vEP. The `read_only` bit is a proven
format field and the right mechanism for *sharing without write access*
(e.g., shared prefix blocks), but its enforcement needs the M7 experiment
before we depend on it.

### 4.3 The full stack

The requested chain, end to end:

```
┌─────────────────────────────────────────────────────────────────────────────┐
│ GUEST                                                                       │
│                                                                             │
│  extended vLLM (serving engine)                                             │
│    ├─ per-tenant sequences + block tables (unchanged algorithms)            │
│    ├─ agent-aware KV allocator: blocks come from per-agent VA pools         │
│    └─ calls oiv-agentd for identity + sharing decisions                     │
│           │ Unix socket (guest-local)                                       │
│  oiv-agentd (NEW guest utility; application-level, no driver patch in A)    │
│    ├─ resolves (tenantId, agentId) -> VA pool + tag records                 │
│    ├─ Design A: publishes tag records to a mailbox in device memory         │
│    └─ Design B: issues new RM controls through the guest KMD               │
│           │ CUDA driver API (stock)      │ RM control (B)                   │
│  guest KMD (stock nvidia.ko; unmodified in A, extended in B [research])     │
│           │ GSP-RM RPC / MMIO                                               │
├───────────┼─────────────────────────────────────────────────────────────────┤
│ vEP HOST (vgsp.ko)                                                          │
│    GSP-RM emulation: controls, objects, page-directory records              │
│    tag registry (per mdev): tenant/agent -> VA ranges, access, shares       │
│    page-table policy: owner checks at SET_PAGE_DIRECTORY / map / submit     │
│    channel binding: channel -> (tenant, agent); arbitration + completion    │
│    fault attribution: fault -> channel -> tenant; owner-only notifier       │
│           │ real page tables / forwarded work                               │
│ GPU: GMMU page directories (topology) + PTE bits (read_only, kind)          │
└─────────────────────────────────────────────────────────────────────────────┘
```

### 4.4 Design A — VA-subtree partitioning (no guest KMD change)

This is the **stock-compatible** design and the research entry point: it is
the only design that needs no guest KMD change, but its prototype is M6 —
post-1.0, dual-gated, and it does not shape baseline 1.0 code (§7.4). It
relies on one idea: **the VA space is the
security domain, and a VA is self-describing because its prefix encodes its
owner.**

* Extended vLLM reserves a disjoint VA range per agent: `cuMemAddressReserve`
  (stock CUDA VMM API) at a base derived from `(tenantId, agentId)`, e.g.
  a deterministic partition of the GPU VA space into tenant subtrees and
  agent subtrees. KV blocks are `cuMemCreate`d and mapped only inside the
  owner's range.
* `oiv-agentd` publishes the binding records (which `(tenantId, agentId)`
  owns which VA range, with what access) to a **tag mailbox**: a device
  allocation whose content the vEP can read. In emulation mode this works
  because guest vidmem *is* host RAM (`kernel/bar_init.c:68-77`; FB
  aperture reads are direct, `svc_read_ap` `kernel/gsp_sema_service.c:786-799`),
  so the vEP can scan the mailbox page through the owner's shadow. A fixed
  mailbox offset (requested from the extension at init) keeps discovery
  trivial.
* vEP policy, enforced when walking/populating page tables and at
  work-submission time:
  1. a page directory is owned by the mdev (L3) and, under this design, by
     a `(tenantId, agentId)` (L7);
  2. a PTE whose VA is in tenant T's subtree may only point at memory
     tagged T (or shared with T);
  3. a channel bound to agent A may only execute work referencing VAs in
     A's subtree (validation at decode/submit, using the existing
     owner-aware claim of §2.2);
  4. violations are refused and attributed to the owning channel.
* Sharing for prefix caching: a shared block lives in the *owner's* range
  and is exposed to the borrower through a new PTE in the borrower's
  directory with `read_only` set (proven format field) — or, until M7
  proves RO enforcement, not exposed at all and prefix sharing is
  downgraded to per-agent copies.

Limits, stated honestly:

* It does not protect against a malicious guest KMD (the KMD builds the
  page tables; it can lie about prefixes). It protects tenants from each
  other *within* the tenant's trust model, which is the vLLM threat model.
* Emulation-mode mailbox discovery does not survive real VRAM backing
  (the host cannot read real VRAM as RAM); the real-VRAM version needs a
  protocol path (Design B) or a host-readable staging mailbox.
* It requires extended vLLM (not stock vLLM) — an application change, not
  a driver change.

### 4.5 Design B — explicit tenant/agent RM controls `[research]`

The explicit design: new vGPU-private RM controls flow through the guest
KMD to the vEP; the vEP stores tags, enforces policy, and can build real
page tables from them. This requires a guest KMD that knows the controls;
therefore it is **post-1.0** (it modifies the guest, which the baseline
guest rule forbids, [`../06-portability/hardware-software-matrix.md` §3](../06-portability/hardware-software-matrix.md)),
dual-gated, and never a 1.0 dependency.

Proposed controls (vGPU-private control range; the proven vGPU-specific
example is `0x90f10106`, `COPY_SERVER_RESERVED_PDES`,
`kernel/gsp_rm_emulation.c:2266`):

| Control | ID (proposed) | Semantics |
|---|---|---|
| `NV90F1_CTRL_CMD_VGPU_TENANT_BIND` | `0x90f10120` | bind the calling client to a `(tenantId, agentId)`; idempotent join; optional exclusive |
| `NV90F1_CTRL_CMD_VGPU_TENANT_TAG_RANGE` | `0x90f10121` | tag a GPU VA range with owner + access + volatility |
| `NV90F1_CTRL_CMD_VGPU_TENANT_UNTAG_RANGE` | `0x90f10122` | remove a tag; range becomes untouchable by non-owners |
| `NV90F1_CTRL_CMD_VGPU_TENANT_SHARE` | `0x90f10123` | grant a borrower a (revocable) view of an owner range |
| `NV90F1_CTRL_CMD_VGPU_AGENT_CHANNEL_BIND` | `0x90f10124` | bind an RM channel to an agent for arbitration/completion attribution |
| `NV90F1_CTRL_CMD_VGPU_TAG_QUERY` | `0x90f10125` | read-only: effective identity, enforcement mode, range tags |

Struct sketches (wire types follow the driver's `NvU32`/`NvU64`/`NvHandle`
conventions; all little-endian, 8-byte aligned):

```c
/* 0x90f10120 */
typedef struct NV_VGPU_TENANT_BIND_PARAMS {
    NvU64 tenantId;      /* stable, VM-scoped; 0 invalid */
    NvU64 agentId;       /* 0 = tenant's default agent */
    NvU32 flags;         /* CREATE 0x1 | JOIN 0x2 | EXCLUSIVE 0x4 */
    NvU32 abiVersion;    /* 1 */
    NvU8  name[32];      /* diagnostic only */
} NV_VGPU_TENANT_BIND_PARAMS;

/* 0x90f10121 / 0x90f10122 */
typedef struct NV_VGPU_TENANT_TAG_RANGE_PARAMS {
    NvU64 tenantId;
    NvU64 agentId;
    NvU64 gpuVa;         /* page aligned */
    NvU64 size;
    NvU32 access;        /* READ 0x1 | WRITE 0x2 (0 invalid) */
    NvU32 flags;         /* SHARED 0x1 | EVICTABLE 0x2 | PINNED 0x4 |
                            KV_BLOCK 0x8 | BLOCK_TABLE 0x10 */
    NvU64 tagCookie;     /* opaque allocator cookie; echoed in faults */
} NV_VGPU_TENANT_TAG_RANGE_PARAMS;

/* 0x90f10123 */
typedef struct NV_VGPU_TENANT_SHARE_PARAMS {
    NvU64 ownerTenantId, ownerAgentId;
    NvU64 borrowerTenantId, borrowerAgentId;
    NvU64 gpuVa;         /* range start as tagged by the owner */
    NvU64 size;
    NvU32 access;        /* borrower view: READ | READ|WRITE */
    NvU32 flags;         /* REVOCABLE 0x1 (default) | COW 0x2 */
} NV_VGPU_TENANT_SHARE_PARAMS;

/* 0x90f10124 */
typedef struct NV_VGPU_AGENT_CHANNEL_BIND_PARAMS {
    NvHandle hChannel;
    NvU64 tenantId, agentId;
    NvU32 flags;         /* EXCLUSIVE 0x1 */
    NvU32 abiVersion;    /* 1 */
} NV_VGPU_AGENT_CHANNEL_BIND_PARAMS;
```

Design A's mailbox record (the stock-compatible carrier):

```c
typedef struct NV_VGPU_TAG_MAILBOX_RECORD {
    NvU32 magic;         /* 'OIV1' = 0x4f495631 */
    NvU32 recordType;    /* 1 bind, 2 tag, 3 untag, 4 share */
    NvU64 seq;           /* producer sequence; consumed strictly in order */
    NvU64 tenantId, agentId;
    NvU64 gpuVa, size;
    NvU32 access, flags;
    NvU64 tagCookie;
    NvU32 crc32;         /* over the record with crc32 field = 0 */
} NV_VGPU_TAG_MAILBOX_RECORD;
```

Message-semantics rules: controls are synchronous (reply = status/params);
tag records are monotonic (later record for a range wins; untag terminal);
shares are revocable, revocation delivered to the borrower's event object;
every mutating control is forwarded to the vEP, whose reply is authoritative
— not the echo/zeroed-success fallback used for unknown IDs today
(`kernel/gsp_rm_emulation.c:2829`; `11-physical-backend-guest-wiring.md`
§2.1 #7, #13).

### 4.6 Where enforcement lives

| Layer | Mechanism | Strength |
|---|---|---|
| Guest application (vLLM + agentd) | VA-pool discipline, block-table construction | advisory; defeated by a buggy kernel |
| Guest KMD (Design B) | control validation, identity binding, forwarding | local; guest-trusted |
| vEP tag registry + policy | owner checks at directory record / map / submit / completion | **the enforcement point** |
| GMMU topology | separate page directory (+ optional sub-directory per agent) | hardware-enforced for context separation |
| GMMU PTE bits | `read_only` for shares; `kind` for memory type | hardware-enforced *if* M7 confirms |
| Completion/fault paths | owner-keyed resolution and attribution | software, load-bearing |
| Audit | periodic owner-vs-PTE audit; canary pages | detection; catches policy escapes |

### 4.7 The three granularities

#### 4.7.1 Per-KV-block (page/PTE level)

*Tag every block; enforce at map time; share with read-only where hardware
allows.*

| Aspect | Assessment |
|---|---|
| Benefits | finest sharing (prefix cache reuse across agents); per-block revoke for eviction; fault granularity is the block |
| Drawbacks | per-block metadata and per-map validation cost; vLLM allocates blocks in bursts — validation must be batched or the map path becomes the bottleneck; sharing needs RO enforcement (M7) or degrades to copies |
| Performance | map-time cost per block (currently an emulated RPC is tens of µs — `10-interaction-and-scheduling.md` §6 — and tagging must not add a round trip per block); no extra TLB misses if topology is unchanged, but per-block VA fragmentation reduces big-page opportunities, which *does* cost TLB reach |
| Implementation cost | high: vLLM allocator change, agentd, mailbox/controls, vEP registry, audit |
| Portability | the *concept* is portable; the enforcement is software+topology, because no PTE tenant field exists (proven absence, §4.2); `read_only` mapping must be re-derived per generation |
| Failure modes | stale tag after reallocation (block freed, VA reused); share revoked but a borrower PTE survives (no invalidate, I1 violation); tag/PTE divergence after migration; audit misses a crafted PTE |

#### 4.7.2 Per-tenant (sub-address-space: separate channel + page directory)

*One tenant = one page directory + at least one channel; agents inside a
tenant share the directory (and are separated only by VA-subtree convention
unless per-agent directories are added).*

| Aspect | Assessment |
|---|---|
| Benefits | **the only proven hardware lever** (§4.2); low steady-state overhead (matches the driver's own per-context model); naturally portable; fault and completion attribution are trivial (directory owner = tenant) |
| Drawbacks | no fine-grained sharing; a tenant is a scheduling unit (context switches); TLB working set per tenant; intra-tenant agents are NOT isolated |
| Performance | launch path unchanged; extra contexts cost context-switch and TLB refill; no per-block map overhead |
| Implementation cost | medium: owner keys in the vEP (§2.1–2.6) plus per-mdev backend clients; Design A needs the allocator/agentd work only for the VA pools |
| Portability | high: object model and directory-per-context survive driver-version and (probably) generation changes; exact PTE encoding does not matter |
| Failure modes | directory reuse on handle collision (today's bug class); a page accidentally mapped into two directories; one tenant's RPC wedge stalls others (single backend session); eviction invalidating a sibling's PTE without quiesce |

#### 4.7.3 Per-agent (finest grouping inside a tenant)

*One agent = one channel; optionally one sub-directory per agent within the
tenant directory.*

| Aspect | Assessment |
|---|---|
| Benefits | fault containment per agent (a crashed agent context does not kill siblings); QoS per agent; completion attributed per agent; the natural mapping of CUDA contexts/streams |
| Drawbacks | more channels/contexts = more scheduler state and context switches; per-agent directories multiply TLB pressure and page-table memory; vLLM must be structured so each agent is its own context (a serving-engine change) |
| Performance | overhead grows with agent count; launch unchanged; arbitration and completion bookkeeping per channel |
| Implementation cost | medium–high: channel binding, arbitration policy, per-agent counter pools; vLLM work |
| Portability | high (channels/streams are stock); optional per-agent directories depend on the guest building them (trusted guest) or the host owning page tables (Design B/E1) |
| Failure modes | misbinding a channel to the wrong agent; handle collisions across agents (the ISSUE-0002 class); a shared channel silently breaks agent isolation; completion counters reused across agents |

**Degradation ladder** (what you lose when you can only do a coarser
grain):

```
per-KV-block + RO shares   best sharing, software-enforced identity, needs M7
        │  (no RO hw -> drop sharing, keep tagging)
per-tenant directories     real hardware context separation; agents inside
        │                  a tenant share a directory -> intra-tenant
        │                  isolation is software-only
per-agent channels         real fault/QoS separation at channel granularity,
        │                  no per-block sharing
per-mdev (1.0 baseline)    real VM isolation; intra-guest tenants are
                           application-trusted only
```

### 4.8 Failure modes across all granularities

* **TOCTOU on guest-owned tables.** The guest can rewrite its PTEs between
  validation and execution. Mitigation: host-owned page tables for protected
  ranges (server-owned contract exists: `COPY_SERVER_RESERVED_PDES`,
  `kernel/gsp_rm_emulation.c:2237-2262`; real population is P6c,
  `11-physical-backend-guest-wiring.md` §2.1 #9/#10) or validate at
  submission and hold the entry (deferred-release hold).
* **Identity laundering.** An agent claims another's `(tenantId, agentId)`.
  The identity source is provisioning (guest-local Unix socket in A; KMD
  control validation in B); the vEP binds identity to the *client/context*
  at bind time, and a second bind for the same range from a different
  client is refused.
* **Stale-share escape.** Revoke without invalidate leaves a borrower PTE.
  Invariant I1/I2; until invalidate exists, shares must be configured as
  copy-on-write or non-revocable-at-runtime (drained between batches).
* **Handle collision.** Two guests/agents pick the same RM handles —
  observed. Owner-keyed tables + per-client reset (issue-number-2).
* **Migration inconsistency.** Saved state without tags/residency can
  reattach pages to the wrong owner. Invariant I6.

---

## 5. End-to-end walked example: two tenants, `paged_attention_v2`, one guest

### 5.1 Setup

One guest, one mdev. Two tenants: **T1** with agent **A1** (model X), **T2**
with agent **A2** (model Y). Each runs its own serving engine process; the
extended vLLM allocates KV blocks from the agent's VA pool. `oiv-agentd`
binds `(T1,A1)` and `(T2,A2)` and publishes tag records for their pools
into the mailbox (Design A) or issues `TENANT_BIND`/`TAG_RANGE` (Design B).
Channels: A1 and A2 each own a compute channel and a copy channel; both are
bound to their agent (`AGENT_CHANNEL_BIND`).

```
GUEST VA SPACE (one directory per tenant under Design B, or per agent)
  T1 subtree / pool        T2 subtree / pool
  ┌───────────────┐        ┌───────────────┐
  │ block B11 ... │        │ block B21 ... │
  └───────────────┘        └───────────────┘
  block tables (int32)     block tables (int32)
  ├─ seq s1: [B11,B12,..]  ├─ seq s3: [B21,B22,..]
  └─ seq s2: [B13,..]      └─ seq s4: [B21,...]  (prefix share? B21 RO)
```

### 5.2 Flow (happy path)

```mermaid
sequenceDiagram
  autonumber
  participant V1 as vLLM engine T1
  participant V2 as vLLM engine T2
  participant AG as oiv-agentd
  participant KMD as guest KMD (stock in A)
  participant EP as vEP host (vgsp.ko)
  participant MMU as GMMU page tables
  participant GPU as engines

  V1->>AG: register (T1, A1); request VA pool
  V2->>AG: register (T2, A2); request VA pool
  AG->>EP: tag records: T1/A1 -> VA range R1, T2/A2 -> R2 (mailbox or RM ctrl)
  EP->>EP: registry: owner(R1)=T1/A1, owner(R2)=T2/A2
  V1->>KMD: cuMemCreate/cuMemMap KV blocks inside R1
  V2->>KMD: cuMemCreate/cuMemMap KV blocks inside R2
  KMD->>EP: SET_PAGE_DIRECTORY (recorded, owner = mdev)
  EP->>MMU: (backend) PTEs for R1 only in T1 dir, R2 only in T2 dir
  V1->>KMD: launch decode kernel on A1 channel (block table in R1)
  V2->>KMD: launch decode kernel on A2 channel (block table in R2)
  EP->>EP: submit-time check: channel A1 -> VA in R1 only; A2 -> R2 only
  EP->>GPU: forward work
  GPU-->>EP: completions
  EP-->>V1: release -> A1's semaphores only
  EP-->>V2: release -> A2's semaphores only
```

### 5.3 Malicious attempt, and which layer stops it

Scenario: tenant T1 crafts a decode kernel whose block table contains a VA
inside R2, or directly dereferences an R2 VA.

1. **Application layer (none).** vLLM's own bookkeeping does not stop it —
   this is the gap being fixed.
2. **GMMU topology.** In T1's page directory there is **no PTE** for an R2
   VA. The engine's access faults (invalid leaf / unmapped). Under
   per-tenant directories this is the primary stop; the fault carries the
   faulting VA and the channel.
3. **vEP submit-time policy.** Before forwarding, the decoded work is
   checked against the channel's allowed VA set (owner map + tags). The
   submission is refused and counted *before* any engine access is
   programmed; this also covers accesses that would otherwise appear legal
   because of a stale PTE.
4. **vEP map-time policy.** If T1 instead tries to *map* an R2-owned block
   into R1 (e.g., by getting the guest KMD to place a PTE pointing at
   T2's memory through a shared directory), the tag registry rejects the
   mapping: owner(R2 physical page) ≠ T1. Refused at population time.
5. **Fault attribution.** The fault (from step 2/3) is attributed to
   `(mdev, channel)` → `(T1, A1)`; the notifier and event go only to A1's
   event object; T2's channels and completions are untouched. A1's context
   is the only thing torn down (policy: kill the offending agent, not the
   server).
6. **Audit.** A periodic owner-vs-PTE audit (canary pages at pool
   boundaries, checksummed) detects any escape that steps 2–4 missed and
   fails closed for the owning tenant.

What this stop costs: a fault is ms-class (engine fault → status queue →
vEP relay → guest notifier), and the refused submission costs one
validation pass at decode time (class: µs, same order as the existing
per-entry decode, `10-interaction-and-scheduling.md` §6). Neither number
is measured multi-tenant yet.

### 5.4 Degradation: if only per-tenant separation is available

Assume the hardware/format experiments (M6–M7) do not land and we ship
only per-tenant directories plus owner-keyed software:

* T1 and T2 remain separated **if and only if** they are in different
  tenants **and** each tenant's agents do not share a directory (the
  deployable unit is one agent per tenant, or per-agent contexts).
* If two agents inside one tenant share a directory (the common vLLM
  single-process case), the cross-agent guarantee downgrades to
  **app-level**: block tables are validated at submit time against the
  agent's VA pool, but a kernel dereferencing a sibling's VA resolves (the
  PTE exists). The stop becomes detection (audit) + kill, not prevention.
* Prefix-cache sharing across agents degrades to copies (no proven RO
  enforcement), costing VRAM and copy bandwidth.
* Honest recommendation for the degraded mode: **one agent per tenant, one
  tenant per context** (vLLM tenant count = context count) — a
  configuration contract, not a hardware guarantee.

---

## 6. Portability

### 6.1 What is portable vs not

| Artifact | Portable? | Why / what to re-derive |
|---|---|---|
| Object model (client, device, vaspace, channel, events) | **yes** | stable across 550–580 as a family; verified live only on 570.144 |
| Page-directory contract (`SET_PAGE_DIRECTORY`/`UNSET`, `COPY_SERVER_RESERVED_PDES`) | **yes, versioned** | controls carry `hVASpace`/`chId`/geometry; re-verify field offsets per version |
| Our GMMU walker/decoder approach | **yes** | decode captured tables; field catalogue per generation (Turing VER2 proven; others re-derive) |
| PTE/PDE bit encodings | **no** | generation-specific (VER2 on Turing; newer formats expected on Blackwell); re-derive from captures + format docs per device |
| Invalidate semantics / fetch-line behavior | **no** | measured per implementation; current 128-byte fetch-line, no-invalidate result is Turing+our-path-specific |
| Tag controls / mailbox (this design) | **concept yes; wire no** | new controls are per-version additions; mailbox depends on guest vidmem being host-readable (emulation only) |
| Per-tenant/per-agent policy | **yes** | policy is ours; mechanism depends on the above |
| mdev type/VRAM sizes | **yes** | per-type metadata registered by the host driver; sizes are policy, not hardware |

### 6.2 Driver 610

RM 610 is under a **hard gate**: no acquisition, analysis, emulation edit,
physical work or support claim before explicit named-human approval
(`../../debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md`). Nothing here
authorizes 610 work; the port plan for this design is M8, gated. Expected
to change and need re-derivation when approved: control inventories and
object model extensions, engine caps, the accelerator/CCS object family,
and possibly the GMMU entry format
(`../../debug/risc-v-gsp-rm-vm/docs/34-BLACKWELL-SOURCES.md`).

### 6.3 Grace-Hopper / Grace-Blackwell / GB10

Targets per the matrix
([`../06-portability/hardware-software-matrix.md` §2](../06-portability/hardware-software-matrix.md));
what matters here:

* **aarch64 host.** Page-size/cache-attribute differences are a known risk
  ([`hardware-software-matrix.md` §8](../06-portability/hardware-software-matrix.md));
  the VA↔PA discovery technique (guest direct-map arithmetic,
  [`06-host-module-and-hybrid-model.md` §4.2](06-host-module-and-hybrid-model.md))
  must be re-verified.
* **Memory hierarchy.** Coherent NVLink-C2C blurs sysmem/vidmem; the
  aperture model and the §3 oversubscription design must be re-derived
  around C2C coherence. `[hypothesis]`
* **GMMU formats.** Entry formats/page sizes are expected to differ;
  anything per-PTE (including a possible ACL-like field) must be re-derived
  before being relied on. `[hypothesis]`
* **Confidential computing (CCS).** CoCo changes the trust boundary in our
  favour (hardware memory encryption) and against us: the host cannot read
  guest memory content, which breaks the content-scan discovery techniques
  in the current memory model ([`06-host-module-and-hybrid-model.md`
  §4.4](06-host-module-and-hybrid-model.md)). The `encrypted` PTE bit
  exists (proven format); semantics are unstudied. `[hypothesis]`
* **MIG** on datacenter parts: an alternative hardware partitioning
  substrate the policy layer can use where available
  ([`09-reference/glossary-and-faq.md`](../09-reference/glossary-and-faq.md),
  "What about MIG?"). `[design]`

### 6.4 Re-derivation checklist (per new device/version)

Capture page tables and decode them; confirm the encoding and write down
explicitly whether any owner/ACL-like field exists. Confirm the directory
is still per-context and the controls still carry the VA space and channel.
Confirm invalidate semantics with a controlled experiment (no-op until
proven; M5/M7). Re-run the owner-keying review against the new control/RPC
inventory, then the two-guest gate and the §5 walkthrough, attaching
sessions and M1–M6-style tables per the portability discipline
(`../../debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md`).

---

## 7. Current and future challenges, tooling gaps, roadmap

### 7.1 Current challenges

1. **The service-state split is the immediate blocker.** `gsp_sema_service.c`
   is global-boot-state code (single `svc`, global counters, value-keyed
   tables); splitting it per mdev is review-heavy, with latent assumptions
   like the ISSUE-0002 root cause.
2. **GR execution is upstream of every engine-isolation claim.** Until guest
   compute executes on real engines (`../04-implementation/03-execution-frontier.md`),
   tenant engine isolation can only be validated on the emulation plane.
3. **No per-tenant fault injection.** The backend's `gsp_phys_fault_inject`
   (`06-host-module-and-hybrid-model.md` §6) is tied to one session; there is
   no "inject as tenant B, prove A unaffected".
4. **No GMMU permission-probe methodology.** The `read_only` bit is a proven
   format field but unexercised, and we have no owner/ACL search protocol
   beyond decode-and-eyeball.
5. **No two-guest harness.** Tools assume one VM/one mdev
   (`../../scripts/vgpu_vm.py`); the 1.0 gate needs a two-VM driver.
6. **No side-channel measurement** — no methodology, no data; explicitly
   deferred to measurement by
   [`../05-operations/01-performance-correctness-security.md` §4](../05-operations/01-performance-correctness-security.md).
7. **Real VRAM backing is not implemented** (`06-host-module-and-hybrid-model.md`
   §8): today isolation-by-address is over emulated RAM; real DMA, real page
   tables and shared scratch pools arrive with the backend
   (`11-physical-backend-guest-wiring.md` §P6b).
8. **vLLM integration cost.** The extension touches allocator/block
   internals; budget a pinned fork plus a thin plugin, not an upstream bet.

### 7.2 What our tools do not address today

`docs/NN` below = `../../debug/risc-v-gsp-rm-vm/docs/NN-*.md`.

| Existing tool (public) | Useful for | Gap |
|---|---|---|
| GMMU PTE decoder (`../../debug/risc-v-gsp-rm-vm/docs/43`) | decode captured entries; field catalogue | decode-only; no permission semantics, no walker dereference |
| GMMU oracle workflow (`docs/44`) | compare simulated vs observed register/value traces | no enforcement experiment; oracle capture is supervised-human |
| Validation predicate probe (`docs/47`) / surgical rehost (`docs/46`) | run one firmware validation branch offline with crafted state | firmware-side; not a GMMU permission probe (candidate substrate, needs a design) |
| Control discovery (`docs/19`) / reach (`docs/19`) | hidden control IDs; symbol reachability | simulator surface only; not real-RM |
| Host debug parameters (`06-host-module-and-hybrid-model.md` §6) | inspect/poke memory, channels, semaphores | single-device oriented; scans are device-selected by first-open |
| Two-guest harness / tenant fault injector / side-channel harness | — | **none of the three exists** |

### 7.3 Roadmap

Each stage has an observable acceptance criterion; no stage advances on
intention. **[1.0]** = baseline programme; **[research]** = post-1.0,
dual-gated, duplicate-not-replace, never required by the baseline.

| Stage | Scope | Acceptance (measurable) | Class |
|---|---|---|---|
| **M1** | Owner-key every vEP table; split `svc` into per-mdev contexts (or keyed tables); per-mdev pump/service; per-client reset | two concurrent mdevs both get service (no first-open starvation); crafted handle/VA collision test fails closed; ISSUE-0002 sequential acceptance stays green; one-guest regression suite green | [1.0] |
| **M2** | Two-guest gate harness + the 2G-01…2G-09 matrix (§2.9) | all tests pass 3× consecutive with evidence; ladder rung 5 "isolation" checklist partially met (fairness still open) | [1.0] |
| **M3** | Per-mdev real GSP clients; per-tenant RPC accounting and bounded rates; head-of-line measurements | two guests' forwarded CE work both completes; p99 cross-tenant delay measured and within the published bound; no cross-tenant completion (canaries) | [1.0, after GR/backend] |
| **M4** | Independent VRAM sizes end to end + sparse backing v1 + capacity admission | two mdevs with different types; sum(declared) > capacity with refusal-only admission; per-mdev accounting counters; no leak under 20 start/stop cycles | [1.0-adjacent] |
| **M5** | Real invalidate path + eviction/reclaim for KV blocks; residency invariants I1–I6 tested | controlled page-out/page-in cycle with hardware work verified byte-exact; no PTE outlives its block; hold timeouts bounded | [1.0-adjacent, needs backend] |
| **M6** | Design A prototype: extended vLLM + `oiv-agentd` + mailbox; VA-subtree policy in the vEP | cross-tenant VA access faults (T2 block from T1 channel) and is attributed to T1; differential test shows no foreign block readable/writable; overhead vs unextended vLLM measured and published | [research] |
| **M7** | GMMU permission/predicate probe: supervised experiment for `read_only` (and any ACL-like field, per generation) | a controlled write through an RO leaf behaves as documented (or the bit is declared unusable); result recorded with sessions and the fidelity label | [research] |
| **M8** | Design B (new RM controls + guest KMD extension) and/or CCS/610 port, only when approved | controls round-trip with enforcement at map and submit; fault attribution per agent; 610/CCS work follows its own gate (`docs/23`) | [research, gated] |

Kill criteria (so research can end): M6 dies if tagging overhead exceeds
the project's sync-heavy factor without a security gain over M1–M5 (which
deliver VM/tenant separation with stock drivers); M7 dies on a family with
no RO-equivalent and no owner field if the capture shows no enforcement
(per-PTE protection impossible; the design then rests on topology, which
M6 must state); M8 dies if M6 meets the same guarantee within the
baseline's envelope and no CCS requirement materialises.

### 7.4 Baseline discipline (mandatory)

Per `AGENTS.md` index entry 11 and
`../10-post-1.0/experimental-host-acceleration.md`, **post-1.0 goals are
not design inputs**: nothing in sections 4.5–4.8, 5.3's research stops, or
stages M6–M8 may shape baseline code (no "just in case" fields, hooks or
parameters). Research stages are dual-gated (compile-time + module-load
flags), duplicate rather than replace, and are never required by the
baseline build or 1.0 acceptance. The 1.0 programme owns M1–M5 only; M6+
starts after 1.0 goals are met, on an explicit human decision.

---

## 8. Cross-links

* [`13-vexecution-plane.md`](13-vexecution-plane.md) — vEP vocabulary and
  the execution plane this document polices (written in parallel).
* [`05-gmmu-amap-hal-gr.md`](05-gmmu-amap-hal-gr.md) — GMMU, page tables,
  channels, scheduling; the hardware background for §4.
* [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md) —
  backend ownership, isolation target (§4.8), open questions;
  [`11-physical-backend-guest-wiring.md`](11-physical-backend-guest-wiring.md) —
  the guest-wiring seams (P6a/P6b/P6c) and risk guards this design needs.
* [`10-interaction-and-scheduling.md`](10-interaction-and-scheduling.md) —
  concurrency, ownership rules, multi-tenant scheduling model (§7);
  [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) —
  memory model, VA↔PA discovery, debug parameters, exactly-once rules.
* [`../05-operations/01-performance-correctness-security.md`](../05-operations/01-performance-correctness-security.md) —
  evaluation ladder (rung 5), threat model, measurement discipline;
  [`../06-portability/hardware-software-matrix.md`](../06-portability/hardware-software-matrix.md) —
  support ladder, new-device/version procedures, guest rule;
  [`../01-orientation/01-purpose-and-scope.md` §5](../01-orientation/01-purpose-and-scope.md) —
  multi-tenancy model.
* `../../debug/issue/issue-number-1.md`, `../../debug/issue/issue-number-2.md` — the
  CUDA reproducibility and per-client state issues M1 resolves;
  `../../debug/risc-v-gsp-rm-vm/docs/{43-GMMU-PTE-DECODER,44-GMMU-ORACLE-AND-WORKFLOW,23-RM610-READINESS}.md` —
  tooling and gates cited above.

## Change log

| Date | Change |
|---|---|
| 2026-10-08 | Created: multi-mdev service-state ownership rules, isolation-layer model, VRAM oversubscription design, paged-attention tenant/agent protection stack (Design A/B, three granularities), walked example, portability, M1–M8 roadmap. Tree `66388560`. |
| 2026-10-08 | Review pass: status hash refreshed to `master @ 1f571dda`; drifted line anchors updated for the post-`66388560` code (pump loop, doorbell, `svc_sem_write`); the vEP vocabulary aligned with doc 13 (13 is the vocabulary authority, this document the GMMU-policy authority); Design A classified as the research entry point (M6), not a 1.0 programme item. |