summaryrefslogtreecommitdiffstats
path: root/docs/03-architecture
diff options
context:
space:
mode:
authorArthur Hanson Rasmusson <arthur@vgpu.io>2026-09-29 15:29:00 -0400
committerArthur Hanson Rasmusson <arthur@vgpu.io>2026-09-29 15:29:00 -0400
commit1fdbf4ea8c17dab2b8fca6b5e58c2a6d3bc7596d (patch)
treecd16ecce1bbbf8e71c17091df843441965f871c1 /docs/03-architecture
parentf4803e303e1e4bfde9f241ec3b6226399d01825d (diff)
downloadopen-iov-1fdbf4ea8c17dab2b8fca6b5e58c2a6d3bc7596d.tar.gz
open-iov-1fdbf4ea8c17dab2b8fca6b5e58c2a6d3bc7596d.zip
docs: record physical-GPU integration mechanism and refresh project status
- add docs/03-architecture/08-physical-gpu-integration.md: interfaces that exist, mechanism options A-E, chosen path (module-owned GPU driven through its real GSP firmware), sharing policy, decision record - update 06-host-module-and-hybrid-model, 02-practical-challenges, system overview, known-knowns, docs/README reading order and CUDA-side docs to match - correct stale engine-topology/COPY14 blocker claims across the tree - add STATE-CURRENT.md as the dated public status snapshot
Diffstat (limited to 'docs/03-architecture')
-rw-r--r--docs/03-architecture/01-system-overview.md27
-rw-r--r--docs/03-architecture/02-guest-stack-and-cudasw.md12
-rw-r--r--docs/03-architecture/03-gsp-and-gsp-rm.md11
-rw-r--r--docs/03-architecture/04-uvm.md13
-rw-r--r--docs/03-architecture/05-gmmu-amap-hal-gr.md9
-rw-r--r--docs/03-architecture/06-host-module-and-hybrid-model.md47
-rw-r--r--docs/03-architecture/07-driver-lifecycle-field-guide.md15
-rw-r--r--docs/03-architecture/08-physical-gpu-integration.md175
8 files changed, 251 insertions, 58 deletions
diff --git a/docs/03-architecture/01-system-overview.md b/docs/03-architecture/01-system-overview.md
index c7a847e..5c2ba6c 100644
--- a/docs/03-architecture/01-system-overview.md
+++ b/docs/03-architecture/01-system-overview.md
@@ -61,10 +61,12 @@ HOST MODULE open-iov-vfio.ko
├─ emulation core: GSP boot + GSP-RM (RPC msgq, controls, objects, events)
├─ memory model: guest RAM access, VA↔PA maps, VRAM backing
└─ work path: channel/push forwarding to the physical GPU [goal; software bridge today]
- │
-HOST DRIVER nvidia.ko (unmodified) and/or direct GPU interfaces
- │
-PHYSICAL GPU
+ │ module-owned device + real GSP session
+HOST BACKEND the GPU's own hardware + GSP firmware, driven by this module
+ │ (the distribution nvidia.ko stays unmodified and unbound in
+ │ this mode; see 08-physical-gpu-integration.md)
+ v
+REAL HARDWARE (engines, memory, interrupts)
```
## 3. Host module architecture
@@ -167,11 +169,12 @@ decompiled and mapped (see `wiki/GSP-570.144W-Firmware-Analysis.md` and
`SUCCESS` with zeroed params, and `GSP_RM_ALLOC` echoes the request params
without server-side object management. Where the driver consumes a reply
value, that value is still being corrected (the reply-fidelity ladder).
-5. `nvidia-smi` works; CUDA user-space starts (`cuInit` / `cuDeviceGet`
- succeed). Context creation is currently blocked: `cuCtxCreate` returns
- `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks the
- async copy engine (`COPY14`), so UVM's channel-manager and page-tree
- initialization is as far as the guest gets.
+5. `nvidia-smi -L` enumerates the emulated device and CUDA user-space starts
+ (`cuInit` / `cuDeviceGetCount` / `cuDeviceGet` succeed). The client reaches
+ `cuCtxCreate`, which currently ends in a forced `rc=999` on a completion
+ wait; the copy-engine method streams execute in software as a bridge, and
+ the execution backend is not built yet (see
+ [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)).
The CUDA-side contract and the reply-fidelity rules are documented in
[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md` §9](../../userspace/cuda/docs/03-driver-and-gpu-interface.md)
@@ -192,7 +195,9 @@ That means the host module must:
faults and semaphores back.
The design for this, including what is known and what remains to be learned, is
-in:
+in
[`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md)
and
-[`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md).
+[`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md);
+the chosen execution backend is specified in
+[`08-physical-gpu-integration.md`](08-physical-gpu-integration.md).
diff --git a/docs/03-architecture/02-guest-stack-and-cudasw.md b/docs/03-architecture/02-guest-stack-and-cudasw.md
index 115f639..e841342 100644
--- a/docs/03-architecture/02-guest-stack-and-cudasw.md
+++ b/docs/03-architecture/02-guest-stack-and-cudasw.md
@@ -89,11 +89,13 @@ open /dev/nvidia* → cuInit → UVM registration (per process)
The initialization path is where compatibility is won or lost; the runtime
path is where *execution* is won or lost. Open-IOV's current focus is
-pre-forwarding correctness: `cuInit` / `cuDeviceGet` succeed, but context
-creation returns `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine
-list lacks the async copy engine (`COPY14`); the forwarding decision
-(host-RM proxy vs direct submission) is still ahead (see
-[`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md)).
+pre-forwarding correctness: `cuInit` / `cuDeviceGet` succeed, and the client
+reaches context creation, which currently ends in a forced `rc=999` on a
+completion wait; the engine inventory is complete and the forwarding decision
+is made — a module-owned GPU running its real GSP firmware as the
+resource-manager server (see
+[`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md)
+and [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)).
For the CUDA userspace side of this stack — its architecture, how it is built
and executes, and the exact kernel/GPU interface contract we must satisfy —
diff --git a/docs/03-architecture/03-gsp-and-gsp-rm.md b/docs/03-architecture/03-gsp-and-gsp-rm.md
index aca908f..8809bb7 100644
--- a/docs/03-architecture/03-gsp-and-gsp-rm.md
+++ b/docs/03-architecture/03-gsp-and-gsp-rm.md
@@ -186,17 +186,18 @@ is not lost.
| GSP boot state machine | ✅ | stages complete; the driver's poll loop is satisfied |
| Message queue transport | ✅ | including the non-contiguous page table for the queue buffer |
| RPC request/reply | ✅ | all functions the driver uses during init are answered; unknown functions get a success reply with no payload |
-| Static info tables | 🟡 | device/FB/BUS/INTR/GMMU tables carry plausible TU104 values; the engine inventory is still incomplete (the async copy engine, COPY14, is missing) |
+| Static info tables | 🟡 | device/FB/BUS/INTR/GMMU tables carry plausible TU104 values; the engine inventory is now complete (the async copy engine is included) and agrees with the class list |
| Controls during init | 🟡 | ~160 distinct control IDs observed per CUDA run; 23 are special-cased with real values, the rest fall through to a success reply with zeroed parameters - consumed fields of those are still being corrected |
| Object creation | 🟡 | handles/parameters are echoed consistently (the request parameters must be echoed back - zeroing them corrupts the caller), but there is no server-side object model: allocation is acknowledged, not tracked |
| Page tables for server-owned ranges | 🟡 | contract received and recorded; leaf population is the forwarding work |
| Event/notification delivery | 🟡 | MSI-X works; richer event semantics as forwarding lands |
| Firmware *behaviour* (routing, scheduling, real execution) | 🔴 | the forwarding work; see the implementation docs |
-The current CUDA blocker is the engine-inventory row: `libcuda` filters the
-engine list it receives against the class list and expects the async copy
-engine to be present, because the secondary (async) engine record is otherwise
-zeroed. See
+The engine-inventory row was the CUDA blocker at the time of writing:
+`libcuda` filters the engine list it receives against the class list and
+expects the async copy engine to be present, because the secondary (async)
+engine record is otherwise zeroed. That inventory is now complete; the
+context-creation wait is later, in the completion path. See
[`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md)
section 3.7 for the offsets and the consumed fields, and
[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../../userspace/cuda/docs/03-driver-and-gpu-interface.md)
diff --git a/docs/03-architecture/04-uvm.md b/docs/03-architecture/04-uvm.md
index b879572..4c8a0c9 100644
--- a/docs/03-architecture/04-uvm.md
+++ b/docs/03-architecture/04-uvm.md
@@ -50,17 +50,20 @@ Each stage has a distinct failure signature; our debugging docs map them.
| Requirement | Detail | Our status |
|---|---|---|
| GPU identity | must match the uuid from RM | ✅ |
-| CE capabilities | per-engine support bits; UVM *requires* `supported && !grce && sysmem` for CPU_TO_GPU/GPU_TO_CPU channels | 🟡 the caps we answer are accepted, but the engine list / CE capabilities are missing the async copy engine (`COPY14`) — the current context-creation blocker |
+| CE capabilities | per-engine support bits; UVM *requires* `supported && !grce && sysmem` for CPU_TO_GPU/GPU_TO_CPU channels | 🟡 the caps we answer are accepted and the inventory now includes the async copy engine; completion/semaphore plumbing is the current frontier |
| Channel creation | RM channel objects with GPFIFO/userd/instance parameters | ✅ objects created |
| Pushes executed | UVM's first action on a new channel is an *init push* followed by a semaphore-tracked wait | 🟡 the host decodes method streams and executes memsets, copies and semaphore releases in software (a bridge); no kernel is forwarded to the physical GPU yet |
| Page trees | UVM initializes page trees with its own pushes; the *server-owned* ranges must be populated by us | 🟡 |
| Fault delivery | fault buffer + notification; replay/cancel semantics | 🔴 |
| Semantics of completions | tracker values must advance exactly | 🟡 |
-**Current blocker (2026-09-23):** `cuCtxCreate` returns `rc=999`
-(`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks the async copy
-engine (`COPY14`); teardown of the failed context still oopses in the guest
-kernel in `uvm_pmm_devmem_page_to_va_space` (twice per boot). The CUDA-side
+**Current frontier (2026-09-29):** the client reaches `cuCtxCreate` and
+currently ends in a forced `rc=999` on a completion wait; the engine inventory
+is complete, and the remaining work is completion translation plus the chosen
+execution backend
+([`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)).
+Failed-context teardown has produced guest kernel oopses in
+`uvm_pmm_devmem_page_to_va_space` in earlier runs. The CUDA-side
contract and reply rules are in
[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md` §9](../../userspace/cuda/docs/03-driver-and-gpu-interface.md)
and
diff --git a/docs/03-architecture/05-gmmu-amap-hal-gr.md b/docs/03-architecture/05-gmmu-amap-hal-gr.md
index 6111f54..76d4200 100644
--- a/docs/03-architecture/05-gmmu-amap-hal-gr.md
+++ b/docs/03-architecture/05-gmmu-amap-hal-gr.md
@@ -186,10 +186,11 @@ engine type (e.g. `COPY0 = 0x9`, `GR = 0x1`).
driver builds its per-engine records from the device/engine information the
kernel driver reports, and filters them against the class list; a missing or
zeroed record is not ignored - it becomes an allocation with class `0` and the
-whole context setup rolls back. The current CUDA bring-up blocker is exactly
-this: the async copy engine (COPY14, engine descriptor `0x793ceb0e`, instance
-14) is absent from the emulated engine list, while a bare-metal oracle reports
-it as the seventh engine alongside GR and CE0..4. See
+whole context setup rolls back. This was exactly the CUDA bring-up blocker
+until the engine inventory was completed: the async copy engine (COPY14,
+engine descriptor `0x793ceb0e`, instance 14) was absent from the emulated
+engine list, while a bare-metal oracle reports it as the seventh engine
+alongside GR and CE0..4; it is now included. See
[`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md)
section 3.7 for the record layout the driver's engine walker consumes.
diff --git a/docs/03-architecture/06-host-module-and-hybrid-model.md b/docs/03-architecture/06-host-module-and-hybrid-model.md
index 15545a4..61be96c 100644
--- a/docs/03-architecture/06-host-module-and-hybrid-model.md
+++ b/docs/03-architecture/06-host-module-and-hybrid-model.md
@@ -18,10 +18,12 @@
│ open-iov-vfio.ko │
│ virtual device ── emulation core ── memory model ── work path │
└────┬─────────────────────────────────────┬─────────────────────────┘
- │ host driver interfaces │ (future) vendor-neutral
- ┌────┴────────────────────┐ │
- │ unmodified host nvidia.ko│ direct GPU interfaces / VF backends
- └──────────────────────────┘
+ │ module-owned device + real GSP │ (future) vendor-managed
+ ┌────┴───────────────────────────────┐ │ sharing backends
+ │ physical GPU and its GSP firmware, │ vGPU host driver / SR-IOV
+ │ driven by this module (the host │ (licensed / hardware-gated)
+ │ driver stays unmodified, unbound) │
+ └────────────────────────────────────┘
```
The module registers itself as an **mdev parent** on the physical GPU and
@@ -103,20 +105,24 @@ The handlers currently cover: the boot/static-info exchanges, the device/FB/
BUS/INTR control tables, CE capability and PCE-mask queries, GMMU static info,
vaspace reserved-PDE contracts, channel/CE object allocation, and the general
"unknown control/function returns success with zeroed output" fallback. The
-same fallback now matters for engine inventory: `libcuda` filters the engine
-list against the class list, so a missing async copy engine (COPY14) leaves a
-record zeroed and blocks context creation (`cuCtxCreate` returns `999`); see
+same fallback mattered for engine inventory: `libcuda` filters the engine
+list against the class list, and an incomplete inventory once left a record
+zeroed and blocked context creation (`cuCtxCreate` returned `999`); the
+emulated inventory is now complete (the async copy engine included). See
[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md)
and
[`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md)
-section 3.7.
+section 3.7 for the analysis.
**Current work-execution bridge.** The module also decodes push buffers as
method streams and executes the work it understands in software - memsets,
copies and semaphore releases - and completes the corresponding semaphores.
That is a correctness bridge for memory operations, not the data plane: no
-CUDA kernel is forwarded to the physical GPU yet, and the forwarding strategy
-(host-RM proxy first, direct submission if needed) is in
+CUDA kernel is forwarded to the physical GPU yet; the chosen execution backend
+(a module-owned GPU running its real GSP firmware as the resource-manager
+server) is in
+[`08-physical-gpu-integration.md`](08-physical-gpu-integration.md), with the
+trade-offs in
[`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md)
section 3.
@@ -205,18 +211,14 @@ guest push submitted
→ relay completions (semaphore/fault/interrupt) back into guest memory
```
-Three forwarding backends are candidates, in increasing order of capability:
-
-1. **Host RM/GSP proxy** — use the unmodified host driver's RM to allocate real
- memory and create real channels, then replay guest work through them.
-2. **Direct engine submission** — build our own channel/push streams to real
- engines using the GMMU/HAL definitions, bypassing the host driver.
-3. **Hybrid** — proxy for management (memory, channels) and direct submission
- for hot-path work (grid launches, copies).
-
-The decision between them is driven by the practical challenges document; the
-short version is that (1) is safest for correctness, (2) is best for latency,
-and both must be isolated to be acceptable.
+Backend selection, including the interfaces that actually exist and why the
+distribution host driver cannot be called, is analyzed in
+[`08-physical-gpu-integration.md`](08-physical-gpu-integration.md). In short:
+proxying through the host driver has no callable entry point from kernel
+context, so the chosen mechanism is for the module to own the device and run
+its real GSP firmware as the resource-manager server, while the emulation path
+remains the control plane. Software execution of decoded memory/state
+operations stays a development bridge, not the data plane.
## 6. Debug tooling in the module
@@ -290,5 +292,6 @@ in `kernel-open` first).
| Push decoding + semaphore completion | demonstrated live; being productized |
| Page-table population for server-owned ranges | designed, not implemented |
| GPU work forwarding (GR/CE execution) | designed, not implemented |
+| Execution backend (module-owned GPU + direct real-GSP session) | chosen, not started ([`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)) |
| Real VRAM backing | designed, not implemented |
| Windows guest, Xen transport, aarch64 hosts | designed, not implemented |
diff --git a/docs/03-architecture/07-driver-lifecycle-field-guide.md b/docs/03-architecture/07-driver-lifecycle-field-guide.md
index 9de5edb..1bfe912 100644
--- a/docs/03-architecture/07-driver-lifecycle-field-guide.md
+++ b/docs/03-architecture/07-driver-lifecycle-field-guide.md
@@ -105,7 +105,7 @@ fallback). Representative controls:
| `0x20802a02` | CE PCE mask | one bit per engine |
| `0x20802a08` | CE fault method buffer size | method buffer sizing |
| `0x90f10106` | vaspace reserved-PDE contract | **carries page-table physical addresses** (see GSP-RM doc) |
-| `0x20800123` / `0x20800170` | `GPU_GET_ENGINES` / `GET_ENGINES_V2` | engine inventory (filtered by the class list / classDB); values are consumed — the guest must see the full engine list, including the async copy engine (`COPY14`), and the current `rc=999` blocker lives here |
+| `0x20800123` / `0x20800170` | `GPU_GET_ENGINES` / `GET_ENGINES_V2` | engine inventory (filtered by the class list / classDB); values are consumed — the guest must see the full engine list, including the async copy engine (`COPY14`); that inventory is now complete, and the context-creation wait lives later in the completion path |
| `0x800201` | `GPU_GET_CLASSLIST` | class list (classDB); must agree with the engine list and the CE caps controls |
| `0x2080012b` | GPU_PROMOTE_CTX (560-byte params) | issued immediately before context creation; fallback generic `SUCCESS`, consumed fields not yet coherent |
| `0x20803039` | NVLink masks/VBIOS info | fallback: generic `SUCCESS` with zeroed params |
@@ -166,13 +166,16 @@ their memsets, copies and semaphore releases in software by translating the
semaphore VA to a guest physical address (see the UVM and host-module docs).
This is a bridge, not forwarding: no work reaches the physical GPU yet.
-**Live development front (2026-09-23):** CUDA context creation — `cuCtxCreate`
-returns `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks
-the async copy engine (`COPY14`). The engine-record layout and the alloc
-reply-fidelity checks are the first things to inspect
+**Live development front (2026-09-29):** CUDA context creation — the client
+reaches `cuCtxCreate` and currently ends in a forced `rc=999` on a completion
+wait; the engine inventory is complete and the completion/translation path is
+the focus. The engine-record layout and the alloc reply-fidelity checks are
+documented in
([`../../userspace/cuda/docs/05-570.144-decompilation.md` §3.3/§3.6](../../userspace/cuda/docs/05-570.144-decompilation.md);
start at
-[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md)).
+[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md));
+the execution backend is in
+[`08-physical-gpu-integration.md`](08-physical-gpu-integration.md).
**Log signatures (guest side):**
```
diff --git a/docs/03-architecture/08-physical-gpu-integration.md b/docs/03-architecture/08-physical-gpu-integration.md
new file mode 100644
index 0000000..f156429
--- /dev/null
+++ b/docs/03-architecture/08-physical-gpu-integration.md
@@ -0,0 +1,175 @@
+# Physical GPU Integration: How Work Reaches Real Hardware
+
+> **Status: chosen mechanism, not yet implemented.** The control plane works
+> today (the guest's driver initializes and CUDA user-space reaches context
+> creation); the guest's *work* is currently executed in software as a
+> development bridge. This document records the interfaces that actually
+> exist, the mechanism options, the chosen path, and what has to be built.
+> It fills the gap left open by
+> [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) §5
+> and [`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) §3,
+> which named "the host driver's RM API" as an option without a concrete call
+> path from kernel context.
+
+## 1. The problem and its constraints
+
+The guest's real work (copies, memory operations, grid launches) must execute
+on the physical GPU. The project constraints are:
+
+* one DKMS kernel module on the host; no daemon, coprocess or script is
+ required for the guest to run;
+* host and guest drivers unmodified; no paravirtual enlightenment in the
+ guest — it believes it owns a physical GPU;
+* all emulation and translation logic inside the module;
+* real execution: compute must not be emulated in software.
+
+Any mechanism must therefore be reachable from kernel context, with no new
+host userspace, and without editing the host driver.
+
+## 2. What interfaces actually exist
+
+| Interface | What it offers | Reachable from our module? |
+|---|---|---|
+| Host driver (distribution build) | full resource management, scheduling, memory | **No.** It exports no general RM entry points to third-party modules, has no kernel-client programming interface, and its device ioctl surface is user-pointer based. Intercepting its internal functions is unsupported (and function tracing on its text is not available). |
+| Host driver peer-memory exports | pin/map memory for peer devices | Partially: **memory only, no submission**. |
+| Open-kernel-module build | source linkage | Only when the host driver is built from source and extended — i.e. a modified host driver; excluded by the constraints. |
+| Vendor vGPU host stack | mediated work submission and scheduling | Exists, but requires the licensed vGPU host branch and a vGPU-aware guest — excluded by "no enlightenment / no GRID licensing". |
+| SR-IOV virtual functions | hardware-partitioned device | Not available on this GPU generation. |
+| The GPU's own GSP firmware | the resource manager server itself | **Yes**, once the module owns the device: this is the path used. |
+
+The conclusion: with an unmodified host driver there is **no callable
+submission interface into it**. Real execution therefore requires the module
+to own the device and talk to the hardware itself.
+
+## 3. Mechanism options
+
+```
+ (A) Proxy through the host driver
+ guest push -> our module -> host driver's RM -> real channel -> GPU
+ Not implementable without a modified host driver or a vendor-managed
+ interface (see section 2).
+
+ (B) Own the GPU; own scheduling (no GSP)
+ guest push -> our module -> our own channels/runlists/page tables -> GPU
+ Possible in principle, but the module would reproduce the scheduler,
+ MMU ownership and context switching; large and capability-limiting.
+
+ (C) Hybrid
+ Only meaningful once (A) and (B) both exist; not on the critical path.
+
+ (D) Own the GPU; use its real GSP firmware as the server <- chosen
+ guest push -> our module -> real GSP-RM session -> real channel -> GPU
+ The module becomes the host-side resource-manager client: the same
+ protocol the guest driver speaks, used against the physical GPU's
+ firmware. Scheduling, page tables and context management stay the
+ firmware's job.
+
+ (E) Vendor-managed sharing (vGPU host stack, SR-IOV VFs)
+ Deployment-gated (licence, hardware generation); out of scope for
+ consumer GPUs.
+```
+
+## 4. The chosen mechanism in detail
+
+### 4.1 Ownership and lifecycle
+
+The module binds the physical GPU as its driver. The distribution driver stays
+unmodified and is unbound from that device while the vGPU service is active;
+host display continues on another GPU (integrated graphics on the development
+system). This is the dedicated-GPU deployment model; see section 5. An earlier
+host-side GSP boot path existed in early versions and was disabled precisely
+because it conflicted with the distribution driver owning the card — under
+exclusive ownership that conflict disappears.
+
+### 4.2 Device bring-up
+
+PCI enable, BAR mapping, firmware load (secure boot stage, booter, RISC-V GSP),
+WPR2 setup, message queues in host system memory, doorbell, version exchange —
+the same observable stages the guest emulation already models, with the module
+now on the client side.
+
+### 4.3 Host-side resource-manager client
+
+Using the wire protocol and RPC inventory already mapped for the supported
+family, the module allocates real objects: device/subdevice, virtual address
+spaces, channels, and memory mappings. This replaces "call the host driver's
+RM" with "be the RM client".
+
+### 4.4 Guest mediation
+
+The guest-facing GSP-RM emulation stays as it is for everything the guest
+observes. Objects that need real resources are translated into real GSP-RM
+requests (handle-domain translation, allocation backing). The page-table
+contract the guest already provides for server-owned ranges becomes
+population of the real MMU instead of a shadow.
+
+### 4.5 Work path
+
+Decoded guest pushes are rebuilt into real push streams on real channels
+(where shapes match, passed through); guest GPU virtual addresses are mapped
+to the same physical pages in the real MMU; the real doorbell is rung and the
+GSP firmware schedules the work.
+
+### 4.6 Completion relay
+
+Hardware-completed semaphores land in memory the guest already polls. Events
+and faults that the guest expects as interrupts are relayed through the
+existing MSI-X injection path.
+
+### 4.7 Isolation
+
+One real GSP client per mdev, per-client page tables; host access to guest
+memory stays behind the hypervisor's DMA interface; the firmware provides
+per-context scheduling. Tenant policy and fairness are a later layer on top.
+
+## 5. Sharing policy
+
+* **Dedicated GPU (chosen):** one GPU serves the vGPU service; the host uses
+ another GPU or integrated graphics.
+* **Multi-GPU host:** one GPU for the distribution driver, one for the
+ service.
+* **Shared single GPU with the host driver bound:** no conforming mechanism
+ exists on this generation without vendor interfaces; this would require
+ relaxing a constraint (vendor vGPU stack, SR-IOV hardware, or a host
+ daemon).
+
+## 6. Status and milestones
+
+| Step | Status |
+|---|---|
+| Control plane (GSP boot emulation, RPC, BAR/PCI/MSI-X) | done |
+| Software-executed bridge for decoded memory/state operations | in progress (memsets, copies, releases) |
+| Device ownership + real GSP bring-up by the module | not started (the early prototype is to be rebuilt under exclusive ownership) |
+| Host-side resource-manager client for real objects/channels | not started |
+| Work translation and real submission | not started |
+| Completion relay through real hardware | not started |
+| CUDA kernel execution (`vectorAdd`) | not started |
+
+## 7. Open questions
+
+* Firmware/host driver ABI match for the supported family on the development
+ GPU.
+* GPU unbind/reset lifecycle and recovery after guest faults.
+* Which guest allocations need real VRAM and which can stay in system memory.
+* Fault semantics when work runs on the real firmware.
+* Per-tenant scheduling policy once multiple mdevs share the service GPU.
+
+## 8. Decision record
+
+* **Date:** 2026-09-29.
+* **Decision:** adopt option (D) — a module-owned GPU with a direct real-GSP
+ session as the execution backend. (A) is recorded as not implementable under
+ the constraints; (B) remains a fallback only if (D) proves unworkable;
+ (E) is deployment-gated and out of scope.
+* **Rationale:** only (D) satisfies module-only, no-daemon, unmodified
+ drivers, real execution, and a path to multi-tenant sharing.
+* **Consequences:** the module grows a host-side resource-manager client and a
+ device-ownership lifecycle; the emulation path remains the control plane and
+ is reused.
+
+## See also
+
+- [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) - the host module, memory model, build contracts
+- [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) - known gaps and plans
+- [`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) - forwarding trade-offs
+- [`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md) - the CUDA-side view