diff options
| author | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-09-29 15:29:00 -0400 |
|---|---|---|
| committer | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-09-29 15:29:00 -0400 |
| commit | 1fdbf4ea8c17dab2b8fca6b5e58c2a6d3bc7596d (patch) | |
| tree | cd16ecce1bbbf8e71c17091df843441965f871c1 /docs/03-architecture | |
| parent | f4803e303e1e4bfde9f241ec3b6226399d01825d (diff) | |
| download | open-iov-1fdbf4ea8c17dab2b8fca6b5e58c2a6d3bc7596d.tar.gz open-iov-1fdbf4ea8c17dab2b8fca6b5e58c2a6d3bc7596d.zip | |
docs: record physical-GPU integration mechanism and refresh project status
- add docs/03-architecture/08-physical-gpu-integration.md: interfaces that
exist, mechanism options A-E, chosen path (module-owned GPU driven through
its real GSP firmware), sharing policy, decision record
- update 06-host-module-and-hybrid-model, 02-practical-challenges, system
overview, known-knowns, docs/README reading order and CUDA-side docs to
match
- correct stale engine-topology/COPY14 blocker claims across the tree
- add STATE-CURRENT.md as the dated public status snapshot
Diffstat (limited to 'docs/03-architecture')
| -rw-r--r-- | docs/03-architecture/01-system-overview.md | 27 | ||||
| -rw-r--r-- | docs/03-architecture/02-guest-stack-and-cudasw.md | 12 | ||||
| -rw-r--r-- | docs/03-architecture/03-gsp-and-gsp-rm.md | 11 | ||||
| -rw-r--r-- | docs/03-architecture/04-uvm.md | 13 | ||||
| -rw-r--r-- | docs/03-architecture/05-gmmu-amap-hal-gr.md | 9 | ||||
| -rw-r--r-- | docs/03-architecture/06-host-module-and-hybrid-model.md | 47 | ||||
| -rw-r--r-- | docs/03-architecture/07-driver-lifecycle-field-guide.md | 15 | ||||
| -rw-r--r-- | docs/03-architecture/08-physical-gpu-integration.md | 175 |
8 files changed, 251 insertions, 58 deletions
diff --git a/docs/03-architecture/01-system-overview.md b/docs/03-architecture/01-system-overview.md index c7a847e..5c2ba6c 100644 --- a/docs/03-architecture/01-system-overview.md +++ b/docs/03-architecture/01-system-overview.md @@ -61,10 +61,12 @@ HOST MODULE open-iov-vfio.ko ├─ emulation core: GSP boot + GSP-RM (RPC msgq, controls, objects, events) ├─ memory model: guest RAM access, VA↔PA maps, VRAM backing └─ work path: channel/push forwarding to the physical GPU [goal; software bridge today] - │ -HOST DRIVER nvidia.ko (unmodified) and/or direct GPU interfaces - │ -PHYSICAL GPU + │ module-owned device + real GSP session +HOST BACKEND the GPU's own hardware + GSP firmware, driven by this module + │ (the distribution nvidia.ko stays unmodified and unbound in + │ this mode; see 08-physical-gpu-integration.md) + v +REAL HARDWARE (engines, memory, interrupts) ``` ## 3. Host module architecture @@ -167,11 +169,12 @@ decompiled and mapped (see `wiki/GSP-570.144W-Firmware-Analysis.md` and `SUCCESS` with zeroed params, and `GSP_RM_ALLOC` echoes the request params without server-side object management. Where the driver consumes a reply value, that value is still being corrected (the reply-fidelity ladder). -5. `nvidia-smi` works; CUDA user-space starts (`cuInit` / `cuDeviceGet` - succeed). Context creation is currently blocked: `cuCtxCreate` returns - `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks the - async copy engine (`COPY14`), so UVM's channel-manager and page-tree - initialization is as far as the guest gets. +5. `nvidia-smi -L` enumerates the emulated device and CUDA user-space starts + (`cuInit` / `cuDeviceGetCount` / `cuDeviceGet` succeed). The client reaches + `cuCtxCreate`, which currently ends in a forced `rc=999` on a completion + wait; the copy-engine method streams execute in software as a bridge, and + the execution backend is not built yet (see + [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)). The CUDA-side contract and the reply-fidelity rules are documented in [`../../userspace/cuda/docs/03-driver-and-gpu-interface.md` §9](../../userspace/cuda/docs/03-driver-and-gpu-interface.md) @@ -192,7 +195,9 @@ That means the host module must: faults and semaphores back. The design for this, including what is known and what remains to be learned, is -in: +in [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) and -[`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md). +[`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md); +the chosen execution backend is specified in +[`08-physical-gpu-integration.md`](08-physical-gpu-integration.md). diff --git a/docs/03-architecture/02-guest-stack-and-cudasw.md b/docs/03-architecture/02-guest-stack-and-cudasw.md index 115f639..e841342 100644 --- a/docs/03-architecture/02-guest-stack-and-cudasw.md +++ b/docs/03-architecture/02-guest-stack-and-cudasw.md @@ -89,11 +89,13 @@ open /dev/nvidia* → cuInit → UVM registration (per process) The initialization path is where compatibility is won or lost; the runtime path is where *execution* is won or lost. Open-IOV's current focus is -pre-forwarding correctness: `cuInit` / `cuDeviceGet` succeed, but context -creation returns `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine -list lacks the async copy engine (`COPY14`); the forwarding decision -(host-RM proxy vs direct submission) is still ahead (see -[`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md)). +pre-forwarding correctness: `cuInit` / `cuDeviceGet` succeed, and the client +reaches context creation, which currently ends in a forced `rc=999` on a +completion wait; the engine inventory is complete and the forwarding decision +is made — a module-owned GPU running its real GSP firmware as the +resource-manager server (see +[`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md) +and [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)). For the CUDA userspace side of this stack — its architecture, how it is built and executes, and the exact kernel/GPU interface contract we must satisfy — diff --git a/docs/03-architecture/03-gsp-and-gsp-rm.md b/docs/03-architecture/03-gsp-and-gsp-rm.md index aca908f..8809bb7 100644 --- a/docs/03-architecture/03-gsp-and-gsp-rm.md +++ b/docs/03-architecture/03-gsp-and-gsp-rm.md @@ -186,17 +186,18 @@ is not lost. | GSP boot state machine | ✅ | stages complete; the driver's poll loop is satisfied | | Message queue transport | ✅ | including the non-contiguous page table for the queue buffer | | RPC request/reply | ✅ | all functions the driver uses during init are answered; unknown functions get a success reply with no payload | -| Static info tables | 🟡 | device/FB/BUS/INTR/GMMU tables carry plausible TU104 values; the engine inventory is still incomplete (the async copy engine, COPY14, is missing) | +| Static info tables | 🟡 | device/FB/BUS/INTR/GMMU tables carry plausible TU104 values; the engine inventory is now complete (the async copy engine is included) and agrees with the class list | | Controls during init | 🟡 | ~160 distinct control IDs observed per CUDA run; 23 are special-cased with real values, the rest fall through to a success reply with zeroed parameters - consumed fields of those are still being corrected | | Object creation | 🟡 | handles/parameters are echoed consistently (the request parameters must be echoed back - zeroing them corrupts the caller), but there is no server-side object model: allocation is acknowledged, not tracked | | Page tables for server-owned ranges | 🟡 | contract received and recorded; leaf population is the forwarding work | | Event/notification delivery | 🟡 | MSI-X works; richer event semantics as forwarding lands | | Firmware *behaviour* (routing, scheduling, real execution) | 🔴 | the forwarding work; see the implementation docs | -The current CUDA blocker is the engine-inventory row: `libcuda` filters the -engine list it receives against the class list and expects the async copy -engine to be present, because the secondary (async) engine record is otherwise -zeroed. See +The engine-inventory row was the CUDA blocker at the time of writing: +`libcuda` filters the engine list it receives against the class list and +expects the async copy engine to be present, because the secondary (async) +engine record is otherwise zeroed. That inventory is now complete; the +context-creation wait is later, in the completion path. See [`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md) section 3.7 for the offsets and the consumed fields, and [`../../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../../userspace/cuda/docs/03-driver-and-gpu-interface.md) diff --git a/docs/03-architecture/04-uvm.md b/docs/03-architecture/04-uvm.md index b879572..4c8a0c9 100644 --- a/docs/03-architecture/04-uvm.md +++ b/docs/03-architecture/04-uvm.md @@ -50,17 +50,20 @@ Each stage has a distinct failure signature; our debugging docs map them. | Requirement | Detail | Our status | |---|---|---| | GPU identity | must match the uuid from RM | ✅ | -| CE capabilities | per-engine support bits; UVM *requires* `supported && !grce && sysmem` for CPU_TO_GPU/GPU_TO_CPU channels | 🟡 the caps we answer are accepted, but the engine list / CE capabilities are missing the async copy engine (`COPY14`) — the current context-creation blocker | +| CE capabilities | per-engine support bits; UVM *requires* `supported && !grce && sysmem` for CPU_TO_GPU/GPU_TO_CPU channels | 🟡 the caps we answer are accepted and the inventory now includes the async copy engine; completion/semaphore plumbing is the current frontier | | Channel creation | RM channel objects with GPFIFO/userd/instance parameters | ✅ objects created | | Pushes executed | UVM's first action on a new channel is an *init push* followed by a semaphore-tracked wait | 🟡 the host decodes method streams and executes memsets, copies and semaphore releases in software (a bridge); no kernel is forwarded to the physical GPU yet | | Page trees | UVM initializes page trees with its own pushes; the *server-owned* ranges must be populated by us | 🟡 | | Fault delivery | fault buffer + notification; replay/cancel semantics | 🔴 | | Semantics of completions | tracker values must advance exactly | 🟡 | -**Current blocker (2026-09-23):** `cuCtxCreate` returns `rc=999` -(`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks the async copy -engine (`COPY14`); teardown of the failed context still oopses in the guest -kernel in `uvm_pmm_devmem_page_to_va_space` (twice per boot). The CUDA-side +**Current frontier (2026-09-29):** the client reaches `cuCtxCreate` and +currently ends in a forced `rc=999` on a completion wait; the engine inventory +is complete, and the remaining work is completion translation plus the chosen +execution backend +([`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)). +Failed-context teardown has produced guest kernel oopses in +`uvm_pmm_devmem_page_to_va_space` in earlier runs. The CUDA-side contract and reply rules are in [`../../userspace/cuda/docs/03-driver-and-gpu-interface.md` §9](../../userspace/cuda/docs/03-driver-and-gpu-interface.md) and diff --git a/docs/03-architecture/05-gmmu-amap-hal-gr.md b/docs/03-architecture/05-gmmu-amap-hal-gr.md index 6111f54..76d4200 100644 --- a/docs/03-architecture/05-gmmu-amap-hal-gr.md +++ b/docs/03-architecture/05-gmmu-amap-hal-gr.md @@ -186,10 +186,11 @@ engine type (e.g. `COPY0 = 0x9`, `GR = 0x1`). driver builds its per-engine records from the device/engine information the kernel driver reports, and filters them against the class list; a missing or zeroed record is not ignored - it becomes an allocation with class `0` and the -whole context setup rolls back. The current CUDA bring-up blocker is exactly -this: the async copy engine (COPY14, engine descriptor `0x793ceb0e`, instance -14) is absent from the emulated engine list, while a bare-metal oracle reports -it as the seventh engine alongside GR and CE0..4. See +whole context setup rolls back. This was exactly the CUDA bring-up blocker +until the engine inventory was completed: the async copy engine (COPY14, +engine descriptor `0x793ceb0e`, instance 14) was absent from the emulated +engine list, while a bare-metal oracle reports it as the seventh engine +alongside GR and CE0..4; it is now included. See [`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md) section 3.7 for the record layout the driver's engine walker consumes. diff --git a/docs/03-architecture/06-host-module-and-hybrid-model.md b/docs/03-architecture/06-host-module-and-hybrid-model.md index 15545a4..61be96c 100644 --- a/docs/03-architecture/06-host-module-and-hybrid-model.md +++ b/docs/03-architecture/06-host-module-and-hybrid-model.md @@ -18,10 +18,12 @@ │ open-iov-vfio.ko │ │ virtual device ── emulation core ── memory model ── work path │ └────┬─────────────────────────────────────┬─────────────────────────┘ - │ host driver interfaces │ (future) vendor-neutral - ┌────┴────────────────────┐ │ - │ unmodified host nvidia.ko│ direct GPU interfaces / VF backends - └──────────────────────────┘ + │ module-owned device + real GSP │ (future) vendor-managed + ┌────┴───────────────────────────────┐ │ sharing backends + │ physical GPU and its GSP firmware, │ vGPU host driver / SR-IOV + │ driven by this module (the host │ (licensed / hardware-gated) + │ driver stays unmodified, unbound) │ + └────────────────────────────────────┘ ``` The module registers itself as an **mdev parent** on the physical GPU and @@ -103,20 +105,24 @@ The handlers currently cover: the boot/static-info exchanges, the device/FB/ BUS/INTR control tables, CE capability and PCE-mask queries, GMMU static info, vaspace reserved-PDE contracts, channel/CE object allocation, and the general "unknown control/function returns success with zeroed output" fallback. The -same fallback now matters for engine inventory: `libcuda` filters the engine -list against the class list, so a missing async copy engine (COPY14) leaves a -record zeroed and blocks context creation (`cuCtxCreate` returns `999`); see +same fallback mattered for engine inventory: `libcuda` filters the engine +list against the class list, and an incomplete inventory once left a record +zeroed and blocked context creation (`cuCtxCreate` returned `999`); the +emulated inventory is now complete (the async copy engine included). See [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md) and [`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md) -section 3.7. +section 3.7 for the analysis. **Current work-execution bridge.** The module also decodes push buffers as method streams and executes the work it understands in software - memsets, copies and semaphore releases - and completes the corresponding semaphores. That is a correctness bridge for memory operations, not the data plane: no -CUDA kernel is forwarded to the physical GPU yet, and the forwarding strategy -(host-RM proxy first, direct submission if needed) is in +CUDA kernel is forwarded to the physical GPU yet; the chosen execution backend +(a module-owned GPU running its real GSP firmware as the resource-manager +server) is in +[`08-physical-gpu-integration.md`](08-physical-gpu-integration.md), with the +trade-offs in [`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) section 3. @@ -205,18 +211,14 @@ guest push submitted → relay completions (semaphore/fault/interrupt) back into guest memory ``` -Three forwarding backends are candidates, in increasing order of capability: - -1. **Host RM/GSP proxy** — use the unmodified host driver's RM to allocate real - memory and create real channels, then replay guest work through them. -2. **Direct engine submission** — build our own channel/push streams to real - engines using the GMMU/HAL definitions, bypassing the host driver. -3. **Hybrid** — proxy for management (memory, channels) and direct submission - for hot-path work (grid launches, copies). - -The decision between them is driven by the practical challenges document; the -short version is that (1) is safest for correctness, (2) is best for latency, -and both must be isolated to be acceptable. +Backend selection, including the interfaces that actually exist and why the +distribution host driver cannot be called, is analyzed in +[`08-physical-gpu-integration.md`](08-physical-gpu-integration.md). In short: +proxying through the host driver has no callable entry point from kernel +context, so the chosen mechanism is for the module to own the device and run +its real GSP firmware as the resource-manager server, while the emulation path +remains the control plane. Software execution of decoded memory/state +operations stays a development bridge, not the data plane. ## 6. Debug tooling in the module @@ -290,5 +292,6 @@ in `kernel-open` first). | Push decoding + semaphore completion | demonstrated live; being productized | | Page-table population for server-owned ranges | designed, not implemented | | GPU work forwarding (GR/CE execution) | designed, not implemented | +| Execution backend (module-owned GPU + direct real-GSP session) | chosen, not started ([`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)) | | Real VRAM backing | designed, not implemented | | Windows guest, Xen transport, aarch64 hosts | designed, not implemented | diff --git a/docs/03-architecture/07-driver-lifecycle-field-guide.md b/docs/03-architecture/07-driver-lifecycle-field-guide.md index 9de5edb..1bfe912 100644 --- a/docs/03-architecture/07-driver-lifecycle-field-guide.md +++ b/docs/03-architecture/07-driver-lifecycle-field-guide.md @@ -105,7 +105,7 @@ fallback). Representative controls: | `0x20802a02` | CE PCE mask | one bit per engine | | `0x20802a08` | CE fault method buffer size | method buffer sizing | | `0x90f10106` | vaspace reserved-PDE contract | **carries page-table physical addresses** (see GSP-RM doc) | -| `0x20800123` / `0x20800170` | `GPU_GET_ENGINES` / `GET_ENGINES_V2` | engine inventory (filtered by the class list / classDB); values are consumed — the guest must see the full engine list, including the async copy engine (`COPY14`), and the current `rc=999` blocker lives here | +| `0x20800123` / `0x20800170` | `GPU_GET_ENGINES` / `GET_ENGINES_V2` | engine inventory (filtered by the class list / classDB); values are consumed — the guest must see the full engine list, including the async copy engine (`COPY14`); that inventory is now complete, and the context-creation wait lives later in the completion path | | `0x800201` | `GPU_GET_CLASSLIST` | class list (classDB); must agree with the engine list and the CE caps controls | | `0x2080012b` | GPU_PROMOTE_CTX (560-byte params) | issued immediately before context creation; fallback generic `SUCCESS`, consumed fields not yet coherent | | `0x20803039` | NVLink masks/VBIOS info | fallback: generic `SUCCESS` with zeroed params | @@ -166,13 +166,16 @@ their memsets, copies and semaphore releases in software by translating the semaphore VA to a guest physical address (see the UVM and host-module docs). This is a bridge, not forwarding: no work reaches the physical GPU yet. -**Live development front (2026-09-23):** CUDA context creation — `cuCtxCreate` -returns `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks -the async copy engine (`COPY14`). The engine-record layout and the alloc -reply-fidelity checks are the first things to inspect +**Live development front (2026-09-29):** CUDA context creation — the client +reaches `cuCtxCreate` and currently ends in a forced `rc=999` on a completion +wait; the engine inventory is complete and the completion/translation path is +the focus. The engine-record layout and the alloc reply-fidelity checks are +documented in ([`../../userspace/cuda/docs/05-570.144-decompilation.md` §3.3/§3.6](../../userspace/cuda/docs/05-570.144-decompilation.md); start at -[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md)). +[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md)); +the execution backend is in +[`08-physical-gpu-integration.md`](08-physical-gpu-integration.md). **Log signatures (guest side):** ``` diff --git a/docs/03-architecture/08-physical-gpu-integration.md b/docs/03-architecture/08-physical-gpu-integration.md new file mode 100644 index 0000000..f156429 --- /dev/null +++ b/docs/03-architecture/08-physical-gpu-integration.md @@ -0,0 +1,175 @@ +# Physical GPU Integration: How Work Reaches Real Hardware + +> **Status: chosen mechanism, not yet implemented.** The control plane works +> today (the guest's driver initializes and CUDA user-space reaches context +> creation); the guest's *work* is currently executed in software as a +> development bridge. This document records the interfaces that actually +> exist, the mechanism options, the chosen path, and what has to be built. +> It fills the gap left open by +> [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) §5 +> and [`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) §3, +> which named "the host driver's RM API" as an option without a concrete call +> path from kernel context. + +## 1. The problem and its constraints + +The guest's real work (copies, memory operations, grid launches) must execute +on the physical GPU. The project constraints are: + +* one DKMS kernel module on the host; no daemon, coprocess or script is + required for the guest to run; +* host and guest drivers unmodified; no paravirtual enlightenment in the + guest — it believes it owns a physical GPU; +* all emulation and translation logic inside the module; +* real execution: compute must not be emulated in software. + +Any mechanism must therefore be reachable from kernel context, with no new +host userspace, and without editing the host driver. + +## 2. What interfaces actually exist + +| Interface | What it offers | Reachable from our module? | +|---|---|---| +| Host driver (distribution build) | full resource management, scheduling, memory | **No.** It exports no general RM entry points to third-party modules, has no kernel-client programming interface, and its device ioctl surface is user-pointer based. Intercepting its internal functions is unsupported (and function tracing on its text is not available). | +| Host driver peer-memory exports | pin/map memory for peer devices | Partially: **memory only, no submission**. | +| Open-kernel-module build | source linkage | Only when the host driver is built from source and extended — i.e. a modified host driver; excluded by the constraints. | +| Vendor vGPU host stack | mediated work submission and scheduling | Exists, but requires the licensed vGPU host branch and a vGPU-aware guest — excluded by "no enlightenment / no GRID licensing". | +| SR-IOV virtual functions | hardware-partitioned device | Not available on this GPU generation. | +| The GPU's own GSP firmware | the resource manager server itself | **Yes**, once the module owns the device: this is the path used. | + +The conclusion: with an unmodified host driver there is **no callable +submission interface into it**. Real execution therefore requires the module +to own the device and talk to the hardware itself. + +## 3. Mechanism options + +``` + (A) Proxy through the host driver + guest push -> our module -> host driver's RM -> real channel -> GPU + Not implementable without a modified host driver or a vendor-managed + interface (see section 2). + + (B) Own the GPU; own scheduling (no GSP) + guest push -> our module -> our own channels/runlists/page tables -> GPU + Possible in principle, but the module would reproduce the scheduler, + MMU ownership and context switching; large and capability-limiting. + + (C) Hybrid + Only meaningful once (A) and (B) both exist; not on the critical path. + + (D) Own the GPU; use its real GSP firmware as the server <- chosen + guest push -> our module -> real GSP-RM session -> real channel -> GPU + The module becomes the host-side resource-manager client: the same + protocol the guest driver speaks, used against the physical GPU's + firmware. Scheduling, page tables and context management stay the + firmware's job. + + (E) Vendor-managed sharing (vGPU host stack, SR-IOV VFs) + Deployment-gated (licence, hardware generation); out of scope for + consumer GPUs. +``` + +## 4. The chosen mechanism in detail + +### 4.1 Ownership and lifecycle + +The module binds the physical GPU as its driver. The distribution driver stays +unmodified and is unbound from that device while the vGPU service is active; +host display continues on another GPU (integrated graphics on the development +system). This is the dedicated-GPU deployment model; see section 5. An earlier +host-side GSP boot path existed in early versions and was disabled precisely +because it conflicted with the distribution driver owning the card — under +exclusive ownership that conflict disappears. + +### 4.2 Device bring-up + +PCI enable, BAR mapping, firmware load (secure boot stage, booter, RISC-V GSP), +WPR2 setup, message queues in host system memory, doorbell, version exchange — +the same observable stages the guest emulation already models, with the module +now on the client side. + +### 4.3 Host-side resource-manager client + +Using the wire protocol and RPC inventory already mapped for the supported +family, the module allocates real objects: device/subdevice, virtual address +spaces, channels, and memory mappings. This replaces "call the host driver's +RM" with "be the RM client". + +### 4.4 Guest mediation + +The guest-facing GSP-RM emulation stays as it is for everything the guest +observes. Objects that need real resources are translated into real GSP-RM +requests (handle-domain translation, allocation backing). The page-table +contract the guest already provides for server-owned ranges becomes +population of the real MMU instead of a shadow. + +### 4.5 Work path + +Decoded guest pushes are rebuilt into real push streams on real channels +(where shapes match, passed through); guest GPU virtual addresses are mapped +to the same physical pages in the real MMU; the real doorbell is rung and the +GSP firmware schedules the work. + +### 4.6 Completion relay + +Hardware-completed semaphores land in memory the guest already polls. Events +and faults that the guest expects as interrupts are relayed through the +existing MSI-X injection path. + +### 4.7 Isolation + +One real GSP client per mdev, per-client page tables; host access to guest +memory stays behind the hypervisor's DMA interface; the firmware provides +per-context scheduling. Tenant policy and fairness are a later layer on top. + +## 5. Sharing policy + +* **Dedicated GPU (chosen):** one GPU serves the vGPU service; the host uses + another GPU or integrated graphics. +* **Multi-GPU host:** one GPU for the distribution driver, one for the + service. +* **Shared single GPU with the host driver bound:** no conforming mechanism + exists on this generation without vendor interfaces; this would require + relaxing a constraint (vendor vGPU stack, SR-IOV hardware, or a host + daemon). + +## 6. Status and milestones + +| Step | Status | +|---|---| +| Control plane (GSP boot emulation, RPC, BAR/PCI/MSI-X) | done | +| Software-executed bridge for decoded memory/state operations | in progress (memsets, copies, releases) | +| Device ownership + real GSP bring-up by the module | not started (the early prototype is to be rebuilt under exclusive ownership) | +| Host-side resource-manager client for real objects/channels | not started | +| Work translation and real submission | not started | +| Completion relay through real hardware | not started | +| CUDA kernel execution (`vectorAdd`) | not started | + +## 7. Open questions + +* Firmware/host driver ABI match for the supported family on the development + GPU. +* GPU unbind/reset lifecycle and recovery after guest faults. +* Which guest allocations need real VRAM and which can stay in system memory. +* Fault semantics when work runs on the real firmware. +* Per-tenant scheduling policy once multiple mdevs share the service GPU. + +## 8. Decision record + +* **Date:** 2026-09-29. +* **Decision:** adopt option (D) — a module-owned GPU with a direct real-GSP + session as the execution backend. (A) is recorded as not implementable under + the constraints; (B) remains a fallback only if (D) proves unworkable; + (E) is deployment-gated and out of scope. +* **Rationale:** only (D) satisfies module-only, no-daemon, unmodified + drivers, real execution, and a path to multi-tenant sharing. +* **Consequences:** the module grows a host-side resource-manager client and a + device-ownership lifecycle; the emulation path remains the control plane and + is reused. + +## See also + +- [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) - the host module, memory model, build contracts +- [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) - known gaps and plans +- [`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) - forwarding trade-offs +- [`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md) - the CUDA-side view |
