From 1fdbf4ea8c17dab2b8fca6b5e58c2a6d3bc7596d Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Tue, 29 Sep 2026 15:29:00 -0400 Subject: docs: record physical-GPU integration mechanism and refresh project status - add docs/03-architecture/08-physical-gpu-integration.md: interfaces that exist, mechanism options A-E, chosen path (module-owned GPU driven through its real GSP firmware), sharing policy, decision record - update 06-host-module-and-hybrid-model, 02-practical-challenges, system overview, known-knowns, docs/README reading order and CUDA-side docs to match - correct stale engine-topology/COPY14 blocker claims across the tree - add STATE-CURRENT.md as the dated public status snapshot --- .../06-host-module-and-hybrid-model.md | 47 ++++++++++++---------- 1 file changed, 25 insertions(+), 22 deletions(-) (limited to 'docs/03-architecture/06-host-module-and-hybrid-model.md') diff --git a/docs/03-architecture/06-host-module-and-hybrid-model.md b/docs/03-architecture/06-host-module-and-hybrid-model.md index 15545a4..61be96c 100644 --- a/docs/03-architecture/06-host-module-and-hybrid-model.md +++ b/docs/03-architecture/06-host-module-and-hybrid-model.md @@ -18,10 +18,12 @@ │ open-iov-vfio.ko │ │ virtual device ── emulation core ── memory model ── work path │ └────┬─────────────────────────────────────┬─────────────────────────┘ - │ host driver interfaces │ (future) vendor-neutral - ┌────┴────────────────────┐ │ - │ unmodified host nvidia.ko│ direct GPU interfaces / VF backends - └──────────────────────────┘ + │ module-owned device + real GSP │ (future) vendor-managed + ┌────┴───────────────────────────────┐ │ sharing backends + │ physical GPU and its GSP firmware, │ vGPU host driver / SR-IOV + │ driven by this module (the host │ (licensed / hardware-gated) + │ driver stays unmodified, unbound) │ + └────────────────────────────────────┘ ``` The module registers itself as an **mdev parent** on the physical GPU and @@ -103,20 +105,24 @@ The handlers currently cover: the boot/static-info exchanges, the device/FB/ BUS/INTR control tables, CE capability and PCE-mask queries, GMMU static info, vaspace reserved-PDE contracts, channel/CE object allocation, and the general "unknown control/function returns success with zeroed output" fallback. The -same fallback now matters for engine inventory: `libcuda` filters the engine -list against the class list, so a missing async copy engine (COPY14) leaves a -record zeroed and blocks context creation (`cuCtxCreate` returns `999`); see +same fallback mattered for engine inventory: `libcuda` filters the engine +list against the class list, and an incomplete inventory once left a record +zeroed and blocked context creation (`cuCtxCreate` returned `999`); the +emulated inventory is now complete (the async copy engine included). See [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md) and [`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md) -section 3.7. +section 3.7 for the analysis. **Current work-execution bridge.** The module also decodes push buffers as method streams and executes the work it understands in software - memsets, copies and semaphore releases - and completes the corresponding semaphores. That is a correctness bridge for memory operations, not the data plane: no -CUDA kernel is forwarded to the physical GPU yet, and the forwarding strategy -(host-RM proxy first, direct submission if needed) is in +CUDA kernel is forwarded to the physical GPU yet; the chosen execution backend +(a module-owned GPU running its real GSP firmware as the resource-manager +server) is in +[`08-physical-gpu-integration.md`](08-physical-gpu-integration.md), with the +trade-offs in [`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) section 3. @@ -205,18 +211,14 @@ guest push submitted → relay completions (semaphore/fault/interrupt) back into guest memory ``` -Three forwarding backends are candidates, in increasing order of capability: - -1. **Host RM/GSP proxy** — use the unmodified host driver's RM to allocate real - memory and create real channels, then replay guest work through them. -2. **Direct engine submission** — build our own channel/push streams to real - engines using the GMMU/HAL definitions, bypassing the host driver. -3. **Hybrid** — proxy for management (memory, channels) and direct submission - for hot-path work (grid launches, copies). - -The decision between them is driven by the practical challenges document; the -short version is that (1) is safest for correctness, (2) is best for latency, -and both must be isolated to be acceptable. +Backend selection, including the interfaces that actually exist and why the +distribution host driver cannot be called, is analyzed in +[`08-physical-gpu-integration.md`](08-physical-gpu-integration.md). In short: +proxying through the host driver has no callable entry point from kernel +context, so the chosen mechanism is for the module to own the device and run +its real GSP firmware as the resource-manager server, while the emulation path +remains the control plane. Software execution of decoded memory/state +operations stays a development bridge, not the data plane. ## 6. Debug tooling in the module @@ -290,5 +292,6 @@ in `kernel-open` first). | Push decoding + semaphore completion | demonstrated live; being productized | | Page-table population for server-owned ranges | designed, not implemented | | GPU work forwarding (GR/CE execution) | designed, not implemented | +| Execution backend (module-owned GPU + direct real-GSP session) | chosen, not started ([`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)) | | Real VRAM backing | designed, not implemented | | Windows guest, Xen transport, aarch64 hosts | designed, not implemented | -- cgit v1.2.3