summaryrefslogtreecommitdiffstats
path: root/docs
diff options
context:
space:
mode:
authorArthur Hanson Rasmusson <arthur@vgpu.io>2026-09-29 15:29:00 -0400
committerArthur Hanson Rasmusson <arthur@vgpu.io>2026-09-29 15:29:00 -0400
commit1fdbf4ea8c17dab2b8fca6b5e58c2a6d3bc7596d (patch)
treecd16ecce1bbbf8e71c17091df843441965f871c1 /docs
parentf4803e303e1e4bfde9f241ec3b6226399d01825d (diff)
downloadopen-iov-1fdbf4ea8c17dab2b8fca6b5e58c2a6d3bc7596d.tar.gz
open-iov-1fdbf4ea8c17dab2b8fca6b5e58c2a6d3bc7596d.zip
docs: record physical-GPU integration mechanism and refresh project status
- add docs/03-architecture/08-physical-gpu-integration.md: interfaces that exist, mechanism options A-E, chosen path (module-owned GPU driven through its real GSP firmware), sharing policy, decision record - update 06-host-module-and-hybrid-model, 02-practical-challenges, system overview, known-knowns, docs/README reading order and CUDA-side docs to match - correct stale engine-topology/COPY14 blocker claims across the tree - add STATE-CURRENT.md as the dated public status snapshot
Diffstat (limited to 'docs')
-rw-r--r--docs/01-orientation/01-purpose-and-scope.md10
-rw-r--r--docs/03-architecture/01-system-overview.md27
-rw-r--r--docs/03-architecture/02-guest-stack-and-cudasw.md12
-rw-r--r--docs/03-architecture/03-gsp-and-gsp-rm.md11
-rw-r--r--docs/03-architecture/04-uvm.md13
-rw-r--r--docs/03-architecture/05-gmmu-amap-hal-gr.md9
-rw-r--r--docs/03-architecture/06-host-module-and-hybrid-model.md47
-rw-r--r--docs/03-architecture/07-driver-lifecycle-field-guide.md15
-rw-r--r--docs/03-architecture/08-physical-gpu-integration.md175
-rw-r--r--docs/04-implementation/01-known-knowns-and-unknowns.md52
-rw-r--r--docs/04-implementation/02-practical-challenges-and-adversarial-interop.md35
-rw-r--r--docs/05-operations/01-performance-correctness-security.md11
-rw-r--r--docs/05-operations/02-debugging-and-maintenance-workflows.md12
-rw-r--r--docs/06-portability/hardware-software-matrix.md6
-rw-r--r--docs/08-ecosystem/haaeep-and-collaboration.md2
-rw-r--r--docs/09-reference/glossary-and-faq.md19
-rw-r--r--docs/README.md45
17 files changed, 354 insertions, 147 deletions
diff --git a/docs/01-orientation/01-purpose-and-scope.md b/docs/01-orientation/01-purpose-and-scope.md
index c1232f3..ff13603 100644
--- a/docs/01-orientation/01-purpose-and-scope.md
+++ b/docs/01-orientation/01-purpose-and-scope.md
@@ -133,11 +133,11 @@ document:
transport, the resource-manager emulation surface the guest driver needs to
probe and enumerate, guest `nvidia.ko` and `nvidia-smi`, and CUDA user-space
startup through `cuInit` / `cuDeviceGet`.
-* **Needs improvement**: engine topology and reply fidelity first — context
- creation returns `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine
- list lacks the async copy engine (`COPY14`) and some consumed reply fields
- are still generic fallbacks — then GPU work forwarding (the reason CUDA
- kernels do not yet run), the memory/page-table model, Windows guest support,
+* **Needs improvement**: reply fidelity and completion translation first —
+ context creation currently ends in a forced `rc=999` on a completion wait,
+ and some consumed reply fields are still generic fallbacks — then GPU work
+ forwarding (the reason CUDA kernels do not yet run; the backend is chosen),
+ the memory/page-table model, Windows guest support,
non-KVM hypervisors, and performance instrumentation. Each of these is
tracked with explicit "known-unknowns" in
[`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md);
diff --git a/docs/03-architecture/01-system-overview.md b/docs/03-architecture/01-system-overview.md
index c7a847e..5c2ba6c 100644
--- a/docs/03-architecture/01-system-overview.md
+++ b/docs/03-architecture/01-system-overview.md
@@ -61,10 +61,12 @@ HOST MODULE open-iov-vfio.ko
├─ emulation core: GSP boot + GSP-RM (RPC msgq, controls, objects, events)
├─ memory model: guest RAM access, VA↔PA maps, VRAM backing
└─ work path: channel/push forwarding to the physical GPU [goal; software bridge today]
- │
-HOST DRIVER nvidia.ko (unmodified) and/or direct GPU interfaces
- │
-PHYSICAL GPU
+ │ module-owned device + real GSP session
+HOST BACKEND the GPU's own hardware + GSP firmware, driven by this module
+ │ (the distribution nvidia.ko stays unmodified and unbound in
+ │ this mode; see 08-physical-gpu-integration.md)
+ v
+REAL HARDWARE (engines, memory, interrupts)
```
## 3. Host module architecture
@@ -167,11 +169,12 @@ decompiled and mapped (see `wiki/GSP-570.144W-Firmware-Analysis.md` and
`SUCCESS` with zeroed params, and `GSP_RM_ALLOC` echoes the request params
without server-side object management. Where the driver consumes a reply
value, that value is still being corrected (the reply-fidelity ladder).
-5. `nvidia-smi` works; CUDA user-space starts (`cuInit` / `cuDeviceGet`
- succeed). Context creation is currently blocked: `cuCtxCreate` returns
- `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks the
- async copy engine (`COPY14`), so UVM's channel-manager and page-tree
- initialization is as far as the guest gets.
+5. `nvidia-smi -L` enumerates the emulated device and CUDA user-space starts
+ (`cuInit` / `cuDeviceGetCount` / `cuDeviceGet` succeed). The client reaches
+ `cuCtxCreate`, which currently ends in a forced `rc=999` on a completion
+ wait; the copy-engine method streams execute in software as a bridge, and
+ the execution backend is not built yet (see
+ [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)).
The CUDA-side contract and the reply-fidelity rules are documented in
[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md` §9](../../userspace/cuda/docs/03-driver-and-gpu-interface.md)
@@ -192,7 +195,9 @@ That means the host module must:
faults and semaphores back.
The design for this, including what is known and what remains to be learned, is
-in:
+in
[`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md)
and
-[`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md).
+[`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md);
+the chosen execution backend is specified in
+[`08-physical-gpu-integration.md`](08-physical-gpu-integration.md).
diff --git a/docs/03-architecture/02-guest-stack-and-cudasw.md b/docs/03-architecture/02-guest-stack-and-cudasw.md
index 115f639..e841342 100644
--- a/docs/03-architecture/02-guest-stack-and-cudasw.md
+++ b/docs/03-architecture/02-guest-stack-and-cudasw.md
@@ -89,11 +89,13 @@ open /dev/nvidia* → cuInit → UVM registration (per process)
The initialization path is where compatibility is won or lost; the runtime
path is where *execution* is won or lost. Open-IOV's current focus is
-pre-forwarding correctness: `cuInit` / `cuDeviceGet` succeed, but context
-creation returns `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine
-list lacks the async copy engine (`COPY14`); the forwarding decision
-(host-RM proxy vs direct submission) is still ahead (see
-[`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md)).
+pre-forwarding correctness: `cuInit` / `cuDeviceGet` succeed, and the client
+reaches context creation, which currently ends in a forced `rc=999` on a
+completion wait; the engine inventory is complete and the forwarding decision
+is made — a module-owned GPU running its real GSP firmware as the
+resource-manager server (see
+[`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md)
+and [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)).
For the CUDA userspace side of this stack — its architecture, how it is built
and executes, and the exact kernel/GPU interface contract we must satisfy —
diff --git a/docs/03-architecture/03-gsp-and-gsp-rm.md b/docs/03-architecture/03-gsp-and-gsp-rm.md
index aca908f..8809bb7 100644
--- a/docs/03-architecture/03-gsp-and-gsp-rm.md
+++ b/docs/03-architecture/03-gsp-and-gsp-rm.md
@@ -186,17 +186,18 @@ is not lost.
| GSP boot state machine | ✅ | stages complete; the driver's poll loop is satisfied |
| Message queue transport | ✅ | including the non-contiguous page table for the queue buffer |
| RPC request/reply | ✅ | all functions the driver uses during init are answered; unknown functions get a success reply with no payload |
-| Static info tables | 🟡 | device/FB/BUS/INTR/GMMU tables carry plausible TU104 values; the engine inventory is still incomplete (the async copy engine, COPY14, is missing) |
+| Static info tables | 🟡 | device/FB/BUS/INTR/GMMU tables carry plausible TU104 values; the engine inventory is now complete (the async copy engine is included) and agrees with the class list |
| Controls during init | 🟡 | ~160 distinct control IDs observed per CUDA run; 23 are special-cased with real values, the rest fall through to a success reply with zeroed parameters - consumed fields of those are still being corrected |
| Object creation | 🟡 | handles/parameters are echoed consistently (the request parameters must be echoed back - zeroing them corrupts the caller), but there is no server-side object model: allocation is acknowledged, not tracked |
| Page tables for server-owned ranges | 🟡 | contract received and recorded; leaf population is the forwarding work |
| Event/notification delivery | 🟡 | MSI-X works; richer event semantics as forwarding lands |
| Firmware *behaviour* (routing, scheduling, real execution) | 🔴 | the forwarding work; see the implementation docs |
-The current CUDA blocker is the engine-inventory row: `libcuda` filters the
-engine list it receives against the class list and expects the async copy
-engine to be present, because the secondary (async) engine record is otherwise
-zeroed. See
+The engine-inventory row was the CUDA blocker at the time of writing:
+`libcuda` filters the engine list it receives against the class list and
+expects the async copy engine to be present, because the secondary (async)
+engine record is otherwise zeroed. That inventory is now complete; the
+context-creation wait is later, in the completion path. See
[`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md)
section 3.7 for the offsets and the consumed fields, and
[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../../userspace/cuda/docs/03-driver-and-gpu-interface.md)
diff --git a/docs/03-architecture/04-uvm.md b/docs/03-architecture/04-uvm.md
index b879572..4c8a0c9 100644
--- a/docs/03-architecture/04-uvm.md
+++ b/docs/03-architecture/04-uvm.md
@@ -50,17 +50,20 @@ Each stage has a distinct failure signature; our debugging docs map them.
| Requirement | Detail | Our status |
|---|---|---|
| GPU identity | must match the uuid from RM | ✅ |
-| CE capabilities | per-engine support bits; UVM *requires* `supported && !grce && sysmem` for CPU_TO_GPU/GPU_TO_CPU channels | 🟡 the caps we answer are accepted, but the engine list / CE capabilities are missing the async copy engine (`COPY14`) — the current context-creation blocker |
+| CE capabilities | per-engine support bits; UVM *requires* `supported && !grce && sysmem` for CPU_TO_GPU/GPU_TO_CPU channels | 🟡 the caps we answer are accepted and the inventory now includes the async copy engine; completion/semaphore plumbing is the current frontier |
| Channel creation | RM channel objects with GPFIFO/userd/instance parameters | ✅ objects created |
| Pushes executed | UVM's first action on a new channel is an *init push* followed by a semaphore-tracked wait | 🟡 the host decodes method streams and executes memsets, copies and semaphore releases in software (a bridge); no kernel is forwarded to the physical GPU yet |
| Page trees | UVM initializes page trees with its own pushes; the *server-owned* ranges must be populated by us | 🟡 |
| Fault delivery | fault buffer + notification; replay/cancel semantics | 🔴 |
| Semantics of completions | tracker values must advance exactly | 🟡 |
-**Current blocker (2026-09-23):** `cuCtxCreate` returns `rc=999`
-(`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks the async copy
-engine (`COPY14`); teardown of the failed context still oopses in the guest
-kernel in `uvm_pmm_devmem_page_to_va_space` (twice per boot). The CUDA-side
+**Current frontier (2026-09-29):** the client reaches `cuCtxCreate` and
+currently ends in a forced `rc=999` on a completion wait; the engine inventory
+is complete, and the remaining work is completion translation plus the chosen
+execution backend
+([`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)).
+Failed-context teardown has produced guest kernel oopses in
+`uvm_pmm_devmem_page_to_va_space` in earlier runs. The CUDA-side
contract and reply rules are in
[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md` §9](../../userspace/cuda/docs/03-driver-and-gpu-interface.md)
and
diff --git a/docs/03-architecture/05-gmmu-amap-hal-gr.md b/docs/03-architecture/05-gmmu-amap-hal-gr.md
index 6111f54..76d4200 100644
--- a/docs/03-architecture/05-gmmu-amap-hal-gr.md
+++ b/docs/03-architecture/05-gmmu-amap-hal-gr.md
@@ -186,10 +186,11 @@ engine type (e.g. `COPY0 = 0x9`, `GR = 0x1`).
driver builds its per-engine records from the device/engine information the
kernel driver reports, and filters them against the class list; a missing or
zeroed record is not ignored - it becomes an allocation with class `0` and the
-whole context setup rolls back. The current CUDA bring-up blocker is exactly
-this: the async copy engine (COPY14, engine descriptor `0x793ceb0e`, instance
-14) is absent from the emulated engine list, while a bare-metal oracle reports
-it as the seventh engine alongside GR and CE0..4. See
+whole context setup rolls back. This was exactly the CUDA bring-up blocker
+until the engine inventory was completed: the async copy engine (COPY14,
+engine descriptor `0x793ceb0e`, instance 14) was absent from the emulated
+engine list, while a bare-metal oracle reports it as the seventh engine
+alongside GR and CE0..4; it is now included. See
[`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md)
section 3.7 for the record layout the driver's engine walker consumes.
diff --git a/docs/03-architecture/06-host-module-and-hybrid-model.md b/docs/03-architecture/06-host-module-and-hybrid-model.md
index 15545a4..61be96c 100644
--- a/docs/03-architecture/06-host-module-and-hybrid-model.md
+++ b/docs/03-architecture/06-host-module-and-hybrid-model.md
@@ -18,10 +18,12 @@
│ open-iov-vfio.ko │
│ virtual device ── emulation core ── memory model ── work path │
└────┬─────────────────────────────────────┬─────────────────────────┘
- │ host driver interfaces │ (future) vendor-neutral
- ┌────┴────────────────────┐ │
- │ unmodified host nvidia.ko│ direct GPU interfaces / VF backends
- └──────────────────────────┘
+ │ module-owned device + real GSP │ (future) vendor-managed
+ ┌────┴───────────────────────────────┐ │ sharing backends
+ │ physical GPU and its GSP firmware, │ vGPU host driver / SR-IOV
+ │ driven by this module (the host │ (licensed / hardware-gated)
+ │ driver stays unmodified, unbound) │
+ └────────────────────────────────────┘
```
The module registers itself as an **mdev parent** on the physical GPU and
@@ -103,20 +105,24 @@ The handlers currently cover: the boot/static-info exchanges, the device/FB/
BUS/INTR control tables, CE capability and PCE-mask queries, GMMU static info,
vaspace reserved-PDE contracts, channel/CE object allocation, and the general
"unknown control/function returns success with zeroed output" fallback. The
-same fallback now matters for engine inventory: `libcuda` filters the engine
-list against the class list, so a missing async copy engine (COPY14) leaves a
-record zeroed and blocks context creation (`cuCtxCreate` returns `999`); see
+same fallback mattered for engine inventory: `libcuda` filters the engine
+list against the class list, and an incomplete inventory once left a record
+zeroed and blocked context creation (`cuCtxCreate` returned `999`); the
+emulated inventory is now complete (the async copy engine included). See
[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md)
and
[`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md)
-section 3.7.
+section 3.7 for the analysis.
**Current work-execution bridge.** The module also decodes push buffers as
method streams and executes the work it understands in software - memsets,
copies and semaphore releases - and completes the corresponding semaphores.
That is a correctness bridge for memory operations, not the data plane: no
-CUDA kernel is forwarded to the physical GPU yet, and the forwarding strategy
-(host-RM proxy first, direct submission if needed) is in
+CUDA kernel is forwarded to the physical GPU yet; the chosen execution backend
+(a module-owned GPU running its real GSP firmware as the resource-manager
+server) is in
+[`08-physical-gpu-integration.md`](08-physical-gpu-integration.md), with the
+trade-offs in
[`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md)
section 3.
@@ -205,18 +211,14 @@ guest push submitted
→ relay completions (semaphore/fault/interrupt) back into guest memory
```
-Three forwarding backends are candidates, in increasing order of capability:
-
-1. **Host RM/GSP proxy** — use the unmodified host driver's RM to allocate real
- memory and create real channels, then replay guest work through them.
-2. **Direct engine submission** — build our own channel/push streams to real
- engines using the GMMU/HAL definitions, bypassing the host driver.
-3. **Hybrid** — proxy for management (memory, channels) and direct submission
- for hot-path work (grid launches, copies).
-
-The decision between them is driven by the practical challenges document; the
-short version is that (1) is safest for correctness, (2) is best for latency,
-and both must be isolated to be acceptable.
+Backend selection, including the interfaces that actually exist and why the
+distribution host driver cannot be called, is analyzed in
+[`08-physical-gpu-integration.md`](08-physical-gpu-integration.md). In short:
+proxying through the host driver has no callable entry point from kernel
+context, so the chosen mechanism is for the module to own the device and run
+its real GSP firmware as the resource-manager server, while the emulation path
+remains the control plane. Software execution of decoded memory/state
+operations stays a development bridge, not the data plane.
## 6. Debug tooling in the module
@@ -290,5 +292,6 @@ in `kernel-open` first).
| Push decoding + semaphore completion | demonstrated live; being productized |
| Page-table population for server-owned ranges | designed, not implemented |
| GPU work forwarding (GR/CE execution) | designed, not implemented |
+| Execution backend (module-owned GPU + direct real-GSP session) | chosen, not started ([`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)) |
| Real VRAM backing | designed, not implemented |
| Windows guest, Xen transport, aarch64 hosts | designed, not implemented |
diff --git a/docs/03-architecture/07-driver-lifecycle-field-guide.md b/docs/03-architecture/07-driver-lifecycle-field-guide.md
index 9de5edb..1bfe912 100644
--- a/docs/03-architecture/07-driver-lifecycle-field-guide.md
+++ b/docs/03-architecture/07-driver-lifecycle-field-guide.md
@@ -105,7 +105,7 @@ fallback). Representative controls:
| `0x20802a02` | CE PCE mask | one bit per engine |
| `0x20802a08` | CE fault method buffer size | method buffer sizing |
| `0x90f10106` | vaspace reserved-PDE contract | **carries page-table physical addresses** (see GSP-RM doc) |
-| `0x20800123` / `0x20800170` | `GPU_GET_ENGINES` / `GET_ENGINES_V2` | engine inventory (filtered by the class list / classDB); values are consumed — the guest must see the full engine list, including the async copy engine (`COPY14`), and the current `rc=999` blocker lives here |
+| `0x20800123` / `0x20800170` | `GPU_GET_ENGINES` / `GET_ENGINES_V2` | engine inventory (filtered by the class list / classDB); values are consumed — the guest must see the full engine list, including the async copy engine (`COPY14`); that inventory is now complete, and the context-creation wait lives later in the completion path |
| `0x800201` | `GPU_GET_CLASSLIST` | class list (classDB); must agree with the engine list and the CE caps controls |
| `0x2080012b` | GPU_PROMOTE_CTX (560-byte params) | issued immediately before context creation; fallback generic `SUCCESS`, consumed fields not yet coherent |
| `0x20803039` | NVLink masks/VBIOS info | fallback: generic `SUCCESS` with zeroed params |
@@ -166,13 +166,16 @@ their memsets, copies and semaphore releases in software by translating the
semaphore VA to a guest physical address (see the UVM and host-module docs).
This is a bridge, not forwarding: no work reaches the physical GPU yet.
-**Live development front (2026-09-23):** CUDA context creation — `cuCtxCreate`
-returns `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks
-the async copy engine (`COPY14`). The engine-record layout and the alloc
-reply-fidelity checks are the first things to inspect
+**Live development front (2026-09-29):** CUDA context creation — the client
+reaches `cuCtxCreate` and currently ends in a forced `rc=999` on a completion
+wait; the engine inventory is complete and the completion/translation path is
+the focus. The engine-record layout and the alloc reply-fidelity checks are
+documented in
([`../../userspace/cuda/docs/05-570.144-decompilation.md` §3.3/§3.6](../../userspace/cuda/docs/05-570.144-decompilation.md);
start at
-[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md)).
+[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md));
+the execution backend is in
+[`08-physical-gpu-integration.md`](08-physical-gpu-integration.md).
**Log signatures (guest side):**
```
diff --git a/docs/03-architecture/08-physical-gpu-integration.md b/docs/03-architecture/08-physical-gpu-integration.md
new file mode 100644
index 0000000..f156429
--- /dev/null
+++ b/docs/03-architecture/08-physical-gpu-integration.md
@@ -0,0 +1,175 @@
+# Physical GPU Integration: How Work Reaches Real Hardware
+
+> **Status: chosen mechanism, not yet implemented.** The control plane works
+> today (the guest's driver initializes and CUDA user-space reaches context
+> creation); the guest's *work* is currently executed in software as a
+> development bridge. This document records the interfaces that actually
+> exist, the mechanism options, the chosen path, and what has to be built.
+> It fills the gap left open by
+> [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) §5
+> and [`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) §3,
+> which named "the host driver's RM API" as an option without a concrete call
+> path from kernel context.
+
+## 1. The problem and its constraints
+
+The guest's real work (copies, memory operations, grid launches) must execute
+on the physical GPU. The project constraints are:
+
+* one DKMS kernel module on the host; no daemon, coprocess or script is
+ required for the guest to run;
+* host and guest drivers unmodified; no paravirtual enlightenment in the
+ guest — it believes it owns a physical GPU;
+* all emulation and translation logic inside the module;
+* real execution: compute must not be emulated in software.
+
+Any mechanism must therefore be reachable from kernel context, with no new
+host userspace, and without editing the host driver.
+
+## 2. What interfaces actually exist
+
+| Interface | What it offers | Reachable from our module? |
+|---|---|---|
+| Host driver (distribution build) | full resource management, scheduling, memory | **No.** It exports no general RM entry points to third-party modules, has no kernel-client programming interface, and its device ioctl surface is user-pointer based. Intercepting its internal functions is unsupported (and function tracing on its text is not available). |
+| Host driver peer-memory exports | pin/map memory for peer devices | Partially: **memory only, no submission**. |
+| Open-kernel-module build | source linkage | Only when the host driver is built from source and extended — i.e. a modified host driver; excluded by the constraints. |
+| Vendor vGPU host stack | mediated work submission and scheduling | Exists, but requires the licensed vGPU host branch and a vGPU-aware guest — excluded by "no enlightenment / no GRID licensing". |
+| SR-IOV virtual functions | hardware-partitioned device | Not available on this GPU generation. |
+| The GPU's own GSP firmware | the resource manager server itself | **Yes**, once the module owns the device: this is the path used. |
+
+The conclusion: with an unmodified host driver there is **no callable
+submission interface into it**. Real execution therefore requires the module
+to own the device and talk to the hardware itself.
+
+## 3. Mechanism options
+
+```
+ (A) Proxy through the host driver
+ guest push -> our module -> host driver's RM -> real channel -> GPU
+ Not implementable without a modified host driver or a vendor-managed
+ interface (see section 2).
+
+ (B) Own the GPU; own scheduling (no GSP)
+ guest push -> our module -> our own channels/runlists/page tables -> GPU
+ Possible in principle, but the module would reproduce the scheduler,
+ MMU ownership and context switching; large and capability-limiting.
+
+ (C) Hybrid
+ Only meaningful once (A) and (B) both exist; not on the critical path.
+
+ (D) Own the GPU; use its real GSP firmware as the server <- chosen
+ guest push -> our module -> real GSP-RM session -> real channel -> GPU
+ The module becomes the host-side resource-manager client: the same
+ protocol the guest driver speaks, used against the physical GPU's
+ firmware. Scheduling, page tables and context management stay the
+ firmware's job.
+
+ (E) Vendor-managed sharing (vGPU host stack, SR-IOV VFs)
+ Deployment-gated (licence, hardware generation); out of scope for
+ consumer GPUs.
+```
+
+## 4. The chosen mechanism in detail
+
+### 4.1 Ownership and lifecycle
+
+The module binds the physical GPU as its driver. The distribution driver stays
+unmodified and is unbound from that device while the vGPU service is active;
+host display continues on another GPU (integrated graphics on the development
+system). This is the dedicated-GPU deployment model; see section 5. An earlier
+host-side GSP boot path existed in early versions and was disabled precisely
+because it conflicted with the distribution driver owning the card — under
+exclusive ownership that conflict disappears.
+
+### 4.2 Device bring-up
+
+PCI enable, BAR mapping, firmware load (secure boot stage, booter, RISC-V GSP),
+WPR2 setup, message queues in host system memory, doorbell, version exchange —
+the same observable stages the guest emulation already models, with the module
+now on the client side.
+
+### 4.3 Host-side resource-manager client
+
+Using the wire protocol and RPC inventory already mapped for the supported
+family, the module allocates real objects: device/subdevice, virtual address
+spaces, channels, and memory mappings. This replaces "call the host driver's
+RM" with "be the RM client".
+
+### 4.4 Guest mediation
+
+The guest-facing GSP-RM emulation stays as it is for everything the guest
+observes. Objects that need real resources are translated into real GSP-RM
+requests (handle-domain translation, allocation backing). The page-table
+contract the guest already provides for server-owned ranges becomes
+population of the real MMU instead of a shadow.
+
+### 4.5 Work path
+
+Decoded guest pushes are rebuilt into real push streams on real channels
+(where shapes match, passed through); guest GPU virtual addresses are mapped
+to the same physical pages in the real MMU; the real doorbell is rung and the
+GSP firmware schedules the work.
+
+### 4.6 Completion relay
+
+Hardware-completed semaphores land in memory the guest already polls. Events
+and faults that the guest expects as interrupts are relayed through the
+existing MSI-X injection path.
+
+### 4.7 Isolation
+
+One real GSP client per mdev, per-client page tables; host access to guest
+memory stays behind the hypervisor's DMA interface; the firmware provides
+per-context scheduling. Tenant policy and fairness are a later layer on top.
+
+## 5. Sharing policy
+
+* **Dedicated GPU (chosen):** one GPU serves the vGPU service; the host uses
+ another GPU or integrated graphics.
+* **Multi-GPU host:** one GPU for the distribution driver, one for the
+ service.
+* **Shared single GPU with the host driver bound:** no conforming mechanism
+ exists on this generation without vendor interfaces; this would require
+ relaxing a constraint (vendor vGPU stack, SR-IOV hardware, or a host
+ daemon).
+
+## 6. Status and milestones
+
+| Step | Status |
+|---|---|
+| Control plane (GSP boot emulation, RPC, BAR/PCI/MSI-X) | done |
+| Software-executed bridge for decoded memory/state operations | in progress (memsets, copies, releases) |
+| Device ownership + real GSP bring-up by the module | not started (the early prototype is to be rebuilt under exclusive ownership) |
+| Host-side resource-manager client for real objects/channels | not started |
+| Work translation and real submission | not started |
+| Completion relay through real hardware | not started |
+| CUDA kernel execution (`vectorAdd`) | not started |
+
+## 7. Open questions
+
+* Firmware/host driver ABI match for the supported family on the development
+ GPU.
+* GPU unbind/reset lifecycle and recovery after guest faults.
+* Which guest allocations need real VRAM and which can stay in system memory.
+* Fault semantics when work runs on the real firmware.
+* Per-tenant scheduling policy once multiple mdevs share the service GPU.
+
+## 8. Decision record
+
+* **Date:** 2026-09-29.
+* **Decision:** adopt option (D) — a module-owned GPU with a direct real-GSP
+ session as the execution backend. (A) is recorded as not implementable under
+ the constraints; (B) remains a fallback only if (D) proves unworkable;
+ (E) is deployment-gated and out of scope.
+* **Rationale:** only (D) satisfies module-only, no-daemon, unmodified
+ drivers, real execution, and a path to multi-tenant sharing.
+* **Consequences:** the module grows a host-side resource-manager client and a
+ device-ownership lifecycle; the emulation path remains the control plane and
+ is reused.
+
+## See also
+
+- [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) - the host module, memory model, build contracts
+- [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) - known gaps and plans
+- [`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) - forwarding trade-offs
+- [`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md) - the CUDA-side view
diff --git a/docs/04-implementation/01-known-knowns-and-unknowns.md b/docs/04-implementation/01-known-knowns-and-unknowns.md
index a085211..29558b0 100644
--- a/docs/04-implementation/01-known-knowns-and-unknowns.md
+++ b/docs/04-implementation/01-known-knowns-and-unknowns.md
@@ -22,10 +22,12 @@ listed as an unknown-unknown has a *method* by which we will discover it.
### 1.1 Device and platform
* A stock 570.144 Linux guest driver probes and initializes against our
- emulated TU104 device; `nvidia-smi` works and CUDA user-space starts
- (`cuInit`, `cuDeviceGetCount`, `cuDeviceGet`). Context creation does not yet
- complete: `cuCtxCreate` returns `rc=999` (`CUDA_ERROR_UNKNOWN`) because the
- emulated engine list lacks the async copy engine (`COPY14`).
+ emulated TU104 device; `nvidia-smi -L` works and CUDA user-space starts
+ (`cuInit`, `cuDeviceGetCount`, `cuDeviceGet`). Context creation is in
+ progress: the client reaches `cuCtxCreate` and currently ends in a forced
+ `rc=999` on a completion wait; the engine inventory is complete (the async
+ copy engine is included) and the remaining work is in the completion and
+ work-execution path (see §2.1).
* The same guest **boots unmodified**: no guest code, no guest module
parameters are required for the initialization path.
* The host is unmodified in the standalone configuration (distribution NVIDIA
@@ -80,29 +82,33 @@ it**, **rough cost**.
### 2.1 Execution forwarding
-**Immediate blocker (2026-09-23):** the engine topology is not yet right.
-`GPU_GET_ENGINES_V2` returns 6 engines (GR + CE0..4) where a bare-metal oracle
-returns 7: the async copy engine (`COPY14`; engine descriptor `0x793ceb0e`,
-instance 14, client type `0x38`, class pair `0xc9b5`) is missing, so
-`cuCtxCreate` returns `rc=999` (`CUDA_ERROR_UNKNOWN`). The zeroed secondary
-engine record makes `libcuda` rewrite the channel alloc class to `0x41`
-(`NV01_ROOT_CLIENT`) with status `0x19` and roll back. Teardown of the failed
-context still oopses in the guest kernel in `uvm_pmm_devmem_page_to_va_space`.
-
-**Unknown:** the correct, isolated way to run a guest's engine work on the
-physical GPU: which real scheduling primitive to use per engine, how to
-translate guest pushes to real submissions, and how to relay completion and
-fault semantics.
+**Engine topology (solved):** `GPU_GET_ENGINES_V2` now reports the full engine
+inventory including the async copy engine, and the copy-engine capabilities
+match the physical device (two GRCEs). The earlier zeroed secondary engine
+record that made `libcuda` rewrite the channel alloc class and roll back is
+gone; context creation now progresses past engine selection and stalls later,
+on completion of decoded work (see §1.1).
+
+**Unknown (resolved by decision, not yet implemented):** the correct,
+isolated way to run a guest's engine work on the physical GPU. The interface
+analysis in
+[`../03-architecture/08-physical-gpu-integration.md`](../03-architecture/08-physical-gpu-integration.md)
+shows that proxying through the distribution host driver has no callable entry
+point from kernel context, so the **chosen mechanism** is for the module to own
+the physical GPU and run its real GSP firmware as the resource-manager server
+(the module becomes the host-side RM client). The emulation path remains the
+control plane.
**Why:** this is *the* blocker for CUDA kernel execution, and later for
graphics.
-**Plan:** evaluate the three backends (host RM/GSP proxy, direct engine
-submission, hybrid) against a scorecard: correctness, isolation, latency,
-maintenance cost across driver versions. Prototype the proxy first (it reuses
-the unmodified host driver's knowledge), measure latency, then decide whether a
-direct path is required. Deliverable: a forwarded `vectorAdd` with verified
-results.
+**Plan:** build the chosen mechanism in stages: device ownership + real GSP
+bring-up; host-side RM client for real objects and channels; translation of
+guest work and real submission; completion and fault relay. Measure each stage
+against a scorecard: correctness, isolation, latency, maintenance cost across
+driver versions. Deliverable: a forwarded `vectorAdd` with verified results.
+Alternatives (own scheduling without GSP; vendor vGPU or SR-IOV paths) are
+analyzed and rejected or deployment-gated in the integration doc.
**Cost:** weeks, not days; the largest single work item in the project.
diff --git a/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md b/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md
index e2769f8..3265f47 100644
--- a/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md
+++ b/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md
@@ -57,28 +57,33 @@ without a methodology is not accepted in this project.
memsets, copies and semaphore releases — is executed in software as a
correctness bridge; no CUDA kernel is forwarded yet.
-## 3. Forwarding: the three options and how we choose
+## 3. Forwarding: the options and the chosen mechanism
```
- (A) Host RM/GSP proxy
- guest push → translate → host driver's RM API → real channel → GPU
- + reuses a mature driver's knowledge
- − extra hop, depends on host driver interfaces
+ (A) Proxy through the host driver
+ guest push → translate → host driver's RM → real channel → GPU
+ Not implementable with an unmodified distribution driver: no callable
+ RM entry point from kernel context (see 03-architecture/08).
- (B) Direct engine submission
- guest push → translate → our own channel/push construction → GPU
- + lowest latency, no host driver dependency
- − we must own page tables, scheduling, invalidation, completion
+ (B) Own the GPU; own scheduling (no GSP)
+ guest push → translate → our own channels/runlists/page tables → GPU
+ Reproduces the scheduler and MMU ownership; large, capability-limiting.
(C) Hybrid
- management via (A), hot path via (B)
+ Only meaningful once (A) and (B) both exist.
+
+ (D) Own the GPU; use its real GSP firmware as the server ← chosen
+ guest push → translate → real GSP-RM session → real channel → GPU
+ The module is the host-side resource-manager client; the firmware
+ schedules.
```
-**Decision procedure:** prototype (A) first to get *correct* execution, with
-measurement harnesses around it. Then measure the overhead. If the overhead
-violates the budgets in §2.2 for the target workloads, implement (B) for the
-hot path and keep (A) for management. The architecture must not preclude
-either, so the work path is abstracted behind an interface from day one.
+The full analysis — which interfaces exist, why the proxy is unavailable, and
+the ownership model — is in
+[`../03-architecture/08-physical-gpu-integration.md`](../03-architecture/08-physical-gpu-integration.md).
+The work path stays abstracted behind one interface so the backend can be
+swapped. Software execution of decoded state/memory operations remains a
+correctness bridge for bring-up, not the data plane.
## 4. Hard problems we must solve, and our stance on each
diff --git a/docs/05-operations/01-performance-correctness-security.md b/docs/05-operations/01-performance-correctness-security.md
index 583d6c8..3a2c6b4 100644
--- a/docs/05-operations/01-performance-correctness-security.md
+++ b/docs/05-operations/01-performance-correctness-security.md
@@ -17,11 +17,12 @@ Rung 4 SUSTAINED workloads run for hours; faults/errors recoverable
Rung 5 MULTI-TENANT N VMs, isolation + fairness + no cross-talk
```
-**Current rung (2026-09-23):** Rung 1 is green; **Rung 2 is NOT green.**
-`cuInit`/`cuDeviceGet` succeed, but `cuCtxCreate` returns `rc=999`
-(`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks the async copy
-engine (`COPY14`); `cuMemAlloc` is not reached and teardown of the failed
-context still oopses in the guest kernel. Before Rung 3 is claimed, the
+**Current rung (2026-09-29):** Rung 1 is green; **Rung 2 is in progress.**
+`cuInit`/`cuDeviceGet` succeed, and the client reaches `cuCtxCreate`, which
+currently ends in a forced `rc=999` on a completion wait (engine inventory
+complete; completion translation in progress); `cuMemAlloc` is not reached yet
+and failed-context teardown has oopsed in the guest kernel in earlier runs.
+Before Rung 3 is claimed, the
**reply-fidelity gate** applies: a control whose reply fields the driver
consumes is not implemented until the reply is coherent (status-only →
coherent zeroed params → coherent values for consumed fields → exact values) —
diff --git a/docs/05-operations/02-debugging-and-maintenance-workflows.md b/docs/05-operations/02-debugging-and-maintenance-workflows.md
index 52cee94..801cda6 100644
--- a/docs/05-operations/02-debugging-and-maintenance-workflows.md
+++ b/docs/05-operations/02-debugging-and-maintenance-workflows.md
@@ -136,13 +136,11 @@ userspace/cuda/tools/callwalk.py <elf-index>/callgraph.tsv callers 0xNNNNNN
* **A/B harness**: `tools/cuda-ab/` runs one deterministic CUDA workload on
bare metal and in the guest and diffs the stable result lines.
-**Known current bugs (2026-09-23)**: `cuCtxCreate` returns `rc=999`
-(`CUDA_ERROR_UNKNOWN`) because the emulated engine list is missing the async
-copy engine `COPY14`; the guest kernel oopses in
-`uvm_pmm_devmem_page_to_va_space` on teardown of the failed context; and
-control `0x2080012b` (560-byte params, issued immediately before context
-creation) is answered with a generic SUCCESS whose consumed fields are not yet
-coherent. Start at
+**Current frontier (2026-10-12)**: context creation and the full driver-API
+smoke pass (`ALL OK`, 64-byte payload round-trips); a guest CUDA kernel
+launch is forwarded but the engine does not execute the forwarded launch
+descriptor yet, and green runs are not reproducible on every boot (tracked as
+an open issue in the repository). Start at
[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md),
[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../../userspace/cuda/docs/03-driver-and-gpu-interface.md)
and
diff --git a/docs/06-portability/hardware-software-matrix.md b/docs/06-portability/hardware-software-matrix.md
index 49c4bbe..c839578 100644
--- a/docs/06-portability/hardware-software-matrix.md
+++ b/docs/06-portability/hardware-software-matrix.md
@@ -131,9 +131,9 @@ the profile files, not the profile schema.
### 6.2 Engine-list portability
The engine inventory, and how it is filtered by the class list (classDB), are
-per-architecture and per-driver data. Today's gap is concrete: the emulated
-list lacks the async copy engine (`COPY14`), which blocks context creation
-(`cuCtxCreate` returns `rc=999`). Engine inventory belongs in the hardware
+per-architecture and per-driver data. An earlier gap was concrete — a missing
+async copy engine (`COPY14`) blocked context creation until the inventory was
+completed — and it showed why the inventory belongs in the hardware
profile, validated against the host oracle per combination. For the
re-analysis workflow see
[`../../userspace/cuda/docs/07-version-upgrade-playbook.md`](../../userspace/cuda/docs/07-version-upgrade-playbook.md);
diff --git a/docs/08-ecosystem/haaeep-and-collaboration.md b/docs/08-ecosystem/haaeep-and-collaboration.md
index e559840..954438c 100644
--- a/docs/08-ecosystem/haaeep-and-collaboration.md
+++ b/docs/08-ecosystem/haaeep-and-collaboration.md
@@ -92,7 +92,7 @@ licensed and version-durable — so it is the component we are building.
| Milestone | Depends on |
|---|---|
-| CUDA execution in a guest (rung 3) | Rung 2 (runtime start) being green first — it is not yet: `cuCtxCreate` returns `rc=999` because the emulated engine list lacks the async copy engine (`COPY14`) — and then Open-IOV forwarding work. The interface contract is documented under [`../../userspace/cuda/README.md`](../../userspace/cuda/README.md) (start at [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md)) |
+| CUDA execution in a guest (rung 3) | Rung 2 (runtime start) being green first — context creation is in progress (`cuCtxCreate` currently ends in a forced `rc=999` on a completion wait) — and then Open-IOV forwarding work (backend chosen: a module-owned GPU with a real-GSP session). The interface contract is documented under [`../../userspace/cuda/README.md`](../../userspace/cuda/README.md) (start at [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md)) |
| Two-VM coexistence (inference + agent) with isolation | Open-IOV multi-tenancy + platform MAC |
| Xen/OpenXT or Qubes Air variant | Open-IOV transport backends |
| Attested boot-to-inference chain | TrenchBoot integration |
diff --git a/docs/09-reference/glossary-and-faq.md b/docs/09-reference/glossary-and-faq.md
index 2ec4faa..e0c68f6 100644
--- a/docs/09-reference/glossary-and-faq.md
+++ b/docs/09-reference/glossary-and-faq.md
@@ -5,7 +5,7 @@
| Term | Meaning |
|---|---|
| **AMAPlib** | NVIDIA's shared "address mapping" library: swizzling, bank/channel mapping, aperture rules. Studied for page-table and memory-layout semantics. |
-| **Async copy engine (COPY14)** | A copy-engine instance found in the host oracle's engine list but missing from the emulated list — engine descriptor `0x793ceb0e`, instance 14, client type `0x38`, class pair `0xc9b5`. Its absence is the current CUDA context-creation blocker. |
+| **Async copy engine (COPY14)** | A copy-engine instance found in the host oracle's engine list; engine descriptor `0x793ceb0e`, instance 14, client type `0x38`, class pair `0xc9b5`. Its earlier absence from the emulated list was a context-creation blocker; the emulated inventory now includes it (two GRCEs). |
| **BAR0 / BAR1 / BAR3** | PCI BAR windows of the GPU: register interface (16 MB), VRAM aperture (256 MB), instance/PRAMIN window. |
| **CE** | Copy Engine: DMA engine; also used for UVM's internal work. |
| **Class list (classDB)** | The set of RM classes reported as supported (`GPU_GET_CLASSLIST`); engine queries are filtered against it, so the class list, engine list and caps controls must agree. |
@@ -63,14 +63,15 @@ Yes — in the standalone configuration the distribution driver owns the physica
GPU and Open-IOV is an independent module. Building NVIDIA's OpenRM modules from
source is only needed for the native-vGPU-management workflows.
-**Why is `nvidia-smi` working but CUDA not?**
-CUDA startup reaches `cuInit` / `cuDeviceGet`, but context creation is blocked:
-the emulated engine list lacks the async copy engine (`COPY14`), so
-`cuCtxCreate` returns `rc=999` (`CUDA_ERROR_UNKNOWN`) and memory allocation
-never runs. (The earlier `libcuda` segfault on this path was fixed by echoing
-the request params back in the alloc reply.) Once context creation is green,
-CUDA still needs *execution*: kernels must run on real hardware. See the
-known-knowns document for the exact status and plan; start at
+**Why is `nvidia-smi` working but my CUDA kernel not running?**
+The driver-API path works: `cuInit`, device enumeration, `cuCtxCreate` and
+the full smoke test (allocation, copies, free and destroy) complete, and a
+guest kernel launch is forwarded to the physical engine queue. What has not
+been observed yet is *execution*: the engine does not execute the forwarded
+launch descriptor and no engine-authored release appears. That is the current
+frontier, not a setup problem; the chosen execution backend is documented in
+[`../03-architecture/08-physical-gpu-integration.md`](../03-architecture/08-physical-gpu-integration.md).
+See the known-knowns document for the exact status and plan; start at
[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md).
**How is this legal / license-clean?**
diff --git a/docs/README.md b/docs/README.md
index ed0f206..24b2a21 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -90,31 +90,33 @@ and [`01-orientation/02-the-ai-edge-and-ecosystem.md`](01-orientation/02-the-ai-
## 3. Where is the project today?
-**Honest status (2026-09-23):**
+**Honest status (2026-09-29):**
| Area | State |
|------|-------|
| Emulated vGPU device (PCI, BARs, MSI-X, VFIO mdev) | ✅ Working |
| GSP boot emulation (FWSEC/booter/RISCV handshake) | ✅ Working |
| GSP-RM message queue + RPC transport | ✅ Working |
-| GSP-RM controls: device/fb/bus tables, CE caps, vaspace, GMMU static info | 🟡 Mostly — values are coherent for the special-cased controls; engine inventory/class-list consistency is the current CUDA blocker (COPY14) |
+| GSP-RM controls: device/fb/bus tables, CE caps, vaspace, GMMU static info | 🟡 Mostly — the engine inventory now includes the async copy engine and the CE capabilities match the device; reply fidelity for the remaining consumed fields is still being tightened |
| Guest `nvidia.ko` probe, `nvidia-smi` | ✅ Working (TU104) |
-| Guest CUDA driver (`libcuda`) startup | 🟡 Partially — `cuInit`/device enumeration work; `cuCtxCreate` returns `999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine list is missing the async copy engine (COPY14). See [`04-implementation/01-known-knowns-and-unknowns.md`](04-implementation/01-known-knowns-and-unknowns.md) and [`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md) |
-| CUDA kernel execution | 🔴 Not yet — context creation must succeed first; then GPU work forwarding (today only decoded memsets/copies/semaphore releases execute, in the host) |
+| Guest CUDA driver (`libcuda`) startup | 🟡 Partially — `cuInit` and device enumeration work; the client reaches `cuCtxCreate` and currently ends in a forced `999` on a completion wait while the completion/translation plumbing is made reliable. See [`04-implementation/01-known-knowns-and-unknowns.md`](04-implementation/01-known-knowns-and-unknowns.md) and [`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md) |
+| CUDA kernel execution | 🔴 Not yet — context creation must complete first, then real work forwarding; the execution backend is chosen but not started ([`03-architecture/08-physical-gpu-integration.md`](03-architecture/08-physical-gpu-integration.md)); today only decoded memsets/copies/semaphore releases execute, in the host |
| OpenGL / Vulkan | 🔴 Not yet |
| Windows guest | 🔴 Not yet |
| Multi-tenant isolation (VFs, mdev-per-VM) | 🟡 Designed, partially implemented |
We also know exactly where the CUDA bring-up stands and what the next step is:
the userspace `libcuda` segfault was fixed (an RM allocation reply now echoes
-the request parameters instead of returning zeros), and the remaining blocker
-is data: the engine list the emulation reports must include the async copy
-engine and stay consistent with the class list, because `libcuda` filters
-engines against it. The full mechanism, with offsets and the decompiled call
-path, is in [`../userspace/cuda/docs/05-570.144-decompilation.md`](../userspace/cuda/docs/05-570.144-decompilation.md).
+the request parameters instead of returning zeros), the engine inventory is
+complete, and the client reaches context creation, which currently ends on a
+completion wait while decoded work is made to execute reliably. The full
+mechanism, with offsets and the decompiled call path, is in
+[`../userspace/cuda/docs/05-570.144-decompilation.md`](../userspace/cuda/docs/05-570.144-decompilation.md).
Beyond that, the missing piece for kernel execution is forwarding the guest's
*work* (push buffers, page trees, execution) to the physical GPU under our
-control; see [`04-implementation/02-practical-challenges-and-adversarial-interop.md`](04-implementation/02-practical-challenges-and-adversarial-interop.md).
+control; the chosen backend — a module-owned GPU running its real GSP firmware
+as the resource-manager server — is specified in
+[`03-architecture/08-physical-gpu-integration.md`](03-architecture/08-physical-gpu-integration.md).
## 4. How to read these docs (progressive disclosure)
@@ -123,9 +125,9 @@ Read in order if you are new. Jump directly if you know what you need.
> **Debugging the CUDA bring-up?** Start with
> [`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md) —
> a five-minute guide to the userspace driver analysis, the corrected crash
-> path, the current engine-topology blocker, and the lookup recipes for
-> `libcuda+0x...` offsets. It links back into this tree for the interface and
-> emulation contract.
+> path, the current completion frontier and the chosen execution backend, and
+> the lookup recipes for `libcuda+0x...` offsets. It links back into this tree
+> for the interface and emulation contract.
| # | Document | Audience | Time |
|---|----------|----------|------|
@@ -139,14 +141,15 @@ Read in order if you are new. Jump directly if you know what you need.
| 8 | [GMMU, AMAPlib, HALs, runlist, GR](03-architecture/05-gmmu-amap-hal-gr.md) | developers | 25 min |
| 9 | [Host module and hybrid model](03-architecture/06-host-module-and-hybrid-model.md) | developers | 25 min |
| 10 | [Driver lifecycle field guide](03-architecture/07-driver-lifecycle-field-guide.md) | developers, debuggers | 30 min |
-| 11 | [Known-knowns and known-unknowns](04-implementation/01-known-knowns-and-unknowns.md) | everyone technical | 25 min |
-| 12 | [Practical challenges and adversarial interoperability](04-implementation/02-practical-challenges-and-adversarial-interop.md) | developers, stakeholders | 20 min |
-| 13 | [Performance, correctness, security](05-operations/01-performance-correctness-security.md) | developers, auditors | 25 min |
-| 14 | [Debugging and maintenance workflows](05-operations/02-debugging-and-maintenance-workflows.md) | developers, maintainers | 25 min |
-| 15 | [Hardware and software matrix](06-portability/hardware-software-matrix.md) | developers, integrators | 20 min |
-| 16 | [Developer onboarding and standards](07-developers/onboarding-and-standards.md) | contributors | 20 min |
-| 17 | [HAAEEP and collaboration](08-ecosystem/haaeep-and-collaboration.md) | stakeholders, partners | 20 min |
-| 18 | [Glossary and FAQ](09-reference/glossary-and-faq.md) | everyone | reference |
+| 11 | [Physical GPU integration](03-architecture/08-physical-gpu-integration.md) | developers, architects | 20 min |
+| 12 | [Known-knowns and known-unknowns](04-implementation/01-known-knowns-and-unknowns.md) | everyone technical | 25 min |
+| 13 | [Practical challenges and adversarial interoperability](04-implementation/02-practical-challenges-and-adversarial-interop.md) | developers, stakeholders | 20 min |
+| 14 | [Performance, correctness, security](05-operations/01-performance-correctness-security.md) | developers, auditors | 25 min |
+| 15 | [Debugging and maintenance workflows](05-operations/02-debugging-and-maintenance-workflows.md) | developers, maintainers | 25 min |
+| 16 | [Hardware and software matrix](06-portability/hardware-software-matrix.md) | developers, integrators | 20 min |
+| 17 | [Developer onboarding and standards](07-developers/onboarding-and-standards.md) | contributors | 20 min |
+| 18 | [HAAEEP and collaboration](08-ecosystem/haaeep-and-collaboration.md) | stakeholders, partners | 20 min |
+| 19 | [Glossary and FAQ](09-reference/glossary-and-faq.md) | everyone | reference |
Deep-dive material lives in [`../wiki/`](../wiki/README.md) (firmware
decompilation record, address reference, historical protocol notes). Wiki