summaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorArthur Hanson Rasmusson <arthur@vgpu.io>2026-09-29 15:29:00 -0400
committerArthur Hanson Rasmusson <arthur@vgpu.io>2026-09-29 15:29:00 -0400
commit4738b619c45abc4cf98a549c3118839e48bb7244 (patch)
treecb6004f1a6351edbc482031189d359fd2882dab5
parent8d45a99fe54cb2b19b3024fd975ab11574611c3e (diff)
downloadopen-iov-4738b619c45abc4cf98a549c3118839e48bb7244.tar.gz
open-iov-4738b619c45abc4cf98a549c3118839e48bb7244.zip
docs: record physical-GPU integration mechanism and refresh project status
- add docs/03-architecture/08-physical-gpu-integration.md: interfaces that exist, mechanism options A-E, chosen path (module-owned GPU driven through its real GSP firmware), sharing policy, decision record - update 06-host-module-and-hybrid-model, 02-practical-challenges, system overview, known-knowns, docs/README reading order and CUDA-side docs to match - correct stale engine-topology/COPY14 blocker claims across the tree - add STATE-CURRENT.md as the dated public status snapshot
-rw-r--r--STATE-CURRENT.md241
-rw-r--r--docs/01-orientation/01-purpose-and-scope.md10
-rw-r--r--docs/03-architecture/01-system-overview.md27
-rw-r--r--docs/03-architecture/02-guest-stack-and-cudasw.md12
-rw-r--r--docs/03-architecture/03-gsp-and-gsp-rm.md11
-rw-r--r--docs/03-architecture/04-uvm.md13
-rw-r--r--docs/03-architecture/05-gmmu-amap-hal-gr.md9
-rw-r--r--docs/03-architecture/06-host-module-and-hybrid-model.md47
-rw-r--r--docs/03-architecture/07-driver-lifecycle-field-guide.md15
-rw-r--r--docs/03-architecture/08-physical-gpu-integration.md175
-rw-r--r--docs/04-implementation/01-known-knowns-and-unknowns.md52
-rw-r--r--docs/04-implementation/02-practical-challenges-and-adversarial-interop.md35
-rw-r--r--docs/05-operations/01-performance-correctness-security.md11
-rw-r--r--docs/05-operations/02-debugging-and-maintenance-workflows.md12
-rw-r--r--docs/06-portability/hardware-software-matrix.md6
-rw-r--r--docs/08-ecosystem/haaeep-and-collaboration.md2
-rw-r--r--docs/09-reference/glossary-and-faq.md19
-rw-r--r--docs/README.md45
-rw-r--r--userspace/cuda/570.144/docs/ANALYSIS-cuctxcreate-segv.md7
-rw-r--r--userspace/cuda/READ-ME-FIRST.md24
-rw-r--r--userspace/cuda/README.md2
-rw-r--r--userspace/cuda/docs/05-570.144-decompilation.md15
-rw-r--r--userspace/cuda/docs/08-virtualization-approaches.md41
23 files changed, 651 insertions, 180 deletions
diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md
new file mode 100644
index 0000000..c00231b
--- /dev/null
+++ b/STATE-CURRENT.md
@@ -0,0 +1,241 @@
+# Open-IOV Current State
+
+> **Snapshot: 2026-09-29 15:25.** This is a dated public status page, not live
+> state; it is refreshed on every progress check. Live working notes are
+> maintained outside this repository; the public known-knowns/unknowns document
+> is authoritative here:
+> [`docs/04-implementation/01-known-knowns-and-unknowns.md`](docs/04-implementation/01-known-knowns-and-unknowns.md).
+> The 2026-09-21 snapshot remains in [`STATE.md`](STATE.md) as a historical
+> record.
+
+## Mission
+
+Run **unmodified NVIDIA guest drivers** in VMs with full GPU functionality
+(CUDA, graphics, video, AI/ML) on consumer/data-center GPUs without
+proprietary GRID licenses - via host-side GSP emulation in
+`open-iov-vfio.ko`. The guest stays stock: no guest patches, no guest-side
+Open-IOV code.
+
+Architecture stance: *emulate the interface, pass the work through to real
+hardware, never patch anyone.* The control plane is emulated; the work is
+meant to execute on the physical GPU.
+
+## Milestone ladder (this snapshot)
+
+| Milestone | Status |
+|---|---|
+| GSP boot handshake, GSP-RM message queues/RPC, BAR/PCI/MSI-X emulation | ✅ |
+| Stock 570.144 guest driver probe / `RmInitAdapter` | ✅ |
+| `nvidia-smi -L` enumerates the emulated TU104 | ✅ |
+| CUDA userspace: `cuInit`, `cuDeviceGetCount`, `cuDeviceGet` | ✅ |
+| `cuCtxCreate` returns | ⏳ **frontier** |
+| `cuMemAlloc` + `cuMemcpy` roundtrip (`ALL OK` in the smoke test) | ❌ not reached |
+| Full `nvidia-smi` output | ❌ (faults on the CUDA path) |
+| CUDA kernel execution (`vectorAdd`) | ❌ not started (forwarding layer absent) |
+| Graphics (3D), video | ❌ |
+| Faults, multi-version, Windows, migration, multi-tenancy, performance | ❌ |
+
+Test status is tracked with the CUDA driver-API smoke test
+([`tools/test/cuda_drv_test.c`](tools/test/cuda_drv_test.c)): it exercises
+`cuInit -> cuDeviceGetCount/Get -> cuCtxCreate -> cuMemAlloc -> cuMemcpyHtoD/
+DtoH -> cuMemFree -> cuCtxDestroy` and prints `ALL OK` only when the whole
+chain succeeds.
+
+## Right now (frontier)
+
+- The stock guest driver initializes fully; the CUDA client reaches
+ `cuCtxCreate` within seconds and runs there for minutes before failing with
+ a forced `rc=999` (a timeout path, not an RM error record).
+- The per-channel completion counters in guest memory are enumerated,
+ registered and published; the client's per-channel waits clear, and the
+ per-channel GPFIFO work on copy channels is consumed.
+- Both engine push-buffer streams are now reachable: the copy-engine buffer is
+ resolved deterministically (function-trace pointer chain, all pages pinned),
+ and the compute-engine mapping window is being pinned from guest page tables
+ on every boot.
+- **The copy-engine data path executes**: decoded method streams run in
+ sustained fashion (memsets, copies, releases climbing; thousands of push
+ entries consumed).
+- **Frontier: context creation still times out on one completion.** The final
+ wait is a tracking semaphore whose release was decoded from a readable
+ compute push but could not be applied in time because the counter page had
+ no translation at that moment. The fix in progress is to pin the
+ compute-channel mapping window from guest truth at boot so the release
+ lands before the client's deadline.
+- Active work: the per-boot compute-window pin stream and re-running the full
+ cycle.
+- Execution backend (chosen, not started): the module will own the physical
+ GPU and run its real GSP firmware as the resource-manager server, so real
+ work executes with an unmodified host driver and no host daemon; see
+ [`docs/03-architecture/08-physical-gpu-integration.md`](docs/03-architecture/08-physical-gpu-integration.md).
+- Host stability: the host hang class observed on 2026-09-28 (kernel stack
+ overflow in a channel snapshot path) is fixed and frame-audited; repeated
+ VM load/attach/teardown cycles run without panics or pstore records.
+
+## Initialization, work and teardown path
+
+Status tags: `[done]`, `[frontier]` (actively worked), `[partial]`,
+`[not yet]`.
+
+```
+ HOST SIDE GUEST (stock driver, unmodified)
+ ========= ================================
+
+ [done] build + load open-iov-vfio.ko
+ | (module first: exports the symbols the NVIDIA open modules link against)
+ v
+ [done] mdev create ------------------------------------> /
+ | BAR/PCI/MSI-X emulation; GSP emulation thread
+ v
+ [done] VM attach (QEMU q35 + pcie-root-port) ----------> config space read; BAR mmap
+ |
+ v
+ [done] GSP boot handshake (emulated) ------------------> nvidia.ko probe / RmInitAdapter
+ |
+ v
+ [done] GSP-RM message queues / RPC / controls ---------> object + client setup
+ |
+ v
+ [done] engine + class lists ---------------------------> GPU_GET_ENGINES, class DB
+ | (two-GRCE fix; async copy engine)
+ v
+ [done] ------------------------------------------------> nvidia-smi -L enumerates TU104
+ |
+ v
+ [done] ------------------------------------------------> cuInit / DeviceGetCount / DeviceGet
+ |
+ v
+ [frontier] --------------------------------------------> cuCtxCreate
+ | UVM register GPU; channel manager
+ | GR + CE channels; push buffers
+ | semaphore pool; completion counters
+ v
+ [not yet] ---------------------------------------------> cuMemAlloc + cuMemcpy roundtrip
+ [not yet] ---------------------------------------------> full nvidia-smi output
+```
+
+Work execution (host service vs client synchronization):
+
+```
+ [partial] host decodes guest pushes (method streams): memsets, copies, releases
+ - CE memop pushes now execute (memset/ringops climbing, misses at zero)
+ - GR/compute pushes not yet consumed
+ [partial] GPU VA -> guest PA translation and pinning; server-owned page tables in progress
+ [not yet] real work forwarded to the physical GPU (host-RM proxy or direct submission)
+ [not yet] correct results (vectorAdd)
+```
+
+Teardown and recovery:
+
+```
+ [partial] failure teardown exercised (context rollback, channel stop, notifier status)
+ [done] VM stop -> mdev remove -> rmmod, in order; never rmmod with a live VM attached
+ [done] crash recovery: pstore inspection; the host hang class is fixed and frame-audited
+```
+
+Recent chain of walls removed (2026-09-28 to 2026-09-29):
+
+```
+ host hang (stack overflow) -> pump state persistence -> map collapse (merge rebuild)
+ -> rescue overhaul -> four-word semaphore releases -> counter injection (all 20)
+ -> stale semaphore-pool translation -> fake-only proven insufficient
+ -> push buffer found and validated -> per-page pins -> authoritative pin table
+ -> deterministic chain solved -> pin-drop and release-push fixes
+ -> copy-engine data path executes -> compute-window pinning
+```
+
+## Tools, code paths and fixes behind this progress
+
+```
+ KERNEL MODULE (open-iov-vfio.ko) GUEST-SIDE TOOLS (tracked)
+ ================================ ==========================
+ gsp_emulation.c boot, doorbells, MSI-X tools/test/cuda_drv_test.c smoke test
+ gsp_rm_emulation.c msgq, RPC, controls tools/guest-diag/vmrestart.sh
+ gsp_rm_tu104.h two-GRCE engine fix tools/guest-diag/cuda_cycle.sh
+ gsp_sema_service.c THE WORKHORSE tools/guest-diag/push_guest_tools.sh
+ channel pump, method decode, rescue, tools/guest-diag/inject_counters.sh
+ map rebuild, pins, counters, releases tools/guest-diag/semfix_watch.py
+ bar_emulation.c BAR shadows tools/guest-diag/track_waits.sh
+ main.c mdev/VFIO lifecycle tools/guest-diag/trkinfo.py
+ stack-frame audit (-Wframe-larger-than) tools/guest-diag/bp999.py / scanerr.py
+ dynamic_rpc_dispatcher.c firmware load fix tools/guest-diag/read_va.py
+ tools/guest-diag/guest_kprobes.sh
+
+ DATA STRUCTURES DEBUG PLANES
+ =============== ============
+ channel table (handle, GPFIFO, USERD, module stats: applied / miss / memset /
+ notifier, ring) ringops / nomap / skip / faked /
+ VA->PA map (merge rebuild, hot pages, cache) pins / ipins
+ pin table + authoritative injected pin table debug params: sema_probe, sema_counter,
+ completion counter table (per channel) sema_map_add, chan_dump, shadow_dump
+ semaphore pool page (GPU VA 0x121000000) QMP gva2gpa + guest gdbstub reads
+ push buffer allocation (GPU VA 0x120000000) guest kprobes; /proc/<pid>/pagemap
+ client trackers / RM error record chain marker-based /dev/kmsg parsing
+```
+
+| Wall | Fix |
+|---|---|
+| Host panic: oversized stack frame in a channel snapshot path | static buffers; frame-size audit on every build |
+| Channel pump lost per-pass state (stale entries re-run; rate limit dead) | write-back of rescue/payload state; one-retry skip for compute channels |
+| VA-to-PA map collapsed after client teardown | merge rebuild instead of wholesale replace |
+| Rescue slow / false candidates | hot-page-first scan; full method-stream validation; retry limits |
+| Four-word semaphore releases ignored | decode both release encodings |
+| Only 4 of 20 completion counters ever moved | async injection queue; fast injector; per-boot pin, order fixed |
+| UVM tracker wait never cleared | stale pool translation found; pool page pinned from guest truth each boot |
+| Acknowledging releases without executing work | proven insufficient; pushes must be read and executed |
+| Push buffers unreadable | located by CPU VA; per-page translation; authoritative injected pin table |
+| Injected pins silently dropped once a pin-table cluster grew | authoritative injected-pin table checked first; larger table, longer probe |
+| Real release pushes rejected as implausible (non-zero method start) | decode release-only when the translation came from injected pins |
+| Completion release decoded but deferred (counter page untranslated) | pin the compute-channel mapping window from guest truth each boot |
+
+## What "CUDA works" means (open gates)
+
+Per [`docs/04-implementation/01-known-knowns-and-unknowns.md`](docs/04-implementation/01-known-knowns-and-unknowns.md)
+section 4, a defensible claim needs **all** of:
+
+1. every guest-to-device message for the CUDA path answered or forwarded,
+ with coherent replies for consumed fields;
+2. every work VA translatable and server-owned page-table ranges populated;
+3. launches of a grid of thread blocks (a CUDA kernel dispatch, e.g.
+ `vectorAdd`'s `<<<blocks, threads>>>`) actually executed on the physical
+ GPU's compute engine - not merely acknowledged - and copies executing on
+ real hardware with correct results;
+4. completion (semaphores, events, faults) behaving as the guest expects;
+5. at least two driver families passing;
+6. tenant isolation;
+7. measured performance within an agreed factor of native.
+
+Until then the honest statement is: **initialization works; execution is in
+progress.**
+
+## Environment
+
+- Host GPU: NVIDIA TU104 (RTX 2080 SUPER Mobile), PCI `10de:1ed3`.
+- Host OS: Ubuntu, kernel 6.8.0-142; the distribution NVIDIA driver manages
+ the physical GPU; `open-iov-vfio.ko` loads as an independent module.
+- Guest: Ubuntu 24.04.5 LTS, stock NVIDIA 570.144 driver, QEMU/KVM q35 with a
+ `pcie-root-port` for the vGPU, SeaBIOS, mdev passthrough.
+- Firmware: TU104 GSP 570.144 (public release).
+
+## Build and load order
+
+Build: host module first (it generates the symbol versions the NVIDIA open
+kernel modules link against), then the NVIDIA open kernel modules, then
+userspace.
+
+Load: VFIO prerequisites, then `open-iov-vfio.ko`, then create the mdev and
+attach it to the VM. Unload in reverse; never `rmmod` while a VM holds the
+device.
+
+## See also
+
+- [`docs/README.md`](docs/README.md) - documentation front door
+- [`docs/04-implementation/01-known-knowns-and-unknowns.md`](docs/04-implementation/01-known-knowns-and-unknowns.md) - known-knowns, known-unknowns, definition of done
+- [`userspace/cuda/READ-ME-FIRST.md`](userspace/cuda/READ-ME-FIRST.md) - CUDA analysis entry point
+- [`docs/03-architecture/08-physical-gpu-integration.md`](docs/03-architecture/08-physical-gpu-integration.md) - chosen execution backend (module-owned GPU + real GSP session)
+- [`STATE.md`](STATE.md) - 2026-09-21 historical snapshot
+
+---
+
+*Refresh this page on every progress check (snapshot time in the header);
+prefer moving detail into the `docs/` tree and keeping this page short.*
diff --git a/docs/01-orientation/01-purpose-and-scope.md b/docs/01-orientation/01-purpose-and-scope.md
index c1232f3..ff13603 100644
--- a/docs/01-orientation/01-purpose-and-scope.md
+++ b/docs/01-orientation/01-purpose-and-scope.md
@@ -133,11 +133,11 @@ document:
transport, the resource-manager emulation surface the guest driver needs to
probe and enumerate, guest `nvidia.ko` and `nvidia-smi`, and CUDA user-space
startup through `cuInit` / `cuDeviceGet`.
-* **Needs improvement**: engine topology and reply fidelity first — context
- creation returns `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine
- list lacks the async copy engine (`COPY14`) and some consumed reply fields
- are still generic fallbacks — then GPU work forwarding (the reason CUDA
- kernels do not yet run), the memory/page-table model, Windows guest support,
+* **Needs improvement**: reply fidelity and completion translation first —
+ context creation currently ends in a forced `rc=999` on a completion wait,
+ and some consumed reply fields are still generic fallbacks — then GPU work
+ forwarding (the reason CUDA kernels do not yet run; the backend is chosen),
+ the memory/page-table model, Windows guest support,
non-KVM hypervisors, and performance instrumentation. Each of these is
tracked with explicit "known-unknowns" in
[`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md);
diff --git a/docs/03-architecture/01-system-overview.md b/docs/03-architecture/01-system-overview.md
index c7a847e..5c2ba6c 100644
--- a/docs/03-architecture/01-system-overview.md
+++ b/docs/03-architecture/01-system-overview.md
@@ -61,10 +61,12 @@ HOST MODULE open-iov-vfio.ko
├─ emulation core: GSP boot + GSP-RM (RPC msgq, controls, objects, events)
├─ memory model: guest RAM access, VA↔PA maps, VRAM backing
└─ work path: channel/push forwarding to the physical GPU [goal; software bridge today]
- │
-HOST DRIVER nvidia.ko (unmodified) and/or direct GPU interfaces
- │
-PHYSICAL GPU
+ │ module-owned device + real GSP session
+HOST BACKEND the GPU's own hardware + GSP firmware, driven by this module
+ │ (the distribution nvidia.ko stays unmodified and unbound in
+ │ this mode; see 08-physical-gpu-integration.md)
+ v
+REAL HARDWARE (engines, memory, interrupts)
```
## 3. Host module architecture
@@ -167,11 +169,12 @@ decompiled and mapped (see `wiki/GSP-570.144W-Firmware-Analysis.md` and
`SUCCESS` with zeroed params, and `GSP_RM_ALLOC` echoes the request params
without server-side object management. Where the driver consumes a reply
value, that value is still being corrected (the reply-fidelity ladder).
-5. `nvidia-smi` works; CUDA user-space starts (`cuInit` / `cuDeviceGet`
- succeed). Context creation is currently blocked: `cuCtxCreate` returns
- `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks the
- async copy engine (`COPY14`), so UVM's channel-manager and page-tree
- initialization is as far as the guest gets.
+5. `nvidia-smi -L` enumerates the emulated device and CUDA user-space starts
+ (`cuInit` / `cuDeviceGetCount` / `cuDeviceGet` succeed). The client reaches
+ `cuCtxCreate`, which currently ends in a forced `rc=999` on a completion
+ wait; the copy-engine method streams execute in software as a bridge, and
+ the execution backend is not built yet (see
+ [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)).
The CUDA-side contract and the reply-fidelity rules are documented in
[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md` §9](../../userspace/cuda/docs/03-driver-and-gpu-interface.md)
@@ -192,7 +195,9 @@ That means the host module must:
faults and semaphores back.
The design for this, including what is known and what remains to be learned, is
-in:
+in
[`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md)
and
-[`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md).
+[`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md);
+the chosen execution backend is specified in
+[`08-physical-gpu-integration.md`](08-physical-gpu-integration.md).
diff --git a/docs/03-architecture/02-guest-stack-and-cudasw.md b/docs/03-architecture/02-guest-stack-and-cudasw.md
index 115f639..e841342 100644
--- a/docs/03-architecture/02-guest-stack-and-cudasw.md
+++ b/docs/03-architecture/02-guest-stack-and-cudasw.md
@@ -89,11 +89,13 @@ open /dev/nvidia* → cuInit → UVM registration (per process)
The initialization path is where compatibility is won or lost; the runtime
path is where *execution* is won or lost. Open-IOV's current focus is
-pre-forwarding correctness: `cuInit` / `cuDeviceGet` succeed, but context
-creation returns `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine
-list lacks the async copy engine (`COPY14`); the forwarding decision
-(host-RM proxy vs direct submission) is still ahead (see
-[`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md)).
+pre-forwarding correctness: `cuInit` / `cuDeviceGet` succeed, and the client
+reaches context creation, which currently ends in a forced `rc=999` on a
+completion wait; the engine inventory is complete and the forwarding decision
+is made — a module-owned GPU running its real GSP firmware as the
+resource-manager server (see
+[`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md)
+and [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)).
For the CUDA userspace side of this stack — its architecture, how it is built
and executes, and the exact kernel/GPU interface contract we must satisfy —
diff --git a/docs/03-architecture/03-gsp-and-gsp-rm.md b/docs/03-architecture/03-gsp-and-gsp-rm.md
index aca908f..8809bb7 100644
--- a/docs/03-architecture/03-gsp-and-gsp-rm.md
+++ b/docs/03-architecture/03-gsp-and-gsp-rm.md
@@ -186,17 +186,18 @@ is not lost.
| GSP boot state machine | ✅ | stages complete; the driver's poll loop is satisfied |
| Message queue transport | ✅ | including the non-contiguous page table for the queue buffer |
| RPC request/reply | ✅ | all functions the driver uses during init are answered; unknown functions get a success reply with no payload |
-| Static info tables | 🟡 | device/FB/BUS/INTR/GMMU tables carry plausible TU104 values; the engine inventory is still incomplete (the async copy engine, COPY14, is missing) |
+| Static info tables | 🟡 | device/FB/BUS/INTR/GMMU tables carry plausible TU104 values; the engine inventory is now complete (the async copy engine is included) and agrees with the class list |
| Controls during init | 🟡 | ~160 distinct control IDs observed per CUDA run; 23 are special-cased with real values, the rest fall through to a success reply with zeroed parameters - consumed fields of those are still being corrected |
| Object creation | 🟡 | handles/parameters are echoed consistently (the request parameters must be echoed back - zeroing them corrupts the caller), but there is no server-side object model: allocation is acknowledged, not tracked |
| Page tables for server-owned ranges | 🟡 | contract received and recorded; leaf population is the forwarding work |
| Event/notification delivery | 🟡 | MSI-X works; richer event semantics as forwarding lands |
| Firmware *behaviour* (routing, scheduling, real execution) | 🔴 | the forwarding work; see the implementation docs |
-The current CUDA blocker is the engine-inventory row: `libcuda` filters the
-engine list it receives against the class list and expects the async copy
-engine to be present, because the secondary (async) engine record is otherwise
-zeroed. See
+The engine-inventory row was the CUDA blocker at the time of writing:
+`libcuda` filters the engine list it receives against the class list and
+expects the async copy engine to be present, because the secondary (async)
+engine record is otherwise zeroed. That inventory is now complete; the
+context-creation wait is later, in the completion path. See
[`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md)
section 3.7 for the offsets and the consumed fields, and
[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../../userspace/cuda/docs/03-driver-and-gpu-interface.md)
diff --git a/docs/03-architecture/04-uvm.md b/docs/03-architecture/04-uvm.md
index b879572..4c8a0c9 100644
--- a/docs/03-architecture/04-uvm.md
+++ b/docs/03-architecture/04-uvm.md
@@ -50,17 +50,20 @@ Each stage has a distinct failure signature; our debugging docs map them.
| Requirement | Detail | Our status |
|---|---|---|
| GPU identity | must match the uuid from RM | ✅ |
-| CE capabilities | per-engine support bits; UVM *requires* `supported && !grce && sysmem` for CPU_TO_GPU/GPU_TO_CPU channels | 🟡 the caps we answer are accepted, but the engine list / CE capabilities are missing the async copy engine (`COPY14`) — the current context-creation blocker |
+| CE capabilities | per-engine support bits; UVM *requires* `supported && !grce && sysmem` for CPU_TO_GPU/GPU_TO_CPU channels | 🟡 the caps we answer are accepted and the inventory now includes the async copy engine; completion/semaphore plumbing is the current frontier |
| Channel creation | RM channel objects with GPFIFO/userd/instance parameters | ✅ objects created |
| Pushes executed | UVM's first action on a new channel is an *init push* followed by a semaphore-tracked wait | 🟡 the host decodes method streams and executes memsets, copies and semaphore releases in software (a bridge); no kernel is forwarded to the physical GPU yet |
| Page trees | UVM initializes page trees with its own pushes; the *server-owned* ranges must be populated by us | 🟡 |
| Fault delivery | fault buffer + notification; replay/cancel semantics | 🔴 |
| Semantics of completions | tracker values must advance exactly | 🟡 |
-**Current blocker (2026-09-23):** `cuCtxCreate` returns `rc=999`
-(`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks the async copy
-engine (`COPY14`); teardown of the failed context still oopses in the guest
-kernel in `uvm_pmm_devmem_page_to_va_space` (twice per boot). The CUDA-side
+**Current frontier (2026-09-29):** the client reaches `cuCtxCreate` and
+currently ends in a forced `rc=999` on a completion wait; the engine inventory
+is complete, and the remaining work is completion translation plus the chosen
+execution backend
+([`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)).
+Failed-context teardown has produced guest kernel oopses in
+`uvm_pmm_devmem_page_to_va_space` in earlier runs. The CUDA-side
contract and reply rules are in
[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md` §9](../../userspace/cuda/docs/03-driver-and-gpu-interface.md)
and
diff --git a/docs/03-architecture/05-gmmu-amap-hal-gr.md b/docs/03-architecture/05-gmmu-amap-hal-gr.md
index d6470b6..cd31409 100644
--- a/docs/03-architecture/05-gmmu-amap-hal-gr.md
+++ b/docs/03-architecture/05-gmmu-amap-hal-gr.md
@@ -186,10 +186,11 @@ engine type (e.g. `COPY0 = 0x9`, `GR = 0x1`).
driver builds its per-engine records from the device/engine information the
kernel driver reports, and filters them against the class list; a missing or
zeroed record is not ignored - it becomes an allocation with class `0` and the
-whole context setup rolls back. The current CUDA bring-up blocker is exactly
-this: the async copy engine (COPY14, engine descriptor `0x793ceb0e`, instance
-14) is absent from the emulated engine list, while a bare-metal oracle reports
-it as the seventh engine alongside GR and CE0..4. See
+whole context setup rolls back. This was exactly the CUDA bring-up blocker
+until the engine inventory was completed: the async copy engine (COPY14,
+engine descriptor `0x793ceb0e`, instance 14) was absent from the emulated
+engine list, while a bare-metal oracle reports it as the seventh engine
+alongside GR and CE0..4; it is now included. See
[`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md)
section 3.7 for the record layout the driver's engine walker consumes.
diff --git a/docs/03-architecture/06-host-module-and-hybrid-model.md b/docs/03-architecture/06-host-module-and-hybrid-model.md
index 15545a4..61be96c 100644
--- a/docs/03-architecture/06-host-module-and-hybrid-model.md
+++ b/docs/03-architecture/06-host-module-and-hybrid-model.md
@@ -18,10 +18,12 @@
│ open-iov-vfio.ko │
│ virtual device ── emulation core ── memory model ── work path │
└────┬─────────────────────────────────────┬─────────────────────────┘
- │ host driver interfaces │ (future) vendor-neutral
- ┌────┴────────────────────┐ │
- │ unmodified host nvidia.ko│ direct GPU interfaces / VF backends
- └──────────────────────────┘
+ │ module-owned device + real GSP │ (future) vendor-managed
+ ┌────┴───────────────────────────────┐ │ sharing backends
+ │ physical GPU and its GSP firmware, │ vGPU host driver / SR-IOV
+ │ driven by this module (the host │ (licensed / hardware-gated)
+ │ driver stays unmodified, unbound) │
+ └────────────────────────────────────┘
```
The module registers itself as an **mdev parent** on the physical GPU and
@@ -103,20 +105,24 @@ The handlers currently cover: the boot/static-info exchanges, the device/FB/
BUS/INTR control tables, CE capability and PCE-mask queries, GMMU static info,
vaspace reserved-PDE contracts, channel/CE object allocation, and the general
"unknown control/function returns success with zeroed output" fallback. The
-same fallback now matters for engine inventory: `libcuda` filters the engine
-list against the class list, so a missing async copy engine (COPY14) leaves a
-record zeroed and blocks context creation (`cuCtxCreate` returns `999`); see
+same fallback mattered for engine inventory: `libcuda` filters the engine
+list against the class list, and an incomplete inventory once left a record
+zeroed and blocked context creation (`cuCtxCreate` returned `999`); the
+emulated inventory is now complete (the async copy engine included). See
[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md)
and
[`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md)
-section 3.7.
+section 3.7 for the analysis.
**Current work-execution bridge.** The module also decodes push buffers as
method streams and executes the work it understands in software - memsets,
copies and semaphore releases - and completes the corresponding semaphores.
That is a correctness bridge for memory operations, not the data plane: no
-CUDA kernel is forwarded to the physical GPU yet, and the forwarding strategy
-(host-RM proxy first, direct submission if needed) is in
+CUDA kernel is forwarded to the physical GPU yet; the chosen execution backend
+(a module-owned GPU running its real GSP firmware as the resource-manager
+server) is in
+[`08-physical-gpu-integration.md`](08-physical-gpu-integration.md), with the
+trade-offs in
[`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md)
section 3.
@@ -205,18 +211,14 @@ guest push submitted
→ relay completions (semaphore/fault/interrupt) back into guest memory
```
-Three forwarding backends are candidates, in increasing order of capability:
-
-1. **Host RM/GSP proxy** — use the unmodified host driver's RM to allocate real
- memory and create real channels, then replay guest work through them.
-2. **Direct engine submission** — build our own channel/push streams to real
- engines using the GMMU/HAL definitions, bypassing the host driver.
-3. **Hybrid** — proxy for management (memory, channels) and direct submission
- for hot-path work (grid launches, copies).
-
-The decision between them is driven by the practical challenges document; the
-short version is that (1) is safest for correctness, (2) is best for latency,
-and both must be isolated to be acceptable.
+Backend selection, including the interfaces that actually exist and why the
+distribution host driver cannot be called, is analyzed in
+[`08-physical-gpu-integration.md`](08-physical-gpu-integration.md). In short:
+proxying through the host driver has no callable entry point from kernel
+context, so the chosen mechanism is for the module to own the device and run
+its real GSP firmware as the resource-manager server, while the emulation path
+remains the control plane. Software execution of decoded memory/state
+operations stays a development bridge, not the data plane.
## 6. Debug tooling in the module
@@ -290,5 +292,6 @@ in `kernel-open` first).
| Push decoding + semaphore completion | demonstrated live; being productized |
| Page-table population for server-owned ranges | designed, not implemented |
| GPU work forwarding (GR/CE execution) | designed, not implemented |
+| Execution backend (module-owned GPU + direct real-GSP session) | chosen, not started ([`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)) |
| Real VRAM backing | designed, not implemented |
| Windows guest, Xen transport, aarch64 hosts | designed, not implemented |
diff --git a/docs/03-architecture/07-driver-lifecycle-field-guide.md b/docs/03-architecture/07-driver-lifecycle-field-guide.md
index 9de5edb..1bfe912 100644
--- a/docs/03-architecture/07-driver-lifecycle-field-guide.md
+++ b/docs/03-architecture/07-driver-lifecycle-field-guide.md
@@ -105,7 +105,7 @@ fallback). Representative controls:
| `0x20802a02` | CE PCE mask | one bit per engine |
| `0x20802a08` | CE fault method buffer size | method buffer sizing |
| `0x90f10106` | vaspace reserved-PDE contract | **carries page-table physical addresses** (see GSP-RM doc) |
-| `0x20800123` / `0x20800170` | `GPU_GET_ENGINES` / `GET_ENGINES_V2` | engine inventory (filtered by the class list / classDB); values are consumed — the guest must see the full engine list, including the async copy engine (`COPY14`), and the current `rc=999` blocker lives here |
+| `0x20800123` / `0x20800170` | `GPU_GET_ENGINES` / `GET_ENGINES_V2` | engine inventory (filtered by the class list / classDB); values are consumed — the guest must see the full engine list, including the async copy engine (`COPY14`); that inventory is now complete, and the context-creation wait lives later in the completion path |
| `0x800201` | `GPU_GET_CLASSLIST` | class list (classDB); must agree with the engine list and the CE caps controls |
| `0x2080012b` | GPU_PROMOTE_CTX (560-byte params) | issued immediately before context creation; fallback generic `SUCCESS`, consumed fields not yet coherent |
| `0x20803039` | NVLink masks/VBIOS info | fallback: generic `SUCCESS` with zeroed params |
@@ -166,13 +166,16 @@ their memsets, copies and semaphore releases in software by translating the
semaphore VA to a guest physical address (see the UVM and host-module docs).
This is a bridge, not forwarding: no work reaches the physical GPU yet.
-**Live development front (2026-09-23):** CUDA context creation — `cuCtxCreate`
-returns `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks
-the async copy engine (`COPY14`). The engine-record layout and the alloc
-reply-fidelity checks are the first things to inspect
+**Live development front (2026-09-29):** CUDA context creation — the client
+reaches `cuCtxCreate` and currently ends in a forced `rc=999` on a completion
+wait; the engine inventory is complete and the completion/translation path is
+the focus. The engine-record layout and the alloc reply-fidelity checks are
+documented in
([`../../userspace/cuda/docs/05-570.144-decompilation.md` §3.3/§3.6](../../userspace/cuda/docs/05-570.144-decompilation.md);
start at
-[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md)).
+[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md));
+the execution backend is in
+[`08-physical-gpu-integration.md`](08-physical-gpu-integration.md).
**Log signatures (guest side):**
```
diff --git a/docs/03-architecture/08-physical-gpu-integration.md b/docs/03-architecture/08-physical-gpu-integration.md
new file mode 100644
index 0000000..f156429
--- /dev/null
+++ b/docs/03-architecture/08-physical-gpu-integration.md
@@ -0,0 +1,175 @@
+# Physical GPU Integration: How Work Reaches Real Hardware
+
+> **Status: chosen mechanism, not yet implemented.** The control plane works
+> today (the guest's driver initializes and CUDA user-space reaches context
+> creation); the guest's *work* is currently executed in software as a
+> development bridge. This document records the interfaces that actually
+> exist, the mechanism options, the chosen path, and what has to be built.
+> It fills the gap left open by
+> [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) §5
+> and [`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) §3,
+> which named "the host driver's RM API" as an option without a concrete call
+> path from kernel context.
+
+## 1. The problem and its constraints
+
+The guest's real work (copies, memory operations, grid launches) must execute
+on the physical GPU. The project constraints are:
+
+* one DKMS kernel module on the host; no daemon, coprocess or script is
+ required for the guest to run;
+* host and guest drivers unmodified; no paravirtual enlightenment in the
+ guest — it believes it owns a physical GPU;
+* all emulation and translation logic inside the module;
+* real execution: compute must not be emulated in software.
+
+Any mechanism must therefore be reachable from kernel context, with no new
+host userspace, and without editing the host driver.
+
+## 2. What interfaces actually exist
+
+| Interface | What it offers | Reachable from our module? |
+|---|---|---|
+| Host driver (distribution build) | full resource management, scheduling, memory | **No.** It exports no general RM entry points to third-party modules, has no kernel-client programming interface, and its device ioctl surface is user-pointer based. Intercepting its internal functions is unsupported (and function tracing on its text is not available). |
+| Host driver peer-memory exports | pin/map memory for peer devices | Partially: **memory only, no submission**. |
+| Open-kernel-module build | source linkage | Only when the host driver is built from source and extended — i.e. a modified host driver; excluded by the constraints. |
+| Vendor vGPU host stack | mediated work submission and scheduling | Exists, but requires the licensed vGPU host branch and a vGPU-aware guest — excluded by "no enlightenment / no GRID licensing". |
+| SR-IOV virtual functions | hardware-partitioned device | Not available on this GPU generation. |
+| The GPU's own GSP firmware | the resource manager server itself | **Yes**, once the module owns the device: this is the path used. |
+
+The conclusion: with an unmodified host driver there is **no callable
+submission interface into it**. Real execution therefore requires the module
+to own the device and talk to the hardware itself.
+
+## 3. Mechanism options
+
+```
+ (A) Proxy through the host driver
+ guest push -> our module -> host driver's RM -> real channel -> GPU
+ Not implementable without a modified host driver or a vendor-managed
+ interface (see section 2).
+
+ (B) Own the GPU; own scheduling (no GSP)
+ guest push -> our module -> our own channels/runlists/page tables -> GPU
+ Possible in principle, but the module would reproduce the scheduler,
+ MMU ownership and context switching; large and capability-limiting.
+
+ (C) Hybrid
+ Only meaningful once (A) and (B) both exist; not on the critical path.
+
+ (D) Own the GPU; use its real GSP firmware as the server <- chosen
+ guest push -> our module -> real GSP-RM session -> real channel -> GPU
+ The module becomes the host-side resource-manager client: the same
+ protocol the guest driver speaks, used against the physical GPU's
+ firmware. Scheduling, page tables and context management stay the
+ firmware's job.
+
+ (E) Vendor-managed sharing (vGPU host stack, SR-IOV VFs)
+ Deployment-gated (licence, hardware generation); out of scope for
+ consumer GPUs.
+```
+
+## 4. The chosen mechanism in detail
+
+### 4.1 Ownership and lifecycle
+
+The module binds the physical GPU as its driver. The distribution driver stays
+unmodified and is unbound from that device while the vGPU service is active;
+host display continues on another GPU (integrated graphics on the development
+system). This is the dedicated-GPU deployment model; see section 5. An earlier
+host-side GSP boot path existed in early versions and was disabled precisely
+because it conflicted with the distribution driver owning the card — under
+exclusive ownership that conflict disappears.
+
+### 4.2 Device bring-up
+
+PCI enable, BAR mapping, firmware load (secure boot stage, booter, RISC-V GSP),
+WPR2 setup, message queues in host system memory, doorbell, version exchange —
+the same observable stages the guest emulation already models, with the module
+now on the client side.
+
+### 4.3 Host-side resource-manager client
+
+Using the wire protocol and RPC inventory already mapped for the supported
+family, the module allocates real objects: device/subdevice, virtual address
+spaces, channels, and memory mappings. This replaces "call the host driver's
+RM" with "be the RM client".
+
+### 4.4 Guest mediation
+
+The guest-facing GSP-RM emulation stays as it is for everything the guest
+observes. Objects that need real resources are translated into real GSP-RM
+requests (handle-domain translation, allocation backing). The page-table
+contract the guest already provides for server-owned ranges becomes
+population of the real MMU instead of a shadow.
+
+### 4.5 Work path
+
+Decoded guest pushes are rebuilt into real push streams on real channels
+(where shapes match, passed through); guest GPU virtual addresses are mapped
+to the same physical pages in the real MMU; the real doorbell is rung and the
+GSP firmware schedules the work.
+
+### 4.6 Completion relay
+
+Hardware-completed semaphores land in memory the guest already polls. Events
+and faults that the guest expects as interrupts are relayed through the
+existing MSI-X injection path.
+
+### 4.7 Isolation
+
+One real GSP client per mdev, per-client page tables; host access to guest
+memory stays behind the hypervisor's DMA interface; the firmware provides
+per-context scheduling. Tenant policy and fairness are a later layer on top.
+
+## 5. Sharing policy
+
+* **Dedicated GPU (chosen):** one GPU serves the vGPU service; the host uses
+ another GPU or integrated graphics.
+* **Multi-GPU host:** one GPU for the distribution driver, one for the
+ service.
+* **Shared single GPU with the host driver bound:** no conforming mechanism
+ exists on this generation without vendor interfaces; this would require
+ relaxing a constraint (vendor vGPU stack, SR-IOV hardware, or a host
+ daemon).
+
+## 6. Status and milestones
+
+| Step | Status |
+|---|---|
+| Control plane (GSP boot emulation, RPC, BAR/PCI/MSI-X) | done |
+| Software-executed bridge for decoded memory/state operations | in progress (memsets, copies, releases) |
+| Device ownership + real GSP bring-up by the module | not started (the early prototype is to be rebuilt under exclusive ownership) |
+| Host-side resource-manager client for real objects/channels | not started |
+| Work translation and real submission | not started |
+| Completion relay through real hardware | not started |
+| CUDA kernel execution (`vectorAdd`) | not started |
+
+## 7. Open questions
+
+* Firmware/host driver ABI match for the supported family on the development
+ GPU.
+* GPU unbind/reset lifecycle and recovery after guest faults.
+* Which guest allocations need real VRAM and which can stay in system memory.
+* Fault semantics when work runs on the real firmware.
+* Per-tenant scheduling policy once multiple mdevs share the service GPU.
+
+## 8. Decision record
+
+* **Date:** 2026-09-29.
+* **Decision:** adopt option (D) — a module-owned GPU with a direct real-GSP
+ session as the execution backend. (A) is recorded as not implementable under
+ the constraints; (B) remains a fallback only if (D) proves unworkable;
+ (E) is deployment-gated and out of scope.
+* **Rationale:** only (D) satisfies module-only, no-daemon, unmodified
+ drivers, real execution, and a path to multi-tenant sharing.
+* **Consequences:** the module grows a host-side resource-manager client and a
+ device-ownership lifecycle; the emulation path remains the control plane and
+ is reused.
+
+## See also
+
+- [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) - the host module, memory model, build contracts
+- [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) - known gaps and plans
+- [`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) - forwarding trade-offs
+- [`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md) - the CUDA-side view
diff --git a/docs/04-implementation/01-known-knowns-and-unknowns.md b/docs/04-implementation/01-known-knowns-and-unknowns.md
index a085211..29558b0 100644
--- a/docs/04-implementation/01-known-knowns-and-unknowns.md
+++ b/docs/04-implementation/01-known-knowns-and-unknowns.md
@@ -22,10 +22,12 @@ listed as an unknown-unknown has a *method* by which we will discover it.
### 1.1 Device and platform
* A stock 570.144 Linux guest driver probes and initializes against our
- emulated TU104 device; `nvidia-smi` works and CUDA user-space starts
- (`cuInit`, `cuDeviceGetCount`, `cuDeviceGet`). Context creation does not yet
- complete: `cuCtxCreate` returns `rc=999` (`CUDA_ERROR_UNKNOWN`) because the
- emulated engine list lacks the async copy engine (`COPY14`).
+ emulated TU104 device; `nvidia-smi -L` works and CUDA user-space starts
+ (`cuInit`, `cuDeviceGetCount`, `cuDeviceGet`). Context creation is in
+ progress: the client reaches `cuCtxCreate` and currently ends in a forced
+ `rc=999` on a completion wait; the engine inventory is complete (the async
+ copy engine is included) and the remaining work is in the completion and
+ work-execution path (see §2.1).
* The same guest **boots unmodified**: no guest code, no guest module
parameters are required for the initialization path.
* The host is unmodified in the standalone configuration (distribution NVIDIA
@@ -80,29 +82,33 @@ it**, **rough cost**.
### 2.1 Execution forwarding
-**Immediate blocker (2026-09-23):** the engine topology is not yet right.
-`GPU_GET_ENGINES_V2` returns 6 engines (GR + CE0..4) where a bare-metal oracle
-returns 7: the async copy engine (`COPY14`; engine descriptor `0x793ceb0e`,
-instance 14, client type `0x38`, class pair `0xc9b5`) is missing, so
-`cuCtxCreate` returns `rc=999` (`CUDA_ERROR_UNKNOWN`). The zeroed secondary
-engine record makes `libcuda` rewrite the channel alloc class to `0x41`
-(`NV01_ROOT_CLIENT`) with status `0x19` and roll back. Teardown of the failed
-context still oopses in the guest kernel in `uvm_pmm_devmem_page_to_va_space`.
-
-**Unknown:** the correct, isolated way to run a guest's engine work on the
-physical GPU: which real scheduling primitive to use per engine, how to
-translate guest pushes to real submissions, and how to relay completion and
-fault semantics.
+**Engine topology (solved):** `GPU_GET_ENGINES_V2` now reports the full engine
+inventory including the async copy engine, and the copy-engine capabilities
+match the physical device (two GRCEs). The earlier zeroed secondary engine
+record that made `libcuda` rewrite the channel alloc class and roll back is
+gone; context creation now progresses past engine selection and stalls later,
+on completion of decoded work (see §1.1).
+
+**Unknown (resolved by decision, not yet implemented):** the correct,
+isolated way to run a guest's engine work on the physical GPU. The interface
+analysis in
+[`../03-architecture/08-physical-gpu-integration.md`](../03-architecture/08-physical-gpu-integration.md)
+shows that proxying through the distribution host driver has no callable entry
+point from kernel context, so the **chosen mechanism** is for the module to own
+the physical GPU and run its real GSP firmware as the resource-manager server
+(the module becomes the host-side RM client). The emulation path remains the
+control plane.
**Why:** this is *the* blocker for CUDA kernel execution, and later for
graphics.
-**Plan:** evaluate the three backends (host RM/GSP proxy, direct engine
-submission, hybrid) against a scorecard: correctness, isolation, latency,
-maintenance cost across driver versions. Prototype the proxy first (it reuses
-the unmodified host driver's knowledge), measure latency, then decide whether a
-direct path is required. Deliverable: a forwarded `vectorAdd` with verified
-results.
+**Plan:** build the chosen mechanism in stages: device ownership + real GSP
+bring-up; host-side RM client for real objects and channels; translation of
+guest work and real submission; completion and fault relay. Measure each stage
+against a scorecard: correctness, isolation, latency, maintenance cost across
+driver versions. Deliverable: a forwarded `vectorAdd` with verified results.
+Alternatives (own scheduling without GSP; vendor vGPU or SR-IOV paths) are
+analyzed and rejected or deployment-gated in the integration doc.
**Cost:** weeks, not days; the largest single work item in the project.
diff --git a/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md b/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md
index e2769f8..3265f47 100644
--- a/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md
+++ b/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md
@@ -57,28 +57,33 @@ without a methodology is not accepted in this project.
memsets, copies and semaphore releases — is executed in software as a
correctness bridge; no CUDA kernel is forwarded yet.
-## 3. Forwarding: the three options and how we choose
+## 3. Forwarding: the options and the chosen mechanism
```
- (A) Host RM/GSP proxy
- guest push → translate → host driver's RM API → real channel → GPU
- + reuses a mature driver's knowledge
- − extra hop, depends on host driver interfaces
+ (A) Proxy through the host driver
+ guest push → translate → host driver's RM → real channel → GPU
+ Not implementable with an unmodified distribution driver: no callable
+ RM entry point from kernel context (see 03-architecture/08).
- (B) Direct engine submission
- guest push → translate → our own channel/push construction → GPU
- + lowest latency, no host driver dependency
- − we must own page tables, scheduling, invalidation, completion
+ (B) Own the GPU; own scheduling (no GSP)
+ guest push → translate → our own channels/runlists/page tables → GPU
+ Reproduces the scheduler and MMU ownership; large, capability-limiting.
(C) Hybrid
- management via (A), hot path via (B)
+ Only meaningful once (A) and (B) both exist.
+
+ (D) Own the GPU; use its real GSP firmware as the server ← chosen
+ guest push → translate → real GSP-RM session → real channel → GPU
+ The module is the host-side resource-manager client; the firmware
+ schedules.
```
-**Decision procedure:** prototype (A) first to get *correct* execution, with
-measurement harnesses around it. Then measure the overhead. If the overhead
-violates the budgets in §2.2 for the target workloads, implement (B) for the
-hot path and keep (A) for management. The architecture must not preclude
-either, so the work path is abstracted behind an interface from day one.
+The full analysis — which interfaces exist, why the proxy is unavailable, and
+the ownership model — is in
+[`../03-architecture/08-physical-gpu-integration.md`](../03-architecture/08-physical-gpu-integration.md).
+The work path stays abstracted behind one interface so the backend can be
+swapped. Software execution of decoded state/memory operations remains a
+correctness bridge for bring-up, not the data plane.
## 4. Hard problems we must solve, and our stance on each
diff --git a/docs/05-operations/01-performance-correctness-security.md b/docs/05-operations/01-performance-correctness-security.md
index 583d6c8..3a2c6b4 100644
--- a/docs/05-operations/01-performance-correctness-security.md
+++ b/docs/05-operations/01-performance-correctness-security.md
@@ -17,11 +17,12 @@ Rung 4 SUSTAINED workloads run for hours; faults/errors recoverable
Rung 5 MULTI-TENANT N VMs, isolation + fairness + no cross-talk
```
-**Current rung (2026-09-23):** Rung 1 is green; **Rung 2 is NOT green.**
-`cuInit`/`cuDeviceGet` succeed, but `cuCtxCreate` returns `rc=999`
-(`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks the async copy
-engine (`COPY14`); `cuMemAlloc` is not reached and teardown of the failed
-context still oopses in the guest kernel. Before Rung 3 is claimed, the
+**Current rung (2026-09-29):** Rung 1 is green; **Rung 2 is in progress.**
+`cuInit`/`cuDeviceGet` succeed, and the client reaches `cuCtxCreate`, which
+currently ends in a forced `rc=999` on a completion wait (engine inventory
+complete; completion translation in progress); `cuMemAlloc` is not reached yet
+and failed-context teardown has oopsed in the guest kernel in earlier runs.
+Before Rung 3 is claimed, the
**reply-fidelity gate** applies: a control whose reply fields the driver
consumes is not implemented until the reply is coherent (status-only →
coherent zeroed params → coherent values for consumed fields → exact values) —
diff --git a/docs/05-operations/02-debugging-and-maintenance-workflows.md b/docs/05-operations/02-debugging-and-maintenance-workflows.md
index 52cee94..801cda6 100644
--- a/docs/05-operations/02-debugging-and-maintenance-workflows.md
+++ b/docs/05-operations/02-debugging-and-maintenance-workflows.md
@@ -136,13 +136,11 @@ userspace/cuda/tools/callwalk.py <elf-index>/callgraph.tsv callers 0xNNNNNN
* **A/B harness**: `tools/cuda-ab/` runs one deterministic CUDA workload on
bare metal and in the guest and diffs the stable result lines.
-**Known current bugs (2026-09-23)**: `cuCtxCreate` returns `rc=999`
-(`CUDA_ERROR_UNKNOWN`) because the emulated engine list is missing the async
-copy engine `COPY14`; the guest kernel oopses in
-`uvm_pmm_devmem_page_to_va_space` on teardown of the failed context; and
-control `0x2080012b` (560-byte params, issued immediately before context
-creation) is answered with a generic SUCCESS whose consumed fields are not yet
-coherent. Start at
+**Current frontier (2026-10-12)**: context creation and the full driver-API
+smoke pass (`ALL OK`, 64-byte payload round-trips); a guest CUDA kernel
+launch is forwarded but the engine does not execute the forwarded launch
+descriptor yet, and green runs are not reproducible on every boot (tracked as
+an open issue in the repository). Start at
[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md),
[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../../userspace/cuda/docs/03-driver-and-gpu-interface.md)
and
diff --git a/docs/06-portability/hardware-software-matrix.md b/docs/06-portability/hardware-software-matrix.md
index 49c4bbe..c839578 100644
--- a/docs/06-portability/hardware-software-matrix.md
+++ b/docs/06-portability/hardware-software-matrix.md
@@ -131,9 +131,9 @@ the profile files, not the profile schema.
### 6.2 Engine-list portability
The engine inventory, and how it is filtered by the class list (classDB), are
-per-architecture and per-driver data. Today's gap is concrete: the emulated
-list lacks the async copy engine (`COPY14`), which blocks context creation
-(`cuCtxCreate` returns `rc=999`). Engine inventory belongs in the hardware
+per-architecture and per-driver data. An earlier gap was concrete — a missing
+async copy engine (`COPY14`) blocked context creation until the inventory was
+completed — and it showed why the inventory belongs in the hardware
profile, validated against the host oracle per combination. For the
re-analysis workflow see
[`../../userspace/cuda/docs/07-version-upgrade-playbook.md`](../../userspace/cuda/docs/07-version-upgrade-playbook.md);
diff --git a/docs/08-ecosystem/haaeep-and-collaboration.md b/docs/08-ecosystem/haaeep-and-collaboration.md
index e559840..954438c 100644
--- a/docs/08-ecosystem/haaeep-and-collaboration.md
+++ b/docs/08-ecosystem/haaeep-and-collaboration.md
@@ -92,7 +92,7 @@ licensed and version-durable — so it is the component we are building.
| Milestone | Depends on |
|---|---|
-| CUDA execution in a guest (rung 3) | Rung 2 (runtime start) being green first — it is not yet: `cuCtxCreate` returns `rc=999` because the emulated engine list lacks the async copy engine (`COPY14`) — and then Open-IOV forwarding work. The interface contract is documented under [`../../userspace/cuda/README.md`](../../userspace/cuda/README.md) (start at [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md)) |
+| CUDA execution in a guest (rung 3) | Rung 2 (runtime start) being green first — context creation is in progress (`cuCtxCreate` currently ends in a forced `rc=999` on a completion wait) — and then Open-IOV forwarding work (backend chosen: a module-owned GPU with a real-GSP session). The interface contract is documented under [`../../userspace/cuda/README.md`](../../userspace/cuda/README.md) (start at [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md)) |
| Two-VM coexistence (inference + agent) with isolation | Open-IOV multi-tenancy + platform MAC |
| Xen/OpenXT or Qubes Air variant | Open-IOV transport backends |
| Attested boot-to-inference chain | TrenchBoot integration |
diff --git a/docs/09-reference/glossary-and-faq.md b/docs/09-reference/glossary-and-faq.md
index 2ec4faa..e0c68f6 100644
--- a/docs/09-reference/glossary-and-faq.md
+++ b/docs/09-reference/glossary-and-faq.md
@@ -5,7 +5,7 @@
| Term | Meaning |
|---|---|
| **AMAPlib** | NVIDIA's shared "address mapping" library: swizzling, bank/channel mapping, aperture rules. Studied for page-table and memory-layout semantics. |
-| **Async copy engine (COPY14)** | A copy-engine instance found in the host oracle's engine list but missing from the emulated list — engine descriptor `0x793ceb0e`, instance 14, client type `0x38`, class pair `0xc9b5`. Its absence is the current CUDA context-creation blocker. |
+| **Async copy engine (COPY14)** | A copy-engine instance found in the host oracle's engine list; engine descriptor `0x793ceb0e`, instance 14, client type `0x38`, class pair `0xc9b5`. Its earlier absence from the emulated list was a context-creation blocker; the emulated inventory now includes it (two GRCEs). |
| **BAR0 / BAR1 / BAR3** | PCI BAR windows of the GPU: register interface (16 MB), VRAM aperture (256 MB), instance/PRAMIN window. |
| **CE** | Copy Engine: DMA engine; also used for UVM's internal work. |
| **Class list (classDB)** | The set of RM classes reported as supported (`GPU_GET_CLASSLIST`); engine queries are filtered against it, so the class list, engine list and caps controls must agree. |
@@ -63,14 +63,15 @@ Yes — in the standalone configuration the distribution driver owns the physica
GPU and Open-IOV is an independent module. Building NVIDIA's OpenRM modules from
source is only needed for the native-vGPU-management workflows.
-**Why is `nvidia-smi` working but CUDA not?**
-CUDA startup reaches `cuInit` / `cuDeviceGet`, but context creation is blocked:
-the emulated engine list lacks the async copy engine (`COPY14`), so
-`cuCtxCreate` returns `rc=999` (`CUDA_ERROR_UNKNOWN`) and memory allocation
-never runs. (The earlier `libcuda` segfault on this path was fixed by echoing
-the request params back in the alloc reply.) Once context creation is green,
-CUDA still needs *execution*: kernels must run on real hardware. See the
-known-knowns document for the exact status and plan; start at
+**Why is `nvidia-smi` working but my CUDA kernel not running?**
+The driver-API path works: `cuInit`, device enumeration, `cuCtxCreate` and
+the full smoke test (allocation, copies, free and destroy) complete, and a
+guest kernel launch is forwarded to the physical engine queue. What has not
+been observed yet is *execution*: the engine does not execute the forwarded
+launch descriptor and no engine-authored release appears. That is the current
+frontier, not a setup problem; the chosen execution backend is documented in
+[`../03-architecture/08-physical-gpu-integration.md`](../03-architecture/08-physical-gpu-integration.md).
+See the known-knowns document for the exact status and plan; start at
[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md).
**How is this legal / license-clean?**
diff --git a/docs/README.md b/docs/README.md
index ed0f206..24b2a21 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -90,31 +90,33 @@ and [`01-orientation/02-the-ai-edge-and-ecosystem.md`](01-orientation/02-the-ai-
## 3. Where is the project today?
-**Honest status (2026-09-23):**
+**Honest status (2026-09-29):**
| Area | State |
|------|-------|
| Emulated vGPU device (PCI, BARs, MSI-X, VFIO mdev) | ✅ Working |
| GSP boot emulation (FWSEC/booter/RISCV handshake) | ✅ Working |
| GSP-RM message queue + RPC transport | ✅ Working |
-| GSP-RM controls: device/fb/bus tables, CE caps, vaspace, GMMU static info | 🟡 Mostly — values are coherent for the special-cased controls; engine inventory/class-list consistency is the current CUDA blocker (COPY14) |
+| GSP-RM controls: device/fb/bus tables, CE caps, vaspace, GMMU static info | 🟡 Mostly — the engine inventory now includes the async copy engine and the CE capabilities match the device; reply fidelity for the remaining consumed fields is still being tightened |
| Guest `nvidia.ko` probe, `nvidia-smi` | ✅ Working (TU104) |
-| Guest CUDA driver (`libcuda`) startup | 🟡 Partially — `cuInit`/device enumeration work; `cuCtxCreate` returns `999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine list is missing the async copy engine (COPY14). See [`04-implementation/01-known-knowns-and-unknowns.md`](04-implementation/01-known-knowns-and-unknowns.md) and [`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md) |
-| CUDA kernel execution | 🔴 Not yet — context creation must succeed first; then GPU work forwarding (today only decoded memsets/copies/semaphore releases execute, in the host) |
+| Guest CUDA driver (`libcuda`) startup | 🟡 Partially — `cuInit` and device enumeration work; the client reaches `cuCtxCreate` and currently ends in a forced `999` on a completion wait while the completion/translation plumbing is made reliable. See [`04-implementation/01-known-knowns-and-unknowns.md`](04-implementation/01-known-knowns-and-unknowns.md) and [`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md) |
+| CUDA kernel execution | 🔴 Not yet — context creation must complete first, then real work forwarding; the execution backend is chosen but not started ([`03-architecture/08-physical-gpu-integration.md`](03-architecture/08-physical-gpu-integration.md)); today only decoded memsets/copies/semaphore releases execute, in the host |
| OpenGL / Vulkan | 🔴 Not yet |
| Windows guest | 🔴 Not yet |
| Multi-tenant isolation (VFs, mdev-per-VM) | 🟡 Designed, partially implemented |
We also know exactly where the CUDA bring-up stands and what the next step is:
the userspace `libcuda` segfault was fixed (an RM allocation reply now echoes
-the request parameters instead of returning zeros), and the remaining blocker
-is data: the engine list the emulation reports must include the async copy
-engine and stay consistent with the class list, because `libcuda` filters
-engines against it. The full mechanism, with offsets and the decompiled call
-path, is in [`../userspace/cuda/docs/05-570.144-decompilation.md`](../userspace/cuda/docs/05-570.144-decompilation.md).
+the request parameters instead of returning zeros), the engine inventory is
+complete, and the client reaches context creation, which currently ends on a
+completion wait while decoded work is made to execute reliably. The full
+mechanism, with offsets and the decompiled call path, is in
+[`../userspace/cuda/docs/05-570.144-decompilation.md`](../userspace/cuda/docs/05-570.144-decompilation.md).
Beyond that, the missing piece for kernel execution is forwarding the guest's
*work* (push buffers, page trees, execution) to the physical GPU under our
-control; see [`04-implementation/02-practical-challenges-and-adversarial-interop.md`](04-implementation/02-practical-challenges-and-adversarial-interop.md).
+control; the chosen backend — a module-owned GPU running its real GSP firmware
+as the resource-manager server — is specified in
+[`03-architecture/08-physical-gpu-integration.md`](03-architecture/08-physical-gpu-integration.md).
## 4. How to read these docs (progressive disclosure)
@@ -123,9 +125,9 @@ Read in order if you are new. Jump directly if you know what you need.
> **Debugging the CUDA bring-up?** Start with
> [`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md) —
> a five-minute guide to the userspace driver analysis, the corrected crash
-> path, the current engine-topology blocker, and the lookup recipes for
-> `libcuda+0x...` offsets. It links back into this tree for the interface and
-> emulation contract.
+> path, the current completion frontier and the chosen execution backend, and
+> the lookup recipes for `libcuda+0x...` offsets. It links back into this tree
+> for the interface and emulation contract.
| # | Document | Audience | Time |
|---|----------|----------|------|
@@ -139,14 +141,15 @@ Read in order if you are new. Jump directly if you know what you need.
| 8 | [GMMU, AMAPlib, HALs, runlist, GR](03-architecture/05-gmmu-amap-hal-gr.md) | developers | 25 min |
| 9 | [Host module and hybrid model](03-architecture/06-host-module-and-hybrid-model.md) | developers | 25 min |
| 10 | [Driver lifecycle field guide](03-architecture/07-driver-lifecycle-field-guide.md) | developers, debuggers | 30 min |
-| 11 | [Known-knowns and known-unknowns](04-implementation/01-known-knowns-and-unknowns.md) | everyone technical | 25 min |
-| 12 | [Practical challenges and adversarial interoperability](04-implementation/02-practical-challenges-and-adversarial-interop.md) | developers, stakeholders | 20 min |
-| 13 | [Performance, correctness, security](05-operations/01-performance-correctness-security.md) | developers, auditors | 25 min |
-| 14 | [Debugging and maintenance workflows](05-operations/02-debugging-and-maintenance-workflows.md) | developers, maintainers | 25 min |
-| 15 | [Hardware and software matrix](06-portability/hardware-software-matrix.md) | developers, integrators | 20 min |
-| 16 | [Developer onboarding and standards](07-developers/onboarding-and-standards.md) | contributors | 20 min |
-| 17 | [HAAEEP and collaboration](08-ecosystem/haaeep-and-collaboration.md) | stakeholders, partners | 20 min |
-| 18 | [Glossary and FAQ](09-reference/glossary-and-faq.md) | everyone | reference |
+| 11 | [Physical GPU integration](03-architecture/08-physical-gpu-integration.md) | developers, architects | 20 min |
+| 12 | [Known-knowns and known-unknowns](04-implementation/01-known-knowns-and-unknowns.md) | everyone technical | 25 min |
+| 13 | [Practical challenges and adversarial interoperability](04-implementation/02-practical-challenges-and-adversarial-interop.md) | developers, stakeholders | 20 min |
+| 14 | [Performance, correctness, security](05-operations/01-performance-correctness-security.md) | developers, auditors | 25 min |
+| 15 | [Debugging and maintenance workflows](05-operations/02-debugging-and-maintenance-workflows.md) | developers, maintainers | 25 min |
+| 16 | [Hardware and software matrix](06-portability/hardware-software-matrix.md) | developers, integrators | 20 min |
+| 17 | [Developer onboarding and standards](07-developers/onboarding-and-standards.md) | contributors | 20 min |
+| 18 | [HAAEEP and collaboration](08-ecosystem/haaeep-and-collaboration.md) | stakeholders, partners | 20 min |
+| 19 | [Glossary and FAQ](09-reference/glossary-and-faq.md) | everyone | reference |
Deep-dive material lives in [`../wiki/`](../wiki/README.md) (firmware
decompilation record, address reference, historical protocol notes). Wiki
diff --git a/userspace/cuda/570.144/docs/ANALYSIS-cuctxcreate-segv.md b/userspace/cuda/570.144/docs/ANALYSIS-cuctxcreate-segv.md
index db38802..666a4e2 100644
--- a/userspace/cuda/570.144/docs/ANALYSIS-cuctxcreate-segv.md
+++ b/userspace/cuda/570.144/docs/ANALYSIS-cuctxcreate-segv.md
@@ -3,8 +3,9 @@
> **Status: historical (2026-09-23 PART 9).** The segfault described here was
> fixed in PART 10 (the RM allocation reply now echoes the request parameters,
> which prevents a 16-byte class-params copy-back from zeroing the caller's
-> saved frame pointer). The current blocker is the missing async copy engine in
-> the emulated engine list (`cuCtxCreate` returns `999`). For the corrected,
+> saved frame pointer). The engine-topology blocker that followed was also
+> since resolved (the async copy engine is included); the current frontier is
+> the completion/execution path. For the corrected,
> current analysis see
> [`../../docs/05-570.144-decompilation.md`](../../docs/05-570.144-decompilation.md)
> and [`../../READ-ME-FIRST.md`](../../READ-ME-FIRST.md). The runtime
@@ -22,7 +23,7 @@ First analysis note for the CUDA userspace driver under Open-IOV. Companion to
|---|---|
| `cuInit(0)` | **OK** (previous sessions: failed) |
| `cuDeviceGetCount` / `cuDeviceGet` | **OK** (device 0) |
-| `cuCtxCreate` | **SIGSEGV inside libcuda** - the current blocker |
+| `cuCtxCreate` | **SIGSEGV inside libcuda** - the blocker at this snapshot (since fixed) |
| `cuMemAlloc` / copies / `ALL OK` | not reached yet |
Guest: Ubuntu 24.04, driver **570.144** (open kernel modules + libcuda
diff --git a/userspace/cuda/READ-ME-FIRST.md b/userspace/cuda/READ-ME-FIRST.md
index 2abb797..9adcf75 100644
--- a/userspace/cuda/READ-ME-FIRST.md
+++ b/userspace/cuda/READ-ME-FIRST.md
@@ -24,14 +24,16 @@ tooling.
ioctl traces) are still valid; any conclusion drawn from its decompiled
code must be re-derived. The new tooling translates addresses for you:
`tools/ghidra_decompile_at.sh` and the annotated corpus.
-2. **`docs/05` §3 — the `cuCtxCreate` path: crash solved, engine topology
- blocking.** The old fault at `libcuda+0x453f90` (helper at `+0x453ef0`) is
- solved: the alloc reply must **echo the request's params**, because the
- driver copies 16 bytes of class params back over the caller's 8-byte local
- (the hook's saved `rbp`) — a zeroed reply is what made `rbp=0` (§3.5). The
- current blocker is one step later: the class-filtered engine list is
- missing the async copy engine `COPY14`, so `cuCtxCreate` returns `rc=999`
- (§3.6). §3.3 gives the engine-record layout (`base+0x2fec`,
+2. **`docs/05` §3 — the `cuCtxCreate` path: crash and engine topology
+ solved, completion is the frontier.** The old fault at
+ `libcuda+0x453f90` (helper at `+0x453ef0`) is solved: the alloc reply must
+ **echo the request's params**, because the driver copies 16 bytes of class
+ params back over the caller's 8-byte local (the hook's saved `rbp`) — a
+ zeroed reply is what made `rbp=0` (§3.5). The engine inventory is now
+ complete (the async copy engine is included); context creation reaches a
+ completion wait that currently ends in `rc=999` (§3.6 carries the
+ historical analysis and a status note). §3.3 gives the engine-record layout
+ (`base+0x2fec`,
`+0x300c+(idx-1)*0x14`, `+0x350c`, `+0x351c+(idx-0x16)*0x14`; fields
+0/+8/+0xc) and §3.7 lists exactly what the emulation should check first.
@@ -45,7 +47,7 @@ Read in this order; each builds on the previous one.
|---|---|---|---|
| 1 | `docs/01-architecture.md` | 15 min | The mental model: layers (API → engine → driver-model → hardware), context/device/stream/channel/memory objects, the marker + tracking-semaphore sync model, launch/QMD flow, UVM integration, threading. This is the vocabulary the rest of the docs use. |
| 2 | `docs/03-driver-and-gpu-interface.md` | 15 min | The emulation contract: device nodes and every ioctl CUDA issues, RM object classes, control families, memory/VA/mapping, channels/GPFIFO/USERD/doorbells, semaphores/notifiers/interrupts, UVM, GSP-RM. Ends with an 8-point emulation checklist. |
-| 3 | `docs/05-570.144-decompilation.md` | 10 min | The actual binary: function counts, address conventions, API stub → wrapper → implementation mechanism, the `cuCtxCreate` crash-path resolution and current engine-topology blocker, and how to look anything up in the corpus. |
+| 3 | `docs/05-570.144-decompilation.md` | 10 min | The actual binary: function counts, address conventions, API stub → wrapper → implementation mechanism, the `cuCtxCreate` crash-path resolution and engine-topology fix, the completion/execution frontier, and how to look anything up in the corpus. |
| 4 | `docs/04-virtualization-and-emulation.md` | 15 min | Design doc: how NVIDIA's vGPU split works, how Open-IOV differs (emulate the interface, execute the work), and the anti-fragility/performance/portability/security principles that constrain any fix you make. |
| 5 | `docs/02-build-and-execution.md` | 10 min | How the library is built (generated API wrappers, entry tables, embedded device code) and how it executes in a process (init sequence, threads, interposition, `/dev` access, why LD_PRELOAD can't see its syscalls). |
@@ -148,8 +150,8 @@ userspace/cuda/
## 6. The shortest possible version
-1. `docs/05` §1.1 and §3 — address fix + the `cuCtxCreate` path (crash
- solved; engine-topology blocker) (read now).
+1. `docs/05` §1.1 and §3 — address fix + the `cuCtxCreate` path (crash and
+ engine topology solved; completion/execution frontier) (read now).
2. `docs/01` and `docs/03` — the model and the contract.
3. `docs/04` — the design principles your fix must respect.
4. `docs/06` — the recipes; then `tools/`.
diff --git a/userspace/cuda/README.md b/userspace/cuda/README.md
index 9c3163d..aaeedff 100644
--- a/userspace/cuda/README.md
+++ b/userspace/cuda/README.md
@@ -27,7 +27,7 @@ Decompiled output is never committed.
| [`docs/02-build-and-execution.md`](docs/02-build-and-execution.md) | how the driver is compiled and how it executes in a process (exports, generated wrappers, entry tables, embedded device code) |
| [`docs/03-driver-and-gpu-interface.md`](docs/03-driver-and-gpu-interface.md) | everything the driver touches below itself: device nodes, RM controls, memory, channels, doorbells, UVM, interrupts |
| [`docs/04-virtualization-and-emulation.md`](docs/04-virtualization-and-emulation.md) | NVIDIA's vGPU split, how Open-IOV differs, anti-fragility, performance, portability, security |
-| [`docs/05-570.144-decompilation.md`](docs/05-570.144-decompilation.md) | the 570.144 binary map, key functions, call chains, the solved `cuCtxCreate` crash, and the current engine-topology blocker |
+| [`docs/05-570.144-decompilation.md`](docs/05-570.144-decompilation.md) | the 570.144 binary map, key functions, call chains, the solved `cuCtxCreate` crash and engine-topology blocker, and the completion/execution frontier |
| [`docs/06-tooling.md`](docs/06-tooling.md) | the reusable toolchain: ELF toolkit, Ghidra analysis/decompilation, name transfer, version deltas |
| [`docs/07-version-upgrade-playbook.md`](docs/07-version-upgrade-playbook.md) | step-by-step method to re-run the analysis on a new driver version |
| [`docs/08-virtualization-approaches.md`](docs/08-virtualization-approaches.md) | one-page comparison of execution-forwarding options (software emulation, host-RM proxy, direct submission, hybrid) and where the project stands |
diff --git a/userspace/cuda/docs/05-570.144-decompilation.md b/userspace/cuda/docs/05-570.144-decompilation.md
index 4e23866..deec463 100644
--- a/userspace/cuda/docs/05-570.144-decompilation.md
+++ b/userspace/cuda/docs/05-570.144-decompilation.md
@@ -89,12 +89,13 @@ This is the mechanism described in `docs/01` §2.2/§2.3: the wrapper does the
tracing/tool callbacks and then calls the hand-written implementation. When
chasing a call, follow the slot; when a tool interposes, the slot changes.
-## 3. The `cuCtxCreate` path: crash solved, engine topology blocking
+## 3. The `cuCtxCreate` path: crash and engine topology solved, completion is the frontier
The 2026-09 crash was deterministic: on a fresh guest `cuCtxCreate` segfaulted
-at `libcuda+0x453f90`. That is **solved** (§3.5). The call now returns
-`rc=999` (`CUDA_ERROR_UNKNOWN`), and the active blocker is engine topology —
-the emulated engine list is missing the async copy engine (§3.6).
+at `libcuda+0x453f90`. That is **solved** (§3.5). The engine inventory is now
+complete (the async copy engine is included); the call proceeds into context
+creation and currently ends in a forced `rc=999` on a completion wait. The
+sections below are the historical analysis of that call path.
Fault facts from the crash period (live traces, corrected decompilation):
@@ -233,6 +234,12 @@ the alloc reply's class-size copy-back described above.
### 3.6 Current blocker (2026-09-23): the missing async copy engine
+> **Status update (2026-09-29): resolved.** The emulated inventory now includes
+> the async copy engine (two GRCEs), and the class list, engine list and
+> capabilities agree; the `rc=999` seen today comes from a completion wait in
+> the work/translation path, not from engine topology. The analysis below is
+> kept as the record of how the topology blocker was diagnosed and fixed.
+
With the copy-back fixed, the walker completes but `cuCtxCreate` returns
`rc=999` (`CUDA_ERROR_UNKNOWN`); `cuMemAlloc` and later calls are not reached.
Measured with the guest ioctl trace plus a host A/B oracle (the same libcuda
diff --git a/userspace/cuda/docs/08-virtualization-approaches.md b/userspace/cuda/docs/08-virtualization-approaches.md
index 00ca13f..e67f648 100644
--- a/userspace/cuda/docs/08-virtualization-approaches.md
+++ b/userspace/cuda/docs/08-virtualization-approaches.md
@@ -20,6 +20,13 @@ principles) and to the project implementation docs (`docs/03-architecture/06-hos
fences, some memops) should be satisfied directly in the emulation regardless
of A/B/C — they are microseconds in hardware and would lose by round-tripping.
+**Availability with an unmodified distribution host driver:** approach A has no
+callable entry point from kernel context (see
+[`../../../docs/03-architecture/08-physical-gpu-integration.md`](../../../docs/03-architecture/08-physical-gpu-integration.md));
+approach B requires owning scheduling; the project's chosen mechanism is a
+module-owned device that keeps the GPU's real GSP firmware as the scheduling
+server. Approach E remains deployment-gated.
+
## 2. Design implications
* Where the boundary sits decides what we must own: A keeps ownership in the
@@ -40,25 +47,31 @@ of A/B/C — they are microseconds in hardware and would lose by round-tripping.
patch anyone* — a mediated-passthrough vGPU where PCI config/BARs, the GSP boot
handshake, the GSP-RM protocol and interrupt routing are emulated in
`open-iov-vfio.ko`, while real work is meant to execute on the physical GPU.
-The project's own decision procedure is explicit: **prototype A first for
-correctness, measure, then move the hot path to B if the budgets demand it**
-(option C in practice), with the work path abstracted from day one.
+The original decision procedure was to prototype A first for correctness; the
+interface analysis
+([`../../../docs/03-architecture/08-physical-gpu-integration.md`](../../../docs/03-architecture/08-physical-gpu-integration.md))
+showed that proxying through an unmodified distribution host driver has no
+callable entry point from kernel context, so the chosen mechanism is a
+**module-owned GPU running its real GSP firmware as the resource-manager
+server** (the module becomes the host-side RM client). The work path stays
+abstracted from day one so the backend can be swapped.
-**Current reality (2026-09, honest):** the *control plane* follows the
+**Current reality (2026-09-29, honest):** the *control plane* follows the
architecture — GSP-RM emulation, RPC handling, BAR/PCI/MSI-X emulation and
object/control replies are implemented, and the guest's stock driver +
`libcuda` now get through `cuInit`, `cuDeviceGetCount`/`cuDeviceGet` and into
`cuCtxCreate`. The old `cuCtxCreate` segfault is solved (the alloc reply must
-echo the request's params — `docs/05` §3.5); context creation is blocked one
-step later by engine topology: the class-filtered engine list is missing the
-async copy engine, so the walker reads a zeroed secondary record and the call
-returns `rc=999` (`docs/05` §3.6). The *data plane* is at the bridge stage: the
-host module decodes the guest's push buffers as method streams and **executes
-the small work it understands itself** (memsets, copies, semaphore releases)
-instead of forwarding it to the physical GPU. No CUDA kernel has been
-forwarded yet. So: yes, we are pursuing the intended architecture, but we are
-in the "emulate + software-execute state/memory ops" phase of it, with the
-A→B/C forwarding decision still ahead.
+echo the request's params — `docs/05` §3.5), and the engine inventory is now
+complete (the async copy engine is included; the copy-engine capabilities
+match the device). Context creation currently ends in a forced `rc=999` on a
+completion wait while the completion/translation plumbing is being made
+reliable. The *data plane* is at the bridge stage: the host module decodes the
+guest's push buffers as method streams and executes the small work it
+understands itself (memsets, copies, semaphore releases) instead of forwarding
+it to the physical GPU. No CUDA kernel has been forwarded yet; the execution
+backend is chosen but not started. So: we are still in the
+"emulate + software-execute state/memory ops" phase, with the backend decision
+now made (option D in the integration document).
**Risks to guard now:** (1) software execution silently becoming the data
plane instead of a bridge; (2) the work-path interface being skipped because