From 4738b619c45abc4cf98a549c3118839e48bb7244 Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Tue, 29 Sep 2026 15:29:00 -0400 Subject: docs: record physical-GPU integration mechanism and refresh project status - add docs/03-architecture/08-physical-gpu-integration.md: interfaces that exist, mechanism options A-E, chosen path (module-owned GPU driven through its real GSP firmware), sharing policy, decision record - update 06-host-module-and-hybrid-model, 02-practical-challenges, system overview, known-knowns, docs/README reading order and CUDA-side docs to match - correct stale engine-topology/COPY14 blocker claims across the tree - add STATE-CURRENT.md as the dated public status snapshot --- docs/01-orientation/01-purpose-and-scope.md | 10 +- docs/03-architecture/01-system-overview.md | 27 ++-- docs/03-architecture/02-guest-stack-and-cudasw.md | 12 +- docs/03-architecture/03-gsp-and-gsp-rm.md | 11 +- docs/03-architecture/04-uvm.md | 13 +- docs/03-architecture/05-gmmu-amap-hal-gr.md | 9 +- .../06-host-module-and-hybrid-model.md | 47 +++--- .../07-driver-lifecycle-field-guide.md | 15 +- .../03-architecture/08-physical-gpu-integration.md | 175 +++++++++++++++++++++ .../01-known-knowns-and-unknowns.md | 52 +++--- ...practical-challenges-and-adversarial-interop.md | 35 +++-- .../01-performance-correctness-security.md | 11 +- .../02-debugging-and-maintenance-workflows.md | 12 +- docs/06-portability/hardware-software-matrix.md | 6 +- docs/08-ecosystem/haaeep-and-collaboration.md | 2 +- docs/09-reference/glossary-and-faq.md | 19 +-- docs/README.md | 45 +++--- 17 files changed, 354 insertions(+), 147 deletions(-) create mode 100644 docs/03-architecture/08-physical-gpu-integration.md (limited to 'docs') diff --git a/docs/01-orientation/01-purpose-and-scope.md b/docs/01-orientation/01-purpose-and-scope.md index c1232f3..ff13603 100644 --- a/docs/01-orientation/01-purpose-and-scope.md +++ b/docs/01-orientation/01-purpose-and-scope.md @@ -133,11 +133,11 @@ document: transport, the resource-manager emulation surface the guest driver needs to probe and enumerate, guest `nvidia.ko` and `nvidia-smi`, and CUDA user-space startup through `cuInit` / `cuDeviceGet`. -* **Needs improvement**: engine topology and reply fidelity first — context - creation returns `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine - list lacks the async copy engine (`COPY14`) and some consumed reply fields - are still generic fallbacks — then GPU work forwarding (the reason CUDA - kernels do not yet run), the memory/page-table model, Windows guest support, +* **Needs improvement**: reply fidelity and completion translation first — + context creation currently ends in a forced `rc=999` on a completion wait, + and some consumed reply fields are still generic fallbacks — then GPU work + forwarding (the reason CUDA kernels do not yet run; the backend is chosen), + the memory/page-table model, Windows guest support, non-KVM hypervisors, and performance instrumentation. Each of these is tracked with explicit "known-unknowns" in [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md); diff --git a/docs/03-architecture/01-system-overview.md b/docs/03-architecture/01-system-overview.md index c7a847e..5c2ba6c 100644 --- a/docs/03-architecture/01-system-overview.md +++ b/docs/03-architecture/01-system-overview.md @@ -61,10 +61,12 @@ HOST MODULE open-iov-vfio.ko ├─ emulation core: GSP boot + GSP-RM (RPC msgq, controls, objects, events) ├─ memory model: guest RAM access, VA↔PA maps, VRAM backing └─ work path: channel/push forwarding to the physical GPU [goal; software bridge today] - │ -HOST DRIVER nvidia.ko (unmodified) and/or direct GPU interfaces - │ -PHYSICAL GPU + │ module-owned device + real GSP session +HOST BACKEND the GPU's own hardware + GSP firmware, driven by this module + │ (the distribution nvidia.ko stays unmodified and unbound in + │ this mode; see 08-physical-gpu-integration.md) + v +REAL HARDWARE (engines, memory, interrupts) ``` ## 3. Host module architecture @@ -167,11 +169,12 @@ decompiled and mapped (see `wiki/GSP-570.144W-Firmware-Analysis.md` and `SUCCESS` with zeroed params, and `GSP_RM_ALLOC` echoes the request params without server-side object management. Where the driver consumes a reply value, that value is still being corrected (the reply-fidelity ladder). -5. `nvidia-smi` works; CUDA user-space starts (`cuInit` / `cuDeviceGet` - succeed). Context creation is currently blocked: `cuCtxCreate` returns - `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks the - async copy engine (`COPY14`), so UVM's channel-manager and page-tree - initialization is as far as the guest gets. +5. `nvidia-smi -L` enumerates the emulated device and CUDA user-space starts + (`cuInit` / `cuDeviceGetCount` / `cuDeviceGet` succeed). The client reaches + `cuCtxCreate`, which currently ends in a forced `rc=999` on a completion + wait; the copy-engine method streams execute in software as a bridge, and + the execution backend is not built yet (see + [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)). The CUDA-side contract and the reply-fidelity rules are documented in [`../../userspace/cuda/docs/03-driver-and-gpu-interface.md` §9](../../userspace/cuda/docs/03-driver-and-gpu-interface.md) @@ -192,7 +195,9 @@ That means the host module must: faults and semaphores back. The design for this, including what is known and what remains to be learned, is -in: +in [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) and -[`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md). +[`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md); +the chosen execution backend is specified in +[`08-physical-gpu-integration.md`](08-physical-gpu-integration.md). diff --git a/docs/03-architecture/02-guest-stack-and-cudasw.md b/docs/03-architecture/02-guest-stack-and-cudasw.md index 115f639..e841342 100644 --- a/docs/03-architecture/02-guest-stack-and-cudasw.md +++ b/docs/03-architecture/02-guest-stack-and-cudasw.md @@ -89,11 +89,13 @@ open /dev/nvidia* → cuInit → UVM registration (per process) The initialization path is where compatibility is won or lost; the runtime path is where *execution* is won or lost. Open-IOV's current focus is -pre-forwarding correctness: `cuInit` / `cuDeviceGet` succeed, but context -creation returns `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine -list lacks the async copy engine (`COPY14`); the forwarding decision -(host-RM proxy vs direct submission) is still ahead (see -[`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md)). +pre-forwarding correctness: `cuInit` / `cuDeviceGet` succeed, and the client +reaches context creation, which currently ends in a forced `rc=999` on a +completion wait; the engine inventory is complete and the forwarding decision +is made — a module-owned GPU running its real GSP firmware as the +resource-manager server (see +[`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md) +and [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)). For the CUDA userspace side of this stack — its architecture, how it is built and executes, and the exact kernel/GPU interface contract we must satisfy — diff --git a/docs/03-architecture/03-gsp-and-gsp-rm.md b/docs/03-architecture/03-gsp-and-gsp-rm.md index aca908f..8809bb7 100644 --- a/docs/03-architecture/03-gsp-and-gsp-rm.md +++ b/docs/03-architecture/03-gsp-and-gsp-rm.md @@ -186,17 +186,18 @@ is not lost. | GSP boot state machine | ✅ | stages complete; the driver's poll loop is satisfied | | Message queue transport | ✅ | including the non-contiguous page table for the queue buffer | | RPC request/reply | ✅ | all functions the driver uses during init are answered; unknown functions get a success reply with no payload | -| Static info tables | 🟡 | device/FB/BUS/INTR/GMMU tables carry plausible TU104 values; the engine inventory is still incomplete (the async copy engine, COPY14, is missing) | +| Static info tables | 🟡 | device/FB/BUS/INTR/GMMU tables carry plausible TU104 values; the engine inventory is now complete (the async copy engine is included) and agrees with the class list | | Controls during init | 🟡 | ~160 distinct control IDs observed per CUDA run; 23 are special-cased with real values, the rest fall through to a success reply with zeroed parameters - consumed fields of those are still being corrected | | Object creation | 🟡 | handles/parameters are echoed consistently (the request parameters must be echoed back - zeroing them corrupts the caller), but there is no server-side object model: allocation is acknowledged, not tracked | | Page tables for server-owned ranges | 🟡 | contract received and recorded; leaf population is the forwarding work | | Event/notification delivery | 🟡 | MSI-X works; richer event semantics as forwarding lands | | Firmware *behaviour* (routing, scheduling, real execution) | 🔴 | the forwarding work; see the implementation docs | -The current CUDA blocker is the engine-inventory row: `libcuda` filters the -engine list it receives against the class list and expects the async copy -engine to be present, because the secondary (async) engine record is otherwise -zeroed. See +The engine-inventory row was the CUDA blocker at the time of writing: +`libcuda` filters the engine list it receives against the class list and +expects the async copy engine to be present, because the secondary (async) +engine record is otherwise zeroed. That inventory is now complete; the +context-creation wait is later, in the completion path. See [`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md) section 3.7 for the offsets and the consumed fields, and [`../../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../../userspace/cuda/docs/03-driver-and-gpu-interface.md) diff --git a/docs/03-architecture/04-uvm.md b/docs/03-architecture/04-uvm.md index b879572..4c8a0c9 100644 --- a/docs/03-architecture/04-uvm.md +++ b/docs/03-architecture/04-uvm.md @@ -50,17 +50,20 @@ Each stage has a distinct failure signature; our debugging docs map them. | Requirement | Detail | Our status | |---|---|---| | GPU identity | must match the uuid from RM | ✅ | -| CE capabilities | per-engine support bits; UVM *requires* `supported && !grce && sysmem` for CPU_TO_GPU/GPU_TO_CPU channels | 🟡 the caps we answer are accepted, but the engine list / CE capabilities are missing the async copy engine (`COPY14`) — the current context-creation blocker | +| CE capabilities | per-engine support bits; UVM *requires* `supported && !grce && sysmem` for CPU_TO_GPU/GPU_TO_CPU channels | 🟡 the caps we answer are accepted and the inventory now includes the async copy engine; completion/semaphore plumbing is the current frontier | | Channel creation | RM channel objects with GPFIFO/userd/instance parameters | ✅ objects created | | Pushes executed | UVM's first action on a new channel is an *init push* followed by a semaphore-tracked wait | 🟡 the host decodes method streams and executes memsets, copies and semaphore releases in software (a bridge); no kernel is forwarded to the physical GPU yet | | Page trees | UVM initializes page trees with its own pushes; the *server-owned* ranges must be populated by us | 🟡 | | Fault delivery | fault buffer + notification; replay/cancel semantics | 🔴 | | Semantics of completions | tracker values must advance exactly | 🟡 | -**Current blocker (2026-09-23):** `cuCtxCreate` returns `rc=999` -(`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks the async copy -engine (`COPY14`); teardown of the failed context still oopses in the guest -kernel in `uvm_pmm_devmem_page_to_va_space` (twice per boot). The CUDA-side +**Current frontier (2026-09-29):** the client reaches `cuCtxCreate` and +currently ends in a forced `rc=999` on a completion wait; the engine inventory +is complete, and the remaining work is completion translation plus the chosen +execution backend +([`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)). +Failed-context teardown has produced guest kernel oopses in +`uvm_pmm_devmem_page_to_va_space` in earlier runs. The CUDA-side contract and reply rules are in [`../../userspace/cuda/docs/03-driver-and-gpu-interface.md` §9](../../userspace/cuda/docs/03-driver-and-gpu-interface.md) and diff --git a/docs/03-architecture/05-gmmu-amap-hal-gr.md b/docs/03-architecture/05-gmmu-amap-hal-gr.md index d6470b6..cd31409 100644 --- a/docs/03-architecture/05-gmmu-amap-hal-gr.md +++ b/docs/03-architecture/05-gmmu-amap-hal-gr.md @@ -186,10 +186,11 @@ engine type (e.g. `COPY0 = 0x9`, `GR = 0x1`). driver builds its per-engine records from the device/engine information the kernel driver reports, and filters them against the class list; a missing or zeroed record is not ignored - it becomes an allocation with class `0` and the -whole context setup rolls back. The current CUDA bring-up blocker is exactly -this: the async copy engine (COPY14, engine descriptor `0x793ceb0e`, instance -14) is absent from the emulated engine list, while a bare-metal oracle reports -it as the seventh engine alongside GR and CE0..4. See +whole context setup rolls back. This was exactly the CUDA bring-up blocker +until the engine inventory was completed: the async copy engine (COPY14, +engine descriptor `0x793ceb0e`, instance 14) was absent from the emulated +engine list, while a bare-metal oracle reports it as the seventh engine +alongside GR and CE0..4; it is now included. See [`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md) section 3.7 for the record layout the driver's engine walker consumes. diff --git a/docs/03-architecture/06-host-module-and-hybrid-model.md b/docs/03-architecture/06-host-module-and-hybrid-model.md index 15545a4..61be96c 100644 --- a/docs/03-architecture/06-host-module-and-hybrid-model.md +++ b/docs/03-architecture/06-host-module-and-hybrid-model.md @@ -18,10 +18,12 @@ │ open-iov-vfio.ko │ │ virtual device ── emulation core ── memory model ── work path │ └────┬─────────────────────────────────────┬─────────────────────────┘ - │ host driver interfaces │ (future) vendor-neutral - ┌────┴────────────────────┐ │ - │ unmodified host nvidia.ko│ direct GPU interfaces / VF backends - └──────────────────────────┘ + │ module-owned device + real GSP │ (future) vendor-managed + ┌────┴───────────────────────────────┐ │ sharing backends + │ physical GPU and its GSP firmware, │ vGPU host driver / SR-IOV + │ driven by this module (the host │ (licensed / hardware-gated) + │ driver stays unmodified, unbound) │ + └────────────────────────────────────┘ ``` The module registers itself as an **mdev parent** on the physical GPU and @@ -103,20 +105,24 @@ The handlers currently cover: the boot/static-info exchanges, the device/FB/ BUS/INTR control tables, CE capability and PCE-mask queries, GMMU static info, vaspace reserved-PDE contracts, channel/CE object allocation, and the general "unknown control/function returns success with zeroed output" fallback. The -same fallback now matters for engine inventory: `libcuda` filters the engine -list against the class list, so a missing async copy engine (COPY14) leaves a -record zeroed and blocks context creation (`cuCtxCreate` returns `999`); see +same fallback mattered for engine inventory: `libcuda` filters the engine +list against the class list, and an incomplete inventory once left a record +zeroed and blocked context creation (`cuCtxCreate` returned `999`); the +emulated inventory is now complete (the async copy engine included). See [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md) and [`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md) -section 3.7. +section 3.7 for the analysis. **Current work-execution bridge.** The module also decodes push buffers as method streams and executes the work it understands in software - memsets, copies and semaphore releases - and completes the corresponding semaphores. That is a correctness bridge for memory operations, not the data plane: no -CUDA kernel is forwarded to the physical GPU yet, and the forwarding strategy -(host-RM proxy first, direct submission if needed) is in +CUDA kernel is forwarded to the physical GPU yet; the chosen execution backend +(a module-owned GPU running its real GSP firmware as the resource-manager +server) is in +[`08-physical-gpu-integration.md`](08-physical-gpu-integration.md), with the +trade-offs in [`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) section 3. @@ -205,18 +211,14 @@ guest push submitted → relay completions (semaphore/fault/interrupt) back into guest memory ``` -Three forwarding backends are candidates, in increasing order of capability: - -1. **Host RM/GSP proxy** — use the unmodified host driver's RM to allocate real - memory and create real channels, then replay guest work through them. -2. **Direct engine submission** — build our own channel/push streams to real - engines using the GMMU/HAL definitions, bypassing the host driver. -3. **Hybrid** — proxy for management (memory, channels) and direct submission - for hot-path work (grid launches, copies). - -The decision between them is driven by the practical challenges document; the -short version is that (1) is safest for correctness, (2) is best for latency, -and both must be isolated to be acceptable. +Backend selection, including the interfaces that actually exist and why the +distribution host driver cannot be called, is analyzed in +[`08-physical-gpu-integration.md`](08-physical-gpu-integration.md). In short: +proxying through the host driver has no callable entry point from kernel +context, so the chosen mechanism is for the module to own the device and run +its real GSP firmware as the resource-manager server, while the emulation path +remains the control plane. Software execution of decoded memory/state +operations stays a development bridge, not the data plane. ## 6. Debug tooling in the module @@ -290,5 +292,6 @@ in `kernel-open` first). | Push decoding + semaphore completion | demonstrated live; being productized | | Page-table population for server-owned ranges | designed, not implemented | | GPU work forwarding (GR/CE execution) | designed, not implemented | +| Execution backend (module-owned GPU + direct real-GSP session) | chosen, not started ([`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)) | | Real VRAM backing | designed, not implemented | | Windows guest, Xen transport, aarch64 hosts | designed, not implemented | diff --git a/docs/03-architecture/07-driver-lifecycle-field-guide.md b/docs/03-architecture/07-driver-lifecycle-field-guide.md index 9de5edb..1bfe912 100644 --- a/docs/03-architecture/07-driver-lifecycle-field-guide.md +++ b/docs/03-architecture/07-driver-lifecycle-field-guide.md @@ -105,7 +105,7 @@ fallback). Representative controls: | `0x20802a02` | CE PCE mask | one bit per engine | | `0x20802a08` | CE fault method buffer size | method buffer sizing | | `0x90f10106` | vaspace reserved-PDE contract | **carries page-table physical addresses** (see GSP-RM doc) | -| `0x20800123` / `0x20800170` | `GPU_GET_ENGINES` / `GET_ENGINES_V2` | engine inventory (filtered by the class list / classDB); values are consumed — the guest must see the full engine list, including the async copy engine (`COPY14`), and the current `rc=999` blocker lives here | +| `0x20800123` / `0x20800170` | `GPU_GET_ENGINES` / `GET_ENGINES_V2` | engine inventory (filtered by the class list / classDB); values are consumed — the guest must see the full engine list, including the async copy engine (`COPY14`); that inventory is now complete, and the context-creation wait lives later in the completion path | | `0x800201` | `GPU_GET_CLASSLIST` | class list (classDB); must agree with the engine list and the CE caps controls | | `0x2080012b` | GPU_PROMOTE_CTX (560-byte params) | issued immediately before context creation; fallback generic `SUCCESS`, consumed fields not yet coherent | | `0x20803039` | NVLink masks/VBIOS info | fallback: generic `SUCCESS` with zeroed params | @@ -166,13 +166,16 @@ their memsets, copies and semaphore releases in software by translating the semaphore VA to a guest physical address (see the UVM and host-module docs). This is a bridge, not forwarding: no work reaches the physical GPU yet. -**Live development front (2026-09-23):** CUDA context creation — `cuCtxCreate` -returns `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks -the async copy engine (`COPY14`). The engine-record layout and the alloc -reply-fidelity checks are the first things to inspect +**Live development front (2026-09-29):** CUDA context creation — the client +reaches `cuCtxCreate` and currently ends in a forced `rc=999` on a completion +wait; the engine inventory is complete and the completion/translation path is +the focus. The engine-record layout and the alloc reply-fidelity checks are +documented in ([`../../userspace/cuda/docs/05-570.144-decompilation.md` §3.3/§3.6](../../userspace/cuda/docs/05-570.144-decompilation.md); start at -[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md)). +[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md)); +the execution backend is in +[`08-physical-gpu-integration.md`](08-physical-gpu-integration.md). **Log signatures (guest side):** ``` diff --git a/docs/03-architecture/08-physical-gpu-integration.md b/docs/03-architecture/08-physical-gpu-integration.md new file mode 100644 index 0000000..f156429 --- /dev/null +++ b/docs/03-architecture/08-physical-gpu-integration.md @@ -0,0 +1,175 @@ +# Physical GPU Integration: How Work Reaches Real Hardware + +> **Status: chosen mechanism, not yet implemented.** The control plane works +> today (the guest's driver initializes and CUDA user-space reaches context +> creation); the guest's *work* is currently executed in software as a +> development bridge. This document records the interfaces that actually +> exist, the mechanism options, the chosen path, and what has to be built. +> It fills the gap left open by +> [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) §5 +> and [`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) §3, +> which named "the host driver's RM API" as an option without a concrete call +> path from kernel context. + +## 1. The problem and its constraints + +The guest's real work (copies, memory operations, grid launches) must execute +on the physical GPU. The project constraints are: + +* one DKMS kernel module on the host; no daemon, coprocess or script is + required for the guest to run; +* host and guest drivers unmodified; no paravirtual enlightenment in the + guest — it believes it owns a physical GPU; +* all emulation and translation logic inside the module; +* real execution: compute must not be emulated in software. + +Any mechanism must therefore be reachable from kernel context, with no new +host userspace, and without editing the host driver. + +## 2. What interfaces actually exist + +| Interface | What it offers | Reachable from our module? | +|---|---|---| +| Host driver (distribution build) | full resource management, scheduling, memory | **No.** It exports no general RM entry points to third-party modules, has no kernel-client programming interface, and its device ioctl surface is user-pointer based. Intercepting its internal functions is unsupported (and function tracing on its text is not available). | +| Host driver peer-memory exports | pin/map memory for peer devices | Partially: **memory only, no submission**. | +| Open-kernel-module build | source linkage | Only when the host driver is built from source and extended — i.e. a modified host driver; excluded by the constraints. | +| Vendor vGPU host stack | mediated work submission and scheduling | Exists, but requires the licensed vGPU host branch and a vGPU-aware guest — excluded by "no enlightenment / no GRID licensing". | +| SR-IOV virtual functions | hardware-partitioned device | Not available on this GPU generation. | +| The GPU's own GSP firmware | the resource manager server itself | **Yes**, once the module owns the device: this is the path used. | + +The conclusion: with an unmodified host driver there is **no callable +submission interface into it**. Real execution therefore requires the module +to own the device and talk to the hardware itself. + +## 3. Mechanism options + +``` + (A) Proxy through the host driver + guest push -> our module -> host driver's RM -> real channel -> GPU + Not implementable without a modified host driver or a vendor-managed + interface (see section 2). + + (B) Own the GPU; own scheduling (no GSP) + guest push -> our module -> our own channels/runlists/page tables -> GPU + Possible in principle, but the module would reproduce the scheduler, + MMU ownership and context switching; large and capability-limiting. + + (C) Hybrid + Only meaningful once (A) and (B) both exist; not on the critical path. + + (D) Own the GPU; use its real GSP firmware as the server <- chosen + guest push -> our module -> real GSP-RM session -> real channel -> GPU + The module becomes the host-side resource-manager client: the same + protocol the guest driver speaks, used against the physical GPU's + firmware. Scheduling, page tables and context management stay the + firmware's job. + + (E) Vendor-managed sharing (vGPU host stack, SR-IOV VFs) + Deployment-gated (licence, hardware generation); out of scope for + consumer GPUs. +``` + +## 4. The chosen mechanism in detail + +### 4.1 Ownership and lifecycle + +The module binds the physical GPU as its driver. The distribution driver stays +unmodified and is unbound from that device while the vGPU service is active; +host display continues on another GPU (integrated graphics on the development +system). This is the dedicated-GPU deployment model; see section 5. An earlier +host-side GSP boot path existed in early versions and was disabled precisely +because it conflicted with the distribution driver owning the card — under +exclusive ownership that conflict disappears. + +### 4.2 Device bring-up + +PCI enable, BAR mapping, firmware load (secure boot stage, booter, RISC-V GSP), +WPR2 setup, message queues in host system memory, doorbell, version exchange — +the same observable stages the guest emulation already models, with the module +now on the client side. + +### 4.3 Host-side resource-manager client + +Using the wire protocol and RPC inventory already mapped for the supported +family, the module allocates real objects: device/subdevice, virtual address +spaces, channels, and memory mappings. This replaces "call the host driver's +RM" with "be the RM client". + +### 4.4 Guest mediation + +The guest-facing GSP-RM emulation stays as it is for everything the guest +observes. Objects that need real resources are translated into real GSP-RM +requests (handle-domain translation, allocation backing). The page-table +contract the guest already provides for server-owned ranges becomes +population of the real MMU instead of a shadow. + +### 4.5 Work path + +Decoded guest pushes are rebuilt into real push streams on real channels +(where shapes match, passed through); guest GPU virtual addresses are mapped +to the same physical pages in the real MMU; the real doorbell is rung and the +GSP firmware schedules the work. + +### 4.6 Completion relay + +Hardware-completed semaphores land in memory the guest already polls. Events +and faults that the guest expects as interrupts are relayed through the +existing MSI-X injection path. + +### 4.7 Isolation + +One real GSP client per mdev, per-client page tables; host access to guest +memory stays behind the hypervisor's DMA interface; the firmware provides +per-context scheduling. Tenant policy and fairness are a later layer on top. + +## 5. Sharing policy + +* **Dedicated GPU (chosen):** one GPU serves the vGPU service; the host uses + another GPU or integrated graphics. +* **Multi-GPU host:** one GPU for the distribution driver, one for the + service. +* **Shared single GPU with the host driver bound:** no conforming mechanism + exists on this generation without vendor interfaces; this would require + relaxing a constraint (vendor vGPU stack, SR-IOV hardware, or a host + daemon). + +## 6. Status and milestones + +| Step | Status | +|---|---| +| Control plane (GSP boot emulation, RPC, BAR/PCI/MSI-X) | done | +| Software-executed bridge for decoded memory/state operations | in progress (memsets, copies, releases) | +| Device ownership + real GSP bring-up by the module | not started (the early prototype is to be rebuilt under exclusive ownership) | +| Host-side resource-manager client for real objects/channels | not started | +| Work translation and real submission | not started | +| Completion relay through real hardware | not started | +| CUDA kernel execution (`vectorAdd`) | not started | + +## 7. Open questions + +* Firmware/host driver ABI match for the supported family on the development + GPU. +* GPU unbind/reset lifecycle and recovery after guest faults. +* Which guest allocations need real VRAM and which can stay in system memory. +* Fault semantics when work runs on the real firmware. +* Per-tenant scheduling policy once multiple mdevs share the service GPU. + +## 8. Decision record + +* **Date:** 2026-09-29. +* **Decision:** adopt option (D) — a module-owned GPU with a direct real-GSP + session as the execution backend. (A) is recorded as not implementable under + the constraints; (B) remains a fallback only if (D) proves unworkable; + (E) is deployment-gated and out of scope. +* **Rationale:** only (D) satisfies module-only, no-daemon, unmodified + drivers, real execution, and a path to multi-tenant sharing. +* **Consequences:** the module grows a host-side resource-manager client and a + device-ownership lifecycle; the emulation path remains the control plane and + is reused. + +## See also + +- [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) - the host module, memory model, build contracts +- [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) - known gaps and plans +- [`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) - forwarding trade-offs +- [`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md) - the CUDA-side view diff --git a/docs/04-implementation/01-known-knowns-and-unknowns.md b/docs/04-implementation/01-known-knowns-and-unknowns.md index a085211..29558b0 100644 --- a/docs/04-implementation/01-known-knowns-and-unknowns.md +++ b/docs/04-implementation/01-known-knowns-and-unknowns.md @@ -22,10 +22,12 @@ listed as an unknown-unknown has a *method* by which we will discover it. ### 1.1 Device and platform * A stock 570.144 Linux guest driver probes and initializes against our - emulated TU104 device; `nvidia-smi` works and CUDA user-space starts - (`cuInit`, `cuDeviceGetCount`, `cuDeviceGet`). Context creation does not yet - complete: `cuCtxCreate` returns `rc=999` (`CUDA_ERROR_UNKNOWN`) because the - emulated engine list lacks the async copy engine (`COPY14`). + emulated TU104 device; `nvidia-smi -L` works and CUDA user-space starts + (`cuInit`, `cuDeviceGetCount`, `cuDeviceGet`). Context creation is in + progress: the client reaches `cuCtxCreate` and currently ends in a forced + `rc=999` on a completion wait; the engine inventory is complete (the async + copy engine is included) and the remaining work is in the completion and + work-execution path (see §2.1). * The same guest **boots unmodified**: no guest code, no guest module parameters are required for the initialization path. * The host is unmodified in the standalone configuration (distribution NVIDIA @@ -80,29 +82,33 @@ it**, **rough cost**. ### 2.1 Execution forwarding -**Immediate blocker (2026-09-23):** the engine topology is not yet right. -`GPU_GET_ENGINES_V2` returns 6 engines (GR + CE0..4) where a bare-metal oracle -returns 7: the async copy engine (`COPY14`; engine descriptor `0x793ceb0e`, -instance 14, client type `0x38`, class pair `0xc9b5`) is missing, so -`cuCtxCreate` returns `rc=999` (`CUDA_ERROR_UNKNOWN`). The zeroed secondary -engine record makes `libcuda` rewrite the channel alloc class to `0x41` -(`NV01_ROOT_CLIENT`) with status `0x19` and roll back. Teardown of the failed -context still oopses in the guest kernel in `uvm_pmm_devmem_page_to_va_space`. - -**Unknown:** the correct, isolated way to run a guest's engine work on the -physical GPU: which real scheduling primitive to use per engine, how to -translate guest pushes to real submissions, and how to relay completion and -fault semantics. +**Engine topology (solved):** `GPU_GET_ENGINES_V2` now reports the full engine +inventory including the async copy engine, and the copy-engine capabilities +match the physical device (two GRCEs). The earlier zeroed secondary engine +record that made `libcuda` rewrite the channel alloc class and roll back is +gone; context creation now progresses past engine selection and stalls later, +on completion of decoded work (see §1.1). + +**Unknown (resolved by decision, not yet implemented):** the correct, +isolated way to run a guest's engine work on the physical GPU. The interface +analysis in +[`../03-architecture/08-physical-gpu-integration.md`](../03-architecture/08-physical-gpu-integration.md) +shows that proxying through the distribution host driver has no callable entry +point from kernel context, so the **chosen mechanism** is for the module to own +the physical GPU and run its real GSP firmware as the resource-manager server +(the module becomes the host-side RM client). The emulation path remains the +control plane. **Why:** this is *the* blocker for CUDA kernel execution, and later for graphics. -**Plan:** evaluate the three backends (host RM/GSP proxy, direct engine -submission, hybrid) against a scorecard: correctness, isolation, latency, -maintenance cost across driver versions. Prototype the proxy first (it reuses -the unmodified host driver's knowledge), measure latency, then decide whether a -direct path is required. Deliverable: a forwarded `vectorAdd` with verified -results. +**Plan:** build the chosen mechanism in stages: device ownership + real GSP +bring-up; host-side RM client for real objects and channels; translation of +guest work and real submission; completion and fault relay. Measure each stage +against a scorecard: correctness, isolation, latency, maintenance cost across +driver versions. Deliverable: a forwarded `vectorAdd` with verified results. +Alternatives (own scheduling without GSP; vendor vGPU or SR-IOV paths) are +analyzed and rejected or deployment-gated in the integration doc. **Cost:** weeks, not days; the largest single work item in the project. diff --git a/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md b/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md index e2769f8..3265f47 100644 --- a/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md +++ b/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md @@ -57,28 +57,33 @@ without a methodology is not accepted in this project. memsets, copies and semaphore releases — is executed in software as a correctness bridge; no CUDA kernel is forwarded yet. -## 3. Forwarding: the three options and how we choose +## 3. Forwarding: the options and the chosen mechanism ``` - (A) Host RM/GSP proxy - guest push → translate → host driver's RM API → real channel → GPU - + reuses a mature driver's knowledge - − extra hop, depends on host driver interfaces + (A) Proxy through the host driver + guest push → translate → host driver's RM → real channel → GPU + Not implementable with an unmodified distribution driver: no callable + RM entry point from kernel context (see 03-architecture/08). - (B) Direct engine submission - guest push → translate → our own channel/push construction → GPU - + lowest latency, no host driver dependency - − we must own page tables, scheduling, invalidation, completion + (B) Own the GPU; own scheduling (no GSP) + guest push → translate → our own channels/runlists/page tables → GPU + Reproduces the scheduler and MMU ownership; large, capability-limiting. (C) Hybrid - management via (A), hot path via (B) + Only meaningful once (A) and (B) both exist. + + (D) Own the GPU; use its real GSP firmware as the server ← chosen + guest push → translate → real GSP-RM session → real channel → GPU + The module is the host-side resource-manager client; the firmware + schedules. ``` -**Decision procedure:** prototype (A) first to get *correct* execution, with -measurement harnesses around it. Then measure the overhead. If the overhead -violates the budgets in §2.2 for the target workloads, implement (B) for the -hot path and keep (A) for management. The architecture must not preclude -either, so the work path is abstracted behind an interface from day one. +The full analysis — which interfaces exist, why the proxy is unavailable, and +the ownership model — is in +[`../03-architecture/08-physical-gpu-integration.md`](../03-architecture/08-physical-gpu-integration.md). +The work path stays abstracted behind one interface so the backend can be +swapped. Software execution of decoded state/memory operations remains a +correctness bridge for bring-up, not the data plane. ## 4. Hard problems we must solve, and our stance on each diff --git a/docs/05-operations/01-performance-correctness-security.md b/docs/05-operations/01-performance-correctness-security.md index 583d6c8..3a2c6b4 100644 --- a/docs/05-operations/01-performance-correctness-security.md +++ b/docs/05-operations/01-performance-correctness-security.md @@ -17,11 +17,12 @@ Rung 4 SUSTAINED workloads run for hours; faults/errors recoverable Rung 5 MULTI-TENANT N VMs, isolation + fairness + no cross-talk ``` -**Current rung (2026-09-23):** Rung 1 is green; **Rung 2 is NOT green.** -`cuInit`/`cuDeviceGet` succeed, but `cuCtxCreate` returns `rc=999` -(`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks the async copy -engine (`COPY14`); `cuMemAlloc` is not reached and teardown of the failed -context still oopses in the guest kernel. Before Rung 3 is claimed, the +**Current rung (2026-09-29):** Rung 1 is green; **Rung 2 is in progress.** +`cuInit`/`cuDeviceGet` succeed, and the client reaches `cuCtxCreate`, which +currently ends in a forced `rc=999` on a completion wait (engine inventory +complete; completion translation in progress); `cuMemAlloc` is not reached yet +and failed-context teardown has oopsed in the guest kernel in earlier runs. +Before Rung 3 is claimed, the **reply-fidelity gate** applies: a control whose reply fields the driver consumes is not implemented until the reply is coherent (status-only → coherent zeroed params → coherent values for consumed fields → exact values) — diff --git a/docs/05-operations/02-debugging-and-maintenance-workflows.md b/docs/05-operations/02-debugging-and-maintenance-workflows.md index 52cee94..801cda6 100644 --- a/docs/05-operations/02-debugging-and-maintenance-workflows.md +++ b/docs/05-operations/02-debugging-and-maintenance-workflows.md @@ -136,13 +136,11 @@ userspace/cuda/tools/callwalk.py /callgraph.tsv callers 0xNNNNNN * **A/B harness**: `tools/cuda-ab/` runs one deterministic CUDA workload on bare metal and in the guest and diffs the stable result lines. -**Known current bugs (2026-09-23)**: `cuCtxCreate` returns `rc=999` -(`CUDA_ERROR_UNKNOWN`) because the emulated engine list is missing the async -copy engine `COPY14`; the guest kernel oopses in -`uvm_pmm_devmem_page_to_va_space` on teardown of the failed context; and -control `0x2080012b` (560-byte params, issued immediately before context -creation) is answered with a generic SUCCESS whose consumed fields are not yet -coherent. Start at +**Current frontier (2026-10-12)**: context creation and the full driver-API +smoke pass (`ALL OK`, 64-byte payload round-trips); a guest CUDA kernel +launch is forwarded but the engine does not execute the forwarded launch +descriptor yet, and green runs are not reproducible on every boot (tracked as +an open issue in the repository). Start at [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md), [`../../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../../userspace/cuda/docs/03-driver-and-gpu-interface.md) and diff --git a/docs/06-portability/hardware-software-matrix.md b/docs/06-portability/hardware-software-matrix.md index 49c4bbe..c839578 100644 --- a/docs/06-portability/hardware-software-matrix.md +++ b/docs/06-portability/hardware-software-matrix.md @@ -131,9 +131,9 @@ the profile files, not the profile schema. ### 6.2 Engine-list portability The engine inventory, and how it is filtered by the class list (classDB), are -per-architecture and per-driver data. Today's gap is concrete: the emulated -list lacks the async copy engine (`COPY14`), which blocks context creation -(`cuCtxCreate` returns `rc=999`). Engine inventory belongs in the hardware +per-architecture and per-driver data. An earlier gap was concrete — a missing +async copy engine (`COPY14`) blocked context creation until the inventory was +completed — and it showed why the inventory belongs in the hardware profile, validated against the host oracle per combination. For the re-analysis workflow see [`../../userspace/cuda/docs/07-version-upgrade-playbook.md`](../../userspace/cuda/docs/07-version-upgrade-playbook.md); diff --git a/docs/08-ecosystem/haaeep-and-collaboration.md b/docs/08-ecosystem/haaeep-and-collaboration.md index e559840..954438c 100644 --- a/docs/08-ecosystem/haaeep-and-collaboration.md +++ b/docs/08-ecosystem/haaeep-and-collaboration.md @@ -92,7 +92,7 @@ licensed and version-durable — so it is the component we are building. | Milestone | Depends on | |---|---| -| CUDA execution in a guest (rung 3) | Rung 2 (runtime start) being green first — it is not yet: `cuCtxCreate` returns `rc=999` because the emulated engine list lacks the async copy engine (`COPY14`) — and then Open-IOV forwarding work. The interface contract is documented under [`../../userspace/cuda/README.md`](../../userspace/cuda/README.md) (start at [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md)) | +| CUDA execution in a guest (rung 3) | Rung 2 (runtime start) being green first — context creation is in progress (`cuCtxCreate` currently ends in a forced `rc=999` on a completion wait) — and then Open-IOV forwarding work (backend chosen: a module-owned GPU with a real-GSP session). The interface contract is documented under [`../../userspace/cuda/README.md`](../../userspace/cuda/README.md) (start at [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md)) | | Two-VM coexistence (inference + agent) with isolation | Open-IOV multi-tenancy + platform MAC | | Xen/OpenXT or Qubes Air variant | Open-IOV transport backends | | Attested boot-to-inference chain | TrenchBoot integration | diff --git a/docs/09-reference/glossary-and-faq.md b/docs/09-reference/glossary-and-faq.md index 2ec4faa..e0c68f6 100644 --- a/docs/09-reference/glossary-and-faq.md +++ b/docs/09-reference/glossary-and-faq.md @@ -5,7 +5,7 @@ | Term | Meaning | |---|---| | **AMAPlib** | NVIDIA's shared "address mapping" library: swizzling, bank/channel mapping, aperture rules. Studied for page-table and memory-layout semantics. | -| **Async copy engine (COPY14)** | A copy-engine instance found in the host oracle's engine list but missing from the emulated list — engine descriptor `0x793ceb0e`, instance 14, client type `0x38`, class pair `0xc9b5`. Its absence is the current CUDA context-creation blocker. | +| **Async copy engine (COPY14)** | A copy-engine instance found in the host oracle's engine list; engine descriptor `0x793ceb0e`, instance 14, client type `0x38`, class pair `0xc9b5`. Its earlier absence from the emulated list was a context-creation blocker; the emulated inventory now includes it (two GRCEs). | | **BAR0 / BAR1 / BAR3** | PCI BAR windows of the GPU: register interface (16 MB), VRAM aperture (256 MB), instance/PRAMIN window. | | **CE** | Copy Engine: DMA engine; also used for UVM's internal work. | | **Class list (classDB)** | The set of RM classes reported as supported (`GPU_GET_CLASSLIST`); engine queries are filtered against it, so the class list, engine list and caps controls must agree. | @@ -63,14 +63,15 @@ Yes — in the standalone configuration the distribution driver owns the physica GPU and Open-IOV is an independent module. Building NVIDIA's OpenRM modules from source is only needed for the native-vGPU-management workflows. -**Why is `nvidia-smi` working but CUDA not?** -CUDA startup reaches `cuInit` / `cuDeviceGet`, but context creation is blocked: -the emulated engine list lacks the async copy engine (`COPY14`), so -`cuCtxCreate` returns `rc=999` (`CUDA_ERROR_UNKNOWN`) and memory allocation -never runs. (The earlier `libcuda` segfault on this path was fixed by echoing -the request params back in the alloc reply.) Once context creation is green, -CUDA still needs *execution*: kernels must run on real hardware. See the -known-knowns document for the exact status and plan; start at +**Why is `nvidia-smi` working but my CUDA kernel not running?** +The driver-API path works: `cuInit`, device enumeration, `cuCtxCreate` and +the full smoke test (allocation, copies, free and destroy) complete, and a +guest kernel launch is forwarded to the physical engine queue. What has not +been observed yet is *execution*: the engine does not execute the forwarded +launch descriptor and no engine-authored release appears. That is the current +frontier, not a setup problem; the chosen execution backend is documented in +[`../03-architecture/08-physical-gpu-integration.md`](../03-architecture/08-physical-gpu-integration.md). +See the known-knowns document for the exact status and plan; start at [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md). **How is this legal / license-clean?** diff --git a/docs/README.md b/docs/README.md index ed0f206..24b2a21 100644 --- a/docs/README.md +++ b/docs/README.md @@ -90,31 +90,33 @@ and [`01-orientation/02-the-ai-edge-and-ecosystem.md`](01-orientation/02-the-ai- ## 3. Where is the project today? -**Honest status (2026-09-23):** +**Honest status (2026-09-29):** | Area | State | |------|-------| | Emulated vGPU device (PCI, BARs, MSI-X, VFIO mdev) | ✅ Working | | GSP boot emulation (FWSEC/booter/RISCV handshake) | ✅ Working | | GSP-RM message queue + RPC transport | ✅ Working | -| GSP-RM controls: device/fb/bus tables, CE caps, vaspace, GMMU static info | 🟡 Mostly — values are coherent for the special-cased controls; engine inventory/class-list consistency is the current CUDA blocker (COPY14) | +| GSP-RM controls: device/fb/bus tables, CE caps, vaspace, GMMU static info | 🟡 Mostly — the engine inventory now includes the async copy engine and the CE capabilities match the device; reply fidelity for the remaining consumed fields is still being tightened | | Guest `nvidia.ko` probe, `nvidia-smi` | ✅ Working (TU104) | -| Guest CUDA driver (`libcuda`) startup | 🟡 Partially — `cuInit`/device enumeration work; `cuCtxCreate` returns `999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine list is missing the async copy engine (COPY14). See [`04-implementation/01-known-knowns-and-unknowns.md`](04-implementation/01-known-knowns-and-unknowns.md) and [`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md) | -| CUDA kernel execution | 🔴 Not yet — context creation must succeed first; then GPU work forwarding (today only decoded memsets/copies/semaphore releases execute, in the host) | +| Guest CUDA driver (`libcuda`) startup | 🟡 Partially — `cuInit` and device enumeration work; the client reaches `cuCtxCreate` and currently ends in a forced `999` on a completion wait while the completion/translation plumbing is made reliable. See [`04-implementation/01-known-knowns-and-unknowns.md`](04-implementation/01-known-knowns-and-unknowns.md) and [`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md) | +| CUDA kernel execution | 🔴 Not yet — context creation must complete first, then real work forwarding; the execution backend is chosen but not started ([`03-architecture/08-physical-gpu-integration.md`](03-architecture/08-physical-gpu-integration.md)); today only decoded memsets/copies/semaphore releases execute, in the host | | OpenGL / Vulkan | 🔴 Not yet | | Windows guest | 🔴 Not yet | | Multi-tenant isolation (VFs, mdev-per-VM) | 🟡 Designed, partially implemented | We also know exactly where the CUDA bring-up stands and what the next step is: the userspace `libcuda` segfault was fixed (an RM allocation reply now echoes -the request parameters instead of returning zeros), and the remaining blocker -is data: the engine list the emulation reports must include the async copy -engine and stay consistent with the class list, because `libcuda` filters -engines against it. The full mechanism, with offsets and the decompiled call -path, is in [`../userspace/cuda/docs/05-570.144-decompilation.md`](../userspace/cuda/docs/05-570.144-decompilation.md). +the request parameters instead of returning zeros), the engine inventory is +complete, and the client reaches context creation, which currently ends on a +completion wait while decoded work is made to execute reliably. The full +mechanism, with offsets and the decompiled call path, is in +[`../userspace/cuda/docs/05-570.144-decompilation.md`](../userspace/cuda/docs/05-570.144-decompilation.md). Beyond that, the missing piece for kernel execution is forwarding the guest's *work* (push buffers, page trees, execution) to the physical GPU under our -control; see [`04-implementation/02-practical-challenges-and-adversarial-interop.md`](04-implementation/02-practical-challenges-and-adversarial-interop.md). +control; the chosen backend — a module-owned GPU running its real GSP firmware +as the resource-manager server — is specified in +[`03-architecture/08-physical-gpu-integration.md`](03-architecture/08-physical-gpu-integration.md). ## 4. How to read these docs (progressive disclosure) @@ -123,9 +125,9 @@ Read in order if you are new. Jump directly if you know what you need. > **Debugging the CUDA bring-up?** Start with > [`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md) — > a five-minute guide to the userspace driver analysis, the corrected crash -> path, the current engine-topology blocker, and the lookup recipes for -> `libcuda+0x...` offsets. It links back into this tree for the interface and -> emulation contract. +> path, the current completion frontier and the chosen execution backend, and +> the lookup recipes for `libcuda+0x...` offsets. It links back into this tree +> for the interface and emulation contract. | # | Document | Audience | Time | |---|----------|----------|------| @@ -139,14 +141,15 @@ Read in order if you are new. Jump directly if you know what you need. | 8 | [GMMU, AMAPlib, HALs, runlist, GR](03-architecture/05-gmmu-amap-hal-gr.md) | developers | 25 min | | 9 | [Host module and hybrid model](03-architecture/06-host-module-and-hybrid-model.md) | developers | 25 min | | 10 | [Driver lifecycle field guide](03-architecture/07-driver-lifecycle-field-guide.md) | developers, debuggers | 30 min | -| 11 | [Known-knowns and known-unknowns](04-implementation/01-known-knowns-and-unknowns.md) | everyone technical | 25 min | -| 12 | [Practical challenges and adversarial interoperability](04-implementation/02-practical-challenges-and-adversarial-interop.md) | developers, stakeholders | 20 min | -| 13 | [Performance, correctness, security](05-operations/01-performance-correctness-security.md) | developers, auditors | 25 min | -| 14 | [Debugging and maintenance workflows](05-operations/02-debugging-and-maintenance-workflows.md) | developers, maintainers | 25 min | -| 15 | [Hardware and software matrix](06-portability/hardware-software-matrix.md) | developers, integrators | 20 min | -| 16 | [Developer onboarding and standards](07-developers/onboarding-and-standards.md) | contributors | 20 min | -| 17 | [HAAEEP and collaboration](08-ecosystem/haaeep-and-collaboration.md) | stakeholders, partners | 20 min | -| 18 | [Glossary and FAQ](09-reference/glossary-and-faq.md) | everyone | reference | +| 11 | [Physical GPU integration](03-architecture/08-physical-gpu-integration.md) | developers, architects | 20 min | +| 12 | [Known-knowns and known-unknowns](04-implementation/01-known-knowns-and-unknowns.md) | everyone technical | 25 min | +| 13 | [Practical challenges and adversarial interoperability](04-implementation/02-practical-challenges-and-adversarial-interop.md) | developers, stakeholders | 20 min | +| 14 | [Performance, correctness, security](05-operations/01-performance-correctness-security.md) | developers, auditors | 25 min | +| 15 | [Debugging and maintenance workflows](05-operations/02-debugging-and-maintenance-workflows.md) | developers, maintainers | 25 min | +| 16 | [Hardware and software matrix](06-portability/hardware-software-matrix.md) | developers, integrators | 20 min | +| 17 | [Developer onboarding and standards](07-developers/onboarding-and-standards.md) | contributors | 20 min | +| 18 | [HAAEEP and collaboration](08-ecosystem/haaeep-and-collaboration.md) | stakeholders, partners | 20 min | +| 19 | [Glossary and FAQ](09-reference/glossary-and-faq.md) | everyone | reference | Deep-dive material lives in [`../wiki/`](../wiki/README.md) (firmware decompilation record, address reference, historical protocol notes). Wiki -- cgit v1.2.3