From 1fdbf4ea8c17dab2b8fca6b5e58c2a6d3bc7596d Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Tue, 29 Sep 2026 15:29:00 -0400 Subject: docs: record physical-GPU integration mechanism and refresh project status - add docs/03-architecture/08-physical-gpu-integration.md: interfaces that exist, mechanism options A-E, chosen path (module-owned GPU driven through its real GSP firmware), sharing policy, decision record - update 06-host-module-and-hybrid-model, 02-practical-challenges, system overview, known-knowns, docs/README reading order and CUDA-side docs to match - correct stale engine-topology/COPY14 blocker claims across the tree - add STATE-CURRENT.md as the dated public status snapshot --- .../01-known-knowns-and-unknowns.md | 52 ++++++++++++---------- ...practical-challenges-and-adversarial-interop.md | 35 ++++++++------- 2 files changed, 49 insertions(+), 38 deletions(-) (limited to 'docs/04-implementation') diff --git a/docs/04-implementation/01-known-knowns-and-unknowns.md b/docs/04-implementation/01-known-knowns-and-unknowns.md index a085211..29558b0 100644 --- a/docs/04-implementation/01-known-knowns-and-unknowns.md +++ b/docs/04-implementation/01-known-knowns-and-unknowns.md @@ -22,10 +22,12 @@ listed as an unknown-unknown has a *method* by which we will discover it. ### 1.1 Device and platform * A stock 570.144 Linux guest driver probes and initializes against our - emulated TU104 device; `nvidia-smi` works and CUDA user-space starts - (`cuInit`, `cuDeviceGetCount`, `cuDeviceGet`). Context creation does not yet - complete: `cuCtxCreate` returns `rc=999` (`CUDA_ERROR_UNKNOWN`) because the - emulated engine list lacks the async copy engine (`COPY14`). + emulated TU104 device; `nvidia-smi -L` works and CUDA user-space starts + (`cuInit`, `cuDeviceGetCount`, `cuDeviceGet`). Context creation is in + progress: the client reaches `cuCtxCreate` and currently ends in a forced + `rc=999` on a completion wait; the engine inventory is complete (the async + copy engine is included) and the remaining work is in the completion and + work-execution path (see §2.1). * The same guest **boots unmodified**: no guest code, no guest module parameters are required for the initialization path. * The host is unmodified in the standalone configuration (distribution NVIDIA @@ -80,29 +82,33 @@ it**, **rough cost**. ### 2.1 Execution forwarding -**Immediate blocker (2026-09-23):** the engine topology is not yet right. -`GPU_GET_ENGINES_V2` returns 6 engines (GR + CE0..4) where a bare-metal oracle -returns 7: the async copy engine (`COPY14`; engine descriptor `0x793ceb0e`, -instance 14, client type `0x38`, class pair `0xc9b5`) is missing, so -`cuCtxCreate` returns `rc=999` (`CUDA_ERROR_UNKNOWN`). The zeroed secondary -engine record makes `libcuda` rewrite the channel alloc class to `0x41` -(`NV01_ROOT_CLIENT`) with status `0x19` and roll back. Teardown of the failed -context still oopses in the guest kernel in `uvm_pmm_devmem_page_to_va_space`. - -**Unknown:** the correct, isolated way to run a guest's engine work on the -physical GPU: which real scheduling primitive to use per engine, how to -translate guest pushes to real submissions, and how to relay completion and -fault semantics. +**Engine topology (solved):** `GPU_GET_ENGINES_V2` now reports the full engine +inventory including the async copy engine, and the copy-engine capabilities +match the physical device (two GRCEs). The earlier zeroed secondary engine +record that made `libcuda` rewrite the channel alloc class and roll back is +gone; context creation now progresses past engine selection and stalls later, +on completion of decoded work (see §1.1). + +**Unknown (resolved by decision, not yet implemented):** the correct, +isolated way to run a guest's engine work on the physical GPU. The interface +analysis in +[`../03-architecture/08-physical-gpu-integration.md`](../03-architecture/08-physical-gpu-integration.md) +shows that proxying through the distribution host driver has no callable entry +point from kernel context, so the **chosen mechanism** is for the module to own +the physical GPU and run its real GSP firmware as the resource-manager server +(the module becomes the host-side RM client). The emulation path remains the +control plane. **Why:** this is *the* blocker for CUDA kernel execution, and later for graphics. -**Plan:** evaluate the three backends (host RM/GSP proxy, direct engine -submission, hybrid) against a scorecard: correctness, isolation, latency, -maintenance cost across driver versions. Prototype the proxy first (it reuses -the unmodified host driver's knowledge), measure latency, then decide whether a -direct path is required. Deliverable: a forwarded `vectorAdd` with verified -results. +**Plan:** build the chosen mechanism in stages: device ownership + real GSP +bring-up; host-side RM client for real objects and channels; translation of +guest work and real submission; completion and fault relay. Measure each stage +against a scorecard: correctness, isolation, latency, maintenance cost across +driver versions. Deliverable: a forwarded `vectorAdd` with verified results. +Alternatives (own scheduling without GSP; vendor vGPU or SR-IOV paths) are +analyzed and rejected or deployment-gated in the integration doc. **Cost:** weeks, not days; the largest single work item in the project. diff --git a/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md b/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md index e2769f8..3265f47 100644 --- a/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md +++ b/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md @@ -57,28 +57,33 @@ without a methodology is not accepted in this project. memsets, copies and semaphore releases — is executed in software as a correctness bridge; no CUDA kernel is forwarded yet. -## 3. Forwarding: the three options and how we choose +## 3. Forwarding: the options and the chosen mechanism ``` - (A) Host RM/GSP proxy - guest push → translate → host driver's RM API → real channel → GPU - + reuses a mature driver's knowledge - − extra hop, depends on host driver interfaces + (A) Proxy through the host driver + guest push → translate → host driver's RM → real channel → GPU + Not implementable with an unmodified distribution driver: no callable + RM entry point from kernel context (see 03-architecture/08). - (B) Direct engine submission - guest push → translate → our own channel/push construction → GPU - + lowest latency, no host driver dependency - − we must own page tables, scheduling, invalidation, completion + (B) Own the GPU; own scheduling (no GSP) + guest push → translate → our own channels/runlists/page tables → GPU + Reproduces the scheduler and MMU ownership; large, capability-limiting. (C) Hybrid - management via (A), hot path via (B) + Only meaningful once (A) and (B) both exist. + + (D) Own the GPU; use its real GSP firmware as the server ← chosen + guest push → translate → real GSP-RM session → real channel → GPU + The module is the host-side resource-manager client; the firmware + schedules. ``` -**Decision procedure:** prototype (A) first to get *correct* execution, with -measurement harnesses around it. Then measure the overhead. If the overhead -violates the budgets in §2.2 for the target workloads, implement (B) for the -hot path and keep (A) for management. The architecture must not preclude -either, so the work path is abstracted behind an interface from day one. +The full analysis — which interfaces exist, why the proxy is unavailable, and +the ownership model — is in +[`../03-architecture/08-physical-gpu-integration.md`](../03-architecture/08-physical-gpu-integration.md). +The work path stays abstracted behind one interface so the backend can be +swapped. Software execution of decoded state/memory operations remains a +correctness bridge for bring-up, not the data plane. ## 4. Hard problems we must solve, and our stance on each -- cgit v1.2.3