summaryrefslogtreecommitdiffstats
path: root/docs/04-implementation
diff options
context:
space:
mode:
authorArthur Hanson Rasmusson <arthur@vgpu.io>2026-09-29 15:29:00 -0400
committerArthur Hanson Rasmusson <arthur@vgpu.io>2026-09-29 15:29:00 -0400
commit4738b619c45abc4cf98a549c3118839e48bb7244 (patch)
treecb6004f1a6351edbc482031189d359fd2882dab5 /docs/04-implementation
parent8d45a99fe54cb2b19b3024fd975ab11574611c3e (diff)
downloadopen-iov-4738b619c45abc4cf98a549c3118839e48bb7244.tar.gz
open-iov-4738b619c45abc4cf98a549c3118839e48bb7244.zip
docs: record physical-GPU integration mechanism and refresh project status
- add docs/03-architecture/08-physical-gpu-integration.md: interfaces that exist, mechanism options A-E, chosen path (module-owned GPU driven through its real GSP firmware), sharing policy, decision record - update 06-host-module-and-hybrid-model, 02-practical-challenges, system overview, known-knowns, docs/README reading order and CUDA-side docs to match - correct stale engine-topology/COPY14 blocker claims across the tree - add STATE-CURRENT.md as the dated public status snapshot
Diffstat (limited to 'docs/04-implementation')
-rw-r--r--docs/04-implementation/01-known-knowns-and-unknowns.md52
-rw-r--r--docs/04-implementation/02-practical-challenges-and-adversarial-interop.md35
2 files changed, 49 insertions, 38 deletions
diff --git a/docs/04-implementation/01-known-knowns-and-unknowns.md b/docs/04-implementation/01-known-knowns-and-unknowns.md
index a085211..29558b0 100644
--- a/docs/04-implementation/01-known-knowns-and-unknowns.md
+++ b/docs/04-implementation/01-known-knowns-and-unknowns.md
@@ -22,10 +22,12 @@ listed as an unknown-unknown has a *method* by which we will discover it.
### 1.1 Device and platform
* A stock 570.144 Linux guest driver probes and initializes against our
- emulated TU104 device; `nvidia-smi` works and CUDA user-space starts
- (`cuInit`, `cuDeviceGetCount`, `cuDeviceGet`). Context creation does not yet
- complete: `cuCtxCreate` returns `rc=999` (`CUDA_ERROR_UNKNOWN`) because the
- emulated engine list lacks the async copy engine (`COPY14`).
+ emulated TU104 device; `nvidia-smi -L` works and CUDA user-space starts
+ (`cuInit`, `cuDeviceGetCount`, `cuDeviceGet`). Context creation is in
+ progress: the client reaches `cuCtxCreate` and currently ends in a forced
+ `rc=999` on a completion wait; the engine inventory is complete (the async
+ copy engine is included) and the remaining work is in the completion and
+ work-execution path (see §2.1).
* The same guest **boots unmodified**: no guest code, no guest module
parameters are required for the initialization path.
* The host is unmodified in the standalone configuration (distribution NVIDIA
@@ -80,29 +82,33 @@ it**, **rough cost**.
### 2.1 Execution forwarding
-**Immediate blocker (2026-09-23):** the engine topology is not yet right.
-`GPU_GET_ENGINES_V2` returns 6 engines (GR + CE0..4) where a bare-metal oracle
-returns 7: the async copy engine (`COPY14`; engine descriptor `0x793ceb0e`,
-instance 14, client type `0x38`, class pair `0xc9b5`) is missing, so
-`cuCtxCreate` returns `rc=999` (`CUDA_ERROR_UNKNOWN`). The zeroed secondary
-engine record makes `libcuda` rewrite the channel alloc class to `0x41`
-(`NV01_ROOT_CLIENT`) with status `0x19` and roll back. Teardown of the failed
-context still oopses in the guest kernel in `uvm_pmm_devmem_page_to_va_space`.
-
-**Unknown:** the correct, isolated way to run a guest's engine work on the
-physical GPU: which real scheduling primitive to use per engine, how to
-translate guest pushes to real submissions, and how to relay completion and
-fault semantics.
+**Engine topology (solved):** `GPU_GET_ENGINES_V2` now reports the full engine
+inventory including the async copy engine, and the copy-engine capabilities
+match the physical device (two GRCEs). The earlier zeroed secondary engine
+record that made `libcuda` rewrite the channel alloc class and roll back is
+gone; context creation now progresses past engine selection and stalls later,
+on completion of decoded work (see §1.1).
+
+**Unknown (resolved by decision, not yet implemented):** the correct,
+isolated way to run a guest's engine work on the physical GPU. The interface
+analysis in
+[`../03-architecture/08-physical-gpu-integration.md`](../03-architecture/08-physical-gpu-integration.md)
+shows that proxying through the distribution host driver has no callable entry
+point from kernel context, so the **chosen mechanism** is for the module to own
+the physical GPU and run its real GSP firmware as the resource-manager server
+(the module becomes the host-side RM client). The emulation path remains the
+control plane.
**Why:** this is *the* blocker for CUDA kernel execution, and later for
graphics.
-**Plan:** evaluate the three backends (host RM/GSP proxy, direct engine
-submission, hybrid) against a scorecard: correctness, isolation, latency,
-maintenance cost across driver versions. Prototype the proxy first (it reuses
-the unmodified host driver's knowledge), measure latency, then decide whether a
-direct path is required. Deliverable: a forwarded `vectorAdd` with verified
-results.
+**Plan:** build the chosen mechanism in stages: device ownership + real GSP
+bring-up; host-side RM client for real objects and channels; translation of
+guest work and real submission; completion and fault relay. Measure each stage
+against a scorecard: correctness, isolation, latency, maintenance cost across
+driver versions. Deliverable: a forwarded `vectorAdd` with verified results.
+Alternatives (own scheduling without GSP; vendor vGPU or SR-IOV paths) are
+analyzed and rejected or deployment-gated in the integration doc.
**Cost:** weeks, not days; the largest single work item in the project.
diff --git a/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md b/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md
index e2769f8..3265f47 100644
--- a/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md
+++ b/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md
@@ -57,28 +57,33 @@ without a methodology is not accepted in this project.
memsets, copies and semaphore releases — is executed in software as a
correctness bridge; no CUDA kernel is forwarded yet.
-## 3. Forwarding: the three options and how we choose
+## 3. Forwarding: the options and the chosen mechanism
```
- (A) Host RM/GSP proxy
- guest push → translate → host driver's RM API → real channel → GPU
- + reuses a mature driver's knowledge
- − extra hop, depends on host driver interfaces
+ (A) Proxy through the host driver
+ guest push → translate → host driver's RM → real channel → GPU
+ Not implementable with an unmodified distribution driver: no callable
+ RM entry point from kernel context (see 03-architecture/08).
- (B) Direct engine submission
- guest push → translate → our own channel/push construction → GPU
- + lowest latency, no host driver dependency
- − we must own page tables, scheduling, invalidation, completion
+ (B) Own the GPU; own scheduling (no GSP)
+ guest push → translate → our own channels/runlists/page tables → GPU
+ Reproduces the scheduler and MMU ownership; large, capability-limiting.
(C) Hybrid
- management via (A), hot path via (B)
+ Only meaningful once (A) and (B) both exist.
+
+ (D) Own the GPU; use its real GSP firmware as the server ← chosen
+ guest push → translate → real GSP-RM session → real channel → GPU
+ The module is the host-side resource-manager client; the firmware
+ schedules.
```
-**Decision procedure:** prototype (A) first to get *correct* execution, with
-measurement harnesses around it. Then measure the overhead. If the overhead
-violates the budgets in §2.2 for the target workloads, implement (B) for the
-hot path and keep (A) for management. The architecture must not preclude
-either, so the work path is abstracted behind an interface from day one.
+The full analysis — which interfaces exist, why the proxy is unavailable, and
+the ownership model — is in
+[`../03-architecture/08-physical-gpu-integration.md`](../03-architecture/08-physical-gpu-integration.md).
+The work path stays abstracted behind one interface so the backend can be
+swapped. Software execution of decoded state/memory operations remains a
+correctness bridge for bring-up, not the data plane.
## 4. Hard problems we must solve, and our stance on each