diff options
| author | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-09-29 15:29:00 -0400 |
|---|---|---|
| committer | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-09-29 15:29:00 -0400 |
| commit | 1fdbf4ea8c17dab2b8fca6b5e58c2a6d3bc7596d (patch) | |
| tree | cd16ecce1bbbf8e71c17091df843441965f871c1 /docs/04-implementation | |
| parent | f4803e303e1e4bfde9f241ec3b6226399d01825d (diff) | |
| download | open-iov-1fdbf4ea8c17dab2b8fca6b5e58c2a6d3bc7596d.tar.gz open-iov-1fdbf4ea8c17dab2b8fca6b5e58c2a6d3bc7596d.zip | |
docs: record physical-GPU integration mechanism and refresh project status
- add docs/03-architecture/08-physical-gpu-integration.md: interfaces that
exist, mechanism options A-E, chosen path (module-owned GPU driven through
its real GSP firmware), sharing policy, decision record
- update 06-host-module-and-hybrid-model, 02-practical-challenges, system
overview, known-knowns, docs/README reading order and CUDA-side docs to
match
- correct stale engine-topology/COPY14 blocker claims across the tree
- add STATE-CURRENT.md as the dated public status snapshot
Diffstat (limited to 'docs/04-implementation')
| -rw-r--r-- | docs/04-implementation/01-known-knowns-and-unknowns.md | 52 | ||||
| -rw-r--r-- | docs/04-implementation/02-practical-challenges-and-adversarial-interop.md | 35 |
2 files changed, 49 insertions, 38 deletions
diff --git a/docs/04-implementation/01-known-knowns-and-unknowns.md b/docs/04-implementation/01-known-knowns-and-unknowns.md index a085211..29558b0 100644 --- a/docs/04-implementation/01-known-knowns-and-unknowns.md +++ b/docs/04-implementation/01-known-knowns-and-unknowns.md @@ -22,10 +22,12 @@ listed as an unknown-unknown has a *method* by which we will discover it. ### 1.1 Device and platform * A stock 570.144 Linux guest driver probes and initializes against our - emulated TU104 device; `nvidia-smi` works and CUDA user-space starts - (`cuInit`, `cuDeviceGetCount`, `cuDeviceGet`). Context creation does not yet - complete: `cuCtxCreate` returns `rc=999` (`CUDA_ERROR_UNKNOWN`) because the - emulated engine list lacks the async copy engine (`COPY14`). + emulated TU104 device; `nvidia-smi -L` works and CUDA user-space starts + (`cuInit`, `cuDeviceGetCount`, `cuDeviceGet`). Context creation is in + progress: the client reaches `cuCtxCreate` and currently ends in a forced + `rc=999` on a completion wait; the engine inventory is complete (the async + copy engine is included) and the remaining work is in the completion and + work-execution path (see §2.1). * The same guest **boots unmodified**: no guest code, no guest module parameters are required for the initialization path. * The host is unmodified in the standalone configuration (distribution NVIDIA @@ -80,29 +82,33 @@ it**, **rough cost**. ### 2.1 Execution forwarding -**Immediate blocker (2026-09-23):** the engine topology is not yet right. -`GPU_GET_ENGINES_V2` returns 6 engines (GR + CE0..4) where a bare-metal oracle -returns 7: the async copy engine (`COPY14`; engine descriptor `0x793ceb0e`, -instance 14, client type `0x38`, class pair `0xc9b5`) is missing, so -`cuCtxCreate` returns `rc=999` (`CUDA_ERROR_UNKNOWN`). The zeroed secondary -engine record makes `libcuda` rewrite the channel alloc class to `0x41` -(`NV01_ROOT_CLIENT`) with status `0x19` and roll back. Teardown of the failed -context still oopses in the guest kernel in `uvm_pmm_devmem_page_to_va_space`. - -**Unknown:** the correct, isolated way to run a guest's engine work on the -physical GPU: which real scheduling primitive to use per engine, how to -translate guest pushes to real submissions, and how to relay completion and -fault semantics. +**Engine topology (solved):** `GPU_GET_ENGINES_V2` now reports the full engine +inventory including the async copy engine, and the copy-engine capabilities +match the physical device (two GRCEs). The earlier zeroed secondary engine +record that made `libcuda` rewrite the channel alloc class and roll back is +gone; context creation now progresses past engine selection and stalls later, +on completion of decoded work (see §1.1). + +**Unknown (resolved by decision, not yet implemented):** the correct, +isolated way to run a guest's engine work on the physical GPU. The interface +analysis in +[`../03-architecture/08-physical-gpu-integration.md`](../03-architecture/08-physical-gpu-integration.md) +shows that proxying through the distribution host driver has no callable entry +point from kernel context, so the **chosen mechanism** is for the module to own +the physical GPU and run its real GSP firmware as the resource-manager server +(the module becomes the host-side RM client). The emulation path remains the +control plane. **Why:** this is *the* blocker for CUDA kernel execution, and later for graphics. -**Plan:** evaluate the three backends (host RM/GSP proxy, direct engine -submission, hybrid) against a scorecard: correctness, isolation, latency, -maintenance cost across driver versions. Prototype the proxy first (it reuses -the unmodified host driver's knowledge), measure latency, then decide whether a -direct path is required. Deliverable: a forwarded `vectorAdd` with verified -results. +**Plan:** build the chosen mechanism in stages: device ownership + real GSP +bring-up; host-side RM client for real objects and channels; translation of +guest work and real submission; completion and fault relay. Measure each stage +against a scorecard: correctness, isolation, latency, maintenance cost across +driver versions. Deliverable: a forwarded `vectorAdd` with verified results. +Alternatives (own scheduling without GSP; vendor vGPU or SR-IOV paths) are +analyzed and rejected or deployment-gated in the integration doc. **Cost:** weeks, not days; the largest single work item in the project. diff --git a/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md b/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md index e2769f8..3265f47 100644 --- a/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md +++ b/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md @@ -57,28 +57,33 @@ without a methodology is not accepted in this project. memsets, copies and semaphore releases — is executed in software as a correctness bridge; no CUDA kernel is forwarded yet. -## 3. Forwarding: the three options and how we choose +## 3. Forwarding: the options and the chosen mechanism ``` - (A) Host RM/GSP proxy - guest push → translate → host driver's RM API → real channel → GPU - + reuses a mature driver's knowledge - − extra hop, depends on host driver interfaces + (A) Proxy through the host driver + guest push → translate → host driver's RM → real channel → GPU + Not implementable with an unmodified distribution driver: no callable + RM entry point from kernel context (see 03-architecture/08). - (B) Direct engine submission - guest push → translate → our own channel/push construction → GPU - + lowest latency, no host driver dependency - − we must own page tables, scheduling, invalidation, completion + (B) Own the GPU; own scheduling (no GSP) + guest push → translate → our own channels/runlists/page tables → GPU + Reproduces the scheduler and MMU ownership; large, capability-limiting. (C) Hybrid - management via (A), hot path via (B) + Only meaningful once (A) and (B) both exist. + + (D) Own the GPU; use its real GSP firmware as the server ← chosen + guest push → translate → real GSP-RM session → real channel → GPU + The module is the host-side resource-manager client; the firmware + schedules. ``` -**Decision procedure:** prototype (A) first to get *correct* execution, with -measurement harnesses around it. Then measure the overhead. If the overhead -violates the budgets in §2.2 for the target workloads, implement (B) for the -hot path and keep (A) for management. The architecture must not preclude -either, so the work path is abstracted behind an interface from day one. +The full analysis — which interfaces exist, why the proxy is unavailable, and +the ownership model — is in +[`../03-architecture/08-physical-gpu-integration.md`](../03-architecture/08-physical-gpu-integration.md). +The work path stays abstracted behind one interface so the backend can be +swapped. Software execution of decoded state/memory operations remains a +correctness bridge for bring-up, not the data plane. ## 4. Hard problems we must solve, and our stance on each |
