diff options
| author | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-09-29 15:29:00 -0400 |
|---|---|---|
| committer | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-09-29 15:29:00 -0400 |
| commit | 4738b619c45abc4cf98a549c3118839e48bb7244 (patch) | |
| tree | cb6004f1a6351edbc482031189d359fd2882dab5 /userspace/cuda/docs | |
| parent | 8d45a99fe54cb2b19b3024fd975ab11574611c3e (diff) | |
| download | open-iov-4738b619c45abc4cf98a549c3118839e48bb7244.tar.gz open-iov-4738b619c45abc4cf98a549c3118839e48bb7244.zip | |
docs: record physical-GPU integration mechanism and refresh project status
- add docs/03-architecture/08-physical-gpu-integration.md: interfaces that
exist, mechanism options A-E, chosen path (module-owned GPU driven through
its real GSP firmware), sharing policy, decision record
- update 06-host-module-and-hybrid-model, 02-practical-challenges, system
overview, known-knowns, docs/README reading order and CUDA-side docs to
match
- correct stale engine-topology/COPY14 blocker claims across the tree
- add STATE-CURRENT.md as the dated public status snapshot
Diffstat (limited to 'userspace/cuda/docs')
| -rw-r--r-- | userspace/cuda/docs/05-570.144-decompilation.md | 15 | ||||
| -rw-r--r-- | userspace/cuda/docs/08-virtualization-approaches.md | 41 |
2 files changed, 38 insertions, 18 deletions
diff --git a/userspace/cuda/docs/05-570.144-decompilation.md b/userspace/cuda/docs/05-570.144-decompilation.md index 4e23866..deec463 100644 --- a/userspace/cuda/docs/05-570.144-decompilation.md +++ b/userspace/cuda/docs/05-570.144-decompilation.md @@ -89,12 +89,13 @@ This is the mechanism described in `docs/01` §2.2/§2.3: the wrapper does the tracing/tool callbacks and then calls the hand-written implementation. When chasing a call, follow the slot; when a tool interposes, the slot changes. -## 3. The `cuCtxCreate` path: crash solved, engine topology blocking +## 3. The `cuCtxCreate` path: crash and engine topology solved, completion is the frontier The 2026-09 crash was deterministic: on a fresh guest `cuCtxCreate` segfaulted -at `libcuda+0x453f90`. That is **solved** (§3.5). The call now returns -`rc=999` (`CUDA_ERROR_UNKNOWN`), and the active blocker is engine topology — -the emulated engine list is missing the async copy engine (§3.6). +at `libcuda+0x453f90`. That is **solved** (§3.5). The engine inventory is now +complete (the async copy engine is included); the call proceeds into context +creation and currently ends in a forced `rc=999` on a completion wait. The +sections below are the historical analysis of that call path. Fault facts from the crash period (live traces, corrected decompilation): @@ -233,6 +234,12 @@ the alloc reply's class-size copy-back described above. ### 3.6 Current blocker (2026-09-23): the missing async copy engine +> **Status update (2026-09-29): resolved.** The emulated inventory now includes +> the async copy engine (two GRCEs), and the class list, engine list and +> capabilities agree; the `rc=999` seen today comes from a completion wait in +> the work/translation path, not from engine topology. The analysis below is +> kept as the record of how the topology blocker was diagnosed and fixed. + With the copy-back fixed, the walker completes but `cuCtxCreate` returns `rc=999` (`CUDA_ERROR_UNKNOWN`); `cuMemAlloc` and later calls are not reached. Measured with the guest ioctl trace plus a host A/B oracle (the same libcuda diff --git a/userspace/cuda/docs/08-virtualization-approaches.md b/userspace/cuda/docs/08-virtualization-approaches.md index 00ca13f..e67f648 100644 --- a/userspace/cuda/docs/08-virtualization-approaches.md +++ b/userspace/cuda/docs/08-virtualization-approaches.md @@ -20,6 +20,13 @@ principles) and to the project implementation docs (`docs/03-architecture/06-hos fences, some memops) should be satisfied directly in the emulation regardless of A/B/C — they are microseconds in hardware and would lose by round-tripping. +**Availability with an unmodified distribution host driver:** approach A has no +callable entry point from kernel context (see +[`../../../docs/03-architecture/08-physical-gpu-integration.md`](../../../docs/03-architecture/08-physical-gpu-integration.md)); +approach B requires owning scheduling; the project's chosen mechanism is a +module-owned device that keeps the GPU's real GSP firmware as the scheduling +server. Approach E remains deployment-gated. + ## 2. Design implications * Where the boundary sits decides what we must own: A keeps ownership in the @@ -40,25 +47,31 @@ of A/B/C — they are microseconds in hardware and would lose by round-tripping. patch anyone* — a mediated-passthrough vGPU where PCI config/BARs, the GSP boot handshake, the GSP-RM protocol and interrupt routing are emulated in `open-iov-vfio.ko`, while real work is meant to execute on the physical GPU. -The project's own decision procedure is explicit: **prototype A first for -correctness, measure, then move the hot path to B if the budgets demand it** -(option C in practice), with the work path abstracted from day one. +The original decision procedure was to prototype A first for correctness; the +interface analysis +([`../../../docs/03-architecture/08-physical-gpu-integration.md`](../../../docs/03-architecture/08-physical-gpu-integration.md)) +showed that proxying through an unmodified distribution host driver has no +callable entry point from kernel context, so the chosen mechanism is a +**module-owned GPU running its real GSP firmware as the resource-manager +server** (the module becomes the host-side RM client). The work path stays +abstracted from day one so the backend can be swapped. -**Current reality (2026-09, honest):** the *control plane* follows the +**Current reality (2026-09-29, honest):** the *control plane* follows the architecture — GSP-RM emulation, RPC handling, BAR/PCI/MSI-X emulation and object/control replies are implemented, and the guest's stock driver + `libcuda` now get through `cuInit`, `cuDeviceGetCount`/`cuDeviceGet` and into `cuCtxCreate`. The old `cuCtxCreate` segfault is solved (the alloc reply must -echo the request's params — `docs/05` §3.5); context creation is blocked one -step later by engine topology: the class-filtered engine list is missing the -async copy engine, so the walker reads a zeroed secondary record and the call -returns `rc=999` (`docs/05` §3.6). The *data plane* is at the bridge stage: the -host module decodes the guest's push buffers as method streams and **executes -the small work it understands itself** (memsets, copies, semaphore releases) -instead of forwarding it to the physical GPU. No CUDA kernel has been -forwarded yet. So: yes, we are pursuing the intended architecture, but we are -in the "emulate + software-execute state/memory ops" phase of it, with the -A→B/C forwarding decision still ahead. +echo the request's params — `docs/05` §3.5), and the engine inventory is now +complete (the async copy engine is included; the copy-engine capabilities +match the device). Context creation currently ends in a forced `rc=999` on a +completion wait while the completion/translation plumbing is being made +reliable. The *data plane* is at the bridge stage: the host module decodes the +guest's push buffers as method streams and executes the small work it +understands itself (memsets, copies, semaphore releases) instead of forwarding +it to the physical GPU. No CUDA kernel has been forwarded yet; the execution +backend is chosen but not started. So: we are still in the +"emulate + software-execute state/memory ops" phase, with the backend decision +now made (option D in the integration document). **Risks to guard now:** (1) software execution silently becoming the data plane instead of a bridge; (2) the work-path interface being skipped because |
