summaryrefslogtreecommitdiffstats
path: root/userspace/cuda
diff options
context:
space:
mode:
authorArthur Hanson Rasmusson <arthur@vgpu.io>2026-09-29 15:29:00 -0400
committerArthur Hanson Rasmusson <arthur@vgpu.io>2026-09-29 15:29:00 -0400
commit4738b619c45abc4cf98a549c3118839e48bb7244 (patch)
treecb6004f1a6351edbc482031189d359fd2882dab5 /userspace/cuda
parent8d45a99fe54cb2b19b3024fd975ab11574611c3e (diff)
downloadopen-iov-4738b619c45abc4cf98a549c3118839e48bb7244.tar.gz
open-iov-4738b619c45abc4cf98a549c3118839e48bb7244.zip
docs: record physical-GPU integration mechanism and refresh project status
- add docs/03-architecture/08-physical-gpu-integration.md: interfaces that exist, mechanism options A-E, chosen path (module-owned GPU driven through its real GSP firmware), sharing policy, decision record - update 06-host-module-and-hybrid-model, 02-practical-challenges, system overview, known-knowns, docs/README reading order and CUDA-side docs to match - correct stale engine-topology/COPY14 blocker claims across the tree - add STATE-CURRENT.md as the dated public status snapshot
Diffstat (limited to 'userspace/cuda')
-rw-r--r--userspace/cuda/570.144/docs/ANALYSIS-cuctxcreate-segv.md7
-rw-r--r--userspace/cuda/READ-ME-FIRST.md24
-rw-r--r--userspace/cuda/README.md2
-rw-r--r--userspace/cuda/docs/05-570.144-decompilation.md15
-rw-r--r--userspace/cuda/docs/08-virtualization-approaches.md41
5 files changed, 56 insertions, 33 deletions
diff --git a/userspace/cuda/570.144/docs/ANALYSIS-cuctxcreate-segv.md b/userspace/cuda/570.144/docs/ANALYSIS-cuctxcreate-segv.md
index db38802..666a4e2 100644
--- a/userspace/cuda/570.144/docs/ANALYSIS-cuctxcreate-segv.md
+++ b/userspace/cuda/570.144/docs/ANALYSIS-cuctxcreate-segv.md
@@ -3,8 +3,9 @@
> **Status: historical (2026-09-23 PART 9).** The segfault described here was
> fixed in PART 10 (the RM allocation reply now echoes the request parameters,
> which prevents a 16-byte class-params copy-back from zeroing the caller's
-> saved frame pointer). The current blocker is the missing async copy engine in
-> the emulated engine list (`cuCtxCreate` returns `999`). For the corrected,
+> saved frame pointer). The engine-topology blocker that followed was also
+> since resolved (the async copy engine is included); the current frontier is
+> the completion/execution path. For the corrected,
> current analysis see
> [`../../docs/05-570.144-decompilation.md`](../../docs/05-570.144-decompilation.md)
> and [`../../READ-ME-FIRST.md`](../../READ-ME-FIRST.md). The runtime
@@ -22,7 +23,7 @@ First analysis note for the CUDA userspace driver under Open-IOV. Companion to
|---|---|
| `cuInit(0)` | **OK** (previous sessions: failed) |
| `cuDeviceGetCount` / `cuDeviceGet` | **OK** (device 0) |
-| `cuCtxCreate` | **SIGSEGV inside libcuda** - the current blocker |
+| `cuCtxCreate` | **SIGSEGV inside libcuda** - the blocker at this snapshot (since fixed) |
| `cuMemAlloc` / copies / `ALL OK` | not reached yet |
Guest: Ubuntu 24.04, driver **570.144** (open kernel modules + libcuda
diff --git a/userspace/cuda/READ-ME-FIRST.md b/userspace/cuda/READ-ME-FIRST.md
index 2abb797..9adcf75 100644
--- a/userspace/cuda/READ-ME-FIRST.md
+++ b/userspace/cuda/READ-ME-FIRST.md
@@ -24,14 +24,16 @@ tooling.
ioctl traces) are still valid; any conclusion drawn from its decompiled
code must be re-derived. The new tooling translates addresses for you:
`tools/ghidra_decompile_at.sh` and the annotated corpus.
-2. **`docs/05` §3 — the `cuCtxCreate` path: crash solved, engine topology
- blocking.** The old fault at `libcuda+0x453f90` (helper at `+0x453ef0`) is
- solved: the alloc reply must **echo the request's params**, because the
- driver copies 16 bytes of class params back over the caller's 8-byte local
- (the hook's saved `rbp`) — a zeroed reply is what made `rbp=0` (§3.5). The
- current blocker is one step later: the class-filtered engine list is
- missing the async copy engine `COPY14`, so `cuCtxCreate` returns `rc=999`
- (§3.6). §3.3 gives the engine-record layout (`base+0x2fec`,
+2. **`docs/05` §3 — the `cuCtxCreate` path: crash and engine topology
+ solved, completion is the frontier.** The old fault at
+ `libcuda+0x453f90` (helper at `+0x453ef0`) is solved: the alloc reply must
+ **echo the request's params**, because the driver copies 16 bytes of class
+ params back over the caller's 8-byte local (the hook's saved `rbp`) — a
+ zeroed reply is what made `rbp=0` (§3.5). The engine inventory is now
+ complete (the async copy engine is included); context creation reaches a
+ completion wait that currently ends in `rc=999` (§3.6 carries the
+ historical analysis and a status note). §3.3 gives the engine-record layout
+ (`base+0x2fec`,
`+0x300c+(idx-1)*0x14`, `+0x350c`, `+0x351c+(idx-0x16)*0x14`; fields
+0/+8/+0xc) and §3.7 lists exactly what the emulation should check first.
@@ -45,7 +47,7 @@ Read in this order; each builds on the previous one.
|---|---|---|---|
| 1 | `docs/01-architecture.md` | 15 min | The mental model: layers (API → engine → driver-model → hardware), context/device/stream/channel/memory objects, the marker + tracking-semaphore sync model, launch/QMD flow, UVM integration, threading. This is the vocabulary the rest of the docs use. |
| 2 | `docs/03-driver-and-gpu-interface.md` | 15 min | The emulation contract: device nodes and every ioctl CUDA issues, RM object classes, control families, memory/VA/mapping, channels/GPFIFO/USERD/doorbells, semaphores/notifiers/interrupts, UVM, GSP-RM. Ends with an 8-point emulation checklist. |
-| 3 | `docs/05-570.144-decompilation.md` | 10 min | The actual binary: function counts, address conventions, API stub → wrapper → implementation mechanism, the `cuCtxCreate` crash-path resolution and current engine-topology blocker, and how to look anything up in the corpus. |
+| 3 | `docs/05-570.144-decompilation.md` | 10 min | The actual binary: function counts, address conventions, API stub → wrapper → implementation mechanism, the `cuCtxCreate` crash-path resolution and engine-topology fix, the completion/execution frontier, and how to look anything up in the corpus. |
| 4 | `docs/04-virtualization-and-emulation.md` | 15 min | Design doc: how NVIDIA's vGPU split works, how Open-IOV differs (emulate the interface, execute the work), and the anti-fragility/performance/portability/security principles that constrain any fix you make. |
| 5 | `docs/02-build-and-execution.md` | 10 min | How the library is built (generated API wrappers, entry tables, embedded device code) and how it executes in a process (init sequence, threads, interposition, `/dev` access, why LD_PRELOAD can't see its syscalls). |
@@ -148,8 +150,8 @@ userspace/cuda/
## 6. The shortest possible version
-1. `docs/05` §1.1 and §3 — address fix + the `cuCtxCreate` path (crash
- solved; engine-topology blocker) (read now).
+1. `docs/05` §1.1 and §3 — address fix + the `cuCtxCreate` path (crash and
+ engine topology solved; completion/execution frontier) (read now).
2. `docs/01` and `docs/03` — the model and the contract.
3. `docs/04` — the design principles your fix must respect.
4. `docs/06` — the recipes; then `tools/`.
diff --git a/userspace/cuda/README.md b/userspace/cuda/README.md
index 9c3163d..aaeedff 100644
--- a/userspace/cuda/README.md
+++ b/userspace/cuda/README.md
@@ -27,7 +27,7 @@ Decompiled output is never committed.
| [`docs/02-build-and-execution.md`](docs/02-build-and-execution.md) | how the driver is compiled and how it executes in a process (exports, generated wrappers, entry tables, embedded device code) |
| [`docs/03-driver-and-gpu-interface.md`](docs/03-driver-and-gpu-interface.md) | everything the driver touches below itself: device nodes, RM controls, memory, channels, doorbells, UVM, interrupts |
| [`docs/04-virtualization-and-emulation.md`](docs/04-virtualization-and-emulation.md) | NVIDIA's vGPU split, how Open-IOV differs, anti-fragility, performance, portability, security |
-| [`docs/05-570.144-decompilation.md`](docs/05-570.144-decompilation.md) | the 570.144 binary map, key functions, call chains, the solved `cuCtxCreate` crash, and the current engine-topology blocker |
+| [`docs/05-570.144-decompilation.md`](docs/05-570.144-decompilation.md) | the 570.144 binary map, key functions, call chains, the solved `cuCtxCreate` crash and engine-topology blocker, and the completion/execution frontier |
| [`docs/06-tooling.md`](docs/06-tooling.md) | the reusable toolchain: ELF toolkit, Ghidra analysis/decompilation, name transfer, version deltas |
| [`docs/07-version-upgrade-playbook.md`](docs/07-version-upgrade-playbook.md) | step-by-step method to re-run the analysis on a new driver version |
| [`docs/08-virtualization-approaches.md`](docs/08-virtualization-approaches.md) | one-page comparison of execution-forwarding options (software emulation, host-RM proxy, direct submission, hybrid) and where the project stands |
diff --git a/userspace/cuda/docs/05-570.144-decompilation.md b/userspace/cuda/docs/05-570.144-decompilation.md
index 4e23866..deec463 100644
--- a/userspace/cuda/docs/05-570.144-decompilation.md
+++ b/userspace/cuda/docs/05-570.144-decompilation.md
@@ -89,12 +89,13 @@ This is the mechanism described in `docs/01` §2.2/§2.3: the wrapper does the
tracing/tool callbacks and then calls the hand-written implementation. When
chasing a call, follow the slot; when a tool interposes, the slot changes.
-## 3. The `cuCtxCreate` path: crash solved, engine topology blocking
+## 3. The `cuCtxCreate` path: crash and engine topology solved, completion is the frontier
The 2026-09 crash was deterministic: on a fresh guest `cuCtxCreate` segfaulted
-at `libcuda+0x453f90`. That is **solved** (§3.5). The call now returns
-`rc=999` (`CUDA_ERROR_UNKNOWN`), and the active blocker is engine topology —
-the emulated engine list is missing the async copy engine (§3.6).
+at `libcuda+0x453f90`. That is **solved** (§3.5). The engine inventory is now
+complete (the async copy engine is included); the call proceeds into context
+creation and currently ends in a forced `rc=999` on a completion wait. The
+sections below are the historical analysis of that call path.
Fault facts from the crash period (live traces, corrected decompilation):
@@ -233,6 +234,12 @@ the alloc reply's class-size copy-back described above.
### 3.6 Current blocker (2026-09-23): the missing async copy engine
+> **Status update (2026-09-29): resolved.** The emulated inventory now includes
+> the async copy engine (two GRCEs), and the class list, engine list and
+> capabilities agree; the `rc=999` seen today comes from a completion wait in
+> the work/translation path, not from engine topology. The analysis below is
+> kept as the record of how the topology blocker was diagnosed and fixed.
+
With the copy-back fixed, the walker completes but `cuCtxCreate` returns
`rc=999` (`CUDA_ERROR_UNKNOWN`); `cuMemAlloc` and later calls are not reached.
Measured with the guest ioctl trace plus a host A/B oracle (the same libcuda
diff --git a/userspace/cuda/docs/08-virtualization-approaches.md b/userspace/cuda/docs/08-virtualization-approaches.md
index 00ca13f..e67f648 100644
--- a/userspace/cuda/docs/08-virtualization-approaches.md
+++ b/userspace/cuda/docs/08-virtualization-approaches.md
@@ -20,6 +20,13 @@ principles) and to the project implementation docs (`docs/03-architecture/06-hos
fences, some memops) should be satisfied directly in the emulation regardless
of A/B/C — they are microseconds in hardware and would lose by round-tripping.
+**Availability with an unmodified distribution host driver:** approach A has no
+callable entry point from kernel context (see
+[`../../../docs/03-architecture/08-physical-gpu-integration.md`](../../../docs/03-architecture/08-physical-gpu-integration.md));
+approach B requires owning scheduling; the project's chosen mechanism is a
+module-owned device that keeps the GPU's real GSP firmware as the scheduling
+server. Approach E remains deployment-gated.
+
## 2. Design implications
* Where the boundary sits decides what we must own: A keeps ownership in the
@@ -40,25 +47,31 @@ of A/B/C — they are microseconds in hardware and would lose by round-tripping.
patch anyone* — a mediated-passthrough vGPU where PCI config/BARs, the GSP boot
handshake, the GSP-RM protocol and interrupt routing are emulated in
`open-iov-vfio.ko`, while real work is meant to execute on the physical GPU.
-The project's own decision procedure is explicit: **prototype A first for
-correctness, measure, then move the hot path to B if the budgets demand it**
-(option C in practice), with the work path abstracted from day one.
+The original decision procedure was to prototype A first for correctness; the
+interface analysis
+([`../../../docs/03-architecture/08-physical-gpu-integration.md`](../../../docs/03-architecture/08-physical-gpu-integration.md))
+showed that proxying through an unmodified distribution host driver has no
+callable entry point from kernel context, so the chosen mechanism is a
+**module-owned GPU running its real GSP firmware as the resource-manager
+server** (the module becomes the host-side RM client). The work path stays
+abstracted from day one so the backend can be swapped.
-**Current reality (2026-09, honest):** the *control plane* follows the
+**Current reality (2026-09-29, honest):** the *control plane* follows the
architecture — GSP-RM emulation, RPC handling, BAR/PCI/MSI-X emulation and
object/control replies are implemented, and the guest's stock driver +
`libcuda` now get through `cuInit`, `cuDeviceGetCount`/`cuDeviceGet` and into
`cuCtxCreate`. The old `cuCtxCreate` segfault is solved (the alloc reply must
-echo the request's params — `docs/05` §3.5); context creation is blocked one
-step later by engine topology: the class-filtered engine list is missing the
-async copy engine, so the walker reads a zeroed secondary record and the call
-returns `rc=999` (`docs/05` §3.6). The *data plane* is at the bridge stage: the
-host module decodes the guest's push buffers as method streams and **executes
-the small work it understands itself** (memsets, copies, semaphore releases)
-instead of forwarding it to the physical GPU. No CUDA kernel has been
-forwarded yet. So: yes, we are pursuing the intended architecture, but we are
-in the "emulate + software-execute state/memory ops" phase of it, with the
-A→B/C forwarding decision still ahead.
+echo the request's params — `docs/05` §3.5), and the engine inventory is now
+complete (the async copy engine is included; the copy-engine capabilities
+match the device). Context creation currently ends in a forced `rc=999` on a
+completion wait while the completion/translation plumbing is being made
+reliable. The *data plane* is at the bridge stage: the host module decodes the
+guest's push buffers as method streams and executes the small work it
+understands itself (memsets, copies, semaphore releases) instead of forwarding
+it to the physical GPU. No CUDA kernel has been forwarded yet; the execution
+backend is chosen but not started. So: we are still in the
+"emulate + software-execute state/memory ops" phase, with the backend decision
+now made (option D in the integration document).
**Risks to guard now:** (1) software execution silently becoming the data
plane instead of a bridge; (2) the work-path interface being skipped because