diff options
| author | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-10-06 15:18:39 -0400 |
|---|---|---|
| committer | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-10-06 15:18:39 -0400 |
| commit | 929646875d80408a21c642ca88ef70efbc0d736a (patch) | |
| tree | deb2946cc4407e4edcc6e3b56f37420e321acf40 /docs/03-architecture | |
| parent | 039daee13c5bac50919eff1ca26977ac49e37b06 (diff) | |
| download | open-iov-929646875d80408a21c642ca88ef70efbc0d736a.tar.gz open-iov-929646875d80408a21c642ca88ef70efbc0d736a.zip | |
docs: add the GR compute-launch frontier page and align execution status across the tree
New page docs/04-implementation/03-execution-frontier.md describes the chain
that must fire for a guest CUDA launch (guest push -> GPFIFO/front-end ->
inline launch-descriptor install -> engine read/dispatch -> completion),
where the leading edge is (the descriptor install path), how we know
(front-end vs engine-authored observables and their limits), what is
exonerated (context validity at dispatch, destination window mapping,
fetch alignment, shared-buffer rule, parenting), the ranked issues (verbatim
prelude targets, missing descriptor-virtualization control, sentinel
hygiene), live investigation lines and labelled theories.
Align every page that still said GR execution is blocked on engine-context
validity to the current frontier, and add the new page to the README reading
order (row 15, later rows renumbered). Docs only; no code.
Diffstat (limited to 'docs/03-architecture')
| -rw-r--r-- | docs/03-architecture/01-system-overview.md | 7 | ||||
| -rw-r--r-- | docs/03-architecture/02-guest-stack-and-cudasw.md | 6 | ||||
| -rw-r--r-- | docs/03-architecture/03-gsp-and-gsp-rm.md | 8 | ||||
| -rw-r--r-- | docs/03-architecture/04-uvm.md | 13 | ||||
| -rw-r--r-- | docs/03-architecture/07-driver-lifecycle-field-guide.md | 7 | ||||
| -rw-r--r-- | docs/03-architecture/08-physical-gpu-integration.md | 4 | ||||
| -rw-r--r-- | docs/03-architecture/10-interaction-and-scheduling.md | 2 | ||||
| -rw-r--r-- | docs/03-architecture/11-physical-backend-guest-wiring.md | 8 |
8 files changed, 39 insertions, 16 deletions
diff --git a/docs/03-architecture/01-system-overview.md b/docs/03-architecture/01-system-overview.md index 8205154..71f52f2 100644 --- a/docs/03-architecture/01-system-overview.md +++ b/docs/03-architecture/01-system-overview.md @@ -174,9 +174,10 @@ decompiled and mapped (see `wiki/GSP-570.144W-Firmware-Analysis.md` and completes and the full CUDA driver-API smoke passes (`ALL OK` 64/64, copies matched). The execution backend is live: CE-class copies are forwarded to the physical copy engine (default off) with engine-authored - semaphore releases and a fault relay; kernel execution is blocked on GR - engine-context validity (see - [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)). + semaphore releases and a fault relay; for GR, a forwarded guest launch + reaches the front-end but the inline launch-descriptor install has not been + observed (no grid dispatched; context validity at dispatch is ruled out — + see [`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md)). The CUDA-side contract and the reply-fidelity rules are documented in [`../../userspace/cuda/docs/03-driver-and-gpu-interface.md` §9](../../userspace/cuda/docs/03-driver-and-gpu-interface.md) diff --git a/docs/03-architecture/02-guest-stack-and-cudasw.md b/docs/03-architecture/02-guest-stack-and-cudasw.md index f9fe249..845e00f 100644 --- a/docs/03-architecture/02-guest-stack-and-cudasw.md +++ b/docs/03-architecture/02-guest-stack-and-cudasw.md @@ -95,8 +95,10 @@ module-owned GPU running its real GSP firmware as the resource-manager server (see [`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md) and [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)) — with -CE-class copies forwarded to the physical copy engine (default off) and GR -kernel execution blocked on GR engine-context validity. +CE-class copies forwarded to the physical copy engine (default off); for GR, a +forwarded guest launch reaches the front-end but the inline launch-descriptor +install has not been observed (context validity at dispatch is ruled out; see +[`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md)). For the CUDA userspace side of this stack — its architecture, how it is built and executes, and the exact kernel/GPU interface contract we must satisfy — diff --git a/docs/03-architecture/03-gsp-and-gsp-rm.md b/docs/03-architecture/03-gsp-and-gsp-rm.md index 10e6a73..56e77be 100644 --- a/docs/03-architecture/03-gsp-and-gsp-rm.md +++ b/docs/03-architecture/03-gsp-and-gsp-rm.md @@ -232,9 +232,11 @@ The engine-inventory row was the CUDA blocker at the time of writing: `libcuda` filters the engine list it receives against the class list and expects the async copy engine to be present, because the secondary (async) engine record is otherwise zeroed. That inventory is now complete and context -creation passes; the current frontier is engine execution, where CE-class -copies are forwarded to the physical copy engine (default off) and GR kernel -execution is blocked on GR engine-context validity. See +creation passes; the current frontier is engine execution: CE-class copies are +forwarded to the physical copy engine (default off), and for GR a forwarded +guest launch reaches the front-end but the inline launch-descriptor install has +not been observed (context validity at dispatch is ruled out; see +[`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md)). See [`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md) section 3.7 for the offsets and the consumed fields, and [`../../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../../userspace/cuda/docs/03-driver-and-gpu-interface.md) diff --git a/docs/03-architecture/04-uvm.md b/docs/03-architecture/04-uvm.md index 9396e6a..d4232e0 100644 --- a/docs/03-architecture/04-uvm.md +++ b/docs/03-architecture/04-uvm.md @@ -52,17 +52,20 @@ Each stage has a distinct failure signature; our debugging docs map them. | GPU identity | must match the uuid from RM | ✅ | | CE capabilities | per-engine support bits; UVM *requires* `supported && !grce && sysmem` for CPU_TO_GPU/GPU_TO_CPU channels | 🟡 the caps we answer are accepted, the inventory now includes the async copy engine, and CE copies are forwarded to the physical copy engine (default off) with engine-authored releases | | Channel creation | RM channel objects with GPFIFO/userd/instance parameters | ✅ objects created | -| Pushes executed | UVM's first action on a new channel is an *init push* followed by a semaphore-tracked wait | 🟡 the host decodes method streams and executes memsets, copies and semaphore releases in software (a bridge); CE-class copies and releases are additionally forwarded to the physical copy engine (default off); no GR kernel is forwarded yet (blocked on GR engine-context validity) | +| Pushes executed | UVM's first action on a new channel is an *init push* followed by a semaphore-tracked wait | 🟡 the host decodes method streams and executes memsets, copies and semaphore releases in software (a bridge); CE-class copies and releases are additionally forwarded to the physical copy engine (default off); for GR, a forwarded launch reaches the front-end but the inline launch-descriptor install has not been observed (see [`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md)) | | Page trees | UVM initializes page trees with its own pushes; the *server-owned* ranges must be populated by us | 🟡 | | Fault delivery | fault buffer + notification; replay/cancel semantics | 🟡 backend-side fault observation/relay exists (default off: notifier write + event post); the guest replay/cancel contract is open | | Semantics of completions | tracker values must advance exactly | 🟡 | -**Current frontier (2026-10-02):** context creation and the full CUDA +**Current frontier (2026-10-06):** context creation and the full CUDA driver-API smoke pass (`ALL OK`, copies matched); the execution backend is live and CE-class copies are forwarded to the physical copy engine (default -off) with engine-authored releases and a fault relay. GR kernel execution is -blocked on GR engine-context validity; the remaining UVM work is completion -translation and the guest-side fault replay/cancel contract +off) with engine-authored releases and a fault relay. For GR, a forwarded guest +launch reaches the front-end but the inline launch-descriptor install has not +been observed (context validity at dispatch is ruled out; see +[`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md)). +The remaining UVM work is completion translation and the guest-side fault +replay/cancel contract ([`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)). Failed-context teardown has produced guest kernel oopses in `uvm_pmm_devmem_page_to_va_space` in earlier runs. The CUDA-side diff --git a/docs/03-architecture/07-driver-lifecycle-field-guide.md b/docs/03-architecture/07-driver-lifecycle-field-guide.md index 3a85da6..ff886c6 100644 --- a/docs/03-architecture/07-driver-lifecycle-field-guide.md +++ b/docs/03-architecture/07-driver-lifecycle-field-guide.md @@ -185,6 +185,13 @@ push faults with an uncommitted GR context (`CTXNOTVALID`), and the the designed locked fail-safe (see [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md) §7/§8). +**Update (2026-10-06):** the GR frontier has moved past the context-commit +stage. A forwarded guest CUDA launch now reaches the front-end (ring entry +fetched, methods executed) but the inline launch-descriptor install has not +been observed and no grid has dispatched; context validity at dispatch is ruled +out. See +[`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md). + **Log signatures (guest side):** ``` nvidia-uvm: ... Tracker entry for value N (sema VA 0x...) channel ID ... CE ... diff --git a/docs/03-architecture/08-physical-gpu-integration.md b/docs/03-architecture/08-physical-gpu-integration.md index 6e13e7b..c8005c6 100644 --- a/docs/03-architecture/08-physical-gpu-integration.md +++ b/docs/03-architecture/08-physical-gpu-integration.md @@ -287,9 +287,9 @@ per-context scheduling. Tenant policy and fairness are a later layer on top. | Power-domain lock across the claim window | verified — 3/3 supervised claim/release cycles (2026-10-01; §4.2, §8); latched by default since the G0 bundle (`gsp_phys_repower=0`) | | Real GSP boot by the module | done — full boot through init-done (`0x1001`, `rpc_result=0`) and a clean release with no reboot on every supervised cycle since 2026-10-01 | | Host-side resource-manager client for real objects/channels | done — full static-config reply served; real memory, vaspace, channel-group and channel objects allocated; real CE copies executed on the physical engine | -| Work translation and real submission | CE forwarding and engine-authored release relay implemented default-off and verified live; GR execution open | +| Work translation and real submission | CE forwarding and engine-authored release relay implemented default-off and verified live; for GR, a forwarded guest launch reaches the front-end but the inline launch-descriptor install has not been observed (see [`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md)) | | Completion relay through real hardware | implemented default-off — engine-authored release verified live and forwarded-engine fault observation/relay behind `gsp_phys_fault_relay`, fail-closed to software on any miss; one relay data mismatch under triage | -| CUDA kernel execution (`vectorAdd`) | blocked on GR context validity/execution — the host GR ladder's first push faulted `CTXNOTVALID` and the promote attempt wedged GSP-RM | +| CUDA kernel execution (`vectorAdd`) | open — context validity at dispatch is ruled out; the frontier is the launch descriptor install path (stage 6–7 of [`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md)) | ## 7. Open questions diff --git a/docs/03-architecture/10-interaction-and-scheduling.md b/docs/03-architecture/10-interaction-and-scheduling.md index ffc02d7..617d219 100644 --- a/docs/03-architecture/10-interaction-and-scheduling.md +++ b/docs/03-architecture/10-interaction-and-scheduling.md @@ -22,7 +22,7 @@ to, and where it stands. | Guest doorbell signal (boot, mailbox, channel) | write into BAR0 shadow, observed by the polling service | steering only | [done] | | Guest RPC request | element placed in a guest-sysmem message queue, read by the module over VFIO DMA | 10s of us per round trip | [done] | | RPC reply + interrupt | module writes reply into the queue, signals eventfd, QEMU raises irqfd into KVM, MSI-X lands in the guest | us | [done] | -| Push-buffer work | guest appends GPFIFO entries; the per-channel pump reads, decodes and applies them | per entry, us to low ms | [wip] (CE copies forward default-off; GR execution open) | +| Push-buffer work | guest appends GPFIFO entries; the per-channel pump reads, decodes and applies them | per entry, us to low ms | [wip] (CE copies forward default-off; GR entry fetch live, launch-descriptor install is the frontier — [`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md)) | | Semaphore release to guest memory | module writes the payload word into the resolved guest page; a forwarded CE copy can have the engine author it | us | [done] (relay default off) | | VA-to-PA translation discovery | in-module structure walk, content scan, rescue, pins | us to seconds (scan bounded) | [wip] | | Translation internalization from protocol | derive mappings from RM/UVM objects as they are created | us | [planned] | diff --git a/docs/03-architecture/11-physical-backend-guest-wiring.md b/docs/03-architecture/11-physical-backend-guest-wiring.md index 943dd44..22d06ed 100644 --- a/docs/03-architecture/11-physical-backend-guest-wiring.md +++ b/docs/03-architecture/11-physical-backend-guest-wiring.md @@ -27,6 +27,14 @@ > forwarded-engine fault relay is live green, default off. The P6a trigger > residual is fixed: triggers 6/7 re-translate at trigger time and use fresh > scratch lines. +> +> **Update (2026-10-06): the frontier has moved.** A forwarded guest CUDA +> launch now reaches the front-end (ring entry fetched, methods executed) but +> the inline launch-descriptor (QMD) install has not been observed and no grid +> has dispatched; context validity at dispatch is ruled out. The current wall +> is the form of the forwarded launch method stream — the chain, evidence and +> live investigation lines are in +> [`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md). > The physical backend itself is live: the module-owned > GSP-RM completes initialization on the real TU104, serves the full > static-config reply, and shuts down gracefully. The host resource-manager |
