diff options
| author | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-10-20 05:22:19 -0400 |
|---|---|---|
| committer | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-10-20 05:22:19 -0400 |
| commit | ae534205925529fc06e0539218f0ddf9fe5f835a (patch) | |
| tree | cb9451984c72e995c2bb4ce85d4ad4ad162fc29d /wiki | |
| parent | 16291c723caf6693832467fe5fc023a1c9923888 (diff) | |
| download | open-iov-ae534205925529fc06e0539218f0ddf9fe5f835a.tar.gz open-iov-ae534205925529fc06e0539218f0ddf9fe5f835a.zip | |
site: 2026-10-20 status refresh - guest CUDA kernel execution working
Diffstat (limited to 'wiki')
| -rw-r--r-- | wiki/.publish-manifest.json | 46 | ||||
| -rw-r--r-- | wiki/CUDA-Userspace-Driver.md | 32 | ||||
| -rw-r--r-- | wiki/Execution-Frontier.md | 177 | ||||
| -rw-r--r-- | wiki/Physical-GPU-Backend.md | 37 | ||||
| -rw-r--r-- | wiki/Project-Status.md | 35 |
5 files changed, 181 insertions, 146 deletions
diff --git a/wiki/.publish-manifest.json b/wiki/.publish-manifest.json index 20e9722..4fe571b 100644 --- a/wiki/.publish-manifest.json +++ b/wiki/.publish-manifest.json @@ -19,11 +19,11 @@ }, "CUDA Userspace Driver": { "file": "CUDA-Userspace-Driver.wiki", - "new_revid": 23963, + "new_revid": 23998, "old_revid": 23963, - "source_sha": "31d77a3a672b2943", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:40:06Z", + "source_sha": "d467dbcd9f323768", + "summary": "Status refresh 2026-10-20: guest CUDA kernels execute on the physical GPU (vector-add/matmul/transformer green); boot-poisoning defect fixed", + "timestamp": "2026-10-20T09:19:03Z", "title": "CUDA Userspace Driver" }, "Community": { @@ -163,11 +163,11 @@ }, "Main Page": { "file": "Main-Page.wiki", - "new_revid": 23972, - "old_revid": 23960, - "source_sha": "1b53ef37afca940e", - "summary": "Fix repository links to the cgit tree/repo URLs", - "timestamp": "2026-10-12T04:40:23Z", + "new_revid": 24000, + "old_revid": 23972, + "source_sha": "2e8171cab56c394c", + "summary": "Status table refresh 2026-10-20: guest CUDA kernel execution working", + "timestamp": "2026-10-20T09:19:09Z", "title": "Main Page" }, "MediaWiki:Sidebar": { @@ -199,20 +199,20 @@ }, "Physical GPU Backend and Guest Wiring": { "file": "Physical-GPU-Backend.wiki", - "new_revid": 23985, - "old_revid": 23951, - "source_sha": "4bff4dbfa8b4644e", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:40:59Z", + "new_revid": 23999, + "old_revid": 23985, + "source_sha": "d762abd50cef636d", + "summary": "Status refresh 2026-10-20: guest CUDA kernels execute on the physical GPU (vector-add/matmul/transformer green); boot-poisoning defect fixed", + "timestamp": "2026-10-20T09:19:05Z", "title": "Physical GPU Backend and Guest Wiring" }, "Project Status": { "file": "Project-Status.wiki", - "new_revid": 23986, - "old_revid": 23952, - "source_sha": "6d07f2a0c6f524a9", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:41:00Z", + "new_revid": 23997, + "old_revid": 23986, + "source_sha": "cb714adeebcfdc33", + "summary": "Status refresh 2026-10-20: guest CUDA kernels execute on the physical GPU (vector-add/matmul/transformer green); boot-poisoning defect fixed", + "timestamp": "2026-10-20T09:19:01Z", "title": "Project Status" }, "RPC Message Format": { @@ -262,11 +262,11 @@ }, "The Execution Frontier": { "file": "Execution-Frontier.wiki", - "new_revid": 23966, + "new_revid": 23996, "old_revid": 23966, - "source_sha": "b26ce9bb6fe63f6a", - "summary": "Fix repository links: point to the cgit tree URLs", - "timestamp": "2026-10-12T04:40:07Z", + "source_sha": "8c0007ec287387b9", + "summary": "Status refresh 2026-10-20: guest CUDA kernels execute on the physical GPU (vector-add/matmul/transformer green); boot-poisoning defect fixed", + "timestamp": "2026-10-20T09:19:00Z", "title": "The Execution Frontier" }, "The Guest Stack and CUDASW": { diff --git a/wiki/CUDA-Userspace-Driver.md b/wiki/CUDA-Userspace-Driver.md index 9a041ef..6975a1d 100644 --- a/wiki/CUDA-Userspace-Driver.md +++ b/wiki/CUDA-Userspace-Driver.md @@ -1,16 +1,20 @@ # CUDA Userspace Driver (`libcuda.so`) — Analysis Deep Dive -> **Status (2026-10-12, honest).** The analysis covers the whole library for +> **Status (2026-10-20, honest).** The analysis covers the whole library for > driver 570.144 and the tooling is version-agnostic. The *control plane* gets > the stock guest stack through `cuInit`, device enumeration, `cuCtxCreate` > and the full driver-API smoke test — allocation, copies, free and destroy > complete end to end (`ALL OK`; the 64-byte payload round-trips exactly). > The *data plane* decodes guest push buffers and executes the memory and > semaphore work it understands; CUDA kernel launches are forwarded to the -> physical execution path, where a launch reaches the engine queue -> (front-end deposit and scheduling are observed) but an engine-authored -> release is not yet observed. Green runs are not yet reproducible on every -> boot — an open issue, tracked in the repository. Canonical detail: +> physical execution path and now execute on the physical GPU — the +> engine-authored completions relay to the guest and its synchronisation +> waits resolve. The vector-add (`VADD n=32768 bad=0`), matrix-multiply +> (`MATMUL n=256 bad=0`) and transformer (`dh=256 seq=64 bad=0`, +> `FFN4 layers=4 bad=0`) clients verify green on fresh boots. The +> large-buffer staged-transfer class (multi-megabyte transfers between guest +> memory and device memory) is under repair, and reproducibility across fresh +> boots is being re-measured after the boot-poisoning fix. Canonical detail: > [`01-architecture.md`](../userspace/cuda/docs/01-architecture.md) > through `08`; reading order > [`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md); @@ -172,19 +176,19 @@ the data plane (`docs/08`). emulation and object/control replies are implemented; the stock guest stack completes `cuInit`, device enumeration, `cuCtxCreate` and the driver-API smoke test (allocation, copies, free and destroy). -* **Current frontier:** a guest CUDA kernel launch is forwarded and reaches - the physical engine queue — the front end deposits the launch and the - scheduling path accepts it — but the engine does not yet execute the - forwarded launch descriptor: an engine-authored release is the only proof - of execution and it is not observed. The per-boot green/stall variance on - the emulation path is an open issue, tracked in the repository. The chain, - the evidence and the live state are on +* **Current frontier:** guest CUDA kernel launches execute on the physical + GPU with engine-authored completions relayed to the guest; the verified + clients are green on fresh boots. The leading edge is the large-buffer + staged-transfer class (some large multi-chunk clients still mis-verify), + re-measurement of reproducibility across fresh boots after the + boot-poisoning fix, and per-guest execution isolation. The chain, the + evidence and the live state are on [`Execution-Frontier.md`](Execution-Frontier.md). * **Data plane today:** the host module decodes guest push buffers as method streams and executes the memory and semaphore work it understands (memsets, copies, semaphore releases), and forwards engine-class work to - the physical backend; kernel launches are forwarded but not yet executed - by the engine. + the physical backend; kernel launches now execute on the engine through + the forwarded path, with completions relayed back to the guest. ## 5. The crash-path lesson: replies are consumed data diff --git a/wiki/Execution-Frontier.md b/wiki/Execution-Frontier.md index fe21fda..2d9d80c 100644 --- a/wiki/Execution-Frontier.md +++ b/wiki/Execution-Frontier.md @@ -1,15 +1,16 @@ # The Execution Frontier -> **What this page is.** The current leading edge of guest-facing execution: -> the chain that must fire for a stock guest CUDA kernel to run on the -> physical GPU, what has been proven about it, where it stops, how we know, -> and the open lines of investigation and theories. It is evidence-led: every -> statement carries a live observation, a decompilation finding, or a clearly -> labelled theory. Condensed from +> **What this page is.** The state of guest-facing execution: the chain that +> must fire for a stock guest CUDA kernel to run on the physical GPU, what has +> been proven about it, how we know, and where the leading edge is now. The +> chain fires end to end: verified guest CUDA clients execute on the physical +> GPU and wake up correctly; the leading edge is now breadth and hardening. It +> is evidence-led: every statement carries a live observation, a +> decompilation finding, or a clearly labelled theory. Condensed from > [`../docs/04-implementation/03-execution-frontier.md`](../docs/04-implementation/03-execution-frontier.md); > the docs tree wins on disagreement. -Status: 2026-10-12 (source page dated 2026-10-06), tree `master @ 2f066727`. +Status: 2026-10-20, tree `master @ 16291c72`. Tags: `[live]` observed; `[wip]` partial; `[wall]` current blocker; theories are labelled as such. @@ -45,46 +46,62 @@ guest: builds launch + method stream | 3 | Host decode + forward stream + QMD prewrite + ring/USERD | `[live]` observed | | 4 | Front-end fetches the ring entry | `[live]` observed (ring cursor advances) | | 5 | Front-end executes methods | `[live]` observed (object bind, cache ops, report semaphore) | -| 6 | **Inline-QMD install** (descriptor materialised) | `[wall]` not observed | -| 7 | **Engine reads the QMD** | `[wall]` not observed | +| 6 | **Inline-QMD install** (descriptor materialised) | `[live]` observed | +| 7 | **Engine reads the QMD** | `[live]` observed | | 8 | Context switch loads the channel context | `[live]` context validity at dispatch is ruled in | -| 9 | GR dispatches the grid to the SMs | never observed on this stack | -| 10 | Engine authors release + channel semaphore | absent for forwarded launches | -| 11 | Host relays completion; guest wakes | the guest waits and times out | +| 9 | GR dispatches the grid to the SMs | `[live]` observed (the guest's results verify) | +| 10 | Engine authors release + channel semaphore | `[live]` engine-authored releases relay to the guest | +| 11 | Host relays completion; guest wakes | `[live]` the guest's synchronisation waits resolve | -**One-line summary:** the guest, the emulation, the forwarding path and the -front-end are live; the **launch-descriptor install path** (stages 6–7) is the -leading edge, and with it stages 9–11. +**One-line summary:** the whole chain is live; the leading edge is now breadth +and hardening — the large-buffer staged-transfer class, reproducibility across +boots, and per-guest execution isolation. ## 2. Where the leading edge is -A stock guest program now reaches its kernel launch and submits it through the -forwarding path. On the wire the ring entry is consumed and the front-end -methods run — but the launch descriptor is never materialised, the engine -never reads a QMD, no engine-authored completion appears, and the guest blocks -in its synchronisation wait. - -This is **not** a context-validity wall (ruled out at dispatch) and **not** a -window-mapping wall for the descriptor destination (that install is accepted). -The wall is the **form of the launch method stream**: what the front-end -requires in order to install the inline descriptor, and what the engine -requires in order to read and dispatch it. +A stock, unmodified guest CUDA program executes its kernels on the physical +GPU and verifies its results. On fresh boots the vector-add client +(`victory_mail`) prints `VADD n=32768 bad=0` + `STATUS OK`, the +matrix-multiply client (`victory_matmul`) prints `MATMUL n=256 bad=0` + +`STATUS OK`, and a transformer kernel prints `dh=256 seq=64 bad=0`, +`FFN4 layers=4 bad=0` + `STATUS OK`. The executing engine is the physical GPU +under the module's own real GSP-RM session; the host CUDA bridge is not part +of this path. Engine-authored completions relay to the guest and its +synchronisation waits resolve. + +The leading edge has moved from the launch chain itself — it fires — to +breadth and hardening around it: + +* **The large-buffer staged-transfer class.** Multi-megabyte staged transfers + between guest memory and device memory are under repair: some large + multi-chunk clients still mis-verify. +* **Reproducibility across boots.** A boot-poisoning defect was root-caused + and fixed: under the driver's status-queue mode the emulation read the wrong + queue cursor, so under load the reply ring could desynchronise and the first + CUDA client would fail (the user-visible `STATUS FAIL`-then-freeze report). + Re-measurement across fresh boots is in progress. +* **Per-guest execution isolation.** Multiple mediated devices with different + emulated VRAM sizes and multiple guests already coexist at the device level; + per-guest execution isolation for multiple simultaneous CUDA guests is in + progress. + +No breadth beyond these is claimed: OpenGL/Vulkan, Windows guests, migration +and performance numbers remain planned. ## 3. How we know (observables and their limits) | Observable | What it proves | What it does not prove | |---|---|---| -| Ring entry consumed; ring cursor advances | the front-end is alive and fetching | nothing about the engine | -| Method execution on the ladder channel | the front-end can execute the method vocabulary | **not** engine execution — these signals can be front-end-authored | -| Descriptor buffer prewritten with a detectable sentinel | whether the descriptor was materialised after submission | a read does not modify memory; the sentinel detects installs, not reads | -| Install sentinel unchanged after submit | the inline-install method did not run | it does not say why | -| Engine-authored release word absent; poll times out | the engine did not complete the launch | it does not say whether the engine started | +| Client verification output (`bad=0` + `STATUS OK`) | the guest executed its kernel and verified its results | by itself, which engine produced the results | +| Engine-authored release relayed to the guest | the engine executed the launch and the relay works | nothing about other launches | +| The guest's synchronisation wait resolving | completion reached the guest through the relay | nothing about other launches | +| Ring entry consumed; ring cursor advances | the front-end is alive and fetching | **not** engine execution — front-end signals are necessary but not sufficient | +| Reproduction across fresh boots | the path is not a lucky boot | no "every boot" claim yet while re-measurement is in progress | | No GR fault recorded | the engine did not raise this error | silence is not health: a skipped launch is silent by construction | -| Channel instance context flag set | the channel is context-valid at dispatch | nothing about the descriptor path | -**Consequence:** progress on this frontier must be measured by an -**engine-authored** signal (the release word advancing) — not by front-end -signals, which we have already learned to distinguish. +**Consequence:** acceptance is verified guest results plus the engine-authored +release; a green run counts as progress only when it reproduces across fresh +boots. ## 4. What is exonerated (do not relitigate) @@ -98,68 +115,64 @@ signals, which we have already learned to distinguish. already initialised in the same cycle; not the dispatch delta. * Parenting/TSG for the fetch — both variants fetch identically. * The copy-engine path — CE-class work, releases and fault relay are live - (default off); the GR path is a separate problem. + (default off); the GR path is separate. +* The descriptor-gate line of investigation (the old T1–T4 theories) — the + inline-descriptor install, the engine read, the dispatch, the + engine-authored release and the guest wake-up are all observed; closed by + observation. +* The old boot-poisoning defect — root-caused to the status-queue consumption + cursor and fixed. ## 5. The clear issues (ranked) -**Issue A — the forwarded stream carries a verbatim guest prelude that cannot -complete.** The forwarded stream includes a verbatim copy of the guest's -prelude whose inline-data writes target addresses not present in the engine's -view of the channel's page tables, and the referenced data is not relocated. -The shorter, proven stream shape fires its tail (channel semaphore) but still -does not schedule the descriptor. +**Issue A — the large-buffer staged-transfer class.** Multi-megabyte staged +transfers between guest memory and device memory are under repair: some large +multi-chunk clients still mis-verify. -**Issue B — the descriptor-virtualization control method is absent.** The -method that arms the inline-descriptor install path is not present in the -forwarded stream. Fixing it should materialise the descriptor — falsifiable in -one cycle by watching the sentinel and the engine-read signal. +**Issue B — reproducibility across boots (re-measurement).** The +boot-poisoning defect (wrong status-queue consumption cursor under the +driver's status-queue mode; the reply ring could desynchronise under load, the +first client would fail) was root-caused and fixed. Re-measurement across +fresh boots is in progress; no "every boot" claim is made until it completes. -**Issue C — diagnostic hygiene.** The sentinel placed at word 0 overlaps real -descriptor fields; it must be moved to an unused word so diagnostics do not -perturb what they measure. +**Issue C — per-guest execution isolation.** Multiple mediated devices with +different emulated VRAM sizes and multiple guests coexist at the device level; +per-guest execution isolation for simultaneous CUDA guests is in progress. ## 6. Lines of investigation (live) -1. **Restore the proven launch shape** as the baseline and **re-add the - virtualization control method** — one variable at a time, with the install - sentinel and the engine-read signal as the readout. -2. **Relocate or drop the verbatim prelude** so inline-data writes target - addresses present in the engine's view, or replace it with the host-side - staging the proven shape already uses. -3. **Move the sentinel** and re-run to confirm the readout is clean. -4. **Diff against a known-good reference** — our own ladder dispatching a - descriptor built from the same source material, or a captured working - launch of the stock stack on the same GPU — converting the remaining work - from inference to differential. -5. **Offline probing** of any rule the front-end or engine is believed to - require, using the firmware debugger before spending a supervised cycle - ([`Riscv-GSP-RM-Debugger.md`](Riscv-GSP-RM-Debugger.md)). - -## 7. Theories (labelled; not yet proven) - -* **T1 — the virtualization control is the gate.** The front-end will not - materialise an inline descriptor until the control method enables the - install path. -* **T2 — materialise-then-read.** The engine only schedules a descriptor that - exists in its own view; an install targeting unmapped destinations aborts - silently rather than faulting. -* **T3 — the verbatim prelude poisons the launch.** Its refused writes leave - the launch path unable to schedule the descriptor. -* **T4 — relocation is the missing transform.** The guest's descriptor is - valid for the guest's view of memory; the forwarding path must relocate - referenced data into the channel's view before install. - -Each theory has a one-cycle falsifier: the install sentinel, the engine-read -signal, and the engine-authored release, in that order. +1. **Repair the large-buffer staged-transfer class** — the three green clients + stay the regression battery and must remain green throughout. +2. **Complete the fresh-boot re-measurement** — demonstrate the fixed boot + path across fresh boots before claiming boot-to-boot reproducibility. +3. **Per-guest execution isolation** — build from the device-level coexistence + to isolation for simultaneous CUDA guests. +4. **Hold the acceptance battery** — vector-add, matrix-multiply and the + transformer kernel; each entry is accepted only on its own verification + output (`bad=0` + `STATUS OK`). + +## 7. Theories, resolved + +The descriptor-gate theories (T1–T4) are resolved and are not kept as open: +the inline-descriptor install, the engine's QMD read, the grid dispatch, the +engine-authored release and the guest wake-up are all directly observed, so +the launch chain they questioned is proven end to end. The remaining work is +engineering (large-buffer transfers, boot-to-boot reproducibility, per-guest +isolation), each with an acceptance signal named before the work. ## 8. Claim discipline -* **Engine-authored release is the only proof of execution.** Front-end - signals are necessary but not sufficient. +* **Engine-authored release is the acceptance signal for execution** — now + observed end to end; the guest's own verification (`bad=0` + `STATUS OK`) + is the guest-side acceptance signal. +* **Front-end signals are necessary but not sufficient.** Ring cursor and + front-end method execution can be front-end-authored; they never substitute + for the engine-authored release. * **Silence is not health.** No fault recorded does not mean no fault occurred. * **Emulator verdicts are instruction/memory-level.** They bound behaviour, never timing or device environment. +* **A green run counts only when it reproduces across fresh boots.** * **A wall is closed only by an acceptance signal**, named before the experiment. diff --git a/wiki/Physical-GPU-Backend.md b/wiki/Physical-GPU-Backend.md index 1b83203..bbca084 100644 --- a/wiki/Physical-GPU-Backend.md +++ b/wiki/Physical-GPU-Backend.md @@ -5,7 +5,7 @@ > guest-facing emulation. Canonical sources: [`../docs/03-architecture/08-physical-gpu-integration.md`](../docs/03-architecture/08-physical-gpu-integration.md) > and [`../docs/03-architecture/11-physical-backend-guest-wiring.md`](../docs/03-architecture/11-physical-backend-guest-wiring.md); the docs tree wins on disagreement. -Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` proven with the +Status: 2026-10-20, tree `master @ 16291c72`. Tags: `[live]` proven with the stock guest driver; `[gated]` implemented default-off; `[wip]` partially built; `[wall]` current blocker; `[target]` planned end state. @@ -64,13 +64,15 @@ independently verified: 5. **Guest mediation** — the guest-facing emulation stays unchanged; real-resource objects translate to GSP-RM requests, and server-owned ranges populate the real MMU. `[wip]` 6. **Work path** — decoded guest pushes become real push streams; resolved host pages - map through module-built scratch page tables; the firmware schedules the work. `[gated]` + map through module-built scratch page tables; the firmware schedules the work. + `[live]` observed end to end for the verified guest kernel clients. 7. **Completion relay** — hardware-completed semaphores land in guest-polled memory; - events and faults relay through the existing MSI-X injection path. `[gated]` + events and faults relay through the existing MSI-X injection path. + `[live]` engine-authored completions relay to the guest and its waits resolve. ## 5. The two planes and the wiring -The **emulation plane** serves the guest: BAR shadows, the sysmem msgq protocol, a hand-built TU104 static-configuration model, and a software work pipeline that executes decoded memsets, copies and semaphore releases. The **physical backend plane** owns the device, boots the real GSP-RM and runs the RM client ladder. The wiring joins them at the lifecycle seam (mdev create/remove), a static-config identity overlay, a read-only control whitelist, and — for eligible copy-engine work — the scratch-mapped forwarding path, all default-off and fail-open to software. +The **emulation plane** serves the guest: BAR shadows, the sysmem msgq protocol, a hand-built TU104 static-configuration model, and a software work pipeline that executes decoded memsets, copies and semaphore releases. The **physical backend plane** owns the device, boots the real GSP-RM and runs the RM client ladder. The wiring joins them at the lifecycle seam (mdev create/remove), a static-config identity overlay, a read-only control whitelist, and — for eligible work — the scratch-mapped forwarding path, parameterised and fail-open to software. ```text GUEST (QEMU) HOST: vgsp.ko (module-owned GPU) PHYSICAL GPU @@ -85,8 +87,9 @@ INIT preflight: distribution driver green; no vgsp/mdev/QEMU [live] RM battery, UVM registration [live] WORK guest CUDA app -> libcuda -> nvidia.ko -> GPFIFO -> doorbell host pump decodes; eligible CE copies forwarded via module-built scratch - page tables; GR: launch reaches the front-end, descriptor not seen [wall] -DONE engine authors the release in guest memory; host relays via MSI-X [gated] + page tables; GR: launches execute on the engine, completions relay + to the guest [live] +DONE engine authors the release in guest memory; host relays via MSI-X [live] TEARDOWN mdev remove -> QEMU stop -> module unload -> clean GSP-RM shutdown -> protected region down -> distribution driver rebound [live] FAIL a failed mutating operation parks the device in a locked fail-safe; @@ -104,17 +107,20 @@ FAIL a failed mutating operation parks the device in a locked fail-safe; | Power domain held active across the ownership window | verified across supervised cycles `[live]` | | Real GSP boot by the module, through init-done and clean release | done, no reboot on clean cycles `[live]` | | Host RM client and first real engine work (static config, real objects, 4 KiB CE copy) | done `[live]` | -| Guest-copy forwarding, deferred release, hardware-primary copies, engine-authored relay, fault relay | default-off, verified live; one relayed-copy mismatch under triage `[gated]` | -| CUDA kernel execution (`vectorAdd`) | open `[wall]` — the launch-descriptor install path; correct results are the `[target]` | +| Guest-copy forwarding, deferred release, hardware-primary copies, engine-authored relay, fault relay | verified live (seams parameterised and fail-open); the large-buffer staged-transfer class is under repair `[live]` | +| CUDA kernel execution (`vectorAdd` and further clients) | done `[live]` — stock guest CUDA clients (vector-add, matrix-multiply, transformer) execute on the physical engine with verified results and engine-authored completions relayed to the guest | ## 7. The execution frontier The RM client ladder reaches the compute engine (a GR0 channel plus a -compute-class object). A forwarded guest CUDA launch now -reaches the front-end (ring entry fetched, methods executed) but the inline -launch-descriptor install has not been observed and no grid has dispatched; -context validity at dispatch is ruled out. The wall is the form of the -forwarded launch method stream — see [`Execution-Frontier.md`](Execution-Frontier.md). +compute-class object). Forwarded guest CUDA launches now execute on the +engine: the inline launch descriptor installs and is read, the grid +dispatches, the engine authors the completion, the host relays it, and the +guest's synchronisation wait resolves — verified on fresh boots with the +vector-add, matrix-multiply and transformer clients. The leading edge is the +large-buffer staged-transfer class (under repair), reproducibility across +fresh boots, and per-guest execution isolation — see +[`Execution-Frontier.md`](Execution-Frontier.md). A failed mutating operation on the backend parks the device in a locked fail-safe that is never auto-retried; recovery is a supervised step. @@ -147,6 +153,9 @@ fail-safe that is never auto-retried; recovery is a supervised step. access to guest memory stays behind the hypervisor's per-device DMA interface, and tenant policy is a later layer on top ([`GMMU-Multi-Tenant-Protection.md`](GMMU-Multi-Tenant-Protection.md)). +* Multiple mediated devices with different emulated VRAM sizes and multiple + guests already coexist at the device level; per-guest execution isolation + for simultaneous CUDA guests is in progress. ## 10. Open questions @@ -159,7 +168,7 @@ fail-safe that is never auto-retried; recovery is a supervised step. * [`../docs/03-architecture/08-physical-gpu-integration.md`](../docs/03-architecture/08-physical-gpu-integration.md) — canonical: mechanism choice and device bring-up. * [`../docs/03-architecture/11-physical-backend-guest-wiring.md`](../docs/03-architecture/11-physical-backend-guest-wiring.md) — canonical: the guest-wiring design. -* [`Execution-Frontier.md`](Execution-Frontier.md), [`Execution-Plane.md`](Execution-Plane.md) — the GR wall and the software execution plane. +* [`Execution-Frontier.md`](Execution-Frontier.md), [`Execution-Plane.md`](Execution-Plane.md) — the execution frontier and the software execution plane. * [`Host-Module.md`](Host-Module.md) — the module that owns both planes. * [`Introduction-to-GMMU.md`](Introduction-to-GMMU.md) — page-table seams and scratch mappings. * [`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md), [`Project-Status.md`](Project-Status.md) — support scope and the honest status picture. diff --git a/wiki/Project-Status.md b/wiki/Project-Status.md index d6e60d1..e63a198 100644 --- a/wiki/Project-Status.md +++ b/wiki/Project-Status.md @@ -10,7 +10,7 @@ > (the execution frontier). When this page and the docs disagree, the docs tree > wins. -Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` proven with the +Status: 2026-10-20, tree `master @ 16291c72`. Tags: `[live]` proven with the stock guest driver; `[gated]` implemented, default-off or behind supervised gates; `[wip]` partially implemented; `[wall]` current blocker; `[target]` named success state; `[planned]` designed, not built. @@ -24,9 +24,9 @@ named success state; `[planned]` designed, not built. | GSP-RM message queues and RPC transport | `[live]` working | | Stock guest driver probe and `nvidia-smi` (TU104) | `[live]` working | | CUDA userspace init and the driver-API smoke (`cuInit`, device enumeration, `cuCtxCreate`, allocation, copies, free, destroy) | `[live]` working — the smoke prints `ALL OK` and the 64-byte payload round-trips exactly | -| Guest CUDA kernel execution on real hardware | `[wall]` a forwarded guest launch reaches the engine queue (the front end deposits the methods and the scheduling path accepts them), but the engine does not yet execute the forwarded launch descriptor — an engine-authored release has not been observed. Emulation-mode kernel runs (executed through the host bridge on a no-claim path) have completed green; reproducibility of the green path across boots is an open issue | -| Physical backend on real hardware (module-owned GPU running its real GSP-RM) | `[live]` boot, RPC session, object creation and copy-engine work proven; GR execution is the same frontier as above | -| Multi-tenant isolation (mdev per VM, per-context page directories) | `[wip]` designed; the baseline service-state ownership work is in progress | +| Guest CUDA kernel execution on real hardware | `[live]` working — verified green on fresh boots: the vector-add (`VADD n=32768 bad=0`), matrix-multiply (`MATMUL n=256 bad=0`) and transformer (`dh=256 seq=64 bad=0`, `FFN4 layers=4 bad=0`) clients print `STATUS OK`, engine-authored completions relay to the guest and its synchronisation waits resolve. Remaining: the large-buffer staged-transfer class is under repair (some large multi-chunk clients still mis-verify); fresh-boot re-measurement is in progress after the boot-poisoning fix; per-guest execution isolation for simultaneous CUDA guests is in progress | +| Physical backend on real hardware (module-owned GPU running its real GSP-RM) | `[live]` boot, RPC session, object creation, copy-engine work and GR execution proven — the verified guest CUDA kernel clients execute through this backend | +| Multi-tenant isolation (mdev per VM, per-context page directories) | `[wip]` device-level coexistence works — multiple mediated devices with different emulated VRAM sizes and multiple guests coexist; per-guest execution isolation for simultaneous CUDA guests is in progress | | OpenGL / Vulkan (3D), video | `[planned]` | | Windows guests | `[planned]` (target) | | Xen / Argo / OpenXT / Qubes Air transports | `[planned]` (the emulation core is hypervisor-independent by construction) | @@ -43,23 +43,32 @@ The project's definition of done for CUDA requires all of the following 2. Context creation and the full driver-API surface (`cuMemAlloc`, copies, free, destroy) — done. 3. A guest kernel launch that **executes on real hardware** and is proven by an - engine-authored completion — the current frontier. + engine-authored completion — demonstrated: the verified clients execute on + the physical engine and the engine-authored completions relay to the guest. 4. Memory the guest's work refers to resolving correctly end to end (the VA↔PA model) — partially, for the paths exercised so far. -5. Correct completion, semaphore and fault semantics back to the guest — CE - class work has the relay; GR is open. -6. Reproducibility: every boot, not a lucky one — an open issue. -7. No ad-hoc default-off parameters required for the canonical path. +5. Correct completion, semaphore and fault semantics back to the guest — the + GR completion path now relays (engine-authored releases wake the guest's + synchronisation waits); fault semantics remain partial. +6. Reproducibility: every boot, not a lucky one — root-caused and fixed (the + boot-poisoning defect: the wrong status-queue consumption cursor could + desynchronise the reply ring under load and poison the boot); + re-measurement across fresh boots is in progress. +7. No ad-hoc default-off parameters required for the canonical path — still + open. -Until every gate holds, the honest formulation is: **initialization and the -driver-API path work; engine execution is in progress.** +Until every gate holds, the honest formulation is: **initialization, the +driver-API path and kernel execution on the physical GPU work and verify; +hardening (large-buffer transfers, boot-to-boot reproducibility) and +per-guest isolation remain.** ## 3. How to see it yourself The reference setup takes about half an hour from a fresh host ([`Install-and-First-Run.md`](Install-and-First-Run.md)); the current expected outcome is exactly the ladder above — driver probe, `nvidia-smi`, the full -driver-API smoke, and the execution frontier where it stands. The +driver-API smoke, and guest CUDA kernels executing and verifying on the +physical GPU. The risc-v-gsp-rm-debugger workbench ([`Riscv-GSP-RM-Debugger.md`](Riscv-GSP-RM-Debugger.md)) is the host-side tooling used to study the firmware side of that frontier. @@ -68,7 +77,7 @@ tooling used to study the firmware side of that frontier. Every claim above carries a date and a tag, maps to a maintained document in the repository, and is refreshed as milestones land. The execution story, with -its evidence and its open theories, is on +its evidence, its resolved theories and its open work, is on [`Execution-Frontier.md`](Execution-Frontier.md). ## 5. Related pages |
