summaryrefslogtreecommitdiffstats
path: root/wiki
diff options
context:
space:
mode:
authorArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-20 05:22:19 -0400
committerArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-20 05:22:19 -0400
commitae534205925529fc06e0539218f0ddf9fe5f835a (patch)
treecb9451984c72e995c2bb4ce85d4ad4ad162fc29d /wiki
parent16291c723caf6693832467fe5fc023a1c9923888 (diff)
downloadopen-iov-ae534205925529fc06e0539218f0ddf9fe5f835a.tar.gz
open-iov-ae534205925529fc06e0539218f0ddf9fe5f835a.zip
site: 2026-10-20 status refresh - guest CUDA kernel execution working
Diffstat (limited to 'wiki')
-rw-r--r--wiki/.publish-manifest.json46
-rw-r--r--wiki/CUDA-Userspace-Driver.md32
-rw-r--r--wiki/Execution-Frontier.md177
-rw-r--r--wiki/Physical-GPU-Backend.md37
-rw-r--r--wiki/Project-Status.md35
5 files changed, 181 insertions, 146 deletions
diff --git a/wiki/.publish-manifest.json b/wiki/.publish-manifest.json
index 20e9722..4fe571b 100644
--- a/wiki/.publish-manifest.json
+++ b/wiki/.publish-manifest.json
@@ -19,11 +19,11 @@
},
"CUDA Userspace Driver": {
"file": "CUDA-Userspace-Driver.wiki",
- "new_revid": 23963,
+ "new_revid": 23998,
"old_revid": 23963,
- "source_sha": "31d77a3a672b2943",
- "summary": "Fix repository links: point to the cgit tree URLs",
- "timestamp": "2026-10-12T04:40:06Z",
+ "source_sha": "d467dbcd9f323768",
+ "summary": "Status refresh 2026-10-20: guest CUDA kernels execute on the physical GPU (vector-add/matmul/transformer green); boot-poisoning defect fixed",
+ "timestamp": "2026-10-20T09:19:03Z",
"title": "CUDA Userspace Driver"
},
"Community": {
@@ -163,11 +163,11 @@
},
"Main Page": {
"file": "Main-Page.wiki",
- "new_revid": 23972,
- "old_revid": 23960,
- "source_sha": "1b53ef37afca940e",
- "summary": "Fix repository links to the cgit tree/repo URLs",
- "timestamp": "2026-10-12T04:40:23Z",
+ "new_revid": 24000,
+ "old_revid": 23972,
+ "source_sha": "2e8171cab56c394c",
+ "summary": "Status table refresh 2026-10-20: guest CUDA kernel execution working",
+ "timestamp": "2026-10-20T09:19:09Z",
"title": "Main Page"
},
"MediaWiki:Sidebar": {
@@ -199,20 +199,20 @@
},
"Physical GPU Backend and Guest Wiring": {
"file": "Physical-GPU-Backend.wiki",
- "new_revid": 23985,
- "old_revid": 23951,
- "source_sha": "4bff4dbfa8b4644e",
- "summary": "Fix repository links: point to the cgit tree URLs",
- "timestamp": "2026-10-12T04:40:59Z",
+ "new_revid": 23999,
+ "old_revid": 23985,
+ "source_sha": "d762abd50cef636d",
+ "summary": "Status refresh 2026-10-20: guest CUDA kernels execute on the physical GPU (vector-add/matmul/transformer green); boot-poisoning defect fixed",
+ "timestamp": "2026-10-20T09:19:05Z",
"title": "Physical GPU Backend and Guest Wiring"
},
"Project Status": {
"file": "Project-Status.wiki",
- "new_revid": 23986,
- "old_revid": 23952,
- "source_sha": "6d07f2a0c6f524a9",
- "summary": "Fix repository links: point to the cgit tree URLs",
- "timestamp": "2026-10-12T04:41:00Z",
+ "new_revid": 23997,
+ "old_revid": 23986,
+ "source_sha": "cb714adeebcfdc33",
+ "summary": "Status refresh 2026-10-20: guest CUDA kernels execute on the physical GPU (vector-add/matmul/transformer green); boot-poisoning defect fixed",
+ "timestamp": "2026-10-20T09:19:01Z",
"title": "Project Status"
},
"RPC Message Format": {
@@ -262,11 +262,11 @@
},
"The Execution Frontier": {
"file": "Execution-Frontier.wiki",
- "new_revid": 23966,
+ "new_revid": 23996,
"old_revid": 23966,
- "source_sha": "b26ce9bb6fe63f6a",
- "summary": "Fix repository links: point to the cgit tree URLs",
- "timestamp": "2026-10-12T04:40:07Z",
+ "source_sha": "8c0007ec287387b9",
+ "summary": "Status refresh 2026-10-20: guest CUDA kernels execute on the physical GPU (vector-add/matmul/transformer green); boot-poisoning defect fixed",
+ "timestamp": "2026-10-20T09:19:00Z",
"title": "The Execution Frontier"
},
"The Guest Stack and CUDASW": {
diff --git a/wiki/CUDA-Userspace-Driver.md b/wiki/CUDA-Userspace-Driver.md
index 9a041ef..6975a1d 100644
--- a/wiki/CUDA-Userspace-Driver.md
+++ b/wiki/CUDA-Userspace-Driver.md
@@ -1,16 +1,20 @@
# CUDA Userspace Driver (`libcuda.so`) — Analysis Deep Dive
-> **Status (2026-10-12, honest).** The analysis covers the whole library for
+> **Status (2026-10-20, honest).** The analysis covers the whole library for
> driver 570.144 and the tooling is version-agnostic. The *control plane* gets
> the stock guest stack through `cuInit`, device enumeration, `cuCtxCreate`
> and the full driver-API smoke test — allocation, copies, free and destroy
> complete end to end (`ALL OK`; the 64-byte payload round-trips exactly).
> The *data plane* decodes guest push buffers and executes the memory and
> semaphore work it understands; CUDA kernel launches are forwarded to the
-> physical execution path, where a launch reaches the engine queue
-> (front-end deposit and scheduling are observed) but an engine-authored
-> release is not yet observed. Green runs are not yet reproducible on every
-> boot — an open issue, tracked in the repository. Canonical detail:
+> physical execution path and now execute on the physical GPU — the
+> engine-authored completions relay to the guest and its synchronisation
+> waits resolve. The vector-add (`VADD n=32768 bad=0`), matrix-multiply
+> (`MATMUL n=256 bad=0`) and transformer (`dh=256 seq=64 bad=0`,
+> `FFN4 layers=4 bad=0`) clients verify green on fresh boots. The
+> large-buffer staged-transfer class (multi-megabyte transfers between guest
+> memory and device memory) is under repair, and reproducibility across fresh
+> boots is being re-measured after the boot-poisoning fix. Canonical detail:
> [`01-architecture.md`](../userspace/cuda/docs/01-architecture.md)
> through `08`; reading order
> [`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md);
@@ -172,19 +176,19 @@ the data plane (`docs/08`).
emulation and object/control replies are implemented; the stock guest stack
completes `cuInit`, device enumeration, `cuCtxCreate` and the driver-API
smoke test (allocation, copies, free and destroy).
-* **Current frontier:** a guest CUDA kernel launch is forwarded and reaches
- the physical engine queue — the front end deposits the launch and the
- scheduling path accepts it — but the engine does not yet execute the
- forwarded launch descriptor: an engine-authored release is the only proof
- of execution and it is not observed. The per-boot green/stall variance on
- the emulation path is an open issue, tracked in the repository. The chain,
- the evidence and the live state are on
+* **Current frontier:** guest CUDA kernel launches execute on the physical
+ GPU with engine-authored completions relayed to the guest; the verified
+ clients are green on fresh boots. The leading edge is the large-buffer
+ staged-transfer class (some large multi-chunk clients still mis-verify),
+ re-measurement of reproducibility across fresh boots after the
+ boot-poisoning fix, and per-guest execution isolation. The chain, the
+ evidence and the live state are on
[`Execution-Frontier.md`](Execution-Frontier.md).
* **Data plane today:** the host module decodes guest push buffers as method
streams and executes the memory and semaphore work it understands
(memsets, copies, semaphore releases), and forwards engine-class work to
- the physical backend; kernel launches are forwarded but not yet executed
- by the engine.
+ the physical backend; kernel launches now execute on the engine through
+ the forwarded path, with completions relayed back to the guest.
## 5. The crash-path lesson: replies are consumed data
diff --git a/wiki/Execution-Frontier.md b/wiki/Execution-Frontier.md
index fe21fda..2d9d80c 100644
--- a/wiki/Execution-Frontier.md
+++ b/wiki/Execution-Frontier.md
@@ -1,15 +1,16 @@
# The Execution Frontier
-> **What this page is.** The current leading edge of guest-facing execution:
-> the chain that must fire for a stock guest CUDA kernel to run on the
-> physical GPU, what has been proven about it, where it stops, how we know,
-> and the open lines of investigation and theories. It is evidence-led: every
-> statement carries a live observation, a decompilation finding, or a clearly
-> labelled theory. Condensed from
+> **What this page is.** The state of guest-facing execution: the chain that
+> must fire for a stock guest CUDA kernel to run on the physical GPU, what has
+> been proven about it, how we know, and where the leading edge is now. The
+> chain fires end to end: verified guest CUDA clients execute on the physical
+> GPU and wake up correctly; the leading edge is now breadth and hardening. It
+> is evidence-led: every statement carries a live observation, a
+> decompilation finding, or a clearly labelled theory. Condensed from
> [`../docs/04-implementation/03-execution-frontier.md`](../docs/04-implementation/03-execution-frontier.md);
> the docs tree wins on disagreement.
-Status: 2026-10-12 (source page dated 2026-10-06), tree `master @ 2f066727`.
+Status: 2026-10-20, tree `master @ 16291c72`.
Tags: `[live]` observed; `[wip]` partial; `[wall]` current blocker; theories
are labelled as such.
@@ -45,46 +46,62 @@ guest: builds launch + method stream
| 3 | Host decode + forward stream + QMD prewrite + ring/USERD | `[live]` observed |
| 4 | Front-end fetches the ring entry | `[live]` observed (ring cursor advances) |
| 5 | Front-end executes methods | `[live]` observed (object bind, cache ops, report semaphore) |
-| 6 | **Inline-QMD install** (descriptor materialised) | `[wall]` not observed |
-| 7 | **Engine reads the QMD** | `[wall]` not observed |
+| 6 | **Inline-QMD install** (descriptor materialised) | `[live]` observed |
+| 7 | **Engine reads the QMD** | `[live]` observed |
| 8 | Context switch loads the channel context | `[live]` context validity at dispatch is ruled in |
-| 9 | GR dispatches the grid to the SMs | never observed on this stack |
-| 10 | Engine authors release + channel semaphore | absent for forwarded launches |
-| 11 | Host relays completion; guest wakes | the guest waits and times out |
+| 9 | GR dispatches the grid to the SMs | `[live]` observed (the guest's results verify) |
+| 10 | Engine authors release + channel semaphore | `[live]` engine-authored releases relay to the guest |
+| 11 | Host relays completion; guest wakes | `[live]` the guest's synchronisation waits resolve |
-**One-line summary:** the guest, the emulation, the forwarding path and the
-front-end are live; the **launch-descriptor install path** (stages 6–7) is the
-leading edge, and with it stages 9–11.
+**One-line summary:** the whole chain is live; the leading edge is now breadth
+and hardening — the large-buffer staged-transfer class, reproducibility across
+boots, and per-guest execution isolation.
## 2. Where the leading edge is
-A stock guest program now reaches its kernel launch and submits it through the
-forwarding path. On the wire the ring entry is consumed and the front-end
-methods run — but the launch descriptor is never materialised, the engine
-never reads a QMD, no engine-authored completion appears, and the guest blocks
-in its synchronisation wait.
-
-This is **not** a context-validity wall (ruled out at dispatch) and **not** a
-window-mapping wall for the descriptor destination (that install is accepted).
-The wall is the **form of the launch method stream**: what the front-end
-requires in order to install the inline descriptor, and what the engine
-requires in order to read and dispatch it.
+A stock, unmodified guest CUDA program executes its kernels on the physical
+GPU and verifies its results. On fresh boots the vector-add client
+(`victory_mail`) prints `VADD n=32768 bad=0` + `STATUS OK`, the
+matrix-multiply client (`victory_matmul`) prints `MATMUL n=256 bad=0` +
+`STATUS OK`, and a transformer kernel prints `dh=256 seq=64 bad=0`,
+`FFN4 layers=4 bad=0` + `STATUS OK`. The executing engine is the physical GPU
+under the module's own real GSP-RM session; the host CUDA bridge is not part
+of this path. Engine-authored completions relay to the guest and its
+synchronisation waits resolve.
+
+The leading edge has moved from the launch chain itself — it fires — to
+breadth and hardening around it:
+
+* **The large-buffer staged-transfer class.** Multi-megabyte staged transfers
+ between guest memory and device memory are under repair: some large
+ multi-chunk clients still mis-verify.
+* **Reproducibility across boots.** A boot-poisoning defect was root-caused
+ and fixed: under the driver's status-queue mode the emulation read the wrong
+ queue cursor, so under load the reply ring could desynchronise and the first
+ CUDA client would fail (the user-visible `STATUS FAIL`-then-freeze report).
+ Re-measurement across fresh boots is in progress.
+* **Per-guest execution isolation.** Multiple mediated devices with different
+ emulated VRAM sizes and multiple guests already coexist at the device level;
+ per-guest execution isolation for multiple simultaneous CUDA guests is in
+ progress.
+
+No breadth beyond these is claimed: OpenGL/Vulkan, Windows guests, migration
+and performance numbers remain planned.
## 3. How we know (observables and their limits)
| Observable | What it proves | What it does not prove |
|---|---|---|
-| Ring entry consumed; ring cursor advances | the front-end is alive and fetching | nothing about the engine |
-| Method execution on the ladder channel | the front-end can execute the method vocabulary | **not** engine execution — these signals can be front-end-authored |
-| Descriptor buffer prewritten with a detectable sentinel | whether the descriptor was materialised after submission | a read does not modify memory; the sentinel detects installs, not reads |
-| Install sentinel unchanged after submit | the inline-install method did not run | it does not say why |
-| Engine-authored release word absent; poll times out | the engine did not complete the launch | it does not say whether the engine started |
+| Client verification output (`bad=0` + `STATUS OK`) | the guest executed its kernel and verified its results | by itself, which engine produced the results |
+| Engine-authored release relayed to the guest | the engine executed the launch and the relay works | nothing about other launches |
+| The guest's synchronisation wait resolving | completion reached the guest through the relay | nothing about other launches |
+| Ring entry consumed; ring cursor advances | the front-end is alive and fetching | **not** engine execution — front-end signals are necessary but not sufficient |
+| Reproduction across fresh boots | the path is not a lucky boot | no "every boot" claim yet while re-measurement is in progress |
| No GR fault recorded | the engine did not raise this error | silence is not health: a skipped launch is silent by construction |
-| Channel instance context flag set | the channel is context-valid at dispatch | nothing about the descriptor path |
-**Consequence:** progress on this frontier must be measured by an
-**engine-authored** signal (the release word advancing) — not by front-end
-signals, which we have already learned to distinguish.
+**Consequence:** acceptance is verified guest results plus the engine-authored
+release; a green run counts as progress only when it reproduces across fresh
+boots.
## 4. What is exonerated (do not relitigate)
@@ -98,68 +115,64 @@ signals, which we have already learned to distinguish.
already initialised in the same cycle; not the dispatch delta.
* Parenting/TSG for the fetch — both variants fetch identically.
* The copy-engine path — CE-class work, releases and fault relay are live
- (default off); the GR path is a separate problem.
+ (default off); the GR path is separate.
+* The descriptor-gate line of investigation (the old T1–T4 theories) — the
+ inline-descriptor install, the engine read, the dispatch, the
+ engine-authored release and the guest wake-up are all observed; closed by
+ observation.
+* The old boot-poisoning defect — root-caused to the status-queue consumption
+ cursor and fixed.
## 5. The clear issues (ranked)
-**Issue A — the forwarded stream carries a verbatim guest prelude that cannot
-complete.** The forwarded stream includes a verbatim copy of the guest's
-prelude whose inline-data writes target addresses not present in the engine's
-view of the channel's page tables, and the referenced data is not relocated.
-The shorter, proven stream shape fires its tail (channel semaphore) but still
-does not schedule the descriptor.
+**Issue A — the large-buffer staged-transfer class.** Multi-megabyte staged
+transfers between guest memory and device memory are under repair: some large
+multi-chunk clients still mis-verify.
-**Issue B — the descriptor-virtualization control method is absent.** The
-method that arms the inline-descriptor install path is not present in the
-forwarded stream. Fixing it should materialise the descriptor — falsifiable in
-one cycle by watching the sentinel and the engine-read signal.
+**Issue B — reproducibility across boots (re-measurement).** The
+boot-poisoning defect (wrong status-queue consumption cursor under the
+driver's status-queue mode; the reply ring could desynchronise under load, the
+first client would fail) was root-caused and fixed. Re-measurement across
+fresh boots is in progress; no "every boot" claim is made until it completes.
-**Issue C — diagnostic hygiene.** The sentinel placed at word 0 overlaps real
-descriptor fields; it must be moved to an unused word so diagnostics do not
-perturb what they measure.
+**Issue C — per-guest execution isolation.** Multiple mediated devices with
+different emulated VRAM sizes and multiple guests coexist at the device level;
+per-guest execution isolation for simultaneous CUDA guests is in progress.
## 6. Lines of investigation (live)
-1. **Restore the proven launch shape** as the baseline and **re-add the
- virtualization control method** — one variable at a time, with the install
- sentinel and the engine-read signal as the readout.
-2. **Relocate or drop the verbatim prelude** so inline-data writes target
- addresses present in the engine's view, or replace it with the host-side
- staging the proven shape already uses.
-3. **Move the sentinel** and re-run to confirm the readout is clean.
-4. **Diff against a known-good reference** — our own ladder dispatching a
- descriptor built from the same source material, or a captured working
- launch of the stock stack on the same GPU — converting the remaining work
- from inference to differential.
-5. **Offline probing** of any rule the front-end or engine is believed to
- require, using the firmware debugger before spending a supervised cycle
- ([`Riscv-GSP-RM-Debugger.md`](Riscv-GSP-RM-Debugger.md)).
-
-## 7. Theories (labelled; not yet proven)
-
-* **T1 — the virtualization control is the gate.** The front-end will not
- materialise an inline descriptor until the control method enables the
- install path.
-* **T2 — materialise-then-read.** The engine only schedules a descriptor that
- exists in its own view; an install targeting unmapped destinations aborts
- silently rather than faulting.
-* **T3 — the verbatim prelude poisons the launch.** Its refused writes leave
- the launch path unable to schedule the descriptor.
-* **T4 — relocation is the missing transform.** The guest's descriptor is
- valid for the guest's view of memory; the forwarding path must relocate
- referenced data into the channel's view before install.
-
-Each theory has a one-cycle falsifier: the install sentinel, the engine-read
-signal, and the engine-authored release, in that order.
+1. **Repair the large-buffer staged-transfer class** — the three green clients
+ stay the regression battery and must remain green throughout.
+2. **Complete the fresh-boot re-measurement** — demonstrate the fixed boot
+ path across fresh boots before claiming boot-to-boot reproducibility.
+3. **Per-guest execution isolation** — build from the device-level coexistence
+ to isolation for simultaneous CUDA guests.
+4. **Hold the acceptance battery** — vector-add, matrix-multiply and the
+ transformer kernel; each entry is accepted only on its own verification
+ output (`bad=0` + `STATUS OK`).
+
+## 7. Theories, resolved
+
+The descriptor-gate theories (T1–T4) are resolved and are not kept as open:
+the inline-descriptor install, the engine's QMD read, the grid dispatch, the
+engine-authored release and the guest wake-up are all directly observed, so
+the launch chain they questioned is proven end to end. The remaining work is
+engineering (large-buffer transfers, boot-to-boot reproducibility, per-guest
+isolation), each with an acceptance signal named before the work.
## 8. Claim discipline
-* **Engine-authored release is the only proof of execution.** Front-end
- signals are necessary but not sufficient.
+* **Engine-authored release is the acceptance signal for execution** — now
+ observed end to end; the guest's own verification (`bad=0` + `STATUS OK`)
+ is the guest-side acceptance signal.
+* **Front-end signals are necessary but not sufficient.** Ring cursor and
+ front-end method execution can be front-end-authored; they never substitute
+ for the engine-authored release.
* **Silence is not health.** No fault recorded does not mean no fault
occurred.
* **Emulator verdicts are instruction/memory-level.** They bound behaviour,
never timing or device environment.
+* **A green run counts only when it reproduces across fresh boots.**
* **A wall is closed only by an acceptance signal**, named before the
experiment.
diff --git a/wiki/Physical-GPU-Backend.md b/wiki/Physical-GPU-Backend.md
index 1b83203..bbca084 100644
--- a/wiki/Physical-GPU-Backend.md
+++ b/wiki/Physical-GPU-Backend.md
@@ -5,7 +5,7 @@
> guest-facing emulation. Canonical sources: [`../docs/03-architecture/08-physical-gpu-integration.md`](../docs/03-architecture/08-physical-gpu-integration.md)
> and [`../docs/03-architecture/11-physical-backend-guest-wiring.md`](../docs/03-architecture/11-physical-backend-guest-wiring.md); the docs tree wins on disagreement.
-Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` proven with the
+Status: 2026-10-20, tree `master @ 16291c72`. Tags: `[live]` proven with the
stock guest driver; `[gated]` implemented default-off; `[wip]` partially
built; `[wall]` current blocker; `[target]` planned end state.
@@ -64,13 +64,15 @@ independently verified:
5. **Guest mediation** — the guest-facing emulation stays unchanged; real-resource
objects translate to GSP-RM requests, and server-owned ranges populate the real MMU. `[wip]`
6. **Work path** — decoded guest pushes become real push streams; resolved host pages
- map through module-built scratch page tables; the firmware schedules the work. `[gated]`
+ map through module-built scratch page tables; the firmware schedules the work.
+ `[live]` observed end to end for the verified guest kernel clients.
7. **Completion relay** — hardware-completed semaphores land in guest-polled memory;
- events and faults relay through the existing MSI-X injection path. `[gated]`
+ events and faults relay through the existing MSI-X injection path.
+ `[live]` engine-authored completions relay to the guest and its waits resolve.
## 5. The two planes and the wiring
-The **emulation plane** serves the guest: BAR shadows, the sysmem msgq protocol, a hand-built TU104 static-configuration model, and a software work pipeline that executes decoded memsets, copies and semaphore releases. The **physical backend plane** owns the device, boots the real GSP-RM and runs the RM client ladder. The wiring joins them at the lifecycle seam (mdev create/remove), a static-config identity overlay, a read-only control whitelist, and — for eligible copy-engine work — the scratch-mapped forwarding path, all default-off and fail-open to software.
+The **emulation plane** serves the guest: BAR shadows, the sysmem msgq protocol, a hand-built TU104 static-configuration model, and a software work pipeline that executes decoded memsets, copies and semaphore releases. The **physical backend plane** owns the device, boots the real GSP-RM and runs the RM client ladder. The wiring joins them at the lifecycle seam (mdev create/remove), a static-config identity overlay, a read-only control whitelist, and — for eligible work — the scratch-mapped forwarding path, parameterised and fail-open to software.
```text
GUEST (QEMU) HOST: vgsp.ko (module-owned GPU) PHYSICAL GPU
@@ -85,8 +87,9 @@ INIT preflight: distribution driver green; no vgsp/mdev/QEMU [live]
RM battery, UVM registration [live]
WORK guest CUDA app -> libcuda -> nvidia.ko -> GPFIFO -> doorbell
host pump decodes; eligible CE copies forwarded via module-built scratch
- page tables; GR: launch reaches the front-end, descriptor not seen [wall]
-DONE engine authors the release in guest memory; host relays via MSI-X [gated]
+ page tables; GR: launches execute on the engine, completions relay
+ to the guest [live]
+DONE engine authors the release in guest memory; host relays via MSI-X [live]
TEARDOWN mdev remove -> QEMU stop -> module unload -> clean GSP-RM shutdown
-> protected region down -> distribution driver rebound [live]
FAIL a failed mutating operation parks the device in a locked fail-safe;
@@ -104,17 +107,20 @@ FAIL a failed mutating operation parks the device in a locked fail-safe;
| Power domain held active across the ownership window | verified across supervised cycles `[live]` |
| Real GSP boot by the module, through init-done and clean release | done, no reboot on clean cycles `[live]` |
| Host RM client and first real engine work (static config, real objects, 4 KiB CE copy) | done `[live]` |
-| Guest-copy forwarding, deferred release, hardware-primary copies, engine-authored relay, fault relay | default-off, verified live; one relayed-copy mismatch under triage `[gated]` |
-| CUDA kernel execution (`vectorAdd`) | open `[wall]` — the launch-descriptor install path; correct results are the `[target]` |
+| Guest-copy forwarding, deferred release, hardware-primary copies, engine-authored relay, fault relay | verified live (seams parameterised and fail-open); the large-buffer staged-transfer class is under repair `[live]` |
+| CUDA kernel execution (`vectorAdd` and further clients) | done `[live]` — stock guest CUDA clients (vector-add, matrix-multiply, transformer) execute on the physical engine with verified results and engine-authored completions relayed to the guest |
## 7. The execution frontier
The RM client ladder reaches the compute engine (a GR0 channel plus a
-compute-class object). A forwarded guest CUDA launch now
-reaches the front-end (ring entry fetched, methods executed) but the inline
-launch-descriptor install has not been observed and no grid has dispatched;
-context validity at dispatch is ruled out. The wall is the form of the
-forwarded launch method stream — see [`Execution-Frontier.md`](Execution-Frontier.md).
+compute-class object). Forwarded guest CUDA launches now execute on the
+engine: the inline launch descriptor installs and is read, the grid
+dispatches, the engine authors the completion, the host relays it, and the
+guest's synchronisation wait resolves — verified on fresh boots with the
+vector-add, matrix-multiply and transformer clients. The leading edge is the
+large-buffer staged-transfer class (under repair), reproducibility across
+fresh boots, and per-guest execution isolation — see
+[`Execution-Frontier.md`](Execution-Frontier.md).
A failed mutating operation on the backend parks the device in a locked
fail-safe that is never auto-retried; recovery is a supervised step.
@@ -147,6 +153,9 @@ fail-safe that is never auto-retried; recovery is a supervised step.
access to guest memory stays behind the hypervisor's per-device DMA
interface, and tenant policy is a later layer on top
([`GMMU-Multi-Tenant-Protection.md`](GMMU-Multi-Tenant-Protection.md)).
+* Multiple mediated devices with different emulated VRAM sizes and multiple
+ guests already coexist at the device level; per-guest execution isolation
+ for simultaneous CUDA guests is in progress.
## 10. Open questions
@@ -159,7 +168,7 @@ fail-safe that is never auto-retried; recovery is a supervised step.
* [`../docs/03-architecture/08-physical-gpu-integration.md`](../docs/03-architecture/08-physical-gpu-integration.md) — canonical: mechanism choice and device bring-up.
* [`../docs/03-architecture/11-physical-backend-guest-wiring.md`](../docs/03-architecture/11-physical-backend-guest-wiring.md) — canonical: the guest-wiring design.
-* [`Execution-Frontier.md`](Execution-Frontier.md), [`Execution-Plane.md`](Execution-Plane.md) — the GR wall and the software execution plane.
+* [`Execution-Frontier.md`](Execution-Frontier.md), [`Execution-Plane.md`](Execution-Plane.md) — the execution frontier and the software execution plane.
* [`Host-Module.md`](Host-Module.md) — the module that owns both planes.
* [`Introduction-to-GMMU.md`](Introduction-to-GMMU.md) — page-table seams and scratch mappings.
* [`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md), [`Project-Status.md`](Project-Status.md) — support scope and the honest status picture.
diff --git a/wiki/Project-Status.md b/wiki/Project-Status.md
index d6e60d1..e63a198 100644
--- a/wiki/Project-Status.md
+++ b/wiki/Project-Status.md
@@ -10,7 +10,7 @@
> (the execution frontier). When this page and the docs disagree, the docs tree
> wins.
-Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` proven with the
+Status: 2026-10-20, tree `master @ 16291c72`. Tags: `[live]` proven with the
stock guest driver; `[gated]` implemented, default-off or behind supervised
gates; `[wip]` partially implemented; `[wall]` current blocker; `[target]`
named success state; `[planned]` designed, not built.
@@ -24,9 +24,9 @@ named success state; `[planned]` designed, not built.
| GSP-RM message queues and RPC transport | `[live]` working |
| Stock guest driver probe and `nvidia-smi` (TU104) | `[live]` working |
| CUDA userspace init and the driver-API smoke (`cuInit`, device enumeration, `cuCtxCreate`, allocation, copies, free, destroy) | `[live]` working — the smoke prints `ALL OK` and the 64-byte payload round-trips exactly |
-| Guest CUDA kernel execution on real hardware | `[wall]` a forwarded guest launch reaches the engine queue (the front end deposits the methods and the scheduling path accepts them), but the engine does not yet execute the forwarded launch descriptor — an engine-authored release has not been observed. Emulation-mode kernel runs (executed through the host bridge on a no-claim path) have completed green; reproducibility of the green path across boots is an open issue |
-| Physical backend on real hardware (module-owned GPU running its real GSP-RM) | `[live]` boot, RPC session, object creation and copy-engine work proven; GR execution is the same frontier as above |
-| Multi-tenant isolation (mdev per VM, per-context page directories) | `[wip]` designed; the baseline service-state ownership work is in progress |
+| Guest CUDA kernel execution on real hardware | `[live]` working — verified green on fresh boots: the vector-add (`VADD n=32768 bad=0`), matrix-multiply (`MATMUL n=256 bad=0`) and transformer (`dh=256 seq=64 bad=0`, `FFN4 layers=4 bad=0`) clients print `STATUS OK`, engine-authored completions relay to the guest and its synchronisation waits resolve. Remaining: the large-buffer staged-transfer class is under repair (some large multi-chunk clients still mis-verify); fresh-boot re-measurement is in progress after the boot-poisoning fix; per-guest execution isolation for simultaneous CUDA guests is in progress |
+| Physical backend on real hardware (module-owned GPU running its real GSP-RM) | `[live]` boot, RPC session, object creation, copy-engine work and GR execution proven — the verified guest CUDA kernel clients execute through this backend |
+| Multi-tenant isolation (mdev per VM, per-context page directories) | `[wip]` device-level coexistence works — multiple mediated devices with different emulated VRAM sizes and multiple guests coexist; per-guest execution isolation for simultaneous CUDA guests is in progress |
| OpenGL / Vulkan (3D), video | `[planned]` |
| Windows guests | `[planned]` (target) |
| Xen / Argo / OpenXT / Qubes Air transports | `[planned]` (the emulation core is hypervisor-independent by construction) |
@@ -43,23 +43,32 @@ The project's definition of done for CUDA requires all of the following
2. Context creation and the full driver-API surface (`cuMemAlloc`, copies,
free, destroy) — done.
3. A guest kernel launch that **executes on real hardware** and is proven by an
- engine-authored completion — the current frontier.
+ engine-authored completion — demonstrated: the verified clients execute on
+ the physical engine and the engine-authored completions relay to the guest.
4. Memory the guest's work refers to resolving correctly end to end (the VA↔PA
model) — partially, for the paths exercised so far.
-5. Correct completion, semaphore and fault semantics back to the guest — CE
- class work has the relay; GR is open.
-6. Reproducibility: every boot, not a lucky one — an open issue.
-7. No ad-hoc default-off parameters required for the canonical path.
+5. Correct completion, semaphore and fault semantics back to the guest — the
+ GR completion path now relays (engine-authored releases wake the guest's
+ synchronisation waits); fault semantics remain partial.
+6. Reproducibility: every boot, not a lucky one — root-caused and fixed (the
+ boot-poisoning defect: the wrong status-queue consumption cursor could
+ desynchronise the reply ring under load and poison the boot);
+ re-measurement across fresh boots is in progress.
+7. No ad-hoc default-off parameters required for the canonical path — still
+ open.
-Until every gate holds, the honest formulation is: **initialization and the
-driver-API path work; engine execution is in progress.**
+Until every gate holds, the honest formulation is: **initialization, the
+driver-API path and kernel execution on the physical GPU work and verify;
+hardening (large-buffer transfers, boot-to-boot reproducibility) and
+per-guest isolation remain.**
## 3. How to see it yourself
The reference setup takes about half an hour from a fresh host
([`Install-and-First-Run.md`](Install-and-First-Run.md)); the current expected
outcome is exactly the ladder above — driver probe, `nvidia-smi`, the full
-driver-API smoke, and the execution frontier where it stands. The
+driver-API smoke, and guest CUDA kernels executing and verifying on the
+physical GPU. The
risc-v-gsp-rm-debugger workbench
([`Riscv-GSP-RM-Debugger.md`](Riscv-GSP-RM-Debugger.md)) is the host-side
tooling used to study the firmware side of that frontier.
@@ -68,7 +77,7 @@ tooling used to study the firmware side of that frontier.
Every claim above carries a date and a tag, maps to a maintained document in
the repository, and is refreshed as milestones land. The execution story, with
-its evidence and its open theories, is on
+its evidence, its resolved theories and its open work, is on
[`Execution-Frontier.md`](Execution-Frontier.md).
## 5. Related pages