summaryrefslogtreecommitdiffstats
path: root/wiki/Physical-GPU-Backend.md
diff options
context:
space:
mode:
authorArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-20 05:22:19 -0400
committerArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-20 05:22:19 -0400
commitae534205925529fc06e0539218f0ddf9fe5f835a (patch)
treecb9451984c72e995c2bb4ce85d4ad4ad162fc29d /wiki/Physical-GPU-Backend.md
parent16291c723caf6693832467fe5fc023a1c9923888 (diff)
downloadopen-iov-ae534205925529fc06e0539218f0ddf9fe5f835a.tar.gz
open-iov-ae534205925529fc06e0539218f0ddf9fe5f835a.zip
site: 2026-10-20 status refresh - guest CUDA kernel execution working
Diffstat (limited to 'wiki/Physical-GPU-Backend.md')
-rw-r--r--wiki/Physical-GPU-Backend.md37
1 files changed, 23 insertions, 14 deletions
diff --git a/wiki/Physical-GPU-Backend.md b/wiki/Physical-GPU-Backend.md
index 1b83203..bbca084 100644
--- a/wiki/Physical-GPU-Backend.md
+++ b/wiki/Physical-GPU-Backend.md
@@ -5,7 +5,7 @@
> guest-facing emulation. Canonical sources: [`../docs/03-architecture/08-physical-gpu-integration.md`](../docs/03-architecture/08-physical-gpu-integration.md)
> and [`../docs/03-architecture/11-physical-backend-guest-wiring.md`](../docs/03-architecture/11-physical-backend-guest-wiring.md); the docs tree wins on disagreement.
-Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` proven with the
+Status: 2026-10-20, tree `master @ 16291c72`. Tags: `[live]` proven with the
stock guest driver; `[gated]` implemented default-off; `[wip]` partially
built; `[wall]` current blocker; `[target]` planned end state.
@@ -64,13 +64,15 @@ independently verified:
5. **Guest mediation** — the guest-facing emulation stays unchanged; real-resource
objects translate to GSP-RM requests, and server-owned ranges populate the real MMU. `[wip]`
6. **Work path** — decoded guest pushes become real push streams; resolved host pages
- map through module-built scratch page tables; the firmware schedules the work. `[gated]`
+ map through module-built scratch page tables; the firmware schedules the work.
+ `[live]` observed end to end for the verified guest kernel clients.
7. **Completion relay** — hardware-completed semaphores land in guest-polled memory;
- events and faults relay through the existing MSI-X injection path. `[gated]`
+ events and faults relay through the existing MSI-X injection path.
+ `[live]` engine-authored completions relay to the guest and its waits resolve.
## 5. The two planes and the wiring
-The **emulation plane** serves the guest: BAR shadows, the sysmem msgq protocol, a hand-built TU104 static-configuration model, and a software work pipeline that executes decoded memsets, copies and semaphore releases. The **physical backend plane** owns the device, boots the real GSP-RM and runs the RM client ladder. The wiring joins them at the lifecycle seam (mdev create/remove), a static-config identity overlay, a read-only control whitelist, and — for eligible copy-engine work — the scratch-mapped forwarding path, all default-off and fail-open to software.
+The **emulation plane** serves the guest: BAR shadows, the sysmem msgq protocol, a hand-built TU104 static-configuration model, and a software work pipeline that executes decoded memsets, copies and semaphore releases. The **physical backend plane** owns the device, boots the real GSP-RM and runs the RM client ladder. The wiring joins them at the lifecycle seam (mdev create/remove), a static-config identity overlay, a read-only control whitelist, and — for eligible work — the scratch-mapped forwarding path, parameterised and fail-open to software.
```text
GUEST (QEMU) HOST: vgsp.ko (module-owned GPU) PHYSICAL GPU
@@ -85,8 +87,9 @@ INIT preflight: distribution driver green; no vgsp/mdev/QEMU [live]
RM battery, UVM registration [live]
WORK guest CUDA app -> libcuda -> nvidia.ko -> GPFIFO -> doorbell
host pump decodes; eligible CE copies forwarded via module-built scratch
- page tables; GR: launch reaches the front-end, descriptor not seen [wall]
-DONE engine authors the release in guest memory; host relays via MSI-X [gated]
+ page tables; GR: launches execute on the engine, completions relay
+ to the guest [live]
+DONE engine authors the release in guest memory; host relays via MSI-X [live]
TEARDOWN mdev remove -> QEMU stop -> module unload -> clean GSP-RM shutdown
-> protected region down -> distribution driver rebound [live]
FAIL a failed mutating operation parks the device in a locked fail-safe;
@@ -104,17 +107,20 @@ FAIL a failed mutating operation parks the device in a locked fail-safe;
| Power domain held active across the ownership window | verified across supervised cycles `[live]` |
| Real GSP boot by the module, through init-done and clean release | done, no reboot on clean cycles `[live]` |
| Host RM client and first real engine work (static config, real objects, 4 KiB CE copy) | done `[live]` |
-| Guest-copy forwarding, deferred release, hardware-primary copies, engine-authored relay, fault relay | default-off, verified live; one relayed-copy mismatch under triage `[gated]` |
-| CUDA kernel execution (`vectorAdd`) | open `[wall]` — the launch-descriptor install path; correct results are the `[target]` |
+| Guest-copy forwarding, deferred release, hardware-primary copies, engine-authored relay, fault relay | verified live (seams parameterised and fail-open); the large-buffer staged-transfer class is under repair `[live]` |
+| CUDA kernel execution (`vectorAdd` and further clients) | done `[live]` — stock guest CUDA clients (vector-add, matrix-multiply, transformer) execute on the physical engine with verified results and engine-authored completions relayed to the guest |
## 7. The execution frontier
The RM client ladder reaches the compute engine (a GR0 channel plus a
-compute-class object). A forwarded guest CUDA launch now
-reaches the front-end (ring entry fetched, methods executed) but the inline
-launch-descriptor install has not been observed and no grid has dispatched;
-context validity at dispatch is ruled out. The wall is the form of the
-forwarded launch method stream — see [`Execution-Frontier.md`](Execution-Frontier.md).
+compute-class object). Forwarded guest CUDA launches now execute on the
+engine: the inline launch descriptor installs and is read, the grid
+dispatches, the engine authors the completion, the host relays it, and the
+guest's synchronisation wait resolves — verified on fresh boots with the
+vector-add, matrix-multiply and transformer clients. The leading edge is the
+large-buffer staged-transfer class (under repair), reproducibility across
+fresh boots, and per-guest execution isolation — see
+[`Execution-Frontier.md`](Execution-Frontier.md).
A failed mutating operation on the backend parks the device in a locked
fail-safe that is never auto-retried; recovery is a supervised step.
@@ -147,6 +153,9 @@ fail-safe that is never auto-retried; recovery is a supervised step.
access to guest memory stays behind the hypervisor's per-device DMA
interface, and tenant policy is a later layer on top
([`GMMU-Multi-Tenant-Protection.md`](GMMU-Multi-Tenant-Protection.md)).
+* Multiple mediated devices with different emulated VRAM sizes and multiple
+ guests already coexist at the device level; per-guest execution isolation
+ for simultaneous CUDA guests is in progress.
## 10. Open questions
@@ -159,7 +168,7 @@ fail-safe that is never auto-retried; recovery is a supervised step.
* [`../docs/03-architecture/08-physical-gpu-integration.md`](../docs/03-architecture/08-physical-gpu-integration.md) — canonical: mechanism choice and device bring-up.
* [`../docs/03-architecture/11-physical-backend-guest-wiring.md`](../docs/03-architecture/11-physical-backend-guest-wiring.md) — canonical: the guest-wiring design.
-* [`Execution-Frontier.md`](Execution-Frontier.md), [`Execution-Plane.md`](Execution-Plane.md) — the GR wall and the software execution plane.
+* [`Execution-Frontier.md`](Execution-Frontier.md), [`Execution-Plane.md`](Execution-Plane.md) — the execution frontier and the software execution plane.
* [`Host-Module.md`](Host-Module.md) — the module that owns both planes.
* [`Introduction-to-GMMU.md`](Introduction-to-GMMU.md) — page-table seams and scratch mappings.
* [`Hardware-Software-Matrix.md`](Hardware-Software-Matrix.md), [`Project-Status.md`](Project-Status.md) — support scope and the honest status picture.