diff options
Diffstat (limited to 'STATE-CURRENT.md')
| -rw-r--r-- | STATE-CURRENT.md | 270 |
1 files changed, 198 insertions, 72 deletions
diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md index 2b04c0c..d0e44d8 100644 --- a/STATE-CURRENT.md +++ b/STATE-CURRENT.md @@ -1,13 +1,83 @@ # Open-IOV Current State -> **Snapshot: 2026-10-02 ~22:12.** This is a dated public status page, not live -> state; it is refreshed on every progress check. Live working notes are -> maintained outside this repository; the public known-knowns/unknowns document -> is authoritative here: +> **Snapshot: 2026-10-03 (reorganized; simulator status current 2026-10-03, +> physical campaign data as of 2026-10-02 ~22:12).** This is a dated public +> status page, not live state; it is refreshed on every progress check. Live +> working notes are maintained outside this repository; the public +> known-knowns/unknowns document is authoritative here: > [`docs/04-implementation/01-known-knowns-and-unknowns.md`](docs/04-implementation/01-known-knowns-and-unknowns.md). > The former 2026-09-21 snapshot (`STATE.md`) is retired; its content remains > in the git history. + +## Objective, status and blockers (this snapshot) + +**Objective.** Land the first green execution of an unmodified guest CUDA +program against the physical TU104 backend (a `vectorAdd`-class kernel whose +results verify), then the remaining open gates (multi-VM, 3D, Windows, +isolation, performance, portability). + +**Status.** + +- **Emulation front: GREEN.** The driver-API smoke completes end-to-end + (`ALL OK`, payload 64/64); the runtime client initializes with `rc=0` and + queues a kernel launch, whose execution requires the backend. +- **Physical backend: live on the real TU104.** Supervised full boot lifecycle + (FWSEC -> Booter Load -> RISC-V active -> RM init-done `0x1001`, + `rpc_result=0`), static-config RPC served, RM objects + a real channel + allocated, and a real copy-engine 4 KiB copy executed (`REAL COPY OK`, + semaphore released, destination equals source). Guest CE copies are + hardware-authored with engine-authored releases via the P6b/P6c forwarding + and relay paths (all parameters default off; one relayed copy's data + mismatch is under triage), and the host GR ladder allocates a GR0 channel + + compute object. +- **Guest frontier (2026-10-02 ~22:01).** The first real guest CUDA program + against the backend enumerated device properties (`cc=7.5 sm=48 + mem=134152192`) and then stalled in its first device allocation / UVM + channel path before any kernel executed; teardown was clean. +- **Simulator workstream:** see the section below (separate, development-only). + +**Blockers (current walls).** + +1. **GR context commit/execution.** The promote-side wall is root-caused + (the ladder VA space was externally owned, so the engine-context commit was + skipped and a promote VA was never booked). A default-off shared-management + address space gets past it, but the next RM call (`TURING_COMPUTE_A` + compute-object alloc) wedges the server; three further cycles closed the + client-promote shape space on the default external VAS with a soft + `NV_ERR_INVALID_STATE` (0x40) each. The V5 commit assert `0x52a8b04` is a + secondary failure. Next routes: name the soft-0x40 check, root-cause the + V5 first failure, or rehost the promote flow in risc-v-gsp-rm-debugger. +2. **First device allocation / UVM channel path in the guest** (the frontier + program's stall) - the first green guest execution is the immediate goal. +3. **Relay data-mismatch triage** (one relayed copy of 68 failed the host-side + data compare; handled fail-closed, parameter default-off). +4. **Known hazards, mitigated, never unattended:** P1 async FWSEC boot defect + (`staging not visible before kick`; fail-open, quarantined off the critical + path) and the D0/D3cold power-gating freeze hazard (default-latched locks, + exactly 5 power-domain locks per release). + +## Simulator workstream (risc-v-gsp-rm-debugger) + +Development-only tooling for simulating and inspecting GSP firmware runtime +memory; never shipped, never a guest dependency, no GPU/mdev/kernel-module +touch. Status 2026-10-03: + +- **S1 device model + boot ladder:** the real 570.144 image boots natively to + rung **`B2`** (relocation copy; 14,682 instructions, 0 unknown MMIO, 2-run + determinism), two rungs past the QEMU first fault; `B3/B4` (message loop) + are the targets. Docs: `debug/risc-v-gsp-rm-vm/docs/35` (design, Phase S/G) + and `36` (device model, `B0`-`B2`). +- **Evaluation harness:** `harness/sim_qemu_line.py` runs the simulator-vs-QEMU + A/B on frozen thresholds; verdict **`above`** on the S1 path (docs/37). +- **One-command triage:** `./risc-v-gsp-rm-debugger.sh triage` (or + `python3 harness/triage.py --firmware <new> --out dumps/triage`) chains + identity/probe/native-boot/A-B/version-diff into one report (docs/40). +- **Tests:** **290 collected on 2026-10-03** (`make test` in the tool + directory; audit baseline 155; docs/00 §Status, docs/27). +- **Docs index:** `debug/risc-v-gsp-rm-vm/docs/00-OVERVIEW.md` + README + document map (docs 30-40, one line each). + ## Mission Run **unmodified NVIDIA guest drivers** in VMs with full GPU functionality @@ -31,45 +101,7 @@ meant to execute on the physical GPU. | `cuCtxCreate` returns | ✅ (the whole smoke test now completes) | | `cuMemAlloc` + `cuMemcpy` roundtrip (`ALL OK` in the smoke test) | ✅ payload roundtrip verified 64/64 in the guest | | Full `nvidia-smi` output / kernel launch | 🟡 the static runtime now initializes end-to-end with **real releases** (`cudaGetDeviceCount`/`cudaSetDevice`/3×`cudaMalloc`/2×`cudaMemcpyHtoD`/`kernel launch`/`cudaGetLastError` all `rc=0`) and queues a kernel launch; execution awaits the backend | -| CUDA kernel execution (`vectorAdd`) | 🟡 execution backend: physical ownership + firmware staging verified; the unbind/rebind freeze hazard is mitigated by the supervised D0 power-domain lock (3/3 green claim cycles; the 2026-10-01 P1 mdev-owned release path initially ran outside that pre-lock window and re-armed D3cold, freezing the host at 22:32:53 - the G0 fix bundle now latches the D0/D3cold locks by default; see the frontier section), and the real FB size (8 GiB) and boot layout are verified live. The supervised live secure-verification run raised the protected region (WPR2 up) for the first time, then proved a secure-verification-only carve is not host-recoverable on this fused part, so the fail-safe held the device and an approved reboot restored it (distribution driver rebound, GPU healthy). That one-way stage is now retired: the only device-write path is a single full boot lifecycle (secure verification -> bootloader -> RISC-V start -> resource-manager initialization), whose code is implemented in the module sources and has now run live: the first real boot raised WPR2, passed Booter Load and went RISC-V active, and the module received the first real GSP-RM message (event 0x1002); init-done (0x1001) and a clean release are the next steps (see the 2026-10-01 section). A second live boot on 2026-10-01 proved the full boot sequence again (protected region up, bootloader halt, RISC-V active, first GSP-RM message) and reached the resource-manager's inline CPU-sequencer request; the host executor rejected the real request because of an over-strict internal size cap, which was found with a raw read of the message in host memory, fixed, and rebuilt. Cycles 2-3 then ran the real sequencer and received init-done with an RM init failure (`0x59`), which reproduces on a verified matching firmware image; the server's own shutdown supplied the last handoff bit, so each failed cycle - completed a **clean release without a reboot**. Cycles 4-10 then captured - the resource-manager's own error journal and root-caused the failure: the - boot queue was missing the two pre-boot initialization messages the matching - driver queues before the kernel kick (a system-information message and a - packed registry table). That handshake is now implemented and **verified - live**: the registry transport and packing are proven (a display-disable key - took effect in the server's journal), the GPU timer is programmed before - the kick, and the chipset/bridge block of the system information is - populated from live values. Init now reaches the server's **static-config - collection** and fails there (`0x59`); the device-id override key was - exercised with no change, and a display-disable experiment showed display is - not the first error. A source-level review of the failing collector has now - closed the host side: it reads none of the remaining host-supplied fields - (the one registry key on its call graph can only clear an error), so the - ACPI/workspace/zeroed-field leads are retired. The one hard dependency left - runtime is enabled only after the collector and the protected region blocks - host reads). **Cycles 11-13 resolved this wall**: the resource-manager log - ring had never actually been mapped (its libos init-argument id was - malformed), which is why the first error was invisible; with the ring live - the failure traced to the server's chip device-id reading as zero because - the pre-boot identity fields were sent unpacked instead of as raw PCI - config-space dwords. Fixing the payload made the server report init-done - with a zero result and serve the full static-config reply (GPU name, 8 GiB - FB, internal handles, BAR PDE bases, engine caps), and graceful shutdown - works through the unload RPC. Each of cycles 4-13 released the device - cleanly with no reboot. Since 2026-10-01 17:24 the next rung has both code - and live results: a host resource-manager client creates memory and - channel-group objects on the booted server, allocates and programs the - page-directory base of a virtual-address-space object, and **allocates a real - channel** on the live server. In cycles 18-25 it went all the way to - execution: the channel was bound and scheduled, a work-submit token was - obtained, a copy work item was placed in its GPFIFO and the doorbell rung, - and after the engine's fetch and method-path faults were diagnosed and fixed - the physical copy engine ran it - a 4 KiB copy with the semaphore released - and destination equal to source. Every cycle released the device cleanly - with no reboot; the next rung is wiring the live physical session into the - guest-facing mdev/vfio path - ([`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md)). The guest-wiring rung then advanced: an inert backend-session API was added, and the mediated-device-owned lifecycle (P1, see the frontier section) was implemented and exercised - deferring the physical claim/boot to the first device create, releasing it on the last remove, with fail-open behaviour. The first P1 acceptance cycle exposed two defects - an asynchronous boot marked READY after a failed FWSEC boot, and the P1 release path freezing the host (see the frontier section). The G0 fix bundle (default-latched D0/D3cold locks, post-rebind readiness gate, pre-claim D0 settle, READY gated on a proven boot, pre-kick staging readback) is implemented, built, smoke-tested and now **validated live**: two module-owned supervised cycles (2026-10-02 ~00:57-00:58) were 2/2 green through claim, FWSEC, booter, RISC-V active, init-done (`0x1001`, `rpc_result=0` in ~955 ms) and a latched release with WPR2 down, exactly 5 `power domain left LOCKED` lines, 0 in-window rail cycles, host alive and pstore empty. The retested P1 mdev-owned path - the exact path that hard-froze on 2026-10-01 - now fails open safely (the READY gate releases the claim when no live GSP-RM session exists; no freeze), while the async FWSEC boot defect (`staging not visible before kick` -> `FWSEC/FRTS execution failed: -5`) remains open, quarantined off the critical path; guest wiring proceeds on the module-owned live session. The bundle was later committed (`56766b2`). Guest-wiring P2 (static-config identity overlay) has since landed and verified live (2026-10-02 01:20-01:45): with `gsp_phys_static_live=1` the guest's `nvidia-smi -q` Product Name comes from the live session (`NVIDIA GeForce RTX 2080 Super with Max-Q Design`) versus the emulated `NVIDIA GeForce RTX 2080 SUPER` with it off (A/B; overlay applied 4x/0), while the guest smoke stays GREEN `ALL OK` 64/64; one overlay-OFF full run hit a pre-existing emulated-path flake, watched. P3 (read-only control forwarding) has since landed and verified live (2026-10-02 ~01:33-01:45): the `gsp_phys_ctrl_forward` whitelist served 22 controls live with 2 fallbacks and 0 timeouts while the guest smoke stayed GREEN `ALL OK` 64/64; **P5 wave 1 (guest-submission shadow execution) has since landed and verified live** (2026-10-02 ~02:00-02:20): with `gsp_phys_work_forward=1` the sema pump shadow-executes eligible CE pushes on the physical engine through a dedicated ordered workqueue - 514 real 4 KiB copies during a GREEN guest smoke (`ALL OK` 64/64), 194 requests coalesced/dropped while one was in flight, 0 refusals, 0 timeouts - while the software release path stays authoritative and no guest memory or guest VA is touched; P6 has since proven guest-page DMA live (P6a, 2026-10-02: the physical copy engine copied 4 KiB to and from a real guest RAM page with the semaphore released while the guest smoke stayed GREEN `ALL OK` 64/64 and release stayed latched with no reboot) and then **P6b wave 1** (2026-10-02, run p6run13): with `gsp_phys_work_forward=1 gsp_phys_guest_mem=1` the sema service resolved real guest CE-copy operands to host physical addresses and executed them on the physical copy engine through module-built GMMU scratch slots (`req=42 ok=42 verify_ok=42 verify_fail=0`, 503 ladder shadow copies also ran) while the software release path stayed authoritative - guest CE copies now execute on hardware and are **hardware-authored with verified data** while the software copy remains the fail-closed fallback for unqueued or failed copies: wave 1.5 added deferred, exactly-once release (`engaged=75 applied=75`), and wave 2 added `gsp_phys_work_hw_primary=1`, which skips the software copy for forwarded ops (`copy_skipped=119`, `req=100 ok=100 verify_ok=100`, smoke `matched=64/64` `ALL OK`, 19 inline software fallbacks `copy_soft=19 copy_fail=0`, and a 121-copy fallback-recovery run with guest-RAM ranges unregistered). The guest smoke stayed GREEN `ALL OK` 64/64 and release stayed latched with no reboot. Real page-directory shadowing and guest channels (P6c: completion/fault relay, guest page tables, `vectorAdd`) have since been reached: guest CE copies and their releases are forwarded with release and fault relay in place (P6c C0/C1/C3, default off); the host GR ladder (H1) brings up a GR0 channel and a `TURING_COMPUTE_A` object, but the first push faults `CTXNOTVALID` (the external-VAS context was never committed) and the promote-binding attempt wedged GSP-RM (host assert, fuse-locked WPR2), so the fail-safe held the device until an approved reboot restored it (see the frontier section). Real guest page tables/channels and GR execution remain, along with the P6a trigger-page pinning follow-up, and P4 (alloc/free handle adoption) remains deliberately deferred. | +| CUDA kernel execution (`vectorAdd`) | 🟡 **In progress - the physical backend is live** (full boot + RM objects + a real CE copy + guest CE forwarding); the current wall is the GR context/execution path and the guest's first device allocation. The first green guest CUDA execution is the immediate goal; the chronological detail is in the milestone log below and the pre-2026-10-02 mega-summary is preserved in the archive. | | Graphics (3D), video | ❌ | | Faults, multi-version, Windows, migration, multi-tenancy, performance | ❌ | @@ -79,9 +111,35 @@ Test status is tracked with the CUDA driver-API smoke test DtoH -> cuMemFree -> cuCtxDestroy` and prints `ALL OK` only when the whole chain succeeds. -## Right now (frontier) +## Milestone index (granular IDs; full detail in the milestone log below) + +| ID | Milestone | When | State | +|---|---|---|---| +| MS-20261002-01 | First real guest CUDA program vs the physical backend | 2026-10-02 ~22:01 | device props enumerated; stalls in first alloc/UVM; clean teardown | +| MS-20261002-02 | GR promote-side wall root-caused; current wall shared-mgmt GR alloc | 2026-10-02 ~17:35 | default-off change; server wedge on `TURING_COMPUTE_A`; fail-safe held | +| MS-20261002-03 | Guest-submission relay + fault relay (P6c) | 2026-10-02 07:43-08:07 | landed default-off; `relays=64`; one data-mismatch under triage | +| MS-20261002-04 | Host GR ladder (H1) opened | 2026-10-02 08:29 | GR0 channel + compute object; first submit `CTXNOTVALID` | +| MS-20261002-05 | risc-v-gsp-rm-debugger status (tool) | 2026-10-02 08:36 | superseded 2026-10-03: see the Simulator workstream section | +| MS-20261002-06 | P6a trigger-path fix | 2026-10-02 07:10 | guest G2H/W2G copy green; PTE self-test green | +| MS-20261002-07 | P6b wave 2 (hardware-primary copies) | 2026-10-02 | `copy_skipped=119`; queued guest copies 100/100 verified | +| MS-20261002-08 | P6b wave 1 (guest CE forwarding) | 2026-10-02 | `req=42 ok=42 verify_ok=42`; 512-pair scratch-pool rule | +| MS-20261002-09 | P6a guest-page DMA proven on the physical engine | 2026-10-02 | G2H + W2G 4 KiB with semaphore; guest observed the write | +| MS-20261002-10 | P6 guest-memory first attempts | 2026-10-02 ~02:42 | resolved by P6a | +| MS-20261002-11 | P5 wave 1 guest-submission shadow execution | 2026-10-02 ~02:00 | 514 real CE copies; 194 coalesced drops | +| MS-20261002-12 | P3 read-only control forwarding | 2026-10-02 01:33 | `ok=22 fallback=2 timeout=0`; smoke green | +| MS-20261002-13 | P2 guest-facing identity overlay | 2026-10-02 01:20 | live Product Name A/B (overlay 4x/0) | +| MS-20261002-14 | G0 freeze-fix bundle validated live | 2026-10-02 ~00:57 | 2/2 green cycles; committed `56766b2` | +| MS-20261002-15 | P1 mdev-owned lifecycle (fail-open) | 2026-10-02 ~00:58 | retested after G0; no freeze | +| MS-20261001-01 | Guest-wiring rung opened (P1) | 2026-10-01 22:00 | backend-session API inert; baseline re-verified | +| MS-20261001-02 | Submission rung: real CE copy on the physical engine | 2026-10-01 17:24-21:05 | `REAL COPY OK`; clean release each cycle | +| MS-20261001-03 | GSP-RM serves the static-config RPC | 2026-10-01 16:45 | `rpc_result=0`; 1656-byte config; graceful unload | +| MS-20261001-04 | Driver-API baseline re-verified green | 2026-10-01 14:20 | `ALL OK` 64/64; `faked=0 miss=0 stale=0` | +| MS-20261001-05 | First real GSP boot + live GSP-RM session | 2026-10-01 | WPR2 up; RISC-V active; first event `0x1002` | +| MS-LEGACY-01..06 | Emulation-campaign milestones (pre-2026-10-01) | 2026-09-26..09-30 | details in the milestone log/archive below | -- **First real guest CUDA program exercised against the physical backend +## Milestone log - current frontier and recent milestones (granular IDs in each entry) + +- `MS-20261002-01` **First real guest CUDA program exercised against the physical backend (2026-10-02 ~22:01).** A fresh supervised module-owned cycle (claim + full GSP-RM boot + guest RAM + hardware-primary work forwarding) booted the real server (`stage 3e complete`, init event `0x1001`, `rpc_result=0` in ~952 ms), @@ -95,7 +153,7 @@ chain succeeds. mdev removed, protected region down, host driver healthy, pstore empty). This is the first time a real CUDA program ran that far in the guest against the real backend; it is not yet a green execution. -- **GR context campaign: the promote-side wall is root-caused; the current +- `MS-20261002-02` **GR context campaign: the promote-side wall is root-caused; the current wall is the shared-management GR object alloc (2026-10-02 ~17:35).** The long-running context-promote wall was the ladder's virtual address space being allocated as *externally owned* (the client owns page tables, VA @@ -129,7 +187,7 @@ chain succeeds. enumerates and reads device properties, then stalls in its first device allocation / UVM channel path before any kernel executes - the first green guest execution remains the immediate goal. -- **Guest-submission relay and fault relay landed, default off (P6c +- `MS-20261002-03` **Guest-submission relay and fault relay landed, default off (P6c C0/C1/C3, 2026-10-02 07:43-08:07).** With `gsp_phys_relay=1` the physical engine authors the guest CE copy and the release semaphore: the engine-authored release was observed in the guest page (`relays=64`), @@ -143,7 +201,7 @@ chain succeeds. relays forwarded-engine faults to the guest channel notifier/event path; it delivered a real CE0 fault live (default off), and the auxiliary event post reported `no_event`. The software path is unchanged with all parameters off. -- **Host GR ladder (H1) opened and hit the GR context wall (2026-10-02 +- `MS-20261002-04` **Host GR ladder (H1) opened and hit the GR context wall (2026-10-02 08:29).** The host resource-manager ladder is now selectable to the compute engine (`gsp_phys_rmcall_engine=1`, default off) and brings up a GR0 channel with a `TURING_COMPUTE_A` compute object on the live GSP-RM @@ -157,7 +215,7 @@ chain succeeds. fuse-locked WPR2) - see the top bullet. Current wall: commit the GR context and make the engine execute the submitted compute block. Next: host `vectorAdd`, then guest GR forwarding. -- **Experimental tooling: risc-v-gsp-rm-debugger (2026-10-02 08:36, in development).** +- `MS-20261002-05` **Experimental tooling: risc-v-gsp-rm-debugger (2026-10-02 08:36, in development).** `debug/risc-v-gsp-rm-vm/` boots GSP/uproc firmware in a RISC-V QEMU VM with a GDB stub and adds Frida-like runtime inspection, a scripting/plugin API and a Ghidra bridge for GSP-RM runtime-memory analysis and @@ -165,13 +223,13 @@ chain succeeds. changes); the in-development notices were removed 2026-10-02 after the tool audit and value validation; see [`docs/05-operations/03-risc-v-gsp-rm-vm.md`](docs/05-operations/03-risc-v-gsp-rm-vm.md). -- **P6a trigger-path fix landed (2026-10-02 07:10).** The debug copy +- `MS-20261002-06` **P6a trigger-path fix landed (2026-10-02 07:10).** The debug copy triggers now resolve the guest page host-physical address at trigger time through the fresh-scratch-line path, retiring the stale-PTE data-check failure (live: `GUEST G2H COPY OK`, `GUEST W2G COPY OK verify_ok=1`, PTE self-test and ladder `REAL COPY OK`, clean release). Forwarding was never affected (HPAs are resolved at submit time). -- **Guest CE copies are hardware-authored with verified data, with deferred +- `MS-20261002-07` **Guest CE copies are hardware-authored with verified data, with deferred exactly-once release (P6b waves 1.5 and 2, 2026-10-02, run p6w2d; superseded as a frontier by the P6c relay and H1 bullets above).** Wave 1.5 (run p6w15), with `gsp_phys_work_defer_release=1` (plus @@ -209,7 +267,7 @@ chain succeeds. The trigger-path fix, the P6c relay/fault relay and the host GR ladder have since landed (bullets above); the relay data-mismatch triage and the GR execution wall are the open items. -- **Guest CE copies run on the physical engine with verified data (P6b wave 1, +- `MS-20261002-08` **Guest CE copies run on the physical engine with verified data (P6b wave 1, 2026-10-02, run p6run13; superseded by the P6b wave-2 result above).** With `gsp_phys_work_forward=1 gsp_phys_guest_mem=1` during a full CUDA driver-API smoke, the sema service captures the first eligible copy per push @@ -239,7 +297,7 @@ chain succeeds. can theoretically race guest reuse within the submit window. Module sha256 `9d444908...`. **Next: P6b wave 2** (hardware-primary release relay), then **P6c** (real guest page tables/channels). -- **Guest-page DMA proven on the physical engine (P6a, 2026-10-02; superseded +- `MS-20261002-09` **Guest-page DMA proven on the physical engine (P6a, 2026-10-02; superseded by the P6b wave-1 result above).** The IOMMU/DMA reachability known-unknown is closed on the reference host: the physical GPU's IOMMU group is an identity domain (all-RAM identity) and the @@ -268,7 +326,7 @@ chain succeeds. `e1bce651...`. **Next: P6b** - forwarding guest submissions (page-directory shadowing, real guest channels, push forwarding) - landed afterwards (superseded by the P6b/P6c bullets above). -- **Guest-memory path (P6) first attempts (2026-10-02 ~02:42 onward; resolved +- `MS-20261002-10` **Guest-memory path (P6) first attempts (2026-10-02 ~02:42 onward; resolved by the P6a result above).** The rung puts guest memory into the real execution path: guest RAM is now a QEMU file-backed `pc.ram` region (`/dev/shm/vgpu-guest-ram`, 4096M, @@ -286,7 +344,7 @@ chain succeeds. latter was root-caused in-run). The P6 code was uncommitted at the time of that snapshot; the follow-up attempt resolved both defects (P6a result above). -- **Guest-submission shadow execution (P5 wave 1) landed and verified live +- `MS-20261002-11` **Guest-submission shadow execution (P5 wave 1) landed and verified live (2026-10-02 ~02:00-02:20).** Under the new `gsp_phys_work_forward` parameter (bool, default off) the sema pump requests ONE shadow copy for each eligible CE push through a dedicated ordered workqueue (`gsp-phys-work`), @@ -312,7 +370,7 @@ chain succeeds. (hardware-primary release relay with a non-applying decode mode; **subsumed** by P6b wave 2 + the P6c relay above); **P4** remains deferred. -- **Read-only control forwarding (P3) landed and verified live (2026-10-02 +- `MS-20261002-12` **Read-only control forwarding (P3) landed and verified live (2026-10-02 01:33-01:45).** `GSP_RM_CONTROL` (fn76) forwards a whitelist of side-effect-free queries (`BUS_GET_PCI_BAR_INFO`, `BUS_GET_INFO_V2`, `FB_GET_INFO_V2`, `INTERNAL_GMMU_GET_STATIC_INFO`) from the live GSP-RM @@ -332,7 +390,7 @@ chain succeeds. would change guest-visible handles and risk the currently green guest path. The P2/P3/P5-wave-1 bundles and the G0 freeze fix are now committed (`56766b2`, `82fc7a8`; HEAD `82fc7a8`). -- **Guest-facing identity wiring (P2) landed and verified live (2026-10-02 +- `MS-20261002-13` **Guest-facing identity wiring (P2) landed and verified live (2026-10-02 01:20-01:45).** On a module-owned live GSP-RM session the emulation overlays the server's real static configuration identity into the guest-facing `GET_GSP_STATIC_INFO` reply under the new @@ -354,7 +412,7 @@ chain succeeds. above). At that point guest work was not yet forwarded to the physical engine (the P6b/P6c forwarding results are above). The P2 bundle and the G0 freeze fix have since been committed (`56766b2`). -- **G0 fix bundle built after the P1 release-path freeze (2026-10-02 00:55).** +- `MS-20261002-14` **G0 fix bundle built after the P1 release-path freeze (2026-10-02 00:55).** The first P1 mdev-owned acceptance cycle ended in the known hard freeze (journal stops, pstore empty): the P1 release path ran outside the supervised D0 pre-lock window and the teardown restored `power/control=auto`, re-arming @@ -374,7 +432,7 @@ chain succeeds. `released 0000:01:00.0; rebound to nvidia (rc=1)`, exactly 5 `power domain left LOCKED` lines, 0 in-window rail cycles, host survived, pstore empty; module sha256 `dde36d92...`. Committed in `56766b2`. -- **Guest-wiring lifecycle (P1) implemented and exercised; retested after G0 +- `MS-20261002-15` **Guest-wiring lifecycle (P1) implemented and exercised; retested after G0 (2026-10-02 ~00:58).** A module parameter selects a lifecycle where the physical-GPU claim and boot are deferred to the first mediated device create and released on the last remove, run asynchronously and **fail-open**: if the @@ -404,7 +462,7 @@ chain succeeds. (`gsp_phys_gpu=1` claim+boot at load, proven); the P1 async claim stays fail-open. At that point no guest work was forwarded to the physical engine yet (the P6b/P6c forwarding results are above). -- **Guest-wiring rung opened (2026-10-01 22:00): the first backend-session +- `MS-20261001-01` **Guest-wiring rung opened (2026-10-01 22:00): the first backend-session API code is in the tree, inert.** A skeleton that exposes the live physical session to the device layer (session request entry points plus a static-info snapshot helper) was added behind the existing gates and does not yet alter @@ -418,7 +476,7 @@ chain succeeds. 3D acceleration, and a native-vs-guest performance program; and a host-side acceleration feasibility study) were produced and are kept in the local evidence archive (untracked). -- **Submission-rung milestone (2026-10-01 17:24-21:05): real work now executes +- `MS-20261001-02` **Submission-rung milestone (2026-10-01 17:24-21:05): real work now executes on the physical engine.** The host resource-manager client allocated a virtual-address-space object with a committed page directory, a channel-group object and a real copy-engine channel, bound and scheduled it, obtained a @@ -436,7 +494,7 @@ chain succeeds. [`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md)). Evidence: the local evidence archive (cycle 18 through cycle 25 runs, with per-wall reports). -- **Physical-backend milestone (2026-10-01 16:45): the host-owned GSP-RM +- `MS-20261001-03` **Physical-backend milestone (2026-10-01 16:45): the host-owned GSP-RM completes initialization on the real TU104 and serves the static-config RPC.** Three supervised cycles, no reboot, clean release each time: 1. the resource-manager log ring is now registered correctly (its libos @@ -459,11 +517,11 @@ chain succeeds. send RM RPCs and receive full replies, and shut it down gracefully. The next rung is to drive the guest-facing emulation from this backend. Evidence: the local evidence archive (cycles 11-13). -- **Driver-API baseline re-verified green on the current build** (2026-10-01 +- `MS-20261001-04` **Driver-API baseline re-verified green on the current build** (2026-10-01 14:20): the smoke test passes end-to-end (`ALL OK`, payload 64/64) with real releases (`faked=0`, `miss=0`, `stale=0`); the source-level root-cause campaign (cycles 4-10) did not regress it. -- **Physical-backend milestone (2026-10-01)**: the module-owned GPU path +- `MS-20261001-05` **Physical-backend milestone (2026-10-01)**: the module-owned GPU path reached the first successful secure-verification (FWSEC) carve of the protected region on the real TU104 under the verified D0 power-domain harness. The teardown of a FWSEC-only carve is not host-recoverable on this @@ -517,17 +575,17 @@ chain succeeds. failure was a zero server-side device-id caused by our unpacked identity payload - the PMU path itself was fine. The raw config-space identity fix completed initialization (see the milestone bullet above). -- **The full CUDA driver-API smoke test passes with correct data**: the client +- `MS-LEGACY-01` **The full CUDA driver-API smoke test passes with correct data**: the client completes `cuCtxCreate -> cuMemAlloc -> cuMemcpyHtoD -> cuMemcpyDtoH -> cuMemFree -> cuCtxDestroy` and prints `ALL OK`, with the 64-byte payload roundtripping exactly (64/64 verified in the guest). -- **The last two walls**: the GPU page-table walk's address decode (child +- `MS-LEGACY-02` **The last two walls**: the GPU page-table walk's address decode (child pointers and leaves were 16x too low; fixed with the real entry encoding - walks now succeed at scale) and the device-to-host copy (a copy-engine class transfer bound on a compute channel whose work was denied by the engine-type gate; trusted host-injected pushes may now execute copy-engine work as well). -- **Baseline green and quiet**: the payload roundtrip is back to 64/64 +- `MS-LEGACY-03` **Baseline green and quiet**: the payload roundtrip is back to 64/64 (`ALL OK`), the staging buffer's translation stays resolvable via pins streamed from the guest, and high-frequency diagnostics are now gated behind a default-off debug switch - a normal boot+run produces a ~520-line @@ -542,17 +600,17 @@ chain succeeds. the copy-engine decoder consults pinned translations before the scanned map, so every push-data operation executes (`memset=19473`, `ringops=10551` live). -- **The runtime client now initializes and reaches kernel launch**: with a +- `MS-LEGACY-04` **The runtime client now initializes and reaches kernel launch**: with a static-runtime `vectorAdd` client, `cudaGetDeviceCount`, `cudaSetDevice`, `cudaFree`, three `cudaMalloc`, the HtoD copies and `cudaGetLastError` all return success. The client then parks in `cudaDeviceSynchronize` - the launch is queued but never **executed**, because real execution requires the physical-GPU backend (the next project). -- **What fixed it**: the failing call was never a kernel call - it was the +- `MS-LEGACY-05` **What fixed it**: the failing call was never a kernel call - it was the clock-domain probe inside libcuda (two device attributes) whose three legacy clock controls were answered with zeros; they are now populated with exactly the fields the helper checks. -- **Real-release mode now reaches the same rc=0 chain**: the earlier +- `MS-LEGACY-06` **Real-release mode now reaches the same rc=0 chain**: the earlier per-push 8 s stall was two pump bookkeeping bugs (release detection keyed on an unchanged semaphore VA; rewind replays re-entering the CE grace path) - both fixed, and the runtime init completes in ~1 min with real releases. @@ -642,6 +700,72 @@ chain succeeds. work does not touch the driver binding and is unaffected. See [`docs/03-architecture/08-physical-gpu-integration.md`](docs/03-architecture/08-physical-gpu-integration.md). + +--- + +## Evidence pointers + +- **Module sha256 (latest runs):** `e5318ee0...` (Addendum 30, 2026-10-03, + large-push parser fix); earlier: P6b `5fb5bfbe...`/`9d444908...`, P6a + `e1bce651...`, P3 `30dfd311...`, P2 `69eac176...`, G0 `dde36d92...`, P5 + wave 1 `83952506...`; boot `12cf3a41-...` unchanged across Addenda 28-30. +- **Campaign detail:** `Handoff-vGPU-IO.md` Part 93 Addenda 26-30 (the + authoritative live narrative; this page is the dated public snapshot). +- **Local evidence archive:** `open-iov/backups/evidence-20261002-victory/` + (`*-victory8/9`, `14-guest-dmesg-fail-*`, `20-gsp-phys-state-*`); earlier + cycle 11-25 runs with per-wall reports (local archive, untracked). +- **Simulator sessions:** `debug/risc-v-gsp-rm-vm/dumps/gsp-boot-b1-run1/2`, + `b2-run1/2`, and `debug/risc-v-gsp-rm-vm/harness/qemu_line/` (docs/36 §4, + docs/37 §4.3). +- **Definition of done:** `docs/04-implementation/01-known-knowns-and-unknowns.md` + §4 + the open-gates section (archived below). + +## Archive - historical narrative (pre-2026-10-02) + +The sections from here down are the dated historical record kept for evidence; the current state is in the top sections of this page. The 2026-10-02 milestone-ladder mega-summary is preserved verbatim below. + +### Preserved mega-summary (2026-10-02 milestone-ladder cell, verbatim) + +> | CUDA kernel execution (`vectorAdd`) | 🟡 execution backend: physical ownership + firmware staging verified; the unbind/rebind freeze hazard is mitigated by the supervised D0 power-domain lock (3/3 green claim cycles; the 2026-10-01 P1 mdev-owned release path initially ran outside that pre-lock window and re-armed D3cold, freezing the host at 22:32:53 - the G0 fix bundle now latches the D0/D3cold locks by default; see the frontier section), and the real FB size (8 GiB) and boot layout are verified live. The supervised live secure-verification run raised the protected region (WPR2 up) for the first time, then proved a secure-verification-only carve is not host-recoverable on this fused part, so the fail-safe held the device and an approved reboot restored it (distribution driver rebound, GPU healthy). That one-way stage is now retired: the only device-write path is a single full boot lifecycle (secure verification -> bootloader -> RISC-V start -> resource-manager initialization), whose code is implemented in the module sources and has now run live: the first real boot raised WPR2, passed Booter Load and went RISC-V active, and the module received the first real GSP-RM message (event 0x1002); init-done (0x1001) and a clean release are the next steps (see the 2026-10-01 section). A second live boot on 2026-10-01 proved the full boot sequence again (protected region up, bootloader halt, RISC-V active, first GSP-RM message) and reached the resource-manager's inline CPU-sequencer request; the host executor rejected the real request because of an over-strict internal size cap, which was found with a raw read of the message in host memory, fixed, and rebuilt. Cycles 2-3 then ran the real sequencer and received init-done with an RM init failure (`0x59`), which reproduces on a verified matching firmware image; the server's own shutdown supplied the last handoff bit, so each failed cycle +> completed a **clean release without a reboot**. Cycles 4-10 then captured +> the resource-manager's own error journal and root-caused the failure: the +> boot queue was missing the two pre-boot initialization messages the matching +> driver queues before the kernel kick (a system-information message and a +> packed registry table). That handshake is now implemented and **verified +> live**: the registry transport and packing are proven (a display-disable key +> took effect in the server's journal), the GPU timer is programmed before +> the kick, and the chipset/bridge block of the system information is +> populated from live values. Init now reaches the server's **static-config +> collection** and fails there (`0x59`); the device-id override key was +> exercised with no change, and a display-disable experiment showed display is +> not the first error. A source-level review of the failing collector has now +> closed the host side: it reads none of the remaining host-supplied fields +> (the one registry key on its call graph can only clear an error), so the +> ACPI/workspace/zeroed-field leads are retired. The one hard dependency left +> runtime is enabled only after the collector and the protected region blocks +> host reads). **Cycles 11-13 resolved this wall**: the resource-manager log +> ring had never actually been mapped (its libos init-argument id was +> malformed), which is why the first error was invisible; with the ring live +> the failure traced to the server's chip device-id reading as zero because +> the pre-boot identity fields were sent unpacked instead of as raw PCI +> config-space dwords. Fixing the payload made the server report init-done +> with a zero result and serve the full static-config reply (GPU name, 8 GiB +> FB, internal handles, BAR PDE bases, engine caps), and graceful shutdown +> works through the unload RPC. Each of cycles 4-13 released the device +> cleanly with no reboot. Since 2026-10-01 17:24 the next rung has both code +> and live results: a host resource-manager client creates memory and +> channel-group objects on the booted server, allocates and programs the +> page-directory base of a virtual-address-space object, and **allocates a real +> channel** on the live server. In cycles 18-25 it went all the way to +> execution: the channel was bound and scheduled, a work-submit token was +> obtained, a copy work item was placed in its GPFIFO and the doorbell rung, +> and after the engine's fetch and method-path faults were diagnosed and fixed +> the physical copy engine ran it - a 4 KiB copy with the semaphore released +> and destination equal to source. Every cycle released the device cleanly +> with no reboot; the next rung is wiring the live physical session into the +> guest-facing mdev/vfio path +> ([`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md)). The guest-wiring rung then advanced: an inert backend-session API was added, and the mediated-device-owned lifecycle (P1, see the frontier section) was implemented and exercised - deferring the physical claim/boot to the first device create, releasing it on the last remove, with fail-open behaviour. The first P1 acceptance cycle exposed two defects - an asynchronous boot marked READY after a failed FWSEC boot, and the P1 release path freezing the host (see the frontier section). The G0 fix bundle (default-latched D0/D3cold locks, post-rebind readiness gate, pre-claim D0 settle, READY gated on a proven boot, pre-kick staging readback) is implemented, built, smoke-tested and now **validated live**: two module-owned supervised cycles (2026-10-02 ~00:57-00:58) were 2/2 green through claim, FWSEC, booter, RISC-V active, init-done (`0x1001`, `rpc_result=0` in ~955 ms) and a latched release with WPR2 down, exactly 5 `power domain left LOCKED` lines, 0 in-window rail cycles, host alive and pstore empty. The retested P1 mdev-owned path - the exact path that hard-froze on 2026-10-01 - now fails open safely (the READY gate releases the claim when no live GSP-RM session exists; no freeze), while the async FWSEC boot defect (`staging not visible before kick` -> `FWSEC/FRTS execution failed: -5`) remains open, quarantined off the critical path; guest wiring proceeds on the module-owned live session. The bundle was later committed (`56766b2`). Guest-wiring P2 (static-config identity overlay) has since landed and verified live (2026-10-02 01:20-01:45): with `gsp_phys_static_live=1` the guest's `nvidia-smi -q` Product Name comes from the live session (`NVIDIA GeForce RTX 2080 Super with Max-Q Design`) versus the emulated `NVIDIA GeForce RTX 2080 SUPER` with it off (A/B; overlay applied 4x/0), while the guest smoke stays GREEN `ALL OK` 64/64; one overlay-OFF full run hit a pre-existing emulated-path flake, watched. P3 (read-only control forwarding) has since landed and verified live (2026-10-02 ~01:33-01:45): the `gsp_phys_ctrl_forward` whitelist served 22 controls live with 2 fallbacks and 0 timeouts while the guest smoke stayed GREEN `ALL OK` 64/64; **P5 wave 1 (guest-submission shadow execution) has since landed and verified live** (2026-10-02 ~02:00-02:20): with `gsp_phys_work_forward=1` the sema pump shadow-executes eligible CE pushes on the physical engine through a dedicated ordered workqueue - 514 real 4 KiB copies during a GREEN guest smoke (`ALL OK` 64/64), 194 requests coalesced/dropped while one was in flight, 0 refusals, 0 timeouts - while the software release path stays authoritative and no guest memory or guest VA is touched; P6 has since proven guest-page DMA live (P6a, 2026-10-02: the physical copy engine copied 4 KiB to and from a real guest RAM page with the semaphore released while the guest smoke stayed GREEN `ALL OK` 64/64 and release stayed latched with no reboot) and then **P6b wave 1** (2026-10-02, run p6run13): with `gsp_phys_work_forward=1 gsp_phys_guest_mem=1` the sema service resolved real guest CE-copy operands to host physical addresses and executed them on the physical copy engine through module-built GMMU scratch slots (`req=42 ok=42 verify_ok=42 verify_fail=0`, 503 ladder shadow copies also ran) while the software release path stayed authoritative - guest CE copies now execute on hardware and are **hardware-authored with verified data** while the software copy remains the fail-closed fallback for unqueued or failed copies: wave 1.5 added deferred, exactly-once release (`engaged=75 applied=75`), and wave 2 added `gsp_phys_work_hw_primary=1`, which skips the software copy for forwarded ops (`copy_skipped=119`, `req=100 ok=100 verify_ok=100`, smoke `matched=64/64` `ALL OK`, 19 inline software fallbacks `copy_soft=19 copy_fail=0`, and a 121-copy fallback-recovery run with guest-RAM ranges unregistered). The guest smoke stayed GREEN `ALL OK` 64/64 and release stayed latched with no reboot. Real page-directory shadowing and guest channels (P6c: completion/fault relay, guest page tables, `vectorAdd`) have since been reached: guest CE copies and their releases are forwarded with release and fault relay in place (P6c C0/C1/C3, default off); the host GR ladder (H1) brings up a GR0 channel and a `TURING_COMPUTE_A` object, but the first push faults `CTXNOTVALID` (the external-VAS context was never committed) and the promote-binding attempt wedged GSP-RM (host assert, fuse-locked WPR2), so the fail-safe held the device until an approved reboot restored it (see the frontier section). Real guest page tables/channels and GR execution remain, along with the P6a trigger-page pinning follow-up, and P4 (alloc/free handle adoption) remains deliberately deferred. | + ## First real GSP boot + live GSP-RM session (2026-10-01) Cycle 09:53:27–09:54:59 on the supervised D0 harness (`--boot --keep`), @@ -839,8 +963,10 @@ a GDB stub and adds Frida-like runtime inspection, a scripting/plugin API and a Ghidra bridge for GSP-RM runtime-memory analysis and emulation-accuracy measurement. Status is tracked by the tool audit (`debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md`); the in-development notices -were removed 2026-10-02 after the 194-test suite and end-to-end value -demonstrations (`debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md`); see +were removed 2026-10-02 after the then-194-test suite and end-to-end value +demonstrations (`debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md`); the +2026-10-03 Phase S wave (device model `B0`-`B2`, A/B line, triage; 290 tests) +is summarised in the "Simulator workstream" section above (docs 35-40). See [`docs/05-operations/03-risc-v-gsp-rm-vm.md`](docs/05-operations/03-risc-v-gsp-rm-vm.md). | Wall | Fix | |