From 162c6ca1bc0811001c095c9a52acd5b0025dfc2c Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Thu, 1 Oct 2026 15:44:02 -0400 Subject: STATE-CURRENT: cycles 4-10 - pre-boot RPCs/registry/timer/chipset landed; RM init wall narrowed to static-config collection --- STATE-CURRENT.md | 67 ++++++++++++++++++++++++++++++-------------------------- 1 file changed, 36 insertions(+), 31 deletions(-) diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md index 4d9dc83..61d7217 100644 --- a/STATE-CURRENT.md +++ b/STATE-CURRENT.md @@ -1,6 +1,6 @@ # Open-IOV Current State -> **Snapshot: 2026-10-01 14:20.** This is a dated public status page, not live +> **Snapshot: 2026-10-01 15:45.** This is a dated public status page, not live > state; it is refreshed on every progress check. Live working notes are > maintained outside this repository; the public known-knowns/unknowns document > is authoritative here: @@ -32,18 +32,19 @@ meant to execute on the physical GPU. | `cuMemAlloc` + `cuMemcpy` roundtrip (`ALL OK` in the smoke test) | ✅ payload roundtrip verified 64/64 in the guest | | Full `nvidia-smi` output / kernel launch | 🟡 the static runtime now initializes end-to-end with **real releases** (`cudaGetDeviceCount`/`cudaSetDevice`/3×`cudaMalloc`/2×`cudaMemcpyHtoD`/`kernel launch`/`cudaGetLastError` all `rc=0`) and queues a kernel launch; execution awaits the backend | | CUDA kernel execution (`vectorAdd`) | 🟡 execution backend: physical ownership + firmware staging verified; the unbind/rebind freeze hazard is mitigated by the supervised D0 power-domain lock (3/3 green claim cycles), and the real FB size (8 GiB) and boot layout are verified live. The supervised live secure-verification run raised the protected region (WPR2 up) for the first time, then proved a secure-verification-only carve is not host-recoverable on this fused part, so the fail-safe held the device and an approved reboot restored it (distribution driver rebound, GPU healthy). That one-way stage is now retired: the only device-write path is a single full boot lifecycle (secure verification -> bootloader -> RISC-V start -> resource-manager initialization), whose code is implemented in the module sources and has now run live: the first real boot raised WPR2, passed Booter Load and went RISC-V active, and the module received the first real GSP-RM message (event 0x1002); init-done (0x1001) and a clean release are the next steps (see the 2026-10-01 section). A second live boot on 2026-10-01 proved the full boot sequence again (protected region up, bootloader halt, RISC-V active, first GSP-RM message) and reached the resource-manager's inline CPU-sequencer request; the host executor rejected the real request because of an over-strict internal size cap, which was found with a raw read of the message in host memory, fixed, and rebuilt. Cycles 2-3 then ran the real sequencer and received init-done with an RM init failure (`0x59`), which reproduces on a verified matching firmware image; the server's own shutdown supplied the last handoff bit, so each failed cycle - completed a **clean release without a reboot**. Cycles 4-7 then captured the - resource-manager's own error journal and root-caused the failure: the boot - queue was missing the two pre-boot initialization messages the matching + completed a **clean release without a reboot**. Cycles 4-10 then captured + the resource-manager's own error journal and root-caused the failure: the + boot queue was missing the two pre-boot initialization messages the matching driver queues before the kernel kick (a system-information message and a packed registry table). That handshake is now implemented and **verified live**: the registry transport and packing are proven (a display-disable key - took effect in the server's journal), and the GPU timer is programmed before - the kick. The fatal error is now root-caused to a **device-id/board - validation** in the server's static-config collection, and the corresponding - device-id override key is implemented in the registry profile; the next - supervised cycle is in flight. Each of cycles 4-7 released the device - cleanly with no reboot. | + took effect in the server's journal), the GPU timer is programmed before + the kick, and the chipset/bridge block of the system information is + populated from live values. Init now reaches the server's **static-config + collection** and fails there (`0x59`); the device-id override key was + exercised with no change, and a display-disable experiment showed display is + not the first error. Each of cycles 4-10 released the device cleanly with no + reboot. | | Graphics (3D), video | ❌ | | Faults, multi-version, Windows, migration, multi-tenancy, performance | ❌ | @@ -55,10 +56,10 @@ chain succeeds. ## Right now (frontier) -- **Driver-API baseline re-verified on the current build**: the smoke test - passes end-to-end (`ALL OK`, payload 64/64) with real releases - (`faked=0`, `miss=0`, `stale=0`), confirming the recent pump/parser and - rewind changes did not regress the baseline. +- **Driver-API baseline re-verified green on the current build** (2026-10-01 + 14:20): the smoke test passes end-to-end (`ALL OK`, payload 64/64) with + real releases (`faked=0`, `miss=0`, `stale=0`); the source-level root-cause + campaign (cycles 4-10) did not regress it. - **Physical-backend milestone (2026-10-01)**: the module-owned GPU path reached the first successful secure-verification (FWSEC) carve of the protected region on the real TU104 under the verified D0 power-domain @@ -88,24 +89,28 @@ chain succeeds. reboot**: the server's own shutdown sets the remaining secure-scratch handoff bit, so all four handoff bits are set and a single `rmmod` clears the protected region, rebinds the distribution driver and releases the - power-domain locks. The error-journal capture has since run on four - supervised cycles (each one attempt, each released cleanly with no reboot): - the journal was read, and the failure was root-caused to two pre-boot - initialization messages the matching driver queues before the kernel kick (a + power-domain locks. The error-journal capture has since run on ten supervised + cycles (each one attempt, each released cleanly with no reboot): the journal + was read, and the failure was root-caused to two pre-boot initialization + messages the matching driver queues before the kernel kick (a system-information message and a packed registry table) - our boot queue was - empty. That handshake is now implemented and **verified live**: the boot - consumes a true system-information message and a real registry profile, the - registry transport and packing are proven (a display-disable key took effect - in the server's journal), the journal collapsed from 29 unique/851 total - records to 3 unique/4, and the GPU timer is programmed before the kick (the - live timer line was observed, and the generation-correct zero timer offset - was confirmed). The fatal error is now root-caused to a **device-id/board - validation** inside the server's static-config collection, and the - corresponding device-id override key is implemented in the registry - profile. The next supervised cycle is in flight. If it still fails, the - fallback leads are the device-id mirror read and the board/panel-ID tables, - the remaining zeroed system-information fields, the video-memory workspace - layout branch, and a build-matched log database for the fatal path. + empty. That handshake is + now implemented and **verified live**: the boot consumes a true + system-information message and a real registry profile, the registry + transport and packing are proven (a display-disable key took effect in the + server's journal), the journal collapsed from 29 unique/851 total records to + 4 unique/5, the pre-kick GPU timer is programmed (the live timer line was + observed, and the generation-correct zero timer offset was confirmed), and + the chipset/bridge block of the system information is populated from live + values. Init now reaches the server's **static-config collection** and fails + there (`0x59`); the device-id override key was exercised with no change, and + a display-disable experiment showed display is not the first error. The + collector's failing step is not visible in the server journal (its log ring + is enabled only after the collector, and the protected region blocks host + reads). Next leads: enumerate the collector's registry reads (in flight), + port the ACPI method-data path if indicated, and gain deeper server + visibility (enable the log ring; a server memory read if the protected + region allows). - **The full CUDA driver-API smoke test passes with correct data**: the client completes `cuCtxCreate -> cuMemAlloc -> cuMemcpyHtoD -> cuMemcpyDtoH -> cuMemFree -> cuCtxDestroy` and prints `ALL OK`, with the 64-byte payload -- cgit v1.2.3