From 66172eaba3cf6ff08a5cd4b7080e05704fa3813f Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Thu, 1 Oct 2026 10:07:15 -0400 Subject: docs: record the first real GSP boot and the live GSP-RM session STATE-CURRENT and the architecture docs now describe the observed order (FWSEC -> Booter Load -> RISC-V active -> event 0x1002 run-sequencer -> 0x1001 init-done), the four-handoff-bit clean-release constraint, and the --boot/--keep harness contract. --- STATE-CURRENT.md | 57 +++++++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 52 insertions(+), 5 deletions(-) (limited to 'STATE-CURRENT.md') diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md index 6771f53..c86311b 100644 --- a/STATE-CURRENT.md +++ b/STATE-CURRENT.md @@ -1,6 +1,6 @@ # Open-IOV Current State -> **Snapshot: 2026-10-01 09:12.** This is a dated public status page, not live +> **Snapshot: 2026-10-01 09:59.** This is a dated public status page, not live > state; it is refreshed on every progress check. Live working notes are > maintained outside this repository; the public known-knowns/unknowns document > is authoritative here: @@ -31,7 +31,7 @@ meant to execute on the physical GPU. | `cuCtxCreate` returns | βœ… (the whole smoke test now completes) | | `cuMemAlloc` + `cuMemcpy` roundtrip (`ALL OK` in the smoke test) | βœ… payload roundtrip verified 64/64 in the guest | | Full `nvidia-smi` output / kernel launch | 🟑 the static runtime now initializes end-to-end with **real releases** (`cudaGetDeviceCount`/`cudaSetDevice`/3Γ—`cudaMalloc`/2Γ—`cudaMemcpyHtoD`/`kernel launch`/`cudaGetLastError` all `rc=0`) and queues a kernel launch; execution awaits the backend | -| CUDA kernel execution (`vectorAdd`) | 🟑 execution backend: physical ownership + firmware staging verified; the unbind/rebind freeze hazard is mitigated by the supervised D0 power-domain lock (3/3 green claim cycles), and the real FB size (8 GiB) and boot layout are verified live. The supervised live secure-verification run raised the protected region (WPR2 up) for the first time, then proved a secure-verification-only carve is not host-recoverable on this fused part, so the fail-safe held the device and an approved reboot restored it (distribution driver rebound, GPU healthy). That one-way stage is now retired: the only device-write path is a single full boot lifecycle (secure verification -> bootloader -> RISC-V start -> resource-manager initialization), which is being implemented. | +| CUDA kernel execution (`vectorAdd`) | 🟑 execution backend: physical ownership + firmware staging verified; the unbind/rebind freeze hazard is mitigated by the supervised D0 power-domain lock (3/3 green claim cycles), and the real FB size (8 GiB) and boot layout are verified live. The supervised live secure-verification run raised the protected region (WPR2 up) for the first time, then proved a secure-verification-only carve is not host-recoverable on this fused part, so the fail-safe held the device and an approved reboot restored it (distribution driver rebound, GPU healthy). That one-way stage is now retired: the only device-write path is a single full boot lifecycle (secure verification -> bootloader -> RISC-V start -> resource-manager initialization), whose code is implemented in the module sources and has now run live: the first real boot raised WPR2, passed Booter Load and went RISC-V active, and the module received the first real GSP-RM message (event 0x1002); init-done (0x1001) and a clean release are the next steps (see the 2026-10-01 section). | | Graphics (3D), video | ❌ | | Faults, multi-version, Windows, migration, multi-tenancy, performance | ❌ | @@ -58,8 +58,10 @@ chain succeeds. host healthy, no crash record). The one-way stage is now retired: the only device-write path is a single supervised full boot lifecycle (secure verification -> bootloader -> RISC-V start -> resource-manager - initialization), whose next stage is being implemented so the normal boot - lifecycle makes teardown valid. + initialization). That lifecycle is now implemented in the module sources + (bootloader runner + generated verification/boot image data) and has run + live under the supervised harness: the first real boot succeeded (see the + section below). - **The full CUDA driver-API smoke test passes with correct data**: the client completes `cuCtxCreate -> cuMemAlloc -> cuMemcpyHtoD -> cuMemcpyDtoH -> cuMemFree -> cuCtxDestroy` and prints `ALL OK`, with the 64-byte payload @@ -153,7 +155,12 @@ chain succeeds. (distribution driver unbound, power-domain locks held) rather than hand it back; a rescue path is being debugged (or, failing that, an approved reboot). A distribution-driver bus-master quirk on the claim path was also - found and fixed in this run. The booter and RISC-V start follow. + found and fixed in this run. The bootloader/RISC-V/resource-manager-init + stages are now implemented in the module sources (bootloader runner plus + generated boot and verification image data) and have now run live: the + boot completed through an active RISC-V core and the module received the + first real GSP-RM message (see the section below); a clean release is + pending the server shutdown RPC. - Host stability: the host hang class observed on 2026-09-28 (kernel stack overflow in a channel snapshot path) is fixed and frame-audited; repeated VM load/attach/teardown cycles run without panics or pstore records. A @@ -180,6 +187,46 @@ chain succeeds. work does not touch the driver binding and is unaffected. See [`docs/03-architecture/08-physical-gpu-integration.md`](docs/03-architecture/08-physical-gpu-integration.md). +## First real GSP boot + live GSP-RM session (2026-10-01) + +Cycle 09:53:27–09:54:59 on the supervised D0 harness (`--boot --keep`), +`open-iov-vfio.ko` sha256 `7815dacc…`, srcversion `2993AF5C…`. The boot order +observed on the physical TU104: + +1. **FWSEC** β€” WPR2 came up; FRTS error code 0; `WPR2_LO=0x01ffe000`, + `WPR2_HI=0x01ffee00`. +2. **Booter Load** on SEC2 β€” halted with `M0=0x0` after 146 ms. +3. **RISC-V active** β€” `RISCV_STATUS=0x00000035` (active bit 0), + `FALCON_OS=0`; secure scratch14 raw `0x11000000` (booter-load handoff bit + 24 = 1, booter binary version 1). +4. **First real GSP-RM message** β€” the staged status queue linked and the + server's first event was `function=0x1002` + (`NV_VGPU_MSG_EVENT_GSP_RUN_CPU_SEQUENCER`, `rpc_result=0`), carrying a + command buffer the host must execute; `GSP_INIT_DONE` (0x1001) follows only + after that sequencer runs and was not observed in the 60 s poll. + +This is the first host↔real-GSP-RM exchange on the physical path. A clean +release is not yet possible: the Booter Unload gate requires all four secure +scratch14 handoff bits (24–27); a host-driven boot sets only bit 24, and bit +27 is set by the server's shutdown path (host RPC `UNLOADING_GUEST_DRIVER`, +not yet implemented). The host-side seed/unlock alternative is fuse-locked on +this part. The session is therefore intentionally left claimed (`--keep`): the +GPU is bound to `open-iov-gsp-phys` and runtime-active; `nvidia` stays unbound +(host headless; `nvidia-smi` fails by design); host healthy, pstore empty. A +fresh live cycle needs a power cycle. + +**Next steps:** (a) execute the 0x1002 sequencer command buffer in the module +and log 0x1001 as `stage 3e complete`; (b) implement the command-queue TX path +and send `UNLOADING_GUEST_DRIVER` on release so teardown passes the handoff +gate and the GPU can be handed back without a reboot; (c) reboot (with +approval, handoff first) when ready to test the next live cycle. + +Ending the claimed session (harness notice): `sudo rmmod open_iov_vfio`, then +`echo auto | sudo tee /sys/bus/pci/devices/0000:01:00.0/power/control` and the +same for `0000:00:01.0`, then `sudo systemctl start nvidia-persistenced`. +Because the shutdown RPC is missing, a plain release cannot pass the handoff +gate, so the reliable route back to the distribution driver is a power cycle. + ## Initialization, work and teardown path Status tags: `[done]`, `[frontier]` (actively worked), `[partial]`, -- cgit v1.2.3