From 66172eaba3cf6ff08a5cd4b7080e05704fa3813f Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Thu, 1 Oct 2026 10:07:15 -0400 Subject: docs: record the first real GSP boot and the live GSP-RM session STATE-CURRENT and the architecture docs now describe the observed order (FWSEC -> Booter Load -> RISC-V active -> event 0x1002 run-sequencer -> 0x1001 init-done), the four-handoff-bit clean-release constraint, and the --boot/--keep harness contract. --- STATE-CURRENT.md | 57 ++++++++++++-- docs/03-architecture/03-gsp-and-gsp-rm.md | 10 +++ .../03-architecture/08-physical-gpu-integration.md | 86 +++++++++++++++------- 3 files changed, 123 insertions(+), 30 deletions(-) diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md index 6771f53..c86311b 100644 --- a/STATE-CURRENT.md +++ b/STATE-CURRENT.md @@ -1,6 +1,6 @@ # Open-IOV Current State -> **Snapshot: 2026-10-01 09:12.** This is a dated public status page, not live +> **Snapshot: 2026-10-01 09:59.** This is a dated public status page, not live > state; it is refreshed on every progress check. Live working notes are > maintained outside this repository; the public known-knowns/unknowns document > is authoritative here: @@ -31,7 +31,7 @@ meant to execute on the physical GPU. | `cuCtxCreate` returns | βœ… (the whole smoke test now completes) | | `cuMemAlloc` + `cuMemcpy` roundtrip (`ALL OK` in the smoke test) | βœ… payload roundtrip verified 64/64 in the guest | | Full `nvidia-smi` output / kernel launch | 🟑 the static runtime now initializes end-to-end with **real releases** (`cudaGetDeviceCount`/`cudaSetDevice`/3Γ—`cudaMalloc`/2Γ—`cudaMemcpyHtoD`/`kernel launch`/`cudaGetLastError` all `rc=0`) and queues a kernel launch; execution awaits the backend | -| CUDA kernel execution (`vectorAdd`) | 🟑 execution backend: physical ownership + firmware staging verified; the unbind/rebind freeze hazard is mitigated by the supervised D0 power-domain lock (3/3 green claim cycles), and the real FB size (8 GiB) and boot layout are verified live. The supervised live secure-verification run raised the protected region (WPR2 up) for the first time, then proved a secure-verification-only carve is not host-recoverable on this fused part, so the fail-safe held the device and an approved reboot restored it (distribution driver rebound, GPU healthy). That one-way stage is now retired: the only device-write path is a single full boot lifecycle (secure verification -> bootloader -> RISC-V start -> resource-manager initialization), which is being implemented. | +| CUDA kernel execution (`vectorAdd`) | 🟑 execution backend: physical ownership + firmware staging verified; the unbind/rebind freeze hazard is mitigated by the supervised D0 power-domain lock (3/3 green claim cycles), and the real FB size (8 GiB) and boot layout are verified live. The supervised live secure-verification run raised the protected region (WPR2 up) for the first time, then proved a secure-verification-only carve is not host-recoverable on this fused part, so the fail-safe held the device and an approved reboot restored it (distribution driver rebound, GPU healthy). That one-way stage is now retired: the only device-write path is a single full boot lifecycle (secure verification -> bootloader -> RISC-V start -> resource-manager initialization), whose code is implemented in the module sources and has now run live: the first real boot raised WPR2, passed Booter Load and went RISC-V active, and the module received the first real GSP-RM message (event 0x1002); init-done (0x1001) and a clean release are the next steps (see the 2026-10-01 section). | | Graphics (3D), video | ❌ | | Faults, multi-version, Windows, migration, multi-tenancy, performance | ❌ | @@ -58,8 +58,10 @@ chain succeeds. host healthy, no crash record). The one-way stage is now retired: the only device-write path is a single supervised full boot lifecycle (secure verification -> bootloader -> RISC-V start -> resource-manager - initialization), whose next stage is being implemented so the normal boot - lifecycle makes teardown valid. + initialization). That lifecycle is now implemented in the module sources + (bootloader runner + generated verification/boot image data) and has run + live under the supervised harness: the first real boot succeeded (see the + section below). - **The full CUDA driver-API smoke test passes with correct data**: the client completes `cuCtxCreate -> cuMemAlloc -> cuMemcpyHtoD -> cuMemcpyDtoH -> cuMemFree -> cuCtxDestroy` and prints `ALL OK`, with the 64-byte payload @@ -153,7 +155,12 @@ chain succeeds. (distribution driver unbound, power-domain locks held) rather than hand it back; a rescue path is being debugged (or, failing that, an approved reboot). A distribution-driver bus-master quirk on the claim path was also - found and fixed in this run. The booter and RISC-V start follow. + found and fixed in this run. The bootloader/RISC-V/resource-manager-init + stages are now implemented in the module sources (bootloader runner plus + generated boot and verification image data) and have now run live: the + boot completed through an active RISC-V core and the module received the + first real GSP-RM message (see the section below); a clean release is + pending the server shutdown RPC. - Host stability: the host hang class observed on 2026-09-28 (kernel stack overflow in a channel snapshot path) is fixed and frame-audited; repeated VM load/attach/teardown cycles run without panics or pstore records. A @@ -180,6 +187,46 @@ chain succeeds. work does not touch the driver binding and is unaffected. See [`docs/03-architecture/08-physical-gpu-integration.md`](docs/03-architecture/08-physical-gpu-integration.md). +## First real GSP boot + live GSP-RM session (2026-10-01) + +Cycle 09:53:27–09:54:59 on the supervised D0 harness (`--boot --keep`), +`open-iov-vfio.ko` sha256 `7815dacc…`, srcversion `2993AF5C…`. The boot order +observed on the physical TU104: + +1. **FWSEC** β€” WPR2 came up; FRTS error code 0; `WPR2_LO=0x01ffe000`, + `WPR2_HI=0x01ffee00`. +2. **Booter Load** on SEC2 β€” halted with `M0=0x0` after 146 ms. +3. **RISC-V active** β€” `RISCV_STATUS=0x00000035` (active bit 0), + `FALCON_OS=0`; secure scratch14 raw `0x11000000` (booter-load handoff bit + 24 = 1, booter binary version 1). +4. **First real GSP-RM message** β€” the staged status queue linked and the + server's first event was `function=0x1002` + (`NV_VGPU_MSG_EVENT_GSP_RUN_CPU_SEQUENCER`, `rpc_result=0`), carrying a + command buffer the host must execute; `GSP_INIT_DONE` (0x1001) follows only + after that sequencer runs and was not observed in the 60 s poll. + +This is the first host↔real-GSP-RM exchange on the physical path. A clean +release is not yet possible: the Booter Unload gate requires all four secure +scratch14 handoff bits (24–27); a host-driven boot sets only bit 24, and bit +27 is set by the server's shutdown path (host RPC `UNLOADING_GUEST_DRIVER`, +not yet implemented). The host-side seed/unlock alternative is fuse-locked on +this part. The session is therefore intentionally left claimed (`--keep`): the +GPU is bound to `open-iov-gsp-phys` and runtime-active; `nvidia` stays unbound +(host headless; `nvidia-smi` fails by design); host healthy, pstore empty. A +fresh live cycle needs a power cycle. + +**Next steps:** (a) execute the 0x1002 sequencer command buffer in the module +and log 0x1001 as `stage 3e complete`; (b) implement the command-queue TX path +and send `UNLOADING_GUEST_DRIVER` on release so teardown passes the handoff +gate and the GPU can be handed back without a reboot; (c) reboot (with +approval, handoff first) when ready to test the next live cycle. + +Ending the claimed session (harness notice): `sudo rmmod open_iov_vfio`, then +`echo auto | sudo tee /sys/bus/pci/devices/0000:01:00.0/power/control` and the +same for `0000:00:01.0`, then `sudo systemctl start nvidia-persistenced`. +Because the shutdown RPC is missing, a plain release cannot pass the handoff +gate, so the reliable route back to the distribution driver is a power cycle. + ## Initialization, work and teardown path Status tags: `[done]`, `[frontier]` (actively worked), `[partial]`, diff --git a/docs/03-architecture/03-gsp-and-gsp-rm.md b/docs/03-architecture/03-gsp-and-gsp-rm.md index fe51838..7f206ba 100644 --- a/docs/03-architecture/03-gsp-and-gsp-rm.md +++ b/docs/03-architecture/03-gsp-and-gsp-rm.md @@ -54,6 +54,16 @@ Our emulation implements stage completion rather than the stages themselves β€” which is legitimate because the *only observable* to the driver is the state machine it polls. +On the physical path (2026-10-01), the same handshake ran against real +firmware: FWSEC raised WPR2 with a zero FRTS error code, the SEC2 booter halted +with mailbox0 = 0, the RISC-V core reported active (`RISCV_STATUS=0x00000035`, +active bit 0) and `FALCON_OS` read 0. The server's first message was then a +status-queue event, function `0x1002` (`NV_VGPU_MSG_EVENT_GSP_RUN_CPU_SEQUENCER`, +`rpc_result=0`), carrying a command buffer the host must execute; +`GSP_INIT_DONE` (`0x1001`) follows only after that sequencer runs. The +host-side queue pair behind the exchange is one shared page table plus a 256 KB +command queue and a 256 KB status queue of 4 KB elements. + ```mermaid sequenceDiagram autonumber diff --git a/docs/03-architecture/08-physical-gpu-integration.md b/docs/03-architecture/08-physical-gpu-integration.md index 5c9e385..0c18e84 100644 --- a/docs/03-architecture/08-physical-gpu-integration.md +++ b/docs/03-architecture/08-physical-gpu-integration.md @@ -1,15 +1,16 @@ # Physical GPU Integration: How Work Reaches Real Hardware > **Status: mechanism chosen; ownership + firmware staging implemented; -> power-domain lock implemented, pending supervised verification; real GSP -> boot not started.** The control plane works today: the guest's driver -> initializes and the CUDA driver-API baseline passes end to end (context -> creation, allocation, copies, 64/64 payload round-trip), and the guest's -> decoded work is executed in software as a development bridge. This document -> records the interfaces that actually exist, the mechanism options, the -> chosen path, the platform power-domain hazard the ownership path exposes on -> the development host (Β§4.2), and what has to be built. It fills the gap -> left open by +> power-domain lock verified; first real GSP boot and first live GSP-RM +> exchange complete (2026-10-01); host-side initialization follow-up and a +> clean release are in progress.** The control plane works today: the guest's +> driver initializes and the CUDA driver-API baseline passes end to end +> (context creation, allocation, copies, 64/64 payload round-trip), and the +> guest's decoded work is executed in software as a development bridge. This +> document records the interfaces that actually exist, the mechanism options, +> the chosen path, the platform power-domain hazard the ownership path +> exposes on the development host (Β§4.2), and what has to be built. It fills +> the gap left open by > [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) Β§5 > and [`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) Β§3, > which named "the host driver's RM API" as an option without a concrete call @@ -150,12 +151,9 @@ that boot: an out-of-band platform power cut bypasses the lock and would produce the identical signature, and retrying would be a second dice roll on a host that has already lost the board. -The lock is implemented in the ownership path and awaits supervised -verification (Β§6, Β§8); a claim is refused unless both the claim parameter and -the hazard acknowledgement are set (Β§8). The exact resource names, controller -commands, captured timelines and the mitigation's code-level design live in -the adjacent local analysis notes (`kernel/LOCAL-GSP-BOOT.md`); the -adjacent document is part of the local analysis. +The lock is implemented in the ownership path and verified across three +supervised claim/release cycles on 2026-10-01 (Β§6, Β§8); a claim is refused +unless both the claim parameter and the hazard acknowledgement are set (Β§8). ### 4.3 Device bring-up @@ -189,8 +187,28 @@ The bring-up is staged so each step is reversible and independently verified: Stages 1 and 2 are implemented in the module and verified live (a claim/release cycle leaves the host driver working on the clean cycles; the staged image -parses as expected). The power-domain lock of Β§4.2 guards the claim window; -stage 3 β€” the real boot β€” has not been started. +parses as expected). The power-domain lock of Β§4.2 guards the claim window and +is verified across three supervised cycles. Stage 3 has now run live +(2026-10-01), in the observed order **FWSEC β†’ Booter Load β†’ RISC-V active β†’ +first server event**: FWSEC raised WPR2 with a zero FRTS error code +(`WPR2_LO=0x01ffe000`, `WPR2_HI=0x01ffee00`), the SEC2 Booter Load halted with +`M0=0` after 146 ms, the RISC-V core reported active +(`RISCV_STATUS=0x00000035`) with `FALCON_OS=0`, and a host-driven boot sets +only the booter-load handoff bit (bit 24) in secure scratch14. Stage 4 has +begun: the staged command/status queues linked and the first real GSP-RM +message arrived (event 0x1002, run-CPU-sequencer, `rpc_result=0`). The server +waits for the host to execute the sequencer command buffer carried in that +event before it raises `GSP_INIT_DONE` (0x1001). **A plain release is not yet +possible:** the Booter Unload gate requires all four secure scratch14 handoff +bits (24–27); a +host-driven boot sets only bit 24, and bit 27 is set by the server's shutdown +path (host RPC `UNLOADING_GUEST_DRIVER`), which is not implemented, while the +host-side seed/unlock alternative is fuse-locked on this part. Until that RPC +lands, a green session is left claimed with the harness `--keep` flag and +returning the GPU to the distribution driver takes a power cycle. The +supervised harness modes are `--boot` (the only write mode: full boot +lifecycle), `--boot-dryrun` (zero-write staging rehearsal) and `--keep` +(leave a verified claimed session running); `--fwsec` is retired. ### 4.4 Host-side resource-manager client @@ -245,9 +263,9 @@ per-context scheduling. Tenant policy and fairness are a later layer on top. | Driver-API baseline (init, devices, context, allocation, copies, free, destroy) | done β€” re-verified 64/64 payload round-trip on the current build | | Software-executed bridge for decoded memory/state operations | in progress (memsets, copies, releases) | | Device ownership + firmware staging | verified β€” claim/release and staged-image parse both verified live; the claim path is hazard-gated (Β§4.2) | -| Power-domain lock across the claim window | implemented β€” pending supervised verification (Β§4.2, Β§8) | -| Real GSP boot by the module | not started | -| Host-side resource-manager client for real objects/channels | not started | +| Power-domain lock across the claim window | verified β€” 3/3 supervised claim/release cycles (2026-10-01; Β§4.2, Β§8) | +| Real GSP boot by the module | first live boot 2026-10-01 β€” FWSEC WPR2 up, Booter Load `M0=0`, RISC-V active; clean release pending the server shutdown RPC | +| Host-side resource-manager client for real objects/channels | transport alive 2026-10-01 β€” first real GSP-RM event (0x1002) received; init-done (0x1001) awaits host sequencer execution | | Work translation and real submission | not started | | Completion relay through real hardware | not started | | CUDA kernel execution (`vectorAdd`) | not started | @@ -265,11 +283,11 @@ per-context scheduling. Tenant policy and fairness are a later layer on top. controller action that does not go through the root port's D3 path β€” would produce the identical power-loss signature; the claim must abort on that signature and must not be retried in the same boot. -* **Supervised verification of the lock:** the lock is implemented but not yet - empirically verified. The first claim/release with it active must be - supervised: explicit human go-ahead, other host work warned, a written - recovery plan in place, a held-open window long enough to prove the board - stayed powered, and pstore armed. +* **Supervised operation of the lock:** the lock is implemented and verified + across three supervised claim/release cycles (2026-10-01). Every claim + remains supervised: explicit human go-ahead, other host work warned, a + written recovery plan in place, a held-open window long enough to prove the + board stayed powered, and pstore armed. ## 8. Decision record @@ -289,6 +307,24 @@ per-context scheduling. Tenant policy and fairness are a later layer on top. plan. The claim path refuses without the acknowledgement. At the first power-loss signature: abort and do not retry in that boot; a host reboot requires explicit human approval and a written recovery plan updated first. +* **Update (2026-10-01) β€” FWSEC-only retired:** stopping after the + secure-verification carve alone is a one-way door on this fused part (the + protected region is left up and is not host-recoverable). The supervised + claim harness therefore refuses `--fwsec` before any module load. The only + device-write mode is `--boot` (`gsp_phys_boot=1`, the full boot lifecycle + through resource-manager init); the zero-write rehearsal is `--boot-dryrun`. + Never stop after FWSEC alone on this host. +* **Update (2026-10-01) β€” first live boot; clean release gated:** the + supervised `--boot --keep` cycle completed the first real boot (FWSEC β†’ + Booter Load β†’ RISC-V active) and received the first real GSP-RM event + (0x1002, run-CPU-sequencer); `GSP_INIT_DONE` (0x1001) awaits the host + executing the sequencer buffer carried in that event. Clean release is not + yet possible: the Booter Unload gate requires all four secure scratch14 + handoff bits (24–27), a host-driven boot sets only bit 24, bit 27 is set by + the server's shutdown path (host RPC `UNLOADING_GUEST_DRIVER`, not + implemented), and host-side seed/unlock is fuse-locked on this part. The + live session is therefore left claimed (`--keep`) and a fresh cycle needs a + power cycle. See Β§4.3. ## 9. Interaction sequence (end state) -- cgit v1.2.3