summaryrefslogtreecommitdiffstats
path: root/STATE-CURRENT.md
diff options
context:
space:
mode:
authorArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-02 01:24:26 -0400
committerArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-02 01:24:26 -0400
commitfe3055491bfa7d0753fae8069cf6ed30d1b668dd (patch)
treeb26d7cdbb331ed4757d55f8c4e6181ac90b6b7df /STATE-CURRENT.md
parentd02cb5ee46654c8dd0fd4ed16403ede0692c9ff8 (diff)
downloadopen-iov-fe3055491bfa7d0753fae8069cf6ed30d1b668dd.tar.gz
open-iov-fe3055491bfa7d0753fae8069cf6ed30d1b668dd.zip
STATE-CURRENT: 01:25 check-back - G0 freeze fix validated 2/2; P2 guest identity overlay live (GREEN); LIVE=0 control did not pass
Diffstat (limited to 'STATE-CURRENT.md')
-rw-r--r--STATE-CURRENT.md83
1 files changed, 67 insertions, 16 deletions
diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md
index 622f997..fcd9957 100644
--- a/STATE-CURRENT.md
+++ b/STATE-CURRENT.md
@@ -1,6 +1,6 @@
# Open-IOV Current State
-> **Snapshot: 2026-10-01 23:12.** This is a dated public status page, not live
+> **Snapshot: 2026-10-02 01:25.** This is a dated public status page, not live
> state; it is refreshed on every progress check. Live working notes are
> maintained outside this repository; the public known-knowns/unknowns document
> is authoritative here:
@@ -31,7 +31,7 @@ meant to execute on the physical GPU.
| `cuCtxCreate` returns | ✅ (the whole smoke test now completes) |
| `cuMemAlloc` + `cuMemcpy` roundtrip (`ALL OK` in the smoke test) | ✅ payload roundtrip verified 64/64 in the guest |
| Full `nvidia-smi` output / kernel launch | 🟡 the static runtime now initializes end-to-end with **real releases** (`cudaGetDeviceCount`/`cudaSetDevice`/3×`cudaMalloc`/2×`cudaMemcpyHtoD`/`kernel launch`/`cudaGetLastError` all `rc=0`) and queues a kernel launch; execution awaits the backend |
-| CUDA kernel execution (`vectorAdd`) | 🟡 execution backend: physical ownership + firmware staging verified; the unbind/rebind freeze hazard is mitigated by the supervised D0 power-domain lock (3/3 green claim cycles), and the real FB size (8 GiB) and boot layout are verified live. The supervised live secure-verification run raised the protected region (WPR2 up) for the first time, then proved a secure-verification-only carve is not host-recoverable on this fused part, so the fail-safe held the device and an approved reboot restored it (distribution driver rebound, GPU healthy). That one-way stage is now retired: the only device-write path is a single full boot lifecycle (secure verification -> bootloader -> RISC-V start -> resource-manager initialization), whose code is implemented in the module sources and has now run live: the first real boot raised WPR2, passed Booter Load and went RISC-V active, and the module received the first real GSP-RM message (event 0x1002); init-done (0x1001) and a clean release are the next steps (see the 2026-10-01 section). A second live boot on 2026-10-01 proved the full boot sequence again (protected region up, bootloader halt, RISC-V active, first GSP-RM message) and reached the resource-manager's inline CPU-sequencer request; the host executor rejected the real request because of an over-strict internal size cap, which was found with a raw read of the message in host memory, fixed, and rebuilt. Cycles 2-3 then ran the real sequencer and received init-done with an RM init failure (`0x59`), which reproduces on a verified matching firmware image; the server's own shutdown supplied the last handoff bit, so each failed cycle
+| CUDA kernel execution (`vectorAdd`) | 🟡 execution backend: physical ownership + firmware staging verified; the unbind/rebind freeze hazard is mitigated by the supervised D0 power-domain lock (3/3 green claim cycles; the 2026-10-01 P1 mdev-owned release path initially ran outside that pre-lock window and re-armed D3cold, freezing the host at 22:32:53 - the G0 fix bundle now latches the D0/D3cold locks by default; see the frontier section), and the real FB size (8 GiB) and boot layout are verified live. The supervised live secure-verification run raised the protected region (WPR2 up) for the first time, then proved a secure-verification-only carve is not host-recoverable on this fused part, so the fail-safe held the device and an approved reboot restored it (distribution driver rebound, GPU healthy). That one-way stage is now retired: the only device-write path is a single full boot lifecycle (secure verification -> bootloader -> RISC-V start -> resource-manager initialization), whose code is implemented in the module sources and has now run live: the first real boot raised WPR2, passed Booter Load and went RISC-V active, and the module received the first real GSP-RM message (event 0x1002); init-done (0x1001) and a clean release are the next steps (see the 2026-10-01 section). A second live boot on 2026-10-01 proved the full boot sequence again (protected region up, bootloader halt, RISC-V active, first GSP-RM message) and reached the resource-manager's inline CPU-sequencer request; the host executor rejected the real request because of an over-strict internal size cap, which was found with a raw read of the message in host memory, fixed, and rebuilt. Cycles 2-3 then ran the real sequencer and received init-done with an RM init failure (`0x59`), which reproduces on a verified matching firmware image; the server's own shutdown supplied the last handoff bit, so each failed cycle
completed a **clean release without a reboot**. Cycles 4-10 then captured
the resource-manager's own error journal and root-caused the failure: the
boot queue was missing the two pre-boot initialization messages the matching
@@ -69,7 +69,7 @@ meant to execute on the physical GPU.
and destination equal to source. Every cycle released the device cleanly
with no reboot; the next rung is wiring the live physical session into the
guest-facing mdev/vfio path
- ([`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md)). The guest-wiring rung then advanced: an inert backend-session API was added, and the mediated-device-owned lifecycle (P1, see the frontier section) was implemented and exercised - deferring the physical claim/boot to the first device create, releasing it on the last remove, with fail-open behaviour. |
+ ([`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md)). The guest-wiring rung then advanced: an inert backend-session API was added, and the mediated-device-owned lifecycle (P1, see the frontier section) was implemented and exercised - deferring the physical claim/boot to the first device create, releasing it on the last remove, with fail-open behaviour. The first P1 acceptance cycle exposed two defects - an asynchronous boot marked READY after a failed FWSEC boot, and the P1 release path freezing the host (see the frontier section). The G0 fix bundle (default-latched D0/D3cold locks, post-rebind readiness gate, pre-claim D0 settle, READY gated on a proven boot, pre-kick staging readback) is implemented, built, smoke-tested and now **validated live**: two module-owned supervised cycles (2026-10-02 ~00:57-00:58) were 2/2 green through claim, FWSEC, booter, RISC-V active, init-done (`0x1001`, `rpc_result=0` in ~955 ms) and a latched release with WPR2 down, exactly 5 `power domain left LOCKED` lines, 0 in-window rail cycles, host alive and pstore empty. The retested P1 mdev-owned path - the exact path that hard-froze on 2026-10-01 - now fails open safely (the READY gate releases the claim when no live GSP-RM session exists; no freeze), while the async FWSEC boot defect (`staging not visible before kick` -> `FWSEC/FRTS execution failed: -5`) remains open, quarantined off the critical path; guest wiring proceeds on the module-owned live session. The bundle is still uncommitted. |
| Graphics (3D), video | ❌ |
| Faults, multi-version, Windows, migration, multi-tenancy, performance | ❌ |
@@ -81,19 +81,70 @@ chain succeeds.
## Right now (frontier)
-- **Guest-wiring lifecycle (P1) implemented and exercised (2026-10-01 22:22).**
- A module parameter selects a lifecycle where the physical-GPU claim and boot
- are deferred to the first mediated device create and released on the last
- remove, run asynchronously and **fail-open**: if the backend boot fails the
- device keeps serving the emulation path. An acceptance cycle verified the
- deferred claim, the fail-open behaviour (the boot failed at the secure
- verification stage, yet the smoke test still passed `ALL OK` payload 64/64
- while the backend stayed off), and the release on remove. Two follow-ups
- opened: the asynchronous boot failed where the supervised cycles had
- succeeded, and the release/rebind was followed by a host power-management
- freeze of the known kind (the GPU power-gating hazard), which needed an
- unplanned reboot; the release path must keep the board powered through the
- driver resume. No guest work is forwarded to the physical engine yet.
+- **Guest-facing identity wiring (P2) validated live (2026-10-02
+ 01:11-01:15).** On a module-owned live GSP-RM session the emulation now
+ overlays the server's real static configuration into the guest-facing
+ identity path: the guest's `nvidia-smi -L` reports `NVIDIA GeForce RTX 2080
+ Super with Max-Q Design` (with a synthetic UUID) instead of the generic
+ emulated identity, while the full driver-API smoke test still passes `ALL
+ OK`. The cycle ran under the supervised harness and released with the
+ latched locks (protected region down, distribution driver rebound, 5 power
+ locks intentionally left held, 0 in-window rail cycles, host stable, no
+ freeze, pstore empty). This is the first live data path of the guest-wiring
+ rung; a control run with the live overlay disabled is in progress (outcome
+ pending). Guest work is still **not** forwarded to the physical engine -
+ the submission/relay wiring is the next rung. The bundle (the default-latched
+ D0/D3cold freeze fix plus the identity overlay) is implemented, built and
+ exercised live, but not yet committed.
+- **G0 fix bundle built after the P1 release-path freeze (2026-10-02 00:55).**
+ The first P1 mdev-owned acceptance cycle ended in the known hard freeze
+ (journal stops, pstore empty): the P1 release path ran outside the supervised
+ D0 pre-lock window and the teardown restored `power/control=auto`, re-arming
+ D3cold; the resume ladder ran 22:32:11-22:32:44 and the host froze at
+ 22:32:53. The same cycle's first async boot failed at FWSEC - the staging DMA
+ never landed after an RTD3 rail cycle - yet the backend was marked READY. The
+ G0 bundle: a new `gsp_phys_repower` parameter (default 0) makes the power
+ unlock latch the D0/D3cold locks instead of releasing them; release gains a
+ 10 s post-rebind readiness gate; claim gains a 2 s D0 settle/verify before
+ the unbind; READY is granted only when the boot proves a live RM session;
+ FWSEC reads back the IMEM/DMEM staging before the kick; and the claim
+ harness leaves the board latched by default (`--restore-power` opts out).
+ Implemented, rebuilt clean, two clean no-param insmod/rmmod smoke cycles at
+ 00:55, then **validated live** (2026-10-02 ~00:57-00:58): two module-owned
+ supervised claim/boot cycles were 2/2 green - claim, FWSEC, booter, RISC-V
+ active, init-done (`0x1001`, `rpc_result=0` in ~955 ms), WPR2 down,
+ `released 0000:01:00.0; rebound to nvidia (rc=1)`, exactly 5 `power domain
+ left LOCKED` lines, 0 in-window rail cycles, host survived, pstore empty;
+ module sha256 `dde36d92...`. Uncommitted.
+- **Guest-wiring lifecycle (P1) implemented and exercised; retested after G0
+ (2026-10-02 ~00:58).** A module parameter selects a lifecycle where the
+ physical-GPU claim and boot are deferred to the first mediated device create
+ and released on the last remove, run asynchronously and **fail-open**: if the
+ backend boot fails the device keeps serving the emulation path. The first
+ acceptance cycle verified the deferred claim, the fail-open behaviour (the
+ boot failed at the secure verification stage, yet the smoke test still passed
+ `ALL OK` payload 64/64 while the backend stayed off), and the release on
+ remove; the release/rebind was followed by a host power-management freeze of
+ the known kind (the GPU power-gating hazard), which needed an unplanned
+ reboot; the release path now latches the D0/D3cold locks by default (the new
+ `gsp_phys_repower=0` default in the G0 bundle above), keeping the board
+ powered through the driver resume. Retested with the fixed module, the exact
+ path that hard-froze now **fails open safely**: the first mdev create
+ schedules the async claim, the new READY gate finds no live GSP-RM session
+ (`rm_init_done=0 unload_submitted=0`) and releases the claim latched (5
+ `power domain left LOCKED`, `released 0000:01:00.0; rebound to nvidia
+ (rc=1)`); the host survives, no freeze. The async FWSEC boot itself still
+ fails (`staging not visible before kick` - DMEM0/IMEM all zero, the falcon
+ is not accepting DMA - aborting before STARTCPU -> `FWSEC/FRTS execution
+ failed: -5`), surfaced in milliseconds by the new pre-kick readback instead
+ of the old 5 s halt timeout; it is an open, fail-open-safe defect off the
+ critical path (an isolation experiment ruled out the obvious hypotheses -
+ stopping `nvidia-persistenced` and setting `power/control=on` plus waiting
+ runtime active before mdev create gives the identical signature, and the
+ module-owned path succeeds with the very same preconditions in the same
+ boot). Guest wiring proceeds on the module-owned live session
+ (`gsp_phys_gpu=1` claim+boot at load, proven); the P1 async claim stays
+ fail-open. No guest work is forwarded to the physical engine yet.
- **Guest-wiring rung opened (2026-10-01 22:00): the first backend-session
API code is in the tree, inert.** A skeleton that exposes the live physical
session to the device layer (session request entry points plus a static-info