diff options
| -rw-r--r-- | STATE-CURRENT.md | 83 |
1 files changed, 67 insertions, 16 deletions
diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md index 622f997..fcd9957 100644 --- a/STATE-CURRENT.md +++ b/STATE-CURRENT.md @@ -1,6 +1,6 @@ # Open-IOV Current State -> **Snapshot: 2026-10-01 23:12.** This is a dated public status page, not live +> **Snapshot: 2026-10-02 01:25.** This is a dated public status page, not live > state; it is refreshed on every progress check. Live working notes are > maintained outside this repository; the public known-knowns/unknowns document > is authoritative here: @@ -31,7 +31,7 @@ meant to execute on the physical GPU. | `cuCtxCreate` returns | ✅ (the whole smoke test now completes) | | `cuMemAlloc` + `cuMemcpy` roundtrip (`ALL OK` in the smoke test) | ✅ payload roundtrip verified 64/64 in the guest | | Full `nvidia-smi` output / kernel launch | 🟡 the static runtime now initializes end-to-end with **real releases** (`cudaGetDeviceCount`/`cudaSetDevice`/3×`cudaMalloc`/2×`cudaMemcpyHtoD`/`kernel launch`/`cudaGetLastError` all `rc=0`) and queues a kernel launch; execution awaits the backend | -| CUDA kernel execution (`vectorAdd`) | 🟡 execution backend: physical ownership + firmware staging verified; the unbind/rebind freeze hazard is mitigated by the supervised D0 power-domain lock (3/3 green claim cycles), and the real FB size (8 GiB) and boot layout are verified live. The supervised live secure-verification run raised the protected region (WPR2 up) for the first time, then proved a secure-verification-only carve is not host-recoverable on this fused part, so the fail-safe held the device and an approved reboot restored it (distribution driver rebound, GPU healthy). That one-way stage is now retired: the only device-write path is a single full boot lifecycle (secure verification -> bootloader -> RISC-V start -> resource-manager initialization), whose code is implemented in the module sources and has now run live: the first real boot raised WPR2, passed Booter Load and went RISC-V active, and the module received the first real GSP-RM message (event 0x1002); init-done (0x1001) and a clean release are the next steps (see the 2026-10-01 section). A second live boot on 2026-10-01 proved the full boot sequence again (protected region up, bootloader halt, RISC-V active, first GSP-RM message) and reached the resource-manager's inline CPU-sequencer request; the host executor rejected the real request because of an over-strict internal size cap, which was found with a raw read of the message in host memory, fixed, and rebuilt. Cycles 2-3 then ran the real sequencer and received init-done with an RM init failure (`0x59`), which reproduces on a verified matching firmware image; the server's own shutdown supplied the last handoff bit, so each failed cycle +| CUDA kernel execution (`vectorAdd`) | 🟡 execution backend: physical ownership + firmware staging verified; the unbind/rebind freeze hazard is mitigated by the supervised D0 power-domain lock (3/3 green claim cycles; the 2026-10-01 P1 mdev-owned release path initially ran outside that pre-lock window and re-armed D3cold, freezing the host at 22:32:53 - the G0 fix bundle now latches the D0/D3cold locks by default; see the frontier section), and the real FB size (8 GiB) and boot layout are verified live. The supervised live secure-verification run raised the protected region (WPR2 up) for the first time, then proved a secure-verification-only carve is not host-recoverable on this fused part, so the fail-safe held the device and an approved reboot restored it (distribution driver rebound, GPU healthy). That one-way stage is now retired: the only device-write path is a single full boot lifecycle (secure verification -> bootloader -> RISC-V start -> resource-manager initialization), whose code is implemented in the module sources and has now run live: the first real boot raised WPR2, passed Booter Load and went RISC-V active, and the module received the first real GSP-RM message (event 0x1002); init-done (0x1001) and a clean release are the next steps (see the 2026-10-01 section). A second live boot on 2026-10-01 proved the full boot sequence again (protected region up, bootloader halt, RISC-V active, first GSP-RM message) and reached the resource-manager's inline CPU-sequencer request; the host executor rejected the real request because of an over-strict internal size cap, which was found with a raw read of the message in host memory, fixed, and rebuilt. Cycles 2-3 then ran the real sequencer and received init-done with an RM init failure (`0x59`), which reproduces on a verified matching firmware image; the server's own shutdown supplied the last handoff bit, so each failed cycle completed a **clean release without a reboot**. Cycles 4-10 then captured the resource-manager's own error journal and root-caused the failure: the boot queue was missing the two pre-boot initialization messages the matching @@ -69,7 +69,7 @@ meant to execute on the physical GPU. and destination equal to source. Every cycle released the device cleanly with no reboot; the next rung is wiring the live physical session into the guest-facing mdev/vfio path - ([`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md)). The guest-wiring rung then advanced: an inert backend-session API was added, and the mediated-device-owned lifecycle (P1, see the frontier section) was implemented and exercised - deferring the physical claim/boot to the first device create, releasing it on the last remove, with fail-open behaviour. | + ([`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md)). The guest-wiring rung then advanced: an inert backend-session API was added, and the mediated-device-owned lifecycle (P1, see the frontier section) was implemented and exercised - deferring the physical claim/boot to the first device create, releasing it on the last remove, with fail-open behaviour. The first P1 acceptance cycle exposed two defects - an asynchronous boot marked READY after a failed FWSEC boot, and the P1 release path freezing the host (see the frontier section). The G0 fix bundle (default-latched D0/D3cold locks, post-rebind readiness gate, pre-claim D0 settle, READY gated on a proven boot, pre-kick staging readback) is implemented, built, smoke-tested and now **validated live**: two module-owned supervised cycles (2026-10-02 ~00:57-00:58) were 2/2 green through claim, FWSEC, booter, RISC-V active, init-done (`0x1001`, `rpc_result=0` in ~955 ms) and a latched release with WPR2 down, exactly 5 `power domain left LOCKED` lines, 0 in-window rail cycles, host alive and pstore empty. The retested P1 mdev-owned path - the exact path that hard-froze on 2026-10-01 - now fails open safely (the READY gate releases the claim when no live GSP-RM session exists; no freeze), while the async FWSEC boot defect (`staging not visible before kick` -> `FWSEC/FRTS execution failed: -5`) remains open, quarantined off the critical path; guest wiring proceeds on the module-owned live session. The bundle is still uncommitted. | | Graphics (3D), video | ❌ | | Faults, multi-version, Windows, migration, multi-tenancy, performance | ❌ | @@ -81,19 +81,70 @@ chain succeeds. ## Right now (frontier) -- **Guest-wiring lifecycle (P1) implemented and exercised (2026-10-01 22:22).** - A module parameter selects a lifecycle where the physical-GPU claim and boot - are deferred to the first mediated device create and released on the last - remove, run asynchronously and **fail-open**: if the backend boot fails the - device keeps serving the emulation path. An acceptance cycle verified the - deferred claim, the fail-open behaviour (the boot failed at the secure - verification stage, yet the smoke test still passed `ALL OK` payload 64/64 - while the backend stayed off), and the release on remove. Two follow-ups - opened: the asynchronous boot failed where the supervised cycles had - succeeded, and the release/rebind was followed by a host power-management - freeze of the known kind (the GPU power-gating hazard), which needed an - unplanned reboot; the release path must keep the board powered through the - driver resume. No guest work is forwarded to the physical engine yet. +- **Guest-facing identity wiring (P2) validated live (2026-10-02 + 01:11-01:15).** On a module-owned live GSP-RM session the emulation now + overlays the server's real static configuration into the guest-facing + identity path: the guest's `nvidia-smi -L` reports `NVIDIA GeForce RTX 2080 + Super with Max-Q Design` (with a synthetic UUID) instead of the generic + emulated identity, while the full driver-API smoke test still passes `ALL + OK`. The cycle ran under the supervised harness and released with the + latched locks (protected region down, distribution driver rebound, 5 power + locks intentionally left held, 0 in-window rail cycles, host stable, no + freeze, pstore empty). This is the first live data path of the guest-wiring + rung; a control run with the live overlay disabled is in progress (outcome + pending). Guest work is still **not** forwarded to the physical engine - + the submission/relay wiring is the next rung. The bundle (the default-latched + D0/D3cold freeze fix plus the identity overlay) is implemented, built and + exercised live, but not yet committed. +- **G0 fix bundle built after the P1 release-path freeze (2026-10-02 00:55).** + The first P1 mdev-owned acceptance cycle ended in the known hard freeze + (journal stops, pstore empty): the P1 release path ran outside the supervised + D0 pre-lock window and the teardown restored `power/control=auto`, re-arming + D3cold; the resume ladder ran 22:32:11-22:32:44 and the host froze at + 22:32:53. The same cycle's first async boot failed at FWSEC - the staging DMA + never landed after an RTD3 rail cycle - yet the backend was marked READY. The + G0 bundle: a new `gsp_phys_repower` parameter (default 0) makes the power + unlock latch the D0/D3cold locks instead of releasing them; release gains a + 10 s post-rebind readiness gate; claim gains a 2 s D0 settle/verify before + the unbind; READY is granted only when the boot proves a live RM session; + FWSEC reads back the IMEM/DMEM staging before the kick; and the claim + harness leaves the board latched by default (`--restore-power` opts out). + Implemented, rebuilt clean, two clean no-param insmod/rmmod smoke cycles at + 00:55, then **validated live** (2026-10-02 ~00:57-00:58): two module-owned + supervised claim/boot cycles were 2/2 green - claim, FWSEC, booter, RISC-V + active, init-done (`0x1001`, `rpc_result=0` in ~955 ms), WPR2 down, + `released 0000:01:00.0; rebound to nvidia (rc=1)`, exactly 5 `power domain + left LOCKED` lines, 0 in-window rail cycles, host survived, pstore empty; + module sha256 `dde36d92...`. Uncommitted. +- **Guest-wiring lifecycle (P1) implemented and exercised; retested after G0 + (2026-10-02 ~00:58).** A module parameter selects a lifecycle where the + physical-GPU claim and boot are deferred to the first mediated device create + and released on the last remove, run asynchronously and **fail-open**: if the + backend boot fails the device keeps serving the emulation path. The first + acceptance cycle verified the deferred claim, the fail-open behaviour (the + boot failed at the secure verification stage, yet the smoke test still passed + `ALL OK` payload 64/64 while the backend stayed off), and the release on + remove; the release/rebind was followed by a host power-management freeze of + the known kind (the GPU power-gating hazard), which needed an unplanned + reboot; the release path now latches the D0/D3cold locks by default (the new + `gsp_phys_repower=0` default in the G0 bundle above), keeping the board + powered through the driver resume. Retested with the fixed module, the exact + path that hard-froze now **fails open safely**: the first mdev create + schedules the async claim, the new READY gate finds no live GSP-RM session + (`rm_init_done=0 unload_submitted=0`) and releases the claim latched (5 + `power domain left LOCKED`, `released 0000:01:00.0; rebound to nvidia + (rc=1)`); the host survives, no freeze. The async FWSEC boot itself still + fails (`staging not visible before kick` - DMEM0/IMEM all zero, the falcon + is not accepting DMA - aborting before STARTCPU -> `FWSEC/FRTS execution + failed: -5`), surfaced in milliseconds by the new pre-kick readback instead + of the old 5 s halt timeout; it is an open, fail-open-safe defect off the + critical path (an isolation experiment ruled out the obvious hypotheses - + stopping `nvidia-persistenced` and setting `power/control=on` plus waiting + runtime active before mdev create gives the identical signature, and the + module-owned path succeeds with the very same preconditions in the same + boot). Guest wiring proceeds on the module-owned live session + (`gsp_phys_gpu=1` claim+boot at load, proven); the P1 async claim stays + fail-open. No guest work is forwarded to the physical engine yet. - **Guest-wiring rung opened (2026-10-01 22:00): the first backend-session API code is in the tree, inert.** A skeleton that exposes the live physical session to the device layer (session request entry points plus a static-info |