diff options
| author | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-10-02 02:07:55 -0400 |
|---|---|---|
| committer | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-10-02 02:07:55 -0400 |
| commit | e950e404140a8adbcb7b178beb153d2216d23f57 (patch) | |
| tree | b71d83d3ff48a703f0b029b8a40491db997f81d3 | |
| parent | 129991a063cbef7cdfc7f10faafdb97107d15e14 (diff) | |
| download | open-iov-e950e404140a8adbcb7b178beb153d2216d23f57.tar.gz open-iov-e950e404140a8adbcb7b178beb153d2216d23f57.zip | |
STATE-CURRENT + docs: record the freeze fix and the P2/P3 guest-wiring landings
- STATE-CURRENT snapshot moves to the P3 frontier: the host-freeze latch
and readiness gates are validated, the static identity overlay and the
read-only control forwarding are landed and live-verified
- the physical-backend guest-wiring design is now marked P0-P3 implemented
with the P2/P3 acceptance results
| -rw-r--r-- | STATE-CURRENT.md | 60 | ||||
| -rw-r--r-- | docs/03-architecture/11-physical-backend-guest-wiring.md | 56 |
2 files changed, 89 insertions, 27 deletions
diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md index fcd9957..f02c742 100644 --- a/STATE-CURRENT.md +++ b/STATE-CURRENT.md @@ -1,6 +1,6 @@ # Open-IOV Current State -> **Snapshot: 2026-10-02 01:25.** This is a dated public status page, not live +> **Snapshot: 2026-10-02 01:55.** This is a dated public status page, not live > state; it is refreshed on every progress check. Live working notes are > maintained outside this repository; the public known-knowns/unknowns document > is authoritative here: @@ -69,7 +69,7 @@ meant to execute on the physical GPU. and destination equal to source. Every cycle released the device cleanly with no reboot; the next rung is wiring the live physical session into the guest-facing mdev/vfio path - ([`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md)). The guest-wiring rung then advanced: an inert backend-session API was added, and the mediated-device-owned lifecycle (P1, see the frontier section) was implemented and exercised - deferring the physical claim/boot to the first device create, releasing it on the last remove, with fail-open behaviour. The first P1 acceptance cycle exposed two defects - an asynchronous boot marked READY after a failed FWSEC boot, and the P1 release path freezing the host (see the frontier section). The G0 fix bundle (default-latched D0/D3cold locks, post-rebind readiness gate, pre-claim D0 settle, READY gated on a proven boot, pre-kick staging readback) is implemented, built, smoke-tested and now **validated live**: two module-owned supervised cycles (2026-10-02 ~00:57-00:58) were 2/2 green through claim, FWSEC, booter, RISC-V active, init-done (`0x1001`, `rpc_result=0` in ~955 ms) and a latched release with WPR2 down, exactly 5 `power domain left LOCKED` lines, 0 in-window rail cycles, host alive and pstore empty. The retested P1 mdev-owned path - the exact path that hard-froze on 2026-10-01 - now fails open safely (the READY gate releases the claim when no live GSP-RM session exists; no freeze), while the async FWSEC boot defect (`staging not visible before kick` -> `FWSEC/FRTS execution failed: -5`) remains open, quarantined off the critical path; guest wiring proceeds on the module-owned live session. The bundle is still uncommitted. | + ([`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md)). The guest-wiring rung then advanced: an inert backend-session API was added, and the mediated-device-owned lifecycle (P1, see the frontier section) was implemented and exercised - deferring the physical claim/boot to the first device create, releasing it on the last remove, with fail-open behaviour. The first P1 acceptance cycle exposed two defects - an asynchronous boot marked READY after a failed FWSEC boot, and the P1 release path freezing the host (see the frontier section). The G0 fix bundle (default-latched D0/D3cold locks, post-rebind readiness gate, pre-claim D0 settle, READY gated on a proven boot, pre-kick staging readback) is implemented, built, smoke-tested and now **validated live**: two module-owned supervised cycles (2026-10-02 ~00:57-00:58) were 2/2 green through claim, FWSEC, booter, RISC-V active, init-done (`0x1001`, `rpc_result=0` in ~955 ms) and a latched release with WPR2 down, exactly 5 `power domain left LOCKED` lines, 0 in-window rail cycles, host alive and pstore empty. The retested P1 mdev-owned path - the exact path that hard-froze on 2026-10-01 - now fails open safely (the READY gate releases the claim when no live GSP-RM session exists; no freeze), while the async FWSEC boot defect (`staging not visible before kick` -> `FWSEC/FRTS execution failed: -5`) remains open, quarantined off the critical path; guest wiring proceeds on the module-owned live session. The bundle is still uncommitted. Guest-wiring P2 (static-config identity overlay) has since landed and verified live (2026-10-02 01:20-01:45): with `gsp_phys_static_live=1` the guest's `nvidia-smi -q` Product Name comes from the live session (`NVIDIA GeForce RTX 2080 Super with Max-Q Design`) versus the emulated `NVIDIA GeForce RTX 2080 SUPER` with it off (A/B; overlay applied 4x/0), while the guest smoke stays GREEN `ALL OK` 64/64; one overlay-OFF full run hit a pre-existing emulated-path flake, watched. P3 (read-only control forwarding) has since landed and verified live (2026-10-02 ~01:33-01:45): the `gsp_phys_ctrl_forward` whitelist served 22 controls live with 2 fallbacks and 0 timeouts while the guest smoke stayed GREEN `ALL OK` 64/64; the next rung is P5 work forwarding + completion relay, with P4 (alloc/free handle adoption) deliberately deferred. | | Graphics (3D), video | ❌ | | Faults, multi-version, Windows, migration, multi-tenancy, performance | ❌ | @@ -81,21 +81,47 @@ chain succeeds. ## Right now (frontier) -- **Guest-facing identity wiring (P2) validated live (2026-10-02 - 01:11-01:15).** On a module-owned live GSP-RM session the emulation now - overlays the server's real static configuration into the guest-facing - identity path: the guest's `nvidia-smi -L` reports `NVIDIA GeForce RTX 2080 - Super with Max-Q Design` (with a synthetic UUID) instead of the generic - emulated identity, while the full driver-API smoke test still passes `ALL - OK`. The cycle ran under the supervised harness and released with the - latched locks (protected region down, distribution driver rebound, 5 power - locks intentionally left held, 0 in-window rail cycles, host stable, no - freeze, pstore empty). This is the first live data path of the guest-wiring - rung; a control run with the live overlay disabled is in progress (outcome - pending). Guest work is still **not** forwarded to the physical engine - - the submission/relay wiring is the next rung. The bundle (the default-latched - D0/D3cold freeze fix plus the identity overlay) is implemented, built and - exercised live, but not yet committed. +- **Read-only control forwarding (P3) landed and verified live (2026-10-02 + 01:33-01:45).** `GSP_RM_CONTROL` (fn76) forwards a whitelist of + side-effect-free queries (`BUS_GET_PCI_BAR_INFO`, `BUS_GET_INFO_V2`, + `FB_GET_INFO_V2`, `INTERNAL_GMMU_GET_STATIC_INFO`) from the live GSP-RM + under the new `gsp_phys_ctrl_forward` parameter (bool, default off). The + guest's emulated internal handles are translated to the snapshot's adopted + server handles before the forward, and the server's echoed params become the + guest reply; everything else falls back to the existing emulated handler + (fail-open), counted as forwarded/fallback/timeout. A/B on the full guest + smoke: forwarding ON stayed GREEN `ALL OK` 64/64 with counters ok=22 + fallback=2 timeout=0, zero reply-send failures and `nvidia-smi -q` working; + OFF showed zero forward counters and a healthy guest. Release ran latched as + designed: WPR2 down, exactly 5 power locks intentionally held, 0 in-window + rail cycles, host stable, no reboot. Module sha256 `30dfd311...`. **Next: + P5** work forwarding + completion relay (guest submissions shadow-executed + on the physical copy engine). **P4 is deliberately deferred**: the P2-P5 + plan itself states P5 wave 1 does not require it, and adopting real handles + would change guest-visible handles and risk the currently green guest path. + The P2/P3 bundles and the G0 freeze fix remain uncommitted in the working + tree. +- **Guest-facing identity wiring (P2) landed and verified live (2026-10-02 + 01:20-01:45).** On a module-owned live GSP-RM session the emulation + overlays the server's real static configuration identity into the + guest-facing `GET_GSP_STATIC_INFO` reply under the new + `gsp_phys_static_live` parameter (bool, default off): a bounded 2 s wait on + the READY-time snapshot, fail-open; the snapshot cache is generation-keyed + and invalidated on session changes. A/B on the full guest smoke: with the + overlay ON the guest's `nvidia-smi -q` Product Name is `NVIDIA GeForce RTX + 2080 Super with Max-Q Design` (live; overlay applied 4x) and with it OFF it + is `NVIDIA GeForce RTX 2080 SUPER` (overlay applied 0); the driver-API + smoke stays GREEN `ALL OK` 64/64. FB geometry, masks, engine caps, PDE + bases and handles remain emulated in P2. One overlay-OFF full run hit a + pre-existing emulated-path flake (the guest wedged in `cuInit` with + `rmapiControlCacheSet ... NV_ERR_INVALID_PARAMETER`; no host abort); a + subsequent OFF quick run was green, so it is not a P2 regression and is + being watched. Release ran latched as designed: WPR2 down, distribution + driver rebound, exactly 5 power locks intentionally left held, 0 in-window + rail cycles, host stable, no reboot. Module sha256 `69eac176...`. P3 + read-only control forwarding has since landed and verified live (bullet + above). Guest work is still **not** forwarded to the physical engine. The + P2 bundle and the G0 freeze fix remain uncommitted in the working tree. - **G0 fix bundle built after the P1 release-path freeze (2026-10-02 00:55).** The first P1 mdev-owned acceptance cycle ended in the known hard freeze (journal stops, pstore empty): the P1 release path ran outside the supervised diff --git a/docs/03-architecture/11-physical-backend-guest-wiring.md b/docs/03-architecture/11-physical-backend-guest-wiring.md index da440eb..55700b5 100644 --- a/docs/03-architecture/11-physical-backend-guest-wiring.md +++ b/docs/03-architecture/11-physical-backend-guest-wiring.md @@ -1,7 +1,12 @@ # Physical Backend Guest Wiring: Driving the Guest-Facing Emulation from the Live GSP-RM Session -> **Status: design for implementation (2026-10-01). No code from this design is -> in the tree yet.** The physical backend itself is live: the module-owned +> **Status: P0-P3 implemented (2026-10-02).** The backend API skeleton (P0), +> the mdev-owned lifecycle (P1, fail-open; the async FWSEC boot on that path +> is an open defect off the critical path), the static-config identity +> overlay (P2, verified live in an A/B run) and the read-only control +> forwarding whitelist (P3, default off, verified live: ok=22/fallback=2, +> smoke `ALL OK` 64/64) are in the tree. The physical +> backend itself is live: the module-owned > GSP-RM completes initialization on the real TU104, serves the full > static-config reply, and shuts down gracefully. The host resource-manager > client ladder has since allocated memory, a virtual-address-space object with @@ -19,7 +24,8 @@ > The design was written against the tree at commit `f8e5d3b`; line numbers are > advisory and have moved with later commits — anchor by function name when > re-grepping. Behaviour claims below are grounded in live evidence from the -> 2026-10-01 cycles; anything not yet exercised is labelled "not yet run". +> 2026-10-01 and 2026-10-02 cycles; anything not yet exercised is labelled +> "not yet run". --- @@ -93,12 +99,11 @@ ### 0.3 The gap -The two planes never touch. The guest's static answers, controls and objects -are emulated; the physical session's real handles, real FB geometry, real -engine caps and real work capability are only visible to debugfs. The bridge -must be added at exactly three places (see §2): the reply point -`gsp_rm_send()`, the inbound dispatch point `gsp_rm_handle_element()`, and the -lifecycle points in `main.c` / `gsp_phys_gpu.c`. +The two planes now meet only at the lifecycle seam (P0/P1, mdev-owned +claim/release) and the static-config identity overlay (P2); controls, objects, +FB geometry, engine caps and work are still emulated. The remaining bridge +must be added at the reply point `gsp_rm_send()`, the inbound dispatch point +`gsp_rm_handle_element()`, and the work-plane seams (see §2). --- @@ -181,7 +186,7 @@ generation counter` so a release cannot deadlock against a forward in flight. the existing release still quiesces, runs the FWSEC teardown and refuses to rebind with WPR2 up (`gsp_phys_gpu.c:939`). A reboot then needs explicit human approval plus a handoff update. -* **Static-config served before READY:** bounded wait (recommend 2 s) then +* **Static-config served before READY:** bounded wait (2 s, implemented) then pure emulation fallback, with a stat counter; never block the emulation thread longer than the guest's own RPC poll can tolerate. @@ -332,6 +337,11 @@ Two grounded warnings: Keep emulated caps until the backend can actually service the resulting demands (P4+). +**P2 status (2026-10-02):** the name, short-name and VBIOS identity fields are +now overlaid live behind `gsp_phys_static_live` (A/B verified); the FB +geometry, masks and engine caps above remain emulated, per these warnings, +until P4/P6. + ### 4.3 Proposed flow 1. At READY, cache the full fn65 payload (new bytes in the backend session, @@ -364,6 +374,9 @@ go-ahead, recovery plan, pstore, never unattended (`docs/03-architecture/08-physical-gpu-integration.md` §7/§8). ### P0 - Backend API skeleton (inert) +* **Status (2026-10-02): implemented.** Session state/refcount plus the locked + backend RPC and static-snapshot helpers are in the tree and refuse when the + backend is OFF/not-init-done/unloading. * **Scope:** new state/refcount/lock in `gsp_phys_gpu.c`; locked `backend_rpc()`/`static_snapshot()` in `gsp_phys_booter.c`; no caller changes; no behavior change. @@ -374,6 +387,12 @@ go-ahead, recovery plan, pstore, never unattended * **Rollback:** trivial. ### P1 - Lifecycle wiring (claim on mdev create, release on last remove) +* **Status (2026-10-02): implemented and exercised live; fail-open.** The + first mdev create starts the claim/boot asynchronously and the last remove + releases it. The async boot currently fails at FWSEC on this path + (`staging not visible before kick`) and the READY gate releases the claim + fail-open; the module-owned live session is the proven path (and what P2 + uses). * **Scope:** `main.c` create/remove/init_work/cleanup_work/exit ordering (§3 items 3-6); no live answers yet. * **Acceptance (supervised, harness `phys_claim_cycle.sh` style):** fresh @@ -392,6 +411,14 @@ go-ahead, recovery plan, pstore, never unattended only; `mdev` emits `-EOPNOTSUPP` from `backend_start()`. ### P2 - Static-config identity overlay +* **Status (2026-10-02): implemented and verified live (A/B).** The READY-time + snapshot cache and the `gsp_phys_static_live` parameter (bool, default 0) + are in the tree. With the overlay on, the guest `nvidia-smi -q` Product + Name is the live `NVIDIA GeForce RTX 2080 Super with Max-Q Design`; with it + off it is the emulated `NVIDIA GeForce RTX 2080 SUPER`; the driver-API + smoke stays green (`ALL OK`, payload 64/64). Only name, short name and + VBIOS identity are overlaid; FB geometry, masks, engine caps, PDE bases and + handles stay emulated. * **Scope:** cache snapshot at READY; overlay allow-list in `gsp_rm_build_static_info()`/`gsp_rm_handle_get_static_info()`; module parameter; stats. Bounded wait (2 s) then emulation fallback. @@ -404,6 +431,15 @@ go-ahead, recovery plan, pstore, never unattended * **Rollback:** parameter off. ### P3 - Read-only control forwarding (whitelist) +* **Status (2026-10-02): implemented and verified live (default off).** With + `gsp_phys_ctrl_forward=1` the whitelist (`BUS_GET_PCI_BAR_INFO`, + `BUS_GET_INFO_V2`, `FB_GET_INFO_V2`, `INTERNAL_GMMU_GET_STATIC_INFO`) is + forwarded over fn76 with the guest's emulated internal handles translated + to the snapshot's adopted server handles; the server's echoed params become + the guest reply, and everything else falls back to the emulated handler + (fail-open; counters forwarded/fallback/timeout). Live acceptance: guest + smoke GREEN `ALL OK` 64/64, counters ok=22 fallback=2 timeout=0, guest + `nvidia-smi -q` worked. * **Scope:** in `gsp_rm_handle_rm_control()` (`:2611`), before the per-cmd switch: for an allow-list of side-effect-free queries (FB/BUS/KGR/GMMU/ CE/display statics), call `backend_rpc()` with the **adopted internal |
