summaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-02 02:07:55 -0400
committerArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-02 02:07:55 -0400
commitfdce9fec4d95e5e7141939d55e333e734122aa6e (patch)
tree8c430a5ab6f5dcd66d6d10e08560644ea93a47b7
parentc886760efda836b060fee03bfce852f16f04b5b8 (diff)
downloadopen-iov-fdce9fec4d95e5e7141939d55e333e734122aa6e.tar.gz
open-iov-fdce9fec4d95e5e7141939d55e333e734122aa6e.zip
STATE-CURRENT + docs: record the freeze fix and the P2/P3 guest-wiring landings
- STATE-CURRENT snapshot moves to the P3 frontier: the host-freeze latch and readiness gates are validated, the static identity overlay and the read-only control forwarding are landed and live-verified - the physical-backend guest-wiring design is now marked P0-P3 implemented with the P2/P3 acceptance results
-rw-r--r--STATE-CURRENT.md60
-rw-r--r--docs/03-architecture/11-physical-backend-guest-wiring.md56
2 files changed, 89 insertions, 27 deletions
diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md
index fcd9957..f02c742 100644
--- a/STATE-CURRENT.md
+++ b/STATE-CURRENT.md
@@ -1,6 +1,6 @@
# Open-IOV Current State
-> **Snapshot: 2026-10-02 01:25.** This is a dated public status page, not live
+> **Snapshot: 2026-10-02 01:55.** This is a dated public status page, not live
> state; it is refreshed on every progress check. Live working notes are
> maintained outside this repository; the public known-knowns/unknowns document
> is authoritative here:
@@ -69,7 +69,7 @@ meant to execute on the physical GPU.
and destination equal to source. Every cycle released the device cleanly
with no reboot; the next rung is wiring the live physical session into the
guest-facing mdev/vfio path
- ([`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md)). The guest-wiring rung then advanced: an inert backend-session API was added, and the mediated-device-owned lifecycle (P1, see the frontier section) was implemented and exercised - deferring the physical claim/boot to the first device create, releasing it on the last remove, with fail-open behaviour. The first P1 acceptance cycle exposed two defects - an asynchronous boot marked READY after a failed FWSEC boot, and the P1 release path freezing the host (see the frontier section). The G0 fix bundle (default-latched D0/D3cold locks, post-rebind readiness gate, pre-claim D0 settle, READY gated on a proven boot, pre-kick staging readback) is implemented, built, smoke-tested and now **validated live**: two module-owned supervised cycles (2026-10-02 ~00:57-00:58) were 2/2 green through claim, FWSEC, booter, RISC-V active, init-done (`0x1001`, `rpc_result=0` in ~955 ms) and a latched release with WPR2 down, exactly 5 `power domain left LOCKED` lines, 0 in-window rail cycles, host alive and pstore empty. The retested P1 mdev-owned path - the exact path that hard-froze on 2026-10-01 - now fails open safely (the READY gate releases the claim when no live GSP-RM session exists; no freeze), while the async FWSEC boot defect (`staging not visible before kick` -> `FWSEC/FRTS execution failed: -5`) remains open, quarantined off the critical path; guest wiring proceeds on the module-owned live session. The bundle is still uncommitted. |
+ ([`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md)). The guest-wiring rung then advanced: an inert backend-session API was added, and the mediated-device-owned lifecycle (P1, see the frontier section) was implemented and exercised - deferring the physical claim/boot to the first device create, releasing it on the last remove, with fail-open behaviour. The first P1 acceptance cycle exposed two defects - an asynchronous boot marked READY after a failed FWSEC boot, and the P1 release path freezing the host (see the frontier section). The G0 fix bundle (default-latched D0/D3cold locks, post-rebind readiness gate, pre-claim D0 settle, READY gated on a proven boot, pre-kick staging readback) is implemented, built, smoke-tested and now **validated live**: two module-owned supervised cycles (2026-10-02 ~00:57-00:58) were 2/2 green through claim, FWSEC, booter, RISC-V active, init-done (`0x1001`, `rpc_result=0` in ~955 ms) and a latched release with WPR2 down, exactly 5 `power domain left LOCKED` lines, 0 in-window rail cycles, host alive and pstore empty. The retested P1 mdev-owned path - the exact path that hard-froze on 2026-10-01 - now fails open safely (the READY gate releases the claim when no live GSP-RM session exists; no freeze), while the async FWSEC boot defect (`staging not visible before kick` -> `FWSEC/FRTS execution failed: -5`) remains open, quarantined off the critical path; guest wiring proceeds on the module-owned live session. The bundle is still uncommitted. Guest-wiring P2 (static-config identity overlay) has since landed and verified live (2026-10-02 01:20-01:45): with `gsp_phys_static_live=1` the guest's `nvidia-smi -q` Product Name comes from the live session (`NVIDIA GeForce RTX 2080 Super with Max-Q Design`) versus the emulated `NVIDIA GeForce RTX 2080 SUPER` with it off (A/B; overlay applied 4x/0), while the guest smoke stays GREEN `ALL OK` 64/64; one overlay-OFF full run hit a pre-existing emulated-path flake, watched. P3 (read-only control forwarding) has since landed and verified live (2026-10-02 ~01:33-01:45): the `gsp_phys_ctrl_forward` whitelist served 22 controls live with 2 fallbacks and 0 timeouts while the guest smoke stayed GREEN `ALL OK` 64/64; the next rung is P5 work forwarding + completion relay, with P4 (alloc/free handle adoption) deliberately deferred. |
| Graphics (3D), video | ❌ |
| Faults, multi-version, Windows, migration, multi-tenancy, performance | ❌ |
@@ -81,21 +81,47 @@ chain succeeds.
## Right now (frontier)
-- **Guest-facing identity wiring (P2) validated live (2026-10-02
- 01:11-01:15).** On a module-owned live GSP-RM session the emulation now
- overlays the server's real static configuration into the guest-facing
- identity path: the guest's `nvidia-smi -L` reports `NVIDIA GeForce RTX 2080
- Super with Max-Q Design` (with a synthetic UUID) instead of the generic
- emulated identity, while the full driver-API smoke test still passes `ALL
- OK`. The cycle ran under the supervised harness and released with the
- latched locks (protected region down, distribution driver rebound, 5 power
- locks intentionally left held, 0 in-window rail cycles, host stable, no
- freeze, pstore empty). This is the first live data path of the guest-wiring
- rung; a control run with the live overlay disabled is in progress (outcome
- pending). Guest work is still **not** forwarded to the physical engine -
- the submission/relay wiring is the next rung. The bundle (the default-latched
- D0/D3cold freeze fix plus the identity overlay) is implemented, built and
- exercised live, but not yet committed.
+- **Read-only control forwarding (P3) landed and verified live (2026-10-02
+ 01:33-01:45).** `GSP_RM_CONTROL` (fn76) forwards a whitelist of
+ side-effect-free queries (`BUS_GET_PCI_BAR_INFO`, `BUS_GET_INFO_V2`,
+ `FB_GET_INFO_V2`, `INTERNAL_GMMU_GET_STATIC_INFO`) from the live GSP-RM
+ under the new `gsp_phys_ctrl_forward` parameter (bool, default off). The
+ guest's emulated internal handles are translated to the snapshot's adopted
+ server handles before the forward, and the server's echoed params become the
+ guest reply; everything else falls back to the existing emulated handler
+ (fail-open), counted as forwarded/fallback/timeout. A/B on the full guest
+ smoke: forwarding ON stayed GREEN `ALL OK` 64/64 with counters ok=22
+ fallback=2 timeout=0, zero reply-send failures and `nvidia-smi -q` working;
+ OFF showed zero forward counters and a healthy guest. Release ran latched as
+ designed: WPR2 down, exactly 5 power locks intentionally held, 0 in-window
+ rail cycles, host stable, no reboot. Module sha256 `30dfd311...`. **Next:
+ P5** work forwarding + completion relay (guest submissions shadow-executed
+ on the physical copy engine). **P4 is deliberately deferred**: the P2-P5
+ plan itself states P5 wave 1 does not require it, and adopting real handles
+ would change guest-visible handles and risk the currently green guest path.
+ The P2/P3 bundles and the G0 freeze fix remain uncommitted in the working
+ tree.
+- **Guest-facing identity wiring (P2) landed and verified live (2026-10-02
+ 01:20-01:45).** On a module-owned live GSP-RM session the emulation
+ overlays the server's real static configuration identity into the
+ guest-facing `GET_GSP_STATIC_INFO` reply under the new
+ `gsp_phys_static_live` parameter (bool, default off): a bounded 2 s wait on
+ the READY-time snapshot, fail-open; the snapshot cache is generation-keyed
+ and invalidated on session changes. A/B on the full guest smoke: with the
+ overlay ON the guest's `nvidia-smi -q` Product Name is `NVIDIA GeForce RTX
+ 2080 Super with Max-Q Design` (live; overlay applied 4x) and with it OFF it
+ is `NVIDIA GeForce RTX 2080 SUPER` (overlay applied 0); the driver-API
+ smoke stays GREEN `ALL OK` 64/64. FB geometry, masks, engine caps, PDE
+ bases and handles remain emulated in P2. One overlay-OFF full run hit a
+ pre-existing emulated-path flake (the guest wedged in `cuInit` with
+ `rmapiControlCacheSet ... NV_ERR_INVALID_PARAMETER`; no host abort); a
+ subsequent OFF quick run was green, so it is not a P2 regression and is
+ being watched. Release ran latched as designed: WPR2 down, distribution
+ driver rebound, exactly 5 power locks intentionally left held, 0 in-window
+ rail cycles, host stable, no reboot. Module sha256 `69eac176...`. P3
+ read-only control forwarding has since landed and verified live (bullet
+ above). Guest work is still **not** forwarded to the physical engine. The
+ P2 bundle and the G0 freeze fix remain uncommitted in the working tree.
- **G0 fix bundle built after the P1 release-path freeze (2026-10-02 00:55).**
The first P1 mdev-owned acceptance cycle ended in the known hard freeze
(journal stops, pstore empty): the P1 release path ran outside the supervised
diff --git a/docs/03-architecture/11-physical-backend-guest-wiring.md b/docs/03-architecture/11-physical-backend-guest-wiring.md
index da440eb..55700b5 100644
--- a/docs/03-architecture/11-physical-backend-guest-wiring.md
+++ b/docs/03-architecture/11-physical-backend-guest-wiring.md
@@ -1,7 +1,12 @@
# Physical Backend Guest Wiring: Driving the Guest-Facing Emulation from the Live GSP-RM Session
-> **Status: design for implementation (2026-10-01). No code from this design is
-> in the tree yet.** The physical backend itself is live: the module-owned
+> **Status: P0-P3 implemented (2026-10-02).** The backend API skeleton (P0),
+> the mdev-owned lifecycle (P1, fail-open; the async FWSEC boot on that path
+> is an open defect off the critical path), the static-config identity
+> overlay (P2, verified live in an A/B run) and the read-only control
+> forwarding whitelist (P3, default off, verified live: ok=22/fallback=2,
+> smoke `ALL OK` 64/64) are in the tree. The physical
+> backend itself is live: the module-owned
> GSP-RM completes initialization on the real TU104, serves the full
> static-config reply, and shuts down gracefully. The host resource-manager
> client ladder has since allocated memory, a virtual-address-space object with
@@ -19,7 +24,8 @@
> The design was written against the tree at commit `f8e5d3b`; line numbers are
> advisory and have moved with later commits — anchor by function name when
> re-grepping. Behaviour claims below are grounded in live evidence from the
-> 2026-10-01 cycles; anything not yet exercised is labelled "not yet run".
+> 2026-10-01 and 2026-10-02 cycles; anything not yet exercised is labelled
+> "not yet run".
---
@@ -93,12 +99,11 @@
### 0.3 The gap
-The two planes never touch. The guest's static answers, controls and objects
-are emulated; the physical session's real handles, real FB geometry, real
-engine caps and real work capability are only visible to debugfs. The bridge
-must be added at exactly three places (see §2): the reply point
-`gsp_rm_send()`, the inbound dispatch point `gsp_rm_handle_element()`, and the
-lifecycle points in `main.c` / `gsp_phys_gpu.c`.
+The two planes now meet only at the lifecycle seam (P0/P1, mdev-owned
+claim/release) and the static-config identity overlay (P2); controls, objects,
+FB geometry, engine caps and work are still emulated. The remaining bridge
+must be added at the reply point `gsp_rm_send()`, the inbound dispatch point
+`gsp_rm_handle_element()`, and the work-plane seams (see §2).
---
@@ -181,7 +186,7 @@ generation counter` so a release cannot deadlock against a forward in flight.
the existing release still quiesces, runs the FWSEC teardown and refuses to
rebind with WPR2 up (`gsp_phys_gpu.c:939`). A reboot then needs explicit
human approval plus a handoff update.
-* **Static-config served before READY:** bounded wait (recommend 2 s) then
+* **Static-config served before READY:** bounded wait (2 s, implemented) then
pure emulation fallback, with a stat counter; never block the emulation
thread longer than the guest's own RPC poll can tolerate.
@@ -332,6 +337,11 @@ Two grounded warnings:
Keep emulated caps until the backend can actually service the resulting
demands (P4+).
+**P2 status (2026-10-02):** the name, short-name and VBIOS identity fields are
+now overlaid live behind `gsp_phys_static_live` (A/B verified); the FB
+geometry, masks and engine caps above remain emulated, per these warnings,
+until P4/P6.
+
### 4.3 Proposed flow
1. At READY, cache the full fn65 payload (new bytes in the backend session,
@@ -364,6 +374,9 @@ go-ahead, recovery plan, pstore, never unattended
(`docs/03-architecture/08-physical-gpu-integration.md` §7/§8).
### P0 - Backend API skeleton (inert)
+* **Status (2026-10-02): implemented.** Session state/refcount plus the locked
+ backend RPC and static-snapshot helpers are in the tree and refuse when the
+ backend is OFF/not-init-done/unloading.
* **Scope:** new state/refcount/lock in `gsp_phys_gpu.c`; locked
`backend_rpc()`/`static_snapshot()` in `gsp_phys_booter.c`; no caller
changes; no behavior change.
@@ -374,6 +387,12 @@ go-ahead, recovery plan, pstore, never unattended
* **Rollback:** trivial.
### P1 - Lifecycle wiring (claim on mdev create, release on last remove)
+* **Status (2026-10-02): implemented and exercised live; fail-open.** The
+ first mdev create starts the claim/boot asynchronously and the last remove
+ releases it. The async boot currently fails at FWSEC on this path
+ (`staging not visible before kick`) and the READY gate releases the claim
+ fail-open; the module-owned live session is the proven path (and what P2
+ uses).
* **Scope:** `main.c` create/remove/init_work/cleanup_work/exit ordering
(§3 items 3-6); no live answers yet.
* **Acceptance (supervised, harness `phys_claim_cycle.sh` style):** fresh
@@ -392,6 +411,14 @@ go-ahead, recovery plan, pstore, never unattended
only; `mdev` emits `-EOPNOTSUPP` from `backend_start()`.
### P2 - Static-config identity overlay
+* **Status (2026-10-02): implemented and verified live (A/B).** The READY-time
+ snapshot cache and the `gsp_phys_static_live` parameter (bool, default 0)
+ are in the tree. With the overlay on, the guest `nvidia-smi -q` Product
+ Name is the live `NVIDIA GeForce RTX 2080 Super with Max-Q Design`; with it
+ off it is the emulated `NVIDIA GeForce RTX 2080 SUPER`; the driver-API
+ smoke stays green (`ALL OK`, payload 64/64). Only name, short name and
+ VBIOS identity are overlaid; FB geometry, masks, engine caps, PDE bases and
+ handles stay emulated.
* **Scope:** cache snapshot at READY; overlay allow-list in
`gsp_rm_build_static_info()`/`gsp_rm_handle_get_static_info()`; module
parameter; stats. Bounded wait (2 s) then emulation fallback.
@@ -404,6 +431,15 @@ go-ahead, recovery plan, pstore, never unattended
* **Rollback:** parameter off.
### P3 - Read-only control forwarding (whitelist)
+* **Status (2026-10-02): implemented and verified live (default off).** With
+ `gsp_phys_ctrl_forward=1` the whitelist (`BUS_GET_PCI_BAR_INFO`,
+ `BUS_GET_INFO_V2`, `FB_GET_INFO_V2`, `INTERNAL_GMMU_GET_STATIC_INFO`) is
+ forwarded over fn76 with the guest's emulated internal handles translated
+ to the snapshot's adopted server handles; the server's echoed params become
+ the guest reply, and everything else falls back to the emulated handler
+ (fail-open; counters forwarded/fallback/timeout). Live acceptance: guest
+ smoke GREEN `ALL OK` 64/64, counters ok=22 fallback=2 timeout=0, guest
+ `nvidia-smi -q` worked.
* **Scope:** in `gsp_rm_handle_rm_control()` (`:2611`), before the per-cmd
switch: for an allow-list of side-effect-free queries (FB/BUS/KGR/GMMU/
CE/display statics), call `backend_rpc()` with the **adopted internal