summaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-02 02:13:03 -0400
committerArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-02 02:13:03 -0400
commita9998b380c16ff7e9f6d26953643a3f204735efa (patch)
tree50388aaf7a509e978baa0c99ea888a1b709d51ee
parentfdce9fec4d95e5e7141939d55e333e734122aa6e (diff)
downloadopen-iov-a9998b380c16ff7e9f6d26953643a3f204735efa.tar.gz
open-iov-a9998b380c16ff7e9f6d26953643a3f204735efa.zip
STATE-CURRENT + docs: record P2 identity overlay, P3 control forwarding and P5 wave-1 shadow execution
-rw-r--r--STATE-CURRENT.md43
-rw-r--r--docs/03-architecture/11-physical-backend-guest-wiring.md20
2 files changed, 49 insertions, 14 deletions
diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md
index f02c742..f15a403 100644
--- a/STATE-CURRENT.md
+++ b/STATE-CURRENT.md
@@ -1,6 +1,6 @@
# Open-IOV Current State
-> **Snapshot: 2026-10-02 01:55.** This is a dated public status page, not live
+> **Snapshot: 2026-10-02 02:25.** This is a dated public status page, not live
> state; it is refreshed on every progress check. Live working notes are
> maintained outside this repository; the public known-knowns/unknowns document
> is authoritative here:
@@ -69,7 +69,7 @@ meant to execute on the physical GPU.
and destination equal to source. Every cycle released the device cleanly
with no reboot; the next rung is wiring the live physical session into the
guest-facing mdev/vfio path
- ([`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md)). The guest-wiring rung then advanced: an inert backend-session API was added, and the mediated-device-owned lifecycle (P1, see the frontier section) was implemented and exercised - deferring the physical claim/boot to the first device create, releasing it on the last remove, with fail-open behaviour. The first P1 acceptance cycle exposed two defects - an asynchronous boot marked READY after a failed FWSEC boot, and the P1 release path freezing the host (see the frontier section). The G0 fix bundle (default-latched D0/D3cold locks, post-rebind readiness gate, pre-claim D0 settle, READY gated on a proven boot, pre-kick staging readback) is implemented, built, smoke-tested and now **validated live**: two module-owned supervised cycles (2026-10-02 ~00:57-00:58) were 2/2 green through claim, FWSEC, booter, RISC-V active, init-done (`0x1001`, `rpc_result=0` in ~955 ms) and a latched release with WPR2 down, exactly 5 `power domain left LOCKED` lines, 0 in-window rail cycles, host alive and pstore empty. The retested P1 mdev-owned path - the exact path that hard-froze on 2026-10-01 - now fails open safely (the READY gate releases the claim when no live GSP-RM session exists; no freeze), while the async FWSEC boot defect (`staging not visible before kick` -> `FWSEC/FRTS execution failed: -5`) remains open, quarantined off the critical path; guest wiring proceeds on the module-owned live session. The bundle is still uncommitted. Guest-wiring P2 (static-config identity overlay) has since landed and verified live (2026-10-02 01:20-01:45): with `gsp_phys_static_live=1` the guest's `nvidia-smi -q` Product Name comes from the live session (`NVIDIA GeForce RTX 2080 Super with Max-Q Design`) versus the emulated `NVIDIA GeForce RTX 2080 SUPER` with it off (A/B; overlay applied 4x/0), while the guest smoke stays GREEN `ALL OK` 64/64; one overlay-OFF full run hit a pre-existing emulated-path flake, watched. P3 (read-only control forwarding) has since landed and verified live (2026-10-02 ~01:33-01:45): the `gsp_phys_ctrl_forward` whitelist served 22 controls live with 2 fallbacks and 0 timeouts while the guest smoke stayed GREEN `ALL OK` 64/64; the next rung is P5 work forwarding + completion relay, with P4 (alloc/free handle adoption) deliberately deferred. |
+ ([`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md)). The guest-wiring rung then advanced: an inert backend-session API was added, and the mediated-device-owned lifecycle (P1, see the frontier section) was implemented and exercised - deferring the physical claim/boot to the first device create, releasing it on the last remove, with fail-open behaviour. The first P1 acceptance cycle exposed two defects - an asynchronous boot marked READY after a failed FWSEC boot, and the P1 release path freezing the host (see the frontier section). The G0 fix bundle (default-latched D0/D3cold locks, post-rebind readiness gate, pre-claim D0 settle, READY gated on a proven boot, pre-kick staging readback) is implemented, built, smoke-tested and now **validated live**: two module-owned supervised cycles (2026-10-02 ~00:57-00:58) were 2/2 green through claim, FWSEC, booter, RISC-V active, init-done (`0x1001`, `rpc_result=0` in ~955 ms) and a latched release with WPR2 down, exactly 5 `power domain left LOCKED` lines, 0 in-window rail cycles, host alive and pstore empty. The retested P1 mdev-owned path - the exact path that hard-froze on 2026-10-01 - now fails open safely (the READY gate releases the claim when no live GSP-RM session exists; no freeze), while the async FWSEC boot defect (`staging not visible before kick` -> `FWSEC/FRTS execution failed: -5`) remains open, quarantined off the critical path; guest wiring proceeds on the module-owned live session. The bundle was later committed (`56766b2`). Guest-wiring P2 (static-config identity overlay) has since landed and verified live (2026-10-02 01:20-01:45): with `gsp_phys_static_live=1` the guest's `nvidia-smi -q` Product Name comes from the live session (`NVIDIA GeForce RTX 2080 Super with Max-Q Design`) versus the emulated `NVIDIA GeForce RTX 2080 SUPER` with it off (A/B; overlay applied 4x/0), while the guest smoke stays GREEN `ALL OK` 64/64; one overlay-OFF full run hit a pre-existing emulated-path flake, watched. P3 (read-only control forwarding) has since landed and verified live (2026-10-02 ~01:33-01:45): the `gsp_phys_ctrl_forward` whitelist served 22 controls live with 2 fallbacks and 0 timeouts while the guest smoke stayed GREEN `ALL OK` 64/64; **P5 wave 1 (guest-submission shadow execution) has since landed and verified live** (2026-10-02 ~02:00-02:20): with `gsp_phys_work_forward=1` the sema pump shadow-executes eligible CE pushes on the physical engine through a dedicated ordered workqueue - 514 real 4 KiB copies during a GREEN guest smoke (`ALL OK` 64/64), 194 requests coalesced/dropped while one was in flight, 0 refusals, 0 timeouts - while the software release path stays authoritative and no guest memory or guest VA is touched; P6 (guest page tables/memory into the real MMU, including the IOMMU/DMA reachability known-unknown) is the next end-to-end rung, P5 wave 2 (hardware-primary release relay) is optional, and P4 (alloc/free handle adoption) remains deliberately deferred. |
| Graphics (3D), video | ❌ |
| Faults, multi-version, Windows, migration, multi-tenancy, performance | ❌ |
@@ -81,6 +81,31 @@ chain succeeds.
## Right now (frontier)
+- **Guest-submission shadow execution (P5 wave 1) landed and verified live
+ (2026-10-02 ~02:00-02:20).** Under the new `gsp_phys_work_forward` parameter
+ (bool, default off) the sema pump requests ONE shadow copy for each eligible
+ CE push through a dedicated ordered workqueue (`gsp-phys-work`),
+ coalescing/dropping requests while one is in flight; the work item runs the
+ ladder channel's module-owned 4 KiB copy via the shared refactored
+ `gsp_phys_backend_ce_copy_submit()` (rm lock only for clear+methods+GPFIFO+
+ GP_PUT+doorbell, then a lock-free semaphore poll <=200 ms). The software
+ release path remains authoritative and unchanged; refusals/timeouts/drops
+ are counted; no guest memory or guest VA is touched in wave 1. Full guest
+ smoke stayed GREEN `ALL OK` 64/64 while the shadow ran **514 real CE copies**
+ on the physical engine (bytes=4096, elapsed 7-9 ms), with **194 requests
+ dropped while one was in flight, 0 submit refusals, 0 copy timeouts**.
+ Release ran latched as designed: rebound `nvidia` (rc=1), exactly 5 `power
+ domain left LOCKED`, WPR2 down, 0 in-window rail cycles, host stable, no
+ reboot. Module sha256 `83952506...`; code HEAD `82fc7a8` (`56766b2` carries
+ the G0+P2/P3/P5-wave-1 code, `82fc7a8` the STATE-CURRENT/design-doc update).
+ Ladder regression order: trigger 5 (adopt handles) -> trigger 3 (VAS/TSG/
+ channel, variant V5 engine COPY0) -> trigger 4 (`REAL COPY OK`); trigger 4
+ alone on a fresh session fails at stage 1 with `NV01_ROOT 0x23`, which is
+ pre-existing expected behavior. **Next: P6** (guest page tables/memory into
+ the real MMU so guest data itself executes on hardware; includes the
+ IOMMU/DMA reachability known-unknown), with optional **P5 wave 2**
+ (hardware-primary release relay with a non-applying decode mode); **P4**
+ remains deferred.
- **Read-only control forwarding (P3) landed and verified live (2026-10-02
01:33-01:45).** `GSP_RM_CONTROL` (fn76) forwards a whitelist of
side-effect-free queries (`BUS_GET_PCI_BAR_INFO`, `BUS_GET_INFO_V2`,
@@ -94,13 +119,13 @@ chain succeeds.
fallback=2 timeout=0, zero reply-send failures and `nvidia-smi -q` working;
OFF showed zero forward counters and a healthy guest. Release ran latched as
designed: WPR2 down, exactly 5 power locks intentionally held, 0 in-window
- rail cycles, host stable, no reboot. Module sha256 `30dfd311...`. **Next:
- P5** work forwarding + completion relay (guest submissions shadow-executed
- on the physical copy engine). **P4 is deliberately deferred**: the P2-P5
+ rail cycles, host stable, no reboot. Module sha256 `30dfd311...`. **P5 wave 1
+ has since landed** (guest submissions shadow-executed on the physical copy
+ engine; bullet above). **P4 is deliberately deferred**: the P2-P5
plan itself states P5 wave 1 does not require it, and adopting real handles
would change guest-visible handles and risk the currently green guest path.
- The P2/P3 bundles and the G0 freeze fix remain uncommitted in the working
- tree.
+ The P2/P3/P5-wave-1 bundles and the G0 freeze fix are now committed
+ (`56766b2`, `82fc7a8`; HEAD `82fc7a8`).
- **Guest-facing identity wiring (P2) landed and verified live (2026-10-02
01:20-01:45).** On a module-owned live GSP-RM session the emulation
overlays the server's real static configuration identity into the
@@ -121,7 +146,7 @@ chain succeeds.
rail cycles, host stable, no reboot. Module sha256 `69eac176...`. P3
read-only control forwarding has since landed and verified live (bullet
above). Guest work is still **not** forwarded to the physical engine. The
- P2 bundle and the G0 freeze fix remain uncommitted in the working tree.
+ P2 bundle and the G0 freeze fix have since been committed (`56766b2`).
- **G0 fix bundle built after the P1 release-path freeze (2026-10-02 00:55).**
The first P1 mdev-owned acceptance cycle ended in the known hard freeze
(journal stops, pstore empty): the P1 release path ran outside the supervised
@@ -141,7 +166,7 @@ chain succeeds.
active, init-done (`0x1001`, `rpc_result=0` in ~955 ms), WPR2 down,
`released 0000:01:00.0; rebound to nvidia (rc=1)`, exactly 5 `power domain
left LOCKED` lines, 0 in-window rail cycles, host survived, pstore empty;
- module sha256 `dde36d92...`. Uncommitted.
+ module sha256 `dde36d92...`. Committed in `56766b2`.
- **Guest-wiring lifecycle (P1) implemented and exercised; retested after G0
(2026-10-02 ~00:58).** A module parameter selects a lifecycle where the
physical-GPU claim and boot are deferred to the first mediated device create
diff --git a/docs/03-architecture/11-physical-backend-guest-wiring.md b/docs/03-architecture/11-physical-backend-guest-wiring.md
index 55700b5..8c38680 100644
--- a/docs/03-architecture/11-physical-backend-guest-wiring.md
+++ b/docs/03-architecture/11-physical-backend-guest-wiring.md
@@ -1,11 +1,13 @@
# Physical Backend Guest Wiring: Driving the Guest-Facing Emulation from the Live GSP-RM Session
-> **Status: P0-P3 implemented (2026-10-02).** The backend API skeleton (P0),
-> the mdev-owned lifecycle (P1, fail-open; the async FWSEC boot on that path
-> is an open defect off the critical path), the static-config identity
-> overlay (P2, verified live in an A/B run) and the read-only control
+> **Status: P0-P5(w1) implemented (2026-10-02).** The backend API skeleton
+> (P0), the mdev-owned lifecycle (P1, fail-open; the async FWSEC boot on that
+> path is an open defect off the critical path), the static-config identity
+> overlay (P2, verified live in an A/B run), the read-only control
> forwarding whitelist (P3, default off, verified live: ok=22/fallback=2,
-> smoke `ALL OK` 64/64) are in the tree. The physical
+> smoke `ALL OK` 64/64) and the P5 wave-1 guest-submission shadow execution
+> (default off, verified live: 514 real CE copies during a green guest smoke
+> `ALL OK` 64/64) are in the tree. The physical
> backend itself is live: the module-owned
> GSP-RM completes initialization on the real TU104, serves the full
> static-config reply, and shuts down gracefully. The host resource-manager
@@ -473,6 +475,14 @@ go-ahead, recovery plan, pstore, never unattended
* **Rollback:** parameter off (guest returns to echo path).
### P5 - Work forwarding + completion relay (interim, host buffers)
+* **Status (2026-10-02): wave 1 implemented and verified live (default off).**
+ With `gsp_phys_work_forward=1` the sema pump shadow-executes eligible CE
+ pushes on the physical engine through one ordered work item running the
+ module-owned ladder copy; the software release path stays authoritative and
+ no guest memory/VA is touched. Live: guest smoke GREEN `ALL OK` 64/64 with
+ 514 real CE copies (194 requests coalesced/dropped while one was in flight,
+ 0 refusals, 0 timeouts). Wave 2 (hardware-primary release relay) and P6
+ remain.
* **Scope:** the debugfs ladder's real-submission half is **done** (cycle 25:
channel allocates, page directory commits, `REAL COPY OK` on the physical
CE in VA mode). The remaining work: bind each guest CE/GR channel to a real