From c17da06d6cd8b9c69449904637bf37729a962989 Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Fri, 2 Oct 2026 11:28:49 -0400 Subject: docs: engineering-operations playbook and project-wide status accuracy pass - docs/eng-ops.md gains Part II, live physical-backend engineering operations: hard rules, the full module-parameter matrix, the routine supervised cycle and clean-release checklist, the debugfs inventory, how to read gsp-phys-state, a debugging scenario catalogue (CTXNOTVALID/promote wedge, fail-safe WPR2, unload hangs, guest wedges, MMU/PTE and PBDMA faults, relay symptoms), a hazard register (crash causes and debugging side effects), the physical-backend OODA loop, and the work-context map with porting flows - architecture, implementation, operations, orientation, getting-started, portability, ecosystem and reference docs corrected to the current state: CUDA driver-API smoke green, CE-class work forwarding and engine-authored releases plus fault relay in place (default off), GR execution blocked on engine-context validity, the promote wedge and designed locked fail-safe, and P5 wave-2 subsumed - internal-reference cleanup in tracked docs (no analysis-source or local-archive references); historical architecture document bannered - README reading list and STATE-CURRENT frontier refreshed; new pages (runbook, green-eye user guide, porting sequence, porting/packaging notes) linked from the tree Link check: scripts/check-docs-links.py passes (56 files, no broken links). --- STATE-CURRENT.md | 78 ++++++++++++++++++++++++++++++-------------------------- 1 file changed, 42 insertions(+), 36 deletions(-) (limited to 'STATE-CURRENT.md') diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md index 7aaf9f1..9f5ea01 100644 --- a/STATE-CURRENT.md +++ b/STATE-CURRENT.md @@ -1,6 +1,6 @@ # Open-IOV Current State -> **Snapshot: 2026-10-02 10:55.** This is a dated public status page, not live +> **Snapshot: 2026-10-02 ~11:00.** This is a dated public status page, not live > state; it is refreshed on every progress check. Live working notes are > maintained outside this repository; the public known-knowns/unknowns document > is authoritative here: @@ -69,7 +69,7 @@ meant to execute on the physical GPU. and destination equal to source. Every cycle released the device cleanly with no reboot; the next rung is wiring the live physical session into the guest-facing mdev/vfio path - ([`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md)). The guest-wiring rung then advanced: an inert backend-session API was added, and the mediated-device-owned lifecycle (P1, see the frontier section) was implemented and exercised - deferring the physical claim/boot to the first device create, releasing it on the last remove, with fail-open behaviour. The first P1 acceptance cycle exposed two defects - an asynchronous boot marked READY after a failed FWSEC boot, and the P1 release path freezing the host (see the frontier section). The G0 fix bundle (default-latched D0/D3cold locks, post-rebind readiness gate, pre-claim D0 settle, READY gated on a proven boot, pre-kick staging readback) is implemented, built, smoke-tested and now **validated live**: two module-owned supervised cycles (2026-10-02 ~00:57-00:58) were 2/2 green through claim, FWSEC, booter, RISC-V active, init-done (`0x1001`, `rpc_result=0` in ~955 ms) and a latched release with WPR2 down, exactly 5 `power domain left LOCKED` lines, 0 in-window rail cycles, host alive and pstore empty. The retested P1 mdev-owned path - the exact path that hard-froze on 2026-10-01 - now fails open safely (the READY gate releases the claim when no live GSP-RM session exists; no freeze), while the async FWSEC boot defect (`staging not visible before kick` -> `FWSEC/FRTS execution failed: -5`) remains open, quarantined off the critical path; guest wiring proceeds on the module-owned live session. The bundle was later committed (`56766b2`). Guest-wiring P2 (static-config identity overlay) has since landed and verified live (2026-10-02 01:20-01:45): with `gsp_phys_static_live=1` the guest's `nvidia-smi -q` Product Name comes from the live session (`NVIDIA GeForce RTX 2080 Super with Max-Q Design`) versus the emulated `NVIDIA GeForce RTX 2080 SUPER` with it off (A/B; overlay applied 4x/0), while the guest smoke stays GREEN `ALL OK` 64/64; one overlay-OFF full run hit a pre-existing emulated-path flake, watched. P3 (read-only control forwarding) has since landed and verified live (2026-10-02 ~01:33-01:45): the `gsp_phys_ctrl_forward` whitelist served 22 controls live with 2 fallbacks and 0 timeouts while the guest smoke stayed GREEN `ALL OK` 64/64; **P5 wave 1 (guest-submission shadow execution) has since landed and verified live** (2026-10-02 ~02:00-02:20): with `gsp_phys_work_forward=1` the sema pump shadow-executes eligible CE pushes on the physical engine through a dedicated ordered workqueue - 514 real 4 KiB copies during a GREEN guest smoke (`ALL OK` 64/64), 194 requests coalesced/dropped while one was in flight, 0 refusals, 0 timeouts - while the software release path stays authoritative and no guest memory or guest VA is touched; P6 has since proven guest-page DMA live (P6a, 2026-10-02: the physical copy engine copied 4 KiB to and from a real guest RAM page with the semaphore released while the guest smoke stayed GREEN `ALL OK` 64/64 and release stayed latched with no reboot) and then **P6b wave 1** (2026-10-02, run p6run13): with `gsp_phys_work_forward=1 gsp_phys_guest_mem=1` the sema service resolved real guest CE-copy operands to host physical addresses and executed them on the physical copy engine through module-built GMMU scratch slots (`req=42 ok=42 verify_ok=42 verify_fail=0`, 503 ladder shadow copies also ran) while the software release path stayed authoritative - guest CE copies now execute on hardware and are **hardware-authored with verified data** while the software copy remains the fail-closed fallback for unqueued or failed copies: wave 1.5 added deferred, exactly-once release (`engaged=75 applied=75`), and wave 2 added `gsp_phys_work_hw_primary=1`, which skips the software copy for forwarded ops (`copy_skipped=119`, `req=100 ok=100 verify_ok=100`, smoke `matched=64/64` `ALL OK`, 19 inline software fallbacks `copy_soft=19 copy_fail=0`, and a 121-copy fallback-recovery run with guest-RAM ranges unregistered). The guest smoke stayed GREEN `ALL OK` 64/64 and release stayed latched with no reboot. Real page-directory shadowing and guest channels (P6c: completion/fault relay, guest page tables, `vectorAdd`) remain, along with the P6a trigger-page pinning follow-up, and P4 (alloc/free handle adoption) remains deliberately deferred. | + ([`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md)). The guest-wiring rung then advanced: an inert backend-session API was added, and the mediated-device-owned lifecycle (P1, see the frontier section) was implemented and exercised - deferring the physical claim/boot to the first device create, releasing it on the last remove, with fail-open behaviour. The first P1 acceptance cycle exposed two defects - an asynchronous boot marked READY after a failed FWSEC boot, and the P1 release path freezing the host (see the frontier section). The G0 fix bundle (default-latched D0/D3cold locks, post-rebind readiness gate, pre-claim D0 settle, READY gated on a proven boot, pre-kick staging readback) is implemented, built, smoke-tested and now **validated live**: two module-owned supervised cycles (2026-10-02 ~00:57-00:58) were 2/2 green through claim, FWSEC, booter, RISC-V active, init-done (`0x1001`, `rpc_result=0` in ~955 ms) and a latched release with WPR2 down, exactly 5 `power domain left LOCKED` lines, 0 in-window rail cycles, host alive and pstore empty. The retested P1 mdev-owned path - the exact path that hard-froze on 2026-10-01 - now fails open safely (the READY gate releases the claim when no live GSP-RM session exists; no freeze), while the async FWSEC boot defect (`staging not visible before kick` -> `FWSEC/FRTS execution failed: -5`) remains open, quarantined off the critical path; guest wiring proceeds on the module-owned live session. The bundle was later committed (`56766b2`). Guest-wiring P2 (static-config identity overlay) has since landed and verified live (2026-10-02 01:20-01:45): with `gsp_phys_static_live=1` the guest's `nvidia-smi -q` Product Name comes from the live session (`NVIDIA GeForce RTX 2080 Super with Max-Q Design`) versus the emulated `NVIDIA GeForce RTX 2080 SUPER` with it off (A/B; overlay applied 4x/0), while the guest smoke stays GREEN `ALL OK` 64/64; one overlay-OFF full run hit a pre-existing emulated-path flake, watched. P3 (read-only control forwarding) has since landed and verified live (2026-10-02 ~01:33-01:45): the `gsp_phys_ctrl_forward` whitelist served 22 controls live with 2 fallbacks and 0 timeouts while the guest smoke stayed GREEN `ALL OK` 64/64; **P5 wave 1 (guest-submission shadow execution) has since landed and verified live** (2026-10-02 ~02:00-02:20): with `gsp_phys_work_forward=1` the sema pump shadow-executes eligible CE pushes on the physical engine through a dedicated ordered workqueue - 514 real 4 KiB copies during a GREEN guest smoke (`ALL OK` 64/64), 194 requests coalesced/dropped while one was in flight, 0 refusals, 0 timeouts - while the software release path stays authoritative and no guest memory or guest VA is touched; P6 has since proven guest-page DMA live (P6a, 2026-10-02: the physical copy engine copied 4 KiB to and from a real guest RAM page with the semaphore released while the guest smoke stayed GREEN `ALL OK` 64/64 and release stayed latched with no reboot) and then **P6b wave 1** (2026-10-02, run p6run13): with `gsp_phys_work_forward=1 gsp_phys_guest_mem=1` the sema service resolved real guest CE-copy operands to host physical addresses and executed them on the physical copy engine through module-built GMMU scratch slots (`req=42 ok=42 verify_ok=42 verify_fail=0`, 503 ladder shadow copies also ran) while the software release path stayed authoritative - guest CE copies now execute on hardware and are **hardware-authored with verified data** while the software copy remains the fail-closed fallback for unqueued or failed copies: wave 1.5 added deferred, exactly-once release (`engaged=75 applied=75`), and wave 2 added `gsp_phys_work_hw_primary=1`, which skips the software copy for forwarded ops (`copy_skipped=119`, `req=100 ok=100 verify_ok=100`, smoke `matched=64/64` `ALL OK`, 19 inline software fallbacks `copy_soft=19 copy_fail=0`, and a 121-copy fallback-recovery run with guest-RAM ranges unregistered). The guest smoke stayed GREEN `ALL OK` 64/64 and release stayed latched with no reboot. Real page-directory shadowing and guest channels (P6c: completion/fault relay, guest page tables, `vectorAdd`) have since been reached: guest CE copies and their releases are forwarded with release and fault relay in place (P6c C0/C1/C3, default off); the host GR ladder (H1) brings up a GR0 channel and a `TURING_COMPUTE_A` object, but the first push faults `CTXNOTVALID` (the external-VAS context was never committed) and the promote-binding attempt wedged GSP-RM (host assert, fuse-locked WPR2), so the device is held in the designed locked fail-safe pending a supervised reboot (see the frontier section). Real guest page tables/channels and GR execution remain, along with the P6a trigger-page pinning follow-up, and P4 (alloc/free handle adoption) remains deliberately deferred. | | Graphics (3D), video | ❌ | | Faults, multi-version, Windows, migration, multi-tenancy, performance | ❌ | @@ -97,36 +97,37 @@ chain succeeds. cycles. The acting session deliberately paused here; the reboot request is awaiting approval because other work is running, and no further live GPU cycles are planned this boot. The promote-assert bisection plan is ready - for the next boot. -- **Guest-submission relay and fault replay landed, default off (P6c + for the next boot. The promote is gated by `gsp_phys_rmcall_gr_promote` + (default 0), and the parameter documentation warns that it can wedge the + device. +- **Guest-submission relay and fault relay landed, default off (P6c C0/C1/C3, 2026-10-02 07:43-08:07).** With `gsp_phys_relay=1` the physical engine authors the guest CE copy and the release semaphore: the engine-authored release was observed in the guest page (`relays=64`), releases are delivered exactly once (deferred hold), and the relay-on run stayed green (`requested=68 ok=67 fallback=9`, smoke `ALL OK` 64/64). A forced-unresolvable control proved the software fallback recovers every - copy exactly once. One relayed copy released its semaphore but failed the - host-side data compare, so the parameter stays default-off pending triage; - verification hardening now logs the mismatch (sem target/payload/poll, - first differing byte, live/snapshot dumps) and falls back to the software - apply. `gsp_phys_fault_relay=1` relays forwarded-engine faults to the guest - channel notifier/event path (live green, default off). The software path is - unchanged with all parameters off. -- **Host GR ladder (H1) opened and hit the next wall (2026-10-02 08:29).** - The host resource-manager ladder is now selectable to the compute engine - (`gsp_phys_rmcall_engine=1`, default off) and allocates a GR0 channel with - a compute-class object on the live GSP-RM (`engine=GR0 hGr=0xcaf0002c`), but - the first host GR semaphore push did not complete: the submit timed out - (`-110`) with no release. The copy-engine control run stayed green - (`REAL COPY OK`), and both runs released latched and clean (protected - region down, distribution driver rebound, exactly 5 power-domain locks, - 0 rail cycles, same boot). Current wall: make the GR engine execute the - submitted compute block - bisect GR allocation parameters, the release - method encoding and the method order/context init, using the drained GR - fault records. The timeout was then root-caused to an uncommitted GR engine - context (`CTXNOTVALID`); the follow-up promote-context attempt wedged the - server - see the top bullet. Next: host `vectorAdd`, then guest GR - forwarding. + copy exactly once. One relayed copy of 68 released its semaphore but failed + the host-side data compare; it was handled fail-closed (software apply) and + the parameter stays default-off pending triage, with hardening now in + place: retry-once, snapshot capture and relay counters. `gsp_phys_fault_relay=1` + relays forwarded-engine faults to the guest channel notifier/event path; it + delivered a real CE0 fault live (default off), and the auxiliary event post + reported `no_event`. The software path is unchanged with all parameters off. +- **Host GR ladder (H1) opened and hit the GR context wall (2026-10-02 + 08:29).** The host resource-manager ladder is now selectable to the compute + engine (`gsp_phys_rmcall_engine=1`, default off) and brings up a GR0 + channel with a `TURING_COMPUTE_A` compute object on the live GSP-RM + (`engine=GR0 hGr=0xcaf0002c`), but the first host GR semaphore push faults: + the submit timed out (`-110`) with no release and the drained fault records + read `CTXNOTVALID` - the external-VAS context was never committed. The + copy-engine control run stayed green (`REAL COPY OK`), and both runs + released latched and clean (protected region down, distribution driver + rebound, exactly 5 power-domain locks, 0 rail cycles, same boot). The + follow-up context-promote binding attempt wedged GSP-RM (host assert, + fuse-locked WPR2) - see the top bullet. Current wall: commit the GR context + and make the engine execute the submitted compute block. Next: host + `vectorAdd`, then guest GR forwarding. - **Experimental tooling: green-eye (2026-10-02 08:36, in development).** `debug/risc-v-gsp-rm-vm/` boots GSP/uproc firmware in a RISC-V QEMU VM with a GDB stub and adds Frida-like runtime inspection, a scripting/plugin @@ -142,7 +143,8 @@ chain succeeds. self-test and ladder `REAL COPY OK`, clean release). Forwarding was never affected (HPAs are resolved at submit time). - **Guest CE copies are hardware-authored with verified data, with deferred - exactly-once release (P6b waves 1.5 and 2, 2026-10-02, run p6w2d).** Wave + exactly-once release (P6b waves 1.5 and 2, 2026-10-02, run p6w2d; + superseded as a frontier by the P6c relay and H1 bullets above).** Wave 1.5 (run p6w15), with `gsp_phys_work_defer_release=1` (plus `gsp_phys_work_forward=1 gsp_phys_guest_mem=1`): a forwarded guest push's releases are recorded and its GPFIFO entry is held until the hardware copy @@ -235,7 +237,8 @@ chain succeeds. distribution driver rebound, protected region down, exactly 5 power-domain locks intentionally held, 0 in-window rail cycles, no reboot. Module sha256 `e1bce651...`. **Next: P6b** - forwarding guest submissions (page-directory - shadowing, real guest channels, push forwarding) - not implemented yet. + shadowing, real guest channels, push forwarding) - landed afterwards + (superseded by the P6b/P6c bullets above). - **Guest-memory path (P6) first attempts (2026-10-02 ~02:42 onward; resolved by the P6a result above).** The rung puts guest memory into the real execution path: guest RAM is now a @@ -277,7 +280,8 @@ chain succeeds. pre-existing expected behavior. **Next: P6** (guest page tables/memory into the real MMU so guest data itself executes on hardware; includes the IOMMU/DMA reachability known-unknown), with optional **P5 wave 2** - (hardware-primary release relay with a non-applying decode mode); **P4** + (hardware-primary release relay with a non-applying decode mode; **subsumed** + by P6b wave 2 + the P6c relay above); **P4** remains deferred. - **Read-only control forwarding (P3) landed and verified live (2026-10-02 01:33-01:45).** `GSP_RM_CONTROL` (fn76) forwards a whitelist of @@ -318,7 +322,8 @@ chain succeeds. driver rebound, exactly 5 power locks intentionally left held, 0 in-window rail cycles, host stable, no reboot. Module sha256 `69eac176...`. P3 read-only control forwarding has since landed and verified live (bullet - above). Guest work is still **not** forwarded to the physical engine. The + above). At that point guest work was not yet forwarded to the physical + engine (the P6b/P6c forwarding results are above). The P2 bundle and the G0 freeze fix have since been committed (`56766b2`). - **G0 fix bundle built after the P1 release-path freeze (2026-10-02 00:55).** The first P1 mdev-owned acceptance cycle ended in the known hard freeze @@ -368,7 +373,8 @@ chain succeeds. module-owned path succeeds with the very same preconditions in the same boot). Guest wiring proceeds on the module-owned live session (`gsp_phys_gpu=1` claim+boot at load, proven); the P1 async claim stays - fail-open. No guest work is forwarded to the physical engine yet. + fail-open. At that point no guest work was forwarded to the physical engine + yet (the P6b/P6c forwarding results are above). - **Guest-wiring rung opened (2026-10-01 22:00): the first backend-session API code is in the tree, inert.** A skeleton that exposes the live physical session to the device layer (session request entry points plus a static-info @@ -381,7 +387,8 @@ chain succeeds. [`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md). Post-single-VM-CUDA research notes (concurrent multi-VM virtual functions, 3D acceleration, and a native-vs-guest performance program; and a host-side - acceleration feasibility study) were produced under `backups/` (untracked). + acceleration feasibility study) were produced and are kept in the local + evidence archive (untracked). - **Submission-rung milestone (2026-10-01 17:24-21:05): real work now executes on the physical engine.** The host resource-manager client allocated a virtual-address-space object with a committed page directory, a channel-group @@ -398,8 +405,8 @@ chain succeeds. wiring this live session into the guest-facing mdev/vfio path so guest submissions execute on the physical engine (phased plan: [`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md)). - Evidence: `backups/evidence-20261001-cycle18/` through `...-cycle25/`, with - per-wall reports in `backups/evidence-20261001-reports/next/`. + Evidence: the local evidence archive (cycle 18 through cycle 25 runs, with + per-wall reports). - **Physical-backend milestone (2026-10-01 16:45): the host-owned GSP-RM completes initialization on the real TU104 and serves the static-config RPC.** Three supervised cycles, no reboot, clean release each time: @@ -422,8 +429,7 @@ chain succeeds. The physical backend is now a live, queryable GSP-RM: the host can boot it, send RM RPCs and receive full replies, and shut it down gracefully. The next rung is to drive the guest-facing emulation from this backend. - Evidence: `backups/evidence-20261001-cycle11/`, `...-cycle12/`, - `...-cycle13/`. + Evidence: the local evidence archive (cycles 11-13). - **Driver-API baseline re-verified green on the current build** (2026-10-01 14:20): the smoke test passes end-to-end (`ALL OK`, payload 64/64) with real releases (`faked=0`, `miss=0`, `stale=0`); the source-level root-cause -- cgit v1.2.3