docs: engineering-operations playbook and project-wide status accuracy pass

- docs/eng-ops.md gains Part II, live physical-backend engineering
  operations: hard rules, the full module-parameter matrix, the routine
  supervised cycle and clean-release checklist, the debugfs inventory,
  how to read gsp-phys-state, a debugging scenario catalogue
  (CTXNOTVALID/promote wedge, fail-safe WPR2, unload hangs, guest wedges,
  MMU/PTE and PBDMA faults, relay symptoms), a hazard register (crash
  causes and debugging side effects), the physical-backend OODA loop, and
  the work-context map with porting flows
- architecture, implementation, operations, orientation, getting-started,
  portability, ecosystem and reference docs corrected to the current
  state: CUDA driver-API smoke green, CE-class work forwarding and
  engine-authored releases plus fault relay in place (default off), GR
  execution blocked on engine-context validity, the promote wedge and
  designed locked fail-safe, and P5 wave-2 subsumed
- internal-reference cleanup in tracked docs (no analysis-source or
  local-archive references); historical architecture document bannered
- README reading list and STATE-CURRENT frontier refreshed; new pages
  (runbook, green-eye user guide, porting sequence, porting/packaging
  notes) linked from the tree

Link check: scripts/check-docs-links.py passes (56 files, no broken links).

1 files changed, 42 insertions, 36 deletions

diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md
index 7aaf9f1..9f5ea01 100644
--- a/STATE-CURRENT.md
+++ b/STATE-CURRENT.md
@@ -1,6 +1,6 @@
 # Open-IOV Current State
 
-> **Snapshot: 2026-10-02 10:55.** This is a dated public status page, not live
+> **Snapshot: 2026-10-02 ~11:00.** This is a dated public status page, not live
 > state; it is refreshed on every progress check. Live working notes are
 > maintained outside this repository; the public known-knowns/unknowns document
 > is authoritative here:
@@ -69,7 +69,7 @@ meant to execute on the physical GPU.
    and destination equal to source. Every cycle released the device cleanly
    with no reboot; the next rung is wiring the live physical session into the
    guest-facing mdev/vfio path
-   ([`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md)). The guest-wiring rung then advanced: an inert backend-session API was added, and the mediated-device-owned lifecycle (P1, see the frontier section) was implemented and exercised - deferring the physical claim/boot to the first device create, releasing it on the last remove, with fail-open behaviour. The first P1 acceptance cycle exposed two defects - an asynchronous boot marked READY after a failed FWSEC boot, and the P1 release path freezing the host (see the frontier section). The G0 fix bundle (default-latched D0/D3cold locks, post-rebind readiness gate, pre-claim D0 settle, READY gated on a proven boot, pre-kick staging readback) is implemented, built, smoke-tested and now **validated live**: two module-owned supervised cycles (2026-10-02 ~00:57-00:58) were 2/2 green through claim, FWSEC, booter, RISC-V active, init-done (`0x1001`, `rpc_result=0` in ~955 ms) and a latched release with WPR2 down, exactly 5 `power domain left LOCKED` lines, 0 in-window rail cycles, host alive and pstore empty. The retested P1 mdev-owned path - the exact path that hard-froze on 2026-10-01 - now fails open safely (the READY gate releases the claim when no live GSP-RM session exists; no freeze), while the async FWSEC boot defect (`staging not visible before kick` -> `FWSEC/FRTS execution failed: -5`) remains open, quarantined off the critical path; guest wiring proceeds on the module-owned live session. The bundle was later committed (`56766b2`). Guest-wiring P2 (static-config identity overlay) has since landed and verified live (2026-10-02 01:20-01:45): with `gsp_phys_static_live=1` the guest's `nvidia-smi -q` Product Name comes from the live session (`NVIDIA GeForce RTX 2080 Super with Max-Q Design`) versus the emulated `NVIDIA GeForce RTX 2080 SUPER` with it off (A/B; overlay applied 4x/0), while the guest smoke stays GREEN `ALL OK` 64/64; one overlay-OFF full run hit a pre-existing emulated-path flake, watched. P3 (read-only control forwarding) has since landed and verified live (2026-10-02 ~01:33-01:45): the `gsp_phys_ctrl_forward` whitelist served 22 controls live with 2 fallbacks and 0 timeouts while the guest smoke stayed GREEN `ALL OK` 64/64; **P5 wave 1 (guest-submission shadow execution) has since landed and verified live** (2026-10-02 ~02:00-02:20): with `gsp_phys_work_forward=1` the sema pump shadow-executes eligible CE pushes on the physical engine through a dedicated ordered workqueue - 514 real 4 KiB copies during a GREEN guest smoke (`ALL OK` 64/64), 194 requests coalesced/dropped while one was in flight, 0 refusals, 0 timeouts - while the software release path stays authoritative and no guest memory or guest VA is touched; P6 has since proven guest-page DMA live (P6a, 2026-10-02: the physical copy engine copied 4 KiB to and from a real guest RAM page with the semaphore released while the guest smoke stayed GREEN `ALL OK` 64/64 and release stayed latched with no reboot) and then **P6b wave 1** (2026-10-02, run p6run13): with `gsp_phys_work_forward=1 gsp_phys_guest_mem=1` the sema service resolved real guest CE-copy operands to host physical addresses and executed them on the physical copy engine through module-built GMMU scratch slots (`req=42 ok=42 verify_ok=42 verify_fail=0`, 503 ladder shadow copies also ran) while the software release path stayed authoritative - guest CE copies now execute on hardware and are **hardware-authored with verified data** while the software copy remains the fail-closed fallback for unqueued or failed copies: wave 1.5 added deferred, exactly-once release (`engaged=75 applied=75`), and wave 2 added `gsp_phys_work_hw_primary=1`, which skips the software copy for forwarded ops (`copy_skipped=119`, `req=100 ok=100 verify_ok=100`, smoke `matched=64/64` `ALL OK`, 19 inline software fallbacks `copy_soft=19 copy_fail=0`, and a 121-copy fallback-recovery run with guest-RAM ranges unregistered). The guest smoke stayed GREEN `ALL OK` 64/64 and release stayed latched with no reboot. Real page-directory shadowing and guest channels (P6c: completion/fault relay, guest page tables, `vectorAdd`) remain, along with the P6a trigger-page pinning follow-up, and P4 (alloc/free handle adoption) remains deliberately deferred. |
+   ([`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md)). The guest-wiring rung then advanced: an inert backend-session API was added, and the mediated-device-owned lifecycle (P1, see the frontier section) was implemented and exercised - deferring the physical claim/boot to the first device create, releasing it on the last remove, with fail-open behaviour. The first P1 acceptance cycle exposed two defects - an asynchronous boot marked READY after a failed FWSEC boot, and the P1 release path freezing the host (see the frontier section). The G0 fix bundle (default-latched D0/D3cold locks, post-rebind readiness gate, pre-claim D0 settle, READY gated on a proven boot, pre-kick staging readback) is implemented, built, smoke-tested and now **validated live**: two module-owned supervised cycles (2026-10-02 ~00:57-00:58) were 2/2 green through claim, FWSEC, booter, RISC-V active, init-done (`0x1001`, `rpc_result=0` in ~955 ms) and a latched release with WPR2 down, exactly 5 `power domain left LOCKED` lines, 0 in-window rail cycles, host alive and pstore empty. The retested P1 mdev-owned path - the exact path that hard-froze on 2026-10-01 - now fails open safely (the READY gate releases the claim when no live GSP-RM session exists; no freeze), while the async FWSEC boot defect (`staging not visible before kick` -> `FWSEC/FRTS execution failed: -5`) remains open, quarantined off the critical path; guest wiring proceeds on the module-owned live session. The bundle was later committed (`56766b2`). Guest-wiring P2 (static-config identity overlay) has since landed and verified live (2026-10-02 01:20-01:45): with `gsp_phys_static_live=1` the guest's `nvidia-smi -q` Product Name comes from the live session (`NVIDIA GeForce RTX 2080 Super with Max-Q Design`) versus the emulated `NVIDIA GeForce RTX 2080 SUPER` with it off (A/B; overlay applied 4x/0), while the guest smoke stays GREEN `ALL OK` 64/64; one overlay-OFF full run hit a pre-existing emulated-path flake, watched. P3 (read-only control forwarding) has since landed and verified live (2026-10-02 ~01:33-01:45): the `gsp_phys_ctrl_forward` whitelist served 22 controls live with 2 fallbacks and 0 timeouts while the guest smoke stayed GREEN `ALL OK` 64/64; **P5 wave 1 (guest-submission shadow execution) has since landed and verified live** (2026-10-02 ~02:00-02:20): with `gsp_phys_work_forward=1` the sema pump shadow-executes eligible CE pushes on the physical engine through a dedicated ordered workqueue - 514 real 4 KiB copies during a GREEN guest smoke (`ALL OK` 64/64), 194 requests coalesced/dropped while one was in flight, 0 refusals, 0 timeouts - while the software release path stays authoritative and no guest memory or guest VA is touched; P6 has since proven guest-page DMA live (P6a, 2026-10-02: the physical copy engine copied 4 KiB to and from a real guest RAM page with the semaphore released while the guest smoke stayed GREEN `ALL OK` 64/64 and release stayed latched with no reboot) and then **P6b wave 1** (2026-10-02, run p6run13): with `gsp_phys_work_forward=1 gsp_phys_guest_mem=1` the sema service resolved real guest CE-copy operands to host physical addresses and executed them on the physical copy engine through module-built GMMU scratch slots (`req=42 ok=42 verify_ok=42 verify_fail=0`, 503 ladder shadow copies also ran) while the software release path stayed authoritative - guest CE copies now execute on hardware and are **hardware-authored with verified data** while the software copy remains the fail-closed fallback for unqueued or failed copies: wave 1.5 added deferred, exactly-once release (`engaged=75 applied=75`), and wave 2 added `gsp_phys_work_hw_primary=1`, which skips the software copy for forwarded ops (`copy_skipped=119`, `req=100 ok=100 verify_ok=100`, smoke `matched=64/64` `ALL OK`, 19 inline software fallbacks `copy_soft=19 copy_fail=0`, and a 121-copy fallback-recovery run with guest-RAM ranges unregistered). The guest smoke stayed GREEN `ALL OK` 64/64 and release stayed latched with no reboot. Real page-directory shadowing and guest channels (P6c: completion/fault relay, guest page tables, `vectorAdd`) have since been reached: guest CE copies and their releases are forwarded with release and fault relay in place (P6c C0/C1/C3, default off); the host GR ladder (H1) brings up a GR0 channel and a `TURING_COMPUTE_A` object, but the first push faults `CTXNOTVALID` (the external-VAS context was never committed) and the promote-binding attempt wedged GSP-RM (host assert, fuse-locked WPR2), so the device is held in the designed locked fail-safe pending a supervised reboot (see the frontier section). Real guest page tables/channels and GR execution remain, along with the P6a trigger-page pinning follow-up, and P4 (alloc/free handle adoption) remains deliberately deferred. |
 | Graphics (3D), video | ❌ |
 | Faults, multi-version, Windows, migration, multi-tenancy, performance | ❌ |
 
@@ -97,36 +97,37 @@ chain succeeds.
   cycles. The acting session deliberately paused here; the reboot request is
   awaiting approval because other work is running, and no further live GPU
   cycles are planned this boot. The promote-assert bisection plan is ready
-  for the next boot.
-- **Guest-submission relay and fault replay landed, default off (P6c
+  for the next boot. The promote is gated by `gsp_phys_rmcall_gr_promote`
+  (default 0), and the parameter documentation warns that it can wedge the
+  device.
+- **Guest-submission relay and fault relay landed, default off (P6c
   C0/C1/C3, 2026-10-02 07:43-08:07).** With `gsp_phys_relay=1` the physical
   engine authors the guest CE copy and the release semaphore: the
   engine-authored release was observed in the guest page (`relays=64`),
   releases are delivered exactly once (deferred hold), and the relay-on run
   stayed green (`requested=68 ok=67 fallback=9`, smoke `ALL OK` 64/64). A
   forced-unresolvable control proved the software fallback recovers every
-  copy exactly once. One relayed copy released its semaphore but failed the
-  host-side data compare, so the parameter stays default-off pending triage;
-  verification hardening now logs the mismatch (sem target/payload/poll,
-  first differing byte, live/snapshot dumps) and falls back to the software
-  apply. `gsp_phys_fault_relay=1` relays forwarded-engine faults to the guest
-  channel notifier/event path (live green, default off). The software path is
-  unchanged with all parameters off.
-- **Host GR ladder (H1) opened and hit the next wall (2026-10-02 08:29).**
-  The host resource-manager ladder is now selectable to the compute engine
-  (`gsp_phys_rmcall_engine=1`, default off) and allocates a GR0 channel with
-  a compute-class object on the live GSP-RM (`engine=GR0 hGr=0xcaf0002c`), but
-  the first host GR semaphore push did not complete: the submit timed out
-  (`-110`) with no release. The copy-engine control run stayed green
-  (`REAL COPY OK`), and both runs released latched and clean (protected
-  region down, distribution driver rebound, exactly 5 power-domain locks,
-  0 rail cycles, same boot). Current wall: make the GR engine execute the
-  submitted compute block - bisect GR allocation parameters, the release
-  method encoding and the method order/context init, using the drained GR
-  fault records. The timeout was then root-caused to an uncommitted GR engine
-  context (`CTXNOTVALID`); the follow-up promote-context attempt wedged the
-  server - see the top bullet. Next: host `vectorAdd`, then guest GR
-  forwarding.
+  copy exactly once. One relayed copy of 68 released its semaphore but failed
+  the host-side data compare; it was handled fail-closed (software apply) and
+  the parameter stays default-off pending triage, with hardening now in
+  place: retry-once, snapshot capture and relay counters. `gsp_phys_fault_relay=1`
+  relays forwarded-engine faults to the guest channel notifier/event path; it
+  delivered a real CE0 fault live (default off), and the auxiliary event post
+  reported `no_event`. The software path is unchanged with all parameters off.
+- **Host GR ladder (H1) opened and hit the GR context wall (2026-10-02
+  08:29).** The host resource-manager ladder is now selectable to the compute
+  engine (`gsp_phys_rmcall_engine=1`, default off) and brings up a GR0
+  channel with a `TURING_COMPUTE_A` compute object on the live GSP-RM
+  (`engine=GR0 hGr=0xcaf0002c`), but the first host GR semaphore push faults:
+  the submit timed out (`-110`) with no release and the drained fault records
+  read `CTXNOTVALID` - the external-VAS context was never committed. The
+  copy-engine control run stayed green (`REAL COPY OK`), and both runs
+  released latched and clean (protected region down, distribution driver
+  rebound, exactly 5 power-domain locks, 0 rail cycles, same boot). The
+  follow-up context-promote binding attempt wedged GSP-RM (host assert,
+  fuse-locked WPR2) - see the top bullet. Current wall: commit the GR context
+  and make the engine execute the submitted compute block. Next: host
+  `vectorAdd`, then guest GR forwarding.
 - **Experimental tooling: green-eye (2026-10-02 08:36, in development).**
   `debug/risc-v-gsp-rm-vm/` boots GSP/uproc firmware in a RISC-V QEMU VM
   with a GDB stub and adds Frida-like runtime inspection, a scripting/plugin
@@ -142,7 +143,8 @@ chain succeeds.
   self-test and ladder `REAL COPY OK`, clean release). Forwarding was never
   affected (HPAs are resolved at submit time).
 - **Guest CE copies are hardware-authored with verified data, with deferred
-  exactly-once release (P6b waves 1.5 and 2, 2026-10-02, run p6w2d).** Wave
+  exactly-once release (P6b waves 1.5 and 2, 2026-10-02, run p6w2d;
+  superseded as a frontier by the P6c relay and H1 bullets above).** Wave
   1.5 (run p6w15), with `gsp_phys_work_defer_release=1` (plus
   `gsp_phys_work_forward=1 gsp_phys_guest_mem=1`): a forwarded guest push's
   releases are recorded and its GPFIFO entry is held until the hardware copy
@@ -235,7 +237,8 @@ chain succeeds.
   distribution driver rebound, protected region down, exactly 5 power-domain
   locks intentionally held, 0 in-window rail cycles, no reboot. Module sha256
   `e1bce651...`. **Next: P6b** - forwarding guest submissions (page-directory
-  shadowing, real guest channels, push forwarding) - not implemented yet.
+  shadowing, real guest channels, push forwarding) - landed afterwards
+  (superseded by the P6b/P6c bullets above).
 - **Guest-memory path (P6) first attempts (2026-10-02 ~02:42 onward; resolved
   by the P6a result above).** The
   rung puts guest memory into the real execution path: guest RAM is now a
@@ -277,7 +280,8 @@ chain succeeds.
   pre-existing expected behavior. **Next: P6** (guest page tables/memory into
   the real MMU so guest data itself executes on hardware; includes the
   IOMMU/DMA reachability known-unknown), with optional **P5 wave 2**
-  (hardware-primary release relay with a non-applying decode mode); **P4**
+  (hardware-primary release relay with a non-applying decode mode; **subsumed**
+  by P6b wave 2 + the P6c relay above); **P4**
   remains deferred.
 - **Read-only control forwarding (P3) landed and verified live (2026-10-02
   01:33-01:45).** `GSP_RM_CONTROL` (fn76) forwards a whitelist of
@@ -318,7 +322,8 @@ chain succeeds.
   driver rebound, exactly 5 power locks intentionally left held, 0 in-window
   rail cycles, host stable, no reboot. Module sha256 `69eac176...`. P3
   read-only control forwarding has since landed and verified live (bullet
-  above). Guest work is still **not** forwarded to the physical engine. The
+  above). At that point guest work was not yet forwarded to the physical
+  engine (the P6b/P6c forwarding results are above). The
   P2 bundle and the G0 freeze fix have since been committed (`56766b2`).
 - **G0 fix bundle built after the P1 release-path freeze (2026-10-02 00:55).**
   The first P1 mdev-owned acceptance cycle ended in the known hard freeze
@@ -368,7 +373,8 @@ chain succeeds.
   module-owned path succeeds with the very same preconditions in the same
   boot). Guest wiring proceeds on the module-owned live session
   (`gsp_phys_gpu=1` claim+boot at load, proven); the P1 async claim stays
-  fail-open. No guest work is forwarded to the physical engine yet.
+  fail-open. At that point no guest work was forwarded to the physical engine
+  yet (the P6b/P6c forwarding results are above).
 - **Guest-wiring rung opened (2026-10-01 22:00): the first backend-session
   API code is in the tree, inert.** A skeleton that exposes the live physical
   session to the device layer (session request entry points plus a static-info
@@ -381,7 +387,8 @@ chain succeeds.
   [`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md).
   Post-single-VM-CUDA research notes (concurrent multi-VM virtual functions,
   3D acceleration, and a native-vs-guest performance program; and a host-side
-  acceleration feasibility study) were produced under `backups/` (untracked).
+  acceleration feasibility study) were produced and are kept in the local
+  evidence archive (untracked).
 - **Submission-rung milestone (2026-10-01 17:24-21:05): real work now executes
   on the physical engine.** The host resource-manager client allocated a
   virtual-address-space object with a committed page directory, a channel-group
@@ -398,8 +405,8 @@ chain succeeds.
   wiring this live session into the guest-facing mdev/vfio path so guest
   submissions execute on the physical engine (phased plan:
   [`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md)).
-  Evidence: `backups/evidence-20261001-cycle18/` through `...-cycle25/`, with
-  per-wall reports in `backups/evidence-20261001-reports/next/`.
+  Evidence: the local evidence archive (cycle 18 through cycle 25 runs, with
+  per-wall reports).
 - **Physical-backend milestone (2026-10-01 16:45): the host-owned GSP-RM
   completes initialization on the real TU104 and serves the static-config
   RPC.** Three supervised cycles, no reboot, clean release each time:
@@ -422,8 +429,7 @@ chain succeeds.
   The physical backend is now a live, queryable GSP-RM: the host can boot it,
   send RM RPCs and receive full replies, and shut it down gracefully. The
   next rung is to drive the guest-facing emulation from this backend.
-  Evidence: `backups/evidence-20261001-cycle11/`, `...-cycle12/`,
-  `...-cycle13/`.
+  Evidence: the local evidence archive (cycles 11-13).
 - **Driver-API baseline re-verified green on the current build** (2026-10-01
   14:20): the smoke test passes end-to-end (`ALL OK`, payload 64/64) with
   real releases (`faked=0`, `miss=0`, `stale=0`); the source-level root-cause