summaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-02 11:28:49 -0400
committerArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-02 11:28:49 -0400
commitc17da06d6cd8b9c69449904637bf37729a962989 (patch)
treeb51618fd63b776cd545f64fd40729af6154996b2
parent002e3d7d13c414ea874abbee8d82803dd3a72fe5 (diff)
downloadopen-iov-c17da06d6cd8b9c69449904637bf37729a962989.tar.gz
open-iov-c17da06d6cd8b9c69449904637bf37729a962989.zip
docs: engineering-operations playbook and project-wide status accuracy pass
- docs/eng-ops.md gains Part II, live physical-backend engineering operations: hard rules, the full module-parameter matrix, the routine supervised cycle and clean-release checklist, the debugfs inventory, how to read gsp-phys-state, a debugging scenario catalogue (CTXNOTVALID/promote wedge, fail-safe WPR2, unload hangs, guest wedges, MMU/PTE and PBDMA faults, relay symptoms), a hazard register (crash causes and debugging side effects), the physical-backend OODA loop, and the work-context map with porting flows - architecture, implementation, operations, orientation, getting-started, portability, ecosystem and reference docs corrected to the current state: CUDA driver-API smoke green, CE-class work forwarding and engine-authored releases plus fault relay in place (default off), GR execution blocked on engine-context validity, the promote wedge and designed locked fail-safe, and P5 wave-2 subsumed - internal-reference cleanup in tracked docs (no analysis-source or local-archive references); historical architecture document bannered - README reading list and STATE-CURRENT frontier refreshed; new pages (runbook, green-eye user guide, porting sequence, porting/packaging notes) linked from the tree Link check: scripts/check-docs-links.py passes (56 files, no broken links).
-rw-r--r--STATE-CURRENT.md78
-rw-r--r--debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md319
-rw-r--r--debug/risc-v-gsp-rm-vm/docs/29-PACKAGING-AND-TRIALING.md246
-rw-r--r--docs/01-orientation/01-purpose-and-scope.md19
-rw-r--r--docs/02-getting-started/01-install-and-first-run.md13
-rw-r--r--docs/03-architecture/01-system-overview.md12
-rw-r--r--docs/03-architecture/02-guest-stack-and-cudasw.md15
-rw-r--r--docs/03-architecture/03-gsp-and-gsp-rm.md12
-rw-r--r--docs/03-architecture/04-uvm.md16
-rw-r--r--docs/03-architecture/05-gmmu-amap-hal-gr.md3
-rw-r--r--docs/03-architecture/06-host-module-and-hybrid-model.md37
-rw-r--r--docs/03-architecture/07-driver-lifecycle-field-guide.md10
-rw-r--r--docs/03-architecture/08-physical-gpu-integration.md91
-rw-r--r--docs/03-architecture/09-full-system-picture.md38
-rw-r--r--docs/03-architecture/10-interaction-and-scheduling.md24
-rw-r--r--docs/03-architecture/11-physical-backend-guest-wiring.md109
-rw-r--r--docs/04-implementation/01-known-knowns-and-unknowns.md64
-rw-r--r--docs/04-implementation/02-practical-challenges-and-adversarial-interop.md8
-rw-r--r--docs/05-operations/01-performance-correctness-security.md23
-rw-r--r--docs/05-operations/02-debugging-and-maintenance-workflows.md28
-rw-r--r--docs/05-operations/04-green-eye-user-guide.md439
-rw-r--r--docs/06-portability/03-device-validation-and-porting-sequence.md232
-rw-r--r--docs/06-portability/hardware-software-matrix.md46
-rw-r--r--docs/07-developers/02-development-and-analysis-environment.md8
-rw-r--r--docs/07-developers/onboarding-and-standards.md12
-rw-r--r--docs/08-ecosystem/haaeep-and-collaboration.md2
-rw-r--r--docs/09-reference/glossary-and-faq.md16
-rw-r--r--docs/README.md64
-rw-r--r--docs/RUNBOOK.md744
-rw-r--r--docs/eng-ops.md850
-rw-r--r--docs/future-eng-ops.md261
31 files changed, 3603 insertions, 236 deletions
diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md
index 7aaf9f1..9f5ea01 100644
--- a/STATE-CURRENT.md
+++ b/STATE-CURRENT.md
@@ -1,6 +1,6 @@
# Open-IOV Current State
-> **Snapshot: 2026-10-02 10:55.** This is a dated public status page, not live
+> **Snapshot: 2026-10-02 ~11:00.** This is a dated public status page, not live
> state; it is refreshed on every progress check. Live working notes are
> maintained outside this repository; the public known-knowns/unknowns document
> is authoritative here:
@@ -69,7 +69,7 @@ meant to execute on the physical GPU.
and destination equal to source. Every cycle released the device cleanly
with no reboot; the next rung is wiring the live physical session into the
guest-facing mdev/vfio path
- ([`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md)). The guest-wiring rung then advanced: an inert backend-session API was added, and the mediated-device-owned lifecycle (P1, see the frontier section) was implemented and exercised - deferring the physical claim/boot to the first device create, releasing it on the last remove, with fail-open behaviour. The first P1 acceptance cycle exposed two defects - an asynchronous boot marked READY after a failed FWSEC boot, and the P1 release path freezing the host (see the frontier section). The G0 fix bundle (default-latched D0/D3cold locks, post-rebind readiness gate, pre-claim D0 settle, READY gated on a proven boot, pre-kick staging readback) is implemented, built, smoke-tested and now **validated live**: two module-owned supervised cycles (2026-10-02 ~00:57-00:58) were 2/2 green through claim, FWSEC, booter, RISC-V active, init-done (`0x1001`, `rpc_result=0` in ~955 ms) and a latched release with WPR2 down, exactly 5 `power domain left LOCKED` lines, 0 in-window rail cycles, host alive and pstore empty. The retested P1 mdev-owned path - the exact path that hard-froze on 2026-10-01 - now fails open safely (the READY gate releases the claim when no live GSP-RM session exists; no freeze), while the async FWSEC boot defect (`staging not visible before kick` -> `FWSEC/FRTS execution failed: -5`) remains open, quarantined off the critical path; guest wiring proceeds on the module-owned live session. The bundle was later committed (`56766b2`). Guest-wiring P2 (static-config identity overlay) has since landed and verified live (2026-10-02 01:20-01:45): with `gsp_phys_static_live=1` the guest's `nvidia-smi -q` Product Name comes from the live session (`NVIDIA GeForce RTX 2080 Super with Max-Q Design`) versus the emulated `NVIDIA GeForce RTX 2080 SUPER` with it off (A/B; overlay applied 4x/0), while the guest smoke stays GREEN `ALL OK` 64/64; one overlay-OFF full run hit a pre-existing emulated-path flake, watched. P3 (read-only control forwarding) has since landed and verified live (2026-10-02 ~01:33-01:45): the `gsp_phys_ctrl_forward` whitelist served 22 controls live with 2 fallbacks and 0 timeouts while the guest smoke stayed GREEN `ALL OK` 64/64; **P5 wave 1 (guest-submission shadow execution) has since landed and verified live** (2026-10-02 ~02:00-02:20): with `gsp_phys_work_forward=1` the sema pump shadow-executes eligible CE pushes on the physical engine through a dedicated ordered workqueue - 514 real 4 KiB copies during a GREEN guest smoke (`ALL OK` 64/64), 194 requests coalesced/dropped while one was in flight, 0 refusals, 0 timeouts - while the software release path stays authoritative and no guest memory or guest VA is touched; P6 has since proven guest-page DMA live (P6a, 2026-10-02: the physical copy engine copied 4 KiB to and from a real guest RAM page with the semaphore released while the guest smoke stayed GREEN `ALL OK` 64/64 and release stayed latched with no reboot) and then **P6b wave 1** (2026-10-02, run p6run13): with `gsp_phys_work_forward=1 gsp_phys_guest_mem=1` the sema service resolved real guest CE-copy operands to host physical addresses and executed them on the physical copy engine through module-built GMMU scratch slots (`req=42 ok=42 verify_ok=42 verify_fail=0`, 503 ladder shadow copies also ran) while the software release path stayed authoritative - guest CE copies now execute on hardware and are **hardware-authored with verified data** while the software copy remains the fail-closed fallback for unqueued or failed copies: wave 1.5 added deferred, exactly-once release (`engaged=75 applied=75`), and wave 2 added `gsp_phys_work_hw_primary=1`, which skips the software copy for forwarded ops (`copy_skipped=119`, `req=100 ok=100 verify_ok=100`, smoke `matched=64/64` `ALL OK`, 19 inline software fallbacks `copy_soft=19 copy_fail=0`, and a 121-copy fallback-recovery run with guest-RAM ranges unregistered). The guest smoke stayed GREEN `ALL OK` 64/64 and release stayed latched with no reboot. Real page-directory shadowing and guest channels (P6c: completion/fault relay, guest page tables, `vectorAdd`) remain, along with the P6a trigger-page pinning follow-up, and P4 (alloc/free handle adoption) remains deliberately deferred. |
+ ([`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md)). The guest-wiring rung then advanced: an inert backend-session API was added, and the mediated-device-owned lifecycle (P1, see the frontier section) was implemented and exercised - deferring the physical claim/boot to the first device create, releasing it on the last remove, with fail-open behaviour. The first P1 acceptance cycle exposed two defects - an asynchronous boot marked READY after a failed FWSEC boot, and the P1 release path freezing the host (see the frontier section). The G0 fix bundle (default-latched D0/D3cold locks, post-rebind readiness gate, pre-claim D0 settle, READY gated on a proven boot, pre-kick staging readback) is implemented, built, smoke-tested and now **validated live**: two module-owned supervised cycles (2026-10-02 ~00:57-00:58) were 2/2 green through claim, FWSEC, booter, RISC-V active, init-done (`0x1001`, `rpc_result=0` in ~955 ms) and a latched release with WPR2 down, exactly 5 `power domain left LOCKED` lines, 0 in-window rail cycles, host alive and pstore empty. The retested P1 mdev-owned path - the exact path that hard-froze on 2026-10-01 - now fails open safely (the READY gate releases the claim when no live GSP-RM session exists; no freeze), while the async FWSEC boot defect (`staging not visible before kick` -> `FWSEC/FRTS execution failed: -5`) remains open, quarantined off the critical path; guest wiring proceeds on the module-owned live session. The bundle was later committed (`56766b2`). Guest-wiring P2 (static-config identity overlay) has since landed and verified live (2026-10-02 01:20-01:45): with `gsp_phys_static_live=1` the guest's `nvidia-smi -q` Product Name comes from the live session (`NVIDIA GeForce RTX 2080 Super with Max-Q Design`) versus the emulated `NVIDIA GeForce RTX 2080 SUPER` with it off (A/B; overlay applied 4x/0), while the guest smoke stays GREEN `ALL OK` 64/64; one overlay-OFF full run hit a pre-existing emulated-path flake, watched. P3 (read-only control forwarding) has since landed and verified live (2026-10-02 ~01:33-01:45): the `gsp_phys_ctrl_forward` whitelist served 22 controls live with 2 fallbacks and 0 timeouts while the guest smoke stayed GREEN `ALL OK` 64/64; **P5 wave 1 (guest-submission shadow execution) has since landed and verified live** (2026-10-02 ~02:00-02:20): with `gsp_phys_work_forward=1` the sema pump shadow-executes eligible CE pushes on the physical engine through a dedicated ordered workqueue - 514 real 4 KiB copies during a GREEN guest smoke (`ALL OK` 64/64), 194 requests coalesced/dropped while one was in flight, 0 refusals, 0 timeouts - while the software release path stays authoritative and no guest memory or guest VA is touched; P6 has since proven guest-page DMA live (P6a, 2026-10-02: the physical copy engine copied 4 KiB to and from a real guest RAM page with the semaphore released while the guest smoke stayed GREEN `ALL OK` 64/64 and release stayed latched with no reboot) and then **P6b wave 1** (2026-10-02, run p6run13): with `gsp_phys_work_forward=1 gsp_phys_guest_mem=1` the sema service resolved real guest CE-copy operands to host physical addresses and executed them on the physical copy engine through module-built GMMU scratch slots (`req=42 ok=42 verify_ok=42 verify_fail=0`, 503 ladder shadow copies also ran) while the software release path stayed authoritative - guest CE copies now execute on hardware and are **hardware-authored with verified data** while the software copy remains the fail-closed fallback for unqueued or failed copies: wave 1.5 added deferred, exactly-once release (`engaged=75 applied=75`), and wave 2 added `gsp_phys_work_hw_primary=1`, which skips the software copy for forwarded ops (`copy_skipped=119`, `req=100 ok=100 verify_ok=100`, smoke `matched=64/64` `ALL OK`, 19 inline software fallbacks `copy_soft=19 copy_fail=0`, and a 121-copy fallback-recovery run with guest-RAM ranges unregistered). The guest smoke stayed GREEN `ALL OK` 64/64 and release stayed latched with no reboot. Real page-directory shadowing and guest channels (P6c: completion/fault relay, guest page tables, `vectorAdd`) have since been reached: guest CE copies and their releases are forwarded with release and fault relay in place (P6c C0/C1/C3, default off); the host GR ladder (H1) brings up a GR0 channel and a `TURING_COMPUTE_A` object, but the first push faults `CTXNOTVALID` (the external-VAS context was never committed) and the promote-binding attempt wedged GSP-RM (host assert, fuse-locked WPR2), so the device is held in the designed locked fail-safe pending a supervised reboot (see the frontier section). Real guest page tables/channels and GR execution remain, along with the P6a trigger-page pinning follow-up, and P4 (alloc/free handle adoption) remains deliberately deferred. |
| Graphics (3D), video | ❌ |
| Faults, multi-version, Windows, migration, multi-tenancy, performance | ❌ |
@@ -97,36 +97,37 @@ chain succeeds.
cycles. The acting session deliberately paused here; the reboot request is
awaiting approval because other work is running, and no further live GPU
cycles are planned this boot. The promote-assert bisection plan is ready
- for the next boot.
-- **Guest-submission relay and fault replay landed, default off (P6c
+ for the next boot. The promote is gated by `gsp_phys_rmcall_gr_promote`
+ (default 0), and the parameter documentation warns that it can wedge the
+ device.
+- **Guest-submission relay and fault relay landed, default off (P6c
C0/C1/C3, 2026-10-02 07:43-08:07).** With `gsp_phys_relay=1` the physical
engine authors the guest CE copy and the release semaphore: the
engine-authored release was observed in the guest page (`relays=64`),
releases are delivered exactly once (deferred hold), and the relay-on run
stayed green (`requested=68 ok=67 fallback=9`, smoke `ALL OK` 64/64). A
forced-unresolvable control proved the software fallback recovers every
- copy exactly once. One relayed copy released its semaphore but failed the
- host-side data compare, so the parameter stays default-off pending triage;
- verification hardening now logs the mismatch (sem target/payload/poll,
- first differing byte, live/snapshot dumps) and falls back to the software
- apply. `gsp_phys_fault_relay=1` relays forwarded-engine faults to the guest
- channel notifier/event path (live green, default off). The software path is
- unchanged with all parameters off.
-- **Host GR ladder (H1) opened and hit the next wall (2026-10-02 08:29).**
- The host resource-manager ladder is now selectable to the compute engine
- (`gsp_phys_rmcall_engine=1`, default off) and allocates a GR0 channel with
- a compute-class object on the live GSP-RM (`engine=GR0 hGr=0xcaf0002c`), but
- the first host GR semaphore push did not complete: the submit timed out
- (`-110`) with no release. The copy-engine control run stayed green
- (`REAL COPY OK`), and both runs released latched and clean (protected
- region down, distribution driver rebound, exactly 5 power-domain locks,
- 0 rail cycles, same boot). Current wall: make the GR engine execute the
- submitted compute block - bisect GR allocation parameters, the release
- method encoding and the method order/context init, using the drained GR
- fault records. The timeout was then root-caused to an uncommitted GR engine
- context (`CTXNOTVALID`); the follow-up promote-context attempt wedged the
- server - see the top bullet. Next: host `vectorAdd`, then guest GR
- forwarding.
+ copy exactly once. One relayed copy of 68 released its semaphore but failed
+ the host-side data compare; it was handled fail-closed (software apply) and
+ the parameter stays default-off pending triage, with hardening now in
+ place: retry-once, snapshot capture and relay counters. `gsp_phys_fault_relay=1`
+ relays forwarded-engine faults to the guest channel notifier/event path; it
+ delivered a real CE0 fault live (default off), and the auxiliary event post
+ reported `no_event`. The software path is unchanged with all parameters off.
+- **Host GR ladder (H1) opened and hit the GR context wall (2026-10-02
+ 08:29).** The host resource-manager ladder is now selectable to the compute
+ engine (`gsp_phys_rmcall_engine=1`, default off) and brings up a GR0
+ channel with a `TURING_COMPUTE_A` compute object on the live GSP-RM
+ (`engine=GR0 hGr=0xcaf0002c`), but the first host GR semaphore push faults:
+ the submit timed out (`-110`) with no release and the drained fault records
+ read `CTXNOTVALID` - the external-VAS context was never committed. The
+ copy-engine control run stayed green (`REAL COPY OK`), and both runs
+ released latched and clean (protected region down, distribution driver
+ rebound, exactly 5 power-domain locks, 0 rail cycles, same boot). The
+ follow-up context-promote binding attempt wedged GSP-RM (host assert,
+ fuse-locked WPR2) - see the top bullet. Current wall: commit the GR context
+ and make the engine execute the submitted compute block. Next: host
+ `vectorAdd`, then guest GR forwarding.
- **Experimental tooling: green-eye (2026-10-02 08:36, in development).**
`debug/risc-v-gsp-rm-vm/` boots GSP/uproc firmware in a RISC-V QEMU VM
with a GDB stub and adds Frida-like runtime inspection, a scripting/plugin
@@ -142,7 +143,8 @@ chain succeeds.
self-test and ladder `REAL COPY OK`, clean release). Forwarding was never
affected (HPAs are resolved at submit time).
- **Guest CE copies are hardware-authored with verified data, with deferred
- exactly-once release (P6b waves 1.5 and 2, 2026-10-02, run p6w2d).** Wave
+ exactly-once release (P6b waves 1.5 and 2, 2026-10-02, run p6w2d;
+ superseded as a frontier by the P6c relay and H1 bullets above).** Wave
1.5 (run p6w15), with `gsp_phys_work_defer_release=1` (plus
`gsp_phys_work_forward=1 gsp_phys_guest_mem=1`): a forwarded guest push's
releases are recorded and its GPFIFO entry is held until the hardware copy
@@ -235,7 +237,8 @@ chain succeeds.
distribution driver rebound, protected region down, exactly 5 power-domain
locks intentionally held, 0 in-window rail cycles, no reboot. Module sha256
`e1bce651...`. **Next: P6b** - forwarding guest submissions (page-directory
- shadowing, real guest channels, push forwarding) - not implemented yet.
+ shadowing, real guest channels, push forwarding) - landed afterwards
+ (superseded by the P6b/P6c bullets above).
- **Guest-memory path (P6) first attempts (2026-10-02 ~02:42 onward; resolved
by the P6a result above).** The
rung puts guest memory into the real execution path: guest RAM is now a
@@ -277,7 +280,8 @@ chain succeeds.
pre-existing expected behavior. **Next: P6** (guest page tables/memory into
the real MMU so guest data itself executes on hardware; includes the
IOMMU/DMA reachability known-unknown), with optional **P5 wave 2**
- (hardware-primary release relay with a non-applying decode mode); **P4**
+ (hardware-primary release relay with a non-applying decode mode; **subsumed**
+ by P6b wave 2 + the P6c relay above); **P4**
remains deferred.
- **Read-only control forwarding (P3) landed and verified live (2026-10-02
01:33-01:45).** `GSP_RM_CONTROL` (fn76) forwards a whitelist of
@@ -318,7 +322,8 @@ chain succeeds.
driver rebound, exactly 5 power locks intentionally left held, 0 in-window
rail cycles, host stable, no reboot. Module sha256 `69eac176...`. P3
read-only control forwarding has since landed and verified live (bullet
- above). Guest work is still **not** forwarded to the physical engine. The
+ above). At that point guest work was not yet forwarded to the physical
+ engine (the P6b/P6c forwarding results are above). The
P2 bundle and the G0 freeze fix have since been committed (`56766b2`).
- **G0 fix bundle built after the P1 release-path freeze (2026-10-02 00:55).**
The first P1 mdev-owned acceptance cycle ended in the known hard freeze
@@ -368,7 +373,8 @@ chain succeeds.
module-owned path succeeds with the very same preconditions in the same
boot). Guest wiring proceeds on the module-owned live session
(`gsp_phys_gpu=1` claim+boot at load, proven); the P1 async claim stays
- fail-open. No guest work is forwarded to the physical engine yet.
+ fail-open. At that point no guest work was forwarded to the physical engine
+ yet (the P6b/P6c forwarding results are above).
- **Guest-wiring rung opened (2026-10-01 22:00): the first backend-session
API code is in the tree, inert.** A skeleton that exposes the live physical
session to the device layer (session request entry points plus a static-info
@@ -381,7 +387,8 @@ chain succeeds.
[`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md).
Post-single-VM-CUDA research notes (concurrent multi-VM virtual functions,
3D acceleration, and a native-vs-guest performance program; and a host-side
- acceleration feasibility study) were produced under `backups/` (untracked).
+ acceleration feasibility study) were produced and are kept in the local
+ evidence archive (untracked).
- **Submission-rung milestone (2026-10-01 17:24-21:05): real work now executes
on the physical engine.** The host resource-manager client allocated a
virtual-address-space object with a committed page directory, a channel-group
@@ -398,8 +405,8 @@ chain succeeds.
wiring this live session into the guest-facing mdev/vfio path so guest
submissions execute on the physical engine (phased plan:
[`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md)).
- Evidence: `backups/evidence-20261001-cycle18/` through `...-cycle25/`, with
- per-wall reports in `backups/evidence-20261001-reports/next/`.
+ Evidence: the local evidence archive (cycle 18 through cycle 25 runs, with
+ per-wall reports).
- **Physical-backend milestone (2026-10-01 16:45): the host-owned GSP-RM
completes initialization on the real TU104 and serves the static-config
RPC.** Three supervised cycles, no reboot, clean release each time:
@@ -422,8 +429,7 @@ chain succeeds.
The physical backend is now a live, queryable GSP-RM: the host can boot it,
send RM RPCs and receive full replies, and shut it down gracefully. The
next rung is to drive the guest-facing emulation from this backend.
- Evidence: `backups/evidence-20261001-cycle11/`, `...-cycle12/`,
- `...-cycle13/`.
+ Evidence: the local evidence archive (cycles 11-13).
- **Driver-API baseline re-verified green on the current build** (2026-10-01
14:20): the smoke test passes end-to-end (`ALL OK`, payload 64/64) with
real releases (`faked=0`, `miss=0`, `stale=0`); the source-level root-cause
diff --git a/debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md b/debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md
new file mode 100644
index 0000000..4669f93
--- /dev/null
+++ b/debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md
@@ -0,0 +1,319 @@
+# 28 — Porting automation: driver-version sweep and new-device scaffold
+
+> **# TOOL IN DEVELOPMENT - NOT READY YET!** See 00 for status and
+> [`GREEN-EYE-ROADMAP.md`](../GREEN-EYE-ROADMAP.md) H4/R6 for the program.
+
+Two stdlib-only orchestrators turn the docs/07 P1/P2 playbooks and the
+docs/22 porting guide into one repeatable chain: a **driver-version sweep**
+(`harness/port_driver_versions.py`) and a **device-port scaffold**
+(`harness/port_new_device.py`). They reuse the version-diff, portability-matrix
+and mutation machinery already documented in docs/18 and docs/21, add the
+RM 610 gate verbatim from `version_diff.py`, and never download firmware,
+never touch hardware, and write only under `--out`. Both are **unadmitted
+pilots** under `eng-ops.md` §2 — the keep/retire evaluation is F2/F3 in
+[`future-eng-ops.md`](../../../docs/future-eng-ops.md), and a sweep is not
+routine release tooling until that record exists.
+
+| Tool | Roadmap | What it does |
+|---|---|---|
+| `harness/port_driver_versions.py` | **R6**, H4.1 | Finds each driver/firmware version's local image, runs version_diff vs a baseline, the portability matrix and a mutation smoke, and assembles per-version evidence bundles plus a cross-version summary with the docs/06 M1–M6 skeleton. |
+| `harness/port_new_device.py` | **R6**, H4.1, docs/07 P2 | Creates/uses a user-supplied device profile, probes the device firmware's first touch, builds a symbol inventory, runs the matrix against a reference, and emits a port checklist with a go/no-go recommendation. |
+
+Both are dry-run by default, resume-safe (sweep), bounded by QEMU budgets plus
+a per-step wall-clock net, and simulator-only (no `/dev`, kernel module, mdev,
+guest or physical GPU).
+
+---
+
+## 1. The chain
+
+```text
+new driver / GSP image lands on disk (runfile, .fwimage, re-hosted ELF)
+ |
+ +-- port_driver_versions --dry-run plan: inputs found/missing, steps, artifacts
+ | |
+ | +-- --run per version:
+ | 1. target discovery/identification (never downloads)
+ | 2. version_diff vs the chosen baseline
+ | 3. portability_matrix (baseline + candidate)
+ | 4. mutation_smoke (bounded, seeded)
+ | 5. per-version bundle-manifest.json
+ | +-- sweep-summary.json/.md (M1-M6 skeleton, pass/fail interpretation)
+ |
+ +-- port_new_device --dry-run plan: profile fields, firmware, steps
+ |
+ +-- --run profile + probe-first-touch + symbols + matrix + checklist
+```
+
+The chain is the automation of docs/07 P1 (new GSP version), P2 (same version,
+new device) and the docs/22 port checklists; the metric contract stays
+docs/06 §5. Evidence is always attachable to the docs/07 checklist and the
+docs/14 §6 pack.
+
+---
+
+## 2. Driver-version sweep
+
+### 2.1 Known branch cadence
+
+The default version list documents the branch cadence and where to edit it —
+the tool never invents an image or downloads one:
+
+| branch | expected image | default sweep |
+|---|---|---|
+| 540 | earlier branch | yes |
+| 550 | earlier branch | yes |
+| 560 | earlier branch | yes |
+| 570 | current anchor (570.144 and patches) | yes |
+| 580 | expected cadence, no local image assumed | no |
+| 610 | RM 610 gate, refused without approval (docs/23) | no |
+
+Edit `DEFAULT_VERSIONS` in `harness/port_driver_versions.py`, or override on
+the command line / in a config file. A branch with no local image is reported
+`no-input`; it is not an error and nothing is fetched.
+
+### 2.2 Invocation
+
+Dry-run first (the default; reads input directories only):
+
+```bash
+cd debug/risc-v-gsp-rm-vm
+python3 harness/port_driver_versions.py --input-dir ../../firmware/57X-GSP \
+ --versions 540,550,560,570
+```
+
+Execute, resume-safe, with explicit targets and per-version symbol DBs:
+
+```bash
+python3 harness/port_driver_versions.py --run \
+ --versions 570.144 \
+ --target 570.144=../../firmware/57X-GSP/570.144/gsp-570.144.fwimage.bin \
+ --baseline 570.144 \
+ --symbols 570.144=symbols-570.144.json \
+ --out dumps/sweep-570.144
+```
+
+Cron (weekly, quiet, resume-safe; completed versions are skipped):
+
+```cron
+17 3 * * 1 cd /path/to/open-iov/debug/risc-v-gsp-rm-vm && \
+ python3 harness/port_driver_versions.py --run \
+ --input-dir ../../firmware/57X-GSP --versions 540,550,560,570 \
+ --out dumps/sweep-570 >> dumps/sweep-570/cron.log 2>&1
+```
+
+Agent (docs/20): run the dry-run to get the plan, use
+`portability_matrix_run`/`mutation_run` for single steps if the sweep itself is
+not exposed by MCP yet, then read `sweep-summary.json` and the per-version
+`bundle-manifest.json` from the bundle instead of recomputing. The dry-run plan
+is machine-readable (`greeneye-port-sweep-plan/1`) and side-effect free.
+
+Flags that matter:
+
+* `--versions` / `--config` — explicit list or JSON config (below);
+* `--input-dir` — recursive discovery of local `.fwimage`, `.fwimage.bin`,
+ `.elf`, `.bin` and `.run` files; version matching uses file and directory
+ names (`570` matches `570.144`);
+* `--target VERSION=PATH`, `--symbols VERSION=PATH`,
+ `--ghidra-exports VERSION=PATH` — explicit inputs, no discovery;
+* `--baseline` — version diffed against (default: earliest version found);
+* `--extract-runfiles` — opt in to local `sh <runfile> --extract-only`
+ extraction when only a driver runfile is present (still never downloads);
+* `--force` — re-run versions whose `version.json` reports `ok`/`partial`;
+* `--step-timeout` — wall-clock safety net per step (`0` disables; QEMU
+ budgets remain the first bound);
+* the usual `version_diff`/`matrix`/`mutation` budgets.
+
+### 2.3 Configuration file
+
+```json
+{
+ "format": "greeneye-port-sweep-config/1",
+ "versions": ["540", "550", "560", "570.144"],
+ "baseline": "570.144",
+ "input_dir": "../../firmware/57X-GSP",
+ "targets": {"570.144": ["../../firmware/57X-GSP/570.144/gsp-570.144.fwimage.bin"]},
+ "symbols": {"570.144": "symbols-570.144.json"}
+}
+```
+
+Unknown keys are refused so a typo cannot silently change the sweep. CLI flags
+override config values.
+
+### 2.4 Output and evidence bundle
+
+```text
+<out>/
+ sweep-summary.json cross-version report (also .md)
+ bundle-manifest.json sha256 + size of every artifact below
+ <version>/
+ version.json status, inputs, steps, M1-M6, sha256
+ version.md rendered per-version report
+ bundle-manifest.json per-version evidence bundle
+ version-diff/ docs/21 §2.3 tree
+ portability-matrix/ portability-report.json|.md, symbols/, sessions/, compare/
+ mutation-smoke/ findings.jsonl, summary.md, sessions/
+```
+
+`sweep-summary.json` carries per-version status, the cross-version M1–M6
+skeleton (every agreement metric `not measured` without a reference capture,
+docs/06 §5), the RM 610 gate record, and an explicit `interpretation` string:
+`pass`, `fail` (with the failing versions) or `not-run`.
+
+### 2.5 Reading failures
+
+| Status | Meaning | Action |
+|---|---|---|
+| `ok` | all steps ran, none errored | attach the bundle to the docs/07 checklist |
+| `partial` | a step skipped (e.g. no QEMU, no mutation kernel) | explain the skip; re-run where the capability exists |
+| `error` | a step raised or timed out | inspect `<version>/version.json`, then the step report (`version-diff.json`, `portability-report.json`, `findings.jsonl`) |
+| `no-input` | no local image found for the version | add the image under `--input-dir`, pass `--target`, or drop the version from the list |
+| resumed | `version.json` already reported `ok`/`partial` | use `--force` to re-run |
+
+`--run` exits non-zero when any executed version errored or when nothing ran
+at all; skipped steps alone do not fail the sweep, but they are always listed.
+An unexplained delta blocks the docs/07 acceptance: an empty
+unexplained-delta list is a claim, not a default.
+
+### 2.6 RM 610 gate (hard, testable)
+
+Versions numerically `>= 610` or labelled `rm610`/`RM-610` are refused before
+any target file is opened — in dry-run too — unless `--approval-token
+RM610-APPROVED` is passed or `GREENEYE_RM610_APPROVED=1` is set. The refusal
+text points at [`docs/23-RM610-READINESS.md`](23-RM610-READINESS.md) and
+states that nothing was downloaded and no target was opened. Do not put a 610
+input in a default list or a cron line before the §6 approval message exists.
+
+---
+
+## 3. New-device scaffold
+
+### 3.1 Profile
+
+`--device <name>` uses `--profile FILE` or `<out>/profiles/<name>.json`. A
+missing profile is created as a scaffold (`greeneye-device-profile/1`) with the
+user-supplied sections and a `where_to_fill` map:
+
+* `pci` — vendor/device/subsystem IDs (`lspci`, vendor docs);
+* `bars` — BAR layout and sizes (guest `lspci -vvv`, host sysfs);
+* `fb` — VRAM/BAR1 geometry (host oracle `rmtool mem stats --format=json`);
+* `engines` — engine inventory (host oracle `rmtool engine list --format=json`);
+* `interrupts` — interrupt topology (MSI-X);
+* `rom` — ROM/VBIOS notes and hash;
+* `firmware` / `reference` — local image paths (or `--firmware`/`--reference`);
+* `firmware_family` / `firmware_version` — the pairing record (docs/07 P2.2);
+* `physical_validation` — supervised physical evidence, recorded by the human,
+ never by this tool.
+
+Profile data is user-supplied; the tool never guesses a device ID, BAR or
+engine list. `config/hardware/*.yaml` remains a design target (see
+hardware-software-matrix §6) — do not treat it as live configuration.
+
+### 3.2 Invocation
+
+```bash
+# dry-run: check profile fields and inputs, create nothing
+python3 harness/port_new_device.py --device rtx4090 --profile profiles/rtx4090.json
+
+# execute: scaffold profile (if absent), probe, symbols, matrix, checklist
+python3 harness/port_new_device.py --run --device rtx4090 \
+ --firmware ../../firmware/57X-GSP/570.144/gsp-570.144.fwimage.bin \
+ --reference firmware/uproc_demo/build/uproc_demo.elf \
+ --out dumps/port-rtx4090
+```
+
+Output tree:
+
+```text
+<out>/<device>/
+ device-port.json / device-port.md
+ port-checklist.json / port-checklist.md
+ symbols/<device>.json merged ELF/Ghidra/green-eye DB + provenance
+ sessions/<firmware>-probe/ recorded first-touch session
+ portability-matrix/ matrix report + sessions + compare
+ bundle-manifest.json sha256 of every artifact
+<out>/profiles/<device>.json scaffolded profile (when absent)
+```
+
+The checklist items are the docs/07 evidence checklist plus the docs/22 §3.1
+profile fields; the recommendation is `hold` until the simulator-side items
+are complete, `simulator-evidence-complete` while physical validation is
+pending, and `go` only when `physical_validation.status` is recorded. The last
+row is deliberately outside this tool's reach: simulator-only evidence never
+justifies a support claim (docs/22 §3.5, hardware-software-matrix §5.1).
+
+---
+
+## 4. Tests, CI and MCP ties
+
+```bash
+python3 -m unittest tests.test_porting_automation -v
+make test # includes these with the rest of the suite
+```
+
+`tests/test_porting_automation.py` is deterministic: `--help` clean,
+dry-runs side-effect free, missing-input handling without tracebacks, resume
+detection, config validation, the RM 610 gate (refusal, token, env), and a
+tiny one-version QEMU sweep against the demo ELF that must produce the
+evidence bundle; QEMU-dependent tests skip cleanly when QEMU or the demo ELF
+is absent. The repository CI workflow (docs/21 §5) runs the suite on pushes
+and PRs touching the tool tree; a full sweep against real firmware is driven
+manually/agentically, dry-run first, because it needs versioned inputs and
+symbol DBs — the same policy as `ci_release` (docs/21 §4.1).
+
+MCP (docs/20) exposes the step-level tools (`probe_firmware`,
+`portability_matrix_run`, `mutation_run`); the sweep is the CLI/agent entry
+point that sequences them and owns the bundle. An agent should read the dry-run
+plan, run, then read `sweep-summary.json`/`device-port.json` and the manifests
+rather than recompute hashes.
+
+---
+
+## 5. When to extend
+
+* **New branch cadence** — add the version to `--versions`/config; if it should
+ be swept by default, edit `DEFAULT_VERSIONS` and the cadence table above.
+ Never add an image you do not have; never add 580/610 to a default.
+* **New per-version artifact** — add a step function in
+ `harness/port_driver_versions.py`, register it in `run_version`, and list the
+ path in `_expected_artifacts`.
+* **New device profile field** — add it to `scaffold_profile`, `PROFILE_HELP`
+ and `profile_missing_fields`, and surface it in the checklist.
+* **New architecture (GB10-class, aarch64)** — follow docs/07 P3 first; the
+ sweep handles the ISA/image shape and the matrix records the first-touch
+ deltas, but PMU/SOE and partition-model work belongs to the roadmap.
+* **New product behaviour** — out of scope: these tools produce evidence, they
+ never edit `open-iov-vfio`. Product changes need the docs/07 P4 escalation
+ and a physical reference.
+
+---
+
+## 6. Safety and limits
+
+* No downloads: discovery and `--extract-runfiles` only read files already on
+ disk; a driver runfile is never fetched.
+* Simulator-only: own QEMU processes on demo/re-hosted ELF or raw firmware;
+ never a guest, kernel module, mdev, existing gdbstub or physical GPU.
+* Bounded: QEMU probe/scenario/mutation budgets plus `--step-timeout`; the
+ mutation harness refuses non-demo ELFs unless explicitly allowed.
+* No support claims: the sweep records what changed; the device checklist
+ holds until physical validation; the matrix and RUNBOOK process own the
+ "Supported" label (hardware-software-matrix §1).
+* Fidelity limits are docs/06: L0/L1 observations only, L2/L3 (timing,
+ interrupts, device model) explicitly not claimed.
+
+## 7. Roadmap mapping
+
+| Item | Where this doc/tool contributes |
+|---|---|
+| **R6 — multi-version orchestration** | One sweep over 540→current with version_diff + matrix + mutation and a cross-version report |
+| **H4.1 — semantic diffing** | Per-version symbol/corpus/behaviour deltas and recorded sessions |
+| **H4.3 — mutation/fuzzing** | Seeded mutation smoke per version |
+| **docs/07 P1/P2** | The playbooks become dry-run plans and evidence bundles |
+| **docs/23 C5/C7** | Release pipeline extended by a sweep-scale orchestrator; 610 gate reused |
+| **H7 — agentic platform** | Machine-readable plans, summaries and bundles for monitored agent runs |
+
+<!-- change (2026-10-02): state the unadmitted-pilot status and link the
+future-eng-ops F2/F3 evaluation from the document that ships the scaffolds;
+scope: intro only; benefit: no reader mistakes the sweep for admitted release
+tooling. -->
diff --git a/debug/risc-v-gsp-rm-vm/docs/29-PACKAGING-AND-TRIALING.md b/debug/risc-v-gsp-rm-vm/docs/29-PACKAGING-AND-TRIALING.md
new file mode 100644
index 0000000..8be3704
--- /dev/null
+++ b/debug/risc-v-gsp-rm-vm/docs/29-PACKAGING-AND-TRIALING.md
@@ -0,0 +1,246 @@
+# 29 — Packaging and trialing: sim + injected code as a deliverable
+
+> **# TOOL IN DEVELOPMENT - NOT READY YET!** See 00 for status.
+> This is a development-only convenience layer over docs/12 (injection SDK),
+> docs/16 (native simulator) and docs/24 (device map). It makes no hardware,
+> timing or security claim; the real `.fwimage` path is honestly blocked on
+> H2.2 and says so in every result.
+
+## 1. Purpose
+
+green-eye can already load an injected service into the native simulator and
+invoke it (docs/12, docs/16 §1.1, `harness/value_demos/s06_inject_service.py`).
+That evidence lives in the tool tree: a reviewer needs the checkout, the built
+demo and SDK, and the exact command sequence. `harness/package_sim.py` turns
+"the s06 scenario" into a self-contained, hash-addressed package that a third
+party can run, inspect and compare:
+
+* one directory (or deterministic `.tar.gz`) containing the target, the service
+ blob, the scenario and a manifest of hashes;
+* one entry point (`run.sh`) that boots the target **natively**, registers the
+ injected service, invokes it directly and through the firmware dispatcher,
+ runs a hook/watch pass, and writes `results.json` + `results.md`;
+* a machine-readable manifest so a recipient can verify exactly which inputs
+ produced a result before trusting it.
+
+It exists to make a code addition trialable and reviewable, not to ship
+anything: the eventual target is a modified, **unsigned** GSP image for the
+simulator with tested injected code (§7).
+
+## 2. Exact usage
+
+All commands run from `open-iov/debug/risc-v-gsp-rm-vm`. Prerequisites:
+`make demo && make sdk` (or pass `--service`), Python 3.10+; no QEMU needed.
+
+```bash
+# 1) Demo trial (defaults: demo uproc + sdk/build/injected_demo.elf, slot 6)
+python3 harness/package_sim.py --out /tmp/greeneye-trial-demo
+
+# 2) Build, archive and immediately run the trial
+python3 harness/package_sim.py --out /tmp/greeneye-trial-demo --tar --run
+/tmp/greeneye-trial-demo/run.sh # or run it yourself
+
+# 3) Combined artifact: the service is pre-injected at a fixed offset
+python3 harness/package_sim.py --pack-image --out /tmp/greeneye-trial-packed
+python3 harness/package_sim.py --pack-image --tar --run --out /tmp/greeneye-trial-packed
+
+# 4) Real 570.144 fwimage: image + service + lenient device map (blocked, see §5)
+python3 harness/package_sim.py --target image \
+ --image /path/to/gsp-570.144.fwimage.bin \
+ --device-map dumps/570.144-device-map.json \
+ --out /tmp/greeneye-trial-image
+
+# 5) Custom scenario and slot
+python3 harness/package_sim.py --script my_scenario.py --slot 6 --out /tmp/my-trial
+
+# 6) Tests
+python3 -m unittest tests.test_package_sim -v # 6 tests, ~4 s
+```
+
+Measured on the reference host (2026-10-02, CPython 3.12): a demo build plus
+`run.sh` reports `trial: ok` in ~1 s wall; the test file (6 tests, including
+the real-image bounded probe) runs in ~4.1 s.
+
+Flags: `--target demo|image` (default demo), `--image`, `--service`,
+`--device-map`, `--script`, `--slot 0..7` (default 6), `--pack-image`,
+`--tar`, `--run`, `--run-timeout`. `--out` must be empty or absent; builds do
+not overwrite.
+
+## 3. Package layout and manifest schema
+
+```
+<out>/
+ README.md what this is, how to run, what to expect, safety
+ run.sh single entry; finds the tool root (GREENEYE_HOME) and runs scenario.py
+ manifest.json format/version + hashes + component list + run contract
+ scenario.py the pre-scripted scenario as run (builtin or --script)
+ payload/
+ <target>.elf|fwimage the target as supplied
+ injected_demo.elf the service blob
+ device_map.json only for image packages when no --device-map was given
+ combined.elf only with --pack-image
+ preinject.json only with --pack-image (placement config)
+ results/ produced by run.sh: results.json, results.md, session/
+```
+
+`manifest.json` (`greeneye-trial/1`):
+
+| Key | Contents |
+|---|---|
+| `format`, `greeneye_version`, `created`, `results_schema` | identity; `created` is the only timestamp |
+| `target` | `kind` (demo/image), `file`, `input` (repo-relative when possible), `sha256`, `entry`, `min_vaddr`, `max_vaddr`, `bias`, `runnable`, `limitations` |
+| `service` | `file`, `input`, `sha256`, `entry`, `entry_symbol` (`ge_demo_service_dispatch`), `slot` |
+| `device_map` | `file`, `input`, `sha256`, `lenient`, `learn` (null for a plain demo) |
+| `script` | `kind` (builtin/custom), `name`, `input`, `sha256` |
+| `preinject` | `file`, `runnable`, `reason`, `target_load_addr`, `target_entry`, `service_load_addr`, `service_offset`, `service_entry`, `entry_addr`, `slot`, `size`, `segments` (null without `--pack-image`) |
+| `scenario` | demo boot/settle seconds, image instruction budget, `memory_mb`, expected call values |
+| `run` | `command`, expected result paths, `exit_nonzero_on: "assertion failure"` |
+| `safety`, `determinism` | policy text carried into the package |
+| `components` | sorted `{path, role, bytes, sha256}` for every packaged file except the manifest itself |
+
+`results.json` (`greeneye-trial-results/1`): `status` (`ok`/`failed`/
+`blocked`), `success`, `backend`, `target`, `service` (load/entry addresses,
+`preinjected`), `calls` (remote vs firmware with expected/agree), `hooks`
+(`rpc_dispatch` hits and first entries), `watches` (`gsp_ctrl.result` hits),
+`control`, `handlers_slot`, `uart_excerpt`, `session`, `assertions`
+(`checks`, `failed`), `limitations`, plus `error` and the blocked fields
+(§5) where applicable. `results.md` renders the same.
+
+## 4. How a user trials it and reads results
+
+1. Receive `<package>.tar.gz` plus its sha256 out of band; verify
+ `sha256sum <package>.tar.gz`.
+2. Extract and run: `./run.sh` (add `--budget N` for image packages). The
+ script finds the green-eye tree by walking up, or via
+ `GREENEYE_HOME=/path/to/risc-v-gsp-rm-vm`. Exit codes: `0` pass (or the
+ documented blocked image status), `1` assertion failure, `3` tree not found.
+3. Read `results/results.md` first, then `results/results.json` for machines.
+ Anything other than `status: ok` on a demo package is a finding, not a
+ flake: `assertions.failed` names the check (`handlers_slot_registered`,
+ `remote_ping`, `routes_agree`, `dispatch_hook_hits`, `uart_banner`, ...).
+4. For deeper review, point the tools at `results/session/`:
+ `./greeneye.sh analyze <pkg>/results/session` and
+ `python3 -m greeneye compare baseline/results/session candidate/results/session`.
+5. Verify provenance: each `components[].sha256` must match the packaged file;
+ `target.sha256`/`service.sha256` identify the inputs.
+
+The demo package asserts: the service is registered in
+`gsp_ctrl.handlers[slot]`; direct and mailbox invocations agree with the SDK
+ABI (`cmd 1: a0 ^ 0xbeef`, `cmd 2: a0 + a1`); at least one `rpc_dispatch`
+hook hit and one `gsp_ctrl.result` watch hit; and the UART banner/self-test
+lines are present. A bisected or broken service fails these and exits 1.
+
+## 5. The `--pack-image` path and its honest limits
+
+For the demo, `--pack-image` writes `payload/combined.elf`: the demo's PT_LOAD
+segments plus the service's segments relocated to `target_base + 0x01000000`
+(the same address `greeneye.inject.load_service` uses by default), merged into
+one ELF64. `payload/preinject.json` records the placement, and the scenario
+boots the combined ELF, writes the entry into the handler slot and skips the
+separate blob load. Round-trip verified in
+`tests/test_package_sim.py::test_pack_image_round_trip`: call results are
+identical to the load/register path and `service.preinjected` is `true` in
+`results.json`. This is the trial starting with the injected code already part
+of the simulated system.
+
+For the real `.fwimage`, the package contains the image, the service blob and
+a lenient `greeneye-device-map/1` map (given or emitted empty with
+`learn: true`). **The native simulator cannot boot it yet**: the H2.2 device
+model is absent. The scenario therefore runs a bounded, deterministic lenient
+first-touch probe and records `status: blocked`,
+`blocked_on_device_model: true`, `instructions_executed`, the stop PC, the
+unmodelled MMIO list and (when learning) `results/learned_device_map.json`.
+Reference measurement: a 200 000-instruction budget (199 998 executed),
+stopped at `pc=0x84009070` (firmware `0x4009070`), exactly one unmodelled
+access at `0x84009014` reading `0x2000000000000a00`, matching docs/24 §5. No
+service is loaded, because no RM service registration point exists yet (M5) and the default load
+address would overwrite the image. Exit is 0 for the documented blocked
+status; a thrown error is recorded and exits 1.
+
+## 6. Sharing the results of a code addition
+
+1. Author builds the package with the service under review
+ (`service.sha256` is the code addition's identity) and runs it.
+2. Attach `manifest.json`, `results/results.json`, `results/results.md` and
+ the `results/session/` directory; pre-injection evidence is in the session
+ as `service_injected`, `call_enter`/`call_return`, `watch_hit` and `bp_hit`
+ events, so the claim is replayable.
+3. A baseline package with the previous service is diffed with
+ `greeneye compare`; the delta is the behavior change.
+4. A reviewer re-runs the received package on the tool tree, verifies hashes
+ and expects the same `status`/call results. Divergence is a finding against
+ the package or the tree, not a nuance.
+5. Nothing here is product evidence by itself: any conclusion that would
+ change `open-iov-vfio` still escalates to a physical capture (docs/06 §6,
+ docs/25 §2.1). The package makes that physical confirmation cheap, it does
+ not replace it.
+
+## 7. Eventual path: a modified GSP image for the simulator
+
+Once H2.2 can boot the real image (docs/16 M4) and the RM service registry
+exists (M5), the same pipeline packages a **modified, unsigned** image:
+
+1. take the original `.fwimage` (hash in `target.sha256`); never a signed
+ production image, never a copy that can be flashed;
+2. record the patch/injection recipe (`preinject`) and the verified device map
+ as versioned profile data keyed by `(device, firmware-family, profile)` per
+ docs/16 §7;
+3. run the package's scenario, which boots the image under the H2.2 device
+ model, registers the tested service with the simulated RM core, and asserts
+ its behavior end to end;
+4. attach the session as evidence and escalate any product-relevant finding to
+ hardware.
+
+Until M4/M5, image packages are deliberately non-runnable and labelled so; the
+`preinject.runnable: false` + `reason` fields and the blocked result keep the
+gap visible instead of implying a boot.
+
+## 8. Security and safety
+
+* The package executes only inside the in-process simulator: no hardware
+ access, no mdev, no guest, no kernel module, no network. `run.sh` invokes
+ Python in the tool tree found locally.
+* Injected code runs with the simulator's full privileges and may corrupt
+ simulated firmware state (docs/12 §9). Treat every package as untrusted
+ code: run only packages whose sha256 and `manifest.json` you have verified,
+ in a scratch copy of the tree, and restart clean after a crash.
+* Never use a package to inject into a guest, a shipped environment or real
+ hardware; never sign, flash or install a modified `.fwimage`.
+* `/tmp` outputs are scratch; packages and sessions contain firmware bytes and
+ are development artifacts, not for redistribution outside the project.
+
+## 9. Review/evaluation criteria (keep vs prune)
+
+Keep the feature only while it earns its maintenance. This is an **unadmitted
+pilot** under `eng-ops.md` §2: the evaluation recorded in
+[`future-eng-ops.md`](../../../docs/future-eng-ops.md) F4 is the admission
+decision. It is worth keeping if:
+
+* `python3 -m unittest tests.test_package_sim` stays green in CI in well under
+ a minute, including the bounded real-image probe;
+* two builds under a fixed `SOURCE_DATE_EPOCH` are byte-identical
+ (manifest and `.tar.gz`) and an intentionally broken service fails the
+ assertions and exits 1 (add that negative test if it is missing);
+* at least one code addition was actually reviewed from a package by someone
+ who did not author it, with hashes verified and results reproduced;
+* the tar carries enough provenance (`manifest.components`, target/service/
+ device-map/script hashes, `preinject` placement) for a reviewer to trust a
+ result without touching the author's checkout;
+* the image path converts into a real H2.2 boot when M4/M5 land, and its
+ `blocked` status is removed rather than quietly left in place.
+
+Prune or rework it if: packages are never shared or reviewed (the s06 demo is
+enough); the M5 native service registry subsumes `--pack-image`; the manifest
+format churns without consumers; the real-image path stays permanently blocked
+and adds only maintenance; or package results are being quoted as product or
+hardware evidence.
+
+Evidence for this document: `harness/package_sim.py`,
+`tests/test_package_sim.py` (6 tests), docs/12, docs/16 §1.1, docs/24 §4–5,
+docs/25 §5–6.
+
+<!-- change (2026-10-02): use the tree's stdlib unittest runner instead of an
+undeclared pytest dependency, and state the unadmitted-pilot status with the
+future-eng-ops F4 evaluation; scope: §2 command, §9 only; benefit: the test
+command runs with the same interpreter as CI and the keep/prune decision is
+linked. -->
diff --git a/docs/01-orientation/01-purpose-and-scope.md b/docs/01-orientation/01-purpose-and-scope.md
index ff13603..278ad0d 100644
--- a/docs/01-orientation/01-purpose-and-scope.md
+++ b/docs/01-orientation/01-purpose-and-scope.md
@@ -131,15 +131,16 @@ document:
* **Working today**: the virtual device, the GSP boot handshake, the RPC
transport, the resource-manager emulation surface the guest driver needs to
- probe and enumerate, guest `nvidia.ko` and `nvidia-smi`, and CUDA user-space
- startup through `cuInit` / `cuDeviceGet`.
-* **Needs improvement**: reply fidelity and completion translation first —
- context creation currently ends in a forced `rc=999` on a completion wait,
- and some consumed reply fields are still generic fallbacks — then GPU work
- forwarding (the reason CUDA kernels do not yet run; the backend is chosen),
- the memory/page-table model, Windows guest support,
- non-KVM hypervisors, and performance instrumentation. Each of these is
- tracked with explicit "known-unknowns" in
+ probe and enumerate, guest `nvidia.ko` and `nvidia-smi`, and the full CUDA
+ driver-API path through context creation, allocation and verified copies
+ (the driver-API smoke passes).
+* **Needs improvement**: engine execution first — the backend is live and
+ CE-class copies and their releases are forwarded to the physical copy engine
+ (default off) with a fault relay, but CUDA kernel execution is blocked on GR
+ engine-context validity — then the remaining reply fidelity (some consumed
+ fields are still generic fallbacks), the memory/page-table model, Windows
+ guest support, non-KVM hypervisors, and performance instrumentation. Each of
+ these is tracked with explicit "known-unknowns" in
[`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md);
for the CUDA analysis area, start at
[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md).
diff --git a/docs/02-getting-started/01-install-and-first-run.md b/docs/02-getting-started/01-install-and-first-run.md
index c1a8f5d..9ae81c2 100644
--- a/docs/02-getting-started/01-install-and-first-run.md
+++ b/docs/02-getting-started/01-install-and-first-run.md
@@ -142,10 +142,11 @@ cd cuda-samples/Samples/1_Utilities/deviceQuery && make && ./deviceQuery
```
**Expectation today:** driver probe, `nvidia-smi`, device enumeration, and the
-first CUDA calls (`cuInit`, device queries) work. `cuCtxCreate` still fails
-with `999` because the emulated engine list is missing the async copy engine;
-full kernel execution is not reached yet. Before drawing conclusions from a
-CUDA failure, read the honest status in
+full CUDA driver-API smoke (`ALL OK` 64/64, copies matched) work.
+Engine execution is the remaining gap: CE-class copies are forwarded to the
+physical copy engine (default off), while CUDA kernel execution is blocked on
+GR engine-context validity. Before drawing conclusions from a CUDA failure,
+read the honest status in
[`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md)
and the debugger's guide [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md).
@@ -183,8 +184,8 @@ Developer debug knobs (module parameters) are documented in
| Module will not load | `dmesg`, `modinfo open-iov-vfio.ko`, stale module state → `cleanup-crash` |
| No mdev type for your GPU | [hardware matrix](../06-portability/hardware-software-matrix.md) |
| Guest sees wrong PCIe link speed/width | root-port attachment + config-space mirror (debugging doc) |
-| `nvidia.ko` probe fails in guest | guest dmesg + `lspci -vvv` + handoff "driver lifecycle" |
-| `nvidia-smi` works, CUDA does not | expected today (`cuCtxCreate` = 999, engine-list gap); see [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) and [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md) |
+| `nvidia.ko` probe fails in guest | guest dmesg + `lspci -vvv` + [driver lifecycle field guide](../03-architecture/07-driver-lifecycle-field-guide.md) |
+| `nvidia-smi` works, CUDA kernel execution does not | expected today: the driver-API smoke passes; CE copies are forwarded (default off) and GR execution is blocked on engine-context validity; see [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) and [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md) |
| VM boot fails or the guest hangs | `scripts/stack_manager.sh cleanup-crash`, then reboot |
## 10. Uninstall
diff --git a/docs/03-architecture/01-system-overview.md b/docs/03-architecture/01-system-overview.md
index 5c2ba6c..fa328ae 100644
--- a/docs/03-architecture/01-system-overview.md
+++ b/docs/03-architecture/01-system-overview.md
@@ -60,7 +60,7 @@ HOST MODULE open-iov-vfio.ko
├─ virtual device: config space, BAR0/BAR1/BAR3 windows, MSI-X
├─ emulation core: GSP boot + GSP-RM (RPC msgq, controls, objects, events)
├─ memory model: guest RAM access, VA↔PA maps, VRAM backing
- └─ work path: channel/push forwarding to the physical GPU [goal; software bridge today]
+ └─ work path: channel/push forwarding to the physical GPU [CE copies forwarded (default off); GR open]
│ module-owned device + real GSP session
HOST BACKEND the GPU's own hardware + GSP firmware, driven by this module
│ (the distribution nvidia.ko stays unmodified and unbound in
@@ -170,10 +170,12 @@ decompiled and mapped (see `wiki/GSP-570.144W-Firmware-Analysis.md` and
without server-side object management. Where the driver consumes a reply
value, that value is still being corrected (the reply-fidelity ladder).
5. `nvidia-smi -L` enumerates the emulated device and CUDA user-space starts
- (`cuInit` / `cuDeviceGetCount` / `cuDeviceGet` succeed). The client reaches
- `cuCtxCreate`, which currently ends in a forced `rc=999` on a completion
- wait; the copy-engine method streams execute in software as a bridge, and
- the execution backend is not built yet (see
+ (`cuInit` / `cuDeviceGetCount` / `cuDeviceGet` succeed); context creation
+ completes and the full CUDA driver-API smoke passes (`ALL OK` 64/64, copies
+ matched). The execution backend is live: CE-class copies are
+ forwarded to the physical copy engine (default off) with engine-authored
+ semaphore releases and a fault relay; kernel execution is blocked on GR
+ engine-context validity (see
[`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)).
The CUDA-side contract and the reply-fidelity rules are documented in
diff --git a/docs/03-architecture/02-guest-stack-and-cudasw.md b/docs/03-architecture/02-guest-stack-and-cudasw.md
index e841342..f9fe249 100644
--- a/docs/03-architecture/02-guest-stack-and-cudasw.md
+++ b/docs/03-architecture/02-guest-stack-and-cudasw.md
@@ -88,14 +88,15 @@ open /dev/nvidia* → cuInit → UVM registration (per process)
```
The initialization path is where compatibility is won or lost; the runtime
-path is where *execution* is won or lost. Open-IOV's current focus is
-pre-forwarding correctness: `cuInit` / `cuDeviceGet` succeed, and the client
-reaches context creation, which currently ends in a forced `rc=999` on a
-completion wait; the engine inventory is complete and the forwarding decision
-is made — a module-owned GPU running its real GSP firmware as the
-resource-manager server (see
+path is where *execution* is won or lost. Context creation and the full CUDA
+driver-API smoke now pass (`ALL OK`, copies matched), so the current focus is
+execution: the forwarding decision is made and the backend is live — a
+module-owned GPU running its real GSP firmware as the resource-manager server
+(see
[`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md)
-and [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)).
+and [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)) — with
+CE-class copies forwarded to the physical copy engine (default off) and GR
+kernel execution blocked on GR engine-context validity.
For the CUDA userspace side of this stack — its architecture, how it is built
and executes, and the exact kernel/GPU interface contract we must satisfy —
diff --git a/docs/03-architecture/03-gsp-and-gsp-rm.md b/docs/03-architecture/03-gsp-and-gsp-rm.md
index 7f206ba..10e6a73 100644
--- a/docs/03-architecture/03-gsp-and-gsp-rm.md
+++ b/docs/03-architecture/03-gsp-and-gsp-rm.md
@@ -59,8 +59,10 @@ firmware: FWSEC raised WPR2 with a zero FRTS error code, the SEC2 booter halted
with mailbox0 = 0, the RISC-V core reported active (`RISCV_STATUS=0x00000035`,
active bit 0) and `FALCON_OS` read 0. The server's first message was then a
status-queue event, function `0x1002` (`NV_VGPU_MSG_EVENT_GSP_RUN_CPU_SEQUENCER`,
-`rpc_result=0`), carrying a command buffer the host must execute;
-`GSP_INIT_DONE` (`0x1001`) follows only after that sequencer runs. The
+`rpc_result=0`), carrying a command buffer the host must execute. Once that
+sequencer runs, `GSP_INIT_DONE` (`0x1001`) is delivered with `rpc_result=0`,
+the static-configuration reply is served from the observed device tables, and
+the session remains live until the unload RPC releases it at teardown. The
host-side queue pair behind the exchange is one shared page table plus a 256 KB
command queue and a 256 KB status queue of 4 KB elements.
@@ -229,8 +231,10 @@ is not lost.
The engine-inventory row was the CUDA blocker at the time of writing:
`libcuda` filters the engine list it receives against the class list and
expects the async copy engine to be present, because the secondary (async)
-engine record is otherwise zeroed. That inventory is now complete; the
-context-creation wait is later, in the completion path. See
+engine record is otherwise zeroed. That inventory is now complete and context
+creation passes; the current frontier is engine execution, where CE-class
+copies are forwarded to the physical copy engine (default off) and GR kernel
+execution is blocked on GR engine-context validity. See
[`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md)
section 3.7 for the offsets and the consumed fields, and
[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../../userspace/cuda/docs/03-driver-and-gpu-interface.md)
diff --git a/docs/03-architecture/04-uvm.md b/docs/03-architecture/04-uvm.md
index 4c8a0c9..9396e6a 100644
--- a/docs/03-architecture/04-uvm.md
+++ b/docs/03-architecture/04-uvm.md
@@ -50,17 +50,19 @@ Each stage has a distinct failure signature; our debugging docs map them.
| Requirement | Detail | Our status |
|---|---|---|
| GPU identity | must match the uuid from RM | ✅ |
-| CE capabilities | per-engine support bits; UVM *requires* `supported && !grce && sysmem` for CPU_TO_GPU/GPU_TO_CPU channels | 🟡 the caps we answer are accepted and the inventory now includes the async copy engine; completion/semaphore plumbing is the current frontier |
+| CE capabilities | per-engine support bits; UVM *requires* `supported && !grce && sysmem` for CPU_TO_GPU/GPU_TO_CPU channels | 🟡 the caps we answer are accepted, the inventory now includes the async copy engine, and CE copies are forwarded to the physical copy engine (default off) with engine-authored releases |
| Channel creation | RM channel objects with GPFIFO/userd/instance parameters | ✅ objects created |
-| Pushes executed | UVM's first action on a new channel is an *init push* followed by a semaphore-tracked wait | 🟡 the host decodes method streams and executes memsets, copies and semaphore releases in software (a bridge); no kernel is forwarded to the physical GPU yet |
+| Pushes executed | UVM's first action on a new channel is an *init push* followed by a semaphore-tracked wait | 🟡 the host decodes method streams and executes memsets, copies and semaphore releases in software (a bridge); CE-class copies and releases are additionally forwarded to the physical copy engine (default off); no GR kernel is forwarded yet (blocked on GR engine-context validity) |
| Page trees | UVM initializes page trees with its own pushes; the *server-owned* ranges must be populated by us | 🟡 |
-| Fault delivery | fault buffer + notification; replay/cancel semantics | 🔴 |
+| Fault delivery | fault buffer + notification; replay/cancel semantics | 🟡 backend-side fault observation/relay exists (default off: notifier write + event post); the guest replay/cancel contract is open |
| Semantics of completions | tracker values must advance exactly | 🟡 |
-**Current frontier (2026-09-29):** the client reaches `cuCtxCreate` and
-currently ends in a forced `rc=999` on a completion wait; the engine inventory
-is complete, and the remaining work is completion translation plus the chosen
-execution backend
+**Current frontier (2026-10-02):** context creation and the full CUDA
+driver-API smoke pass (`ALL OK`, copies matched); the execution backend is
+live and CE-class copies are forwarded to the physical copy engine (default
+off) with engine-authored releases and a fault relay. GR kernel execution is
+blocked on GR engine-context validity; the remaining UVM work is completion
+translation and the guest-side fault replay/cancel contract
([`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)).
Failed-context teardown has produced guest kernel oopses in
`uvm_pmm_devmem_page_to_va_space` in earlier runs. The CUDA-side
diff --git a/docs/03-architecture/05-gmmu-amap-hal-gr.md b/docs/03-architecture/05-gmmu-amap-hal-gr.md
index 76d4200..67ceeca 100644
--- a/docs/03-architecture/05-gmmu-amap-hal-gr.md
+++ b/docs/03-architecture/05-gmmu-amap-hal-gr.md
@@ -206,5 +206,4 @@ section 3.7 for the record layout the driver's engine walker consumes.
| Firmware structure and dispatchers | `wiki/GSP-570.144W-Firmware-Analysis.md` |
**Rule:** tracked documentation describes components and behaviour, with our
-own observations, IDs and layouts. Exact analysis sources live in the
-maintainers' study notes.
+own observations, IDs and layouts.
diff --git a/docs/03-architecture/06-host-module-and-hybrid-model.md b/docs/03-architecture/06-host-module-and-hybrid-model.md
index d6b96e7..0dbc89a 100644
--- a/docs/03-architecture/06-host-module-and-hybrid-model.md
+++ b/docs/03-architecture/06-host-module-and-hybrid-model.md
@@ -140,7 +140,10 @@ section 3.7 for the analysis.
method streams and executes the work it understands in software - memsets,
copies and semaphore releases - and completes the corresponding semaphores.
That is a correctness bridge for memory operations, not the data plane: no
-CUDA kernel is forwarded to the physical GPU yet; the chosen execution backend
+CUDA kernel is forwarded to the physical GPU yet (default-off CE copy
+forwarding, engine-authored release relay and fault relay are implemented; see
+[`11-physical-backend-guest-wiring.md`](11-physical-backend-guest-wiring.md));
+the chosen execution backend
(a module-owned GPU running its real GSP firmware as the resource-manager
server) is in
[`08-physical-gpu-integration.md`](08-physical-gpu-integration.md), with the
@@ -149,10 +152,11 @@ trade-offs in
section 3.
**Subsystem status note.** Some components are compiled but not currently
-wired into the live path: the physical-GSP load/boot path is skipped in favour
-of emulation, and the runtime firmware analyzer, version tables, hardware
-abstraction tables and dynamic RPC dispatcher are not yet called by the live
-flow. Treat their descriptions here as intended design, not active code.
+wired into the live path: the physical-GSP load/boot path is default-off and
+runs only behind the supervised claim gates, and the runtime firmware
+analyzer, version tables, hardware abstraction tables and dynamic RPC
+dispatcher are not yet called by the live flow. Treat their descriptions here
+as intended design, not active code.
### 3.3 Event channels and interrupts
@@ -263,6 +267,22 @@ emulated device inspectable without rebuilding:
| `sema_map_add=<va>,<pa>` | teach the service one verified `{GPU VA -> guest PA}` pair |
| `dump_guest=<gpa>,<len>` | dump guest physical memory (length capped) |
+The physical-backend path adds a second, default-off parameter set:
+`gsp_phys_gpu`, `gsp_phys_hazard_ack`, `gsp_phys_fw`, `gsp_phys_mdev_owned`,
+`gsp_phys_static_live`, `gsp_phys_ctrl_forward`, `gsp_phys_work_forward`,
+`gsp_phys_guest_mem`, `gsp_phys_work_defer_release`,
+`gsp_phys_work_hw_primary`, `gsp_phys_relay`, `gsp_phys_fault_relay`,
+`gsp_phys_fault_inject`, `gsp_phys_rmcall_engine`,
+`gsp_phys_rmcall_gr_promote`, `gsp_phys_repower`, `gsp_phys_layout`,
+`gsp_phys_boot`, `gsp_phys_boot_dryrun`, `gsp_phys_nocat` and the
+`gsp_phys_fwsec*` stage controls. Their semantics, the physical-backend
+debugfs inventory (`gsp-phys-state`, `gsp-phys-rm-probe`, `gsp-phys-rm-call`,
+`gsp-phys-unload-rpc`, `gsp-phys-guest-ram`/`gsp-phys-guest-gpa`,
+`gsp-phys-log-dump`, `gsp-phys-status-drain`) and the supervised runbooks are
+maintained in the engineering-operations document
+([`../eng-ops.md`](../eng-ops.md), Part II); this document does not duplicate
+those tables.
+
These are development-grade but production-useful: they are how the memory
model was reverse-engineered, and they remain the fastest way to diagnose a
mismatch. Their workflows are documented in
@@ -312,8 +332,9 @@ in `kernel-open` first).
| Message queues + RPC + all init controls observed from stock 570.144 | implemented |
| Guest sysmem access + VA↔PA discovery | implemented (debug-grade, to be productized) |
| Push decoding + semaphore completion | demonstrated live; being productized |
-| Page-table population for server-owned ranges | designed, not implemented |
-| GPU work forwarding (GR/CE execution) | designed, not implemented |
-| Execution backend (module-owned GPU + direct real-GSP session) | mechanism chosen; ownership + firmware staging verified live; power-domain lock implemented, pending supervised verification; real GSP boot not started ([`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)) |
+| Guest-RAM physical path | implemented default-off: guest RAM is file-backed shared memory and GPA→HPA goes through its page cache, verified live on the CE |
+| Page-table population for server-owned ranges | implemented default-off: forwarded CE copies map resolved host pages through module-built GMMU scratch page tables; real guest page tables remain |
+| GPU work forwarding (GR/CE execution) | CE forwarding, engine-authored release relay and fault relay implemented default-off and verified live; GR execution open |
+| Execution backend (module-owned GPU + direct real-GSP session) | live: boot, init-done, full static-config reply and clean releases proven; RM client allocates objects/channels and has executed real CE copies; currently in the designed locked fail-safe after a GR context-promote wedge ([`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)) |
| Real VRAM backing | designed, not implemented |
| Windows guest, Xen transport, aarch64 hosts | designed, not implemented |
diff --git a/docs/03-architecture/07-driver-lifecycle-field-guide.md b/docs/03-architecture/07-driver-lifecycle-field-guide.md
index 1bfe912..73b007e 100644
--- a/docs/03-architecture/07-driver-lifecycle-field-guide.md
+++ b/docs/03-architecture/07-driver-lifecycle-field-guide.md
@@ -107,7 +107,7 @@ fallback). Representative controls:
| `0x90f10106` | vaspace reserved-PDE contract | **carries page-table physical addresses** (see GSP-RM doc) |
| `0x20800123` / `0x20800170` | `GPU_GET_ENGINES` / `GET_ENGINES_V2` | engine inventory (filtered by the class list / classDB); values are consumed — the guest must see the full engine list, including the async copy engine (`COPY14`); that inventory is now complete, and the context-creation wait lives later in the completion path |
| `0x800201` | `GPU_GET_CLASSLIST` | class list (classDB); must agree with the engine list and the CE caps controls |
-| `0x2080012b` | GPU_PROMOTE_CTX (560-byte params) | issued immediately before context creation; fallback generic `SUCCESS`, consumed fields not yet coherent |
+| `0x2080012b` | GPU_PROMOTE_CTX (560-byte params) | historically issued immediately before context creation and answered by the generic fallback. Context creation now returns; in the host GR ladder this promote is the load-bearing GR context-validity operation, and a wrong shape is a known server wedge (no retry; reboot policy - [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md) §7) |
| `0x20803039` | NVLink masks/VBIOS info | fallback: generic `SUCCESS` with zeroed params |
| `0x20800a70` | BUS flush with sysmembar | fallback: generic `SUCCESS` with zeroed params; UVM issues these frequently |
@@ -177,6 +177,14 @@ start at
the execution backend is in
[`08-physical-gpu-integration.md`](08-physical-gpu-integration.md).
+**Update (2026-10-02):** context creation now returns (the driver-API smoke
+completes end to end) and the frontier is execution. The host GR ladder
+allocates a GR0 channel and a compute-class object, but its first semaphore
+push faults with an uncommitted GR context (`CTXNOTVALID`), and the
+`GPU_PROMOTE_CTX` attempt that followed wedged GSP-RM; the device then sits in
+the designed locked fail-safe (see
+[`08-physical-gpu-integration.md`](08-physical-gpu-integration.md) §7/§8).
+
**Log signatures (guest side):**
```
nvidia-uvm: ... Tracker entry for value N (sema VA 0x...) channel ID ... CE ...
diff --git a/docs/03-architecture/08-physical-gpu-integration.md b/docs/03-architecture/08-physical-gpu-integration.md
index 2ea44a1..532cd77 100644
--- a/docs/03-architecture/08-physical-gpu-integration.md
+++ b/docs/03-architecture/08-physical-gpu-integration.md
@@ -1,10 +1,12 @@
# Physical GPU Integration: How Work Reaches Real Hardware
-> **Status: mechanism chosen; ownership + firmware staging implemented;
-> power-domain lock verified; first real GSP boot and first live GSP-RM
-> exchange complete (2026-10-01); host-side initialization follow-up and a
-> clean release are in progress.** The control plane works today: the guest's
-> driver initializes and the CUDA driver-API baseline passes end to end
+> **Status: backend live; init-done, static configuration and clean releases
+> proven; CE work forwarding with an engine-authored release and fault relay
+> implemented default-off; GR execution is the open wall. The device is
+> currently in the designed locked fail-safe awaiting a supervised reboot
+> after a GR context-promote attempt wedged GSP-RM.** The control plane works
+> today: the guest's driver initializes and the CUDA driver-API baseline
+> passes end to end
> (context creation, allocation, copies, 64/64 payload round-trip), and the
> guest's decoded work is executed in software as a development bridge. This
> document records the interfaces that actually exist, the mechanism options,
@@ -145,8 +147,10 @@ re-bind. While the module owns the device it verifies that the GPU still
answers config space before proceeding and aborts on any power-loss signature
(the companion controller reporting a lost root hub, or a failed config read).
The distribution driver is then rebound with the lock still held, its probe is
-allowed to complete, and only then is runtime power management restored to
-automatic. Any power-loss signature aborts the sequence and forbids a retry in
+allowed to complete, and the D0/D3cold locks stay latched by default
+(`gsp_phys_repower=0`); restoring runtime power management is an explicit
+opt-in through the power-restore path, taken only when the operator chooses
+it. Any power-loss signature aborts the sequence and forbids a retry in
that boot: an out-of-band platform power cut bypasses the lock and would
produce the identical signature, and retrying would be a second dice roll on a
host that has already lost the board.
@@ -210,19 +214,22 @@ bridge of the GPU's domain), built from the live PCI topology
method-data block is sent invalid/zero: the boot path deliberately makes no
ACPI/AML calls and every consumer tolerates the no-data encoding. The server
waits for the host to execute the sequencer command buffer carried in that
-event before it raises `GSP_INIT_DONE` (0x1001). **A plain release is not yet
-possible:** the Booter Unload gate requires all four secure scratch14 handoff
-bits (24–27); a
-host-driven boot sets only bit 24, and bit 27 is set by the server's shutdown
-path (host RPC `UNLOADING_GUEST_DRIVER`; implemented in the module as the
-rung-4b command-queue TX + shutdown path and pending its first live run),
-while the
-host-side seed/unlock alternative is fuse-locked on this part. Until that RPC
-has run in a live session, a green session is left claimed with the harness
-`--keep` flag and returning the GPU to the distribution driver takes a power
-cycle. The supervised harness modes are `--boot` (the only write mode: full
-boot lifecycle), `--boot-dryrun` (zero-write staging rehearsal) and `--keep`
-(leave a verified claimed session running); `--fwsec` is retired.
+event before it raises `GSP_INIT_DONE` (0x1001). **Clean release is
+proven:** since 2026-10-01 the unload RPC path delivers a clean release with
+no reboot on every cycle — the
+Booter Unload gate requires all four secure scratch14 handoff bits (24–27); a
+host-driven boot sets bit 24, and the server's shutdown path (host RPC
+`UNLOADING_GUEST_DRIVER`, rung 4b, §8) sets the rest, while the host-side
+seed/unlock alternative is fuse-locked on this part. The exception is a wedged
+server: if the server never sets its bit, teardown cannot clear the protected
+region and the fail-safe refuses the hand-back (no rebind, D0 locks held, a
+reboot with approval required; §7). The supervised harness modes are `--boot`
+(the only write mode: full boot lifecycle), `--boot-dryrun` (zero-write
+staging rehearsal) and `--keep` (leave a verified claimed session running);
+`--fwsec` is retired. The full parameter, debugfs and runbook inventory for
+this path is maintained in the engineering-operations document
+([`../eng-ops.md`](../eng-ops.md), Part II); this document does not duplicate
+those tables.
### 4.4 Host-side resource-manager client
@@ -277,12 +284,12 @@ per-context scheduling. Tenant policy and fairness are a later layer on top.
| Driver-API baseline (init, devices, context, allocation, copies, free, destroy) | done — re-verified 64/64 payload round-trip on the current build |
| Software-executed bridge for decoded memory/state operations | in progress (memsets, copies, releases) |
| Device ownership + firmware staging | verified — claim/release and staged-image parse both verified live; the claim path is hazard-gated (§4.2) |
-| Power-domain lock across the claim window | verified — 3/3 supervised claim/release cycles (2026-10-01; §4.2, §8) |
-| Real GSP boot by the module | first live boot 2026-10-01 — FWSEC WPR2 up, Booter Load `M0=0`, RISC-V active; clean release pending the server shutdown RPC |
-| Host-side resource-manager client for real objects/channels | transport alive 2026-10-01 — first real GSP-RM event (0x1002) received; init-done (0x1001) awaits host sequencer execution |
-| Work translation and real submission | not started |
-| Completion relay through real hardware | not started |
-| CUDA kernel execution (`vectorAdd`) | not started |
+| Power-domain lock across the claim window | verified — 3/3 supervised claim/release cycles (2026-10-01; §4.2, §8); latched by default since the G0 bundle (`gsp_phys_repower=0`) |
+| Real GSP boot by the module | done — full boot through init-done (`0x1001`, `rpc_result=0`) and a clean release with no reboot on every supervised cycle since 2026-10-01 |
+| Host-side resource-manager client for real objects/channels | done — full static-config reply served; real memory, vaspace, channel-group and channel objects allocated; real CE copies executed on the physical engine |
+| Work translation and real submission | CE forwarding and engine-authored release relay implemented default-off and verified live; GR execution open |
+| Completion relay through real hardware | implemented default-off — engine-authored release verified live and forwarded-engine fault observation/relay behind `gsp_phys_fault_relay`, fail-closed to software on any miss; one relay data mismatch under triage |
+| CUDA kernel execution (`vectorAdd`) | blocked on GR context validity/execution — the host GR ladder's first push faulted `CTXNOTVALID` and the promote attempt wedged GSP-RM |
## 7. Open questions
@@ -297,6 +304,12 @@ per-context scheduling. Tenant policy and fairness are a later layer on top.
controller action that does not go through the root port's D3 path — would
produce the identical power-loss signature; the claim must abort on that
signature and must not be retried in the same boot.
+* **Wedged server / protected-region fail-safe:** if a live session wedges the
+ server so its shutdown path never completes, WPR2 cannot be cleared (the
+ host-side unlock is fuse-locked on this part). The release then refuses to
+ rebind and keeps the D0 locks; never retry in that boot, and return to the
+ distribution driver only through a reboot governed by the project's
+ reboot-approval policy (a written reason and post-reboot plan first).
* **Supervised operation of the lock:** the lock is implemented and verified
across three supervised claim/release cycles (2026-10-01). Every claim
remains supervised: explicit human go-ahead, other host work warned, a
@@ -338,7 +351,8 @@ per-context scheduling. Tenant policy and fairness are a later layer on top.
the server's shutdown path (host RPC `UNLOADING_GUEST_DRIVER`), and
host-side seed/unlock is fuse-locked on this part. The live session is
therefore left claimed (`--keep`) and a fresh cycle needs a power cycle.
- See §4.3.
+ See §4.3. *(Historical: superseded by the clean-release result in the
+ unload-RPC update below.)*
* **Update (2026-10-01) — rung 4b implemented (build-only):** the module now
produces real command-queue records to GSP-RM and attempts a clean shutdown
on release. The producer writes the 48-byte transport element header plus
@@ -355,8 +369,10 @@ per-context scheduling. Tenant policy and fairness are a later layer on top.
behaviour is unchanged. A write-only debugfs trigger
(`open-iov-vfio/gsp-phys-unload-rpc`, write "1") lets a live claimed
session test the shutdown without touching WPR2. All of it is inert unless
- the full boot stage ran; the code has not yet run live, and this bullet
- must be updated with the live outcome.
+ the full boot stage ran. **Live-verified (2026-10-01):** the unload RPC
+ shuts the server down (GSP-RM processor suspended in ~220 ms, handoff bits
+ complete), so WPR2 goes down and the distribution driver is rebound with no
+ reboot on every clean cycle.
* **Update (2026-10-01) — rung 4c slice 1 implemented (build-only):** for the
next cycle a host resource-manager static-info probe is available: the
write-only debugfs file `open-iov-vfio/gsp-phys-rm-probe` (write "1", only
@@ -365,8 +381,20 @@ per-context scheduling. Tenant policy and fairness are a later layer on top.
handles, split-VA flag, BAR1/2 PDE bases, framebuffer geometry, engine
caps), falling back once to `SET_GUEST_SYSTEM_INFO` plus one retry; the
command-queue producer now splits records across up to 16 queue elements
- and the status-queue walk coalesces multi-element records. Not yet run
- live; the module was built only.
+ and the status-queue walk coalesces multi-element records. **Live-verified:**
+ init-done with the full 1656-byte static-config reply was observed on the
+ physical server, and the probe decodes its static configuration (internal
+ handles, BAR PDE bases and engine caps); the multi-element producer and
+ status-walk coalescing also run live.
+* **Update (2026-10-02) — GR execution is the wall; device in locked
+ fail-safe:** the resource-manager client ladder reaches the compute engine
+ (GR0 channel plus a compute-class object), but the first GR semaphore push
+ faulted with an uncommitted context (`CTXNOTVALID`), and the follow-up
+ `GPU_PROMOTE_CTX` attempt wedged GSP-RM (internal assert `0x52a8b04`, no
+ reply). Teardown could not clear WPR2, so the fail-safe held the device
+ (module unloaded, GPU unbound, protected region up, D0 locks held). Recovery
+ is a supervised reboot under the reboot-approval policy; the promote
+ bisection resumes after that.
## 9. Interaction sequence (end state)
@@ -398,6 +426,7 @@ system map is in [`09-full-system-picture.md`](09-full-system-picture.md).
## See also
- [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) - the host module, memory model, build contracts
+- [`../eng-ops.md`](../eng-ops.md) - engineering operations; Part II carries the physical-backend parameter, debugfs and runbook inventory
- [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) - known gaps and plans
- [`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) - forwarding trade-offs
- [`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md) - the CUDA-side view
diff --git a/docs/03-architecture/09-full-system-picture.md b/docs/03-architecture/09-full-system-picture.md
index e03a68e..3c15830 100644
--- a/docs/03-architecture/09-full-system-picture.md
+++ b/docs/03-architecture/09-full-system-picture.md
@@ -80,7 +80,7 @@ flowchart TB
class VDEV,EMU done
class MEMM,WORK wip
- class BE plan
+ class BE done
class IOMMU,VFIO done
class GSP,ENG,GMMU,VRAM,DB hw
class ND,USR,GKO,APP vendor
@@ -125,8 +125,9 @@ stateDiagram-v2
Emulated today: the module drives the
stage transitions and answers status
reads from the BAR0 shadow. The real
- backend boots the physical GSP in
- parallel [planned].
+ backend boots the physical GSP as its
+ own live session (init-done, static
+ configuration and clean releases proven).
end note
```
@@ -262,14 +263,14 @@ flowchart LR
F --> G["Write releases and completions<br/>into guest memory"]
G --> H["Guest trackers advance"]
- D -.->|"planned"| I["Forward to a real engine"]
+ D -.->|"wip/default-off"| I["Forward to a real engine"]
classDef done fill:#d5f5d5,stroke:#1e7a1e
classDef wip fill:#fff3cd,stroke:#9a7b1a
classDef plan fill:#f8d7da,stroke:#9a1a1a
class A,B,D done
class E,F,G,H wip
- class I plan
+ class I wip
```
This is a correctness bridge: it advances synchronization and memory
@@ -329,22 +330,30 @@ flowchart TB
R4 --> R5["Real submissions scheduled by GSP"]
R5 --> R6["Completion and faults relayed to guest"]
end
- G -.->|"translation [planned]"| R4
+ G -.->|"translation [wip/default-off]"| R4
classDef done fill:#d5f5d5,stroke:#1e7a1e
+ classDef wip fill:#fff3cd,stroke:#9a7b1a
classDef plan fill:#f8d7da,stroke:#9a1a1a
class A,B,C,D,E done
class F,G wip
- class R1,R2,R3,R4,R5,R6 plan
+ class R1,R2,R3 done
+ class R4,R5,R6 wip
```
+Backend boot, the real RM client, channel allocation and real CE copies are
+live; guest-forwarding and relay are default-off; GR execution and real guest
+page tables are the open `[planned]` parts (see
+[`11-physical-backend-guest-wiring.md`](11-physical-backend-guest-wiring.md)).
+
## 5. Scheduling and multi-tenancy
There is no hardware partitioning (no SR-IOV) on this GPU generation, so
tenancy is a **logical VF model**: each VM holds one mdev instance, and the
module gives each instance its own GSP-RM session, channels, page tables and
scheduling policy. “Per-VF GSP emulation” therefore means per-mdev emulation
-context plus a per-mdev real GSP client in the backend.
+context plus a per-mdev real GSP client in the backend. Today the backend is
+a single per-GPU session (live); per-mdev clients are planned.
```mermaid
flowchart TB
@@ -358,7 +367,7 @@ flowchart TB
Q2["Emulation context 2"]
Q3["Emulation context N"]
POL["Fairness policy [planned]<br/>time slices, quotas, priority, burst"]
- C1["Real GSP client 1 [planned]"]
+ C1["Real GSP client 1<br/>single per-GPU session [live]"]
C2["Real GSP client 2 [planned]"]
C3["Real GSP client N [planned]"]
end
@@ -381,7 +390,8 @@ flowchart TB
classDef wip fill:#fff3cd,stroke:#9a7b1a
classDef plan fill:#f8d7da,stroke:#9a1a1a
class Q1,Q2,Q3 done
- class C1,C2,C3,POL,SCHED plan
+ class C1 done
+ class C2,C3,POL,SCHED plan
```
**How multi-tenant performance works (and is intended to):**
@@ -636,10 +646,10 @@ against a stock guest driver with evidence in the repository.
| Guest memory access + VA-to-PA discovery | partial | yes | protocol-derived discovery, no external pins |
| Push decode + semaphore completion | partial | yes | complete GR/compute encodings |
| Software execution bridge (memsets, copies, releases) | partial | yes | remains bridge only |
-| Page-table population for server-owned ranges | no | no | design + validator, then implement |
-| Execution backend (own GPU + real GSP) | no | no | device ownership, real GSP bring-up, RM client |
-| Real work forwarding (GR and CE) | no | no | after backend bring-up |
-| Fault semantics (buffers, replay, cancel) | no | no | after forwarding |
+| Page-table population for server-owned ranges | partial (module-built scratch page tables, default off) | yes (default off) | real guest page tables |
+| Execution backend (own GPU + real GSP) | yes (single per-GPU session) | yes | per-mdev clients |
+| Real work forwarding (GR and CE) | partial (CE default-off) | yes (default off) | GR context validity/execution |
+| Fault semantics (buffers, replay, cancel) | partial (forwarded-engine observation/relay default-off) | yes (default off) | GR fault semantics and triage |
| Multi-tenant fairness policy | no | no | after multi-tenant forwarding |
| Version matrix (2+ driver families) | partial | partial | per-version profiles + A/B |
| Windows guest | no | no | after Linux execution path |
diff --git a/docs/03-architecture/10-interaction-and-scheduling.md b/docs/03-architecture/10-interaction-and-scheduling.md
index 3608dc5..ffc02d7 100644
--- a/docs/03-architecture/10-interaction-and-scheduling.md
+++ b/docs/03-architecture/10-interaction-and-scheduling.md
@@ -22,12 +22,12 @@ to, and where it stands.
| Guest doorbell signal (boot, mailbox, channel) | write into BAR0 shadow, observed by the polling service | steering only | [done] |
| Guest RPC request | element placed in a guest-sysmem message queue, read by the module over VFIO DMA | 10s of us per round trip | [done] |
| RPC reply + interrupt | module writes reply into the queue, signals eventfd, QEMU raises irqfd into KVM, MSI-X lands in the guest | us | [done] |
-| Push-buffer work | guest appends GPFIFO entries; the per-channel pump reads, decodes and applies them | per entry, us to low ms | [wip] |
-| Semaphore release to guest memory | module writes the payload word into the resolved guest page | us | [done] |
+| Push-buffer work | guest appends GPFIFO entries; the per-channel pump reads, decodes and applies them | per entry, us to low ms | [wip] (CE copies forward default-off; GR execution open) |
+| Semaphore release to guest memory | module writes the payload word into the resolved guest page; a forwarded CE copy can have the engine author it | us | [done] (relay default off) |
| VA-to-PA translation discovery | in-module structure walk, content scan, rescue, pins | us to seconds (scan bounded) | [wip] |
| Translation internalization from protocol | derive mappings from RM/UVM objects as they are created | us | [planned] |
-| Real channel submission | translated push stream on a real channel, real doorbell | us + hardware | [planned] |
-| Physical GPU interrupt | real MSI-X into the module's IRQ handler, relayed to guest trackers | us | [planned] |
+| Real channel submission | translated push stream on a real channel, real doorbell; the ladder's real submission is live | us + hardware | [wip] (guest forwarding default-off) |
+| Physical GPU interrupt | real MSI-X into the module's IRQ handler, relayed to guest trackers; forwarded-engine fault observation/relay exists | us | [wip] (default off) |
| mdev create / destroy | sysfs plus VFIO device registration | ms | [done] |
| Guest RAM mapping | VFIO / IOMMUFD maps guest pages for DMA | ms once | [done] |
| Teardown | ordered stop: VM, device close, mdev remove, module unload | s | [done] |
@@ -129,7 +129,7 @@ stateDiagram-v2
note right of Applying
State and notification operations are
applied from the emulation. Real engine
- work is forwarded [planned].
+ work is forwarded [wip/default-off].
end note
```
@@ -198,7 +198,7 @@ Rules the design enforces:
| Push entry decode | us per entry | thousands of entries per context |
| Rescue of an unreadable entry | bounded ms | only on translation misses |
| MSI-X to guest | us | completion latency floor |
-| Real submission + completion relay | hardware + us | the forwarded-work path [planned] |
+| Real submission + completion relay | hardware + us | the forwarded-work path [wip/default-off] |
The rule of thumb the architecture is built around: **state operations are
cheaper in the module than they would be on hardware; real work belongs on
@@ -250,7 +250,7 @@ flowchart TB
F --> G["KVM injects MSI-X vector"]
G --> H["Guest ISR runs"]
end
- subgraph PHY["Module and physical GPU [planned]"]
+ subgraph PHY["Module and physical GPU [wip/default-off]"]
I["Real channel doorbell"] --> J["GPU executes"]
J --> K["Physical MSI-X"]
K --> L["Module IRQ handler"]
@@ -260,9 +260,9 @@ flowchart TB
M -.-> E
classDef done fill:#d5f5d5,stroke:#1e7a1e
- classDef plan fill:#f8d7da,stroke:#9a1a1a
+ classDef wip fill:#fff3cd,stroke:#9a7b1a
class A,B,C,D,E,F,G,H done
- class I,J,K,L,M plan
+ class I,J,K,L,M wip
```
## 9. Lifecycle and teardown ordering
@@ -301,8 +301,8 @@ sequenceDiagram
participant GD as Guest driver
participant EMU as GSP-RM emulation
participant VT as Version profile
- participant BE as Execution backend [planned]
- participant FW as Real GSP firmware [planned]
+ participant BE as Execution backend (single live session)
+ participant FW as Real GSP firmware (live session)
GD->>EMU: version and capability queries
EMU->>VT: select profile for this driver family
@@ -327,7 +327,7 @@ sequenceDiagram
| Pin tables | authoritative injected translations | debug tooling; later protocol discovery | lookup fast path | [wip] |
| Pending release queue | releases awaiting a translation | decoder | service drain | [done] |
| Completion counters | per-channel progress published to guest pages | service | guest trackers | [done] |
-| Real channels and page tables | forwarding state | backend | real GPU | [planned] |
+| Real channels and page tables | forwarding state (real channel live; guest page tables open) | backend | real GPU | [wip] |
## See also
diff --git a/docs/03-architecture/11-physical-backend-guest-wiring.md b/docs/03-architecture/11-physical-backend-guest-wiring.md
index cbb4b33..2229365 100644
--- a/docs/03-architecture/11-physical-backend-guest-wiring.md
+++ b/docs/03-architecture/11-physical-backend-guest-wiring.md
@@ -1,8 +1,11 @@
# Physical Backend Guest Wiring: Driving the Guest-Facing Emulation from the Live GSP-RM Session
-> **Status: P0-P5(w1) implemented; P6a guest-page reachability and P6b waves
+> **Status: P0-P5(w1) implemented; P6a guest-page reachability (trigger
+> residual fixed) and P6b waves
> 1/1.5/2 (guest-copy forwarding, deferred release, hardware-primary copies)
-> proven live (2026-10-02).** The backend API skeleton
+> proven live (2026-10-02); P6c C0/C1 (engine-authored guest CE semaphore
+> relay) and C3 (forwarded-engine fault observation/relay) implemented
+> default-off; H1 host GR ladder open.** The backend API skeleton
> (P0), the mdev-owned lifecycle (P1, fail-open; the async FWSEC boot on that
> path is an open defect off the critical path), the static-config identity
> overlay (P2, verified live in an A/B run), the read-only control
@@ -16,7 +19,14 @@
> releases exactly once) and the wave-2 hardware-primary copies (default off,
> verified live: copy_skipped=119, queued req=100 ok=100 verify_ok=100
> verify_fail=0 fallback=0 timeouts=0, smoke `matched=64/64` `ALL OK`, with
-> the software fallback for unqueued/failed copies) are in the tree.
+> the software fallback for unqueued/failed copies) are in the tree. The P6c
+> relay is live-verified default-off — the physical engine authored the
+> guest CE copy's semaphore release (`relays=64`), failed/refused copies fall
+> back to the software apply exactly once, and one relayed-copy data mismatch
+> is under triage (mismatch logging plus software fallback in charge); the C3
+> forwarded-engine fault relay is live green, default off. The P6a trigger
+> residual is fixed: triggers 6/7 re-translate at trigger time and use fresh
+> scratch lines.
> The physical backend itself is live: the module-owned
> GSP-RM completes initialization on the real TU104, serves the full
> static-config reply, and shuts down gracefully. The host resource-manager
@@ -24,7 +34,12 @@
> a committed page directory, a channel group and a real copy-engine channel,
> and has executed the **first real work on the physical engine** — a 4 KiB
> copy in virtual-address mode with the semaphore released and destination
-> equal to source (cycle 25). What this document specifies is the remaining
+> equal to source (cycle 25). The H1 host GR ladder then allocated a GR0
+> channel with a compute-class object, but its first semaphore push faulted
+> with an uncommitted context (`CTXNOTVALID`), and the follow-up
+> `GPU_PROMOTE_CTX` attempt wedged GSP-RM; the GR execution wall is open and
+> the device sits in the designed locked fail-safe. What this document
+> specifies is the remaining
> bridge: wiring that live session into the guest-facing mdev/VFIO path so that
> guest submissions execute on the physical GPU. It extends
> [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md), which
@@ -102,6 +117,15 @@
(`:5068`). Stage 4 now runs end to end: the channel allocates, the page
directory is committed, and a 4 KiB virtual-address-mode copy executes on
the physical CE with the semaphore observed (`REAL COPY OK`, cycle 25).
+ `gsp_phys_rmcall_engine=1` selects a GR0 ladder variant (stage 3 builds the
+ channel and a `TURING_COMPUTE_A` compute object with `engineType=1`, stage 4
+ pushes a GR semaphore release). Its first live push faulted with an
+ uncommitted context (`CTXNOTVALID`). `gsp_phys_rmcall_gr_promote=1..3`
+ selects opt-in `GPU_PROMOTE_CTX` shapes to commit that context; the first
+ attempt wedged GSP-RM (internal assert, no reply) and teardown could not
+ clear WPR2, so the fail-safe held the device. Debugfs trigger 9 re-runs the
+ GR semaphore push on an already-built GR ladder without touching WPR2, for
+ use once a promote shape is viable.
* Release: `gsp_phys_gpu_release()` (`gsp_phys_gpu.c:876`) sends
`UNLOADING_GUEST_DRIVER` once (`:909`), quiesces (`:910`), tears down WPR2
(`:911`), frees staging (`:914`), and only rebinds the distribution driver
@@ -114,10 +138,12 @@ The two planes now meet at the lifecycle seam (P0/P1, mdev-owned
claim/release), the static-config identity overlay (P2), the read-only control
whitelist (P3) and, for eligible CE copies, the P6b scratch-mapped work path
(waves 1/1.5/2: guest operands, deferred exactly-once release,
-hardware-primary copies); objects, FB geometry, engine caps, real guest
-page tables and completion/fault relay are still emulated. The remaining
-bridge must be added at the reply point `gsp_rm_send()`, the inbound dispatch
-point `gsp_rm_handle_element()`, and the work-plane seams (see §2).
+hardware-primary copies) plus the P6c relay (C0/C1 engine-authored guest
+release, C3 forwarded-engine fault observation/relay), all default off;
+objects, FB geometry, engine caps and real guest page tables are still
+emulated. The remaining bridge must be added at the reply point
+`gsp_rm_send()`, the inbound dispatch point `gsp_rm_handle_element()`, and the
+work-plane seams (see §2).
---
@@ -141,7 +167,9 @@ point `gsp_rm_handle_element()`, and the work-plane seams (see §2).
BACKEND_STOPPING stop emulation forwarding; UNLOADING_GUEST_DRIVER;
| quiesce; WPR2 teardown; staging free
v
- BACKEND_OFF power unlock; distribution driver rebound and settled
+ BACKEND_OFF power unlock only when gsp_phys_repower=1 (default
+ 0 latches the D0/D3cold locks); distribution driver
+ rebound and settled
```
`BACKEND_READY` is the single gate for every live answer. In every other
@@ -198,8 +226,9 @@ generation counter` so a release cannot deadlock against a forward in flight.
parent/class teardown.
* **Server wedge (the 0x40 GSP-owned incident):** if `rm_unload` times out,
the existing release still quiesces, runs the FWSEC teardown and refuses to
- rebind with WPR2 up (`gsp_phys_gpu.c:939`). A reboot then needs explicit
- human approval plus a handoff update.
+ rebind with WPR2 up (`gsp_phys_gpu.c:939`). A reboot then follows the
+ project reboot-approval policy (a written reason and post-reboot plan before
+ approval).
* **Static-config served before READY:** bounded wait (2 s, implemented) then
pure emulation fallback, with a stat counter; never block the emulation
thread longer than the guest's own RPC poll can tolerate.
@@ -495,8 +524,10 @@ go-ahead, recovery plan, pstore, never unattended
514 real CE copies (194 requests coalesced/dropped while one was in flight,
0 refusals, 0 timeouts). The P6b waves 1/1.5/2 (guest-copy forwarding,
deferred exactly-once release, hardware-primary copies with a software
- fallback) have since landed (P6 status below); the real guest-page-table
- path (P6c) remains.
+ fallback) have since landed (P6 status below), and P5 wave 2 (hardware-
+ primary release relay) is subsumed by the P6b wave-2 hardware-primary mode
+ and the P6c engine-authored relay; the real guest-page-table path and GR
+ execution remain.
* **Scope:** the debugfs ladder's real-submission half is **done** (cycle 25:
channel allocates, page directory commits, `REAL COPY OK` on the physical
CE in VA mode). The remaining work: bind each guest CE/GR channel to a real
@@ -587,20 +618,35 @@ go-ahead, recovery plan, pstore, never unattended
with the guest-RAM ranges unregistered proved that fallback again
(`copy_skipped=121`, recovered once each by the software copy,
`copy_fail=0`, smoke still green), so the software fallback remains for any
- unqueued or failed copy. **Residual (P6a trigger path):**
- the debug-trigger path pre-builds its page-table slots once during ladder
- stage 3; a guest page's host physical address can change between
- translations (observed `0x13a591000` then `0x3b8318000` for the same GPA),
- and a runtime PTE update into an already-fetched 128-byte line is invisible,
- so the later write-direction trigger failed its data check. Forwarding is
- unaffected (it resolves HPAs at submit time); the follow-up is to give the
- debug triggers fresh scratch lines (the forwarding path) or to pin the
- translated page. **Remaining:** P6c - completion/fault relay, real
- page-directory shadowing, real guest channels and push forwarding with
- guest page tables in the real MMU, and `vectorAdd` correct results;
- guest-submission forwarding through the real guest page tables is not
- implemented yet. All new parameters default off; with them off the software
- path is unchanged.
+ unqueued or failed copy. **P6a trigger residual — fixed (2026-10-02):**
+ the debug triggers now re-translate the guest page's host physical address
+ at trigger time through the fresh-scratch-line path (triggers 6/7), retiring
+ the stale-PTE data-check failure; live `GUEST G2H COPY OK` and
+ `GUEST W2G COPY OK verify_ok=1`, with the PTE self-test and ladder
+ `REAL COPY OK` on a clean release. Forwarding was never affected (HPAs are
+ resolved at submit time). **P6c C0/C1 (engine-authored guest CE semaphore
+ relay, default off):** with `gsp_phys_relay=1` the guest-visible semaphore
+ page is mapped into the CE channel and the engine authors one selected
+ release; the deferred hold verifies the guest word and skips the software
+ apply of that record, and any refusal, failure, timeout or failed
+ verification leaves the unchanged software apply in charge, so the release
+ stays exactly-once. Live: the engine-authored release was observed in the
+ guest page (`relays=64`), the relay-on run stayed green
+ (`requested=68 ok=67 fallback=9`, smoke `ALL OK` 64/64), and a
+ forced-unresolvable control proved the fallback recovers every copy once;
+ one relayed copy released its semaphore but failed the host-side data
+ compare, so the parameter stays default-off pending triage, with the
+ mismatch logged and the software apply still in charge. **P6c C3
+ (forwarded-engine fault observation/relay, default off):** with
+ `gsp_phys_fault_relay=1` a timed-out forwarded copy drains the GSP-RM
+ status queue for the engine's fault record, matches it to the ladder
+ channel and writes the guest channel notifier/event through the existing
+ MSI-X path (live green, default off). **Remaining:** real guest page tables
+ and channels, real page-directory shadowing, GR context validity/execution
+ and `vectorAdd` correct results; guest-submission forwarding through the
+ real guest page tables is not implemented yet, and the GR promote attempt
+ wedged the server (§0.2). All new parameters default off; with them off the
+ software path is unchanged.
* **Scope:** map guest page-table pages/records so the physical GPU can DMA
them; forward `SET_PAGE_DIRECTORY` (`gsp_rm_emulation.c:2975`) to the real
server; allocate real memory objects for guest allocations (fn4 with real
@@ -677,13 +723,18 @@ go-ahead, recovery plan, pstore, never unattended
order).
8. **Debuggability and evidence.** Keep every real RPC logged at the TX/RX
seams (`booter:3050`, `:3127`, status-walk logs), keep the read-only
- `gsp-phys-state`/`gsp-phys-log-dump` files, and keep the evidence tree
- (`backups/evidence-*`) current on every live cycle.
+ `gsp-phys-state`/`gsp-phys-log-dump` files, and attach the evidence
+ artifacts to the change on every live cycle.
---
## Appendix A - line anchor map (function -> file:line)
+> **Historical anchors.** The line numbers below were taken from the
+> working-tree revision used for the original design and are stale; they are
+> kept as a historical map only. Anchor by function name when re-grepping and
+> treat the numbers as untrusted.
+
`kernel/gsp_phys_booter.c` (working-tree revision used for this design):
`gsp_phys_boot_prime` 1349, `gsp_phys_boot_run` 1524, `gsp_phys_status_queue_init`
2467, `_link` 2490, `_next` 2554, `gsp_phys_status_wait` 2670,
diff --git a/docs/04-implementation/01-known-knowns-and-unknowns.md b/docs/04-implementation/01-known-knowns-and-unknowns.md
index 1a5967b..0c8e50f 100644
--- a/docs/04-implementation/01-known-knowns-and-unknowns.md
+++ b/docs/04-implementation/01-known-knowns-and-unknowns.md
@@ -22,12 +22,14 @@ listed as an unknown-unknown has a *method* by which we will discover it.
### 1.1 Device and platform
* A stock 570.144 Linux guest driver probes and initializes against our
- emulated TU104 device; `nvidia-smi -L` works and CUDA user-space starts
- (`cuInit`, `cuDeviceGetCount`, `cuDeviceGet`). Context creation is in
- progress: the client reaches `cuCtxCreate` and currently ends in a forced
- `rc=999` on a completion wait; the engine inventory is complete (the async
- copy engine is included) and the remaining work is in the completion and
- work-execution path (see §2.1).
+ emulated TU104 device; `nvidia-smi -L` works and the full CUDA driver-API
+ smoke passes (`ALL OK` 64/64, copies matched) — `cuInit`, device queries,
+ context creation, allocation, copies, free and destroy complete, and the
+ copies are verified byte-for-byte. The execution backend is live: CE-class copies are
+ forwarded to the physical copy engine (default off) with engine-authored
+ semaphore releases and a fault relay; the remaining frontier is engine
+ execution — GR kernel execution is blocked on GR engine-context validity
+ (see §2.1).
* The same guest **boots unmodified**: no guest code, no guest module
parameters are required for the initialization path.
* The host is unmodified in the standalone configuration (distribution NVIDIA
@@ -86,29 +88,30 @@ it**, **rough cost**.
inventory including the async copy engine, and the copy-engine capabilities
match the physical device (two GRCEs). The earlier zeroed secondary engine
record that made `libcuda` rewrite the channel alloc class and roll back is
-gone; context creation now progresses past engine selection and stalls later,
-on completion of decoded work (see §1.1).
+gone; context creation completes and the driver-API smoke passes (see §1.1).
-**Unknown (resolved by decision, not yet implemented):** the correct,
-isolated way to run a guest's engine work on the physical GPU. The interface
-analysis in
+**Partially delivered:** the correct, isolated way to run a guest's engine
+work on the physical GPU. The interface analysis in
[`../03-architecture/08-physical-gpu-integration.md`](../03-architecture/08-physical-gpu-integration.md)
shows that proxying through the distribution host driver has no callable entry
point from kernel context, so the **chosen mechanism** is for the module to own
the physical GPU and run its real GSP firmware as the resource-manager server
-(the module becomes the host-side RM client). The emulation path remains the
-control plane.
+(the module becomes the host-side RM client). That backend is now live: CE-class
+copies and their releases are forwarded to the physical copy engine (default
+off) with a fault relay, while the emulation path remains the control plane.
+What is still open is **GR (kernel) execution**, blocked on GR engine-context
+validity.
-**Why:** this is *the* blocker for CUDA kernel execution, and later for
+**Why:** GR execution is *the* blocker for CUDA kernels, and later for
graphics.
-**Plan:** build the chosen mechanism in stages: device ownership + real GSP
-bring-up; host-side RM client for real objects and channels; translation of
-guest work and real submission; completion and fault relay. Measure each stage
-against a scorecard: correctness, isolation, latency, maintenance cost across
-driver versions. Deliverable: a forwarded `vectorAdd` with verified results.
-Alternatives (own scheduling without GSP; vendor vGPU or SR-IOV paths) are
-analyzed and rejected or deployment-gated in the integration doc.
+**Plan:** finish the backend in stages: GR engine-context validity and kernel
+submission; translation of the remaining guest work; completion and fault relay
+for every engine class. Measure each stage against a scorecard: correctness,
+isolation, latency, maintenance cost across driver versions. Deliverable: a
+forwarded `vectorAdd` with verified results. Alternatives (own scheduling
+without GSP; vendor vGPU or SR-IOV paths) are analyzed and rejected or
+deployment-gated in the integration doc.
**Cost:** weeks, not days; the largest single work item in the project.
@@ -146,16 +149,19 @@ remain the top risk; the division-of-labour question above is unchanged.
### 2.3 Fault semantics
-**Unknown:** fault buffer formats and replay/cancel semantics as the *guest*
-expects them in our virtualized setting (we know the buffer sizes and that
-UVM programs them, but not yet the end-to-end contract under forwarding).
+**Partly known:** the backend-side fault observation and relay now exists
+(default off — a notifier write plus an event post; the auxiliary event post
+can fail with `no_event`). **Still unknown:** the replay/cancel contract as the
+*guest* expects it in our virtualized setting (we know the buffer sizes and
+that UVM programs them, but not the end-to-end guest-side contract under
+forwarding).
**Why:** GPU page faults happen constantly in real CUDA workloads (UVM
oversubscription, migrations); correct handling is required for robustness.
**Plan:** reproduce faults deliberately (map a VA to nothing, touch it from a
-forwarded kernel), capture both sides, implement replay; validate with
-`uvm_fault`-heavy tests (e.g. oversubscribed memory).
+forwarded kernel), capture both sides, implement the guest-side replay/cancel
+contract; validate with `uvm_fault`-heavy tests (e.g. oversubscribed memory).
**Cost:** weeks.
@@ -320,9 +326,9 @@ To make a defensible statement, we need all of:
7. **Performance:** within an agreed factor of native for the supported cases,
measured with a published methodology.
-Until all seven hold, we say "initialization works; execution is in progress",
-and we link here. The CUDA-side checklist behind these gates and the current
-blocker are in
+Until all seven hold, we say "initialization and the driver-API path work;
+engine execution is in progress", and we link here. The CUDA-side checklist
+behind these gates and the current execution frontier are in
[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md` §9](../../userspace/cuda/docs/03-driver-and-gpu-interface.md);
start at
[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md).
diff --git a/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md b/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md
index 3265f47..4b886b3 100644
--- a/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md
+++ b/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md
@@ -53,9 +53,11 @@ without a methodology is not accepted in this project.
are answered from cached/derived state where the hardware has no side effects
the guest can observe.
* **Real work** (grid launches, copies) is meant to be forwarded to the
- physical GPU (see §3). Today only the work the host decoder understands —
- memsets, copies and semaphore releases — is executed in software as a
- correctness bridge; no CUDA kernel is forwarded yet.
+ physical GPU (see §3). CE-class copies and their semaphore releases are now
+ forwarded to the physical copy engine (default off) with a host-side fault
+ relay, and memsets/copies/releases the host decoder understands still run in
+ software as a fail-closed correctness bridge. No CUDA kernel is forwarded
+ yet: GR kernel execution is blocked on GR engine-context validity.
## 3. Forwarding: the options and the chosen mechanism
diff --git a/docs/05-operations/01-performance-correctness-security.md b/docs/05-operations/01-performance-correctness-security.md
index 3a2c6b4..21afd06 100644
--- a/docs/05-operations/01-performance-correctness-security.md
+++ b/docs/05-operations/01-performance-correctness-security.md
@@ -17,12 +17,14 @@ Rung 4 SUSTAINED workloads run for hours; faults/errors recoverable
Rung 5 MULTI-TENANT N VMs, isolation + fairness + no cross-talk
```
-**Current rung (2026-09-29):** Rung 1 is green; **Rung 2 is in progress.**
-`cuInit`/`cuDeviceGet` succeed, and the client reaches `cuCtxCreate`, which
-currently ends in a forced `rc=999` on a completion wait (engine inventory
-complete; completion translation in progress); `cuMemAlloc` is not reached yet
-and failed-context teardown has oopsed in the guest kernel in earlier runs.
-Before Rung 3 is claimed, the
+**Current rung (2026-10-02):** Rungs 1–2 are green: the stock driver probes,
+`nvidia-smi` enumerates, and the full CUDA driver-API smoke passes
+(`ALL OK` 64/64, copies matched) — `cuInit`, device queries, context creation,
+allocation, copies, free and destroy complete, and the copies are verified
+byte-for-byte. **Rung 3 is partial:** CE-class copies execute on the physical
+copy engine (default off) with engine-authored releases and a fault relay, but
+CUDA kernel execution is blocked on GR engine-context validity. Before Rung 3
+is claimed, the
**reply-fidelity gate** applies: a control whose reply fields the driver
consumes is not implemented until the reply is coherent (status-only →
coherent zeroed params → coherent values for consumed fields → exact values) —
@@ -64,8 +66,9 @@ New to CUDA analysis here: start at
`./scripts/stack_manager.sh test` runs the host suite (`make tests` only builds
the test binaries). CI (`.github/workflows/ci.yml`) runs on push and pull
requests only — there is currently no scheduled guest suite and no
-stock-driver contract test. A rung regression is a release blocker; the
-handoff documents carry the current rung.
+stock-driver contract test. A rung regression is a release blocker;
+[`STATE-CURRENT.md`](../../STATE-CURRENT.md) and this docs tree carry the
+current rung.
## 3. Performance
@@ -93,6 +96,10 @@ handoff documents carry the current rung.
* **Publish the harness**: the benchmark scripts live in `tools/`, and a
result is not comparable unless produced by them (or an equivalent,
documented harness).
+* **Runbooks over methodology**: the live physical-backend runbooks, scenario
+ scripts, debugfs surfaces and module parameters are collected in
+ [`eng-ops.md` Part II](../eng-ops.md); repeatable procedures live there, and
+ this document stays the measurement contract.
### 3.3 Targets
diff --git a/docs/05-operations/02-debugging-and-maintenance-workflows.md b/docs/05-operations/02-debugging-and-maintenance-workflows.md
index 8d711c6..cfc452d 100644
--- a/docs/05-operations/02-debugging-and-maintenance-workflows.md
+++ b/docs/05-operations/02-debugging-and-maintenance-workflows.md
@@ -322,6 +322,34 @@ version × each hardware family, run the step-5 bring-up sequence. When adding
a version, extend the matrix rather than replacing the old column —
regressions across versions are the cheapest bugs to catch.
+### 4.1.1 Green-eye-assisted upgrade/porting workflow (experimental)
+
+<!-- change (2026-10-02): add the green-eye probe -> symbols -> version diff ->
+portability matrix -> decide pipeline to the upgrade runbook, with pointers to
+the new RUNBOOK and green-eye user guide; additive only. -->
+
+For firmware-side questions the experimental green-eye tooling (see
+[`03-risc-v-gsp-rm-vm.md`](03-risc-v-gsp-rm-vm.md)) turns steps 3–5 above into
+a repeatable pipeline that produces attachable evidence before any emulation
+change. It never replaces the physical backend when silicon truth is required,
+and it never touches a guest, a kernel module or the installed driver.
+
+| Rung | Command (from `debug/risc-v-gsp-rm-vm/`) | Output |
+|---|---|---|
+| Probe | `./greeneye.sh probe <new.fwimage> --symbols symbols-<ver>.json --record` | first-divergence report; first-touch map |
+| Symbols | `./greeneye.sh symbols export --index <export.tsv> --bias 0 -o symbols-<ver>.json` | runtime-address symbol DB with provenance |
+| Version diff | `python3 harness/version_diff.py --old <old> --new <new> --out dumps/vdiff-<ver>` | shape/symbol/corpus/probe/behavior diff plus the M1–M6 skeleton |
+| Portability matrix | `python3 harness/portability_matrix.py --target <old> --target <new> --out dumps/matrix-<ver>` | per-target sessions and per-pair `compare` reports |
+| Decide | `./greeneye.sh compare dumps/<old-run> dumps/<new-run> --symbols` | symbol-attributed deltas; a product change only after physical confirmation |
+
+Decision discipline: verify the pairing first; behaviour that moved without a
+code change is usually a state/ordering fix; changed code on a live path is
+prioritised; a symbol never hit in either run is a coverage gap, not dead code.
+The full incident procedures (symptoms, evidence, response order, rollback) are
+in the [Open-IOV Defensive Operations Runbook](../RUNBOOK.md) S1 and its OODA
+ordering; the quickstart and tool chooser are in the
+[green-eye user guide](04-green-eye-user-guide.md).
+
### 4.2 New guest driver version
1. Install the stock driver in a guest (no patches).
diff --git a/docs/05-operations/04-green-eye-user-guide.md b/docs/05-operations/04-green-eye-user-guide.md
new file mode 100644
index 0000000..a9bf1dc
--- /dev/null
+++ b/docs/05-operations/04-green-eye-user-guide.md
@@ -0,0 +1,439 @@
+# green-eye User Guide (experimental)
+
+> **# TOOL IN DEVELOPMENT - NOT READY YET!**
+> green-eye is the development environment for running GSP/uproc firmware
+> off-hardware, observing it, injecting into it and comparing runs as evidence.
+> It is never shipped, never a driver dependency and never present in a guest;
+> interfaces and file formats may change without notice. For incident response
+> procedures see the [Open-IOV Defensive Operations Runbook](../RUNBOOK.md).
+
+This guide is the single front door for a new user: what green-eye is, how to
+get a first recorded run in ten minutes, which tool to use for which task, how
+to treat sessions as evidence, and what to do when something is wrong. The
+authoritative details live in the tool's own documentation set
+([documentation map](#10-documentation-map)).
+
+## 1. What green-eye is
+
+green-eye runs GSP/uproc firmware in a controllable machine, instruments it
+with Frida-like primitives, records a durable session, changes the machine's
+inputs (including injected code), and compares runs so that emulation changes
+in `open-iov-vfio` land with evidence instead of anecdotes.
+
+| It is | It is not |
+|---|---|
+| A firmware runtime microscope (execution + memory: L0/L1) | A timing, interrupt or device model (L2/L3 absent) |
+| A scriptable `Session` API with two backends (QEMU-RSP and native) | A GDB wrapper; there is no GDB dependency |
+| A recorded-evidence system with a versioned session format | A product component, a guest agent or a shipped dependency |
+| A static/dynamic join for Ghidra | A decompiler or a CUDA-corpus replacement |
+| An automation layer (harnesses, MCP, CI) | A fuzzing service against hardware; simulator/re-hosted code only |
+
+Ghidra answers *what is the code*; green-eye answers *what did it do*. Guest
+diagnostics answer *what did the driver request*; a supervised physical cycle
+answers *what does silicon do*. green-eye makes the middle question repeatable
+and safe.
+
+## 2. Ten-minute quickstart
+
+Requirements: Linux, Python 3.10+ (stdlib only), `qemu-system-riscv64`
+(Ubuntu `qemu-system-misc`), `clang` and `lld`. No GPU, no driver and no
+kernel module are touched.
+
+```bash
+cd open-iov/debug/risc-v-gsp-rm-vm
+
+./greeneye.sh info # environment check: QEMU, toolchain, paths
+make demo && make sdk # build the bundled RV64 uproc demo and injection SDK
+
+./greeneye.sh demo # canned run: boot, hooks, watchpoints, session
+./greeneye.sh analyze dumps/<printed-session-dir>
+
+./greeneye.sh probe # real GSP image: report the first divergence
+./greeneye.sh inject # load and invoke a position-independent service
+```
+
+What "good" looks like:
+
+1. `info` reports QEMU and the demo paths;
+2. `demo` prints hook/branch lines and a session directory under `dumps/`;
+3. `analyze` renders hook/watch/breakpoint events and a coherent summary;
+4. `probe` reports a bounded first divergence (for the TU104 570.144 image, a
+ load-access fault at firmware vaddr `0x4009014` reading
+ `0x2000000000000A00`) — that is expected data, not a boot;
+5. `inject` returns matching values on the direct and firmware-dispatched
+ paths.
+
+Next steps: [choose a tool](#5-choosing-the-right-tool) for your task and read
+[Common recipes](#7-common-recipes).
+
+## 3. Mental model
+
+### 3.1 Two backends, one session API
+
+```text
+CLI / REPL / Python ─┐
+MCP (stdio) ────────┼─► GreenEyeAPI ─► Session ─┬─► QEMU-RSP backend (qemu-system-riscv64)
+Ghidra bridge ──────┤ hooks, watchpoints, └─► native backend (sim/machine.py)
+harness/ + rmharness┘ memory, symbols, recorder ──► session directory
+```
+
+* **QEMU-RSP** is the default: full RV64 execution, gdbstub transport without
+ GDB, TCG trace plugin optional.
+* **native** (`--backend native`) is the in-process interpreter: deterministic
+ virtual time (instruction counts), fast startup, an exec/access trace, and
+ lenient MMIO for first-touch exploration.
+* Both satisfy the same `Session` primitives (registers/memory, breakpoints,
+ watchpoints, step/continue, snapshots), so scripts, probes, plugins,
+ recordings and comparisons survive a backend change.
+
+### 3.2 The primitives
+
+| Primitive | Use it for |
+|---|---|
+| `breakpoint` / `hook` | did this function run? with what arguments and return value? |
+| `watch` (`w`/`r`/`a`) | when did this field change, from which store PC? |
+| `run` / `run_until` | bounded observation; stop at a chosen address |
+| `call` | Frida-like remote call to unit-test a function in isolation |
+| `step` / `note` | single-step with dispatch; annotate the event stream |
+| `Memory.read/write/hexdump/search/snapshot/diff` | inspect and diff memory regions |
+| `SymbolDB` | resolve file/program/runtime addresses across sources |
+| `Recorder` | write the durable session (events, snapshots, blobs) |
+| injection SDK | load, register and invoke your own PIC RV64 service |
+
+Instructions between stops are invisible unless a trace is on: the TCG plugin
+(`-plugin qemu/greeneye-trace.so`) on QEMU, or `--trace` on the native backend.
+
+### 3.3 Fidelity levels (what a result can mean)
+
+| Level | Property | green-eye today |
+|---|---|---|
+| **L0** | instruction semantics | high (QEMU TCG, same ISA family) |
+| **L1** | memory behaviour | high for RAM-resident code, conditional on mapping |
+| **L2** | time/concurrency | **not modelled** (do not claim) |
+| **L3** | device environment (falcon, WPR2, CSB, PMU, timers, DMA, IRQ) | **absent** (device model is roadmap) |
+
+A simulator observation is evidence about the firmware, not about silicon;
+anything that would change the product is reproduced on a physical/vfio-mdev
+reference first. Never quote a green-eye session for timing, interrupt or
+security claims.
+
+### 3.4 A session is the unit of work
+
+Every run records a `greeneye-session/1` directory:
+
+```text
+dumps/<stamp>-<name>/
+ session.json format, name, counts, snapshot inventory, meta
+ events.jsonl append-only event stream (hooks, watches, memory, notes)
+ snapshots/*.bin.z zlib-compressed region captures with sha256
+ blobs/* QEMU logs, trace.jsonl, host-oracle stdout
+ summary.md rendered analysis
+```
+
+Partial sessions stay readable; the format evolves additively. A copied
+`events.jsonl` without its `session.json`, snapshots and metadata is not
+evidence.
+
+## 4. Tool inventory
+
+Run everything from `open-iov/debug/risc-v-gsp-rm-vm/` unless stated.
+
+### 4.1 CLI (`./greeneye.sh`)
+
+| Command | One-line purpose |
+|---|---|
+| `info` | environment/tooling check (QEMU, paths, capabilities) |
+| `demo` | canned boot + hooks + watchpoints + recording on the bundled uproc |
+| `run <script.py>` | run a Python instrumentation session against the demo/ELF |
+| `attach` / `launch` | interactive REPL; keep a target alive and print its gdbstub port |
+| `record` | boot and record with watches/snapshots (and `--trace` on native) |
+| `probe <image>` | bounded run of a raw image; first-divergence report |
+| `analyze <session>` | session to symbol-annotated Markdown report |
+| `compare <a> <b>` | symbolic A/B delta of two recorded sessions |
+| `symbols export\|resolve` | build/inspect a runtime-address symbol DB |
+| `inject` | load, register and invoke an SDK service (direct + mailbox) |
+| `ghidra` | serve the live session to GhidraScripts over localhost |
+| `plugins` | list discoverable plugins |
+
+### 4.2 Harnesses (`harness/`)
+
+| Tool | One-line purpose |
+|---|---|
+| `portability_matrix.py` | probe + fixed scenario + compare across targets, with an M1–M6 skeleton |
+| `mutation_harness.py` | seeded one-input mutations with outcome classes and per-attempt sessions |
+| `control_discovery.py` | sweep a call surface to find known/unknown/injected handler IDs |
+| `reach.py` | is one symbol reachable, what triggers it, with what arguments and effects |
+| `version_diff.py` | old vs new image: shape, symbols, corpus hints, probe, behavior |
+| `ci_release.py` | version_diff → matrix → mutation smoke, hashed into one bundle |
+| `port_driver_versions.py`, `port_new_device.py`, `package_sim.py` | porting/packaging scaffolds (dry-run default; pilot status and evaluation in [docs/28](../../debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md), [docs/29](../../debug/risc-v-gsp-rm-vm/docs/29-PACKAGING-AND-TRIALING.md)) |
+| `value_demos/run_all.py` | ten fixed scenarios s01–s10 with measured time-to-value |
+| `common.py` | shared helpers (flat relocation, sessions, bounded log reads) |
+
+### 4.3 Native simulator, plugins, host oracle, bridges
+
+| Tool | One-line purpose |
+|---|---|
+| `sim/` (`--backend native`, `--trace`) | deterministic RV64IMAC interpreter with exec/access trace and lenient MMIO |
+| `python3 -m greeneye.device_map harvest\|show\|merge\|demo` | first-touch MMIO map: authoring, inspection, merge |
+| `qemu/greeneye_trace.c` (`make -C qemu`) | full TCG exec/access JSONL trace with `start=`/`end=` filters |
+| `plugins/example_plugin.py` | plugin authoring reference |
+| `plugins/mcp_server.py` | MCP server for agents (stdio or loopback; read-only by default) |
+| `plugins/rmharness_plugin.py` | REPL commands for the host RM oracle |
+| `rmharness/harness.sh` + `rmtool`/`rmtop`/`rmq` | read-only host oracle captures appended into a session |
+| `ghidra/ExportForGreenEye.java`, `MarkGreenEyeHits.java`, `GreenEyeControl.java` | symbol export, hit annotation and decompiler-driven execution |
+| `sdk/` + `greeneye inject` | position-independent service ABI v1 |
+
+## 5. Choosing the right tool
+
+| Task | Tool | Notes |
+|---|---|---|
+| "Does this run?" / first divergence of a raw image | `./greeneye.sh probe` | no device model: expected early fault is data |
+| "What does this firmware do on this stimulus?" | `record` / `run` + `analyze` | fixed stimulus, bounded budgets, hooks/watchpoints |
+| "Which command IDs exist / what is undiscovered?" | `harness/control_discovery.py` | fresh session per probe; deterministic seed |
+| "Is this function reachable, and how?" | `harness/reach.py` | `auto|stimulus|trigger|run|call`; emits a replayable probe spec |
+| "What changed between two firmware versions?" | `harness/version_diff.py`, `./greeneye.sh compare` | static inventory + dynamic profile |
+| "Does the new firmware move behaviour or code?" | `harness/version_diff.py` + docs/14 decision table | code vs behaviour classification |
+| "Prove an error path / fail-closed rule" | `harness/mutation_harness.py` | simulator only; seeded and replayable |
+| "Prototype an RM service before product code" | `sdk/` + `./greeneye.sh inject` | PIC blob; direct and mailbox paths |
+| "Join Ghidra static addresses to runtime hits" | `./greeneye.sh ghidra` + Java bridges | localhost only; no auth |
+| "What does a control reply look like on real RM?" | `rmharness/harness.sh … rmtool` | read-only host oracle; safe beside a CUDA job |
+| "Explore MMIO the firmware touches first" | `probe` → `python3 -m greeneye.device_map` | versioned JSON; feeds the device model |
+| "Triage a release end to end" | `harness/ci_release.py` | always `--dry-run` first |
+| "Let an agent monitor a run" | `plugins/mcp_server.py` | read/observe by default; writes gated |
+
+Decision rule: static reading is a hypothesis; a recorded session is evidence;
+at least an A/B pair is a conclusion.
+
+## 6. Sessions as evidence
+
+1. **Record with a fixed stimulus.** The same script, seed, budgets and image
+ hash make runs comparable; a self-comparison must be empty.
+2. **Annotate as you go.** `s.note(...)` inside the session explains what was
+ asked and what was mutated; a chat log is not evidence.
+3. **Attach the provenance.** Firmware SHA-256, version string, pairing notes,
+ symbol DB source/hash/bias, and the exact command or script.
+4. **Keep the directory together.** `session.json` + `events.jsonl` +
+ `snapshots/` + blobs are one artifact; publish a SHA-256 list when moving
+ it.
+5. **State the fidelity.** L0/L1 exercised; L2/L3 not claimed; reference
+ capture named (hardware, vfio-mdev, host oracle or none).
+6. **Report deltas honestly.** `compare` marks non-semantic fields (time, host
+ paths, backend identity) as such; every remaining delta is explained or
+ listed as open. An empty unexplained-delta list is a claim that must be
+ earned.
+7. **Bundle for triage.** A release bundle is
+ `bundle-manifest.json` + `release-report.json` + version-diff + portability
+ matrix + mutation smoke; see
+ [docs/21](../../debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md) §4.3.
+
+The M1–M6 metric contract (inventory, reply, ordering, state, error-path and
+memory-side-effect agreement) is defined in
+[docs/06-ACCURACY](../../debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md) §5 and is
+authoritative for every claim.
+
+## 7. Common recipes
+
+### 7.1 First record and analyze
+
+```bash
+cd debug/risc-v-gsp-rm-vm
+make demo
+./greeneye.sh demo # prints the session directory
+./greeneye.sh analyze dumps/<session> # symbol-annotated report
+```
+
+### 7.2 Probe a new firmware image
+
+```bash
+./greeneye.sh symbols export --index <export.tsv> --bias 0 -o symbols-<ver>.json
+./greeneye.sh probe <gsp-<VER>.fwimage.bin> --symbols symbols-<ver>.json --record
+python3 harness/version_diff.py --old <old> --new <new> --out dumps/vdiff-<ver> \
+ --old-symbols symbols-<old>.json --new-symbols symbols-<new>.json
+```
+
+### 7.3 Discover controls and prove reachability
+
+```bash
+python3 harness/control_discovery.py --out dumps/discovery --max-id 32 \
+ --patterns zero --seed 5
+python3 harness/reach.py --target rpc_ping --method call --args 0x41 \
+ --out dumps/reach-ping
+```
+
+### 7.4 Mutate an input surface (simulator only)
+
+```bash
+python3 harness/mutation_harness.py --out dumps/mutation-7 \
+ --iterations 64 --seed 7 --timeout 0.25 --settle 0.2
+```
+
+### 7.5 Portability pipeline across targets
+
+```bash
+python3 harness/portability_matrix.py \
+ --target <reference-fwimage> --target <candidate-fwimage> \
+ --out dumps/matrix-candidate --probe-instructions 200 --scenario-duration 0.2
+```
+
+### 7.6 Inject and debug a service
+
+```bash
+make -C sdk
+make -C sdk relocs
+./greeneye.sh inject --service sdk/build/injected_demo.elf --cmd 1 --arg0 0x41 --arg1 0x42
+```
+
+The direct call is the unit test; the mailbox call is the integration test.
+Both are recorded.
+
+### 7.7 Drive execution from Ghidra
+
+```bash
+# terminal 1
+./greeneye.sh ghidra --kernel <elf> --bridge-port 7000
+# terminal 2 (Ghidra headless)
+$GHIDRA/support/analyzeHeadless <proj> <name> -process <program> -noanalysis \
+ -scriptPath open-iov/debug/risc-v-gsp-rm-vm/ghidra \
+ -postScript GreenEyeControl.java 7000 0x<address>
+```
+
+### 7.8 Host oracle capture (read-only)
+
+```bash
+./rmharness/harness.sh --session dumps/real-<anchor> --name before \
+ -- rmtool engine list --format=json
+./rmharness/harness.sh --session dumps/real-<anchor> -- rmtop --once
+```
+
+### 7.9 Release rehearsal
+
+```bash
+python3 harness/ci_release.py --old <old> --new <new> --out dumps/release-<ver> \
+ --old-label <OLD> --new-label <NEW> --dry-run
+```
+
+`--dry-run` prints the plan and creates nothing; the RM 610 gate refuses 610
+labels (exit 3) until a human approves in writing.
+
+### 7.10 Let an agent monitor
+
+```bash
+python3 plugins/mcp_server.py --stdio --backend native --record
+# read/observe by default; --allow-write and --allow-inject are explicit opt-ins
+```
+
+### 7.11 First-touch device map
+
+```bash
+./greeneye.sh probe <fwimage> --record
+python3 -m greeneye.device_map harvest dumps/<probe-session> --out dumps/<ver>-device-map.json
+python3 -m greeneye.device_map show dumps/<ver>-device-map.json
+```
+
+## 8. Troubleshooting
+
+| Symptom | Likely cause | Fix |
+|---|---|---|
+| `qemu-system-riscv64 not found` | QEMU absent or not on `PATH` | install `qemu-system-misc` or point `GREENEYE_QEMU` at the binary |
+| QEMU starts but the script cannot attach | wrong port / target exited | use the port `launch`/`demo` printed; re-run `launch` and `attach --port <port>` |
+| `Address already in use` on a bridge/trace port | stale bridge or another session | choose another `--bridge-port`; check `pgrep -fa greeneye` and `pgrep -fa qemu-system-riscv64` |
+| QEMU survives Ctrl-C of a script | `launch` keeps the target alive by design | close the session or stop the reported PID; do not kill an unrelated QEMU |
+| Session analyzer shows no hits/events | recording off, or the stimulus never ran | use the CLI (records by default); check hooks/watchpoints with a known symbol (`rpc_sum`) |
+| `NameError: __file__` in `run` scripts | script invoked outside `greeneye.sh run` | run through `./greeneye.sh run`, keep scripts inside the tool tree |
+| Empty TCG trace blob | plugin window missed or target never resumed | `make -C qemu`; widen `start=`/`end=`, set `exec=on`, resume the target |
+| MCP writes refused / no tools | server read-only by default; stdout polluted | pass `--allow-write`/`--allow-inject` deliberately; keep stdout protocol-only |
+| `greeneye error:` on a valid path | wrong relative path or `--db` file | `./greeneye.sh info` shows resolved paths; bad symbol DBs fail with a clear error |
+| Harness findings not reproducible | seed/budget changed | pin `--seed` and budgets; compare `findings.jsonl` byte-for-byte |
+| Everything "unreachable" | wrong surface/timeouts | run `--surface demo` on the built demo first; raise `--settle`/`--timeout` |
+| `Unknown symbol` in the optional OpenRM build | module-tag drift | re-pin build glue; see the [RUNBOOK](../RUNBOOK.md) S2 |
+
+Marker status: every tool document carries the
+**# TOOL IN DEVELOPMENT - NOT READY YET!** banner. The marker is removed only
+when the acceptance list in
+[docs/00-OVERVIEW](../../debug/risc-v-gsp-rm-vm/docs/00-OVERVIEW.md) passes on a
+clean checkout, the usage recipes verify end-to-end, a second person can run
+the quickstart unaided, and the session format and CLI are declared stable.
+
+## 9. Safety
+
+* **Simulator only.** The tools launch their own QEMU or native machine
+ against the bundled demo or an explicitly allowed re-hosted ELF. They never
+ attach to hardware, a guest, a kernel module, an mdev or an existing
+ gdbstub.
+* **No guest, no hardware, no product.** Nothing runs in a guest; injected
+ services are prototypes, not shipped code; product changes go through
+ `open-iov-vfio` with physical confirmation.
+* **Read by default; writes gated.** MCP `write_mem`/`call_function` need
+ `--allow-write`, injection needs `--allow-inject`, and every write is
+ recorded in the session.
+* **Bounded everywhere.** Runs have durations/stop caps; reads and snapshots
+ have size caps; harness subprocesses have timeouts; a wedged target is left
+ stopped and restarted clean rather than poked.
+* **Local only.** Ghidra and MCP transports are localhost/stdio; the bridge is
+ unauthenticated by design, so never expose it on a network interface.
+* **No confidentiality claims.** The simulator is a study environment, never a
+ security or confidentiality boundary; never use injection to defeat a
+ hardware security boundary.
+* **Fuzzing is gated.** Mutation runs only against the simulator or re-hosted
+ code, behind explicit harnesses, with recorded evidence.
+* **RM 610 is a hard human gate.** No acquisition, analysis, emulation edit,
+ physical work or support claim before written approval; the gate is enforced
+ by the CLIs.
+
+## 10. Documentation map
+
+Tool documentation (`debug/risc-v-gsp-rm-vm/docs/`): [00 overview](../../debug/risc-v-gsp-rm-vm/docs/00-OVERVIEW.md),
+[01 design](../../debug/risc-v-gsp-rm-vm/docs/01-DESIGN.md),
+[02 usage](../../debug/risc-v-gsp-rm-vm/docs/02-USAGE.md),
+[03 scripting](../../debug/risc-v-gsp-rm-vm/docs/03-SCRIPTING.md),
+[04 plugins](../../debug/risc-v-gsp-rm-vm/docs/04-PLUGINS.md),
+[05 Ghidra bridge](../../debug/risc-v-gsp-rm-vm/docs/05-GHIDRA-BRIDGE.md),
+[06 accuracy](../../debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md),
+[07 portability playbooks](../../debug/risc-v-gsp-rm-vm/docs/07-PORTABILITY-PLAYBOOKS.md),
+[08 roadmap](../../debug/risc-v-gsp-rm-vm/docs/08-ROADMAP.md),
+[09 firmware notes](../../debug/risc-v-gsp-rm-vm/docs/09-GSP-FIRMWARE-NOTES.md),
+[10 related work](../../debug/risc-v-gsp-rm-vm/docs/10-RELATED-WORK.md),
+[11 RM service guide](../../debug/risc-v-gsp-rm-vm/docs/11-GSP-RM-SERVICE-DEVELOPER-GUIDE.md),
+[12 injection SDK](../../debug/risc-v-gsp-rm-vm/docs/12-INJECTION-SDK.md),
+[13 Ghidra control](../../debug/risc-v-gsp-rm-vm/docs/13-GHIDRA-CONTROL.md),
+[14 adversarial interop](../../debug/risc-v-gsp-rm-vm/docs/14-ADVERSARIAL-INTEROP-GUIDE.md),
+[15 agentic GSP IDE](../../debug/risc-v-gsp-rm-vm/docs/15-AGENTIC-GSP-IDE.md),
+[16 custom simulator](../../debug/risc-v-gsp-rm-vm/docs/16-CUSTOM-SIMULATOR.md),
+[17 RM harness](../../debug/risc-v-gsp-rm-vm/docs/17-RM-HARNESS.md),
+[18 harness automation](../../debug/risc-v-gsp-rm-vm/docs/18-HARNESS-AUTOMATION.md),
+[19 control discovery](../../debug/risc-v-gsp-rm-vm/docs/19-CONTROL-DISCOVERY.md),
+[20 MCP server](../../debug/risc-v-gsp-rm-vm/docs/20-MCP-SERVER.md),
+[21 CI/release pipeline](../../debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md),
+[22 porting CUDA and devices](../../debug/risc-v-gsp-rm-vm/docs/22-PORTING-CUDA-AND-DEVICES.md),
+[23 RM 610 readiness](../../debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md),
+[24 device map](../../debug/risc-v-gsp-rm-vm/docs/24-DEVICE-MAP.md),
+[25 value validation](../../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md),
+[26 architecture](../../debug/risc-v-gsp-rm-vm/docs/26-ARCHITECTURE.md),
+[27 tool audit](../../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md),
+[28 porting automation](../../debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md),
+[29 packaging and trialing](../../debug/risc-v-gsp-rm-vm/docs/29-PACKAGING-AND-TRIALING.md), and the
+[long-term roadmap](../../debug/risc-v-gsp-rm-vm/GREEN-EYE-ROADMAP.md).
+
+Project documentation:
+
+* [Open-IOV Defensive Operations Runbook](../RUNBOOK.md) — incident scenarios
+ S1–S10, evidence bundles, escalation tree;
+* [GSP in a RISC-V VM](03-risc-v-gsp-rm-vm.md) — project-level purpose, status
+ and limits;
+* [Debugging and maintenance workflows](02-debugging-and-maintenance-workflows.md)
+ — captures, failure taxonomy and the upgrade runbook;
+* [Hardware and software matrix](../06-portability/hardware-software-matrix.md)
+ — support ladder and the portability pipeline;
+* [Adversarial interoperability](../04-implementation/02-practical-challenges-and-adversarial-interop.md)
+ — the no-patch rule;
+* [Engineering operations](../eng-ops.md) — tool admission, workflows and
+ retirement (the cross-cutting engineering-operations document);
+* CUDA playbook:
+ [`userspace/cuda/docs/07-version-upgrade-playbook.md`](../../userspace/cuda/docs/07-version-upgrade-playbook.md)
+ and [`userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md);
+* [Documentation front door](../README.md) — the full reading order.
+
+<!-- change (2026-10-02): correct the TCG plugin filename to the actual
+qemu/greeneye_trace.c, list the docs 28/29 scaffolds in the tool inventory and
+documentation map; scope: §4.2, §4.3, §10 only; benefit: copied commands and
+doc-map navigation match the tool tree. -->
diff --git a/docs/06-portability/03-device-validation-and-porting-sequence.md b/docs/06-portability/03-device-validation-and-porting-sequence.md
new file mode 100644
index 0000000..3c5d1aa
--- /dev/null
+++ b/docs/06-portability/03-device-validation-and-porting-sequence.md
@@ -0,0 +1,232 @@
+# Device Validation and Porting Sequence
+
+> How Open-IOV takes a new GPU device from "available to test" to a documented
+> support decision: which devices can be rented or borrowed, the physical
+> preparation, the supervised test-and-port sequence from driver **540** to the
+> current anchor, the evidence collected per device, and the go/no-go rules.
+>
+> Scope note: this document describes the **validation process**; the support
+> ladder and the current status of each combination live in
+> [`hardware-software-matrix.md`](hardware-software-matrix.md). Nothing here
+> moves a device up the ladder by itself. The companion automation is
+> [`../../debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md`](../../debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md).
+
+## 1. Rules that never bend
+
+1. **No unattended claim cycles.** Every cycle that will be used to claim
+ support runs with a human supervising the device; the simulator harness may
+ run unattended, the physical device may not.
+2. **Reboots need approval.** Host or device-host reboots, driver
+ bind/unbind, VM restarts and mdev claim/release for a claim cycle are
+ approved by the device owner before the cycle starts. One approved cycle at
+ a time.
+3. **The guest never contains Open-IOV code.** No guest agent, no patched
+ driver, no extra kernel parameters beyond what the stock driver needs.
+4. **A claim is evidence, not an intention.** "Tested and documented" only:
+ a device is Supported after this sequence's evidence exists and the matrix
+ records it with date, driver family, firmware hash and tested rung.
+5. **Simulator evidence is evidence about the firmware, never about silicon.**
+ Physical capture or explicit "not physically validated" is stated in every
+ report (see [`../05-operations/01-performance-correctness-security.md`](../05-operations/01-performance-correctness-security.md)
+ and the accuracy contract in the tool's docs/06).
+
+The operational steps for a supervised cycle (preflight, capture, recovery) are
+owned by the [Open-IOV Defensive Operations Runbook](../RUNBOOK.md) (§P7 and
+the physical-cycle response path). This document is the porting sequence that
+wraps that cycle.
+
+## 2. Device classes we can rent or borrow
+
+Support cost is dominated by *access to hardware*, so the sequence starts by
+naming where each class can come from. Eligibility is about PCIe passthrough
+and VFIO/mdev, not the marketing tier:
+
+| Class | Examples | Access path | Known constraints |
+|---|---|---|---|
+| Turing (RTX 20) | TU104 (RTX 2080-class) | **Owned reference**; further cards bought used | The current Reference profile and development host |
+| Ampere consumer (RTX 30) | GA102/GA104 | Lab loan or purchase; workstation rental | Same passthrough path as the reference; engine inventory differs (docs/07 P2) |
+| Ada consumer (RTX 40) | AD102/AD103/AD104 | Lab loan or purchase; workstation rental | ROM/VBIOS variants across board partners; inventory is profile data |
+| Blackwell consumer (RTX 50) | GB202/GB203 | Vendor eval, partner loan, purchase | Newer driver/firmware family; validate the pairing before debugging |
+| GB10-class | DGX Spark, Surface-class RTX laptops | Vendor eval / short loan, retail where available | aarch64 hosts, laptop ROM/VBIOS quirks, possible RV64GC and FMC partition model (docs/07 P3) |
+| Datacenter compute | A100, H100/H200, L40S | Cloud bare-metal rental or partner lab; short loans | Rental instances frequently expose MIG/vGPU rather than raw PCIe passthrough — confirm before paying for a cycle |
+| Datacenter/rack scale | GB200-class, NVL systems | Partner lab access only | Multi-tenant policy and MIG-aware handling; out of reach for casual rental |
+
+Procurement rules:
+
+* Prefer a **bare-metal rental with documented PCIe passthrough and IOMMU** over
+ a vGPU/mdev instance; the latter cannot exercise the VFIO mdev path.
+* A borrowed card arrives with an unknown ROM/VBIOS and firmware pairing:
+ collect and hash everything in §4 before touching the emulation.
+* One device per architecture/family at a time is enough to move the matrix;
+ do not accumulate hardware before the previous evidence bundle is complete.
+
+## 3. Physical preparation
+
+Preflight is a checklist, not a ritual (see the RUNBOOK for the host-specific
+commands):
+
+1. **Host qualification** — x86_64 or aarch64 host, IOMMU enabled
+ (`intel_iommu=on`/`amd_iommu=on`, `iommu=pt` on aarch64), ACS/isolation
+ confirmed, kernel with `vfio-pci`; record `uname -a`, kernel, and QEMU
+ version.
+2. **Slot and power** — card seated and powered per vendor spec; auxiliary
+ power connected; sufficient PSU headroom; thermals monitored for the cycle
+ (a throttled device invalidates timing-flavoured observations; timing is not
+ claimed anyway, docs/06 M7).
+3. **Isolation** — bind the target GPU to `vfio-pci`; keep the host distro
+ driver reserved as the **oracle**, never as the emulated anchor (anchor
+ discipline, docs/22 §1).
+4. **Baseline capture** — before any guest runs: host oracle captures
+ (`rmtool engine list --format=json`, `rmtool mem stats --format=json`
+ via `rmharness`, docs/17), `lspci -vvv` for the physical and virtual
+ functions, ROM/VBIOS hash, firmware family and version string.
+5. **Guest image** — stock guest driver of the exact version under test, a
+ CUDA smoke workload and the ioctl/control tracer
+ (`tools/guest-diag/ioctl_trace.c`).
+6. **Supervision plan** — named operator, one-cycle approval, expected
+ duration, watchdog, and the rollback path (old anchor directories and
+ config untouched; never reuse old-version addresses).
+7. **Simulator rehearsal first** — run the sub-sections below in `--dry-run`,
+ then in simulator-only mode, before the physical cycle. A plan that cannot
+ complete in simulation is not ready for silicon.
+
+## 4. Test-and-port sequence: driver 540 to current
+
+The sequence is **ascending**: each branch becomes the reference for the next,
+so a failure is attributed to one step, not to a jump. The current anchor is
+**570.144**; the sweep continues to the newest locally available image, and
+610+ stays behind the RM 610 approval gate (docs/23) until that approval
+exists. The `harness/` commands below run from
+`debug/risc-v-gsp-rm-vm/`; the supervised-cycle commands run on the host and
+device under test.
+
+Per driver/firmware version `N`:
+
+1. **Acquire and version** — obtain the runfile and matching GSP `.fwimage`
+ through the project's acquisition rules; lay out
+ `firmware/<SERIES>-GSP/<N>/` and `config/versions/<N>.yaml` (documentation
+ until the load path reads it). Never overwrite a working version; hash
+ every input immediately.
+2. **Static triage** — extract the image, build/locate the symbol DB, and run
+ the release pipeline in dry-run first:
+ `harness/version_diff.py` and `harness/ci_release.py` (docs/21). The sweep
+ orchestrator wraps this per version:
+ `python3 harness/port_driver_versions.py --versions <N> --target <N>=<fwimage> --dry-run`
+ (docs/28 §2; unadmitted pilot, evaluation in future-eng-ops F2).
+3. **Simulator evidence** — execute the sweep against the previous branch and
+ the anchor:
+ `python3 harness/port_driver_versions.py --run --baseline <N-1> ...`;
+ the per-version bundle carries `version-diff/`, `portability-matrix/` and
+ `mutation-smoke/` results plus the M1–M6 skeleton.
+4. **Device profile** — for a new device run
+ `python3 harness/port_new_device.py --run --device <name> --firmware <N-image> --reference <anchor-image>`
+ (docs/28 §3; unadmitted pilot, evaluation in future-eng-ops F3); fill the
+ profile from §4 evidence and reconcile the engine inventory against the
+ host oracle.
+5. **Supervised physical cycle** (approved, attended):
+ * boot the guest with the stock driver of version `N` on the device bound
+ to `vfio-pci`; run the CUDA smoke and the ioctl tracer;
+ * capture host module trace (`gsp-resman-vfio:`), guest dmesg,
+ `nvidia-smi -q`, `lspci -vvv`, and the control/ioctl sequence;
+ * compare the physical captures with the simulator sessions
+ (`greeneye compare`, M1–M6) and resolve every delta as profile, logic or
+ fidelity gap (docs/07 P4).
+6. **No new guest regression** — run the existing guest/CUDA ladder rungs and
+ confirm no rung regressed before the next version.
+7. **Rollback point** — keep the previous version's directories, config and
+ guest driver untouched until the next version's bundle is complete.
+
+Only after the current branch's bundle is complete does it become the
+reference for the next branch. 610 is not attempted here: see
+[`../../debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md`](../../debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md).
+
+## 5. Sequence diagram
+
+```mermaid
+sequenceDiagram
+ autonumber
+ participant O as Operator
+ participant H as Ops host
+ participant D as Device under test
+ participant G as Guest VM (stock driver)
+ participant S as green-eye harness
+ participant E as Evidence store
+
+ O->>H: approve one supervised cycle (reboot/bind approval)
+ O->>H: preflight (IOMMU, vfio-pci bind, baselines)
+ H->>D: bind target GPU to vfio-pci
+ O->>S: sweep --dry-run for version N
+ S-->>E: plan, input hashes
+ O->>S: sweep --run vs baseline (simulator only)
+ S-->>E: version-diff + matrix + mutation bundles
+ O->>G: start guest with stock driver N, run CUDA smoke
+ G->>D: GSP RPC / control traffic
+ O->>H: capture host trace, guest dmesg, lspci, oracle
+ H-->>E: physical captures
+ O->>S: compare physical vs simulator (M1-M6)
+ S-->>E: delta report, fidelity gaps
+ O->>O: go/no-go decision
+ alt go
+ O->>E: publish bundle + matrix entry
+ else hold
+ O->>E: record gap, keep previous anchor
+ end
+ D->>H: rollback bind, restore host driver
+```
+
+## 6. Evidence collected per device
+
+| Item | Source | Artifact |
+|---|---|---|
+| Device profile (PCI IDs, BARs, FB geometry, engines, interrupts, ROM) | `port_new_device` profile + host sysfs | `profiles/<device>.json` |
+| Admission/authorisation record | Ops process | cycle approval ID |
+| Physical host facts | preflight | `lspci -vvv`, ROM/VBIOS hash, kernel/QEMU versions |
+| Host oracle snapshots | `rmharness`/`rmtool` | `engine list`, `mem stats` JSON in a session |
+| Firmware pairing | image hash + family + version string | top of the port report |
+| Simulator bundle | `port_driver_versions`/`port_new_device` | `bundle-manifest.json` + tree, docs/28 |
+| Physical captures | supervised cycle | host module trace, guest dmesg, `nvidia-smi -q`, ioctl sequence |
+| Metric table | comparison | M1–M6 with reference-capture method and L2/L3 limits (docs/06 §5) |
+| Unexplained-delta list | review | explicit, possibly empty; empty is a claim |
+| Support decision | review | matrix entry with rung, date, driver/firmware, evidence hash |
+
+## 7. Go/no-go criteria
+
+Proceed to the next driver or publish a matrix entry only when **all** hold:
+
+* pairing verified and hashed before debugging (family, version string, FMC
+ manifest where applicable);
+* every probe first-touch delta explained as profile data vs logic;
+* engine/RPC inventory complete and reconciled against the host oracle;
+* M1–M6 do not regress on the reference workload, or each regression has a
+ decided response;
+* no guest-ladder rung regressed; rollback path intact;
+* the evidence bundle is complete and hashed, including the physical capture;
+* the physical cycle was supervised and approved.
+
+Hold (keep the previous anchor) when any of: ambiguous pairing, an unexplained
+probe or inventory delta, a matrix/M-metric regression without a response, an
+incomplete bundle, or a simulator-only claim with no physical capture. A
+**fidelity gap** (simulator diverges, physical cannot reproduce) is recorded
+as a gap, never "fixed" in the product from simulator evidence alone
+(docs/07 P4.5).
+
+## 8. References
+
+* Support ladder and per-combination status:
+ [`hardware-software-matrix.md`](hardware-software-matrix.md)
+* Current ops entry point and host procedures:
+ [`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md),
+ the [Open-IOV Defensive Operations Runbook](../RUNBOOK.md), and the
+ [green-eye user guide](../05-operations/04-green-eye-user-guide.md)
+* Porting automation (sweep + device scaffold, bundles, M1–M6):
+ [`../../debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md`](../../debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md)
+* Playbooks and porting guide:
+ [`../../debug/risc-v-gsp-rm-vm/docs/07-PORTABILITY-PLAYBOOKS.md`](../../debug/risc-v-gsp-rm-vm/docs/07-PORTABILITY-PLAYBOOKS.md),
+ [`../../debug/risc-v-gsp-rm-vm/docs/22-PORTING-CUDA-AND-DEVICES.md`](../../debug/risc-v-gsp-rm-vm/docs/22-PORTING-CUDA-AND-DEVICES.md)
+* RM 610 gate:
+ [`../../debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md`](../../debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md)
+
+<!-- change (2026-10-02): correct the guest-diag tracer path, state the working
+directory for the harness commands, and mark the porting scripts as unadmitted
+pilots with their evaluation pointers; scope: §3.1, §4 only; benefit: commands
+copy correctly and the scaffolds are not read as routine pipeline steps. -->
diff --git a/docs/06-portability/hardware-software-matrix.md b/docs/06-portability/hardware-software-matrix.md
index c839578..8fca9e9 100644
--- a/docs/06-portability/hardware-software-matrix.md
+++ b/docs/06-portability/hardware-software-matrix.md
@@ -35,7 +35,7 @@ yet and selection is compile-time (see §6).
| Guest | Status |
|---|---|
-| Linux x86_64 (stock `nvidia.ko`, CUDA) | **Experimental** (bring-up; probe and `nvidia-smi` work, but there is no continuous guest testing and context creation is currently blocked — see §6.1) |
+| Linux x86_64 (stock `nvidia.ko`, CUDA) | **Experimental** (bring-up; probe, `nvidia-smi` and the driver-API smoke pass, and CE copies are forwarded default-off, but kernel execution is blocked on GR engine-context validity and there is no continuous guest testing — see §6.1) |
| Linux aarch64 | **Target** |
| Windows x86_64 (WDDM driver, CUDA, 3D) | **Target** |
| Other Unix (FreeBSD, ...) | Out of scope until demand |
@@ -70,11 +70,34 @@ so new transports are a small, isolated implementation.
3. Record the differences; prefer configuration over code.
4. Publish the profile, the evidence, and any workarounds.
+**Green-eye portability pipeline (experimental).**
+
+<!-- change (2026-10-02): add the green-eye portability pipeline to the new-GPU
+procedure and point at the RUNBOOK/user guide; additive only. -->
+
+Before the profile change lands, run the firmware-side pipeline from
+`debug/risc-v-gsp-rm-vm/` and attach its output:
+`./greeneye.sh probe <image> --symbols <db> --record` for the first-touch map;
+`./greeneye.sh symbols export ...` for a provenance-carrying symbol DB;
+`python3 harness/portability_matrix.py --target <reference> --target <candidate>
+--out dumps/matrix-<candidate>` for per-pair sessions and `compare` reports;
+then classify profile-versus-logic deltas with the decision table in
+`debug/risc-v-gsp-rm-vm/docs/14-ADVERSARIAL-INTEROP-GUIDE.md` §4.1. The
+command-level walkthrough is in the
+[green-eye user guide](../05-operations/04-green-eye-user-guide.md); the
+incident procedure, evidence expectations and rollback rules are in the
+[Open-IOV Defensive Operations Runbook](../RUNBOOK.md) S5.
+
### 5.2 New firmware version
See [`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md) §4.1.
Firmware images are versioned under `firmware/<SERIES>-GSP/<VERSION>/` with
-signatures and analysis artifacts.
+signatures and analysis artifacts. The defensive procedure, evidence bundle
+and rollback rules for this path are in the
+[Open-IOV Defensive Operations Runbook](../RUNBOOK.md) S1.
+
+<!-- change (2026-10-02): cross-link the new firmware-version procedure to the
+defensive-operations RUNBOOK S1; additive only. -->
### 5.3 New driver version
@@ -118,6 +141,25 @@ config/
A profile is the *unit of support* — what is missing is the code that reads
the profile files, not the profile schema.
+**Evidence expectations for adding support.**
+
+<!-- change (2026-10-02): state the evidence required before a combination is
+entered in the matrix, and link the RUNBOOK bundle checklist and green-eye
+portability pipeline; additive only. -->
+
+A combination is entered in the matrix only with: the profile fields recorded;
+the firmware/device pairing verified by hash and version string before any
+debugging; a first-touch diff whose every changed address is explained as
+profile or logic; a portability matrix report with per-pair `compare` results
+and session hashes; the M1–M6 table with the reference-capture method and the
+L2/L3 limits stated; and a supervised physical validation for the support
+claim. "Tested and documented" is literal: an untested or simulator-only
+combination stays **Target** or **Experimental**. The full bundle checklist
+and fault tree are in the
+[Open-IOV Defensive Operations Runbook](../RUNBOOK.md) §6–§7; the pipeline
+commands are in the
+[green-eye user guide](../05-operations/04-green-eye-user-guide.md).
+
### 6.1 Userspace driver facts
* **Guest CUDA driver** (`libcuda`) under test: **570.144** (md5
diff --git a/docs/07-developers/02-development-and-analysis-environment.md b/docs/07-developers/02-development-and-analysis-environment.md
index 370cefc..114345b 100644
--- a/docs/07-developers/02-development-and-analysis-environment.md
+++ b/docs/07-developers/02-development-and-analysis-environment.md
@@ -189,6 +189,7 @@ capture them into sessions.
make -C tools/rmtool && make -C tools/rmtop
tools/rmtool/rmtool gpu list --format=json
tools/rmtool/rmtool engine list && tools/rmtool/rmtool mem stats
+cd debug/risc-v-gsp-rm-vm
./rmharness/harness.sh --session dumps/real-rm -- rmtool engine list --format=json
```
@@ -312,7 +313,7 @@ Detailed procedures are not duplicated here.
|---|---|---|---|
| New CUDA userspace (`libcuda` bump) | `cudatool.py info/delta`, `fdematch`, `diff_corpus.py`, name-map apply | guest smoke (`cuda_drv_test.c`), `cuda-ab`, symbol spot-checks | [`userspace/cuda/docs/07`](../../userspace/cuda/docs/07-version-upgrade-playbook.md); green-eye [`22`](../../debug/risc-v-gsp-rm-vm/docs/22-PORTING-CUDA-AND-DEVICES.md) §2 |
| New GSP/RM firmware version | decompile + diff RPC dispatcher/controls; Ghidra symbol inventory | `greeneye probe` first-touch, scripted/re-hosted `record`, `compare`, `version_diff.py` | [`05-operations/02` §4.1](../05-operations/02-debugging-and-maintenance-workflows.md) (upgrade runbook); green-eye [`07 P1`](../../debug/risc-v-gsp-rm-vm/docs/07-PORTABILITY-PLAYBOOKS.md), [`21`](../../debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md), [`22`](../../debug/risc-v-gsp-rm-vm/docs/22-PORTING-CUDA-AND-DEVICES.md) §3 |
-| New device / architecture | device profile data, ISA/image shape, PMU/SOE deltas | portability matrix, probe first-touch on the new image | [`06-portability/hardware-software-matrix.md`](../06-portability/hardware-software-matrix.md); green-eye [`07 P2/P3`](../../debug/risc-v-gsp-rm-vm/docs/07-PORTABILITY-PLAYBOOKS.md), [`18`](../../debug/risc-v-gsp-rm-vm/docs/18-HARNESS-AUTOMATION.md) |
+| New device / architecture | device profile data, ISA/image shape, PMU/SOE deltas | portability matrix, probe first-touch on the new image | [`06-portability/hardware-software-matrix.md`](../06-portability/hardware-software-matrix.md) and [`03-device-validation-and-porting-sequence.md`](../06-portability/03-device-validation-and-porting-sequence.md); green-eye [`07 P2/P3`](../../debug/risc-v-gsp-rm-vm/docs/07-PORTABILITY-PLAYBOOKS.md), [`18`](../../debug/risc-v-gsp-rm-vm/docs/18-HARNESS-AUTOMATION.md) |
| A/B accuracy ritual (real vs emulated) | M1–M6 metric definition | hardware/VFIO reference capture vs green-eye session | [`05-operations/01`](../05-operations/01-performance-correctness-security.md); green-eye [`06`](../../debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md), [`07 P4`](../../debug/risc-v-gsp-rm-vm/docs/07-PORTABILITY-PLAYBOOKS.md), [`25`](../../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md) |
| RM 610 family | — | gate first; nothing opened before approval | green-eye [`23`](../../debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md) (hard gate), [`21` §3](../../debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md) |
@@ -361,3 +362,8 @@ dynamic session disposes, and physical confirmation gates any product change.
Every path in this document was verified at the time of writing; when a tool
moves, update this table first — it is the front door.
+
+<!-- change (2026-10-02): fix the host-oracle capture command to run from the
+tool root and link the device-validation/porting sequence from the porting
+table; scope: §7, §12 only; benefit: the copied rmharness command resolves the
+wrapper and the device-port path is discoverable from the environment map. -->
diff --git a/docs/07-developers/onboarding-and-standards.md b/docs/07-developers/onboarding-and-standards.md
index e62ec28..78100c4 100644
--- a/docs/07-developers/onboarding-and-standards.md
+++ b/docs/07-developers/onboarding-and-standards.md
@@ -62,8 +62,8 @@ open-iov/
├── scripts/ build/load/test automation, VM harness
├── config/ hardware/version profiles (the portability surface)
├── firmware/ versioned firmware images + decompilation artifacts
-├── (private) reverse-engineering study notes live untracked;
-│ see the maintainers' study index
+├── (private) reverse-engineering study notes live untracked,
+│ outside the published tree
├── docs/ THIS documentation tree (start at docs/README.md)
└── wiki/ long-form firmware/protocol deep dives
```
@@ -156,7 +156,7 @@ The following rules each come from a real bug and are enforced in review:
| `tools/cuda-ab/` | host-vs-guest CUDA A/B harness (one deterministic workload, diffed results) |
| `tools/rmq/` | host RM-API oracle for ground-truth GPU/RM data |
| `tools/firmware_analyzer.py` + Ghidra scripts | firmware analysis pipeline |
-| module debug parameters | guest-memory scanning/poking, VRAM inspection |
+| module debug parameters | guest-memory scanning/poking, VRAM inspection; physical-backend parameters and debugfs state/control files (see [`../eng-ops.md`](../eng-ops.md) Part II) |
| QEMU gdbstub workflow | guest virtual-memory inspection/writing |
| `debug/risc-v-gsp-rm-vm/` (green-eye) | **# TOOL IN DEVELOPMENT - NOT READY YET!** — RISC-V GSP/uproc VM runtime inspection, session recording and `gsp-rm-vfio` accuracy measurement (see [`../05-operations/03-risc-v-gsp-rm-vm.md`](../05-operations/03-risc-v-gsp-rm-vm.md)) |
| `wiki/` documents | protocol and firmware deep dives |
@@ -193,6 +193,12 @@ make clean
* Protocol changes: diff the RPC inventory before/after; attach both.
* Memory-model changes: re-run the VA↔PA discovery tooling and a guest CUDA
start-up; attach the trace.
+* Live physical-backend cycles are **supervised only**: one cycle at a time,
+ never unattended, with the module sha256 recorded before and after and the
+ clean-release checklist walked every time. Do not retry after a
+ mutating-RPC failure or a wedge. A requested reboot needs a written reason
+ and a post-reboot plan before approval is asked (project hard rule). See
+ [`../eng-ops.md`](../eng-ops.md) Part II.
* Never land a change that makes a rung regress, even "temporarily".
## 8. Security hygiene for contributors
diff --git a/docs/08-ecosystem/haaeep-and-collaboration.md b/docs/08-ecosystem/haaeep-and-collaboration.md
index 954438c..ee847d4 100644
--- a/docs/08-ecosystem/haaeep-and-collaboration.md
+++ b/docs/08-ecosystem/haaeep-and-collaboration.md
@@ -92,7 +92,7 @@ licensed and version-durable — so it is the component we are building.
| Milestone | Depends on |
|---|---|
-| CUDA execution in a guest (rung 3) | Rung 2 (runtime start) being green first — context creation is in progress (`cuCtxCreate` currently ends in a forced `rc=999` on a completion wait) — and then Open-IOV forwarding work (backend chosen: a module-owned GPU with a real-GSP session). The interface contract is documented under [`../../userspace/cuda/README.md`](../../userspace/cuda/README.md) (start at [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md)) |
+| CUDA execution in a guest (rung 3) | Rung 2 (runtime start) is green — the driver-API smoke test completes end to end (`ALL OK`, payload 64/64). Copy-engine-class work and its releases are forwarded to real hardware with release and fault relay in place (parameters default off); graphics/compute (GR) execution is the open frontier. The interface contract is documented under [`../../userspace/cuda/README.md`](../../userspace/cuda/README.md) (start at [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md)) |
| Two-VM coexistence (inference + agent) with isolation | Open-IOV multi-tenancy + platform MAC |
| Xen/OpenXT or Qubes Air variant | Open-IOV transport backends |
| Attested boot-to-inference chain | TrenchBoot integration |
diff --git a/docs/09-reference/glossary-and-faq.md b/docs/09-reference/glossary-and-faq.md
index e0c68f6..1668e35 100644
--- a/docs/09-reference/glossary-and-faq.md
+++ b/docs/09-reference/glossary-and-faq.md
@@ -97,16 +97,24 @@ MIG is a hardware partitioning mechanism; Open-IOV's scheduling policy can use
it where available. It is part of the multi-tenancy work.
**How fast is it?**
-No published numbers yet: execution forwarding is not implemented, and even
-the initialization path has not been measured under the project's methodology.
-Any performance claim will come with the methodology and the measurements
-together (see
+No published numbers yet: copy-engine-class work forwarding and engine-authored
+release + fault relay are implemented but default off, and GR/kernel execution
+is still the open frontier; even the initialization path has not been measured
+under the project's methodology. Any performance claim will come with the
+methodology and the measurements together (see
[`../05-operations/01-performance-correctness-security.md`](../05-operations/01-performance-correctness-security.md)).
**Where do I report bugs or ask for help?**
Open an issue with the standard captures listed in the debugging document. A
reproduction plus logs is worth more than a description.
+**Can I run the live hardware cycles?**
+Live physical-backend cycles are supervised only — one at a time, never
+unattended — and a system is never rebooted without a written reason, a
+post-reboot plan and explicit human approval. The operational detail is in the
+engineering-operations document, Part II
+([`../eng-ops.md`](../eng-ops.md)).
+
**Can my company use this commercially?**
Yes, under BSD-3-Clause (kernel modules GPL-2.0), with no fee. We also welcome
co-development; see the ecosystem document.
diff --git a/docs/README.md b/docs/README.md
index 61aa529..77f4069 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -98,7 +98,10 @@ and [`01-orientation/02-the-ai-edge-and-ecosystem.md`](01-orientation/02-the-ai-
## 3. Where is the project today?
-**Honest status (2026-10-01):**
+**Honest status (2026-10-02):**
+
+A dated snapshot of this status, refreshed on every progress check, is kept at
+[`../STATE-CURRENT.md`](../STATE-CURRENT.md).
| Area | State |
|------|-------|
@@ -107,8 +110,8 @@ and [`01-orientation/02-the-ai-edge-and-ecosystem.md`](01-orientation/02-the-ai-
| GSP-RM message queue + RPC transport | ✅ Working |
| GSP-RM controls: device/fb/bus tables, CE caps, vaspace, GMMU static info | 🟡 Mostly — the engine inventory now includes the async copy engine and the CE capabilities match the device; reply fidelity for the remaining consumed fields is still being tightened |
| Guest `nvidia.ko` probe, `nvidia-smi` | ✅ Working (TU104) |
-| Guest CUDA driver (`libcuda`) startup | ✅ Driver-API baseline — `cuInit`, device enumeration, `cuCtxCreate`, `cuMemAlloc`, copies, free and destroy complete end to end; the smoke test prints `ALL OK` and the 64-byte payload round-trips exactly (64/64, re-verified 2026-10-01 on the current build). See [`04-implementation/01-known-knowns-and-unknowns.md`](04-implementation/01-known-knowns-and-unknowns.md) and [`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md) |
-| CUDA kernel execution | 🟡 The backend is live: the module owns the physical GPU and its real GSP-RM initialized, serves the full static-config reply and shuts down cleanly; a host resource-manager client now allocates memory/VAS/page-directory/TSG/channel and has executed the first real copy on the physical copy engine (4 KiB, semaphore released, dst==src, 2026-10-01). What remains is wiring that session into the guest-facing path so guest submissions execute there ([`03-architecture/11-physical-backend-guest-wiring.md`](03-architecture/11-physical-backend-guest-wiring.md)); see also [`03-architecture/08-physical-gpu-integration.md`](03-architecture/08-physical-gpu-integration.md) |
+| Guest CUDA driver (`libcuda`) startup | ✅ Driver-API baseline — `cuInit`, device enumeration, `cuCtxCreate`, `cuMemAlloc`, copies, free and destroy complete end to end; the smoke test prints `ALL OK` and the 64-byte payload round-trips exactly (64/64, re-verified 2026-10-02 on the current build). See [`04-implementation/01-known-knowns-and-unknowns.md`](04-implementation/01-known-knowns-and-unknowns.md) and [`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md) |
+| CUDA kernel execution | 🟡 Copy-engine-class guest copies and their releases are forwarded to the real hardware with release and fault relay in place (parameters default off); GR (graphics/compute) execution is the open wall — the host GR ladder brings up a GR0 channel with a compute-class object, but its first push faults on a context that was never committed. The device is currently held in the designed locked fail-safe awaiting a supervised reboot. See [`03-architecture/11-physical-backend-guest-wiring.md`](03-architecture/11-physical-backend-guest-wiring.md), [`03-architecture/08-physical-gpu-integration.md`](03-architecture/08-physical-gpu-integration.md) and [`../STATE-CURRENT.md`](../STATE-CURRENT.md) |
| OpenGL / Vulkan | 🔴 Not yet |
| Windows guest | 🔴 Not yet |
| Multi-tenant isolation (VFs, mdev-per-VM) | 🟡 Designed, partially implemented |
@@ -118,14 +121,16 @@ the userspace `libcuda` segfault was fixed (an RM allocation reply now echoes
the request parameters instead of returning zeros), the engine inventory is
complete, and the driver-API smoke test now completes end to end — `cuCtxCreate`
returns and the 64-byte payload round-trips exactly (64/64, re-verified
-2026-10-01 on the current build). The full
+2026-10-02 on the current build). The full
mechanism, with offsets and the decompiled call path, is in
[`../userspace/cuda/docs/05-570.144-decompilation.md`](../userspace/cuda/docs/05-570.144-decompilation.md).
-The remaining frontier is guest-facing execution: the runtime client initializes
-and queues a kernel launch, and the physical backend now executes real work (the
-first copy-engine transfer completed on hardware — 4 KiB with the semaphore
-released and dst==src, 2026-10-01) — but the two planes are not yet wired
-together. The backend, a module-owned GPU running its real GSP firmware as the
+The remaining frontier is guest-facing execution: copy-engine-class copies and
+their releases are forwarded to real hardware with release relay and fault
+relay in place (parameters default off); GR execution is the open wall — the
+host GR ladder brings up a GR0 channel with a compute-class object, but its
+first push faults on an engine context that was never committed. The device is
+currently held in the designed locked fail-safe awaiting a supervised reboot.
+The backend, a module-owned GPU running its real GSP firmware as the
resource-manager server, is specified in
[`03-architecture/08-physical-gpu-integration.md`](03-architecture/08-physical-gpu-integration.md);
the bridge from the live session to the guest-facing mdev/VFIO path is specified
@@ -170,16 +175,36 @@ Read in order if you are new. Jump directly if you know what you need.
| 18 | [Performance, correctness, security](05-operations/01-performance-correctness-security.md) | developers, auditors | 25 min |
| 19 | [Debugging and maintenance workflows](05-operations/02-debugging-and-maintenance-workflows.md) | developers, maintainers | 25 min |
| 20 | [GSP in a RISC-V VM (green-eye, experimental)](05-operations/03-risc-v-gsp-rm-vm.md) — **# TOOL IN DEVELOPMENT - NOT READY YET!** | developers, analysts | 15 min |
-| 21 | [Hardware and software matrix](06-portability/hardware-software-matrix.md) | developers, integrators | 20 min |
-| 22 | [Developer onboarding and standards](07-developers/onboarding-and-standards.md) | contributors | 20 min |
-| 23 | [HAAEEP and collaboration](08-ecosystem/haaeep-and-collaboration.md) | stakeholders, partners | 20 min |
-| 24 | [Glossary and FAQ](09-reference/glossary-and-faq.md) | everyone | reference |
+| 21 | [green-eye user guide (experimental)](05-operations/04-green-eye-user-guide.md) — **# TOOL IN DEVELOPMENT - NOT READY YET!** | developers, analysts | 15 min |
+| 22 | [Defensive Operations Runbook](RUNBOOK.md) | maintainers, operators | 20 min |
+| 23 | [Engineering operations](eng-ops.md) — tool admission and workflows (Part I) plus live physical-backend operations (Part II) | maintainers, developers | 20 min |
+| 24 | [Hardware and software matrix](06-portability/hardware-software-matrix.md) | developers, integrators | 20 min |
+| 25 | [Developer onboarding and standards](07-developers/onboarding-and-standards.md) | contributors | 20 min |
+| 26 | [HAAEEP and collaboration](08-ecosystem/haaeep-and-collaboration.md) | stakeholders, partners | 20 min |
+| 27 | [Glossary and FAQ](09-reference/glossary-and-faq.md) | everyone | reference |
+
+<!-- change (2026-10-02): insert the green-eye user guide and the Open-IOV
+Defensive Operations Runbook into the reading list; renumber rows 21-24 to
+23-26; additive only. -->
+
+<!-- change (2026-10-02): insert the engineering-operations document into the
+operations group; renumber rows 23-26 to 24-27; additive only. -->
Row 15 is a **deferred future goal**: host NVIDIA-driver acceleration over a
virtual function. The feasibility verdict and a first design sketch are
recorded there for planning; it is not on the active critical path (guest CUDA
execution, multi-VM, 3D, performance come first).
+> **Tool documentation is under development.** The per-tool pages under
+> [`tools/`](tools/README.md) (for example `rmtool.md` and `rmtop.md`) are
+> being expanded; until they are complete, start with the
+> [Defensive Operations Runbook](RUNBOOK.md) and the
+> [green-eye user guide](05-operations/04-green-eye-user-guide.md).
+
+<!-- change (2026-10-02): state that the per-tool documentation section is
+under development and point readers at the RUNBOOK and the green-eye user
+guide; additive only. -->
+
Deep-dive material lives in [`../wiki/`](../wiki/README.md) (firmware
decompilation record, address reference, historical protocol notes). Wiki
pages that predate the current implementation carry a status banner naming
@@ -191,6 +216,15 @@ The pre-reboot flat documents have been merged into this tree and removed
* debugging: gdbstub/NMI workflows, the EPT-livelock case study and the
failure taxonomy →
[`05-operations/02-debugging-and-maintenance-workflows.md`](05-operations/02-debugging-and-maintenance-workflows.md)
+* defensive operations: incident scenarios, the OODA ordering, evidence
+ bundles and the escalation tree →
+ [`RUNBOOK.md`](RUNBOOK.md); the green-eye quickstart, tool chooser and
+ troubleshooting →
+ [`05-operations/04-green-eye-user-guide.md`](05-operations/04-green-eye-user-guide.md)
+
+<!-- change (2026-10-02): point the post-merge document map at the new RUNBOOK
+and green-eye user guide; additive only. -->
+
* firmware runtime analysis (experimental): running GSP/uproc firmware in a
RISC-V QEMU VM with hooks, watchpoints, memory dumps and `gsp-rm-vfio`
accuracy measurement — **# TOOL IN DEVELOPMENT - NOT READY YET!** →
@@ -213,7 +247,9 @@ The pre-reboot flat documents have been merged into this tree and removed
and [`06-portability/hardware-software-matrix.md`](06-portability/hardware-software-matrix.md).
What remains historical is the pre-supersession wiki material and
-`old/ARCHITECTURE.md`; their status banners name the canonical document. (The
+`old/ARCHITECTURE.md`; the latter carries an explicit superseded banner
+pointing at this page, and the older wiki pages carry status banners naming
+the canonical document. (The
former 2026-09-21 `STATE.md` snapshot is retired; its content lives on in the
git history and its replacements above.)
diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md
new file mode 100644
index 0000000..79c1735
--- /dev/null
+++ b/docs/RUNBOOK.md
@@ -0,0 +1,744 @@
+# Open-IOV Defensive Operations Runbook
+
+> The per-incident book for operating an independent IP block (Open-IOV)
+> against ecosystems that change on their own schedule. NVIDIA ships new GSP
+> firmware, kernel modules, CUDA userspace, devices and ISA revisions without
+> coordinating with us; any of them can break the emulation while the others
+> stand still. This runbook is the ordered response.
+>
+> **Related:** [green-eye user guide](05-operations/04-green-eye-user-guide.md)
+> · [GSP in a RISC-V VM (experimental)](05-operations/03-risc-v-gsp-rm-vm.md)
+> · [Debugging and maintenance workflows](05-operations/02-debugging-and-maintenance-workflows.md)
+> · [Hardware and software matrix](06-portability/hardware-software-matrix.md)
+> · [Adversarial interoperability](04-implementation/02-practical-challenges-and-adversarial-interop.md)
+> · [Developer onboarding and standards](07-developers/onboarding-and-standards.md)
+> · [Engineering operations](eng-ops.md)
+>
+> **# TOOL IN DEVELOPMENT - NOT READY YET!** The green-eye tooling cited here
+> is experimental; its interfaces and file formats may change. Nothing in this
+> runbook authorises a product change, a hardware write or work on RM 610.
+
+## 0. How to use this runbook
+
+Audience: maintainers, release engineers and on-call operators of the Open-IOV
+emulation and its evidence pipeline.
+
+Rules for reading under pressure:
+
+1. Find the symptom in §5 (scenario catalogue) or §9 (one-page decision tree).
+2. Follow the **response order** exactly; do not skip the pairing/hash check.
+3. Before any action that could change `open-iov-vfio`, complete the evidence
+ bundle (§6) and pass the escalation gates (§7).
+4. Record the outcome where the scenario says to record it — a finding without
+ a record is lost work.
+5. If an action is not in this runbook and not in the linked project docs,
+ treat it as a candidate for a new runbook entry, not as improvisation.
+
+Command conventions: `open-iov/` is the repository root. Commands run from the
+repository root unless a block starts with `cd`. Angle-bracket values
+(`<VER>`, `<session>`, `<image>`) are supplied by the task; never hardcode
+private paths, host names or serial numbers into tracked documents.
+
+## 1. Operating principles and hard rules
+
+These apply to every scenario; where a scenario conflicts with them, the rules
+win.
+
+| # | Principle | Consequence |
+|---|---|---|
+| P1 | **Anchor discipline.** The lock-step is *guest driver ↔ emulated firmware profile*. The host distribution driver is the A/B oracle, not the anchor. | Never evaluate a new `libcuda` against an old emulated firmware profile and call the mismatch an emulation bug. |
+| P2 | **Never patch what we do not own.** Guest driver, host driver, GPU firmware, hypervisor, device model and guest OS stay unmodified. | Product changes live in our module, userspace, tools, configuration, tests and docs only. |
+| P3 | **Evidence over anecdote.** Every conclusion needs at least an A/B pair and an explicit fidelity level (L0–L3). | A single run, a static reading or a simulator-only delta cannot land a product change. |
+| P4 | **Physical confirmation for behaviour changes.** A simulator observation is evidence about the firmware, not about silicon. | Reproduction on a supervised physical/vfio-mdev cycle is required before a change that alters emulation behaviour. |
+| P5 | **The simulator is a study environment, never a security boundary.** | No confidentiality, side-channel or security claims; never use injection to defeat a hardware security boundary. |
+| P6 | **Development-only tooling.** green-eye is never shipped, never a dependency of the emulation, never in a guest. | Keep the in-development banner on tool docs until the acceptance list passes. |
+| P7 | **No unattended physical-GPU cycles.** Physical work is supervised, with fail-safe release and explicit reboot approval. | Stop and ask; never bind/unbind drivers or claim the GPU as an experiment. |
+| P8 | **RM 610 is a hard human gate.** Until a named human approves in writing, no acquisition, analysis, emulation edit, physical work or support claim for 610. | The gate is enforced in code by `version_diff`/`ci_release`; do not work around it. |
+| P9 | **Fail closed, record honestly.** Unverified host info, unpaired firmware, unexplained deltas and absent references stay open. | "No unexplained deltas" is a claim that must be earned, not a default. |
+
+## 2. Maintainer roles and responsibilities
+
+One name per role at any time; handovers are a written change note in the
+affected doc or PR.
+
+| Role | Owns | Key duties | Gates they hold |
+|---|---|---|---|
+| **Anchor maintainer** | guest driver ↔ emulated firmware anchor; version surface | keeps the anchor current; approves anchor bumps; keeps `config/versions/*.yaml` labelled as design targets until read by code | any change to the anchor |
+| **Firmware triage maintainer** | versioned firmware directories; symbol DBs; probe/version-diff runs | acquires and hashes new images; builds symbol DBs with provenance; runs the P1 loop; owns the pairing decision | pairing verdict; "new logic" vs "behaviour moved" |
+| **green-eye tooling maintainer** | `debug/risc-v-gsp-rm-vm/`; test suite; session format | keeps `make test` green; guards format/API stability; keeps the marker and doc map honest; maintains golden sessions | tool status claims; removal of the in-development marker |
+| **CI/release maintainer** | `harness/version_diff.py`, `harness/ci_release.py`, `greeneye.yml`, goldens | runs dry-run before real runs; hashes evidence bundles; diffs seeded corpora; keeps CI paths and budgets bounded | release triage sign-off; RM 610 gate enforcement |
+| **Physical-cycle supervisor** | supervised physical/vfio-mdev cycles | enforces fail-safe release, D0/D3cold handling and reboot approvals; captures reference traces | all physical execution; reboot requests |
+| **Evidence archivist** | session and bundle retention | keeps SHA-256 inventories; enforces provenance (no private paths, no unsanitised host data); applies retention/prune policy | publication of evidence |
+| **Docs steward** | this tree | enforces change notes, link checks and doc-map updates; retires stale claims | docs are part of "done" |
+
+Escalation order for an ambiguous incident: triage maintainer → anchor
+maintainer → physical-cycle supervisor (if hardware is involved) → project
+maintainer channels. Escalate with a reproduction and an evidence pack, never
+with a summary alone.
+
+## 3. Routine maintenance
+
+### 3.1 Golden sessions and determinism
+
+| Item | Cadence | Procedure | Pass condition |
+|---|---|---|---|
+| Demo golden session | every tool change and weekly | `./greeneye.sh demo` and compare against the archived session/expected events | UART banner and control-block liveness present; hook/watch/breakpoint sequence coherent |
+| Portability self-comparison | per PR touching `harness/`, `greeneye/`, `sim/` | `python3 harness/portability_matrix.py --target firmware/uproc_demo/build/uproc_demo.elf --target firmware/uproc_demo/build/uproc_demo.elf --out dumps/matrix-selfcheck` | `all_zero` deltas in `portability-report.json` |
+| Mutation replay | per PR touching mutation/harness code | run `mutation_harness.py` twice with the same seed and budgets | `findings.jsonl` byte-identical |
+| Version-diff self-check | per release-triage change | `version_diff.py` old == new | zero symbol/probe/behavior deltas |
+| Host oracle capture | per anchor change | `./rmharness/harness.sh --session dumps/real-<anchor> --name before -- rmtool engine list --format=json` (repeat for `mem stats`) | capture recorded with sizes and SHA-256 |
+
+Golden artifacts are append-only history: never edit a stored golden session
+(§3.2 uses SHA-256). Re-baseline by creating a new session with a change note.
+
+### 3.2 Session hygiene
+
+* Sessions are directories (`dumps/<stamp>-<name>/`) with `session.json`,
+ `events.jsonl`, `snapshots/`, `blobs/` and `summary.md`. Keep them together;
+ a copied `events.jsonl` without its `session.json` is not evidence.
+* Name sessions for the question asked, not the host or the operator.
+* Keep sessions bounded: fixed stimulus, fixed budgets, region-scoped
+ snapshots; a multi-gigabyte trace is a run that was not scoped.
+* Publish a SHA-256 list (or the `bundle-manifest.json`) with any session that
+ leaves the machine.
+* Write findings into `note` events inside the session (self-describing packs);
+ never rely on a chat log to explain a recording.
+* Apply the provenance rule everywhere: tracked docs describe components and
+ behaviour; sessions contain no host-specific paths in their metadata.
+* Bound growth deliberately: `dumps/` is gitignored scratch, so keep every
+ golden session and release bundle, publish the SHA-256 list before deleting
+ anything, and prune only scratch sessions past the review window — a
+ multi-gigabyte session is a scoping failure (fixed stimulus, fixed budgets,
+ region-scoped snapshots), not a storage problem to solve with new tooling.
+
+### 3.3 Toolchain updates
+
+| Dependency | Check | Action |
+|---|---|---|
+| QEMU | `./greeneye.sh info`; run `make test` | install/renew `qemu-system-riscv64` (Ubuntu `qemu-system-misc`); set `GREENEYE_QEMU` if the binary is elsewhere |
+| QEMU plugin API | `make -C qemu` after a QEMU upgrade | rebuild `qemu/greeneye-trace.so` against the vendored `qemu-plugin.h`; re-run a trace smoke |
+| Python | `python3 --version` (3.10+; stdlib only) | no pip packages required; keep plugins/harnesses stdlib-only |
+| clang/lld | `make demo`, `make sdk` | reinstall toolchain if the demo or SDK stops building |
+| Test suite | `python3 -m unittest discover -s tests -t .` | all green; skips for absent QEMU/demo explained, not ignored |
+| Native parity | `python3 examples/scripts/sim_diff.py` | `PARITY: OK` (or a clean skip when QEMU is absent) |
+| Docs | `python3 scripts/check-docs-links.py` from `open-iov/` | no broken relative links |
+
+### 3.4 Documentation and marker discipline
+
+* The **# TOOL IN DEVELOPMENT - NOT READY YET!** banner stays on tool docs
+ until the acceptance list in
+ `debug/risc-v-gsp-rm-vm/docs/00-OVERVIEW.md` passes; removal is a reviewed
+ change, not an edit.
+* Every behaviour change updates the doc that describes it (docs are part of
+ the definition of done); planned work is labelled `planned (docs/xx)`.
+* Status tables must match the code on the current commit (commands, flags,
+ test counts); stale claims are bugs.
+* Keep the doc map complete when adding documents; keep change notes
+ (`<!-- change (YYYY-MM-DD): purpose; scope; benefit -->`) at edited
+ locations.
+* Record reproducible discoveries in the failure taxonomy of
+ [`05-operations/02-debugging-and-maintenance-workflows.md`](05-operations/02-debugging-and-maintenance-workflows.md)
+ §5, not only in a session directory.
+* Tool admission, extension, trialing and retirement are governed by
+ [`eng-ops.md`](eng-ops.md); procedural detail that is not incident response
+ belongs there, not here.
+
+## 4. The OODA ordering for a hostile update
+
+"Hostile" here means *unplanned firmware change plus deliberate opacity* —
+new images on a hidden schedule, stripped and signed, hardware-coupled,
+possibly with anti-analysis or decoy behaviour. It does not assume a malicious
+actor, and it does not justify shortcuts.
+
+| Stage | Order | Actions | Exit criterion |
+|---|---|---|---|
+| **Observe** | 1 | Freeze the anchor; acquire the new input through the approved channel; record hash, version string and family; do not edit the emulation | Input is versioned and hashed in a parallel directory |
+| | 2 | Capture the baseline: real-backend reference traces for the fixed workload (or the archived previous capture) | Baseline exists with a provenance note |
+| | 3 | Record the symptom in a session note and a reproduction | Reproduction is repeatable and bounded |
+| **Orient** | 4 | Static map: symbol DB/corpus diff per version | Inventory deltas listed |
+| | 5 | Dynamic map: probe first-touch, then fixed-stimulus sessions on both versions | `compare` report with hit/watch/write deltas |
+| | 6 | Classify each delta with the decision table (code changed vs behaviour moved) | Every delta has a class or is explicitly open |
+| **Decide** | 7 | Fill the M1–M6 skeleton with the reference-capture method; list unexplained deltas | Metric table and open list attached |
+| | 8 | Choose: no change, version-keyed handler, profile data, or fidelity gap | Decision names the evidence that drove it |
+| | 9 | If the decision changes emulation behaviour, require physical reproduction; if it touches RM 610, stop at the gate | Gate passed or work parked |
+| **Act** | 10 | Implement in `open-iov-vfio` with the evidence bundle linked; never in the simulator | PR carries the bundle and the metric table |
+| | 11 | Re-run the identical stimulus; require profiles to improve or stay equal; re-run the guest ladder | No rung regression; CI green |
+| | 12 | Archive the bundle and update the matrix/version docs only where earned | Recorded and reversible |
+
+Ordering rules that keep this honest:
+
+1. Never orient before the input is hashed and paired.
+2. Never decide from a single session (P3).
+3. Never act on the product from simulator-only evidence (P4).
+4. Never let "the new version" become the anchor implicitly; the anchor moves
+ by explicit decision.
+
+## 5. Scenario catalogue
+
+Each scenario follows the same template: symptoms, detection, evidence,
+response order, tools/commands, decision criteria, rollback, where to record.
+Scenarios are incidents to run, not roadmap claims.
+
+### S1 — New GSP/RM firmware release
+
+| Attribute | Detail |
+|---|---|
+| **Symptoms** | Init fails with a generic status (the real 0x59 chase: `GSP_INIT_DONE rpc_result=0x59`), handler entries or reply shapes shift, a previously green guest step now fails, control inventory gains/removes IDs. |
+| **Detection** | Firmware hash/version string changed; probe first-divergence address changed; symbol/corpus inventory delta; fixed-stimulus behaviour delta. |
+| **Evidence** | Firmware SHA-256 + version string + pairing; old/new symbol DBs with provenance and bias; probe sessions; version-diff report; behaviour `compare`; M1–M6 skeleton; unexplained-delta list. |
+| **Decision criteria** | Pairing mismatch → stop. Code changed and live profile changed → prioritise new logic. Code unchanged and profile changed → behaviour moved (state/ordering/environment fix). Symbol never hit either side → coverage gap, widen the stimulus first. |
+| **Rollback** | The previous version's directory and configuration stay untouched; roll back by returning to the old anchor commit/tag and the old guest driver. A forward attempt never destroys the working setup. |
+| **Record in** | The version's `firmware/<SERIES>-GSP/<VER>/` notes, the PR body (link the bundle), and the matrix entry only once support is earned. |
+
+**Response order**
+
+1. Freeze the anchor and the workload; do not edit emulation code.
+2. Acquire the image through the approved channel; lay down a new versioned
+ directory; hash it.
+3. Build a symbol DB for the new image and the previous one.
+4. Probe both images with identical settings; diff the first-touch maps.
+5. Run the static triage (dispatcher/handler/control/corpus diff).
+6. Run the same stimulus on both (re-hosted or demo analog) and compare.
+7. Fill M1–M6 with the reference-capture method; classify every delta.
+8. Escalate any product-behaviour change to a physical reproduction.
+9. Land the change (or record the fidelity gap) with the evidence bundle; add
+ a version-keyed handler only where the diff requires it.
+
+**Commands**
+
+```bash
+cd debug/risc-v-gsp-rm-vm
+
+# 1) symbol DBs, one per version
+./greeneye.sh symbols export --index <export-VER.tsv> --bias 0 -o symbols-<VER>.json
+
+# 2) bounded probe of each image (records a session)
+./greeneye.sh probe <gsp-<VER>.fwimage.bin> --symbols symbols-<VER>.json --record
+
+# 3) full old-vs-new triage (shape, symbols, corpus, probe, behavior, M1-M6)
+python3 harness/version_diff.py \
+ --old <gsp-<OLD>.fwimage.bin> --new <gsp-<NEW>.fwimage.bin> \
+ --old-symbols symbols-<OLD>.json --new-symbols symbols-<NEW>.json \
+ --old-label <OLD> --new-label <NEW> \
+ --out dumps/vdiff-<OLD>-<NEW>
+
+# 4) behaviour compare and canonical report
+./greeneye.sh compare dumps/<old-run> dumps/<new-run> --symbols
+./greeneye.sh analyze dumps/<new-run> --symbols symbols-<NEW>.json
+```
+
+### S2 — New kernel driver / open-kernel-modules release
+
+| Attribute | Detail |
+|---|---|
+| **Symptoms** | The optional OpenRM build fails with `Unknown symbol`; exports move or disappear between tags; the pinned build glue no longer matches the installed modules; `modinfo`/`dmesg` report version drift. |
+| **Detection** | `./scripts/build_nvidia_openrm.sh` failure; `dmesg | grep 'Unknown symbol'`; `kernel-open` tree diff for the two tags; SBOM tag mismatch. |
+| **Evidence** | Old/new tag identifiers; build transcript; the `Unknown symbol` list; a recorded classification of each moved export; confirmation that the runtime path still uses the stock distribution driver. |
+| **Decision criteria** | If only build glue breaks and the runtime path is unaffected → repair or re-pin the glue. If an exported interface used by the emulation changed → treat as S1/S3 (functionality moved). If the distribution driver moved → it is the oracle; do not "upgrade" the anchor to match it. |
+| **Rollback** | Restore the previously pinned module tag and rebuild; revert the glue commit. |
+| **Record in** | `SBOM.md` tag updates, the build glue commit message, and the failure taxonomy if it is a reusable lesson. |
+
+**Response order**
+
+1. Verify which driver is actually in use (guest stock driver vs optional host
+ OpenRM modules) before debugging.
+2. Reproduce the build failure with a clean tree; capture the full error.
+3. Diff the `kernel-open` trees for the old and new tags (publish only the
+ symbol/interface delta).
+4. Classify: build-glue-only, symbol rename, interface removal, or new
+ functionality.
+5. Repair build glue if and only if the runtime contract is unchanged;
+ otherwise route to S3.
+6. Re-run `make kernel`, the host test suite, and the optional OpenRM build.
+
+**Commands**
+
+```bash
+# optional OpenRM build (build glue; never a runtime dependency)
+./scripts/build_nvidia_openrm.sh
+make kernel
+./scripts/stack_manager.sh test
+
+# evidence of the failure
+sudo dmesg -T | grep -i 'Unknown symbol'
+
+# runtime path sanity: guest remains stock
+# (guest: stock nvidia.ko; no Open-IOV code in the guest, ever)
+```
+
+### S3 — Functionality moving between kernel and GSP, or between userspace versions
+
+| Attribute | Detail |
+|---|---|
+| **Symptoms** | A control that used to be answered by the guest kernel is now requested from the firmware (or vice versa); new RPCs/controls appear; reply fields that used to be populated are ignored; a workload passes on one component mix and fails on another. |
+| **Detection** | Ioctl trace changes; control sweep finds new/removed IDs; dispatcher hit profile changes while static code is unchanged; `compare` shows hit/watch deltas with no source delta. |
+| **Evidence** | Recorded ioctl/control diff; `discovery-report.json` old vs new; `reach.py` reports for moved targets; session A/B pair; M1/M3/M4 rows with the reference method. |
+| **Decision criteria** | Same code and new hit profile → behaviour moved to a new caller/state; new handler ID → new logic; `default-path` result change → fail-closed/default behaviour changed; coverage gap → widen the sweep before deciding. |
+| **Rollback** | Keep both component versions in parallel directories; revert the mix to the last green combination; never convert a version-keyed handler into a version-blind one. |
+| **Record in** | The version's control-inventory notes, the interface docs for the exact version, and the PR that adds/updates the version-keyed handler. |
+
+**Response order**
+
+1. Capture the ioctl/control stream on the failing mix and on the last green
+ mix.
+2. Sweep the call surface (control discovery) and diff the inventory.
+3. Use `reach.py` to prove each moved target is reachable under the exact
+ stimulus and to record its trigger.
+4. Attribute every delta to kernel, userspace or firmware with a session;
+ fill M1/M3/M4.
+5. Update the version-keyed handler or capability table only for the classified
+ deltas; physical-confirm any behaviour change.
+
+**Commands**
+
+```bash
+cd debug/risc-v-gsp-rm-vm
+
+# unknown-call sweep and inventory diff between versions
+python3 harness/control_discovery.py --out dumps/discovery-<VER> --seed <seed>
+
+# reachability/trigger/args for a moved symbol
+python3 harness/reach.py --target <symbol> --method auto --out dumps/reach-<symbol>
+
+# symbol-attributed A/B
+./greeneye.sh compare dumps/<old-run> dumps/<new-run> --symbols
+```
+
+### S4 — New CUDA userspace / toolkit interacting unexpectedly with the GSP
+
+| Attribute | Detail |
+|---|---|
+| **Symptoms** | `cuInit` succeeds but context creation fails; a generic SUCCESS reply with incoherent consumed fields; a new or reordered control appears immediately before a failure; `libcuda` crashes (`lib[base+size] ip`); result mismatch in the A/B workload. |
+| **Detection** | `cudatool.py info`/`delta` on the new `libcuda`; `fdematch` transfer measurement; corpus diff; ioctl trace diff; `ab.sh` result comparison. |
+| **Evidence** | `delta` report (exports added/removed/moved, `.text` growth, build-id); `fdematch` table; wrapper re-resolution evidence; ioctl/control diff explained line by line; `cdt` output ending `ALL OK`; `ab.sh` `RESULT`/`STATUS` match; symbol DB with bias. |
+| **Decision criteria** | From the delta: no export moves + small `.text` growth → patch (name map transfer, spot checks); new/removed exports → feature (decode the new surface first); many moves → reshuffle (full re-analysis; never transfer wrapper addresses). Anchor mismatch (new `libcuda` vs old firmware profile) → fix the pairing first. |
+| **Rollback** | Keep `userspace/cuda/<version>/` trees in parallel; revert to the last green userspace version. Never overwrite an analysed version. |
+| **Record in** | The version's decompilation map/`names.tsv`, the version-upgrade playbook notes, and the firmware-anchor note when the anchor also moves. |
+
+**Response order**
+
+1. Cheap triage first (minutes): `info` + `delta`; classify the port.
+2. Measure name transfer before copying anything (`fdematch`).
+3. Build the green-eye symbol DB for the new version so runtime addresses
+ resolve; spot-check transferred addresses.
+4. Confirm dynamically: `cdt` with the ioctl tracer, then `ab.sh all`, against
+ the **same** emulated firmware profile (anchor discipline).
+5. Diff the ioctl/control sequence old vs new; explain every change.
+6. Only then touch the emulation, with physical confirmation per P4.
+
+**Commands**
+
+```bash
+cd userspace/cuda
+tools/cudatool.py info <new>/run/libcuda.so.<new>
+tools/cudatool.py delta <old>/run/libcuda.so.<old> <new>/run/libcuda.so.<new> > <new>/run/delta.txt
+tools/cudatool.py fdematch <old>/run/libcuda.so.<old> <new>/run/libcuda.so.<new> \
+ --names <old>/annotations/names.tsv
+tools/cudatool.py wrappers <new>/run/libcuda.so.<new>
+
+cd ../../debug/risc-v-gsp-rm-vm
+./greeneye.sh symbols export \
+ --names ../../userspace/cuda/<new>/annotations/names.tsv \
+ --index ../../userspace/cuda/<new>/run/analysis/decompiled/decompiled_index.tsv \
+ --bias 0x100000 -o symbols-<new>.json
+./greeneye.sh symbols resolve --db symbols-<new>.json 0x<transferred-addr>
+
+cd ../..
+tools/cuda-ab/ab.sh all # host oracle + guest run + compare
+```
+
+### S5 — New GPU model or architecture (RTX 5090 / GB10-class, aarch64 hosts)
+
+| Attribute | Detail |
+|---|---|
+| **Symptoms** | Probe first-touch addresses differ from the reference device; a firmware family/partition-manifest mismatch; ISA string differs (RV64GC vs RV64IMAC); aarch64 host reports `qemu-system-riscv64 not found`; PMU/SOE provisioning assumptions break. |
+| **Detection** | `file`/`readelf` image shape; pairing check (family, FMC partition manifest, version string); probe first-touch diff; host tooling check (`./greeneye.sh info`); `rmtool` host-oracle inventory. |
+| **Evidence** | Device profile fields (PCI IDs, BARs, FB geometry, engine inventory, interrupt topology, ROM quirks); pairing notes; reference vs candidate first-touch maps; portability matrix report; physical validation record before any support claim. |
+| **Decision criteria** | Changed first-touch on profile registers → profile data, not logic. New firmware family/partition model → new family inputs, not a re-badged image. RV64GC or new CSRs → ISA/decoder work (S6). No physical validation → status stays Target/Experimental, never Supported. |
+| **Rollback** | Keep the reference device/firmware directories untouched; revert the candidate profile; a support claim is only removed with a matrix change note. |
+| **Record in** | [`06-portability/hardware-software-matrix.md`](06-portability/hardware-software-matrix.md) and the device profile notes; the evaluation ladder result for the combination. |
+
+**Response order**
+
+1. Collect the device profile; verify the firmware pairing before debugging
+ anything (a mismatched pairing looks exactly like an emulation bug).
+2. Confirm ISA/image shape (`file`, `readelf`); note FMC/manifest and boot
+ model; diff PMU/SOE images.
+3. Probe the new image and the reference with identical budgets; diff the
+ first-touch maps.
+4. Run the portability pipeline across reference and candidate.
+5. Update profile data first; only add code when an interface genuinely
+ differs.
+6. Validate on the physical device under supervision before claiming support.
+
+**Commands**
+
+```bash
+file <device-fwimage> && readelf -h <device-elf> && readelf -l <device-elf>
+cd debug/risc-v-gsp-rm-vm
+./greeneye.sh info
+./greeneye.sh probe <device-fwimage> --symbols symbols-<device>.json --record
+python3 harness/portability_matrix.py \
+ --target <reference-fwimage> --target <device-fwimage> \
+ --out dumps/matrix-<device>
+./rmharness/harness.sh --session dumps/real-<device> --name before \
+ -- rmtool engine list --format=json
+# aarch64 host without qemu-system-riscv64:
+# install qemu-system-misc, or point GREENEYE_QEMU at the binary
+```
+
+### S6 — New instructions or ISA changes in NV-RISCV / falcon / GSP
+
+| Attribute | Detail |
+|---|---|
+| **Symptoms** | Probe faults with an illegal/unimplemented instruction; the native backend diverges from QEMU on the same image; compressed/float extensions or new CSRs appear; a re-hosted build built with the old `-march` fails or misbehaves. |
+| **Detection** | ISA string from `file`/`readelf`; probe fault cause/`desc`; QEMU-vs-native differential (`sim_diff.py`); decoder unit tests. |
+| **Evidence** | Image shape (ISA, segments, entry); fault report with cause, `epc`/`epc_firmware` and access address; `PARITY: OK` transcript or the exact divergence; a fidelity-gap note when the native decoder cannot execute the new instruction. |
+| **Decision criteria** | RV64IMAC on QEMU TCG → L0 high, proceed. RV64GC → QEMU supports it; update rehost `-march` and re-run parity. Native decoder gap → record as a fidelity gap; do not claim native coverage. Illegal instruction in QEMU itself → confirm ISA first, then escalate as a tool bug. |
+| **Rollback** | Revert rehost `-march`/build flags; keep the previous probe/session as the reference. |
+| **Record in** | The tool's accuracy/fidelity-gap list (`debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md` §8) and the image's version notes. |
+
+**Response order**
+
+1. Confirm the ISA and image shape; identify the exact faulting instruction and
+ address.
+2. Reproduce under QEMU (the reference execution engine) with a bounded probe.
+3. Run the QEMU-vs-native differential; if it diverges, capture both sessions.
+4. If the instruction is unsupported natively, record a fidelity gap with the
+ session as evidence; keep QEMU as the execution engine for that image.
+5. Update rehost build flags and re-run parity and the tool test suite.
+
+**Commands**
+
+```bash
+cd debug/risc-v-gsp-rm-vm
+./greeneye.sh probe <fwimage> --instructions <budget> --record
+python3 examples/scripts/sim_diff.py # requires QEMU; prints PARITY: OK on agreement
+make test
+```
+
+### S7 — Significant VBIOS/ROM changes
+
+| Attribute | Detail |
+|---|---|
+| **Symptoms** | The guest driver reads different device identity/VBIOS fields; probe-time register values move; ROM/VBIOS quirks change (laptop/Surface variants); a previously green probe hangs on a different register. |
+| **Detection** | Guest `lspci -vvv -d 10de:` and init-trace diff; snapshot diff of the identity/static-config region; `rmtool` read-only `BUS_GET_PCI_INFO` oracle; ROM hash change. |
+| **Evidence** | ROM hash and identity fields; before/after snapshot with a schema-attributed diff; pairing notes; the exact guest-observed values and the trace that consumed them. |
+| **Decision criteria** | Changed fields consumed by the driver → profile/override work, physically confirmed. Fields not consumed → record and monitor. Anything that would require impersonating a licensed product → forbidden; present a coherent device instead. |
+| **Rollback** | Revert the profile/override; previous ROM/profile stays archived with its hash. |
+| **Record in** | The device profile and the matrix entry; the failure taxonomy if the symptom was misleading. |
+
+**Response order**
+
+1. Hash the ROM/VBIOS and collect the identity fields.
+2. Capture a before/after snapshot of the firmware-side static-config region
+ and diff it field by field.
+3. Compare with the host oracle (read-only) and the guest's observed values.
+4. Change only profile data that a consumer demonstrably uses; confirm on
+ hardware.
+5. Record unexplained fields as open, not as noise.
+
+**Commands**
+
+```bash
+# guest
+sudo lspci -vvv -d 10de:
+# host oracle (read-only; verified queries from docs/17)
+# build once from the repo root: make -C tools/rmtool
+tools/rmtool/rmtool gpu info -d 0
+tools/rmtool/rmtool engine list
+
+cd debug/risc-v-gsp-rm-vm
+# snapshot the identity/static-config region before and after
+# (schema decode from the bundled/demo schema; real regions are version-keyed)
+./greeneye.sh compare dumps/<old-run> dumps/<new-run> --symbols
+```
+
+### S8 — Encrypted/protected memory or protected instructions (measurement plan; non-claims)
+
+| Attribute | Detail |
+|---|---|
+| **Symptoms** | The image consumes identity material that is provisioned encrypted (the PMU-delivered encrypted device identity in the real 0x59 campaign); protected-memory or protected-instruction paths are reached; a lenient run returns zeros where a real device returns provisioned values; boot halts at a secret-dependent step. |
+| **Detection** | First-touch/device-map analysis; lenient-MMIO `mmio_log` and `mmio_unknown`; boot-chain shim assumptions in session metadata; probe fault reports identifying the consuming PC. |
+| **Evidence** | First-touch map entry per protected address; `mmio_log` with PC and access; the exact access that consumes protected state; a written measurement plan and an explicit non-claims list. |
+| **Decision criteria** | Secret-dependent branch without provisioned input → blocked; record as a fidelity gap and a device-model requirement, not a bug. Zero-value success under lenient MMIO → hypothesis only. Any request to extract, defeat or prove protected state → refuse (P5/P7). |
+| **Rollback** | No product changes are made in this scenario; the only rollback is reverting experiment notes and keeping the map as data. |
+| **Record in** | The fidelity-gap list, the device-map file (`greeneye-device-map/1`), and the H5 cross-controller plan; never in a security-claims section. |
+
+**Measurement plan (what to measure, and what is not claimed)**
+
+| Measure | Method | Non-claim |
+|---|---|---|
+| Which address/instruction consumes protected state | bounded probe + device map + `mmio_log` | not a statement about the real value |
+| Whether execution continues when the value is absent/zero | lenient native run with `unknown_mmio="lenient"` | zeros are a hypothesis, not read-back semantics |
+| Which subsystem provisions the value | cross-controller inventory (PMU/SEC2/ACR/SOE) and boot-order notes | not a model of the secure state machine |
+| Whether the path is security-relevant | static map + recorded access ordering | not a side-channel, confidentiality or attack claim |
+
+**Response order**
+
+1. Classify: protected read, protected instruction, or a secret-dependent
+ branch. Do not attempt extraction; do not patch anything.
+2. Capture the first-touch map and the lenient `mmio_log` in a recorded
+ session.
+3. Write the measurement plan and the non-claims into the session note.
+4. Escalate the missing provisioning as a device-model requirement (H2/H5),
+ not as an emulation defect.
+5. Stop. Physical work on protected paths requires the supervisor and a
+ separate approved plan.
+
+**Commands**
+
+```bash
+cd debug/risc-v-gsp-rm-vm
+python3 -m greeneye.device_map demo --instructions 200000
+python3 -m greeneye.device_map harvest dumps/<probe-session> --out dumps/<ver>-device-map.json
+python3 -m greeneye.device_map show dumps/<ver>-device-map.json
+# lenient native continuation (programmatic):
+# ge.launch(kernel=..., backend="native",
+# device_map="dumps/<ver>-device-map.json",
+# unknown_mmio="lenient")
+# session.target.mmio_log / session.target.mmio_unknown
+```
+
+### S9 — New microcontrollers and new inter-/intra-microcontroller APIs and DMA lines
+
+| Attribute | Detail |
+|---|---|
+| **Symptoms** | The image stalls waiting on PMU/SEC2/ACR/SOE; falcon staging DMA does not land (the real failure mode: the SEC2 boot-loader block is not visible, the falcon does not halt); WPR2/security state differs from the successful path; new mailbox functions or DMA descriptors appear. |
+| **Detection** | Probe first-touch; boot-chain shim assumptions recorded in session metadata; compare against the last successful boot path; device-map diffs; cross-controller event absence in sessions. |
+| **Evidence** | First-touch addresses per controller window; mailbox/DMA access map; boot-order notes; shim assumptions in the session; the successful-path comparison; H2.2/H5 scope note. |
+| **Decision criteria** | GSP-dependent stall → PMU/SEC2 stub work first (H2.2 entry). New mailbox/API surface → inventory it before modelling. Security-chain semantics → shim/documented assumption, never a claimed secure model. Unmodelled DMA line → device-map entry plus model work, not a product change. |
+| **Rollback** | Revert any shim assumption change; the previous session/boot note stays as the reference. |
+| **Record in** | The cross-controller inventory and boot-order notes; the device map; the H5/H2.2 roadmap scope. |
+
+**Response order**
+
+1. Inventory the controllers and DMA lines the image touches first
+ (PMU, SEC2, ACR/SOE, booter).
+2. Reproduce the stall with a bounded probe and record shim assumptions in the
+ session.
+3. Compare against the last successful path (e.g. WPR2 state, SEC2 staging
+ visibility) and name the missing exchange.
+4. Model the minimum stub that lets the image proceed; keep it documented
+ data, not a silent special case.
+5. Gate real security-chain semantics behind H2.2/H5; escalate any conclusion
+ that would change the product.
+
+**Commands**
+
+```bash
+cd debug/risc-v-gsp-rm-vm
+./greeneye.sh probe <fwimage> --record
+python3 -m greeneye.device_map harvest dumps/<probe-session> --out dumps/<ver>-device-map.json
+python3 -m greeneye.device_map merge dumps/<ref>-device-map.json dumps/<ver>-device-map.json \
+ --out dumps/merged-device-map.json
+./greeneye.sh compare dumps/<last-good-run> dumps/<stall-run> --symbols
+```
+
+### S10 — Multi-version matrix operations and CI (RM 610 requires explicit human approval)
+
+| Attribute | Detail |
+|---|---|
+| **Symptoms** | A new matrix column regresses an existing combination; CI drifts from the local run; someone proposes a 610-labelled input or command; the release bundle is incomplete. |
+| **Detection** | `version_diff`/`portability_matrix` deltas; `ci_release --dry-run` plan inspection; the RM 610 gate refusal (`exit 3`); test transcripts. |
+| **Evidence** | `version-diff.json`, `portability-report.json`, `release-report.json`, `bundle-manifest.json`, seeded `findings.jsonl`, test transcripts, approval record for 610. |
+| **Decision criteria** | Dry-run clean before a real run. A step that raises is `error`, not skipped. Self-comparison must be `all_zero`; seeded replays byte-identical. **610: no acquisition, analysis, emulation edit, physical work or support claim before a named human approves in writing; an open-ended approval is invalid.** |
+| **Rollback** | Extend the matrix, never replace a column; revert the candidate commit and keep the previous anchor/tag/driver; a 610 attempt parks without touching 570.144. |
+| **Record in** | The port log/approval thread (610), the release bundle directory, and the matrix with its test date and rung. |
+
+**Response order**
+
+1. Run the release rehearsal on current-version labels first; inspect the
+ dry-run plan.
+2. Run `version_diff` and `portability_matrix` with fixed budgets; require the
+ self-comparison to be zero.
+3. Run the mutation smoke with a pinned seed; verify replay.
+4. Assemble and hash the bundle; attach it to the PR.
+5. For 610: stop at the gate; obtain the approval message; only then acquire
+ inputs and repeat the pipeline with 610 labels.
+
+**Commands**
+
+```bash
+cd debug/risc-v-gsp-rm-vm
+
+# rehearsal first: dry-run never opens a target and creates no output dir
+python3 harness/ci_release.py --old <old> --new <new> --out dumps/release-<ver> \
+ --old-label <OLD> --new-label <NEW> --dry-run
+
+# real pipeline (dry-run clean)
+python3 harness/ci_release.py --old <old> --new <new> --out dumps/release-<ver> \
+ --old-label <OLD> --new-label <NEW> \
+ --old-symbols symbols-<OLD>.json --new-symbols symbols-<NEW>.json \
+ --mutation-kernel firmware/uproc_demo/build/uproc_demo.elf \
+ --mutation-iterations 8 --mutation-seed 7
+
+# RM 610 gate: refused by default; explicit approval token required
+python3 harness/version_diff.py --old <old> --new <new> --out dumps/vdiff-610 \
+ --old-label RM-610 --new-label 570.144
+# exit 3 unless: --approval-token RM610-APPROVED or GREENEYE_RM610_APPROVED=1
+```
+
+## 6. Evidence-bundle requirements
+
+A bundle is the minimum attachable proof for a version/device/architecture
+triage or a product change. It combines the docs/07 evidence checklist, the
+docs/14 §6 adversarial pack and the docs/21 release tree.
+
+```
+<out>/
+ bundle-manifest.json sha256 + bytes of every file below
+ release-report.json pipeline summary (steps, gate, labels)
+ version-diff/ version-diff.json|.md, symbols/, sessions/, compare/
+ portability-matrix/ portability-report.json|.md, symbols/, sessions/, compare/
+ mutation-smoke/ findings.jsonl, summary.md, sessions/
+```
+
+Checklist — every box must be true or explicitly waived with a reason:
+
+* [ ] firmware identity: SHA-256, version string, family and pairing notes;
+* [ ] input provenance: channel, date, and the exact file names analysed;
+* [ ] symbol DB(s): hash, source (ELF/Ghidra/JSON), `bias` and provenance;
+* [ ] baseline and candidate session directories (or their SHA-256 list);
+* [ ] exact stimulus (script/spec plus mutation values) and budgets;
+* [ ] `compare` report and relevant `analyze` reports;
+* [ ] M1–M6 table with the reference-capture method and L2/L3 limits stated;
+* [ ] `greeneye_hits.tsv` and any Ghidra annotations produced;
+* [ ] `bundle-manifest.json` written last, covering the final reports;
+* [ ] unexplained-delta list (empty is a claim, not a default);
+* [ ] decision and product action (or explicit "no change") recorded;
+* [ ] for 610: approval ID, approver, date and scope limits.
+
+Fidelity statement template (copy into every report):
+
+```text
+Fidelity: L0 (instruction semantics) and L1 (memory) exercised; L2 (timing,
+concurrency) NOT modelled; L3 (device environment) absent for <registers>.
+Reference capture: <hardware / vfio-mdev / host oracle / none>.
+Physical reproduction required before product change: <yes/no + why>.
+```
+
+## 7. Fault and escalation tree
+
+```text
+Failure / new input observed
+|
++- Is the input hashed, versioned and paired? NO -> STOP: acquire/hash/pair first (S1)
+|
++- Did it come from RM 610 or a 610-labelled path? YES -> STOP: hard human gate (S10)
+|
++- Is a physical GPU / driver bind / mdev involved?
+| YES -> STOP: supervised physical cycle + reboot approval (P7); never unattended
+|
++- Does the question need timing, interrupts, protected state or real device
+| read-back semantics (L2/L3)?
+| YES -> simulator evidence is a hypothesis only (P3/P4/P5); record the
+| fidelity gap; product change requires physical reproduction
+|
++- Does it reproduce under a fixed stimulus in green-eye?
+| NO -> widen the stimulus / capture the real trace before theorising;
+| absence of a hit is not proof of dead code
+| YES -> classify with the S1 decision table (code changed vs behaviour moved)
+|
++- Would a fix change emulation behaviour?
+| YES -> physical/vfio-mdev reproduction + evidence bundle before landing
+| NO -> docs/profile/test-only change with the bundle attached
+|
++- Does any vendor component need modification?
+ YES -> forbidden (P2): redesign to emulate the interface instead
+```
+
+Stop conditions (do not proceed, do not "temporarily" work around):
+
+1. Unverified pairing or unhashed input.
+2. RM 610 without written approval.
+3. A physical cycle without the supervisor and fail-safe plan.
+4. A request to patch a vendor component or a guest.
+5. A security/confidentiality claim based on simulator state.
+6. A product change with no physical reproduction and no bundle.
+
+## 8. When NOT to use the tooling
+
+| Situation | Why not | Use instead |
+|---|---|---|
+| You need silicon truth (timing, interrupts, real register read-back) | L2/L3 are absent by design | supervised physical cycle + reference traces |
+| You are about to mutate or inject against hardware, a guest, a kernel module or an existing gdbstub | Tooling is simulator-only; hardware writes are prohibited | bounded QEMU/native harness with recorded sessions |
+| You need to defeat, extract or prove a security boundary | The simulator is not a confidentiality boundary | stop; escalate as a design question |
+| You need a product fix "now" without evidence | Violates P3/P4 | evidence bundle first; the anchor and matrix can wait |
+| The input has not been hashed/versioned/paired | Every later claim is contaminated | S1 steps 1–3 |
+| RM 610 has no written approval | Hard gate | request approval with the readiness checklist |
+| The physical GPU is held by another job | Shared-oracle safety and fail-safe discipline | host-side read-only oracle (`rmtool gpu list`, `harness.sh`) |
+| You need type recovery or decompilation | green-eye is dynamic ground truth, not a decompiler | Ghidra corpus and symbol tooling |
+| You need guest-driver behaviour | green-eye observes firmware, not the guest | `tools/guest-diag/`, ioctl tracer |
+| A one-line question can be answered by `analyze` on an existing session | Do not produce new runs for no reason | reuse recorded sessions and `compare` |
+
+## 9. One-page quick-reference decision tree
+
+| Symptom (first sign) | First command | Decision | Scenario |
+|---|---|---|---|
+| New image/version appears | `./greeneye.sh symbols export ...` + `probe --record` | hash+pair first, then version diff | S1 |
+| OpenRM build breaks, `Unknown symbol` | `./scripts/build_nvidia_openrm.sh`; `dmesg` | glue-only vs interface change | S2 |
+| Control/ioctl inventory shifts | `python3 harness/control_discovery.py` | moved vs new logic | S3 |
+| `libcuda` bump / context failure | `tools/cudatool.py delta` + `fdematch` | patch / feature / reshuffle | S4 |
+| New device/architecture | `file`/`readelf`, `probe` | profile vs logic; physical validation for support | S5 |
+| Illegal/unknown instruction | `./greeneye.sh probe --instructions N` | ISA/decoder; fidelity gap if native-only | S6 |
+| Identity/ROM fields differ | guest `lspci -vvv`, snapshot diff | consumed field → profile; else record | S7 |
+| Protected/encrypted state blocks | device map + lenient `mmio_log` | record fidelity gap; no extraction; no claims | S8 |
+| Early stall on PMU/SEC2/ACR/SOE | `probe --record` + device map | stub the missing exchange; gate security semantics | S9 |
+| Matrix/CI drift; 610 mentioned | `ci_release --dry-run` | dry-run clean; 610 hard gate | S10 |
+
+Golden rule: **freeze, hash, pair, reproduce, classify, evidence, then act** —
+and never let the last step touch anything we do not own.
+
+## 10. Related documentation
+
+Green-eye suite (experimental, in development):
+
+* [green-eye user guide](05-operations/04-green-eye-user-guide.md) — what it
+ is, quickstart, tool chooser and troubleshooting;
+* [GSP in a RISC-V VM](05-operations/03-risc-v-gsp-rm-vm.md) — project-level
+ purpose, status and limits;
+* [`debug/risc-v-gsp-rm-vm/docs/00-OVERVIEW.md`](../debug/risc-v-gsp-rm-vm/docs/00-OVERVIEW.md)
+ — tool status and doc map;
+* [`debug/risc-v-gsp-rm-vm/GREEN-EYE-ROADMAP.md`](../debug/risc-v-gsp-rm-vm/GREEN-EYE-ROADMAP.md)
+ — H1–H7 program;
+* [`debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md`](../debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md)
+ — fidelity levels and the M1–M6 contract;
+* [`debug/risc-v-gsp-rm-vm/docs/07-PORTABILITY-PLAYBOOKS.md`](../debug/risc-v-gsp-rm-vm/docs/07-PORTABILITY-PLAYBOOKS.md)
+ — P1–P4 porting playbooks;
+* [`debug/risc-v-gsp-rm-vm/docs/14-ADVERSARIAL-INTEROP-GUIDE.md`](../debug/risc-v-gsp-rm-vm/docs/14-ADVERSARIAL-INTEROP-GUIDE.md)
+ — adversarial navigation and the evidence pack;
+* [`debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md`](../debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md)
+ — release triage and bundles;
+* [`debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md`](../debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md)
+ — the 610 gate and approval format;
+* [`debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md`](../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md)
+ — tool audit, symptom-to-fix table and provenance rule;
+* [`debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md`](../debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md)
+ — sweep/device-port scaffolds (unadmitted pilots, dry-run default);
+* [`debug/risc-v-gsp-rm-vm/docs/29-PACKAGING-AND-TRIALING.md`](../debug/risc-v-gsp-rm-vm/docs/29-PACKAGING-AND-TRIALING.md)
+ — trial simulation packages (unadmitted pilot).
+
+Project documentation:
+
+* [Debugging and maintenance workflows](05-operations/02-debugging-and-maintenance-workflows.md)
+ — captures, failure taxonomy, version discipline;
+* [Performance, correctness, security](05-operations/01-performance-correctness-security.md)
+ — the evaluation ladder;
+* [Hardware and software matrix](06-portability/hardware-software-matrix.md)
+ — support definitions and the portability pipeline;
+* [Practical challenges and adversarial interoperability](04-implementation/02-practical-challenges-and-adversarial-interop.md)
+ — the no-patch rule;
+* [Developer onboarding and standards](07-developers/onboarding-and-standards.md)
+ — coding rules and the landing checklist;
+* [Documentation front door](README.md) — the full reading order;
+* [Engineering operations](eng-ops.md) — tool admission, workflows and
+ retirement (the cross-cutting engineering-operations document);
+* CUDA playbook:
+ [`userspace/cuda/docs/07-version-upgrade-playbook.md`](../userspace/cuda/docs/07-version-upgrade-playbook.md)
+ and [`userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md).
+
+<!-- change (2026-10-02): fix the S4 fdematch invocation (two libraries plus
+--names, per tools/cudatool.py), add a session/disk-growth bound to §3.2, and
+link docs 28/29 from §10; scope: RUNBOOK §3.2, §5 S4, §10 only; benefit: the
+copied triage command runs as written and the retention rule is explicit. -->
diff --git a/docs/eng-ops.md b/docs/eng-ops.md
new file mode 100644
index 0000000..117bd0c
--- /dev/null
+++ b/docs/eng-ops.md
@@ -0,0 +1,850 @@
+# Engineering Operations: tooling admission, workflows and retirement
+
+> How Open-IOV decides which engineering tools it uses, extends and retires,
+> and how it measures whether a tool pays for its maintenance. Maintainers
+> decide and record; agents execute bounded loops and produce evidence.
+> Subsystem manuals are linked, not duplicated.
+
+## 1. Purpose and scope
+
+**Who uses it.** Maintainers decide, record and review; agents (MCP, CI, cron)
+execute bounded loops and produce sessions, reports and bundles. The same rule
+binds both: evidence or it did not happen ([`onboarding-and-standards.md`](07-developers/onboarding-and-standards.md) §5, [`15-AGENTIC-GSP-IDE.md`](../debug/risc-v-gsp-rm-vm/docs/15-AGENTIC-GSP-IDE.md) §5).
+
+| Governed here | Owned elsewhere (authoritative) |
+|---|---|
+| Admission, extension, redesign, trialing and retirement of tools | Architecture ([`03-architecture/01-system-overview.md`](03-architecture/01-system-overview.md)); tool manuals: CUDA ([`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md)), green-eye ([`26-ARCHITECTURE.md`](../debug/risc-v-gsp-rm-vm/docs/26-ARCHITECTURE.md), [user guide](../debug/risc-v-gsp-rm-vm/docs/02-USAGE.md), [scripting](../debug/risc-v-gsp-rm-vm/docs/03-SCRIPTING.md)), rmharness ([`README`](../debug/risc-v-gsp-rm-vm/rmharness/README.md)) |
+| Which tool applies in which loop; automation and packaging | Accuracy M1–M6 ([`06-ACCURACY.md`](../debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md)); performance method ([`05-operations/01`](05-operations/01-performance-correctness-security.md) §3); version discipline ([`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §4) |
+| Evidence bundles, version-span gates, frontier admission | Per-tool docs and the periodic [`27-TOOL-AUDIT.md`](../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md) |
+| Live physical-backend engineering operations: supervised-cycle mechanics, parameter gates, debugfs controls and the hazard register (**Part II** below) | Physical-backend architecture ([`03-architecture/08`](03-architecture/08-physical-gpu-integration.md), [`03-architecture/11`](03-architecture/11-physical-backend-guest-wiring.md)); defensive/incident scenarios ([`RUNBOOK.md`](RUNBOOK.md)); current frontier ([`STATE-CURRENT.md`](STATE-CURRENT.md)) |
+
+**Explicitly out of scope.** (1) Security claims: the simulator and its device
+maps are study environments, never a confidentiality or isolation boundary
+([`06-ACCURACY.md`](../debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md) §6,
+[`GREEN-EYE-ROADMAP.md`](../debug/risc-v-gsp-rm-vm/GREEN-EYE-ROADMAP.md) V4).
+(2) Timing claims: L2 is not modelled; order-of-events only
+([`06-ACCURACY.md`](../debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md) §2).
+(3) Shipping injected code: development-only, never a firmware patch, never in
+a guest ([`12-INJECTION-SDK.md`](../debug/risc-v-gsp-rm-vm/docs/12-INJECTION-SDK.md) §1).
+(4) Support and licensing: tested-and-documented only
+([`hardware-software-matrix.md`](06-portability/hardware-software-matrix.md) §1);
+legal rules in [`legal-and-licensing.md`](09-reference/legal-and-licensing.md).
+
+## 2. The core rule: no insertion without a measurable outcome
+
+A tool, or an insertion of an existing tool into a workflow, is admitted only
+with all five fields below. Missing any field rejects the proposal; it is not
+deferred until the metric is invented later.
+
+| Field | Meaning | Example |
+|---|---|---|
+| **Scenario** | exact question, target, version and loop where the tool is inserted | "classify one new GSP image's dispatch-table delta before the emulation is touched (portability loop)" |
+| **Measurable outcome** | result with a unit and an observer | "image drop → `version-diff.md` delta table; sessions per finding" |
+| **Baseline** | the replaced method and its measured/recorded cost | "static corpus diff + one supervised boot per hypothesis; baseline in [`docs/25`](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md) §3" |
+| **Evaluation threshold** | the number that decides keep/extend/retire, with sample size and environment | "N ≥ 3; median TTFA ≤ 25 % of baseline; self-diff zero; M-table improves or stays equal" |
+| **Failure signal** | observable "not useful" outcome during the pilot | "unexplained `compare` deltas after two triage passes, or the tool needs hardware to run" |
+
+Rules: one scenario per record; the threshold is recorded **before** the first
+run and moving it needs a new record; every admission starts as `admit-pilot`
+with an expiry of one quarter or five sessions; the baseline must be observable
+(measure it first if unrecorded); a missed threshold triggers §4.4/§4.5.
+
+```text
+TOOL ADMISSION RECORD id: TOOL-<yyyy>-<nn>
+tool: <name or path; existing | extension | new>
+loop: static-RE | dynamic-RE | porting | performance | correctness | CI | guest-compat
+owner/decider: <maintainer>; executor: <human|agent|cron>
+scenario: <one sentence: question, target, version, trigger>
+measurable: <metric + unit + observer>
+baseline: <replaced method; cost source and date>
+threshold: keep if <...>; reject if <...>; N=<runs>; environment=<...>
+failure signal: <observable not-useful outcome>
+evidence: <session dir / report / manifest + sha256>
+decision: admit-pilot | admit | extend | redesign | retire | do-not-build
+review-by: <date or N sessions> trials-dir: <path>
+```
+
+The record is stored with the evidence (session directory, bundle or PR
+description); decisions are recorded by a human maintainer and mirrored in the
+register (§4.5) and revision notes (§9.3). At expiry: met → admit; not met →
+redesign or retire; no signal → retire. No unused pilot survives.
+
+## 3. Value axes: what tooling is for, and how it is measured
+
+Each admission names exactly one axis. Every axis reduces to moving evidence
+through a §3.3 loop; if it cannot be expressed as a time or a count, it is not
+admissible.
+
+| # | Axis | Concrete example (real tooling) | Metric (recorded) | Loop |
+|---|---|---|---|---|
+| 1 | Faster reverse engineering | `version_diff.py` + `compare` on two GSP images; Ghidra corpus diff | firmware drop → classified delta (TTFA); symbols resolved vs Ghidra-only | RE |
+| 2 | Deeper / more correct analysis | watchpoints + TCG/native trace; capability inference ([`docs/14`](../debug/risc-v-gsp-rm-vm/docs/14-ADVERSARIAL-INTEROP-GUIDE.md) §4.2) | attributed bytes/stores; M6 agreement; unexplained-delta count | RE |
+| 3 | Code-artifact porting time | `cudatool.py fdematch`, `diff_corpus.py`, `symbols export` | transfer rate per class; wall time old→new symbol DB; wrapper re-resolution errors | portability |
+| 4 | Bug-analysis time | `reach.py`, `run_until`/`call`, `ioctl_trace.c`; taxonomy in [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §5 | crash/question → first evidence-backed answer (TTFA); supervised cycles avoided | adversarial |
+| 5 | Development velocity on the simulated GSP | injection SDK; `greeneye.sh inject`; value demo s06 | seconds from service edit to direct+mailbox agreement; sessions per prototype | RE |
+| 6 | Correctness validation of virtualization | M1–M6 table, `compare`, golden sessions | M-table per change; rung ladder ([`05-operations/01`](05-operations/01-performance-correctness-security.md) §1) | correctness |
+| 7 | Performance analysis and improvement | A/B method, `tools/benchmark_infrastructure.sh`, simulator overhead ([`docs/25`](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md) §6) | B/A % per metric; p50/p99 RPC; trace overhead; before/after same machine | performance |
+| 8 | Userspace/API compatibility friction | capability negotiation ([`../userspace/cuda/docs/04-virtualization-and-emulation.md`](../userspace/cuda/docs/04-virtualization-and-emulation.md) §1), symbol DB, ioctl diffs | controls with coherent replies vs observed; TTFA for a new capability bit; peer scenarios (RDMA guest, DPU/NIC/NVMe page sharing) counted only once a peer model exists | guest-compat |
+| 9 | Hidden / undocumented / unauthorized code, side effects, backdoors | `control_discovery.py`, mutation harness, decoy handling ([`docs/14`](../debug/risc-v-gsp-rm-vm/docs/14-ADVERSARIAL-INTEROP-GUIDE.md) §4.4) | unknown IDs and injected-slot aliases found; capability-model deltas; corpus entries with sessions | adversarial |
+| 10 | Architectural understanding and SBOM/doc improvement | symbol DB, `export_hits`, firmware notes ([`docs/09`](../debug/risc-v-gsp-rm-vm/docs/09-GSP-FIRMWARE-NOTES.md)), [`SBOM.md`](../SBOM.md) | doc/SBOM rows updated per finding; onboarding time; subsystem coverage | RE |
+| 11 | Security fixes / hardening time | capability inference → fail-closed rules ([`04-implementation/02`](04-implementation/02-practical-challenges-and-adversarial-interop.md) §4), mutation corpus | error paths covered; finding → test time; physical confirmation before product change | adversarial |
+| 12 | New GSP support time | `probe` + `version_diff` + `portability_matrix` + `ci_release` bundle | hours from image to triage table; M-table completeness; regression lead time | portability |
+| 13 | New hardware-device support time | device profile capture, `rmharness`, matrix, [`docs/22`](../debug/risc-v-gsp-rm-vm/docs/22-PORTING-CUDA-AND-DEVICES.md) §3, [device sequence](06-portability/03-device-validation-and-porting-sequence.md) | profile fields captured; engine inventory validated; first-touch deltas explained; tested-at-rung date | portability |
+| 14 | CUDA / runtime / VM-environment fixes | `userspace/cuda/docs/07`, `cuda_drv_test.c`, `tools/cuda-ab/ab.sh`, guest diagnostics | crash → named function time; `RESULT`/`STATUS` match; rung movement | guest-compat |
+
+Axis 8 note: GPU-direct RDMA availability is negotiated via the stock guest's
+vGPU capability block; page-table sharing with simulated DPU/NIC/NVMe peers is
+future, so its metric starts only when a peer model exists (§8).
+
+**Performance axis (axis 7).** Profiling order is the project's existing one
+([`05-operations/01`](05-operations/01-performance-correctness-security.md) §3.6):
+RPC dispatch → doorbell cadence → eventfd/MSI-X delivery → BAR0 shadows →
+memory-model gap. Method: (1) define the metric and A/B pair before code
+changes — same binary, inputs, governor, clocks and boot where possible, and
+report B/A %, never absolutes; (2) run A via `tools/benchmark_infrastructure.sh`
+and B via `scripts/vgpu_vm.py exec`, logging versions and
+`nvidia-smi -q -d CLOCK`; (3) attribute with host counters
+(`gsp_emu_get_stats()`, `trace_printk`, `/sys/kernel/debug/tracing`) and the
+measured simulator overhead ([`docs/16`](../debug/risc-v-gsp-rm-vm/docs/16-CUSTOM-SIMULATOR.md) §1.1a, [`docs/25`](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md) §6);
+(4) keep the numbers with the change and re-run correctness; (5) reject single
+runs, cross-host comparisons, bare-metal-only data and simulator timing claims.
+
+**OODA reduction.** Every axis moves evidence through one of three loops. Each
+loop's compression is the project's program-level metric
+([`GREEN-EYE-ROADMAP.md`](../debug/risc-v-gsp-rm-vm/GREEN-EYE-ROADMAP.md) success
+criteria), measured in [`docs/25`](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md) §7:
+
+| Loop | Observe → Orient → Decide → Act | Compression metric |
+|---|---|---|
+| RE-constrained (static/dynamic interlock) | probe/hooks/watch/trace → `analyze`/snapshots → `compare`/triage → reimplementation | question → reviewable answer; sessions per finding |
+| Adversarial-interop (opaque/decoy firmware) | discovery/mutation/replay → classification ([`docs/14`](../debug/risc-v-gsp-rm-vm/docs/14-ADVERSARIAL-INTEROP-GUIDE.md) §4.1) → fail-closed or no change → emulation change + physical confirm | unknown-ID → classified route; error paths covered |
+| Portability (GSP/device/architecture churn) | probe first-touch/symbol diff → pairing check/delta class → anchor decision → versioned dirs + handlers | image drop → triage table (hours) |
+
+## 4. Use, extend, build, redesign, retire
+
+```mermaid
+flowchart TD
+ P["Friction + scenario + measurable outcome"] --> Q1{"Existing tool covers the scenario?"}
+ Q1 -->|"yes"| USE["Admit the insertion; pilot + metric record"]
+ Q1 -->|"no"| Q2{"A close tool has a public extension point?"}
+ Q2 -->|"yes"| EXT["Extend in place: plugin, harness Surface, probe spec, SDK"]
+ Q2 -->|"no"| Q3{"Repeated scenario and no cheaper alternative?"}
+ Q3 -->|"no"| NB["Do not build: record rejection and the trigger to revisit"]
+ Q3 -->|"yes"| NEW["Admit a new tool as a bounded pilot"]
+ USE & EXT & NEW --> REV{"Threshold met after the pilot?"}
+ REV -->|"yes"| KEEP["Admit; add a second metric over time"]
+ REV -->|"no"| Q4{"Failure signal points at the design, not the idea?"}
+ Q4 -->|"yes"| RD["Redesign with a migration note; re-pilot"]
+ Q4 -->|"no"| PRUNE["Retire / prune; keep evidence read-only"]
+```
+
+| Decision | Minimum evidence | Decider | Record |
+|---|---|---|---|
+| (a) Use existing | scenario + measured baseline + one recorded trial session | maintainer; agent may run the trial | admission record + register row; link the session |
+| (b) Extend existing | ≥ 3 uses or ≥ 2 users; public extension point (plugin, `Surface`, probe spec, SDK); cost ≤ new tool | maintainer | record names the extension point and keeps the old tests green |
+| (c) Build new | record complete; bounded pilot; golden/replay evidence; stdlib/local-only unless justified | maintainer | record + register + revision note; tests added to the tool suite |
+| (d) Redesign | failure signal twice, or a fidelity gap that changes conclusions; migration note (`greeneye-session/1` evolves additively) | maintainer | before/after metric; old artifacts stay readable |
+| (e) Stop / prune | threshold missed, no evidence in the window, duplicated capability, or maintenance > value | maintainer at the audit | retire row; `27-TOOL-AUDIT.md` status; keep sessions/reports |
+| (f) Do not build | any trigger below is true | maintainer | rejection note naming the trigger that would reopen it |
+
+**When NOT to build (workflow risk).** Reject if any holds; the absence of a
+time-to-value process is the risk this document removes: (1) no scenario, unit
+or baseline — the pilot cannot fail; (2) the tool would become a shipped
+dependency, guest component or firmware patch
+([`04-implementation/02`](04-implementation/02-practical-challenges-and-adversarial-interop.md) §5.1);
+(3) it needs physical GPU, mdev or guest access for its primary value
+([`docs/15`](../debug/risc-v-gsp-rm-vm/docs/15-AGENTIC-GSP-IDE.md) §5);
+(4) it bypasses a security boundary or turns a study observation into a
+security claim; (5) it duplicates a tool without a measured improvement;
+(6) total cost is not plausibly below the replaced OODA cost; (7) its only
+evidence is prose — evidence must be a session, report or bundle.
+
+**Redesign and retirement.** Redesign rather than patch when the failure
+signal repeats, a fidelity gap invalidates an accepted conclusion, a public
+interface must change incompatibly, or the tool drifted outside its admitted
+loop. Retire when no artifact references the tool, the threshold was missed
+twice, another tool covers the scenario better, or maintenance exceeds the
+saving. Retirement is revocation of admission, not deletion: keep tool, tests
+and artifacts read-only, mark the register row `retired` with date and reason,
+and reconcile at the next [`27-TOOL-AUDIT.md`](../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md).
+
+**Register (how decisions are recorded).** The durable record is the register
+row, the evidence directory/bundle it names and the revision note in §9.3;
+the audit is the periodic reconciliation.
+
+| Admitted family | Loop | Value evidence (current) | Review / retire trigger |
+|---|---|---|---|
+| Ghidra corpora + scripts (`tools/ghidra`, `userspace/cuda/tools`) | RE, portability | docs 05/07; `diff_corpus`, name map | corpus diff no longer changes port class |
+| green-eye core + session format | dynamic-RE | `docs/25` s01–s10; 155-test suite | format unused by any loop |
+| Harness (`version_diff`, matrix, mutation, discovery, reach, `ci_release`) | portability, adversarial | docs 18/19/21; deterministic replay tests | a report is never attached to a change |
+| Injection SDK + demo uproc | dynamic-RE | docs 11/12; s06 | prototypes stop using it |
+| Ghidra bridges (in/out) | RE | docs 05/13 | static/dynamic join replaced |
+| Device map (`greeneye-device-map/1`) | portability | docs 24; H2.1 first-touch | H2.2 model stops consuming maps |
+| `rmharness` + `rmtool`/`rmtop`/`rmq` | correctness, guest-compat | docs 17; M1/M6 reference captures | reference capture unused/unavailable |
+| MCP server + agent workflows | all loops | docs 15/20; release monitoring | write gating or audit trail removed |
+| CI (`ci.yml`, `greeneye.yml`) | correctness | release-blocking tests | suite non-reproducible or skipped |
+
+## 5. Workflow map: where each tool applies
+
+### 5.1 Context of the development and reverse-engineering environment
+
+```mermaid
+flowchart LR
+ subgraph STATIC["Static"]
+ GH["Ghidra CUDA corpus + GSP firmware"]
+ end
+ subgraph DYN["Dynamic (green-eye, experimental)"]
+ GE["Session + QEMU/native backends"]
+ HR["harness/*.py"]
+ end
+ subgraph HOST["Host and guest"]
+ MOD["open-iov-vfio.ko"]
+ VM["stock guest + guest-diag"]
+ RMH["rmtool/rmtop/rmq + rmharness"]
+ end
+ subgraph AUTO["Automation"]
+ CI["ci.yml / greeneye.yml"]
+ MCP["MCP server + agents/cron"]
+ end
+ EVID["Evidence: sessions, reports, bundles"]
+ GH -->|"names, index, symbol DB"| GE
+ GE -->|"export_hits, fault PCs"| GH
+ GE --> HR --> EVID
+ GE --> EVID
+ RMH --> EVID
+ VM -->|"stimulus and failures"| GE
+ MOD --> VM
+ EVID --> AUTO
+ AUTO -->|"bounded runs"| HR
+ EVID -->|"product-change gate"| MOD
+```
+
+### 5.2 Static reverse-engineering loop
+
+```mermaid
+flowchart LR
+ BIN["libcuda / .fwimage"] --> IMP["Ghidra import + analysis"]
+ IMP --> IDX["names.tsv, decompiled_index.tsv, callgraph.tsv"]
+ IDX --> DIFF["corpus diff / fdematch / RPC dispatch diff"]
+ DIFF --> HYP["Hypothesis: functions, tables, layouts"]
+ HYP --> SYM["greeneye symbols export (bias recorded)"]
+ SYM --> DYN["dynamic loop"]
+ DYN -->|"hits, watch PCs, fault PCs"| HYP
+```
+
+### 5.3 Dynamic firmware loop (OODA)
+
+```mermaid
+flowchart LR
+ Q["Question + target + fidelity statement"] --> OBS["Observe: probe, hooks, watch, trace"]
+ OBS --> ORI["Orient: analyze, symbol decode, snapshots"]
+ ORI --> DEC["Decide: compare, M1-M6, mutation"]
+ DEC --> ACT["Act: emulation change with evidence"]
+ ACT --> REG["Regression: same stimulus, same budgets"]
+ REG -->|"improved or equal"| PACK["Evidence pack + session hashes"]
+ REG -->|"regressed"| Q
+ PACK --> Q
+```
+
+### 5.4 Porting loop (GSP firmware, device, architecture)
+
+```mermaid
+flowchart TB
+ ACQ["Acquire + hash; versioned dirs, never overwrite"] --> TRI["Triage: delta, fdematch, pairing check"]
+ TRI --> PRB["Probe first-touch old vs new"]
+ PRB --> RUN["Rehost/demo: identical stimulus + record"]
+ RUN --> CMP["compare + M1-M6 skeleton"]
+ CMP --> BUN["version_diff -> matrix -> mutation -> bundle"]
+ BUN --> REV{"Every delta explained?"}
+ REV -->|"no"| TRI
+ REV -->|"yes"| ANCHOR["Anchor decision + docs + matrix entry"]
+```
+
+### 5.5 Performance loop
+
+```mermaid
+flowchart LR
+ DEF["Define metric + A/B pair"] --> BASE["A: bare metal baseline"]
+ DEF --> CAND["B: vGPU / candidate"]
+ BASE --> RATIO["B/A ratio, variance, N runs"]
+ CAND --> RATIO
+ RATIO --> PROF["Attribute: RPC, doorbell, eventfd, shadows"]
+ PROF --> KEEP["Keep numbers with the change"]
+ KEEP --> REG["Re-run correctness corpus + ladder"]
+```
+
+### 5.6 Correctness and CI loop
+
+```mermaid
+flowchart LR
+ CH["Change"] --> HOST["Host suite: stack_manager.sh test"]
+ HOST --> GUEST["Guest ladder: probe, nvidia-smi, cdt"]
+ GUEST --> AB["cuda-ab: RESULT/STATUS must match"]
+ AB --> SESS["Session compare / golden data"]
+ SESS --> GATE{"Rung or metric regressed?"}
+ GATE -->|"no"| LAND["Land with evidence"]
+ GATE -->|"yes"| CH
+```
+
+### 5.7 Guest-compatibility loop
+
+```mermaid
+flowchart LR
+ GD["guest-diag: ioctl trace, engines, pmap"] --> CTRL["Control/ioctl inventory"]
+ CTRL --> EMU["Emulated reply fidelity"]
+ EMU --> GD2["New driver/CUDA version trace"]
+ GD2 --> DIFF["Control diff old vs new"]
+ DIFF --> CTRL
+ CTRL --> CORPUS["CUDA corpus + symbol DB"]
+```
+
+### 5.8 Loop index and detailed docs
+
+| Loop | Entry points | Evidence | Gate | Detail |
+|---|---|---|---|---|
+| Static RE | `tools/ghidra/*`, `userspace/cuda/tools/*`, `analyzeHeadless` | corpus TSVs, name map, symbol DB | addresses re-derived per build | [`userspace/cuda/docs/07`](../userspace/cuda/docs/07-version-upgrade-playbook.md), green-eye docs 05/13 |
+| Dynamic RE | `greeneye.sh` probe/record/analyze/compare/inject/ghidra, Python `Session` | `greeneye-session/1` + `summary.md` | A/B pair; L2/L3 statement | [`26-ARCHITECTURE.md`](../debug/risc-v-gsp-rm-vm/docs/26-ARCHITECTURE.md) §11, [`11`](../debug/risc-v-gsp-rm-vm/docs/11-GSP-RM-SERVICE-DEVELOPER-GUIDE.md), [`12`](../debug/risc-v-gsp-rm-vm/docs/12-INJECTION-SDK.md), [`13`](../debug/risc-v-gsp-rm-vm/docs/13-GHIDRA-CONTROL.md), [`14`](../debug/risc-v-gsp-rm-vm/docs/14-ADVERSARIAL-INTEROP-GUIDE.md) §4.6, [`06`](../debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md) §5 |
+| Porting | `version_diff.py`, `portability_matrix.py`, `mutation_harness.py`, `ci_release.py` | reports + `bundle-manifest.json` | every delta explained | [`07`](../debug/risc-v-gsp-rm-vm/docs/07-PORTABILITY-PLAYBOOKS.md), [`22`](../debug/risc-v-gsp-rm-vm/docs/22-PORTING-CUDA-AND-DEVICES.md), [`21`](../debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md), runbook [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §4.1 |
+| Performance | `benchmark_infrastructure.sh`, module counters, `vgpu_vm.py` | B/A table with environment | same-machine A/B; corpus green | [`05-operations/01`](05-operations/01-performance-correctness-security.md) §3 |
+| Correctness | `stack_manager.sh test`, `make tests`, `cuda_drv_test.c`, `cuda-ab` | rung + trace excerpts | no rung regression | [`05-operations/01`](05-operations/01-performance-correctness-security.md) §1–2 |
+| CI | `.github/workflows/ci.yml`, `.github/workflows/greeneye.yml` | workflow logs | push/PR path triggers | [`21`](../debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md) §5 |
+| Guest-compat | `tools/guest-diag/*`, `cudatool.py`, `diff_corpus.py` | ioctl/control diff, corpus delta | stock guest only | [`READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md) |
+
+## 6. Automation and agent invocation
+
+```mermaid
+flowchart LR
+ PRE["Preflight: greeneye.sh info; RM610 gate"] --> DRY["ci_release.py --dry-run (plan only)"]
+ DRY --> REAL["ci_release.py real run"]
+ REAL --> VD["version_diff"]
+ REAL --> PM["portability_matrix"]
+ REAL --> MU["mutation smoke"]
+ VD & PM & MU --> MAN["release-report.json + bundle-manifest.json"]
+ MAN --> REV["agent/human review"]
+ REV -->|"accepted"| REG["anchor/register/docs updated"]
+ REV -->|"rejected"| GAP["fidelity gap or fix, recorded"]
+```
+
+Run everything from `debug/risc-v-gsp-rm-vm/`; scripts are stdlib-only and
+launch their own QEMU. Never re-use an existing gdbstub, a guest, a kernel
+module or hardware.
+
+**Scripts that exist** (success and failure are read, not assumed):
+
+| Script | Safe invocation | Success | Failure signal / limit |
+|---|---|---|---|
+| `harness/version_diff.py` | `python3 harness/version_diff.py --old <old> --new <new> --out dumps/vdiff` | sections `ok`; self-diff zero; deltas explained | exit 2 tool error, 3 RM610 refusal; raw `.fwimage` faults early by design |
+| `harness/portability_matrix.py` | `python3 harness/portability_matrix.py --target <a> --target <b> --out dumps/matrix` | self-comparison `all_zero`; M1–M6 `not measured` without reference | non-zero target deltas or per-target `status: error` |
+| `harness/mutation_harness.py` | `python3 harness/mutation_harness.py --out dumps/mut --iterations 64 --seed 7` | deterministic `findings.jsonl`; declared outcome classes | seed divergence; non-demo ELF refused without `--allow-foreign-elf` |
+| `harness/control_discovery.py` | `python3 harness/control_discovery.py --surface demo --out dumps/disc --seed 7` | known IDs resolve; unknown/injected classes listed | RM surface exits cleanly until H1.3; misclassification or unreplayable probe |
+| `harness/reach.py` | `python3 harness/reach.py --target rpc_ping --method call --args 0x41 --out dumps/reach` | `reachable` with recorded route and args | timeout means "not under this stimulus", not unreachable |
+| `harness/ci_release.py` | dry-run first: `python3 harness/ci_release.py --old <old> --new <new> --out dumps/release --dry-run` | dry-run touches no target; real run writes the bundle | exit 1 step error, 2 tool error, 3 RM610 refusal; `skipped` is not `pass` |
+| `greeneye.sh` | `./greeneye.sh info`/`probe`/`record`/`analyze`/`compare`/`inject` | session dir with `session.json` + `summary.md` | empty sessions or missed hooks; budgets required |
+| `rmharness/harness.sh` | `./rmharness/harness.sh --session dumps/real -- rmtool engine list --format=json` | `rm_harness_run` event + stream blobs | read-only only; permission errors reported, never escalated |
+| `plugins/mcp_server.py` | `python3 plugins/mcp_server.py --stdio --record` | 24-tool JSON-RPC surface; read/observe default | stdout protocol-only; writes need `--allow-write`/`--allow-inject` |
+| `harness/value_demos/run_all.py` | `python3 harness/value_demos/run_all.py` | one `RESULT_JSON` per demo; suite ≤ 180 s | any demo `success: false`; re-run before quoting numbers |
+
+**Scaffolded scripts (landed, not admitted yet).** The three scripts below
+landed with docs 28/29 and are pilots until they have a §2 record; the first
+two are dry-run by default (`--run` executes). `--help` is the authoritative
+flag list; the outputs and failure readings are in
+[`docs/28`](../debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md) and
+[`docs/29`](../debug/risc-v-gsp-rm-vm/docs/29-PACKAGING-AND-TRIALING.md).
+
+| Script | Purpose | Interface | Outputs | Exit codes |
+|---|---|---|---|---|
+| `harness/port_driver_versions.py` | sweep a version list through `version_diff` + matrix + mutation, one evidence set per version | `--versions <list>` or `--config <json>`, `--baseline <ver>`, `--input-dir <dir>`, `--target <ver>=<path>`, `--symbols <ver>=<path>`, `--ghidra-exports <ver>=<path>`, `--out <dir>`, `--run` (default: dry-run), `--force`, `--extract-runfiles`, `--approval-token` | per-version `version-diff/` + `portability-matrix/` + `mutation-smoke/` + `bundle-manifest.json`, `sweep-summary.json|.md` | 0 complete, 1 step/threshold failure, 2 tool error, 3 RM610 refusal |
+| `harness/port_new_device.py` | per-device port: profile, pairing check, first-touch probe, symbols, matrix, checklist | `--device <label>`, `--profile <json>`, `--firmware <path>`, `--reference <path>`, `--symbols <json>`, `--ghidra-exports <tsv>`, `--out <dir>`, `--run` (default: dry-run), `--approval-token` | `device-port.json|.md`, `port-checklist.json|.md`, `symbols/`, sessions, matrix, `bundle-manifest.json` | as above; 1 when pairing is ambiguous or a delta unexplained |
+| `harness/package_sim.py` | build (and optionally run) a trial simulation package (§7) | `--target demo\|image`, `--image <path>`, `--service <elf>`, `--device-map <json>`, `--script <py>`, `--slot N`, `--pack-image`, `--tar`, `--run`, `--out <dir>` | package tree + `manifest.json`; `results.json|.md` after `run.sh` | 0 built/run, 1 replay or assertion failure, 2 tool error, 3 gate refusal |
+
+All three: stdlib-only, simulator/local only, deterministic budgets recorded in
+the report, reusing existing harness functions, and running the RM610 label check first.
+Their admission evaluation is recorded in
+[`future-eng-ops.md`](future-eng-ops.md) F2–F4.
+
+**Version span and the RM 610 gate.** The automated span is **540 → current**
+(anchor 570.144, [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §4.1);
+inputs are versioned under `firmware/<SERIES>-GSP/<VER>/` and never overwritten.
+**RM 610 requires explicit human approval**: before approval nothing fetches,
+opens or analyses a 610 image
+([`23-RM610-READINESS.md`](../debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md)).
+`version_diff`/`ci_release` refuse any `rm610`-matching label with exit 3
+unless `--approval-token RM610-APPROVED` or `GREENEYE_RM610_APPROVED=1` is
+present; the check runs in `--dry-run` and before any target is opened. An
+agent may prepare the approval request and run current-anchor rehearsals; only
+a named human approves.
+
+**Evidence bundles.** `ci_release.py` writes `release-report.json`,
+`bundle-manifest.json` (sha256 + size of every file) and the step trees
+([`docs/21`](../debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md) §4.3).
+Exit 0 with all steps `ok` is complete, but read `skipped`; exit 1 means a step
+errored (reviewable, not evidence for that section); exit 2 no claim; exit 3 a
+gate refusal, not a bug. Verify hashes before sharing and attach the bundle.
+
+**MCP monitoring.** `plugins/mcp_server.py` exposes the live session
+(read/observe by default) and shells out to `probe_firmware`,
+`portability_matrix_run`, `mutation_run`, `analyze_current` and
+`compare_sessions` ([`docs/20`](../debug/risc-v-gsp-rm-vm/docs/20-MCP-SERVER.md) §5).
+Release pattern: matrix → pinned mutation smoke → compare the recorded pair →
+read the bundle (`version-diff.json` / `release-report.json` /
+`bundle-manifest.json`); the agent does not recompute it. Absent capabilities
+return `not_available`, so planning is safe.
+
+**Cron/agent safety.** Default to read/observe and `--dry-run`; write/inject
+flags are explicit and the recording is the audit trail. Physical GPU, mdev
+churn, module load/unload and guest access are never unattended. Budgets
+everywhere; a wedged target is left stopped. Product changes are proposals —
+only humans merge, after physical confirmation of behaviour changes. Local
+only: stdio MCP, loopback TCP at most; never expose it to a network.
+
+## 7. Packaging and trialing
+
+A trial package lets a user run one **pre-scripted whole-GSP simulation** with
+injected code without the lab. It is a developer artifact: local, simulator
+only, no hardware, guest or kernel module. It is not a product component and
+not a support claim.
+
+```text
+dist/<pkg>/
+ README.md scenario, expected result, budgets, L2/L3 caveat
+ run.sh deterministic runner (greeneye.sh run + settle)
+ scenario.py fixed stimulus, fixed budgets, records a session
+ firmware/<image> ELF or .fwimage, hash-pinned
+ service/<service>.elf injected service (PIC, ABI v1), hash-pinned
+ device-map/<map>.json greeneye-device-map/1 (bias, entries, fw sha256)
+ golden/session/ optional reference session for compare
+ package-manifest.json sha256 + bytes per member (bundle shape, docs/21)
+```
+
+Runner and scenario follow the injection contract: PIC blob, no libc, no
+absolute addresses, one dispatch entry per slot, every write recorded
+([`docs/12`](../debug/risc-v-gsp-rm-vm/docs/12-INJECTION-SDK.md) §2–5); the device map
+carries provenance so a lenient run is not mistaken for device-modelled behaviour
+([`docs/24`](../debug/risc-v-gsp-rm-vm/docs/24-DEVICE-MAP.md) §1).
+
+```text
+python3 harness/package_sim.py --out dist/<pkg> --service <service.elf> \
+ --device-map <map.json> --script <scenario.py> [--pack-image] [--tar] [--run]
+./dist/<pkg>/run.sh
+```
+
+Success: the packaged `run.sh` reports `status: ok` (demo target) or the
+documented blocked status (real image), and every `manifest.json` component
+hash matches. Failure: hash mismatch, nondeterministic replay, unresolved
+symbol or missing member. The scaffold has no `--verify` subcommand yet:
+verification is re-running `run.sh` and `sha256sum` against the manifest
+([`docs/29`](../debug/risc-v-gsp-rm-vm/docs/29-PACKAGING-AND-TRIALING.md) §2–4),
+and it stays a pilot until future-eng-ops F4's threshold is met.
+
+**Sharing and judging results.** Share the package (or its manifest) with the
+session it produces. The recipient judges manifest hashes, the session
+`summary.md`, the golden comparison and the fidelity statement (L0/L1 only,
+L2/L3 absent). Results are evidence about firmware in the simulator; a product
+change still requires the accuracy loop and physical confirmation.
+
+**Packing a modified GSP image.** When a GSP modification (injected service,
+shim or patched firmware) is considered ready to ship, `package_sim.py` may
+pack the modified image, the tested injected code, device map, scenario and
+golden session so the recipient can reproduce the exact run before any product
+decision. **Caveat, in every package README: this is a simulation package, not
+a signed production image.** It is not a firmware patch, must never be flashed
+or injected into a guest, and is not bootable on silicon. Injected code is
+development-only ([`docs/12`](../debug/risc-v-gsp-rm-vm/docs/12-INJECTION-SDK.md) §1);
+productization goes through `open-iov-vfio` + physical confirmation, never
+through a simulator package.
+
+Local-only, budgeted, session-recorded; never touches `/dev`, a module, an
+mdev, a guest or a GPU. No confidentiality, isolation or timing claim derives
+from it; distribution follows per-file licences ([`SBOM.md`](../SBOM.md),
+[`legal-and-licensing.md`](09-reference/legal-and-licensing.md)).
+
+## 8. Frontier capabilities
+
+Everything here is **future** and off the critical path. Each capability enters
+the §2 process only when its near-term need is real and its gate is passed;
+details are tracked in the companion [`future-eng-ops.md`](future-eng-ops.md)
+(same directory) and
+[`GREEN-EYE-ROADMAP.md`](../debug/risc-v-gsp-rm-vm/GREEN-EYE-ROADMAP.md) H5/H6.
+Deferred context: [`10-post-1.0/experimental-host-acceleration.md`](10-post-1.0/experimental-host-acceleration.md).
+
+| Frontier capability | Near-term need | Precondition / gate | Evaluation criterion (if funded) |
+|---|---|---|---|
+| SIMD cores and CUDA kernel execution on device | Execute a real kernel launch path (`vectorAdd`-class) in simulation before the physical backend is finished | H1/H2 stable; injection ABI usable for an engine model | kernel executes within an instruction budget; output matches the native/golden result byte-for-byte; deterministic replay |
+| FPGA soft-processor simulation | A cycle-accuracy question worth funding (timing/security/interoperability claim needing silicon-equivalent behaviour) | H3 validated (native parity) and H2 device model proven | the same `Session` scripts run against the RTL backend; execution parity with native; cycle counts recorded as a distinct event class, docs/06 fidelity revisited |
+| Accurate Verilog recreation of all components the GSP interacts with, plus the NVRISC environment and intra/inter-micro-controller interconnects | Boot the real image with silicon-equivalent register, interrupt and mailbox semantics; exercise PMU/SEC2/ACR cross-controller boot flows | H2/H5 proven, validated device map, stable `Backend` abstraction | firmware reaches its message loop in the soft environment; M1–M6 improve over the QEMU/native baseline; one cross-controller exchange reproduced and compared with a physical capture |
+
+## 9. Review discipline
+
+**Adversarial-review loop.** (1) The author proposes a change with its
+revision note (§9.3). (2) A second person or agent attacks it: broken links,
+unverifiable claims, thresholds without metrics, tools without a scenario,
+missing register rows, scope creep into the manuals. (3) The attacker
+reproduces one item end-to-end (link check plus one claimed command);
+"looks fine" is not a review. (4) The author resolves findings; the reviewer
+signs the note with the evidence used.
+
+**Proportionate review (no needless overhead).**
+
+| Change class | Examples | Required review |
+|---|---|---|
+| A — editorial | wording, formatting, link fixes | `python3 scripts/check-docs-links.py` from the repo root; no design review |
+| B — decision | admission, threshold, register, retire changes | one adversarial reviewer + evidence named in the note |
+| C — scope/security | out-of-scope changes, gates, packaging caveats | maintainer sign-off + updated admission records |
+
+Cadence: review at each tool audit
+([`27-TOOL-AUDIT.md`](../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md)) and
+after any firmware anchor change. Expired pilots must appear in the register;
+an empty register of decisions means this document is not being used.
+
+**Revision notes.** Every change records purpose, scope and benefit; a change
+that cannot state a benefit is rejected as overhead.
+
+| Date | Revision | Purpose / scope | Benefit | Evidence |
+|---|---|---|---|---|
+| 2026-10-02 | v1 | Establish the admission rule, value axes, decision framework, workflow maps, automation, packaging, frontier gates and review discipline | Tooling decisions become auditable, measurable and reversible; no tool without an evaluation | This document plus linked tool docs and the [`docs/25`](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md) baselines |
+| 2026-10-02 | v2 (Part II) | Add the live physical-backend operations: hard rules and scope, build/load order and the full `gsp_phys_*` parameter matrix, the routine supervised cycle with its clean-release checklist, the debugfs inventory and `gsp-phys-state` reference, the physical-backend scenario catalogue and hazard register, the OODA loop and the work-context/porting map | Physical-backend execution becomes auditable and fail-safe: one checklist, a named signal per failure, and no retry of a wedged path | Kernel sources cited per parameter and debugfs file; the current frontier in this document and [`STATE-CURRENT.md`](STATE-CURRENT.md) |
+
+```bash
+python3 scripts/check-docs-links.py # repo root
+cd debug/risc-v-gsp-rm-vm && make demo sdk && make test # tool claims
+./greeneye.sh info # environment
+```
+
+---
+
+# Part II — Live physical-backend operations
+
+> **Scope.** This part is the operating manual for cycles against the **live
+> physical backend**: the module-owned physical GPU running its real GSP-RM
+> session (the chosen execution mechanism in
+> [`03-architecture/08`](03-architecture/08-physical-gpu-integration.md), wired
+> toward the guest plane in
+> [`03-architecture/11`](03-architecture/11-physical-backend-guest-wiring.md)).
+> Part I (§§1–9) governs tooling admission, value measurement and retirement.
+> Defensive/incident scenarios are owned by [`RUNBOOK.md`](RUNBOOK.md) and are
+> cross-linked, not duplicated, here; the current frontier is
+> [`STATE-CURRENT.md`](STATE-CURRENT.md); the known unknowns are
+> [`04-implementation/01`](04-implementation/01-known-knowns-and-unknowns.md).
+> Everything in this part is supervised, bounded and fail-safe-first; nothing
+> here authorises unattended hardware work.
+
+## 10. Live physical-backend operations
+
+### 10.1 Hard rules and scope
+
+These apply to every physical cycle; where a step below conflicts with them,
+the rules win.
+
+| # | Rule | Consequence / practice |
+|---|---|---|
+| 1 | **Supervised only, one GPU cycle at a time.** Never unattended, never parallel with other host work, never as a background agent job. | A named human is present for the cycle; the pre-flight and clean-release checklists (§10.3) are completed for each single attempt. |
+| 2 | **Reboot is gated by the written approval policy.** A reboot request MUST first record, in the project's live working record, the reason and the exact post-reboot plan; only then is a named human asked to approve. | No reboot while other work is running; the device is left untouched in the meantime; an unapproved reboot is never initiated by an agent. |
+| 3 | **Never enable `gsp_phys_repower`; never re-arm D3cold.** The default 0 latches the pre-claim D0 policy until reboot. | Re-enabling runtime PM/D3cold after a claim can power-gate the board and hard-freeze the host ([`03-architecture/08`](03-architecture/08-physical-gpu-integration.md) §4.2). |
+| 4 | **Never retry a failed mutating RPC.** One attempt per staged step; abort the ladder from the first failure. | Retrying has wedged GSP-RM (channel-flag and context-promote incidents); `UNLOADING_GUEST_DRIVER` + `rmmod` and, if needed, an approved reboot remain the only recovery. |
+| 5 | **Stop on hard signatures.** `board unresponsive`, `refusing claim`, `do not retry in this boot`, any panic/oops. | Do not write the device again in that boot; keep the locks; request reboot approval per rule 2. |
+| 6 | **Verify module identity before and after a cycle.** Record the built module's sha256 (and `srcversion`) before load and re-read it after unload. | A hash mismatch invalidates the cycle's evidence; stop and rebuild from the reviewed source. |
+| 7 | **A wedged device is left untouched for the reboot.** No `rmmod` retry loops, no bind/unbind experiments, no power-policy pokes. | The fail-safe (no rebind, D0 locks + runtime reference held, WPR2 up) is the designed resting state. |
+| 8 | **Scope.** This part covers the claim/boot, rm-call ladder (host-side real submission), guest-memory forwarding, relay/fault-relay and release paths. Guest-facing emulation debugging is [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md); firmware runtime study is [green-eye](05-operations/03-risc-v-gsp-rm-vm.md). | All other physical work belongs to the scenario owner named in §14. |
+
+### 10.2 Build and load order, and the module-parameter matrix
+
+**Build order (physical-backend work needs the kernel module only):**
+
+1. Build `open-iov-vfio.ko` first; it generates the `Module.symvers` the
+ optional OpenRM modules link against. `scripts/stack_manager.sh
+ build-kernel`.
+2. Build the optional OpenRM `nvidia.ko` with
+ `KBUILD_EXTRA_SYMBOLS` only if that management path is in use; the normal
+ consent path runs the distribution driver for the physical GPU and the
+ emulation module independently (`scripts/stack_manager.sh build-nvidia`).
+3. Build userspace (`libiov`/examples) independently
+ (`scripts/stack_manager.sh build-userspace`).
+4. Stage the GSP boot image into the firmware search path
+ (`scripts/stack_manager.sh install-firmware`).
+
+**Load order:**
+
+1. Prerequisites: `vfio`, `vfio_iommu_type1`, `vfio_pci`, `mdev`
+ (`scripts/stack_manager.sh load-prereqs`).
+2. `open-iov-vfio.ko` with the exact reviewed parameter set
+ (`scripts/stack_manager.sh load-open-iov`). The backend claim is
+ **module-owned** at load with `gsp_phys_gpu=1 gsp_phys_hazard_ack=1`, or
+ **mdev-owned** and deferred to the first mdev create with
+ `gsp_phys_mdev_owned=1` (fail-open).
+3. Record the loaded module identity and the starting state (§10.3 step 0).
+
+**Module-parameter matrix.** Every current `gsp_phys_*` parameter is listed
+below, verified against the kernel sources on the 2026-10-02 tree. The exact
+firmware image name used by `gsp_phys_fw` is deliberately not repeated here
+(it is the distribution driver's Turing-generation artifact); the source is
+authoritative for the literal default.
+
+| Parameter | Default | Effect | Gate / dependency |
+|---|---|---|---|
+| `gsp_phys_gpu` | 0 | Claim the physical GPU at module load: bind `open-iov-gsp-phys`, map BARs, stage firmware and run the requested stages; module-owned claim is released only at `rmmod`. | Refuses without `gsp_phys_hazard_ack=1`; refuses if the GPU or its root port is already runtime-suspended. With `gsp_phys_mdev_owned=1` it only arms the deferred path. |
+| `gsp_phys_hazard_ack` | 0 | Acknowledges the unbind/rebind host-freeze hazard; second gate for every claim. | Required with `gsp_phys_gpu=1`; without it the claim logs `REFUSED` and returns `-EPERM`. |
+| `gsp_phys_boot` | 0 | Full real-GSP boot: FWSEC → Booter Load → RISC-V start → RM-init poll (device writes). | Requires a claim; implies the layout stage; mutually exclusive with `gsp_phys_boot_dryrun`; FWSEC-only is refused because that carve is not host-recoverable. |
+| `gsp_phys_boot_dryrun` | 0 | Stage-3d host-memory staging rehearsal (radix3, queues, boot args) with no BAR0 writes and no falcon resets. | Requires a claim; mutually exclusive with `gsp_phys_boot`. |
+| `gsp_phys_layout` | 0 | Allocate the boot-argument pages and compute/log the WPR/FB layout; the only device access is a read-only video-memory-size sample. | Requires a claim; implied by the boot and dry-run paths. |
+| `gsp_phys_nocat` | 0 | Log every NOCAT journal record (function `0x1020`) individually during the RM-init poll; the unique-record summary is always accumulated and dumped when the poll ends. | Boot path only; diagnostic. |
+| `gsp_phys_fw` | distribution GSP image | GSP ELF the backend stages; override with `<name>` relative to the firmware search path. | Requires a claim; a missing/invalid image leaves the backend at ownership rung 1. |
+| `gsp_phys_mdev_owned` | 0 | Deferred, refcounted lifecycle: first mdev create claims+boots asynchronously, last remove releases; every failure is fail-open to the emulation path. | Requires `gsp_phys_gpu=1` + `gsp_phys_hazard_ack=1`; a module-owned claim stays up until `rmmod`. |
+| `gsp_phys_fwsec` | 0 | Run the FWSEC/FRTS secure-verification carve (device writes). | Requires the layout to have succeeded; **refused unless `gsp_phys_boot=1`**. |
+| `gsp_phys_fwsec_dryrun` | 0 | Read-only validation of the VBIOS/BIT/FWSEC parse and the staging/placement math. | Requires a claim; no device writes. |
+| `gsp_phys_fwsec_verbose` | 0 | Extra bounded read-only FWSEC kick state dump (falcon regs, scratch, DMEM/IMEM readback) after a successful halt. | Diagnostic; failure path always dumps. |
+| `gsp_phys_static_live` | 0 | Overlay the live GPU name/short-name/VBIOS identity from the READY snapshot onto the emulated `GET_GSP_STATIC_INFO` reply (bounded 2 s wait, fail-open). | Requires a READY backend snapshot; other static fields stay emulated. |
+| `gsp_phys_ctrl_forward` | 0 | Forward a whitelist of side-effect-free `GSP_RM_CONTROL` queries (bus BAR info, bus info, FB info, GMMU static info) with the adopted server handles; every miss falls back to the emulated handler. | Requires READY; fail-open; per-cmd forwarded/fallback counters. |
+| `gsp_phys_work_forward` | 0 | Request one real CE shadow copy per eligible decoded guest push on the ladder channel; the software release path stays authoritative. | Requires a live claim + RM init-done + a built ladder channel; all CE forwarding paths refuse cleanly on a GR ladder. |
+| `gsp_phys_guest_mem` | 0 | Let the ladder's guest triggers (6/7) and the forwarding path operate on a pinned guest GPA resolved through the guest-RAM backing file. | Triggers require READY, ladder stage ≥ 3, an identity DMA domain, an active guest-RAM file and a pinned GPA (6/7). |
+| `gsp_phys_work_defer_release` | 0 | Defer a forwarded push's guest-visible release until the hardware copy completes (bounded 500 ms), then apply it from the service thread. | Requires `gsp_phys_work_forward=1` + `gsp_phys_guest_mem=1`; with 0 the release applies inline. |
+| `gsp_phys_work_hw_primary` | 0 | Skip the software copy for a forwarded guest CE push (hardware authors the data); one software fallback copy on any hardware/queue failure. | Requires `gsp_phys_work_forward=1` + `gsp_phys_guest_mem=1` + `gsp_phys_work_defer_release=1`. |
+| `gsp_phys_relay` | 0 | Let the engine author the selected guest-visible semaphore release into the guest page (third PTE on the same fresh line); the hold verifies the word and skips that record's software apply. | Requires `gsp_phys_work_forward=1` + `gsp_phys_guest_mem=1` + `gsp_phys_work_defer_release=1`; default off pending mismatch triage. |
+| `gsp_phys_fault_relay` | 0 | On a relay timeout, drain the status queue for the engine fault (`0x1006`, corroborated by `0x1020`), match runlistId/chid and write the guest channel notifier + event. | Requires `gsp_phys_relay=1` and the three parameters above. |
+| `gsp_phys_fault_inject` | 0 | Debug: write the relayed pair's source PTE as zero to force an engine fault on the next relay submission; only relayed pairs are affected. | Requires the relay path; default 0 programs the real source HPA. |
+| `gsp_phys_rmcall_engine` | 0 | Ladder engine: 0 = COPY0 (CE copy submission), 1 = GR0 (compute object, BIND engineType 1, GR semaphore release). Snapshotted when stage 3 starts; mid-session changes are ignored. | Requires boot + init-done; CE forwarding paths refuse on a GR ladder. |
+| `gsp_phys_rmcall_gr_promote` | 0 (clamp 0..3) | GR context-promote bisection: **0 never sends `PROMOTE_CTX`** (the ladder stops after the geometry queries, context buffer/PT allocation, PDB mapping and channel/compute alloc, before BIND); 1 = single MAIN entry, initialize/no-nonmapped, size raw+subctx; 2 = two-phase PA-only then VA-only; 3 = single entry with the exact raw size. Nonzero enables can assert GSP-RM (`0x52a8b04`) and wedge the device. | GR ladder only; value snapshotted at stage 3; **requires confirmed reboot readiness** before enabling. |
+| `gsp_phys_repower` | 0 (latched) | Restore the pre-claim runtime-PM/D3cold policy when releasing the GPU; with 0 the D0 locks and runtime reference are deliberately left held until reboot. | **Never enable** (rule 3); re-arming D3cold after a claim can hard-freeze the host. |
+
+The emulation-side `sema_*` parameters (ring discovery, release recovery,
+diagnostic watches, grace windows) are not physical-backend controls; their
+canonical description is [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md)
+§3.2 and [`03-architecture/06`](03-architecture/06-host-module-and-hybrid-model.md).
+
+### 10.3 The routine supervised cycle
+
+One cycle is one attempt; there is no second attempt in the same boot after a
+hard signature.
+
+**Step 0 — pre-flight (before anything is loaded):**
+
+* Host clean: no QEMU process, no mdev under `/sys/bus/mdev/devices/`, the
+ module not loaded, the physical GPU bound to the distribution driver.
+* Record `boot_id`, pstore contents, module sha256/`srcversion`, and the
+ planned parameter set in the cycle log.
+* Confirm supervision (present human), a written recovery plan, reboot
+ readiness per rule 2, and that other host work is finished.
+
+**Step 1 — build and identity check.** Rebuild the reviewed source; record the
+module hash; never run a cycle against a stale or unknown module.
+
+**Step 2 — load.** Prerequisites, then `open-iov-vfio.ko` with the reviewed
+parameters. Module-owned: the claim and requested stages run at load.
+Mdev-owned: load, then create the mdev (the async claim/boot starts) only when
+no VM is live.
+
+**Step 3 — claim/boot prerequisites.** In the cycle log confirm: `claimed`,
+`PCI_COMMAND` bus-master on, BAR lengths, `BOOT0`/`BOOT1`, both falcons, WPR2
+down at entry; the D0 settle check passed (claim refuses otherwise); and after
+the boot: `stage: primed=1 kicked=1 ran=1 riscv_active=1 rm_init_done=1`, WPR2
+up, `scratch14` hand-off bits advancing. Any claim refusal (`refusing claim`,
+`board unresponsive`) ends the cycle (rule 5).
+
+**Step 4 — ladder (host-side real submission).** Write `gsp-phys-rm-call`
+values 1..4 cumulatively; the ladder remembers the highest completed stage:
+
+| Write | Stage | Green signal |
+|---|---|---|
+| 1 | root/device/subdevice objects | stage 1 complete, handles logged (`hClient`/`hDevice`/`hSubdevice`). |
+| 2 | memory registration | stage 2 complete. |
+| 3 | VAS + TSG + channel (COPY0 variant walk; GR0 single attempt) | stage 3 complete (`variant`, `hChannel`, GR: `hGr`, `rm_call_ctx`). |
+| 4 | real submission: CE copy or GR semaphore release | `REAL COPY OK` (CE) / `REAL GR SEM OK` (GR); trigger 9 re-runs only the GR release. |
+| 5 | adopt static-config handles as stage 1 | stage 5 complete; a later 4 runs stages 2–4 with adopted handles. |
+| 6 | guest write direction (W2G) on the pinned GPA | `GUEST W2G COPY OK`. |
+| 7 | guest read direction (G2H) on the pinned GPA | `GUEST G2H COPY OK`. |
+| 8 | fresh-line PTE self-test (no GPA) | `GUEST PTE SELFTEST OK` / `selftest_ok=1`. |
+| 9 | GR semaphore re-run on a built GR ladder | `REAL GR SEM OK` or a decoded failure; no retry. |
+
+6–8 require `gsp_phys_guest_mem=1`, a READY backend and ladder stage ≥ 3;
+6/7 additionally require the GPA pinned via `gsp-phys-guest-gpa`. On the first
+failed stage: stop, drain the status queue, and end the ladder (rule 4).
+
+**Step 5 — guest attach (only if the rung needs it).** Start the VM with a
+file-backed shared guest RAM region (`share=yes, prealloc=on`), attach the
+mdev, configure `gsp-phys-guest-ram <path> <gpa_base> <offset> <size>`, pin
+the GPA, and run the driver-API smoke; the guest must stay green (`ALL OK`
+64/64) and any forwarded counters must be explained. Remove the mdev before
+`rmmod`; never unload with a live VM.
+
+**Step 6 — evidence capture.** Keep with the cycle: parameter set, module
+identity, boot id, `gsp-phys-state`, ladder traces, `gsp-phys-status-drain`
+records, NOCAT/`0x1006` decodes, `gsp-phys-log-dump`, and the clean-release
+checklist below. Evidence is property of the change that follows.
+
+**Step 7 — teardown/unload.** Remove mdevs/stop the VM, then `rmmod`
+(`scripts/stack_manager.sh unload`). The release sends
+`UNLOADING_GUEST_DRIVER` once (only after a boot), quiesces the falcon, runs
+the FWSEC teardown, frees staging, and rebinds the distribution driver only
+after a WPR2-down and board-liveness check, with the D0 locks held through a
+bounded post-rebind readiness gate.
+
+**Mandatory clean-release checklist** (every line must hold; any failure stops
+the cycle and invokes the fail-safe):
+
+| Check | Green value | If it fails |
+|---|---|---|
+| Boot id | unchanged from pre-flight | the device was power-cycled mid-cycle: discard in-window counters and rebuild the plan; expect a fresh scratch pool at the next boot. |
+| Module identity | sha256/`srcversion` identical before/after | wrong build ran: stop, rebuild, restart the cycle (rule 6). |
+| Protected region | `wpr2: ... state=down`; GSP falcon quiesced | fail-safe: **no rebind**, D0 locks + runtime reference stay held, WPR2 up; reboot approval per rule 2; never retry in this boot. |
+| Unload RPC | `rm_unload_submitted=1`, processor suspended (mailbox `0x80000000`) | wedged server; fail-safe as above (the release log says `teardown did not clear WPR2` / `refusing the hand-back`). |
+| Driver rebind | `released <bdf>; rebound to nvidia (rc=1)` | rebind failure/timeout: locks stay held; the module logs the manual `modprobe nvidia` path; reboot is the clean recovery. |
+| Power-domain locks | exactly 5 `power domain left LOCKED (gsp_phys_repower=0):` lines on this host (GPU + board functions + root port) | wrong count: stop; audit the locks before any further cycle. |
+| Rail cycles in window | 0 | any rail cycle rewrote the power risk: treat the cycle as unsafe and stop. |
+| pstore / leftovers | pstore empty; no QEMU, no mdev, guest-RAM file released | a crash record or leaked state exists: capture it, clean with `stack_manager.sh cleanup-crash`, and do not start another cycle until the host is clean. |
+
+Fail-safe summary: on any WPR2/unresponsive/rebind failure the release does
+**not** hand the device back, keeps the power-domain locks (and the runtime-PM
+reference) held, and the correct next action is an approved reboot with the
+written post-reboot plan - never a second attempt in the same boot.
+
+### 10.4 debugfs inventory
+
+All files live under the module's debugfs directory
+`/sys/kernel/debug/open-iov-vfio/`. **The directory and every file are
+root-only on this system**: existence and read checks must run as root (a
+permission error can look like an absent file). Every writer is inert unless
+its gates hold, and every accepted write performs a real RPC or device
+action - log it in the cycle record.
+
+| File | Mode | Purpose |
+|---|---|---|
+| `gsp-phys-state` | 0444 | Read-only state snapshot (§10.5): stage/boot state, RM probe/call, GR context, guest copy and relay counters, fault counters, guest-RAM ranges, WPR2/RISCV/falcon/scratch14, queue headers. |
+| `gsp-phys-log-dump` | 0444 | Read-only libos task log buffers (`LOGINIT`, `LOGRM`), raw words plus a decoded view. The raw view is authoritative; decode offline where a script is provided by the evidence set. |
+| `gsp-phys-status-drain` | 0444 | One bounded status-queue drain per read, printing the decoded `0x1006` engine-fault and `0x1020` NOCAT records; advances the host read pointer (normal consumer protocol). |
+| `gsp-phys-unload-rpc` | 0200 | Write `1`: send `UNLOADING_GUEST_DRIVER` once on a live claimed session; every other value is a logged no-op. |
+| `gsp-phys-rm-probe` | 0200 | Write `1`: run one `SET_GUEST_SYSTEM_INFO`/`GET_GSP_STATIC_INFO` probe cycle and cache the static-config snapshot. |
+| `gsp-phys-rm-call` | 0200 | Write `1`..`9`: the ladder stages, handle adoption, guest triggers and GR re-run of §10.3 step 4. |
+| `gsp-phys-guest-gpa` | 0200 | Write `0x<hex>`: pin one 4 KB-aligned guest GPA for triggers 6/7; one GPA per boot, a different GPA is refused. |
+| `gsp-phys-guest-ram` | 0200 | Write `<path> <gpa_base> <offset> <size>`: configure the guest RAM backing file and one GPA window (file path fixed for the module cycle); `-` clears. |
+
+### 10.5 Reading `gsp-phys-state`
+
+The first lines say whether the device snapshot is meaningful at all
+(`inactive: physical GPU not claimed` / `stage-3d staging not primed`); a
+snapshot is only read from a primed, claimed session. Then, in order:
+
+| Line | Fields | Meaning and invariants |
+|---|---|---|
+| `claimed` / `boot_param` / `dryrun_param` | claim and requested-stage flags | `claimed=0` with a live ladder stage means a release already ran: stop reading device state. |
+| `stage:` | `primed`, `kicked`, `ran`, `riscv_active`, `rm_init_done`, `rm_unload_submitted`, `dryrun`, `tx_seq_num` | The boot lifecycle position. READY work needs `primed=1 ran=1 rm_init_done=1`; `rm_unload_submitted=1` means the session is ending and no new forward is legal. |
+| `rm_probe:` | `attempted`, `last_ret`, `last_function`, `last_payload_len` | Static-config probe outcome; function 65 with a nonzero payload is the snapshot source. |
+| `rm_call:` | `ran`, `stage`, `last_stage`, `last_ret`, `rpcs`, `hClient`/`hDevice`/`hSubdevice`/`hVas`/`hTsg`/`hChannel`, `variant`, `engine`, `hGr`, `token`, `notifier`, `base` | Highest completed ladder stage and the object handles; `engine` is the stage-3 snapshot (COPY0/GR0). A GR `hChannel` with `token` but a faulting push is the context-commit wall, not a transport failure. |
+| `rm_call_ctx:` | `prepared`, `valid`, `pa`, `va`, `size`, `pages`, `ptPages`, `gr_promote_param`, `step`, `entrySize` | GR context buffer/PDB; printed only when a GR context was prepared. `gr_promote_param=0` means `PROMOTE_CTX` is never sent; `step` names the last accepted promote step. |
+| `rm_call_guest:` | `gpa`, `hpa`, `g2h_ok`, `w2g_ok`, `selftest_ok`, `gpa_valid`, `phys_domain_identity` | The last guest trigger's translation and proof flags; `phys_domain_identity=1` is the prerequisite for programming host PAs into the engine page tables. |
+| `work_guest:` | `req`, `fallback`, `ok`, `verify_ok`, `verify_fail`, `pool_exhausted`, `refused`, `cand_none`/`cand_sys_sys`/`cand_mixed`/`cand_vid_vid`, `resolve_fail`, `next_line`, `capacity` | Forwarded guest copies. Green run: `verify_fail=0`, `pool_exhausted=0`, `refused=0`, `resolve_fail=0`; `next_line ≤ capacity` (512 lines per boot, never reused). |
+| `work_guard:` | `copy_skipped`, `deferred`, `applied`, `fallback`, `pending`, `overflow`, `engaged`, `timeouts`, `canceled`, `defer_param`, `copy_soft`, `copy_fail`, `hw_primary_param` | Deferred-release hold. **Invariant: `applied + fallback == engaged`.** `pending` is the live hold gauge (should return to 0); `timeouts`/`overflow`/`canceled` must be 0; `copy_soft` counts the designed inline software fallbacks. |
+| `work_relay:` | `requested`, `ok`, `fallback`, `skipped_sw`, `verify_fail`, `verify_transient`, `hold_verify_fail`, `resolve_fail`, `timeouts`, `pool_exhausted`, `refused`, `relay_param` | Engine-authored semaphore relay. Green run: `verify_fail=0`, `hold_verify_fail=0`, `timeouts=0`, `pool_exhausted=0`, `refused=0`, `resolve_fail=0`; `verify_transient` is a retry-cleared mismatch and should be rare and explained. |
+| `faults:` | `seen`, `stale`, `drained`, `wait_unmatched`, `epoch` | Fault observation at the status queue: records seen vs discarded as stale, records drained, unmatched waits, and the epoch used to match a relay submission to the fault it caused. |
+| `fault_relay:` | `matched`, `unmatched`, `delivered`, `no_event`, `late_release`, `relay_param`, `fault_relay_param`, `fault_inject_param` | Guest delivery: `delivered` is a written notifier **and** event; `no_event` means the notifier was written but the auxiliary event post produced no MSI-X; `late_release` means the release landed before the fault could be relayed (no fault is delivered). |
+| `guest_ram:` | `active=1 path=... ranges=...` or `active=0` | Backing-file and GPA-window configuration; `active=0` disables every guest-page path. |
+| `wpr2:` / `riscv:` / `falcon:` / `scratch14:` | `state=up/down`; active bit; both falcons' CPUCTL/MBOX/FALCON_OS; secure-scratch bit fields | The fail-safe state. A green release ends with `state=down`, both falcons quiesced, and the booter-unload gate bits complete. |
+| `cmd_tx` / `stat_tx` | queue header words | Queue producer/consumer cursors; a stalled status read pointer with an advanced command read pointer means "consumed but never replied" (the wedge signature). |
+
+**Relay data-mismatch semantics (fail-closed).** The relay result is verified
+host-side against a source window snapshotted *before* the doorbell. A
+mismatch is retried once after 2 ms: if the second read matches, it is a
+`verify_transient` (the live guest page was mutated under the first compare)
+and the copy stays verified; if it persists it counts `verify_fail`, the
+submission returns an error, and the sema service falls back to the software
+release. The same fallback covers `hold_verify_fail` (the guest semaphore
+re-read did not carry the expected payload). **A software release apply is
+never lost or duplicated**, which is why the relay defaults off until
+`verify_fail` is triaged.
+
+## 11. Debugging scenario catalogue
+
+Physical-backend symptoms, their exact signal, and the bounded response. The
+defensive/new-input scenarios (firmware release, OpenRM build, userspace
+bump, device/ISA/ROM change, protected state, matrix/CI) are owned by
+[`RUNBOOK.md`](RUNBOOK.md) S1–S10 and are not duplicated here.
+
+| Symptom | Exact signal to look for | Decode / likely cause | First diagnostic | Action |
+|---|---|---|---|---|
+| Guest GR push faults | `func=0x1006 ... exceptType=0x20 runlistId=0 chid=1 errString="... intr1 80000000"`; NOCAT `recType=4 errorCode=0x20 faultingEngine="GR"` | `0x20` = robust-channel PBDMA error; `intr1` bit 31 = `CTXNOTVALID`; the external-VAS GR engine context was never committed (`ENG_CTX_VALID=0`), so the guest binds via `PROMOTE_CTX` (`0x2080012b`). CE channels are unaffected. | `gsp-phys-status-drain`; `gsp-phys-state` `rm_call`/`rm_call_ctx` | Decode first, no retry. Only a proven promote shape commits the context; keep the client promote path gated by `gsp_phys_rmcall_gr_promote=0` until a variant is proven. |
+| `PROMOTE_CTX` wedge | `gsp-nocat: rec seq=N recType=5 errorCode=0x52a8b04 source="ASSERT" faultingEngine=""`, then no fn76 reply (timeout `-110`); status ring stops, server command read pointer advanced | A host/server-RM assert in the promote path - not an engine fault (no `0x1006`). The server consumed the command and wrote no reply. Teardown cannot clear WPR2 (its unlock is fuse-locked on this part). | `gsp-phys-state` (`cmd_tx`/`stat_tx`, `wpr2`), release log | Ladder aborts by design; **do not retry**. Fail-safe holds the device; reboot with written approval (rule 2); bisect exactly one promote variant per cycle after reboot. |
+| WPR2 / clean-release failure | `teardown did not clear WPR2 (HI=0x...)`; `release: WPR2 still up after teardown ... refusing the hand-back`; `NOT rebinding ... power domain stays LOCKED` | The booter-unload gate bits were never completed by the wedge, or the board stopped answering; the host-side unlock is fuse-locked. | `gsp-phys-state` `wpr2`, `scratch14`, `stage:` | Fail-safe is the designed state: no rebind, locks held, WPR2 up. Leave the device; reboot with approval. |
+| Module unload hang | `rmmod` does not return; `ps` shows the emulation/relay kthreads or work items; mdev still present | Teardown ordering (kthread/workqueue) or an open mdev/VM; the emulation-side cleanup applies. | `ps -e \| grep gsp_emu`; mdev list; `dmesg` tail | Remove mdev/stop VM first; `scripts/stack_manager.sh cleanup-crash`; reboot with approval if it persists (rule 5). |
+| Guest `cuInit` wedge / guest flake | guest `cuInit rc=999`, `rmapiControlCacheSet ... NV_ERR_INVALID_PARAMETER`, or a host `gsp-phys-fwsec: unexpected WPR2 already up`; a one-off guest Oops in `free_swap_cache` cleared on retry | On the emulated path this is a known guest-side flake; the WPR2-up line is the FWSEC refusal on a board already in a bad state. | guest dmesg; host `gsp-phys-state` `wpr2`; retry the guest once | If only the guest flaked, restart the guest once. If `unexpected WPR2 already up` appears, stop the cycle and inspect state; never retry FWSEC (rule 4/5). |
+| GSP boot failure | `gsp-phys-fwsec: staging not visible before kick (DMEM0 ... IMEM ...)` then `FWSEC/FRTS execution failed: -5`; historical `GSP_INIT_DONE rpc_result=0x59` | Staging DMA did not land (async mdev-owned FWSEC path) or, historically, the pre-boot handshake/identity payload was wrong (`0x59` = operating-system error). | `gsp-phys-state` `stage:`, `gsp-phys-log-dump` | Use the module-owned claim path (proven); the async FWSEC defect is fail-open and off the critical path. Do not repeat a failed boot in the same boot after a hard signature. |
+| MMU/PTE faults and the fetch-line rule | engine copy times out with no semaphore; `GUEST CE COPY FAILED: scratch-PT pool exhausted ... capacity=512`; re-translated GPA yields a different HPA | The GMMU caches PTE fetches in 128-byte lines and there is no invalidate: a runtime PTE write into an already-fetched line is invisible. A guest page's HPA can change between translations. | `rm-call: guest HPA changed: gpa=... was ... now ...`; `work_guest` `pool_exhausted`/`next_line` | Always use a fresh never-reused line per pair; pre-build guest-window slots before engine access; pin translated pages; treat pool exhaustion as end-of-budget, not a bug. |
+| PBDMA method/class errors | `0x1006` with `intr0` nonzero (a method/class was rejected) versus `intr1` bit 31 (`CTXNOTVALID`); NOCAT `faultingEngine="GR"`; errString `"Channel ID ... intr0 ... intr1 ..."` | Wrong method encoding, class/subchannel or channel state; `intr0 != 0` means the engine rejected the method, so look at the pushed method block, not the context only. | `gsp-phys-status-drain`; compare the method encoding with the engine's class | Fix the method/class encoding; no retry against the same wedged channel; re-run from a fresh ladder. |
+| Relay symptoms | `GUEST CE RELAY FAILED: semaphore released ... does not match ... (retried after 2 ms)`; `verify_fail`/`verify_transient`/`hold_verify_fail`; `fallback`/`skipped_sw`; `pool_exhausted`; `guest HPA changed` | One persistent data mismatch (shadow-page/concurrent-guest race or decode offset edge) is the open residual; anything else is the designed fail-closed fallback to the software apply. | `work_relay` line; the relay failure log includes sem target/payload/poll, first differing byte, live/snapshot dumps, `src_mutated`, `snap_vs_dst` | Keep default off until the mismatch is triaged. Abort the feature (`gsp_phys_relay=0`, then `gsp_phys_work_forward=0`) if `timeouts>0` or `verify_fail>0` ever appears. |
+| Fault-relay symptoms | `fault_relay delivered=1` versus `no_event=1`; `late_release`; `matched`/`unmatched` by runlistId/chid; `faults epoch` | `delivered` = notifier written and event posted; `no_event` = notifier written but the auxiliary event post found no event object (no MSI-X reaches the guest); `late_release` = release landed first, so no fault is delivered (by design). | `fault_relay` line; `gsp-phys-status-drain` | A `no_event` is an event-path problem, not a fault-decode problem: fix the channel notifier/event wiring. Use `gsp_phys_fault_inject=1` to reproduce the path on demand. |
+| Guest driver probe failure | guest sees no PCI device / BAR map fails / probe hangs; host `board unresponsive`, mdev `-EUSERS`; missing `SET_IRQS wired` | Config space, BAR mapping, MSI-X wiring or mdev state on the emulated path - not the physical claim. | guest `lspci -vvv`; host `05-operations/02` §5.1 taxonomy | Follow the taxonomy in [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §5.1; never start a physical claim to debug a guest-side probe failure. |
+
+## 12. Hazard register (crash causes and debugging side effects)
+
+| Hazard | Mechanism | Mitigation / practice |
+|---|---|---|
+| D3cold re-arm / power-gate host freeze | Releasing the D0 policy or re-enabling runtime PM after a claim lets the root port enter D3; the platform can cut the board rail, and the rebind can fail with a host freeze minutes later (no oops, pstore empty). | `gsp_phys_repower=0` (latched) is the default; the D0 locks cover GPU + board functions + root port; the post-rebind readiness gate must pass before anything is released; never unbind/rebind outside the supervised claim; pstore armed each cycle. |
+| Failed mutating RPC wedges GSP-RM | One bad mutating operation (channel-flag incident; `PROMOTE_CTX` assert `0x52a8b04`) leaves the server unable to reply or suspend while WPR2 stays up. | One mutating attempt per staged step; never retry a failed mutating RPC; abort the ladder from the first failure; the release fail-safe keeps the locks and refuses the hand-back. |
+| debugfs write side effects | Every 0200 file performs a real RPC/device action when its gates hold; a "test" write changes device or guest state. | Root-only; read `gsp-phys-state` and record the intended effect before any write; one write per action; no trigger writes outside a supervised cycle. |
+| `pkill -f qemu-system-x86_64` self-match | The pattern also matches the invoking shell's own command line, killing the agent/session instead of the VM. | Use bracket patterns (`pgrep/pkill -f '[q]emu-system-x86_64'`) or explicit PIDs; stop VMs through `scripts/vgpu_vm.py`. |
+| Root-only debugfs | The whole `open-iov-vfio` debugfs directory and its files are root-only; a non-root existence check reports absence. | Run existence/read checks under root; treat "missing" as a permission question first. |
+| Module reload/unload hazards | Kthread/workqueue teardown can hang; the IOMMU identity-domain property and the guest-RAM file must not outlive the session; unloading with a live mdev/VM corrupts the teardown order. | Remove mdevs and stop VMs first; `gsp_sema_service_work_quiesce()` before staging is freed (release does this); use `scripts/stack_manager.sh unload`/`cleanup-crash`; audit leftovers after every unload. |
+| pstore/journal persistence | A freeze leaves no oops but usually a persistent pstore record; the journal stops. | Check `/sys/fs/pstore` before and after every cycle; never clear it; capture it before any reboot or retry - it is the only post-freeze evidence. |
+| Guest-RAM shared-file requirement | The engine can reach guest pages only through the QEMU file-backed shared region; a non-shared or non-preallocated RAM backend has no host page cache to translate. | Start QEMU with `memory-backend-file,share=yes,prealloc=on`; configure the file via `gsp-phys-guest-ram`; one file per module cycle; translation is gated on an open mdev and an active range. |
+| Release-before-verify race | A hardware copy that times out cannot be canceled, so a late engine write is theoretically possible after the software fallback already applied. | The deferred-release hold withholds the guest release until completion or the bounded window; the software apply is never lost; if `timeouts`/`verify_fail` appears, abort the feature rather than tune around it. |
+| Single-inflight limit / deferred-release hold | One forwarded request is in flight at a time; while a hold waits, new forwards are refused and apply their releases inline. | Designed fail-closed: `copy_soft` fallback, 500 ms cap, `pending` returns to 0. Do not deepen the queue without a measured need. |
+| Scratch-pool exhaustion | 512 fresh 128-byte PTE lines per boot and no reuse; each copy pair consumes one whole line. | Size a cycle's guest-copy workload to the pool; treat `pool_exhausted` as the designed end-of-budget, then reload/reboot for a fresh pool. |
+
+## 13. The physical-backend OODA loop
+
+| Stage | What to do | Exit criterion |
+|---|---|---|
+| **Observe** | Collect: the `gsp-phys-state` snapshot (§10.5), ladder traces and stage numbers, `gsp-phys-status-drain` records, decoded `0x1006`/`0x1020` faults, `gsp-phys-log-dump`, the clean-release checklist, module sha/boot id, and the evidence set. | The cycle's question can be stated in one discriminating sentence with its parameter set and run identity. |
+| **Orient** | Separate static from dynamic deltas. Classify failures: **engine fault** (`0x1006` + NOCAT `faultingEngine` set), **host/server assert** (NOCAT `recType=5`, `faultingEngine=""`, no `0x1006`), **transport** (timeout `-110`, status read pointer stalled while the command pointer advanced), or **data-verify** (`verify_fail`/`hold_verify_fail`). Match relay faults by `epoch` plus runlistId/chid and honour the external-VAS rule (`ENG_CTX_VALID` must be committed by a proven promote before a GR push can complete). | Every delta has exactly one class, or is recorded as an open gap. |
+| **Decide** | Choose one: fix / observe more / extend tooling / record an accepted gap. Bisect with exactly one variable per cycle (parameter variant, one promote shape, one method encoding); never retry a wedged path, and never decide "fix" from a single run or a static reading. | The decision names the evidence and the single changed variable. |
+| **Act** | Land the code change with the evidence set attached; commit only after a live green result on the same stimulus; fail-safe first (the release must refuse the hand-back before any optimisation). | Same stimulus is green, the clean-release checklist holds, and the change carries its evidence. |
+| **Stop rules** | Hard signatures (rule 5), the timebox (three observe/compare iterations without a discriminating delta = stop and write down what is missing), the reboot policy (rule 2), and **no unattended runs** (rule 1). | Nothing physical runs past a stop rule; the state and hypotheses are recorded for the next supervised window. |
+
+## 14. Work-context map and porting flows
+
+### 14.1 Which document owns which debugging context
+
+| Debugging context | Owning document |
+|---|---|
+| Emulated-driver debugging (guest driver vs the emulated GSP-RM) | [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §1–3, §5 (observability, captures, gdbstub, failure taxonomy). |
+| Firmware runtime (GSP/uproc in a RISC-V VM, hooks/watchpoints/sessions) | [`05-operations/03`](05-operations/03-risc-v-gsp-rm-vm.md) and the [green-eye user guide](05-operations/04-green-eye-user-guide.md) (experimental). |
+| Live physical backend (claim/boot, ladder, forwarding, relay, release) | **This Part II.** |
+| Portability (firmware, hardware, driver version, userspace version) | [`06-portability/*`](06-portability/hardware-software-matrix.md) and [`RUNBOOK.md`](RUNBOOK.md) S1–S10. |
+| Performance and correctness validation | [`05-operations/01`](05-operations/01-performance-correctness-security.md) (evaluation ladder, metrics, A/B method). |
+| Architecture of the backend and the guest wiring | [`03-architecture/08`](03-architecture/08-physical-gpu-integration.md), [`03-architecture/11`](03-architecture/11-physical-backend-guest-wiring.md). |
+| Current frontier and dated status | [`STATE-CURRENT.md`](STATE-CURRENT.md). |
+
+### 14.2 Triage flow for a CUDA/userspace symptom
+
+1. **Guest diagnostics first** ([`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §2–3): guest dmesg, PCI config, `nvidia-smi`, the driver-API smoke result, and the exact failing call.
+2. **Classify**: emulated reply (guest failed against the emulation plane) vs forwarded reply (`gsp_phys_static_live`/`gsp_phys_ctrl_forward`/`gsp_phys_work_forward` counters moved) vs engine fault (`0x1006`/NOCAT) vs guest-side flake (WPR2 untouched).
+3. **Check the owning state/log**: for forwarded paths `work_guest`/`work_guard`/`work_relay`/`fault_relay` in `gsp-phys-state`; for faults `gsp-phys-status-drain`; for boot/claim `stage:`/`wpr2:`; for emulation-only symptoms the emulation trace.
+4. **Next step**: emulation handler change under [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) discipline; forwarding parameter/ladder re-run under §10.3; fault decode and no-retry path under §11; guest-flake restart once. A physical claim is never the first response to a guest-side probe failure.
+
+### 14.3 Porting flows
+
+| New input | What changes | What must be validated | Procedure |
+|---|---|---|---|
+| Firmware image | The staged ELF, its signature section and the boot handshake; the static-config snapshot content. | ELF64 RISC-V shape and pairing; FWSEC/booter/RISC-V stages; RM init-done `rpc_result=0`; full static-config reply; one supervised cycle before any emulation overlay change. | [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §4.1; [`06-portability`](06-portability/hardware-software-matrix.md); [`RUNBOOK.md`](RUNBOOK.md) S1. |
+| GPU hardware | Device profile (PCI IDs, BAR geometry, real FB size), power-domain topology (lock count), engine inventory, identity overlay. | Claim/D0 lock and clean release on the new board; boot layout; engine capabilities; a supervised cycle at the target rung; matrix entry only once earned. | [`06-portability`](06-portability/hardware-software-matrix.md); [`RUNBOOK.md`](RUNBOOK.md) S5. |
+| Kernel driver version | The distribution driver that is unbound/rebound; module build against the kernel; OpenRM build glue if that path is used (never a runtime dependency). | Build order and symbols; claim/release hand-back; no regression on the module-owned session. | [`RUNBOOK.md`](RUNBOOK.md) S2; [`03-architecture/06`](03-architecture/06-host-module-and-hybrid-model.md) §7. |
+| CUDA/userspace version | Guest-visible controls/replies and the userspace crash surface; the emulation's reply fidelity. | Driver-API smoke `ALL OK` 64/64; control/ioctl inventory diff explained; A/B on the same anchor. | [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §4.2; userspace CUDA playbook; [`RUNBOOK.md`](RUNBOOK.md) S4. |
+
+### 14.4 Current frontier (2026-10-02)
+
+Guest CE copies are **hardware-authored with verified data** and the relay and
+fault-relay paths are in place, all default off (the deferred-release and
+hardware-primary invariants hold; one relay data mismatch is the open
+triage residual). The host GR ladder reaches the **context-promote wall**:
+the GR0 channel and compute object allocate and the engine path is encoded,
+but the external-VAS context is never committed, and the first host promote
+attempt asserted GSP-RM (`0x52a8b04`) without a reply and left the protected
+region up. The device is therefore parked in the **designed locked
+fail-safe** (module unloaded, GPU unbound, D0 locks held, WPR2 up) awaiting a
+supervised reboot; the promote path stays gated behind
+`gsp_phys_rmcall_gr_promote=0` until a bisected variant is proven. Live detail
+and evidence pointers: [`STATE-CURRENT.md`](STATE-CURRENT.md).
+
+<!-- change (2026-10-02): correct the porting/packaging tool section from
+planned interfaces that no longer match the landed scaffolds (docs 28/29) to
+the actual flags and outputs, link the device-validation sequence and
+future-eng-ops admission evaluations, and mark the scaffolds unadmitted;
+scope: eng-ops §3 axis 13, §6, §7, §8 only; benefit: readers no longer build
+against non-existent flags or treat unadmitted scripts as routine pipeline. -->
diff --git a/docs/future-eng-ops.md b/docs/future-eng-ops.md
new file mode 100644
index 0000000..0aa961a
--- /dev/null
+++ b/docs/future-eng-ops.md
@@ -0,0 +1,261 @@
+# Future engineering operations: green-eye capabilities not yet earning their keep
+
+> Companion to [`eng-ops.md`](eng-ops.md). That document records what the
+> project uses now; this one records green-eye capabilities that are built but
+> unused, partially used, planned or research, each tied to an impending
+> project need, scoped explicitly, and carrying an evaluation that can fail —
+> plus a kill criterion if it never pays off. Nothing here is a product
+> commitment, and no simulation observation becomes an accuracy claim without
+> the physical-confirmation rules of
+> [`06-ACCURACY.md`](../debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md).
+
+Status snapshot: 2026-10-02. Sources are the green-eye roadmap, tool docs
+16–27, the `harness/` entry points and the tool audit (doc 27). The tool tree
+is development-only: never shipped, never a dependency of `open-iov-vfio`,
+never present in a guest. RM 610 work is gated by human approval
+([`docs/21` §3](../debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md),
+[`docs/23`](../debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md)).
+
+## 1. Status vocabulary
+
+| Status | Meaning |
+|---|---|
+| `built, not used yet` | Implemented and tested, but no workflow calls it yet; the decision to keep is recorded. |
+| `partially used` | Some pieces run in a real workflow; the capability as named is incomplete. |
+| `planned` | Scoped in a roadmap/design document, not implemented. |
+| `research` | No committed design; needs a funded use case before it becomes `planned`. |
+
+Priority scale: **P1** do when the gate lands; **P2** next, capacity
+permitting; **P3** when a consuming workflow exists; **P4** research or
+funded-only. Effort is engineering-time for one developer familiar with the
+tree, excluding multi-quarter research.
+
+Every item below uses the same six fields: status, need, scope, evaluation
+(metric / baseline / threshold / kill), gates, effort and priority.
+
+## 2. Catalogue at a glance
+
+| # | Capability | Status | Gate | Effort | Priority | Kill when |
+|---|---|---|---|---|---|---|
+| F1 | MCP automation of full port chains | `partially used` | CI pipeline (done), RM 610 approval | 1–2 wk | P2 | two releases show no MCP-only artifact |
+| F2 | Multi-version driver port automation (540→current) | `partially used` | versioned inputs, anchor policy | 2–3 wk | P2 | two versions still need unabsorbed manual steps |
+| F3 | Device-port automation scripts | `partially used` | reference device, GB10 availability | 1–2 wk | P2 | capture needs privileged/unsafe operations |
+| F4 | Packing injected code and modified GSP images; trial bundles | `partially used` | H1.3, SDK | 2–3 wk | P3 | re-run needs host state or is non-deterministic |
+| F5 | Control discovery against the real RM surface | `built, not used yet` | H1.3 + H2.2 | 1–2 wk | P1 | no dispatch table exposed after H2.2 |
+| F6 | Value demos extended to rehosted uproc | `planned` | rehost shim (H2.3/R2) | 2–3 wk | P1 | fewer than 4 classes transfer |
+| F7 | Native simulator M2+ (map DSL, snapshots, scheduler) | `partially used` | H2.2 exit (hard) | 24–36 wk | P1 | moments cannot restore identically by M3 |
+| F8 | Adjacent-microcontroller models | `planned` | H2.2 | 6–10 wk | P2 | PMU/SEC2 semantics unprovable in one milestone |
+| F9 | Performance analysis tooling (before/after) | `partially used` | H2.2 for real firmware | 2–3 wk | P3 | reports disagree with physical benchmarks |
+| F10 | Guest/userspace friction: RDMA, shared page tables, peers | `research` | funded platform case | 8+ wk | P4 | no funded scenario at review |
+| F11 | SIMD cores and CUDA kernel execution on device | `research` | H6/H3, funded reason | unbounded | P4 | first kernel prototype fails bit-exactness |
+| F12 | FPGA soft-processor simulation | `planned` | H2/H3 proven, funded timing claim | multi-quarter | P4 | no funded timing claim at review |
+| F13 | Verilog recreation of GSP components and NVRISC fabric | `research` | F12 progress, legal review | multi-quarter | P4 | accuracy cannot be bounded after first milestone |
+| F14 | Hidden/undocumented/unauthorized-code discovery | `partially used` | H1.3 + H2.2 | 3–5 wk | P2 | two real releases yield no actionable finding |
+| F15 | SBOM/architectural understanding from observations | `planned` | H2.2, legal review | 2–3 wk | P3 | observation rows rejected twice by SBOM review |
+
+Status counts: `built, not used yet` ×1 (F5); `partially used` ×7 (F1, F2,
+F3, F4, F7, F9, F14); `planned` ×4 (F6, F8, F12, F15); `research` ×3 (F10,
+F11, F13). Each item carries exactly one evaluation criterion.
+
+## 3. Items
+
+### F1 — MCP automation of full port chains
+
+| Field | Detail |
+|---|---|
+| **Status** | `partially used`. The MCP server is implemented and tested (24 tools, read/observe by default, writes and injection gated; [`20-MCP-SERVER.md`](../debug/risc-v-gsp-rm-vm/docs/20-MCP-SERVER.md)), and it exposes `probe_firmware`, `portability_matrix_run`, `mutation_run` and `compare_sessions`. No agent has driven a complete release chain through it, and `version_diff`/`ci_release` have no MCP tool. |
+| **Need** | RM 610 readiness criteria C4/C5 require the MCP path and the release pipeline to be exercised, not merely present ([`23-RM610-READINESS.md`](../debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md)); every future GSP/driver release should be monitorable by an agent while a human keeps the merge gate. Waiting means each release is driven by hand and the 610 evidence pack stays unproven. |
+| **Scope** | Does orchestrate the existing MCP tools plus a new `version_diff`/`ci_release` entry into one named, budgeted pipeline ending in a hashed bundle; does not add network transport, does not relax write/inject gating, does not approve product changes. |
+| **Evaluation** | Metric: pipeline steps completed through MCP per release with a bundle manifest. Baseline: 0 complete chains (CLI only). Threshold: one 570.144 self-pair rehearsal completes probe → matrix → mutation → compare → bundle through MCP with artifact hashes equal to the CLI chain. Kill: if two releases produce no artifact the CLI chain lacks, retire the pipeline tool and keep MCP session-level only. |
+| **Gates** | MCP server (done) and release pipeline (done, [`21`](../debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md)); human approval before any RM 610 input; recorded session for every write. |
+| **Effort / priority** | 1–2 engineering-weeks; **P2** (unblocks RM 610 C4/C5). |
+
+### F2 — Multi-version driver port automation (540-series → 570.144 → 580/610)
+
+| Field | Detail |
+|---|---|
+| **Status** | `partially used`. `cudatool.py info/delta/fdematch`, `diff_corpus.py`, `annotate_corpus.py` and `symbols export` are in production use for the 570.144 anchor, but the docs/22 §2 runbook is ten manual steps. A concurrent `harness/port_driver_versions.py` scaffold is landing as an unadmitted pilot (dry-run default; [`docs/28`](../debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md) §2); no chained multi-version run is admitted or exercised yet and there is no 540-era regression corpus in-tree. |
+| **Need** | The anchor is 570.144; deployed guests and the anticipated 580/610 families each repeat the same runbook, and wrapper re-resolution (~1% of wrapper names transfer by address) is a known trap ([`22-PORTING-CUDA-AND-DEVICES.md`](../debug/risc-v-gsp-rm-vm/docs/22-PORTING-CUDA-AND-DEVICES.md)). Each repeat costs days of human time and risks silently moving wrapper names. |
+| **Scope** | Does chain the existing CUDA tools into one per-version report (exports, FDE transfer matrix, corpus delta, name map, symbol DB, smoke results) and diff reports across versions; does not decompile unknown binaries automatically, does not change the firmware anchor, does not run a guest smoke without a supervised guest. |
+| **Evaluation** | Metric: wall-clock and human minutes per version triage to a complete evidence pack. Baseline: the recorded 570.144 → 570.211 manual runbook. Threshold: one rerun (or 540-era reference) triaged in ≤50% of the baseline human minutes with wrapper re-resolution warnings preserved. Kill: if two consecutive versions need manual steps the chain cannot absorb, drop the chain and keep per-step tools. |
+| **Gates** | Versioned parallel directories and anchor policy ([`22`](../debug/risc-v-gsp-rm-vm/docs/22-PORTING-CUDA-AND-DEVICES.md) §1); RM 610 gate; no new acquisition policy. |
+| **Effort / priority** | 2–3 engineering-weeks; **P2**. |
+
+### F3 — Device-port automation scripts
+
+| Field | Detail |
+|---|---|
+| **Status** | `partially used`. Probe first-touch, portability matrix and version diff exist for firmware targets, but the device profile fields of docs/22 §3.1 are collected by hand (`lspci`, sysfs, `rmtool` JSON, guest diagnostics) and `config/hardware/*.yaml` is a design target, not read by code. A concurrent `harness/port_new_device.py` scaffold is landing as an unadmitted pilot (dry-run default; [`docs/28`](../debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md) §3); it is not yet exercised and no profile capture has been automated in a workflow. |
+| **Need** | TU104 → GB202/GB10-class ports each require the profile table plus a pairing check; a missed field surfaces later as an "emulation bug". Pairing errors are already a known failure mode (570.144 image vs 570.211 driver). |
+| **Scope** | Does script the read-only profile capture (PCI IDs, BAR layout, FB geometry, engine inventory, MSI-X topology, ROM quirks, probe-time registers), verify firmware pairing, and run the matrix across reference and candidate to emit a candidate profile plus evidence. Does not bind drivers, does not write to the device, does not claim support. |
+| **Evaluation** | Metric: profile fields captured automatically out of the eight in docs/22 §3.1. Baseline: fully manual table. Threshold: ≥7/8 fields plus a pairing verdict and matrix report from one command on the reference TU104. Kill: if any required capture needs privileged writes or is not read-only-safe, retire the script to the manual table. |
+| **Gates** | Supervised access to the reference device; GB10 hardware availability; the 610 gate where labels match. |
+| **Effort / priority** | 1–2 engineering-weeks; **P2**. |
+
+### F4 — Packing injected code and modified GSP images; trial bundles
+
+| Field | Detail |
+|---|---|
+| **Status** | `partially used`. The PIC service SDK, `greeneye inject`, `elf.write_flat_kernel` and newly tested `write_relocated_elf` exist, and version diff already byte-patches an image ad hoc (docs/25 s07). A concurrent `harness/package_sim.py` scaffold is landing as an unadmitted pilot (trial directory plus `--pack-image`; [`docs/29`](../debug/risc-v-gsp-rm-vm/docs/29-PACKAGING-AND-TRIALING.md)); no packer or trial bundle is admitted, hash-verified in CI, or used by a reviewer workflow yet. |
+| **Need** | Injection experiments today are demo-slot recipes; RM-core injection (H1.3) and reviewer replay need a reproducible artifact, the docs/23 C8 evidence bundle needs one-command reproduction, and hand-built byte patches do not carry an ABI/version contract. |
+| **Scope** | Does build a deterministic image-pack (input hash, relocation map, injected segments, patched dispatch entries, blob ABI version) and a trial bundle (tool revision, profile, stimulus, expected session); does not sign images, does not bypass firmware checks, does not make an image runnable on hardware, does not ship. |
+| **Evaluation** | Metric: manual steps to reproduce a reviewer's injected-service run from the artifact. Baseline: the ad-hoc patch + inject recipe. Threshold: one command yields a bundle whose rerun reproduces a byte-identical session on a clean host. Kill: if a rerun depends on host-specific state or is non-deterministic, retire the pack format and keep sessions as the artifact. |
+| **Gates** | H1.3 for RM dispatch patching; `make sdk`; the docs/12 blob contract; native M2 moments optional. |
+| **Effort / priority** | 2–3 engineering-weeks; **P3**. |
+
+### F5 — Control discovery against the real RM surface
+
+| Field | Detail |
+|---|---|
+| **Status** | `built, not used yet`. The `Surface` abstraction, sweep, classification, reporting and determinism tests are complete; `--surface rm` exists but exits cleanly because no RM control/method entry point is exposed ([`19-CONTROL-DISCOVERY.md`](../debug/risc-v-gsp-rm-vm/docs/19-CONTROL-DISCOVERY.md) §5). |
+| **Need** | Every new GSP image can add or re-point controls silently; static table reading is per-image work and does not reveal triggers or effects. Control inventory (M1) is the input surface the emulation must handle, and the RM 610 port will need it first. |
+| **Scope** | Does sweep RM control/method IDs by class with deterministic argument patterns, capture handler entry, selected object, status and effect snapshots, and replay a found route via `reach.py`; does not call hardware or a live RM, does not treat unreachability as proof, does not mutate replies. |
+| **Evaluation** | Metric: RM controls classified with handler entry and effect snapshot versus the `rmtool` host-oracle inventory. Baseline: demo-only sweep (189 probes ≈ 36 s, docs/19 §3.4). Threshold: after H1.3/H2.2, one real control space swept with ≥90% of oracle controls attributed or explicitly default-path, plus one previously unknown control reached and replayed with a recorded session. Kill: if H1.3/H2.2 do not expose a dispatch table, retire `--surface rm` and re-scope discovery to rehosted dispatch code. |
+| **Gates** | H1.3 + H2.2 (hard); device map ([`24`](../debug/risc-v-gsp-rm-vm/docs/24-DEVICE-MAP.md)); human approval for 610. |
+| **Effort / priority** | 1–2 engineering-weeks after the gates; **P1** when H2.2 lands. |
+
+### F6 — Value demos extended to rehosted uproc
+
+| Field | Detail |
+|---|---|
+| **Status** | `planned`. s01–s10 run and are measured on the bundled `uproc_demo`; every real-firmware case in [`25-VALUE-VALIDATION.md`](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md) is labelled blocked or projected. No demo runs on a rehosted GSP-derived uproc build. |
+| **Need** | The value claim is currently analog-only. Reviewers, the RM 610 C8 evidence pack and the project's "hours, not weeks" success criterion need the same ten question classes answered on NVIDIA-derived code before the physical path is required. |
+| **Scope** | Does port the demo classes (reply consumer, counter publication, unknown commands, run-until, snapshot diff, injection, version diff, mutation, Ghidra join, evidence pack) to a rehosted uproc ELF with identical success/failure tables; does not require the full image to boot, does not claim GSP-RM fidelity beyond the rehosted subset. |
+| **Evaluation** | Metric: demos passing against a rehosted uproc with the expected outcome. Baseline: 10/10 on the demo analog only. Threshold: ≥6/10 on rehosted uproc, the remainder explicitly device-model-blocked with reasons, per-demo wall clock ≤2× analog. Kill: if fewer than four classes transfer, keep the demo corpus as a tooling regression suite, stop calling it value validation and re-scope docs/25. |
+| **Gates** | A rehost shim and a stable symbol DB for the rehosted build (roadmap R2/H2.3). |
+| **Effort / priority** | 2–3 engineering-weeks; **P1**. |
+
+### F7 — Native simulator M2+ (map DSL, snapshots/time-travel, deterministic scheduler)
+
+| Field | Detail |
+|---|---|
+| **Status** | `partially used`. M1 (RV64IMAC interpreter + `Session` parity) and M1.5 (native exec/access trace) are implemented and differential-tested; the device map and lenient MMIO mode exist. M2–M6 — map DSL, falcon/CSB/PTIMER devices, moments, deterministic scheduler, injection ABI, performance hardening — are designed but not built ([`16-CUSTOM-SIMULATOR.md`](../debug/risc-v-gsp-rm-vm/docs/16-CUSTOM-SIMULATOR.md)). |
+| **Need** | QEMU has no cheap snapshots, deterministic replay or in-process device callbacks, so the fuzzing and discovery harnesses pay a fresh session per probe (measured ~36 s/189 probes) and branch exploration is impossible. H4/H5 work and the R5 fuzzing corpus are gated on cheap forks and replay. |
+| **Scope** | Does M2 map DSL + falcon/CSB/PTIMER + moments; M3 deterministic scheduler + event log/replay; M4 device-model parity with H2.2; M5 injection ABI/interception; M6 performance and CLI. Does not claim L2 timing or cycle accuracy, does not replace QEMU before M4, no realtime behaviour. |
+| **Evaluation** | Metric: (a) double-run canonical hash equality, (b) moment save/restore latency against the docs/16 §14 targets (≤100 ms save / ≤50 ms restore at 256 MiB sparse), (c) rehosted boot from a profile plus restore reproducing the event-stream hash. Baseline: fresh-session cost and QEMU. Threshold: all M2/M3 exit criteria in docs/16 §13 met on the rehosted uproc. Kill: if moments cannot restore byte-identical sessions by the M3 exit, drop snapshots and keep deterministic replay-only. |
+| **Gates** | H2.2 exit (hard, cannot start before); docs/16 §13. |
+| **Effort / priority** | 24–36 engineering-weeks for M2–M6; **P1** after H2.2, not before. |
+
+### F8 — Adjacent-microcontroller models
+
+| Field | Detail |
+|---|---|
+| **Status** | `planned`. H5 defines PMU/SEC2/ACR/SOE inventory, per-controller maps and cross-controller scenarios; none is modelled. H2.2 will force partial PMU/SEC2 stubs to pass early GSP init. |
+| **Need** | GSP-RM early init reads PMU identity/clock provisioning and SEC2 state; GB10-class and confidential-compute partitions follow the Init → ACR → RM ordering. Cross-controller regressions are invisible until a hardware cycle fails, which is exactly the expensive class this tool exists to remove. |
+| **Scope** | Does per-controller memory maps, mailbox protocols and boot state machines as versioned profile data, plus recorded cross-controller sessions; does not fully model the security chain, does not make confidentiality or attestation claims, does not run real ACR/SOE. |
+| **Evaluation** | Metric: cross-controller exchanges reproduced and compared against hardware evidence. Baseline: 0. Threshold: one exchange (GSP ↔ PMU identity/clock) reproduced with an A/B session pair reaching the same state as the hardware reference under the docs/06 metric table. Kill: if PMU/SEC2 semantics cannot be established from public evidence within one milestone, keep only documented H2.3 shims and mark adjacent-controller claims out of scope. |
+| **Gates** | H2.2; public/authorised source material; a hardware reference capture; docs/23 gate for any 610-family controller. |
+| **Effort / priority** | 6–10 engineering-weeks for PMU/SEC2 stubs; **P2**. |
+
+### F9 — Performance analysis tooling (profiling path, before/after methodology)
+
+| Field | Detail |
+|---|---|
+| **Status** | `partially used`. `sim_bench.py`, the trace-overhead measurements (docs/16 §1.1a, docs/25 §6) and the product A/B methodology in [`01-performance-correctness-security.md`](05-operations/01-performance-correctness-security.md) §3 exist. There is no green-eye path that profiles a product run off-hardware or produces an automated before/after report. |
+| **Need** | Every emulation change is measured against docs/05 targets (e.g. init within 2× native) using physical cycles and a manual harness; pre-screening structural regressions off-hardware would reduce failed physical cycles. M7 timing is never claimed, so the useful signal is structural (instruction/event counts, session bytes, written pages), not wall clock. |
+| **Scope** | Does capture deterministic structural metrics per session and diff them before/after with an explicit fidelity label; does not make timing or wall-clock claims about silicon, does not replace the physical benchmark in docs/05. |
+| **Evaluation** | Metric: fraction of perf-sensitive emulation changes that receive a green-eye before/after report before the physical benchmark. Baseline: 0. Threshold: five consecutive changes carry a report, and structural regressions ≥10% are flagged before the hardware run. Kill: if reports agree with the physical benchmark less than 50% of the time, retire them to debug annotations only. |
+| **Gates** | H2.2 for real firmware; docs/06 M7 rule; docs/05 methodology is authoritative. |
+| **Effort / priority** | 2–3 engineering-weeks; **P3**. |
+
+### F10 — Guest/userspace compatibility friction: RDMA, shared page tables, simulated peers
+
+| Field | Detail |
+|---|---|
+| **Status** | `research`. No peer-device simulation (DPU/NIC/NVMe), no RDMA path; page-table work exists only on the product GMMU/IOMMU side and in the physical-backend wiring. |
+| **Need** | Edge-AI platforms place GPU, networking and storage VMs in one trust domain with device-to-device data paths ([`haaeep-and-collaboration.md`](08-ecosystem/haaeep-and-collaboration.md)); guest RDMA and shared-page-table flows are a known friction class, and failures are expensive to debug on hardware. A simulated peer would let the page-table contract be regression-tested early. |
+| **Scope** | Does model DPU/NIC/NVMe peers enough to exercise page-table sharing and a GPUDirect-style map → peer DMA → completion path against the simulated GSP; does not implement real network/storage protocols, does not claim performance, does not modify guests. |
+| **Evaluation** | Metric: peer-interaction failure classes reproduced off-hardware that matched a hardware capture. Baseline: 0. Threshold: one page-table-sharing interaction reproduced and compared against hardware. Kill: if no funded platform scenario needs it at the next review, keep it as a design note and delete the experiment scaffolding. |
+| **Gates** | H2.2/H3; a funded platform use case; no confidential-compute claim; a hardware reference capture. |
+| **Effort / priority** | 8+ engineering-weeks; **P4** research. |
+
+### F11 — SIMD cores and CUDA kernel execution on device
+
+| Field | Detail |
+|---|---|
+| **Status** | `research`. The native core is scalar RV64IMAC; there is no SM/SIMT or vector execution model. The product's guest-to-physical execution wiring is in progress ([`11-physical-backend-guest-wiring.md`](03-architecture/11-physical-backend-guest-wiring.md)), so command streams exist but nothing executes kernels off-hardware. |
+| **Need** | Executing a real CUDA kernel against a device model would validate the command/memory path beyond copy-engine correctness and enable fail-closed testing of kernel submissions without a physical cycle. Waiting is acceptable while hardware backends exist, which is why this stays research. |
+| **Scope** | Does explore a device-side execution model (SIMT/SIMD core plus memory model) able to run a minimal kernel from an emulation-produced command stream; does not claim performance or cycle accuracy, does not replace hardware for CUDA, not on the critical path. |
+| **Evaluation** | Metric: vectorAdd-class kernels producing byte-identical results to a hardware reference. Baseline: 0. Threshold: one deterministic kernel correct on the simulator from an emulation-produced command stream. Kill: if the first bounded prototype cannot produce bit-identical results for a 64-element vectorAdd, stop and record why; only a funded cycle-accuracy or thesis use case may restart it. |
+| **Gates** | H6/H3; a funded reason; simulator-security posture (docs/16 §16). |
+| **Effort / priority** | Unbounded research; **P4**. |
+
+### F12 — FPGA soft-processor simulation
+
+| Field | Detail |
+|---|---|
+| **Status** | `planned`, far later. The H6 annex defines an RTL RV64 core with TCM/MPU/timers, falcon/CSB/ACR windows, DMA and WPR2/FWSEC state, Verilator and FPGA run modes, and a co-simulation bridge. Nothing is built or started. |
+| **Need** | When a funded timing/interoperability claim appears (customer-facing timing, hardware validation of an emulated behaviour), functional models are insufficient. Until then, starting would be effort creep against the roadmap's explicit non-critical-path note. |
+| **Scope** | Does build/choose an RV64 core and device fabric, pure-RTL simulation for determinism, FPGA for speed, and a `Backend` bridge so the same hooks, sessions and comparisons apply; does not claim confidentiality or isolation, does not ship, never present in a guest. |
+| **Evaluation** | Metric: cycle accounting agreement between RTL and a hardware capture on a bounded scenario. Baseline: none. Threshold: one bounded scenario (e.g. mailbox round trip) cycle-accounted within a declared tolerance, or the variance fully explained. Kill: if no funded timing claim exists at the next review, keep the design annex and close the effort without building. |
+| **Gates** | H2/H3 proven; validated device map; explicit funded reason; docs/16 §16 H6.3. |
+| **Effort / priority** | Multi-quarter; **P4**, funded-only. |
+
+### F13 — Verilog recreation of GSP-interacting components and the NVRISC fabric
+
+| Field | Detail |
+|---|---|
+| **Status** | `research`. H6.1 names the GSP-interacting blocks, but there is no model of the NVRISC environment, interconnect ordering, interrupt fabric or doorbell/mailbox races, and no tracked public design to build one from without a provenance review. |
+| **Need** | Deep timing, security and interoperability questions (interconnect ordering, boot-order races, doorbell semantics) cannot be answered by functional models; a documented recreation is the only off-hardware route to them. No funded scenario exists yet, so it is research. |
+| **Scope** | Does define and build behavioural/cycle-aware RTL for GSP-interacting blocks and the NVRISC fabric environment with a declared accuracy envelope; does not reproduce secret silicon internals, does not claim confidentiality/attestation, never on the critical path. |
+| **Evaluation** | Metric: interconnect-order or race observations reproduced against hardware captures within the stated envelope. Baseline: 0. Threshold: one documented race/ordering case explained and validated on hardware. Kill: if accuracy cannot be stated as a bounded envelope after the first milestone, retire to a paper design and record the decision. |
+| **Gates** | F12 progress; a funded reason; legal review (tracked docs describe behaviour only). |
+| **Effort / priority** | Multi-quarter research; **P4**. |
+
+### F14 — Hidden/undocumented/unauthorized-code discovery
+
+| Field | Detail |
+|---|---|
+| **Status** | `partially used`. Control discovery, reachability and seeded mutation are built and measured on the demo (injected-slot aliases, default-path `~id`, sentinel-pointer faults; [`19-CONTROL-DISCOVERY.md`](../debug/risc-v-gsp-rm-vm/docs/19-CONTROL-DISCOVERY.md)). Capability inference is designed, decoy/anti-analysis handling is unbuilt, and no real image can be swept before H2.2. |
+| **Need** | Firmware is stripped and signed; new images can add handlers, re-route dispatch entries, or insert anti-analysis checks. A missed handler is a silent emulation regression, and the fail-closed rules the product needs come from error paths that only adversarial input reaches. |
+| **Scope** | Does sweep inputs, classify unknown/default paths, infer capabilities from watchpoints and trace, catalogue observed anti-analysis techniques and feed fail-closed rules; does not bypass hardware protections, does not attack a live device or guest, does not publish exploit material. |
+| **Evaluation** | Metric: hidden/undocumented behaviours found per image that changed a modelling or security decision. Baseline: demo findings only. Threshold: ≥1 new real-image behaviour per release that lands with a replayable probe spec and a recorded emulation/security decision. Kill: if two real-image releases yield no actionable finding, reduce to the fast CI inventory smoke and stop the broader campaign. |
+| **Gates** | H1.3/H2.2; docs/14 ethics and safety rules; human review before publication. |
+| **Effort / priority** | 3–5 engineering-weeks plus campaign time; **P2**. |
+
+### F15 — SBOM/architectural understanding from observations
+
+| Field | Detail |
+|---|---|
+| **Status** | `planned`. The repository has a tracked SBOM ([`SBOM.md`](../SBOM.md), CycloneDX) and the wiki GPU SBOM is superseded; there is no observation-derived inventory per firmware image that links services, symbols and dispatch tables to the SBOM and architecture docs. |
+| **Need** | Version bumps change firmware components and external tooling dependencies; provenance, license and vulnerability response need to know what an image contains and what changed. The architecture docs drift from the binary until a costly re-analysis rediscovers the delta. |
+| **Scope** | Does derive an observation-backed component/capability inventory per image (hashes, symbols, services, dispatch tables, external dependencies) and link it to SBOM/architecture updates; does not assert legal conclusions, does not name local sources, does not auto-publish un-reviewed findings. |
+| **Evaluation** | Metric: SBOM/architecture deltas per release that are observation-derived and reviewable. Baseline: manual entries. Threshold: one release where every added/removed firmware component row carries a hash plus evidence link from a green-eye session. Kill: if observation rows are rejected by SBOM review twice, drop to a provenance annex inside the session evidence. |
+| **Gates** | H2.2 for real images; legal-and-licensing review ([`legal-and-licensing.md`](09-reference/legal-and-licensing.md)); local-provenance rules. |
+| **Effort / priority** | 2–3 engineering-weeks; **P3**. |
+
+## 4. Pruning and retirement
+
+An item is retired at the review where its evaluation fails or its gate has
+not landed for two consecutive reviews. Retirement is a recorded action, not a
+silent drift:
+
+| Outcome | Trigger | Action | Record |
+|---|---|---|---|
+| **Prune** | Threshold missed, or gate unlanded after two reviews, with no funded exception | Delete the code path and its tests; mark the capability `retired` here; remove it from roadmap commitments | Decision log row with date, evidence link, owner |
+| **Demote** | Need is real but not now | Move the item to a research appendix; stop maintaining scaffolding; keep the design pointer | Decision log row |
+| **Keep** | Evaluation passes, or a named funded exception exists | Retain with owner and next check date; state the exception's expiry | Decision log row |
+
+No item has been retired yet (first review pending). The smallest
+`built, not used yet` surface found by the tool audit (`Session.symbol_at`, an
+alias with no caller) is the first prune candidate: if it still has no caller
+at the first review, delete it rather than documenting API symmetry.
+
+## 5. Review cadence
+
+| Trigger | Who | What is re-checked |
+|---|---|---|
+| H2.2 exit | green-eye maintainer + eng-ops owner | F5, F6, F7, F8, F14, F15 gates open; statuses move to `partially used`/`planned` execution |
+| Each GSP/driver release | green-eye maintainer | F1/F2/F3 chain cost, F14 findings, F15 SBOM deltas; kill criteria evaluated |
+| RM 610 approval request | project lead + maintainer | F1 evidence (C4/C5), bundles for C8; no 610 input before approval |
+| Quarterly | eng-ops owner | Re-run or explicitly defer every evaluation; prune/demote/keep decisions recorded in §4 |
+| New funded use case | project lead | Admit a new item with the same six fields; research items become `planned` only with funding |
+
+The doc is reviewed against the tool audit ([`27-TOOL-AUDIT.md`](../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md))
+and the roadmap ([`GREEN-EYE-ROADMAP.md`](../debug/risc-v-gsp-rm-vm/GREEN-EYE-ROADMAP.md))
+so the "what we use now / not yet" split stays true.
+
+<!-- change (2026-10-02): point F2–F4 at the landed docs 28/29 scaffolds so the
+catalogue entries and the tree agree; scope: F2/F3/F4 status rows only;
+benefit: a reader can find the pilot code and its evaluation from the catalogue
+without treating it as admitted tooling. -->