docs: engineering-operations playbook and project-wide status accuracy pass

- docs/eng-ops.md gains Part II, live physical-backend engineering
  operations: hard rules, the full module-parameter matrix, the routine
  supervised cycle and clean-release checklist, the debugfs inventory,
  how to read gsp-phys-state, a debugging scenario catalogue
  (CTXNOTVALID/promote wedge, fail-safe WPR2, unload hangs, guest wedges,
  MMU/PTE and PBDMA faults, relay symptoms), a hazard register (crash
  causes and debugging side effects), the physical-backend OODA loop, and
  the work-context map with porting flows
- architecture, implementation, operations, orientation, getting-started,
  portability, ecosystem and reference docs corrected to the current
  state: CUDA driver-API smoke green, CE-class work forwarding and
  engine-authored releases plus fault relay in place (default off), GR
  execution blocked on engine-context validity, the promote wedge and
  designed locked fail-safe, and P5 wave-2 subsumed
- internal-reference cleanup in tracked docs (no analysis-source or
  local-archive references); historical architecture document bannered
- README reading list and STATE-CURRENT frontier refreshed; new pages
  (runbook, green-eye user guide, porting sequence, porting/packaging
  notes) linked from the tree

Link check: scripts/check-docs-links.py passes (56 files, no broken links).

1 files changed, 850 insertions, 0 deletions

diff --git a/docs/eng-ops.md b/docs/eng-ops.md
new file mode 100644
index 0000000..117bd0c
--- /dev/null
+++ b/docs/eng-ops.md
@@ -0,0 +1,850 @@
+# Engineering Operations: tooling admission, workflows and retirement
+
+> How Open-IOV decides which engineering tools it uses, extends and retires,
+> and how it measures whether a tool pays for its maintenance. Maintainers
+> decide and record; agents execute bounded loops and produce evidence.
+> Subsystem manuals are linked, not duplicated.
+
+## 1. Purpose and scope
+
+**Who uses it.** Maintainers decide, record and review; agents (MCP, CI, cron)
+execute bounded loops and produce sessions, reports and bundles. The same rule
+binds both: evidence or it did not happen ([`onboarding-and-standards.md`](07-developers/onboarding-and-standards.md) §5, [`15-AGENTIC-GSP-IDE.md`](../debug/risc-v-gsp-rm-vm/docs/15-AGENTIC-GSP-IDE.md) §5).
+
+| Governed here | Owned elsewhere (authoritative) |
+|---|---|
+| Admission, extension, redesign, trialing and retirement of tools | Architecture ([`03-architecture/01-system-overview.md`](03-architecture/01-system-overview.md)); tool manuals: CUDA ([`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md)), green-eye ([`26-ARCHITECTURE.md`](../debug/risc-v-gsp-rm-vm/docs/26-ARCHITECTURE.md), [user guide](../debug/risc-v-gsp-rm-vm/docs/02-USAGE.md), [scripting](../debug/risc-v-gsp-rm-vm/docs/03-SCRIPTING.md)), rmharness ([`README`](../debug/risc-v-gsp-rm-vm/rmharness/README.md)) |
+| Which tool applies in which loop; automation and packaging | Accuracy M1–M6 ([`06-ACCURACY.md`](../debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md)); performance method ([`05-operations/01`](05-operations/01-performance-correctness-security.md) §3); version discipline ([`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §4) |
+| Evidence bundles, version-span gates, frontier admission | Per-tool docs and the periodic [`27-TOOL-AUDIT.md`](../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md) |
+| Live physical-backend engineering operations: supervised-cycle mechanics, parameter gates, debugfs controls and the hazard register (**Part II** below) | Physical-backend architecture ([`03-architecture/08`](03-architecture/08-physical-gpu-integration.md), [`03-architecture/11`](03-architecture/11-physical-backend-guest-wiring.md)); defensive/incident scenarios ([`RUNBOOK.md`](RUNBOOK.md)); current frontier ([`STATE-CURRENT.md`](STATE-CURRENT.md)) |
+
+**Explicitly out of scope.** (1) Security claims: the simulator and its device
+maps are study environments, never a confidentiality or isolation boundary
+([`06-ACCURACY.md`](../debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md) §6,
+[`GREEN-EYE-ROADMAP.md`](../debug/risc-v-gsp-rm-vm/GREEN-EYE-ROADMAP.md) V4).
+(2) Timing claims: L2 is not modelled; order-of-events only
+([`06-ACCURACY.md`](../debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md) §2).
+(3) Shipping injected code: development-only, never a firmware patch, never in
+a guest ([`12-INJECTION-SDK.md`](../debug/risc-v-gsp-rm-vm/docs/12-INJECTION-SDK.md) §1).
+(4) Support and licensing: tested-and-documented only
+([`hardware-software-matrix.md`](06-portability/hardware-software-matrix.md) §1);
+legal rules in [`legal-and-licensing.md`](09-reference/legal-and-licensing.md).
+
+## 2. The core rule: no insertion without a measurable outcome
+
+A tool, or an insertion of an existing tool into a workflow, is admitted only
+with all five fields below. Missing any field rejects the proposal; it is not
+deferred until the metric is invented later.
+
+| Field | Meaning | Example |
+|---|---|---|
+| **Scenario** | exact question, target, version and loop where the tool is inserted | "classify one new GSP image's dispatch-table delta before the emulation is touched (portability loop)" |
+| **Measurable outcome** | result with a unit and an observer | "image drop → `version-diff.md` delta table; sessions per finding" |
+| **Baseline** | the replaced method and its measured/recorded cost | "static corpus diff + one supervised boot per hypothesis; baseline in [`docs/25`](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md) §3" |
+| **Evaluation threshold** | the number that decides keep/extend/retire, with sample size and environment | "N ≥ 3; median TTFA ≤ 25 % of baseline; self-diff zero; M-table improves or stays equal" |
+| **Failure signal** | observable "not useful" outcome during the pilot | "unexplained `compare` deltas after two triage passes, or the tool needs hardware to run" |
+
+Rules: one scenario per record; the threshold is recorded **before** the first
+run and moving it needs a new record; every admission starts as `admit-pilot`
+with an expiry of one quarter or five sessions; the baseline must be observable
+(measure it first if unrecorded); a missed threshold triggers §4.4/§4.5.
+
+```text
+TOOL ADMISSION RECORD                                    id: TOOL-<yyyy>-<nn>
+tool:           <name or path; existing | extension | new>
+loop:           static-RE | dynamic-RE | porting | performance | correctness | CI | guest-compat
+owner/decider:  <maintainer>; executor: <human|agent|cron>
+scenario:       <one sentence: question, target, version, trigger>
+measurable:     <metric + unit + observer>
+baseline:       <replaced method; cost source and date>
+threshold:      keep if <...>; reject if <...>; N=<runs>; environment=<...>
+failure signal: <observable not-useful outcome>
+evidence:       <session dir / report / manifest + sha256>
+decision:       admit-pilot | admit | extend | redesign | retire | do-not-build
+review-by:      <date or N sessions>              trials-dir: <path>
+```
+
+The record is stored with the evidence (session directory, bundle or PR
+description); decisions are recorded by a human maintainer and mirrored in the
+register (§4.5) and revision notes (§9.3). At expiry: met → admit; not met →
+redesign or retire; no signal → retire. No unused pilot survives.
+
+## 3. Value axes: what tooling is for, and how it is measured
+
+Each admission names exactly one axis. Every axis reduces to moving evidence
+through a §3.3 loop; if it cannot be expressed as a time or a count, it is not
+admissible.
+
+| # | Axis | Concrete example (real tooling) | Metric (recorded) | Loop |
+|---|---|---|---|---|
+| 1 | Faster reverse engineering | `version_diff.py` + `compare` on two GSP images; Ghidra corpus diff | firmware drop → classified delta (TTFA); symbols resolved vs Ghidra-only | RE |
+| 2 | Deeper / more correct analysis | watchpoints + TCG/native trace; capability inference ([`docs/14`](../debug/risc-v-gsp-rm-vm/docs/14-ADVERSARIAL-INTEROP-GUIDE.md) §4.2) | attributed bytes/stores; M6 agreement; unexplained-delta count | RE |
+| 3 | Code-artifact porting time | `cudatool.py fdematch`, `diff_corpus.py`, `symbols export` | transfer rate per class; wall time old→new symbol DB; wrapper re-resolution errors | portability |
+| 4 | Bug-analysis time | `reach.py`, `run_until`/`call`, `ioctl_trace.c`; taxonomy in [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §5 | crash/question → first evidence-backed answer (TTFA); supervised cycles avoided | adversarial |
+| 5 | Development velocity on the simulated GSP | injection SDK; `greeneye.sh inject`; value demo s06 | seconds from service edit to direct+mailbox agreement; sessions per prototype | RE |
+| 6 | Correctness validation of virtualization | M1–M6 table, `compare`, golden sessions | M-table per change; rung ladder ([`05-operations/01`](05-operations/01-performance-correctness-security.md) §1) | correctness |
+| 7 | Performance analysis and improvement | A/B method, `tools/benchmark_infrastructure.sh`, simulator overhead ([`docs/25`](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md) §6) | B/A % per metric; p50/p99 RPC; trace overhead; before/after same machine | performance |
+| 8 | Userspace/API compatibility friction | capability negotiation ([`../userspace/cuda/docs/04-virtualization-and-emulation.md`](../userspace/cuda/docs/04-virtualization-and-emulation.md) §1), symbol DB, ioctl diffs | controls with coherent replies vs observed; TTFA for a new capability bit; peer scenarios (RDMA guest, DPU/NIC/NVMe page sharing) counted only once a peer model exists | guest-compat |
+| 9 | Hidden / undocumented / unauthorized code, side effects, backdoors | `control_discovery.py`, mutation harness, decoy handling ([`docs/14`](../debug/risc-v-gsp-rm-vm/docs/14-ADVERSARIAL-INTEROP-GUIDE.md) §4.4) | unknown IDs and injected-slot aliases found; capability-model deltas; corpus entries with sessions | adversarial |
+| 10 | Architectural understanding and SBOM/doc improvement | symbol DB, `export_hits`, firmware notes ([`docs/09`](../debug/risc-v-gsp-rm-vm/docs/09-GSP-FIRMWARE-NOTES.md)), [`SBOM.md`](../SBOM.md) | doc/SBOM rows updated per finding; onboarding time; subsystem coverage | RE |
+| 11 | Security fixes / hardening time | capability inference → fail-closed rules ([`04-implementation/02`](04-implementation/02-practical-challenges-and-adversarial-interop.md) §4), mutation corpus | error paths covered; finding → test time; physical confirmation before product change | adversarial |
+| 12 | New GSP support time | `probe` + `version_diff` + `portability_matrix` + `ci_release` bundle | hours from image to triage table; M-table completeness; regression lead time | portability |
+| 13 | New hardware-device support time | device profile capture, `rmharness`, matrix, [`docs/22`](../debug/risc-v-gsp-rm-vm/docs/22-PORTING-CUDA-AND-DEVICES.md) §3, [device sequence](06-portability/03-device-validation-and-porting-sequence.md) | profile fields captured; engine inventory validated; first-touch deltas explained; tested-at-rung date | portability |
+| 14 | CUDA / runtime / VM-environment fixes | `userspace/cuda/docs/07`, `cuda_drv_test.c`, `tools/cuda-ab/ab.sh`, guest diagnostics | crash → named function time; `RESULT`/`STATUS` match; rung movement | guest-compat |
+
+Axis 8 note: GPU-direct RDMA availability is negotiated via the stock guest's
+vGPU capability block; page-table sharing with simulated DPU/NIC/NVMe peers is
+future, so its metric starts only when a peer model exists (§8).
+
+**Performance axis (axis 7).** Profiling order is the project's existing one
+([`05-operations/01`](05-operations/01-performance-correctness-security.md) §3.6):
+RPC dispatch → doorbell cadence → eventfd/MSI-X delivery → BAR0 shadows →
+memory-model gap. Method: (1) define the metric and A/B pair before code
+changes — same binary, inputs, governor, clocks and boot where possible, and
+report B/A %, never absolutes; (2) run A via `tools/benchmark_infrastructure.sh`
+and B via `scripts/vgpu_vm.py exec`, logging versions and
+`nvidia-smi -q -d CLOCK`; (3) attribute with host counters
+(`gsp_emu_get_stats()`, `trace_printk`, `/sys/kernel/debug/tracing`) and the
+measured simulator overhead ([`docs/16`](../debug/risc-v-gsp-rm-vm/docs/16-CUSTOM-SIMULATOR.md) §1.1a, [`docs/25`](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md) §6);
+(4) keep the numbers with the change and re-run correctness; (5) reject single
+runs, cross-host comparisons, bare-metal-only data and simulator timing claims.
+
+**OODA reduction.** Every axis moves evidence through one of three loops. Each
+loop's compression is the project's program-level metric
+([`GREEN-EYE-ROADMAP.md`](../debug/risc-v-gsp-rm-vm/GREEN-EYE-ROADMAP.md) success
+criteria), measured in [`docs/25`](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md) §7:
+
+| Loop | Observe → Orient → Decide → Act | Compression metric |
+|---|---|---|
+| RE-constrained (static/dynamic interlock) | probe/hooks/watch/trace → `analyze`/snapshots → `compare`/triage → reimplementation | question → reviewable answer; sessions per finding |
+| Adversarial-interop (opaque/decoy firmware) | discovery/mutation/replay → classification ([`docs/14`](../debug/risc-v-gsp-rm-vm/docs/14-ADVERSARIAL-INTEROP-GUIDE.md) §4.1) → fail-closed or no change → emulation change + physical confirm | unknown-ID → classified route; error paths covered |
+| Portability (GSP/device/architecture churn) | probe first-touch/symbol diff → pairing check/delta class → anchor decision → versioned dirs + handlers | image drop → triage table (hours) |
+
+## 4. Use, extend, build, redesign, retire
+
+```mermaid
+flowchart TD
+  P["Friction + scenario + measurable outcome"] --> Q1{"Existing tool covers the scenario?"}
+  Q1 -->|"yes"| USE["Admit the insertion; pilot + metric record"]
+  Q1 -->|"no"| Q2{"A close tool has a public extension point?"}
+  Q2 -->|"yes"| EXT["Extend in place: plugin, harness Surface, probe spec, SDK"]
+  Q2 -->|"no"| Q3{"Repeated scenario and no cheaper alternative?"}
+  Q3 -->|"no"| NB["Do not build: record rejection and the trigger to revisit"]
+  Q3 -->|"yes"| NEW["Admit a new tool as a bounded pilot"]
+  USE & EXT & NEW --> REV{"Threshold met after the pilot?"}
+  REV -->|"yes"| KEEP["Admit; add a second metric over time"]
+  REV -->|"no"| Q4{"Failure signal points at the design, not the idea?"}
+  Q4 -->|"yes"| RD["Redesign with a migration note; re-pilot"]
+  Q4 -->|"no"| PRUNE["Retire / prune; keep evidence read-only"]
+```
+
+| Decision | Minimum evidence | Decider | Record |
+|---|---|---|---|
+| (a) Use existing | scenario + measured baseline + one recorded trial session | maintainer; agent may run the trial | admission record + register row; link the session |
+| (b) Extend existing | ≥ 3 uses or ≥ 2 users; public extension point (plugin, `Surface`, probe spec, SDK); cost ≤ new tool | maintainer | record names the extension point and keeps the old tests green |
+| (c) Build new | record complete; bounded pilot; golden/replay evidence; stdlib/local-only unless justified | maintainer | record + register + revision note; tests added to the tool suite |
+| (d) Redesign | failure signal twice, or a fidelity gap that changes conclusions; migration note (`greeneye-session/1` evolves additively) | maintainer | before/after metric; old artifacts stay readable |
+| (e) Stop / prune | threshold missed, no evidence in the window, duplicated capability, or maintenance > value | maintainer at the audit | retire row; `27-TOOL-AUDIT.md` status; keep sessions/reports |
+| (f) Do not build | any trigger below is true | maintainer | rejection note naming the trigger that would reopen it |
+
+**When NOT to build (workflow risk).** Reject if any holds; the absence of a
+time-to-value process is the risk this document removes: (1) no scenario, unit
+or baseline — the pilot cannot fail; (2) the tool would become a shipped
+dependency, guest component or firmware patch
+([`04-implementation/02`](04-implementation/02-practical-challenges-and-adversarial-interop.md) §5.1);
+(3) it needs physical GPU, mdev or guest access for its primary value
+([`docs/15`](../debug/risc-v-gsp-rm-vm/docs/15-AGENTIC-GSP-IDE.md) §5);
+(4) it bypasses a security boundary or turns a study observation into a
+security claim; (5) it duplicates a tool without a measured improvement;
+(6) total cost is not plausibly below the replaced OODA cost; (7) its only
+evidence is prose — evidence must be a session, report or bundle.
+
+**Redesign and retirement.** Redesign rather than patch when the failure
+signal repeats, a fidelity gap invalidates an accepted conclusion, a public
+interface must change incompatibly, or the tool drifted outside its admitted
+loop. Retire when no artifact references the tool, the threshold was missed
+twice, another tool covers the scenario better, or maintenance exceeds the
+saving. Retirement is revocation of admission, not deletion: keep tool, tests
+and artifacts read-only, mark the register row `retired` with date and reason,
+and reconcile at the next [`27-TOOL-AUDIT.md`](../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md).
+
+**Register (how decisions are recorded).** The durable record is the register
+row, the evidence directory/bundle it names and the revision note in §9.3;
+the audit is the periodic reconciliation.
+
+| Admitted family | Loop | Value evidence (current) | Review / retire trigger |
+|---|---|---|---|
+| Ghidra corpora + scripts (`tools/ghidra`, `userspace/cuda/tools`) | RE, portability | docs 05/07; `diff_corpus`, name map | corpus diff no longer changes port class |
+| green-eye core + session format | dynamic-RE | `docs/25` s01–s10; 155-test suite | format unused by any loop |
+| Harness (`version_diff`, matrix, mutation, discovery, reach, `ci_release`) | portability, adversarial | docs 18/19/21; deterministic replay tests | a report is never attached to a change |
+| Injection SDK + demo uproc | dynamic-RE | docs 11/12; s06 | prototypes stop using it |
+| Ghidra bridges (in/out) | RE | docs 05/13 | static/dynamic join replaced |
+| Device map (`greeneye-device-map/1`) | portability | docs 24; H2.1 first-touch | H2.2 model stops consuming maps |
+| `rmharness` + `rmtool`/`rmtop`/`rmq` | correctness, guest-compat | docs 17; M1/M6 reference captures | reference capture unused/unavailable |
+| MCP server + agent workflows | all loops | docs 15/20; release monitoring | write gating or audit trail removed |
+| CI (`ci.yml`, `greeneye.yml`) | correctness | release-blocking tests | suite non-reproducible or skipped |
+
+## 5. Workflow map: where each tool applies
+
+### 5.1 Context of the development and reverse-engineering environment
+
+```mermaid
+flowchart LR
+  subgraph STATIC["Static"]
+    GH["Ghidra CUDA corpus + GSP firmware"]
+  end
+  subgraph DYN["Dynamic (green-eye, experimental)"]
+    GE["Session + QEMU/native backends"]
+    HR["harness/*.py"]
+  end
+  subgraph HOST["Host and guest"]
+    MOD["open-iov-vfio.ko"]
+    VM["stock guest + guest-diag"]
+    RMH["rmtool/rmtop/rmq + rmharness"]
+  end
+  subgraph AUTO["Automation"]
+    CI["ci.yml / greeneye.yml"]
+    MCP["MCP server + agents/cron"]
+  end
+  EVID["Evidence: sessions, reports, bundles"]
+  GH -->|"names, index, symbol DB"| GE
+  GE -->|"export_hits, fault PCs"| GH
+  GE --> HR --> EVID
+  GE --> EVID
+  RMH --> EVID
+  VM -->|"stimulus and failures"| GE
+  MOD --> VM
+  EVID --> AUTO
+  AUTO -->|"bounded runs"| HR
+  EVID -->|"product-change gate"| MOD
+```
+
+### 5.2 Static reverse-engineering loop
+
+```mermaid
+flowchart LR
+  BIN["libcuda / .fwimage"] --> IMP["Ghidra import + analysis"]
+  IMP --> IDX["names.tsv, decompiled_index.tsv, callgraph.tsv"]
+  IDX --> DIFF["corpus diff / fdematch / RPC dispatch diff"]
+  DIFF --> HYP["Hypothesis: functions, tables, layouts"]
+  HYP --> SYM["greeneye symbols export (bias recorded)"]
+  SYM --> DYN["dynamic loop"]
+  DYN -->|"hits, watch PCs, fault PCs"| HYP
+```
+
+### 5.3 Dynamic firmware loop (OODA)
+
+```mermaid
+flowchart LR
+  Q["Question + target + fidelity statement"] --> OBS["Observe: probe, hooks, watch, trace"]
+  OBS --> ORI["Orient: analyze, symbol decode, snapshots"]
+  ORI --> DEC["Decide: compare, M1-M6, mutation"]
+  DEC --> ACT["Act: emulation change with evidence"]
+  ACT --> REG["Regression: same stimulus, same budgets"]
+  REG -->|"improved or equal"| PACK["Evidence pack + session hashes"]
+  REG -->|"regressed"| Q
+  PACK --> Q
+```
+
+### 5.4 Porting loop (GSP firmware, device, architecture)
+
+```mermaid
+flowchart TB
+  ACQ["Acquire + hash; versioned dirs, never overwrite"] --> TRI["Triage: delta, fdematch, pairing check"]
+  TRI --> PRB["Probe first-touch old vs new"]
+  PRB --> RUN["Rehost/demo: identical stimulus + record"]
+  RUN --> CMP["compare + M1-M6 skeleton"]
+  CMP --> BUN["version_diff -> matrix -> mutation -> bundle"]
+  BUN --> REV{"Every delta explained?"}
+  REV -->|"no"| TRI
+  REV -->|"yes"| ANCHOR["Anchor decision + docs + matrix entry"]
+```
+
+### 5.5 Performance loop
+
+```mermaid
+flowchart LR
+  DEF["Define metric + A/B pair"] --> BASE["A: bare metal baseline"]
+  DEF --> CAND["B: vGPU / candidate"]
+  BASE --> RATIO["B/A ratio, variance, N runs"]
+  CAND --> RATIO
+  RATIO --> PROF["Attribute: RPC, doorbell, eventfd, shadows"]
+  PROF --> KEEP["Keep numbers with the change"]
+  KEEP --> REG["Re-run correctness corpus + ladder"]
+```
+
+### 5.6 Correctness and CI loop
+
+```mermaid
+flowchart LR
+  CH["Change"] --> HOST["Host suite: stack_manager.sh test"]
+  HOST --> GUEST["Guest ladder: probe, nvidia-smi, cdt"]
+  GUEST --> AB["cuda-ab: RESULT/STATUS must match"]
+  AB --> SESS["Session compare / golden data"]
+  SESS --> GATE{"Rung or metric regressed?"}
+  GATE -->|"no"| LAND["Land with evidence"]
+  GATE -->|"yes"| CH
+```
+
+### 5.7 Guest-compatibility loop
+
+```mermaid
+flowchart LR
+  GD["guest-diag: ioctl trace, engines, pmap"] --> CTRL["Control/ioctl inventory"]
+  CTRL --> EMU["Emulated reply fidelity"]
+  EMU --> GD2["New driver/CUDA version trace"]
+  GD2 --> DIFF["Control diff old vs new"]
+  DIFF --> CTRL
+  CTRL --> CORPUS["CUDA corpus + symbol DB"]
+```
+
+### 5.8 Loop index and detailed docs
+
+| Loop | Entry points | Evidence | Gate | Detail |
+|---|---|---|---|---|
+| Static RE | `tools/ghidra/*`, `userspace/cuda/tools/*`, `analyzeHeadless` | corpus TSVs, name map, symbol DB | addresses re-derived per build | [`userspace/cuda/docs/07`](../userspace/cuda/docs/07-version-upgrade-playbook.md), green-eye docs 05/13 |
+| Dynamic RE | `greeneye.sh` probe/record/analyze/compare/inject/ghidra, Python `Session` | `greeneye-session/1` + `summary.md` | A/B pair; L2/L3 statement | [`26-ARCHITECTURE.md`](../debug/risc-v-gsp-rm-vm/docs/26-ARCHITECTURE.md) §11, [`11`](../debug/risc-v-gsp-rm-vm/docs/11-GSP-RM-SERVICE-DEVELOPER-GUIDE.md), [`12`](../debug/risc-v-gsp-rm-vm/docs/12-INJECTION-SDK.md), [`13`](../debug/risc-v-gsp-rm-vm/docs/13-GHIDRA-CONTROL.md), [`14`](../debug/risc-v-gsp-rm-vm/docs/14-ADVERSARIAL-INTEROP-GUIDE.md) §4.6, [`06`](../debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md) §5 |
+| Porting | `version_diff.py`, `portability_matrix.py`, `mutation_harness.py`, `ci_release.py` | reports + `bundle-manifest.json` | every delta explained | [`07`](../debug/risc-v-gsp-rm-vm/docs/07-PORTABILITY-PLAYBOOKS.md), [`22`](../debug/risc-v-gsp-rm-vm/docs/22-PORTING-CUDA-AND-DEVICES.md), [`21`](../debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md), runbook [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §4.1 |
+| Performance | `benchmark_infrastructure.sh`, module counters, `vgpu_vm.py` | B/A table with environment | same-machine A/B; corpus green | [`05-operations/01`](05-operations/01-performance-correctness-security.md) §3 |
+| Correctness | `stack_manager.sh test`, `make tests`, `cuda_drv_test.c`, `cuda-ab` | rung + trace excerpts | no rung regression | [`05-operations/01`](05-operations/01-performance-correctness-security.md) §1–2 |
+| CI | `.github/workflows/ci.yml`, `.github/workflows/greeneye.yml` | workflow logs | push/PR path triggers | [`21`](../debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md) §5 |
+| Guest-compat | `tools/guest-diag/*`, `cudatool.py`, `diff_corpus.py` | ioctl/control diff, corpus delta | stock guest only | [`READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md) |
+
+## 6. Automation and agent invocation
+
+```mermaid
+flowchart LR
+  PRE["Preflight: greeneye.sh info; RM610 gate"] --> DRY["ci_release.py --dry-run (plan only)"]
+  DRY --> REAL["ci_release.py real run"]
+  REAL --> VD["version_diff"]
+  REAL --> PM["portability_matrix"]
+  REAL --> MU["mutation smoke"]
+  VD & PM & MU --> MAN["release-report.json + bundle-manifest.json"]
+  MAN --> REV["agent/human review"]
+  REV -->|"accepted"| REG["anchor/register/docs updated"]
+  REV -->|"rejected"| GAP["fidelity gap or fix, recorded"]
+```
+
+Run everything from `debug/risc-v-gsp-rm-vm/`; scripts are stdlib-only and
+launch their own QEMU. Never re-use an existing gdbstub, a guest, a kernel
+module or hardware.
+
+**Scripts that exist** (success and failure are read, not assumed):
+
+| Script | Safe invocation | Success | Failure signal / limit |
+|---|---|---|---|
+| `harness/version_diff.py` | `python3 harness/version_diff.py --old <old> --new <new> --out dumps/vdiff` | sections `ok`; self-diff zero; deltas explained | exit 2 tool error, 3 RM610 refusal; raw `.fwimage` faults early by design |
+| `harness/portability_matrix.py` | `python3 harness/portability_matrix.py --target <a> --target <b> --out dumps/matrix` | self-comparison `all_zero`; M1–M6 `not measured` without reference | non-zero target deltas or per-target `status: error` |
+| `harness/mutation_harness.py` | `python3 harness/mutation_harness.py --out dumps/mut --iterations 64 --seed 7` | deterministic `findings.jsonl`; declared outcome classes | seed divergence; non-demo ELF refused without `--allow-foreign-elf` |
+| `harness/control_discovery.py` | `python3 harness/control_discovery.py --surface demo --out dumps/disc --seed 7` | known IDs resolve; unknown/injected classes listed | RM surface exits cleanly until H1.3; misclassification or unreplayable probe |
+| `harness/reach.py` | `python3 harness/reach.py --target rpc_ping --method call --args 0x41 --out dumps/reach` | `reachable` with recorded route and args | timeout means "not under this stimulus", not unreachable |
+| `harness/ci_release.py` | dry-run first: `python3 harness/ci_release.py --old <old> --new <new> --out dumps/release --dry-run` | dry-run touches no target; real run writes the bundle | exit 1 step error, 2 tool error, 3 RM610 refusal; `skipped` is not `pass` |
+| `greeneye.sh` | `./greeneye.sh info`/`probe`/`record`/`analyze`/`compare`/`inject` | session dir with `session.json` + `summary.md` | empty sessions or missed hooks; budgets required |
+| `rmharness/harness.sh` | `./rmharness/harness.sh --session dumps/real -- rmtool engine list --format=json` | `rm_harness_run` event + stream blobs | read-only only; permission errors reported, never escalated |
+| `plugins/mcp_server.py` | `python3 plugins/mcp_server.py --stdio --record` | 24-tool JSON-RPC surface; read/observe default | stdout protocol-only; writes need `--allow-write`/`--allow-inject` |
+| `harness/value_demos/run_all.py` | `python3 harness/value_demos/run_all.py` | one `RESULT_JSON` per demo; suite ≤ 180 s | any demo `success: false`; re-run before quoting numbers |
+
+**Scaffolded scripts (landed, not admitted yet).** The three scripts below
+landed with docs 28/29 and are pilots until they have a §2 record; the first
+two are dry-run by default (`--run` executes). `--help` is the authoritative
+flag list; the outputs and failure readings are in
+[`docs/28`](../debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md) and
+[`docs/29`](../debug/risc-v-gsp-rm-vm/docs/29-PACKAGING-AND-TRIALING.md).
+
+| Script | Purpose | Interface | Outputs | Exit codes |
+|---|---|---|---|---|
+| `harness/port_driver_versions.py` | sweep a version list through `version_diff` + matrix + mutation, one evidence set per version | `--versions <list>` or `--config <json>`, `--baseline <ver>`, `--input-dir <dir>`, `--target <ver>=<path>`, `--symbols <ver>=<path>`, `--ghidra-exports <ver>=<path>`, `--out <dir>`, `--run` (default: dry-run), `--force`, `--extract-runfiles`, `--approval-token` | per-version `version-diff/` + `portability-matrix/` + `mutation-smoke/` + `bundle-manifest.json`, `sweep-summary.json|.md` | 0 complete, 1 step/threshold failure, 2 tool error, 3 RM610 refusal |
+| `harness/port_new_device.py` | per-device port: profile, pairing check, first-touch probe, symbols, matrix, checklist | `--device <label>`, `--profile <json>`, `--firmware <path>`, `--reference <path>`, `--symbols <json>`, `--ghidra-exports <tsv>`, `--out <dir>`, `--run` (default: dry-run), `--approval-token` | `device-port.json|.md`, `port-checklist.json|.md`, `symbols/`, sessions, matrix, `bundle-manifest.json` | as above; 1 when pairing is ambiguous or a delta unexplained |
+| `harness/package_sim.py` | build (and optionally run) a trial simulation package (§7) | `--target demo\|image`, `--image <path>`, `--service <elf>`, `--device-map <json>`, `--script <py>`, `--slot N`, `--pack-image`, `--tar`, `--run`, `--out <dir>` | package tree + `manifest.json`; `results.json|.md` after `run.sh` | 0 built/run, 1 replay or assertion failure, 2 tool error, 3 gate refusal |
+
+All three: stdlib-only, simulator/local only, deterministic budgets recorded in
+the report, reusing existing harness functions, and running the RM610 label check first.
+Their admission evaluation is recorded in
+[`future-eng-ops.md`](future-eng-ops.md) F2–F4.
+
+**Version span and the RM 610 gate.** The automated span is **540 → current**
+(anchor 570.144, [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §4.1);
+inputs are versioned under `firmware/<SERIES>-GSP/<VER>/` and never overwritten.
+**RM 610 requires explicit human approval**: before approval nothing fetches,
+opens or analyses a 610 image
+([`23-RM610-READINESS.md`](../debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md)).
+`version_diff`/`ci_release` refuse any `rm610`-matching label with exit 3
+unless `--approval-token RM610-APPROVED` or `GREENEYE_RM610_APPROVED=1` is
+present; the check runs in `--dry-run` and before any target is opened. An
+agent may prepare the approval request and run current-anchor rehearsals; only
+a named human approves.
+
+**Evidence bundles.** `ci_release.py` writes `release-report.json`,
+`bundle-manifest.json` (sha256 + size of every file) and the step trees
+([`docs/21`](../debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md) §4.3).
+Exit 0 with all steps `ok` is complete, but read `skipped`; exit 1 means a step
+errored (reviewable, not evidence for that section); exit 2 no claim; exit 3 a
+gate refusal, not a bug. Verify hashes before sharing and attach the bundle.
+
+**MCP monitoring.** `plugins/mcp_server.py` exposes the live session
+(read/observe by default) and shells out to `probe_firmware`,
+`portability_matrix_run`, `mutation_run`, `analyze_current` and
+`compare_sessions` ([`docs/20`](../debug/risc-v-gsp-rm-vm/docs/20-MCP-SERVER.md) §5).
+Release pattern: matrix → pinned mutation smoke → compare the recorded pair →
+read the bundle (`version-diff.json` / `release-report.json` /
+`bundle-manifest.json`); the agent does not recompute it. Absent capabilities
+return `not_available`, so planning is safe.
+
+**Cron/agent safety.** Default to read/observe and `--dry-run`; write/inject
+flags are explicit and the recording is the audit trail. Physical GPU, mdev
+churn, module load/unload and guest access are never unattended. Budgets
+everywhere; a wedged target is left stopped. Product changes are proposals —
+only humans merge, after physical confirmation of behaviour changes. Local
+only: stdio MCP, loopback TCP at most; never expose it to a network.
+
+## 7. Packaging and trialing
+
+A trial package lets a user run one **pre-scripted whole-GSP simulation** with
+injected code without the lab. It is a developer artifact: local, simulator
+only, no hardware, guest or kernel module. It is not a product component and
+not a support claim.
+
+```text
+dist/<pkg>/
+  README.md                 scenario, expected result, budgets, L2/L3 caveat
+  run.sh                    deterministic runner (greeneye.sh run + settle)
+  scenario.py               fixed stimulus, fixed budgets, records a session
+  firmware/<image>          ELF or .fwimage, hash-pinned
+  service/<service>.elf     injected service (PIC, ABI v1), hash-pinned
+  device-map/<map>.json     greeneye-device-map/1 (bias, entries, fw sha256)
+  golden/session/           optional reference session for compare
+  package-manifest.json     sha256 + bytes per member (bundle shape, docs/21)
+```
+
+Runner and scenario follow the injection contract: PIC blob, no libc, no
+absolute addresses, one dispatch entry per slot, every write recorded
+([`docs/12`](../debug/risc-v-gsp-rm-vm/docs/12-INJECTION-SDK.md) §2–5); the device map
+carries provenance so a lenient run is not mistaken for device-modelled behaviour
+([`docs/24`](../debug/risc-v-gsp-rm-vm/docs/24-DEVICE-MAP.md) §1).
+
+```text
+python3 harness/package_sim.py --out dist/<pkg> --service <service.elf> \
+    --device-map <map.json> --script <scenario.py> [--pack-image] [--tar] [--run]
+./dist/<pkg>/run.sh
+```
+
+Success: the packaged `run.sh` reports `status: ok` (demo target) or the
+documented blocked status (real image), and every `manifest.json` component
+hash matches. Failure: hash mismatch, nondeterministic replay, unresolved
+symbol or missing member. The scaffold has no `--verify` subcommand yet:
+verification is re-running `run.sh` and `sha256sum` against the manifest
+([`docs/29`](../debug/risc-v-gsp-rm-vm/docs/29-PACKAGING-AND-TRIALING.md) §2–4),
+and it stays a pilot until future-eng-ops F4's threshold is met.
+
+**Sharing and judging results.** Share the package (or its manifest) with the
+session it produces. The recipient judges manifest hashes, the session
+`summary.md`, the golden comparison and the fidelity statement (L0/L1 only,
+L2/L3 absent). Results are evidence about firmware in the simulator; a product
+change still requires the accuracy loop and physical confirmation.
+
+**Packing a modified GSP image.** When a GSP modification (injected service,
+shim or patched firmware) is considered ready to ship, `package_sim.py` may
+pack the modified image, the tested injected code, device map, scenario and
+golden session so the recipient can reproduce the exact run before any product
+decision. **Caveat, in every package README: this is a simulation package, not
+a signed production image.** It is not a firmware patch, must never be flashed
+or injected into a guest, and is not bootable on silicon. Injected code is
+development-only ([`docs/12`](../debug/risc-v-gsp-rm-vm/docs/12-INJECTION-SDK.md) §1);
+productization goes through `open-iov-vfio` + physical confirmation, never
+through a simulator package.
+
+Local-only, budgeted, session-recorded; never touches `/dev`, a module, an
+mdev, a guest or a GPU. No confidentiality, isolation or timing claim derives
+from it; distribution follows per-file licences ([`SBOM.md`](../SBOM.md),
+[`legal-and-licensing.md`](09-reference/legal-and-licensing.md)).
+
+## 8. Frontier capabilities
+
+Everything here is **future** and off the critical path. Each capability enters
+the §2 process only when its near-term need is real and its gate is passed;
+details are tracked in the companion [`future-eng-ops.md`](future-eng-ops.md)
+(same directory) and
+[`GREEN-EYE-ROADMAP.md`](../debug/risc-v-gsp-rm-vm/GREEN-EYE-ROADMAP.md) H5/H6.
+Deferred context: [`10-post-1.0/experimental-host-acceleration.md`](10-post-1.0/experimental-host-acceleration.md).
+
+| Frontier capability | Near-term need | Precondition / gate | Evaluation criterion (if funded) |
+|---|---|---|---|
+| SIMD cores and CUDA kernel execution on device | Execute a real kernel launch path (`vectorAdd`-class) in simulation before the physical backend is finished | H1/H2 stable; injection ABI usable for an engine model | kernel executes within an instruction budget; output matches the native/golden result byte-for-byte; deterministic replay |
+| FPGA soft-processor simulation | A cycle-accuracy question worth funding (timing/security/interoperability claim needing silicon-equivalent behaviour) | H3 validated (native parity) and H2 device model proven | the same `Session` scripts run against the RTL backend; execution parity with native; cycle counts recorded as a distinct event class, docs/06 fidelity revisited |
+| Accurate Verilog recreation of all components the GSP interacts with, plus the NVRISC environment and intra/inter-micro-controller interconnects | Boot the real image with silicon-equivalent register, interrupt and mailbox semantics; exercise PMU/SEC2/ACR cross-controller boot flows | H2/H5 proven, validated device map, stable `Backend` abstraction | firmware reaches its message loop in the soft environment; M1–M6 improve over the QEMU/native baseline; one cross-controller exchange reproduced and compared with a physical capture |
+
+## 9. Review discipline
+
+**Adversarial-review loop.** (1) The author proposes a change with its
+revision note (§9.3). (2) A second person or agent attacks it: broken links,
+unverifiable claims, thresholds without metrics, tools without a scenario,
+missing register rows, scope creep into the manuals. (3) The attacker
+reproduces one item end-to-end (link check plus one claimed command);
+"looks fine" is not a review. (4) The author resolves findings; the reviewer
+signs the note with the evidence used.
+
+**Proportionate review (no needless overhead).**
+
+| Change class | Examples | Required review |
+|---|---|---|
+| A — editorial | wording, formatting, link fixes | `python3 scripts/check-docs-links.py` from the repo root; no design review |
+| B — decision | admission, threshold, register, retire changes | one adversarial reviewer + evidence named in the note |
+| C — scope/security | out-of-scope changes, gates, packaging caveats | maintainer sign-off + updated admission records |
+
+Cadence: review at each tool audit
+([`27-TOOL-AUDIT.md`](../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md)) and
+after any firmware anchor change. Expired pilots must appear in the register;
+an empty register of decisions means this document is not being used.
+
+**Revision notes.** Every change records purpose, scope and benefit; a change
+that cannot state a benefit is rejected as overhead.
+
+| Date | Revision | Purpose / scope | Benefit | Evidence |
+|---|---|---|---|---|
+| 2026-10-02 | v1 | Establish the admission rule, value axes, decision framework, workflow maps, automation, packaging, frontier gates and review discipline | Tooling decisions become auditable, measurable and reversible; no tool without an evaluation | This document plus linked tool docs and the [`docs/25`](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md) baselines |
+| 2026-10-02 | v2 (Part II) | Add the live physical-backend operations: hard rules and scope, build/load order and the full `gsp_phys_*` parameter matrix, the routine supervised cycle with its clean-release checklist, the debugfs inventory and `gsp-phys-state` reference, the physical-backend scenario catalogue and hazard register, the OODA loop and the work-context/porting map | Physical-backend execution becomes auditable and fail-safe: one checklist, a named signal per failure, and no retry of a wedged path | Kernel sources cited per parameter and debugfs file; the current frontier in this document and [`STATE-CURRENT.md`](STATE-CURRENT.md) |
+
+```bash
+python3 scripts/check-docs-links.py                       # repo root
+cd debug/risc-v-gsp-rm-vm && make demo sdk && make test   # tool claims
+./greeneye.sh info                                        # environment
+```
+
+---
+
+# Part II — Live physical-backend operations
+
+> **Scope.** This part is the operating manual for cycles against the **live
+> physical backend**: the module-owned physical GPU running its real GSP-RM
+> session (the chosen execution mechanism in
+> [`03-architecture/08`](03-architecture/08-physical-gpu-integration.md), wired
+> toward the guest plane in
+> [`03-architecture/11`](03-architecture/11-physical-backend-guest-wiring.md)).
+> Part I (§§1–9) governs tooling admission, value measurement and retirement.
+> Defensive/incident scenarios are owned by [`RUNBOOK.md`](RUNBOOK.md) and are
+> cross-linked, not duplicated, here; the current frontier is
+> [`STATE-CURRENT.md`](STATE-CURRENT.md); the known unknowns are
+> [`04-implementation/01`](04-implementation/01-known-knowns-and-unknowns.md).
+> Everything in this part is supervised, bounded and fail-safe-first; nothing
+> here authorises unattended hardware work.
+
+## 10. Live physical-backend operations
+
+### 10.1 Hard rules and scope
+
+These apply to every physical cycle; where a step below conflicts with them,
+the rules win.
+
+| # | Rule | Consequence / practice |
+|---|---|---|
+| 1 | **Supervised only, one GPU cycle at a time.** Never unattended, never parallel with other host work, never as a background agent job. | A named human is present for the cycle; the pre-flight and clean-release checklists (§10.3) are completed for each single attempt. |
+| 2 | **Reboot is gated by the written approval policy.** A reboot request MUST first record, in the project's live working record, the reason and the exact post-reboot plan; only then is a named human asked to approve. | No reboot while other work is running; the device is left untouched in the meantime; an unapproved reboot is never initiated by an agent. |
+| 3 | **Never enable `gsp_phys_repower`; never re-arm D3cold.** The default 0 latches the pre-claim D0 policy until reboot. | Re-enabling runtime PM/D3cold after a claim can power-gate the board and hard-freeze the host ([`03-architecture/08`](03-architecture/08-physical-gpu-integration.md) §4.2). |
+| 4 | **Never retry a failed mutating RPC.** One attempt per staged step; abort the ladder from the first failure. | Retrying has wedged GSP-RM (channel-flag and context-promote incidents); `UNLOADING_GUEST_DRIVER` + `rmmod` and, if needed, an approved reboot remain the only recovery. |
+| 5 | **Stop on hard signatures.** `board unresponsive`, `refusing claim`, `do not retry in this boot`, any panic/oops. | Do not write the device again in that boot; keep the locks; request reboot approval per rule 2. |
+| 6 | **Verify module identity before and after a cycle.** Record the built module's sha256 (and `srcversion`) before load and re-read it after unload. | A hash mismatch invalidates the cycle's evidence; stop and rebuild from the reviewed source. |
+| 7 | **A wedged device is left untouched for the reboot.** No `rmmod` retry loops, no bind/unbind experiments, no power-policy pokes. | The fail-safe (no rebind, D0 locks + runtime reference held, WPR2 up) is the designed resting state. |
+| 8 | **Scope.** This part covers the claim/boot, rm-call ladder (host-side real submission), guest-memory forwarding, relay/fault-relay and release paths. Guest-facing emulation debugging is [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md); firmware runtime study is [green-eye](05-operations/03-risc-v-gsp-rm-vm.md). | All other physical work belongs to the scenario owner named in §14. |
+
+### 10.2 Build and load order, and the module-parameter matrix
+
+**Build order (physical-backend work needs the kernel module only):**
+
+1. Build `open-iov-vfio.ko` first; it generates the `Module.symvers` the
+   optional OpenRM modules link against. `scripts/stack_manager.sh
+   build-kernel`.
+2. Build the optional OpenRM `nvidia.ko` with
+   `KBUILD_EXTRA_SYMBOLS` only if that management path is in use; the normal
+   consent path runs the distribution driver for the physical GPU and the
+   emulation module independently (`scripts/stack_manager.sh build-nvidia`).
+3. Build userspace (`libiov`/examples) independently
+   (`scripts/stack_manager.sh build-userspace`).
+4. Stage the GSP boot image into the firmware search path
+   (`scripts/stack_manager.sh install-firmware`).
+
+**Load order:**
+
+1. Prerequisites: `vfio`, `vfio_iommu_type1`, `vfio_pci`, `mdev`
+   (`scripts/stack_manager.sh load-prereqs`).
+2. `open-iov-vfio.ko` with the exact reviewed parameter set
+   (`scripts/stack_manager.sh load-open-iov`). The backend claim is
+   **module-owned** at load with `gsp_phys_gpu=1 gsp_phys_hazard_ack=1`, or
+   **mdev-owned** and deferred to the first mdev create with
+   `gsp_phys_mdev_owned=1` (fail-open).
+3. Record the loaded module identity and the starting state (§10.3 step 0).
+
+**Module-parameter matrix.** Every current `gsp_phys_*` parameter is listed
+below, verified against the kernel sources on the 2026-10-02 tree. The exact
+firmware image name used by `gsp_phys_fw` is deliberately not repeated here
+(it is the distribution driver's Turing-generation artifact); the source is
+authoritative for the literal default.
+
+| Parameter | Default | Effect | Gate / dependency |
+|---|---|---|---|
+| `gsp_phys_gpu` | 0 | Claim the physical GPU at module load: bind `open-iov-gsp-phys`, map BARs, stage firmware and run the requested stages; module-owned claim is released only at `rmmod`. | Refuses without `gsp_phys_hazard_ack=1`; refuses if the GPU or its root port is already runtime-suspended. With `gsp_phys_mdev_owned=1` it only arms the deferred path. |
+| `gsp_phys_hazard_ack` | 0 | Acknowledges the unbind/rebind host-freeze hazard; second gate for every claim. | Required with `gsp_phys_gpu=1`; without it the claim logs `REFUSED` and returns `-EPERM`. |
+| `gsp_phys_boot` | 0 | Full real-GSP boot: FWSEC → Booter Load → RISC-V start → RM-init poll (device writes). | Requires a claim; implies the layout stage; mutually exclusive with `gsp_phys_boot_dryrun`; FWSEC-only is refused because that carve is not host-recoverable. |
+| `gsp_phys_boot_dryrun` | 0 | Stage-3d host-memory staging rehearsal (radix3, queues, boot args) with no BAR0 writes and no falcon resets. | Requires a claim; mutually exclusive with `gsp_phys_boot`. |
+| `gsp_phys_layout` | 0 | Allocate the boot-argument pages and compute/log the WPR/FB layout; the only device access is a read-only video-memory-size sample. | Requires a claim; implied by the boot and dry-run paths. |
+| `gsp_phys_nocat` | 0 | Log every NOCAT journal record (function `0x1020`) individually during the RM-init poll; the unique-record summary is always accumulated and dumped when the poll ends. | Boot path only; diagnostic. |
+| `gsp_phys_fw` | distribution GSP image | GSP ELF the backend stages; override with `<name>` relative to the firmware search path. | Requires a claim; a missing/invalid image leaves the backend at ownership rung 1. |
+| `gsp_phys_mdev_owned` | 0 | Deferred, refcounted lifecycle: first mdev create claims+boots asynchronously, last remove releases; every failure is fail-open to the emulation path. | Requires `gsp_phys_gpu=1` + `gsp_phys_hazard_ack=1`; a module-owned claim stays up until `rmmod`. |
+| `gsp_phys_fwsec` | 0 | Run the FWSEC/FRTS secure-verification carve (device writes). | Requires the layout to have succeeded; **refused unless `gsp_phys_boot=1`**. |
+| `gsp_phys_fwsec_dryrun` | 0 | Read-only validation of the VBIOS/BIT/FWSEC parse and the staging/placement math. | Requires a claim; no device writes. |
+| `gsp_phys_fwsec_verbose` | 0 | Extra bounded read-only FWSEC kick state dump (falcon regs, scratch, DMEM/IMEM readback) after a successful halt. | Diagnostic; failure path always dumps. |
+| `gsp_phys_static_live` | 0 | Overlay the live GPU name/short-name/VBIOS identity from the READY snapshot onto the emulated `GET_GSP_STATIC_INFO` reply (bounded 2 s wait, fail-open). | Requires a READY backend snapshot; other static fields stay emulated. |
+| `gsp_phys_ctrl_forward` | 0 | Forward a whitelist of side-effect-free `GSP_RM_CONTROL` queries (bus BAR info, bus info, FB info, GMMU static info) with the adopted server handles; every miss falls back to the emulated handler. | Requires READY; fail-open; per-cmd forwarded/fallback counters. |
+| `gsp_phys_work_forward` | 0 | Request one real CE shadow copy per eligible decoded guest push on the ladder channel; the software release path stays authoritative. | Requires a live claim + RM init-done + a built ladder channel; all CE forwarding paths refuse cleanly on a GR ladder. |
+| `gsp_phys_guest_mem` | 0 | Let the ladder's guest triggers (6/7) and the forwarding path operate on a pinned guest GPA resolved through the guest-RAM backing file. | Triggers require READY, ladder stage ≥ 3, an identity DMA domain, an active guest-RAM file and a pinned GPA (6/7). |
+| `gsp_phys_work_defer_release` | 0 | Defer a forwarded push's guest-visible release until the hardware copy completes (bounded 500 ms), then apply it from the service thread. | Requires `gsp_phys_work_forward=1` + `gsp_phys_guest_mem=1`; with 0 the release applies inline. |
+| `gsp_phys_work_hw_primary` | 0 | Skip the software copy for a forwarded guest CE push (hardware authors the data); one software fallback copy on any hardware/queue failure. | Requires `gsp_phys_work_forward=1` + `gsp_phys_guest_mem=1` + `gsp_phys_work_defer_release=1`. |
+| `gsp_phys_relay` | 0 | Let the engine author the selected guest-visible semaphore release into the guest page (third PTE on the same fresh line); the hold verifies the word and skips that record's software apply. | Requires `gsp_phys_work_forward=1` + `gsp_phys_guest_mem=1` + `gsp_phys_work_defer_release=1`; default off pending mismatch triage. |
+| `gsp_phys_fault_relay` | 0 | On a relay timeout, drain the status queue for the engine fault (`0x1006`, corroborated by `0x1020`), match runlistId/chid and write the guest channel notifier + event. | Requires `gsp_phys_relay=1` and the three parameters above. |
+| `gsp_phys_fault_inject` | 0 | Debug: write the relayed pair's source PTE as zero to force an engine fault on the next relay submission; only relayed pairs are affected. | Requires the relay path; default 0 programs the real source HPA. |
+| `gsp_phys_rmcall_engine` | 0 | Ladder engine: 0 = COPY0 (CE copy submission), 1 = GR0 (compute object, BIND engineType 1, GR semaphore release). Snapshotted when stage 3 starts; mid-session changes are ignored. | Requires boot + init-done; CE forwarding paths refuse on a GR ladder. |
+| `gsp_phys_rmcall_gr_promote` | 0 (clamp 0..3) | GR context-promote bisection: **0 never sends `PROMOTE_CTX`** (the ladder stops after the geometry queries, context buffer/PT allocation, PDB mapping and channel/compute alloc, before BIND); 1 = single MAIN entry, initialize/no-nonmapped, size raw+subctx; 2 = two-phase PA-only then VA-only; 3 = single entry with the exact raw size. Nonzero enables can assert GSP-RM (`0x52a8b04`) and wedge the device. | GR ladder only; value snapshotted at stage 3; **requires confirmed reboot readiness** before enabling. |
+| `gsp_phys_repower` | 0 (latched) | Restore the pre-claim runtime-PM/D3cold policy when releasing the GPU; with 0 the D0 locks and runtime reference are deliberately left held until reboot. | **Never enable** (rule 3); re-arming D3cold after a claim can hard-freeze the host. |
+
+The emulation-side `sema_*` parameters (ring discovery, release recovery,
+diagnostic watches, grace windows) are not physical-backend controls; their
+canonical description is [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md)
+§3.2 and [`03-architecture/06`](03-architecture/06-host-module-and-hybrid-model.md).
+
+### 10.3 The routine supervised cycle
+
+One cycle is one attempt; there is no second attempt in the same boot after a
+hard signature.
+
+**Step 0 — pre-flight (before anything is loaded):**
+
+* Host clean: no QEMU process, no mdev under `/sys/bus/mdev/devices/`, the
+  module not loaded, the physical GPU bound to the distribution driver.
+* Record `boot_id`, pstore contents, module sha256/`srcversion`, and the
+  planned parameter set in the cycle log.
+* Confirm supervision (present human), a written recovery plan, reboot
+  readiness per rule 2, and that other host work is finished.
+
+**Step 1 — build and identity check.** Rebuild the reviewed source; record the
+module hash; never run a cycle against a stale or unknown module.
+
+**Step 2 — load.** Prerequisites, then `open-iov-vfio.ko` with the reviewed
+parameters. Module-owned: the claim and requested stages run at load.
+Mdev-owned: load, then create the mdev (the async claim/boot starts) only when
+no VM is live.
+
+**Step 3 — claim/boot prerequisites.** In the cycle log confirm: `claimed`,
+`PCI_COMMAND` bus-master on, BAR lengths, `BOOT0`/`BOOT1`, both falcons, WPR2
+down at entry; the D0 settle check passed (claim refuses otherwise); and after
+the boot: `stage: primed=1 kicked=1 ran=1 riscv_active=1 rm_init_done=1`, WPR2
+up, `scratch14` hand-off bits advancing. Any claim refusal (`refusing claim`,
+`board unresponsive`) ends the cycle (rule 5).
+
+**Step 4 — ladder (host-side real submission).** Write `gsp-phys-rm-call`
+values 1..4 cumulatively; the ladder remembers the highest completed stage:
+
+| Write | Stage | Green signal |
+|---|---|---|
+| 1 | root/device/subdevice objects | stage 1 complete, handles logged (`hClient`/`hDevice`/`hSubdevice`). |
+| 2 | memory registration | stage 2 complete. |
+| 3 | VAS + TSG + channel (COPY0 variant walk; GR0 single attempt) | stage 3 complete (`variant`, `hChannel`, GR: `hGr`, `rm_call_ctx`). |
+| 4 | real submission: CE copy or GR semaphore release | `REAL COPY OK` (CE) / `REAL GR SEM OK` (GR); trigger 9 re-runs only the GR release. |
+| 5 | adopt static-config handles as stage 1 | stage 5 complete; a later 4 runs stages 2–4 with adopted handles. |
+| 6 | guest write direction (W2G) on the pinned GPA | `GUEST W2G COPY OK`. |
+| 7 | guest read direction (G2H) on the pinned GPA | `GUEST G2H COPY OK`. |
+| 8 | fresh-line PTE self-test (no GPA) | `GUEST PTE SELFTEST OK` / `selftest_ok=1`. |
+| 9 | GR semaphore re-run on a built GR ladder | `REAL GR SEM OK` or a decoded failure; no retry. |
+
+6–8 require `gsp_phys_guest_mem=1`, a READY backend and ladder stage ≥ 3;
+6/7 additionally require the GPA pinned via `gsp-phys-guest-gpa`. On the first
+failed stage: stop, drain the status queue, and end the ladder (rule 4).
+
+**Step 5 — guest attach (only if the rung needs it).** Start the VM with a
+file-backed shared guest RAM region (`share=yes, prealloc=on`), attach the
+mdev, configure `gsp-phys-guest-ram <path> <gpa_base> <offset> <size>`, pin
+the GPA, and run the driver-API smoke; the guest must stay green (`ALL OK`
+64/64) and any forwarded counters must be explained. Remove the mdev before
+`rmmod`; never unload with a live VM.
+
+**Step 6 — evidence capture.** Keep with the cycle: parameter set, module
+identity, boot id, `gsp-phys-state`, ladder traces, `gsp-phys-status-drain`
+records, NOCAT/`0x1006` decodes, `gsp-phys-log-dump`, and the clean-release
+checklist below. Evidence is property of the change that follows.
+
+**Step 7 — teardown/unload.** Remove mdevs/stop the VM, then `rmmod`
+(`scripts/stack_manager.sh unload`). The release sends
+`UNLOADING_GUEST_DRIVER` once (only after a boot), quiesces the falcon, runs
+the FWSEC teardown, frees staging, and rebinds the distribution driver only
+after a WPR2-down and board-liveness check, with the D0 locks held through a
+bounded post-rebind readiness gate.
+
+**Mandatory clean-release checklist** (every line must hold; any failure stops
+the cycle and invokes the fail-safe):
+
+| Check | Green value | If it fails |
+|---|---|---|
+| Boot id | unchanged from pre-flight | the device was power-cycled mid-cycle: discard in-window counters and rebuild the plan; expect a fresh scratch pool at the next boot. |
+| Module identity | sha256/`srcversion` identical before/after | wrong build ran: stop, rebuild, restart the cycle (rule 6). |
+| Protected region | `wpr2: ... state=down`; GSP falcon quiesced | fail-safe: **no rebind**, D0 locks + runtime reference stay held, WPR2 up; reboot approval per rule 2; never retry in this boot. |
+| Unload RPC | `rm_unload_submitted=1`, processor suspended (mailbox `0x80000000`) | wedged server; fail-safe as above (the release log says `teardown did not clear WPR2` / `refusing the hand-back`). |
+| Driver rebind | `released <bdf>; rebound to nvidia (rc=1)` | rebind failure/timeout: locks stay held; the module logs the manual `modprobe nvidia` path; reboot is the clean recovery. |
+| Power-domain locks | exactly 5 `power domain left LOCKED (gsp_phys_repower=0):` lines on this host (GPU + board functions + root port) | wrong count: stop; audit the locks before any further cycle. |
+| Rail cycles in window | 0 | any rail cycle rewrote the power risk: treat the cycle as unsafe and stop. |
+| pstore / leftovers | pstore empty; no QEMU, no mdev, guest-RAM file released | a crash record or leaked state exists: capture it, clean with `stack_manager.sh cleanup-crash`, and do not start another cycle until the host is clean. |
+
+Fail-safe summary: on any WPR2/unresponsive/rebind failure the release does
+**not** hand the device back, keeps the power-domain locks (and the runtime-PM
+reference) held, and the correct next action is an approved reboot with the
+written post-reboot plan - never a second attempt in the same boot.
+
+### 10.4 debugfs inventory
+
+All files live under the module's debugfs directory
+`/sys/kernel/debug/open-iov-vfio/`. **The directory and every file are
+root-only on this system**: existence and read checks must run as root (a
+permission error can look like an absent file). Every writer is inert unless
+its gates hold, and every accepted write performs a real RPC or device
+action - log it in the cycle record.
+
+| File | Mode | Purpose |
+|---|---|---|
+| `gsp-phys-state` | 0444 | Read-only state snapshot (§10.5): stage/boot state, RM probe/call, GR context, guest copy and relay counters, fault counters, guest-RAM ranges, WPR2/RISCV/falcon/scratch14, queue headers. |
+| `gsp-phys-log-dump` | 0444 | Read-only libos task log buffers (`LOGINIT`, `LOGRM`), raw words plus a decoded view. The raw view is authoritative; decode offline where a script is provided by the evidence set. |
+| `gsp-phys-status-drain` | 0444 | One bounded status-queue drain per read, printing the decoded `0x1006` engine-fault and `0x1020` NOCAT records; advances the host read pointer (normal consumer protocol). |
+| `gsp-phys-unload-rpc` | 0200 | Write `1`: send `UNLOADING_GUEST_DRIVER` once on a live claimed session; every other value is a logged no-op. |
+| `gsp-phys-rm-probe` | 0200 | Write `1`: run one `SET_GUEST_SYSTEM_INFO`/`GET_GSP_STATIC_INFO` probe cycle and cache the static-config snapshot. |
+| `gsp-phys-rm-call` | 0200 | Write `1`..`9`: the ladder stages, handle adoption, guest triggers and GR re-run of §10.3 step 4. |
+| `gsp-phys-guest-gpa` | 0200 | Write `0x<hex>`: pin one 4 KB-aligned guest GPA for triggers 6/7; one GPA per boot, a different GPA is refused. |
+| `gsp-phys-guest-ram` | 0200 | Write `<path> <gpa_base> <offset> <size>`: configure the guest RAM backing file and one GPA window (file path fixed for the module cycle); `-` clears. |
+
+### 10.5 Reading `gsp-phys-state`
+
+The first lines say whether the device snapshot is meaningful at all
+(`inactive: physical GPU not claimed` / `stage-3d staging not primed`); a
+snapshot is only read from a primed, claimed session. Then, in order:
+
+| Line | Fields | Meaning and invariants |
+|---|---|---|
+| `claimed` / `boot_param` / `dryrun_param` | claim and requested-stage flags | `claimed=0` with a live ladder stage means a release already ran: stop reading device state. |
+| `stage:` | `primed`, `kicked`, `ran`, `riscv_active`, `rm_init_done`, `rm_unload_submitted`, `dryrun`, `tx_seq_num` | The boot lifecycle position. READY work needs `primed=1 ran=1 rm_init_done=1`; `rm_unload_submitted=1` means the session is ending and no new forward is legal. |
+| `rm_probe:` | `attempted`, `last_ret`, `last_function`, `last_payload_len` | Static-config probe outcome; function 65 with a nonzero payload is the snapshot source. |
+| `rm_call:` | `ran`, `stage`, `last_stage`, `last_ret`, `rpcs`, `hClient`/`hDevice`/`hSubdevice`/`hVas`/`hTsg`/`hChannel`, `variant`, `engine`, `hGr`, `token`, `notifier`, `base` | Highest completed ladder stage and the object handles; `engine` is the stage-3 snapshot (COPY0/GR0). A GR `hChannel` with `token` but a faulting push is the context-commit wall, not a transport failure. |
+| `rm_call_ctx:` | `prepared`, `valid`, `pa`, `va`, `size`, `pages`, `ptPages`, `gr_promote_param`, `step`, `entrySize` | GR context buffer/PDB; printed only when a GR context was prepared. `gr_promote_param=0` means `PROMOTE_CTX` is never sent; `step` names the last accepted promote step. |
+| `rm_call_guest:` | `gpa`, `hpa`, `g2h_ok`, `w2g_ok`, `selftest_ok`, `gpa_valid`, `phys_domain_identity` | The last guest trigger's translation and proof flags; `phys_domain_identity=1` is the prerequisite for programming host PAs into the engine page tables. |
+| `work_guest:` | `req`, `fallback`, `ok`, `verify_ok`, `verify_fail`, `pool_exhausted`, `refused`, `cand_none`/`cand_sys_sys`/`cand_mixed`/`cand_vid_vid`, `resolve_fail`, `next_line`, `capacity` | Forwarded guest copies. Green run: `verify_fail=0`, `pool_exhausted=0`, `refused=0`, `resolve_fail=0`; `next_line ≤ capacity` (512 lines per boot, never reused). |
+| `work_guard:` | `copy_skipped`, `deferred`, `applied`, `fallback`, `pending`, `overflow`, `engaged`, `timeouts`, `canceled`, `defer_param`, `copy_soft`, `copy_fail`, `hw_primary_param` | Deferred-release hold. **Invariant: `applied + fallback == engaged`.** `pending` is the live hold gauge (should return to 0); `timeouts`/`overflow`/`canceled` must be 0; `copy_soft` counts the designed inline software fallbacks. |
+| `work_relay:` | `requested`, `ok`, `fallback`, `skipped_sw`, `verify_fail`, `verify_transient`, `hold_verify_fail`, `resolve_fail`, `timeouts`, `pool_exhausted`, `refused`, `relay_param` | Engine-authored semaphore relay. Green run: `verify_fail=0`, `hold_verify_fail=0`, `timeouts=0`, `pool_exhausted=0`, `refused=0`, `resolve_fail=0`; `verify_transient` is a retry-cleared mismatch and should be rare and explained. |
+| `faults:` | `seen`, `stale`, `drained`, `wait_unmatched`, `epoch` | Fault observation at the status queue: records seen vs discarded as stale, records drained, unmatched waits, and the epoch used to match a relay submission to the fault it caused. |
+| `fault_relay:` | `matched`, `unmatched`, `delivered`, `no_event`, `late_release`, `relay_param`, `fault_relay_param`, `fault_inject_param` | Guest delivery: `delivered` is a written notifier **and** event; `no_event` means the notifier was written but the auxiliary event post produced no MSI-X; `late_release` means the release landed before the fault could be relayed (no fault is delivered). |
+| `guest_ram:` | `active=1 path=... ranges=...` or `active=0` | Backing-file and GPA-window configuration; `active=0` disables every guest-page path. |
+| `wpr2:` / `riscv:` / `falcon:` / `scratch14:` | `state=up/down`; active bit; both falcons' CPUCTL/MBOX/FALCON_OS; secure-scratch bit fields | The fail-safe state. A green release ends with `state=down`, both falcons quiesced, and the booter-unload gate bits complete. |
+| `cmd_tx` / `stat_tx` | queue header words | Queue producer/consumer cursors; a stalled status read pointer with an advanced command read pointer means "consumed but never replied" (the wedge signature). |
+
+**Relay data-mismatch semantics (fail-closed).** The relay result is verified
+host-side against a source window snapshotted *before* the doorbell. A
+mismatch is retried once after 2 ms: if the second read matches, it is a
+`verify_transient` (the live guest page was mutated under the first compare)
+and the copy stays verified; if it persists it counts `verify_fail`, the
+submission returns an error, and the sema service falls back to the software
+release. The same fallback covers `hold_verify_fail` (the guest semaphore
+re-read did not carry the expected payload). **A software release apply is
+never lost or duplicated**, which is why the relay defaults off until
+`verify_fail` is triaged.
+
+## 11. Debugging scenario catalogue
+
+Physical-backend symptoms, their exact signal, and the bounded response. The
+defensive/new-input scenarios (firmware release, OpenRM build, userspace
+bump, device/ISA/ROM change, protected state, matrix/CI) are owned by
+[`RUNBOOK.md`](RUNBOOK.md) S1–S10 and are not duplicated here.
+
+| Symptom | Exact signal to look for | Decode / likely cause | First diagnostic | Action |
+|---|---|---|---|---|
+| Guest GR push faults | `func=0x1006 ... exceptType=0x20 runlistId=0 chid=1 errString="... intr1 80000000"`; NOCAT `recType=4 errorCode=0x20 faultingEngine="GR"` | `0x20` = robust-channel PBDMA error; `intr1` bit 31 = `CTXNOTVALID`; the external-VAS GR engine context was never committed (`ENG_CTX_VALID=0`), so the guest binds via `PROMOTE_CTX` (`0x2080012b`). CE channels are unaffected. | `gsp-phys-status-drain`; `gsp-phys-state` `rm_call`/`rm_call_ctx` | Decode first, no retry. Only a proven promote shape commits the context; keep the client promote path gated by `gsp_phys_rmcall_gr_promote=0` until a variant is proven. |
+| `PROMOTE_CTX` wedge | `gsp-nocat: rec seq=N recType=5 errorCode=0x52a8b04 source="ASSERT" faultingEngine=""`, then no fn76 reply (timeout `-110`); status ring stops, server command read pointer advanced | A host/server-RM assert in the promote path - not an engine fault (no `0x1006`). The server consumed the command and wrote no reply. Teardown cannot clear WPR2 (its unlock is fuse-locked on this part). | `gsp-phys-state` (`cmd_tx`/`stat_tx`, `wpr2`), release log | Ladder aborts by design; **do not retry**. Fail-safe holds the device; reboot with written approval (rule 2); bisect exactly one promote variant per cycle after reboot. |
+| WPR2 / clean-release failure | `teardown did not clear WPR2 (HI=0x...)`; `release: WPR2 still up after teardown ... refusing the hand-back`; `NOT rebinding ... power domain stays LOCKED` | The booter-unload gate bits were never completed by the wedge, or the board stopped answering; the host-side unlock is fuse-locked. | `gsp-phys-state` `wpr2`, `scratch14`, `stage:` | Fail-safe is the designed state: no rebind, locks held, WPR2 up. Leave the device; reboot with approval. |
+| Module unload hang | `rmmod` does not return; `ps` shows the emulation/relay kthreads or work items; mdev still present | Teardown ordering (kthread/workqueue) or an open mdev/VM; the emulation-side cleanup applies. | `ps -e \| grep gsp_emu`; mdev list; `dmesg` tail | Remove mdev/stop VM first; `scripts/stack_manager.sh cleanup-crash`; reboot with approval if it persists (rule 5). |
+| Guest `cuInit` wedge / guest flake | guest `cuInit rc=999`, `rmapiControlCacheSet ... NV_ERR_INVALID_PARAMETER`, or a host `gsp-phys-fwsec: unexpected WPR2 already up`; a one-off guest Oops in `free_swap_cache` cleared on retry | On the emulated path this is a known guest-side flake; the WPR2-up line is the FWSEC refusal on a board already in a bad state. | guest dmesg; host `gsp-phys-state` `wpr2`; retry the guest once | If only the guest flaked, restart the guest once. If `unexpected WPR2 already up` appears, stop the cycle and inspect state; never retry FWSEC (rule 4/5). |
+| GSP boot failure | `gsp-phys-fwsec: staging not visible before kick (DMEM0 ... IMEM ...)` then `FWSEC/FRTS execution failed: -5`; historical `GSP_INIT_DONE rpc_result=0x59` | Staging DMA did not land (async mdev-owned FWSEC path) or, historically, the pre-boot handshake/identity payload was wrong (`0x59` = operating-system error). | `gsp-phys-state` `stage:`, `gsp-phys-log-dump` | Use the module-owned claim path (proven); the async FWSEC defect is fail-open and off the critical path. Do not repeat a failed boot in the same boot after a hard signature. |
+| MMU/PTE faults and the fetch-line rule | engine copy times out with no semaphore; `GUEST CE COPY FAILED: scratch-PT pool exhausted ... capacity=512`; re-translated GPA yields a different HPA | The GMMU caches PTE fetches in 128-byte lines and there is no invalidate: a runtime PTE write into an already-fetched line is invisible. A guest page's HPA can change between translations. | `rm-call: guest HPA changed: gpa=... was ... now ...`; `work_guest` `pool_exhausted`/`next_line` | Always use a fresh never-reused line per pair; pre-build guest-window slots before engine access; pin translated pages; treat pool exhaustion as end-of-budget, not a bug. |
+| PBDMA method/class errors | `0x1006` with `intr0` nonzero (a method/class was rejected) versus `intr1` bit 31 (`CTXNOTVALID`); NOCAT `faultingEngine="GR"`; errString `"Channel ID ... intr0 ... intr1 ..."` | Wrong method encoding, class/subchannel or channel state; `intr0 != 0` means the engine rejected the method, so look at the pushed method block, not the context only. | `gsp-phys-status-drain`; compare the method encoding with the engine's class | Fix the method/class encoding; no retry against the same wedged channel; re-run from a fresh ladder. |
+| Relay symptoms | `GUEST CE RELAY FAILED: semaphore released ... does not match ... (retried after 2 ms)`; `verify_fail`/`verify_transient`/`hold_verify_fail`; `fallback`/`skipped_sw`; `pool_exhausted`; `guest HPA changed` | One persistent data mismatch (shadow-page/concurrent-guest race or decode offset edge) is the open residual; anything else is the designed fail-closed fallback to the software apply. | `work_relay` line; the relay failure log includes sem target/payload/poll, first differing byte, live/snapshot dumps, `src_mutated`, `snap_vs_dst` | Keep default off until the mismatch is triaged. Abort the feature (`gsp_phys_relay=0`, then `gsp_phys_work_forward=0`) if `timeouts>0` or `verify_fail>0` ever appears. |
+| Fault-relay symptoms | `fault_relay delivered=1` versus `no_event=1`; `late_release`; `matched`/`unmatched` by runlistId/chid; `faults epoch` | `delivered` = notifier written and event posted; `no_event` = notifier written but the auxiliary event post found no event object (no MSI-X reaches the guest); `late_release` = release landed first, so no fault is delivered (by design). | `fault_relay` line; `gsp-phys-status-drain` | A `no_event` is an event-path problem, not a fault-decode problem: fix the channel notifier/event wiring. Use `gsp_phys_fault_inject=1` to reproduce the path on demand. |
+| Guest driver probe failure | guest sees no PCI device / BAR map fails / probe hangs; host `board unresponsive`, mdev `-EUSERS`; missing `SET_IRQS wired` | Config space, BAR mapping, MSI-X wiring or mdev state on the emulated path - not the physical claim. | guest `lspci -vvv`; host `05-operations/02` §5.1 taxonomy | Follow the taxonomy in [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §5.1; never start a physical claim to debug a guest-side probe failure. |
+
+## 12. Hazard register (crash causes and debugging side effects)
+
+| Hazard | Mechanism | Mitigation / practice |
+|---|---|---|
+| D3cold re-arm / power-gate host freeze | Releasing the D0 policy or re-enabling runtime PM after a claim lets the root port enter D3; the platform can cut the board rail, and the rebind can fail with a host freeze minutes later (no oops, pstore empty). | `gsp_phys_repower=0` (latched) is the default; the D0 locks cover GPU + board functions + root port; the post-rebind readiness gate must pass before anything is released; never unbind/rebind outside the supervised claim; pstore armed each cycle. |
+| Failed mutating RPC wedges GSP-RM | One bad mutating operation (channel-flag incident; `PROMOTE_CTX` assert `0x52a8b04`) leaves the server unable to reply or suspend while WPR2 stays up. | One mutating attempt per staged step; never retry a failed mutating RPC; abort the ladder from the first failure; the release fail-safe keeps the locks and refuses the hand-back. |
+| debugfs write side effects | Every 0200 file performs a real RPC/device action when its gates hold; a "test" write changes device or guest state. | Root-only; read `gsp-phys-state` and record the intended effect before any write; one write per action; no trigger writes outside a supervised cycle. |
+| `pkill -f qemu-system-x86_64` self-match | The pattern also matches the invoking shell's own command line, killing the agent/session instead of the VM. | Use bracket patterns (`pgrep/pkill -f '[q]emu-system-x86_64'`) or explicit PIDs; stop VMs through `scripts/vgpu_vm.py`. |
+| Root-only debugfs | The whole `open-iov-vfio` debugfs directory and its files are root-only; a non-root existence check reports absence. | Run existence/read checks under root; treat "missing" as a permission question first. |
+| Module reload/unload hazards | Kthread/workqueue teardown can hang; the IOMMU identity-domain property and the guest-RAM file must not outlive the session; unloading with a live mdev/VM corrupts the teardown order. | Remove mdevs and stop VMs first; `gsp_sema_service_work_quiesce()` before staging is freed (release does this); use `scripts/stack_manager.sh unload`/`cleanup-crash`; audit leftovers after every unload. |
+| pstore/journal persistence | A freeze leaves no oops but usually a persistent pstore record; the journal stops. | Check `/sys/fs/pstore` before and after every cycle; never clear it; capture it before any reboot or retry - it is the only post-freeze evidence. |
+| Guest-RAM shared-file requirement | The engine can reach guest pages only through the QEMU file-backed shared region; a non-shared or non-preallocated RAM backend has no host page cache to translate. | Start QEMU with `memory-backend-file,share=yes,prealloc=on`; configure the file via `gsp-phys-guest-ram`; one file per module cycle; translation is gated on an open mdev and an active range. |
+| Release-before-verify race | A hardware copy that times out cannot be canceled, so a late engine write is theoretically possible after the software fallback already applied. | The deferred-release hold withholds the guest release until completion or the bounded window; the software apply is never lost; if `timeouts`/`verify_fail` appears, abort the feature rather than tune around it. |
+| Single-inflight limit / deferred-release hold | One forwarded request is in flight at a time; while a hold waits, new forwards are refused and apply their releases inline. | Designed fail-closed: `copy_soft` fallback, 500 ms cap, `pending` returns to 0. Do not deepen the queue without a measured need. |
+| Scratch-pool exhaustion | 512 fresh 128-byte PTE lines per boot and no reuse; each copy pair consumes one whole line. | Size a cycle's guest-copy workload to the pool; treat `pool_exhausted` as the designed end-of-budget, then reload/reboot for a fresh pool. |
+
+## 13. The physical-backend OODA loop
+
+| Stage | What to do | Exit criterion |
+|---|---|---|
+| **Observe** | Collect: the `gsp-phys-state` snapshot (§10.5), ladder traces and stage numbers, `gsp-phys-status-drain` records, decoded `0x1006`/`0x1020` faults, `gsp-phys-log-dump`, the clean-release checklist, module sha/boot id, and the evidence set. | The cycle's question can be stated in one discriminating sentence with its parameter set and run identity. |
+| **Orient** | Separate static from dynamic deltas. Classify failures: **engine fault** (`0x1006` + NOCAT `faultingEngine` set), **host/server assert** (NOCAT `recType=5`, `faultingEngine=""`, no `0x1006`), **transport** (timeout `-110`, status read pointer stalled while the command pointer advanced), or **data-verify** (`verify_fail`/`hold_verify_fail`). Match relay faults by `epoch` plus runlistId/chid and honour the external-VAS rule (`ENG_CTX_VALID` must be committed by a proven promote before a GR push can complete). | Every delta has exactly one class, or is recorded as an open gap. |
+| **Decide** | Choose one: fix / observe more / extend tooling / record an accepted gap. Bisect with exactly one variable per cycle (parameter variant, one promote shape, one method encoding); never retry a wedged path, and never decide "fix" from a single run or a static reading. | The decision names the evidence and the single changed variable. |
+| **Act** | Land the code change with the evidence set attached; commit only after a live green result on the same stimulus; fail-safe first (the release must refuse the hand-back before any optimisation). | Same stimulus is green, the clean-release checklist holds, and the change carries its evidence. |
+| **Stop rules** | Hard signatures (rule 5), the timebox (three observe/compare iterations without a discriminating delta = stop and write down what is missing), the reboot policy (rule 2), and **no unattended runs** (rule 1). | Nothing physical runs past a stop rule; the state and hypotheses are recorded for the next supervised window. |
+
+## 14. Work-context map and porting flows
+
+### 14.1 Which document owns which debugging context
+
+| Debugging context | Owning document |
+|---|---|
+| Emulated-driver debugging (guest driver vs the emulated GSP-RM) | [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §1–3, §5 (observability, captures, gdbstub, failure taxonomy). |
+| Firmware runtime (GSP/uproc in a RISC-V VM, hooks/watchpoints/sessions) | [`05-operations/03`](05-operations/03-risc-v-gsp-rm-vm.md) and the [green-eye user guide](05-operations/04-green-eye-user-guide.md) (experimental). |
+| Live physical backend (claim/boot, ladder, forwarding, relay, release) | **This Part II.** |
+| Portability (firmware, hardware, driver version, userspace version) | [`06-portability/*`](06-portability/hardware-software-matrix.md) and [`RUNBOOK.md`](RUNBOOK.md) S1–S10. |
+| Performance and correctness validation | [`05-operations/01`](05-operations/01-performance-correctness-security.md) (evaluation ladder, metrics, A/B method). |
+| Architecture of the backend and the guest wiring | [`03-architecture/08`](03-architecture/08-physical-gpu-integration.md), [`03-architecture/11`](03-architecture/11-physical-backend-guest-wiring.md). |
+| Current frontier and dated status | [`STATE-CURRENT.md`](STATE-CURRENT.md). |
+
+### 14.2 Triage flow for a CUDA/userspace symptom
+
+1. **Guest diagnostics first** ([`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §2–3): guest dmesg, PCI config, `nvidia-smi`, the driver-API smoke result, and the exact failing call.
+2. **Classify**: emulated reply (guest failed against the emulation plane) vs forwarded reply (`gsp_phys_static_live`/`gsp_phys_ctrl_forward`/`gsp_phys_work_forward` counters moved) vs engine fault (`0x1006`/NOCAT) vs guest-side flake (WPR2 untouched).
+3. **Check the owning state/log**: for forwarded paths `work_guest`/`work_guard`/`work_relay`/`fault_relay` in `gsp-phys-state`; for faults `gsp-phys-status-drain`; for boot/claim `stage:`/`wpr2:`; for emulation-only symptoms the emulation trace.
+4. **Next step**: emulation handler change under [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) discipline; forwarding parameter/ladder re-run under §10.3; fault decode and no-retry path under §11; guest-flake restart once. A physical claim is never the first response to a guest-side probe failure.
+
+### 14.3 Porting flows
+
+| New input | What changes | What must be validated | Procedure |
+|---|---|---|---|
+| Firmware image | The staged ELF, its signature section and the boot handshake; the static-config snapshot content. | ELF64 RISC-V shape and pairing; FWSEC/booter/RISC-V stages; RM init-done `rpc_result=0`; full static-config reply; one supervised cycle before any emulation overlay change. | [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §4.1; [`06-portability`](06-portability/hardware-software-matrix.md); [`RUNBOOK.md`](RUNBOOK.md) S1. |
+| GPU hardware | Device profile (PCI IDs, BAR geometry, real FB size), power-domain topology (lock count), engine inventory, identity overlay. | Claim/D0 lock and clean release on the new board; boot layout; engine capabilities; a supervised cycle at the target rung; matrix entry only once earned. | [`06-portability`](06-portability/hardware-software-matrix.md); [`RUNBOOK.md`](RUNBOOK.md) S5. |
+| Kernel driver version | The distribution driver that is unbound/rebound; module build against the kernel; OpenRM build glue if that path is used (never a runtime dependency). | Build order and symbols; claim/release hand-back; no regression on the module-owned session. | [`RUNBOOK.md`](RUNBOOK.md) S2; [`03-architecture/06`](03-architecture/06-host-module-and-hybrid-model.md) §7. |
+| CUDA/userspace version | Guest-visible controls/replies and the userspace crash surface; the emulation's reply fidelity. | Driver-API smoke `ALL OK` 64/64; control/ioctl inventory diff explained; A/B on the same anchor. | [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §4.2; userspace CUDA playbook; [`RUNBOOK.md`](RUNBOOK.md) S4. |
+
+### 14.4 Current frontier (2026-10-02)
+
+Guest CE copies are **hardware-authored with verified data** and the relay and
+fault-relay paths are in place, all default off (the deferred-release and
+hardware-primary invariants hold; one relay data mismatch is the open
+triage residual). The host GR ladder reaches the **context-promote wall**:
+the GR0 channel and compute object allocate and the engine path is encoded,
+but the external-VAS context is never committed, and the first host promote
+attempt asserted GSP-RM (`0x52a8b04`) without a reply and left the protected
+region up. The device is therefore parked in the **designed locked
+fail-safe** (module unloaded, GPU unbound, D0 locks held, WPR2 up) awaiting a
+supervised reboot; the promote path stays gated behind
+`gsp_phys_rmcall_gr_promote=0` until a bisected variant is proven. Live detail
+and evidence pointers: [`STATE-CURRENT.md`](STATE-CURRENT.md).
+
+<!-- change (2026-10-02): correct the porting/packaging tool section from
+planned interfaces that no longer match the landed scaffolds (docs 28/29) to
+the actual flags and outputs, link the device-validation sequence and
+future-eng-ops admission evaluations, and mark the scaffolds unadmitted;
+scope: eng-ops §3 axis 13, §6, §7, §8 only; benefit: readers no longer build
+against non-existent flags or treat unadmitted scripts as routine pipeline. -->