path: root/docs/eng-ops.md

# Engineering Operations: tooling admission, workflows and retirement

> How Open-IOV decides which engineering tools it uses, extends and retires,
> and how it measures whether a tool pays for its maintenance. Maintainers
> decide and record; agents execute bounded loops and produce evidence.
> Subsystem manuals are linked, not duplicated.

## 1. Purpose and scope

**Who uses it.** Maintainers decide, record and review; agents (MCP, CI, cron)
execute bounded loops and produce sessions, reports and bundles. The same rule
binds both: evidence or it did not happen ([`onboarding-and-standards.md`](07-developers/onboarding-and-standards.md) §5, [`15-AGENTIC-GSP-IDE.md`](../debug/risc-v-gsp-rm-vm/docs/15-AGENTIC-GSP-IDE.md) §5).

| Governed here | Owned elsewhere (authoritative) |
|---|---|
| Admission, extension, redesign, trialing and retirement of tools | Architecture ([`03-architecture/01-system-overview.md`](03-architecture/01-system-overview.md)); tool manuals: CUDA ([`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md)), green-eye ([`26-ARCHITECTURE.md`](../debug/risc-v-gsp-rm-vm/docs/26-ARCHITECTURE.md), [user guide](../debug/risc-v-gsp-rm-vm/docs/02-USAGE.md), [scripting](../debug/risc-v-gsp-rm-vm/docs/03-SCRIPTING.md)), rmharness ([`README`](../debug/risc-v-gsp-rm-vm/rmharness/README.md)) |
| Which tool applies in which loop; automation and packaging | Accuracy M1–M6 ([`06-ACCURACY.md`](../debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md)); performance method ([`05-operations/01`](05-operations/01-performance-correctness-security.md) §3); version discipline ([`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §4) |
| Evidence bundles, version-span gates, frontier admission | Per-tool docs and the periodic [`27-TOOL-AUDIT.md`](../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md) |
| Live physical-backend engineering operations: supervised-cycle mechanics, parameter gates, debugfs controls and the hazard register (**Part II** below) | Physical-backend architecture ([`03-architecture/08`](03-architecture/08-physical-gpu-integration.md), [`03-architecture/11`](03-architecture/11-physical-backend-guest-wiring.md)); defensive/incident scenarios ([`RUNBOOK.md`](RUNBOOK.md)); current frontier ([`STATE-CURRENT.md`](STATE-CURRENT.md)) |

**Explicitly out of scope.** (1) Security claims: the simulator and its device
maps are study environments, never a confidentiality or isolation boundary
([`06-ACCURACY.md`](../debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md) §6,
[`GREEN-EYE-ROADMAP.md`](../debug/risc-v-gsp-rm-vm/GREEN-EYE-ROADMAP.md) V4).
(2) Timing claims: L2 is not modelled; order-of-events only
([`06-ACCURACY.md`](../debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md) §2).
(3) Shipping injected code: development-only, never a firmware patch, never in
a guest ([`12-INJECTION-SDK.md`](../debug/risc-v-gsp-rm-vm/docs/12-INJECTION-SDK.md) §1).
(4) Support and licensing: tested-and-documented only
([`hardware-software-matrix.md`](06-portability/hardware-software-matrix.md) §1);
legal rules in [`legal-and-licensing.md`](09-reference/legal-and-licensing.md).

## 2. The core rule: no insertion without a measurable outcome

A tool, or an insertion of an existing tool into a workflow, is admitted only
with all five fields below. Missing any field rejects the proposal; it is not
deferred until the metric is invented later.

| Field | Meaning | Example |
|---|---|---|
| **Scenario** | exact question, target, version and loop where the tool is inserted | "classify one new GSP image's dispatch-table delta before the emulation is touched (portability loop)" |
| **Measurable outcome** | result with a unit and an observer | "image drop → `version-diff.md` delta table; sessions per finding" |
| **Baseline** | the replaced method and its measured/recorded cost | "static corpus diff + one supervised boot per hypothesis; baseline in [`docs/25`](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md) §3" |
| **Evaluation threshold** | the number that decides keep/extend/retire, with sample size and environment | "N ≥ 3; median TTFA ≤ 25 % of baseline; self-diff zero; M-table improves or stays equal" |
| **Failure signal** | observable "not useful" outcome during the pilot | "unexplained `compare` deltas after two triage passes, or the tool needs hardware to run" |

Rules: one scenario per record; the threshold is recorded **before** the first
run and moving it needs a new record; every admission starts as `admit-pilot`
with an expiry of one quarter or five sessions; the baseline must be observable
(measure it first if unrecorded); a missed threshold triggers §4.4/§4.5.

```text
TOOL ADMISSION RECORD                                    id: TOOL-<yyyy>-<nn>
tool:           <name or path; existing | extension | new>
loop:           static-RE | dynamic-RE | porting | performance | correctness | CI | guest-compat
owner/decider:  <maintainer>; executor: <human|agent|cron>
scenario:       <one sentence: question, target, version, trigger>
measurable:     <metric + unit + observer>
baseline:       <replaced method; cost source and date>
threshold:      keep if <...>; reject if <...>; N=<runs>; environment=<...>
failure signal: <observable not-useful outcome>
evidence:       <session dir / report / manifest + sha256>
decision:       admit-pilot | admit | extend | redesign | retire | do-not-build
review-by:      <date or N sessions>              trials-dir: <path>
```

The record is stored with the evidence (session directory, bundle or PR
description); decisions are recorded by a human maintainer and mirrored in the
register (§4.5) and revision notes (§9.3). At expiry: met → admit; not met →
redesign or retire; no signal → retire. No unused pilot survives.

## 3. Value axes: what tooling is for, and how it is measured

Each admission names exactly one axis. Every axis reduces to moving evidence
through a §3.3 loop; if it cannot be expressed as a time or a count, it is not
admissible.

| # | Axis | Concrete example (real tooling) | Metric (recorded) | Loop |
|---|---|---|---|---|
| 1 | Faster reverse engineering | `version_diff.py` + `compare` on two GSP images; Ghidra corpus diff | firmware drop → classified delta (TTFA); symbols resolved vs Ghidra-only | RE |
| 2 | Deeper / more correct analysis | watchpoints + TCG/native trace; capability inference ([`docs/14`](../debug/risc-v-gsp-rm-vm/docs/14-ADVERSARIAL-INTEROP-GUIDE.md) §4.2) | attributed bytes/stores; M6 agreement; unexplained-delta count | RE |
| 3 | Code-artifact porting time | `cudatool.py fdematch`, `diff_corpus.py`, `symbols export` | transfer rate per class; wall time old→new symbol DB; wrapper re-resolution errors | portability |
| 4 | Bug-analysis time | `reach.py`, `run_until`/`call`, `ioctl_trace.c`; taxonomy in [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §5 | crash/question → first evidence-backed answer (TTFA); supervised cycles avoided | adversarial |
| 5 | Development velocity on the simulated GSP | injection SDK; `greeneye.sh inject`; value demo s06 | seconds from service edit to direct+mailbox agreement; sessions per prototype | RE |
| 6 | Correctness validation of virtualization | M1–M6 table, `compare`, golden sessions | M-table per change; rung ladder ([`05-operations/01`](05-operations/01-performance-correctness-security.md) §1) | correctness |
| 7 | Performance analysis and improvement | A/B method, `tools/benchmark_infrastructure.sh`, simulator overhead ([`docs/25`](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md) §6) | B/A % per metric; p50/p99 RPC; trace overhead; before/after same machine | performance |
| 8 | Userspace/API compatibility friction | capability negotiation ([`../userspace/cuda/docs/04-virtualization-and-emulation.md`](../userspace/cuda/docs/04-virtualization-and-emulation.md) §1), symbol DB, ioctl diffs | controls with coherent replies vs observed; TTFA for a new capability bit; peer scenarios (RDMA guest, DPU/NIC/NVMe page sharing) counted only once a peer model exists | guest-compat |
| 9 | Hidden / undocumented / unauthorized code, side effects, backdoors | `control_discovery.py`, mutation harness, decoy handling ([`docs/14`](../debug/risc-v-gsp-rm-vm/docs/14-ADVERSARIAL-INTEROP-GUIDE.md) §4.4) | unknown IDs and injected-slot aliases found; capability-model deltas; corpus entries with sessions | adversarial |
| 10 | Architectural understanding and SBOM/doc improvement | symbol DB, `export_hits`, firmware notes ([`docs/09`](../debug/risc-v-gsp-rm-vm/docs/09-GSP-FIRMWARE-NOTES.md)), [`SBOM.md`](../SBOM.md) | doc/SBOM rows updated per finding; onboarding time; subsystem coverage | RE |
| 11 | Security fixes / hardening time | capability inference → fail-closed rules ([`04-implementation/02`](04-implementation/02-practical-challenges-and-adversarial-interop.md) §4), mutation corpus | error paths covered; finding → test time; physical confirmation before product change | adversarial |
| 12 | New GSP support time | `probe` + `version_diff` + `portability_matrix` + `ci_release` bundle | hours from image to triage table; M-table completeness; regression lead time | portability |
| 13 | New hardware-device support time | device profile capture, `rmharness`, matrix, [`docs/22`](../debug/risc-v-gsp-rm-vm/docs/22-PORTING-CUDA-AND-DEVICES.md) §3, [device sequence](06-portability/03-device-validation-and-porting-sequence.md) | profile fields captured; engine inventory validated; first-touch deltas explained; tested-at-rung date | portability |
| 14 | CUDA / runtime / VM-environment fixes | `userspace/cuda/docs/07`, `cuda_drv_test.c`, `tools/cuda-ab/ab.sh`, guest diagnostics | crash → named function time; `RESULT`/`STATUS` match; rung movement | guest-compat |

Axis 8 note: GPU-direct RDMA availability is negotiated via the stock guest's
vGPU capability block; page-table sharing with simulated DPU/NIC/NVMe peers is
future, so its metric starts only when a peer model exists (§8).

**Performance axis (axis 7).** Profiling order is the project's existing one
([`05-operations/01`](05-operations/01-performance-correctness-security.md) §3.6):
RPC dispatch → doorbell cadence → eventfd/MSI-X delivery → BAR0 shadows →
memory-model gap. Method: (1) define the metric and A/B pair before code
changes — same binary, inputs, governor, clocks and boot where possible, and
report B/A %, never absolutes; (2) run A via `tools/benchmark_infrastructure.sh`
and B via `scripts/vgpu_vm.py exec`, logging versions and
`nvidia-smi -q -d CLOCK`; (3) attribute with host counters
(`gsp_emu_get_stats()`, `trace_printk`, `/sys/kernel/debug/tracing`) and the
measured simulator overhead ([`docs/16`](../debug/risc-v-gsp-rm-vm/docs/16-CUSTOM-SIMULATOR.md) §1.1a, [`docs/25`](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md) §6);
(4) keep the numbers with the change and re-run correctness; (5) reject single
runs, cross-host comparisons, bare-metal-only data and simulator timing claims.

**OODA reduction.** Every axis moves evidence through one of three loops. Each
loop's compression is the project's program-level metric
([`GREEN-EYE-ROADMAP.md`](../debug/risc-v-gsp-rm-vm/GREEN-EYE-ROADMAP.md) success
criteria), measured in [`docs/25`](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md) §7:

| Loop | Observe → Orient → Decide → Act | Compression metric |
|---|---|---|
| RE-constrained (static/dynamic interlock) | probe/hooks/watch/trace → `analyze`/snapshots → `compare`/triage → reimplementation | question → reviewable answer; sessions per finding |
| Adversarial-interop (opaque/decoy firmware) | discovery/mutation/replay → classification ([`docs/14`](../debug/risc-v-gsp-rm-vm/docs/14-ADVERSARIAL-INTEROP-GUIDE.md) §4.1) → fail-closed or no change → emulation change + physical confirm | unknown-ID → classified route; error paths covered |
| Portability (GSP/device/architecture churn) | probe first-touch/symbol diff → pairing check/delta class → anchor decision → versioned dirs + handlers | image drop → triage table (hours) |

## 4. Use, extend, build, redesign, retire

```mermaid
flowchart TD
  P["Friction + scenario + measurable outcome"] --> Q1{"Existing tool covers the scenario?"}
  Q1 -->|"yes"| USE["Admit the insertion; pilot + metric record"]
  Q1 -->|"no"| Q2{"A close tool has a public extension point?"}
  Q2 -->|"yes"| EXT["Extend in place: plugin, harness Surface, probe spec, SDK"]
  Q2 -->|"no"| Q3{"Repeated scenario and no cheaper alternative?"}
  Q3 -->|"no"| NB["Do not build: record rejection and the trigger to revisit"]
  Q3 -->|"yes"| NEW["Admit a new tool as a bounded pilot"]
  USE & EXT & NEW --> REV{"Threshold met after the pilot?"}
  REV -->|"yes"| KEEP["Admit; add a second metric over time"]
  REV -->|"no"| Q4{"Failure signal points at the design, not the idea?"}
  Q4 -->|"yes"| RD["Redesign with a migration note; re-pilot"]
  Q4 -->|"no"| PRUNE["Retire / prune; keep evidence read-only"]
```

| Decision | Minimum evidence | Decider | Record |
|---|---|---|---|
| (a) Use existing | scenario + measured baseline + one recorded trial session | maintainer; agent may run the trial | admission record + register row; link the session |
| (b) Extend existing | ≥ 3 uses or ≥ 2 users; public extension point (plugin, `Surface`, probe spec, SDK); cost ≤ new tool | maintainer | record names the extension point and keeps the old tests green |
| (c) Build new | record complete; bounded pilot; golden/replay evidence; stdlib/local-only unless justified | maintainer | record + register + revision note; tests added to the tool suite |
| (d) Redesign | failure signal twice, or a fidelity gap that changes conclusions; migration note (`greeneye-session/1` evolves additively) | maintainer | before/after metric; old artifacts stay readable |
| (e) Stop / prune | threshold missed, no evidence in the window, duplicated capability, or maintenance > value | maintainer at the audit | retire row; `27-TOOL-AUDIT.md` status; keep sessions/reports |
| (f) Do not build | any trigger below is true | maintainer | rejection note naming the trigger that would reopen it |

**When NOT to build (workflow risk).** Reject if any holds; the absence of a
time-to-value process is the risk this document removes: (1) no scenario, unit
or baseline — the pilot cannot fail; (2) the tool would become a shipped
dependency, guest component or firmware patch
([`04-implementation/02`](04-implementation/02-practical-challenges-and-adversarial-interop.md) §5.1);
(3) it needs physical GPU, mdev or guest access for its primary value
([`docs/15`](../debug/risc-v-gsp-rm-vm/docs/15-AGENTIC-GSP-IDE.md) §5);
(4) it bypasses a security boundary or turns a study observation into a
security claim; (5) it duplicates a tool without a measured improvement;
(6) total cost is not plausibly below the replaced OODA cost; (7) its only
evidence is prose — evidence must be a session, report or bundle.

**Redesign and retirement.** Redesign rather than patch when the failure
signal repeats, a fidelity gap invalidates an accepted conclusion, a public
interface must change incompatibly, or the tool drifted outside its admitted
loop. Retire when no artifact references the tool, the threshold was missed
twice, another tool covers the scenario better, or maintenance exceeds the
saving. Retirement is revocation of admission, not deletion: keep tool, tests
and artifacts read-only, mark the register row `retired` with date and reason,
and reconcile at the next [`27-TOOL-AUDIT.md`](../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md).

**Register (how decisions are recorded).** The durable record is the register
row, the evidence directory/bundle it names and the revision note in §9.3;
the audit is the periodic reconciliation.

| Admitted family | Loop | Value evidence (current) | Review / retire trigger |
|---|---|---|---|
| Ghidra corpora + scripts (`tools/ghidra`, `userspace/cuda/tools`) | RE, portability | docs 05/07; `diff_corpus`, name map | corpus diff no longer changes port class |
| green-eye core + session format | dynamic-RE | `docs/25` s01–s10; 155-test suite | format unused by any loop |
| Harness (`version_diff`, matrix, mutation, discovery, reach, `ci_release`) | portability, adversarial | docs 18/19/21; deterministic replay tests | a report is never attached to a change |
| Injection SDK + demo uproc | dynamic-RE | docs 11/12; s06 | prototypes stop using it |
| Ghidra bridges (in/out) | RE | docs 05/13 | static/dynamic join replaced |
| Device map (`greeneye-device-map/1`) | portability | docs 24; H2.1 first-touch | H2.2 model stops consuming maps |
| `rmharness` + `rmtool`/`rmtop`/`rmq` | correctness, guest-compat | docs 17; M1/M6 reference captures | reference capture unused/unavailable |
| MCP server + agent workflows | all loops | docs 15/20; release monitoring | write gating or audit trail removed |
| CI (`ci.yml`, `greeneye.yml`) | correctness | release-blocking tests | suite non-reproducible or skipped |

## 5. Workflow map: where each tool applies

### 5.1 Context of the development and reverse-engineering environment

```mermaid
flowchart LR
  subgraph STATIC["Static"]
    GH["Ghidra CUDA corpus + GSP firmware"]
  end
  subgraph DYN["Dynamic (green-eye, experimental)"]
    GE["Session + QEMU/native backends"]
    HR["harness/*.py"]
  end
  subgraph HOST["Host and guest"]
    MOD["open-iov-vfio.ko"]
    VM["stock guest + guest-diag"]
    RMH["rmtool/rmtop/rmq + rmharness"]
  end
  subgraph AUTO["Automation"]
    CI["ci.yml / greeneye.yml"]
    MCP["MCP server + agents/cron"]
  end
  EVID["Evidence: sessions, reports, bundles"]
  GH -->|"names, index, symbol DB"| GE
  GE -->|"export_hits, fault PCs"| GH
  GE --> HR --> EVID
  GE --> EVID
  RMH --> EVID
  VM -->|"stimulus and failures"| GE
  MOD --> VM
  EVID --> AUTO
  AUTO -->|"bounded runs"| HR
  EVID -->|"product-change gate"| MOD
```

### 5.2 Static reverse-engineering loop

```mermaid
flowchart LR
  BIN["libcuda / .fwimage"] --> IMP["Ghidra import + analysis"]
  IMP --> IDX["names.tsv, decompiled_index.tsv, callgraph.tsv"]
  IDX --> DIFF["corpus diff / fdematch / RPC dispatch diff"]
  DIFF --> HYP["Hypothesis: functions, tables, layouts"]
  HYP --> SYM["greeneye symbols export (bias recorded)"]
  SYM --> DYN["dynamic loop"]
  DYN -->|"hits, watch PCs, fault PCs"| HYP
```

### 5.3 Dynamic firmware loop (OODA)

```mermaid
flowchart LR
  Q["Question + target + fidelity statement"] --> OBS["Observe: probe, hooks, watch, trace"]
  OBS --> ORI["Orient: analyze, symbol decode, snapshots"]
  ORI --> DEC["Decide: compare, M1-M6, mutation"]
  DEC --> ACT["Act: emulation change with evidence"]
  ACT --> REG["Regression: same stimulus, same budgets"]
  REG -->|"improved or equal"| PACK["Evidence pack + session hashes"]
  REG -->|"regressed"| Q
  PACK --> Q
```

### 5.4 Porting loop (GSP firmware, device, architecture)

```mermaid
flowchart TB
  ACQ["Acquire + hash; versioned dirs, never overwrite"] --> TRI["Triage: delta, fdematch, pairing check"]
  TRI --> PRB["Probe first-touch old vs new"]
  PRB --> RUN["Rehost/demo: identical stimulus + record"]
  RUN --> CMP["compare + M1-M6 skeleton"]
  CMP --> BUN["version_diff -> matrix -> mutation -> bundle"]
  BUN --> REV{"Every delta explained?"}
  REV -->|"no"| TRI
  REV -->|"yes"| ANCHOR["Anchor decision + docs + matrix entry"]
```

### 5.5 Performance loop

```mermaid
flowchart LR
  DEF["Define metric + A/B pair"] --> BASE["A: bare metal baseline"]
  DEF --> CAND["B: vGPU / candidate"]
  BASE --> RATIO["B/A ratio, variance, N runs"]
  CAND --> RATIO
  RATIO --> PROF["Attribute: RPC, doorbell, eventfd, shadows"]
  PROF --> KEEP["Keep numbers with the change"]
  KEEP --> REG["Re-run correctness corpus + ladder"]
```

### 5.6 Correctness and CI loop

```mermaid
flowchart LR
  CH["Change"] --> HOST["Host suite: stack_manager.sh test"]
  HOST --> GUEST["Guest ladder: probe, nvidia-smi, cdt"]
  GUEST --> AB["cuda-ab: RESULT/STATUS must match"]
  AB --> SESS["Session compare / golden data"]
  SESS --> GATE{"Rung or metric regressed?"}
  GATE -->|"no"| LAND["Land with evidence"]
  GATE -->|"yes"| CH
```

### 5.7 Guest-compatibility loop

```mermaid
flowchart LR
  GD["guest-diag: ioctl trace, engines, pmap"] --> CTRL["Control/ioctl inventory"]
  CTRL --> EMU["Emulated reply fidelity"]
  EMU --> GD2["New driver/CUDA version trace"]
  GD2 --> DIFF["Control diff old vs new"]
  DIFF --> CTRL
  CTRL --> CORPUS["CUDA corpus + symbol DB"]
```

### 5.8 Loop index and detailed docs

| Loop | Entry points | Evidence | Gate | Detail |
|---|---|---|---|---|
| Static RE | `tools/ghidra/*`, `userspace/cuda/tools/*`, `analyzeHeadless` | corpus TSVs, name map, symbol DB | addresses re-derived per build | [`userspace/cuda/docs/07`](../userspace/cuda/docs/07-version-upgrade-playbook.md), green-eye docs 05/13 |
| Dynamic RE | `greeneye.sh` probe/record/analyze/compare/inject/ghidra, Python `Session` | `greeneye-session/1` + `summary.md` | A/B pair; L2/L3 statement | [`26-ARCHITECTURE.md`](../debug/risc-v-gsp-rm-vm/docs/26-ARCHITECTURE.md) §11, [`11`](../debug/risc-v-gsp-rm-vm/docs/11-GSP-RM-SERVICE-DEVELOPER-GUIDE.md), [`12`](../debug/risc-v-gsp-rm-vm/docs/12-INJECTION-SDK.md), [`13`](../debug/risc-v-gsp-rm-vm/docs/13-GHIDRA-CONTROL.md), [`14`](../debug/risc-v-gsp-rm-vm/docs/14-ADVERSARIAL-INTEROP-GUIDE.md) §4.6, [`06`](../debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md) §5 |
| Porting | `version_diff.py`, `portability_matrix.py`, `mutation_harness.py`, `ci_release.py` | reports + `bundle-manifest.json` | every delta explained | [`07`](../debug/risc-v-gsp-rm-vm/docs/07-PORTABILITY-PLAYBOOKS.md), [`22`](../debug/risc-v-gsp-rm-vm/docs/22-PORTING-CUDA-AND-DEVICES.md), [`21`](../debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md), runbook [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §4.1 |
| Performance | `benchmark_infrastructure.sh`, module counters, `vgpu_vm.py` | B/A table with environment | same-machine A/B; corpus green | [`05-operations/01`](05-operations/01-performance-correctness-security.md) §3 |
| Correctness | `stack_manager.sh test`, `make tests`, `cuda_drv_test.c`, `cuda-ab` | rung + trace excerpts | no rung regression | [`05-operations/01`](05-operations/01-performance-correctness-security.md) §1–2 |
| CI | `.github/workflows/ci.yml`, `.github/workflows/greeneye.yml` | workflow logs | push/PR path triggers | [`21`](../debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md) §5 |
| Guest-compat | `tools/guest-diag/*`, `cudatool.py`, `diff_corpus.py` | ioctl/control diff, corpus delta | stock guest only | [`READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md) |

## 6. Automation and agent invocation

```mermaid
flowchart LR
  PRE["Preflight: greeneye.sh info; RM610 gate"] --> DRY["ci_release.py --dry-run (plan only)"]
  DRY --> REAL["ci_release.py real run"]
  REAL --> VD["version_diff"]
  REAL --> PM["portability_matrix"]
  REAL --> MU["mutation smoke"]
  VD & PM & MU --> MAN["release-report.json + bundle-manifest.json"]
  MAN --> REV["agent/human review"]
  REV -->|"accepted"| REG["anchor/register/docs updated"]
  REV -->|"rejected"| GAP["fidelity gap or fix, recorded"]
```

Run everything from `debug/risc-v-gsp-rm-vm/`; scripts are stdlib-only and
launch their own QEMU. Never re-use an existing gdbstub, a guest, a kernel
module or hardware.

**Scripts that exist** (success and failure are read, not assumed):

| Script | Safe invocation | Success | Failure signal / limit |
|---|---|---|---|
| `harness/version_diff.py` | `python3 harness/version_diff.py --old <old> --new <new> --out dumps/vdiff` | sections `ok`; self-diff zero; deltas explained | exit 2 tool error, 3 RM610 refusal; raw `.fwimage` faults early by design |
| `harness/portability_matrix.py` | `python3 harness/portability_matrix.py --target <a> --target <b> --out dumps/matrix` | self-comparison `all_zero`; M1–M6 `not measured` without reference | non-zero target deltas or per-target `status: error` |
| `harness/mutation_harness.py` | `python3 harness/mutation_harness.py --out dumps/mut --iterations 64 --seed 7` | deterministic `findings.jsonl`; declared outcome classes | seed divergence; non-demo ELF refused without `--allow-foreign-elf` |
| `harness/control_discovery.py` | `python3 harness/control_discovery.py --surface demo --out dumps/disc --seed 7` | known IDs resolve; unknown/injected classes listed | RM surface exits cleanly until H1.3; misclassification or unreplayable probe |
| `harness/reach.py` | `python3 harness/reach.py --target rpc_ping --method call --args 0x41 --out dumps/reach` | `reachable` with recorded route and args | timeout means "not under this stimulus", not unreachable |
| `harness/ci_release.py` | dry-run first: `python3 harness/ci_release.py --old <old> --new <new> --out dumps/release --dry-run` | dry-run touches no target; real run writes the bundle | exit 1 step error, 2 tool error, 3 RM610 refusal; `skipped` is not `pass` |
| `greeneye.sh` | `./greeneye.sh info`/`probe`/`record`/`analyze`/`compare`/`inject` | session dir with `session.json` + `summary.md` | empty sessions or missed hooks; budgets required |
| `rmharness/harness.sh` | `./rmharness/harness.sh --session dumps/real -- rmtool engine list --format=json` | `rm_harness_run` event + stream blobs | read-only only; permission errors reported, never escalated |
| `plugins/mcp_server.py` | `python3 plugins/mcp_server.py --stdio --record` | 24-tool JSON-RPC surface; read/observe default | stdout protocol-only; writes need `--allow-write`/`--allow-inject` |
| `harness/value_demos/run_all.py` | `python3 harness/value_demos/run_all.py` | one `RESULT_JSON` per demo; suite ≤ 180 s | any demo `success: false`; re-run before quoting numbers |

**Scaffolded scripts (landed, not admitted yet).** The three scripts below
landed with docs 28/29 and are pilots until they have a §2 record; the first
two are dry-run by default (`--run` executes). `--help` is the authoritative
flag list; the outputs and failure readings are in
[`docs/28`](../debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md) and
[`docs/29`](../debug/risc-v-gsp-rm-vm/docs/29-PACKAGING-AND-TRIALING.md).

| Script | Purpose | Interface | Outputs | Exit codes |
|---|---|---|---|---|
| `harness/port_driver_versions.py` | sweep a version list through `version_diff` + matrix + mutation, one evidence set per version | `--versions <list>` or `--config <json>`, `--baseline <ver>`, `--input-dir <dir>`, `--target <ver>=<path>`, `--symbols <ver>=<path>`, `--ghidra-exports <ver>=<path>`, `--out <dir>`, `--run` (default: dry-run), `--force`, `--extract-runfiles`, `--approval-token` | per-version `version-diff/` + `portability-matrix/` + `mutation-smoke/` + `bundle-manifest.json`, `sweep-summary.json|.md` | 0 complete, 1 step/threshold failure, 2 tool error, 3 RM610 refusal |
| `harness/port_new_device.py` | per-device port: profile, pairing check, first-touch probe, symbols, matrix, checklist | `--device <label>`, `--profile <json>`, `--firmware <path>`, `--reference <path>`, `--symbols <json>`, `--ghidra-exports <tsv>`, `--out <dir>`, `--run` (default: dry-run), `--approval-token` | `device-port.json|.md`, `port-checklist.json|.md`, `symbols/`, sessions, matrix, `bundle-manifest.json` | as above; 1 when pairing is ambiguous or a delta unexplained |
| `harness/package_sim.py` | build (and optionally run) a trial simulation package (§7) | `--target demo\|image`, `--image <path>`, `--service <elf>`, `--device-map <json>`, `--script <py>`, `--slot N`, `--pack-image`, `--tar`, `--run`, `--out <dir>` | package tree + `manifest.json`; `results.json|.md` after `run.sh` | 0 built/run, 1 replay or assertion failure, 2 tool error, 3 gate refusal |

All three: stdlib-only, simulator/local only, deterministic budgets recorded in
the report, reusing existing harness functions, and running the RM610 label check first.
Their admission evaluation is recorded in
[`future-eng-ops.md`](future-eng-ops.md) F2–F4.

**Version span and the RM 610 gate.** The automated span is **540 → current**
(anchor 570.144, [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §4.1);
inputs are versioned under `firmware/<SERIES>-GSP/<VER>/` and never overwritten.
**RM 610 requires explicit human approval**: before approval nothing fetches,
opens or analyses a 610 image
([`23-RM610-READINESS.md`](../debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md)).
`version_diff`/`ci_release` refuse any `rm610`-matching label with exit 3
unless `--approval-token RM610-APPROVED` or `GREENEYE_RM610_APPROVED=1` is
present; the check runs in `--dry-run` and before any target is opened. An
agent may prepare the approval request and run current-anchor rehearsals; only
a named human approves.

**Evidence bundles.** `ci_release.py` writes `release-report.json`,
`bundle-manifest.json` (sha256 + size of every file) and the step trees
([`docs/21`](../debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md) §4.3).
Exit 0 with all steps `ok` is complete, but read `skipped`; exit 1 means a step
errored (reviewable, not evidence for that section); exit 2 no claim; exit 3 a
gate refusal, not a bug. Verify hashes before sharing and attach the bundle.

**MCP monitoring.** `plugins/mcp_server.py` exposes the live session
(read/observe by default) and shells out to `probe_firmware`,
`portability_matrix_run`, `mutation_run`, `analyze_current` and
`compare_sessions` ([`docs/20`](../debug/risc-v-gsp-rm-vm/docs/20-MCP-SERVER.md) §5).
Release pattern: matrix → pinned mutation smoke → compare the recorded pair →
read the bundle (`version-diff.json` / `release-report.json` /
`bundle-manifest.json`); the agent does not recompute it. Absent capabilities
return `not_available`, so planning is safe.

**Cron/agent safety.** Default to read/observe and `--dry-run`; write/inject
flags are explicit and the recording is the audit trail. Physical GPU, mdev
churn, module load/unload and guest access are never unattended. Budgets
everywhere; a wedged target is left stopped. Product changes are proposals —
only humans merge, after physical confirmation of behaviour changes. Local
only: stdio MCP, loopback TCP at most; never expose it to a network.

## 7. Packaging and trialing

A trial package lets a user run one **pre-scripted whole-GSP simulation** with
injected code without the lab. It is a developer artifact: local, simulator
only, no hardware, guest or kernel module. It is not a product component and
not a support claim.

```text
dist/<pkg>/
  README.md                 scenario, expected result, budgets, L2/L3 caveat
  run.sh                    deterministic runner (greeneye.sh run + settle)
  scenario.py               fixed stimulus, fixed budgets, records a session
  firmware/<image>          ELF or .fwimage, hash-pinned
  service/<service>.elf     injected service (PIC, ABI v1), hash-pinned
  device-map/<map>.json     greeneye-device-map/1 (bias, entries, fw sha256)
  golden/session/           optional reference session for compare
  package-manifest.json     sha256 + bytes per member (bundle shape, docs/21)
```

Runner and scenario follow the injection contract: PIC blob, no libc, no
absolute addresses, one dispatch entry per slot, every write recorded
([`docs/12`](../debug/risc-v-gsp-rm-vm/docs/12-INJECTION-SDK.md) §2–5); the device map
carries provenance so a lenient run is not mistaken for device-modelled behaviour
([`docs/24`](../debug/risc-v-gsp-rm-vm/docs/24-DEVICE-MAP.md) §1).

```text
python3 harness/package_sim.py --out dist/<pkg> --service <service.elf> \
    --device-map <map.json> --script <scenario.py> [--pack-image] [--tar] [--run]
./dist/<pkg>/run.sh
```

Success: the packaged `run.sh` reports `status: ok` (demo target) or the
documented blocked status (real image), and every `manifest.json` component
hash matches. Failure: hash mismatch, nondeterministic replay, unresolved
symbol or missing member. The scaffold has no `--verify` subcommand yet:
verification is re-running `run.sh` and `sha256sum` against the manifest
([`docs/29`](../debug/risc-v-gsp-rm-vm/docs/29-PACKAGING-AND-TRIALING.md) §2–4),
and it stays a pilot until future-eng-ops F4's threshold is met.

**Sharing and judging results.** Share the package (or its manifest) with the
session it produces. The recipient judges manifest hashes, the session
`summary.md`, the golden comparison and the fidelity statement (L0/L1 only,
L2/L3 absent). Results are evidence about firmware in the simulator; a product
change still requires the accuracy loop and physical confirmation.

**Packing a modified GSP image.** When a GSP modification (injected service,
shim or patched firmware) is considered ready to ship, `package_sim.py` may
pack the modified image, the tested injected code, device map, scenario and
golden session so the recipient can reproduce the exact run before any product
decision. **Caveat, in every package README: this is a simulation package, not
a signed production image.** It is not a firmware patch, must never be flashed
or injected into a guest, and is not bootable on silicon. Injected code is
development-only ([`docs/12`](../debug/risc-v-gsp-rm-vm/docs/12-INJECTION-SDK.md) §1);
productization goes through `open-iov-vfio` + physical confirmation, never
through a simulator package.

Local-only, budgeted, session-recorded; never touches `/dev`, a module, an
mdev, a guest or a GPU. No confidentiality, isolation or timing claim derives
from it; distribution follows per-file licences ([`SBOM.md`](../SBOM.md),
[`legal-and-licensing.md`](09-reference/legal-and-licensing.md)).

## 8. Frontier capabilities

Everything here is **future** and off the critical path. Each capability enters
the §2 process only when its near-term need is real and its gate is passed;
details are tracked in the companion [`future-eng-ops.md`](future-eng-ops.md)
(same directory) and
[`GREEN-EYE-ROADMAP.md`](../debug/risc-v-gsp-rm-vm/GREEN-EYE-ROADMAP.md) H5/H6.
Deferred context: [`10-post-1.0/experimental-host-acceleration.md`](10-post-1.0/experimental-host-acceleration.md).

| Frontier capability | Near-term need | Precondition / gate | Evaluation criterion (if funded) |
|---|---|---|---|
| SIMD cores and CUDA kernel execution on device | Execute a real kernel launch path (`vectorAdd`-class) in simulation before the physical backend is finished | H1/H2 stable; injection ABI usable for an engine model | kernel executes within an instruction budget; output matches the native/golden result byte-for-byte; deterministic replay |
| FPGA soft-processor simulation | A cycle-accuracy question worth funding (timing/security/interoperability claim needing silicon-equivalent behaviour) | H3 validated (native parity) and H2 device model proven | the same `Session` scripts run against the RTL backend; execution parity with native; cycle counts recorded as a distinct event class, docs/06 fidelity revisited |
| Accurate Verilog recreation of all components the GSP interacts with, plus the NVRISC environment and intra/inter-micro-controller interconnects | Boot the real image with silicon-equivalent register, interrupt and mailbox semantics; exercise PMU/SEC2/ACR cross-controller boot flows | H2/H5 proven, validated device map, stable `Backend` abstraction | firmware reaches its message loop in the soft environment; M1–M6 improve over the QEMU/native baseline; one cross-controller exchange reproduced and compared with a physical capture |

## 9. Review discipline

**Adversarial-review loop.** (1) The author proposes a change with its
revision note (§9.3). (2) A second person or agent attacks it: broken links,
unverifiable claims, thresholds without metrics, tools without a scenario,
missing register rows, scope creep into the manuals. (3) The attacker
reproduces one item end-to-end (link check plus one claimed command);
"looks fine" is not a review. (4) The author resolves findings; the reviewer
signs the note with the evidence used.

**Proportionate review (no needless overhead).**

| Change class | Examples | Required review |
|---|---|---|
| A — editorial | wording, formatting, link fixes | `python3 scripts/check-docs-links.py` from the repo root; no design review |
| B — decision | admission, threshold, register, retire changes | one adversarial reviewer + evidence named in the note |
| C — scope/security | out-of-scope changes, gates, packaging caveats | maintainer sign-off + updated admission records |

Cadence: review at each tool audit
([`27-TOOL-AUDIT.md`](../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md)) and
after any firmware anchor change. Expired pilots must appear in the register;
an empty register of decisions means this document is not being used.

**Revision notes.** Every change records purpose, scope and benefit; a change
that cannot state a benefit is rejected as overhead.

| Date | Revision | Purpose / scope | Benefit | Evidence |
|---|---|---|---|---|
| 2026-10-02 | v1 | Establish the admission rule, value axes, decision framework, workflow maps, automation, packaging, frontier gates and review discipline | Tooling decisions become auditable, measurable and reversible; no tool without an evaluation | This document plus linked tool docs and the [`docs/25`](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md) baselines |
| 2026-10-02 | v2 (Part II) | Add the live physical-backend operations: hard rules and scope, build/load order and the full `gsp_phys_*` parameter matrix, the routine supervised cycle with its clean-release checklist, the debugfs inventory and `gsp-phys-state` reference, the physical-backend scenario catalogue and hazard register, the OODA loop and the work-context/porting map | Physical-backend execution becomes auditable and fail-safe: one checklist, a named signal per failure, and no retry of a wedged path | Kernel sources cited per parameter and debugfs file; the current frontier in this document and [`STATE-CURRENT.md`](STATE-CURRENT.md) |

```bash
python3 scripts/check-docs-links.py                       # repo root
cd debug/risc-v-gsp-rm-vm && make demo sdk && make test   # tool claims
./greeneye.sh info                                        # environment
```

---

# Part II — Live physical-backend operations

> **Scope.** This part is the operating manual for cycles against the **live
> physical backend**: the module-owned physical GPU running its real GSP-RM
> session (the chosen execution mechanism in
> [`03-architecture/08`](03-architecture/08-physical-gpu-integration.md), wired
> toward the guest plane in
> [`03-architecture/11`](03-architecture/11-physical-backend-guest-wiring.md)).
> Part I (§§1–9) governs tooling admission, value measurement and retirement.
> Defensive/incident scenarios are owned by [`RUNBOOK.md`](RUNBOOK.md) and are
> cross-linked, not duplicated, here; the current frontier is
> [`STATE-CURRENT.md`](STATE-CURRENT.md); the known unknowns are
> [`04-implementation/01`](04-implementation/01-known-knowns-and-unknowns.md).
> Everything in this part is supervised, bounded and fail-safe-first; nothing
> here authorises unattended hardware work.

## 10. Live physical-backend operations

### 10.1 Hard rules and scope

These apply to every physical cycle; where a step below conflicts with them,
the rules win.

| # | Rule | Consequence / practice |
|---|---|---|
| 1 | **Supervised only, one GPU cycle at a time.** Never unattended, never parallel with other host work, never as a background agent job. | A named human is present for the cycle; the pre-flight and clean-release checklists (§10.3) are completed for each single attempt. |
| 2 | **Reboot is gated by the written approval policy.** A reboot request MUST first record, in the project's live working record, the reason and the exact post-reboot plan; only then is a named human asked to approve. | No reboot while other work is running; the device is left untouched in the meantime; an unapproved reboot is never initiated by an agent. |
| 3 | **Never enable `gsp_phys_repower`; never re-arm D3cold.** The default 0 latches the pre-claim D0 policy until reboot. | Re-enabling runtime PM/D3cold after a claim can power-gate the board and hard-freeze the host ([`03-architecture/08`](03-architecture/08-physical-gpu-integration.md) §4.2). |
| 4 | **Never retry a failed mutating RPC.** One attempt per staged step; abort the ladder from the first failure. | Retrying has wedged GSP-RM (channel-flag and context-promote incidents); `UNLOADING_GUEST_DRIVER` + `rmmod` and, if needed, an approved reboot remain the only recovery. |
| 5 | **Stop on hard signatures.** `board unresponsive`, `refusing claim`, `do not retry in this boot`, any panic/oops. | Do not write the device again in that boot; keep the locks; request reboot approval per rule 2. |
| 6 | **Verify module identity before and after a cycle.** Record the built module's sha256 (and `srcversion`) before load and re-read it after unload. | A hash mismatch invalidates the cycle's evidence; stop and rebuild from the reviewed source. |
| 7 | **A wedged device is left untouched for the reboot.** No `rmmod` retry loops, no bind/unbind experiments, no power-policy pokes. | The fail-safe (no rebind, D0 locks + runtime reference held, WPR2 up) is the designed resting state. |
| 8 | **Scope.** This part covers the claim/boot, rm-call ladder (host-side real submission), guest-memory forwarding, relay/fault-relay and release paths. Guest-facing emulation debugging is [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md); firmware runtime study is [green-eye](05-operations/03-risc-v-gsp-rm-vm.md). | All other physical work belongs to the scenario owner named in §14. |

### 10.2 Build and load order, and the module-parameter matrix

**Build order (physical-backend work needs the kernel module only):**

1. Build `open-iov-vfio.ko` first; it generates the `Module.symvers` the
   optional OpenRM modules link against. `scripts/stack_manager.sh
   build-kernel`.
2. Build the optional OpenRM `nvidia.ko` with
   `KBUILD_EXTRA_SYMBOLS` only if that management path is in use; the normal
   consent path runs the distribution driver for the physical GPU and the
   emulation module independently (`scripts/stack_manager.sh build-nvidia`).
3. Build userspace (`libiov`/examples) independently
   (`scripts/stack_manager.sh build-userspace`).
4. Stage the GSP boot image into the firmware search path
   (`scripts/stack_manager.sh install-firmware`).

**Load order:**

1. Prerequisites: `vfio`, `vfio_iommu_type1`, `vfio_pci`, `mdev`
   (`scripts/stack_manager.sh load-prereqs`).
2. `open-iov-vfio.ko` with the exact reviewed parameter set
   (`scripts/stack_manager.sh load-open-iov`). The backend claim is
   **module-owned** at load with `gsp_phys_gpu=1 gsp_phys_hazard_ack=1`, or
   **mdev-owned** and deferred to the first mdev create with
   `gsp_phys_mdev_owned=1` (fail-open).
3. Record the loaded module identity and the starting state (§10.3 step 0).

**Module-parameter matrix.** Every current `gsp_phys_*` parameter is listed
below, verified against the kernel sources on the 2026-10-02 tree. The exact
firmware image name used by `gsp_phys_fw` is deliberately not repeated here
(it is the distribution driver's Turing-generation artifact); the source is
authoritative for the literal default.

| Parameter | Default | Effect | Gate / dependency |
|---|---|---|---|
| `gsp_phys_gpu` | 0 | Claim the physical GPU at module load: bind `open-iov-gsp-phys`, map BARs, stage firmware and run the requested stages; module-owned claim is released only at `rmmod`. | Refuses without `gsp_phys_hazard_ack=1`; refuses if the GPU or its root port is already runtime-suspended. With `gsp_phys_mdev_owned=1` it only arms the deferred path. |
| `gsp_phys_hazard_ack` | 0 | Acknowledges the unbind/rebind host-freeze hazard; second gate for every claim. | Required with `gsp_phys_gpu=1`; without it the claim logs `REFUSED` and returns `-EPERM`. |
| `gsp_phys_boot` | 0 | Full real-GSP boot: FWSEC → Booter Load → RISC-V start → RM-init poll (device writes). | Requires a claim; implies the layout stage; mutually exclusive with `gsp_phys_boot_dryrun`; FWSEC-only is refused because that carve is not host-recoverable. |
| `gsp_phys_boot_dryrun` | 0 | Stage-3d host-memory staging rehearsal (radix3, queues, boot args) with no BAR0 writes and no falcon resets. | Requires a claim; mutually exclusive with `gsp_phys_boot`. |
| `gsp_phys_layout` | 0 | Allocate the boot-argument pages and compute/log the WPR/FB layout; the only device access is a read-only video-memory-size sample. | Requires a claim; implied by the boot and dry-run paths. |
| `gsp_phys_nocat` | 0 | Log every NOCAT journal record (function `0x1020`) individually during the RM-init poll; the unique-record summary is always accumulated and dumped when the poll ends. | Boot path only; diagnostic. |
| `gsp_phys_fw` | distribution GSP image | GSP ELF the backend stages; override with `<name>` relative to the firmware search path. | Requires a claim; a missing/invalid image leaves the backend at ownership rung 1. |
| `gsp_phys_mdev_owned` | 0 | Deferred, refcounted lifecycle: first mdev create claims+boots asynchronously, last remove releases; every failure is fail-open to the emulation path. | Requires `gsp_phys_gpu=1` + `gsp_phys_hazard_ack=1`; a module-owned claim stays up until `rmmod`. |
| `gsp_phys_fwsec` | 0 | Run the FWSEC/FRTS secure-verification carve (device writes). | Requires the layout to have succeeded; **refused unless `gsp_phys_boot=1`**. |
| `gsp_phys_fwsec_dryrun` | 0 | Read-only validation of the VBIOS/BIT/FWSEC parse and the staging/placement math. | Requires a claim; no device writes. |
| `gsp_phys_fwsec_verbose` | 0 | Extra bounded read-only FWSEC kick state dump (falcon regs, scratch, DMEM/IMEM readback) after a successful halt. | Diagnostic; failure path always dumps. |
| `gsp_phys_static_live` | 0 | Overlay the live GPU name/short-name/VBIOS identity from the READY snapshot onto the emulated `GET_GSP_STATIC_INFO` reply (bounded 2 s wait, fail-open). | Requires a READY backend snapshot; other static fields stay emulated. |
| `gsp_phys_ctrl_forward` | 0 | Forward a whitelist of side-effect-free `GSP_RM_CONTROL` queries (bus BAR info, bus info, FB info, GMMU static info) with the adopted server handles; every miss falls back to the emulated handler. | Requires READY; fail-open; per-cmd forwarded/fallback counters. |
| `gsp_phys_work_forward` | 0 | Request one real CE shadow copy per eligible decoded guest push on the ladder channel; the software release path stays authoritative. | Requires a live claim + RM init-done + a built ladder channel; all CE forwarding paths refuse cleanly on a GR ladder. |
| `gsp_phys_guest_mem` | 0 | Let the ladder's guest triggers (6/7) and the forwarding path operate on a pinned guest GPA resolved through the guest-RAM backing file. | Triggers require READY, ladder stage ≥ 3, an identity DMA domain, an active guest-RAM file and a pinned GPA (6/7). |
| `gsp_phys_work_defer_release` | 0 | Defer a forwarded push's guest-visible release until the hardware copy completes (bounded 500 ms), then apply it from the service thread. | Requires `gsp_phys_work_forward=1` + `gsp_phys_guest_mem=1`; with 0 the release applies inline. |
| `gsp_phys_work_hw_primary` | 0 | Skip the software copy for a forwarded guest CE push (hardware authors the data); one software fallback copy on any hardware/queue failure. | Requires `gsp_phys_work_forward=1` + `gsp_phys_guest_mem=1` + `gsp_phys_work_defer_release=1`. |
| `gsp_phys_relay` | 0 | Let the engine author the selected guest-visible semaphore release into the guest page (third PTE on the same fresh line); the hold verifies the word and skips that record's software apply. | Requires `gsp_phys_work_forward=1` + `gsp_phys_guest_mem=1` + `gsp_phys_work_defer_release=1`; default off pending mismatch triage. |
| `gsp_phys_fault_relay` | 0 | On a relay timeout, drain the status queue for the engine fault (`0x1006`, corroborated by `0x1020`), match runlistId/chid and write the guest channel notifier + event. | Requires `gsp_phys_relay=1` and the three parameters above. |
| `gsp_phys_fault_inject` | 0 | Debug: write the relayed pair's source PTE as zero to force an engine fault on the next relay submission; only relayed pairs are affected. | Requires the relay path; default 0 programs the real source HPA. |
| `gsp_phys_rmcall_engine` | 0 | Ladder engine: 0 = COPY0 (CE copy submission), 1 = GR0 (compute object, BIND engineType 1, GR semaphore release). Snapshotted when stage 3 starts; mid-session changes are ignored. | Requires boot + init-done; CE forwarding paths refuse on a GR ladder. |
| `gsp_phys_rmcall_gr_promote` | 0 (clamp 0..3) | GR context-promote bisection: **0 never sends `PROMOTE_CTX`** (the ladder stops after the geometry queries, context buffer/PT allocation, PDB mapping and channel/compute alloc, before BIND); 1 = single MAIN entry, initialize/no-nonmapped, size raw+subctx; 2 = two-phase PA-only then VA-only; 3 = single entry with the exact raw size. Nonzero enables can assert GSP-RM (`0x52a8b04`) and wedge the device. | GR ladder only; value snapshotted at stage 3; **requires confirmed reboot readiness** before enabling. |
| `gsp_phys_repower` | 0 (latched) | Restore the pre-claim runtime-PM/D3cold policy when releasing the GPU; with 0 the D0 locks and runtime reference are deliberately left held until reboot. | **Never enable** (rule 3); re-arming D3cold after a claim can hard-freeze the host. |

The emulation-side `sema_*` parameters (ring discovery, release recovery,
diagnostic watches, grace windows) are not physical-backend controls; their
canonical description is [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md)
§3.2 and [`03-architecture/06`](03-architecture/06-host-module-and-hybrid-model.md).

### 10.3 The routine supervised cycle

One cycle is one attempt; there is no second attempt in the same boot after a
hard signature.

**Step 0 — pre-flight (before anything is loaded):**

* Host clean: no QEMU process, no mdev under `/sys/bus/mdev/devices/`, the
  module not loaded, the physical GPU bound to the distribution driver.
* Record `boot_id`, pstore contents, module sha256/`srcversion`, and the
  planned parameter set in the cycle log.
* Confirm supervision (present human), a written recovery plan, reboot
  readiness per rule 2, and that other host work is finished.

**Step 1 — build and identity check.** Rebuild the reviewed source; record the
module hash; never run a cycle against a stale or unknown module.

**Step 2 — load.** Prerequisites, then `open-iov-vfio.ko` with the reviewed
parameters. Module-owned: the claim and requested stages run at load.
Mdev-owned: load, then create the mdev (the async claim/boot starts) only when
no VM is live.

**Step 3 — claim/boot prerequisites.** In the cycle log confirm: `claimed`,
`PCI_COMMAND` bus-master on, BAR lengths, `BOOT0`/`BOOT1`, both falcons, WPR2
down at entry; the D0 settle check passed (claim refuses otherwise); and after
the boot: `stage: primed=1 kicked=1 ran=1 riscv_active=1 rm_init_done=1`, WPR2
up, `scratch14` hand-off bits advancing. Any claim refusal (`refusing claim`,
`board unresponsive`) ends the cycle (rule 5).

**Step 4 — ladder (host-side real submission).** Write `gsp-phys-rm-call`
values 1..4 cumulatively; the ladder remembers the highest completed stage:

| Write | Stage | Green signal |
|---|---|---|
| 1 | root/device/subdevice objects | stage 1 complete, handles logged (`hClient`/`hDevice`/`hSubdevice`). |
| 2 | memory registration | stage 2 complete. |
| 3 | VAS + TSG + channel (COPY0 variant walk; GR0 single attempt) | stage 3 complete (`variant`, `hChannel`, GR: `hGr`, `rm_call_ctx`). |
| 4 | real submission: CE copy or GR semaphore release | `REAL COPY OK` (CE) / `REAL GR SEM OK` (GR); trigger 9 re-runs only the GR release. |
| 5 | adopt static-config handles as stage 1 | stage 5 complete; a later 4 runs stages 2–4 with adopted handles. |
| 6 | guest write direction (W2G) on the pinned GPA | `GUEST W2G COPY OK`. |
| 7 | guest read direction (G2H) on the pinned GPA | `GUEST G2H COPY OK`. |
| 8 | fresh-line PTE self-test (no GPA) | `GUEST PTE SELFTEST OK` / `selftest_ok=1`. |
| 9 | GR semaphore re-run on a built GR ladder | `REAL GR SEM OK` or a decoded failure; no retry. |

6–8 require `gsp_phys_guest_mem=1`, a READY backend and ladder stage ≥ 3;
6/7 additionally require the GPA pinned via `gsp-phys-guest-gpa`. On the first
failed stage: stop, drain the status queue, and end the ladder (rule 4).

**Step 5 — guest attach (only if the rung needs it).** Start the VM with a
file-backed shared guest RAM region (`share=yes, prealloc=on`), attach the
mdev, configure `gsp-phys-guest-ram <path> <gpa_base> <offset> <size>`, pin
the GPA, and run the driver-API smoke; the guest must stay green (`ALL OK`
64/64) and any forwarded counters must be explained. Remove the mdev before
`rmmod`; never unload with a live VM.

**Step 6 — evidence capture.** Keep with the cycle: parameter set, module
identity, boot id, `gsp-phys-state`, ladder traces, `gsp-phys-status-drain`
records, NOCAT/`0x1006` decodes, `gsp-phys-log-dump`, and the clean-release
checklist below. Evidence is property of the change that follows.

**Step 7 — teardown/unload.** Remove mdevs/stop the VM, then `rmmod`
(`scripts/stack_manager.sh unload`). The release sends
`UNLOADING_GUEST_DRIVER` once (only after a boot), quiesces the falcon, runs
the FWSEC teardown, frees staging, and rebinds the distribution driver only
after a WPR2-down and board-liveness check, with the D0 locks held through a
bounded post-rebind readiness gate.

**Mandatory clean-release checklist** (every line must hold; any failure stops
the cycle and invokes the fail-safe):

| Check | Green value | If it fails |
|---|---|---|
| Boot id | unchanged from pre-flight | the device was power-cycled mid-cycle: discard in-window counters and rebuild the plan; expect a fresh scratch pool at the next boot. |
| Module identity | sha256/`srcversion` identical before/after | wrong build ran: stop, rebuild, restart the cycle (rule 6). |
| Protected region | `wpr2: ... state=down`; GSP falcon quiesced | fail-safe: **no rebind**, D0 locks + runtime reference stay held, WPR2 up; reboot approval per rule 2; never retry in this boot. |
| Unload RPC | `rm_unload_submitted=1`, processor suspended (mailbox `0x80000000`) | wedged server; fail-safe as above (the release log says `teardown did not clear WPR2` / `refusing the hand-back`). |
| Driver rebind | `released <bdf>; rebound to nvidia (rc=1)` | rebind failure/timeout: locks stay held; the module logs the manual `modprobe nvidia` path; reboot is the clean recovery. |
| Power-domain locks | exactly 5 `power domain left LOCKED (gsp_phys_repower=0):` lines on this host (GPU + board functions + root port) | wrong count: stop; audit the locks before any further cycle. |
| Rail cycles in window | 0 | any rail cycle rewrote the power risk: treat the cycle as unsafe and stop. |
| pstore / leftovers | pstore empty; no QEMU, no mdev, guest-RAM file released | a crash record or leaked state exists: capture it, clean with `stack_manager.sh cleanup-crash`, and do not start another cycle until the host is clean. |

Fail-safe summary: on any WPR2/unresponsive/rebind failure the release does
**not** hand the device back, keeps the power-domain locks (and the runtime-PM
reference) held, and the correct next action is an approved reboot with the
written post-reboot plan - never a second attempt in the same boot.

### 10.4 debugfs inventory

All files live under the module's debugfs directory
`/sys/kernel/debug/open-iov-vfio/`. **The directory and every file are
root-only on this system**: existence and read checks must run as root (a
permission error can look like an absent file). Every writer is inert unless
its gates hold, and every accepted write performs a real RPC or device
action - log it in the cycle record.

| File | Mode | Purpose |
|---|---|---|
| `gsp-phys-state` | 0444 | Read-only state snapshot (§10.5): stage/boot state, RM probe/call, GR context, guest copy and relay counters, fault counters, guest-RAM ranges, WPR2/RISCV/falcon/scratch14, queue headers. |
| `gsp-phys-log-dump` | 0444 | Read-only libos task log buffers (`LOGINIT`, `LOGRM`), raw words plus a decoded view. The raw view is authoritative; decode offline where a script is provided by the evidence set. |
| `gsp-phys-status-drain` | 0444 | One bounded status-queue drain per read, printing the decoded `0x1006` engine-fault and `0x1020` NOCAT records; advances the host read pointer (normal consumer protocol). |
| `gsp-phys-unload-rpc` | 0200 | Write `1`: send `UNLOADING_GUEST_DRIVER` once on a live claimed session; every other value is a logged no-op. |
| `gsp-phys-rm-probe` | 0200 | Write `1`: run one `SET_GUEST_SYSTEM_INFO`/`GET_GSP_STATIC_INFO` probe cycle and cache the static-config snapshot. |
| `gsp-phys-rm-call` | 0200 | Write `1`..`9`: the ladder stages, handle adoption, guest triggers and GR re-run of §10.3 step 4. |
| `gsp-phys-guest-gpa` | 0200 | Write `0x<hex>`: pin one 4 KB-aligned guest GPA for triggers 6/7; one GPA per boot, a different GPA is refused. |
| `gsp-phys-guest-ram` | 0200 | Write `<path> <gpa_base> <offset> <size>`: configure the guest RAM backing file and one GPA window (file path fixed for the module cycle); `-` clears. |

### 10.5 Reading `gsp-phys-state`

The first lines say whether the device snapshot is meaningful at all
(`inactive: physical GPU not claimed` / `stage-3d staging not primed`); a
snapshot is only read from a primed, claimed session. Then, in order:

| Line | Fields | Meaning and invariants |
|---|---|---|
| `claimed` / `boot_param` / `dryrun_param` | claim and requested-stage flags | `claimed=0` with a live ladder stage means a release already ran: stop reading device state. |
| `stage:` | `primed`, `kicked`, `ran`, `riscv_active`, `rm_init_done`, `rm_unload_submitted`, `dryrun`, `tx_seq_num` | The boot lifecycle position. READY work needs `primed=1 ran=1 rm_init_done=1`; `rm_unload_submitted=1` means the session is ending and no new forward is legal. |
| `rm_probe:` | `attempted`, `last_ret`, `last_function`, `last_payload_len` | Static-config probe outcome; function 65 with a nonzero payload is the snapshot source. |
| `rm_call:` | `ran`, `stage`, `last_stage`, `last_ret`, `rpcs`, `hClient`/`hDevice`/`hSubdevice`/`hVas`/`hTsg`/`hChannel`, `variant`, `engine`, `hGr`, `token`, `notifier`, `base` | Highest completed ladder stage and the object handles; `engine` is the stage-3 snapshot (COPY0/GR0). A GR `hChannel` with `token` but a faulting push is the context-commit wall, not a transport failure. |
| `rm_call_ctx:` | `prepared`, `valid`, `pa`, `va`, `size`, `pages`, `ptPages`, `gr_promote_param`, `step`, `entrySize` | GR context buffer/PDB; printed only when a GR context was prepared. `gr_promote_param=0` means `PROMOTE_CTX` is never sent; `step` names the last accepted promote step. |
| `rm_call_guest:` | `gpa`, `hpa`, `g2h_ok`, `w2g_ok`, `selftest_ok`, `gpa_valid`, `phys_domain_identity` | The last guest trigger's translation and proof flags; `phys_domain_identity=1` is the prerequisite for programming host PAs into the engine page tables. |
| `work_guest:` | `req`, `fallback`, `ok`, `verify_ok`, `verify_fail`, `pool_exhausted`, `refused`, `cand_none`/`cand_sys_sys`/`cand_mixed`/`cand_vid_vid`, `resolve_fail`, `next_line`, `capacity` | Forwarded guest copies. Green run: `verify_fail=0`, `pool_exhausted=0`, `refused=0`, `resolve_fail=0`; `next_line ≤ capacity` (512 lines per boot, never reused). |
| `work_guard:` | `copy_skipped`, `deferred`, `applied`, `fallback`, `pending`, `overflow`, `engaged`, `timeouts`, `canceled`, `defer_param`, `copy_soft`, `copy_fail`, `hw_primary_param` | Deferred-release hold. **Invariant: `applied + fallback == engaged`.** `pending` is the live hold gauge (should return to 0); `timeouts`/`overflow`/`canceled` must be 0; `copy_soft` counts the designed inline software fallbacks. |
| `work_relay:` | `requested`, `ok`, `fallback`, `skipped_sw`, `verify_fail`, `verify_transient`, `hold_verify_fail`, `resolve_fail`, `timeouts`, `pool_exhausted`, `refused`, `relay_param` | Engine-authored semaphore relay. Green run: `verify_fail=0`, `hold_verify_fail=0`, `timeouts=0`, `pool_exhausted=0`, `refused=0`, `resolve_fail=0`; `verify_transient` is a retry-cleared mismatch and should be rare and explained. |
| `faults:` | `seen`, `stale`, `drained`, `wait_unmatched`, `epoch` | Fault observation at the status queue: records seen vs discarded as stale, records drained, unmatched waits, and the epoch used to match a relay submission to the fault it caused. |
| `fault_relay:` | `matched`, `unmatched`, `delivered`, `no_event`, `late_release`, `relay_param`, `fault_relay_param`, `fault_inject_param` | Guest delivery: `delivered` is a written notifier **and** event; `no_event` means the notifier was written but the auxiliary event post produced no MSI-X; `late_release` means the release landed before the fault could be relayed (no fault is delivered). |
| `guest_ram:` | `active=1 path=... ranges=...` or `active=0` | Backing-file and GPA-window configuration; `active=0` disables every guest-page path. |
| `wpr2:` / `riscv:` / `falcon:` / `scratch14:` | `state=up/down`; active bit; both falcons' CPUCTL/MBOX/FALCON_OS; secure-scratch bit fields | The fail-safe state. A green release ends with `state=down`, both falcons quiesced, and the booter-unload gate bits complete. |
| `cmd_tx` / `stat_tx` | queue header words | Queue producer/consumer cursors; a stalled status read pointer with an advanced command read pointer means "consumed but never replied" (the wedge signature). |

**Relay data-mismatch semantics (fail-closed).** The relay result is verified
host-side against a source window snapshotted *before* the doorbell. A
mismatch is retried once after 2 ms: if the second read matches, it is a
`verify_transient` (the live guest page was mutated under the first compare)
and the copy stays verified; if it persists it counts `verify_fail`, the
submission returns an error, and the sema service falls back to the software
release. The same fallback covers `hold_verify_fail` (the guest semaphore
re-read did not carry the expected payload). **A software release apply is
never lost or duplicated**, which is why the relay defaults off until
`verify_fail` is triaged.

## 11. Debugging scenario catalogue

Physical-backend symptoms, their exact signal, and the bounded response. The
defensive/new-input scenarios (firmware release, OpenRM build, userspace
bump, device/ISA/ROM change, protected state, matrix/CI) are owned by
[`RUNBOOK.md`](RUNBOOK.md) S1–S10 and are not duplicated here.

| Symptom | Exact signal to look for | Decode / likely cause | First diagnostic | Action |
|---|---|---|---|---|
| Guest GR push faults | `func=0x1006 ... exceptType=0x20 runlistId=0 chid=1 errString="... intr1 80000000"`; NOCAT `recType=4 errorCode=0x20 faultingEngine="GR"` | `0x20` = robust-channel PBDMA error; `intr1` bit 31 = `CTXNOTVALID`; the external-VAS GR engine context was never committed (`ENG_CTX_VALID=0`), so the guest binds via `PROMOTE_CTX` (`0x2080012b`). CE channels are unaffected. | `gsp-phys-status-drain`; `gsp-phys-state` `rm_call`/`rm_call_ctx` | Decode first, no retry. Only a proven promote shape commits the context; keep the client promote path gated by `gsp_phys_rmcall_gr_promote=0` until a variant is proven. |
| `PROMOTE_CTX` wedge | `gsp-nocat: rec seq=N recType=5 errorCode=0x52a8b04 source="ASSERT" faultingEngine=""`, then no fn76 reply (timeout `-110`); status ring stops, server command read pointer advanced | A host/server-RM assert in the promote path - not an engine fault (no `0x1006`). The server consumed the command and wrote no reply. Teardown cannot clear WPR2 (its unlock is fuse-locked on this part). | `gsp-phys-state` (`cmd_tx`/`stat_tx`, `wpr2`), release log | Ladder aborts by design; **do not retry**. Fail-safe holds the device; reboot with written approval (rule 2); bisect exactly one promote variant per cycle after reboot. |
| WPR2 / clean-release failure | `teardown did not clear WPR2 (HI=0x...)`; `release: WPR2 still up after teardown ... refusing the hand-back`; `NOT rebinding ... power domain stays LOCKED` | The booter-unload gate bits were never completed by the wedge, or the board stopped answering; the host-side unlock is fuse-locked. | `gsp-phys-state` `wpr2`, `scratch14`, `stage:` | Fail-safe is the designed state: no rebind, locks held, WPR2 up. Leave the device; reboot with approval. |
| Module unload hang | `rmmod` does not return; `ps` shows the emulation/relay kthreads or work items; mdev still present | Teardown ordering (kthread/workqueue) or an open mdev/VM; the emulation-side cleanup applies. | `ps -e \| grep gsp_emu`; mdev list; `dmesg` tail | Remove mdev/stop VM first; `scripts/stack_manager.sh cleanup-crash`; reboot with approval if it persists (rule 5). |
| Guest `cuInit` wedge / guest flake | guest `cuInit rc=999`, `rmapiControlCacheSet ... NV_ERR_INVALID_PARAMETER`, or a host `gsp-phys-fwsec: unexpected WPR2 already up`; a one-off guest Oops in `free_swap_cache` cleared on retry | On the emulated path this is a known guest-side flake; the WPR2-up line is the FWSEC refusal on a board already in a bad state. | guest dmesg; host `gsp-phys-state` `wpr2`; retry the guest once | If only the guest flaked, restart the guest once. If `unexpected WPR2 already up` appears, stop the cycle and inspect state; never retry FWSEC (rule 4/5). |
| GSP boot failure | `gsp-phys-fwsec: staging not visible before kick (DMEM0 ... IMEM ...)` then `FWSEC/FRTS execution failed: -5`; historical `GSP_INIT_DONE rpc_result=0x59` | Staging DMA did not land (async mdev-owned FWSEC path) or, historically, the pre-boot handshake/identity payload was wrong (`0x59` = operating-system error). | `gsp-phys-state` `stage:`, `gsp-phys-log-dump` | Use the module-owned claim path (proven); the async FWSEC defect is fail-open and off the critical path. Do not repeat a failed boot in the same boot after a hard signature. |
| MMU/PTE faults and the fetch-line rule | engine copy times out with no semaphore; `GUEST CE COPY FAILED: scratch-PT pool exhausted ... capacity=512`; re-translated GPA yields a different HPA | The GMMU caches PTE fetches in 128-byte lines and there is no invalidate: a runtime PTE write into an already-fetched line is invisible. A guest page's HPA can change between translations. | `rm-call: guest HPA changed: gpa=... was ... now ...`; `work_guest` `pool_exhausted`/`next_line` | Always use a fresh never-reused line per pair; pre-build guest-window slots before engine access; pin translated pages; treat pool exhaustion as end-of-budget, not a bug. |
| PBDMA method/class errors | `0x1006` with `intr0` nonzero (a method/class was rejected) versus `intr1` bit 31 (`CTXNOTVALID`); NOCAT `faultingEngine="GR"`; errString `"Channel ID ... intr0 ... intr1 ..."` | Wrong method encoding, class/subchannel or channel state; `intr0 != 0` means the engine rejected the method, so look at the pushed method block, not the context only. | `gsp-phys-status-drain`; compare the method encoding with the engine's class | Fix the method/class encoding; no retry against the same wedged channel; re-run from a fresh ladder. |
| Relay symptoms | `GUEST CE RELAY FAILED: semaphore released ... does not match ... (retried after 2 ms)`; `verify_fail`/`verify_transient`/`hold_verify_fail`; `fallback`/`skipped_sw`; `pool_exhausted`; `guest HPA changed` | One persistent data mismatch (shadow-page/concurrent-guest race or decode offset edge) is the open residual; anything else is the designed fail-closed fallback to the software apply. | `work_relay` line; the relay failure log includes sem target/payload/poll, first differing byte, live/snapshot dumps, `src_mutated`, `snap_vs_dst` | Keep default off until the mismatch is triaged. Abort the feature (`gsp_phys_relay=0`, then `gsp_phys_work_forward=0`) if `timeouts>0` or `verify_fail>0` ever appears. |
| Fault-relay symptoms | `fault_relay delivered=1` versus `no_event=1`; `late_release`; `matched`/`unmatched` by runlistId/chid; `faults epoch` | `delivered` = notifier written and event posted; `no_event` = notifier written but the auxiliary event post found no event object (no MSI-X reaches the guest); `late_release` = release landed first, so no fault is delivered (by design). | `fault_relay` line; `gsp-phys-status-drain` | A `no_event` is an event-path problem, not a fault-decode problem: fix the channel notifier/event wiring. Use `gsp_phys_fault_inject=1` to reproduce the path on demand. |
| Guest driver probe failure | guest sees no PCI device / BAR map fails / probe hangs; host `board unresponsive`, mdev `-EUSERS`; missing `SET_IRQS wired` | Config space, BAR mapping, MSI-X wiring or mdev state on the emulated path - not the physical claim. | guest `lspci -vvv`; host `05-operations/02` §5.1 taxonomy | Follow the taxonomy in [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §5.1; never start a physical claim to debug a guest-side probe failure. |

## 12. Hazard register (crash causes and debugging side effects)

| Hazard | Mechanism | Mitigation / practice |
|---|---|---|
| D3cold re-arm / power-gate host freeze | Releasing the D0 policy or re-enabling runtime PM after a claim lets the root port enter D3; the platform can cut the board rail, and the rebind can fail with a host freeze minutes later (no oops, pstore empty). | `gsp_phys_repower=0` (latched) is the default; the D0 locks cover GPU + board functions + root port; the post-rebind readiness gate must pass before anything is released; never unbind/rebind outside the supervised claim; pstore armed each cycle. |
| Failed mutating RPC wedges GSP-RM | One bad mutating operation (channel-flag incident; `PROMOTE_CTX` assert `0x52a8b04`) leaves the server unable to reply or suspend while WPR2 stays up. | One mutating attempt per staged step; never retry a failed mutating RPC; abort the ladder from the first failure; the release fail-safe keeps the locks and refuses the hand-back. |
| debugfs write side effects | Every 0200 file performs a real RPC/device action when its gates hold; a "test" write changes device or guest state. | Root-only; read `gsp-phys-state` and record the intended effect before any write; one write per action; no trigger writes outside a supervised cycle. |
| `pkill -f qemu-system-x86_64` self-match | The pattern also matches the invoking shell's own command line, killing the agent/session instead of the VM. | Use bracket patterns (`pgrep/pkill -f '[q]emu-system-x86_64'`) or explicit PIDs; stop VMs through `scripts/vgpu_vm.py`. |
| Root-only debugfs | The whole `open-iov-vfio` debugfs directory and its files are root-only; a non-root existence check reports absence. | Run existence/read checks under root; treat "missing" as a permission question first. |
| Module reload/unload hazards | Kthread/workqueue teardown can hang; the IOMMU identity-domain property and the guest-RAM file must not outlive the session; unloading with a live mdev/VM corrupts the teardown order. | Remove mdevs and stop VMs first; `gsp_sema_service_work_quiesce()` before staging is freed (release does this); use `scripts/stack_manager.sh unload`/`cleanup-crash`; audit leftovers after every unload. |
| pstore/journal persistence | A freeze leaves no oops but usually a persistent pstore record; the journal stops. | Check `/sys/fs/pstore` before and after every cycle; never clear it; capture it before any reboot or retry - it is the only post-freeze evidence. |
| Guest-RAM shared-file requirement | The engine can reach guest pages only through the QEMU file-backed shared region; a non-shared or non-preallocated RAM backend has no host page cache to translate. | Start QEMU with `memory-backend-file,share=yes,prealloc=on`; configure the file via `gsp-phys-guest-ram`; one file per module cycle; translation is gated on an open mdev and an active range. |
| Release-before-verify race | A hardware copy that times out cannot be canceled, so a late engine write is theoretically possible after the software fallback already applied. | The deferred-release hold withholds the guest release until completion or the bounded window; the software apply is never lost; if `timeouts`/`verify_fail` appears, abort the feature rather than tune around it. |
| Single-inflight limit / deferred-release hold | One forwarded request is in flight at a time; while a hold waits, new forwards are refused and apply their releases inline. | Designed fail-closed: `copy_soft` fallback, 500 ms cap, `pending` returns to 0. Do not deepen the queue without a measured need. |
| Scratch-pool exhaustion | 512 fresh 128-byte PTE lines per boot and no reuse; each copy pair consumes one whole line. | Size a cycle's guest-copy workload to the pool; treat `pool_exhausted` as the designed end-of-budget, then reload/reboot for a fresh pool. |

## 13. The physical-backend OODA loop

| Stage | What to do | Exit criterion |
|---|---|---|
| **Observe** | Collect: the `gsp-phys-state` snapshot (§10.5), ladder traces and stage numbers, `gsp-phys-status-drain` records, decoded `0x1006`/`0x1020` faults, `gsp-phys-log-dump`, the clean-release checklist, module sha/boot id, and the evidence set. | The cycle's question can be stated in one discriminating sentence with its parameter set and run identity. |
| **Orient** | Separate static from dynamic deltas. Classify failures: **engine fault** (`0x1006` + NOCAT `faultingEngine` set), **host/server assert** (NOCAT `recType=5`, `faultingEngine=""`, no `0x1006`), **transport** (timeout `-110`, status read pointer stalled while the command pointer advanced), or **data-verify** (`verify_fail`/`hold_verify_fail`). Match relay faults by `epoch` plus runlistId/chid and honour the external-VAS rule (`ENG_CTX_VALID` must be committed by a proven promote before a GR push can complete). | Every delta has exactly one class, or is recorded as an open gap. |
| **Decide** | Choose one: fix / observe more / extend tooling / record an accepted gap. Bisect with exactly one variable per cycle (parameter variant, one promote shape, one method encoding); never retry a wedged path, and never decide "fix" from a single run or a static reading. | The decision names the evidence and the single changed variable. |
| **Act** | Land the code change with the evidence set attached; commit only after a live green result on the same stimulus; fail-safe first (the release must refuse the hand-back before any optimisation). | Same stimulus is green, the clean-release checklist holds, and the change carries its evidence. |
| **Stop rules** | Hard signatures (rule 5), the timebox (three observe/compare iterations without a discriminating delta = stop and write down what is missing), the reboot policy (rule 2), and **no unattended runs** (rule 1). | Nothing physical runs past a stop rule; the state and hypotheses are recorded for the next supervised window. |

## 14. Work-context map and porting flows

### 14.1 Which document owns which debugging context

| Debugging context | Owning document |
|---|---|
| Emulated-driver debugging (guest driver vs the emulated GSP-RM) | [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §1–3, §5 (observability, captures, gdbstub, failure taxonomy). |
| Firmware runtime (GSP/uproc in a RISC-V VM, hooks/watchpoints/sessions) | [`05-operations/03`](05-operations/03-risc-v-gsp-rm-vm.md) and the [green-eye user guide](05-operations/04-green-eye-user-guide.md) (experimental). |
| Live physical backend (claim/boot, ladder, forwarding, relay, release) | **This Part II.** |
| Portability (firmware, hardware, driver version, userspace version) | [`06-portability/*`](06-portability/hardware-software-matrix.md) and [`RUNBOOK.md`](RUNBOOK.md) S1–S10. |
| Performance and correctness validation | [`05-operations/01`](05-operations/01-performance-correctness-security.md) (evaluation ladder, metrics, A/B method). |
| Architecture of the backend and the guest wiring | [`03-architecture/08`](03-architecture/08-physical-gpu-integration.md), [`03-architecture/11`](03-architecture/11-physical-backend-guest-wiring.md). |
| Current frontier and dated status | [`STATE-CURRENT.md`](STATE-CURRENT.md). |

### 14.2 Triage flow for a CUDA/userspace symptom

1. **Guest diagnostics first** ([`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §2–3): guest dmesg, PCI config, `nvidia-smi`, the driver-API smoke result, and the exact failing call.
2. **Classify**: emulated reply (guest failed against the emulation plane) vs forwarded reply (`gsp_phys_static_live`/`gsp_phys_ctrl_forward`/`gsp_phys_work_forward` counters moved) vs engine fault (`0x1006`/NOCAT) vs guest-side flake (WPR2 untouched).
3. **Check the owning state/log**: for forwarded paths `work_guest`/`work_guard`/`work_relay`/`fault_relay` in `gsp-phys-state`; for faults `gsp-phys-status-drain`; for boot/claim `stage:`/`wpr2:`; for emulation-only symptoms the emulation trace.
4. **Next step**: emulation handler change under [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) discipline; forwarding parameter/ladder re-run under §10.3; fault decode and no-retry path under §11; guest-flake restart once. A physical claim is never the first response to a guest-side probe failure.

### 14.3 Porting flows

| New input | What changes | What must be validated | Procedure |
|---|---|---|---|
| Firmware image | The staged ELF, its signature section and the boot handshake; the static-config snapshot content. | ELF64 RISC-V shape and pairing; FWSEC/booter/RISC-V stages; RM init-done `rpc_result=0`; full static-config reply; one supervised cycle before any emulation overlay change. | [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §4.1; [`06-portability`](06-portability/hardware-software-matrix.md); [`RUNBOOK.md`](RUNBOOK.md) S1. |
| GPU hardware | Device profile (PCI IDs, BAR geometry, real FB size), power-domain topology (lock count), engine inventory, identity overlay. | Claim/D0 lock and clean release on the new board; boot layout; engine capabilities; a supervised cycle at the target rung; matrix entry only once earned. | [`06-portability`](06-portability/hardware-software-matrix.md); [`RUNBOOK.md`](RUNBOOK.md) S5. |
| Kernel driver version | The distribution driver that is unbound/rebound; module build against the kernel; OpenRM build glue if that path is used (never a runtime dependency). | Build order and symbols; claim/release hand-back; no regression on the module-owned session. | [`RUNBOOK.md`](RUNBOOK.md) S2; [`03-architecture/06`](03-architecture/06-host-module-and-hybrid-model.md) §7. |
| CUDA/userspace version | Guest-visible controls/replies and the userspace crash surface; the emulation's reply fidelity. | Driver-API smoke `ALL OK` 64/64; control/ioctl inventory diff explained; A/B on the same anchor. | [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §4.2; userspace CUDA playbook; [`RUNBOOK.md`](RUNBOOK.md) S4. |

### 14.4 Current frontier (2026-10-02)

Guest CE copies are **hardware-authored with verified data** and the relay and
fault-relay paths are in place, all default off (the deferred-release and
hardware-primary invariants hold; one relay data mismatch is the open
triage residual). The host GR ladder reaches the **context-promote wall**:
the GR0 channel and compute object allocate and the engine path is encoded,
but the external-VAS context is never committed, and the first host promote
attempt asserted GSP-RM (`0x52a8b04`) without a reply and left the protected
region up. The device is therefore parked in the **designed locked
fail-safe** (module unloaded, GPU unbound, D0 locks held, WPR2 up) awaiting a
supervised reboot; the promote path stays gated behind
`gsp_phys_rmcall_gr_promote=0` until a bisected variant is proven. Live detail
and evidence pointers: [`STATE-CURRENT.md`](STATE-CURRENT.md).

<!-- change (2026-10-02): correct the porting/packaging tool section from
planned interfaces that no longer match the landed scaffolds (docs 28/29) to
the actual flags and outputs, link the device-validation sequence and
future-eng-ops admission evaluations, and mark the scaffolds unadmitted;
scope: eng-ops §3 axis 13, §6, §7, §8 only; benefit: readers no longer build
against non-existent flags or treat unadmitted scripts as routine pipeline. -->