From c17da06d6cd8b9c69449904637bf37729a962989 Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Fri, 2 Oct 2026 11:28:49 -0400 Subject: docs: engineering-operations playbook and project-wide status accuracy pass - docs/eng-ops.md gains Part II, live physical-backend engineering operations: hard rules, the full module-parameter matrix, the routine supervised cycle and clean-release checklist, the debugfs inventory, how to read gsp-phys-state, a debugging scenario catalogue (CTXNOTVALID/promote wedge, fail-safe WPR2, unload hangs, guest wedges, MMU/PTE and PBDMA faults, relay symptoms), a hazard register (crash causes and debugging side effects), the physical-backend OODA loop, and the work-context map with porting flows - architecture, implementation, operations, orientation, getting-started, portability, ecosystem and reference docs corrected to the current state: CUDA driver-API smoke green, CE-class work forwarding and engine-authored releases plus fault relay in place (default off), GR execution blocked on engine-context validity, the promote wedge and designed locked fail-safe, and P5 wave-2 subsumed - internal-reference cleanup in tracked docs (no analysis-source or local-archive references); historical architecture document bannered - README reading list and STATE-CURRENT frontier refreshed; new pages (runbook, green-eye user guide, porting sequence, porting/packaging notes) linked from the tree Link check: scripts/check-docs-links.py passes (56 files, no broken links). --- docs/eng-ops.md | 850 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 850 insertions(+) create mode 100644 docs/eng-ops.md (limited to 'docs/eng-ops.md') diff --git a/docs/eng-ops.md b/docs/eng-ops.md new file mode 100644 index 0000000..117bd0c --- /dev/null +++ b/docs/eng-ops.md @@ -0,0 +1,850 @@ +# Engineering Operations: tooling admission, workflows and retirement + +> How Open-IOV decides which engineering tools it uses, extends and retires, +> and how it measures whether a tool pays for its maintenance. Maintainers +> decide and record; agents execute bounded loops and produce evidence. +> Subsystem manuals are linked, not duplicated. + +## 1. Purpose and scope + +**Who uses it.** Maintainers decide, record and review; agents (MCP, CI, cron) +execute bounded loops and produce sessions, reports and bundles. The same rule +binds both: evidence or it did not happen ([`onboarding-and-standards.md`](07-developers/onboarding-and-standards.md) §5, [`15-AGENTIC-GSP-IDE.md`](../debug/risc-v-gsp-rm-vm/docs/15-AGENTIC-GSP-IDE.md) §5). + +| Governed here | Owned elsewhere (authoritative) | +|---|---| +| Admission, extension, redesign, trialing and retirement of tools | Architecture ([`03-architecture/01-system-overview.md`](03-architecture/01-system-overview.md)); tool manuals: CUDA ([`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md)), green-eye ([`26-ARCHITECTURE.md`](../debug/risc-v-gsp-rm-vm/docs/26-ARCHITECTURE.md), [user guide](../debug/risc-v-gsp-rm-vm/docs/02-USAGE.md), [scripting](../debug/risc-v-gsp-rm-vm/docs/03-SCRIPTING.md)), rmharness ([`README`](../debug/risc-v-gsp-rm-vm/rmharness/README.md)) | +| Which tool applies in which loop; automation and packaging | Accuracy M1–M6 ([`06-ACCURACY.md`](../debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md)); performance method ([`05-operations/01`](05-operations/01-performance-correctness-security.md) §3); version discipline ([`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §4) | +| Evidence bundles, version-span gates, frontier admission | Per-tool docs and the periodic [`27-TOOL-AUDIT.md`](../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md) | +| Live physical-backend engineering operations: supervised-cycle mechanics, parameter gates, debugfs controls and the hazard register (**Part II** below) | Physical-backend architecture ([`03-architecture/08`](03-architecture/08-physical-gpu-integration.md), [`03-architecture/11`](03-architecture/11-physical-backend-guest-wiring.md)); defensive/incident scenarios ([`RUNBOOK.md`](RUNBOOK.md)); current frontier ([`STATE-CURRENT.md`](STATE-CURRENT.md)) | + +**Explicitly out of scope.** (1) Security claims: the simulator and its device +maps are study environments, never a confidentiality or isolation boundary +([`06-ACCURACY.md`](../debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md) §6, +[`GREEN-EYE-ROADMAP.md`](../debug/risc-v-gsp-rm-vm/GREEN-EYE-ROADMAP.md) V4). +(2) Timing claims: L2 is not modelled; order-of-events only +([`06-ACCURACY.md`](../debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md) §2). +(3) Shipping injected code: development-only, never a firmware patch, never in +a guest ([`12-INJECTION-SDK.md`](../debug/risc-v-gsp-rm-vm/docs/12-INJECTION-SDK.md) §1). +(4) Support and licensing: tested-and-documented only +([`hardware-software-matrix.md`](06-portability/hardware-software-matrix.md) §1); +legal rules in [`legal-and-licensing.md`](09-reference/legal-and-licensing.md). + +## 2. The core rule: no insertion without a measurable outcome + +A tool, or an insertion of an existing tool into a workflow, is admitted only +with all five fields below. Missing any field rejects the proposal; it is not +deferred until the metric is invented later. + +| Field | Meaning | Example | +|---|---|---| +| **Scenario** | exact question, target, version and loop where the tool is inserted | "classify one new GSP image's dispatch-table delta before the emulation is touched (portability loop)" | +| **Measurable outcome** | result with a unit and an observer | "image drop → `version-diff.md` delta table; sessions per finding" | +| **Baseline** | the replaced method and its measured/recorded cost | "static corpus diff + one supervised boot per hypothesis; baseline in [`docs/25`](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md) §3" | +| **Evaluation threshold** | the number that decides keep/extend/retire, with sample size and environment | "N ≥ 3; median TTFA ≤ 25 % of baseline; self-diff zero; M-table improves or stays equal" | +| **Failure signal** | observable "not useful" outcome during the pilot | "unexplained `compare` deltas after two triage passes, or the tool needs hardware to run" | + +Rules: one scenario per record; the threshold is recorded **before** the first +run and moving it needs a new record; every admission starts as `admit-pilot` +with an expiry of one quarter or five sessions; the baseline must be observable +(measure it first if unrecorded); a missed threshold triggers §4.4/§4.5. + +```text +TOOL ADMISSION RECORD id: TOOL-- +tool: +loop: static-RE | dynamic-RE | porting | performance | correctness | CI | guest-compat +owner/decider: ; executor: +scenario: +measurable: +baseline: +threshold: keep if <...>; reject if <...>; N=; environment=<...> +failure signal: +evidence: +decision: admit-pilot | admit | extend | redesign | retire | do-not-build +review-by: trials-dir: +``` + +The record is stored with the evidence (session directory, bundle or PR +description); decisions are recorded by a human maintainer and mirrored in the +register (§4.5) and revision notes (§9.3). At expiry: met → admit; not met → +redesign or retire; no signal → retire. No unused pilot survives. + +## 3. Value axes: what tooling is for, and how it is measured + +Each admission names exactly one axis. Every axis reduces to moving evidence +through a §3.3 loop; if it cannot be expressed as a time or a count, it is not +admissible. + +| # | Axis | Concrete example (real tooling) | Metric (recorded) | Loop | +|---|---|---|---|---| +| 1 | Faster reverse engineering | `version_diff.py` + `compare` on two GSP images; Ghidra corpus diff | firmware drop → classified delta (TTFA); symbols resolved vs Ghidra-only | RE | +| 2 | Deeper / more correct analysis | watchpoints + TCG/native trace; capability inference ([`docs/14`](../debug/risc-v-gsp-rm-vm/docs/14-ADVERSARIAL-INTEROP-GUIDE.md) §4.2) | attributed bytes/stores; M6 agreement; unexplained-delta count | RE | +| 3 | Code-artifact porting time | `cudatool.py fdematch`, `diff_corpus.py`, `symbols export` | transfer rate per class; wall time old→new symbol DB; wrapper re-resolution errors | portability | +| 4 | Bug-analysis time | `reach.py`, `run_until`/`call`, `ioctl_trace.c`; taxonomy in [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §5 | crash/question → first evidence-backed answer (TTFA); supervised cycles avoided | adversarial | +| 5 | Development velocity on the simulated GSP | injection SDK; `greeneye.sh inject`; value demo s06 | seconds from service edit to direct+mailbox agreement; sessions per prototype | RE | +| 6 | Correctness validation of virtualization | M1–M6 table, `compare`, golden sessions | M-table per change; rung ladder ([`05-operations/01`](05-operations/01-performance-correctness-security.md) §1) | correctness | +| 7 | Performance analysis and improvement | A/B method, `tools/benchmark_infrastructure.sh`, simulator overhead ([`docs/25`](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md) §6) | B/A % per metric; p50/p99 RPC; trace overhead; before/after same machine | performance | +| 8 | Userspace/API compatibility friction | capability negotiation ([`../userspace/cuda/docs/04-virtualization-and-emulation.md`](../userspace/cuda/docs/04-virtualization-and-emulation.md) §1), symbol DB, ioctl diffs | controls with coherent replies vs observed; TTFA for a new capability bit; peer scenarios (RDMA guest, DPU/NIC/NVMe page sharing) counted only once a peer model exists | guest-compat | +| 9 | Hidden / undocumented / unauthorized code, side effects, backdoors | `control_discovery.py`, mutation harness, decoy handling ([`docs/14`](../debug/risc-v-gsp-rm-vm/docs/14-ADVERSARIAL-INTEROP-GUIDE.md) §4.4) | unknown IDs and injected-slot aliases found; capability-model deltas; corpus entries with sessions | adversarial | +| 10 | Architectural understanding and SBOM/doc improvement | symbol DB, `export_hits`, firmware notes ([`docs/09`](../debug/risc-v-gsp-rm-vm/docs/09-GSP-FIRMWARE-NOTES.md)), [`SBOM.md`](../SBOM.md) | doc/SBOM rows updated per finding; onboarding time; subsystem coverage | RE | +| 11 | Security fixes / hardening time | capability inference → fail-closed rules ([`04-implementation/02`](04-implementation/02-practical-challenges-and-adversarial-interop.md) §4), mutation corpus | error paths covered; finding → test time; physical confirmation before product change | adversarial | +| 12 | New GSP support time | `probe` + `version_diff` + `portability_matrix` + `ci_release` bundle | hours from image to triage table; M-table completeness; regression lead time | portability | +| 13 | New hardware-device support time | device profile capture, `rmharness`, matrix, [`docs/22`](../debug/risc-v-gsp-rm-vm/docs/22-PORTING-CUDA-AND-DEVICES.md) §3, [device sequence](06-portability/03-device-validation-and-porting-sequence.md) | profile fields captured; engine inventory validated; first-touch deltas explained; tested-at-rung date | portability | +| 14 | CUDA / runtime / VM-environment fixes | `userspace/cuda/docs/07`, `cuda_drv_test.c`, `tools/cuda-ab/ab.sh`, guest diagnostics | crash → named function time; `RESULT`/`STATUS` match; rung movement | guest-compat | + +Axis 8 note: GPU-direct RDMA availability is negotiated via the stock guest's +vGPU capability block; page-table sharing with simulated DPU/NIC/NVMe peers is +future, so its metric starts only when a peer model exists (§8). + +**Performance axis (axis 7).** Profiling order is the project's existing one +([`05-operations/01`](05-operations/01-performance-correctness-security.md) §3.6): +RPC dispatch → doorbell cadence → eventfd/MSI-X delivery → BAR0 shadows → +memory-model gap. Method: (1) define the metric and A/B pair before code +changes — same binary, inputs, governor, clocks and boot where possible, and +report B/A %, never absolutes; (2) run A via `tools/benchmark_infrastructure.sh` +and B via `scripts/vgpu_vm.py exec`, logging versions and +`nvidia-smi -q -d CLOCK`; (3) attribute with host counters +(`gsp_emu_get_stats()`, `trace_printk`, `/sys/kernel/debug/tracing`) and the +measured simulator overhead ([`docs/16`](../debug/risc-v-gsp-rm-vm/docs/16-CUSTOM-SIMULATOR.md) §1.1a, [`docs/25`](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md) §6); +(4) keep the numbers with the change and re-run correctness; (5) reject single +runs, cross-host comparisons, bare-metal-only data and simulator timing claims. + +**OODA reduction.** Every axis moves evidence through one of three loops. Each +loop's compression is the project's program-level metric +([`GREEN-EYE-ROADMAP.md`](../debug/risc-v-gsp-rm-vm/GREEN-EYE-ROADMAP.md) success +criteria), measured in [`docs/25`](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md) §7: + +| Loop | Observe → Orient → Decide → Act | Compression metric | +|---|---|---| +| RE-constrained (static/dynamic interlock) | probe/hooks/watch/trace → `analyze`/snapshots → `compare`/triage → reimplementation | question → reviewable answer; sessions per finding | +| Adversarial-interop (opaque/decoy firmware) | discovery/mutation/replay → classification ([`docs/14`](../debug/risc-v-gsp-rm-vm/docs/14-ADVERSARIAL-INTEROP-GUIDE.md) §4.1) → fail-closed or no change → emulation change + physical confirm | unknown-ID → classified route; error paths covered | +| Portability (GSP/device/architecture churn) | probe first-touch/symbol diff → pairing check/delta class → anchor decision → versioned dirs + handlers | image drop → triage table (hours) | + +## 4. Use, extend, build, redesign, retire + +```mermaid +flowchart TD + P["Friction + scenario + measurable outcome"] --> Q1{"Existing tool covers the scenario?"} + Q1 -->|"yes"| USE["Admit the insertion; pilot + metric record"] + Q1 -->|"no"| Q2{"A close tool has a public extension point?"} + Q2 -->|"yes"| EXT["Extend in place: plugin, harness Surface, probe spec, SDK"] + Q2 -->|"no"| Q3{"Repeated scenario and no cheaper alternative?"} + Q3 -->|"no"| NB["Do not build: record rejection and the trigger to revisit"] + Q3 -->|"yes"| NEW["Admit a new tool as a bounded pilot"] + USE & EXT & NEW --> REV{"Threshold met after the pilot?"} + REV -->|"yes"| KEEP["Admit; add a second metric over time"] + REV -->|"no"| Q4{"Failure signal points at the design, not the idea?"} + Q4 -->|"yes"| RD["Redesign with a migration note; re-pilot"] + Q4 -->|"no"| PRUNE["Retire / prune; keep evidence read-only"] +``` + +| Decision | Minimum evidence | Decider | Record | +|---|---|---|---| +| (a) Use existing | scenario + measured baseline + one recorded trial session | maintainer; agent may run the trial | admission record + register row; link the session | +| (b) Extend existing | ≥ 3 uses or ≥ 2 users; public extension point (plugin, `Surface`, probe spec, SDK); cost ≤ new tool | maintainer | record names the extension point and keeps the old tests green | +| (c) Build new | record complete; bounded pilot; golden/replay evidence; stdlib/local-only unless justified | maintainer | record + register + revision note; tests added to the tool suite | +| (d) Redesign | failure signal twice, or a fidelity gap that changes conclusions; migration note (`greeneye-session/1` evolves additively) | maintainer | before/after metric; old artifacts stay readable | +| (e) Stop / prune | threshold missed, no evidence in the window, duplicated capability, or maintenance > value | maintainer at the audit | retire row; `27-TOOL-AUDIT.md` status; keep sessions/reports | +| (f) Do not build | any trigger below is true | maintainer | rejection note naming the trigger that would reopen it | + +**When NOT to build (workflow risk).** Reject if any holds; the absence of a +time-to-value process is the risk this document removes: (1) no scenario, unit +or baseline — the pilot cannot fail; (2) the tool would become a shipped +dependency, guest component or firmware patch +([`04-implementation/02`](04-implementation/02-practical-challenges-and-adversarial-interop.md) §5.1); +(3) it needs physical GPU, mdev or guest access for its primary value +([`docs/15`](../debug/risc-v-gsp-rm-vm/docs/15-AGENTIC-GSP-IDE.md) §5); +(4) it bypasses a security boundary or turns a study observation into a +security claim; (5) it duplicates a tool without a measured improvement; +(6) total cost is not plausibly below the replaced OODA cost; (7) its only +evidence is prose — evidence must be a session, report or bundle. + +**Redesign and retirement.** Redesign rather than patch when the failure +signal repeats, a fidelity gap invalidates an accepted conclusion, a public +interface must change incompatibly, or the tool drifted outside its admitted +loop. Retire when no artifact references the tool, the threshold was missed +twice, another tool covers the scenario better, or maintenance exceeds the +saving. Retirement is revocation of admission, not deletion: keep tool, tests +and artifacts read-only, mark the register row `retired` with date and reason, +and reconcile at the next [`27-TOOL-AUDIT.md`](../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md). + +**Register (how decisions are recorded).** The durable record is the register +row, the evidence directory/bundle it names and the revision note in §9.3; +the audit is the periodic reconciliation. + +| Admitted family | Loop | Value evidence (current) | Review / retire trigger | +|---|---|---|---| +| Ghidra corpora + scripts (`tools/ghidra`, `userspace/cuda/tools`) | RE, portability | docs 05/07; `diff_corpus`, name map | corpus diff no longer changes port class | +| green-eye core + session format | dynamic-RE | `docs/25` s01–s10; 155-test suite | format unused by any loop | +| Harness (`version_diff`, matrix, mutation, discovery, reach, `ci_release`) | portability, adversarial | docs 18/19/21; deterministic replay tests | a report is never attached to a change | +| Injection SDK + demo uproc | dynamic-RE | docs 11/12; s06 | prototypes stop using it | +| Ghidra bridges (in/out) | RE | docs 05/13 | static/dynamic join replaced | +| Device map (`greeneye-device-map/1`) | portability | docs 24; H2.1 first-touch | H2.2 model stops consuming maps | +| `rmharness` + `rmtool`/`rmtop`/`rmq` | correctness, guest-compat | docs 17; M1/M6 reference captures | reference capture unused/unavailable | +| MCP server + agent workflows | all loops | docs 15/20; release monitoring | write gating or audit trail removed | +| CI (`ci.yml`, `greeneye.yml`) | correctness | release-blocking tests | suite non-reproducible or skipped | + +## 5. Workflow map: where each tool applies + +### 5.1 Context of the development and reverse-engineering environment + +```mermaid +flowchart LR + subgraph STATIC["Static"] + GH["Ghidra CUDA corpus + GSP firmware"] + end + subgraph DYN["Dynamic (green-eye, experimental)"] + GE["Session + QEMU/native backends"] + HR["harness/*.py"] + end + subgraph HOST["Host and guest"] + MOD["open-iov-vfio.ko"] + VM["stock guest + guest-diag"] + RMH["rmtool/rmtop/rmq + rmharness"] + end + subgraph AUTO["Automation"] + CI["ci.yml / greeneye.yml"] + MCP["MCP server + agents/cron"] + end + EVID["Evidence: sessions, reports, bundles"] + GH -->|"names, index, symbol DB"| GE + GE -->|"export_hits, fault PCs"| GH + GE --> HR --> EVID + GE --> EVID + RMH --> EVID + VM -->|"stimulus and failures"| GE + MOD --> VM + EVID --> AUTO + AUTO -->|"bounded runs"| HR + EVID -->|"product-change gate"| MOD +``` + +### 5.2 Static reverse-engineering loop + +```mermaid +flowchart LR + BIN["libcuda / .fwimage"] --> IMP["Ghidra import + analysis"] + IMP --> IDX["names.tsv, decompiled_index.tsv, callgraph.tsv"] + IDX --> DIFF["corpus diff / fdematch / RPC dispatch diff"] + DIFF --> HYP["Hypothesis: functions, tables, layouts"] + HYP --> SYM["greeneye symbols export (bias recorded)"] + SYM --> DYN["dynamic loop"] + DYN -->|"hits, watch PCs, fault PCs"| HYP +``` + +### 5.3 Dynamic firmware loop (OODA) + +```mermaid +flowchart LR + Q["Question + target + fidelity statement"] --> OBS["Observe: probe, hooks, watch, trace"] + OBS --> ORI["Orient: analyze, symbol decode, snapshots"] + ORI --> DEC["Decide: compare, M1-M6, mutation"] + DEC --> ACT["Act: emulation change with evidence"] + ACT --> REG["Regression: same stimulus, same budgets"] + REG -->|"improved or equal"| PACK["Evidence pack + session hashes"] + REG -->|"regressed"| Q + PACK --> Q +``` + +### 5.4 Porting loop (GSP firmware, device, architecture) + +```mermaid +flowchart TB + ACQ["Acquire + hash; versioned dirs, never overwrite"] --> TRI["Triage: delta, fdematch, pairing check"] + TRI --> PRB["Probe first-touch old vs new"] + PRB --> RUN["Rehost/demo: identical stimulus + record"] + RUN --> CMP["compare + M1-M6 skeleton"] + CMP --> BUN["version_diff -> matrix -> mutation -> bundle"] + BUN --> REV{"Every delta explained?"} + REV -->|"no"| TRI + REV -->|"yes"| ANCHOR["Anchor decision + docs + matrix entry"] +``` + +### 5.5 Performance loop + +```mermaid +flowchart LR + DEF["Define metric + A/B pair"] --> BASE["A: bare metal baseline"] + DEF --> CAND["B: vGPU / candidate"] + BASE --> RATIO["B/A ratio, variance, N runs"] + CAND --> RATIO + RATIO --> PROF["Attribute: RPC, doorbell, eventfd, shadows"] + PROF --> KEEP["Keep numbers with the change"] + KEEP --> REG["Re-run correctness corpus + ladder"] +``` + +### 5.6 Correctness and CI loop + +```mermaid +flowchart LR + CH["Change"] --> HOST["Host suite: stack_manager.sh test"] + HOST --> GUEST["Guest ladder: probe, nvidia-smi, cdt"] + GUEST --> AB["cuda-ab: RESULT/STATUS must match"] + AB --> SESS["Session compare / golden data"] + SESS --> GATE{"Rung or metric regressed?"} + GATE -->|"no"| LAND["Land with evidence"] + GATE -->|"yes"| CH +``` + +### 5.7 Guest-compatibility loop + +```mermaid +flowchart LR + GD["guest-diag: ioctl trace, engines, pmap"] --> CTRL["Control/ioctl inventory"] + CTRL --> EMU["Emulated reply fidelity"] + EMU --> GD2["New driver/CUDA version trace"] + GD2 --> DIFF["Control diff old vs new"] + DIFF --> CTRL + CTRL --> CORPUS["CUDA corpus + symbol DB"] +``` + +### 5.8 Loop index and detailed docs + +| Loop | Entry points | Evidence | Gate | Detail | +|---|---|---|---|---| +| Static RE | `tools/ghidra/*`, `userspace/cuda/tools/*`, `analyzeHeadless` | corpus TSVs, name map, symbol DB | addresses re-derived per build | [`userspace/cuda/docs/07`](../userspace/cuda/docs/07-version-upgrade-playbook.md), green-eye docs 05/13 | +| Dynamic RE | `greeneye.sh` probe/record/analyze/compare/inject/ghidra, Python `Session` | `greeneye-session/1` + `summary.md` | A/B pair; L2/L3 statement | [`26-ARCHITECTURE.md`](../debug/risc-v-gsp-rm-vm/docs/26-ARCHITECTURE.md) §11, [`11`](../debug/risc-v-gsp-rm-vm/docs/11-GSP-RM-SERVICE-DEVELOPER-GUIDE.md), [`12`](../debug/risc-v-gsp-rm-vm/docs/12-INJECTION-SDK.md), [`13`](../debug/risc-v-gsp-rm-vm/docs/13-GHIDRA-CONTROL.md), [`14`](../debug/risc-v-gsp-rm-vm/docs/14-ADVERSARIAL-INTEROP-GUIDE.md) §4.6, [`06`](../debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md) §5 | +| Porting | `version_diff.py`, `portability_matrix.py`, `mutation_harness.py`, `ci_release.py` | reports + `bundle-manifest.json` | every delta explained | [`07`](../debug/risc-v-gsp-rm-vm/docs/07-PORTABILITY-PLAYBOOKS.md), [`22`](../debug/risc-v-gsp-rm-vm/docs/22-PORTING-CUDA-AND-DEVICES.md), [`21`](../debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md), runbook [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §4.1 | +| Performance | `benchmark_infrastructure.sh`, module counters, `vgpu_vm.py` | B/A table with environment | same-machine A/B; corpus green | [`05-operations/01`](05-operations/01-performance-correctness-security.md) §3 | +| Correctness | `stack_manager.sh test`, `make tests`, `cuda_drv_test.c`, `cuda-ab` | rung + trace excerpts | no rung regression | [`05-operations/01`](05-operations/01-performance-correctness-security.md) §1–2 | +| CI | `.github/workflows/ci.yml`, `.github/workflows/greeneye.yml` | workflow logs | push/PR path triggers | [`21`](../debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md) §5 | +| Guest-compat | `tools/guest-diag/*`, `cudatool.py`, `diff_corpus.py` | ioctl/control diff, corpus delta | stock guest only | [`READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md) | + +## 6. Automation and agent invocation + +```mermaid +flowchart LR + PRE["Preflight: greeneye.sh info; RM610 gate"] --> DRY["ci_release.py --dry-run (plan only)"] + DRY --> REAL["ci_release.py real run"] + REAL --> VD["version_diff"] + REAL --> PM["portability_matrix"] + REAL --> MU["mutation smoke"] + VD & PM & MU --> MAN["release-report.json + bundle-manifest.json"] + MAN --> REV["agent/human review"] + REV -->|"accepted"| REG["anchor/register/docs updated"] + REV -->|"rejected"| GAP["fidelity gap or fix, recorded"] +``` + +Run everything from `debug/risc-v-gsp-rm-vm/`; scripts are stdlib-only and +launch their own QEMU. Never re-use an existing gdbstub, a guest, a kernel +module or hardware. + +**Scripts that exist** (success and failure are read, not assumed): + +| Script | Safe invocation | Success | Failure signal / limit | +|---|---|---|---| +| `harness/version_diff.py` | `python3 harness/version_diff.py --old --new --out dumps/vdiff` | sections `ok`; self-diff zero; deltas explained | exit 2 tool error, 3 RM610 refusal; raw `.fwimage` faults early by design | +| `harness/portability_matrix.py` | `python3 harness/portability_matrix.py --target --target --out dumps/matrix` | self-comparison `all_zero`; M1–M6 `not measured` without reference | non-zero target deltas or per-target `status: error` | +| `harness/mutation_harness.py` | `python3 harness/mutation_harness.py --out dumps/mut --iterations 64 --seed 7` | deterministic `findings.jsonl`; declared outcome classes | seed divergence; non-demo ELF refused without `--allow-foreign-elf` | +| `harness/control_discovery.py` | `python3 harness/control_discovery.py --surface demo --out dumps/disc --seed 7` | known IDs resolve; unknown/injected classes listed | RM surface exits cleanly until H1.3; misclassification or unreplayable probe | +| `harness/reach.py` | `python3 harness/reach.py --target rpc_ping --method call --args 0x41 --out dumps/reach` | `reachable` with recorded route and args | timeout means "not under this stimulus", not unreachable | +| `harness/ci_release.py` | dry-run first: `python3 harness/ci_release.py --old --new --out dumps/release --dry-run` | dry-run touches no target; real run writes the bundle | exit 1 step error, 2 tool error, 3 RM610 refusal; `skipped` is not `pass` | +| `greeneye.sh` | `./greeneye.sh info`/`probe`/`record`/`analyze`/`compare`/`inject` | session dir with `session.json` + `summary.md` | empty sessions or missed hooks; budgets required | +| `rmharness/harness.sh` | `./rmharness/harness.sh --session dumps/real -- rmtool engine list --format=json` | `rm_harness_run` event + stream blobs | read-only only; permission errors reported, never escalated | +| `plugins/mcp_server.py` | `python3 plugins/mcp_server.py --stdio --record` | 24-tool JSON-RPC surface; read/observe default | stdout protocol-only; writes need `--allow-write`/`--allow-inject` | +| `harness/value_demos/run_all.py` | `python3 harness/value_demos/run_all.py` | one `RESULT_JSON` per demo; suite ≤ 180 s | any demo `success: false`; re-run before quoting numbers | + +**Scaffolded scripts (landed, not admitted yet).** The three scripts below +landed with docs 28/29 and are pilots until they have a §2 record; the first +two are dry-run by default (`--run` executes). `--help` is the authoritative +flag list; the outputs and failure readings are in +[`docs/28`](../debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md) and +[`docs/29`](../debug/risc-v-gsp-rm-vm/docs/29-PACKAGING-AND-TRIALING.md). + +| Script | Purpose | Interface | Outputs | Exit codes | +|---|---|---|---|---| +| `harness/port_driver_versions.py` | sweep a version list through `version_diff` + matrix + mutation, one evidence set per version | `--versions ` or `--config `, `--baseline `, `--input-dir `, `--target =`, `--symbols =`, `--ghidra-exports =`, `--out `, `--run` (default: dry-run), `--force`, `--extract-runfiles`, `--approval-token` | per-version `version-diff/` + `portability-matrix/` + `mutation-smoke/` + `bundle-manifest.json`, `sweep-summary.json|.md` | 0 complete, 1 step/threshold failure, 2 tool error, 3 RM610 refusal | +| `harness/port_new_device.py` | per-device port: profile, pairing check, first-touch probe, symbols, matrix, checklist | `--device