debug: green-eye wave 3 - RM harness, local simulator, device map, MCP plugin, value-validation demos, CI pipeline, toolchain docs; add host RM tools rmtool/rmtop

1 files changed, 284 insertions, 0 deletions

diff --git a/debug/risc-v-gsp-rm-vm/docs/18-HARNESS-AUTOMATION.md b/debug/risc-v-gsp-rm-vm/docs/18-HARNESS-AUTOMATION.md
new file mode 100644
index 0000000..ecd0614
--- /dev/null
+++ b/debug/risc-v-gsp-rm-vm/docs/18-HARNESS-AUTOMATION.md
@@ -0,0 +1,284 @@
+# 18 — Harness automation: portability matrix and mutation harness
+
+> **# TOOL IN DEVELOPMENT - NOT READY YET!** See 00 for status.
+
+Two stdlib-only orchestration tools in `harness/` turn the manual playbooks
+into repeatable runs:
+
+| Tool | Roadmap | What it does |
+|---|---|---|
+| `harness/portability_matrix.py` | **R6**, H4.1 | Runs probe + scripted demo + recorded sessions against several firmware ELFs and emits a cross-target delta report with the M1–M6 skeleton. |
+| `harness/mutation_harness.py` | **R5**, H4.3 | Deterministically mutates the demo firmware's mailbox input surface, classifies abnormal outcomes, and records every attempt as a session plus a compact finding. |
+
+Both use the **QEMU backend only** (the same `probe`/`demo` machinery as the
+CLI). They intentionally do not depend on `greeneye.sim_backend`, and they
+never attach to an existing gdbstub, hardware, a guest or a kernel module.
+
+Requirements: `qemu-system-riscv64`, Python 3.10+, and (for the bundled demo)
+`make demo`. All commands run from `open-iov/debug/risc-v-gsp-rm-vm/`.
+
+---
+
+## 1. Portability matrix
+
+### 1.1 Usage
+
+Self-comparison of one build (a determinism check):
+
+```bash
+python3 harness/portability_matrix.py \
+    --target firmware/uproc_demo/build/uproc_demo.elf \
+    --target firmware/uproc_demo/build/uproc_demo.elf \
+    --out dumps/matrix-demo \
+    --probe-instructions 200 --scenario-duration 0.2
+```
+
+Old vs new firmware with the project's symbol DBs (one DB applies to every
+target in the invocation; run separate invocations for per-version DBs):
+
+```bash
+python3 harness/portability_matrix.py \
+    --target firmware/57X-GSP/570.144/gsp-570.144.fwimage.bin \
+    --target firmware/57X-GSP/570.211/gsp-570.211.fwimage.bin \
+    --ghidra-index analysis/570.144/decompiled_index.tsv --ghidra-bias 0 \
+    --symbols symbols-570.144.json \
+    --out dumps/matrix-570
+```
+
+Replacing the builtin scenario with a fixed green-eye script:
+
+```bash
+python3 harness/portability_matrix.py --target <rehosted.elf> \
+    --scenario-script examples/scripts/trace_rpc.py --out dumps/matrix-rehosted
+```
+
+Flags: `--probe-instructions N` (0 = wall-clock `--probe-duration`),
+`--scenario-duration`, `--max-stops`, `--max-rows`, `--symbols`,
+`--ghidra-names/--ghidra-index/--ghidra-exports/--ghidra-callgraph`
+(`--ghidra-bias`), and repeatable `--target` (positional targets also work).
+
+### 1.2 What each target does
+
+1. **Probe** — the ELF is flattened and relocated to `0x80000000` exactly as
+   `greeneye probe` does, then run for a fixed instruction budget (default
+   2000) or a wall-clock budget. The first exception is parsed with
+   `greeneye.analyze.parse_qemu_faults`; the stopped PC is annotated from the
+   symbol DB. Raw `.fwimage` targets fault early here; that is the expected
+   first-touch data, not a harness failure.
+2. **Symbols** — ELF `.symtab` plus optional Ghidra exports and/or an
+   existing JSON DB, merged and saved to `<out>/symbols/<target>.json`
+   (runtime addresses; the relocation `bias` is recorded in the DB meta).
+3. **Scenario** — the target boots under QEMU with hooks on every `rpc_*`
+   handler that resolves and a write watchpoint on `gsp_ctrl.rpc_count`, runs
+   its self-test, then receives one deterministic host stimulus
+   (`mailbox = PING (1)`, `arg0 = 0x1234`). Everything is recorded as a
+   normal `greeneye-session/1` directory. A re-hosted ELF without `rpc_*`/
+   `gsp_ctrl` symbols still runs; it simply produces an empty profile.
+
+### 1.3 Output format
+
+```text
+<out>/
+  portability-report.json        stable machine-readable report
+  portability-report.md          rendered report
+  symbols/<target>.json          merged symbol DB per target
+  sessions/<target>-probe/       recorded probe session
+  sessions/<target>-scenario/    recorded scripted session
+  compare/<a>--<b>.md            greeneye compare report per target pair
+```
+
+JSON shape (stable keys, targets in CLI order, tables sorted):
+
+```json
+{
+  "format": "greeneye-portability-matrix/1",
+  "tool": "harness/portability_matrix.py",
+  "backend": "qemu",
+  "targets": [{
+    "name": "...", "path": "...", "sha256": "...", "entry": "0x...",
+    "status": "ok",
+    "probe": {"status": "...", "stopped_pc": 0, "stopped_symbol": "...",
+              "fault_count": 0, "first_fault": null},
+    "symbols": {"count": 13, "sources": ["elf"], "path": "symbols/x.json"},
+    "scenario": {"session": "sessions/x-scenario", "event_counts": {},
+                 "bp": {}, "watch": {}, "writes": {}, "ctrl": {}},
+    "metrics": [{"id": "M1", "status": "not measured", "observed": {}}]
+  }],
+  "comparisons": [{"a": "...", "b": "...", "deltas": {"bp": [], "watch": [], "writes": []},
+                   "all_zero": true, "metrics": []}],
+  "skipped": []
+}
+```
+
+Determinism: fixed probe instructions, fixed scenario phases and the same
+stimulus make hit/watch/write profiles reproducible, so an identical target
+compared with itself reports **zero deltas**. The M6 control-block snapshot
+hash is *not* a fixed point while the firmware heartbeats; compare snapshots
+only between quiesced runs. Metrics are marked `"not measured"` unless a
+reference capture exists (docs/06 §5); `observed` carries the local facts the
+matrix could gather.
+
+Targets that are missing, not RISC-V ELF, or too large are reported with
+`status: "skipped"` and a `skip_reason`; a failing step yields `"error"` for
+that step and does not abort the matrix.
+
+---
+
+## 2. Mutation harness
+
+### 2.1 Usage
+
+```bash
+python3 harness/mutation_harness.py --out dumps/mutation-7 \
+    --iterations 64 --seed 7 --timeout 0.25 --settle 0.2
+```
+
+Targeting the command/args surface plus an optional RAM window:
+
+```bash
+python3 harness/mutation_harness.py --out dumps/mutation-ram \
+    --iterations 128 --seed 11 --memory-region 0x80000900:0x100
+```
+
+Budgets: `--iterations` (1..10000), `--seed`, `--timeout` (0.01..10 s total
+observation per mutation; a short first window plus one confirmation window
+for slow handlers), `--settle` (per attempt, until `SELFTEST DONE` is seen),
+and `--max-log-bytes` (QEMU log window per fault check; trap loops flood
+`-d int`). `--kernel` defaults to the bundled demo; a non-demo ELF is refused
+unless `--allow-foreign-elf` is given, and remains QEMU-only.
+
+### 2.2 Strategies
+
+| Strategy | Input surface |
+|---|---|
+| `cmd-invalid` | command IDs outside the demo's valid range (`0, 7..15, 24+, 0x100, 0xffff, 2^64-1`) |
+| `arg-boundary` | one of arg0/arg1/arg2 set to `0, 1, 2, 0xff, 0x100, 0x7fffffff, 2^63, 2^64-1`, ... |
+| `bit-flip` | one bit flipped in the command or an argument of a valid command |
+| `buffer-length` | ECHO/FILL/SUM against `gsp_ctrl.buf` with lengths `0, 1, 31, 32, 33, 255, 256, 257, 4096, 2^32, ...` |
+| `mem-region` | one byte bit-flip inside `--memory-region` (when supplied) |
+
+Each iteration uses a **fresh QEMU session**, settles the demo through its
+self-test, applies exactly one mutation, runs bounded, then reads back the
+control block. No hardware watchpoints are installed, so the only stops are
+budget interrupts and real traps.
+
+### 2.3 Outcome classes
+
+| Class | Detection |
+|---|---|
+| `ok` | command dispatched (or `cmd=0`) and the demo oracle, when one exists, matched |
+| `trap` | first new exception is a breakpoint (`ebreak`), or control transferred below `0x100` |
+| `fault` | first new QEMU exception is a load/store/fetch/illegal fault (`parse_qemu_faults`) |
+| `no-progress` | command written but `rpc_count` did not advance within both windows (no fault) |
+| `unexpected-result` | result/last_rpc contradict the documented demo oracle for `cmd ∈ {0,1,3,5, invalid}` |
+| `exit` / `settle-failed` / `error` | target exited; self-test never observed; iteration exception |
+
+The oracle is deliberately small: handlers that dereference `arg0` (ECHO,
+SUM) and injected slots 0–5 have no oracle, so their successful executions
+are `ok` with `expected: null`.
+
+### 2.4 Output format
+
+```text
+<out>/
+  findings.jsonl      one compact finding per iteration (deterministic order)
+  summary.md          unique outcome classes, strategy coverage, examples, replay command
+  sessions/iter-0000/ recorded greeneye session per iteration
+```
+
+Finding shape:
+
+```json
+{"iteration": 5, "strategy": "arg-boundary",
+ "mutation": {"cmd": 4, "arg0": 0, "arg1": 255, "arg2": 0, "region": null},
+ "outcome": "fault",
+ "detail": {"observed": {"result": 48, "last_rpc": 5, "rpc_count_delta": 0},
+            "fault": {"cause": 5, "desc": "fault_load", "epc": 2147483860,
+                      "tval": 255, "access": {"rw": "read", "access_addr": 255,
+                      "size": 1, "reason": "rejected"}}},
+ "session": "sessions/iter-0005"}
+```
+
+Replay is re-running the same command: the per-iteration RNG is seeded from
+`(seed, index)`, finding fields avoid wall-clock data, and `findings.jsonl`
+is byte-identical for the same seed/budgets on the same host. Session
+directories under `sessions/` contain the usual `events.jsonl`, `session.json`
+and `summary.md`; they are the evidence behind each finding.
+
+---
+
+## 3. Tests
+
+```bash
+python3 -m unittest tests.test_harness -v
+make test        # includes the harness tests with the rest of the suite
+```
+
+The integration tests boot the demo twice for a self-comparison (must show
+zero deltas) and run 12 seeded iterations twice (findings file, at least one
+outcome class, byte-identical replay). They skip cleanly when QEMU or the
+demo ELF is absent.
+
+---
+
+## 4. Attaching reports to a portability change
+
+For the docs/07 P1/P2/P3 loop and the docs/14 §6 evidence pack:
+
+1. Run one matrix invocation per version/device/architecture family with the
+   same scenario options, and keep `<out>/portability-report.json`, the
+   per-pair compare reports and the session directories (or their sha256
+   list).
+2. Record the firmware hash (`sha256` in each target entry), the symbol
+   source list and `bias` from `<out>/symbols/*.json` meta.
+3. Fill the M1–M6 table using `observed` from the matrix plus the real
+   reference capture; the matrix marks agreement metrics "not measured" on
+   purpose. Cite docs/06 L2/L3 limits in any claim.
+4. For an input-surface change, run the mutation harness before and after the
+   change and diff the outcome-class tables; attach `findings.jsonl` as the
+   corpus entry list, with each finding's session as evidence.
+5. Re-run the unchanged stimulus after the emulation change: profiles must
+   improve or stay equal (the docs/06 accuracy loop).
+
+---
+
+## 5. Safety
+
+* Demo / re-hosted firmware under QEMU only. Never physical hardware, never
+  a guest, never a kernel module, never an existing gdbstub or the native
+  simulator.
+* The mutation harness refuses non-demo ELFs unless `--allow-foreign-elf` is
+  passed explicitly, and still refuses non-ELF images.
+* Memory-region mutation is bounds-checked against the QEMU RAM window; no
+  host memory is touched outside the emulated target.
+* Every write is recorded in a session (`mem_write`, `note`, mutation fields);
+  budgets (`duration`/`max_stops`/`--timeout`) bound every run.
+* Only temporary work directories and `<out>/` are written on the host.
+
+---
+
+## 6. Limits and fidelity
+
+* L2/L3 claims are out of scope: no timing, interrupt or device-model
+  conclusions (docs/06). The matrix observes RAM-resident behaviour and the
+  first unmapped access, nothing more.
+* A QEMU log window is bounded; a fault not in the window is recorded as its
+  secondary signal (`trap_loop`/`no-progress`) rather than guessed.
+* Mutation covers one mailbox write per iteration; multi-write races and
+  concurrency are future work (H2/H4.4).
+* The demo oracle is small and documented; "unexpected-result" is only
+  meaningful for the listed commands.
+
+---
+
+## 7. Roadmap mapping
+
+| Item | Where this doc/tool contributes |
+|---|---|
+| **R5 — fuzzing and mutation harness** | `mutation_harness.py`: bounded seeded mutation, outcome classes, recorded corpus (`findings.jsonl` + sessions) |
+| **R6 — multi-version orchestration** | `portability_matrix.py`: probe + rehost/demo + compare + M1–M6 for a target set |
+| **H4.1 — semantic diffing** | identical stimulus per target, `compare_sessions` plus hit/watch/write deltas in one report |
+| **H4.3 — mutation/fuzzing** | error-path corpus with symbol-annotated sessions, feeding fail-closed rules |
+| **H4.5 — reimplementation loop** | reports are the attachable evidence for the docs/07 checklist |
+| **V1 — metrics/golden data** | deterministic profiles and seeded findings support golden comparisons |
+| **H7 — agentic harness** | self-describing JSON/markdown artifacts designed for an agent to drive and review |