# 18 — Harness automation: portability matrix and mutation harness > Status: implemented, documented and tested (audit: docs/27-TOOL-AUDIT.md; value demos: docs/25-VALUE-VALIDATION.md). Two stdlib-only orchestration tools in `harness/` turn the manual playbooks into repeatable runs: | Tool | Roadmap | What it does | |---|---|---| | `harness/portability_matrix.py` | **R6**, H4.1 | Runs probe + scripted demo + recorded sessions against several firmware ELFs and emits a cross-target delta report with the M1–M6 skeleton. | | `harness/mutation_harness.py` | **R5**, H4.3 | Deterministically mutates the demo firmware's mailbox input surface, classifies abnormal outcomes, and records every attempt as a session plus a compact finding. | Both use the **QEMU backend only** (the same `probe`/`demo` machinery as the CLI). They intentionally do not depend on `greeneye.sim_backend`, and they never attach to an existing gdbstub, hardware, a guest or a kernel module. Requirements: `qemu-system-riscv64`, Python 3.10+, and (for the bundled demo) `make demo`. All commands run from `open-iov/debug/risc-v-gsp-rm-vm/`. --- ## 1. Portability matrix ### 1.1 Usage Self-comparison of one build (a determinism check): ```bash python3 harness/portability_matrix.py \ --target firmware/uproc_demo/build/uproc_demo.elf \ --target firmware/uproc_demo/build/uproc_demo.elf \ --out dumps/matrix-demo \ --probe-instructions 200 --scenario-duration 0.2 ``` Old vs new firmware with the project's symbol DBs (one DB applies to every target in the invocation; run separate invocations for per-version DBs): ```bash python3 harness/portability_matrix.py \ --target firmware/57X-GSP/570.144/gsp-570.144.fwimage.bin \ --target firmware/57X-GSP/570.211/gsp-570.211.fwimage.bin \ --ghidra-index analysis/570.144/decompiled_index.tsv --ghidra-bias 0 \ --symbols symbols-570.144.json \ --out dumps/matrix-570 ``` Replacing the builtin scenario with a fixed risc-v-gsp-rm-debugger script: ```bash python3 harness/portability_matrix.py --target \ --scenario-script examples/scripts/trace_rpc.py --out dumps/matrix-rehosted ``` Flags: `--probe-instructions N` (0 = wall-clock `--probe-duration`), `--scenario-duration`, `--max-stops`, `--max-rows`, `--symbols`, `--ghidra-names/--ghidra-index/--ghidra-exports/--ghidra-callgraph` (`--ghidra-bias`), and repeatable `--target` (positional targets also work). ### 1.2 What each target does 1. **Probe** — the ELF is flattened and relocated to `0x80000000` exactly as `./risc-v-gsp-rm-debugger.sh probe` does, then run for a fixed instruction budget (default 2000) or a wall-clock budget. The first exception is parsed with `greeneye.analyze.parse_qemu_faults`; the stopped PC is annotated from the symbol DB. Raw `.fwimage` targets fault early here; that is the expected first-touch data, not a harness failure. 2. **Symbols** — ELF `.symtab` plus optional Ghidra exports and/or an existing JSON DB, merged and saved to `/symbols/.json` (runtime addresses; the relocation `bias` is recorded in the DB meta). 3. **Scenario** — the target boots under QEMU with hooks on every `rpc_*` handler that resolves and a write watchpoint on `gsp_ctrl.rpc_count`, runs its self-test, then receives one deterministic host stimulus (`mailbox = PING (1)`, `arg0 = 0x1234`). Everything is recorded as a normal `greeneye-session/1` directory. A re-hosted ELF without `rpc_*`/ `gsp_ctrl` symbols still runs; it simply produces an empty profile. ### 1.3 Output format ```text / portability-report.json stable machine-readable report portability-report.md rendered report symbols/.json merged symbol DB per target sessions/-probe/ recorded probe session sessions/-scenario/ recorded scripted session compare/--.md risc-v-gsp-rm-debugger.sh compare report per target pair ``` JSON shape (stable keys, targets in CLI order, tables sorted): ```json { "format": "greeneye-portability-matrix/1", "tool": "harness/portability_matrix.py", "backend": "qemu", "targets": [{ "name": "...", "path": "...", "sha256": "...", "entry": "0x...", "status": "ok", "probe": {"status": "...", "stopped_pc": 0, "stopped_symbol": "...", "fault_count": 0, "first_fault": null}, "symbols": {"count": 13, "sources": ["elf"], "path": "symbols/x.json"}, "scenario": {"session": "sessions/x-scenario", "event_counts": {}, "bp": {}, "watch": {}, "writes": {}, "ctrl": {}}, "metrics": [{"id": "M1", "status": "not measured", "observed": {}}] }], "comparisons": [{"a": "...", "b": "...", "deltas": {"bp": [], "watch": [], "writes": []}, "all_zero": true, "metrics": []}], "skipped": [] } ``` Determinism: fixed probe instructions, fixed scenario phases and the same stimulus make hit/watch/write profiles reproducible, so an identical target compared with itself reports **zero deltas**. The M6 control-block snapshot hash is *not* a fixed point while the firmware heartbeats; compare snapshots only between quiesced runs. Metrics are marked `"not measured"` unless a reference capture exists (docs/06 §5); `observed` carries the local facts the matrix could gather. Targets that are missing, not RISC-V ELF, or too large are reported with `status: "skipped"` and a `skip_reason`; a failing step yields `"error"` for that step and does not abort the matrix. --- ## 2. Mutation harness ### 2.1 Usage ```bash python3 harness/mutation_harness.py --out dumps/mutation-7 \ --iterations 64 --seed 7 --timeout 0.25 --settle 0.2 ``` Targeting the command/args surface plus an optional RAM window: ```bash python3 harness/mutation_harness.py --out dumps/mutation-ram \ --iterations 128 --seed 11 --memory-region 0x80000900:0x100 ``` Budgets: `--iterations` (1..10000), `--seed`, `--timeout` (0.01..10 s total observation per mutation; a short first window plus one confirmation window for slow handlers), `--settle` (per attempt, until `SELFTEST DONE` is seen), and `--max-log-bytes` (QEMU log window per fault check; trap loops flood `-d int`). `--kernel` defaults to the bundled demo; a non-demo ELF is refused unless `--allow-foreign-elf` is given, and remains QEMU-only. ### 2.2 Strategies | Strategy | Input surface | |---|---| | `cmd-invalid` | command IDs outside the demo's valid range (`0, 7..15, 24+, 0x100, 0xffff, 2^64-1`) | | `arg-boundary` | one of arg0/arg1/arg2 set to `0, 1, 2, 0xff, 0x100, 0x7fffffff, 2^63, 2^64-1`, ... | | `bit-flip` | one bit flipped in the command or an argument of a valid command | | `buffer-length` | ECHO/FILL/SUM against `gsp_ctrl.buf` with lengths `0, 1, 31, 32, 33, 255, 256, 257, 4096, 2^32, ...` | | `mem-region` | one byte bit-flip inside `--memory-region` (when supplied) | Each iteration uses a **fresh QEMU session**, settles the demo through its self-test, applies exactly one mutation, runs bounded, then reads back the control block. No hardware watchpoints are installed, so the only stops are budget interrupts and real traps. ### 2.3 Outcome classes | Class | Detection | |---|---| | `ok` | command dispatched (or `cmd=0`) and the demo oracle, when one exists, matched | | `trap` | first new exception is a breakpoint (`ebreak`), or control transferred below `0x100` | | `fault` | first new QEMU exception is a load/store/fetch/illegal fault (`parse_qemu_faults`) | | `no-progress` | command written but `rpc_count` did not advance within both windows (no fault) | | `unexpected-result` | result/last_rpc contradict the documented demo oracle for `cmd ∈ {0,1,3,5, invalid}` | | `exit` / `settle-failed` / `error` | target exited; self-test never observed; iteration exception | The oracle is deliberately small: handlers that dereference `arg0` (ECHO, SUM) and injected slots 0–5 have no oracle, so their successful executions are `ok` with `expected: null`. ### 2.4 Output format ```text / findings.jsonl one compact finding per iteration (deterministic order) summary.md unique outcome classes, strategy coverage, examples, replay command sessions/iter-0000/ recorded risc-v-gsp-rm-debugger session per iteration ``` Finding shape: ```json {"iteration": 5, "strategy": "arg-boundary", "mutation": {"cmd": 4, "arg0": 0, "arg1": 255, "arg2": 0, "region": null}, "outcome": "fault", "detail": {"observed": {"result": 48, "last_rpc": 5, "rpc_count_delta": 0}, "fault": {"cause": 5, "desc": "fault_load", "epc": 2147483860, "tval": 255, "access": {"rw": "read", "access_addr": 255, "size": 1, "reason": "rejected"}}}, "session": "sessions/iter-0005"} ``` Replay is re-running the same command: the per-iteration RNG is seeded from `(seed, index)`, finding fields avoid wall-clock data, and `findings.jsonl` is byte-identical for the same seed/budgets on the same host. Session directories under `sessions/` contain the usual `events.jsonl`, `session.json` and `summary.md`; they are the evidence behind each finding. --- ## 3. Tests ```bash python3 -m unittest tests.test_harness -v make test # includes the harness tests with the rest of the suite ``` The integration tests boot the demo twice for a self-comparison (must show zero deltas) and run 12 seeded iterations twice (findings file, at least one outcome class, byte-identical replay). They skip cleanly when QEMU or the demo ELF is absent. --- ## 4. Attaching reports to a portability change For the docs/07 P1/P2/P3 loop and the docs/14 §6 evidence pack: 1. Run one matrix invocation per version/device/architecture family with the same scenario options, and keep `/portability-report.json`, the per-pair compare reports and the session directories (or their sha256 list). 2. Record the firmware hash (`sha256` in each target entry), the symbol source list and `bias` from `/symbols/*.json` meta. 3. Fill the M1–M6 table using `observed` from the matrix plus the real reference capture; the matrix marks agreement metrics "not measured" on purpose. Cite docs/06 L2/L3 limits in any claim. 4. For an input-surface change, run the mutation harness before and after the change and diff the outcome-class tables; attach `findings.jsonl` as the corpus entry list, with each finding's session as evidence. 5. Re-run the unchanged stimulus after the emulation change: profiles must improve or stay equal (the docs/06 accuracy loop). --- ## 5. Safety * Demo / re-hosted firmware under QEMU only. Never physical hardware, never a guest, never a kernel module, never an existing gdbstub or the native simulator. * The mutation harness refuses non-demo ELFs unless `--allow-foreign-elf` is passed explicitly, and still refuses non-ELF images. * Memory-region mutation is bounds-checked against the QEMU RAM window; no host memory is touched outside the emulated target. * Every write is recorded in a session (`mem_write`, `note`, mutation fields); budgets (`duration`/`max_stops`/`--timeout`) bound every run. * Only temporary work directories and `/` are written on the host. --- ## 6. Limits and fidelity * L2/L3 claims are out of scope: no timing, interrupt or device-model conclusions (docs/06). The matrix observes RAM-resident behaviour and the first unmapped access, nothing more. * A QEMU log window is bounded; a fault not in the window is recorded as its secondary signal (`trap_loop`/`no-progress`) rather than guessed. * Mutation covers one mailbox write per iteration; multi-write races and concurrency are future work (H2/H4.4). * The demo oracle is small and documented; "unexpected-result" is only meaningful for the listed commands. --- ## 7. Roadmap mapping | Item | Where this doc/tool contributes | |---|---| | **R5 — fuzzing and mutation harness** | `mutation_harness.py`: bounded seeded mutation, outcome classes, recorded corpus (`findings.jsonl` + sessions) | | **R6 — multi-version orchestration** | `portability_matrix.py`: probe + rehost/demo + compare + M1–M6 for a target set | | **H4.1 — semantic diffing** | identical stimulus per target, `compare_sessions` plus hit/watch/write deltas in one report | | **H4.3 — mutation/fuzzing** | error-path corpus with symbol-annotated sessions, feeding fail-closed rules | | **H4.5 — reimplementation loop** | reports are the attachable evidence for the docs/07 checklist | | **V1 — metrics/golden data** | deterministic profiles and seeded findings support golden comparisons | | **H7 — agentic harness** | self-describing JSON/markdown artifacts designed for an agent to drive and review |