path: root/debug/risc-v-gsp-rm-vm/docs/18-HARNESS-AUTOMATION.md

# 18 — Harness automation: portability matrix and mutation harness

> Status: implemented, documented and tested (audit: docs/27-TOOL-AUDIT.md; value demos: docs/25-VALUE-VALIDATION.md).

Two stdlib-only orchestration tools in `harness/` turn the manual playbooks
into repeatable runs:

| Tool | Roadmap | What it does |
|---|---|---|
| `harness/portability_matrix.py` | **R6**, H4.1 | Runs probe + scripted demo + recorded sessions against several firmware ELFs and emits a cross-target delta report with the M1–M6 skeleton. |
| `harness/mutation_harness.py` | **R5**, H4.3 | Deterministically mutates the demo firmware's mailbox input surface, classifies abnormal outcomes, and records every attempt as a session plus a compact finding. |

Both use the **QEMU backend only** (the same `probe`/`demo` machinery as the
CLI). They intentionally do not depend on `greeneye.sim_backend`, and they
never attach to an existing gdbstub, hardware, a guest or a kernel module.

Requirements: `qemu-system-riscv64`, Python 3.10+, and (for the bundled demo)
`make demo`. All commands run from `open-iov/debug/risc-v-gsp-rm-vm/`.

---

## 1. Portability matrix

### 1.1 Usage

Self-comparison of one build (a determinism check):

```bash
python3 harness/portability_matrix.py \
    --target firmware/uproc_demo/build/uproc_demo.elf \
    --target firmware/uproc_demo/build/uproc_demo.elf \
    --out dumps/matrix-demo \
    --probe-instructions 200 --scenario-duration 0.2
```

Old vs new firmware with the project's symbol DBs (one DB applies to every
target in the invocation; run separate invocations for per-version DBs):

```bash
python3 harness/portability_matrix.py \
    --target firmware/57X-GSP/570.144/gsp-570.144.fwimage.bin \
    --target firmware/57X-GSP/570.211/gsp-570.211.fwimage.bin \
    --ghidra-index analysis/570.144/decompiled_index.tsv --ghidra-bias 0 \
    --symbols symbols-570.144.json \
    --out dumps/matrix-570
```

Replacing the builtin scenario with a fixed risc-v-gsp-rm-debugger script:

```bash
python3 harness/portability_matrix.py --target <rehosted.elf> \
    --scenario-script examples/scripts/trace_rpc.py --out dumps/matrix-rehosted
```

Flags: `--probe-instructions N` (0 = wall-clock `--probe-duration`),
`--scenario-duration`, `--max-stops`, `--max-rows`, `--symbols`,
`--ghidra-names/--ghidra-index/--ghidra-exports/--ghidra-callgraph`
(`--ghidra-bias`), and repeatable `--target` (positional targets also work).

### 1.2 What each target does

1. **Probe** — the ELF is flattened and relocated to `0x80000000` exactly as
   `./risc-v-gsp-rm-debugger.sh probe` does, then run for a fixed instruction budget (default
   2000) or a wall-clock budget. The first exception is parsed with
   `greeneye.analyze.parse_qemu_faults`; the stopped PC is annotated from the
   symbol DB. Raw `.fwimage` targets fault early here; that is the expected
   first-touch data, not a harness failure.
2. **Symbols** — ELF `.symtab` plus optional Ghidra exports and/or an
   existing JSON DB, merged and saved to `<out>/symbols/<target>.json`
   (runtime addresses; the relocation `bias` is recorded in the DB meta).
3. **Scenario** — the target boots under QEMU with hooks on every `rpc_*`
   handler that resolves and a write watchpoint on `gsp_ctrl.rpc_count`, runs
   its self-test, then receives one deterministic host stimulus
   (`mailbox = PING (1)`, `arg0 = 0x1234`). Everything is recorded as a
   normal `greeneye-session/1` directory. A re-hosted ELF without `rpc_*`/
   `gsp_ctrl` symbols still runs; it simply produces an empty profile.

### 1.3 Output format

```text
<out>/
  portability-report.json        stable machine-readable report
  portability-report.md          rendered report
  symbols/<target>.json          merged symbol DB per target
  sessions/<target>-probe/       recorded probe session
  sessions/<target>-scenario/    recorded scripted session
  compare/<a>--<b>.md            risc-v-gsp-rm-debugger.sh compare report per target pair
```

JSON shape (stable keys, targets in CLI order, tables sorted):

```json
{
  "format": "greeneye-portability-matrix/1",
  "tool": "harness/portability_matrix.py",
  "backend": "qemu",
  "targets": [{
    "name": "...", "path": "...", "sha256": "...", "entry": "0x...",
    "status": "ok",
    "probe": {"status": "...", "stopped_pc": 0, "stopped_symbol": "...",
              "fault_count": 0, "first_fault": null},
    "symbols": {"count": 13, "sources": ["elf"], "path": "symbols/x.json"},
    "scenario": {"session": "sessions/x-scenario", "event_counts": {},
                 "bp": {}, "watch": {}, "writes": {}, "ctrl": {}},
    "metrics": [{"id": "M1", "status": "not measured", "observed": {}}]
  }],
  "comparisons": [{"a": "...", "b": "...", "deltas": {"bp": [], "watch": [], "writes": []},
                   "all_zero": true, "metrics": []}],
  "skipped": []
}
```

Determinism: fixed probe instructions, fixed scenario phases and the same
stimulus make hit/watch/write profiles reproducible, so an identical target
compared with itself reports **zero deltas**. The M6 control-block snapshot
hash is *not* a fixed point while the firmware heartbeats; compare snapshots
only between quiesced runs. Metrics are marked `"not measured"` unless a
reference capture exists (docs/06 §5); `observed` carries the local facts the
matrix could gather.

Targets that are missing, not RISC-V ELF, or too large are reported with
`status: "skipped"` and a `skip_reason`; a failing step yields `"error"` for
that step and does not abort the matrix.

---

## 2. Mutation harness

### 2.1 Usage

```bash
python3 harness/mutation_harness.py --out dumps/mutation-7 \
    --iterations 64 --seed 7 --timeout 0.25 --settle 0.2
```

Targeting the command/args surface plus an optional RAM window:

```bash
python3 harness/mutation_harness.py --out dumps/mutation-ram \
    --iterations 128 --seed 11 --memory-region 0x80000900:0x100
```

Budgets: `--iterations` (1..10000), `--seed`, `--timeout` (0.01..10 s total
observation per mutation; a short first window plus one confirmation window
for slow handlers), `--settle` (per attempt, until `SELFTEST DONE` is seen),
and `--max-log-bytes` (QEMU log window per fault check; trap loops flood
`-d int`). `--kernel` defaults to the bundled demo; a non-demo ELF is refused
unless `--allow-foreign-elf` is given, and remains QEMU-only.

### 2.2 Strategies

| Strategy | Input surface |
|---|---|
| `cmd-invalid` | command IDs outside the demo's valid range (`0, 7..15, 24+, 0x100, 0xffff, 2^64-1`) |
| `arg-boundary` | one of arg0/arg1/arg2 set to `0, 1, 2, 0xff, 0x100, 0x7fffffff, 2^63, 2^64-1`, ... |
| `bit-flip` | one bit flipped in the command or an argument of a valid command |
| `buffer-length` | ECHO/FILL/SUM against `gsp_ctrl.buf` with lengths `0, 1, 31, 32, 33, 255, 256, 257, 4096, 2^32, ...` |
| `mem-region` | one byte bit-flip inside `--memory-region` (when supplied) |

Each iteration uses a **fresh QEMU session**, settles the demo through its
self-test, applies exactly one mutation, runs bounded, then reads back the
control block. No hardware watchpoints are installed, so the only stops are
budget interrupts and real traps.

### 2.3 Outcome classes

| Class | Detection |
|---|---|
| `ok` | command dispatched (or `cmd=0`) and the demo oracle, when one exists, matched |
| `trap` | first new exception is a breakpoint (`ebreak`), or control transferred below `0x100` |
| `fault` | first new QEMU exception is a load/store/fetch/illegal fault (`parse_qemu_faults`) |
| `no-progress` | command written but `rpc_count` did not advance within both windows (no fault) |
| `unexpected-result` | result/last_rpc contradict the documented demo oracle for `cmd ∈ {0,1,3,5, invalid}` |
| `exit` / `settle-failed` / `error` | target exited; self-test never observed; iteration exception |

The oracle is deliberately small: handlers that dereference `arg0` (ECHO,
SUM) and injected slots 0–5 have no oracle, so their successful executions
are `ok` with `expected: null`.

### 2.4 Output format

```text
<out>/
  findings.jsonl      one compact finding per iteration (deterministic order)
  summary.md          unique outcome classes, strategy coverage, examples, replay command
  sessions/iter-0000/ recorded risc-v-gsp-rm-debugger session per iteration
```

Finding shape:

```json
{"iteration": 5, "strategy": "arg-boundary",
 "mutation": {"cmd": 4, "arg0": 0, "arg1": 255, "arg2": 0, "region": null},
 "outcome": "fault",
 "detail": {"observed": {"result": 48, "last_rpc": 5, "rpc_count_delta": 0},
            "fault": {"cause": 5, "desc": "fault_load", "epc": 2147483860,
                      "tval": 255, "access": {"rw": "read", "access_addr": 255,
                      "size": 1, "reason": "rejected"}}},
 "session": "sessions/iter-0005"}
```

Replay is re-running the same command: the per-iteration RNG is seeded from
`(seed, index)`, finding fields avoid wall-clock data, and `findings.jsonl`
is byte-identical for the same seed/budgets on the same host. Session
directories under `sessions/` contain the usual `events.jsonl`, `session.json`
and `summary.md`; they are the evidence behind each finding.

---

## 3. Tests

```bash
python3 -m unittest tests.test_harness -v
make test        # includes the harness tests with the rest of the suite
```

The integration tests boot the demo twice for a self-comparison (must show
zero deltas) and run 12 seeded iterations twice (findings file, at least one
outcome class, byte-identical replay). They skip cleanly when QEMU or the
demo ELF is absent.

---

## 4. Attaching reports to a portability change

For the docs/07 P1/P2/P3 loop and the docs/14 §6 evidence pack:

1. Run one matrix invocation per version/device/architecture family with the
   same scenario options, and keep `<out>/portability-report.json`, the
   per-pair compare reports and the session directories (or their sha256
   list).
2. Record the firmware hash (`sha256` in each target entry), the symbol
   source list and `bias` from `<out>/symbols/*.json` meta.
3. Fill the M1–M6 table using `observed` from the matrix plus the real
   reference capture; the matrix marks agreement metrics "not measured" on
   purpose. Cite docs/06 L2/L3 limits in any claim.
4. For an input-surface change, run the mutation harness before and after the
   change and diff the outcome-class tables; attach `findings.jsonl` as the
   corpus entry list, with each finding's session as evidence.
5. Re-run the unchanged stimulus after the emulation change: profiles must
   improve or stay equal (the docs/06 accuracy loop).

---

## 5. Safety

* Demo / re-hosted firmware under QEMU only. Never physical hardware, never
  a guest, never a kernel module, never an existing gdbstub or the native
  simulator.
* The mutation harness refuses non-demo ELFs unless `--allow-foreign-elf` is
  passed explicitly, and still refuses non-ELF images.
* Memory-region mutation is bounds-checked against the QEMU RAM window; no
  host memory is touched outside the emulated target.
* Every write is recorded in a session (`mem_write`, `note`, mutation fields);
  budgets (`duration`/`max_stops`/`--timeout`) bound every run.
* Only temporary work directories and `<out>/` are written on the host.

---

## 6. Limits and fidelity

* L2/L3 claims are out of scope: no timing, interrupt or device-model
  conclusions (docs/06). The matrix observes RAM-resident behaviour and the
  first unmapped access, nothing more.
* A QEMU log window is bounded; a fault not in the window is recorded as its
  secondary signal (`trap_loop`/`no-progress`) rather than guessed.
* Mutation covers one mailbox write per iteration; multi-write races and
  concurrency are future work (H2/H4.4).
* The demo oracle is small and documented; "unexpected-result" is only
  meaningful for the listed commands.

---

## 7. Roadmap mapping

| Item | Where this doc/tool contributes |
|---|---|
| **R5 — fuzzing and mutation harness** | `mutation_harness.py`: bounded seeded mutation, outcome classes, recorded corpus (`findings.jsonl` + sessions) |
| **R6 — multi-version orchestration** | `portability_matrix.py`: probe + rehost/demo + compare + M1–M6 for a target set |
| **H4.1 — semantic diffing** | identical stimulus per target, `compare_sessions` plus hit/watch/write deltas in one report |
| **H4.3 — mutation/fuzzing** | error-path corpus with symbol-annotated sessions, feeding fail-closed rules |
| **H4.5 — reimplementation loop** | reports are the attachable evidence for the docs/07 checklist |
| **V1 — metrics/golden data** | deterministic profiles and seeded findings support golden comparisons |
| **H7 — agentic harness** | self-describing JSON/markdown artifacts designed for an agent to drive and review |