path: root/debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md

# 21 — CI and release pipeline (version diff + evidence bundle)

> **# TOOL IN DEVELOPMENT - NOT READY YET!** See 00 for status.

## 1. Purpose

This is the release-triage path for a new GSP firmware image (roadmap R6 /
H4.1, docs/23 C5/C7). Two stdlib-only tools turn the manual docs/07 P1/P2
playbook into one repeatable run and one attachable bundle:

| Tool | What it does |
|---|---|
| `harness/version_diff.py` | old vs new target, section by section: image/shape, symbol inventory, decompiled corpus hints, bounded QEMU probe, shared-scenario behavior, M1–M6 skeleton |
| `harness/ci_release.py` | runs version_diff → portability matrix → mutation smoke and hashes every artifact into `bundle-manifest.json` |

`.github/workflows/greeneye.yml` runs the tool's test suite on pushes/PRs that
touch `debug/risc-v-gsp-rm-vm/**` or `tools/rm*/**`. The repository-level
`.github/workflows/ci.yml` is independent and unchanged.

Everything here is local: the tools never download firmware, never attach to
hardware, a guest or a kernel module, and use their own QEMU processes.

---

## 2. Version diff

### 2.1 Usage

Self-diff of the demo (a determinism/regression check; must be all-zero):

```bash
python3 harness/version_diff.py \
    --old firmware/uproc_demo/build/uproc_demo.elf \
    --new firmware/uproc_demo/build/uproc_demo.elf \
    --out dumps/vdiff-selfcheck \
    --probe-instructions 200 --behavior-duration 0.2
```

Old vs new firmware with per-version symbol DBs and Ghidra exports:

```bash
python3 harness/version_diff.py \
    --old firmware/57X-GSP/570.144/gsp-570.144.fwimage.bin \
    --new firmware/57X-GSP/570.211/gsp-570.211.fwimage.bin \
    --old-label 570.144 --new-label 570.211 \
    --old-ghidra-exports analysis/570.144/symbols.tsv \
    --new-ghidra-exports analysis/570.211/symbols.tsv \
    --corpus-diff analysis/570.144/decompiled_index.tsv analysis/570.211/decompiled_index.tsv \
    --out dumps/vdiff-570
```

Attaching a fixed stimulus and a pre-produced corpus report:

```bash
python3 harness/version_diff.py --old <rehosted-old.elf> --new <rehosted-new.elf> \
    --stimulus examples/scripts/trace_rpc.py \
    --corpus-report run/corpus-delta-570.144-570.211.txt \
    --out dumps/vdiff-rehosted
```

Targets are RISC-V ELF files or raw images (`.fwimage`). A raw target is
relocated to `0x80000000` as one implicit load; its firmware-vaddr translation
is `runtime - 0x80000000`. Symbols may be absent for raw images; the symbol
section then reports `skipped` with a reason unless Ghidra/JSON sources are
supplied.

Flags: `--old/--new` (required), `--out` (required), `--old-label/--new-label/
--release`, `--old-symbols/--new-symbols` (green-eye JSON), `--old-ghidra-names`,
`--new-ghidra-names`, `--old-ghidra-index`, `--new-ghidra-index`,
`--old-ghidra-exports`, `--new-ghidra-exports` (`--old-ghidra-bias`/
`--new-ghidra-bias`), `--corpus-diff OLD NEW`, `--corpus-report FILE`,
`--stimulus`, `--probe-instructions` (`0` = `--probe-duration` wall clock),
`--behavior-duration`, `--max-stops`, `--max-rows`, `--approval-token`.

### 2.2 Sections

Every section reports `status: ok | skipped | error`, and a skipped section
never aborts the others. A failing QEMU probe or scenario is recorded as
`error` for that section only.

1. **Image / shape.** Entry, span, file size/hash, PT_LOAD layout (vaddr,
   filesz/memsz, flags), and the ELF section table (name/type/addr/size).
   Raw images have no section table; that is stated, not guessed.
2. **Symbol inventory.** ELF `.symtab` plus Ghidra `names.tsv`,
   `decompiled_index.tsv`/bucket TSVs, 4-column `exports.tsv` and the
   3-column `ExportForGreenEye` form (`addr<TAB>name<TAB>size`), plus
   green-eye `greeneye-symbols/1` JSON DBs. Matching follows
   `userspace/cuda/tools/diff_corpus.py`: named functions match by name
   (duplicates in address order), the address shift is the median of named
   deltas, and `FUN_*` functions match at the shifted address. The report
   lists added/removed/moved/resized (named and anonymous), the shift and its
   dominance, match rates and the named-transfer rate. Merged per-side DBs are
   saved under `<out>/symbols/`.
3. **Decompiled-change hints.** `--corpus-diff A B` runs the same matching
   discipline over two corpus indexes and writes `<out>/corpus/corpus-diff.json`;
   `--corpus-report FILE` attaches a previously produced report (hash +
   preview). With neither flag the section is `skipped` with reason
   `not supplied: ...`.
4. **Probe first-touch.** Both targets are flattened, relocated to
   `0x80000000`, stepped for the bounded budget, and their QEMU logs parsed
   with `greeneye.analyze.parse_qemu_faults`. The first fault is reported with
   cause/desc, `epc` **and** `epc_firmware`, the access address **and**
   `access_firmware` (translated only when the access address falls inside the
   image). The diff says whether the stopped PC/first fault changed.
5. **Behavior.** The shared demo scenario from `portability_matrix` (hooks on
   every resolvable `rpc_*`, a write watchpoint on `gsp_ctrl.rpc_count`, the
   self-test, one deterministic PING) or a `--stimulus` script is run on both
   targets and recorded as normal `greeneye-session/1` directories. The report
   includes `analyze.compare_sessions` output under `<out>/compare/`, explicit
   hit/watch/written-page deltas, `all_zero`, and the docs/06 M1–M6 skeleton
   with every agreement metric marked `not measured` (a reference capture is
   required, docs/06 §5).

### 2.3 Output

```text
<out>/
  version-diff.json              stable machine-readable report
  version-diff.md                rendered report
  symbols/<target>.json          merged symbol DB per side (when non-empty)
  sessions/<target>-probe/       recorded probe session
  sessions/<target>-behavior/    recorded scenario session
  compare/<old>--<new>.md        compare_sessions report
  corpus/corpus-diff.json        when --corpus-diff is used
```

Exit status: `0` for a normal report (skips included), `2` for a tool error,
`3` for an RM 610 gate refusal.

---

## 3. RM 610 gate (hard, testable)

The gate is enforced by `version_diff` and `ci_release` **before any target
file is opened** and before `--out` is created:

* a target/release label matching `rm610` / `RM-610` / `rm_610` etc. (the
  `--release`, `--old-label`, `--new-label` values and the target basenames)
  is refused unless
  * `--approval-token RM610-APPROVED` is passed, or
  * `GREENEYE_RM610_APPROVED=1` is set;
* the refusal text points at [`docs/23-RM610-READINESS.md`](23-RM610-READINESS.md)
  and states that nothing was downloaded and no target file was opened;
* the same check runs in `--dry-run` mode.

Nothing in either tool performs network access, and `--dry-run` never touches
a target. Current-version rehearsal runs use `570.144`-labelled inputs; 610
inputs require the §6 approval message in docs/23.

---

## 4. Release pipeline

### 4.1 Dry run first

```bash
python3 harness/ci_release.py --old <old> --new <new> --out dumps/release-570 \
    --old-label 570.144 --new-label 570.211 --dry-run
```

`--dry-run` prints a deterministic JSON plan (the three step commands and the
artifact paths) and exits `0` without creating `--out` or opening a target.
This is the docs/23 §4 step 6 rehearsal.

### 4.2 Real run

```bash
python3 harness/ci_release.py --old <old> --new <new> --out dumps/release-570 \
    --old-label 570.144 --new-label 570.211 \
    --old-symbols symbols-570.144.json --new-symbols symbols-570.211.json \
    --stimulus examples/scripts/trace_rpc.py \
    --mutation-kernel firmware/uproc_demo/build/uproc_demo.elf \
    --mutation-iterations 8 --mutation-seed 7
```

Steps, in order:

1. **version_diff** — writes `<out>/version-diff/` (section flags above);
2. **portability_matrix** — old and new as two targets, same budgets, writes
   `<out>/portability-matrix/portability-report.json|.md` and per-pair compare
   reports; the self-comparison must show `all_zero`;
3. **mutation_smoke** — `mutation_harness` on the demo (or `--mutation-kernel`,
   with `--allow-foreign-mutation` for a re-hosted ELF) with a small budget;
   writes `<out>/mutation-smoke/findings.jsonl|summary.md`.

A step that raises is recorded as `status: error` and the later steps still
run; the process exits `1` when any step errored, `0` otherwise (`skipped`
steps do not fail the run, but their reasons are in the report). Exit `3` is
the RM 610 refusal.

`ci_release` flags: the `version_diff` inputs above plus `--matrix-symbols`
(one JSON DB applied to both matrix targets), `--mutation-kernel` (default:
the bundled demo), `--mutation-iterations`/`--mutation-seed`/
`--mutation-timeout`/`--mutation-settle`, and `--allow-foreign-mutation` for a
non-demo re-hosted ELF.

### 4.3 Evidence bundle format

```text
<out>/
  release-report.json            pipeline summary (steps, gate, labels)
  bundle-manifest.json           sha256 + byte size of every other file
  version-diff/                  docs/21 §2.3 tree
  portability-matrix/            portability-report.json|.md, symbols/, sessions/, compare/
  mutation-smoke/                findings.jsonl, summary.md, sessions/
```

`bundle-manifest.json`:

```json
{
  "format": "greeneye-release-bundle/1",
  "tool": "harness/ci_release.py",
  "files": [{"path": "version-diff/version-diff.json", "bytes": 12345, "sha256": "..."}]
}
```

`release-report.json` shape:

```json
{
  "format": "greeneye-ci-release/1",
  "dry_run": false,
  "rm610_gate": {"status": "passed", "labels": ["570.144", "570.211"]},
  "old": {"label": "570.144", "path": "gsp-570.144.fwimage.bin"},
  "new": {"label": "570.211", "path": "gsp-570.211.fwimage.bin"},
  "steps": [{"name": "version_diff", "status": "ok",
             "sections": {"image": "ok", "symbols": "ok", "corpus": "ok",
                          "probe": "ok", "behavior": "ok"}}],
  "errors": [],
  "manifest": "bundle-manifest.json"
}
```

The manifest is written last, so its hashes cover the final release report.
Attach the directory (or its manifest) to the docs/07 evidence checklist and
the docs/14 §6 pack; the docs/23 §4 step 7 expected-artifact list maps 1:1.

---

## 5. CI workflow

`.github/workflows/greeneye.yml` (new; `ci.yml` is untouched):

* triggers on push/PR paths `debug/risc-v-gsp-rm-vm/**` and `tools/rm*/**`;
* `ubuntu-latest`, no secrets, no hardware;
* installs `qemu-system-misc clang lld-18`;
* builds `make -C debug/risc-v-gsp-rm-vm sdk demo`;
* runs `python3 -m unittest discover -s tests -t .` from the tool root (QEMU
  integration tests run; if QEMU were absent they skip cleanly).

The intended release gates this workflow covers are the fast inventory smoke
from docs/19 §6 (`control_discovery --max-id 32 --patterns zero`), the harness
suites, and the docs/21 dry-run. A full `ci_release` run against real firmware
is driven manually/agentically because it needs the versioned inputs and
symbol DBs; CI-adjacent automation should call it with `--dry-run` first.

---

## 6. MCP monitoring

`plugins/mcp_server.py` (docs/20) exposes the same machinery to an agent:
`probe_firmware`, `portability_matrix_run`, `mutation_run`, `compare_sessions`
and `analyze_current`. The agent pattern for a release:

1. `portability_matrix_run` with the old/new targets and fixed budgets;
2. `mutation_run` on the demo with the pinned seed and budget;
3. `compare_sessions` for the recorded evidence pair;
4. read `version-diff.json` / `release-report.json` / `bundle-manifest.json`
   from the run directory (agent reads the bundle, it does not recompute it).

The MCP server remains read/observe by default and simulator-only (docs/20 §7).

---

## 7. Determinism, limits and safety

* Fixed probe budgets, fixed scenario phases and the same stimulus make the
  self-diff zero: an unchanged target reports zero symbol/probe/behavior
  deltas (verified by `tests/test_version_diff.py`).
* Agreement metrics M1–M6 are **not measured** without a reference capture
  (docs/06 §5); the skeleton carries local `observed` facts only. L2/L3 claims
  (timing, interrupts, device model) are out of scope.
* Raw `.fwimage` targets execute as flat blobs; a firmware linked differently
  may fault earlier than on silicon — that is first-touch data, not a boot
  (docs/06 §3).
* Only temporary work directories and `<out>/` are written; QEMU is always
  launched and killed by the tools; log reads are bounded
  (`--log-bytes`).
* No downloads, no `/dev`, no kernel module, no existing gdbstub, no physical
  GPU. The 610 gate is part of both CLIs, not documentation-only.

---

## 8. Tests

```bash
python3 -m unittest tests.test_version_diff -v
make test        # includes these with the rest of the suite
```

`tests/test_version_diff.py` covers the gate (default refusal, token/env
approval, gate-before-open), the `diff_corpus.py` matching discipline
(moved/resized/added/removed, anonymous shift), the 3-column export parser,
missing/incompatible/raw target handling, corpus hints, `ci_release --dry-run`
side-effect freedom, clean `--help`, and a demo self-diff asserting zero
inventory and behavior deltas. QEMU integration tests skip cleanly when QEMU
or the demo ELF is absent.

---

## 9. Roadmap mapping

| Item | Where this doc/tool contributes |
|---|---|
| **R6 — multi-version orchestration** | `version_diff` + `ci_release` prefix the matrix with a version-scoped diff |
| **H4.1 — semantic diffing** | symbol/corpus inventory deltas and behavior profiles for two firmware versions |
| **V1 — metrics/golden data** | deterministic self-diff and stable JSON reports support golden comparisons |
| **docs/23 C5/C7** | release pipeline implemented and exercisable; dry-run rehearsal path |
| **H7 — agentic harness** | self-describing bundle + docs/20 MCP tools for monitored runs |