# 21 — CI and release pipeline (version diff + evidence bundle) > **# TOOL IN DEVELOPMENT - NOT READY YET!** See 00 for status. ## 1. Purpose This is the release-triage path for a new GSP firmware image (roadmap R6 / H4.1, docs/23 C5/C7). Two stdlib-only tools turn the manual docs/07 P1/P2 playbook into one repeatable run and one attachable bundle: | Tool | What it does | |---|---| | `harness/version_diff.py` | old vs new target, section by section: image/shape, symbol inventory, decompiled corpus hints, bounded QEMU probe, shared-scenario behavior, M1–M6 skeleton | | `harness/ci_release.py` | runs version_diff → portability matrix → mutation smoke and hashes every artifact into `bundle-manifest.json` | `.github/workflows/risc-v-gsp-rm-debugger.yml` runs the tool's test suite on pushes/PRs that touch `debug/risc-v-gsp-rm-vm/**` or `tools/rm*/**`. The repository-level `.github/workflows/ci.yml` is independent and unchanged. Everything here is local: the tools never download firmware, never attach to hardware, a guest or a kernel module, and use their own QEMU processes. --- ## 2. Version diff ### 2.1 Usage Self-diff of the demo (a determinism/regression check; must be all-zero): ```bash python3 harness/version_diff.py \ --old firmware/uproc_demo/build/uproc_demo.elf \ --new firmware/uproc_demo/build/uproc_demo.elf \ --out dumps/vdiff-selfcheck \ --probe-instructions 200 --behavior-duration 0.2 ``` Old vs new firmware with per-version symbol DBs and Ghidra exports: ```bash python3 harness/version_diff.py \ --old firmware/57X-GSP/570.144/gsp-570.144.fwimage.bin \ --new firmware/57X-GSP/570.211/gsp-570.211.fwimage.bin \ --old-label 570.144 --new-label 570.211 \ --old-ghidra-exports analysis/570.144/symbols.tsv \ --new-ghidra-exports analysis/570.211/symbols.tsv \ --corpus-diff analysis/570.144/decompiled_index.tsv analysis/570.211/decompiled_index.tsv \ --out dumps/vdiff-570 ``` Attaching a fixed stimulus and a pre-produced corpus report: ```bash python3 harness/version_diff.py --old --new \ --stimulus examples/scripts/trace_rpc.py \ --corpus-report run/corpus-delta-570.144-570.211.txt \ --out dumps/vdiff-rehosted ``` Targets are RISC-V ELF files or raw images (`.fwimage`). A raw target is relocated to `0x80000000` as one implicit load; its firmware-vaddr translation is `runtime - 0x80000000`. Symbols may be absent for raw images; the symbol section then reports `skipped` with a reason unless Ghidra/JSON sources are supplied. Flags: `--old/--new` (required), `--out` (required), `--old-label/--new-label/ --release`, `--old-symbols/--new-symbols` (risc-v-gsp-rm-debugger JSON), `--old-ghidra-names`, `--new-ghidra-names`, `--old-ghidra-index`, `--new-ghidra-index`, `--old-ghidra-exports`, `--new-ghidra-exports` (`--old-ghidra-bias`/ `--new-ghidra-bias`), `--corpus-diff OLD NEW`, `--corpus-report FILE`, `--stimulus`, `--probe-instructions` (`0` = `--probe-duration` wall clock), `--behavior-duration`, `--max-stops`, `--max-rows`, `--approval-token`. ### 2.2 Sections Every section reports `status: ok | skipped | error`, and a skipped section never aborts the others. A failing QEMU probe or scenario is recorded as `error` for that section only. 1. **Image / shape.** Entry, span, file size/hash, PT_LOAD layout (vaddr, filesz/memsz, flags), and the ELF section table (name/type/addr/size). Raw images have no section table; that is stated, not guessed. 2. **Symbol inventory.** ELF `.symtab` plus Ghidra `names.tsv`, `decompiled_index.tsv`/bucket TSVs, 4-column `exports.tsv` and the 3-column `ExportForGreenEye` form (`addrnamesize`), plus risc-v-gsp-rm-debugger `greeneye-symbols/1` JSON DBs. Matching follows `userspace/cuda/tools/diff_corpus.py`: named functions match by name (duplicates in address order), the address shift is the median of named deltas, and `FUN_*` functions match at the shifted address. The report lists added/removed/moved/resized (named and anonymous), the shift and its dominance, match rates and the named-transfer rate. Merged per-side DBs are saved under `/symbols/`. 3. **Decompiled-change hints.** `--corpus-diff A B` runs the same matching discipline over two corpus indexes and writes `/corpus/corpus-diff.json`; `--corpus-report FILE` attaches a previously produced report (hash + preview). With neither flag the section is `skipped` with reason `not supplied: ...`. 4. **Probe first-touch.** Both targets are flattened, relocated to `0x80000000`, stepped for the bounded budget, and their QEMU logs parsed with `greeneye.analyze.parse_qemu_faults`. The first fault is reported with cause/desc, `epc` **and** `epc_firmware`, the access address **and** `access_firmware` (translated only when the access address falls inside the image). The diff says whether the stopped PC/first fault changed. 5. **Behavior.** The shared demo scenario from `portability_matrix` (hooks on every resolvable `rpc_*`, a write watchpoint on `gsp_ctrl.rpc_count`, the self-test, one deterministic PING) or a `--stimulus` script is run on both targets and recorded as normal `greeneye-session/1` directories. The report includes `analyze.compare_sessions` output under `/compare/`, explicit hit/watch/written-page deltas, `all_zero`, and the docs/06 M1–M6 skeleton with every agreement metric marked `not measured` (a reference capture is required, docs/06 §5). ### 2.3 Output ```text / version-diff.json stable machine-readable report version-diff.md rendered report symbols/.json merged symbol DB per side (when non-empty) sessions/-probe/ recorded probe session sessions/-behavior/ recorded scenario session compare/--.md compare_sessions report corpus/corpus-diff.json when --corpus-diff is used ``` Exit status: `0` for a normal report (skips included), `2` for a tool error, `3` for an RM 610 gate refusal. --- ## 3. RM 610 gate (hard, testable) The gate is enforced by `version_diff` and `ci_release` **before any target file is opened** and before `--out` is created: * a target/release label matching `rm610` / `RM-610` / `rm_610` etc. (the `--release`, `--old-label`, `--new-label` values and the target basenames) is refused unless * `--approval-token RM610-APPROVED` is passed, or * `GREENEYE_RM610_APPROVED=1` is set; * the refusal text points at [`docs/23-RM610-READINESS.md`](23-RM610-READINESS.md) and states that nothing was downloaded and no target file was opened; * the same check runs in `--dry-run` mode. Nothing in either tool performs network access, and `--dry-run` never touches a target. Current-version rehearsal runs use `570.144`-labelled inputs; 610 inputs require the §6 approval message in docs/23. --- ## 4. Release pipeline ### 4.1 Dry run first ```bash python3 harness/ci_release.py --old --new --out dumps/release-570 \ --old-label 570.144 --new-label 570.211 --dry-run ``` `--dry-run` prints a deterministic JSON plan (the three step commands and the artifact paths) and exits `0` without creating `--out` or opening a target. This is the docs/23 §4 step 6 rehearsal. ### 4.2 Real run ```bash python3 harness/ci_release.py --old --new --out dumps/release-570 \ --old-label 570.144 --new-label 570.211 \ --old-symbols symbols-570.144.json --new-symbols symbols-570.211.json \ --stimulus examples/scripts/trace_rpc.py \ --mutation-kernel firmware/uproc_demo/build/uproc_demo.elf \ --mutation-iterations 8 --mutation-seed 7 ``` Steps, in order: 1. **version_diff** — writes `/version-diff/` (section flags above); 2. **portability_matrix** — old and new as two targets, same budgets, writes `/portability-matrix/portability-report.json|.md` and per-pair compare reports; the self-comparison must show `all_zero`; 3. **mutation_smoke** — `mutation_harness` on the demo (or `--mutation-kernel`, with `--allow-foreign-mutation` for a re-hosted ELF) with a small budget; writes `/mutation-smoke/findings.jsonl|summary.md`. A step that raises is recorded as `status: error` and the later steps still run; the process exits `1` when any step errored, `0` otherwise (`skipped` steps do not fail the run, but their reasons are in the report). Exit `3` is the RM 610 refusal. `ci_release` flags: the `version_diff` inputs above plus `--matrix-symbols` (one JSON DB applied to both matrix targets), `--mutation-kernel` (default: the bundled demo), `--mutation-iterations`/`--mutation-seed`/ `--mutation-timeout`/`--mutation-settle`, and `--allow-foreign-mutation` for a non-demo re-hosted ELF. ### 4.3 Evidence bundle format ```text / release-report.json pipeline summary (steps, gate, labels) bundle-manifest.json sha256 + byte size of every other file version-diff/ docs/21 §2.3 tree portability-matrix/ portability-report.json|.md, symbols/, sessions/, compare/ mutation-smoke/ findings.jsonl, summary.md, sessions/ ``` `bundle-manifest.json`: ```json { "format": "greeneye-release-bundle/1", "tool": "harness/ci_release.py", "files": [{"path": "version-diff/version-diff.json", "bytes": 12345, "sha256": "..."}] } ``` `release-report.json` shape: ```json { "format": "greeneye-ci-release/1", "dry_run": false, "rm610_gate": {"status": "passed", "labels": ["570.144", "570.211"]}, "old": {"label": "570.144", "path": "gsp-570.144.fwimage.bin"}, "new": {"label": "570.211", "path": "gsp-570.211.fwimage.bin"}, "steps": [{"name": "version_diff", "status": "ok", "sections": {"image": "ok", "symbols": "ok", "corpus": "ok", "probe": "ok", "behavior": "ok"}}], "errors": [], "manifest": "bundle-manifest.json" } ``` The manifest is written last, so its hashes cover the final release report. Attach the directory (or its manifest) to the docs/07 evidence checklist and the docs/14 §6 pack; the docs/23 §4 step 7 expected-artifact list maps 1:1. --- ## 5. CI workflow `.github/workflows/risc-v-gsp-rm-debugger.yml` (new; `ci.yml` is untouched): * triggers on push/PR paths `debug/risc-v-gsp-rm-vm/**` and `tools/rm*/**`; * `ubuntu-latest`, no secrets, no hardware; * installs `qemu-system-misc clang lld-18`; * builds `make -C debug/risc-v-gsp-rm-vm sdk demo`; * runs `python3 -m unittest discover -s tests -t .` from the tool root (QEMU integration tests run; if QEMU were absent they skip cleanly). The release gates this workflow reaches are the harness suites: the docs/19 §6 fast inventory smoke runs inside `tests/test_discovery.py`, and the docs/21 dry-run is covered by `tests/test_version_diff.py`. The workflow does not run `control_discovery` or `ci_release` as standalone steps. A full `ci_release` run against real firmware is driven manually/agentically because it needs the versioned inputs and symbol DBs; CI-adjacent automation should call it with `--dry-run` first. --- ## 6. MCP monitoring `plugins/mcp_server.py` (docs/20) exposes the same machinery to an agent: `probe_firmware`, `portability_matrix_run`, `mutation_run`, `compare_sessions` and `analyze_current`. The agent pattern for a release: 1. `portability_matrix_run` with the old/new targets and fixed budgets; 2. `mutation_run` on the demo with the pinned seed and budget; 3. `compare_sessions` for the recorded evidence pair; 4. read `version-diff.json` / `release-report.json` / `bundle-manifest.json` from the run directory (agent reads the bundle, it does not recompute it). The MCP server remains read/observe by default and simulator-only (docs/20 §7). --- ## 7. Determinism, limits and safety * Fixed probe budgets, fixed scenario phases and the same stimulus make the self-diff zero: an unchanged target reports zero symbol/probe/behavior deltas (verified by `tests/test_version_diff.py`). * Agreement metrics M1–M6 are **not measured** without a reference capture (docs/06 §5); the skeleton carries local `observed` facts only. L2/L3 claims (timing, interrupts, device model) are out of scope. * Raw `.fwimage` targets execute as flat blobs; a firmware linked differently may fault earlier than on silicon — that is first-touch data, not a boot (docs/06 §3). * Only temporary work directories and `/` are written; QEMU is always launched and killed by the tools; log reads are bounded (`--log-bytes`). * No downloads, no `/dev`, no kernel module, no existing gdbstub, no physical GPU. The 610 gate is part of both CLIs, not documentation-only. --- ## 8. Tests ```bash python3 -m unittest tests.test_version_diff -v make test # includes these with the rest of the suite ``` `tests/test_version_diff.py` covers the gate (default refusal, token/env approval, gate-before-open), the `diff_corpus.py` matching discipline (moved/resized/added/removed, anonymous shift), the 3-column export parser, missing/incompatible/raw target handling, corpus hints, `ci_release --dry-run` side-effect freedom, clean `--help`, and a demo self-diff asserting zero inventory and behavior deltas. QEMU integration tests skip cleanly when QEMU or the demo ELF is absent. --- ## 9. Roadmap mapping | Item | Where this doc/tool contributes | |---|---| | **R6 — multi-version orchestration** | `version_diff` + `ci_release` prefix the matrix with a version-scoped diff | | **H4.1 — semantic diffing** | symbol/corpus inventory deltas and behavior profiles for two firmware versions | | **V1 — metrics/golden data** | deterministic self-diff and stable JSON reports support golden comparisons | | **docs/23 C5/C7** | release pipeline implemented and exercisable; dry-run rehearsal path | | **H7 — agentic harness** | self-describing bundle + docs/20 MCP tools for monitored runs |