path: root/debug/risc-v-gsp-rm-vm/docs/40-NEW-VERSION-TRIAGE.md

# 40 — New-version triage (one command)

> **Status: in development (2026-10-03).** Implemented, tested and measured on
> the TU104 570.144 image; the measured value proof is section 5. Interfaces
> and report fields may still change. Simulator-only: never touches hardware,
> GPU, mdev, a kernel module or a guest; nothing is downloaded.
>
> **Audience:** agents and maintainers of risc-v-gsp-rm-debugger.
>
> This document is public-safe: it describes components, behaviour and
> measured numbers only.

---

## 1. Why this exists (U46)

The program has one goal: cut the time from a new NVIDIA RM driver / GSP
firmware release to supported in `vgsp`. Today a new image is triaged by hand
by reading docs/07 P1 and invoking four different tools in sequence (probe,
native boot, A/B, version diff), then collecting session paths and writing a
summary.

`harness/triage.py` turns that into **one bounded command** that reuses the
existing tools - it does not reimplement any of them:

| Step | Reuses | Answers |
|---|---|---|
| `identity` | `greeneye.elf`, `sha256` | what is the image, which version, which baseline |
| `probe` | `harness/sim_qemu_line.py:run_qemu_baseline` (the v0.1 probe path) | where does it first fault / first touch under QEMU |
| `native_boot` | `python3 -m sim.gsp_boot` (S1 device model + boot ladder) | how far does the native ladder get (B0–B4) |
| `ab_eval` | `harness/sim_qemu_line.py:run_harness` | QEMU vs native A/B, determinism, frozen thresholds |
| `version_diff` | `harness/version_diff.py` (only with `--baseline`) | image/symbol/corpus/probe/behavior delta, RM 610 gate respected |
| `features` | none (introspection) | hooks/watch/memory/session API, native sim, TCG plugin, MCP, demo |
| `support_readiness` | derived | blockers and ordered next actions for `vgsp` support work |

Every step is wrapped: a failing step is recorded as `status: error`, the
remaining steps still run, and the report is still written. Every session a
step used is listed with its sha256 hashes.

---

## 2. Usage

Run from the tool root (`open-iov/debug/risc-v-gsp-rm-vm`):

```bash
# Plan only (no targets opened, no --out created)
python3 harness/triage.py \
    --firmware ../../firmware/57X-GSP/570.144/gsp-570.144.fwimage.bin \
    --version 570.144 --out dumps/triage-570.144 --dry-run

# Single new image: identity + probe + bootstrap ladder + A/B + features
python3 harness/triage.py \
    --firmware <new.fwimage.bin> --version <label> --out dumps/triage-<label>

# With a baseline: adds the full version diff (docs/21)
python3 harness/triage.py \
    --firmware <new.fwimage.bin> --version <new-label> \
    --baseline <old.fwimage.bin> --baseline-label <old-label> \
    --out dumps/triage-<old>-<new>

# One command via the launcher or make
./risc-v-gsp-rm-debugger.sh triage --firmware <new> --out dumps/triage-new
make triage FW=<new> OUT=dumps/triage-new TRIAGE_ARGS="--baseline <old>"
```

Flags: `--firmware` (required), `--baseline`, `--version`, `--baseline-label`,
`--out` (required unless `--dry-run`), `--budget` (native instructions per
attempt, default 200000, hard cap 2000000), `--qemu-duration` (default 0.5s),
`--step-timeout` (per-step wall-clock net, default 300s), `--release`,
`--approval-token`, `--dry-run`.

Safety: `--dry-run` opens no target and creates no `--out`. The RM 610 gate
runs first in every mode (docs/23) and refuses with exit `3`; labels checked
are `--release`, `--version`, `--baseline-label` and both file basenames.
Exit codes: `0` report written (skipped steps included), `1` at least one step
errored, `2` tool/input error (or firmware absent), `3` RM 610 refusal.

Output tree under `--out`:

```text
<out>/
  triage-report.json        stable machine-readable report
  triage-report.md          rendered report
  sessions/qemu-probe/      QEMU probe session
  sessions/native-boot/     S1 boot-ladder session
  ab-eval/                  docs/37 A/B report + sim-qemu-line.json|.md + sessions/
  version-diff/             only with --baseline (docs/21 tree)
```

---

## 3. Report schema (`greeneye-new-version-triage/1`)

```json
{
  "format": "greeneye-new-version-triage/1",
  "tool": "harness/triage.py",
  "generated_at": "2026-10-03T04:44:24Z",
  "rm610_gate": {"status": "passed", "labels": ["570.144"]},
  "input": {
    "firmware": {"path": "...", "sha256": "...", "size": 28528288,
                 "version": "570.144W", "version_source": "gsp_version.txt sidecar"},
    "baseline": null, "baseline_supplied": false,
    "budget": 200000, "qemu_duration": 0.5, "triage_command": "..."
  },
  "thresholds": {"frozen_on": "2026-10-03", "manual_baseline_s": 1800.0,
                 "triage_wall_s_max": 600.0},
  "steps": [
    {"name": "identity", "status": "ok", "wall_s": 0.099, "firmware": {...}, "elf": {...}},
    {"name": "probe", "status": "ok", "rung": "pre-B0", "first_fault": {...},
     "stopped_pc": 0, "sessions": [...]},
    {"name": "native_boot", "status": "ok", "rung": "B2",
     "instructions_executed": 14682, "stop": {...}},
    {"name": "ab_eval", "status": "ok", "native_rung": "B2",
     "native_identical": true, "above_qemu_line": true, "verdict": "above"},
    {"name": "version_diff", "status": "skipped", "reason": "no --baseline supplied"},
    {"name": "features", "status": "ok", "checks": {...}, "unavailable": []}
  ],
  "sessions": [{"step": "probe", "name": "qemu-probe", "dir": "sessions/qemu-probe",
                "present": true, "hashes": {"session.json": "...", "events.jsonl": "...", "summary.md": "..."}}],
  "support_readiness": {"verdict": "sim-observable", "baseline_compared": false,
                        "observable_rung": "B2", "blockers": [], "next_actions": ["..."]},
  "threshold_checks": {"triage_wall_s": 2.781, "triage_wall_s_max": 600.0,
                       "wall_within_max": true, "manual_baseline_s": 1800.0,
                       "speedup_x": 647.2, "beats_manual_baseline": true,
                       "required_steps_missing": []},
  "wall_s": 2.781,
  "artifacts": {"report_json": "<out>/triage-report.json", "report_md": "<out>/triage-report.md"}
}
```

The `wall_s`/`threshold_checks` values above are the **measured 2026-10-03
sample** and are host-load dependent, not fixed constants.

Key fields:

* `steps[].status` is `ok | skipped | error`; `skipped` carries a reason and
  never fails the run.
* `support_readiness.verdict` is `blocked` (identity/step errors),
  `first-touch-only` (no runtime observation yet) or `sim-observable` (native
  ladder reached `B2`+, the point where the firmware runs past boot and can be
  instrumented).
* `support_readiness.next_actions` is the ordered work list for `vgsp` support
  (device-model gaps, symbol DBs, A/B above-the-line work, missing inputs).
* `sessions[].hashes` is the docs/07 evidence-checklist hash list; attach it
  with the recorded sessions.

---

## 4. Worked example — 570.144 (measured 2026-10-03)

Commands (real, read-only; `/usr/bin/time` around the wrapper):

```bash
./risc-v-gsp-rm-debugger.sh triage \
    --firmware ../../firmware/57X-GSP/570.144/gsp-570.144.fwimage.bin \
    --version 570.144 --out dumps/triage-570.144
```

Measured output (trimmed):

```text
[triage] identity: status=ok
[triage] probe: status=ok reason=ok
[triage] native_boot: status=ok
[triage] ab_eval: status=ok
[triage] version_diff: status=skipped reason=no --baseline supplied
[triage] features: status=ok
[triage] readiness: sim-observable (baseline_compared=False)
[triage] next: native ladder is at B2 (runtime-observable): inspect the milestone/stop with hooks ...
[triage] next: native A/B is not yet above the QEMU line (verdict=above_lenient_only): device model ...
[triage] next: re-run with --baseline <previous.fwimage> for the symbol/behavior delta ...
[triage] wall: 2.781s vs manual baseline 1800.0s (speedup 647.2x)
```

The `2.781s`/`647.2x` line is the **measured 2026-10-03 sample** (host-load
dependent, not a fixed constant).

What the report recorded:

| Step | Measured | Meaning |
|---|---|---|
| identity | `570.144W` via `gsp_version.txt`, sha256 + 28,528,288 bytes | image paired and hashed before any run |
| probe | QEMU first fault `pc_fw 0x4009014`, load access at `0x2000000000000A00` | matches the docs/37 QEMU reference first touch |
| native_boot | rung **B2**, 14,682 instructions, stop fault `pc_fw 0x400EA0C`, 0 unknown MMIO | S1 ladder is two rungs past the QEMU fault; boot shim consumed, relocation milestone hit |
| ab_eval | native `B2` (14,682 instructions, 0 unknown MMIO), identical across the repeat, `above_qemu_line: true`, verdict `above` | post-S1 re-run (2026-10-03, later): the discovered `sim/profiles` model + shim now satisfy the docs/37 line contract; the pre-S1 run was `B0`/`above_lenient_only` |
| features | greeneye API, native backend, boot ladder, A/B, version_diff, TCG plugin, MCP, demo, QEMU all present | capability surface for the follow-up work |

Post-S1 re-run (2026-10-03, later; same command, after S1 and the harness
wiring landed): `ab_eval` reports native rung **B2**, all five frozen
thresholds true and `above_qemu_line: true`; the "native A/B is not yet above
the QEMU line" next-action line no longer appears. Wall ≈2.7 s
(host-load dependent). The pre-S1 trace above is kept as the recorded
before/after evidence.

Baseline variant (same image both sides, an unchanged-version rehearsal with
the docs/21 path):

```bash
./risc-v-gsp-rm-debugger.sh triage \
    --firmware ../../firmware/57X-GSP/570.144/gsp-570.144.fwimage.bin \
    --baseline ../../firmware/57X-GSP/570.144/gsp-570.144.fwimage.bin \
    --version 570.144 --baseline-label 570.144 --out dumps/triage-570.144-baseline
```

Measured (2026-10-03, host-load dependent): **11.388s** internal wall
(11.64s including interpreter startup);
`version_diff` = 8.526s, sections `image/probe/behavior: ok`,
`symbols/corpus: skipped` with reasons; image self-diff `same: true`; 7 session
directories hashed (probe, native boot, 2 A/B native, A/B QEMU, version-diff
probe + behavior). Readiness stayed `sim-observable`, `baseline_compared:
true`, with next actions for the missing symbol/corpus inputs.

Frozen thresholds (recorded in `harness/triage.py` on 2026-10-03 **before**
the runs, per handoff §14.5): manual baseline `1800s`, triage wall max
`600s`. Result: `wall_within_max: true`, `beats_manual_baseline: true`,
speedup `647x` (single image) / `158x` (with baseline) — both wall times are
**measured 2026-10-03 samples (host-load dependent)**, not fixed constants. The
manual baseline is
a conservative understatement of the steps replaced (read docs/07 P1, choose
budgets, run probe + boot ladder + A/B, run version diff, collect session
hashes and write the summary for one image).

---

## 5. Value proof and its limits

* The frozen contract is in the report itself (`thresholds`,
  `threshold_checks`); the same numbers are restated in the module docstring
  and here. Never move them silently.
* The measured numbers above are from this checkout on 2026-10-03; reproduce
  with the exact commands in section 4.
* **Conservative baseline:** 1800s is an *estimate of manual effort*, not a
  timed human study. The speedup numbers are reported against it, and the
  report flags them as such.
* **What the run proves:** one bounded command produces the identity, probe,
  boot-ladder, A/B and capability evidence with hashed sessions in seconds -
  the mechanical part of docs/07 P1.
* **What it does not prove:** that the remaining human work (interpreting the
  deltas, extending the device model, the accuracy loop, the `vgsp` change) is
  automated; it is not, and the next-actions list is where it stays explicit.

---

## 6. Release-to-support mapping

| Triage output | Next workflow (docs) |
|---|---|
| identity + hash + version | docs/07 P1 step 1, docs/22 pairing checks, docs/28 sweep input |
| probe first fault / first touch | docs/07 P1 step 2, docs/24 device map harvest |
| native boot rung + stop | docs/35 §3.3 ladder; the `vgsp` support scope starts at `B2`+ |
| A/B verdict | docs/37 line contract (U50) |
| version_diff sections + symbol/corpus skips | docs/21, docs/14 §6 evidence pack |
| readiness next actions | docs/07 P1.7 acceptance, the actual `vgsp` support work items |
| RM 610 gate | docs/23 (never bypassed) |

---

## 7. Not verified (explicit)

* **No real 570.211+ new-version pair exists locally**; the cross-version
  delta path is exercised only by the self-diff rehearsal in section 4.
* **Symbols and corpus are skipped for raw `.fwimage` targets** (no `.symtab`,
  no Ghidra exports supplied): the handler-inventory delta is not measured
  until those inputs are supplied.
* **The S4 A/B verdict is `above` on the frozen checks after S1 landed**
  (B2, documented device model + shim, deterministic, within the wall
  bound); B3/B4 are not reached and triage makes no "beats QEMU" claim beyond
  the documented rung contract (`above_lenient_only` was the pre-S1 result).
* **Fidelity:** L0/L1 execution facts only; no L2 timing and no full L3
  device-environment claims (docs/06, docs/36). Cycles are modeled; the boot
  shim values are declared assumptions.
* **No physical backend, `/dev`, module, mdev, guest or network access** was
  used or is performed by the tool.

---

## 8. Tests

```bash
python3 -m unittest tests.test_triage -v   # 14 tests
make test                                  # full suite
```

`tests/test_triage.py` covers the version-label extraction chain, the report
shape and artifact writes, session hashing, partial-failure handling
(injected step failure still yields a report), missing-firmware behaviour
(reported, not raised), budget bounds, RM 610 refusal (exit `3`), dry-run
side-effect freedom, and a tiny-budget integration run against the real
570.144 image that skips cleanly when the image or QEMU is absent.

<!-- change (2026-10-03): new document recording the one-command new-version
triage workflow (U46), its measured 570.144 value proof and its limits;
scope: docs/40 only; benefit: the release-to-support loop has one documented
entry point and one attachable evidence bundle. -->