path: root/debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md

# 25 — Value validation: measurable time-to-value evidence

> Status: implemented, documented and tested (audit: docs/27-TOOL-AUDIT.md; value demos: docs/25-VALUE-VALIDATION.md).
> Every number in this document is
> either **measured** on the bundled demo analog (recorded in
> `harness/value_demos/artifacts/`) or explicitly labelled a **projection**
> for the real case. No demo claims a real-GSP result the simulator cannot
> reach; device-model-blocked scenarios say so in their result.

## 1. Why this document exists

risc-v-gsp-rm-debugger is only worth its maintenance if it turns real firmware questions
into answers faster than static analysis and supervised hardware cycles. The
project handoffs record those questions and their costs in wall-clock and
cycles: the 0x59 RM-init chase (PART 91, 2026-10-01), the 20 GR counters
(PARTs 46–64, 2026-09-24…29), the GPFIFO GET fault (PART 91 ADDENDUM 9), the
P6 service wiring (PART 92 ADDENDA 5–11, 2026-10-02), and the version pairing
question (570.144 vs 570.211). This document maps each of them to a runnable
demonstration under [`harness/value_demos/`](../harness/value_demos/), states
the baseline cost from the handoffs, and measures the risc-v-gsp-rm-debugger path.

## 2. Methodology

### 2.1 What is measured, what is projected

| Class | Meaning | Where it appears |
|---|---|---|
| **Measured (analog)** | Wall-clock of the risc-v-gsp-rm-debugger path on the bundled `uproc_demo` firmware, native RV64 interpreter or QEMU; answers, counts, session bytes, trace slowdown | `result.json` per demo, `artifacts/results.json` |
| **Recorded (real baseline)** | Cost actually paid on the physical path, as recorded in the handoffs (dates, cycles, PARTs, wall spans) | `real_case.static_baseline` in each `result.json`, §5 below |
| **Projected** | What the same tool path would save on the real firmware once H2.2 executes it; labelled `projected` | `real_case.projection`; never presented as a result |
| **Blocked** | Real firmware access requires the H2.2 QEMU device model; `blocked_on_device_model: true` | every real `.fwimage` scenario here |

Fidelity is exactly docs/06: L0 instruction semantics and L1 memory behaviour
are exercised; **L2 time/concurrency is not modelled** and **L3 device
environment is absent**. A risc-v-gsp-rm-debugger observation is evidence about the
firmware, and anything that would change `vgsp` is escalated to a
physical capture before the product changes (docs/06 §6, docs/14 §7).

### 2.2 Run contract

* Each demo is a stdlib-only script run from the tool root; it prints the
  question, commands, answer, evidence and elapsed time, then one
  `RESULT_JSON: {...}` line.
* `run_all.py` executes each demo in a subprocess with a hard timeout,
  aggregates `artifacts/results.json`, and enforces the suite budget
  (per-demo ≤ 45 s, suite ≤ 180 s).
* Demos are deterministic: native sessions use virtual time (instruction
  count), stimuli are fixed, and the version-diff patch site is verified
  unique before use. Artifacts contain no host paths beyond the repo.

### 2.3 Scenario selection and “why not edge cases”

Every class below recurs across multiple handoff PARTs and sits on the
critical path to `vectorAdd`. The mapping is direct:

| Demo | Recurs in | Why it is on the vectorAdd path |
|---|---|---|
| s01 reply consumer | PART 91 cycles 2–13; status-queue decode cycles 22–25 | RM init must complete (`0x1001`, `rpc_result=0`) before any RPC; a generic failure hides the first error |
| s02 counter publication | PARTs 46–64 | `cuCtxCreate` completes on per-channel counters; 4/20 blocked the client |
| s03 unknown commands | every new image; docs/19 | control inventory (M1) is the input surface the emulation must handle |
| s04 run-until | PART 91 ADDENDUM 9 (cycles 18–25) | each ladder rung (VAS, PDB, channel, token, GPFIFO, semaphore) is a reachability question |
| s05 snapshot diff | PART 91 ADDENDUM 6/7 (identity, static config) | static-config fields feed guest wiring; a zeroed field kept the device-id read at zero |
| s06 inject service | PART 92 ADDENDA 5–11 (P6a–P6c) | guest-originated work executing on the engine is the shape of the load |
| s07 version diff | PART 91 cycles 2–3; every GSP release | the emulation must track firmware changes without regressing |
| s08 mutation triage | docs/18 R5; docs/19 §3.4; PARTs 3–9 | fail-closed rules come from error paths, not happy paths |
| s09 Ghidra address | PART 91 ADDENDUM 9 cycle 24 | the GPFIFO GET fix was accepted only after the runtime fault PC existed |
| s10 evidence pack | docs/14 §6; every milestone | a finding without sessions/hashes/M1–M6 is not actionable |

None of these is a bizarre edge case: they are the same observe → decode →
change loop the project runs on every firmware version, and each one is
cited more than once in the handoffs.

## 3. Scenario → tool → baseline → measured

Measured on the reference host (CPython 3.12); numbers regenerate with
`python3 harness/value_demos/run_all.py`. Times below are from the
`run_all` aggregate (~18.4 s total).

| # | Real scenario (source, date) | Risc-V GSP-RM debugger tool | Static/Ghidra-only baseline (recorded) | Analog (measured) | Real-case projection |
|---|---|---|---|---|---|
| s01 | 0x59 RM-init chase; fn72/fn73 (PART 91 + ADDENDA 2–7, 2026-10-01) | hook dispatcher, watch `result`, decode reply | cycles 2–13, ~11:40→16:52 ≈ 5 h 10 m; 10+ supervised boots (~5–6 min each); reply was generic `0x59` | **1.14 s**; 3/3 commands consumed and decoded; FAULT traps before the reply store | one hook + one watchpoint per command once H2.2 boots the image (blocked) |
| s02 | 20 GR per-channel counters (PARTs 46–64, 2026-09-24…29) | watch publication field, per-slot readback | ~5 days; PART 49 “only 4 of 20 counters ever moved”; ~2–3 s pin vs ~2 s client deadline | **1.34 s**; 20/20 slots published, 28 stores, store PCs captured | per-channel page watchpoints on real counter pages (blocked) |
| s03 | unknown control IDs / injected slots (docs/19, 2026-10-02) | sweep + dispatcher/handler hooks + classify | 189 real probes in ~36 s; static-only requires the indirect table per image | **5.96 s** for 24 IDs: 6 known, 8 injected-slot (aliases shown), 10 default | RM control/method sweep once H1.3/H2.2 exposes the table (blocked) |
| s04 | RM-client ladder reachability (PART 91 ADDENDUM 9, 2026-10-01) | `run_until`, trigger, direct `call` | cycles 18–25 = 19:12→20:53 ≈ 1 h 41 m, 8 supervised boots | **0.24 s**; natural hit / timeout / trigger / direct call | every ladder rung becomes a probe spec (blocked) |
| s05 | fn72 identity + 1656-B static config (PART 91 ADDENDUM 6/7, 2026-10-01) | before/after snapshot + schema diff | malformed fields suspected statically; acceptance required a live boot per iteration (10 NOCAT cycles) | **0.50 s**; 6 named fields changed, 7 byte runs fully attributed | same diff over real RM control/reply pages (blocked) |
| s06 | P6 service wiring / guest CE copies (PART 92 ADDENDA 5–11, 2026-10-02) | service injection SDK, both invocation routes | kernel rebuild + supervised cycle (~5–8 min) per rung; P6a→P6c spanned ~02:20–08:30 | **0.63 s**; blob loaded/registered; remote and mailbox results agree | RM service registry (M5) once H2.2 exists (blocked) |
| s07 | 570.144 vs 570.211 pairing (PART 91 cycles 2–3, 2026-10-01) | same stimulus on two builds, `compare_sessions` | one full boot (~5–6 min) per version plus manual log comparison | **1.14 s**; only the patched reply field moved; decision-table row = “prioritise” | real versions triaged on re-hosted firmware; symbols/sessions are the artifact (blocked for the raw image) |
| s08 | mutation / fail-closed corpus (R5; docs/18 §2; docs/19 §3.4) | seeded one-field mutations, signal classification | hand-derived asserts/NOCAT across PARTs 3–9; a 64-iteration QEMU run is the tracked harness baseline | **1.82 s**; 7 mutations → {ok:4, fault:2, no-progress:1}, all expectations matched | real controls land in the same harness (H1.3 surface, blocked) |
| s09 | GPFIFO GET fault `0x40000` vs `0x101000` (PART 91 ADDENDUM 9 cycle 24) | Ghidra TSV + bias + raw-address hook + `export_hits` | static hypothesis unproven; the decoded fault ended 8 supervised cycles (~1 h 41 m) | **0.72 s**; biased address resolves, hook fires at the exact PC, TSV annotated | names.tsv + bias loop once the image executes (blocked) |
| s10 | milestone evidence pack (docs/14 §6; cycles 11–13, 2026-10-01) | one recorded session → analyze/compare/hits/manifest/M1–M6 | hashes, sessions, logs and compare gathered by hand across boots | **4.92 s**; 12,276-byte session, A/B deterministic, all artifacts + overhead numbers | same pack against real sessions (blocked) |

## 4. Exact commands (per demo)

All from `open-iov/debug/risc-v-gsp-rm-vm`. `VALUE_DEMO_BACKEND` selects the
backend (`native` default, `qemu` where noted).

```bash
# s01 — reply consumer (native; QEMU also supported)
python3 harness/value_demos/s01_reply_consumer.py
#   hook("rpc_dispatch"); watch(gsp_ctrl+0x40, 8, 'w')
#   mailbox cmd=1 arg0=0x1234 -> 0xacdb; cmd=7 -> ~7; cmd=6 -> trap, no result store

# s02 — counter publication
python3 harness/value_demos/s02_counter_publication.py
#   watch(gsp_ctrl+0x40, 8, 'w'); 20 x (mailbox cmd=5, run) -> 20/20 published

# s03 — unknown command discovery
python3 harness/value_demos/s03_unknown_command.py
#   fresh session per ID 1..24; hook dispatcher + 6 handlers; classify
#   (findings.jsonl, control-discovery.md)

# s04 — run-until / reachability
python3 harness/value_demos/s04_run_until_function.py
#   run_until("rpc_sum"); run_until("rpc_fault", timeout=0.2) -> RspTimeout;
#   mailbox cmd=6 + run_until; call("rpc_ping", (0x41,))

# s05 — snapshot diff
python3 harness/value_demos/s05_snapshot_diff.py
#   snapshot before/after ECHO; Memory.diff; decode with gsp_ctrl.schema.json

# s06 — service injection (needs `make sdk`)
python3 harness/value_demos/s06_inject_service.py
#   load_service(...slot=6); invoke_remote; invoke_via_firmware

# s07 — version diff
python3 harness/value_demos/s07_version_diff.py
#   byte-patch the PING key (0xbeef -> 0xbef0) into version B; identical
#   stimulus; decoded field diff + compare_sessions

# s08 — mutation triage
python3 harness/value_demos/s08_mutation_triage.py
#   7 seeded one-field mutations; classify by stop signal + rpc_count

# s09 — Ghidra address join (QEMU by default)
python3 harness/value_demos/s09_grabbing_a_ghidra_address.py
#   names.tsv (program = runtime - 0x1000); load_names_tsv(bias=0x1000);
#   hook(ghidra + bias); export_hits -> greeneye_hits.tsv

# s10 — full evidence pack (QEMU by default)
python3 harness/value_demos/s10_full_session_evidence.py
#   two identical recorded sessions; analyze + compare + hits + manifest + M1–M6
#   + startup/session/trace overhead

# all of them
python3 harness/value_demos/run_all.py
python3 -m unittest tests.test_value_demos -v
```

## 5. Per-scenario detail

### s01 — reply consumer (0x59 first-error visibility)

The physical path sent `GSP_SET_SYSTEM_INFO` (fn72) and `SET_REGISTRY`
(fn73), and the only visible reply was `GSP_INIT_DONE rpc_result=0x59`
(`NV_ERR_OPERATING_SYSTEM`), wrapped generically by `RmStartup`; the RM log
ring was not mapped until the LOGRM id fix (`0x4c4f474d` “LOGM” vs
`0x4c4f47524d` “LOGRM”), so the failing collector step was invisible
(PART 91 ADDENDA 2–6). Ten-plus supervised boots later, the identity fix
(raw config-space dwords instead of unpacked fields) made init-done succeed
in 950 ms. The analog consumes each posted command, watches the result
field, and decodes the exact reply — including the case where the fault
handler traps before any reply store. **Success:** 3/3 dispatches, replies
match the ABI, FAULT has no result store. **Failure:** a command is not
consumed or a reply disagrees.

### s02 — counter publication (20 slots)

`cuCtxCreate` waits on 20 GR per-channel counters; the host saw the client’s
releases but could not translate the counter page, and for days only 4 of 20
client-visible slots moved (PART 49, 2026-09-29). The page VA
(`0x20400f000`, slots at `+0xec0`, 16-byte stride) and the different GPA per
boot were only proven by a raw live page dump (PARTs 51, 64). The analog
publishes 20 slots and attributes each store to a PC. **Success:** 20/20
published with one `rpc_count` increment per slot. **Failure:** any slot
missing or count delta disagreeing.

### s03 — unknown command discovery

New firmware can silently add or re-point dispatch entries; static reading of
an indirect table is per-image work. The 2026-10-02 sweep (189 probes, ~36 s)
found IDs 16–23 falling through to the injected-service handler table
(16–21 alias built-ins, 22–23 NULL returning `~id`) and 7–15/24–63 on the
default path (docs/19 §3.4). The analog sweeps 24 IDs with fresh sessions
and the same hook shape. **Success:** expected classes per ID. **Failure:**
a misclassification or an unreplayable probe.

### s04 — run-until reachability

The RM-client ladder asked per rung whether symbolized code executed under
the current stimulus; cycles 18–25 cost ~1 h 41 m of supervised boots
(PART 91 ADDENDUM 9). The analog shows three answers in one session: reached
on the natural path, timed out (not reached under this stimulus — docs/14 §5
explicitly says this is not unreachability), reached via a trigger, and a
direct Frida-like call. **Success:** all four behaviors agree with the ABI.
**Failure:** a route reports reachable without a hit, or a direct call
disagrees.

### s05 — snapshot diff

The fn72 payload carried unpacked identity fields where the reference sends
raw config-space dwords; the server read the device id back as zero, and the
fix was confirmed by rejecting hypotheses field-by-field (PART 91 ADDENDUM 6,
cycle 12), then by decoding the full 1656-byte static-config reply (cycle
13). The analog diffs two snapshots and attributes every changed byte run to
a schema field. **Success:** `rpc_count`/`last_rpc`/`result`/`buf` change,
`result` equals the ECHO length, no unattributed run. **Failure:** an
unattributed changed region.

### s06 — service injection

P6 wired guest-originated copies through the real engine; every rung was a
kernel rebuild plus a supervised cycle with fail-safe release discipline
(PART 92 ADDENDA 5–11, 2026-10-02). The analog loads `injected_demo.elf`,
registers it in the dispatch table, and invokes it directly and through the
mailbox. **Success:** both routes agree with the SDK ABI. **Failure:** a
route disagrees, the slot is not registered, or the blob cannot load
(skipped with a reason when `make sdk` has not run).

### s07 — version diff

Cycles 2 (570.211) and 3 (570.144) both failed init with `0x59`, disproving
the image-pairing hypothesis at the cost of one full boot each, and the
project’s H4.1 decision table exists to separate “the code changed” from
“the behaviour moved” (docs/14 §4.1). The analog patches the PING reply
constant into a labelled version B and replays an identical stimulus; only
the patched field moves. **Success:** exactly the patched reply moves.
**Failure:** unrelated movement or an unreplayable version.

### s08 — mutation triage

The fail-closed rules come from error paths; the tracked harness is bounded
seeded mutation with outcome classes (docs/18 §2), and the discovery sweep
found the sentinel/boundary pointer faults (docs/19 §3.4). The analog runs 7
one-field mutations in fresh sessions with `max_stops=1` (a faulted target
re-faults at the same PC, so the run stops at the first signal) and
classifies ok/trap/fault/no-progress. **Success:** every mutation matches
its case table. **Failure:** a contradicting class.

### s09 — grabbing a Ghidra address

The GPFIFO GET hypothesis (`0x101000`) was only settled when the decoded
status queue showed the fetch at `0x40000`; fix `6f9cbfa` followed
(PART 91 ADDENDUM 9, cycle 24). The analog exports a Ghidra-style
`names.tsv` with a deliberate base offset, loads it with the matching bias,
hooks the raw static address, triggers the path, and writes
`greeneye_hits.tsv` back for Ghidra. **Success:** biased address equals the
real function, hook records the hit, TSV annotates it. **Failure:** a hit at
a different address or an empty TSV.

### s10 — full session evidence

The evidence pack contract (docs/14 §6) was satisfied by hand for every
milestone (cycle evidence dirs, P6 dirs). The analog produces it in one run:
two deterministic sessions, analyze, compare, hits TSV, firmware/artifact
hashes, the M1–M6 skeleton (M2/M5 marked not measured without a physical
reference), and the measured overhead. **Success:** ABI reply, A/B
agreement, all artifacts, empty unexplained-delta list. **Failure:**
nondeterminism or a missing artifact.

## 6. Overhead analysis (measured, with numbers)

| Overhead | Measured on the analog | Real-path comparison |
|---|---|---|
| Session startup | native ≈ 0.001 s; QEMU ≈ 0.13 s per launch | one supervised hardware cycle ≈ 5–6 min (PART 91/92) |
| Session size | 12,276 B per recorded session (s10), including 2 snapshots and 16 event types | per-cycle evidence dirs hold multi-MB logs, rings and decodes |
| Trace slowdown | 3.1× run-only on the demo workload; 13.2× end-to-end including writing `blobs/trace.jsonl` (~200 k exec lines) | docs/16: ~2.4× exec trace on the demo workload (1.53 → 0.54 MIPS); guard ~7% |
| CI cost | `tests/test_value_demos.py`: 5 tests in ≈ 4.5 s; `run_all.py`: 10 demos in ≈ 18.4 s (budget 180 s) | a single physical cycle is 300–400× the entire demo suite |
| Host footprint | stdlib only; artifacts under `harness/value_demos/artifacts/`; no hardware, no guest, no kernel module | physical cycles hold D0/D3cold locks, mdev and a QEMU guest |

**Why the overhead is dominated by the time saved.** The expensive baselines
are per-question hardware cycles and multi-hour manual campaigns (s01 ≈ 5 h,
s02 ≈ 5 days, s04/s09 ≈ 1 h 41 m each, s06 ≈ 6 h). The tool path for the same
question class costs 0.2–6.0 s on the analog plus ≤0.13 s of launch — under
7 s end to end for every demo, or ~0.02 % of one 5–6-minute cycle even
before the campaign time. Trace logging at 3–13× of a sub-second run is
still seconds; a 12 KB session replaces a multi-MB manual capture. The only
place the tools add real cost is the once-per-question scripting in
`harness/value_demos/` and the 4.5 s test, which run in CI and are reused by
every later version.

## 7. OODA-loop compression

| Stage | Before (real path, as recorded) | After (risc-v-gsp-rm-debugger analog, measured) |
|---|---|---|
| **Observe** | one supervised boot + capture per hypothesis: 5–6 min, fail-safe discipline, one device-write experiment per cycle | launch 0.001–0.13 s, then hooks/watchpoints and bounded runs: 0.2–5.8 s per question |
| **Orient** | manual LOGINIT/LOGRM/NOCAT ring decode, static reports (REPORT3–10), hours per campaign; the ring was invisible until cycle 11 | `analyze`/`compare`/`Memory.diff` with named symbols: seconds, e.g. s05 attributes every changed byte to a schema field |
| **Decide** | one candidate fix per supervised cycle, accepted/rejected by another boot | version diff (s07), mutation matrix (s08) and per-slot publication (s02) test the whole candidate set in one deterministic run |
| **Act** | rebuild + supervised boot; escalate every conclusion to a physical capture | re-run the session (seconds); product change still gated on physical confirmation (docs/06 §6, docs/14 §7) |

The compression is not “the simulator replaces the hardware”; it is the loop
around each physical experiment. risc-v-gsp-rm-debugger reduces the number of physical
cycles needed to reach a confident hypothesis and produces the evidence pack
that makes the eventual physical confirmation cheap (s10).

## 8. Success/failure criteria summary

| Demo | Pass | Fail |
|---|---|---|
| s01 | 3/3 dispatches; PING/unknown replies match; FAULT traps before the store | command not consumed or reply disagrees |
| s02 | 20/20 slots published, `rpc_count` +20 | any slot missing / count mismatch |
| s03 | expected class per ID (6 known, 8 injected, 10 default) | misclassification or unreplayable probe |
| s04 | natural hit, timeout, trigger hit, direct call value | false reachability or wrong call value |
| s05 | required fields change, result = ECHO length, no unattributed run | unattributed changed region |
| s06 | remote and mailbox invocation agree with the SDK ABI | route disagreement or unloadable blob (skipped if SDK absent) |
| s07 | only the patched field moves | unrelated movement / unreplayable version |
| s08 | every mutation matches its expected class | contradicting class |
| s09 | biased address resolves; hook fires; TSV annotated | wrong address / empty TSV |
| s10 | ABI reply; A/B agreement; artifacts complete; no unexplained deltas | nondeterminism / missing artifact |

## 9. Reproducing and extending

```bash
cd open-iov/debug/risc-v-gsp-rm-vm
make demo && make sdk
python3 harness/value_demos/run_all.py            # writes artifacts/results.json
python3 -m unittest tests.test_value_demos -v     # fast gate
```

`artifacts/results.json` is the machine-readable source for the table in §3:
per-demo `elapsed_s`, `success`, `real_case`, answers, and artifact paths.
When the H2.2 device model lands, each demo's `projected` note becomes the
runbook for replacing its analog with the real image; the result format and
evidence contract do not change.

## 10. Limits and fidelity statement

* The analog firmware is not GSP-RM: no RM objects, status queue, object
  model or GPU device model. Demos reproduce the *shape* of each scenario
  (hook the dispatcher, watch the field, decode the reply, diff versions,
  mutate, inject) and are labelled as such.
* L2 timing/concurrency is not modelled; no demo makes a timing claim.
* L3 device environment is absent; every real-firmware result is
  `blocked_on_device_model`.
* Absence of a hit is a fact about that run at that fidelity, not proof of
  dead code (docs/14 §2).
* Full suite status and per-demo times are regenerated by
  `run_all.py`; do not quote a stale number without re-running it.