diff options
Diffstat (limited to 'debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md')
| -rw-r--r-- | debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md | 337 |
1 files changed, 337 insertions, 0 deletions
diff --git a/debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md b/debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md new file mode 100644 index 0000000..d47b8ea --- /dev/null +++ b/debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md @@ -0,0 +1,337 @@ +# 25 — Value validation: measurable time-to-value evidence + +> **# TOOL IN DEVELOPMENT - NOT READY YET!** See +> [00-OVERVIEW](00-OVERVIEW.md) for status. Every number in this document is +> either **measured** on the bundled demo analog (recorded in +> `harness/value_demos/artifacts/`) or explicitly labelled a **projection** +> for the real case. No demo claims a real-GSP result the simulator cannot +> reach; device-model-blocked scenarios say so in their result. + +## 1. Why this document exists + +green-eye is only worth its maintenance if it turns real firmware questions +into answers faster than static analysis and supervised hardware cycles. The +project handoffs record those questions and their costs in wall-clock and +cycles: the 0x59 RM-init chase (PART 91, 2026-10-01), the 20 GR counters +(PARTs 46–64, 2026-09-24…29), the GPFIFO GET fault (PART 91 ADDENDUM 9), the +P6 service wiring (PART 92 ADDENDA 5–11, 2026-10-02), and the version pairing +question (570.144 vs 570.211). This document maps each of them to a runnable +demonstration under [`harness/value_demos/`](../harness/value_demos/), states +the baseline cost from the handoffs, and measures the green-eye path. + +## 2. Methodology + +### 2.1 What is measured, what is projected + +| Class | Meaning | Where it appears | +|---|---|---| +| **Measured (analog)** | Wall-clock of the green-eye path on the bundled `uproc_demo` firmware, native RV64 interpreter or QEMU; answers, counts, session bytes, trace slowdown | `result.json` per demo, `artifacts/results.json` | +| **Recorded (real baseline)** | Cost actually paid on the physical path, as recorded in the handoffs (dates, cycles, PARTs, wall spans) | `real_case.static_baseline` in each `result.json`, §5 below | +| **Projected** | What the same tool path would save on the real firmware once H2.2 executes it; labelled `projected` | `real_case.projection`; never presented as a result | +| **Blocked** | Real firmware access requires the H2.2 QEMU device model; `blocked_on_device_model: true` | every real `.fwimage` scenario here | + +Fidelity is exactly docs/06: L0 instruction semantics and L1 memory behaviour +are exercised; **L2 time/concurrency is not modelled** and **L3 device +environment is absent**. A green-eye observation is evidence about the +firmware, and anything that would change `open-iov-vfio` is escalated to a +physical capture before the product changes (docs/06 §6, docs/14 §7). + +### 2.2 Run contract + +* Each demo is a stdlib-only script run from the tool root; it prints the + question, commands, answer, evidence and elapsed time, then one + `RESULT_JSON: {...}` line. +* `run_all.py` executes each demo in a subprocess with a hard timeout, + aggregates `artifacts/results.json`, and enforces the suite budget + (per-demo ≤ 45 s, suite ≤ 180 s). +* Demos are deterministic: native sessions use virtual time (instruction + count), stimuli are fixed, and the version-diff patch site is verified + unique before use. Artifacts contain no host paths beyond the repo. + +### 2.3 Scenario selection and “why not edge cases” + +Every class below recurs across multiple handoff PARTs and sits on the +critical path to `vectorAdd`. The mapping is direct: + +| Demo | Recurs in | Why it is on the vectorAdd path | +|---|---|---| +| s01 reply consumer | PART 91 cycles 2–13; status-queue decode cycles 22–25 | RM init must complete (`0x1001`, `rpc_result=0`) before any RPC; a generic failure hides the first error | +| s02 counter publication | PARTs 46–64 | `cuCtxCreate` completes on per-channel counters; 4/20 blocked the client | +| s03 unknown commands | every new image; docs/19 | control inventory (M1) is the input surface the emulation must handle | +| s04 run-until | PART 91 ADDENDUM 9 (cycles 18–25) | each ladder rung (VAS, PDB, channel, token, GPFIFO, semaphore) is a reachability question | +| s05 snapshot diff | PART 91 ADDENDUM 6/7 (identity, static config) | static-config fields feed guest wiring; a zeroed field kept the device-id read at zero | +| s06 inject service | PART 92 ADDENDA 5–11 (P6a–P6c) | guest-originated work executing on the engine is the shape of the load | +| s07 version diff | PART 91 cycles 2–3; every GSP release | the emulation must track firmware changes without regressing | +| s08 mutation triage | docs/18 R5; docs/19 §3.4; PARTs 3–9 | fail-closed rules come from error paths, not happy paths | +| s09 Ghidra address | PART 91 ADDENDUM 9 cycle 24 | the GPFIFO GET fix was accepted only after the runtime fault PC existed | +| s10 evidence pack | docs/14 §6; every milestone | a finding without sessions/hashes/M1–M6 is not actionable | + +None of these is a bizarre edge case: they are the same observe → decode → +change loop the project runs on every firmware version, and each one is +cited more than once in the handoffs. + +## 3. Scenario → tool → baseline → measured + +Measured on the reference host (CPython 3.12); numbers regenerate with +`python3 harness/value_demos/run_all.py`. Times below are from the +`run_all` aggregate (~18.4 s total). + +| # | Real scenario (source, date) | Green-eye tool | Static/Ghidra-only baseline (recorded) | Analog (measured) | Real-case projection | +|---|---|---|---|---|---| +| s01 | 0x59 RM-init chase; fn72/fn73 (PART 91 + ADDENDA 2–7, 2026-10-01) | hook dispatcher, watch `result`, decode reply | cycles 2–13, ~11:40→16:52 ≈ 5 h 10 m; 10+ supervised boots (~5–6 min each); reply was generic `0x59` | **1.14 s**; 3/3 commands consumed and decoded; FAULT traps before the reply store | one hook + one watchpoint per command once H2.2 boots the image (blocked) | +| s02 | 20 GR per-channel counters (PARTs 46–64, 2026-09-24…29) | watch publication field, per-slot readback | ~5 days; PART 49 “only 4 of 20 counters ever moved”; ~2–3 s pin vs ~2 s client deadline | **1.34 s**; 20/20 slots published, 28 stores, store PCs captured | per-channel page watchpoints on real counter pages (blocked) | +| s03 | unknown control IDs / injected slots (docs/19, 2026-10-02) | sweep + dispatcher/handler hooks + classify | 189 real probes in ~36 s; static-only requires the indirect table per image | **5.96 s** for 24 IDs: 6 known, 8 injected-slot (aliases shown), 10 default | RM control/method sweep once H1.3/H2.2 exposes the table (blocked) | +| s04 | RM-client ladder reachability (PART 91 ADDENDUM 9, 2026-10-01) | `run_until`, trigger, direct `call` | cycles 18–25 = 19:12→20:53 ≈ 1 h 41 m, 8 supervised boots | **0.24 s**; natural hit / timeout / trigger / direct call | every ladder rung becomes a probe spec (blocked) | +| s05 | fn72 identity + 1656-B static config (PART 91 ADDENDUM 6/7, 2026-10-01) | before/after snapshot + schema diff | malformed fields suspected statically; acceptance required a live boot per iteration (10 NOCAT cycles) | **0.50 s**; 6 named fields changed, 7 byte runs fully attributed | same diff over real RM control/reply pages (blocked) | +| s06 | P6 service wiring / guest CE copies (PART 92 ADDENDA 5–11, 2026-10-02) | service injection SDK, both invocation routes | kernel rebuild + supervised cycle (~5–8 min) per rung; P6a→P6c spanned ~02:20–08:30 | **0.63 s**; blob loaded/registered; remote and mailbox results agree | RM service registry (M5) once H2.2 exists (blocked) | +| s07 | 570.144 vs 570.211 pairing (PART 91 cycles 2–3, 2026-10-01) | same stimulus on two builds, `compare_sessions` | one full boot (~5–6 min) per version plus manual log comparison | **1.14 s**; only the patched reply field moved; decision-table row = “prioritise” | real versions triaged on re-hosted firmware; symbols/sessions are the artifact (blocked for the raw image) | +| s08 | mutation / fail-closed corpus (R5; docs/18 §2; docs/19 §3.4) | seeded one-field mutations, signal classification | hand-derived asserts/NOCAT across PARTs 3–9; a 64-iteration QEMU run is the tracked harness baseline | **1.82 s**; 7 mutations → {ok:4, fault:2, no-progress:1}, all expectations matched | real controls land in the same harness (H1.3 surface, blocked) | +| s09 | GPFIFO GET fault `0x40000` vs `0x101000` (PART 91 ADDENDUM 9 cycle 24) | Ghidra TSV + bias + raw-address hook + `export_hits` | static hypothesis unproven; the decoded fault ended 8 supervised cycles (~1 h 41 m) | **0.72 s**; biased address resolves, hook fires at the exact PC, TSV annotated | names.tsv + bias loop once the image executes (blocked) | +| s10 | milestone evidence pack (docs/14 §6; cycles 11–13, 2026-10-01) | one recorded session → analyze/compare/hits/manifest/M1–M6 | hashes, sessions, logs and compare gathered by hand across boots | **4.92 s**; 12,276-byte session, A/B deterministic, all artifacts + overhead numbers | same pack against real sessions (blocked) | + +## 4. Exact commands (per demo) + +All from `open-iov/debug/risc-v-gsp-rm-vm`. `VALUE_DEMO_BACKEND` selects the +backend (`native` default, `qemu` where noted). + +```bash +# s01 — reply consumer (native; QEMU also supported) +python3 harness/value_demos/s01_reply_consumer.py +# hook("rpc_dispatch"); watch(gsp_ctrl+0x40, 8, 'w') +# mailbox cmd=1 arg0=0x1234 -> 0xacdb; cmd=7 -> ~7; cmd=6 -> trap, no result store + +# s02 — counter publication +python3 harness/value_demos/s02_counter_publication.py +# watch(gsp_ctrl+0x40, 8, 'w'); 20 x (mailbox cmd=5, run) -> 20/20 published + +# s03 — unknown command discovery +python3 harness/value_demos/s03_unknown_command.py +# fresh session per ID 1..24; hook dispatcher + 6 handlers; classify +# (findings.jsonl, control-discovery.md) + +# s04 — run-until / reachability +python3 harness/value_demos/s04_run_until_function.py +# run_until("rpc_sum"); run_until("rpc_fault", timeout=0.2) -> RspTimeout; +# mailbox cmd=6 + run_until; call("rpc_ping", (0x41,)) + +# s05 — snapshot diff +python3 harness/value_demos/s05_snapshot_diff.py +# snapshot before/after ECHO; Memory.diff; decode with gsp_ctrl.schema.json + +# s06 — service injection (needs `make sdk`) +python3 harness/value_demos/s06_inject_service.py +# load_service(...slot=6); invoke_remote; invoke_via_firmware + +# s07 — version diff +python3 harness/value_demos/s07_version_diff.py +# byte-patch the PING key (0xbeef -> 0xbef0) into version B; identical +# stimulus; decoded field diff + compare_sessions + +# s08 — mutation triage +python3 harness/value_demos/s08_mutation_triage.py +# 7 seeded one-field mutations; classify by stop signal + rpc_count + +# s09 — Ghidra address join (QEMU by default) +python3 harness/value_demos/s09_grabbing_a_ghidra_address.py +# names.tsv (program = runtime - 0x1000); load_names_tsv(bias=0x1000); +# hook(ghidra + bias); export_hits -> greeneye_hits.tsv + +# s10 — full evidence pack (QEMU by default) +python3 harness/value_demos/s10_full_session_evidence.py +# two identical recorded sessions; analyze + compare + hits + manifest + M1–M6 +# + startup/session/trace overhead + +# all of them +python3 harness/value_demos/run_all.py +python3 -m unittest tests.test_value_demos -v +``` + +## 5. Per-scenario detail + +### s01 — reply consumer (0x59 first-error visibility) + +The physical path sent `GSP_SET_SYSTEM_INFO` (fn72) and `SET_REGISTRY` +(fn73), and the only visible reply was `GSP_INIT_DONE rpc_result=0x59` +(`NV_ERR_OPERATING_SYSTEM`), wrapped generically by `RmStartup`; the RM log +ring was not mapped until the LOGRM id fix (`0x4c4f474d` “LOGM” vs +`0x4c4f47524d` “LOGRM”), so the failing collector step was invisible +(PART 91 ADDENDA 2–6). Ten-plus supervised boots later, the identity fix +(raw config-space dwords instead of unpacked fields) made init-done succeed +in 950 ms. The analog consumes each posted command, watches the result +field, and decodes the exact reply — including the case where the fault +handler traps before any reply store. **Success:** 3/3 dispatches, replies +match the ABI, FAULT has no result store. **Failure:** a command is not +consumed or a reply disagrees. + +### s02 — counter publication (20 slots) + +`cuCtxCreate` waits on 20 GR per-channel counters; the host saw the client’s +releases but could not translate the counter page, and for days only 4 of 20 +client-visible slots moved (PART 49, 2026-09-29). The page VA +(`0x20400f000`, slots at `+0xec0`, 16-byte stride) and the different GPA per +boot were only proven by a raw live page dump (PARTs 51, 64). The analog +publishes 20 slots and attributes each store to a PC. **Success:** 20/20 +published with one `rpc_count` increment per slot. **Failure:** any slot +missing or count delta disagreeing. + +### s03 — unknown command discovery + +New firmware can silently add or re-point dispatch entries; static reading of +an indirect table is per-image work. The 2026-10-02 sweep (189 probes, ~36 s) +found IDs 16–23 falling through to the injected-service handler table +(16–21 alias built-ins, 22–23 NULL returning `~id`) and 7–15/24–63 on the +default path (docs/19 §3.4). The analog sweeps 24 IDs with fresh sessions +and the same hook shape. **Success:** expected classes per ID. **Failure:** +a misclassification or an unreplayable probe. + +### s04 — run-until reachability + +The RM-client ladder asked per rung whether symbolized code executed under +the current stimulus; cycles 18–25 cost ~1 h 41 m of supervised boots +(PART 91 ADDENDUM 9). The analog shows three answers in one session: reached +on the natural path, timed out (not reached under this stimulus — docs/14 §5 +explicitly says this is not unreachability), reached via a trigger, and a +direct Frida-like call. **Success:** all four behaviors agree with the ABI. +**Failure:** a route reports reachable without a hit, or a direct call +disagrees. + +### s05 — snapshot diff + +The fn72 payload carried unpacked identity fields where the reference sends +raw config-space dwords; the server read the device id back as zero, and the +fix was confirmed by rejecting hypotheses field-by-field (PART 91 ADDENDUM 6, +cycle 12), then by decoding the full 1656-byte static-config reply (cycle +13). The analog diffs two snapshots and attributes every changed byte run to +a schema field. **Success:** `rpc_count`/`last_rpc`/`result`/`buf` change, +`result` equals the ECHO length, no unattributed run. **Failure:** an +unattributed changed region. + +### s06 — service injection + +P6 wired guest-originated copies through the real engine; every rung was a +kernel rebuild plus a supervised cycle with fail-safe release discipline +(PART 92 ADDENDA 5–11, 2026-10-02). The analog loads `injected_demo.elf`, +registers it in the dispatch table, and invokes it directly and through the +mailbox. **Success:** both routes agree with the SDK ABI. **Failure:** a +route disagrees, the slot is not registered, or the blob cannot load +(skipped with a reason when `make sdk` has not run). + +### s07 — version diff + +Cycles 2 (570.211) and 3 (570.144) both failed init with `0x59`, disproving +the image-pairing hypothesis at the cost of one full boot each, and the +project’s H4.1 decision table exists to separate “the code changed” from +“the behaviour moved” (docs/14 §4.1). The analog patches the PING reply +constant into a labelled version B and replays an identical stimulus; only +the patched field moves. **Success:** exactly the patched reply moves. +**Failure:** unrelated movement or an unreplayable version. + +### s08 — mutation triage + +The fail-closed rules come from error paths; the tracked harness is bounded +seeded mutation with outcome classes (docs/18 §2), and the discovery sweep +found the sentinel/boundary pointer faults (docs/19 §3.4). The analog runs 7 +one-field mutations in fresh sessions with `max_stops=1` (a faulted target +re-faults at the same PC, so the run stops at the first signal) and +classifies ok/trap/fault/no-progress. **Success:** every mutation matches +its case table. **Failure:** a contradicting class. + +### s09 — grabbing a Ghidra address + +The GPFIFO GET hypothesis (`0x101000`) was only settled when the decoded +status queue showed the fetch at `0x40000`; fix `6f9cbfa` followed +(PART 91 ADDENDUM 9, cycle 24). The analog exports a Ghidra-style +`names.tsv` with a deliberate base offset, loads it with the matching bias, +hooks the raw static address, triggers the path, and writes +`greeneye_hits.tsv` back for Ghidra. **Success:** biased address equals the +real function, hook records the hit, TSV annotates it. **Failure:** a hit at +a different address or an empty TSV. + +### s10 — full session evidence + +The evidence pack contract (docs/14 §6) was satisfied by hand for every +milestone (cycle evidence dirs, P6 dirs). The analog produces it in one run: +two deterministic sessions, analyze, compare, hits TSV, firmware/artifact +hashes, the M1–M6 skeleton (M2/M5 marked not measured without a physical +reference), and the measured overhead. **Success:** ABI reply, A/B +agreement, all artifacts, empty unexplained-delta list. **Failure:** +nondeterminism or a missing artifact. + +## 6. Overhead analysis (measured, with numbers) + +| Overhead | Measured on the analog | Real-path comparison | +|---|---|---| +| Session startup | native ≈ 0.001 s; QEMU ≈ 0.13 s per launch | one supervised hardware cycle ≈ 5–6 min (PART 91/92) | +| Session size | 12,276 B per recorded session (s10), including 2 snapshots and 16 event types | per-cycle evidence dirs hold multi-MB logs, rings and decodes | +| Trace slowdown | 3.1× run-only on the demo workload; 13.2× end-to-end including writing `blobs/trace.jsonl` (~200 k exec lines) | docs/16: ~2.4× exec trace on the demo workload (1.53 → 0.54 MIPS); guard ~7% | +| CI cost | `tests/test_value_demos.py`: 5 tests in ≈ 4.5 s; `run_all.py`: 10 demos in ≈ 18.4 s (budget 180 s) | a single physical cycle is 300–400× the entire demo suite | +| Host footprint | stdlib only; artifacts under `harness/value_demos/artifacts/`; no hardware, no guest, no kernel module | physical cycles hold D0/D3cold locks, mdev and a QEMU guest | + +**Why the overhead is dominated by the time saved.** The expensive baselines +are per-question hardware cycles and multi-hour manual campaigns (s01 ≈ 5 h, +s02 ≈ 5 days, s04/s09 ≈ 1 h 41 m each, s06 ≈ 6 h). The tool path for the same +question class costs 0.2–6.0 s on the analog plus ≤0.13 s of launch — under +7 s end to end for every demo, or ~0.02 % of one 5–6-minute cycle even +before the campaign time. Trace logging at 3–13× of a sub-second run is +still seconds; a 12 KB session replaces a multi-MB manual capture. The only +place the tools add real cost is the once-per-question scripting in +`harness/value_demos/` and the 4.5 s test, which run in CI and are reused by +every later version. + +## 7. OODA-loop compression + +| Stage | Before (real path, as recorded) | After (green-eye analog, measured) | +|---|---|---| +| **Observe** | one supervised boot + capture per hypothesis: 5–6 min, fail-safe discipline, one device-write experiment per cycle | launch 0.001–0.13 s, then hooks/watchpoints and bounded runs: 0.2–5.8 s per question | +| **Orient** | manual LOGINIT/LOGRM/NOCAT ring decode, static reports (REPORT3–10), hours per campaign; the ring was invisible until cycle 11 | `analyze`/`compare`/`Memory.diff` with named symbols: seconds, e.g. s05 attributes every changed byte to a schema field | +| **Decide** | one candidate fix per supervised cycle, accepted/rejected by another boot | version diff (s07), mutation matrix (s08) and per-slot publication (s02) test the whole candidate set in one deterministic run | +| **Act** | rebuild + supervised boot; escalate every conclusion to a physical capture | re-run the session (seconds); product change still gated on physical confirmation (docs/06 §6, docs/14 §7) | + +The compression is not “the simulator replaces the hardware”; it is the loop +around each physical experiment. green-eye reduces the number of physical +cycles needed to reach a confident hypothesis and produces the evidence pack +that makes the eventual physical confirmation cheap (s10). + +## 8. Success/failure criteria summary + +| Demo | Pass | Fail | +|---|---|---| +| s01 | 3/3 dispatches; PING/unknown replies match; FAULT traps before the store | command not consumed or reply disagrees | +| s02 | 20/20 slots published, `rpc_count` +20 | any slot missing / count mismatch | +| s03 | expected class per ID (6 known, 8 injected, 10 default) | misclassification or unreplayable probe | +| s04 | natural hit, timeout, trigger hit, direct call value | false reachability or wrong call value | +| s05 | required fields change, result = ECHO length, no unattributed run | unattributed changed region | +| s06 | remote and mailbox invocation agree with the SDK ABI | route disagreement or unloadable blob (skipped if SDK absent) | +| s07 | only the patched field moves | unrelated movement / unreplayable version | +| s08 | every mutation matches its expected class | contradicting class | +| s09 | biased address resolves; hook fires; TSV annotated | wrong address / empty TSV | +| s10 | ABI reply; A/B agreement; artifacts complete; no unexplained deltas | nondeterminism / missing artifact | + +## 9. Reproducing and extending + +```bash +cd open-iov/debug/risc-v-gsp-rm-vm +make demo && make sdk +python3 harness/value_demos/run_all.py # writes artifacts/results.json +python3 -m unittest tests.test_value_demos -v # fast gate +``` + +`artifacts/results.json` is the machine-readable source for the table in §3: +per-demo `elapsed_s`, `success`, `real_case`, answers, and artifact paths. +When the H2.2 device model lands, each demo's `projected` note becomes the +runbook for replacing its analog with the real image; the result format and +evidence contract do not change. + +## 10. Limits and fidelity statement + +* The analog firmware is not GSP-RM: no RM objects, status queue, object + model or GPU device model. Demos reproduce the *shape* of each scenario + (hook the dispatcher, watch the field, decode the reply, diff versions, + mutate, inject) and are labelled as such. +* L2 timing/concurrency is not modelled; no demo makes a timing claim. +* L3 device environment is absent; every real-firmware result is + `blocked_on_device_model`. +* Absence of a hit is a fact about that run at that fidelity, not proof of + dead code (docs/14 §2). +* Full suite status and per-demo times are regenerated by + `run_all.py`; do not quote a stale number without re-running it. |