diff options
Diffstat (limited to 'debug/risc-v-gsp-rm-vm/docs/53-FIRMWARE-OBSERVABILITY-WORKFLOW.md')
| -rw-r--r-- | debug/risc-v-gsp-rm-vm/docs/53-FIRMWARE-OBSERVABILITY-WORKFLOW.md | 245 |
1 files changed, 245 insertions, 0 deletions
diff --git a/debug/risc-v-gsp-rm-vm/docs/53-FIRMWARE-OBSERVABILITY-WORKFLOW.md b/debug/risc-v-gsp-rm-vm/docs/53-FIRMWARE-OBSERVABILITY-WORKFLOW.md new file mode 100644 index 0000000..4106244 --- /dev/null +++ b/debug/risc-v-gsp-rm-vm/docs/53-FIRMWARE-OBSERVABILITY-WORKFLOW.md @@ -0,0 +1,245 @@ +# Firmware Observability Workflow: using the emulator in the present problem space + +> **Status: 2026-10-06.** This guide says where the emulator belongs in the +> Open-IOV engineering loop, what it can and cannot answer, and exactly which +> commands to run for each question. It introduces four tools added this +> session (`state_decode`, `log_decode`, `server_snapshot`, `msgq_attach`) and +> wires them to the existing ones (`valprobe`, `rehost`, `state_seed`, +> `launch_audit`, `gmmu_walk`, `host_traffic`, `rm_capture`, `contracts`, +> `triage`, `version_diff`). It is offline-only: no GPU, module, guest or +> network. Fidelity is L0/L1 (instruction/memory); never quote timing or +> device-environment results from it. +> +> Index registration in `docs/00-OVERVIEW.md`/`README.md` is intentionally +> left to the in-flight docs edit owned by the simulator workstream. + +--- + +## 1. The layer map: what belongs where + +The project sits on three layers; each layer has one honest oracle. + +| Layer | What it decides | Oracle | Emulator role | +|---|---|---|---| +| **Firmware / server** (GSP-RM: controls, objects, context commit, logs) | "What will the firmware do with this control/state?" | the emulator (predicates, rehosts, seeded handlers, message loop) + a live log ring | first-class: dumps, probes, replay | +| **Front-end / engine** (PBDMA fetch, inline-QMD install, GR dispatch, SMs) | "Does the engine execute this work?" | **silicon only** — the emulator has no PBDMA/GR model by design | evidence preparation and diffing only; *never* a verdict | +| **Guest / host software** | "Did the guest see correct completion?" | the live guest plus the acceptance strings | offline decode of what happened | + +**Consequence:** the emulator is the right tool for *server decisions* and +*evidence reduction*; it is the wrong tool for *engine dispatch*. When a wall +is engine-side, the emulator's job is to (a) exonerate the server half and +(b) reduce the live search space, not to declare a fix. + +The **differential loop** that makes this practical: + +``` + live cycle / captured evidence ──► snapshot + decode + log join + ▲ │ + │ diff / query + │ ▼ + one hardware test ◄── ranked candidates ◄── emulator probes/rehost (server half) +``` + +--- + +## 2. The four new tools + +All four are stdlib-only, deterministic, session-log friendly, and each has +tests in `tests/` (84 tests across the four as of 2026-10-06). Run everything +from the tool root. + +### 2.1 `harness/state_decode.py` — turn channel state into named fields + +Decodes INSTANCE / RAMFC / context-buffer words into field tables and diffs +two snapshots. The bundled table is deliberately small and evidence-only +(INSTANCE magic and GPFIFO VA; RAMFC_TARGET bits 16/17 = ENG/CE context valid); +everything else comes from an extensible `--map` (`greeneye-state-map/1`). + +```bash +# Decode a captured ladder dump (finds INSTANCE/RAMFC blocks automatically) +python3 harness/state_decode.py decode --input <evidence>/06-ladder-<run>.txt --json out.json + +# Diff two decoded states (or two raw inputs) -> greeneye-state-diff/1 +python3 harness/state_decode.py diff a.json b.json +``` + +Use it whenever a run's verdict is "silence": the field table shows context +validity bits and the words that correlate with success, and the diff shows +what a cycle changed. + +### 2.2 `harness/log_decode.py` — the server's stderr, readable + +Normalizes the two real dump shapes (raw ring dumps with `LOGINIT-raw:`/ +`LOGRM-raw:` headers and hexdump rows; textual NOCAT/record lines incl. +continuation call stacks and aggregate entries) into `greeneye-log-decode/1`, +with histograms, filters, and optional nearest-symbol resolution. + +```bash +python3 harness/log_decode.py decode <evidence>/21-logdump-<run>.txt \ + --rec-type 5 --json log.json # just the asserts +python3 harness/log_decode.py decode <log> --symbols symbols.json +``` + +Use it to read *why* the server refused or asserted: recover addresses become +function names, and record histograms expose repeated failure signatures. + +### 2.3 `harness/server_snapshot.py` — canonical server-state snapshot + diff + +Collects chunk dumps and record lines from evidence directories/files into one +`greeneye-server-snapshot/1` JSON (labels → words + records), and diffs two +snapshots into `greeneye-server-snapshot-diff/1`. + +```bash +python3 harness/server_snapshot.py collect --file <evidence>/06-ladder-a.txt \ + --created 2026-10-06T00:00:00Z --json snap-a.json +python3 harness/server_snapshot.py collect --file <evidence>/06-ladder-b.txt \ + --created 2026-10-06T00:00:00Z --json snap-b.json +python3 harness/server_snapshot.py diff --a snap-a.json --b snap-b.json +``` + +This is the A/B instrument for the differential loop: two cycles, one diff, +sorted per-chunk added/removed/changed words. Record differences (e.g. dmesg +timestamps) are visible and honest rather than silently merged. + +Note: `20-gsp-phys-state-*.txt` is an aggregate key:value state, not a memory +chunk; it yields no chunks by design. Pair it with `state_decode` only after +converting its contents to a chunk form. + +### 2.4 `sim/msgq_attach.py` — boot the real firmware to its message loop + +Implements docs/51 §7 steps 1–3: boots the real 570.144 image to the B3 +message-loop port wait, attaches the host doorbell/queue rings before the run, +rings the doorbell, continues under a bounded budget, and classifies the first +poll (`greeneye-msgq-attach/1`). It reports honestly: loop reached or first +divergence, with the reason. + +```bash +python3 -m sim.msgq_attach run --doorbell --json attach.json --record <sess-dir> +python3 -m sim.msgq_attach demo # bounded run + honest skip if no image +``` + +Measured today: **B3 reached** (~474k instructions), doorbell write performed +and checksum-verified; the **libos port model** then completes the +external-interrupt receive handoff (`wait @ 0x4019df4: completed shuttle 10 +(extintr) size=0`), and the loop task's next syscall is `TASK_STATE_QUERY` at +`0x401a46c` — the libos trap/report path. **First ring poll: still none**, and +the report says exactly why (trap state for the query; the command-queue +consumer task is never scheduled in this single-hart model; the +`GSP_ARGUMENTS_CACHED` address handoff is not delivered). No fabricated poll; +two runs are byte-identical. + +--- + +## 3. Run-to-the-wall procedure (reproducible) + +This is the exact sequence to bring the emulator to the current frontier and +produce its verdict on what it *can* decide. + +```bash +cd open-iov/debug/risc-v-gsp-rm-vm +EVID=<our evidence directory> + +# 1. Server state at the current runs, decoded +python3 harness/state_decode.py decode --input $EVID/06-ladder-<run>.txt + +# 2. Server log records around the promote/launch +python3 harness/log_decode.py decode $EVID/21-logdump-<run>.txt + +# 3. A/B the two runs that differ in launch capture +python3 harness/server_snapshot.py collect --file $EVID/06-ladder-<a>.txt --json snap-a.json +python3 harness/server_snapshot.py collect --file $EVID/06-ladder-<b>.txt --json snap-b.json +python3 harness/server_snapshot.py diff --a snap-a.json --b snap-b.json + +# 4. Real image to the message loop (bounded; honest boundary report) +python3 -m sim.msgq_attach run --doorbell +``` + +What today's run of this procedure established: + +* INSTANCE decodes cleanly (magic `0xface`, GPFIFO VA `0x100000`); the + `0x00030000` marker appears at instance words `0x4c`/`0xac` — the same + success marker the live campaign reads, now a named field candidate rather + than an unexplained word. The RAMFC chunk's own `RAMFC_TARGET` is zero, + consistent with the server programming target bits through the instance map. +* The log dump decodes to 122 records with histograms and frame addresses — + the server's assert stream is now machine-readable and symbolizable. +* The two ladder runs diff cleanly at word level — the A/B instrument works. +* The real firmware boots to its message-loop wait deterministically, and the + first thing it needs is **libos port delivery**, not more MMIO. + +**Emulator verdict on the present wall:** the wall is the *forwarded launch's +inline-descriptor install* (front-end/engine physics). The emulator can and did +exonerate the server half (context validity markers, control acceptance, +mapping), but it **cannot decide engine dispatch** and will not be asked to. +The candidate fixes remain the live-lane tests (restore the proven stream +shape, add the descriptor-virtualization control, relocate the prelude) and, +if those fail, the qmdwin variant — all hardware-side. + +--- + +## 4. The full tool chooser (existing + new) + +| Question you actually have | Tool | Kind | +|---|---|---| +| What fields does this chunk hold? What changed between runs? | `state_decode` | new | +| Why did the firmware assert/refuse? What do its records say? | `log_decode` | new | +| Give me one canonical state snapshot per cycle, and diff it | `server_snapshot` | new | +| Does the real image reach its message loop? What does it poll first? | `msgq_attach` | new | +| Which entry/control forms does the firmware accept, on which fields? | `valprobe` | existing | +| Run one whole handler with crafted state; first missing state is the finding | `rehost` + `state_seed` | existing | +| Where would the engine read/write each launch leg? | `launch_audit` (+ `gmmu_walk`) | existing | +| Walk a captured page table through a VA | `gmmu_walk` | existing | +| Replay captured host traffic against the transport model | `host_traffic` | existing | +| Decode/compare real-client RPC captures | `rm_capture` | existing | +| Re-validate proven tools before a cycle | `contracts` / `run_contracts` | existing | +| New image triage / version delta | `triage` / `version_diff` | existing | + +--- + +## 5. Next emulation iterations (the continuation loop) + +Ordered by what each unblocks; each is bounded and session-recorded. + +1. **libos port delivery — landed (partial):** the port-role model + (`sim/libos_port.py`) now carries the doorbell wake through the + external-interrupt receive contract. Measured next blockers: the + `TASK_STATE_QUERY` trap state (`xcause`/`xepc` from the wait point) and + scheduling the command-queue consumer task (the single-hart model runs one + task). Success for this iteration = the consumer's first ring read + (`addr/width/value`), not a fabricated poll. +2. **`GSP_ARGUMENTS_CACHED` / queue-init parsing**: relocate the rings to the + addresses the host actually gives the firmware instead of the fixture block. +3. **CSB/TRANSCOM read-back semantics** for the loop's poll window. +4. **One served RPC**: dispatcher stub returning a real captured reply + (`rm_capture` stimulus), recorded as the B5 session. +5. **MSI-X/event delivery and IRQ arming** so replies can be pushed, not only + polled. +6. **Front-end/engine model — decision gate, not a default.** Reproducing the + inline-descriptor install (PBDMA method execution, IQ2M) would let the + emulator rank engine-side fixes offline, but the project has explicitly + excluded an engine model (surface map §6; it is silicon physics, and offline + engine claims are forbidden). Treat this as a gated, admitted research item + only if the live lane's next tests leave an unresolved ambiguity that only a + model could separate — never as a routine build. + +--- + +## 6. Fidelity, honesty and safety rules + +* Emulator outputs are L0/L1: instruction and memory semantics of the firmware. + No timing (L2), no device environment (L3), no silicon behaviour. +* Silence is not health: a missing fault does not mean success. Prefer + engine-authored observables; front-end-authored signals are not proof. +* Every run keeps its session/report; a claim without a session is an anecdote. +* Offline only: never a GPU, module, mdev, guest, network or capture from this + tree. Injected code is development-only and stays here. +* The differential loop is only as good as the captured evidence; a verdict on + a partial capture names the next page (or field) to capture, not a cause. + +--- + +## 7. Change log + +| Date | Change | +|---|---| +| 2026-10-06 | Created with the four new tools (`state_decode`, `log_decode`, `server_snapshot`, `msgq_attach`), the layer map, the run-to-the-wall procedure and the continuation loop. | |