# Firmware Observability Workflow: using the emulator in the present problem space > **Status: 2026-10-06.** This guide says where the emulator belongs in the > Open-IOV engineering loop, what it can and cannot answer, and exactly which > commands to run for each question. It introduces four tools added this > session (`state_decode`, `log_decode`, `server_snapshot`, `msgq_attach`) and > wires them to the existing ones (`valprobe`, `rehost`, `state_seed`, > `launch_audit`, `gmmu_walk`, `host_traffic`, `rm_capture`, `contracts`, > `triage`, `version_diff`). It is offline-only: no GPU, module, guest or > network. Fidelity is L0/L1 (instruction/memory); never quote timing or > device-environment results from it. > > Index registration in `docs/00-OVERVIEW.md`/`README.md` is intentionally > left to the in-flight docs edit owned by the simulator workstream. --- ## 1. The layer map: what belongs where The project sits on three layers; each layer has one honest oracle. | Layer | What it decides | Oracle | Emulator role | |---|---|---|---| | **Firmware / server** (GSP-RM: controls, objects, context commit, logs) | "What will the firmware do with this control/state?" | the emulator (predicates, rehosts, seeded handlers, message loop) + a live log ring | first-class: dumps, probes, replay | | **Front-end / engine** (PBDMA fetch, inline-QMD install, GR dispatch, SMs) | "Does the engine execute this work?" | **silicon only** — the emulator has no PBDMA/GR model by design | evidence preparation and diffing only; *never* a verdict | | **Guest / host software** | "Did the guest see correct completion?" | the live guest plus the acceptance strings | offline decode of what happened | **Consequence:** the emulator is the right tool for *server decisions* and *evidence reduction*; it is the wrong tool for *engine dispatch*. When a wall is engine-side, the emulator's job is to (a) exonerate the server half and (b) reduce the live search space, not to declare a fix. The **differential loop** that makes this practical: ``` live cycle / captured evidence ──► snapshot + decode + log join ▲ │ │ diff / query │ ▼ one hardware test ◄── ranked candidates ◄── emulator probes/rehost (server half) ``` --- ## 2. The four new tools All four are stdlib-only, deterministic, session-log friendly, and each has tests in `tests/` (84 tests across the four as of 2026-10-06). Run everything from the tool root. ### 2.1 `harness/state_decode.py` — turn channel state into named fields Decodes INSTANCE / RAMFC / context-buffer words into field tables and diffs two snapshots. The bundled table is deliberately small and evidence-only (INSTANCE magic and GPFIFO VA; RAMFC_TARGET bits 16/17 = ENG/CE context valid); everything else comes from an extensible `--map` (`greeneye-state-map/1`). ```bash # Decode a captured ladder dump (finds INSTANCE/RAMFC blocks automatically) python3 harness/state_decode.py decode --input /06-ladder-.txt --json out.json # Diff two decoded states (or two raw inputs) -> greeneye-state-diff/1 python3 harness/state_decode.py diff a.json b.json ``` Use it whenever a run's verdict is "silence": the field table shows context validity bits and the words that correlate with success, and the diff shows what a cycle changed. ### 2.2 `harness/log_decode.py` — the server's stderr, readable Normalizes the two real dump shapes (raw ring dumps with `LOGINIT-raw:`/ `LOGRM-raw:` headers and hexdump rows; textual NOCAT/record lines incl. continuation call stacks and aggregate entries) into `greeneye-log-decode/1`, with histograms, filters, and optional nearest-symbol resolution. ```bash python3 harness/log_decode.py decode /21-logdump-.txt \ --rec-type 5 --json log.json # just the asserts python3 harness/log_decode.py decode --symbols symbols.json ``` Use it to read *why* the server refused or asserted: recover addresses become function names, and record histograms expose repeated failure signatures. ### 2.3 `harness/server_snapshot.py` — canonical server-state snapshot + diff Collects chunk dumps and record lines from evidence directories/files into one `greeneye-server-snapshot/1` JSON (labels → words + records), and diffs two snapshots into `greeneye-server-snapshot-diff/1`. ```bash python3 harness/server_snapshot.py collect --file /06-ladder-a.txt \ --created 2026-10-06T00:00:00Z --json snap-a.json python3 harness/server_snapshot.py collect --file /06-ladder-b.txt \ --created 2026-10-06T00:00:00Z --json snap-b.json python3 harness/server_snapshot.py diff --a snap-a.json --b snap-b.json ``` This is the A/B instrument for the differential loop: two cycles, one diff, sorted per-chunk added/removed/changed words. Record differences (e.g. dmesg timestamps) are visible and honest rather than silently merged. Note: `20-gsp-phys-state-*.txt` is an aggregate key:value state, not a memory chunk; it yields no chunks by design. Pair it with `state_decode` only after converting its contents to a chunk form. ### 2.4 `sim/msgq_attach.py` — boot the real firmware to its message loop Implements docs/51 §7 steps 1–3: boots the real 570.144 image to the B3 message-loop port wait, attaches the host doorbell/queue rings before the run, rings the doorbell, continues under a bounded budget, and classifies the first poll (`greeneye-msgq-attach/1`). It reports honestly: loop reached or first divergence, with the reason. ```bash python3 -m sim.msgq_attach run --doorbell --json attach.json --record python3 -m sim.msgq_attach demo # bounded run + honest skip if no image ``` Measured today: **B3 reached** (~474k instructions), doorbell write performed and checksum-verified; the **libos port model** completes the external-interrupt receive handoff (`wait @ 0x4019df4: completed shuttle 10 (extintr) size=0`) and serves the following `TASK_STATE_QUERY` with a modelled spurious/EINT trap state, so the init task consumes the report and re-blocks. The image's **two linked tasks are discovered from its own section/manifest records** (an init task and the RM task); the **scheduled second task** then completes its boot port handshake with the init task inside the bounded budget (`modelled:cross-task-send` / `modelled:cross-task-reply`, payloads copied from firmware memory), after which its own start-up continues past the 2M instruction cap before its first further port syscall. **First ring poll: still none**, and the report says exactly why (`missing_semantic.stage = second-task-frontier`: the consumer start-up is not reachable from its entry inside the cap; the `GSP_ARGUMENTS_CACHED` queue-address handoff is not delivered; the doorbell→consumer binding is not modelled). No fabricated poll; two runs are byte-identical. --- ## 3. Run-to-the-wall procedure (reproducible) This is the exact sequence to bring the emulator to the current frontier and produce its verdict on what it *can* decide. ```bash cd open-iov/debug/risc-v-gsp-rm-vm EVID= # 1. Server state at the current runs, decoded python3 harness/state_decode.py decode --input $EVID/06-ladder-.txt # 2. Server log records around the promote/launch python3 harness/log_decode.py decode $EVID/21-logdump-.txt # 3. A/B the two runs that differ in launch capture python3 harness/server_snapshot.py collect --file $EVID/06-ladder-.txt --json snap-a.json python3 harness/server_snapshot.py collect --file $EVID/06-ladder-.txt --json snap-b.json python3 harness/server_snapshot.py diff --a snap-a.json --b snap-b.json # 4. Real image to the message loop (bounded; honest boundary report) python3 -m sim.msgq_attach run --doorbell ``` What today's run of this procedure established: * INSTANCE decodes cleanly (magic `0xface`, GPFIFO VA `0x100000`); the `0x00030000` marker appears at instance words `0x4c`/`0xac` — the same success marker the live campaign reads, now a named field candidate rather than an unexplained word. The RAMFC chunk's own `RAMFC_TARGET` is zero, consistent with the server programming target bits through the instance map. * The log dump decodes to 122 records with histograms and frame addresses — the server's assert stream is now machine-readable and symbolizable. * The two ladder runs diff cleanly at word level — the A/B instrument works. * The real firmware boots to its message-loop wait deterministically, and the first thing it needs is **libos port delivery**, not more MMIO. **Emulator verdict on the present wall:** the wall is the *forwarded launch's inline-descriptor install* (front-end/engine physics). The emulator can and did exonerate the server half (context validity markers, control acceptance, mapping), but it **cannot decide engine dispatch** and will not be asked to. The candidate fixes remain the live-lane tests (restore the proven stream shape, add the descriptor-virtualization control, relocate the prelude) and, if those fail, the qmdwin variant — all hardware-side. --- ## 4. The full tool chooser (existing + new) | Question you actually have | Tool | Kind | |---|---|---| | What fields does this chunk hold? What changed between runs? | `state_decode` | new | | Why did the firmware assert/refuse? What do its records say? | `log_decode` | new | | Give me one canonical state snapshot per cycle, and diff it | `server_snapshot` | new | | Does the real image reach its message loop? What does it poll first? | `msgq_attach` | new | | Which entry/control forms does the firmware accept, on which fields? | `valprobe` | existing | | Run one whole handler with crafted state; first missing state is the finding | `rehost` + `state_seed` | existing | | Where would the engine read/write each launch leg? | `launch_audit` (+ `gmmu_walk`) | existing | | Walk a captured page table through a VA | `gmmu_walk` | existing | | Replay captured host traffic against the transport model | `host_traffic` | existing | | Decode/compare real-client RPC captures | `rm_capture` | existing | | Re-validate proven tools before a cycle | `contracts` / `run_contracts` | existing | | New image triage / version delta | `triage` / `version_diff` | existing | --- ## 5. Next emulation iterations (the continuation loop) Ordered by what each unblocks; each is bounded and session-recorded. 1. **libos port delivery — landed (partial):** the port-role model (`sim/libos_port.py`) now carries the doorbell wake through the external-interrupt receive contract. Measured next blockers: the `TASK_STATE_QUERY` trap state (`xcause`/`xepc` from the wait point) and scheduling the command-queue consumer task (the single-hart model runs one task). Success for this iteration = the consumer's first ring read (`addr/width/value`), not a fabricated poll. 2. **`GSP_ARGUMENTS_CACHED` / queue-init parsing**: relocate the rings to the addresses the host actually gives the firmware instead of the fixture block. 3. **CSB/TRANSCOM read-back semantics** for the loop's poll window. 4. **One served RPC**: dispatcher stub returning a real captured reply (`rm_capture` stimulus), recorded as the B5 session. 5. **MSI-X/event delivery and IRQ arming** so replies can be pushed, not only polled. 6. **Front-end/engine model — decision gate, not a default.** Reproducing the inline-descriptor install (PBDMA method execution, IQ2M) would let the emulator rank engine-side fixes offline, but the project has explicitly excluded an engine model (surface map §6; it is silicon physics, and offline engine claims are forbidden). Treat this as a gated, admitted research item only if the live lane's next tests leave an unresolved ambiguity that only a model could separate — never as a routine build. --- ## 6. Fidelity, honesty and safety rules * Emulator outputs are L0/L1: instruction and memory semantics of the firmware. No timing (L2), no device environment (L3), no silicon behaviour. * Silence is not health: a missing fault does not mean success. Prefer engine-authored observables; front-end-authored signals are not proof. * Every run keeps its session/report; a claim without a session is an anecdote. * Offline only: never a GPU, module, mdev, guest, network or capture from this tree. Injected code is development-only and stays here. * The differential loop is only as good as the captured evidence; a verdict on a partial capture names the next page (or field) to capture, not a cause. --- ## 7. Change log | Date | Change | |---|---| | 2026-10-06 | Created with the four new tools (`state_decode`, `log_decode`, `server_snapshot`, `msgq_attach`), the layer map, the run-to-the-wall procedure and the continuation loop. | | 2026-10-06 | Continuation: `sim/libos_port.py` + two-task B5 attach (extintr handoff completes; first poll still blocked on post-init resume + `GSP_ARGUMENTS_CACHED`); `state_decode` gains aggregate `key: value` state parsing (m26→m29 diff). |