debug: green-eye wave 3 - RM harness, local simulator, device map, MCP plugin, value-validation demos, CI pipeline, toolchain docs; add host RM tools rmtool/rmtop

1 files changed, 148 insertions, 8 deletions

diff --git a/debug/risc-v-gsp-rm-vm/docs/16-CUSTOM-SIMULATOR.md b/debug/risc-v-gsp-rm-vm/docs/16-CUSTOM-SIMULATOR.md
index 2b77ca3..2f99f7b 100644
--- a/debug/risc-v-gsp-rm-vm/docs/16-CUSTOM-SIMULATOR.md
+++ b/debug/risc-v-gsp-rm-vm/docs/16-CUSTOM-SIMULATOR.md
@@ -9,10 +9,116 @@ backend: a small deterministic RISC-V machine (`libgreenmachine`) implementing
 the same `Session` contract as today's QEMU backend. It expands
 `GREEN-EYE-ROADMAP.md` §H3 (H3.1–H3.4) and the §H6 soft-processor annex.
 
-**Nothing here exists today.** The only supported backend is `QemuTarget` +
-`RspClient` (`greeneye/target.py`, `greeneye/rsp.py`); native is gated behind
-H3 entry — H2.2, the QEMU device model that makes the real GSP image reach its
-message loop — and must not be started ad hoc. This document only scopes it.
+**Beyond M1.5 (§1.1), nothing here exists yet.** The only production backend is
+`QemuTarget` + `RspClient` (`greeneye/target.py`, `greeneye/rsp.py`); M2+ of
+the native backend is gated behind H3 entry — H2.2, the QEMU device model that
+makes the real GSP image reach its message loop — and must not be started ad
+hoc. This document scopes it.
+
+### 1.1 M1 status (implemented in this tree)
+
+M1 was explicitly commissioned and is implemented behind `--backend native`;
+the status marker above still applies to the remaining milestones (M2–M6) and
+to any use on real GSP images.
+
+* **What exists.** A pure-Python, stdlib-only RV64IMAC interpreter under
+  `sim/` (`sim/decode.py` decoder, `sim/machine.py` 256 MiB RAM + NS16550A
+  UART + execution loop) with an `RspClient`-shaped adapter in
+  `greeneye/sim_backend.py` (`SimClient`, `SimTarget`, `launch_native`).
+  `greeneye.session.Session` runs unchanged on it: registers by name, memory,
+  SW/HW breakpoints, watchpoints, step/continue/interrupt, `run_until`/`call`.
+* **Firmware parity.** The bundled `uproc_demo` boots unmodified; a scripted
+  hook/watch/leave scenario produces an event-for-event identical profile on
+  QEMU and native (`tests/test_sim_backend.py::DifferentialTests`), including
+  handler arguments and return values. Watchpoints stop *before* the accessing
+  instruction, which reproduces QEMU's `watch` stop and the Session
+  clear/step/re-arm dance exactly.
+* **Deviations from this design.** M1 here is the *reference* interpreter, not
+  the Rust product core of §12; no MMU, FPU, GC, snapshots/moments, map DSL,
+  scheduler, event bus or injection ABI (those are M2–M5). There is no
+  self-modifying code: the per-PC decode cache is invalidated by debugger
+  writes only.
+* **Time model.** Virtual time is instruction count: `continue_(timeout)`
+  executes at most `timeout * 1_000_000` instructions (SIM_IPS), then reports
+  a budget timeout. `Session.run(duration=...)` therefore bounds native runs
+  by virtual time, deterministically; QEMU's wall-clock execution naturally
+  reaches different heartbeat counts in the same duration (see §14).
+* **Measured performance** (one host thread, CPython 3.12, this machine):
+  ~2.3 MIPS tight RAM loop, ~1.5 MIPS sustained demo workload
+  (`examples/scripts/sim_bench.py`), ~1.25 MIPS with a hook + watchpoint
+  active. The ≥10 MIPS M1 target in §14 is a compiled-core target and is not
+  met by the Python reference; the demo self-test completes in ~2k
+  instructions, so the parity suite still runs in seconds.
+* **Stop semantics.** `ebreak` (the demo's `rpc_fault`) stops *at* the trap as
+  a SIGTRAP like the gdbstub, and `step` executes over it; `wfi`, `fence` and
+  `fence.i` are no-ops (single hart, no ordering model); unmapped accesses and
+  illegal instructions stop as SIGSEGV/SIGILL stops instead of raising.
+
+### 1.1a M1.5 status - native exec/access trace (implemented in this tree)
+
+The roadmap R4 full-trace path now exists natively, without the QEMU TCG
+plugin and without leaving the process:
+
+* **Engine.** `sim/trace.py` (`Trace`, `make_trace`) collects compact JSONL
+  lines from the execution loop in `sim/machine.py`:
+  `{"t":"exec","vcpu":0,"pc":"0x...","vt":N}` and
+  `{"t":"read"|"write","vcpu":0,"addr":"0x...","size":S,"pc":"0x...","vt":N}`,
+  ending in the plugin's `{"t":"summary",...}` line. `vt` is the machine's
+  deterministic instruction count; nothing reads the host clock.
+* **Knobs (same semantics as `qemu/greeneye_trace.c`).** `exec=on|off`
+  (default on in the CLI), `start=`/`end=` inclusive PC filters applied to the
+  executing instruction - so memory accesses are filtered by *instruction* PC
+  exactly as the plugin filters at translation time - and a ring budget
+  `max_events` after which lines are counted as `dropped` (default 1,000,000,
+  `0` = unlimited).
+* **Delivery.** The trace is written at Session close as the session blob
+  `blobs/trace.jsonl` (referenced from `session.json` as `trace_blob`, with
+  `trace` config/counts in the metadata), so it never swamps `events.jsonl`;
+  folded `exec`/`access`/`read`/`write` events in `events.jsonl` are also
+  accepted by the analyzer.
+* **Analyzer.** `greeneye analyze` renders an `## Execution trace` section:
+  exec/read/write totals, unique PCs grouped by symbol, and accessed 4 KiB
+  pages with read/write counts. `greeneye compare` adds a trace-stream totals
+  delta and includes trace writes in the written-page delta.
+* **API.** `ge.launch(..., backend="native", trace=...)` and
+  `launch_native(trace=...)` take `True`, a dict
+  `{"exec", "start", "end", "max_events"}`, or a `sim.trace.Trace`.
+* **Measured cost.** On the tight one-instruction loop microbenchmark the
+  always-present `if trace` guard costs ~7% versus an unguarded copy; on the
+  demo workload (best-of runs) untraced throughput is unchanged within noise
+  (1.53 MIPS before, 1.55 MIPS after). Tracing itself costs ~2.4x on the demo
+  workload and ~3x on the microbenchmark. `examples/scripts/sim_bench.py
+  --trace` reports both.
+
+### 1.2 Usage
+
+```bash
+./greeneye.sh demo --backend native          # the canned demo, native backend
+./greeneye.sh run examples/scripts/trace_rpc.py --backend native
+printf 'bp rpc_sum\nc 0.2\nquit\n' | ./greeneye.sh attach --backend native
+./greeneye.sh record --backend native --trace --duration 0.5
+./greeneye.sh record --backend native --trace --trace-exec off \
+    --trace-start 0x80000000 --trace-end 0x80010000
+./greeneye.sh analyze dumps/<stamp>-record   # renders the trace coverage section
+python3 examples/scripts/sim_bench.py --trace    # throughput + trace overhead
+python3 examples/scripts/sim_diff.py             # QEMU/native profile diff
+```
+
+```python
+import greeneye as ge
+s = ge.launch(kernel="firmware/uproc_demo/build/uproc_demo.elf", backend="native",
+              record="dumps/traced", trace={"exec": True, "start": 0x80000000})
+s.hook("rpc_sum", on_enter=lambda hit: print(hit.arg(0), hit.arg(1)))
+s.watch(s.symbols.lookup("gsp_ctrl") + 0x10, size=8, kind="w")
+s.run(duration=1.0)
+s.close()   # writes blobs/trace.jsonl
+```
+
+`--backend` is accepted by `demo`, `run`, `attach`, `record`, `inject` and
+`ghidra`; the default stays `qemu` and the QEMU path is behaviourally
+unchanged. `attach --port` stays QEMU-only (there is no server to attach to),
+and `--trace` is native-only (QEMU uses the TCG plugin). Everything below
+remains the design for M2+.
 
 ## 2. Why a custom simulator
 
@@ -147,6 +253,13 @@ class Backend(Protocol):
 
 Compatibility decisions:
 
+* **M1 implementation note.** Instead of the `Backend` protocol and typed
+  `Session(backend=...)` sketched below, M1 implements the narrower, already
+  exercised `RspClient` method surface (`greeneye/sim_backend.py`) and leaves
+  `Session` untouched. The `Backend` refactor is deferred until a second
+  non-RSP backend (e.g. Verilator, §16) actually needs it; §5's table is
+  still the M2+ target.
+
 * `Session(client=...)` keeps working; `Session(backend=...)` is added and
   `client=` becomes an alias for `RspBackend(client)`. `Memory` is re-typed
   against `Backend`; its public methods do not change.
@@ -291,16 +404,17 @@ single producer for stop-based and full-access events:
 |---|---|---|
 | `session_start`, `image_loaded`, `hook`, `watch`, `note`, `exit`, `session_end` | existing | unchanged |
 | `bp_hit`, `leave`, `watch_hit`, `stop`, `step`, `mem_read`, `mem_write`, `snapshot*`, `probe*`, `plugin*` | existing | unchanged; `mem_*` may now come from the bus, not only tooling |
-| `exec` | new (M1, optional) | `pc`, `insn`, `vt`, `core`, `symbol` |
-| `access` | new (M2) | `pc`, `addr`, `size`, `rw`, `region`, `vt`, `core` |
+| `exec` | native M1.5 (blob) | `pc`, `vt` (+ `symbol` when folded into an event) |
+| `access` | native M1.5 (`read`/`write` lines or folded events) | `pc`, `addr`, `size`, `vt`; `rw` only for the folded `access` form |
 | `device` | new (M2) | `device`, `name`, `rw`, `value`, `vt` |
 | `irq` / `schedule` | new (M3) | `line`, `state`, `core`, `vt`, `reason` |
 | `inject_*` | existing shape (docs/12) | `service_injected`, `call_enter`, `call_return` unchanged |
 
 `t` stays the recorder's monotonic seconds **filled from virtual time**, so
 the same run yields the same file. `exec`/`access` volume is controlled by
-profile logging and a ring budget; the analyzer gains coverage/heatmap
-rendering (roadmap R4). `greeneye compare` treats backend identity, `t` and
+profile logging and a ring budget; for the native backend that is implemented
+(M1.5, §1.1a): the stream is a session blob and the analyzer renders coverage
+and accessed pages. `greeneye compare` treats backend identity, `t` and
 host paths as non-semantic, so cross-backend comparison is first-class.
 
 ## 11. Injection ABI
@@ -352,12 +466,20 @@ familiar with the codebase; milestones are sequential.
 | # | Scope | Exit criteria | Effort |
 |---|---|---|---|
 | M1 | RV64IMAC interpreter, RAM, NS16550A-compatible UART, loader; `Session` adapter with `Backend` | bundled demo uproc boots unmodified; the canned instrumentation script (`03 §12`) produces the same hook/watch/stop timeline and memory snapshots as QEMU (canonical comparison); determinism test green | 4–6 wk |
+| M1.5 | Native exec/access trace (R4 equivalent): `--trace`, filters, blob + analyzer coverage | trace determinism test green; analyzer renders unique PCs/pages; non-trace path unregressed (`sim_bench.py --trace`) | 1–2 wk |
 | M2 | Memory-map DSL + `falcon`/`csb`/`ptimer` devices + profile validation; native region snapshots; moments v1 | a re-hosted uproc build boots from a profile with no core changes; moment save/restore round-trips; restoring reproduces the event-stream hash | 4–6 wk |
 | M3 | Deterministic scheduler (budgets, timers, IRQ events) + event log/replay | multi-task test profile replays identically from the log on two hosts; tie-break and replay tests in CI | 3–4 wk |
 | M4 | Device-model parity with H2.2 + `dma`/`pmu_stub` + boot shims | real GSP image reaches the same stable state as the QEMU H2 exit under the same profile; differential suite passes for probe and boot fixtures; L3 parity documented, L2 still not claimed | 6–10 wk |
 | M5 | Injection ABI + service registry + `log`/`stub`/`forward` interception | an SDK v1 blob loads and serves a control end-to-end; each interception mode has a recorded acceptance session; fuzzing seed/restore demonstrated | 4–6 wk |
 | M6 | Performance/ergonomics: profiling, fast paths, `--backend native`, docs + flag-removal criteria | §14 targets met; `greeneye demo --backend native` in the differential CI job; user docs updated | 3–4 wk |
 
+> **M1 status: done (pure-Python reference implementation).** Ported without
+> the `Backend` refactor (see §5 note); differential parity and a determinism
+> test are in `tests/test_sim_backend.py`; performance is §1.1. **M1.5 (native
+> trace) is done** in §1.1a: `sim/trace.py`, `tests/test_sim_trace.py`,
+> analyzer coverage rendering and `examples/scripts/sim_diff.py`. M2–M6 above
+> are unchanged.
+
 Total: roughly 24–36 engineering-weeks, consistent with the roadmap's
 "months; explicitly later" classification; none of it starts before the H3
 entry gate, and if H2 slips this plan waits.
@@ -389,6 +511,24 @@ harness, trended in CI):
 | MMIO access vs RAM access | ≤5× | M2 |
 | Event bus throughput | ≥1 M events/s buffered, lossless under budget | M3 |
 
+> **M1 measured (pure-Python reference).** 2.3 MIPS tight RAM loop, 1.5 MIPS
+> sustained demo workload, 1.25 MIPS with a hook + watchpoint active
+> (`examples/scripts/sim_bench.py`, CPython 3.12). The ≥10 MIPS M1 row is a
+> compiled-core gate and remains open for the Rust core; the Python reference
+> exists to pin semantics and differentially test the firmware, which it does
+> at parity. QEMU executes comparatively more instructions per wall second on
+> this host, so equal wall-clock `run` durations produce different heartbeat
+> counts across backends — the hook/watch/leave event profile is identical.
+>
+> **M1.5 trace overhead (same host).** The tracing guard is a single local
+> boolean test per instruction plus one per memory access; A/B against an
+> otherwise identical loop copy: ~2.4 MIPS vs ~2.6 MIPS on a pathological
+> one-instruction loop (~7%), while the demo workload is unchanged within
+> run-to-run noise (1.53 vs 1.55 MIPS best-of). With `exec=on` the demo
+> workload runs at ~0.54 MIPS (~2.4x slower); an access-only trace is
+> proportionally cheaper. Use `--trace-start/--trace-end` to bound the logged
+> region, and the default 1M-event ring budget to bound memory.
+
 ## 15. CI strategy
 
 * The Python package and QEMU path remain the default, always-on suite; no