path: root/debug/risc-v-gsp-rm-vm/docs/35-GREENEYE-SIMULATOR-DESIGN.md

# 35 — GreenEye simulator design (single design document)

> **Status:** in development (2026-10-03). Phase S (standalone GSP simulator)
> is the only implementation scope until its pause gate; Phase G (GPGPU-Sim
> board model) is designed and gated, not started.
>
> **Audience:** agents and maintainers of risc-v-gsp-rm-debugger.
>

---

## 0. What this document is

One consolidated design for the simulator program requested for
risc-v-gsp-rm-debugger. It covers, in order of execution:

1. **Phase S — standalone GSP simulator** (now): execution of the real
   firmware under the modelled, evidence-labelled device environment (L0/L1;
   silicon accuracy is **not** claimed), full tool/workflow integration, and
   the above-the-QEMU-line contract on the frozen checks (docs/37 §3).
   **The program pauses at Phase S exit** before any GPGPU-Sim work.
2. **Phase G — GPGPU-Sim full-board model** (later, gated): TU104 board
   devices (GMMU, FBIF, FALCON→GMMU DMA, vidmem heap, VBIOS), GSP↔core
   interactions, PCIe/VFIO bridge for a vGSP VF, QEMU-gdb time sync, MCP
   extension, multi-GPU/fabric backlog.
3. **Plan B** — the narrowly scoped reference plan for "a simulator simply
   better than the QEMU baseline for the open-iov workflow"
   (`../Handoff-Simulator-Plan-B.md`, private), with an explicit Plan A/Plan B
   comparison and decision procedure.

Owner directives are numbered U1–U45 in the private handoff (§1 and §13);
this document restates the operative ones and never contradicts them.

## 1. Definitions

### 1.1 The QEMU line (the bar to beat)

The **QEMU baseline** is the v0.1/v0.2 tool behaviour measured on the real
GSP image: RSP transport, hooks/breakpoints/watchpoints, memory/struct/
snapshot tooling, sessions, symbols/analyze/compare, probes/plugins, Ghidra
symbol + control bridges, TCG trace, injection SDK, `Session.call`/
`run_until`; and on the real `gsp-570.144.fwimage.bin`: relocation-free flat
load, first exception at firmware vaddr `0x4009014` (load access fault
reading `0x2000000000000A00`), then `mtvec=0` fetch loop. Planned QEMU
features (device model H2.2, boot shims H2.3, deterministic multi-core H2.4,
RM service injection H1.3, trace→session folding R4, debugger parity H1.5)
are **part of the bar** even though QEMU has not implemented them.

**"Above the QEMU line"** means, at minimum:

* executes the real 570.144 image deterministically to at least one recorded
  milestone **past** the QEMU first fault (ladder in §3.3), with the device
  model and shims documented;
* every QEMU feature available against the native backend, plus the planned
  QEMU features above;
* every run produces `greeneye-session/1` evidence with simulator build hash,
  firmware hash, profile/hash, shim assumptions and fidelity labels;
* run-to-run determinism proven; cycle counts emitted and labelled *modeled*;
* a real-workload demonstration whose eng-ops baseline/threshold were frozen
  before the run.

### 1.2 Fidelity labels

`L0` instruction semantics, `L1` memory behaviour, `L2` time/concurrency,
`L3` device environment (extended from docs/06). The standalone GSP
simulator targets **L0–L2 for the GSP core and its device model**, with L2
explicitly labelled *modeled* (table/source-derived) until calibrated. The
GPGPU-Sim board model adds board-level L3. No silicon-accuracy claim without
calibration evidence and a declared uncertainty band.

### 1.3 Accuracy metrics

* `M1–M6` — the existing metric contract (inventory, replies, ordering, state,
  error paths, memory side effects; docs/06).
* `C1–C6` — cycle calibration (instruction latency agreement, memory latency,
  bandwidth, kernel correlation, determinism, clock-domain ratio); bands are
  frozen in the private handoff §9.2 and must be recorded before first use.

## 2. The workflow this exists to serve

The simulator must move one of the project's OODA loops faster with evidence
(eng-ops §3): **dynamic RE** (what does the firmware do?), **portability**
(new firmware/device triage), **adversarial** (unexpected behaviour), and
**correctness** (does our emulation match?). Time-to-value is the currency:
every capability below names the baseline it replaces and the number that
decides keep/redesign/retire.

## 3. Phase S — standalone GSP simulator (NOW)

### 3.1 Architecture

```
greeneye control plane (Python; unchanged public API)
  Session · hooks · bp/watch · probes · plugins · recorder · analyze/compare
  Ghidra bridge · MCP · injection SDK
        │ Backend ABI (typed; QEMU and native are peers)
        ▼
native backend (sim/)
  machine.py       RV64IMAC core, deterministic virtual time, cycle model
  devices/         NEW: GSP device model (falcon/SEC2 windows, CSB/TRANSCOM,
                   WPR2/FWSEC state, PTIMER, IRQ routing, DMA/IOMMU, PMU stub)
  profiles/        NEW: versioned device profiles (JSON) + boot shims
  trace.py         exec/access/cycle events → session blob
  moments (S2)     snapshot/restore + deterministic replay
```

*The QEMU backend stays untouched and remains the differential reference.*
Backend selection is explicit (`--backend qemu|native`); no shared mutable
state; the session format evolves additively (`greeneye-session/1` + new
event classes `cycle`, `device`, `mcu`).

### 3.2 Work packages

| WP | Name | Deliverable | Agent |
|---|---|---|---|
| S0 | baseline (done) | native core, session parity, M1.5 trace, modeled cycles, real-image bounded run | — |
| S1 | GSP device model + boot ladder | `sim/devices/` + `sim/profiles/` + milestone sessions + tests | S1 |
| S2 | debugger parity | catchpoints, core/thread selection, memory-region metadata, moments/replay, parity matrix | S2 |
| S3 | MCP + scripting + injection | MCP tools against native backend; injection against the simulated GSP; docs | S3 |
| S4 | real-workload validation | boot-ladder A/B vs QEMU probe; scenario harness; value demo + report | S4 |
| S5 | stability/packaging | long-run soak, error taxonomy, session size budgets, quickstart, release notes | successor |

### 3.3 Boot ladder (S1 acceptance)

Each rung is a recorded session with a session hash; each rung must be
reproducible twice identically and have its device requirements listed.

| Rung | Milestone | Current status |
|---|---|---|
| B0 | flat image loads; first MMIO observed at `0x2000000000000A00` | done (native, lenient) |
| B1 | device model serves the whole boot path: 0 unknown MMIO, 0 unhandled in windows; deterministic 2M-instruction run stopping at the boot-halt/report loop (`0x4010928`) | done (S1 device model; sessions `dumps/gsp-boot-b1-run1/2`) |
| B2 | boot stage progress: vector table/TCM/MPU/trap handling exercised with the shim; passes the bootargs consumer and reaches the relocation copy | done (S1 device model + XMPU model + bootstrap-args shim; reaches B3) |
| B3 | libos init completes (TCM registration, pagetable registration, DMA flush, external-interrupt arming) and the message-loop task blocks in the port wait | done (sessions `dumps/gsp-boot-b3-run1/2`: 474 043 insns, stop `ecall` at `0x4018038`, two-run identical) |
| B4 | first CSB/TRANSCOM mailbox poll served with correct read-back semantics (host-traffic-triggered doorbell) | target |
| B5 | one scripted host-side RM message served end-to-end with a recorded reply | target (gated by S3 injection) |

**Ladder revision (2026-10-03, B3).** The original B3 wording assumed the
firmware polls CSB/TRANSCOM during a silent boot. The 570.144 image does not:
after arming interrupts and completing its libos services it blocks in the
message-loop port wait, and the CSB/TRANSCOM poll is *triggered by host
traffic*. B3 is therefore anchored on the observed wait (a real, reproducible
milestone) and the CSB poll moved to B4, where a scripted doorbell will
produce it. The pre-revision text is kept in history; this note is the
operative definition.

The ladder is deliberately versioned-data driven: a new firmware version is a
new profile + shim record, not a code fork.

### 3.4 What is explicitly not in Phase S

GPGPU-Sim changes, board devices beyond the GSP complex, guest/PCIe models,
VBIOS, VF bridge, Ampere/Blackwell, fabric — all Phase G or backlog (§10).

### 3.5 Pause gate

Phase S ends when: S1–S4 exits met; suite green incl. new tests; a real
workload demo exists with a frozen baseline and measured time-to-value;
`docs/30–40` updated; adversarial review (§9) signed off. **Pause and report
before any Phase G work.**

## 4. Phase G — GPGPU-Sim board model (later, gated)

This is the previously requested GPGPU-Sim program, now explicitly sequenced
after Phase S. It is *designed* now so Phase S artifacts (device map,
profiles, session/event classes, metrics, MCP surface) fit it without rework.

### 4.1 Board device inventory (TU104 family)

Expanded from owner requirements (GMMU, FBIF, FALCON→GMMU DMA, vidmem heap)
and the porting dossier.

| Device | Model target | Evidence class |
|---|---|---|
| GMMU (+ PTEs, page tables, ATS/IOMMU seam) | page-table walk, TLB, fault reporting, VA→PA translation for firmware descriptors | register/field maps + formats; A/B vs probe |
| FBIF (FB interface, compression, ECC) | FB access path, alignment/partition hashing, vidmem heap model | register maps; documented assumptions |
| FALCON→GMMU DMA line | descriptor-driven DMA with cycle/latency accounting | boot/mainline firmware usage; source traces |
| vidmem heap stack | allocator map, reserved regions, WPR2 carve-outs | boot state + shims |
| GSP core + CSB/TRANSCOM | message queues, doorbells, interrupts | Phase S artifacts |
| PMU / SEC2 / booter / ACR/SOE | interfaces, mailboxes, state machines, latencies | Phase S MCU models |
| VBIOS/PROM | image layout, init tables, device declaration for guests | public parsing + profile |
| PCIe endpoint | config space, BARs/MMIO windows, MSI-X, TLPs, D2D | descriptor + QEMU/GDB observations |
| VF bridge | vGSP VF traffic → simulator BAR/MMIO instead of real GPU | design-only until human approval |

### 4.2 Light-touch VF bridge (design)

* Userspace gpgpu-sim hosts the board model; a future (separately approved)
  kernel module presents a PCIe/VFIO device FD.
* Protocol: shared-memory queue + eventfd doorbells; simulator services
  MMIO/BAR reads/writes, MSI-X injection and DMA windows as messages; no
  changes to `vgsp` primitives or data paths.
* **Hard rule:** no `vgsp` kernel-module edits without explicit human
  approval. Phase S/G implement everything up to that boundary; the module is
  design + tests on stubs only.
* A conformance harness replays recorded QEMU/VFIO traffic against the
  simulator model and compares replies (accuracy loop), so the bridge is
  proven before any kernel work is proposed.

### 4.3 QEMU-gdb time sync (design)

Guest QEMU and the simulator exchange a clock-sync protocol (periodic
time-base beacons + event ordering barriers). Deterministic mode: simulator
virtual time is the master; QEMU is granted step/sync credits. Never a
security boundary; no confidentiality claims.

### 4.4 MCP extension (design)

The existing 31-tool MCP surface grows simulator-native tools: run/step with
budget, cycle queries, device-state dumps, MCU state, boot-ladder report,
board topology, DMA/TLP traces. Read/observe default; writes/step gated by
flags; every call attributable to a session.

## 5. Feature parity matrix

| Capability | QEMU (current) | QEMU (planned) | Native GSP (now) | Native GSP target |
|---|---|---|---|---|
| RSP-free transports/launch | yes | — | launch/attach | yes |
| Hooks/watchpoints/breakpoints | yes | catchpoints/core select | hooks/watch/bp | full parity (S2) |
| Memory/struct/snapshots | yes | region metadata | memory/struct | snapshots/moments (S2) |
| Sessions/analyze/compare | yes | trace folding | yes + cycles | trace folded, coverage (S2) |
| Probe/record/plugins | yes | — | backend peer | yes |
| Ghidra symbol/control bridges | yes | watch/heat-map/nav | protocol-compatible | full (S2/S3) |
| Injection SDK | demo | RM-core interception H1.3 | demo backend | services vs GSP model (S3) |
| Device model / real boot | no (faults) | H2.2/H2.3/H2.4 planned | ladder B0–B3 (S1, device model + XMPU + libos services) | B4/B5 (S1/S3) |
| Determinism/replay | wall-clock | H2.4 budgets | instruction-count | event replay (S2) |
| MCP | yes (QEMU) | — | native sim tools + backend selection (20) | wider GSP-model tools (S3) |
| Tiering | — | — | modeled cycles | calibrated bands (S4) |

## 6. Boundaries and invariants

1. All changes live under `debug/risc-v-gsp-rm-vm/` (private handoff copies at
   workspace root excepted). Never edit project directories outside it.
2. **No `vgsp` kernel-module edits without explicit human approval** (U33).
3. No physical GPU, mdev, guest, module loads, `/dev` or reboots from agents;
   RM 610 is a no-fly zone (docs/23); no unattended cycles (RUNBOOK P1–P9).
4. The simulator is a study environment, never a product dependency, never in
   a guest, no security claims (RUNBOOK P5/P6).
5. Upstream GPGPU-Sim stays a pinned read-only checkout; our changes are
   tracked patches + overlay (Phase G), never a silent fork.
6. The QEMU path must remain green while native features land (differential
   suite is the regression net).

## 7. eng-ops admissions

`gpgpu-sim/ADMISSION.md` holds draft records TOOL-2026-SIM-01..05 (build/run,
TU104 calibration, GSP-in-sim, MCU model, debugger backend) with the five
required fields; Phase S adds one admission record per S work package (S1
device model, S2 parity, S3 MCP/injection, S4 validation). Thresholds are
frozen before the first run and never moved silently; every result carries a
session directory and hashes.

## 8. Real-workload proof (S4)

The "real workload" for Phase S is the open-iov workflow on the real image:
boot the 570.144 GSP to the highest ladder rung, expose device/MCU state
through the standard tooling (CLI/Python/MCP/Ghidra), serve one scripted
message, and record sessions. Baselines: (a) QEMU `probe` first fault;
(b) static analysis time estimates from the project handoff. The demo is
accepted only if it beats the frozen threshold; otherwise the failure signal
triggers redesign per eng-ops §4.

## 9. Adversarial review

| Risk | Failure signal | Mitigation |
|---|---|---|
| "Accuracy" theatre (lenient MMIO masquerades as a model) | rung claimed with lenient policy | each rung requires explicit device semantics; lenient runs labelled |
| Device model guesses without evidence | unexplained delta in A/B | every register backed by source or observed firmware use; record assumption |
| Parity breaks the QEMU path | differential suite red | QEMU untouched; feature-flag additions |
| Scope creep into Phase G | files outside `sim/`+docs | gate checks; private handoff scope contract |
| Non-determinism | replay delta non-empty | virtual-time only; double-run tests |
| False "above QEMU" claim | missing A/B session | acceptance requires session pair + hashes |
| Kernel-module temptation | any `open-iov/kernel/` diff | hard rule U33; reviews |
| Time-to-value negative | measured baseline worse | per-WP admission thresholds |

## 10. Plan B comparison plan

`Handoff-Simulator-Plan-B.md` (private) defines the narrow fallback: the
smallest simulator that is simply better than QEMU for the open-iov workflow.
At Phase S exit (and earlier if a work package fails twice), compare Plan A
output vs the Plan B design on: rung reached, feature coverage, determinism,
cycle evidence, integration effort, maintenance cost, and measured
time-to-value. Decision: **keep Plan A**, **apply Plan B lessons in place**,
or **start the Plan B implementation** (with a migration note and re-pilot).
The comparison and decision are recorded in the private handoff.

## 11. Status and wave log

| Wave | Date | Result |
|---|---|---|
| Recon + master handoff | 2026-10-03 | master handoff; source register |
| W1 GPGPU-Sim build | 2026-10-03 | CUDA 12 port patches; `vectorAdd` functional+perf green |
| W1 TU104 config/overlay | 2026-10-03 | config scaffold + sanity 26/26 + admissions draft |
| W1 GSP cycle-0 | 2026-10-03 | modeled cycles; real image 200k instr deterministic; 298 tests green (2026-10-03 suite run; count varies with prerequisites) |
| W1 dossier | 2026-10-03 | Turing/Ampere/Blackwell parameter docs + cross-generation survey |
| W2 Phase S (S1 + S4) | 2026-10-03 | `sim/devices` + `sim/profiles` device model: B0–B2 on the real image (0 unknown MMIO), 2-run determinism; A/B harness discovers the profile and records verdict `above` on the frozen checks (docs/36, docs/37 §4.3, docs/40) |
| W2 Phase S (4c: XMPU + libos) | 2026-10-03 | `sim/mpu.py` + `sim/libos.py` + bootstrap-args shim: B3 on the real image (474 043 instr, message-loop port wait, 0 unknown MMIO, 2-run identical); A/B re-run at `--instructions 2000000` verdict `above` (docs/37 §4.4); 417 tests green |
| W2 Phase S | launched | S2 parity, S3 MCP/injection, S4 validation |

Backlog (recorded, not started): Ampere/Blackwell models; fabric (PCIe,
NVLink/C2C, RDMA/NIC/switch/cable/transceiver, NVMe TLPs, bounce buffers,
DRAM tiers, rendezvous hashing); 8-GPU chassis; H100 SuperPOD RA; QEMU-gdb
time sync; VBIOS; VF bridge module; Verilog/FPGA; multi-version sweeps.