diff options
Diffstat (limited to 'debug/risc-v-gsp-rm-vm/docs/15-AGENTIC-GSP-IDE.md')
| -rw-r--r-- | debug/risc-v-gsp-rm-vm/docs/15-AGENTIC-GSP-IDE.md | 387 |
1 files changed, 387 insertions, 0 deletions
diff --git a/debug/risc-v-gsp-rm-vm/docs/15-AGENTIC-GSP-IDE.md b/debug/risc-v-gsp-rm-vm/docs/15-AGENTIC-GSP-IDE.md new file mode 100644 index 0000000..79d17b8 --- /dev/null +++ b/debug/risc-v-gsp-rm-vm/docs/15-AGENTIC-GSP-IDE.md @@ -0,0 +1,387 @@ +# 15 — green-eye as an agentic integrated development, analysis and debugging environment for the GPU System Processor + +> **# TOOL IN DEVELOPMENT - NOT READY YET!** See 00 for status. + +## 1. Vision and what "agentic" means here + +green-eye already has the two properties that make an agent loop possible: +a **deterministic, scriptable harness** (`ge.launch`/`ge.attach`, `Session`, +`record`, `compare`) and a **durable evidence format** (the session directory). +"Agentic" here means: + +* AI agents drive the deterministic harness — they choose instrumentation, + run stimuli, compare sessions and write findings; +* sessions are the shared state between agents, humans and Ghidra, so every + claim is inspectable after the fact; +* **humans keep product-change authority**: an agent may propose an + `open-iov-vfio` change with evidence, but only a human merges it, and the + physical-confirmation rules of docs/06 and docs/07 still apply. + +Non-goals: autonomous hardware cycles, unattended reboots, agents that "fix" +the emulation directly, or replacing the engineer reading the decompiler. +This document describes the environment that exists today plus the roadmap +hooks that would make a full agent platform real. + +## 2. Architecture + +``` + +-------------------- agents (triage / hypothesis / RE / portability) ----+ + | | + | CLI: ./greeneye.sh info|demo|run|attach|launch|probe|record|analyze| | + | compare|symbols|plugins|inject|ghidra | + | Python: ge.launch/attach, Session.*, memory.*, symbols.*, structs, | + | probes, plugins, ghidra.export_hits, inject.* | + | REPL: bp/hook/watch/call/x/snap/events/note | + | Ghidra bridge: JSON lines over localhost (docs/13) | + | MCP: planned plugin only (roadmap R3) — does not exist in v0.1 | + +----------------------------------+---------------------------------------+ + v + session directory (events.jsonl + session.json + snapshots + summary.md) +``` + +### 2.1 The session directory is the shared state + +| Artifact | Content | Consumers | +|---|---|---| +| `events.jsonl` | append-only event stream (`bp_hit`, `watch_hit`, `mem_write`, `call_enter/return`, `note`, `service_injected`, …) | `analyze`, `compare`, `export_hits`, plugins | +| `session.json` | format, name, counts, snapshot inventory, meta | report tooling, provenance | +| `snapshots/*.bin.z` | zlib-compressed region captures with sha256 | offline diff, field decode | +| `blobs/*` | plugin/probe dumps (e.g. qemu.log) | forensics | +| `summary.md` | rendered report | humans, PR attachments | + +`greeneye compare dumps/<a> dumps/<b> --symbols` is the primary agent +primitive: it turns two sessions into hit/watch/write deltas. Format +`greeneye-session/1` evolves additively (roadmap V2). + +### 2.2 Surfaces an agent may use + +* `./greeneye.sh …` — the stable CLI (all real subcommands listed above). +* `./greeneye.sh run script.py` — the preloaded Python namespace (`ge`, + `session`/`s`, `memory`, `symbols`, `recorder`) makes a harness file the + unit of automation. +* `./greeneye.sh ghidra` + `GreenEyeControl.java` — JSON-lines bridge for the + decompiler loop (`run_until`, `bp_set`, `read_mem`, `call`, `hits`, + `export_hits`; protocol in docs/13 §3). +* Plugins — the extension point (docs/04). A plugin's `register(ge)` can add + instrumentation, CLI commands (`ge.add_command`) and recorded events; plugin + output lands in the same `events.jsonl`. +* **MCP is planned, not present.** Roadmap R3/H7 reserves an MCP plugin that + would expose sessions as local stdio tools. Until that rung is scheduled, do + not add MCP dependencies to the core package (docs/04 §3d). + +### 2.3 Why the evidence model matters for agents + +A human can hold context across steps; an agent cannot. The session directory +makes the context external: an agent can be restarted, handed a directory, and +re-derive the same conclusion without replaying its own reasoning. This is the +difference between an agent that reports opinions and one that produces +reviewable evidence. + +## 3. Agent roles and workflows + +| Role | Input | Tools | Output | Stop condition | +|---|---|---|---|---| +| Triage | question + firmware version(s) | `symbols export`, `probe`, `analyze` | delta candidates, prioritised symbol list | every changed symbol classified | +| Hypothesis | one behaviour question | `run` scripts, hooks, watchpoints, `compare` | session pair + verdict | delta attributed to one variable | +| RE | decompiled function | `greeneye ghidra`, `run_until`, breakpoints, `export_hits` | hits TSV + annotation, reachability answer | function hit or timeout recorded | +| Portability | version/device pair | docs/07 playbook, baseline + candidate sessions, M1–M6 | portability report | metric table complete | +| Injection | service idea | `sdk/`, `inject`, `load_service`, `invoke_remote`, `invoke_via_firmware` | injected-service session pair | direct and firmware paths agree | +| Verification | another agent's finding | golden sessions, `compare`, `analyze` | pass/fail + counterexample | finding reproduced or refuted | + +Role separation is deliberate: the agent that made a claim should not be the +only one that checks it. Verification requires re-running the original +stimulus, not re-reading the original report. + +## 4. Runbooks + +Each runbook is written so a human or an agent can execute it start to finish, +and each ends with the evidence pack from docs/14 §6. + +### 4.1 Answer an emulation reply question + +Question template: "For control `<X>`, does the firmware set field `<Y>` of +the reply/object, and in what order relative to `<Z>`?" + +1. Frame it as A/B: the emulation's reply (reference capture, docs/07 P4) and + the firmware's reply in green-eye. One session alone cannot answer it. +2. Resolve the handler and object from the symbol DB; never hardcode: + + ```bash + ./greeneye.sh symbols resolve --db symbols-<ver>.json <handler-or-addr> + ``` + +3. Capture the firmware side with a hook + before/after snapshots: + + ```python + # reply_q.py -- ./greeneye.sh run reply_q.py --kernel <rehosted.elf> + import greeneye as ge + from greeneye import Memory + + s = ge.launch(kernel="<rehosted.elf>", record=True, name="reply-q") + try: + ctrl = s.symbols.lookup("<control-symbol>") # demo: gsp_ctrl + s.hook("<handler-symbol>", + on_enter=lambda h: s.note("enter", a0=h.arg(0)), + on_leave=lambda h: s.note("leave", retval=h.retval)) + before = s.snapshot(ctrl, 0x100, "before") + s.memory.write_u64(ctrl + 0x20, 1) # stimulus + s.run(duration=0.5) + after = s.snapshot(ctrl, 0x100, "after") + print(Memory.render_diff(Memory.diff(before, after, base=ctrl), s.symbols)) + finally: + s.close() + ``` + +4. Decode the captured bytes with the structure schema so the answer is + field-by-field, not byte noise: + + ```python + import json + from greeneye.structs import schema_from_json, decode, render + schema = schema_from_json(json.load(open("firmware/uproc_demo/gsp_ctrl.schema.json"))) + values = decode(schema, s.memory.read(ctrl, schema.size), base=ctrl) + print(render(schema, values)) + ``` + +5. Compare against the reference capture with + `greeneye.analyze.compare_sessions(...)` / `./greeneye.sh compare …` for + ordering (M3) and state the field diff in words (M2). If the reply is in a + different object, repeat step 3 on that object's address. +6. Report: question, session pair, `compare` output, M2/M3 rows, fidelity + statement, explicit "cannot answer because L3 absent" where applicable. + +### 4.2 Find why a handler disappeared + +1. Build/refresh the static inventory for both firmware versions: + + ```bash + $GHIDRA/support/analyzeHeadless <proj> <name> -process <program> -noanalysis \ + -scriptPath debug/risc-v-gsp-rm-vm/ghidra \ + -postScript ExportForGreenEye.java /tmp/symbols-<ver>.tsv + ./greeneye.sh symbols export --index /tmp/symbols-<ver>.tsv --bias 0 -o symbols-<ver>.json + diff <(cut -f2 /tmp/symbols-old.tsv | sort) <(cut -f2 /tmp/symbols-new.tsv | sort) + ``` + +2. Probe the new image for early structure changes: + + ```bash + ./greeneye.sh probe <new.fwimage> --symbols symbols-<ver>.json --record + ``` + +3. Record the same stimulus against each version. Hook the old handler name; + a missing symbol is itself evidence — catch `SymbolNotFound` and log it + rather than aborting the run. Also hook the dispatcher and the old + handler's neighbours from the inventory diff. +4. Compare the sessions: + + ```bash + ./greeneye.sh compare dumps/<old-run> dumps/<new-run> --symbols + ./greeneye.sh analyze dumps/<new-run> --symbols symbols-<ver>.json --out new.md + ``` + +5. Classify the disappearance with the table below, then confirm in Ghidra: + + | Observation | Likely cause | Next step | + |---|---|---| + | name absent from new symbol DB | removed or renamed before build | match by size/callgraph in Ghidra | + | old function exists, dispatcher hits moved to another symbol | merged or re-routed | `run_until` the new dispatcher entry, `bp_set` there | + | old function exists, zero hits under the standard stimulus | coverage gap or decoy | widen stimulus, check xrefs from the dispatch table | + | dispatch table entry changed but code identical | behaviour moved (state/input) | inspect the table with bridge `read_mem`/snapshot | + +6. With the bridge running (`./greeneye.sh ghidra --kernel <elf>`), execute + until the dispatch site from the decompiler and set breakpoints: + + ```bash + $GHIDRA/support/analyzeHeadless <proj> <name> -process <program> -noanalysis \ + -scriptPath debug/risc-v-gsp-rm-vm/ghidra \ + -postScript GreenEyeControl.java 7000 0x<dispatch-address> + ``` + +7. Export hits and annotate the old function's fate: + + ```bash + ./greeneye.sh analyze dumps/<new-run> --symbols symbols-<ver>.json --out new.md + python3 - <<'PY' + from greeneye.ghidra import export_hits + from greeneye.symbols import SymbolDB + db = SymbolDB(); db.load_json("symbols-<ver>.json") + print(export_hits("dumps/<new-run>", "greeneye_hits.tsv", symbols=db)) + PY + ``` + +8. Report: static delta, dynamic delta, classification, and whether the + emulation must change (with the docs/14 evidence pack). + +### 4.3 Prototype a service + +1. Build the SDK reference service and prove position independence: + + ```bash + make -C sdk + make -C sdk relocs # must print "no relocations in this file" + ``` + +2. Inject and invoke through both paths (direct debugger call, then firmware + mailbox dispatch): + + ```bash + ./greeneye.sh inject --service sdk/build/injected_demo.elf \ + --cmd 1 --arg0 0x41 --arg1 0x42 + ``` + +3. Iterate programmatically when the CLI shape is not enough: + + ```python + from greeneye.inject import load_service, invoke_remote, invoke_via_firmware + svc = load_service(s, "sdk/build/injected_demo.elf", slot=6) + print(hex(invoke_remote(s, svc, 2, 0x10, 0x20))) + s.run(duration=0.3) + print(hex(invoke_via_firmware(s, svc, 2, 0x10, 0x20))) + ``` + +4. Instrument the service like firmware: hooks on the entry, watchpoints on + its data, snapshots before/after each command (docs/11 §7). +5. Compare two versions of the service to see the behavioural delta: + + ```bash + ./greeneye.sh compare dumps/<svc-v1> dumps/<svc-v2> --symbols + ``` + +6. Stop condition: direct and firmware-dispatched results agree, the session + is recorded, and the prototype is documented as development-only. Product + code goes to `open-iov-vfio` only through the accuracy loop (docs/11 §8, + docs/12 §9); GSP-RM registration is roadmap H1.3. + +### 4.4 Prepare a portability report + +1. Follow the P1 loop in docs/07-PORTABILITY-PLAYBOOKS.md with real commands: + + ```bash + ./greeneye.sh probe <old.fwimage> --symbols symbols-old.json --record + ./greeneye.sh probe <new.fwimage> --symbols symbols-new.json --record + ./greeneye.sh record --kernel <rehosted-old.elf> --duration 2.0 \ + --watch gsp_ctrl+0x10:8 --snapshot gsp_ctrl:392 + ./greeneye.sh record --kernel <rehosted-new.elf> --duration 2.0 \ + --watch gsp_ctrl+0x10:8 --snapshot gsp_ctrl:392 + ./greeneye.sh compare dumps/<old-run> dumps/<new-run> --symbols + ``` + +2. Keep the stimulus identical across versions: use one script/probe spec and + record it in the pack (docs/14 §4.1). +3. Fill the M1–M6 table from docs/06-ACCURACY.md. M7 (timing) is never + claimed; L2/L3 statements are mandatory. +4. Produce the report with the docs/14 §6 template and attach session hashes, + symbol DB hashes and the Ghidra export provenance. +5. Verification agent step: replay the candidate stimulus against the checked-in + golden session (roadmap V1) and require `compare` to match expectations. + +## 5. Guardrails + +Hard rules (no agent may override them): + +* **Never claim cycles, time or reboots from the simulator.** L2 is not + modelled; L3 is absent until H2 (docs/06). +* **No guest modifications** and no instrumentation of a guest driver; the + tool is never shipped and never present in a guest (docs/00, V4). +* **Sessions are mandatory evidence.** A finding without a session directory + and a `compare` report is not a finding (docs/11 §8). +* **No unattended hardware.** The agent runs against the simulator by default; + physical GPU cycles are supervised and human-driven (H7). +* **Human approval for product changes.** Agents produce proposals and + evidence packs; only humans merge `open-iov-vfio` changes. +* **Budgets everywhere.** Every run uses `duration=`/`max_stops=`/`timeout=` + and a wall-clock cap; a wedged target leaves the session stopped and is + restarted clean rather than poked. + +Capability manifest for actions (proposed file for the harness; see §7): + +| Class | Examples | Who runs it | Preconditions | +|---|---|---|---| +| Read | `info`, `plugins`, `symbols resolve`, `analyze`, `compare` | agent, unattended | path/DB provenance recorded | +| Observe | `probe`, `record`, `demo`, hooks/watch/step | agent, unattended on simulator | timeout, session recording on | +| Write (target) | `memory.write*`, `call`, `run_until`, `inject`, mailbox stimulus | agent with manifest | declared address/range, fresh session, before/after snapshots, mutation values noted | +| Product | `open-iov-vfio` patch | human only | M1–M6 + physical confirmation rules (docs/07 P4.5) | +| Hardware | physical cycles/reboots | human only | supervised, never unattended | + +A write action's manifest entry names the session, the target range, the +purpose and the budget; the session records the action, so the manifest and +the evidence can be cross-checked. + +## 6. Evaluation + +An agent workflow is judged on the same axis as the human one: how long until +a reviewable answer, and does the answer survive verification. + +| Metric | Definition | Instrument | +|---|---|---| +| TTFA | question → first evidence-backed answer | wall clock; session count | +| TTEP | question → complete evidence pack (docs/14 §6) | pack checklist | +| Evidence yield | sessions/`compare` reports per accepted finding | `dumps/` inventory | +| Verification pass rate | findings reproduced by the verification role | replay runs | +| Coverage | inventory observed vs emulated (M1) | docs/06 table | +| Accuracy trend | M1–M6 per release | docs/06 tables | +| Regression lead time | firmware drop → delta report | probe/compare timestamps | +| Human minutes | manual minutes per accepted finding | benchmark log | + +Benchmark protocol: take five questions from recent history, run each twice — +once with the agent (all actions recorded) and once manually — on the same +machine and firmware set, then compare TTFA/TTEP/evidence yield. Re-run the +manual side offline against the agent's sessions to separate speed from +correctness. Publish the ratios with the raw session directories, not just +the numbers (the project's "never a single run" rule applies to tooling too). + +Golden data is the regression bed: the demo session and re-hosted workloads +are checked in CI (roadmap V1), and every new stimulus script should grow a +golden expectation once it is stable. + +## 7. Roadmap hooks (H7) and what would make this real + +| H7 item | What exists today | What is missing | +|---|---|---| +| MCP plugin (R3) | JSON-lines Ghidra bridge; sessions as data | MCP server plugin (`plugins/mcp_server.py` when built) | +| deterministic harnesses | `run` scripts, probes, plugins, `record`/`compare` | named harness entry points, budgets, manifests | +| sessions as evidence | `greeneye-session/1`, `summary.md`, `compare` | session index, signed/hashed pack tooling | +| autonomous loops | hooks/watchpoints/compare primitives | hypothesis/verification controllers, stop conditions | +| safety | local-only bridge, no network, no hardware | enforced capability manifest, process isolation | + +What would make this real: + +1. **MCP tool schema** (sketch; planned only, no code in v0.1): + + ```json + { + "transport": "local stdio; no network exposure by default", + "tools": [ + {"name": "list_sessions", "args": {}}, + {"name": "open_session", "args": {"dir": "string"}}, + {"name": "symbols_resolve", "args": {"session": "string", "name": "string"}}, + {"name": "memory_read", "args": {"session": "string", "addr": "string", "size": "int"}}, + {"name": "memory_hexdump", "args": {"session": "string", "addr": "string", "size": "int"}}, + {"name": "set_breakpoint", "args": {"session": "string", "target": "string", "kind": "sw|hw"}}, + {"name": "watch", "args": {"session": "string", "target": "string", "size": "int", "kind": "w|r|a"}}, + {"name": "run", "args": {"session": "string", "duration": "float", "max_stops": "int"}}, + {"name": "run_until", "args": {"session": "string", "target": "string", "timeout": "float"}}, + {"name": "snapshot", "args": {"session": "string", "addr": "string", "size": "int", "label": "string"}}, + {"name": "compare_sessions", "args": {"a": "string", "b": "string", "symbols": "bool"}}, + {"name": "export_hits", "args": {"session": "string", "out": "string"}} + ] + } + ``` + + Baseline names follow docs/08 R3; `watch` and `run_until` follow the + bridge command set and H7. + +2. **Harness scripts** with fixed names and budgets: `stimulus.py` (one + workload), `probe_matrix.py` (version × device), `golden_check.py` + (compare against expected profiles). Today these are written ad hoc. +3. **Golden data**: checked-in expectation files for the demo and at least one + re-hosted workload, wired into `make test` (roadmap V1). +4. **Capability manifest** file the harness reads before any write action + (§5 table), with the session recording as the audit trail. +5. **Session index** for agents: a small JSONL listing session dirs, questions + and outcomes, so triage starts from history instead of from zero. +6. **Physical-escalation gate**: a write-up checklist that must be complete + before an agent-proposed change reaches a supervised hardware cycle. + +Until R3 lands, the agent loop is: CLI/scripts + Ghidra bridge + sessions — +nothing more, and the documentation should keep saying so. |