diff options
Diffstat (limited to 'docs/RUNBOOK.md')
| -rw-r--r-- | docs/RUNBOOK.md | 744 |
1 files changed, 744 insertions, 0 deletions
diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md new file mode 100644 index 0000000..79c1735 --- /dev/null +++ b/docs/RUNBOOK.md @@ -0,0 +1,744 @@ +# Open-IOV Defensive Operations Runbook + +> The per-incident book for operating an independent IP block (Open-IOV) +> against ecosystems that change on their own schedule. NVIDIA ships new GSP +> firmware, kernel modules, CUDA userspace, devices and ISA revisions without +> coordinating with us; any of them can break the emulation while the others +> stand still. This runbook is the ordered response. +> +> **Related:** [green-eye user guide](05-operations/04-green-eye-user-guide.md) +> · [GSP in a RISC-V VM (experimental)](05-operations/03-risc-v-gsp-rm-vm.md) +> · [Debugging and maintenance workflows](05-operations/02-debugging-and-maintenance-workflows.md) +> · [Hardware and software matrix](06-portability/hardware-software-matrix.md) +> · [Adversarial interoperability](04-implementation/02-practical-challenges-and-adversarial-interop.md) +> · [Developer onboarding and standards](07-developers/onboarding-and-standards.md) +> · [Engineering operations](eng-ops.md) +> +> **# TOOL IN DEVELOPMENT - NOT READY YET!** The green-eye tooling cited here +> is experimental; its interfaces and file formats may change. Nothing in this +> runbook authorises a product change, a hardware write or work on RM 610. + +## 0. How to use this runbook + +Audience: maintainers, release engineers and on-call operators of the Open-IOV +emulation and its evidence pipeline. + +Rules for reading under pressure: + +1. Find the symptom in §5 (scenario catalogue) or §9 (one-page decision tree). +2. Follow the **response order** exactly; do not skip the pairing/hash check. +3. Before any action that could change `open-iov-vfio`, complete the evidence + bundle (§6) and pass the escalation gates (§7). +4. Record the outcome where the scenario says to record it — a finding without + a record is lost work. +5. If an action is not in this runbook and not in the linked project docs, + treat it as a candidate for a new runbook entry, not as improvisation. + +Command conventions: `open-iov/` is the repository root. Commands run from the +repository root unless a block starts with `cd`. Angle-bracket values +(`<VER>`, `<session>`, `<image>`) are supplied by the task; never hardcode +private paths, host names or serial numbers into tracked documents. + +## 1. Operating principles and hard rules + +These apply to every scenario; where a scenario conflicts with them, the rules +win. + +| # | Principle | Consequence | +|---|---|---| +| P1 | **Anchor discipline.** The lock-step is *guest driver ↔ emulated firmware profile*. The host distribution driver is the A/B oracle, not the anchor. | Never evaluate a new `libcuda` against an old emulated firmware profile and call the mismatch an emulation bug. | +| P2 | **Never patch what we do not own.** Guest driver, host driver, GPU firmware, hypervisor, device model and guest OS stay unmodified. | Product changes live in our module, userspace, tools, configuration, tests and docs only. | +| P3 | **Evidence over anecdote.** Every conclusion needs at least an A/B pair and an explicit fidelity level (L0–L3). | A single run, a static reading or a simulator-only delta cannot land a product change. | +| P4 | **Physical confirmation for behaviour changes.** A simulator observation is evidence about the firmware, not about silicon. | Reproduction on a supervised physical/vfio-mdev cycle is required before a change that alters emulation behaviour. | +| P5 | **The simulator is a study environment, never a security boundary.** | No confidentiality, side-channel or security claims; never use injection to defeat a hardware security boundary. | +| P6 | **Development-only tooling.** green-eye is never shipped, never a dependency of the emulation, never in a guest. | Keep the in-development banner on tool docs until the acceptance list passes. | +| P7 | **No unattended physical-GPU cycles.** Physical work is supervised, with fail-safe release and explicit reboot approval. | Stop and ask; never bind/unbind drivers or claim the GPU as an experiment. | +| P8 | **RM 610 is a hard human gate.** Until a named human approves in writing, no acquisition, analysis, emulation edit, physical work or support claim for 610. | The gate is enforced in code by `version_diff`/`ci_release`; do not work around it. | +| P9 | **Fail closed, record honestly.** Unverified host info, unpaired firmware, unexplained deltas and absent references stay open. | "No unexplained deltas" is a claim that must be earned, not a default. | + +## 2. Maintainer roles and responsibilities + +One name per role at any time; handovers are a written change note in the +affected doc or PR. + +| Role | Owns | Key duties | Gates they hold | +|---|---|---|---| +| **Anchor maintainer** | guest driver ↔ emulated firmware anchor; version surface | keeps the anchor current; approves anchor bumps; keeps `config/versions/*.yaml` labelled as design targets until read by code | any change to the anchor | +| **Firmware triage maintainer** | versioned firmware directories; symbol DBs; probe/version-diff runs | acquires and hashes new images; builds symbol DBs with provenance; runs the P1 loop; owns the pairing decision | pairing verdict; "new logic" vs "behaviour moved" | +| **green-eye tooling maintainer** | `debug/risc-v-gsp-rm-vm/`; test suite; session format | keeps `make test` green; guards format/API stability; keeps the marker and doc map honest; maintains golden sessions | tool status claims; removal of the in-development marker | +| **CI/release maintainer** | `harness/version_diff.py`, `harness/ci_release.py`, `greeneye.yml`, goldens | runs dry-run before real runs; hashes evidence bundles; diffs seeded corpora; keeps CI paths and budgets bounded | release triage sign-off; RM 610 gate enforcement | +| **Physical-cycle supervisor** | supervised physical/vfio-mdev cycles | enforces fail-safe release, D0/D3cold handling and reboot approvals; captures reference traces | all physical execution; reboot requests | +| **Evidence archivist** | session and bundle retention | keeps SHA-256 inventories; enforces provenance (no private paths, no unsanitised host data); applies retention/prune policy | publication of evidence | +| **Docs steward** | this tree | enforces change notes, link checks and doc-map updates; retires stale claims | docs are part of "done" | + +Escalation order for an ambiguous incident: triage maintainer → anchor +maintainer → physical-cycle supervisor (if hardware is involved) → project +maintainer channels. Escalate with a reproduction and an evidence pack, never +with a summary alone. + +## 3. Routine maintenance + +### 3.1 Golden sessions and determinism + +| Item | Cadence | Procedure | Pass condition | +|---|---|---|---| +| Demo golden session | every tool change and weekly | `./greeneye.sh demo` and compare against the archived session/expected events | UART banner and control-block liveness present; hook/watch/breakpoint sequence coherent | +| Portability self-comparison | per PR touching `harness/`, `greeneye/`, `sim/` | `python3 harness/portability_matrix.py --target firmware/uproc_demo/build/uproc_demo.elf --target firmware/uproc_demo/build/uproc_demo.elf --out dumps/matrix-selfcheck` | `all_zero` deltas in `portability-report.json` | +| Mutation replay | per PR touching mutation/harness code | run `mutation_harness.py` twice with the same seed and budgets | `findings.jsonl` byte-identical | +| Version-diff self-check | per release-triage change | `version_diff.py` old == new | zero symbol/probe/behavior deltas | +| Host oracle capture | per anchor change | `./rmharness/harness.sh --session dumps/real-<anchor> --name before -- rmtool engine list --format=json` (repeat for `mem stats`) | capture recorded with sizes and SHA-256 | + +Golden artifacts are append-only history: never edit a stored golden session +(§3.2 uses SHA-256). Re-baseline by creating a new session with a change note. + +### 3.2 Session hygiene + +* Sessions are directories (`dumps/<stamp>-<name>/`) with `session.json`, + `events.jsonl`, `snapshots/`, `blobs/` and `summary.md`. Keep them together; + a copied `events.jsonl` without its `session.json` is not evidence. +* Name sessions for the question asked, not the host or the operator. +* Keep sessions bounded: fixed stimulus, fixed budgets, region-scoped + snapshots; a multi-gigabyte trace is a run that was not scoped. +* Publish a SHA-256 list (or the `bundle-manifest.json`) with any session that + leaves the machine. +* Write findings into `note` events inside the session (self-describing packs); + never rely on a chat log to explain a recording. +* Apply the provenance rule everywhere: tracked docs describe components and + behaviour; sessions contain no host-specific paths in their metadata. +* Bound growth deliberately: `dumps/` is gitignored scratch, so keep every + golden session and release bundle, publish the SHA-256 list before deleting + anything, and prune only scratch sessions past the review window — a + multi-gigabyte session is a scoping failure (fixed stimulus, fixed budgets, + region-scoped snapshots), not a storage problem to solve with new tooling. + +### 3.3 Toolchain updates + +| Dependency | Check | Action | +|---|---|---| +| QEMU | `./greeneye.sh info`; run `make test` | install/renew `qemu-system-riscv64` (Ubuntu `qemu-system-misc`); set `GREENEYE_QEMU` if the binary is elsewhere | +| QEMU plugin API | `make -C qemu` after a QEMU upgrade | rebuild `qemu/greeneye-trace.so` against the vendored `qemu-plugin.h`; re-run a trace smoke | +| Python | `python3 --version` (3.10+; stdlib only) | no pip packages required; keep plugins/harnesses stdlib-only | +| clang/lld | `make demo`, `make sdk` | reinstall toolchain if the demo or SDK stops building | +| Test suite | `python3 -m unittest discover -s tests -t .` | all green; skips for absent QEMU/demo explained, not ignored | +| Native parity | `python3 examples/scripts/sim_diff.py` | `PARITY: OK` (or a clean skip when QEMU is absent) | +| Docs | `python3 scripts/check-docs-links.py` from `open-iov/` | no broken relative links | + +### 3.4 Documentation and marker discipline + +* The **# TOOL IN DEVELOPMENT - NOT READY YET!** banner stays on tool docs + until the acceptance list in + `debug/risc-v-gsp-rm-vm/docs/00-OVERVIEW.md` passes; removal is a reviewed + change, not an edit. +* Every behaviour change updates the doc that describes it (docs are part of + the definition of done); planned work is labelled `planned (docs/xx)`. +* Status tables must match the code on the current commit (commands, flags, + test counts); stale claims are bugs. +* Keep the doc map complete when adding documents; keep change notes + (`<!-- change (YYYY-MM-DD): purpose; scope; benefit -->`) at edited + locations. +* Record reproducible discoveries in the failure taxonomy of + [`05-operations/02-debugging-and-maintenance-workflows.md`](05-operations/02-debugging-and-maintenance-workflows.md) + §5, not only in a session directory. +* Tool admission, extension, trialing and retirement are governed by + [`eng-ops.md`](eng-ops.md); procedural detail that is not incident response + belongs there, not here. + +## 4. The OODA ordering for a hostile update + +"Hostile" here means *unplanned firmware change plus deliberate opacity* — +new images on a hidden schedule, stripped and signed, hardware-coupled, +possibly with anti-analysis or decoy behaviour. It does not assume a malicious +actor, and it does not justify shortcuts. + +| Stage | Order | Actions | Exit criterion | +|---|---|---|---| +| **Observe** | 1 | Freeze the anchor; acquire the new input through the approved channel; record hash, version string and family; do not edit the emulation | Input is versioned and hashed in a parallel directory | +| | 2 | Capture the baseline: real-backend reference traces for the fixed workload (or the archived previous capture) | Baseline exists with a provenance note | +| | 3 | Record the symptom in a session note and a reproduction | Reproduction is repeatable and bounded | +| **Orient** | 4 | Static map: symbol DB/corpus diff per version | Inventory deltas listed | +| | 5 | Dynamic map: probe first-touch, then fixed-stimulus sessions on both versions | `compare` report with hit/watch/write deltas | +| | 6 | Classify each delta with the decision table (code changed vs behaviour moved) | Every delta has a class or is explicitly open | +| **Decide** | 7 | Fill the M1–M6 skeleton with the reference-capture method; list unexplained deltas | Metric table and open list attached | +| | 8 | Choose: no change, version-keyed handler, profile data, or fidelity gap | Decision names the evidence that drove it | +| | 9 | If the decision changes emulation behaviour, require physical reproduction; if it touches RM 610, stop at the gate | Gate passed or work parked | +| **Act** | 10 | Implement in `open-iov-vfio` with the evidence bundle linked; never in the simulator | PR carries the bundle and the metric table | +| | 11 | Re-run the identical stimulus; require profiles to improve or stay equal; re-run the guest ladder | No rung regression; CI green | +| | 12 | Archive the bundle and update the matrix/version docs only where earned | Recorded and reversible | + +Ordering rules that keep this honest: + +1. Never orient before the input is hashed and paired. +2. Never decide from a single session (P3). +3. Never act on the product from simulator-only evidence (P4). +4. Never let "the new version" become the anchor implicitly; the anchor moves + by explicit decision. + +## 5. Scenario catalogue + +Each scenario follows the same template: symptoms, detection, evidence, +response order, tools/commands, decision criteria, rollback, where to record. +Scenarios are incidents to run, not roadmap claims. + +### S1 — New GSP/RM firmware release + +| Attribute | Detail | +|---|---| +| **Symptoms** | Init fails with a generic status (the real 0x59 chase: `GSP_INIT_DONE rpc_result=0x59`), handler entries or reply shapes shift, a previously green guest step now fails, control inventory gains/removes IDs. | +| **Detection** | Firmware hash/version string changed; probe first-divergence address changed; symbol/corpus inventory delta; fixed-stimulus behaviour delta. | +| **Evidence** | Firmware SHA-256 + version string + pairing; old/new symbol DBs with provenance and bias; probe sessions; version-diff report; behaviour `compare`; M1–M6 skeleton; unexplained-delta list. | +| **Decision criteria** | Pairing mismatch → stop. Code changed and live profile changed → prioritise new logic. Code unchanged and profile changed → behaviour moved (state/ordering/environment fix). Symbol never hit either side → coverage gap, widen the stimulus first. | +| **Rollback** | The previous version's directory and configuration stay untouched; roll back by returning to the old anchor commit/tag and the old guest driver. A forward attempt never destroys the working setup. | +| **Record in** | The version's `firmware/<SERIES>-GSP/<VER>/` notes, the PR body (link the bundle), and the matrix entry only once support is earned. | + +**Response order** + +1. Freeze the anchor and the workload; do not edit emulation code. +2. Acquire the image through the approved channel; lay down a new versioned + directory; hash it. +3. Build a symbol DB for the new image and the previous one. +4. Probe both images with identical settings; diff the first-touch maps. +5. Run the static triage (dispatcher/handler/control/corpus diff). +6. Run the same stimulus on both (re-hosted or demo analog) and compare. +7. Fill M1–M6 with the reference-capture method; classify every delta. +8. Escalate any product-behaviour change to a physical reproduction. +9. Land the change (or record the fidelity gap) with the evidence bundle; add + a version-keyed handler only where the diff requires it. + +**Commands** + +```bash +cd debug/risc-v-gsp-rm-vm + +# 1) symbol DBs, one per version +./greeneye.sh symbols export --index <export-VER.tsv> --bias 0 -o symbols-<VER>.json + +# 2) bounded probe of each image (records a session) +./greeneye.sh probe <gsp-<VER>.fwimage.bin> --symbols symbols-<VER>.json --record + +# 3) full old-vs-new triage (shape, symbols, corpus, probe, behavior, M1-M6) +python3 harness/version_diff.py \ + --old <gsp-<OLD>.fwimage.bin> --new <gsp-<NEW>.fwimage.bin> \ + --old-symbols symbols-<OLD>.json --new-symbols symbols-<NEW>.json \ + --old-label <OLD> --new-label <NEW> \ + --out dumps/vdiff-<OLD>-<NEW> + +# 4) behaviour compare and canonical report +./greeneye.sh compare dumps/<old-run> dumps/<new-run> --symbols +./greeneye.sh analyze dumps/<new-run> --symbols symbols-<NEW>.json +``` + +### S2 — New kernel driver / open-kernel-modules release + +| Attribute | Detail | +|---|---| +| **Symptoms** | The optional OpenRM build fails with `Unknown symbol`; exports move or disappear between tags; the pinned build glue no longer matches the installed modules; `modinfo`/`dmesg` report version drift. | +| **Detection** | `./scripts/build_nvidia_openrm.sh` failure; `dmesg | grep 'Unknown symbol'`; `kernel-open` tree diff for the two tags; SBOM tag mismatch. | +| **Evidence** | Old/new tag identifiers; build transcript; the `Unknown symbol` list; a recorded classification of each moved export; confirmation that the runtime path still uses the stock distribution driver. | +| **Decision criteria** | If only build glue breaks and the runtime path is unaffected → repair or re-pin the glue. If an exported interface used by the emulation changed → treat as S1/S3 (functionality moved). If the distribution driver moved → it is the oracle; do not "upgrade" the anchor to match it. | +| **Rollback** | Restore the previously pinned module tag and rebuild; revert the glue commit. | +| **Record in** | `SBOM.md` tag updates, the build glue commit message, and the failure taxonomy if it is a reusable lesson. | + +**Response order** + +1. Verify which driver is actually in use (guest stock driver vs optional host + OpenRM modules) before debugging. +2. Reproduce the build failure with a clean tree; capture the full error. +3. Diff the `kernel-open` trees for the old and new tags (publish only the + symbol/interface delta). +4. Classify: build-glue-only, symbol rename, interface removal, or new + functionality. +5. Repair build glue if and only if the runtime contract is unchanged; + otherwise route to S3. +6. Re-run `make kernel`, the host test suite, and the optional OpenRM build. + +**Commands** + +```bash +# optional OpenRM build (build glue; never a runtime dependency) +./scripts/build_nvidia_openrm.sh +make kernel +./scripts/stack_manager.sh test + +# evidence of the failure +sudo dmesg -T | grep -i 'Unknown symbol' + +# runtime path sanity: guest remains stock +# (guest: stock nvidia.ko; no Open-IOV code in the guest, ever) +``` + +### S3 — Functionality moving between kernel and GSP, or between userspace versions + +| Attribute | Detail | +|---|---| +| **Symptoms** | A control that used to be answered by the guest kernel is now requested from the firmware (or vice versa); new RPCs/controls appear; reply fields that used to be populated are ignored; a workload passes on one component mix and fails on another. | +| **Detection** | Ioctl trace changes; control sweep finds new/removed IDs; dispatcher hit profile changes while static code is unchanged; `compare` shows hit/watch deltas with no source delta. | +| **Evidence** | Recorded ioctl/control diff; `discovery-report.json` old vs new; `reach.py` reports for moved targets; session A/B pair; M1/M3/M4 rows with the reference method. | +| **Decision criteria** | Same code and new hit profile → behaviour moved to a new caller/state; new handler ID → new logic; `default-path` result change → fail-closed/default behaviour changed; coverage gap → widen the sweep before deciding. | +| **Rollback** | Keep both component versions in parallel directories; revert the mix to the last green combination; never convert a version-keyed handler into a version-blind one. | +| **Record in** | The version's control-inventory notes, the interface docs for the exact version, and the PR that adds/updates the version-keyed handler. | + +**Response order** + +1. Capture the ioctl/control stream on the failing mix and on the last green + mix. +2. Sweep the call surface (control discovery) and diff the inventory. +3. Use `reach.py` to prove each moved target is reachable under the exact + stimulus and to record its trigger. +4. Attribute every delta to kernel, userspace or firmware with a session; + fill M1/M3/M4. +5. Update the version-keyed handler or capability table only for the classified + deltas; physical-confirm any behaviour change. + +**Commands** + +```bash +cd debug/risc-v-gsp-rm-vm + +# unknown-call sweep and inventory diff between versions +python3 harness/control_discovery.py --out dumps/discovery-<VER> --seed <seed> + +# reachability/trigger/args for a moved symbol +python3 harness/reach.py --target <symbol> --method auto --out dumps/reach-<symbol> + +# symbol-attributed A/B +./greeneye.sh compare dumps/<old-run> dumps/<new-run> --symbols +``` + +### S4 — New CUDA userspace / toolkit interacting unexpectedly with the GSP + +| Attribute | Detail | +|---|---| +| **Symptoms** | `cuInit` succeeds but context creation fails; a generic SUCCESS reply with incoherent consumed fields; a new or reordered control appears immediately before a failure; `libcuda` crashes (`lib[base+size] ip`); result mismatch in the A/B workload. | +| **Detection** | `cudatool.py info`/`delta` on the new `libcuda`; `fdematch` transfer measurement; corpus diff; ioctl trace diff; `ab.sh` result comparison. | +| **Evidence** | `delta` report (exports added/removed/moved, `.text` growth, build-id); `fdematch` table; wrapper re-resolution evidence; ioctl/control diff explained line by line; `cdt` output ending `ALL OK`; `ab.sh` `RESULT`/`STATUS` match; symbol DB with bias. | +| **Decision criteria** | From the delta: no export moves + small `.text` growth → patch (name map transfer, spot checks); new/removed exports → feature (decode the new surface first); many moves → reshuffle (full re-analysis; never transfer wrapper addresses). Anchor mismatch (new `libcuda` vs old firmware profile) → fix the pairing first. | +| **Rollback** | Keep `userspace/cuda/<version>/` trees in parallel; revert to the last green userspace version. Never overwrite an analysed version. | +| **Record in** | The version's decompilation map/`names.tsv`, the version-upgrade playbook notes, and the firmware-anchor note when the anchor also moves. | + +**Response order** + +1. Cheap triage first (minutes): `info` + `delta`; classify the port. +2. Measure name transfer before copying anything (`fdematch`). +3. Build the green-eye symbol DB for the new version so runtime addresses + resolve; spot-check transferred addresses. +4. Confirm dynamically: `cdt` with the ioctl tracer, then `ab.sh all`, against + the **same** emulated firmware profile (anchor discipline). +5. Diff the ioctl/control sequence old vs new; explain every change. +6. Only then touch the emulation, with physical confirmation per P4. + +**Commands** + +```bash +cd userspace/cuda +tools/cudatool.py info <new>/run/libcuda.so.<new> +tools/cudatool.py delta <old>/run/libcuda.so.<old> <new>/run/libcuda.so.<new> > <new>/run/delta.txt +tools/cudatool.py fdematch <old>/run/libcuda.so.<old> <new>/run/libcuda.so.<new> \ + --names <old>/annotations/names.tsv +tools/cudatool.py wrappers <new>/run/libcuda.so.<new> + +cd ../../debug/risc-v-gsp-rm-vm +./greeneye.sh symbols export \ + --names ../../userspace/cuda/<new>/annotations/names.tsv \ + --index ../../userspace/cuda/<new>/run/analysis/decompiled/decompiled_index.tsv \ + --bias 0x100000 -o symbols-<new>.json +./greeneye.sh symbols resolve --db symbols-<new>.json 0x<transferred-addr> + +cd ../.. +tools/cuda-ab/ab.sh all # host oracle + guest run + compare +``` + +### S5 — New GPU model or architecture (RTX 5090 / GB10-class, aarch64 hosts) + +| Attribute | Detail | +|---|---| +| **Symptoms** | Probe first-touch addresses differ from the reference device; a firmware family/partition-manifest mismatch; ISA string differs (RV64GC vs RV64IMAC); aarch64 host reports `qemu-system-riscv64 not found`; PMU/SOE provisioning assumptions break. | +| **Detection** | `file`/`readelf` image shape; pairing check (family, FMC partition manifest, version string); probe first-touch diff; host tooling check (`./greeneye.sh info`); `rmtool` host-oracle inventory. | +| **Evidence** | Device profile fields (PCI IDs, BARs, FB geometry, engine inventory, interrupt topology, ROM quirks); pairing notes; reference vs candidate first-touch maps; portability matrix report; physical validation record before any support claim. | +| **Decision criteria** | Changed first-touch on profile registers → profile data, not logic. New firmware family/partition model → new family inputs, not a re-badged image. RV64GC or new CSRs → ISA/decoder work (S6). No physical validation → status stays Target/Experimental, never Supported. | +| **Rollback** | Keep the reference device/firmware directories untouched; revert the candidate profile; a support claim is only removed with a matrix change note. | +| **Record in** | [`06-portability/hardware-software-matrix.md`](06-portability/hardware-software-matrix.md) and the device profile notes; the evaluation ladder result for the combination. | + +**Response order** + +1. Collect the device profile; verify the firmware pairing before debugging + anything (a mismatched pairing looks exactly like an emulation bug). +2. Confirm ISA/image shape (`file`, `readelf`); note FMC/manifest and boot + model; diff PMU/SOE images. +3. Probe the new image and the reference with identical budgets; diff the + first-touch maps. +4. Run the portability pipeline across reference and candidate. +5. Update profile data first; only add code when an interface genuinely + differs. +6. Validate on the physical device under supervision before claiming support. + +**Commands** + +```bash +file <device-fwimage> && readelf -h <device-elf> && readelf -l <device-elf> +cd debug/risc-v-gsp-rm-vm +./greeneye.sh info +./greeneye.sh probe <device-fwimage> --symbols symbols-<device>.json --record +python3 harness/portability_matrix.py \ + --target <reference-fwimage> --target <device-fwimage> \ + --out dumps/matrix-<device> +./rmharness/harness.sh --session dumps/real-<device> --name before \ + -- rmtool engine list --format=json +# aarch64 host without qemu-system-riscv64: +# install qemu-system-misc, or point GREENEYE_QEMU at the binary +``` + +### S6 — New instructions or ISA changes in NV-RISCV / falcon / GSP + +| Attribute | Detail | +|---|---| +| **Symptoms** | Probe faults with an illegal/unimplemented instruction; the native backend diverges from QEMU on the same image; compressed/float extensions or new CSRs appear; a re-hosted build built with the old `-march` fails or misbehaves. | +| **Detection** | ISA string from `file`/`readelf`; probe fault cause/`desc`; QEMU-vs-native differential (`sim_diff.py`); decoder unit tests. | +| **Evidence** | Image shape (ISA, segments, entry); fault report with cause, `epc`/`epc_firmware` and access address; `PARITY: OK` transcript or the exact divergence; a fidelity-gap note when the native decoder cannot execute the new instruction. | +| **Decision criteria** | RV64IMAC on QEMU TCG → L0 high, proceed. RV64GC → QEMU supports it; update rehost `-march` and re-run parity. Native decoder gap → record as a fidelity gap; do not claim native coverage. Illegal instruction in QEMU itself → confirm ISA first, then escalate as a tool bug. | +| **Rollback** | Revert rehost `-march`/build flags; keep the previous probe/session as the reference. | +| **Record in** | The tool's accuracy/fidelity-gap list (`debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md` §8) and the image's version notes. | + +**Response order** + +1. Confirm the ISA and image shape; identify the exact faulting instruction and + address. +2. Reproduce under QEMU (the reference execution engine) with a bounded probe. +3. Run the QEMU-vs-native differential; if it diverges, capture both sessions. +4. If the instruction is unsupported natively, record a fidelity gap with the + session as evidence; keep QEMU as the execution engine for that image. +5. Update rehost build flags and re-run parity and the tool test suite. + +**Commands** + +```bash +cd debug/risc-v-gsp-rm-vm +./greeneye.sh probe <fwimage> --instructions <budget> --record +python3 examples/scripts/sim_diff.py # requires QEMU; prints PARITY: OK on agreement +make test +``` + +### S7 — Significant VBIOS/ROM changes + +| Attribute | Detail | +|---|---| +| **Symptoms** | The guest driver reads different device identity/VBIOS fields; probe-time register values move; ROM/VBIOS quirks change (laptop/Surface variants); a previously green probe hangs on a different register. | +| **Detection** | Guest `lspci -vvv -d 10de:` and init-trace diff; snapshot diff of the identity/static-config region; `rmtool` read-only `BUS_GET_PCI_INFO` oracle; ROM hash change. | +| **Evidence** | ROM hash and identity fields; before/after snapshot with a schema-attributed diff; pairing notes; the exact guest-observed values and the trace that consumed them. | +| **Decision criteria** | Changed fields consumed by the driver → profile/override work, physically confirmed. Fields not consumed → record and monitor. Anything that would require impersonating a licensed product → forbidden; present a coherent device instead. | +| **Rollback** | Revert the profile/override; previous ROM/profile stays archived with its hash. | +| **Record in** | The device profile and the matrix entry; the failure taxonomy if the symptom was misleading. | + +**Response order** + +1. Hash the ROM/VBIOS and collect the identity fields. +2. Capture a before/after snapshot of the firmware-side static-config region + and diff it field by field. +3. Compare with the host oracle (read-only) and the guest's observed values. +4. Change only profile data that a consumer demonstrably uses; confirm on + hardware. +5. Record unexplained fields as open, not as noise. + +**Commands** + +```bash +# guest +sudo lspci -vvv -d 10de: +# host oracle (read-only; verified queries from docs/17) +# build once from the repo root: make -C tools/rmtool +tools/rmtool/rmtool gpu info -d 0 +tools/rmtool/rmtool engine list + +cd debug/risc-v-gsp-rm-vm +# snapshot the identity/static-config region before and after +# (schema decode from the bundled/demo schema; real regions are version-keyed) +./greeneye.sh compare dumps/<old-run> dumps/<new-run> --symbols +``` + +### S8 — Encrypted/protected memory or protected instructions (measurement plan; non-claims) + +| Attribute | Detail | +|---|---| +| **Symptoms** | The image consumes identity material that is provisioned encrypted (the PMU-delivered encrypted device identity in the real 0x59 campaign); protected-memory or protected-instruction paths are reached; a lenient run returns zeros where a real device returns provisioned values; boot halts at a secret-dependent step. | +| **Detection** | First-touch/device-map analysis; lenient-MMIO `mmio_log` and `mmio_unknown`; boot-chain shim assumptions in session metadata; probe fault reports identifying the consuming PC. | +| **Evidence** | First-touch map entry per protected address; `mmio_log` with PC and access; the exact access that consumes protected state; a written measurement plan and an explicit non-claims list. | +| **Decision criteria** | Secret-dependent branch without provisioned input → blocked; record as a fidelity gap and a device-model requirement, not a bug. Zero-value success under lenient MMIO → hypothesis only. Any request to extract, defeat or prove protected state → refuse (P5/P7). | +| **Rollback** | No product changes are made in this scenario; the only rollback is reverting experiment notes and keeping the map as data. | +| **Record in** | The fidelity-gap list, the device-map file (`greeneye-device-map/1`), and the H5 cross-controller plan; never in a security-claims section. | + +**Measurement plan (what to measure, and what is not claimed)** + +| Measure | Method | Non-claim | +|---|---|---| +| Which address/instruction consumes protected state | bounded probe + device map + `mmio_log` | not a statement about the real value | +| Whether execution continues when the value is absent/zero | lenient native run with `unknown_mmio="lenient"` | zeros are a hypothesis, not read-back semantics | +| Which subsystem provisions the value | cross-controller inventory (PMU/SEC2/ACR/SOE) and boot-order notes | not a model of the secure state machine | +| Whether the path is security-relevant | static map + recorded access ordering | not a side-channel, confidentiality or attack claim | + +**Response order** + +1. Classify: protected read, protected instruction, or a secret-dependent + branch. Do not attempt extraction; do not patch anything. +2. Capture the first-touch map and the lenient `mmio_log` in a recorded + session. +3. Write the measurement plan and the non-claims into the session note. +4. Escalate the missing provisioning as a device-model requirement (H2/H5), + not as an emulation defect. +5. Stop. Physical work on protected paths requires the supervisor and a + separate approved plan. + +**Commands** + +```bash +cd debug/risc-v-gsp-rm-vm +python3 -m greeneye.device_map demo --instructions 200000 +python3 -m greeneye.device_map harvest dumps/<probe-session> --out dumps/<ver>-device-map.json +python3 -m greeneye.device_map show dumps/<ver>-device-map.json +# lenient native continuation (programmatic): +# ge.launch(kernel=..., backend="native", +# device_map="dumps/<ver>-device-map.json", +# unknown_mmio="lenient") +# session.target.mmio_log / session.target.mmio_unknown +``` + +### S9 — New microcontrollers and new inter-/intra-microcontroller APIs and DMA lines + +| Attribute | Detail | +|---|---| +| **Symptoms** | The image stalls waiting on PMU/SEC2/ACR/SOE; falcon staging DMA does not land (the real failure mode: the SEC2 boot-loader block is not visible, the falcon does not halt); WPR2/security state differs from the successful path; new mailbox functions or DMA descriptors appear. | +| **Detection** | Probe first-touch; boot-chain shim assumptions recorded in session metadata; compare against the last successful boot path; device-map diffs; cross-controller event absence in sessions. | +| **Evidence** | First-touch addresses per controller window; mailbox/DMA access map; boot-order notes; shim assumptions in the session; the successful-path comparison; H2.2/H5 scope note. | +| **Decision criteria** | GSP-dependent stall → PMU/SEC2 stub work first (H2.2 entry). New mailbox/API surface → inventory it before modelling. Security-chain semantics → shim/documented assumption, never a claimed secure model. Unmodelled DMA line → device-map entry plus model work, not a product change. | +| **Rollback** | Revert any shim assumption change; the previous session/boot note stays as the reference. | +| **Record in** | The cross-controller inventory and boot-order notes; the device map; the H5/H2.2 roadmap scope. | + +**Response order** + +1. Inventory the controllers and DMA lines the image touches first + (PMU, SEC2, ACR/SOE, booter). +2. Reproduce the stall with a bounded probe and record shim assumptions in the + session. +3. Compare against the last successful path (e.g. WPR2 state, SEC2 staging + visibility) and name the missing exchange. +4. Model the minimum stub that lets the image proceed; keep it documented + data, not a silent special case. +5. Gate real security-chain semantics behind H2.2/H5; escalate any conclusion + that would change the product. + +**Commands** + +```bash +cd debug/risc-v-gsp-rm-vm +./greeneye.sh probe <fwimage> --record +python3 -m greeneye.device_map harvest dumps/<probe-session> --out dumps/<ver>-device-map.json +python3 -m greeneye.device_map merge dumps/<ref>-device-map.json dumps/<ver>-device-map.json \ + --out dumps/merged-device-map.json +./greeneye.sh compare dumps/<last-good-run> dumps/<stall-run> --symbols +``` + +### S10 — Multi-version matrix operations and CI (RM 610 requires explicit human approval) + +| Attribute | Detail | +|---|---| +| **Symptoms** | A new matrix column regresses an existing combination; CI drifts from the local run; someone proposes a 610-labelled input or command; the release bundle is incomplete. | +| **Detection** | `version_diff`/`portability_matrix` deltas; `ci_release --dry-run` plan inspection; the RM 610 gate refusal (`exit 3`); test transcripts. | +| **Evidence** | `version-diff.json`, `portability-report.json`, `release-report.json`, `bundle-manifest.json`, seeded `findings.jsonl`, test transcripts, approval record for 610. | +| **Decision criteria** | Dry-run clean before a real run. A step that raises is `error`, not skipped. Self-comparison must be `all_zero`; seeded replays byte-identical. **610: no acquisition, analysis, emulation edit, physical work or support claim before a named human approves in writing; an open-ended approval is invalid.** | +| **Rollback** | Extend the matrix, never replace a column; revert the candidate commit and keep the previous anchor/tag/driver; a 610 attempt parks without touching 570.144. | +| **Record in** | The port log/approval thread (610), the release bundle directory, and the matrix with its test date and rung. | + +**Response order** + +1. Run the release rehearsal on current-version labels first; inspect the + dry-run plan. +2. Run `version_diff` and `portability_matrix` with fixed budgets; require the + self-comparison to be zero. +3. Run the mutation smoke with a pinned seed; verify replay. +4. Assemble and hash the bundle; attach it to the PR. +5. For 610: stop at the gate; obtain the approval message; only then acquire + inputs and repeat the pipeline with 610 labels. + +**Commands** + +```bash +cd debug/risc-v-gsp-rm-vm + +# rehearsal first: dry-run never opens a target and creates no output dir +python3 harness/ci_release.py --old <old> --new <new> --out dumps/release-<ver> \ + --old-label <OLD> --new-label <NEW> --dry-run + +# real pipeline (dry-run clean) +python3 harness/ci_release.py --old <old> --new <new> --out dumps/release-<ver> \ + --old-label <OLD> --new-label <NEW> \ + --old-symbols symbols-<OLD>.json --new-symbols symbols-<NEW>.json \ + --mutation-kernel firmware/uproc_demo/build/uproc_demo.elf \ + --mutation-iterations 8 --mutation-seed 7 + +# RM 610 gate: refused by default; explicit approval token required +python3 harness/version_diff.py --old <old> --new <new> --out dumps/vdiff-610 \ + --old-label RM-610 --new-label 570.144 +# exit 3 unless: --approval-token RM610-APPROVED or GREENEYE_RM610_APPROVED=1 +``` + +## 6. Evidence-bundle requirements + +A bundle is the minimum attachable proof for a version/device/architecture +triage or a product change. It combines the docs/07 evidence checklist, the +docs/14 §6 adversarial pack and the docs/21 release tree. + +``` +<out>/ + bundle-manifest.json sha256 + bytes of every file below + release-report.json pipeline summary (steps, gate, labels) + version-diff/ version-diff.json|.md, symbols/, sessions/, compare/ + portability-matrix/ portability-report.json|.md, symbols/, sessions/, compare/ + mutation-smoke/ findings.jsonl, summary.md, sessions/ +``` + +Checklist — every box must be true or explicitly waived with a reason: + +* [ ] firmware identity: SHA-256, version string, family and pairing notes; +* [ ] input provenance: channel, date, and the exact file names analysed; +* [ ] symbol DB(s): hash, source (ELF/Ghidra/JSON), `bias` and provenance; +* [ ] baseline and candidate session directories (or their SHA-256 list); +* [ ] exact stimulus (script/spec plus mutation values) and budgets; +* [ ] `compare` report and relevant `analyze` reports; +* [ ] M1–M6 table with the reference-capture method and L2/L3 limits stated; +* [ ] `greeneye_hits.tsv` and any Ghidra annotations produced; +* [ ] `bundle-manifest.json` written last, covering the final reports; +* [ ] unexplained-delta list (empty is a claim, not a default); +* [ ] decision and product action (or explicit "no change") recorded; +* [ ] for 610: approval ID, approver, date and scope limits. + +Fidelity statement template (copy into every report): + +```text +Fidelity: L0 (instruction semantics) and L1 (memory) exercised; L2 (timing, +concurrency) NOT modelled; L3 (device environment) absent for <registers>. +Reference capture: <hardware / vfio-mdev / host oracle / none>. +Physical reproduction required before product change: <yes/no + why>. +``` + +## 7. Fault and escalation tree + +```text +Failure / new input observed +| ++- Is the input hashed, versioned and paired? NO -> STOP: acquire/hash/pair first (S1) +| ++- Did it come from RM 610 or a 610-labelled path? YES -> STOP: hard human gate (S10) +| ++- Is a physical GPU / driver bind / mdev involved? +| YES -> STOP: supervised physical cycle + reboot approval (P7); never unattended +| ++- Does the question need timing, interrupts, protected state or real device +| read-back semantics (L2/L3)? +| YES -> simulator evidence is a hypothesis only (P3/P4/P5); record the +| fidelity gap; product change requires physical reproduction +| ++- Does it reproduce under a fixed stimulus in green-eye? +| NO -> widen the stimulus / capture the real trace before theorising; +| absence of a hit is not proof of dead code +| YES -> classify with the S1 decision table (code changed vs behaviour moved) +| ++- Would a fix change emulation behaviour? +| YES -> physical/vfio-mdev reproduction + evidence bundle before landing +| NO -> docs/profile/test-only change with the bundle attached +| ++- Does any vendor component need modification? + YES -> forbidden (P2): redesign to emulate the interface instead +``` + +Stop conditions (do not proceed, do not "temporarily" work around): + +1. Unverified pairing or unhashed input. +2. RM 610 without written approval. +3. A physical cycle without the supervisor and fail-safe plan. +4. A request to patch a vendor component or a guest. +5. A security/confidentiality claim based on simulator state. +6. A product change with no physical reproduction and no bundle. + +## 8. When NOT to use the tooling + +| Situation | Why not | Use instead | +|---|---|---| +| You need silicon truth (timing, interrupts, real register read-back) | L2/L3 are absent by design | supervised physical cycle + reference traces | +| You are about to mutate or inject against hardware, a guest, a kernel module or an existing gdbstub | Tooling is simulator-only; hardware writes are prohibited | bounded QEMU/native harness with recorded sessions | +| You need to defeat, extract or prove a security boundary | The simulator is not a confidentiality boundary | stop; escalate as a design question | +| You need a product fix "now" without evidence | Violates P3/P4 | evidence bundle first; the anchor and matrix can wait | +| The input has not been hashed/versioned/paired | Every later claim is contaminated | S1 steps 1–3 | +| RM 610 has no written approval | Hard gate | request approval with the readiness checklist | +| The physical GPU is held by another job | Shared-oracle safety and fail-safe discipline | host-side read-only oracle (`rmtool gpu list`, `harness.sh`) | +| You need type recovery or decompilation | green-eye is dynamic ground truth, not a decompiler | Ghidra corpus and symbol tooling | +| You need guest-driver behaviour | green-eye observes firmware, not the guest | `tools/guest-diag/`, ioctl tracer | +| A one-line question can be answered by `analyze` on an existing session | Do not produce new runs for no reason | reuse recorded sessions and `compare` | + +## 9. One-page quick-reference decision tree + +| Symptom (first sign) | First command | Decision | Scenario | +|---|---|---|---| +| New image/version appears | `./greeneye.sh symbols export ...` + `probe --record` | hash+pair first, then version diff | S1 | +| OpenRM build breaks, `Unknown symbol` | `./scripts/build_nvidia_openrm.sh`; `dmesg` | glue-only vs interface change | S2 | +| Control/ioctl inventory shifts | `python3 harness/control_discovery.py` | moved vs new logic | S3 | +| `libcuda` bump / context failure | `tools/cudatool.py delta` + `fdematch` | patch / feature / reshuffle | S4 | +| New device/architecture | `file`/`readelf`, `probe` | profile vs logic; physical validation for support | S5 | +| Illegal/unknown instruction | `./greeneye.sh probe --instructions N` | ISA/decoder; fidelity gap if native-only | S6 | +| Identity/ROM fields differ | guest `lspci -vvv`, snapshot diff | consumed field → profile; else record | S7 | +| Protected/encrypted state blocks | device map + lenient `mmio_log` | record fidelity gap; no extraction; no claims | S8 | +| Early stall on PMU/SEC2/ACR/SOE | `probe --record` + device map | stub the missing exchange; gate security semantics | S9 | +| Matrix/CI drift; 610 mentioned | `ci_release --dry-run` | dry-run clean; 610 hard gate | S10 | + +Golden rule: **freeze, hash, pair, reproduce, classify, evidence, then act** — +and never let the last step touch anything we do not own. + +## 10. Related documentation + +Green-eye suite (experimental, in development): + +* [green-eye user guide](05-operations/04-green-eye-user-guide.md) — what it + is, quickstart, tool chooser and troubleshooting; +* [GSP in a RISC-V VM](05-operations/03-risc-v-gsp-rm-vm.md) — project-level + purpose, status and limits; +* [`debug/risc-v-gsp-rm-vm/docs/00-OVERVIEW.md`](../debug/risc-v-gsp-rm-vm/docs/00-OVERVIEW.md) + — tool status and doc map; +* [`debug/risc-v-gsp-rm-vm/GREEN-EYE-ROADMAP.md`](../debug/risc-v-gsp-rm-vm/GREEN-EYE-ROADMAP.md) + — H1–H7 program; +* [`debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md`](../debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md) + — fidelity levels and the M1–M6 contract; +* [`debug/risc-v-gsp-rm-vm/docs/07-PORTABILITY-PLAYBOOKS.md`](../debug/risc-v-gsp-rm-vm/docs/07-PORTABILITY-PLAYBOOKS.md) + — P1–P4 porting playbooks; +* [`debug/risc-v-gsp-rm-vm/docs/14-ADVERSARIAL-INTEROP-GUIDE.md`](../debug/risc-v-gsp-rm-vm/docs/14-ADVERSARIAL-INTEROP-GUIDE.md) + — adversarial navigation and the evidence pack; +* [`debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md`](../debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md) + — release triage and bundles; +* [`debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md`](../debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md) + — the 610 gate and approval format; +* [`debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md`](../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md) + — tool audit, symptom-to-fix table and provenance rule; +* [`debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md`](../debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md) + — sweep/device-port scaffolds (unadmitted pilots, dry-run default); +* [`debug/risc-v-gsp-rm-vm/docs/29-PACKAGING-AND-TRIALING.md`](../debug/risc-v-gsp-rm-vm/docs/29-PACKAGING-AND-TRIALING.md) + — trial simulation packages (unadmitted pilot). + +Project documentation: + +* [Debugging and maintenance workflows](05-operations/02-debugging-and-maintenance-workflows.md) + — captures, failure taxonomy, version discipline; +* [Performance, correctness, security](05-operations/01-performance-correctness-security.md) + — the evaluation ladder; +* [Hardware and software matrix](06-portability/hardware-software-matrix.md) + — support definitions and the portability pipeline; +* [Practical challenges and adversarial interoperability](04-implementation/02-practical-challenges-and-adversarial-interop.md) + — the no-patch rule; +* [Developer onboarding and standards](07-developers/onboarding-and-standards.md) + — coding rules and the landing checklist; +* [Documentation front door](README.md) — the full reading order; +* [Engineering operations](eng-ops.md) — tool admission, workflows and + retirement (the cross-cutting engineering-operations document); +* CUDA playbook: + [`userspace/cuda/docs/07-version-upgrade-playbook.md`](../userspace/cuda/docs/07-version-upgrade-playbook.md) + and [`userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md). + +<!-- change (2026-10-02): fix the S4 fdematch invocation (two libraries plus +--names, per tools/cudatool.py), add a session/disk-growth bound to §3.2, and +link docs 28/29 from §10; scope: RUNBOOK §3.2, §5 S4, §10 only; benefit: the +copied triage command runs as written and the retention rule is explicit. --> |