path: root/docs/RUNBOOK.md

# Open-IOV Defensive Operations Runbook

> The per-incident book for operating an independent IP block (Open-IOV)
> against ecosystems that change on their own schedule. NVIDIA ships new GSP
> firmware, kernel modules, CUDA userspace, devices and ISA revisions without
> coordinating with us; any of them can break the emulation while the others
> stand still. This runbook is the ordered response.
>
> **Related:** [green-eye user guide](05-operations/04-green-eye-user-guide.md)
> · [GSP in a RISC-V VM (experimental)](05-operations/03-risc-v-gsp-rm-vm.md)
> · [Debugging and maintenance workflows](05-operations/02-debugging-and-maintenance-workflows.md)
> · [Hardware and software matrix](06-portability/hardware-software-matrix.md)
> · [Adversarial interoperability](04-implementation/02-practical-challenges-and-adversarial-interop.md)
> · [Developer onboarding and standards](07-developers/onboarding-and-standards.md)
> · [Engineering operations](eng-ops.md)
>
> **# TOOL IN DEVELOPMENT - NOT READY YET!** The green-eye tooling cited here
> is experimental; its interfaces and file formats may change. Nothing in this
> runbook authorises a product change, a hardware write or work on RM 610.

## 0. How to use this runbook

Audience: maintainers, release engineers and on-call operators of the Open-IOV
emulation and its evidence pipeline.

Rules for reading under pressure:

1. Find the symptom in §5 (scenario catalogue) or §9 (one-page decision tree).
2. Follow the **response order** exactly; do not skip the pairing/hash check.
3. Before any action that could change `open-iov-vfio`, complete the evidence
   bundle (§6) and pass the escalation gates (§7).
4. Record the outcome where the scenario says to record it — a finding without
   a record is lost work.
5. If an action is not in this runbook and not in the linked project docs,
   treat it as a candidate for a new runbook entry, not as improvisation.

Command conventions: `open-iov/` is the repository root. Commands run from the
repository root unless a block starts with `cd`. Angle-bracket values
(`<VER>`, `<session>`, `<image>`) are supplied by the task; never hardcode
private paths, host names or serial numbers into tracked documents.

## 1. Operating principles and hard rules

These apply to every scenario; where a scenario conflicts with them, the rules
win.

| # | Principle | Consequence |
|---|---|---|
| P1 | **Anchor discipline.** The lock-step is *guest driver ↔ emulated firmware profile*. The host distribution driver is the A/B oracle, not the anchor. | Never evaluate a new `libcuda` against an old emulated firmware profile and call the mismatch an emulation bug. |
| P2 | **Never patch what we do not own.** Guest driver, host driver, GPU firmware, hypervisor, device model and guest OS stay unmodified. | Product changes live in our module, userspace, tools, configuration, tests and docs only. |
| P3 | **Evidence over anecdote.** Every conclusion needs at least an A/B pair and an explicit fidelity level (L0–L3). | A single run, a static reading or a simulator-only delta cannot land a product change. |
| P4 | **Physical confirmation for behaviour changes.** A simulator observation is evidence about the firmware, not about silicon. | Reproduction on a supervised physical/vfio-mdev cycle is required before a change that alters emulation behaviour. |
| P5 | **The simulator is a study environment, never a security boundary.** | No confidentiality, side-channel or security claims; never use injection to defeat a hardware security boundary. |
| P6 | **Development-only tooling.** green-eye is never shipped, never a dependency of the emulation, never in a guest. | Keep the in-development banner on tool docs until the acceptance list passes. |
| P7 | **No unattended physical-GPU cycles.** Physical work is supervised, with fail-safe release and explicit reboot approval. | Stop and ask; never bind/unbind drivers or claim the GPU as an experiment. |
| P8 | **RM 610 is a hard human gate.** Until a named human approves in writing, no acquisition, analysis, emulation edit, physical work or support claim for 610. | The gate is enforced in code by `version_diff`/`ci_release`; do not work around it. |
| P9 | **Fail closed, record honestly.** Unverified host info, unpaired firmware, unexplained deltas and absent references stay open. | "No unexplained deltas" is a claim that must be earned, not a default. |

## 2. Maintainer roles and responsibilities

One name per role at any time; handovers are a written change note in the
affected doc or PR.

| Role | Owns | Key duties | Gates they hold |
|---|---|---|---|
| **Anchor maintainer** | guest driver ↔ emulated firmware anchor; version surface | keeps the anchor current; approves anchor bumps; keeps `config/versions/*.yaml` labelled as design targets until read by code | any change to the anchor |
| **Firmware triage maintainer** | versioned firmware directories; symbol DBs; probe/version-diff runs | acquires and hashes new images; builds symbol DBs with provenance; runs the P1 loop; owns the pairing decision | pairing verdict; "new logic" vs "behaviour moved" |
| **green-eye tooling maintainer** | `debug/risc-v-gsp-rm-vm/`; test suite; session format | keeps `make test` green; guards format/API stability; keeps the marker and doc map honest; maintains golden sessions | tool status claims; removal of the in-development marker |
| **CI/release maintainer** | `harness/version_diff.py`, `harness/ci_release.py`, `greeneye.yml`, goldens | runs dry-run before real runs; hashes evidence bundles; diffs seeded corpora; keeps CI paths and budgets bounded | release triage sign-off; RM 610 gate enforcement |
| **Physical-cycle supervisor** | supervised physical/vfio-mdev cycles | enforces fail-safe release, D0/D3cold handling and reboot approvals; captures reference traces | all physical execution; reboot requests |
| **Evidence archivist** | session and bundle retention | keeps SHA-256 inventories; enforces provenance (no private paths, no unsanitised host data); applies retention/prune policy | publication of evidence |
| **Docs steward** | this tree | enforces change notes, link checks and doc-map updates; retires stale claims | docs are part of "done" |

Escalation order for an ambiguous incident: triage maintainer → anchor
maintainer → physical-cycle supervisor (if hardware is involved) → project
maintainer channels. Escalate with a reproduction and an evidence pack, never
with a summary alone.

## 3. Routine maintenance

### 3.1 Golden sessions and determinism

| Item | Cadence | Procedure | Pass condition |
|---|---|---|---|
| Demo golden session | every tool change and weekly | `./greeneye.sh demo` and compare against the archived session/expected events | UART banner and control-block liveness present; hook/watch/breakpoint sequence coherent |
| Portability self-comparison | per PR touching `harness/`, `greeneye/`, `sim/` | `python3 harness/portability_matrix.py --target firmware/uproc_demo/build/uproc_demo.elf --target firmware/uproc_demo/build/uproc_demo.elf --out dumps/matrix-selfcheck` | `all_zero` deltas in `portability-report.json` |
| Mutation replay | per PR touching mutation/harness code | run `mutation_harness.py` twice with the same seed and budgets | `findings.jsonl` byte-identical |
| Version-diff self-check | per release-triage change | `version_diff.py` old == new | zero symbol/probe/behavior deltas |
| Host oracle capture | per anchor change | `./rmharness/harness.sh --session dumps/real-<anchor> --name before -- rmtool engine list --format=json` (repeat for `mem stats`) | capture recorded with sizes and SHA-256 |

Golden artifacts are append-only history: never edit a stored golden session
(§3.2 uses SHA-256). Re-baseline by creating a new session with a change note.

### 3.2 Session hygiene

* Sessions are directories (`dumps/<stamp>-<name>/`) with `session.json`,
  `events.jsonl`, `snapshots/`, `blobs/` and `summary.md`. Keep them together;
  a copied `events.jsonl` without its `session.json` is not evidence.
* Name sessions for the question asked, not the host or the operator.
* Keep sessions bounded: fixed stimulus, fixed budgets, region-scoped
  snapshots; a multi-gigabyte trace is a run that was not scoped.
* Publish a SHA-256 list (or the `bundle-manifest.json`) with any session that
  leaves the machine.
* Write findings into `note` events inside the session (self-describing packs);
  never rely on a chat log to explain a recording.
* Apply the provenance rule everywhere: tracked docs describe components and
  behaviour; sessions contain no host-specific paths in their metadata.
* Bound growth deliberately: `dumps/` is gitignored scratch, so keep every
  golden session and release bundle, publish the SHA-256 list before deleting
  anything, and prune only scratch sessions past the review window — a
  multi-gigabyte session is a scoping failure (fixed stimulus, fixed budgets,
  region-scoped snapshots), not a storage problem to solve with new tooling.

### 3.3 Toolchain updates

| Dependency | Check | Action |
|---|---|---|
| QEMU | `./greeneye.sh info`; run `make test` | install/renew `qemu-system-riscv64` (Ubuntu `qemu-system-misc`); set `GREENEYE_QEMU` if the binary is elsewhere |
| QEMU plugin API | `make -C qemu` after a QEMU upgrade | rebuild `qemu/greeneye-trace.so` against the vendored `qemu-plugin.h`; re-run a trace smoke |
| Python | `python3 --version` (3.10+; stdlib only) | no pip packages required; keep plugins/harnesses stdlib-only |
| clang/lld | `make demo`, `make sdk` | reinstall toolchain if the demo or SDK stops building |
| Test suite | `python3 -m unittest discover -s tests -t .` | all green; skips for absent QEMU/demo explained, not ignored |
| Native parity | `python3 examples/scripts/sim_diff.py` | `PARITY: OK` (or a clean skip when QEMU is absent) |
| Docs | `python3 scripts/check-docs-links.py` from `open-iov/` | no broken relative links |

### 3.4 Documentation and marker discipline

* The **# TOOL IN DEVELOPMENT - NOT READY YET!** banner stays on tool docs
  until the acceptance list in
  `debug/risc-v-gsp-rm-vm/docs/00-OVERVIEW.md` passes; removal is a reviewed
  change, not an edit.
* Every behaviour change updates the doc that describes it (docs are part of
  the definition of done); planned work is labelled `planned (docs/xx)`.
* Status tables must match the code on the current commit (commands, flags,
  test counts); stale claims are bugs.
* Keep the doc map complete when adding documents; keep change notes
  (`<!-- change (YYYY-MM-DD): purpose; scope; benefit -->`) at edited
  locations.
* Record reproducible discoveries in the failure taxonomy of
  [`05-operations/02-debugging-and-maintenance-workflows.md`](05-operations/02-debugging-and-maintenance-workflows.md)
  §5, not only in a session directory.
* Tool admission, extension, trialing and retirement are governed by
  [`eng-ops.md`](eng-ops.md); procedural detail that is not incident response
  belongs there, not here.

## 4. The OODA ordering for a hostile update

"Hostile" here means *unplanned firmware change plus deliberate opacity* —
new images on a hidden schedule, stripped and signed, hardware-coupled,
possibly with anti-analysis or decoy behaviour. It does not assume a malicious
actor, and it does not justify shortcuts.

| Stage | Order | Actions | Exit criterion |
|---|---|---|---|
| **Observe** | 1 | Freeze the anchor; acquire the new input through the approved channel; record hash, version string and family; do not edit the emulation | Input is versioned and hashed in a parallel directory |
| | 2 | Capture the baseline: real-backend reference traces for the fixed workload (or the archived previous capture) | Baseline exists with a provenance note |
| | 3 | Record the symptom in a session note and a reproduction | Reproduction is repeatable and bounded |
| **Orient** | 4 | Static map: symbol DB/corpus diff per version | Inventory deltas listed |
| | 5 | Dynamic map: probe first-touch, then fixed-stimulus sessions on both versions | `compare` report with hit/watch/write deltas |
| | 6 | Classify each delta with the decision table (code changed vs behaviour moved) | Every delta has a class or is explicitly open |
| **Decide** | 7 | Fill the M1–M6 skeleton with the reference-capture method; list unexplained deltas | Metric table and open list attached |
| | 8 | Choose: no change, version-keyed handler, profile data, or fidelity gap | Decision names the evidence that drove it |
| | 9 | If the decision changes emulation behaviour, require physical reproduction; if it touches RM 610, stop at the gate | Gate passed or work parked |
| **Act** | 10 | Implement in `open-iov-vfio` with the evidence bundle linked; never in the simulator | PR carries the bundle and the metric table |
| | 11 | Re-run the identical stimulus; require profiles to improve or stay equal; re-run the guest ladder | No rung regression; CI green |
| | 12 | Archive the bundle and update the matrix/version docs only where earned | Recorded and reversible |

Ordering rules that keep this honest:

1. Never orient before the input is hashed and paired.
2. Never decide from a single session (P3).
3. Never act on the product from simulator-only evidence (P4).
4. Never let "the new version" become the anchor implicitly; the anchor moves
   by explicit decision.

## 5. Scenario catalogue

Each scenario follows the same template: symptoms, detection, evidence,
response order, tools/commands, decision criteria, rollback, where to record.
Scenarios are incidents to run, not roadmap claims.

### S1 — New GSP/RM firmware release

| Attribute | Detail |
|---|---|
| **Symptoms** | Init fails with a generic status (the real 0x59 chase: `GSP_INIT_DONE rpc_result=0x59`), handler entries or reply shapes shift, a previously green guest step now fails, control inventory gains/removes IDs. |
| **Detection** | Firmware hash/version string changed; probe first-divergence address changed; symbol/corpus inventory delta; fixed-stimulus behaviour delta. |
| **Evidence** | Firmware SHA-256 + version string + pairing; old/new symbol DBs with provenance and bias; probe sessions; version-diff report; behaviour `compare`; M1–M6 skeleton; unexplained-delta list. |
| **Decision criteria** | Pairing mismatch → stop. Code changed and live profile changed → prioritise new logic. Code unchanged and profile changed → behaviour moved (state/ordering/environment fix). Symbol never hit either side → coverage gap, widen the stimulus first. |
| **Rollback** | The previous version's directory and configuration stay untouched; roll back by returning to the old anchor commit/tag and the old guest driver. A forward attempt never destroys the working setup. |
| **Record in** | The version's `firmware/<SERIES>-GSP/<VER>/` notes, the PR body (link the bundle), and the matrix entry only once support is earned. |

**Response order**

1. Freeze the anchor and the workload; do not edit emulation code.
2. Acquire the image through the approved channel; lay down a new versioned
   directory; hash it.
3. Build a symbol DB for the new image and the previous one.
4. Probe both images with identical settings; diff the first-touch maps.
5. Run the static triage (dispatcher/handler/control/corpus diff).
6. Run the same stimulus on both (re-hosted or demo analog) and compare.
7. Fill M1–M6 with the reference-capture method; classify every delta.
8. Escalate any product-behaviour change to a physical reproduction.
9. Land the change (or record the fidelity gap) with the evidence bundle; add
   a version-keyed handler only where the diff requires it.

**Commands**

```bash
cd debug/risc-v-gsp-rm-vm

# 1) symbol DBs, one per version
./greeneye.sh symbols export --index <export-VER.tsv> --bias 0 -o symbols-<VER>.json

# 2) bounded probe of each image (records a session)
./greeneye.sh probe <gsp-<VER>.fwimage.bin> --symbols symbols-<VER>.json --record

# 3) full old-vs-new triage (shape, symbols, corpus, probe, behavior, M1-M6)
python3 harness/version_diff.py \
    --old <gsp-<OLD>.fwimage.bin> --new <gsp-<NEW>.fwimage.bin> \
    --old-symbols symbols-<OLD>.json --new-symbols symbols-<NEW>.json \
    --old-label <OLD> --new-label <NEW> \
    --out dumps/vdiff-<OLD>-<NEW>

# 4) behaviour compare and canonical report
./greeneye.sh compare dumps/<old-run> dumps/<new-run> --symbols
./greeneye.sh analyze dumps/<new-run> --symbols symbols-<NEW>.json
```

### S2 — New kernel driver / open-kernel-modules release

| Attribute | Detail |
|---|---|
| **Symptoms** | The optional OpenRM build fails with `Unknown symbol`; exports move or disappear between tags; the pinned build glue no longer matches the installed modules; `modinfo`/`dmesg` report version drift. |
| **Detection** | `./scripts/build_nvidia_openrm.sh` failure; `dmesg | grep 'Unknown symbol'`; `kernel-open` tree diff for the two tags; SBOM tag mismatch. |
| **Evidence** | Old/new tag identifiers; build transcript; the `Unknown symbol` list; a recorded classification of each moved export; confirmation that the runtime path still uses the stock distribution driver. |
| **Decision criteria** | If only build glue breaks and the runtime path is unaffected → repair or re-pin the glue. If an exported interface used by the emulation changed → treat as S1/S3 (functionality moved). If the distribution driver moved → it is the oracle; do not "upgrade" the anchor to match it. |
| **Rollback** | Restore the previously pinned module tag and rebuild; revert the glue commit. |
| **Record in** | `SBOM.md` tag updates, the build glue commit message, and the failure taxonomy if it is a reusable lesson. |

**Response order**

1. Verify which driver is actually in use (guest stock driver vs optional host
   OpenRM modules) before debugging.
2. Reproduce the build failure with a clean tree; capture the full error.
3. Diff the `kernel-open` trees for the old and new tags (publish only the
   symbol/interface delta).
4. Classify: build-glue-only, symbol rename, interface removal, or new
   functionality.
5. Repair build glue if and only if the runtime contract is unchanged;
   otherwise route to S3.
6. Re-run `make kernel`, the host test suite, and the optional OpenRM build.

**Commands**

```bash
# optional OpenRM build (build glue; never a runtime dependency)
./scripts/build_nvidia_openrm.sh
make kernel
./scripts/stack_manager.sh test

# evidence of the failure
sudo dmesg -T | grep -i 'Unknown symbol'

# runtime path sanity: guest remains stock
# (guest: stock nvidia.ko; no Open-IOV code in the guest, ever)
```

### S3 — Functionality moving between kernel and GSP, or between userspace versions

| Attribute | Detail |
|---|---|
| **Symptoms** | A control that used to be answered by the guest kernel is now requested from the firmware (or vice versa); new RPCs/controls appear; reply fields that used to be populated are ignored; a workload passes on one component mix and fails on another. |
| **Detection** | Ioctl trace changes; control sweep finds new/removed IDs; dispatcher hit profile changes while static code is unchanged; `compare` shows hit/watch deltas with no source delta. |
| **Evidence** | Recorded ioctl/control diff; `discovery-report.json` old vs new; `reach.py` reports for moved targets; session A/B pair; M1/M3/M4 rows with the reference method. |
| **Decision criteria** | Same code and new hit profile → behaviour moved to a new caller/state; new handler ID → new logic; `default-path` result change → fail-closed/default behaviour changed; coverage gap → widen the sweep before deciding. |
| **Rollback** | Keep both component versions in parallel directories; revert the mix to the last green combination; never convert a version-keyed handler into a version-blind one. |
| **Record in** | The version's control-inventory notes, the interface docs for the exact version, and the PR that adds/updates the version-keyed handler. |

**Response order**

1. Capture the ioctl/control stream on the failing mix and on the last green
   mix.
2. Sweep the call surface (control discovery) and diff the inventory.
3. Use `reach.py` to prove each moved target is reachable under the exact
   stimulus and to record its trigger.
4. Attribute every delta to kernel, userspace or firmware with a session;
   fill M1/M3/M4.
5. Update the version-keyed handler or capability table only for the classified
   deltas; physical-confirm any behaviour change.

**Commands**

```bash
cd debug/risc-v-gsp-rm-vm

# unknown-call sweep and inventory diff between versions
python3 harness/control_discovery.py --out dumps/discovery-<VER> --seed <seed>

# reachability/trigger/args for a moved symbol
python3 harness/reach.py --target <symbol> --method auto --out dumps/reach-<symbol>

# symbol-attributed A/B
./greeneye.sh compare dumps/<old-run> dumps/<new-run> --symbols
```

### S4 — New CUDA userspace / toolkit interacting unexpectedly with the GSP

| Attribute | Detail |
|---|---|
| **Symptoms** | `cuInit` succeeds but context creation fails; a generic SUCCESS reply with incoherent consumed fields; a new or reordered control appears immediately before a failure; `libcuda` crashes (`lib[base+size] ip`); result mismatch in the A/B workload. |
| **Detection** | `cudatool.py info`/`delta` on the new `libcuda`; `fdematch` transfer measurement; corpus diff; ioctl trace diff; `ab.sh` result comparison. |
| **Evidence** | `delta` report (exports added/removed/moved, `.text` growth, build-id); `fdematch` table; wrapper re-resolution evidence; ioctl/control diff explained line by line; `cdt` output ending `ALL OK`; `ab.sh` `RESULT`/`STATUS` match; symbol DB with bias. |
| **Decision criteria** | From the delta: no export moves + small `.text` growth → patch (name map transfer, spot checks); new/removed exports → feature (decode the new surface first); many moves → reshuffle (full re-analysis; never transfer wrapper addresses). Anchor mismatch (new `libcuda` vs old firmware profile) → fix the pairing first. |
| **Rollback** | Keep `userspace/cuda/<version>/` trees in parallel; revert to the last green userspace version. Never overwrite an analysed version. |
| **Record in** | The version's decompilation map/`names.tsv`, the version-upgrade playbook notes, and the firmware-anchor note when the anchor also moves. |

**Response order**

1. Cheap triage first (minutes): `info` + `delta`; classify the port.
2. Measure name transfer before copying anything (`fdematch`).
3. Build the green-eye symbol DB for the new version so runtime addresses
   resolve; spot-check transferred addresses.
4. Confirm dynamically: `cdt` with the ioctl tracer, then `ab.sh all`, against
   the **same** emulated firmware profile (anchor discipline).
5. Diff the ioctl/control sequence old vs new; explain every change.
6. Only then touch the emulation, with physical confirmation per P4.

**Commands**

```bash
cd userspace/cuda
tools/cudatool.py info   <new>/run/libcuda.so.<new>
tools/cudatool.py delta  <old>/run/libcuda.so.<old> <new>/run/libcuda.so.<new> > <new>/run/delta.txt
tools/cudatool.py fdematch <old>/run/libcuda.so.<old> <new>/run/libcuda.so.<new> \
    --names <old>/annotations/names.tsv
tools/cudatool.py wrappers <new>/run/libcuda.so.<new>

cd ../../debug/risc-v-gsp-rm-vm
./greeneye.sh symbols export \
    --names ../../userspace/cuda/<new>/annotations/names.tsv \
    --index ../../userspace/cuda/<new>/run/analysis/decompiled/decompiled_index.tsv \
    --bias 0x100000 -o symbols-<new>.json
./greeneye.sh symbols resolve --db symbols-<new>.json 0x<transferred-addr>

cd ../..
tools/cuda-ab/ab.sh all      # host oracle + guest run + compare
```

### S5 — New GPU model or architecture (RTX 5090 / GB10-class, aarch64 hosts)

| Attribute | Detail |
|---|---|
| **Symptoms** | Probe first-touch addresses differ from the reference device; a firmware family/partition-manifest mismatch; ISA string differs (RV64GC vs RV64IMAC); aarch64 host reports `qemu-system-riscv64 not found`; PMU/SOE provisioning assumptions break. |
| **Detection** | `file`/`readelf` image shape; pairing check (family, FMC partition manifest, version string); probe first-touch diff; host tooling check (`./greeneye.sh info`); `rmtool` host-oracle inventory. |
| **Evidence** | Device profile fields (PCI IDs, BARs, FB geometry, engine inventory, interrupt topology, ROM quirks); pairing notes; reference vs candidate first-touch maps; portability matrix report; physical validation record before any support claim. |
| **Decision criteria** | Changed first-touch on profile registers → profile data, not logic. New firmware family/partition model → new family inputs, not a re-badged image. RV64GC or new CSRs → ISA/decoder work (S6). No physical validation → status stays Target/Experimental, never Supported. |
| **Rollback** | Keep the reference device/firmware directories untouched; revert the candidate profile; a support claim is only removed with a matrix change note. |
| **Record in** | [`06-portability/hardware-software-matrix.md`](06-portability/hardware-software-matrix.md) and the device profile notes; the evaluation ladder result for the combination. |

**Response order**

1. Collect the device profile; verify the firmware pairing before debugging
   anything (a mismatched pairing looks exactly like an emulation bug).
2. Confirm ISA/image shape (`file`, `readelf`); note FMC/manifest and boot
   model; diff PMU/SOE images.
3. Probe the new image and the reference with identical budgets; diff the
   first-touch maps.
4. Run the portability pipeline across reference and candidate.
5. Update profile data first; only add code when an interface genuinely
   differs.
6. Validate on the physical device under supervision before claiming support.

**Commands**

```bash
file <device-fwimage> && readelf -h <device-elf> && readelf -l <device-elf>
cd debug/risc-v-gsp-rm-vm
./greeneye.sh info
./greeneye.sh probe <device-fwimage> --symbols symbols-<device>.json --record
python3 harness/portability_matrix.py \
    --target <reference-fwimage> --target <device-fwimage> \
    --out dumps/matrix-<device>
./rmharness/harness.sh --session dumps/real-<device> --name before \
    -- rmtool engine list --format=json
# aarch64 host without qemu-system-riscv64:
#   install qemu-system-misc, or point GREENEYE_QEMU at the binary
```

### S6 — New instructions or ISA changes in NV-RISCV / falcon / GSP

| Attribute | Detail |
|---|---|
| **Symptoms** | Probe faults with an illegal/unimplemented instruction; the native backend diverges from QEMU on the same image; compressed/float extensions or new CSRs appear; a re-hosted build built with the old `-march` fails or misbehaves. |
| **Detection** | ISA string from `file`/`readelf`; probe fault cause/`desc`; QEMU-vs-native differential (`sim_diff.py`); decoder unit tests. |
| **Evidence** | Image shape (ISA, segments, entry); fault report with cause, `epc`/`epc_firmware` and access address; `PARITY: OK` transcript or the exact divergence; a fidelity-gap note when the native decoder cannot execute the new instruction. |
| **Decision criteria** | RV64IMAC on QEMU TCG → L0 high, proceed. RV64GC → QEMU supports it; update rehost `-march` and re-run parity. Native decoder gap → record as a fidelity gap; do not claim native coverage. Illegal instruction in QEMU itself → confirm ISA first, then escalate as a tool bug. |
| **Rollback** | Revert rehost `-march`/build flags; keep the previous probe/session as the reference. |
| **Record in** | The tool's accuracy/fidelity-gap list (`debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md` §8) and the image's version notes. |

**Response order**

1. Confirm the ISA and image shape; identify the exact faulting instruction and
   address.
2. Reproduce under QEMU (the reference execution engine) with a bounded probe.
3. Run the QEMU-vs-native differential; if it diverges, capture both sessions.
4. If the instruction is unsupported natively, record a fidelity gap with the
   session as evidence; keep QEMU as the execution engine for that image.
5. Update rehost build flags and re-run parity and the tool test suite.

**Commands**

```bash
cd debug/risc-v-gsp-rm-vm
./greeneye.sh probe <fwimage> --instructions <budget> --record
python3 examples/scripts/sim_diff.py        # requires QEMU; prints PARITY: OK on agreement
make test
```

### S7 — Significant VBIOS/ROM changes

| Attribute | Detail |
|---|---|
| **Symptoms** | The guest driver reads different device identity/VBIOS fields; probe-time register values move; ROM/VBIOS quirks change (laptop/Surface variants); a previously green probe hangs on a different register. |
| **Detection** | Guest `lspci -vvv -d 10de:` and init-trace diff; snapshot diff of the identity/static-config region; `rmtool` read-only `BUS_GET_PCI_INFO` oracle; ROM hash change. |
| **Evidence** | ROM hash and identity fields; before/after snapshot with a schema-attributed diff; pairing notes; the exact guest-observed values and the trace that consumed them. |
| **Decision criteria** | Changed fields consumed by the driver → profile/override work, physically confirmed. Fields not consumed → record and monitor. Anything that would require impersonating a licensed product → forbidden; present a coherent device instead. |
| **Rollback** | Revert the profile/override; previous ROM/profile stays archived with its hash. |
| **Record in** | The device profile and the matrix entry; the failure taxonomy if the symptom was misleading. |

**Response order**

1. Hash the ROM/VBIOS and collect the identity fields.
2. Capture a before/after snapshot of the firmware-side static-config region
   and diff it field by field.
3. Compare with the host oracle (read-only) and the guest's observed values.
4. Change only profile data that a consumer demonstrably uses; confirm on
   hardware.
5. Record unexplained fields as open, not as noise.

**Commands**

```bash
# guest
sudo lspci -vvv -d 10de:
# host oracle (read-only; verified queries from docs/17)
# build once from the repo root: make -C tools/rmtool
tools/rmtool/rmtool gpu info -d 0
tools/rmtool/rmtool engine list

cd debug/risc-v-gsp-rm-vm
# snapshot the identity/static-config region before and after
# (schema decode from the bundled/demo schema; real regions are version-keyed)
./greeneye.sh compare dumps/<old-run> dumps/<new-run> --symbols
```

### S8 — Encrypted/protected memory or protected instructions (measurement plan; non-claims)

| Attribute | Detail |
|---|---|
| **Symptoms** | The image consumes identity material that is provisioned encrypted (the PMU-delivered encrypted device identity in the real 0x59 campaign); protected-memory or protected-instruction paths are reached; a lenient run returns zeros where a real device returns provisioned values; boot halts at a secret-dependent step. |
| **Detection** | First-touch/device-map analysis; lenient-MMIO `mmio_log` and `mmio_unknown`; boot-chain shim assumptions in session metadata; probe fault reports identifying the consuming PC. |
| **Evidence** | First-touch map entry per protected address; `mmio_log` with PC and access; the exact access that consumes protected state; a written measurement plan and an explicit non-claims list. |
| **Decision criteria** | Secret-dependent branch without provisioned input → blocked; record as a fidelity gap and a device-model requirement, not a bug. Zero-value success under lenient MMIO → hypothesis only. Any request to extract, defeat or prove protected state → refuse (P5/P7). |
| **Rollback** | No product changes are made in this scenario; the only rollback is reverting experiment notes and keeping the map as data. |
| **Record in** | The fidelity-gap list, the device-map file (`greeneye-device-map/1`), and the H5 cross-controller plan; never in a security-claims section. |

**Measurement plan (what to measure, and what is not claimed)**

| Measure | Method | Non-claim |
|---|---|---|
| Which address/instruction consumes protected state | bounded probe + device map + `mmio_log` | not a statement about the real value |
| Whether execution continues when the value is absent/zero | lenient native run with `unknown_mmio="lenient"` | zeros are a hypothesis, not read-back semantics |
| Which subsystem provisions the value | cross-controller inventory (PMU/SEC2/ACR/SOE) and boot-order notes | not a model of the secure state machine |
| Whether the path is security-relevant | static map + recorded access ordering | not a side-channel, confidentiality or attack claim |

**Response order**

1. Classify: protected read, protected instruction, or a secret-dependent
   branch. Do not attempt extraction; do not patch anything.
2. Capture the first-touch map and the lenient `mmio_log` in a recorded
   session.
3. Write the measurement plan and the non-claims into the session note.
4. Escalate the missing provisioning as a device-model requirement (H2/H5),
   not as an emulation defect.
5. Stop. Physical work on protected paths requires the supervisor and a
   separate approved plan.

**Commands**

```bash
cd debug/risc-v-gsp-rm-vm
python3 -m greeneye.device_map demo --instructions 200000
python3 -m greeneye.device_map harvest dumps/<probe-session> --out dumps/<ver>-device-map.json
python3 -m greeneye.device_map show dumps/<ver>-device-map.json
# lenient native continuation (programmatic):
#   ge.launch(kernel=..., backend="native",
#             device_map="dumps/<ver>-device-map.json",
#             unknown_mmio="lenient")
#   session.target.mmio_log / session.target.mmio_unknown
```

### S9 — New microcontrollers and new inter-/intra-microcontroller APIs and DMA lines

| Attribute | Detail |
|---|---|
| **Symptoms** | The image stalls waiting on PMU/SEC2/ACR/SOE; falcon staging DMA does not land (the real failure mode: the SEC2 boot-loader block is not visible, the falcon does not halt); WPR2/security state differs from the successful path; new mailbox functions or DMA descriptors appear. |
| **Detection** | Probe first-touch; boot-chain shim assumptions recorded in session metadata; compare against the last successful boot path; device-map diffs; cross-controller event absence in sessions. |
| **Evidence** | First-touch addresses per controller window; mailbox/DMA access map; boot-order notes; shim assumptions in the session; the successful-path comparison; H2.2/H5 scope note. |
| **Decision criteria** | GSP-dependent stall → PMU/SEC2 stub work first (H2.2 entry). New mailbox/API surface → inventory it before modelling. Security-chain semantics → shim/documented assumption, never a claimed secure model. Unmodelled DMA line → device-map entry plus model work, not a product change. |
| **Rollback** | Revert any shim assumption change; the previous session/boot note stays as the reference. |
| **Record in** | The cross-controller inventory and boot-order notes; the device map; the H5/H2.2 roadmap scope. |

**Response order**

1. Inventory the controllers and DMA lines the image touches first
   (PMU, SEC2, ACR/SOE, booter).
2. Reproduce the stall with a bounded probe and record shim assumptions in the
   session.
3. Compare against the last successful path (e.g. WPR2 state, SEC2 staging
   visibility) and name the missing exchange.
4. Model the minimum stub that lets the image proceed; keep it documented
   data, not a silent special case.
5. Gate real security-chain semantics behind H2.2/H5; escalate any conclusion
   that would change the product.

**Commands**

```bash
cd debug/risc-v-gsp-rm-vm
./greeneye.sh probe <fwimage> --record
python3 -m greeneye.device_map harvest dumps/<probe-session> --out dumps/<ver>-device-map.json
python3 -m greeneye.device_map merge dumps/<ref>-device-map.json dumps/<ver>-device-map.json \
    --out dumps/merged-device-map.json
./greeneye.sh compare dumps/<last-good-run> dumps/<stall-run> --symbols
```

### S10 — Multi-version matrix operations and CI (RM 610 requires explicit human approval)

| Attribute | Detail |
|---|---|
| **Symptoms** | A new matrix column regresses an existing combination; CI drifts from the local run; someone proposes a 610-labelled input or command; the release bundle is incomplete. |
| **Detection** | `version_diff`/`portability_matrix` deltas; `ci_release --dry-run` plan inspection; the RM 610 gate refusal (`exit 3`); test transcripts. |
| **Evidence** | `version-diff.json`, `portability-report.json`, `release-report.json`, `bundle-manifest.json`, seeded `findings.jsonl`, test transcripts, approval record for 610. |
| **Decision criteria** | Dry-run clean before a real run. A step that raises is `error`, not skipped. Self-comparison must be `all_zero`; seeded replays byte-identical. **610: no acquisition, analysis, emulation edit, physical work or support claim before a named human approves in writing; an open-ended approval is invalid.** |
| **Rollback** | Extend the matrix, never replace a column; revert the candidate commit and keep the previous anchor/tag/driver; a 610 attempt parks without touching 570.144. |
| **Record in** | The port log/approval thread (610), the release bundle directory, and the matrix with its test date and rung. |

**Response order**

1. Run the release rehearsal on current-version labels first; inspect the
   dry-run plan.
2. Run `version_diff` and `portability_matrix` with fixed budgets; require the
   self-comparison to be zero.
3. Run the mutation smoke with a pinned seed; verify replay.
4. Assemble and hash the bundle; attach it to the PR.
5. For 610: stop at the gate; obtain the approval message; only then acquire
   inputs and repeat the pipeline with 610 labels.

**Commands**

```bash
cd debug/risc-v-gsp-rm-vm

# rehearsal first: dry-run never opens a target and creates no output dir
python3 harness/ci_release.py --old <old> --new <new> --out dumps/release-<ver> \
    --old-label <OLD> --new-label <NEW> --dry-run

# real pipeline (dry-run clean)
python3 harness/ci_release.py --old <old> --new <new> --out dumps/release-<ver> \
    --old-label <OLD> --new-label <NEW> \
    --old-symbols symbols-<OLD>.json --new-symbols symbols-<NEW>.json \
    --mutation-kernel firmware/uproc_demo/build/uproc_demo.elf \
    --mutation-iterations 8 --mutation-seed 7

# RM 610 gate: refused by default; explicit approval token required
python3 harness/version_diff.py --old <old> --new <new> --out dumps/vdiff-610 \
    --old-label RM-610 --new-label 570.144
# exit 3 unless:  --approval-token RM610-APPROVED   or   GREENEYE_RM610_APPROVED=1
```

## 6. Evidence-bundle requirements

A bundle is the minimum attachable proof for a version/device/architecture
triage or a product change. It combines the docs/07 evidence checklist, the
docs/14 §6 adversarial pack and the docs/21 release tree.

```
<out>/
  bundle-manifest.json            sha256 + bytes of every file below
  release-report.json             pipeline summary (steps, gate, labels)
  version-diff/                   version-diff.json|.md, symbols/, sessions/, compare/
  portability-matrix/             portability-report.json|.md, symbols/, sessions/, compare/
  mutation-smoke/                 findings.jsonl, summary.md, sessions/
```

Checklist — every box must be true or explicitly waived with a reason:

* [ ] firmware identity: SHA-256, version string, family and pairing notes;
* [ ] input provenance: channel, date, and the exact file names analysed;
* [ ] symbol DB(s): hash, source (ELF/Ghidra/JSON), `bias` and provenance;
* [ ] baseline and candidate session directories (or their SHA-256 list);
* [ ] exact stimulus (script/spec plus mutation values) and budgets;
* [ ] `compare` report and relevant `analyze` reports;
* [ ] M1–M6 table with the reference-capture method and L2/L3 limits stated;
* [ ] `greeneye_hits.tsv` and any Ghidra annotations produced;
* [ ] `bundle-manifest.json` written last, covering the final reports;
* [ ] unexplained-delta list (empty is a claim, not a default);
* [ ] decision and product action (or explicit "no change") recorded;
* [ ] for 610: approval ID, approver, date and scope limits.

Fidelity statement template (copy into every report):

```text
Fidelity: L0 (instruction semantics) and L1 (memory) exercised; L2 (timing,
concurrency) NOT modelled; L3 (device environment) absent for <registers>.
Reference capture: <hardware / vfio-mdev / host oracle / none>.
Physical reproduction required before product change: <yes/no + why>.
```

## 7. Fault and escalation tree

```text
Failure / new input observed
|
+- Is the input hashed, versioned and paired?  NO -> STOP: acquire/hash/pair first (S1)
|
+- Did it come from RM 610 or a 610-labelled path?  YES -> STOP: hard human gate (S10)
|
+- Is a physical GPU / driver bind / mdev involved?
|     YES -> STOP: supervised physical cycle + reboot approval (P7); never unattended
|
+- Does the question need timing, interrupts, protected state or real device
|  read-back semantics (L2/L3)?
|     YES -> simulator evidence is a hypothesis only (P3/P4/P5); record the
|            fidelity gap; product change requires physical reproduction
|
+- Does it reproduce under a fixed stimulus in green-eye?
|     NO  -> widen the stimulus / capture the real trace before theorising;
|            absence of a hit is not proof of dead code
|     YES -> classify with the S1 decision table (code changed vs behaviour moved)
|
+- Would a fix change emulation behaviour?
|     YES -> physical/vfio-mdev reproduction + evidence bundle before landing
|     NO  -> docs/profile/test-only change with the bundle attached
|
+- Does any vendor component need modification?
      YES -> forbidden (P2): redesign to emulate the interface instead
```

Stop conditions (do not proceed, do not "temporarily" work around):

1. Unverified pairing or unhashed input.
2. RM 610 without written approval.
3. A physical cycle without the supervisor and fail-safe plan.
4. A request to patch a vendor component or a guest.
5. A security/confidentiality claim based on simulator state.
6. A product change with no physical reproduction and no bundle.

## 8. When NOT to use the tooling

| Situation | Why not | Use instead |
|---|---|---|
| You need silicon truth (timing, interrupts, real register read-back) | L2/L3 are absent by design | supervised physical cycle + reference traces |
| You are about to mutate or inject against hardware, a guest, a kernel module or an existing gdbstub | Tooling is simulator-only; hardware writes are prohibited | bounded QEMU/native harness with recorded sessions |
| You need to defeat, extract or prove a security boundary | The simulator is not a confidentiality boundary | stop; escalate as a design question |
| You need a product fix "now" without evidence | Violates P3/P4 | evidence bundle first; the anchor and matrix can wait |
| The input has not been hashed/versioned/paired | Every later claim is contaminated | S1 steps 1–3 |
| RM 610 has no written approval | Hard gate | request approval with the readiness checklist |
| The physical GPU is held by another job | Shared-oracle safety and fail-safe discipline | host-side read-only oracle (`rmtool gpu list`, `harness.sh`) |
| You need type recovery or decompilation | green-eye is dynamic ground truth, not a decompiler | Ghidra corpus and symbol tooling |
| You need guest-driver behaviour | green-eye observes firmware, not the guest | `tools/guest-diag/`, ioctl tracer |
| A one-line question can be answered by `analyze` on an existing session | Do not produce new runs for no reason | reuse recorded sessions and `compare` |

## 9. One-page quick-reference decision tree

| Symptom (first sign) | First command | Decision | Scenario |
|---|---|---|---|
| New image/version appears | `./greeneye.sh symbols export ...` + `probe --record` | hash+pair first, then version diff | S1 |
| OpenRM build breaks, `Unknown symbol` | `./scripts/build_nvidia_openrm.sh`; `dmesg` | glue-only vs interface change | S2 |
| Control/ioctl inventory shifts | `python3 harness/control_discovery.py` | moved vs new logic | S3 |
| `libcuda` bump / context failure | `tools/cudatool.py delta` + `fdematch` | patch / feature / reshuffle | S4 |
| New device/architecture | `file`/`readelf`, `probe` | profile vs logic; physical validation for support | S5 |
| Illegal/unknown instruction | `./greeneye.sh probe --instructions N` | ISA/decoder; fidelity gap if native-only | S6 |
| Identity/ROM fields differ | guest `lspci -vvv`, snapshot diff | consumed field → profile; else record | S7 |
| Protected/encrypted state blocks | device map + lenient `mmio_log` | record fidelity gap; no extraction; no claims | S8 |
| Early stall on PMU/SEC2/ACR/SOE | `probe --record` + device map | stub the missing exchange; gate security semantics | S9 |
| Matrix/CI drift; 610 mentioned | `ci_release --dry-run` | dry-run clean; 610 hard gate | S10 |

Golden rule: **freeze, hash, pair, reproduce, classify, evidence, then act** —
and never let the last step touch anything we do not own.

## 10. Related documentation

Green-eye suite (experimental, in development):

* [green-eye user guide](05-operations/04-green-eye-user-guide.md) — what it
  is, quickstart, tool chooser and troubleshooting;
* [GSP in a RISC-V VM](05-operations/03-risc-v-gsp-rm-vm.md) — project-level
  purpose, status and limits;
* [`debug/risc-v-gsp-rm-vm/docs/00-OVERVIEW.md`](../debug/risc-v-gsp-rm-vm/docs/00-OVERVIEW.md)
  — tool status and doc map;
* [`debug/risc-v-gsp-rm-vm/GREEN-EYE-ROADMAP.md`](../debug/risc-v-gsp-rm-vm/GREEN-EYE-ROADMAP.md)
  — H1–H7 program;
* [`debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md`](../debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md)
  — fidelity levels and the M1–M6 contract;
* [`debug/risc-v-gsp-rm-vm/docs/07-PORTABILITY-PLAYBOOKS.md`](../debug/risc-v-gsp-rm-vm/docs/07-PORTABILITY-PLAYBOOKS.md)
  — P1–P4 porting playbooks;
* [`debug/risc-v-gsp-rm-vm/docs/14-ADVERSARIAL-INTEROP-GUIDE.md`](../debug/risc-v-gsp-rm-vm/docs/14-ADVERSARIAL-INTEROP-GUIDE.md)
  — adversarial navigation and the evidence pack;
* [`debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md`](../debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md)
  — release triage and bundles;
* [`debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md`](../debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md)
  — the 610 gate and approval format;
* [`debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md`](../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md)
  — tool audit, symptom-to-fix table and provenance rule;
* [`debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md`](../debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md)
  — sweep/device-port scaffolds (unadmitted pilots, dry-run default);
* [`debug/risc-v-gsp-rm-vm/docs/29-PACKAGING-AND-TRIALING.md`](../debug/risc-v-gsp-rm-vm/docs/29-PACKAGING-AND-TRIALING.md)
  — trial simulation packages (unadmitted pilot).

Project documentation:

* [Debugging and maintenance workflows](05-operations/02-debugging-and-maintenance-workflows.md)
  — captures, failure taxonomy, version discipline;
* [Performance, correctness, security](05-operations/01-performance-correctness-security.md)
  — the evaluation ladder;
* [Hardware and software matrix](06-portability/hardware-software-matrix.md)
  — support definitions and the portability pipeline;
* [Practical challenges and adversarial interoperability](04-implementation/02-practical-challenges-and-adversarial-interop.md)
  — the no-patch rule;
* [Developer onboarding and standards](07-developers/onboarding-and-standards.md)
  — coding rules and the landing checklist;
* [Documentation front door](README.md) — the full reading order;
* [Engineering operations](eng-ops.md) — tool admission, workflows and
  retirement (the cross-cutting engineering-operations document);
* CUDA playbook:
  [`userspace/cuda/docs/07-version-upgrade-playbook.md`](../userspace/cuda/docs/07-version-upgrade-playbook.md)
  and [`userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md).

<!-- change (2026-10-02): fix the S4 fdematch invocation (two libraries plus
--names, per tools/cudatool.py), add a session/disk-growth bound to §3.2, and
link docs 28/29 from §10; scope: RUNBOOK §3.2, §5 S4, §10 only; benefit: the
copied triage command runs as written and the retention rule is explicit. -->