summaryrefslogtreecommitdiffstats
path: root/docs/RUNBOOK.md
diff options
context:
space:
mode:
authorArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-02 11:28:49 -0400
committerArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-02 11:28:49 -0400
commitc17da06d6cd8b9c69449904637bf37729a962989 (patch)
treeb51618fd63b776cd545f64fd40729af6154996b2 /docs/RUNBOOK.md
parent002e3d7d13c414ea874abbee8d82803dd3a72fe5 (diff)
downloadopen-iov-c17da06d6cd8b9c69449904637bf37729a962989.tar.gz
open-iov-c17da06d6cd8b9c69449904637bf37729a962989.zip
docs: engineering-operations playbook and project-wide status accuracy pass
- docs/eng-ops.md gains Part II, live physical-backend engineering operations: hard rules, the full module-parameter matrix, the routine supervised cycle and clean-release checklist, the debugfs inventory, how to read gsp-phys-state, a debugging scenario catalogue (CTXNOTVALID/promote wedge, fail-safe WPR2, unload hangs, guest wedges, MMU/PTE and PBDMA faults, relay symptoms), a hazard register (crash causes and debugging side effects), the physical-backend OODA loop, and the work-context map with porting flows - architecture, implementation, operations, orientation, getting-started, portability, ecosystem and reference docs corrected to the current state: CUDA driver-API smoke green, CE-class work forwarding and engine-authored releases plus fault relay in place (default off), GR execution blocked on engine-context validity, the promote wedge and designed locked fail-safe, and P5 wave-2 subsumed - internal-reference cleanup in tracked docs (no analysis-source or local-archive references); historical architecture document bannered - README reading list and STATE-CURRENT frontier refreshed; new pages (runbook, green-eye user guide, porting sequence, porting/packaging notes) linked from the tree Link check: scripts/check-docs-links.py passes (56 files, no broken links).
Diffstat (limited to 'docs/RUNBOOK.md')
-rw-r--r--docs/RUNBOOK.md744
1 files changed, 744 insertions, 0 deletions
diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md
new file mode 100644
index 0000000..79c1735
--- /dev/null
+++ b/docs/RUNBOOK.md
@@ -0,0 +1,744 @@
+# Open-IOV Defensive Operations Runbook
+
+> The per-incident book for operating an independent IP block (Open-IOV)
+> against ecosystems that change on their own schedule. NVIDIA ships new GSP
+> firmware, kernel modules, CUDA userspace, devices and ISA revisions without
+> coordinating with us; any of them can break the emulation while the others
+> stand still. This runbook is the ordered response.
+>
+> **Related:** [green-eye user guide](05-operations/04-green-eye-user-guide.md)
+> · [GSP in a RISC-V VM (experimental)](05-operations/03-risc-v-gsp-rm-vm.md)
+> · [Debugging and maintenance workflows](05-operations/02-debugging-and-maintenance-workflows.md)
+> · [Hardware and software matrix](06-portability/hardware-software-matrix.md)
+> · [Adversarial interoperability](04-implementation/02-practical-challenges-and-adversarial-interop.md)
+> · [Developer onboarding and standards](07-developers/onboarding-and-standards.md)
+> · [Engineering operations](eng-ops.md)
+>
+> **# TOOL IN DEVELOPMENT - NOT READY YET!** The green-eye tooling cited here
+> is experimental; its interfaces and file formats may change. Nothing in this
+> runbook authorises a product change, a hardware write or work on RM 610.
+
+## 0. How to use this runbook
+
+Audience: maintainers, release engineers and on-call operators of the Open-IOV
+emulation and its evidence pipeline.
+
+Rules for reading under pressure:
+
+1. Find the symptom in §5 (scenario catalogue) or §9 (one-page decision tree).
+2. Follow the **response order** exactly; do not skip the pairing/hash check.
+3. Before any action that could change `open-iov-vfio`, complete the evidence
+ bundle (§6) and pass the escalation gates (§7).
+4. Record the outcome where the scenario says to record it — a finding without
+ a record is lost work.
+5. If an action is not in this runbook and not in the linked project docs,
+ treat it as a candidate for a new runbook entry, not as improvisation.
+
+Command conventions: `open-iov/` is the repository root. Commands run from the
+repository root unless a block starts with `cd`. Angle-bracket values
+(`<VER>`, `<session>`, `<image>`) are supplied by the task; never hardcode
+private paths, host names or serial numbers into tracked documents.
+
+## 1. Operating principles and hard rules
+
+These apply to every scenario; where a scenario conflicts with them, the rules
+win.
+
+| # | Principle | Consequence |
+|---|---|---|
+| P1 | **Anchor discipline.** The lock-step is *guest driver ↔ emulated firmware profile*. The host distribution driver is the A/B oracle, not the anchor. | Never evaluate a new `libcuda` against an old emulated firmware profile and call the mismatch an emulation bug. |
+| P2 | **Never patch what we do not own.** Guest driver, host driver, GPU firmware, hypervisor, device model and guest OS stay unmodified. | Product changes live in our module, userspace, tools, configuration, tests and docs only. |
+| P3 | **Evidence over anecdote.** Every conclusion needs at least an A/B pair and an explicit fidelity level (L0–L3). | A single run, a static reading or a simulator-only delta cannot land a product change. |
+| P4 | **Physical confirmation for behaviour changes.** A simulator observation is evidence about the firmware, not about silicon. | Reproduction on a supervised physical/vfio-mdev cycle is required before a change that alters emulation behaviour. |
+| P5 | **The simulator is a study environment, never a security boundary.** | No confidentiality, side-channel or security claims; never use injection to defeat a hardware security boundary. |
+| P6 | **Development-only tooling.** green-eye is never shipped, never a dependency of the emulation, never in a guest. | Keep the in-development banner on tool docs until the acceptance list passes. |
+| P7 | **No unattended physical-GPU cycles.** Physical work is supervised, with fail-safe release and explicit reboot approval. | Stop and ask; never bind/unbind drivers or claim the GPU as an experiment. |
+| P8 | **RM 610 is a hard human gate.** Until a named human approves in writing, no acquisition, analysis, emulation edit, physical work or support claim for 610. | The gate is enforced in code by `version_diff`/`ci_release`; do not work around it. |
+| P9 | **Fail closed, record honestly.** Unverified host info, unpaired firmware, unexplained deltas and absent references stay open. | "No unexplained deltas" is a claim that must be earned, not a default. |
+
+## 2. Maintainer roles and responsibilities
+
+One name per role at any time; handovers are a written change note in the
+affected doc or PR.
+
+| Role | Owns | Key duties | Gates they hold |
+|---|---|---|---|
+| **Anchor maintainer** | guest driver ↔ emulated firmware anchor; version surface | keeps the anchor current; approves anchor bumps; keeps `config/versions/*.yaml` labelled as design targets until read by code | any change to the anchor |
+| **Firmware triage maintainer** | versioned firmware directories; symbol DBs; probe/version-diff runs | acquires and hashes new images; builds symbol DBs with provenance; runs the P1 loop; owns the pairing decision | pairing verdict; "new logic" vs "behaviour moved" |
+| **green-eye tooling maintainer** | `debug/risc-v-gsp-rm-vm/`; test suite; session format | keeps `make test` green; guards format/API stability; keeps the marker and doc map honest; maintains golden sessions | tool status claims; removal of the in-development marker |
+| **CI/release maintainer** | `harness/version_diff.py`, `harness/ci_release.py`, `greeneye.yml`, goldens | runs dry-run before real runs; hashes evidence bundles; diffs seeded corpora; keeps CI paths and budgets bounded | release triage sign-off; RM 610 gate enforcement |
+| **Physical-cycle supervisor** | supervised physical/vfio-mdev cycles | enforces fail-safe release, D0/D3cold handling and reboot approvals; captures reference traces | all physical execution; reboot requests |
+| **Evidence archivist** | session and bundle retention | keeps SHA-256 inventories; enforces provenance (no private paths, no unsanitised host data); applies retention/prune policy | publication of evidence |
+| **Docs steward** | this tree | enforces change notes, link checks and doc-map updates; retires stale claims | docs are part of "done" |
+
+Escalation order for an ambiguous incident: triage maintainer → anchor
+maintainer → physical-cycle supervisor (if hardware is involved) → project
+maintainer channels. Escalate with a reproduction and an evidence pack, never
+with a summary alone.
+
+## 3. Routine maintenance
+
+### 3.1 Golden sessions and determinism
+
+| Item | Cadence | Procedure | Pass condition |
+|---|---|---|---|
+| Demo golden session | every tool change and weekly | `./greeneye.sh demo` and compare against the archived session/expected events | UART banner and control-block liveness present; hook/watch/breakpoint sequence coherent |
+| Portability self-comparison | per PR touching `harness/`, `greeneye/`, `sim/` | `python3 harness/portability_matrix.py --target firmware/uproc_demo/build/uproc_demo.elf --target firmware/uproc_demo/build/uproc_demo.elf --out dumps/matrix-selfcheck` | `all_zero` deltas in `portability-report.json` |
+| Mutation replay | per PR touching mutation/harness code | run `mutation_harness.py` twice with the same seed and budgets | `findings.jsonl` byte-identical |
+| Version-diff self-check | per release-triage change | `version_diff.py` old == new | zero symbol/probe/behavior deltas |
+| Host oracle capture | per anchor change | `./rmharness/harness.sh --session dumps/real-<anchor> --name before -- rmtool engine list --format=json` (repeat for `mem stats`) | capture recorded with sizes and SHA-256 |
+
+Golden artifacts are append-only history: never edit a stored golden session
+(§3.2 uses SHA-256). Re-baseline by creating a new session with a change note.
+
+### 3.2 Session hygiene
+
+* Sessions are directories (`dumps/<stamp>-<name>/`) with `session.json`,
+ `events.jsonl`, `snapshots/`, `blobs/` and `summary.md`. Keep them together;
+ a copied `events.jsonl` without its `session.json` is not evidence.
+* Name sessions for the question asked, not the host or the operator.
+* Keep sessions bounded: fixed stimulus, fixed budgets, region-scoped
+ snapshots; a multi-gigabyte trace is a run that was not scoped.
+* Publish a SHA-256 list (or the `bundle-manifest.json`) with any session that
+ leaves the machine.
+* Write findings into `note` events inside the session (self-describing packs);
+ never rely on a chat log to explain a recording.
+* Apply the provenance rule everywhere: tracked docs describe components and
+ behaviour; sessions contain no host-specific paths in their metadata.
+* Bound growth deliberately: `dumps/` is gitignored scratch, so keep every
+ golden session and release bundle, publish the SHA-256 list before deleting
+ anything, and prune only scratch sessions past the review window — a
+ multi-gigabyte session is a scoping failure (fixed stimulus, fixed budgets,
+ region-scoped snapshots), not a storage problem to solve with new tooling.
+
+### 3.3 Toolchain updates
+
+| Dependency | Check | Action |
+|---|---|---|
+| QEMU | `./greeneye.sh info`; run `make test` | install/renew `qemu-system-riscv64` (Ubuntu `qemu-system-misc`); set `GREENEYE_QEMU` if the binary is elsewhere |
+| QEMU plugin API | `make -C qemu` after a QEMU upgrade | rebuild `qemu/greeneye-trace.so` against the vendored `qemu-plugin.h`; re-run a trace smoke |
+| Python | `python3 --version` (3.10+; stdlib only) | no pip packages required; keep plugins/harnesses stdlib-only |
+| clang/lld | `make demo`, `make sdk` | reinstall toolchain if the demo or SDK stops building |
+| Test suite | `python3 -m unittest discover -s tests -t .` | all green; skips for absent QEMU/demo explained, not ignored |
+| Native parity | `python3 examples/scripts/sim_diff.py` | `PARITY: OK` (or a clean skip when QEMU is absent) |
+| Docs | `python3 scripts/check-docs-links.py` from `open-iov/` | no broken relative links |
+
+### 3.4 Documentation and marker discipline
+
+* The **# TOOL IN DEVELOPMENT - NOT READY YET!** banner stays on tool docs
+ until the acceptance list in
+ `debug/risc-v-gsp-rm-vm/docs/00-OVERVIEW.md` passes; removal is a reviewed
+ change, not an edit.
+* Every behaviour change updates the doc that describes it (docs are part of
+ the definition of done); planned work is labelled `planned (docs/xx)`.
+* Status tables must match the code on the current commit (commands, flags,
+ test counts); stale claims are bugs.
+* Keep the doc map complete when adding documents; keep change notes
+ (`<!-- change (YYYY-MM-DD): purpose; scope; benefit -->`) at edited
+ locations.
+* Record reproducible discoveries in the failure taxonomy of
+ [`05-operations/02-debugging-and-maintenance-workflows.md`](05-operations/02-debugging-and-maintenance-workflows.md)
+ §5, not only in a session directory.
+* Tool admission, extension, trialing and retirement are governed by
+ [`eng-ops.md`](eng-ops.md); procedural detail that is not incident response
+ belongs there, not here.
+
+## 4. The OODA ordering for a hostile update
+
+"Hostile" here means *unplanned firmware change plus deliberate opacity* —
+new images on a hidden schedule, stripped and signed, hardware-coupled,
+possibly with anti-analysis or decoy behaviour. It does not assume a malicious
+actor, and it does not justify shortcuts.
+
+| Stage | Order | Actions | Exit criterion |
+|---|---|---|---|
+| **Observe** | 1 | Freeze the anchor; acquire the new input through the approved channel; record hash, version string and family; do not edit the emulation | Input is versioned and hashed in a parallel directory |
+| | 2 | Capture the baseline: real-backend reference traces for the fixed workload (or the archived previous capture) | Baseline exists with a provenance note |
+| | 3 | Record the symptom in a session note and a reproduction | Reproduction is repeatable and bounded |
+| **Orient** | 4 | Static map: symbol DB/corpus diff per version | Inventory deltas listed |
+| | 5 | Dynamic map: probe first-touch, then fixed-stimulus sessions on both versions | `compare` report with hit/watch/write deltas |
+| | 6 | Classify each delta with the decision table (code changed vs behaviour moved) | Every delta has a class or is explicitly open |
+| **Decide** | 7 | Fill the M1–M6 skeleton with the reference-capture method; list unexplained deltas | Metric table and open list attached |
+| | 8 | Choose: no change, version-keyed handler, profile data, or fidelity gap | Decision names the evidence that drove it |
+| | 9 | If the decision changes emulation behaviour, require physical reproduction; if it touches RM 610, stop at the gate | Gate passed or work parked |
+| **Act** | 10 | Implement in `open-iov-vfio` with the evidence bundle linked; never in the simulator | PR carries the bundle and the metric table |
+| | 11 | Re-run the identical stimulus; require profiles to improve or stay equal; re-run the guest ladder | No rung regression; CI green |
+| | 12 | Archive the bundle and update the matrix/version docs only where earned | Recorded and reversible |
+
+Ordering rules that keep this honest:
+
+1. Never orient before the input is hashed and paired.
+2. Never decide from a single session (P3).
+3. Never act on the product from simulator-only evidence (P4).
+4. Never let "the new version" become the anchor implicitly; the anchor moves
+ by explicit decision.
+
+## 5. Scenario catalogue
+
+Each scenario follows the same template: symptoms, detection, evidence,
+response order, tools/commands, decision criteria, rollback, where to record.
+Scenarios are incidents to run, not roadmap claims.
+
+### S1 — New GSP/RM firmware release
+
+| Attribute | Detail |
+|---|---|
+| **Symptoms** | Init fails with a generic status (the real 0x59 chase: `GSP_INIT_DONE rpc_result=0x59`), handler entries or reply shapes shift, a previously green guest step now fails, control inventory gains/removes IDs. |
+| **Detection** | Firmware hash/version string changed; probe first-divergence address changed; symbol/corpus inventory delta; fixed-stimulus behaviour delta. |
+| **Evidence** | Firmware SHA-256 + version string + pairing; old/new symbol DBs with provenance and bias; probe sessions; version-diff report; behaviour `compare`; M1–M6 skeleton; unexplained-delta list. |
+| **Decision criteria** | Pairing mismatch → stop. Code changed and live profile changed → prioritise new logic. Code unchanged and profile changed → behaviour moved (state/ordering/environment fix). Symbol never hit either side → coverage gap, widen the stimulus first. |
+| **Rollback** | The previous version's directory and configuration stay untouched; roll back by returning to the old anchor commit/tag and the old guest driver. A forward attempt never destroys the working setup. |
+| **Record in** | The version's `firmware/<SERIES>-GSP/<VER>/` notes, the PR body (link the bundle), and the matrix entry only once support is earned. |
+
+**Response order**
+
+1. Freeze the anchor and the workload; do not edit emulation code.
+2. Acquire the image through the approved channel; lay down a new versioned
+ directory; hash it.
+3. Build a symbol DB for the new image and the previous one.
+4. Probe both images with identical settings; diff the first-touch maps.
+5. Run the static triage (dispatcher/handler/control/corpus diff).
+6. Run the same stimulus on both (re-hosted or demo analog) and compare.
+7. Fill M1–M6 with the reference-capture method; classify every delta.
+8. Escalate any product-behaviour change to a physical reproduction.
+9. Land the change (or record the fidelity gap) with the evidence bundle; add
+ a version-keyed handler only where the diff requires it.
+
+**Commands**
+
+```bash
+cd debug/risc-v-gsp-rm-vm
+
+# 1) symbol DBs, one per version
+./greeneye.sh symbols export --index <export-VER.tsv> --bias 0 -o symbols-<VER>.json
+
+# 2) bounded probe of each image (records a session)
+./greeneye.sh probe <gsp-<VER>.fwimage.bin> --symbols symbols-<VER>.json --record
+
+# 3) full old-vs-new triage (shape, symbols, corpus, probe, behavior, M1-M6)
+python3 harness/version_diff.py \
+ --old <gsp-<OLD>.fwimage.bin> --new <gsp-<NEW>.fwimage.bin> \
+ --old-symbols symbols-<OLD>.json --new-symbols symbols-<NEW>.json \
+ --old-label <OLD> --new-label <NEW> \
+ --out dumps/vdiff-<OLD>-<NEW>
+
+# 4) behaviour compare and canonical report
+./greeneye.sh compare dumps/<old-run> dumps/<new-run> --symbols
+./greeneye.sh analyze dumps/<new-run> --symbols symbols-<NEW>.json
+```
+
+### S2 — New kernel driver / open-kernel-modules release
+
+| Attribute | Detail |
+|---|---|
+| **Symptoms** | The optional OpenRM build fails with `Unknown symbol`; exports move or disappear between tags; the pinned build glue no longer matches the installed modules; `modinfo`/`dmesg` report version drift. |
+| **Detection** | `./scripts/build_nvidia_openrm.sh` failure; `dmesg | grep 'Unknown symbol'`; `kernel-open` tree diff for the two tags; SBOM tag mismatch. |
+| **Evidence** | Old/new tag identifiers; build transcript; the `Unknown symbol` list; a recorded classification of each moved export; confirmation that the runtime path still uses the stock distribution driver. |
+| **Decision criteria** | If only build glue breaks and the runtime path is unaffected → repair or re-pin the glue. If an exported interface used by the emulation changed → treat as S1/S3 (functionality moved). If the distribution driver moved → it is the oracle; do not "upgrade" the anchor to match it. |
+| **Rollback** | Restore the previously pinned module tag and rebuild; revert the glue commit. |
+| **Record in** | `SBOM.md` tag updates, the build glue commit message, and the failure taxonomy if it is a reusable lesson. |
+
+**Response order**
+
+1. Verify which driver is actually in use (guest stock driver vs optional host
+ OpenRM modules) before debugging.
+2. Reproduce the build failure with a clean tree; capture the full error.
+3. Diff the `kernel-open` trees for the old and new tags (publish only the
+ symbol/interface delta).
+4. Classify: build-glue-only, symbol rename, interface removal, or new
+ functionality.
+5. Repair build glue if and only if the runtime contract is unchanged;
+ otherwise route to S3.
+6. Re-run `make kernel`, the host test suite, and the optional OpenRM build.
+
+**Commands**
+
+```bash
+# optional OpenRM build (build glue; never a runtime dependency)
+./scripts/build_nvidia_openrm.sh
+make kernel
+./scripts/stack_manager.sh test
+
+# evidence of the failure
+sudo dmesg -T | grep -i 'Unknown symbol'
+
+# runtime path sanity: guest remains stock
+# (guest: stock nvidia.ko; no Open-IOV code in the guest, ever)
+```
+
+### S3 — Functionality moving between kernel and GSP, or between userspace versions
+
+| Attribute | Detail |
+|---|---|
+| **Symptoms** | A control that used to be answered by the guest kernel is now requested from the firmware (or vice versa); new RPCs/controls appear; reply fields that used to be populated are ignored; a workload passes on one component mix and fails on another. |
+| **Detection** | Ioctl trace changes; control sweep finds new/removed IDs; dispatcher hit profile changes while static code is unchanged; `compare` shows hit/watch deltas with no source delta. |
+| **Evidence** | Recorded ioctl/control diff; `discovery-report.json` old vs new; `reach.py` reports for moved targets; session A/B pair; M1/M3/M4 rows with the reference method. |
+| **Decision criteria** | Same code and new hit profile → behaviour moved to a new caller/state; new handler ID → new logic; `default-path` result change → fail-closed/default behaviour changed; coverage gap → widen the sweep before deciding. |
+| **Rollback** | Keep both component versions in parallel directories; revert the mix to the last green combination; never convert a version-keyed handler into a version-blind one. |
+| **Record in** | The version's control-inventory notes, the interface docs for the exact version, and the PR that adds/updates the version-keyed handler. |
+
+**Response order**
+
+1. Capture the ioctl/control stream on the failing mix and on the last green
+ mix.
+2. Sweep the call surface (control discovery) and diff the inventory.
+3. Use `reach.py` to prove each moved target is reachable under the exact
+ stimulus and to record its trigger.
+4. Attribute every delta to kernel, userspace or firmware with a session;
+ fill M1/M3/M4.
+5. Update the version-keyed handler or capability table only for the classified
+ deltas; physical-confirm any behaviour change.
+
+**Commands**
+
+```bash
+cd debug/risc-v-gsp-rm-vm
+
+# unknown-call sweep and inventory diff between versions
+python3 harness/control_discovery.py --out dumps/discovery-<VER> --seed <seed>
+
+# reachability/trigger/args for a moved symbol
+python3 harness/reach.py --target <symbol> --method auto --out dumps/reach-<symbol>
+
+# symbol-attributed A/B
+./greeneye.sh compare dumps/<old-run> dumps/<new-run> --symbols
+```
+
+### S4 — New CUDA userspace / toolkit interacting unexpectedly with the GSP
+
+| Attribute | Detail |
+|---|---|
+| **Symptoms** | `cuInit` succeeds but context creation fails; a generic SUCCESS reply with incoherent consumed fields; a new or reordered control appears immediately before a failure; `libcuda` crashes (`lib[base+size] ip`); result mismatch in the A/B workload. |
+| **Detection** | `cudatool.py info`/`delta` on the new `libcuda`; `fdematch` transfer measurement; corpus diff; ioctl trace diff; `ab.sh` result comparison. |
+| **Evidence** | `delta` report (exports added/removed/moved, `.text` growth, build-id); `fdematch` table; wrapper re-resolution evidence; ioctl/control diff explained line by line; `cdt` output ending `ALL OK`; `ab.sh` `RESULT`/`STATUS` match; symbol DB with bias. |
+| **Decision criteria** | From the delta: no export moves + small `.text` growth → patch (name map transfer, spot checks); new/removed exports → feature (decode the new surface first); many moves → reshuffle (full re-analysis; never transfer wrapper addresses). Anchor mismatch (new `libcuda` vs old firmware profile) → fix the pairing first. |
+| **Rollback** | Keep `userspace/cuda/<version>/` trees in parallel; revert to the last green userspace version. Never overwrite an analysed version. |
+| **Record in** | The version's decompilation map/`names.tsv`, the version-upgrade playbook notes, and the firmware-anchor note when the anchor also moves. |
+
+**Response order**
+
+1. Cheap triage first (minutes): `info` + `delta`; classify the port.
+2. Measure name transfer before copying anything (`fdematch`).
+3. Build the green-eye symbol DB for the new version so runtime addresses
+ resolve; spot-check transferred addresses.
+4. Confirm dynamically: `cdt` with the ioctl tracer, then `ab.sh all`, against
+ the **same** emulated firmware profile (anchor discipline).
+5. Diff the ioctl/control sequence old vs new; explain every change.
+6. Only then touch the emulation, with physical confirmation per P4.
+
+**Commands**
+
+```bash
+cd userspace/cuda
+tools/cudatool.py info <new>/run/libcuda.so.<new>
+tools/cudatool.py delta <old>/run/libcuda.so.<old> <new>/run/libcuda.so.<new> > <new>/run/delta.txt
+tools/cudatool.py fdematch <old>/run/libcuda.so.<old> <new>/run/libcuda.so.<new> \
+ --names <old>/annotations/names.tsv
+tools/cudatool.py wrappers <new>/run/libcuda.so.<new>
+
+cd ../../debug/risc-v-gsp-rm-vm
+./greeneye.sh symbols export \
+ --names ../../userspace/cuda/<new>/annotations/names.tsv \
+ --index ../../userspace/cuda/<new>/run/analysis/decompiled/decompiled_index.tsv \
+ --bias 0x100000 -o symbols-<new>.json
+./greeneye.sh symbols resolve --db symbols-<new>.json 0x<transferred-addr>
+
+cd ../..
+tools/cuda-ab/ab.sh all # host oracle + guest run + compare
+```
+
+### S5 — New GPU model or architecture (RTX 5090 / GB10-class, aarch64 hosts)
+
+| Attribute | Detail |
+|---|---|
+| **Symptoms** | Probe first-touch addresses differ from the reference device; a firmware family/partition-manifest mismatch; ISA string differs (RV64GC vs RV64IMAC); aarch64 host reports `qemu-system-riscv64 not found`; PMU/SOE provisioning assumptions break. |
+| **Detection** | `file`/`readelf` image shape; pairing check (family, FMC partition manifest, version string); probe first-touch diff; host tooling check (`./greeneye.sh info`); `rmtool` host-oracle inventory. |
+| **Evidence** | Device profile fields (PCI IDs, BARs, FB geometry, engine inventory, interrupt topology, ROM quirks); pairing notes; reference vs candidate first-touch maps; portability matrix report; physical validation record before any support claim. |
+| **Decision criteria** | Changed first-touch on profile registers → profile data, not logic. New firmware family/partition model → new family inputs, not a re-badged image. RV64GC or new CSRs → ISA/decoder work (S6). No physical validation → status stays Target/Experimental, never Supported. |
+| **Rollback** | Keep the reference device/firmware directories untouched; revert the candidate profile; a support claim is only removed with a matrix change note. |
+| **Record in** | [`06-portability/hardware-software-matrix.md`](06-portability/hardware-software-matrix.md) and the device profile notes; the evaluation ladder result for the combination. |
+
+**Response order**
+
+1. Collect the device profile; verify the firmware pairing before debugging
+ anything (a mismatched pairing looks exactly like an emulation bug).
+2. Confirm ISA/image shape (`file`, `readelf`); note FMC/manifest and boot
+ model; diff PMU/SOE images.
+3. Probe the new image and the reference with identical budgets; diff the
+ first-touch maps.
+4. Run the portability pipeline across reference and candidate.
+5. Update profile data first; only add code when an interface genuinely
+ differs.
+6. Validate on the physical device under supervision before claiming support.
+
+**Commands**
+
+```bash
+file <device-fwimage> && readelf -h <device-elf> && readelf -l <device-elf>
+cd debug/risc-v-gsp-rm-vm
+./greeneye.sh info
+./greeneye.sh probe <device-fwimage> --symbols symbols-<device>.json --record
+python3 harness/portability_matrix.py \
+ --target <reference-fwimage> --target <device-fwimage> \
+ --out dumps/matrix-<device>
+./rmharness/harness.sh --session dumps/real-<device> --name before \
+ -- rmtool engine list --format=json
+# aarch64 host without qemu-system-riscv64:
+# install qemu-system-misc, or point GREENEYE_QEMU at the binary
+```
+
+### S6 — New instructions or ISA changes in NV-RISCV / falcon / GSP
+
+| Attribute | Detail |
+|---|---|
+| **Symptoms** | Probe faults with an illegal/unimplemented instruction; the native backend diverges from QEMU on the same image; compressed/float extensions or new CSRs appear; a re-hosted build built with the old `-march` fails or misbehaves. |
+| **Detection** | ISA string from `file`/`readelf`; probe fault cause/`desc`; QEMU-vs-native differential (`sim_diff.py`); decoder unit tests. |
+| **Evidence** | Image shape (ISA, segments, entry); fault report with cause, `epc`/`epc_firmware` and access address; `PARITY: OK` transcript or the exact divergence; a fidelity-gap note when the native decoder cannot execute the new instruction. |
+| **Decision criteria** | RV64IMAC on QEMU TCG → L0 high, proceed. RV64GC → QEMU supports it; update rehost `-march` and re-run parity. Native decoder gap → record as a fidelity gap; do not claim native coverage. Illegal instruction in QEMU itself → confirm ISA first, then escalate as a tool bug. |
+| **Rollback** | Revert rehost `-march`/build flags; keep the previous probe/session as the reference. |
+| **Record in** | The tool's accuracy/fidelity-gap list (`debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md` §8) and the image's version notes. |
+
+**Response order**
+
+1. Confirm the ISA and image shape; identify the exact faulting instruction and
+ address.
+2. Reproduce under QEMU (the reference execution engine) with a bounded probe.
+3. Run the QEMU-vs-native differential; if it diverges, capture both sessions.
+4. If the instruction is unsupported natively, record a fidelity gap with the
+ session as evidence; keep QEMU as the execution engine for that image.
+5. Update rehost build flags and re-run parity and the tool test suite.
+
+**Commands**
+
+```bash
+cd debug/risc-v-gsp-rm-vm
+./greeneye.sh probe <fwimage> --instructions <budget> --record
+python3 examples/scripts/sim_diff.py # requires QEMU; prints PARITY: OK on agreement
+make test
+```
+
+### S7 — Significant VBIOS/ROM changes
+
+| Attribute | Detail |
+|---|---|
+| **Symptoms** | The guest driver reads different device identity/VBIOS fields; probe-time register values move; ROM/VBIOS quirks change (laptop/Surface variants); a previously green probe hangs on a different register. |
+| **Detection** | Guest `lspci -vvv -d 10de:` and init-trace diff; snapshot diff of the identity/static-config region; `rmtool` read-only `BUS_GET_PCI_INFO` oracle; ROM hash change. |
+| **Evidence** | ROM hash and identity fields; before/after snapshot with a schema-attributed diff; pairing notes; the exact guest-observed values and the trace that consumed them. |
+| **Decision criteria** | Changed fields consumed by the driver → profile/override work, physically confirmed. Fields not consumed → record and monitor. Anything that would require impersonating a licensed product → forbidden; present a coherent device instead. |
+| **Rollback** | Revert the profile/override; previous ROM/profile stays archived with its hash. |
+| **Record in** | The device profile and the matrix entry; the failure taxonomy if the symptom was misleading. |
+
+**Response order**
+
+1. Hash the ROM/VBIOS and collect the identity fields.
+2. Capture a before/after snapshot of the firmware-side static-config region
+ and diff it field by field.
+3. Compare with the host oracle (read-only) and the guest's observed values.
+4. Change only profile data that a consumer demonstrably uses; confirm on
+ hardware.
+5. Record unexplained fields as open, not as noise.
+
+**Commands**
+
+```bash
+# guest
+sudo lspci -vvv -d 10de:
+# host oracle (read-only; verified queries from docs/17)
+# build once from the repo root: make -C tools/rmtool
+tools/rmtool/rmtool gpu info -d 0
+tools/rmtool/rmtool engine list
+
+cd debug/risc-v-gsp-rm-vm
+# snapshot the identity/static-config region before and after
+# (schema decode from the bundled/demo schema; real regions are version-keyed)
+./greeneye.sh compare dumps/<old-run> dumps/<new-run> --symbols
+```
+
+### S8 — Encrypted/protected memory or protected instructions (measurement plan; non-claims)
+
+| Attribute | Detail |
+|---|---|
+| **Symptoms** | The image consumes identity material that is provisioned encrypted (the PMU-delivered encrypted device identity in the real 0x59 campaign); protected-memory or protected-instruction paths are reached; a lenient run returns zeros where a real device returns provisioned values; boot halts at a secret-dependent step. |
+| **Detection** | First-touch/device-map analysis; lenient-MMIO `mmio_log` and `mmio_unknown`; boot-chain shim assumptions in session metadata; probe fault reports identifying the consuming PC. |
+| **Evidence** | First-touch map entry per protected address; `mmio_log` with PC and access; the exact access that consumes protected state; a written measurement plan and an explicit non-claims list. |
+| **Decision criteria** | Secret-dependent branch without provisioned input → blocked; record as a fidelity gap and a device-model requirement, not a bug. Zero-value success under lenient MMIO → hypothesis only. Any request to extract, defeat or prove protected state → refuse (P5/P7). |
+| **Rollback** | No product changes are made in this scenario; the only rollback is reverting experiment notes and keeping the map as data. |
+| **Record in** | The fidelity-gap list, the device-map file (`greeneye-device-map/1`), and the H5 cross-controller plan; never in a security-claims section. |
+
+**Measurement plan (what to measure, and what is not claimed)**
+
+| Measure | Method | Non-claim |
+|---|---|---|
+| Which address/instruction consumes protected state | bounded probe + device map + `mmio_log` | not a statement about the real value |
+| Whether execution continues when the value is absent/zero | lenient native run with `unknown_mmio="lenient"` | zeros are a hypothesis, not read-back semantics |
+| Which subsystem provisions the value | cross-controller inventory (PMU/SEC2/ACR/SOE) and boot-order notes | not a model of the secure state machine |
+| Whether the path is security-relevant | static map + recorded access ordering | not a side-channel, confidentiality or attack claim |
+
+**Response order**
+
+1. Classify: protected read, protected instruction, or a secret-dependent
+ branch. Do not attempt extraction; do not patch anything.
+2. Capture the first-touch map and the lenient `mmio_log` in a recorded
+ session.
+3. Write the measurement plan and the non-claims into the session note.
+4. Escalate the missing provisioning as a device-model requirement (H2/H5),
+ not as an emulation defect.
+5. Stop. Physical work on protected paths requires the supervisor and a
+ separate approved plan.
+
+**Commands**
+
+```bash
+cd debug/risc-v-gsp-rm-vm
+python3 -m greeneye.device_map demo --instructions 200000
+python3 -m greeneye.device_map harvest dumps/<probe-session> --out dumps/<ver>-device-map.json
+python3 -m greeneye.device_map show dumps/<ver>-device-map.json
+# lenient native continuation (programmatic):
+# ge.launch(kernel=..., backend="native",
+# device_map="dumps/<ver>-device-map.json",
+# unknown_mmio="lenient")
+# session.target.mmio_log / session.target.mmio_unknown
+```
+
+### S9 — New microcontrollers and new inter-/intra-microcontroller APIs and DMA lines
+
+| Attribute | Detail |
+|---|---|
+| **Symptoms** | The image stalls waiting on PMU/SEC2/ACR/SOE; falcon staging DMA does not land (the real failure mode: the SEC2 boot-loader block is not visible, the falcon does not halt); WPR2/security state differs from the successful path; new mailbox functions or DMA descriptors appear. |
+| **Detection** | Probe first-touch; boot-chain shim assumptions recorded in session metadata; compare against the last successful boot path; device-map diffs; cross-controller event absence in sessions. |
+| **Evidence** | First-touch addresses per controller window; mailbox/DMA access map; boot-order notes; shim assumptions in the session; the successful-path comparison; H2.2/H5 scope note. |
+| **Decision criteria** | GSP-dependent stall → PMU/SEC2 stub work first (H2.2 entry). New mailbox/API surface → inventory it before modelling. Security-chain semantics → shim/documented assumption, never a claimed secure model. Unmodelled DMA line → device-map entry plus model work, not a product change. |
+| **Rollback** | Revert any shim assumption change; the previous session/boot note stays as the reference. |
+| **Record in** | The cross-controller inventory and boot-order notes; the device map; the H5/H2.2 roadmap scope. |
+
+**Response order**
+
+1. Inventory the controllers and DMA lines the image touches first
+ (PMU, SEC2, ACR/SOE, booter).
+2. Reproduce the stall with a bounded probe and record shim assumptions in the
+ session.
+3. Compare against the last successful path (e.g. WPR2 state, SEC2 staging
+ visibility) and name the missing exchange.
+4. Model the minimum stub that lets the image proceed; keep it documented
+ data, not a silent special case.
+5. Gate real security-chain semantics behind H2.2/H5; escalate any conclusion
+ that would change the product.
+
+**Commands**
+
+```bash
+cd debug/risc-v-gsp-rm-vm
+./greeneye.sh probe <fwimage> --record
+python3 -m greeneye.device_map harvest dumps/<probe-session> --out dumps/<ver>-device-map.json
+python3 -m greeneye.device_map merge dumps/<ref>-device-map.json dumps/<ver>-device-map.json \
+ --out dumps/merged-device-map.json
+./greeneye.sh compare dumps/<last-good-run> dumps/<stall-run> --symbols
+```
+
+### S10 — Multi-version matrix operations and CI (RM 610 requires explicit human approval)
+
+| Attribute | Detail |
+|---|---|
+| **Symptoms** | A new matrix column regresses an existing combination; CI drifts from the local run; someone proposes a 610-labelled input or command; the release bundle is incomplete. |
+| **Detection** | `version_diff`/`portability_matrix` deltas; `ci_release --dry-run` plan inspection; the RM 610 gate refusal (`exit 3`); test transcripts. |
+| **Evidence** | `version-diff.json`, `portability-report.json`, `release-report.json`, `bundle-manifest.json`, seeded `findings.jsonl`, test transcripts, approval record for 610. |
+| **Decision criteria** | Dry-run clean before a real run. A step that raises is `error`, not skipped. Self-comparison must be `all_zero`; seeded replays byte-identical. **610: no acquisition, analysis, emulation edit, physical work or support claim before a named human approves in writing; an open-ended approval is invalid.** |
+| **Rollback** | Extend the matrix, never replace a column; revert the candidate commit and keep the previous anchor/tag/driver; a 610 attempt parks without touching 570.144. |
+| **Record in** | The port log/approval thread (610), the release bundle directory, and the matrix with its test date and rung. |
+
+**Response order**
+
+1. Run the release rehearsal on current-version labels first; inspect the
+ dry-run plan.
+2. Run `version_diff` and `portability_matrix` with fixed budgets; require the
+ self-comparison to be zero.
+3. Run the mutation smoke with a pinned seed; verify replay.
+4. Assemble and hash the bundle; attach it to the PR.
+5. For 610: stop at the gate; obtain the approval message; only then acquire
+ inputs and repeat the pipeline with 610 labels.
+
+**Commands**
+
+```bash
+cd debug/risc-v-gsp-rm-vm
+
+# rehearsal first: dry-run never opens a target and creates no output dir
+python3 harness/ci_release.py --old <old> --new <new> --out dumps/release-<ver> \
+ --old-label <OLD> --new-label <NEW> --dry-run
+
+# real pipeline (dry-run clean)
+python3 harness/ci_release.py --old <old> --new <new> --out dumps/release-<ver> \
+ --old-label <OLD> --new-label <NEW> \
+ --old-symbols symbols-<OLD>.json --new-symbols symbols-<NEW>.json \
+ --mutation-kernel firmware/uproc_demo/build/uproc_demo.elf \
+ --mutation-iterations 8 --mutation-seed 7
+
+# RM 610 gate: refused by default; explicit approval token required
+python3 harness/version_diff.py --old <old> --new <new> --out dumps/vdiff-610 \
+ --old-label RM-610 --new-label 570.144
+# exit 3 unless: --approval-token RM610-APPROVED or GREENEYE_RM610_APPROVED=1
+```
+
+## 6. Evidence-bundle requirements
+
+A bundle is the minimum attachable proof for a version/device/architecture
+triage or a product change. It combines the docs/07 evidence checklist, the
+docs/14 §6 adversarial pack and the docs/21 release tree.
+
+```
+<out>/
+ bundle-manifest.json sha256 + bytes of every file below
+ release-report.json pipeline summary (steps, gate, labels)
+ version-diff/ version-diff.json|.md, symbols/, sessions/, compare/
+ portability-matrix/ portability-report.json|.md, symbols/, sessions/, compare/
+ mutation-smoke/ findings.jsonl, summary.md, sessions/
+```
+
+Checklist — every box must be true or explicitly waived with a reason:
+
+* [ ] firmware identity: SHA-256, version string, family and pairing notes;
+* [ ] input provenance: channel, date, and the exact file names analysed;
+* [ ] symbol DB(s): hash, source (ELF/Ghidra/JSON), `bias` and provenance;
+* [ ] baseline and candidate session directories (or their SHA-256 list);
+* [ ] exact stimulus (script/spec plus mutation values) and budgets;
+* [ ] `compare` report and relevant `analyze` reports;
+* [ ] M1–M6 table with the reference-capture method and L2/L3 limits stated;
+* [ ] `greeneye_hits.tsv` and any Ghidra annotations produced;
+* [ ] `bundle-manifest.json` written last, covering the final reports;
+* [ ] unexplained-delta list (empty is a claim, not a default);
+* [ ] decision and product action (or explicit "no change") recorded;
+* [ ] for 610: approval ID, approver, date and scope limits.
+
+Fidelity statement template (copy into every report):
+
+```text
+Fidelity: L0 (instruction semantics) and L1 (memory) exercised; L2 (timing,
+concurrency) NOT modelled; L3 (device environment) absent for <registers>.
+Reference capture: <hardware / vfio-mdev / host oracle / none>.
+Physical reproduction required before product change: <yes/no + why>.
+```
+
+## 7. Fault and escalation tree
+
+```text
+Failure / new input observed
+|
++- Is the input hashed, versioned and paired? NO -> STOP: acquire/hash/pair first (S1)
+|
++- Did it come from RM 610 or a 610-labelled path? YES -> STOP: hard human gate (S10)
+|
++- Is a physical GPU / driver bind / mdev involved?
+| YES -> STOP: supervised physical cycle + reboot approval (P7); never unattended
+|
++- Does the question need timing, interrupts, protected state or real device
+| read-back semantics (L2/L3)?
+| YES -> simulator evidence is a hypothesis only (P3/P4/P5); record the
+| fidelity gap; product change requires physical reproduction
+|
++- Does it reproduce under a fixed stimulus in green-eye?
+| NO -> widen the stimulus / capture the real trace before theorising;
+| absence of a hit is not proof of dead code
+| YES -> classify with the S1 decision table (code changed vs behaviour moved)
+|
++- Would a fix change emulation behaviour?
+| YES -> physical/vfio-mdev reproduction + evidence bundle before landing
+| NO -> docs/profile/test-only change with the bundle attached
+|
++- Does any vendor component need modification?
+ YES -> forbidden (P2): redesign to emulate the interface instead
+```
+
+Stop conditions (do not proceed, do not "temporarily" work around):
+
+1. Unverified pairing or unhashed input.
+2. RM 610 without written approval.
+3. A physical cycle without the supervisor and fail-safe plan.
+4. A request to patch a vendor component or a guest.
+5. A security/confidentiality claim based on simulator state.
+6. A product change with no physical reproduction and no bundle.
+
+## 8. When NOT to use the tooling
+
+| Situation | Why not | Use instead |
+|---|---|---|
+| You need silicon truth (timing, interrupts, real register read-back) | L2/L3 are absent by design | supervised physical cycle + reference traces |
+| You are about to mutate or inject against hardware, a guest, a kernel module or an existing gdbstub | Tooling is simulator-only; hardware writes are prohibited | bounded QEMU/native harness with recorded sessions |
+| You need to defeat, extract or prove a security boundary | The simulator is not a confidentiality boundary | stop; escalate as a design question |
+| You need a product fix "now" without evidence | Violates P3/P4 | evidence bundle first; the anchor and matrix can wait |
+| The input has not been hashed/versioned/paired | Every later claim is contaminated | S1 steps 1–3 |
+| RM 610 has no written approval | Hard gate | request approval with the readiness checklist |
+| The physical GPU is held by another job | Shared-oracle safety and fail-safe discipline | host-side read-only oracle (`rmtool gpu list`, `harness.sh`) |
+| You need type recovery or decompilation | green-eye is dynamic ground truth, not a decompiler | Ghidra corpus and symbol tooling |
+| You need guest-driver behaviour | green-eye observes firmware, not the guest | `tools/guest-diag/`, ioctl tracer |
+| A one-line question can be answered by `analyze` on an existing session | Do not produce new runs for no reason | reuse recorded sessions and `compare` |
+
+## 9. One-page quick-reference decision tree
+
+| Symptom (first sign) | First command | Decision | Scenario |
+|---|---|---|---|
+| New image/version appears | `./greeneye.sh symbols export ...` + `probe --record` | hash+pair first, then version diff | S1 |
+| OpenRM build breaks, `Unknown symbol` | `./scripts/build_nvidia_openrm.sh`; `dmesg` | glue-only vs interface change | S2 |
+| Control/ioctl inventory shifts | `python3 harness/control_discovery.py` | moved vs new logic | S3 |
+| `libcuda` bump / context failure | `tools/cudatool.py delta` + `fdematch` | patch / feature / reshuffle | S4 |
+| New device/architecture | `file`/`readelf`, `probe` | profile vs logic; physical validation for support | S5 |
+| Illegal/unknown instruction | `./greeneye.sh probe --instructions N` | ISA/decoder; fidelity gap if native-only | S6 |
+| Identity/ROM fields differ | guest `lspci -vvv`, snapshot diff | consumed field → profile; else record | S7 |
+| Protected/encrypted state blocks | device map + lenient `mmio_log` | record fidelity gap; no extraction; no claims | S8 |
+| Early stall on PMU/SEC2/ACR/SOE | `probe --record` + device map | stub the missing exchange; gate security semantics | S9 |
+| Matrix/CI drift; 610 mentioned | `ci_release --dry-run` | dry-run clean; 610 hard gate | S10 |
+
+Golden rule: **freeze, hash, pair, reproduce, classify, evidence, then act** —
+and never let the last step touch anything we do not own.
+
+## 10. Related documentation
+
+Green-eye suite (experimental, in development):
+
+* [green-eye user guide](05-operations/04-green-eye-user-guide.md) — what it
+ is, quickstart, tool chooser and troubleshooting;
+* [GSP in a RISC-V VM](05-operations/03-risc-v-gsp-rm-vm.md) — project-level
+ purpose, status and limits;
+* [`debug/risc-v-gsp-rm-vm/docs/00-OVERVIEW.md`](../debug/risc-v-gsp-rm-vm/docs/00-OVERVIEW.md)
+ — tool status and doc map;
+* [`debug/risc-v-gsp-rm-vm/GREEN-EYE-ROADMAP.md`](../debug/risc-v-gsp-rm-vm/GREEN-EYE-ROADMAP.md)
+ — H1–H7 program;
+* [`debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md`](../debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md)
+ — fidelity levels and the M1–M6 contract;
+* [`debug/risc-v-gsp-rm-vm/docs/07-PORTABILITY-PLAYBOOKS.md`](../debug/risc-v-gsp-rm-vm/docs/07-PORTABILITY-PLAYBOOKS.md)
+ — P1–P4 porting playbooks;
+* [`debug/risc-v-gsp-rm-vm/docs/14-ADVERSARIAL-INTEROP-GUIDE.md`](../debug/risc-v-gsp-rm-vm/docs/14-ADVERSARIAL-INTEROP-GUIDE.md)
+ — adversarial navigation and the evidence pack;
+* [`debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md`](../debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md)
+ — release triage and bundles;
+* [`debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md`](../debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md)
+ — the 610 gate and approval format;
+* [`debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md`](../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md)
+ — tool audit, symptom-to-fix table and provenance rule;
+* [`debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md`](../debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md)
+ — sweep/device-port scaffolds (unadmitted pilots, dry-run default);
+* [`debug/risc-v-gsp-rm-vm/docs/29-PACKAGING-AND-TRIALING.md`](../debug/risc-v-gsp-rm-vm/docs/29-PACKAGING-AND-TRIALING.md)
+ — trial simulation packages (unadmitted pilot).
+
+Project documentation:
+
+* [Debugging and maintenance workflows](05-operations/02-debugging-and-maintenance-workflows.md)
+ — captures, failure taxonomy, version discipline;
+* [Performance, correctness, security](05-operations/01-performance-correctness-security.md)
+ — the evaluation ladder;
+* [Hardware and software matrix](06-portability/hardware-software-matrix.md)
+ — support definitions and the portability pipeline;
+* [Practical challenges and adversarial interoperability](04-implementation/02-practical-challenges-and-adversarial-interop.md)
+ — the no-patch rule;
+* [Developer onboarding and standards](07-developers/onboarding-and-standards.md)
+ — coding rules and the landing checklist;
+* [Documentation front door](README.md) — the full reading order;
+* [Engineering operations](eng-ops.md) — tool admission, workflows and
+ retirement (the cross-cutting engineering-operations document);
+* CUDA playbook:
+ [`userspace/cuda/docs/07-version-upgrade-playbook.md`](../userspace/cuda/docs/07-version-upgrade-playbook.md)
+ and [`userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md).
+
+<!-- change (2026-10-02): fix the S4 fdematch invocation (two libraries plus
+--names, per tools/cudatool.py), add a session/disk-growth bound to §3.2, and
+link docs 28/29 from §10; scope: RUNBOOK §3.2, §5 S4, §10 only; benefit: the
+copied triage command runs as written and the retention rule is explicit. -->