diff options
31 files changed, 3603 insertions, 236 deletions
diff --git a/STATE-CURRENT.md b/STATE-CURRENT.md index 7aaf9f1..9f5ea01 100644 --- a/STATE-CURRENT.md +++ b/STATE-CURRENT.md @@ -1,6 +1,6 @@ # Open-IOV Current State -> **Snapshot: 2026-10-02 10:55.** This is a dated public status page, not live +> **Snapshot: 2026-10-02 ~11:00.** This is a dated public status page, not live > state; it is refreshed on every progress check. Live working notes are > maintained outside this repository; the public known-knowns/unknowns document > is authoritative here: @@ -69,7 +69,7 @@ meant to execute on the physical GPU. and destination equal to source. Every cycle released the device cleanly with no reboot; the next rung is wiring the live physical session into the guest-facing mdev/vfio path - ([`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md)). The guest-wiring rung then advanced: an inert backend-session API was added, and the mediated-device-owned lifecycle (P1, see the frontier section) was implemented and exercised - deferring the physical claim/boot to the first device create, releasing it on the last remove, with fail-open behaviour. The first P1 acceptance cycle exposed two defects - an asynchronous boot marked READY after a failed FWSEC boot, and the P1 release path freezing the host (see the frontier section). The G0 fix bundle (default-latched D0/D3cold locks, post-rebind readiness gate, pre-claim D0 settle, READY gated on a proven boot, pre-kick staging readback) is implemented, built, smoke-tested and now **validated live**: two module-owned supervised cycles (2026-10-02 ~00:57-00:58) were 2/2 green through claim, FWSEC, booter, RISC-V active, init-done (`0x1001`, `rpc_result=0` in ~955 ms) and a latched release with WPR2 down, exactly 5 `power domain left LOCKED` lines, 0 in-window rail cycles, host alive and pstore empty. The retested P1 mdev-owned path - the exact path that hard-froze on 2026-10-01 - now fails open safely (the READY gate releases the claim when no live GSP-RM session exists; no freeze), while the async FWSEC boot defect (`staging not visible before kick` -> `FWSEC/FRTS execution failed: -5`) remains open, quarantined off the critical path; guest wiring proceeds on the module-owned live session. The bundle was later committed (`56766b2`). Guest-wiring P2 (static-config identity overlay) has since landed and verified live (2026-10-02 01:20-01:45): with `gsp_phys_static_live=1` the guest's `nvidia-smi -q` Product Name comes from the live session (`NVIDIA GeForce RTX 2080 Super with Max-Q Design`) versus the emulated `NVIDIA GeForce RTX 2080 SUPER` with it off (A/B; overlay applied 4x/0), while the guest smoke stays GREEN `ALL OK` 64/64; one overlay-OFF full run hit a pre-existing emulated-path flake, watched. P3 (read-only control forwarding) has since landed and verified live (2026-10-02 ~01:33-01:45): the `gsp_phys_ctrl_forward` whitelist served 22 controls live with 2 fallbacks and 0 timeouts while the guest smoke stayed GREEN `ALL OK` 64/64; **P5 wave 1 (guest-submission shadow execution) has since landed and verified live** (2026-10-02 ~02:00-02:20): with `gsp_phys_work_forward=1` the sema pump shadow-executes eligible CE pushes on the physical engine through a dedicated ordered workqueue - 514 real 4 KiB copies during a GREEN guest smoke (`ALL OK` 64/64), 194 requests coalesced/dropped while one was in flight, 0 refusals, 0 timeouts - while the software release path stays authoritative and no guest memory or guest VA is touched; P6 has since proven guest-page DMA live (P6a, 2026-10-02: the physical copy engine copied 4 KiB to and from a real guest RAM page with the semaphore released while the guest smoke stayed GREEN `ALL OK` 64/64 and release stayed latched with no reboot) and then **P6b wave 1** (2026-10-02, run p6run13): with `gsp_phys_work_forward=1 gsp_phys_guest_mem=1` the sema service resolved real guest CE-copy operands to host physical addresses and executed them on the physical copy engine through module-built GMMU scratch slots (`req=42 ok=42 verify_ok=42 verify_fail=0`, 503 ladder shadow copies also ran) while the software release path stayed authoritative - guest CE copies now execute on hardware and are **hardware-authored with verified data** while the software copy remains the fail-closed fallback for unqueued or failed copies: wave 1.5 added deferred, exactly-once release (`engaged=75 applied=75`), and wave 2 added `gsp_phys_work_hw_primary=1`, which skips the software copy for forwarded ops (`copy_skipped=119`, `req=100 ok=100 verify_ok=100`, smoke `matched=64/64` `ALL OK`, 19 inline software fallbacks `copy_soft=19 copy_fail=0`, and a 121-copy fallback-recovery run with guest-RAM ranges unregistered). The guest smoke stayed GREEN `ALL OK` 64/64 and release stayed latched with no reboot. Real page-directory shadowing and guest channels (P6c: completion/fault relay, guest page tables, `vectorAdd`) remain, along with the P6a trigger-page pinning follow-up, and P4 (alloc/free handle adoption) remains deliberately deferred. | + ([`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md)). The guest-wiring rung then advanced: an inert backend-session API was added, and the mediated-device-owned lifecycle (P1, see the frontier section) was implemented and exercised - deferring the physical claim/boot to the first device create, releasing it on the last remove, with fail-open behaviour. The first P1 acceptance cycle exposed two defects - an asynchronous boot marked READY after a failed FWSEC boot, and the P1 release path freezing the host (see the frontier section). The G0 fix bundle (default-latched D0/D3cold locks, post-rebind readiness gate, pre-claim D0 settle, READY gated on a proven boot, pre-kick staging readback) is implemented, built, smoke-tested and now **validated live**: two module-owned supervised cycles (2026-10-02 ~00:57-00:58) were 2/2 green through claim, FWSEC, booter, RISC-V active, init-done (`0x1001`, `rpc_result=0` in ~955 ms) and a latched release with WPR2 down, exactly 5 `power domain left LOCKED` lines, 0 in-window rail cycles, host alive and pstore empty. The retested P1 mdev-owned path - the exact path that hard-froze on 2026-10-01 - now fails open safely (the READY gate releases the claim when no live GSP-RM session exists; no freeze), while the async FWSEC boot defect (`staging not visible before kick` -> `FWSEC/FRTS execution failed: -5`) remains open, quarantined off the critical path; guest wiring proceeds on the module-owned live session. The bundle was later committed (`56766b2`). Guest-wiring P2 (static-config identity overlay) has since landed and verified live (2026-10-02 01:20-01:45): with `gsp_phys_static_live=1` the guest's `nvidia-smi -q` Product Name comes from the live session (`NVIDIA GeForce RTX 2080 Super with Max-Q Design`) versus the emulated `NVIDIA GeForce RTX 2080 SUPER` with it off (A/B; overlay applied 4x/0), while the guest smoke stays GREEN `ALL OK` 64/64; one overlay-OFF full run hit a pre-existing emulated-path flake, watched. P3 (read-only control forwarding) has since landed and verified live (2026-10-02 ~01:33-01:45): the `gsp_phys_ctrl_forward` whitelist served 22 controls live with 2 fallbacks and 0 timeouts while the guest smoke stayed GREEN `ALL OK` 64/64; **P5 wave 1 (guest-submission shadow execution) has since landed and verified live** (2026-10-02 ~02:00-02:20): with `gsp_phys_work_forward=1` the sema pump shadow-executes eligible CE pushes on the physical engine through a dedicated ordered workqueue - 514 real 4 KiB copies during a GREEN guest smoke (`ALL OK` 64/64), 194 requests coalesced/dropped while one was in flight, 0 refusals, 0 timeouts - while the software release path stays authoritative and no guest memory or guest VA is touched; P6 has since proven guest-page DMA live (P6a, 2026-10-02: the physical copy engine copied 4 KiB to and from a real guest RAM page with the semaphore released while the guest smoke stayed GREEN `ALL OK` 64/64 and release stayed latched with no reboot) and then **P6b wave 1** (2026-10-02, run p6run13): with `gsp_phys_work_forward=1 gsp_phys_guest_mem=1` the sema service resolved real guest CE-copy operands to host physical addresses and executed them on the physical copy engine through module-built GMMU scratch slots (`req=42 ok=42 verify_ok=42 verify_fail=0`, 503 ladder shadow copies also ran) while the software release path stayed authoritative - guest CE copies now execute on hardware and are **hardware-authored with verified data** while the software copy remains the fail-closed fallback for unqueued or failed copies: wave 1.5 added deferred, exactly-once release (`engaged=75 applied=75`), and wave 2 added `gsp_phys_work_hw_primary=1`, which skips the software copy for forwarded ops (`copy_skipped=119`, `req=100 ok=100 verify_ok=100`, smoke `matched=64/64` `ALL OK`, 19 inline software fallbacks `copy_soft=19 copy_fail=0`, and a 121-copy fallback-recovery run with guest-RAM ranges unregistered). The guest smoke stayed GREEN `ALL OK` 64/64 and release stayed latched with no reboot. Real page-directory shadowing and guest channels (P6c: completion/fault relay, guest page tables, `vectorAdd`) have since been reached: guest CE copies and their releases are forwarded with release and fault relay in place (P6c C0/C1/C3, default off); the host GR ladder (H1) brings up a GR0 channel and a `TURING_COMPUTE_A` object, but the first push faults `CTXNOTVALID` (the external-VAS context was never committed) and the promote-binding attempt wedged GSP-RM (host assert, fuse-locked WPR2), so the device is held in the designed locked fail-safe pending a supervised reboot (see the frontier section). Real guest page tables/channels and GR execution remain, along with the P6a trigger-page pinning follow-up, and P4 (alloc/free handle adoption) remains deliberately deferred. | | Graphics (3D), video | ❌ | | Faults, multi-version, Windows, migration, multi-tenancy, performance | ❌ | @@ -97,36 +97,37 @@ chain succeeds. cycles. The acting session deliberately paused here; the reboot request is awaiting approval because other work is running, and no further live GPU cycles are planned this boot. The promote-assert bisection plan is ready - for the next boot. -- **Guest-submission relay and fault replay landed, default off (P6c + for the next boot. The promote is gated by `gsp_phys_rmcall_gr_promote` + (default 0), and the parameter documentation warns that it can wedge the + device. +- **Guest-submission relay and fault relay landed, default off (P6c C0/C1/C3, 2026-10-02 07:43-08:07).** With `gsp_phys_relay=1` the physical engine authors the guest CE copy and the release semaphore: the engine-authored release was observed in the guest page (`relays=64`), releases are delivered exactly once (deferred hold), and the relay-on run stayed green (`requested=68 ok=67 fallback=9`, smoke `ALL OK` 64/64). A forced-unresolvable control proved the software fallback recovers every - copy exactly once. One relayed copy released its semaphore but failed the - host-side data compare, so the parameter stays default-off pending triage; - verification hardening now logs the mismatch (sem target/payload/poll, - first differing byte, live/snapshot dumps) and falls back to the software - apply. `gsp_phys_fault_relay=1` relays forwarded-engine faults to the guest - channel notifier/event path (live green, default off). The software path is - unchanged with all parameters off. -- **Host GR ladder (H1) opened and hit the next wall (2026-10-02 08:29).** - The host resource-manager ladder is now selectable to the compute engine - (`gsp_phys_rmcall_engine=1`, default off) and allocates a GR0 channel with - a compute-class object on the live GSP-RM (`engine=GR0 hGr=0xcaf0002c`), but - the first host GR semaphore push did not complete: the submit timed out - (`-110`) with no release. The copy-engine control run stayed green - (`REAL COPY OK`), and both runs released latched and clean (protected - region down, distribution driver rebound, exactly 5 power-domain locks, - 0 rail cycles, same boot). Current wall: make the GR engine execute the - submitted compute block - bisect GR allocation parameters, the release - method encoding and the method order/context init, using the drained GR - fault records. The timeout was then root-caused to an uncommitted GR engine - context (`CTXNOTVALID`); the follow-up promote-context attempt wedged the - server - see the top bullet. Next: host `vectorAdd`, then guest GR - forwarding. + copy exactly once. One relayed copy of 68 released its semaphore but failed + the host-side data compare; it was handled fail-closed (software apply) and + the parameter stays default-off pending triage, with hardening now in + place: retry-once, snapshot capture and relay counters. `gsp_phys_fault_relay=1` + relays forwarded-engine faults to the guest channel notifier/event path; it + delivered a real CE0 fault live (default off), and the auxiliary event post + reported `no_event`. The software path is unchanged with all parameters off. +- **Host GR ladder (H1) opened and hit the GR context wall (2026-10-02 + 08:29).** The host resource-manager ladder is now selectable to the compute + engine (`gsp_phys_rmcall_engine=1`, default off) and brings up a GR0 + channel with a `TURING_COMPUTE_A` compute object on the live GSP-RM + (`engine=GR0 hGr=0xcaf0002c`), but the first host GR semaphore push faults: + the submit timed out (`-110`) with no release and the drained fault records + read `CTXNOTVALID` - the external-VAS context was never committed. The + copy-engine control run stayed green (`REAL COPY OK`), and both runs + released latched and clean (protected region down, distribution driver + rebound, exactly 5 power-domain locks, 0 rail cycles, same boot). The + follow-up context-promote binding attempt wedged GSP-RM (host assert, + fuse-locked WPR2) - see the top bullet. Current wall: commit the GR context + and make the engine execute the submitted compute block. Next: host + `vectorAdd`, then guest GR forwarding. - **Experimental tooling: green-eye (2026-10-02 08:36, in development).** `debug/risc-v-gsp-rm-vm/` boots GSP/uproc firmware in a RISC-V QEMU VM with a GDB stub and adds Frida-like runtime inspection, a scripting/plugin @@ -142,7 +143,8 @@ chain succeeds. self-test and ladder `REAL COPY OK`, clean release). Forwarding was never affected (HPAs are resolved at submit time). - **Guest CE copies are hardware-authored with verified data, with deferred - exactly-once release (P6b waves 1.5 and 2, 2026-10-02, run p6w2d).** Wave + exactly-once release (P6b waves 1.5 and 2, 2026-10-02, run p6w2d; + superseded as a frontier by the P6c relay and H1 bullets above).** Wave 1.5 (run p6w15), with `gsp_phys_work_defer_release=1` (plus `gsp_phys_work_forward=1 gsp_phys_guest_mem=1`): a forwarded guest push's releases are recorded and its GPFIFO entry is held until the hardware copy @@ -235,7 +237,8 @@ chain succeeds. distribution driver rebound, protected region down, exactly 5 power-domain locks intentionally held, 0 in-window rail cycles, no reboot. Module sha256 `e1bce651...`. **Next: P6b** - forwarding guest submissions (page-directory - shadowing, real guest channels, push forwarding) - not implemented yet. + shadowing, real guest channels, push forwarding) - landed afterwards + (superseded by the P6b/P6c bullets above). - **Guest-memory path (P6) first attempts (2026-10-02 ~02:42 onward; resolved by the P6a result above).** The rung puts guest memory into the real execution path: guest RAM is now a @@ -277,7 +280,8 @@ chain succeeds. pre-existing expected behavior. **Next: P6** (guest page tables/memory into the real MMU so guest data itself executes on hardware; includes the IOMMU/DMA reachability known-unknown), with optional **P5 wave 2** - (hardware-primary release relay with a non-applying decode mode); **P4** + (hardware-primary release relay with a non-applying decode mode; **subsumed** + by P6b wave 2 + the P6c relay above); **P4** remains deferred. - **Read-only control forwarding (P3) landed and verified live (2026-10-02 01:33-01:45).** `GSP_RM_CONTROL` (fn76) forwards a whitelist of @@ -318,7 +322,8 @@ chain succeeds. driver rebound, exactly 5 power locks intentionally left held, 0 in-window rail cycles, host stable, no reboot. Module sha256 `69eac176...`. P3 read-only control forwarding has since landed and verified live (bullet - above). Guest work is still **not** forwarded to the physical engine. The + above). At that point guest work was not yet forwarded to the physical + engine (the P6b/P6c forwarding results are above). The P2 bundle and the G0 freeze fix have since been committed (`56766b2`). - **G0 fix bundle built after the P1 release-path freeze (2026-10-02 00:55).** The first P1 mdev-owned acceptance cycle ended in the known hard freeze @@ -368,7 +373,8 @@ chain succeeds. module-owned path succeeds with the very same preconditions in the same boot). Guest wiring proceeds on the module-owned live session (`gsp_phys_gpu=1` claim+boot at load, proven); the P1 async claim stays - fail-open. No guest work is forwarded to the physical engine yet. + fail-open. At that point no guest work was forwarded to the physical engine + yet (the P6b/P6c forwarding results are above). - **Guest-wiring rung opened (2026-10-01 22:00): the first backend-session API code is in the tree, inert.** A skeleton that exposes the live physical session to the device layer (session request entry points plus a static-info @@ -381,7 +387,8 @@ chain succeeds. [`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md). Post-single-VM-CUDA research notes (concurrent multi-VM virtual functions, 3D acceleration, and a native-vs-guest performance program; and a host-side - acceleration feasibility study) were produced under `backups/` (untracked). + acceleration feasibility study) were produced and are kept in the local + evidence archive (untracked). - **Submission-rung milestone (2026-10-01 17:24-21:05): real work now executes on the physical engine.** The host resource-manager client allocated a virtual-address-space object with a committed page directory, a channel-group @@ -398,8 +405,8 @@ chain succeeds. wiring this live session into the guest-facing mdev/vfio path so guest submissions execute on the physical engine (phased plan: [`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md)). - Evidence: `backups/evidence-20261001-cycle18/` through `...-cycle25/`, with - per-wall reports in `backups/evidence-20261001-reports/next/`. + Evidence: the local evidence archive (cycle 18 through cycle 25 runs, with + per-wall reports). - **Physical-backend milestone (2026-10-01 16:45): the host-owned GSP-RM completes initialization on the real TU104 and serves the static-config RPC.** Three supervised cycles, no reboot, clean release each time: @@ -422,8 +429,7 @@ chain succeeds. The physical backend is now a live, queryable GSP-RM: the host can boot it, send RM RPCs and receive full replies, and shut it down gracefully. The next rung is to drive the guest-facing emulation from this backend. - Evidence: `backups/evidence-20261001-cycle11/`, `...-cycle12/`, - `...-cycle13/`. + Evidence: the local evidence archive (cycles 11-13). - **Driver-API baseline re-verified green on the current build** (2026-10-01 14:20): the smoke test passes end-to-end (`ALL OK`, payload 64/64) with real releases (`faked=0`, `miss=0`, `stale=0`); the source-level root-cause diff --git a/debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md b/debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md new file mode 100644 index 0000000..4669f93 --- /dev/null +++ b/debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md @@ -0,0 +1,319 @@ +# 28 — Porting automation: driver-version sweep and new-device scaffold + +> **# TOOL IN DEVELOPMENT - NOT READY YET!** See 00 for status and +> [`GREEN-EYE-ROADMAP.md`](../GREEN-EYE-ROADMAP.md) H4/R6 for the program. + +Two stdlib-only orchestrators turn the docs/07 P1/P2 playbooks and the +docs/22 porting guide into one repeatable chain: a **driver-version sweep** +(`harness/port_driver_versions.py`) and a **device-port scaffold** +(`harness/port_new_device.py`). They reuse the version-diff, portability-matrix +and mutation machinery already documented in docs/18 and docs/21, add the +RM 610 gate verbatim from `version_diff.py`, and never download firmware, +never touch hardware, and write only under `--out`. Both are **unadmitted +pilots** under `eng-ops.md` §2 — the keep/retire evaluation is F2/F3 in +[`future-eng-ops.md`](../../../docs/future-eng-ops.md), and a sweep is not +routine release tooling until that record exists. + +| Tool | Roadmap | What it does | +|---|---|---| +| `harness/port_driver_versions.py` | **R6**, H4.1 | Finds each driver/firmware version's local image, runs version_diff vs a baseline, the portability matrix and a mutation smoke, and assembles per-version evidence bundles plus a cross-version summary with the docs/06 M1–M6 skeleton. | +| `harness/port_new_device.py` | **R6**, H4.1, docs/07 P2 | Creates/uses a user-supplied device profile, probes the device firmware's first touch, builds a symbol inventory, runs the matrix against a reference, and emits a port checklist with a go/no-go recommendation. | + +Both are dry-run by default, resume-safe (sweep), bounded by QEMU budgets plus +a per-step wall-clock net, and simulator-only (no `/dev`, kernel module, mdev, +guest or physical GPU). + +--- + +## 1. The chain + +```text +new driver / GSP image lands on disk (runfile, .fwimage, re-hosted ELF) + | + +-- port_driver_versions --dry-run plan: inputs found/missing, steps, artifacts + | | + | +-- --run per version: + | 1. target discovery/identification (never downloads) + | 2. version_diff vs the chosen baseline + | 3. portability_matrix (baseline + candidate) + | 4. mutation_smoke (bounded, seeded) + | 5. per-version bundle-manifest.json + | +-- sweep-summary.json/.md (M1-M6 skeleton, pass/fail interpretation) + | + +-- port_new_device --dry-run plan: profile fields, firmware, steps + | + +-- --run profile + probe-first-touch + symbols + matrix + checklist +``` + +The chain is the automation of docs/07 P1 (new GSP version), P2 (same version, +new device) and the docs/22 port checklists; the metric contract stays +docs/06 §5. Evidence is always attachable to the docs/07 checklist and the +docs/14 §6 pack. + +--- + +## 2. Driver-version sweep + +### 2.1 Known branch cadence + +The default version list documents the branch cadence and where to edit it — +the tool never invents an image or downloads one: + +| branch | expected image | default sweep | +|---|---|---| +| 540 | earlier branch | yes | +| 550 | earlier branch | yes | +| 560 | earlier branch | yes | +| 570 | current anchor (570.144 and patches) | yes | +| 580 | expected cadence, no local image assumed | no | +| 610 | RM 610 gate, refused without approval (docs/23) | no | + +Edit `DEFAULT_VERSIONS` in `harness/port_driver_versions.py`, or override on +the command line / in a config file. A branch with no local image is reported +`no-input`; it is not an error and nothing is fetched. + +### 2.2 Invocation + +Dry-run first (the default; reads input directories only): + +```bash +cd debug/risc-v-gsp-rm-vm +python3 harness/port_driver_versions.py --input-dir ../../firmware/57X-GSP \ + --versions 540,550,560,570 +``` + +Execute, resume-safe, with explicit targets and per-version symbol DBs: + +```bash +python3 harness/port_driver_versions.py --run \ + --versions 570.144 \ + --target 570.144=../../firmware/57X-GSP/570.144/gsp-570.144.fwimage.bin \ + --baseline 570.144 \ + --symbols 570.144=symbols-570.144.json \ + --out dumps/sweep-570.144 +``` + +Cron (weekly, quiet, resume-safe; completed versions are skipped): + +```cron +17 3 * * 1 cd /path/to/open-iov/debug/risc-v-gsp-rm-vm && \ + python3 harness/port_driver_versions.py --run \ + --input-dir ../../firmware/57X-GSP --versions 540,550,560,570 \ + --out dumps/sweep-570 >> dumps/sweep-570/cron.log 2>&1 +``` + +Agent (docs/20): run the dry-run to get the plan, use +`portability_matrix_run`/`mutation_run` for single steps if the sweep itself is +not exposed by MCP yet, then read `sweep-summary.json` and the per-version +`bundle-manifest.json` from the bundle instead of recomputing. The dry-run plan +is machine-readable (`greeneye-port-sweep-plan/1`) and side-effect free. + +Flags that matter: + +* `--versions` / `--config` — explicit list or JSON config (below); +* `--input-dir` — recursive discovery of local `.fwimage`, `.fwimage.bin`, + `.elf`, `.bin` and `.run` files; version matching uses file and directory + names (`570` matches `570.144`); +* `--target VERSION=PATH`, `--symbols VERSION=PATH`, + `--ghidra-exports VERSION=PATH` — explicit inputs, no discovery; +* `--baseline` — version diffed against (default: earliest version found); +* `--extract-runfiles` — opt in to local `sh <runfile> --extract-only` + extraction when only a driver runfile is present (still never downloads); +* `--force` — re-run versions whose `version.json` reports `ok`/`partial`; +* `--step-timeout` — wall-clock safety net per step (`0` disables; QEMU + budgets remain the first bound); +* the usual `version_diff`/`matrix`/`mutation` budgets. + +### 2.3 Configuration file + +```json +{ + "format": "greeneye-port-sweep-config/1", + "versions": ["540", "550", "560", "570.144"], + "baseline": "570.144", + "input_dir": "../../firmware/57X-GSP", + "targets": {"570.144": ["../../firmware/57X-GSP/570.144/gsp-570.144.fwimage.bin"]}, + "symbols": {"570.144": "symbols-570.144.json"} +} +``` + +Unknown keys are refused so a typo cannot silently change the sweep. CLI flags +override config values. + +### 2.4 Output and evidence bundle + +```text +<out>/ + sweep-summary.json cross-version report (also .md) + bundle-manifest.json sha256 + size of every artifact below + <version>/ + version.json status, inputs, steps, M1-M6, sha256 + version.md rendered per-version report + bundle-manifest.json per-version evidence bundle + version-diff/ docs/21 §2.3 tree + portability-matrix/ portability-report.json|.md, symbols/, sessions/, compare/ + mutation-smoke/ findings.jsonl, summary.md, sessions/ +``` + +`sweep-summary.json` carries per-version status, the cross-version M1–M6 +skeleton (every agreement metric `not measured` without a reference capture, +docs/06 §5), the RM 610 gate record, and an explicit `interpretation` string: +`pass`, `fail` (with the failing versions) or `not-run`. + +### 2.5 Reading failures + +| Status | Meaning | Action | +|---|---|---| +| `ok` | all steps ran, none errored | attach the bundle to the docs/07 checklist | +| `partial` | a step skipped (e.g. no QEMU, no mutation kernel) | explain the skip; re-run where the capability exists | +| `error` | a step raised or timed out | inspect `<version>/version.json`, then the step report (`version-diff.json`, `portability-report.json`, `findings.jsonl`) | +| `no-input` | no local image found for the version | add the image under `--input-dir`, pass `--target`, or drop the version from the list | +| resumed | `version.json` already reported `ok`/`partial` | use `--force` to re-run | + +`--run` exits non-zero when any executed version errored or when nothing ran +at all; skipped steps alone do not fail the sweep, but they are always listed. +An unexplained delta blocks the docs/07 acceptance: an empty +unexplained-delta list is a claim, not a default. + +### 2.6 RM 610 gate (hard, testable) + +Versions numerically `>= 610` or labelled `rm610`/`RM-610` are refused before +any target file is opened — in dry-run too — unless `--approval-token +RM610-APPROVED` is passed or `GREENEYE_RM610_APPROVED=1` is set. The refusal +text points at [`docs/23-RM610-READINESS.md`](23-RM610-READINESS.md) and +states that nothing was downloaded and no target was opened. Do not put a 610 +input in a default list or a cron line before the §6 approval message exists. + +--- + +## 3. New-device scaffold + +### 3.1 Profile + +`--device <name>` uses `--profile FILE` or `<out>/profiles/<name>.json`. A +missing profile is created as a scaffold (`greeneye-device-profile/1`) with the +user-supplied sections and a `where_to_fill` map: + +* `pci` — vendor/device/subsystem IDs (`lspci`, vendor docs); +* `bars` — BAR layout and sizes (guest `lspci -vvv`, host sysfs); +* `fb` — VRAM/BAR1 geometry (host oracle `rmtool mem stats --format=json`); +* `engines` — engine inventory (host oracle `rmtool engine list --format=json`); +* `interrupts` — interrupt topology (MSI-X); +* `rom` — ROM/VBIOS notes and hash; +* `firmware` / `reference` — local image paths (or `--firmware`/`--reference`); +* `firmware_family` / `firmware_version` — the pairing record (docs/07 P2.2); +* `physical_validation` — supervised physical evidence, recorded by the human, + never by this tool. + +Profile data is user-supplied; the tool never guesses a device ID, BAR or +engine list. `config/hardware/*.yaml` remains a design target (see +hardware-software-matrix §6) — do not treat it as live configuration. + +### 3.2 Invocation + +```bash +# dry-run: check profile fields and inputs, create nothing +python3 harness/port_new_device.py --device rtx4090 --profile profiles/rtx4090.json + +# execute: scaffold profile (if absent), probe, symbols, matrix, checklist +python3 harness/port_new_device.py --run --device rtx4090 \ + --firmware ../../firmware/57X-GSP/570.144/gsp-570.144.fwimage.bin \ + --reference firmware/uproc_demo/build/uproc_demo.elf \ + --out dumps/port-rtx4090 +``` + +Output tree: + +```text +<out>/<device>/ + device-port.json / device-port.md + port-checklist.json / port-checklist.md + symbols/<device>.json merged ELF/Ghidra/green-eye DB + provenance + sessions/<firmware>-probe/ recorded first-touch session + portability-matrix/ matrix report + sessions + compare + bundle-manifest.json sha256 of every artifact +<out>/profiles/<device>.json scaffolded profile (when absent) +``` + +The checklist items are the docs/07 evidence checklist plus the docs/22 §3.1 +profile fields; the recommendation is `hold` until the simulator-side items +are complete, `simulator-evidence-complete` while physical validation is +pending, and `go` only when `physical_validation.status` is recorded. The last +row is deliberately outside this tool's reach: simulator-only evidence never +justifies a support claim (docs/22 §3.5, hardware-software-matrix §5.1). + +--- + +## 4. Tests, CI and MCP ties + +```bash +python3 -m unittest tests.test_porting_automation -v +make test # includes these with the rest of the suite +``` + +`tests/test_porting_automation.py` is deterministic: `--help` clean, +dry-runs side-effect free, missing-input handling without tracebacks, resume +detection, config validation, the RM 610 gate (refusal, token, env), and a +tiny one-version QEMU sweep against the demo ELF that must produce the +evidence bundle; QEMU-dependent tests skip cleanly when QEMU or the demo ELF +is absent. The repository CI workflow (docs/21 §5) runs the suite on pushes +and PRs touching the tool tree; a full sweep against real firmware is driven +manually/agentically, dry-run first, because it needs versioned inputs and +symbol DBs — the same policy as `ci_release` (docs/21 §4.1). + +MCP (docs/20) exposes the step-level tools (`probe_firmware`, +`portability_matrix_run`, `mutation_run`); the sweep is the CLI/agent entry +point that sequences them and owns the bundle. An agent should read the dry-run +plan, run, then read `sweep-summary.json`/`device-port.json` and the manifests +rather than recompute hashes. + +--- + +## 5. When to extend + +* **New branch cadence** — add the version to `--versions`/config; if it should + be swept by default, edit `DEFAULT_VERSIONS` and the cadence table above. + Never add an image you do not have; never add 580/610 to a default. +* **New per-version artifact** — add a step function in + `harness/port_driver_versions.py`, register it in `run_version`, and list the + path in `_expected_artifacts`. +* **New device profile field** — add it to `scaffold_profile`, `PROFILE_HELP` + and `profile_missing_fields`, and surface it in the checklist. +* **New architecture (GB10-class, aarch64)** — follow docs/07 P3 first; the + sweep handles the ISA/image shape and the matrix records the first-touch + deltas, but PMU/SOE and partition-model work belongs to the roadmap. +* **New product behaviour** — out of scope: these tools produce evidence, they + never edit `open-iov-vfio`. Product changes need the docs/07 P4 escalation + and a physical reference. + +--- + +## 6. Safety and limits + +* No downloads: discovery and `--extract-runfiles` only read files already on + disk; a driver runfile is never fetched. +* Simulator-only: own QEMU processes on demo/re-hosted ELF or raw firmware; + never a guest, kernel module, mdev, existing gdbstub or physical GPU. +* Bounded: QEMU probe/scenario/mutation budgets plus `--step-timeout`; the + mutation harness refuses non-demo ELFs unless explicitly allowed. +* No support claims: the sweep records what changed; the device checklist + holds until physical validation; the matrix and RUNBOOK process own the + "Supported" label (hardware-software-matrix §1). +* Fidelity limits are docs/06: L0/L1 observations only, L2/L3 (timing, + interrupts, device model) explicitly not claimed. + +## 7. Roadmap mapping + +| Item | Where this doc/tool contributes | +|---|---| +| **R6 — multi-version orchestration** | One sweep over 540→current with version_diff + matrix + mutation and a cross-version report | +| **H4.1 — semantic diffing** | Per-version symbol/corpus/behaviour deltas and recorded sessions | +| **H4.3 — mutation/fuzzing** | Seeded mutation smoke per version | +| **docs/07 P1/P2** | The playbooks become dry-run plans and evidence bundles | +| **docs/23 C5/C7** | Release pipeline extended by a sweep-scale orchestrator; 610 gate reused | +| **H7 — agentic platform** | Machine-readable plans, summaries and bundles for monitored agent runs | + +<!-- change (2026-10-02): state the unadmitted-pilot status and link the +future-eng-ops F2/F3 evaluation from the document that ships the scaffolds; +scope: intro only; benefit: no reader mistakes the sweep for admitted release +tooling. --> diff --git a/debug/risc-v-gsp-rm-vm/docs/29-PACKAGING-AND-TRIALING.md b/debug/risc-v-gsp-rm-vm/docs/29-PACKAGING-AND-TRIALING.md new file mode 100644 index 0000000..8be3704 --- /dev/null +++ b/debug/risc-v-gsp-rm-vm/docs/29-PACKAGING-AND-TRIALING.md @@ -0,0 +1,246 @@ +# 29 — Packaging and trialing: sim + injected code as a deliverable + +> **# TOOL IN DEVELOPMENT - NOT READY YET!** See 00 for status. +> This is a development-only convenience layer over docs/12 (injection SDK), +> docs/16 (native simulator) and docs/24 (device map). It makes no hardware, +> timing or security claim; the real `.fwimage` path is honestly blocked on +> H2.2 and says so in every result. + +## 1. Purpose + +green-eye can already load an injected service into the native simulator and +invoke it (docs/12, docs/16 §1.1, `harness/value_demos/s06_inject_service.py`). +That evidence lives in the tool tree: a reviewer needs the checkout, the built +demo and SDK, and the exact command sequence. `harness/package_sim.py` turns +"the s06 scenario" into a self-contained, hash-addressed package that a third +party can run, inspect and compare: + +* one directory (or deterministic `.tar.gz`) containing the target, the service + blob, the scenario and a manifest of hashes; +* one entry point (`run.sh`) that boots the target **natively**, registers the + injected service, invokes it directly and through the firmware dispatcher, + runs a hook/watch pass, and writes `results.json` + `results.md`; +* a machine-readable manifest so a recipient can verify exactly which inputs + produced a result before trusting it. + +It exists to make a code addition trialable and reviewable, not to ship +anything: the eventual target is a modified, **unsigned** GSP image for the +simulator with tested injected code (§7). + +## 2. Exact usage + +All commands run from `open-iov/debug/risc-v-gsp-rm-vm`. Prerequisites: +`make demo && make sdk` (or pass `--service`), Python 3.10+; no QEMU needed. + +```bash +# 1) Demo trial (defaults: demo uproc + sdk/build/injected_demo.elf, slot 6) +python3 harness/package_sim.py --out /tmp/greeneye-trial-demo + +# 2) Build, archive and immediately run the trial +python3 harness/package_sim.py --out /tmp/greeneye-trial-demo --tar --run +/tmp/greeneye-trial-demo/run.sh # or run it yourself + +# 3) Combined artifact: the service is pre-injected at a fixed offset +python3 harness/package_sim.py --pack-image --out /tmp/greeneye-trial-packed +python3 harness/package_sim.py --pack-image --tar --run --out /tmp/greeneye-trial-packed + +# 4) Real 570.144 fwimage: image + service + lenient device map (blocked, see §5) +python3 harness/package_sim.py --target image \ + --image /path/to/gsp-570.144.fwimage.bin \ + --device-map dumps/570.144-device-map.json \ + --out /tmp/greeneye-trial-image + +# 5) Custom scenario and slot +python3 harness/package_sim.py --script my_scenario.py --slot 6 --out /tmp/my-trial + +# 6) Tests +python3 -m unittest tests.test_package_sim -v # 6 tests, ~4 s +``` + +Measured on the reference host (2026-10-02, CPython 3.12): a demo build plus +`run.sh` reports `trial: ok` in ~1 s wall; the test file (6 tests, including +the real-image bounded probe) runs in ~4.1 s. + +Flags: `--target demo|image` (default demo), `--image`, `--service`, +`--device-map`, `--script`, `--slot 0..7` (default 6), `--pack-image`, +`--tar`, `--run`, `--run-timeout`. `--out` must be empty or absent; builds do +not overwrite. + +## 3. Package layout and manifest schema + +``` +<out>/ + README.md what this is, how to run, what to expect, safety + run.sh single entry; finds the tool root (GREENEYE_HOME) and runs scenario.py + manifest.json format/version + hashes + component list + run contract + scenario.py the pre-scripted scenario as run (builtin or --script) + payload/ + <target>.elf|fwimage the target as supplied + injected_demo.elf the service blob + device_map.json only for image packages when no --device-map was given + combined.elf only with --pack-image + preinject.json only with --pack-image (placement config) + results/ produced by run.sh: results.json, results.md, session/ +``` + +`manifest.json` (`greeneye-trial/1`): + +| Key | Contents | +|---|---| +| `format`, `greeneye_version`, `created`, `results_schema` | identity; `created` is the only timestamp | +| `target` | `kind` (demo/image), `file`, `input` (repo-relative when possible), `sha256`, `entry`, `min_vaddr`, `max_vaddr`, `bias`, `runnable`, `limitations` | +| `service` | `file`, `input`, `sha256`, `entry`, `entry_symbol` (`ge_demo_service_dispatch`), `slot` | +| `device_map` | `file`, `input`, `sha256`, `lenient`, `learn` (null for a plain demo) | +| `script` | `kind` (builtin/custom), `name`, `input`, `sha256` | +| `preinject` | `file`, `runnable`, `reason`, `target_load_addr`, `target_entry`, `service_load_addr`, `service_offset`, `service_entry`, `entry_addr`, `slot`, `size`, `segments` (null without `--pack-image`) | +| `scenario` | demo boot/settle seconds, image instruction budget, `memory_mb`, expected call values | +| `run` | `command`, expected result paths, `exit_nonzero_on: "assertion failure"` | +| `safety`, `determinism` | policy text carried into the package | +| `components` | sorted `{path, role, bytes, sha256}` for every packaged file except the manifest itself | + +`results.json` (`greeneye-trial-results/1`): `status` (`ok`/`failed`/ +`blocked`), `success`, `backend`, `target`, `service` (load/entry addresses, +`preinjected`), `calls` (remote vs firmware with expected/agree), `hooks` +(`rpc_dispatch` hits and first entries), `watches` (`gsp_ctrl.result` hits), +`control`, `handlers_slot`, `uart_excerpt`, `session`, `assertions` +(`checks`, `failed`), `limitations`, plus `error` and the blocked fields +(§5) where applicable. `results.md` renders the same. + +## 4. How a user trials it and reads results + +1. Receive `<package>.tar.gz` plus its sha256 out of band; verify + `sha256sum <package>.tar.gz`. +2. Extract and run: `./run.sh` (add `--budget N` for image packages). The + script finds the green-eye tree by walking up, or via + `GREENEYE_HOME=/path/to/risc-v-gsp-rm-vm`. Exit codes: `0` pass (or the + documented blocked image status), `1` assertion failure, `3` tree not found. +3. Read `results/results.md` first, then `results/results.json` for machines. + Anything other than `status: ok` on a demo package is a finding, not a + flake: `assertions.failed` names the check (`handlers_slot_registered`, + `remote_ping`, `routes_agree`, `dispatch_hook_hits`, `uart_banner`, ...). +4. For deeper review, point the tools at `results/session/`: + `./greeneye.sh analyze <pkg>/results/session` and + `python3 -m greeneye compare baseline/results/session candidate/results/session`. +5. Verify provenance: each `components[].sha256` must match the packaged file; + `target.sha256`/`service.sha256` identify the inputs. + +The demo package asserts: the service is registered in +`gsp_ctrl.handlers[slot]`; direct and mailbox invocations agree with the SDK +ABI (`cmd 1: a0 ^ 0xbeef`, `cmd 2: a0 + a1`); at least one `rpc_dispatch` +hook hit and one `gsp_ctrl.result` watch hit; and the UART banner/self-test +lines are present. A bisected or broken service fails these and exits 1. + +## 5. The `--pack-image` path and its honest limits + +For the demo, `--pack-image` writes `payload/combined.elf`: the demo's PT_LOAD +segments plus the service's segments relocated to `target_base + 0x01000000` +(the same address `greeneye.inject.load_service` uses by default), merged into +one ELF64. `payload/preinject.json` records the placement, and the scenario +boots the combined ELF, writes the entry into the handler slot and skips the +separate blob load. Round-trip verified in +`tests/test_package_sim.py::test_pack_image_round_trip`: call results are +identical to the load/register path and `service.preinjected` is `true` in +`results.json`. This is the trial starting with the injected code already part +of the simulated system. + +For the real `.fwimage`, the package contains the image, the service blob and +a lenient `greeneye-device-map/1` map (given or emitted empty with +`learn: true`). **The native simulator cannot boot it yet**: the H2.2 device +model is absent. The scenario therefore runs a bounded, deterministic lenient +first-touch probe and records `status: blocked`, +`blocked_on_device_model: true`, `instructions_executed`, the stop PC, the +unmodelled MMIO list and (when learning) `results/learned_device_map.json`. +Reference measurement: a 200 000-instruction budget (199 998 executed), +stopped at `pc=0x84009070` (firmware `0x4009070`), exactly one unmodelled +access at `0x84009014` reading `0x2000000000000a00`, matching docs/24 §5. No +service is loaded, because no RM service registration point exists yet (M5) and the default load +address would overwrite the image. Exit is 0 for the documented blocked +status; a thrown error is recorded and exits 1. + +## 6. Sharing the results of a code addition + +1. Author builds the package with the service under review + (`service.sha256` is the code addition's identity) and runs it. +2. Attach `manifest.json`, `results/results.json`, `results/results.md` and + the `results/session/` directory; pre-injection evidence is in the session + as `service_injected`, `call_enter`/`call_return`, `watch_hit` and `bp_hit` + events, so the claim is replayable. +3. A baseline package with the previous service is diffed with + `greeneye compare`; the delta is the behavior change. +4. A reviewer re-runs the received package on the tool tree, verifies hashes + and expects the same `status`/call results. Divergence is a finding against + the package or the tree, not a nuance. +5. Nothing here is product evidence by itself: any conclusion that would + change `open-iov-vfio` still escalates to a physical capture (docs/06 §6, + docs/25 §2.1). The package makes that physical confirmation cheap, it does + not replace it. + +## 7. Eventual path: a modified GSP image for the simulator + +Once H2.2 can boot the real image (docs/16 M4) and the RM service registry +exists (M5), the same pipeline packages a **modified, unsigned** image: + +1. take the original `.fwimage` (hash in `target.sha256`); never a signed + production image, never a copy that can be flashed; +2. record the patch/injection recipe (`preinject`) and the verified device map + as versioned profile data keyed by `(device, firmware-family, profile)` per + docs/16 §7; +3. run the package's scenario, which boots the image under the H2.2 device + model, registers the tested service with the simulated RM core, and asserts + its behavior end to end; +4. attach the session as evidence and escalate any product-relevant finding to + hardware. + +Until M4/M5, image packages are deliberately non-runnable and labelled so; the +`preinject.runnable: false` + `reason` fields and the blocked result keep the +gap visible instead of implying a boot. + +## 8. Security and safety + +* The package executes only inside the in-process simulator: no hardware + access, no mdev, no guest, no kernel module, no network. `run.sh` invokes + Python in the tool tree found locally. +* Injected code runs with the simulator's full privileges and may corrupt + simulated firmware state (docs/12 §9). Treat every package as untrusted + code: run only packages whose sha256 and `manifest.json` you have verified, + in a scratch copy of the tree, and restart clean after a crash. +* Never use a package to inject into a guest, a shipped environment or real + hardware; never sign, flash or install a modified `.fwimage`. +* `/tmp` outputs are scratch; packages and sessions contain firmware bytes and + are development artifacts, not for redistribution outside the project. + +## 9. Review/evaluation criteria (keep vs prune) + +Keep the feature only while it earns its maintenance. This is an **unadmitted +pilot** under `eng-ops.md` §2: the evaluation recorded in +[`future-eng-ops.md`](../../../docs/future-eng-ops.md) F4 is the admission +decision. It is worth keeping if: + +* `python3 -m unittest tests.test_package_sim` stays green in CI in well under + a minute, including the bounded real-image probe; +* two builds under a fixed `SOURCE_DATE_EPOCH` are byte-identical + (manifest and `.tar.gz`) and an intentionally broken service fails the + assertions and exits 1 (add that negative test if it is missing); +* at least one code addition was actually reviewed from a package by someone + who did not author it, with hashes verified and results reproduced; +* the tar carries enough provenance (`manifest.components`, target/service/ + device-map/script hashes, `preinject` placement) for a reviewer to trust a + result without touching the author's checkout; +* the image path converts into a real H2.2 boot when M4/M5 land, and its + `blocked` status is removed rather than quietly left in place. + +Prune or rework it if: packages are never shared or reviewed (the s06 demo is +enough); the M5 native service registry subsumes `--pack-image`; the manifest +format churns without consumers; the real-image path stays permanently blocked +and adds only maintenance; or package results are being quoted as product or +hardware evidence. + +Evidence for this document: `harness/package_sim.py`, +`tests/test_package_sim.py` (6 tests), docs/12, docs/16 §1.1, docs/24 §4–5, +docs/25 §5–6. + +<!-- change (2026-10-02): use the tree's stdlib unittest runner instead of an +undeclared pytest dependency, and state the unadmitted-pilot status with the +future-eng-ops F4 evaluation; scope: §2 command, §9 only; benefit: the test +command runs with the same interpreter as CI and the keep/prune decision is +linked. --> diff --git a/docs/01-orientation/01-purpose-and-scope.md b/docs/01-orientation/01-purpose-and-scope.md index ff13603..278ad0d 100644 --- a/docs/01-orientation/01-purpose-and-scope.md +++ b/docs/01-orientation/01-purpose-and-scope.md @@ -131,15 +131,16 @@ document: * **Working today**: the virtual device, the GSP boot handshake, the RPC transport, the resource-manager emulation surface the guest driver needs to - probe and enumerate, guest `nvidia.ko` and `nvidia-smi`, and CUDA user-space - startup through `cuInit` / `cuDeviceGet`. -* **Needs improvement**: reply fidelity and completion translation first — - context creation currently ends in a forced `rc=999` on a completion wait, - and some consumed reply fields are still generic fallbacks — then GPU work - forwarding (the reason CUDA kernels do not yet run; the backend is chosen), - the memory/page-table model, Windows guest support, - non-KVM hypervisors, and performance instrumentation. Each of these is - tracked with explicit "known-unknowns" in + probe and enumerate, guest `nvidia.ko` and `nvidia-smi`, and the full CUDA + driver-API path through context creation, allocation and verified copies + (the driver-API smoke passes). +* **Needs improvement**: engine execution first — the backend is live and + CE-class copies and their releases are forwarded to the physical copy engine + (default off) with a fault relay, but CUDA kernel execution is blocked on GR + engine-context validity — then the remaining reply fidelity (some consumed + fields are still generic fallbacks), the memory/page-table model, Windows + guest support, non-KVM hypervisors, and performance instrumentation. Each of + these is tracked with explicit "known-unknowns" in [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md); for the CUDA analysis area, start at [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md). diff --git a/docs/02-getting-started/01-install-and-first-run.md b/docs/02-getting-started/01-install-and-first-run.md index c1a8f5d..9ae81c2 100644 --- a/docs/02-getting-started/01-install-and-first-run.md +++ b/docs/02-getting-started/01-install-and-first-run.md @@ -142,10 +142,11 @@ cd cuda-samples/Samples/1_Utilities/deviceQuery && make && ./deviceQuery ``` **Expectation today:** driver probe, `nvidia-smi`, device enumeration, and the -first CUDA calls (`cuInit`, device queries) work. `cuCtxCreate` still fails -with `999` because the emulated engine list is missing the async copy engine; -full kernel execution is not reached yet. Before drawing conclusions from a -CUDA failure, read the honest status in +full CUDA driver-API smoke (`ALL OK` 64/64, copies matched) work. +Engine execution is the remaining gap: CE-class copies are forwarded to the +physical copy engine (default off), while CUDA kernel execution is blocked on +GR engine-context validity. Before drawing conclusions from a CUDA failure, +read the honest status in [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) and the debugger's guide [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md). @@ -183,8 +184,8 @@ Developer debug knobs (module parameters) are documented in | Module will not load | `dmesg`, `modinfo open-iov-vfio.ko`, stale module state → `cleanup-crash` | | No mdev type for your GPU | [hardware matrix](../06-portability/hardware-software-matrix.md) | | Guest sees wrong PCIe link speed/width | root-port attachment + config-space mirror (debugging doc) | -| `nvidia.ko` probe fails in guest | guest dmesg + `lspci -vvv` + handoff "driver lifecycle" | -| `nvidia-smi` works, CUDA does not | expected today (`cuCtxCreate` = 999, engine-list gap); see [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) and [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md) | +| `nvidia.ko` probe fails in guest | guest dmesg + `lspci -vvv` + [driver lifecycle field guide](../03-architecture/07-driver-lifecycle-field-guide.md) | +| `nvidia-smi` works, CUDA kernel execution does not | expected today: the driver-API smoke passes; CE copies are forwarded (default off) and GR execution is blocked on engine-context validity; see [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) and [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md) | | VM boot fails or the guest hangs | `scripts/stack_manager.sh cleanup-crash`, then reboot | ## 10. Uninstall diff --git a/docs/03-architecture/01-system-overview.md b/docs/03-architecture/01-system-overview.md index 5c2ba6c..fa328ae 100644 --- a/docs/03-architecture/01-system-overview.md +++ b/docs/03-architecture/01-system-overview.md @@ -60,7 +60,7 @@ HOST MODULE open-iov-vfio.ko ├─ virtual device: config space, BAR0/BAR1/BAR3 windows, MSI-X ├─ emulation core: GSP boot + GSP-RM (RPC msgq, controls, objects, events) ├─ memory model: guest RAM access, VA↔PA maps, VRAM backing - └─ work path: channel/push forwarding to the physical GPU [goal; software bridge today] + └─ work path: channel/push forwarding to the physical GPU [CE copies forwarded (default off); GR open] │ module-owned device + real GSP session HOST BACKEND the GPU's own hardware + GSP firmware, driven by this module │ (the distribution nvidia.ko stays unmodified and unbound in @@ -170,10 +170,12 @@ decompiled and mapped (see `wiki/GSP-570.144W-Firmware-Analysis.md` and without server-side object management. Where the driver consumes a reply value, that value is still being corrected (the reply-fidelity ladder). 5. `nvidia-smi -L` enumerates the emulated device and CUDA user-space starts - (`cuInit` / `cuDeviceGetCount` / `cuDeviceGet` succeed). The client reaches - `cuCtxCreate`, which currently ends in a forced `rc=999` on a completion - wait; the copy-engine method streams execute in software as a bridge, and - the execution backend is not built yet (see + (`cuInit` / `cuDeviceGetCount` / `cuDeviceGet` succeed); context creation + completes and the full CUDA driver-API smoke passes (`ALL OK` 64/64, copies + matched). The execution backend is live: CE-class copies are + forwarded to the physical copy engine (default off) with engine-authored + semaphore releases and a fault relay; kernel execution is blocked on GR + engine-context validity (see [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)). The CUDA-side contract and the reply-fidelity rules are documented in diff --git a/docs/03-architecture/02-guest-stack-and-cudasw.md b/docs/03-architecture/02-guest-stack-and-cudasw.md index e841342..f9fe249 100644 --- a/docs/03-architecture/02-guest-stack-and-cudasw.md +++ b/docs/03-architecture/02-guest-stack-and-cudasw.md @@ -88,14 +88,15 @@ open /dev/nvidia* → cuInit → UVM registration (per process) ``` The initialization path is where compatibility is won or lost; the runtime -path is where *execution* is won or lost. Open-IOV's current focus is -pre-forwarding correctness: `cuInit` / `cuDeviceGet` succeed, and the client -reaches context creation, which currently ends in a forced `rc=999` on a -completion wait; the engine inventory is complete and the forwarding decision -is made — a module-owned GPU running its real GSP firmware as the -resource-manager server (see +path is where *execution* is won or lost. Context creation and the full CUDA +driver-API smoke now pass (`ALL OK`, copies matched), so the current focus is +execution: the forwarding decision is made and the backend is live — a +module-owned GPU running its real GSP firmware as the resource-manager server +(see [`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md) -and [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)). +and [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)) — with +CE-class copies forwarded to the physical copy engine (default off) and GR +kernel execution blocked on GR engine-context validity. For the CUDA userspace side of this stack — its architecture, how it is built and executes, and the exact kernel/GPU interface contract we must satisfy — diff --git a/docs/03-architecture/03-gsp-and-gsp-rm.md b/docs/03-architecture/03-gsp-and-gsp-rm.md index 7f206ba..10e6a73 100644 --- a/docs/03-architecture/03-gsp-and-gsp-rm.md +++ b/docs/03-architecture/03-gsp-and-gsp-rm.md @@ -59,8 +59,10 @@ firmware: FWSEC raised WPR2 with a zero FRTS error code, the SEC2 booter halted with mailbox0 = 0, the RISC-V core reported active (`RISCV_STATUS=0x00000035`, active bit 0) and `FALCON_OS` read 0. The server's first message was then a status-queue event, function `0x1002` (`NV_VGPU_MSG_EVENT_GSP_RUN_CPU_SEQUENCER`, -`rpc_result=0`), carrying a command buffer the host must execute; -`GSP_INIT_DONE` (`0x1001`) follows only after that sequencer runs. The +`rpc_result=0`), carrying a command buffer the host must execute. Once that +sequencer runs, `GSP_INIT_DONE` (`0x1001`) is delivered with `rpc_result=0`, +the static-configuration reply is served from the observed device tables, and +the session remains live until the unload RPC releases it at teardown. The host-side queue pair behind the exchange is one shared page table plus a 256 KB command queue and a 256 KB status queue of 4 KB elements. @@ -229,8 +231,10 @@ is not lost. The engine-inventory row was the CUDA blocker at the time of writing: `libcuda` filters the engine list it receives against the class list and expects the async copy engine to be present, because the secondary (async) -engine record is otherwise zeroed. That inventory is now complete; the -context-creation wait is later, in the completion path. See +engine record is otherwise zeroed. That inventory is now complete and context +creation passes; the current frontier is engine execution, where CE-class +copies are forwarded to the physical copy engine (default off) and GR kernel +execution is blocked on GR engine-context validity. See [`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md) section 3.7 for the offsets and the consumed fields, and [`../../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../../userspace/cuda/docs/03-driver-and-gpu-interface.md) diff --git a/docs/03-architecture/04-uvm.md b/docs/03-architecture/04-uvm.md index 4c8a0c9..9396e6a 100644 --- a/docs/03-architecture/04-uvm.md +++ b/docs/03-architecture/04-uvm.md @@ -50,17 +50,19 @@ Each stage has a distinct failure signature; our debugging docs map them. | Requirement | Detail | Our status | |---|---|---| | GPU identity | must match the uuid from RM | ✅ | -| CE capabilities | per-engine support bits; UVM *requires* `supported && !grce && sysmem` for CPU_TO_GPU/GPU_TO_CPU channels | 🟡 the caps we answer are accepted and the inventory now includes the async copy engine; completion/semaphore plumbing is the current frontier | +| CE capabilities | per-engine support bits; UVM *requires* `supported && !grce && sysmem` for CPU_TO_GPU/GPU_TO_CPU channels | 🟡 the caps we answer are accepted, the inventory now includes the async copy engine, and CE copies are forwarded to the physical copy engine (default off) with engine-authored releases | | Channel creation | RM channel objects with GPFIFO/userd/instance parameters | ✅ objects created | -| Pushes executed | UVM's first action on a new channel is an *init push* followed by a semaphore-tracked wait | 🟡 the host decodes method streams and executes memsets, copies and semaphore releases in software (a bridge); no kernel is forwarded to the physical GPU yet | +| Pushes executed | UVM's first action on a new channel is an *init push* followed by a semaphore-tracked wait | 🟡 the host decodes method streams and executes memsets, copies and semaphore releases in software (a bridge); CE-class copies and releases are additionally forwarded to the physical copy engine (default off); no GR kernel is forwarded yet (blocked on GR engine-context validity) | | Page trees | UVM initializes page trees with its own pushes; the *server-owned* ranges must be populated by us | 🟡 | -| Fault delivery | fault buffer + notification; replay/cancel semantics | 🔴 | +| Fault delivery | fault buffer + notification; replay/cancel semantics | 🟡 backend-side fault observation/relay exists (default off: notifier write + event post); the guest replay/cancel contract is open | | Semantics of completions | tracker values must advance exactly | 🟡 | -**Current frontier (2026-09-29):** the client reaches `cuCtxCreate` and -currently ends in a forced `rc=999` on a completion wait; the engine inventory -is complete, and the remaining work is completion translation plus the chosen -execution backend +**Current frontier (2026-10-02):** context creation and the full CUDA +driver-API smoke pass (`ALL OK`, copies matched); the execution backend is +live and CE-class copies are forwarded to the physical copy engine (default +off) with engine-authored releases and a fault relay. GR kernel execution is +blocked on GR engine-context validity; the remaining UVM work is completion +translation and the guest-side fault replay/cancel contract ([`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)). Failed-context teardown has produced guest kernel oopses in `uvm_pmm_devmem_page_to_va_space` in earlier runs. The CUDA-side diff --git a/docs/03-architecture/05-gmmu-amap-hal-gr.md b/docs/03-architecture/05-gmmu-amap-hal-gr.md index 76d4200..67ceeca 100644 --- a/docs/03-architecture/05-gmmu-amap-hal-gr.md +++ b/docs/03-architecture/05-gmmu-amap-hal-gr.md @@ -206,5 +206,4 @@ section 3.7 for the record layout the driver's engine walker consumes. | Firmware structure and dispatchers | `wiki/GSP-570.144W-Firmware-Analysis.md` | **Rule:** tracked documentation describes components and behaviour, with our -own observations, IDs and layouts. Exact analysis sources live in the -maintainers' study notes. +own observations, IDs and layouts. diff --git a/docs/03-architecture/06-host-module-and-hybrid-model.md b/docs/03-architecture/06-host-module-and-hybrid-model.md index d6b96e7..0dbc89a 100644 --- a/docs/03-architecture/06-host-module-and-hybrid-model.md +++ b/docs/03-architecture/06-host-module-and-hybrid-model.md @@ -140,7 +140,10 @@ section 3.7 for the analysis. method streams and executes the work it understands in software - memsets, copies and semaphore releases - and completes the corresponding semaphores. That is a correctness bridge for memory operations, not the data plane: no -CUDA kernel is forwarded to the physical GPU yet; the chosen execution backend +CUDA kernel is forwarded to the physical GPU yet (default-off CE copy +forwarding, engine-authored release relay and fault relay are implemented; see +[`11-physical-backend-guest-wiring.md`](11-physical-backend-guest-wiring.md)); +the chosen execution backend (a module-owned GPU running its real GSP firmware as the resource-manager server) is in [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md), with the @@ -149,10 +152,11 @@ trade-offs in section 3. **Subsystem status note.** Some components are compiled but not currently -wired into the live path: the physical-GSP load/boot path is skipped in favour -of emulation, and the runtime firmware analyzer, version tables, hardware -abstraction tables and dynamic RPC dispatcher are not yet called by the live -flow. Treat their descriptions here as intended design, not active code. +wired into the live path: the physical-GSP load/boot path is default-off and +runs only behind the supervised claim gates, and the runtime firmware +analyzer, version tables, hardware abstraction tables and dynamic RPC +dispatcher are not yet called by the live flow. Treat their descriptions here +as intended design, not active code. ### 3.3 Event channels and interrupts @@ -263,6 +267,22 @@ emulated device inspectable without rebuilding: | `sema_map_add=<va>,<pa>` | teach the service one verified `{GPU VA -> guest PA}` pair | | `dump_guest=<gpa>,<len>` | dump guest physical memory (length capped) | +The physical-backend path adds a second, default-off parameter set: +`gsp_phys_gpu`, `gsp_phys_hazard_ack`, `gsp_phys_fw`, `gsp_phys_mdev_owned`, +`gsp_phys_static_live`, `gsp_phys_ctrl_forward`, `gsp_phys_work_forward`, +`gsp_phys_guest_mem`, `gsp_phys_work_defer_release`, +`gsp_phys_work_hw_primary`, `gsp_phys_relay`, `gsp_phys_fault_relay`, +`gsp_phys_fault_inject`, `gsp_phys_rmcall_engine`, +`gsp_phys_rmcall_gr_promote`, `gsp_phys_repower`, `gsp_phys_layout`, +`gsp_phys_boot`, `gsp_phys_boot_dryrun`, `gsp_phys_nocat` and the +`gsp_phys_fwsec*` stage controls. Their semantics, the physical-backend +debugfs inventory (`gsp-phys-state`, `gsp-phys-rm-probe`, `gsp-phys-rm-call`, +`gsp-phys-unload-rpc`, `gsp-phys-guest-ram`/`gsp-phys-guest-gpa`, +`gsp-phys-log-dump`, `gsp-phys-status-drain`) and the supervised runbooks are +maintained in the engineering-operations document +([`../eng-ops.md`](../eng-ops.md), Part II); this document does not duplicate +those tables. + These are development-grade but production-useful: they are how the memory model was reverse-engineered, and they remain the fastest way to diagnose a mismatch. Their workflows are documented in @@ -312,8 +332,9 @@ in `kernel-open` first). | Message queues + RPC + all init controls observed from stock 570.144 | implemented | | Guest sysmem access + VA↔PA discovery | implemented (debug-grade, to be productized) | | Push decoding + semaphore completion | demonstrated live; being productized | -| Page-table population for server-owned ranges | designed, not implemented | -| GPU work forwarding (GR/CE execution) | designed, not implemented | -| Execution backend (module-owned GPU + direct real-GSP session) | mechanism chosen; ownership + firmware staging verified live; power-domain lock implemented, pending supervised verification; real GSP boot not started ([`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)) | +| Guest-RAM physical path | implemented default-off: guest RAM is file-backed shared memory and GPA→HPA goes through its page cache, verified live on the CE | +| Page-table population for server-owned ranges | implemented default-off: forwarded CE copies map resolved host pages through module-built GMMU scratch page tables; real guest page tables remain | +| GPU work forwarding (GR/CE execution) | CE forwarding, engine-authored release relay and fault relay implemented default-off and verified live; GR execution open | +| Execution backend (module-owned GPU + direct real-GSP session) | live: boot, init-done, full static-config reply and clean releases proven; RM client allocates objects/channels and has executed real CE copies; currently in the designed locked fail-safe after a GR context-promote wedge ([`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)) | | Real VRAM backing | designed, not implemented | | Windows guest, Xen transport, aarch64 hosts | designed, not implemented | diff --git a/docs/03-architecture/07-driver-lifecycle-field-guide.md b/docs/03-architecture/07-driver-lifecycle-field-guide.md index 1bfe912..73b007e 100644 --- a/docs/03-architecture/07-driver-lifecycle-field-guide.md +++ b/docs/03-architecture/07-driver-lifecycle-field-guide.md @@ -107,7 +107,7 @@ fallback). Representative controls: | `0x90f10106` | vaspace reserved-PDE contract | **carries page-table physical addresses** (see GSP-RM doc) | | `0x20800123` / `0x20800170` | `GPU_GET_ENGINES` / `GET_ENGINES_V2` | engine inventory (filtered by the class list / classDB); values are consumed — the guest must see the full engine list, including the async copy engine (`COPY14`); that inventory is now complete, and the context-creation wait lives later in the completion path | | `0x800201` | `GPU_GET_CLASSLIST` | class list (classDB); must agree with the engine list and the CE caps controls | -| `0x2080012b` | GPU_PROMOTE_CTX (560-byte params) | issued immediately before context creation; fallback generic `SUCCESS`, consumed fields not yet coherent | +| `0x2080012b` | GPU_PROMOTE_CTX (560-byte params) | historically issued immediately before context creation and answered by the generic fallback. Context creation now returns; in the host GR ladder this promote is the load-bearing GR context-validity operation, and a wrong shape is a known server wedge (no retry; reboot policy - [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md) §7) | | `0x20803039` | NVLink masks/VBIOS info | fallback: generic `SUCCESS` with zeroed params | | `0x20800a70` | BUS flush with sysmembar | fallback: generic `SUCCESS` with zeroed params; UVM issues these frequently | @@ -177,6 +177,14 @@ start at the execution backend is in [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md). +**Update (2026-10-02):** context creation now returns (the driver-API smoke +completes end to end) and the frontier is execution. The host GR ladder +allocates a GR0 channel and a compute-class object, but its first semaphore +push faults with an uncommitted GR context (`CTXNOTVALID`), and the +`GPU_PROMOTE_CTX` attempt that followed wedged GSP-RM; the device then sits in +the designed locked fail-safe (see +[`08-physical-gpu-integration.md`](08-physical-gpu-integration.md) §7/§8). + **Log signatures (guest side):** ``` nvidia-uvm: ... Tracker entry for value N (sema VA 0x...) channel ID ... CE ... diff --git a/docs/03-architecture/08-physical-gpu-integration.md b/docs/03-architecture/08-physical-gpu-integration.md index 2ea44a1..532cd77 100644 --- a/docs/03-architecture/08-physical-gpu-integration.md +++ b/docs/03-architecture/08-physical-gpu-integration.md @@ -1,10 +1,12 @@ # Physical GPU Integration: How Work Reaches Real Hardware -> **Status: mechanism chosen; ownership + firmware staging implemented; -> power-domain lock verified; first real GSP boot and first live GSP-RM -> exchange complete (2026-10-01); host-side initialization follow-up and a -> clean release are in progress.** The control plane works today: the guest's -> driver initializes and the CUDA driver-API baseline passes end to end +> **Status: backend live; init-done, static configuration and clean releases +> proven; CE work forwarding with an engine-authored release and fault relay +> implemented default-off; GR execution is the open wall. The device is +> currently in the designed locked fail-safe awaiting a supervised reboot +> after a GR context-promote attempt wedged GSP-RM.** The control plane works +> today: the guest's driver initializes and the CUDA driver-API baseline +> passes end to end > (context creation, allocation, copies, 64/64 payload round-trip), and the > guest's decoded work is executed in software as a development bridge. This > document records the interfaces that actually exist, the mechanism options, @@ -145,8 +147,10 @@ re-bind. While the module owns the device it verifies that the GPU still answers config space before proceeding and aborts on any power-loss signature (the companion controller reporting a lost root hub, or a failed config read). The distribution driver is then rebound with the lock still held, its probe is -allowed to complete, and only then is runtime power management restored to -automatic. Any power-loss signature aborts the sequence and forbids a retry in +allowed to complete, and the D0/D3cold locks stay latched by default +(`gsp_phys_repower=0`); restoring runtime power management is an explicit +opt-in through the power-restore path, taken only when the operator chooses +it. Any power-loss signature aborts the sequence and forbids a retry in that boot: an out-of-band platform power cut bypasses the lock and would produce the identical signature, and retrying would be a second dice roll on a host that has already lost the board. @@ -210,19 +214,22 @@ bridge of the GPU's domain), built from the live PCI topology method-data block is sent invalid/zero: the boot path deliberately makes no ACPI/AML calls and every consumer tolerates the no-data encoding. The server waits for the host to execute the sequencer command buffer carried in that -event before it raises `GSP_INIT_DONE` (0x1001). **A plain release is not yet -possible:** the Booter Unload gate requires all four secure scratch14 handoff -bits (24–27); a -host-driven boot sets only bit 24, and bit 27 is set by the server's shutdown -path (host RPC `UNLOADING_GUEST_DRIVER`; implemented in the module as the -rung-4b command-queue TX + shutdown path and pending its first live run), -while the -host-side seed/unlock alternative is fuse-locked on this part. Until that RPC -has run in a live session, a green session is left claimed with the harness -`--keep` flag and returning the GPU to the distribution driver takes a power -cycle. The supervised harness modes are `--boot` (the only write mode: full -boot lifecycle), `--boot-dryrun` (zero-write staging rehearsal) and `--keep` -(leave a verified claimed session running); `--fwsec` is retired. +event before it raises `GSP_INIT_DONE` (0x1001). **Clean release is +proven:** since 2026-10-01 the unload RPC path delivers a clean release with +no reboot on every cycle — the +Booter Unload gate requires all four secure scratch14 handoff bits (24–27); a +host-driven boot sets bit 24, and the server's shutdown path (host RPC +`UNLOADING_GUEST_DRIVER`, rung 4b, §8) sets the rest, while the host-side +seed/unlock alternative is fuse-locked on this part. The exception is a wedged +server: if the server never sets its bit, teardown cannot clear the protected +region and the fail-safe refuses the hand-back (no rebind, D0 locks held, a +reboot with approval required; §7). The supervised harness modes are `--boot` +(the only write mode: full boot lifecycle), `--boot-dryrun` (zero-write +staging rehearsal) and `--keep` (leave a verified claimed session running); +`--fwsec` is retired. The full parameter, debugfs and runbook inventory for +this path is maintained in the engineering-operations document +([`../eng-ops.md`](../eng-ops.md), Part II); this document does not duplicate +those tables. ### 4.4 Host-side resource-manager client @@ -277,12 +284,12 @@ per-context scheduling. Tenant policy and fairness are a later layer on top. | Driver-API baseline (init, devices, context, allocation, copies, free, destroy) | done — re-verified 64/64 payload round-trip on the current build | | Software-executed bridge for decoded memory/state operations | in progress (memsets, copies, releases) | | Device ownership + firmware staging | verified — claim/release and staged-image parse both verified live; the claim path is hazard-gated (§4.2) | -| Power-domain lock across the claim window | verified — 3/3 supervised claim/release cycles (2026-10-01; §4.2, §8) | -| Real GSP boot by the module | first live boot 2026-10-01 — FWSEC WPR2 up, Booter Load `M0=0`, RISC-V active; clean release pending the server shutdown RPC | -| Host-side resource-manager client for real objects/channels | transport alive 2026-10-01 — first real GSP-RM event (0x1002) received; init-done (0x1001) awaits host sequencer execution | -| Work translation and real submission | not started | -| Completion relay through real hardware | not started | -| CUDA kernel execution (`vectorAdd`) | not started | +| Power-domain lock across the claim window | verified — 3/3 supervised claim/release cycles (2026-10-01; §4.2, §8); latched by default since the G0 bundle (`gsp_phys_repower=0`) | +| Real GSP boot by the module | done — full boot through init-done (`0x1001`, `rpc_result=0`) and a clean release with no reboot on every supervised cycle since 2026-10-01 | +| Host-side resource-manager client for real objects/channels | done — full static-config reply served; real memory, vaspace, channel-group and channel objects allocated; real CE copies executed on the physical engine | +| Work translation and real submission | CE forwarding and engine-authored release relay implemented default-off and verified live; GR execution open | +| Completion relay through real hardware | implemented default-off — engine-authored release verified live and forwarded-engine fault observation/relay behind `gsp_phys_fault_relay`, fail-closed to software on any miss; one relay data mismatch under triage | +| CUDA kernel execution (`vectorAdd`) | blocked on GR context validity/execution — the host GR ladder's first push faulted `CTXNOTVALID` and the promote attempt wedged GSP-RM | ## 7. Open questions @@ -297,6 +304,12 @@ per-context scheduling. Tenant policy and fairness are a later layer on top. controller action that does not go through the root port's D3 path — would produce the identical power-loss signature; the claim must abort on that signature and must not be retried in the same boot. +* **Wedged server / protected-region fail-safe:** if a live session wedges the + server so its shutdown path never completes, WPR2 cannot be cleared (the + host-side unlock is fuse-locked on this part). The release then refuses to + rebind and keeps the D0 locks; never retry in that boot, and return to the + distribution driver only through a reboot governed by the project's + reboot-approval policy (a written reason and post-reboot plan first). * **Supervised operation of the lock:** the lock is implemented and verified across three supervised claim/release cycles (2026-10-01). Every claim remains supervised: explicit human go-ahead, other host work warned, a @@ -338,7 +351,8 @@ per-context scheduling. Tenant policy and fairness are a later layer on top. the server's shutdown path (host RPC `UNLOADING_GUEST_DRIVER`), and host-side seed/unlock is fuse-locked on this part. The live session is therefore left claimed (`--keep`) and a fresh cycle needs a power cycle. - See §4.3. + See §4.3. *(Historical: superseded by the clean-release result in the + unload-RPC update below.)* * **Update (2026-10-01) — rung 4b implemented (build-only):** the module now produces real command-queue records to GSP-RM and attempts a clean shutdown on release. The producer writes the 48-byte transport element header plus @@ -355,8 +369,10 @@ per-context scheduling. Tenant policy and fairness are a later layer on top. behaviour is unchanged. A write-only debugfs trigger (`open-iov-vfio/gsp-phys-unload-rpc`, write "1") lets a live claimed session test the shutdown without touching WPR2. All of it is inert unless - the full boot stage ran; the code has not yet run live, and this bullet - must be updated with the live outcome. + the full boot stage ran. **Live-verified (2026-10-01):** the unload RPC + shuts the server down (GSP-RM processor suspended in ~220 ms, handoff bits + complete), so WPR2 goes down and the distribution driver is rebound with no + reboot on every clean cycle. * **Update (2026-10-01) — rung 4c slice 1 implemented (build-only):** for the next cycle a host resource-manager static-info probe is available: the write-only debugfs file `open-iov-vfio/gsp-phys-rm-probe` (write "1", only @@ -365,8 +381,20 @@ per-context scheduling. Tenant policy and fairness are a later layer on top. handles, split-VA flag, BAR1/2 PDE bases, framebuffer geometry, engine caps), falling back once to `SET_GUEST_SYSTEM_INFO` plus one retry; the command-queue producer now splits records across up to 16 queue elements - and the status-queue walk coalesces multi-element records. Not yet run - live; the module was built only. + and the status-queue walk coalesces multi-element records. **Live-verified:** + init-done with the full 1656-byte static-config reply was observed on the + physical server, and the probe decodes its static configuration (internal + handles, BAR PDE bases and engine caps); the multi-element producer and + status-walk coalescing also run live. +* **Update (2026-10-02) — GR execution is the wall; device in locked + fail-safe:** the resource-manager client ladder reaches the compute engine + (GR0 channel plus a compute-class object), but the first GR semaphore push + faulted with an uncommitted context (`CTXNOTVALID`), and the follow-up + `GPU_PROMOTE_CTX` attempt wedged GSP-RM (internal assert `0x52a8b04`, no + reply). Teardown could not clear WPR2, so the fail-safe held the device + (module unloaded, GPU unbound, protected region up, D0 locks held). Recovery + is a supervised reboot under the reboot-approval policy; the promote + bisection resumes after that. ## 9. Interaction sequence (end state) @@ -398,6 +426,7 @@ system map is in [`09-full-system-picture.md`](09-full-system-picture.md). ## See also - [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) - the host module, memory model, build contracts +- [`../eng-ops.md`](../eng-ops.md) - engineering operations; Part II carries the physical-backend parameter, debugfs and runbook inventory - [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) - known gaps and plans - [`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) - forwarding trade-offs - [`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md) - the CUDA-side view diff --git a/docs/03-architecture/09-full-system-picture.md b/docs/03-architecture/09-full-system-picture.md index e03a68e..3c15830 100644 --- a/docs/03-architecture/09-full-system-picture.md +++ b/docs/03-architecture/09-full-system-picture.md @@ -80,7 +80,7 @@ flowchart TB class VDEV,EMU done class MEMM,WORK wip - class BE plan + class BE done class IOMMU,VFIO done class GSP,ENG,GMMU,VRAM,DB hw class ND,USR,GKO,APP vendor @@ -125,8 +125,9 @@ stateDiagram-v2 Emulated today: the module drives the stage transitions and answers status reads from the BAR0 shadow. The real - backend boots the physical GSP in - parallel [planned]. + backend boots the physical GSP as its + own live session (init-done, static + configuration and clean releases proven). end note ``` @@ -262,14 +263,14 @@ flowchart LR F --> G["Write releases and completions<br/>into guest memory"] G --> H["Guest trackers advance"] - D -.->|"planned"| I["Forward to a real engine"] + D -.->|"wip/default-off"| I["Forward to a real engine"] classDef done fill:#d5f5d5,stroke:#1e7a1e classDef wip fill:#fff3cd,stroke:#9a7b1a classDef plan fill:#f8d7da,stroke:#9a1a1a class A,B,D done class E,F,G,H wip - class I plan + class I wip ``` This is a correctness bridge: it advances synchronization and memory @@ -329,22 +330,30 @@ flowchart TB R4 --> R5["Real submissions scheduled by GSP"] R5 --> R6["Completion and faults relayed to guest"] end - G -.->|"translation [planned]"| R4 + G -.->|"translation [wip/default-off]"| R4 classDef done fill:#d5f5d5,stroke:#1e7a1e + classDef wip fill:#fff3cd,stroke:#9a7b1a classDef plan fill:#f8d7da,stroke:#9a1a1a class A,B,C,D,E done class F,G wip - class R1,R2,R3,R4,R5,R6 plan + class R1,R2,R3 done + class R4,R5,R6 wip ``` +Backend boot, the real RM client, channel allocation and real CE copies are +live; guest-forwarding and relay are default-off; GR execution and real guest +page tables are the open `[planned]` parts (see +[`11-physical-backend-guest-wiring.md`](11-physical-backend-guest-wiring.md)). + ## 5. Scheduling and multi-tenancy There is no hardware partitioning (no SR-IOV) on this GPU generation, so tenancy is a **logical VF model**: each VM holds one mdev instance, and the module gives each instance its own GSP-RM session, channels, page tables and scheduling policy. “Per-VF GSP emulation” therefore means per-mdev emulation -context plus a per-mdev real GSP client in the backend. +context plus a per-mdev real GSP client in the backend. Today the backend is +a single per-GPU session (live); per-mdev clients are planned. ```mermaid flowchart TB @@ -358,7 +367,7 @@ flowchart TB Q2["Emulation context 2"] Q3["Emulation context N"] POL["Fairness policy [planned]<br/>time slices, quotas, priority, burst"] - C1["Real GSP client 1 [planned]"] + C1["Real GSP client 1<br/>single per-GPU session [live]"] C2["Real GSP client 2 [planned]"] C3["Real GSP client N [planned]"] end @@ -381,7 +390,8 @@ flowchart TB classDef wip fill:#fff3cd,stroke:#9a7b1a classDef plan fill:#f8d7da,stroke:#9a1a1a class Q1,Q2,Q3 done - class C1,C2,C3,POL,SCHED plan + class C1 done + class C2,C3,POL,SCHED plan ``` **How multi-tenant performance works (and is intended to):** @@ -636,10 +646,10 @@ against a stock guest driver with evidence in the repository. | Guest memory access + VA-to-PA discovery | partial | yes | protocol-derived discovery, no external pins | | Push decode + semaphore completion | partial | yes | complete GR/compute encodings | | Software execution bridge (memsets, copies, releases) | partial | yes | remains bridge only | -| Page-table population for server-owned ranges | no | no | design + validator, then implement | -| Execution backend (own GPU + real GSP) | no | no | device ownership, real GSP bring-up, RM client | -| Real work forwarding (GR and CE) | no | no | after backend bring-up | -| Fault semantics (buffers, replay, cancel) | no | no | after forwarding | +| Page-table population for server-owned ranges | partial (module-built scratch page tables, default off) | yes (default off) | real guest page tables | +| Execution backend (own GPU + real GSP) | yes (single per-GPU session) | yes | per-mdev clients | +| Real work forwarding (GR and CE) | partial (CE default-off) | yes (default off) | GR context validity/execution | +| Fault semantics (buffers, replay, cancel) | partial (forwarded-engine observation/relay default-off) | yes (default off) | GR fault semantics and triage | | Multi-tenant fairness policy | no | no | after multi-tenant forwarding | | Version matrix (2+ driver families) | partial | partial | per-version profiles + A/B | | Windows guest | no | no | after Linux execution path | diff --git a/docs/03-architecture/10-interaction-and-scheduling.md b/docs/03-architecture/10-interaction-and-scheduling.md index 3608dc5..ffc02d7 100644 --- a/docs/03-architecture/10-interaction-and-scheduling.md +++ b/docs/03-architecture/10-interaction-and-scheduling.md @@ -22,12 +22,12 @@ to, and where it stands. | Guest doorbell signal (boot, mailbox, channel) | write into BAR0 shadow, observed by the polling service | steering only | [done] | | Guest RPC request | element placed in a guest-sysmem message queue, read by the module over VFIO DMA | 10s of us per round trip | [done] | | RPC reply + interrupt | module writes reply into the queue, signals eventfd, QEMU raises irqfd into KVM, MSI-X lands in the guest | us | [done] | -| Push-buffer work | guest appends GPFIFO entries; the per-channel pump reads, decodes and applies them | per entry, us to low ms | [wip] | -| Semaphore release to guest memory | module writes the payload word into the resolved guest page | us | [done] | +| Push-buffer work | guest appends GPFIFO entries; the per-channel pump reads, decodes and applies them | per entry, us to low ms | [wip] (CE copies forward default-off; GR execution open) | +| Semaphore release to guest memory | module writes the payload word into the resolved guest page; a forwarded CE copy can have the engine author it | us | [done] (relay default off) | | VA-to-PA translation discovery | in-module structure walk, content scan, rescue, pins | us to seconds (scan bounded) | [wip] | | Translation internalization from protocol | derive mappings from RM/UVM objects as they are created | us | [planned] | -| Real channel submission | translated push stream on a real channel, real doorbell | us + hardware | [planned] | -| Physical GPU interrupt | real MSI-X into the module's IRQ handler, relayed to guest trackers | us | [planned] | +| Real channel submission | translated push stream on a real channel, real doorbell; the ladder's real submission is live | us + hardware | [wip] (guest forwarding default-off) | +| Physical GPU interrupt | real MSI-X into the module's IRQ handler, relayed to guest trackers; forwarded-engine fault observation/relay exists | us | [wip] (default off) | | mdev create / destroy | sysfs plus VFIO device registration | ms | [done] | | Guest RAM mapping | VFIO / IOMMUFD maps guest pages for DMA | ms once | [done] | | Teardown | ordered stop: VM, device close, mdev remove, module unload | s | [done] | @@ -129,7 +129,7 @@ stateDiagram-v2 note right of Applying State and notification operations are applied from the emulation. Real engine - work is forwarded [planned]. + work is forwarded [wip/default-off]. end note ``` @@ -198,7 +198,7 @@ Rules the design enforces: | Push entry decode | us per entry | thousands of entries per context | | Rescue of an unreadable entry | bounded ms | only on translation misses | | MSI-X to guest | us | completion latency floor | -| Real submission + completion relay | hardware + us | the forwarded-work path [planned] | +| Real submission + completion relay | hardware + us | the forwarded-work path [wip/default-off] | The rule of thumb the architecture is built around: **state operations are cheaper in the module than they would be on hardware; real work belongs on @@ -250,7 +250,7 @@ flowchart TB F --> G["KVM injects MSI-X vector"] G --> H["Guest ISR runs"] end - subgraph PHY["Module and physical GPU [planned]"] + subgraph PHY["Module and physical GPU [wip/default-off]"] I["Real channel doorbell"] --> J["GPU executes"] J --> K["Physical MSI-X"] K --> L["Module IRQ handler"] @@ -260,9 +260,9 @@ flowchart TB M -.-> E classDef done fill:#d5f5d5,stroke:#1e7a1e - classDef plan fill:#f8d7da,stroke:#9a1a1a + classDef wip fill:#fff3cd,stroke:#9a7b1a class A,B,C,D,E,F,G,H done - class I,J,K,L,M plan + class I,J,K,L,M wip ``` ## 9. Lifecycle and teardown ordering @@ -301,8 +301,8 @@ sequenceDiagram participant GD as Guest driver participant EMU as GSP-RM emulation participant VT as Version profile - participant BE as Execution backend [planned] - participant FW as Real GSP firmware [planned] + participant BE as Execution backend (single live session) + participant FW as Real GSP firmware (live session) GD->>EMU: version and capability queries EMU->>VT: select profile for this driver family @@ -327,7 +327,7 @@ sequenceDiagram | Pin tables | authoritative injected translations | debug tooling; later protocol discovery | lookup fast path | [wip] | | Pending release queue | releases awaiting a translation | decoder | service drain | [done] | | Completion counters | per-channel progress published to guest pages | service | guest trackers | [done] | -| Real channels and page tables | forwarding state | backend | real GPU | [planned] | +| Real channels and page tables | forwarding state (real channel live; guest page tables open) | backend | real GPU | [wip] | ## See also diff --git a/docs/03-architecture/11-physical-backend-guest-wiring.md b/docs/03-architecture/11-physical-backend-guest-wiring.md index cbb4b33..2229365 100644 --- a/docs/03-architecture/11-physical-backend-guest-wiring.md +++ b/docs/03-architecture/11-physical-backend-guest-wiring.md @@ -1,8 +1,11 @@ # Physical Backend Guest Wiring: Driving the Guest-Facing Emulation from the Live GSP-RM Session -> **Status: P0-P5(w1) implemented; P6a guest-page reachability and P6b waves +> **Status: P0-P5(w1) implemented; P6a guest-page reachability (trigger +> residual fixed) and P6b waves > 1/1.5/2 (guest-copy forwarding, deferred release, hardware-primary copies) -> proven live (2026-10-02).** The backend API skeleton +> proven live (2026-10-02); P6c C0/C1 (engine-authored guest CE semaphore +> relay) and C3 (forwarded-engine fault observation/relay) implemented +> default-off; H1 host GR ladder open.** The backend API skeleton > (P0), the mdev-owned lifecycle (P1, fail-open; the async FWSEC boot on that > path is an open defect off the critical path), the static-config identity > overlay (P2, verified live in an A/B run), the read-only control @@ -16,7 +19,14 @@ > releases exactly once) and the wave-2 hardware-primary copies (default off, > verified live: copy_skipped=119, queued req=100 ok=100 verify_ok=100 > verify_fail=0 fallback=0 timeouts=0, smoke `matched=64/64` `ALL OK`, with -> the software fallback for unqueued/failed copies) are in the tree. +> the software fallback for unqueued/failed copies) are in the tree. The P6c +> relay is live-verified default-off — the physical engine authored the +> guest CE copy's semaphore release (`relays=64`), failed/refused copies fall +> back to the software apply exactly once, and one relayed-copy data mismatch +> is under triage (mismatch logging plus software fallback in charge); the C3 +> forwarded-engine fault relay is live green, default off. The P6a trigger +> residual is fixed: triggers 6/7 re-translate at trigger time and use fresh +> scratch lines. > The physical backend itself is live: the module-owned > GSP-RM completes initialization on the real TU104, serves the full > static-config reply, and shuts down gracefully. The host resource-manager @@ -24,7 +34,12 @@ > a committed page directory, a channel group and a real copy-engine channel, > and has executed the **first real work on the physical engine** — a 4 KiB > copy in virtual-address mode with the semaphore released and destination -> equal to source (cycle 25). What this document specifies is the remaining +> equal to source (cycle 25). The H1 host GR ladder then allocated a GR0 +> channel with a compute-class object, but its first semaphore push faulted +> with an uncommitted context (`CTXNOTVALID`), and the follow-up +> `GPU_PROMOTE_CTX` attempt wedged GSP-RM; the GR execution wall is open and +> the device sits in the designed locked fail-safe. What this document +> specifies is the remaining > bridge: wiring that live session into the guest-facing mdev/VFIO path so that > guest submissions execute on the physical GPU. It extends > [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md), which @@ -102,6 +117,15 @@ (`:5068`). Stage 4 now runs end to end: the channel allocates, the page directory is committed, and a 4 KiB virtual-address-mode copy executes on the physical CE with the semaphore observed (`REAL COPY OK`, cycle 25). + `gsp_phys_rmcall_engine=1` selects a GR0 ladder variant (stage 3 builds the + channel and a `TURING_COMPUTE_A` compute object with `engineType=1`, stage 4 + pushes a GR semaphore release). Its first live push faulted with an + uncommitted context (`CTXNOTVALID`). `gsp_phys_rmcall_gr_promote=1..3` + selects opt-in `GPU_PROMOTE_CTX` shapes to commit that context; the first + attempt wedged GSP-RM (internal assert, no reply) and teardown could not + clear WPR2, so the fail-safe held the device. Debugfs trigger 9 re-runs the + GR semaphore push on an already-built GR ladder without touching WPR2, for + use once a promote shape is viable. * Release: `gsp_phys_gpu_release()` (`gsp_phys_gpu.c:876`) sends `UNLOADING_GUEST_DRIVER` once (`:909`), quiesces (`:910`), tears down WPR2 (`:911`), frees staging (`:914`), and only rebinds the distribution driver @@ -114,10 +138,12 @@ The two planes now meet at the lifecycle seam (P0/P1, mdev-owned claim/release), the static-config identity overlay (P2), the read-only control whitelist (P3) and, for eligible CE copies, the P6b scratch-mapped work path (waves 1/1.5/2: guest operands, deferred exactly-once release, -hardware-primary copies); objects, FB geometry, engine caps, real guest -page tables and completion/fault relay are still emulated. The remaining -bridge must be added at the reply point `gsp_rm_send()`, the inbound dispatch -point `gsp_rm_handle_element()`, and the work-plane seams (see §2). +hardware-primary copies) plus the P6c relay (C0/C1 engine-authored guest +release, C3 forwarded-engine fault observation/relay), all default off; +objects, FB geometry, engine caps and real guest page tables are still +emulated. The remaining bridge must be added at the reply point +`gsp_rm_send()`, the inbound dispatch point `gsp_rm_handle_element()`, and the +work-plane seams (see §2). --- @@ -141,7 +167,9 @@ point `gsp_rm_handle_element()`, and the work-plane seams (see §2). BACKEND_STOPPING stop emulation forwarding; UNLOADING_GUEST_DRIVER; | quiesce; WPR2 teardown; staging free v - BACKEND_OFF power unlock; distribution driver rebound and settled + BACKEND_OFF power unlock only when gsp_phys_repower=1 (default + 0 latches the D0/D3cold locks); distribution driver + rebound and settled ``` `BACKEND_READY` is the single gate for every live answer. In every other @@ -198,8 +226,9 @@ generation counter` so a release cannot deadlock against a forward in flight. parent/class teardown. * **Server wedge (the 0x40 GSP-owned incident):** if `rm_unload` times out, the existing release still quiesces, runs the FWSEC teardown and refuses to - rebind with WPR2 up (`gsp_phys_gpu.c:939`). A reboot then needs explicit - human approval plus a handoff update. + rebind with WPR2 up (`gsp_phys_gpu.c:939`). A reboot then follows the + project reboot-approval policy (a written reason and post-reboot plan before + approval). * **Static-config served before READY:** bounded wait (2 s, implemented) then pure emulation fallback, with a stat counter; never block the emulation thread longer than the guest's own RPC poll can tolerate. @@ -495,8 +524,10 @@ go-ahead, recovery plan, pstore, never unattended 514 real CE copies (194 requests coalesced/dropped while one was in flight, 0 refusals, 0 timeouts). The P6b waves 1/1.5/2 (guest-copy forwarding, deferred exactly-once release, hardware-primary copies with a software - fallback) have since landed (P6 status below); the real guest-page-table - path (P6c) remains. + fallback) have since landed (P6 status below), and P5 wave 2 (hardware- + primary release relay) is subsumed by the P6b wave-2 hardware-primary mode + and the P6c engine-authored relay; the real guest-page-table path and GR + execution remain. * **Scope:** the debugfs ladder's real-submission half is **done** (cycle 25: channel allocates, page directory commits, `REAL COPY OK` on the physical CE in VA mode). The remaining work: bind each guest CE/GR channel to a real @@ -587,20 +618,35 @@ go-ahead, recovery plan, pstore, never unattended with the guest-RAM ranges unregistered proved that fallback again (`copy_skipped=121`, recovered once each by the software copy, `copy_fail=0`, smoke still green), so the software fallback remains for any - unqueued or failed copy. **Residual (P6a trigger path):** - the debug-trigger path pre-builds its page-table slots once during ladder - stage 3; a guest page's host physical address can change between - translations (observed `0x13a591000` then `0x3b8318000` for the same GPA), - and a runtime PTE update into an already-fetched 128-byte line is invisible, - so the later write-direction trigger failed its data check. Forwarding is - unaffected (it resolves HPAs at submit time); the follow-up is to give the - debug triggers fresh scratch lines (the forwarding path) or to pin the - translated page. **Remaining:** P6c - completion/fault relay, real - page-directory shadowing, real guest channels and push forwarding with - guest page tables in the real MMU, and `vectorAdd` correct results; - guest-submission forwarding through the real guest page tables is not - implemented yet. All new parameters default off; with them off the software - path is unchanged. + unqueued or failed copy. **P6a trigger residual — fixed (2026-10-02):** + the debug triggers now re-translate the guest page's host physical address + at trigger time through the fresh-scratch-line path (triggers 6/7), retiring + the stale-PTE data-check failure; live `GUEST G2H COPY OK` and + `GUEST W2G COPY OK verify_ok=1`, with the PTE self-test and ladder + `REAL COPY OK` on a clean release. Forwarding was never affected (HPAs are + resolved at submit time). **P6c C0/C1 (engine-authored guest CE semaphore + relay, default off):** with `gsp_phys_relay=1` the guest-visible semaphore + page is mapped into the CE channel and the engine authors one selected + release; the deferred hold verifies the guest word and skips the software + apply of that record, and any refusal, failure, timeout or failed + verification leaves the unchanged software apply in charge, so the release + stays exactly-once. Live: the engine-authored release was observed in the + guest page (`relays=64`), the relay-on run stayed green + (`requested=68 ok=67 fallback=9`, smoke `ALL OK` 64/64), and a + forced-unresolvable control proved the fallback recovers every copy once; + one relayed copy released its semaphore but failed the host-side data + compare, so the parameter stays default-off pending triage, with the + mismatch logged and the software apply still in charge. **P6c C3 + (forwarded-engine fault observation/relay, default off):** with + `gsp_phys_fault_relay=1` a timed-out forwarded copy drains the GSP-RM + status queue for the engine's fault record, matches it to the ladder + channel and writes the guest channel notifier/event through the existing + MSI-X path (live green, default off). **Remaining:** real guest page tables + and channels, real page-directory shadowing, GR context validity/execution + and `vectorAdd` correct results; guest-submission forwarding through the + real guest page tables is not implemented yet, and the GR promote attempt + wedged the server (§0.2). All new parameters default off; with them off the + software path is unchanged. * **Scope:** map guest page-table pages/records so the physical GPU can DMA them; forward `SET_PAGE_DIRECTORY` (`gsp_rm_emulation.c:2975`) to the real server; allocate real memory objects for guest allocations (fn4 with real @@ -677,13 +723,18 @@ go-ahead, recovery plan, pstore, never unattended order). 8. **Debuggability and evidence.** Keep every real RPC logged at the TX/RX seams (`booter:3050`, `:3127`, status-walk logs), keep the read-only - `gsp-phys-state`/`gsp-phys-log-dump` files, and keep the evidence tree - (`backups/evidence-*`) current on every live cycle. + `gsp-phys-state`/`gsp-phys-log-dump` files, and attach the evidence + artifacts to the change on every live cycle. --- ## Appendix A - line anchor map (function -> file:line) +> **Historical anchors.** The line numbers below were taken from the +> working-tree revision used for the original design and are stale; they are +> kept as a historical map only. Anchor by function name when re-grepping and +> treat the numbers as untrusted. + `kernel/gsp_phys_booter.c` (working-tree revision used for this design): `gsp_phys_boot_prime` 1349, `gsp_phys_boot_run` 1524, `gsp_phys_status_queue_init` 2467, `_link` 2490, `_next` 2554, `gsp_phys_status_wait` 2670, diff --git a/docs/04-implementation/01-known-knowns-and-unknowns.md b/docs/04-implementation/01-known-knowns-and-unknowns.md index 1a5967b..0c8e50f 100644 --- a/docs/04-implementation/01-known-knowns-and-unknowns.md +++ b/docs/04-implementation/01-known-knowns-and-unknowns.md @@ -22,12 +22,14 @@ listed as an unknown-unknown has a *method* by which we will discover it. ### 1.1 Device and platform * A stock 570.144 Linux guest driver probes and initializes against our - emulated TU104 device; `nvidia-smi -L` works and CUDA user-space starts - (`cuInit`, `cuDeviceGetCount`, `cuDeviceGet`). Context creation is in - progress: the client reaches `cuCtxCreate` and currently ends in a forced - `rc=999` on a completion wait; the engine inventory is complete (the async - copy engine is included) and the remaining work is in the completion and - work-execution path (see §2.1). + emulated TU104 device; `nvidia-smi -L` works and the full CUDA driver-API + smoke passes (`ALL OK` 64/64, copies matched) — `cuInit`, device queries, + context creation, allocation, copies, free and destroy complete, and the + copies are verified byte-for-byte. The execution backend is live: CE-class copies are + forwarded to the physical copy engine (default off) with engine-authored + semaphore releases and a fault relay; the remaining frontier is engine + execution — GR kernel execution is blocked on GR engine-context validity + (see §2.1). * The same guest **boots unmodified**: no guest code, no guest module parameters are required for the initialization path. * The host is unmodified in the standalone configuration (distribution NVIDIA @@ -86,29 +88,30 @@ it**, **rough cost**. inventory including the async copy engine, and the copy-engine capabilities match the physical device (two GRCEs). The earlier zeroed secondary engine record that made `libcuda` rewrite the channel alloc class and roll back is -gone; context creation now progresses past engine selection and stalls later, -on completion of decoded work (see §1.1). +gone; context creation completes and the driver-API smoke passes (see §1.1). -**Unknown (resolved by decision, not yet implemented):** the correct, -isolated way to run a guest's engine work on the physical GPU. The interface -analysis in +**Partially delivered:** the correct, isolated way to run a guest's engine +work on the physical GPU. The interface analysis in [`../03-architecture/08-physical-gpu-integration.md`](../03-architecture/08-physical-gpu-integration.md) shows that proxying through the distribution host driver has no callable entry point from kernel context, so the **chosen mechanism** is for the module to own the physical GPU and run its real GSP firmware as the resource-manager server -(the module becomes the host-side RM client). The emulation path remains the -control plane. +(the module becomes the host-side RM client). That backend is now live: CE-class +copies and their releases are forwarded to the physical copy engine (default +off) with a fault relay, while the emulation path remains the control plane. +What is still open is **GR (kernel) execution**, blocked on GR engine-context +validity. -**Why:** this is *the* blocker for CUDA kernel execution, and later for +**Why:** GR execution is *the* blocker for CUDA kernels, and later for graphics. -**Plan:** build the chosen mechanism in stages: device ownership + real GSP -bring-up; host-side RM client for real objects and channels; translation of -guest work and real submission; completion and fault relay. Measure each stage -against a scorecard: correctness, isolation, latency, maintenance cost across -driver versions. Deliverable: a forwarded `vectorAdd` with verified results. -Alternatives (own scheduling without GSP; vendor vGPU or SR-IOV paths) are -analyzed and rejected or deployment-gated in the integration doc. +**Plan:** finish the backend in stages: GR engine-context validity and kernel +submission; translation of the remaining guest work; completion and fault relay +for every engine class. Measure each stage against a scorecard: correctness, +isolation, latency, maintenance cost across driver versions. Deliverable: a +forwarded `vectorAdd` with verified results. Alternatives (own scheduling +without GSP; vendor vGPU or SR-IOV paths) are analyzed and rejected or +deployment-gated in the integration doc. **Cost:** weeks, not days; the largest single work item in the project. @@ -146,16 +149,19 @@ remain the top risk; the division-of-labour question above is unchanged. ### 2.3 Fault semantics -**Unknown:** fault buffer formats and replay/cancel semantics as the *guest* -expects them in our virtualized setting (we know the buffer sizes and that -UVM programs them, but not yet the end-to-end contract under forwarding). +**Partly known:** the backend-side fault observation and relay now exists +(default off — a notifier write plus an event post; the auxiliary event post +can fail with `no_event`). **Still unknown:** the replay/cancel contract as the +*guest* expects it in our virtualized setting (we know the buffer sizes and +that UVM programs them, but not the end-to-end guest-side contract under +forwarding). **Why:** GPU page faults happen constantly in real CUDA workloads (UVM oversubscription, migrations); correct handling is required for robustness. **Plan:** reproduce faults deliberately (map a VA to nothing, touch it from a -forwarded kernel), capture both sides, implement replay; validate with -`uvm_fault`-heavy tests (e.g. oversubscribed memory). +forwarded kernel), capture both sides, implement the guest-side replay/cancel +contract; validate with `uvm_fault`-heavy tests (e.g. oversubscribed memory). **Cost:** weeks. @@ -320,9 +326,9 @@ To make a defensible statement, we need all of: 7. **Performance:** within an agreed factor of native for the supported cases, measured with a published methodology. -Until all seven hold, we say "initialization works; execution is in progress", -and we link here. The CUDA-side checklist behind these gates and the current -blocker are in +Until all seven hold, we say "initialization and the driver-API path work; +engine execution is in progress", and we link here. The CUDA-side checklist +behind these gates and the current execution frontier are in [`../../userspace/cuda/docs/03-driver-and-gpu-interface.md` §9](../../userspace/cuda/docs/03-driver-and-gpu-interface.md); start at [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md). diff --git a/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md b/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md index 3265f47..4b886b3 100644 --- a/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md +++ b/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md @@ -53,9 +53,11 @@ without a methodology is not accepted in this project. are answered from cached/derived state where the hardware has no side effects the guest can observe. * **Real work** (grid launches, copies) is meant to be forwarded to the - physical GPU (see §3). Today only the work the host decoder understands — - memsets, copies and semaphore releases — is executed in software as a - correctness bridge; no CUDA kernel is forwarded yet. + physical GPU (see §3). CE-class copies and their semaphore releases are now + forwarded to the physical copy engine (default off) with a host-side fault + relay, and memsets/copies/releases the host decoder understands still run in + software as a fail-closed correctness bridge. No CUDA kernel is forwarded + yet: GR kernel execution is blocked on GR engine-context validity. ## 3. Forwarding: the options and the chosen mechanism diff --git a/docs/05-operations/01-performance-correctness-security.md b/docs/05-operations/01-performance-correctness-security.md index 3a2c6b4..21afd06 100644 --- a/docs/05-operations/01-performance-correctness-security.md +++ b/docs/05-operations/01-performance-correctness-security.md @@ -17,12 +17,14 @@ Rung 4 SUSTAINED workloads run for hours; faults/errors recoverable Rung 5 MULTI-TENANT N VMs, isolation + fairness + no cross-talk ``` -**Current rung (2026-09-29):** Rung 1 is green; **Rung 2 is in progress.** -`cuInit`/`cuDeviceGet` succeed, and the client reaches `cuCtxCreate`, which -currently ends in a forced `rc=999` on a completion wait (engine inventory -complete; completion translation in progress); `cuMemAlloc` is not reached yet -and failed-context teardown has oopsed in the guest kernel in earlier runs. -Before Rung 3 is claimed, the +**Current rung (2026-10-02):** Rungs 1–2 are green: the stock driver probes, +`nvidia-smi` enumerates, and the full CUDA driver-API smoke passes +(`ALL OK` 64/64, copies matched) — `cuInit`, device queries, context creation, +allocation, copies, free and destroy complete, and the copies are verified +byte-for-byte. **Rung 3 is partial:** CE-class copies execute on the physical +copy engine (default off) with engine-authored releases and a fault relay, but +CUDA kernel execution is blocked on GR engine-context validity. Before Rung 3 +is claimed, the **reply-fidelity gate** applies: a control whose reply fields the driver consumes is not implemented until the reply is coherent (status-only → coherent zeroed params → coherent values for consumed fields → exact values) — @@ -64,8 +66,9 @@ New to CUDA analysis here: start at `./scripts/stack_manager.sh test` runs the host suite (`make tests` only builds the test binaries). CI (`.github/workflows/ci.yml`) runs on push and pull requests only — there is currently no scheduled guest suite and no -stock-driver contract test. A rung regression is a release blocker; the -handoff documents carry the current rung. +stock-driver contract test. A rung regression is a release blocker; +[`STATE-CURRENT.md`](../../STATE-CURRENT.md) and this docs tree carry the +current rung. ## 3. Performance @@ -93,6 +96,10 @@ handoff documents carry the current rung. * **Publish the harness**: the benchmark scripts live in `tools/`, and a result is not comparable unless produced by them (or an equivalent, documented harness). +* **Runbooks over methodology**: the live physical-backend runbooks, scenario + scripts, debugfs surfaces and module parameters are collected in + [`eng-ops.md` Part II](../eng-ops.md); repeatable procedures live there, and + this document stays the measurement contract. ### 3.3 Targets diff --git a/docs/05-operations/02-debugging-and-maintenance-workflows.md b/docs/05-operations/02-debugging-and-maintenance-workflows.md index 8d711c6..cfc452d 100644 --- a/docs/05-operations/02-debugging-and-maintenance-workflows.md +++ b/docs/05-operations/02-debugging-and-maintenance-workflows.md @@ -322,6 +322,34 @@ version × each hardware family, run the step-5 bring-up sequence. When adding a version, extend the matrix rather than replacing the old column — regressions across versions are the cheapest bugs to catch. +### 4.1.1 Green-eye-assisted upgrade/porting workflow (experimental) + +<!-- change (2026-10-02): add the green-eye probe -> symbols -> version diff -> +portability matrix -> decide pipeline to the upgrade runbook, with pointers to +the new RUNBOOK and green-eye user guide; additive only. --> + +For firmware-side questions the experimental green-eye tooling (see +[`03-risc-v-gsp-rm-vm.md`](03-risc-v-gsp-rm-vm.md)) turns steps 3–5 above into +a repeatable pipeline that produces attachable evidence before any emulation +change. It never replaces the physical backend when silicon truth is required, +and it never touches a guest, a kernel module or the installed driver. + +| Rung | Command (from `debug/risc-v-gsp-rm-vm/`) | Output | +|---|---|---| +| Probe | `./greeneye.sh probe <new.fwimage> --symbols symbols-<ver>.json --record` | first-divergence report; first-touch map | +| Symbols | `./greeneye.sh symbols export --index <export.tsv> --bias 0 -o symbols-<ver>.json` | runtime-address symbol DB with provenance | +| Version diff | `python3 harness/version_diff.py --old <old> --new <new> --out dumps/vdiff-<ver>` | shape/symbol/corpus/probe/behavior diff plus the M1–M6 skeleton | +| Portability matrix | `python3 harness/portability_matrix.py --target <old> --target <new> --out dumps/matrix-<ver>` | per-target sessions and per-pair `compare` reports | +| Decide | `./greeneye.sh compare dumps/<old-run> dumps/<new-run> --symbols` | symbol-attributed deltas; a product change only after physical confirmation | + +Decision discipline: verify the pairing first; behaviour that moved without a +code change is usually a state/ordering fix; changed code on a live path is +prioritised; a symbol never hit in either run is a coverage gap, not dead code. +The full incident procedures (symptoms, evidence, response order, rollback) are +in the [Open-IOV Defensive Operations Runbook](../RUNBOOK.md) S1 and its OODA +ordering; the quickstart and tool chooser are in the +[green-eye user guide](04-green-eye-user-guide.md). + ### 4.2 New guest driver version 1. Install the stock driver in a guest (no patches). diff --git a/docs/05-operations/04-green-eye-user-guide.md b/docs/05-operations/04-green-eye-user-guide.md new file mode 100644 index 0000000..a9bf1dc --- /dev/null +++ b/docs/05-operations/04-green-eye-user-guide.md @@ -0,0 +1,439 @@ +# green-eye User Guide (experimental) + +> **# TOOL IN DEVELOPMENT - NOT READY YET!** +> green-eye is the development environment for running GSP/uproc firmware +> off-hardware, observing it, injecting into it and comparing runs as evidence. +> It is never shipped, never a driver dependency and never present in a guest; +> interfaces and file formats may change without notice. For incident response +> procedures see the [Open-IOV Defensive Operations Runbook](../RUNBOOK.md). + +This guide is the single front door for a new user: what green-eye is, how to +get a first recorded run in ten minutes, which tool to use for which task, how +to treat sessions as evidence, and what to do when something is wrong. The +authoritative details live in the tool's own documentation set +([documentation map](#10-documentation-map)). + +## 1. What green-eye is + +green-eye runs GSP/uproc firmware in a controllable machine, instruments it +with Frida-like primitives, records a durable session, changes the machine's +inputs (including injected code), and compares runs so that emulation changes +in `open-iov-vfio` land with evidence instead of anecdotes. + +| It is | It is not | +|---|---| +| A firmware runtime microscope (execution + memory: L0/L1) | A timing, interrupt or device model (L2/L3 absent) | +| A scriptable `Session` API with two backends (QEMU-RSP and native) | A GDB wrapper; there is no GDB dependency | +| A recorded-evidence system with a versioned session format | A product component, a guest agent or a shipped dependency | +| A static/dynamic join for Ghidra | A decompiler or a CUDA-corpus replacement | +| An automation layer (harnesses, MCP, CI) | A fuzzing service against hardware; simulator/re-hosted code only | + +Ghidra answers *what is the code*; green-eye answers *what did it do*. Guest +diagnostics answer *what did the driver request*; a supervised physical cycle +answers *what does silicon do*. green-eye makes the middle question repeatable +and safe. + +## 2. Ten-minute quickstart + +Requirements: Linux, Python 3.10+ (stdlib only), `qemu-system-riscv64` +(Ubuntu `qemu-system-misc`), `clang` and `lld`. No GPU, no driver and no +kernel module are touched. + +```bash +cd open-iov/debug/risc-v-gsp-rm-vm + +./greeneye.sh info # environment check: QEMU, toolchain, paths +make demo && make sdk # build the bundled RV64 uproc demo and injection SDK + +./greeneye.sh demo # canned run: boot, hooks, watchpoints, session +./greeneye.sh analyze dumps/<printed-session-dir> + +./greeneye.sh probe # real GSP image: report the first divergence +./greeneye.sh inject # load and invoke a position-independent service +``` + +What "good" looks like: + +1. `info` reports QEMU and the demo paths; +2. `demo` prints hook/branch lines and a session directory under `dumps/`; +3. `analyze` renders hook/watch/breakpoint events and a coherent summary; +4. `probe` reports a bounded first divergence (for the TU104 570.144 image, a + load-access fault at firmware vaddr `0x4009014` reading + `0x2000000000000A00`) — that is expected data, not a boot; +5. `inject` returns matching values on the direct and firmware-dispatched + paths. + +Next steps: [choose a tool](#5-choosing-the-right-tool) for your task and read +[Common recipes](#7-common-recipes). + +## 3. Mental model + +### 3.1 Two backends, one session API + +```text +CLI / REPL / Python ─┐ +MCP (stdio) ────────┼─► GreenEyeAPI ─► Session ─┬─► QEMU-RSP backend (qemu-system-riscv64) +Ghidra bridge ──────┤ hooks, watchpoints, └─► native backend (sim/machine.py) +harness/ + rmharness┘ memory, symbols, recorder ──► session directory +``` + +* **QEMU-RSP** is the default: full RV64 execution, gdbstub transport without + GDB, TCG trace plugin optional. +* **native** (`--backend native`) is the in-process interpreter: deterministic + virtual time (instruction counts), fast startup, an exec/access trace, and + lenient MMIO for first-touch exploration. +* Both satisfy the same `Session` primitives (registers/memory, breakpoints, + watchpoints, step/continue, snapshots), so scripts, probes, plugins, + recordings and comparisons survive a backend change. + +### 3.2 The primitives + +| Primitive | Use it for | +|---|---| +| `breakpoint` / `hook` | did this function run? with what arguments and return value? | +| `watch` (`w`/`r`/`a`) | when did this field change, from which store PC? | +| `run` / `run_until` | bounded observation; stop at a chosen address | +| `call` | Frida-like remote call to unit-test a function in isolation | +| `step` / `note` | single-step with dispatch; annotate the event stream | +| `Memory.read/write/hexdump/search/snapshot/diff` | inspect and diff memory regions | +| `SymbolDB` | resolve file/program/runtime addresses across sources | +| `Recorder` | write the durable session (events, snapshots, blobs) | +| injection SDK | load, register and invoke your own PIC RV64 service | + +Instructions between stops are invisible unless a trace is on: the TCG plugin +(`-plugin qemu/greeneye-trace.so`) on QEMU, or `--trace` on the native backend. + +### 3.3 Fidelity levels (what a result can mean) + +| Level | Property | green-eye today | +|---|---|---| +| **L0** | instruction semantics | high (QEMU TCG, same ISA family) | +| **L1** | memory behaviour | high for RAM-resident code, conditional on mapping | +| **L2** | time/concurrency | **not modelled** (do not claim) | +| **L3** | device environment (falcon, WPR2, CSB, PMU, timers, DMA, IRQ) | **absent** (device model is roadmap) | + +A simulator observation is evidence about the firmware, not about silicon; +anything that would change the product is reproduced on a physical/vfio-mdev +reference first. Never quote a green-eye session for timing, interrupt or +security claims. + +### 3.4 A session is the unit of work + +Every run records a `greeneye-session/1` directory: + +```text +dumps/<stamp>-<name>/ + session.json format, name, counts, snapshot inventory, meta + events.jsonl append-only event stream (hooks, watches, memory, notes) + snapshots/*.bin.z zlib-compressed region captures with sha256 + blobs/* QEMU logs, trace.jsonl, host-oracle stdout + summary.md rendered analysis +``` + +Partial sessions stay readable; the format evolves additively. A copied +`events.jsonl` without its `session.json`, snapshots and metadata is not +evidence. + +## 4. Tool inventory + +Run everything from `open-iov/debug/risc-v-gsp-rm-vm/` unless stated. + +### 4.1 CLI (`./greeneye.sh`) + +| Command | One-line purpose | +|---|---| +| `info` | environment/tooling check (QEMU, paths, capabilities) | +| `demo` | canned boot + hooks + watchpoints + recording on the bundled uproc | +| `run <script.py>` | run a Python instrumentation session against the demo/ELF | +| `attach` / `launch` | interactive REPL; keep a target alive and print its gdbstub port | +| `record` | boot and record with watches/snapshots (and `--trace` on native) | +| `probe <image>` | bounded run of a raw image; first-divergence report | +| `analyze <session>` | session to symbol-annotated Markdown report | +| `compare <a> <b>` | symbolic A/B delta of two recorded sessions | +| `symbols export\|resolve` | build/inspect a runtime-address symbol DB | +| `inject` | load, register and invoke an SDK service (direct + mailbox) | +| `ghidra` | serve the live session to GhidraScripts over localhost | +| `plugins` | list discoverable plugins | + +### 4.2 Harnesses (`harness/`) + +| Tool | One-line purpose | +|---|---| +| `portability_matrix.py` | probe + fixed scenario + compare across targets, with an M1–M6 skeleton | +| `mutation_harness.py` | seeded one-input mutations with outcome classes and per-attempt sessions | +| `control_discovery.py` | sweep a call surface to find known/unknown/injected handler IDs | +| `reach.py` | is one symbol reachable, what triggers it, with what arguments and effects | +| `version_diff.py` | old vs new image: shape, symbols, corpus hints, probe, behavior | +| `ci_release.py` | version_diff → matrix → mutation smoke, hashed into one bundle | +| `port_driver_versions.py`, `port_new_device.py`, `package_sim.py` | porting/packaging scaffolds (dry-run default; pilot status and evaluation in [docs/28](../../debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md), [docs/29](../../debug/risc-v-gsp-rm-vm/docs/29-PACKAGING-AND-TRIALING.md)) | +| `value_demos/run_all.py` | ten fixed scenarios s01–s10 with measured time-to-value | +| `common.py` | shared helpers (flat relocation, sessions, bounded log reads) | + +### 4.3 Native simulator, plugins, host oracle, bridges + +| Tool | One-line purpose | +|---|---| +| `sim/` (`--backend native`, `--trace`) | deterministic RV64IMAC interpreter with exec/access trace and lenient MMIO | +| `python3 -m greeneye.device_map harvest\|show\|merge\|demo` | first-touch MMIO map: authoring, inspection, merge | +| `qemu/greeneye_trace.c` (`make -C qemu`) | full TCG exec/access JSONL trace with `start=`/`end=` filters | +| `plugins/example_plugin.py` | plugin authoring reference | +| `plugins/mcp_server.py` | MCP server for agents (stdio or loopback; read-only by default) | +| `plugins/rmharness_plugin.py` | REPL commands for the host RM oracle | +| `rmharness/harness.sh` + `rmtool`/`rmtop`/`rmq` | read-only host oracle captures appended into a session | +| `ghidra/ExportForGreenEye.java`, `MarkGreenEyeHits.java`, `GreenEyeControl.java` | symbol export, hit annotation and decompiler-driven execution | +| `sdk/` + `greeneye inject` | position-independent service ABI v1 | + +## 5. Choosing the right tool + +| Task | Tool | Notes | +|---|---|---| +| "Does this run?" / first divergence of a raw image | `./greeneye.sh probe` | no device model: expected early fault is data | +| "What does this firmware do on this stimulus?" | `record` / `run` + `analyze` | fixed stimulus, bounded budgets, hooks/watchpoints | +| "Which command IDs exist / what is undiscovered?" | `harness/control_discovery.py` | fresh session per probe; deterministic seed | +| "Is this function reachable, and how?" | `harness/reach.py` | `auto|stimulus|trigger|run|call`; emits a replayable probe spec | +| "What changed between two firmware versions?" | `harness/version_diff.py`, `./greeneye.sh compare` | static inventory + dynamic profile | +| "Does the new firmware move behaviour or code?" | `harness/version_diff.py` + docs/14 decision table | code vs behaviour classification | +| "Prove an error path / fail-closed rule" | `harness/mutation_harness.py` | simulator only; seeded and replayable | +| "Prototype an RM service before product code" | `sdk/` + `./greeneye.sh inject` | PIC blob; direct and mailbox paths | +| "Join Ghidra static addresses to runtime hits" | `./greeneye.sh ghidra` + Java bridges | localhost only; no auth | +| "What does a control reply look like on real RM?" | `rmharness/harness.sh … rmtool` | read-only host oracle; safe beside a CUDA job | +| "Explore MMIO the firmware touches first" | `probe` → `python3 -m greeneye.device_map` | versioned JSON; feeds the device model | +| "Triage a release end to end" | `harness/ci_release.py` | always `--dry-run` first | +| "Let an agent monitor a run" | `plugins/mcp_server.py` | read/observe by default; writes gated | + +Decision rule: static reading is a hypothesis; a recorded session is evidence; +at least an A/B pair is a conclusion. + +## 6. Sessions as evidence + +1. **Record with a fixed stimulus.** The same script, seed, budgets and image + hash make runs comparable; a self-comparison must be empty. +2. **Annotate as you go.** `s.note(...)` inside the session explains what was + asked and what was mutated; a chat log is not evidence. +3. **Attach the provenance.** Firmware SHA-256, version string, pairing notes, + symbol DB source/hash/bias, and the exact command or script. +4. **Keep the directory together.** `session.json` + `events.jsonl` + + `snapshots/` + blobs are one artifact; publish a SHA-256 list when moving + it. +5. **State the fidelity.** L0/L1 exercised; L2/L3 not claimed; reference + capture named (hardware, vfio-mdev, host oracle or none). +6. **Report deltas honestly.** `compare` marks non-semantic fields (time, host + paths, backend identity) as such; every remaining delta is explained or + listed as open. An empty unexplained-delta list is a claim that must be + earned. +7. **Bundle for triage.** A release bundle is + `bundle-manifest.json` + `release-report.json` + version-diff + portability + matrix + mutation smoke; see + [docs/21](../../debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md) §4.3. + +The M1–M6 metric contract (inventory, reply, ordering, state, error-path and +memory-side-effect agreement) is defined in +[docs/06-ACCURACY](../../debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md) §5 and is +authoritative for every claim. + +## 7. Common recipes + +### 7.1 First record and analyze + +```bash +cd debug/risc-v-gsp-rm-vm +make demo +./greeneye.sh demo # prints the session directory +./greeneye.sh analyze dumps/<session> # symbol-annotated report +``` + +### 7.2 Probe a new firmware image + +```bash +./greeneye.sh symbols export --index <export.tsv> --bias 0 -o symbols-<ver>.json +./greeneye.sh probe <gsp-<VER>.fwimage.bin> --symbols symbols-<ver>.json --record +python3 harness/version_diff.py --old <old> --new <new> --out dumps/vdiff-<ver> \ + --old-symbols symbols-<old>.json --new-symbols symbols-<new>.json +``` + +### 7.3 Discover controls and prove reachability + +```bash +python3 harness/control_discovery.py --out dumps/discovery --max-id 32 \ + --patterns zero --seed 5 +python3 harness/reach.py --target rpc_ping --method call --args 0x41 \ + --out dumps/reach-ping +``` + +### 7.4 Mutate an input surface (simulator only) + +```bash +python3 harness/mutation_harness.py --out dumps/mutation-7 \ + --iterations 64 --seed 7 --timeout 0.25 --settle 0.2 +``` + +### 7.5 Portability pipeline across targets + +```bash +python3 harness/portability_matrix.py \ + --target <reference-fwimage> --target <candidate-fwimage> \ + --out dumps/matrix-candidate --probe-instructions 200 --scenario-duration 0.2 +``` + +### 7.6 Inject and debug a service + +```bash +make -C sdk +make -C sdk relocs +./greeneye.sh inject --service sdk/build/injected_demo.elf --cmd 1 --arg0 0x41 --arg1 0x42 +``` + +The direct call is the unit test; the mailbox call is the integration test. +Both are recorded. + +### 7.7 Drive execution from Ghidra + +```bash +# terminal 1 +./greeneye.sh ghidra --kernel <elf> --bridge-port 7000 +# terminal 2 (Ghidra headless) +$GHIDRA/support/analyzeHeadless <proj> <name> -process <program> -noanalysis \ + -scriptPath open-iov/debug/risc-v-gsp-rm-vm/ghidra \ + -postScript GreenEyeControl.java 7000 0x<address> +``` + +### 7.8 Host oracle capture (read-only) + +```bash +./rmharness/harness.sh --session dumps/real-<anchor> --name before \ + -- rmtool engine list --format=json +./rmharness/harness.sh --session dumps/real-<anchor> -- rmtop --once +``` + +### 7.9 Release rehearsal + +```bash +python3 harness/ci_release.py --old <old> --new <new> --out dumps/release-<ver> \ + --old-label <OLD> --new-label <NEW> --dry-run +``` + +`--dry-run` prints the plan and creates nothing; the RM 610 gate refuses 610 +labels (exit 3) until a human approves in writing. + +### 7.10 Let an agent monitor + +```bash +python3 plugins/mcp_server.py --stdio --backend native --record +# read/observe by default; --allow-write and --allow-inject are explicit opt-ins +``` + +### 7.11 First-touch device map + +```bash +./greeneye.sh probe <fwimage> --record +python3 -m greeneye.device_map harvest dumps/<probe-session> --out dumps/<ver>-device-map.json +python3 -m greeneye.device_map show dumps/<ver>-device-map.json +``` + +## 8. Troubleshooting + +| Symptom | Likely cause | Fix | +|---|---|---| +| `qemu-system-riscv64 not found` | QEMU absent or not on `PATH` | install `qemu-system-misc` or point `GREENEYE_QEMU` at the binary | +| QEMU starts but the script cannot attach | wrong port / target exited | use the port `launch`/`demo` printed; re-run `launch` and `attach --port <port>` | +| `Address already in use` on a bridge/trace port | stale bridge or another session | choose another `--bridge-port`; check `pgrep -fa greeneye` and `pgrep -fa qemu-system-riscv64` | +| QEMU survives Ctrl-C of a script | `launch` keeps the target alive by design | close the session or stop the reported PID; do not kill an unrelated QEMU | +| Session analyzer shows no hits/events | recording off, or the stimulus never ran | use the CLI (records by default); check hooks/watchpoints with a known symbol (`rpc_sum`) | +| `NameError: __file__` in `run` scripts | script invoked outside `greeneye.sh run` | run through `./greeneye.sh run`, keep scripts inside the tool tree | +| Empty TCG trace blob | plugin window missed or target never resumed | `make -C qemu`; widen `start=`/`end=`, set `exec=on`, resume the target | +| MCP writes refused / no tools | server read-only by default; stdout polluted | pass `--allow-write`/`--allow-inject` deliberately; keep stdout protocol-only | +| `greeneye error:` on a valid path | wrong relative path or `--db` file | `./greeneye.sh info` shows resolved paths; bad symbol DBs fail with a clear error | +| Harness findings not reproducible | seed/budget changed | pin `--seed` and budgets; compare `findings.jsonl` byte-for-byte | +| Everything "unreachable" | wrong surface/timeouts | run `--surface demo` on the built demo first; raise `--settle`/`--timeout` | +| `Unknown symbol` in the optional OpenRM build | module-tag drift | re-pin build glue; see the [RUNBOOK](../RUNBOOK.md) S2 | + +Marker status: every tool document carries the +**# TOOL IN DEVELOPMENT - NOT READY YET!** banner. The marker is removed only +when the acceptance list in +[docs/00-OVERVIEW](../../debug/risc-v-gsp-rm-vm/docs/00-OVERVIEW.md) passes on a +clean checkout, the usage recipes verify end-to-end, a second person can run +the quickstart unaided, and the session format and CLI are declared stable. + +## 9. Safety + +* **Simulator only.** The tools launch their own QEMU or native machine + against the bundled demo or an explicitly allowed re-hosted ELF. They never + attach to hardware, a guest, a kernel module, an mdev or an existing + gdbstub. +* **No guest, no hardware, no product.** Nothing runs in a guest; injected + services are prototypes, not shipped code; product changes go through + `open-iov-vfio` with physical confirmation. +* **Read by default; writes gated.** MCP `write_mem`/`call_function` need + `--allow-write`, injection needs `--allow-inject`, and every write is + recorded in the session. +* **Bounded everywhere.** Runs have durations/stop caps; reads and snapshots + have size caps; harness subprocesses have timeouts; a wedged target is left + stopped and restarted clean rather than poked. +* **Local only.** Ghidra and MCP transports are localhost/stdio; the bridge is + unauthenticated by design, so never expose it on a network interface. +* **No confidentiality claims.** The simulator is a study environment, never a + security or confidentiality boundary; never use injection to defeat a + hardware security boundary. +* **Fuzzing is gated.** Mutation runs only against the simulator or re-hosted + code, behind explicit harnesses, with recorded evidence. +* **RM 610 is a hard human gate.** No acquisition, analysis, emulation edit, + physical work or support claim before written approval; the gate is enforced + by the CLIs. + +## 10. Documentation map + +Tool documentation (`debug/risc-v-gsp-rm-vm/docs/`): [00 overview](../../debug/risc-v-gsp-rm-vm/docs/00-OVERVIEW.md), +[01 design](../../debug/risc-v-gsp-rm-vm/docs/01-DESIGN.md), +[02 usage](../../debug/risc-v-gsp-rm-vm/docs/02-USAGE.md), +[03 scripting](../../debug/risc-v-gsp-rm-vm/docs/03-SCRIPTING.md), +[04 plugins](../../debug/risc-v-gsp-rm-vm/docs/04-PLUGINS.md), +[05 Ghidra bridge](../../debug/risc-v-gsp-rm-vm/docs/05-GHIDRA-BRIDGE.md), +[06 accuracy](../../debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md), +[07 portability playbooks](../../debug/risc-v-gsp-rm-vm/docs/07-PORTABILITY-PLAYBOOKS.md), +[08 roadmap](../../debug/risc-v-gsp-rm-vm/docs/08-ROADMAP.md), +[09 firmware notes](../../debug/risc-v-gsp-rm-vm/docs/09-GSP-FIRMWARE-NOTES.md), +[10 related work](../../debug/risc-v-gsp-rm-vm/docs/10-RELATED-WORK.md), +[11 RM service guide](../../debug/risc-v-gsp-rm-vm/docs/11-GSP-RM-SERVICE-DEVELOPER-GUIDE.md), +[12 injection SDK](../../debug/risc-v-gsp-rm-vm/docs/12-INJECTION-SDK.md), +[13 Ghidra control](../../debug/risc-v-gsp-rm-vm/docs/13-GHIDRA-CONTROL.md), +[14 adversarial interop](../../debug/risc-v-gsp-rm-vm/docs/14-ADVERSARIAL-INTEROP-GUIDE.md), +[15 agentic GSP IDE](../../debug/risc-v-gsp-rm-vm/docs/15-AGENTIC-GSP-IDE.md), +[16 custom simulator](../../debug/risc-v-gsp-rm-vm/docs/16-CUSTOM-SIMULATOR.md), +[17 RM harness](../../debug/risc-v-gsp-rm-vm/docs/17-RM-HARNESS.md), +[18 harness automation](../../debug/risc-v-gsp-rm-vm/docs/18-HARNESS-AUTOMATION.md), +[19 control discovery](../../debug/risc-v-gsp-rm-vm/docs/19-CONTROL-DISCOVERY.md), +[20 MCP server](../../debug/risc-v-gsp-rm-vm/docs/20-MCP-SERVER.md), +[21 CI/release pipeline](../../debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md), +[22 porting CUDA and devices](../../debug/risc-v-gsp-rm-vm/docs/22-PORTING-CUDA-AND-DEVICES.md), +[23 RM 610 readiness](../../debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md), +[24 device map](../../debug/risc-v-gsp-rm-vm/docs/24-DEVICE-MAP.md), +[25 value validation](../../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md), +[26 architecture](../../debug/risc-v-gsp-rm-vm/docs/26-ARCHITECTURE.md), +[27 tool audit](../../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md), +[28 porting automation](../../debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md), +[29 packaging and trialing](../../debug/risc-v-gsp-rm-vm/docs/29-PACKAGING-AND-TRIALING.md), and the +[long-term roadmap](../../debug/risc-v-gsp-rm-vm/GREEN-EYE-ROADMAP.md). + +Project documentation: + +* [Open-IOV Defensive Operations Runbook](../RUNBOOK.md) — incident scenarios + S1–S10, evidence bundles, escalation tree; +* [GSP in a RISC-V VM](03-risc-v-gsp-rm-vm.md) — project-level purpose, status + and limits; +* [Debugging and maintenance workflows](02-debugging-and-maintenance-workflows.md) + — captures, failure taxonomy and the upgrade runbook; +* [Hardware and software matrix](../06-portability/hardware-software-matrix.md) + — support ladder and the portability pipeline; +* [Adversarial interoperability](../04-implementation/02-practical-challenges-and-adversarial-interop.md) + — the no-patch rule; +* [Engineering operations](../eng-ops.md) — tool admission, workflows and + retirement (the cross-cutting engineering-operations document); +* CUDA playbook: + [`userspace/cuda/docs/07-version-upgrade-playbook.md`](../../userspace/cuda/docs/07-version-upgrade-playbook.md) + and [`userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md); +* [Documentation front door](../README.md) — the full reading order. + +<!-- change (2026-10-02): correct the TCG plugin filename to the actual +qemu/greeneye_trace.c, list the docs 28/29 scaffolds in the tool inventory and +documentation map; scope: §4.2, §4.3, §10 only; benefit: copied commands and +doc-map navigation match the tool tree. --> diff --git a/docs/06-portability/03-device-validation-and-porting-sequence.md b/docs/06-portability/03-device-validation-and-porting-sequence.md new file mode 100644 index 0000000..3c5d1aa --- /dev/null +++ b/docs/06-portability/03-device-validation-and-porting-sequence.md @@ -0,0 +1,232 @@ +# Device Validation and Porting Sequence + +> How Open-IOV takes a new GPU device from "available to test" to a documented +> support decision: which devices can be rented or borrowed, the physical +> preparation, the supervised test-and-port sequence from driver **540** to the +> current anchor, the evidence collected per device, and the go/no-go rules. +> +> Scope note: this document describes the **validation process**; the support +> ladder and the current status of each combination live in +> [`hardware-software-matrix.md`](hardware-software-matrix.md). Nothing here +> moves a device up the ladder by itself. The companion automation is +> [`../../debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md`](../../debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md). + +## 1. Rules that never bend + +1. **No unattended claim cycles.** Every cycle that will be used to claim + support runs with a human supervising the device; the simulator harness may + run unattended, the physical device may not. +2. **Reboots need approval.** Host or device-host reboots, driver + bind/unbind, VM restarts and mdev claim/release for a claim cycle are + approved by the device owner before the cycle starts. One approved cycle at + a time. +3. **The guest never contains Open-IOV code.** No guest agent, no patched + driver, no extra kernel parameters beyond what the stock driver needs. +4. **A claim is evidence, not an intention.** "Tested and documented" only: + a device is Supported after this sequence's evidence exists and the matrix + records it with date, driver family, firmware hash and tested rung. +5. **Simulator evidence is evidence about the firmware, never about silicon.** + Physical capture or explicit "not physically validated" is stated in every + report (see [`../05-operations/01-performance-correctness-security.md`](../05-operations/01-performance-correctness-security.md) + and the accuracy contract in the tool's docs/06). + +The operational steps for a supervised cycle (preflight, capture, recovery) are +owned by the [Open-IOV Defensive Operations Runbook](../RUNBOOK.md) (§P7 and +the physical-cycle response path). This document is the porting sequence that +wraps that cycle. + +## 2. Device classes we can rent or borrow + +Support cost is dominated by *access to hardware*, so the sequence starts by +naming where each class can come from. Eligibility is about PCIe passthrough +and VFIO/mdev, not the marketing tier: + +| Class | Examples | Access path | Known constraints | +|---|---|---|---| +| Turing (RTX 20) | TU104 (RTX 2080-class) | **Owned reference**; further cards bought used | The current Reference profile and development host | +| Ampere consumer (RTX 30) | GA102/GA104 | Lab loan or purchase; workstation rental | Same passthrough path as the reference; engine inventory differs (docs/07 P2) | +| Ada consumer (RTX 40) | AD102/AD103/AD104 | Lab loan or purchase; workstation rental | ROM/VBIOS variants across board partners; inventory is profile data | +| Blackwell consumer (RTX 50) | GB202/GB203 | Vendor eval, partner loan, purchase | Newer driver/firmware family; validate the pairing before debugging | +| GB10-class | DGX Spark, Surface-class RTX laptops | Vendor eval / short loan, retail where available | aarch64 hosts, laptop ROM/VBIOS quirks, possible RV64GC and FMC partition model (docs/07 P3) | +| Datacenter compute | A100, H100/H200, L40S | Cloud bare-metal rental or partner lab; short loans | Rental instances frequently expose MIG/vGPU rather than raw PCIe passthrough — confirm before paying for a cycle | +| Datacenter/rack scale | GB200-class, NVL systems | Partner lab access only | Multi-tenant policy and MIG-aware handling; out of reach for casual rental | + +Procurement rules: + +* Prefer a **bare-metal rental with documented PCIe passthrough and IOMMU** over + a vGPU/mdev instance; the latter cannot exercise the VFIO mdev path. +* A borrowed card arrives with an unknown ROM/VBIOS and firmware pairing: + collect and hash everything in §4 before touching the emulation. +* One device per architecture/family at a time is enough to move the matrix; + do not accumulate hardware before the previous evidence bundle is complete. + +## 3. Physical preparation + +Preflight is a checklist, not a ritual (see the RUNBOOK for the host-specific +commands): + +1. **Host qualification** — x86_64 or aarch64 host, IOMMU enabled + (`intel_iommu=on`/`amd_iommu=on`, `iommu=pt` on aarch64), ACS/isolation + confirmed, kernel with `vfio-pci`; record `uname -a`, kernel, and QEMU + version. +2. **Slot and power** — card seated and powered per vendor spec; auxiliary + power connected; sufficient PSU headroom; thermals monitored for the cycle + (a throttled device invalidates timing-flavoured observations; timing is not + claimed anyway, docs/06 M7). +3. **Isolation** — bind the target GPU to `vfio-pci`; keep the host distro + driver reserved as the **oracle**, never as the emulated anchor (anchor + discipline, docs/22 §1). +4. **Baseline capture** — before any guest runs: host oracle captures + (`rmtool engine list --format=json`, `rmtool mem stats --format=json` + via `rmharness`, docs/17), `lspci -vvv` for the physical and virtual + functions, ROM/VBIOS hash, firmware family and version string. +5. **Guest image** — stock guest driver of the exact version under test, a + CUDA smoke workload and the ioctl/control tracer + (`tools/guest-diag/ioctl_trace.c`). +6. **Supervision plan** — named operator, one-cycle approval, expected + duration, watchdog, and the rollback path (old anchor directories and + config untouched; never reuse old-version addresses). +7. **Simulator rehearsal first** — run the sub-sections below in `--dry-run`, + then in simulator-only mode, before the physical cycle. A plan that cannot + complete in simulation is not ready for silicon. + +## 4. Test-and-port sequence: driver 540 to current + +The sequence is **ascending**: each branch becomes the reference for the next, +so a failure is attributed to one step, not to a jump. The current anchor is +**570.144**; the sweep continues to the newest locally available image, and +610+ stays behind the RM 610 approval gate (docs/23) until that approval +exists. The `harness/` commands below run from +`debug/risc-v-gsp-rm-vm/`; the supervised-cycle commands run on the host and +device under test. + +Per driver/firmware version `N`: + +1. **Acquire and version** — obtain the runfile and matching GSP `.fwimage` + through the project's acquisition rules; lay out + `firmware/<SERIES>-GSP/<N>/` and `config/versions/<N>.yaml` (documentation + until the load path reads it). Never overwrite a working version; hash + every input immediately. +2. **Static triage** — extract the image, build/locate the symbol DB, and run + the release pipeline in dry-run first: + `harness/version_diff.py` and `harness/ci_release.py` (docs/21). The sweep + orchestrator wraps this per version: + `python3 harness/port_driver_versions.py --versions <N> --target <N>=<fwimage> --dry-run` + (docs/28 §2; unadmitted pilot, evaluation in future-eng-ops F2). +3. **Simulator evidence** — execute the sweep against the previous branch and + the anchor: + `python3 harness/port_driver_versions.py --run --baseline <N-1> ...`; + the per-version bundle carries `version-diff/`, `portability-matrix/` and + `mutation-smoke/` results plus the M1–M6 skeleton. +4. **Device profile** — for a new device run + `python3 harness/port_new_device.py --run --device <name> --firmware <N-image> --reference <anchor-image>` + (docs/28 §3; unadmitted pilot, evaluation in future-eng-ops F3); fill the + profile from §4 evidence and reconcile the engine inventory against the + host oracle. +5. **Supervised physical cycle** (approved, attended): + * boot the guest with the stock driver of version `N` on the device bound + to `vfio-pci`; run the CUDA smoke and the ioctl tracer; + * capture host module trace (`gsp-resman-vfio:`), guest dmesg, + `nvidia-smi -q`, `lspci -vvv`, and the control/ioctl sequence; + * compare the physical captures with the simulator sessions + (`greeneye compare`, M1–M6) and resolve every delta as profile, logic or + fidelity gap (docs/07 P4). +6. **No new guest regression** — run the existing guest/CUDA ladder rungs and + confirm no rung regressed before the next version. +7. **Rollback point** — keep the previous version's directories, config and + guest driver untouched until the next version's bundle is complete. + +Only after the current branch's bundle is complete does it become the +reference for the next branch. 610 is not attempted here: see +[`../../debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md`](../../debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md). + +## 5. Sequence diagram + +```mermaid +sequenceDiagram + autonumber + participant O as Operator + participant H as Ops host + participant D as Device under test + participant G as Guest VM (stock driver) + participant S as green-eye harness + participant E as Evidence store + + O->>H: approve one supervised cycle (reboot/bind approval) + O->>H: preflight (IOMMU, vfio-pci bind, baselines) + H->>D: bind target GPU to vfio-pci + O->>S: sweep --dry-run for version N + S-->>E: plan, input hashes + O->>S: sweep --run vs baseline (simulator only) + S-->>E: version-diff + matrix + mutation bundles + O->>G: start guest with stock driver N, run CUDA smoke + G->>D: GSP RPC / control traffic + O->>H: capture host trace, guest dmesg, lspci, oracle + H-->>E: physical captures + O->>S: compare physical vs simulator (M1-M6) + S-->>E: delta report, fidelity gaps + O->>O: go/no-go decision + alt go + O->>E: publish bundle + matrix entry + else hold + O->>E: record gap, keep previous anchor + end + D->>H: rollback bind, restore host driver +``` + +## 6. Evidence collected per device + +| Item | Source | Artifact | +|---|---|---| +| Device profile (PCI IDs, BARs, FB geometry, engines, interrupts, ROM) | `port_new_device` profile + host sysfs | `profiles/<device>.json` | +| Admission/authorisation record | Ops process | cycle approval ID | +| Physical host facts | preflight | `lspci -vvv`, ROM/VBIOS hash, kernel/QEMU versions | +| Host oracle snapshots | `rmharness`/`rmtool` | `engine list`, `mem stats` JSON in a session | +| Firmware pairing | image hash + family + version string | top of the port report | +| Simulator bundle | `port_driver_versions`/`port_new_device` | `bundle-manifest.json` + tree, docs/28 | +| Physical captures | supervised cycle | host module trace, guest dmesg, `nvidia-smi -q`, ioctl sequence | +| Metric table | comparison | M1–M6 with reference-capture method and L2/L3 limits (docs/06 §5) | +| Unexplained-delta list | review | explicit, possibly empty; empty is a claim | +| Support decision | review | matrix entry with rung, date, driver/firmware, evidence hash | + +## 7. Go/no-go criteria + +Proceed to the next driver or publish a matrix entry only when **all** hold: + +* pairing verified and hashed before debugging (family, version string, FMC + manifest where applicable); +* every probe first-touch delta explained as profile data vs logic; +* engine/RPC inventory complete and reconciled against the host oracle; +* M1–M6 do not regress on the reference workload, or each regression has a + decided response; +* no guest-ladder rung regressed; rollback path intact; +* the evidence bundle is complete and hashed, including the physical capture; +* the physical cycle was supervised and approved. + +Hold (keep the previous anchor) when any of: ambiguous pairing, an unexplained +probe or inventory delta, a matrix/M-metric regression without a response, an +incomplete bundle, or a simulator-only claim with no physical capture. A +**fidelity gap** (simulator diverges, physical cannot reproduce) is recorded +as a gap, never "fixed" in the product from simulator evidence alone +(docs/07 P4.5). + +## 8. References + +* Support ladder and per-combination status: + [`hardware-software-matrix.md`](hardware-software-matrix.md) +* Current ops entry point and host procedures: + [`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md), + the [Open-IOV Defensive Operations Runbook](../RUNBOOK.md), and the + [green-eye user guide](../05-operations/04-green-eye-user-guide.md) +* Porting automation (sweep + device scaffold, bundles, M1–M6): + [`../../debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md`](../../debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md) +* Playbooks and porting guide: + [`../../debug/risc-v-gsp-rm-vm/docs/07-PORTABILITY-PLAYBOOKS.md`](../../debug/risc-v-gsp-rm-vm/docs/07-PORTABILITY-PLAYBOOKS.md), + [`../../debug/risc-v-gsp-rm-vm/docs/22-PORTING-CUDA-AND-DEVICES.md`](../../debug/risc-v-gsp-rm-vm/docs/22-PORTING-CUDA-AND-DEVICES.md) +* RM 610 gate: + [`../../debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md`](../../debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md) + +<!-- change (2026-10-02): correct the guest-diag tracer path, state the working +directory for the harness commands, and mark the porting scripts as unadmitted +pilots with their evaluation pointers; scope: §3.1, §4 only; benefit: commands +copy correctly and the scaffolds are not read as routine pipeline steps. --> diff --git a/docs/06-portability/hardware-software-matrix.md b/docs/06-portability/hardware-software-matrix.md index c839578..8fca9e9 100644 --- a/docs/06-portability/hardware-software-matrix.md +++ b/docs/06-portability/hardware-software-matrix.md @@ -35,7 +35,7 @@ yet and selection is compile-time (see §6). | Guest | Status | |---|---| -| Linux x86_64 (stock `nvidia.ko`, CUDA) | **Experimental** (bring-up; probe and `nvidia-smi` work, but there is no continuous guest testing and context creation is currently blocked — see §6.1) | +| Linux x86_64 (stock `nvidia.ko`, CUDA) | **Experimental** (bring-up; probe, `nvidia-smi` and the driver-API smoke pass, and CE copies are forwarded default-off, but kernel execution is blocked on GR engine-context validity and there is no continuous guest testing — see §6.1) | | Linux aarch64 | **Target** | | Windows x86_64 (WDDM driver, CUDA, 3D) | **Target** | | Other Unix (FreeBSD, ...) | Out of scope until demand | @@ -70,11 +70,34 @@ so new transports are a small, isolated implementation. 3. Record the differences; prefer configuration over code. 4. Publish the profile, the evidence, and any workarounds. +**Green-eye portability pipeline (experimental).** + +<!-- change (2026-10-02): add the green-eye portability pipeline to the new-GPU +procedure and point at the RUNBOOK/user guide; additive only. --> + +Before the profile change lands, run the firmware-side pipeline from +`debug/risc-v-gsp-rm-vm/` and attach its output: +`./greeneye.sh probe <image> --symbols <db> --record` for the first-touch map; +`./greeneye.sh symbols export ...` for a provenance-carrying symbol DB; +`python3 harness/portability_matrix.py --target <reference> --target <candidate> +--out dumps/matrix-<candidate>` for per-pair sessions and `compare` reports; +then classify profile-versus-logic deltas with the decision table in +`debug/risc-v-gsp-rm-vm/docs/14-ADVERSARIAL-INTEROP-GUIDE.md` §4.1. The +command-level walkthrough is in the +[green-eye user guide](../05-operations/04-green-eye-user-guide.md); the +incident procedure, evidence expectations and rollback rules are in the +[Open-IOV Defensive Operations Runbook](../RUNBOOK.md) S5. + ### 5.2 New firmware version See [`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md) §4.1. Firmware images are versioned under `firmware/<SERIES>-GSP/<VERSION>/` with -signatures and analysis artifacts. +signatures and analysis artifacts. The defensive procedure, evidence bundle +and rollback rules for this path are in the +[Open-IOV Defensive Operations Runbook](../RUNBOOK.md) S1. + +<!-- change (2026-10-02): cross-link the new firmware-version procedure to the +defensive-operations RUNBOOK S1; additive only. --> ### 5.3 New driver version @@ -118,6 +141,25 @@ config/ A profile is the *unit of support* — what is missing is the code that reads the profile files, not the profile schema. +**Evidence expectations for adding support.** + +<!-- change (2026-10-02): state the evidence required before a combination is +entered in the matrix, and link the RUNBOOK bundle checklist and green-eye +portability pipeline; additive only. --> + +A combination is entered in the matrix only with: the profile fields recorded; +the firmware/device pairing verified by hash and version string before any +debugging; a first-touch diff whose every changed address is explained as +profile or logic; a portability matrix report with per-pair `compare` results +and session hashes; the M1–M6 table with the reference-capture method and the +L2/L3 limits stated; and a supervised physical validation for the support +claim. "Tested and documented" is literal: an untested or simulator-only +combination stays **Target** or **Experimental**. The full bundle checklist +and fault tree are in the +[Open-IOV Defensive Operations Runbook](../RUNBOOK.md) §6–§7; the pipeline +commands are in the +[green-eye user guide](../05-operations/04-green-eye-user-guide.md). + ### 6.1 Userspace driver facts * **Guest CUDA driver** (`libcuda`) under test: **570.144** (md5 diff --git a/docs/07-developers/02-development-and-analysis-environment.md b/docs/07-developers/02-development-and-analysis-environment.md index 370cefc..114345b 100644 --- a/docs/07-developers/02-development-and-analysis-environment.md +++ b/docs/07-developers/02-development-and-analysis-environment.md @@ -189,6 +189,7 @@ capture them into sessions. make -C tools/rmtool && make -C tools/rmtop tools/rmtool/rmtool gpu list --format=json tools/rmtool/rmtool engine list && tools/rmtool/rmtool mem stats +cd debug/risc-v-gsp-rm-vm ./rmharness/harness.sh --session dumps/real-rm -- rmtool engine list --format=json ``` @@ -312,7 +313,7 @@ Detailed procedures are not duplicated here. |---|---|---|---| | New CUDA userspace (`libcuda` bump) | `cudatool.py info/delta`, `fdematch`, `diff_corpus.py`, name-map apply | guest smoke (`cuda_drv_test.c`), `cuda-ab`, symbol spot-checks | [`userspace/cuda/docs/07`](../../userspace/cuda/docs/07-version-upgrade-playbook.md); green-eye [`22`](../../debug/risc-v-gsp-rm-vm/docs/22-PORTING-CUDA-AND-DEVICES.md) §2 | | New GSP/RM firmware version | decompile + diff RPC dispatcher/controls; Ghidra symbol inventory | `greeneye probe` first-touch, scripted/re-hosted `record`, `compare`, `version_diff.py` | [`05-operations/02` §4.1](../05-operations/02-debugging-and-maintenance-workflows.md) (upgrade runbook); green-eye [`07 P1`](../../debug/risc-v-gsp-rm-vm/docs/07-PORTABILITY-PLAYBOOKS.md), [`21`](../../debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md), [`22`](../../debug/risc-v-gsp-rm-vm/docs/22-PORTING-CUDA-AND-DEVICES.md) §3 | -| New device / architecture | device profile data, ISA/image shape, PMU/SOE deltas | portability matrix, probe first-touch on the new image | [`06-portability/hardware-software-matrix.md`](../06-portability/hardware-software-matrix.md); green-eye [`07 P2/P3`](../../debug/risc-v-gsp-rm-vm/docs/07-PORTABILITY-PLAYBOOKS.md), [`18`](../../debug/risc-v-gsp-rm-vm/docs/18-HARNESS-AUTOMATION.md) | +| New device / architecture | device profile data, ISA/image shape, PMU/SOE deltas | portability matrix, probe first-touch on the new image | [`06-portability/hardware-software-matrix.md`](../06-portability/hardware-software-matrix.md) and [`03-device-validation-and-porting-sequence.md`](../06-portability/03-device-validation-and-porting-sequence.md); green-eye [`07 P2/P3`](../../debug/risc-v-gsp-rm-vm/docs/07-PORTABILITY-PLAYBOOKS.md), [`18`](../../debug/risc-v-gsp-rm-vm/docs/18-HARNESS-AUTOMATION.md) | | A/B accuracy ritual (real vs emulated) | M1–M6 metric definition | hardware/VFIO reference capture vs green-eye session | [`05-operations/01`](../05-operations/01-performance-correctness-security.md); green-eye [`06`](../../debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md), [`07 P4`](../../debug/risc-v-gsp-rm-vm/docs/07-PORTABILITY-PLAYBOOKS.md), [`25`](../../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md) | | RM 610 family | — | gate first; nothing opened before approval | green-eye [`23`](../../debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md) (hard gate), [`21` §3](../../debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md) | @@ -361,3 +362,8 @@ dynamic session disposes, and physical confirmation gates any product change. Every path in this document was verified at the time of writing; when a tool moves, update this table first — it is the front door. + +<!-- change (2026-10-02): fix the host-oracle capture command to run from the +tool root and link the device-validation/porting sequence from the porting +table; scope: §7, §12 only; benefit: the copied rmharness command resolves the +wrapper and the device-port path is discoverable from the environment map. --> diff --git a/docs/07-developers/onboarding-and-standards.md b/docs/07-developers/onboarding-and-standards.md index e62ec28..78100c4 100644 --- a/docs/07-developers/onboarding-and-standards.md +++ b/docs/07-developers/onboarding-and-standards.md @@ -62,8 +62,8 @@ open-iov/ ├── scripts/ build/load/test automation, VM harness ├── config/ hardware/version profiles (the portability surface) ├── firmware/ versioned firmware images + decompilation artifacts -├── (private) reverse-engineering study notes live untracked; -│ see the maintainers' study index +├── (private) reverse-engineering study notes live untracked, +│ outside the published tree ├── docs/ THIS documentation tree (start at docs/README.md) └── wiki/ long-form firmware/protocol deep dives ``` @@ -156,7 +156,7 @@ The following rules each come from a real bug and are enforced in review: | `tools/cuda-ab/` | host-vs-guest CUDA A/B harness (one deterministic workload, diffed results) | | `tools/rmq/` | host RM-API oracle for ground-truth GPU/RM data | | `tools/firmware_analyzer.py` + Ghidra scripts | firmware analysis pipeline | -| module debug parameters | guest-memory scanning/poking, VRAM inspection | +| module debug parameters | guest-memory scanning/poking, VRAM inspection; physical-backend parameters and debugfs state/control files (see [`../eng-ops.md`](../eng-ops.md) Part II) | | QEMU gdbstub workflow | guest virtual-memory inspection/writing | | `debug/risc-v-gsp-rm-vm/` (green-eye) | **# TOOL IN DEVELOPMENT - NOT READY YET!** — RISC-V GSP/uproc VM runtime inspection, session recording and `gsp-rm-vfio` accuracy measurement (see [`../05-operations/03-risc-v-gsp-rm-vm.md`](../05-operations/03-risc-v-gsp-rm-vm.md)) | | `wiki/` documents | protocol and firmware deep dives | @@ -193,6 +193,12 @@ make clean * Protocol changes: diff the RPC inventory before/after; attach both. * Memory-model changes: re-run the VA↔PA discovery tooling and a guest CUDA start-up; attach the trace. +* Live physical-backend cycles are **supervised only**: one cycle at a time, + never unattended, with the module sha256 recorded before and after and the + clean-release checklist walked every time. Do not retry after a + mutating-RPC failure or a wedge. A requested reboot needs a written reason + and a post-reboot plan before approval is asked (project hard rule). See + [`../eng-ops.md`](../eng-ops.md) Part II. * Never land a change that makes a rung regress, even "temporarily". ## 8. Security hygiene for contributors diff --git a/docs/08-ecosystem/haaeep-and-collaboration.md b/docs/08-ecosystem/haaeep-and-collaboration.md index 954438c..ee847d4 100644 --- a/docs/08-ecosystem/haaeep-and-collaboration.md +++ b/docs/08-ecosystem/haaeep-and-collaboration.md @@ -92,7 +92,7 @@ licensed and version-durable — so it is the component we are building. | Milestone | Depends on | |---|---| -| CUDA execution in a guest (rung 3) | Rung 2 (runtime start) being green first — context creation is in progress (`cuCtxCreate` currently ends in a forced `rc=999` on a completion wait) — and then Open-IOV forwarding work (backend chosen: a module-owned GPU with a real-GSP session). The interface contract is documented under [`../../userspace/cuda/README.md`](../../userspace/cuda/README.md) (start at [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md)) | +| CUDA execution in a guest (rung 3) | Rung 2 (runtime start) is green — the driver-API smoke test completes end to end (`ALL OK`, payload 64/64). Copy-engine-class work and its releases are forwarded to real hardware with release and fault relay in place (parameters default off); graphics/compute (GR) execution is the open frontier. The interface contract is documented under [`../../userspace/cuda/README.md`](../../userspace/cuda/README.md) (start at [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md)) | | Two-VM coexistence (inference + agent) with isolation | Open-IOV multi-tenancy + platform MAC | | Xen/OpenXT or Qubes Air variant | Open-IOV transport backends | | Attested boot-to-inference chain | TrenchBoot integration | diff --git a/docs/09-reference/glossary-and-faq.md b/docs/09-reference/glossary-and-faq.md index e0c68f6..1668e35 100644 --- a/docs/09-reference/glossary-and-faq.md +++ b/docs/09-reference/glossary-and-faq.md @@ -97,16 +97,24 @@ MIG is a hardware partitioning mechanism; Open-IOV's scheduling policy can use it where available. It is part of the multi-tenancy work. **How fast is it?** -No published numbers yet: execution forwarding is not implemented, and even -the initialization path has not been measured under the project's methodology. -Any performance claim will come with the methodology and the measurements -together (see +No published numbers yet: copy-engine-class work forwarding and engine-authored +release + fault relay are implemented but default off, and GR/kernel execution +is still the open frontier; even the initialization path has not been measured +under the project's methodology. Any performance claim will come with the +methodology and the measurements together (see [`../05-operations/01-performance-correctness-security.md`](../05-operations/01-performance-correctness-security.md)). **Where do I report bugs or ask for help?** Open an issue with the standard captures listed in the debugging document. A reproduction plus logs is worth more than a description. +**Can I run the live hardware cycles?** +Live physical-backend cycles are supervised only — one at a time, never +unattended — and a system is never rebooted without a written reason, a +post-reboot plan and explicit human approval. The operational detail is in the +engineering-operations document, Part II +([`../eng-ops.md`](../eng-ops.md)). + **Can my company use this commercially?** Yes, under BSD-3-Clause (kernel modules GPL-2.0), with no fee. We also welcome co-development; see the ecosystem document. diff --git a/docs/README.md b/docs/README.md index 61aa529..77f4069 100644 --- a/docs/README.md +++ b/docs/README.md @@ -98,7 +98,10 @@ and [`01-orientation/02-the-ai-edge-and-ecosystem.md`](01-orientation/02-the-ai- ## 3. Where is the project today? -**Honest status (2026-10-01):** +**Honest status (2026-10-02):** + +A dated snapshot of this status, refreshed on every progress check, is kept at +[`../STATE-CURRENT.md`](../STATE-CURRENT.md). | Area | State | |------|-------| @@ -107,8 +110,8 @@ and [`01-orientation/02-the-ai-edge-and-ecosystem.md`](01-orientation/02-the-ai- | GSP-RM message queue + RPC transport | ✅ Working | | GSP-RM controls: device/fb/bus tables, CE caps, vaspace, GMMU static info | 🟡 Mostly — the engine inventory now includes the async copy engine and the CE capabilities match the device; reply fidelity for the remaining consumed fields is still being tightened | | Guest `nvidia.ko` probe, `nvidia-smi` | ✅ Working (TU104) | -| Guest CUDA driver (`libcuda`) startup | ✅ Driver-API baseline — `cuInit`, device enumeration, `cuCtxCreate`, `cuMemAlloc`, copies, free and destroy complete end to end; the smoke test prints `ALL OK` and the 64-byte payload round-trips exactly (64/64, re-verified 2026-10-01 on the current build). See [`04-implementation/01-known-knowns-and-unknowns.md`](04-implementation/01-known-knowns-and-unknowns.md) and [`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md) | -| CUDA kernel execution | 🟡 The backend is live: the module owns the physical GPU and its real GSP-RM initialized, serves the full static-config reply and shuts down cleanly; a host resource-manager client now allocates memory/VAS/page-directory/TSG/channel and has executed the first real copy on the physical copy engine (4 KiB, semaphore released, dst==src, 2026-10-01). What remains is wiring that session into the guest-facing path so guest submissions execute there ([`03-architecture/11-physical-backend-guest-wiring.md`](03-architecture/11-physical-backend-guest-wiring.md)); see also [`03-architecture/08-physical-gpu-integration.md`](03-architecture/08-physical-gpu-integration.md) | +| Guest CUDA driver (`libcuda`) startup | ✅ Driver-API baseline — `cuInit`, device enumeration, `cuCtxCreate`, `cuMemAlloc`, copies, free and destroy complete end to end; the smoke test prints `ALL OK` and the 64-byte payload round-trips exactly (64/64, re-verified 2026-10-02 on the current build). See [`04-implementation/01-known-knowns-and-unknowns.md`](04-implementation/01-known-knowns-and-unknowns.md) and [`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md) | +| CUDA kernel execution | 🟡 Copy-engine-class guest copies and their releases are forwarded to the real hardware with release and fault relay in place (parameters default off); GR (graphics/compute) execution is the open wall — the host GR ladder brings up a GR0 channel with a compute-class object, but its first push faults on a context that was never committed. The device is currently held in the designed locked fail-safe awaiting a supervised reboot. See [`03-architecture/11-physical-backend-guest-wiring.md`](03-architecture/11-physical-backend-guest-wiring.md), [`03-architecture/08-physical-gpu-integration.md`](03-architecture/08-physical-gpu-integration.md) and [`../STATE-CURRENT.md`](../STATE-CURRENT.md) | | OpenGL / Vulkan | 🔴 Not yet | | Windows guest | 🔴 Not yet | | Multi-tenant isolation (VFs, mdev-per-VM) | 🟡 Designed, partially implemented | @@ -118,14 +121,16 @@ the userspace `libcuda` segfault was fixed (an RM allocation reply now echoes the request parameters instead of returning zeros), the engine inventory is complete, and the driver-API smoke test now completes end to end — `cuCtxCreate` returns and the 64-byte payload round-trips exactly (64/64, re-verified -2026-10-01 on the current build). The full +2026-10-02 on the current build). The full mechanism, with offsets and the decompiled call path, is in [`../userspace/cuda/docs/05-570.144-decompilation.md`](../userspace/cuda/docs/05-570.144-decompilation.md). -The remaining frontier is guest-facing execution: the runtime client initializes -and queues a kernel launch, and the physical backend now executes real work (the -first copy-engine transfer completed on hardware — 4 KiB with the semaphore -released and dst==src, 2026-10-01) — but the two planes are not yet wired -together. The backend, a module-owned GPU running its real GSP firmware as the +The remaining frontier is guest-facing execution: copy-engine-class copies and +their releases are forwarded to real hardware with release relay and fault +relay in place (parameters default off); GR execution is the open wall — the +host GR ladder brings up a GR0 channel with a compute-class object, but its +first push faults on an engine context that was never committed. The device is +currently held in the designed locked fail-safe awaiting a supervised reboot. +The backend, a module-owned GPU running its real GSP firmware as the resource-manager server, is specified in [`03-architecture/08-physical-gpu-integration.md`](03-architecture/08-physical-gpu-integration.md); the bridge from the live session to the guest-facing mdev/VFIO path is specified @@ -170,16 +175,36 @@ Read in order if you are new. Jump directly if you know what you need. | 18 | [Performance, correctness, security](05-operations/01-performance-correctness-security.md) | developers, auditors | 25 min | | 19 | [Debugging and maintenance workflows](05-operations/02-debugging-and-maintenance-workflows.md) | developers, maintainers | 25 min | | 20 | [GSP in a RISC-V VM (green-eye, experimental)](05-operations/03-risc-v-gsp-rm-vm.md) — **# TOOL IN DEVELOPMENT - NOT READY YET!** | developers, analysts | 15 min | -| 21 | [Hardware and software matrix](06-portability/hardware-software-matrix.md) | developers, integrators | 20 min | -| 22 | [Developer onboarding and standards](07-developers/onboarding-and-standards.md) | contributors | 20 min | -| 23 | [HAAEEP and collaboration](08-ecosystem/haaeep-and-collaboration.md) | stakeholders, partners | 20 min | -| 24 | [Glossary and FAQ](09-reference/glossary-and-faq.md) | everyone | reference | +| 21 | [green-eye user guide (experimental)](05-operations/04-green-eye-user-guide.md) — **# TOOL IN DEVELOPMENT - NOT READY YET!** | developers, analysts | 15 min | +| 22 | [Defensive Operations Runbook](RUNBOOK.md) | maintainers, operators | 20 min | +| 23 | [Engineering operations](eng-ops.md) — tool admission and workflows (Part I) plus live physical-backend operations (Part II) | maintainers, developers | 20 min | +| 24 | [Hardware and software matrix](06-portability/hardware-software-matrix.md) | developers, integrators | 20 min | +| 25 | [Developer onboarding and standards](07-developers/onboarding-and-standards.md) | contributors | 20 min | +| 26 | [HAAEEP and collaboration](08-ecosystem/haaeep-and-collaboration.md) | stakeholders, partners | 20 min | +| 27 | [Glossary and FAQ](09-reference/glossary-and-faq.md) | everyone | reference | + +<!-- change (2026-10-02): insert the green-eye user guide and the Open-IOV +Defensive Operations Runbook into the reading list; renumber rows 21-24 to +23-26; additive only. --> + +<!-- change (2026-10-02): insert the engineering-operations document into the +operations group; renumber rows 23-26 to 24-27; additive only. --> Row 15 is a **deferred future goal**: host NVIDIA-driver acceleration over a virtual function. The feasibility verdict and a first design sketch are recorded there for planning; it is not on the active critical path (guest CUDA execution, multi-VM, 3D, performance come first). +> **Tool documentation is under development.** The per-tool pages under +> [`tools/`](tools/README.md) (for example `rmtool.md` and `rmtop.md`) are +> being expanded; until they are complete, start with the +> [Defensive Operations Runbook](RUNBOOK.md) and the +> [green-eye user guide](05-operations/04-green-eye-user-guide.md). + +<!-- change (2026-10-02): state that the per-tool documentation section is +under development and point readers at the RUNBOOK and the green-eye user +guide; additive only. --> + Deep-dive material lives in [`../wiki/`](../wiki/README.md) (firmware decompilation record, address reference, historical protocol notes). Wiki pages that predate the current implementation carry a status banner naming @@ -191,6 +216,15 @@ The pre-reboot flat documents have been merged into this tree and removed * debugging: gdbstub/NMI workflows, the EPT-livelock case study and the failure taxonomy → [`05-operations/02-debugging-and-maintenance-workflows.md`](05-operations/02-debugging-and-maintenance-workflows.md) +* defensive operations: incident scenarios, the OODA ordering, evidence + bundles and the escalation tree → + [`RUNBOOK.md`](RUNBOOK.md); the green-eye quickstart, tool chooser and + troubleshooting → + [`05-operations/04-green-eye-user-guide.md`](05-operations/04-green-eye-user-guide.md) + +<!-- change (2026-10-02): point the post-merge document map at the new RUNBOOK +and green-eye user guide; additive only. --> + * firmware runtime analysis (experimental): running GSP/uproc firmware in a RISC-V QEMU VM with hooks, watchpoints, memory dumps and `gsp-rm-vfio` accuracy measurement — **# TOOL IN DEVELOPMENT - NOT READY YET!** → @@ -213,7 +247,9 @@ The pre-reboot flat documents have been merged into this tree and removed and [`06-portability/hardware-software-matrix.md`](06-portability/hardware-software-matrix.md). What remains historical is the pre-supersession wiki material and -`old/ARCHITECTURE.md`; their status banners name the canonical document. (The +`old/ARCHITECTURE.md`; the latter carries an explicit superseded banner +pointing at this page, and the older wiki pages carry status banners naming +the canonical document. (The former 2026-09-21 `STATE.md` snapshot is retired; its content lives on in the git history and its replacements above.) diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md new file mode 100644 index 0000000..79c1735 --- /dev/null +++ b/docs/RUNBOOK.md @@ -0,0 +1,744 @@ +# Open-IOV Defensive Operations Runbook + +> The per-incident book for operating an independent IP block (Open-IOV) +> against ecosystems that change on their own schedule. NVIDIA ships new GSP +> firmware, kernel modules, CUDA userspace, devices and ISA revisions without +> coordinating with us; any of them can break the emulation while the others +> stand still. This runbook is the ordered response. +> +> **Related:** [green-eye user guide](05-operations/04-green-eye-user-guide.md) +> · [GSP in a RISC-V VM (experimental)](05-operations/03-risc-v-gsp-rm-vm.md) +> · [Debugging and maintenance workflows](05-operations/02-debugging-and-maintenance-workflows.md) +> · [Hardware and software matrix](06-portability/hardware-software-matrix.md) +> · [Adversarial interoperability](04-implementation/02-practical-challenges-and-adversarial-interop.md) +> · [Developer onboarding and standards](07-developers/onboarding-and-standards.md) +> · [Engineering operations](eng-ops.md) +> +> **# TOOL IN DEVELOPMENT - NOT READY YET!** The green-eye tooling cited here +> is experimental; its interfaces and file formats may change. Nothing in this +> runbook authorises a product change, a hardware write or work on RM 610. + +## 0. How to use this runbook + +Audience: maintainers, release engineers and on-call operators of the Open-IOV +emulation and its evidence pipeline. + +Rules for reading under pressure: + +1. Find the symptom in §5 (scenario catalogue) or §9 (one-page decision tree). +2. Follow the **response order** exactly; do not skip the pairing/hash check. +3. Before any action that could change `open-iov-vfio`, complete the evidence + bundle (§6) and pass the escalation gates (§7). +4. Record the outcome where the scenario says to record it — a finding without + a record is lost work. +5. If an action is not in this runbook and not in the linked project docs, + treat it as a candidate for a new runbook entry, not as improvisation. + +Command conventions: `open-iov/` is the repository root. Commands run from the +repository root unless a block starts with `cd`. Angle-bracket values +(`<VER>`, `<session>`, `<image>`) are supplied by the task; never hardcode +private paths, host names or serial numbers into tracked documents. + +## 1. Operating principles and hard rules + +These apply to every scenario; where a scenario conflicts with them, the rules +win. + +| # | Principle | Consequence | +|---|---|---| +| P1 | **Anchor discipline.** The lock-step is *guest driver ↔ emulated firmware profile*. The host distribution driver is the A/B oracle, not the anchor. | Never evaluate a new `libcuda` against an old emulated firmware profile and call the mismatch an emulation bug. | +| P2 | **Never patch what we do not own.** Guest driver, host driver, GPU firmware, hypervisor, device model and guest OS stay unmodified. | Product changes live in our module, userspace, tools, configuration, tests and docs only. | +| P3 | **Evidence over anecdote.** Every conclusion needs at least an A/B pair and an explicit fidelity level (L0–L3). | A single run, a static reading or a simulator-only delta cannot land a product change. | +| P4 | **Physical confirmation for behaviour changes.** A simulator observation is evidence about the firmware, not about silicon. | Reproduction on a supervised physical/vfio-mdev cycle is required before a change that alters emulation behaviour. | +| P5 | **The simulator is a study environment, never a security boundary.** | No confidentiality, side-channel or security claims; never use injection to defeat a hardware security boundary. | +| P6 | **Development-only tooling.** green-eye is never shipped, never a dependency of the emulation, never in a guest. | Keep the in-development banner on tool docs until the acceptance list passes. | +| P7 | **No unattended physical-GPU cycles.** Physical work is supervised, with fail-safe release and explicit reboot approval. | Stop and ask; never bind/unbind drivers or claim the GPU as an experiment. | +| P8 | **RM 610 is a hard human gate.** Until a named human approves in writing, no acquisition, analysis, emulation edit, physical work or support claim for 610. | The gate is enforced in code by `version_diff`/`ci_release`; do not work around it. | +| P9 | **Fail closed, record honestly.** Unverified host info, unpaired firmware, unexplained deltas and absent references stay open. | "No unexplained deltas" is a claim that must be earned, not a default. | + +## 2. Maintainer roles and responsibilities + +One name per role at any time; handovers are a written change note in the +affected doc or PR. + +| Role | Owns | Key duties | Gates they hold | +|---|---|---|---| +| **Anchor maintainer** | guest driver ↔ emulated firmware anchor; version surface | keeps the anchor current; approves anchor bumps; keeps `config/versions/*.yaml` labelled as design targets until read by code | any change to the anchor | +| **Firmware triage maintainer** | versioned firmware directories; symbol DBs; probe/version-diff runs | acquires and hashes new images; builds symbol DBs with provenance; runs the P1 loop; owns the pairing decision | pairing verdict; "new logic" vs "behaviour moved" | +| **green-eye tooling maintainer** | `debug/risc-v-gsp-rm-vm/`; test suite; session format | keeps `make test` green; guards format/API stability; keeps the marker and doc map honest; maintains golden sessions | tool status claims; removal of the in-development marker | +| **CI/release maintainer** | `harness/version_diff.py`, `harness/ci_release.py`, `greeneye.yml`, goldens | runs dry-run before real runs; hashes evidence bundles; diffs seeded corpora; keeps CI paths and budgets bounded | release triage sign-off; RM 610 gate enforcement | +| **Physical-cycle supervisor** | supervised physical/vfio-mdev cycles | enforces fail-safe release, D0/D3cold handling and reboot approvals; captures reference traces | all physical execution; reboot requests | +| **Evidence archivist** | session and bundle retention | keeps SHA-256 inventories; enforces provenance (no private paths, no unsanitised host data); applies retention/prune policy | publication of evidence | +| **Docs steward** | this tree | enforces change notes, link checks and doc-map updates; retires stale claims | docs are part of "done" | + +Escalation order for an ambiguous incident: triage maintainer → anchor +maintainer → physical-cycle supervisor (if hardware is involved) → project +maintainer channels. Escalate with a reproduction and an evidence pack, never +with a summary alone. + +## 3. Routine maintenance + +### 3.1 Golden sessions and determinism + +| Item | Cadence | Procedure | Pass condition | +|---|---|---|---| +| Demo golden session | every tool change and weekly | `./greeneye.sh demo` and compare against the archived session/expected events | UART banner and control-block liveness present; hook/watch/breakpoint sequence coherent | +| Portability self-comparison | per PR touching `harness/`, `greeneye/`, `sim/` | `python3 harness/portability_matrix.py --target firmware/uproc_demo/build/uproc_demo.elf --target firmware/uproc_demo/build/uproc_demo.elf --out dumps/matrix-selfcheck` | `all_zero` deltas in `portability-report.json` | +| Mutation replay | per PR touching mutation/harness code | run `mutation_harness.py` twice with the same seed and budgets | `findings.jsonl` byte-identical | +| Version-diff self-check | per release-triage change | `version_diff.py` old == new | zero symbol/probe/behavior deltas | +| Host oracle capture | per anchor change | `./rmharness/harness.sh --session dumps/real-<anchor> --name before -- rmtool engine list --format=json` (repeat for `mem stats`) | capture recorded with sizes and SHA-256 | + +Golden artifacts are append-only history: never edit a stored golden session +(§3.2 uses SHA-256). Re-baseline by creating a new session with a change note. + +### 3.2 Session hygiene + +* Sessions are directories (`dumps/<stamp>-<name>/`) with `session.json`, + `events.jsonl`, `snapshots/`, `blobs/` and `summary.md`. Keep them together; + a copied `events.jsonl` without its `session.json` is not evidence. +* Name sessions for the question asked, not the host or the operator. +* Keep sessions bounded: fixed stimulus, fixed budgets, region-scoped + snapshots; a multi-gigabyte trace is a run that was not scoped. +* Publish a SHA-256 list (or the `bundle-manifest.json`) with any session that + leaves the machine. +* Write findings into `note` events inside the session (self-describing packs); + never rely on a chat log to explain a recording. +* Apply the provenance rule everywhere: tracked docs describe components and + behaviour; sessions contain no host-specific paths in their metadata. +* Bound growth deliberately: `dumps/` is gitignored scratch, so keep every + golden session and release bundle, publish the SHA-256 list before deleting + anything, and prune only scratch sessions past the review window — a + multi-gigabyte session is a scoping failure (fixed stimulus, fixed budgets, + region-scoped snapshots), not a storage problem to solve with new tooling. + +### 3.3 Toolchain updates + +| Dependency | Check | Action | +|---|---|---| +| QEMU | `./greeneye.sh info`; run `make test` | install/renew `qemu-system-riscv64` (Ubuntu `qemu-system-misc`); set `GREENEYE_QEMU` if the binary is elsewhere | +| QEMU plugin API | `make -C qemu` after a QEMU upgrade | rebuild `qemu/greeneye-trace.so` against the vendored `qemu-plugin.h`; re-run a trace smoke | +| Python | `python3 --version` (3.10+; stdlib only) | no pip packages required; keep plugins/harnesses stdlib-only | +| clang/lld | `make demo`, `make sdk` | reinstall toolchain if the demo or SDK stops building | +| Test suite | `python3 -m unittest discover -s tests -t .` | all green; skips for absent QEMU/demo explained, not ignored | +| Native parity | `python3 examples/scripts/sim_diff.py` | `PARITY: OK` (or a clean skip when QEMU is absent) | +| Docs | `python3 scripts/check-docs-links.py` from `open-iov/` | no broken relative links | + +### 3.4 Documentation and marker discipline + +* The **# TOOL IN DEVELOPMENT - NOT READY YET!** banner stays on tool docs + until the acceptance list in + `debug/risc-v-gsp-rm-vm/docs/00-OVERVIEW.md` passes; removal is a reviewed + change, not an edit. +* Every behaviour change updates the doc that describes it (docs are part of + the definition of done); planned work is labelled `planned (docs/xx)`. +* Status tables must match the code on the current commit (commands, flags, + test counts); stale claims are bugs. +* Keep the doc map complete when adding documents; keep change notes + (`<!-- change (YYYY-MM-DD): purpose; scope; benefit -->`) at edited + locations. +* Record reproducible discoveries in the failure taxonomy of + [`05-operations/02-debugging-and-maintenance-workflows.md`](05-operations/02-debugging-and-maintenance-workflows.md) + §5, not only in a session directory. +* Tool admission, extension, trialing and retirement are governed by + [`eng-ops.md`](eng-ops.md); procedural detail that is not incident response + belongs there, not here. + +## 4. The OODA ordering for a hostile update + +"Hostile" here means *unplanned firmware change plus deliberate opacity* — +new images on a hidden schedule, stripped and signed, hardware-coupled, +possibly with anti-analysis or decoy behaviour. It does not assume a malicious +actor, and it does not justify shortcuts. + +| Stage | Order | Actions | Exit criterion | +|---|---|---|---| +| **Observe** | 1 | Freeze the anchor; acquire the new input through the approved channel; record hash, version string and family; do not edit the emulation | Input is versioned and hashed in a parallel directory | +| | 2 | Capture the baseline: real-backend reference traces for the fixed workload (or the archived previous capture) | Baseline exists with a provenance note | +| | 3 | Record the symptom in a session note and a reproduction | Reproduction is repeatable and bounded | +| **Orient** | 4 | Static map: symbol DB/corpus diff per version | Inventory deltas listed | +| | 5 | Dynamic map: probe first-touch, then fixed-stimulus sessions on both versions | `compare` report with hit/watch/write deltas | +| | 6 | Classify each delta with the decision table (code changed vs behaviour moved) | Every delta has a class or is explicitly open | +| **Decide** | 7 | Fill the M1–M6 skeleton with the reference-capture method; list unexplained deltas | Metric table and open list attached | +| | 8 | Choose: no change, version-keyed handler, profile data, or fidelity gap | Decision names the evidence that drove it | +| | 9 | If the decision changes emulation behaviour, require physical reproduction; if it touches RM 610, stop at the gate | Gate passed or work parked | +| **Act** | 10 | Implement in `open-iov-vfio` with the evidence bundle linked; never in the simulator | PR carries the bundle and the metric table | +| | 11 | Re-run the identical stimulus; require profiles to improve or stay equal; re-run the guest ladder | No rung regression; CI green | +| | 12 | Archive the bundle and update the matrix/version docs only where earned | Recorded and reversible | + +Ordering rules that keep this honest: + +1. Never orient before the input is hashed and paired. +2. Never decide from a single session (P3). +3. Never act on the product from simulator-only evidence (P4). +4. Never let "the new version" become the anchor implicitly; the anchor moves + by explicit decision. + +## 5. Scenario catalogue + +Each scenario follows the same template: symptoms, detection, evidence, +response order, tools/commands, decision criteria, rollback, where to record. +Scenarios are incidents to run, not roadmap claims. + +### S1 — New GSP/RM firmware release + +| Attribute | Detail | +|---|---| +| **Symptoms** | Init fails with a generic status (the real 0x59 chase: `GSP_INIT_DONE rpc_result=0x59`), handler entries or reply shapes shift, a previously green guest step now fails, control inventory gains/removes IDs. | +| **Detection** | Firmware hash/version string changed; probe first-divergence address changed; symbol/corpus inventory delta; fixed-stimulus behaviour delta. | +| **Evidence** | Firmware SHA-256 + version string + pairing; old/new symbol DBs with provenance and bias; probe sessions; version-diff report; behaviour `compare`; M1–M6 skeleton; unexplained-delta list. | +| **Decision criteria** | Pairing mismatch → stop. Code changed and live profile changed → prioritise new logic. Code unchanged and profile changed → behaviour moved (state/ordering/environment fix). Symbol never hit either side → coverage gap, widen the stimulus first. | +| **Rollback** | The previous version's directory and configuration stay untouched; roll back by returning to the old anchor commit/tag and the old guest driver. A forward attempt never destroys the working setup. | +| **Record in** | The version's `firmware/<SERIES>-GSP/<VER>/` notes, the PR body (link the bundle), and the matrix entry only once support is earned. | + +**Response order** + +1. Freeze the anchor and the workload; do not edit emulation code. +2. Acquire the image through the approved channel; lay down a new versioned + directory; hash it. +3. Build a symbol DB for the new image and the previous one. +4. Probe both images with identical settings; diff the first-touch maps. +5. Run the static triage (dispatcher/handler/control/corpus diff). +6. Run the same stimulus on both (re-hosted or demo analog) and compare. +7. Fill M1–M6 with the reference-capture method; classify every delta. +8. Escalate any product-behaviour change to a physical reproduction. +9. Land the change (or record the fidelity gap) with the evidence bundle; add + a version-keyed handler only where the diff requires it. + +**Commands** + +```bash +cd debug/risc-v-gsp-rm-vm + +# 1) symbol DBs, one per version +./greeneye.sh symbols export --index <export-VER.tsv> --bias 0 -o symbols-<VER>.json + +# 2) bounded probe of each image (records a session) +./greeneye.sh probe <gsp-<VER>.fwimage.bin> --symbols symbols-<VER>.json --record + +# 3) full old-vs-new triage (shape, symbols, corpus, probe, behavior, M1-M6) +python3 harness/version_diff.py \ + --old <gsp-<OLD>.fwimage.bin> --new <gsp-<NEW>.fwimage.bin> \ + --old-symbols symbols-<OLD>.json --new-symbols symbols-<NEW>.json \ + --old-label <OLD> --new-label <NEW> \ + --out dumps/vdiff-<OLD>-<NEW> + +# 4) behaviour compare and canonical report +./greeneye.sh compare dumps/<old-run> dumps/<new-run> --symbols +./greeneye.sh analyze dumps/<new-run> --symbols symbols-<NEW>.json +``` + +### S2 — New kernel driver / open-kernel-modules release + +| Attribute | Detail | +|---|---| +| **Symptoms** | The optional OpenRM build fails with `Unknown symbol`; exports move or disappear between tags; the pinned build glue no longer matches the installed modules; `modinfo`/`dmesg` report version drift. | +| **Detection** | `./scripts/build_nvidia_openrm.sh` failure; `dmesg | grep 'Unknown symbol'`; `kernel-open` tree diff for the two tags; SBOM tag mismatch. | +| **Evidence** | Old/new tag identifiers; build transcript; the `Unknown symbol` list; a recorded classification of each moved export; confirmation that the runtime path still uses the stock distribution driver. | +| **Decision criteria** | If only build glue breaks and the runtime path is unaffected → repair or re-pin the glue. If an exported interface used by the emulation changed → treat as S1/S3 (functionality moved). If the distribution driver moved → it is the oracle; do not "upgrade" the anchor to match it. | +| **Rollback** | Restore the previously pinned module tag and rebuild; revert the glue commit. | +| **Record in** | `SBOM.md` tag updates, the build glue commit message, and the failure taxonomy if it is a reusable lesson. | + +**Response order** + +1. Verify which driver is actually in use (guest stock driver vs optional host + OpenRM modules) before debugging. +2. Reproduce the build failure with a clean tree; capture the full error. +3. Diff the `kernel-open` trees for the old and new tags (publish only the + symbol/interface delta). +4. Classify: build-glue-only, symbol rename, interface removal, or new + functionality. +5. Repair build glue if and only if the runtime contract is unchanged; + otherwise route to S3. +6. Re-run `make kernel`, the host test suite, and the optional OpenRM build. + +**Commands** + +```bash +# optional OpenRM build (build glue; never a runtime dependency) +./scripts/build_nvidia_openrm.sh +make kernel +./scripts/stack_manager.sh test + +# evidence of the failure +sudo dmesg -T | grep -i 'Unknown symbol' + +# runtime path sanity: guest remains stock +# (guest: stock nvidia.ko; no Open-IOV code in the guest, ever) +``` + +### S3 — Functionality moving between kernel and GSP, or between userspace versions + +| Attribute | Detail | +|---|---| +| **Symptoms** | A control that used to be answered by the guest kernel is now requested from the firmware (or vice versa); new RPCs/controls appear; reply fields that used to be populated are ignored; a workload passes on one component mix and fails on another. | +| **Detection** | Ioctl trace changes; control sweep finds new/removed IDs; dispatcher hit profile changes while static code is unchanged; `compare` shows hit/watch deltas with no source delta. | +| **Evidence** | Recorded ioctl/control diff; `discovery-report.json` old vs new; `reach.py` reports for moved targets; session A/B pair; M1/M3/M4 rows with the reference method. | +| **Decision criteria** | Same code and new hit profile → behaviour moved to a new caller/state; new handler ID → new logic; `default-path` result change → fail-closed/default behaviour changed; coverage gap → widen the sweep before deciding. | +| **Rollback** | Keep both component versions in parallel directories; revert the mix to the last green combination; never convert a version-keyed handler into a version-blind one. | +| **Record in** | The version's control-inventory notes, the interface docs for the exact version, and the PR that adds/updates the version-keyed handler. | + +**Response order** + +1. Capture the ioctl/control stream on the failing mix and on the last green + mix. +2. Sweep the call surface (control discovery) and diff the inventory. +3. Use `reach.py` to prove each moved target is reachable under the exact + stimulus and to record its trigger. +4. Attribute every delta to kernel, userspace or firmware with a session; + fill M1/M3/M4. +5. Update the version-keyed handler or capability table only for the classified + deltas; physical-confirm any behaviour change. + +**Commands** + +```bash +cd debug/risc-v-gsp-rm-vm + +# unknown-call sweep and inventory diff between versions +python3 harness/control_discovery.py --out dumps/discovery-<VER> --seed <seed> + +# reachability/trigger/args for a moved symbol +python3 harness/reach.py --target <symbol> --method auto --out dumps/reach-<symbol> + +# symbol-attributed A/B +./greeneye.sh compare dumps/<old-run> dumps/<new-run> --symbols +``` + +### S4 — New CUDA userspace / toolkit interacting unexpectedly with the GSP + +| Attribute | Detail | +|---|---| +| **Symptoms** | `cuInit` succeeds but context creation fails; a generic SUCCESS reply with incoherent consumed fields; a new or reordered control appears immediately before a failure; `libcuda` crashes (`lib[base+size] ip`); result mismatch in the A/B workload. | +| **Detection** | `cudatool.py info`/`delta` on the new `libcuda`; `fdematch` transfer measurement; corpus diff; ioctl trace diff; `ab.sh` result comparison. | +| **Evidence** | `delta` report (exports added/removed/moved, `.text` growth, build-id); `fdematch` table; wrapper re-resolution evidence; ioctl/control diff explained line by line; `cdt` output ending `ALL OK`; `ab.sh` `RESULT`/`STATUS` match; symbol DB with bias. | +| **Decision criteria** | From the delta: no export moves + small `.text` growth → patch (name map transfer, spot checks); new/removed exports → feature (decode the new surface first); many moves → reshuffle (full re-analysis; never transfer wrapper addresses). Anchor mismatch (new `libcuda` vs old firmware profile) → fix the pairing first. | +| **Rollback** | Keep `userspace/cuda/<version>/` trees in parallel; revert to the last green userspace version. Never overwrite an analysed version. | +| **Record in** | The version's decompilation map/`names.tsv`, the version-upgrade playbook notes, and the firmware-anchor note when the anchor also moves. | + +**Response order** + +1. Cheap triage first (minutes): `info` + `delta`; classify the port. +2. Measure name transfer before copying anything (`fdematch`). +3. Build the green-eye symbol DB for the new version so runtime addresses + resolve; spot-check transferred addresses. +4. Confirm dynamically: `cdt` with the ioctl tracer, then `ab.sh all`, against + the **same** emulated firmware profile (anchor discipline). +5. Diff the ioctl/control sequence old vs new; explain every change. +6. Only then touch the emulation, with physical confirmation per P4. + +**Commands** + +```bash +cd userspace/cuda +tools/cudatool.py info <new>/run/libcuda.so.<new> +tools/cudatool.py delta <old>/run/libcuda.so.<old> <new>/run/libcuda.so.<new> > <new>/run/delta.txt +tools/cudatool.py fdematch <old>/run/libcuda.so.<old> <new>/run/libcuda.so.<new> \ + --names <old>/annotations/names.tsv +tools/cudatool.py wrappers <new>/run/libcuda.so.<new> + +cd ../../debug/risc-v-gsp-rm-vm +./greeneye.sh symbols export \ + --names ../../userspace/cuda/<new>/annotations/names.tsv \ + --index ../../userspace/cuda/<new>/run/analysis/decompiled/decompiled_index.tsv \ + --bias 0x100000 -o symbols-<new>.json +./greeneye.sh symbols resolve --db symbols-<new>.json 0x<transferred-addr> + +cd ../.. +tools/cuda-ab/ab.sh all # host oracle + guest run + compare +``` + +### S5 — New GPU model or architecture (RTX 5090 / GB10-class, aarch64 hosts) + +| Attribute | Detail | +|---|---| +| **Symptoms** | Probe first-touch addresses differ from the reference device; a firmware family/partition-manifest mismatch; ISA string differs (RV64GC vs RV64IMAC); aarch64 host reports `qemu-system-riscv64 not found`; PMU/SOE provisioning assumptions break. | +| **Detection** | `file`/`readelf` image shape; pairing check (family, FMC partition manifest, version string); probe first-touch diff; host tooling check (`./greeneye.sh info`); `rmtool` host-oracle inventory. | +| **Evidence** | Device profile fields (PCI IDs, BARs, FB geometry, engine inventory, interrupt topology, ROM quirks); pairing notes; reference vs candidate first-touch maps; portability matrix report; physical validation record before any support claim. | +| **Decision criteria** | Changed first-touch on profile registers → profile data, not logic. New firmware family/partition model → new family inputs, not a re-badged image. RV64GC or new CSRs → ISA/decoder work (S6). No physical validation → status stays Target/Experimental, never Supported. | +| **Rollback** | Keep the reference device/firmware directories untouched; revert the candidate profile; a support claim is only removed with a matrix change note. | +| **Record in** | [`06-portability/hardware-software-matrix.md`](06-portability/hardware-software-matrix.md) and the device profile notes; the evaluation ladder result for the combination. | + +**Response order** + +1. Collect the device profile; verify the firmware pairing before debugging + anything (a mismatched pairing looks exactly like an emulation bug). +2. Confirm ISA/image shape (`file`, `readelf`); note FMC/manifest and boot + model; diff PMU/SOE images. +3. Probe the new image and the reference with identical budgets; diff the + first-touch maps. +4. Run the portability pipeline across reference and candidate. +5. Update profile data first; only add code when an interface genuinely + differs. +6. Validate on the physical device under supervision before claiming support. + +**Commands** + +```bash +file <device-fwimage> && readelf -h <device-elf> && readelf -l <device-elf> +cd debug/risc-v-gsp-rm-vm +./greeneye.sh info +./greeneye.sh probe <device-fwimage> --symbols symbols-<device>.json --record +python3 harness/portability_matrix.py \ + --target <reference-fwimage> --target <device-fwimage> \ + --out dumps/matrix-<device> +./rmharness/harness.sh --session dumps/real-<device> --name before \ + -- rmtool engine list --format=json +# aarch64 host without qemu-system-riscv64: +# install qemu-system-misc, or point GREENEYE_QEMU at the binary +``` + +### S6 — New instructions or ISA changes in NV-RISCV / falcon / GSP + +| Attribute | Detail | +|---|---| +| **Symptoms** | Probe faults with an illegal/unimplemented instruction; the native backend diverges from QEMU on the same image; compressed/float extensions or new CSRs appear; a re-hosted build built with the old `-march` fails or misbehaves. | +| **Detection** | ISA string from `file`/`readelf`; probe fault cause/`desc`; QEMU-vs-native differential (`sim_diff.py`); decoder unit tests. | +| **Evidence** | Image shape (ISA, segments, entry); fault report with cause, `epc`/`epc_firmware` and access address; `PARITY: OK` transcript or the exact divergence; a fidelity-gap note when the native decoder cannot execute the new instruction. | +| **Decision criteria** | RV64IMAC on QEMU TCG → L0 high, proceed. RV64GC → QEMU supports it; update rehost `-march` and re-run parity. Native decoder gap → record as a fidelity gap; do not claim native coverage. Illegal instruction in QEMU itself → confirm ISA first, then escalate as a tool bug. | +| **Rollback** | Revert rehost `-march`/build flags; keep the previous probe/session as the reference. | +| **Record in** | The tool's accuracy/fidelity-gap list (`debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md` §8) and the image's version notes. | + +**Response order** + +1. Confirm the ISA and image shape; identify the exact faulting instruction and + address. +2. Reproduce under QEMU (the reference execution engine) with a bounded probe. +3. Run the QEMU-vs-native differential; if it diverges, capture both sessions. +4. If the instruction is unsupported natively, record a fidelity gap with the + session as evidence; keep QEMU as the execution engine for that image. +5. Update rehost build flags and re-run parity and the tool test suite. + +**Commands** + +```bash +cd debug/risc-v-gsp-rm-vm +./greeneye.sh probe <fwimage> --instructions <budget> --record +python3 examples/scripts/sim_diff.py # requires QEMU; prints PARITY: OK on agreement +make test +``` + +### S7 — Significant VBIOS/ROM changes + +| Attribute | Detail | +|---|---| +| **Symptoms** | The guest driver reads different device identity/VBIOS fields; probe-time register values move; ROM/VBIOS quirks change (laptop/Surface variants); a previously green probe hangs on a different register. | +| **Detection** | Guest `lspci -vvv -d 10de:` and init-trace diff; snapshot diff of the identity/static-config region; `rmtool` read-only `BUS_GET_PCI_INFO` oracle; ROM hash change. | +| **Evidence** | ROM hash and identity fields; before/after snapshot with a schema-attributed diff; pairing notes; the exact guest-observed values and the trace that consumed them. | +| **Decision criteria** | Changed fields consumed by the driver → profile/override work, physically confirmed. Fields not consumed → record and monitor. Anything that would require impersonating a licensed product → forbidden; present a coherent device instead. | +| **Rollback** | Revert the profile/override; previous ROM/profile stays archived with its hash. | +| **Record in** | The device profile and the matrix entry; the failure taxonomy if the symptom was misleading. | + +**Response order** + +1. Hash the ROM/VBIOS and collect the identity fields. +2. Capture a before/after snapshot of the firmware-side static-config region + and diff it field by field. +3. Compare with the host oracle (read-only) and the guest's observed values. +4. Change only profile data that a consumer demonstrably uses; confirm on + hardware. +5. Record unexplained fields as open, not as noise. + +**Commands** + +```bash +# guest +sudo lspci -vvv -d 10de: +# host oracle (read-only; verified queries from docs/17) +# build once from the repo root: make -C tools/rmtool +tools/rmtool/rmtool gpu info -d 0 +tools/rmtool/rmtool engine list + +cd debug/risc-v-gsp-rm-vm +# snapshot the identity/static-config region before and after +# (schema decode from the bundled/demo schema; real regions are version-keyed) +./greeneye.sh compare dumps/<old-run> dumps/<new-run> --symbols +``` + +### S8 — Encrypted/protected memory or protected instructions (measurement plan; non-claims) + +| Attribute | Detail | +|---|---| +| **Symptoms** | The image consumes identity material that is provisioned encrypted (the PMU-delivered encrypted device identity in the real 0x59 campaign); protected-memory or protected-instruction paths are reached; a lenient run returns zeros where a real device returns provisioned values; boot halts at a secret-dependent step. | +| **Detection** | First-touch/device-map analysis; lenient-MMIO `mmio_log` and `mmio_unknown`; boot-chain shim assumptions in session metadata; probe fault reports identifying the consuming PC. | +| **Evidence** | First-touch map entry per protected address; `mmio_log` with PC and access; the exact access that consumes protected state; a written measurement plan and an explicit non-claims list. | +| **Decision criteria** | Secret-dependent branch without provisioned input → blocked; record as a fidelity gap and a device-model requirement, not a bug. Zero-value success under lenient MMIO → hypothesis only. Any request to extract, defeat or prove protected state → refuse (P5/P7). | +| **Rollback** | No product changes are made in this scenario; the only rollback is reverting experiment notes and keeping the map as data. | +| **Record in** | The fidelity-gap list, the device-map file (`greeneye-device-map/1`), and the H5 cross-controller plan; never in a security-claims section. | + +**Measurement plan (what to measure, and what is not claimed)** + +| Measure | Method | Non-claim | +|---|---|---| +| Which address/instruction consumes protected state | bounded probe + device map + `mmio_log` | not a statement about the real value | +| Whether execution continues when the value is absent/zero | lenient native run with `unknown_mmio="lenient"` | zeros are a hypothesis, not read-back semantics | +| Which subsystem provisions the value | cross-controller inventory (PMU/SEC2/ACR/SOE) and boot-order notes | not a model of the secure state machine | +| Whether the path is security-relevant | static map + recorded access ordering | not a side-channel, confidentiality or attack claim | + +**Response order** + +1. Classify: protected read, protected instruction, or a secret-dependent + branch. Do not attempt extraction; do not patch anything. +2. Capture the first-touch map and the lenient `mmio_log` in a recorded + session. +3. Write the measurement plan and the non-claims into the session note. +4. Escalate the missing provisioning as a device-model requirement (H2/H5), + not as an emulation defect. +5. Stop. Physical work on protected paths requires the supervisor and a + separate approved plan. + +**Commands** + +```bash +cd debug/risc-v-gsp-rm-vm +python3 -m greeneye.device_map demo --instructions 200000 +python3 -m greeneye.device_map harvest dumps/<probe-session> --out dumps/<ver>-device-map.json +python3 -m greeneye.device_map show dumps/<ver>-device-map.json +# lenient native continuation (programmatic): +# ge.launch(kernel=..., backend="native", +# device_map="dumps/<ver>-device-map.json", +# unknown_mmio="lenient") +# session.target.mmio_log / session.target.mmio_unknown +``` + +### S9 — New microcontrollers and new inter-/intra-microcontroller APIs and DMA lines + +| Attribute | Detail | +|---|---| +| **Symptoms** | The image stalls waiting on PMU/SEC2/ACR/SOE; falcon staging DMA does not land (the real failure mode: the SEC2 boot-loader block is not visible, the falcon does not halt); WPR2/security state differs from the successful path; new mailbox functions or DMA descriptors appear. | +| **Detection** | Probe first-touch; boot-chain shim assumptions recorded in session metadata; compare against the last successful boot path; device-map diffs; cross-controller event absence in sessions. | +| **Evidence** | First-touch addresses per controller window; mailbox/DMA access map; boot-order notes; shim assumptions in the session; the successful-path comparison; H2.2/H5 scope note. | +| **Decision criteria** | GSP-dependent stall → PMU/SEC2 stub work first (H2.2 entry). New mailbox/API surface → inventory it before modelling. Security-chain semantics → shim/documented assumption, never a claimed secure model. Unmodelled DMA line → device-map entry plus model work, not a product change. | +| **Rollback** | Revert any shim assumption change; the previous session/boot note stays as the reference. | +| **Record in** | The cross-controller inventory and boot-order notes; the device map; the H5/H2.2 roadmap scope. | + +**Response order** + +1. Inventory the controllers and DMA lines the image touches first + (PMU, SEC2, ACR/SOE, booter). +2. Reproduce the stall with a bounded probe and record shim assumptions in the + session. +3. Compare against the last successful path (e.g. WPR2 state, SEC2 staging + visibility) and name the missing exchange. +4. Model the minimum stub that lets the image proceed; keep it documented + data, not a silent special case. +5. Gate real security-chain semantics behind H2.2/H5; escalate any conclusion + that would change the product. + +**Commands** + +```bash +cd debug/risc-v-gsp-rm-vm +./greeneye.sh probe <fwimage> --record +python3 -m greeneye.device_map harvest dumps/<probe-session> --out dumps/<ver>-device-map.json +python3 -m greeneye.device_map merge dumps/<ref>-device-map.json dumps/<ver>-device-map.json \ + --out dumps/merged-device-map.json +./greeneye.sh compare dumps/<last-good-run> dumps/<stall-run> --symbols +``` + +### S10 — Multi-version matrix operations and CI (RM 610 requires explicit human approval) + +| Attribute | Detail | +|---|---| +| **Symptoms** | A new matrix column regresses an existing combination; CI drifts from the local run; someone proposes a 610-labelled input or command; the release bundle is incomplete. | +| **Detection** | `version_diff`/`portability_matrix` deltas; `ci_release --dry-run` plan inspection; the RM 610 gate refusal (`exit 3`); test transcripts. | +| **Evidence** | `version-diff.json`, `portability-report.json`, `release-report.json`, `bundle-manifest.json`, seeded `findings.jsonl`, test transcripts, approval record for 610. | +| **Decision criteria** | Dry-run clean before a real run. A step that raises is `error`, not skipped. Self-comparison must be `all_zero`; seeded replays byte-identical. **610: no acquisition, analysis, emulation edit, physical work or support claim before a named human approves in writing; an open-ended approval is invalid.** | +| **Rollback** | Extend the matrix, never replace a column; revert the candidate commit and keep the previous anchor/tag/driver; a 610 attempt parks without touching 570.144. | +| **Record in** | The port log/approval thread (610), the release bundle directory, and the matrix with its test date and rung. | + +**Response order** + +1. Run the release rehearsal on current-version labels first; inspect the + dry-run plan. +2. Run `version_diff` and `portability_matrix` with fixed budgets; require the + self-comparison to be zero. +3. Run the mutation smoke with a pinned seed; verify replay. +4. Assemble and hash the bundle; attach it to the PR. +5. For 610: stop at the gate; obtain the approval message; only then acquire + inputs and repeat the pipeline with 610 labels. + +**Commands** + +```bash +cd debug/risc-v-gsp-rm-vm + +# rehearsal first: dry-run never opens a target and creates no output dir +python3 harness/ci_release.py --old <old> --new <new> --out dumps/release-<ver> \ + --old-label <OLD> --new-label <NEW> --dry-run + +# real pipeline (dry-run clean) +python3 harness/ci_release.py --old <old> --new <new> --out dumps/release-<ver> \ + --old-label <OLD> --new-label <NEW> \ + --old-symbols symbols-<OLD>.json --new-symbols symbols-<NEW>.json \ + --mutation-kernel firmware/uproc_demo/build/uproc_demo.elf \ + --mutation-iterations 8 --mutation-seed 7 + +# RM 610 gate: refused by default; explicit approval token required +python3 harness/version_diff.py --old <old> --new <new> --out dumps/vdiff-610 \ + --old-label RM-610 --new-label 570.144 +# exit 3 unless: --approval-token RM610-APPROVED or GREENEYE_RM610_APPROVED=1 +``` + +## 6. Evidence-bundle requirements + +A bundle is the minimum attachable proof for a version/device/architecture +triage or a product change. It combines the docs/07 evidence checklist, the +docs/14 §6 adversarial pack and the docs/21 release tree. + +``` +<out>/ + bundle-manifest.json sha256 + bytes of every file below + release-report.json pipeline summary (steps, gate, labels) + version-diff/ version-diff.json|.md, symbols/, sessions/, compare/ + portability-matrix/ portability-report.json|.md, symbols/, sessions/, compare/ + mutation-smoke/ findings.jsonl, summary.md, sessions/ +``` + +Checklist — every box must be true or explicitly waived with a reason: + +* [ ] firmware identity: SHA-256, version string, family and pairing notes; +* [ ] input provenance: channel, date, and the exact file names analysed; +* [ ] symbol DB(s): hash, source (ELF/Ghidra/JSON), `bias` and provenance; +* [ ] baseline and candidate session directories (or their SHA-256 list); +* [ ] exact stimulus (script/spec plus mutation values) and budgets; +* [ ] `compare` report and relevant `analyze` reports; +* [ ] M1–M6 table with the reference-capture method and L2/L3 limits stated; +* [ ] `greeneye_hits.tsv` and any Ghidra annotations produced; +* [ ] `bundle-manifest.json` written last, covering the final reports; +* [ ] unexplained-delta list (empty is a claim, not a default); +* [ ] decision and product action (or explicit "no change") recorded; +* [ ] for 610: approval ID, approver, date and scope limits. + +Fidelity statement template (copy into every report): + +```text +Fidelity: L0 (instruction semantics) and L1 (memory) exercised; L2 (timing, +concurrency) NOT modelled; L3 (device environment) absent for <registers>. +Reference capture: <hardware / vfio-mdev / host oracle / none>. +Physical reproduction required before product change: <yes/no + why>. +``` + +## 7. Fault and escalation tree + +```text +Failure / new input observed +| ++- Is the input hashed, versioned and paired? NO -> STOP: acquire/hash/pair first (S1) +| ++- Did it come from RM 610 or a 610-labelled path? YES -> STOP: hard human gate (S10) +| ++- Is a physical GPU / driver bind / mdev involved? +| YES -> STOP: supervised physical cycle + reboot approval (P7); never unattended +| ++- Does the question need timing, interrupts, protected state or real device +| read-back semantics (L2/L3)? +| YES -> simulator evidence is a hypothesis only (P3/P4/P5); record the +| fidelity gap; product change requires physical reproduction +| ++- Does it reproduce under a fixed stimulus in green-eye? +| NO -> widen the stimulus / capture the real trace before theorising; +| absence of a hit is not proof of dead code +| YES -> classify with the S1 decision table (code changed vs behaviour moved) +| ++- Would a fix change emulation behaviour? +| YES -> physical/vfio-mdev reproduction + evidence bundle before landing +| NO -> docs/profile/test-only change with the bundle attached +| ++- Does any vendor component need modification? + YES -> forbidden (P2): redesign to emulate the interface instead +``` + +Stop conditions (do not proceed, do not "temporarily" work around): + +1. Unverified pairing or unhashed input. +2. RM 610 without written approval. +3. A physical cycle without the supervisor and fail-safe plan. +4. A request to patch a vendor component or a guest. +5. A security/confidentiality claim based on simulator state. +6. A product change with no physical reproduction and no bundle. + +## 8. When NOT to use the tooling + +| Situation | Why not | Use instead | +|---|---|---| +| You need silicon truth (timing, interrupts, real register read-back) | L2/L3 are absent by design | supervised physical cycle + reference traces | +| You are about to mutate or inject against hardware, a guest, a kernel module or an existing gdbstub | Tooling is simulator-only; hardware writes are prohibited | bounded QEMU/native harness with recorded sessions | +| You need to defeat, extract or prove a security boundary | The simulator is not a confidentiality boundary | stop; escalate as a design question | +| You need a product fix "now" without evidence | Violates P3/P4 | evidence bundle first; the anchor and matrix can wait | +| The input has not been hashed/versioned/paired | Every later claim is contaminated | S1 steps 1–3 | +| RM 610 has no written approval | Hard gate | request approval with the readiness checklist | +| The physical GPU is held by another job | Shared-oracle safety and fail-safe discipline | host-side read-only oracle (`rmtool gpu list`, `harness.sh`) | +| You need type recovery or decompilation | green-eye is dynamic ground truth, not a decompiler | Ghidra corpus and symbol tooling | +| You need guest-driver behaviour | green-eye observes firmware, not the guest | `tools/guest-diag/`, ioctl tracer | +| A one-line question can be answered by `analyze` on an existing session | Do not produce new runs for no reason | reuse recorded sessions and `compare` | + +## 9. One-page quick-reference decision tree + +| Symptom (first sign) | First command | Decision | Scenario | +|---|---|---|---| +| New image/version appears | `./greeneye.sh symbols export ...` + `probe --record` | hash+pair first, then version diff | S1 | +| OpenRM build breaks, `Unknown symbol` | `./scripts/build_nvidia_openrm.sh`; `dmesg` | glue-only vs interface change | S2 | +| Control/ioctl inventory shifts | `python3 harness/control_discovery.py` | moved vs new logic | S3 | +| `libcuda` bump / context failure | `tools/cudatool.py delta` + `fdematch` | patch / feature / reshuffle | S4 | +| New device/architecture | `file`/`readelf`, `probe` | profile vs logic; physical validation for support | S5 | +| Illegal/unknown instruction | `./greeneye.sh probe --instructions N` | ISA/decoder; fidelity gap if native-only | S6 | +| Identity/ROM fields differ | guest `lspci -vvv`, snapshot diff | consumed field → profile; else record | S7 | +| Protected/encrypted state blocks | device map + lenient `mmio_log` | record fidelity gap; no extraction; no claims | S8 | +| Early stall on PMU/SEC2/ACR/SOE | `probe --record` + device map | stub the missing exchange; gate security semantics | S9 | +| Matrix/CI drift; 610 mentioned | `ci_release --dry-run` | dry-run clean; 610 hard gate | S10 | + +Golden rule: **freeze, hash, pair, reproduce, classify, evidence, then act** — +and never let the last step touch anything we do not own. + +## 10. Related documentation + +Green-eye suite (experimental, in development): + +* [green-eye user guide](05-operations/04-green-eye-user-guide.md) — what it + is, quickstart, tool chooser and troubleshooting; +* [GSP in a RISC-V VM](05-operations/03-risc-v-gsp-rm-vm.md) — project-level + purpose, status and limits; +* [`debug/risc-v-gsp-rm-vm/docs/00-OVERVIEW.md`](../debug/risc-v-gsp-rm-vm/docs/00-OVERVIEW.md) + — tool status and doc map; +* [`debug/risc-v-gsp-rm-vm/GREEN-EYE-ROADMAP.md`](../debug/risc-v-gsp-rm-vm/GREEN-EYE-ROADMAP.md) + — H1–H7 program; +* [`debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md`](../debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md) + — fidelity levels and the M1–M6 contract; +* [`debug/risc-v-gsp-rm-vm/docs/07-PORTABILITY-PLAYBOOKS.md`](../debug/risc-v-gsp-rm-vm/docs/07-PORTABILITY-PLAYBOOKS.md) + — P1–P4 porting playbooks; +* [`debug/risc-v-gsp-rm-vm/docs/14-ADVERSARIAL-INTEROP-GUIDE.md`](../debug/risc-v-gsp-rm-vm/docs/14-ADVERSARIAL-INTEROP-GUIDE.md) + — adversarial navigation and the evidence pack; +* [`debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md`](../debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md) + — release triage and bundles; +* [`debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md`](../debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md) + — the 610 gate and approval format; +* [`debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md`](../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md) + — tool audit, symptom-to-fix table and provenance rule; +* [`debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md`](../debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md) + — sweep/device-port scaffolds (unadmitted pilots, dry-run default); +* [`debug/risc-v-gsp-rm-vm/docs/29-PACKAGING-AND-TRIALING.md`](../debug/risc-v-gsp-rm-vm/docs/29-PACKAGING-AND-TRIALING.md) + — trial simulation packages (unadmitted pilot). + +Project documentation: + +* [Debugging and maintenance workflows](05-operations/02-debugging-and-maintenance-workflows.md) + — captures, failure taxonomy, version discipline; +* [Performance, correctness, security](05-operations/01-performance-correctness-security.md) + — the evaluation ladder; +* [Hardware and software matrix](06-portability/hardware-software-matrix.md) + — support definitions and the portability pipeline; +* [Practical challenges and adversarial interoperability](04-implementation/02-practical-challenges-and-adversarial-interop.md) + — the no-patch rule; +* [Developer onboarding and standards](07-developers/onboarding-and-standards.md) + — coding rules and the landing checklist; +* [Documentation front door](README.md) — the full reading order; +* [Engineering operations](eng-ops.md) — tool admission, workflows and + retirement (the cross-cutting engineering-operations document); +* CUDA playbook: + [`userspace/cuda/docs/07-version-upgrade-playbook.md`](../userspace/cuda/docs/07-version-upgrade-playbook.md) + and [`userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md). + +<!-- change (2026-10-02): fix the S4 fdematch invocation (two libraries plus +--names, per tools/cudatool.py), add a session/disk-growth bound to §3.2, and +link docs 28/29 from §10; scope: RUNBOOK §3.2, §5 S4, §10 only; benefit: the +copied triage command runs as written and the retention rule is explicit. --> diff --git a/docs/eng-ops.md b/docs/eng-ops.md new file mode 100644 index 0000000..117bd0c --- /dev/null +++ b/docs/eng-ops.md @@ -0,0 +1,850 @@ +# Engineering Operations: tooling admission, workflows and retirement + +> How Open-IOV decides which engineering tools it uses, extends and retires, +> and how it measures whether a tool pays for its maintenance. Maintainers +> decide and record; agents execute bounded loops and produce evidence. +> Subsystem manuals are linked, not duplicated. + +## 1. Purpose and scope + +**Who uses it.** Maintainers decide, record and review; agents (MCP, CI, cron) +execute bounded loops and produce sessions, reports and bundles. The same rule +binds both: evidence or it did not happen ([`onboarding-and-standards.md`](07-developers/onboarding-and-standards.md) §5, [`15-AGENTIC-GSP-IDE.md`](../debug/risc-v-gsp-rm-vm/docs/15-AGENTIC-GSP-IDE.md) §5). + +| Governed here | Owned elsewhere (authoritative) | +|---|---| +| Admission, extension, redesign, trialing and retirement of tools | Architecture ([`03-architecture/01-system-overview.md`](03-architecture/01-system-overview.md)); tool manuals: CUDA ([`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md)), green-eye ([`26-ARCHITECTURE.md`](../debug/risc-v-gsp-rm-vm/docs/26-ARCHITECTURE.md), [user guide](../debug/risc-v-gsp-rm-vm/docs/02-USAGE.md), [scripting](../debug/risc-v-gsp-rm-vm/docs/03-SCRIPTING.md)), rmharness ([`README`](../debug/risc-v-gsp-rm-vm/rmharness/README.md)) | +| Which tool applies in which loop; automation and packaging | Accuracy M1–M6 ([`06-ACCURACY.md`](../debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md)); performance method ([`05-operations/01`](05-operations/01-performance-correctness-security.md) §3); version discipline ([`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §4) | +| Evidence bundles, version-span gates, frontier admission | Per-tool docs and the periodic [`27-TOOL-AUDIT.md`](../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md) | +| Live physical-backend engineering operations: supervised-cycle mechanics, parameter gates, debugfs controls and the hazard register (**Part II** below) | Physical-backend architecture ([`03-architecture/08`](03-architecture/08-physical-gpu-integration.md), [`03-architecture/11`](03-architecture/11-physical-backend-guest-wiring.md)); defensive/incident scenarios ([`RUNBOOK.md`](RUNBOOK.md)); current frontier ([`STATE-CURRENT.md`](STATE-CURRENT.md)) | + +**Explicitly out of scope.** (1) Security claims: the simulator and its device +maps are study environments, never a confidentiality or isolation boundary +([`06-ACCURACY.md`](../debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md) §6, +[`GREEN-EYE-ROADMAP.md`](../debug/risc-v-gsp-rm-vm/GREEN-EYE-ROADMAP.md) V4). +(2) Timing claims: L2 is not modelled; order-of-events only +([`06-ACCURACY.md`](../debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md) §2). +(3) Shipping injected code: development-only, never a firmware patch, never in +a guest ([`12-INJECTION-SDK.md`](../debug/risc-v-gsp-rm-vm/docs/12-INJECTION-SDK.md) §1). +(4) Support and licensing: tested-and-documented only +([`hardware-software-matrix.md`](06-portability/hardware-software-matrix.md) §1); +legal rules in [`legal-and-licensing.md`](09-reference/legal-and-licensing.md). + +## 2. The core rule: no insertion without a measurable outcome + +A tool, or an insertion of an existing tool into a workflow, is admitted only +with all five fields below. Missing any field rejects the proposal; it is not +deferred until the metric is invented later. + +| Field | Meaning | Example | +|---|---|---| +| **Scenario** | exact question, target, version and loop where the tool is inserted | "classify one new GSP image's dispatch-table delta before the emulation is touched (portability loop)" | +| **Measurable outcome** | result with a unit and an observer | "image drop → `version-diff.md` delta table; sessions per finding" | +| **Baseline** | the replaced method and its measured/recorded cost | "static corpus diff + one supervised boot per hypothesis; baseline in [`docs/25`](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md) §3" | +| **Evaluation threshold** | the number that decides keep/extend/retire, with sample size and environment | "N ≥ 3; median TTFA ≤ 25 % of baseline; self-diff zero; M-table improves or stays equal" | +| **Failure signal** | observable "not useful" outcome during the pilot | "unexplained `compare` deltas after two triage passes, or the tool needs hardware to run" | + +Rules: one scenario per record; the threshold is recorded **before** the first +run and moving it needs a new record; every admission starts as `admit-pilot` +with an expiry of one quarter or five sessions; the baseline must be observable +(measure it first if unrecorded); a missed threshold triggers §4.4/§4.5. + +```text +TOOL ADMISSION RECORD id: TOOL-<yyyy>-<nn> +tool: <name or path; existing | extension | new> +loop: static-RE | dynamic-RE | porting | performance | correctness | CI | guest-compat +owner/decider: <maintainer>; executor: <human|agent|cron> +scenario: <one sentence: question, target, version, trigger> +measurable: <metric + unit + observer> +baseline: <replaced method; cost source and date> +threshold: keep if <...>; reject if <...>; N=<runs>; environment=<...> +failure signal: <observable not-useful outcome> +evidence: <session dir / report / manifest + sha256> +decision: admit-pilot | admit | extend | redesign | retire | do-not-build +review-by: <date or N sessions> trials-dir: <path> +``` + +The record is stored with the evidence (session directory, bundle or PR +description); decisions are recorded by a human maintainer and mirrored in the +register (§4.5) and revision notes (§9.3). At expiry: met → admit; not met → +redesign or retire; no signal → retire. No unused pilot survives. + +## 3. Value axes: what tooling is for, and how it is measured + +Each admission names exactly one axis. Every axis reduces to moving evidence +through a §3.3 loop; if it cannot be expressed as a time or a count, it is not +admissible. + +| # | Axis | Concrete example (real tooling) | Metric (recorded) | Loop | +|---|---|---|---|---| +| 1 | Faster reverse engineering | `version_diff.py` + `compare` on two GSP images; Ghidra corpus diff | firmware drop → classified delta (TTFA); symbols resolved vs Ghidra-only | RE | +| 2 | Deeper / more correct analysis | watchpoints + TCG/native trace; capability inference ([`docs/14`](../debug/risc-v-gsp-rm-vm/docs/14-ADVERSARIAL-INTEROP-GUIDE.md) §4.2) | attributed bytes/stores; M6 agreement; unexplained-delta count | RE | +| 3 | Code-artifact porting time | `cudatool.py fdematch`, `diff_corpus.py`, `symbols export` | transfer rate per class; wall time old→new symbol DB; wrapper re-resolution errors | portability | +| 4 | Bug-analysis time | `reach.py`, `run_until`/`call`, `ioctl_trace.c`; taxonomy in [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §5 | crash/question → first evidence-backed answer (TTFA); supervised cycles avoided | adversarial | +| 5 | Development velocity on the simulated GSP | injection SDK; `greeneye.sh inject`; value demo s06 | seconds from service edit to direct+mailbox agreement; sessions per prototype | RE | +| 6 | Correctness validation of virtualization | M1–M6 table, `compare`, golden sessions | M-table per change; rung ladder ([`05-operations/01`](05-operations/01-performance-correctness-security.md) §1) | correctness | +| 7 | Performance analysis and improvement | A/B method, `tools/benchmark_infrastructure.sh`, simulator overhead ([`docs/25`](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md) §6) | B/A % per metric; p50/p99 RPC; trace overhead; before/after same machine | performance | +| 8 | Userspace/API compatibility friction | capability negotiation ([`../userspace/cuda/docs/04-virtualization-and-emulation.md`](../userspace/cuda/docs/04-virtualization-and-emulation.md) §1), symbol DB, ioctl diffs | controls with coherent replies vs observed; TTFA for a new capability bit; peer scenarios (RDMA guest, DPU/NIC/NVMe page sharing) counted only once a peer model exists | guest-compat | +| 9 | Hidden / undocumented / unauthorized code, side effects, backdoors | `control_discovery.py`, mutation harness, decoy handling ([`docs/14`](../debug/risc-v-gsp-rm-vm/docs/14-ADVERSARIAL-INTEROP-GUIDE.md) §4.4) | unknown IDs and injected-slot aliases found; capability-model deltas; corpus entries with sessions | adversarial | +| 10 | Architectural understanding and SBOM/doc improvement | symbol DB, `export_hits`, firmware notes ([`docs/09`](../debug/risc-v-gsp-rm-vm/docs/09-GSP-FIRMWARE-NOTES.md)), [`SBOM.md`](../SBOM.md) | doc/SBOM rows updated per finding; onboarding time; subsystem coverage | RE | +| 11 | Security fixes / hardening time | capability inference → fail-closed rules ([`04-implementation/02`](04-implementation/02-practical-challenges-and-adversarial-interop.md) §4), mutation corpus | error paths covered; finding → test time; physical confirmation before product change | adversarial | +| 12 | New GSP support time | `probe` + `version_diff` + `portability_matrix` + `ci_release` bundle | hours from image to triage table; M-table completeness; regression lead time | portability | +| 13 | New hardware-device support time | device profile capture, `rmharness`, matrix, [`docs/22`](../debug/risc-v-gsp-rm-vm/docs/22-PORTING-CUDA-AND-DEVICES.md) §3, [device sequence](06-portability/03-device-validation-and-porting-sequence.md) | profile fields captured; engine inventory validated; first-touch deltas explained; tested-at-rung date | portability | +| 14 | CUDA / runtime / VM-environment fixes | `userspace/cuda/docs/07`, `cuda_drv_test.c`, `tools/cuda-ab/ab.sh`, guest diagnostics | crash → named function time; `RESULT`/`STATUS` match; rung movement | guest-compat | + +Axis 8 note: GPU-direct RDMA availability is negotiated via the stock guest's +vGPU capability block; page-table sharing with simulated DPU/NIC/NVMe peers is +future, so its metric starts only when a peer model exists (§8). + +**Performance axis (axis 7).** Profiling order is the project's existing one +([`05-operations/01`](05-operations/01-performance-correctness-security.md) §3.6): +RPC dispatch → doorbell cadence → eventfd/MSI-X delivery → BAR0 shadows → +memory-model gap. Method: (1) define the metric and A/B pair before code +changes — same binary, inputs, governor, clocks and boot where possible, and +report B/A %, never absolutes; (2) run A via `tools/benchmark_infrastructure.sh` +and B via `scripts/vgpu_vm.py exec`, logging versions and +`nvidia-smi -q -d CLOCK`; (3) attribute with host counters +(`gsp_emu_get_stats()`, `trace_printk`, `/sys/kernel/debug/tracing`) and the +measured simulator overhead ([`docs/16`](../debug/risc-v-gsp-rm-vm/docs/16-CUSTOM-SIMULATOR.md) §1.1a, [`docs/25`](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md) §6); +(4) keep the numbers with the change and re-run correctness; (5) reject single +runs, cross-host comparisons, bare-metal-only data and simulator timing claims. + +**OODA reduction.** Every axis moves evidence through one of three loops. Each +loop's compression is the project's program-level metric +([`GREEN-EYE-ROADMAP.md`](../debug/risc-v-gsp-rm-vm/GREEN-EYE-ROADMAP.md) success +criteria), measured in [`docs/25`](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md) §7: + +| Loop | Observe → Orient → Decide → Act | Compression metric | +|---|---|---| +| RE-constrained (static/dynamic interlock) | probe/hooks/watch/trace → `analyze`/snapshots → `compare`/triage → reimplementation | question → reviewable answer; sessions per finding | +| Adversarial-interop (opaque/decoy firmware) | discovery/mutation/replay → classification ([`docs/14`](../debug/risc-v-gsp-rm-vm/docs/14-ADVERSARIAL-INTEROP-GUIDE.md) §4.1) → fail-closed or no change → emulation change + physical confirm | unknown-ID → classified route; error paths covered | +| Portability (GSP/device/architecture churn) | probe first-touch/symbol diff → pairing check/delta class → anchor decision → versioned dirs + handlers | image drop → triage table (hours) | + +## 4. Use, extend, build, redesign, retire + +```mermaid +flowchart TD + P["Friction + scenario + measurable outcome"] --> Q1{"Existing tool covers the scenario?"} + Q1 -->|"yes"| USE["Admit the insertion; pilot + metric record"] + Q1 -->|"no"| Q2{"A close tool has a public extension point?"} + Q2 -->|"yes"| EXT["Extend in place: plugin, harness Surface, probe spec, SDK"] + Q2 -->|"no"| Q3{"Repeated scenario and no cheaper alternative?"} + Q3 -->|"no"| NB["Do not build: record rejection and the trigger to revisit"] + Q3 -->|"yes"| NEW["Admit a new tool as a bounded pilot"] + USE & EXT & NEW --> REV{"Threshold met after the pilot?"} + REV -->|"yes"| KEEP["Admit; add a second metric over time"] + REV -->|"no"| Q4{"Failure signal points at the design, not the idea?"} + Q4 -->|"yes"| RD["Redesign with a migration note; re-pilot"] + Q4 -->|"no"| PRUNE["Retire / prune; keep evidence read-only"] +``` + +| Decision | Minimum evidence | Decider | Record | +|---|---|---|---| +| (a) Use existing | scenario + measured baseline + one recorded trial session | maintainer; agent may run the trial | admission record + register row; link the session | +| (b) Extend existing | ≥ 3 uses or ≥ 2 users; public extension point (plugin, `Surface`, probe spec, SDK); cost ≤ new tool | maintainer | record names the extension point and keeps the old tests green | +| (c) Build new | record complete; bounded pilot; golden/replay evidence; stdlib/local-only unless justified | maintainer | record + register + revision note; tests added to the tool suite | +| (d) Redesign | failure signal twice, or a fidelity gap that changes conclusions; migration note (`greeneye-session/1` evolves additively) | maintainer | before/after metric; old artifacts stay readable | +| (e) Stop / prune | threshold missed, no evidence in the window, duplicated capability, or maintenance > value | maintainer at the audit | retire row; `27-TOOL-AUDIT.md` status; keep sessions/reports | +| (f) Do not build | any trigger below is true | maintainer | rejection note naming the trigger that would reopen it | + +**When NOT to build (workflow risk).** Reject if any holds; the absence of a +time-to-value process is the risk this document removes: (1) no scenario, unit +or baseline — the pilot cannot fail; (2) the tool would become a shipped +dependency, guest component or firmware patch +([`04-implementation/02`](04-implementation/02-practical-challenges-and-adversarial-interop.md) §5.1); +(3) it needs physical GPU, mdev or guest access for its primary value +([`docs/15`](../debug/risc-v-gsp-rm-vm/docs/15-AGENTIC-GSP-IDE.md) §5); +(4) it bypasses a security boundary or turns a study observation into a +security claim; (5) it duplicates a tool without a measured improvement; +(6) total cost is not plausibly below the replaced OODA cost; (7) its only +evidence is prose — evidence must be a session, report or bundle. + +**Redesign and retirement.** Redesign rather than patch when the failure +signal repeats, a fidelity gap invalidates an accepted conclusion, a public +interface must change incompatibly, or the tool drifted outside its admitted +loop. Retire when no artifact references the tool, the threshold was missed +twice, another tool covers the scenario better, or maintenance exceeds the +saving. Retirement is revocation of admission, not deletion: keep tool, tests +and artifacts read-only, mark the register row `retired` with date and reason, +and reconcile at the next [`27-TOOL-AUDIT.md`](../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md). + +**Register (how decisions are recorded).** The durable record is the register +row, the evidence directory/bundle it names and the revision note in §9.3; +the audit is the periodic reconciliation. + +| Admitted family | Loop | Value evidence (current) | Review / retire trigger | +|---|---|---|---| +| Ghidra corpora + scripts (`tools/ghidra`, `userspace/cuda/tools`) | RE, portability | docs 05/07; `diff_corpus`, name map | corpus diff no longer changes port class | +| green-eye core + session format | dynamic-RE | `docs/25` s01–s10; 155-test suite | format unused by any loop | +| Harness (`version_diff`, matrix, mutation, discovery, reach, `ci_release`) | portability, adversarial | docs 18/19/21; deterministic replay tests | a report is never attached to a change | +| Injection SDK + demo uproc | dynamic-RE | docs 11/12; s06 | prototypes stop using it | +| Ghidra bridges (in/out) | RE | docs 05/13 | static/dynamic join replaced | +| Device map (`greeneye-device-map/1`) | portability | docs 24; H2.1 first-touch | H2.2 model stops consuming maps | +| `rmharness` + `rmtool`/`rmtop`/`rmq` | correctness, guest-compat | docs 17; M1/M6 reference captures | reference capture unused/unavailable | +| MCP server + agent workflows | all loops | docs 15/20; release monitoring | write gating or audit trail removed | +| CI (`ci.yml`, `greeneye.yml`) | correctness | release-blocking tests | suite non-reproducible or skipped | + +## 5. Workflow map: where each tool applies + +### 5.1 Context of the development and reverse-engineering environment + +```mermaid +flowchart LR + subgraph STATIC["Static"] + GH["Ghidra CUDA corpus + GSP firmware"] + end + subgraph DYN["Dynamic (green-eye, experimental)"] + GE["Session + QEMU/native backends"] + HR["harness/*.py"] + end + subgraph HOST["Host and guest"] + MOD["open-iov-vfio.ko"] + VM["stock guest + guest-diag"] + RMH["rmtool/rmtop/rmq + rmharness"] + end + subgraph AUTO["Automation"] + CI["ci.yml / greeneye.yml"] + MCP["MCP server + agents/cron"] + end + EVID["Evidence: sessions, reports, bundles"] + GH -->|"names, index, symbol DB"| GE + GE -->|"export_hits, fault PCs"| GH + GE --> HR --> EVID + GE --> EVID + RMH --> EVID + VM -->|"stimulus and failures"| GE + MOD --> VM + EVID --> AUTO + AUTO -->|"bounded runs"| HR + EVID -->|"product-change gate"| MOD +``` + +### 5.2 Static reverse-engineering loop + +```mermaid +flowchart LR + BIN["libcuda / .fwimage"] --> IMP["Ghidra import + analysis"] + IMP --> IDX["names.tsv, decompiled_index.tsv, callgraph.tsv"] + IDX --> DIFF["corpus diff / fdematch / RPC dispatch diff"] + DIFF --> HYP["Hypothesis: functions, tables, layouts"] + HYP --> SYM["greeneye symbols export (bias recorded)"] + SYM --> DYN["dynamic loop"] + DYN -->|"hits, watch PCs, fault PCs"| HYP +``` + +### 5.3 Dynamic firmware loop (OODA) + +```mermaid +flowchart LR + Q["Question + target + fidelity statement"] --> OBS["Observe: probe, hooks, watch, trace"] + OBS --> ORI["Orient: analyze, symbol decode, snapshots"] + ORI --> DEC["Decide: compare, M1-M6, mutation"] + DEC --> ACT["Act: emulation change with evidence"] + ACT --> REG["Regression: same stimulus, same budgets"] + REG -->|"improved or equal"| PACK["Evidence pack + session hashes"] + REG -->|"regressed"| Q + PACK --> Q +``` + +### 5.4 Porting loop (GSP firmware, device, architecture) + +```mermaid +flowchart TB + ACQ["Acquire + hash; versioned dirs, never overwrite"] --> TRI["Triage: delta, fdematch, pairing check"] + TRI --> PRB["Probe first-touch old vs new"] + PRB --> RUN["Rehost/demo: identical stimulus + record"] + RUN --> CMP["compare + M1-M6 skeleton"] + CMP --> BUN["version_diff -> matrix -> mutation -> bundle"] + BUN --> REV{"Every delta explained?"} + REV -->|"no"| TRI + REV -->|"yes"| ANCHOR["Anchor decision + docs + matrix entry"] +``` + +### 5.5 Performance loop + +```mermaid +flowchart LR + DEF["Define metric + A/B pair"] --> BASE["A: bare metal baseline"] + DEF --> CAND["B: vGPU / candidate"] + BASE --> RATIO["B/A ratio, variance, N runs"] + CAND --> RATIO + RATIO --> PROF["Attribute: RPC, doorbell, eventfd, shadows"] + PROF --> KEEP["Keep numbers with the change"] + KEEP --> REG["Re-run correctness corpus + ladder"] +``` + +### 5.6 Correctness and CI loop + +```mermaid +flowchart LR + CH["Change"] --> HOST["Host suite: stack_manager.sh test"] + HOST --> GUEST["Guest ladder: probe, nvidia-smi, cdt"] + GUEST --> AB["cuda-ab: RESULT/STATUS must match"] + AB --> SESS["Session compare / golden data"] + SESS --> GATE{"Rung or metric regressed?"} + GATE -->|"no"| LAND["Land with evidence"] + GATE -->|"yes"| CH +``` + +### 5.7 Guest-compatibility loop + +```mermaid +flowchart LR + GD["guest-diag: ioctl trace, engines, pmap"] --> CTRL["Control/ioctl inventory"] + CTRL --> EMU["Emulated reply fidelity"] + EMU --> GD2["New driver/CUDA version trace"] + GD2 --> DIFF["Control diff old vs new"] + DIFF --> CTRL + CTRL --> CORPUS["CUDA corpus + symbol DB"] +``` + +### 5.8 Loop index and detailed docs + +| Loop | Entry points | Evidence | Gate | Detail | +|---|---|---|---|---| +| Static RE | `tools/ghidra/*`, `userspace/cuda/tools/*`, `analyzeHeadless` | corpus TSVs, name map, symbol DB | addresses re-derived per build | [`userspace/cuda/docs/07`](../userspace/cuda/docs/07-version-upgrade-playbook.md), green-eye docs 05/13 | +| Dynamic RE | `greeneye.sh` probe/record/analyze/compare/inject/ghidra, Python `Session` | `greeneye-session/1` + `summary.md` | A/B pair; L2/L3 statement | [`26-ARCHITECTURE.md`](../debug/risc-v-gsp-rm-vm/docs/26-ARCHITECTURE.md) §11, [`11`](../debug/risc-v-gsp-rm-vm/docs/11-GSP-RM-SERVICE-DEVELOPER-GUIDE.md), [`12`](../debug/risc-v-gsp-rm-vm/docs/12-INJECTION-SDK.md), [`13`](../debug/risc-v-gsp-rm-vm/docs/13-GHIDRA-CONTROL.md), [`14`](../debug/risc-v-gsp-rm-vm/docs/14-ADVERSARIAL-INTEROP-GUIDE.md) §4.6, [`06`](../debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md) §5 | +| Porting | `version_diff.py`, `portability_matrix.py`, `mutation_harness.py`, `ci_release.py` | reports + `bundle-manifest.json` | every delta explained | [`07`](../debug/risc-v-gsp-rm-vm/docs/07-PORTABILITY-PLAYBOOKS.md), [`22`](../debug/risc-v-gsp-rm-vm/docs/22-PORTING-CUDA-AND-DEVICES.md), [`21`](../debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md), runbook [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §4.1 | +| Performance | `benchmark_infrastructure.sh`, module counters, `vgpu_vm.py` | B/A table with environment | same-machine A/B; corpus green | [`05-operations/01`](05-operations/01-performance-correctness-security.md) §3 | +| Correctness | `stack_manager.sh test`, `make tests`, `cuda_drv_test.c`, `cuda-ab` | rung + trace excerpts | no rung regression | [`05-operations/01`](05-operations/01-performance-correctness-security.md) §1–2 | +| CI | `.github/workflows/ci.yml`, `.github/workflows/greeneye.yml` | workflow logs | push/PR path triggers | [`21`](../debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md) §5 | +| Guest-compat | `tools/guest-diag/*`, `cudatool.py`, `diff_corpus.py` | ioctl/control diff, corpus delta | stock guest only | [`READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md) | + +## 6. Automation and agent invocation + +```mermaid +flowchart LR + PRE["Preflight: greeneye.sh info; RM610 gate"] --> DRY["ci_release.py --dry-run (plan only)"] + DRY --> REAL["ci_release.py real run"] + REAL --> VD["version_diff"] + REAL --> PM["portability_matrix"] + REAL --> MU["mutation smoke"] + VD & PM & MU --> MAN["release-report.json + bundle-manifest.json"] + MAN --> REV["agent/human review"] + REV -->|"accepted"| REG["anchor/register/docs updated"] + REV -->|"rejected"| GAP["fidelity gap or fix, recorded"] +``` + +Run everything from `debug/risc-v-gsp-rm-vm/`; scripts are stdlib-only and +launch their own QEMU. Never re-use an existing gdbstub, a guest, a kernel +module or hardware. + +**Scripts that exist** (success and failure are read, not assumed): + +| Script | Safe invocation | Success | Failure signal / limit | +|---|---|---|---| +| `harness/version_diff.py` | `python3 harness/version_diff.py --old <old> --new <new> --out dumps/vdiff` | sections `ok`; self-diff zero; deltas explained | exit 2 tool error, 3 RM610 refusal; raw `.fwimage` faults early by design | +| `harness/portability_matrix.py` | `python3 harness/portability_matrix.py --target <a> --target <b> --out dumps/matrix` | self-comparison `all_zero`; M1–M6 `not measured` without reference | non-zero target deltas or per-target `status: error` | +| `harness/mutation_harness.py` | `python3 harness/mutation_harness.py --out dumps/mut --iterations 64 --seed 7` | deterministic `findings.jsonl`; declared outcome classes | seed divergence; non-demo ELF refused without `--allow-foreign-elf` | +| `harness/control_discovery.py` | `python3 harness/control_discovery.py --surface demo --out dumps/disc --seed 7` | known IDs resolve; unknown/injected classes listed | RM surface exits cleanly until H1.3; misclassification or unreplayable probe | +| `harness/reach.py` | `python3 harness/reach.py --target rpc_ping --method call --args 0x41 --out dumps/reach` | `reachable` with recorded route and args | timeout means "not under this stimulus", not unreachable | +| `harness/ci_release.py` | dry-run first: `python3 harness/ci_release.py --old <old> --new <new> --out dumps/release --dry-run` | dry-run touches no target; real run writes the bundle | exit 1 step error, 2 tool error, 3 RM610 refusal; `skipped` is not `pass` | +| `greeneye.sh` | `./greeneye.sh info`/`probe`/`record`/`analyze`/`compare`/`inject` | session dir with `session.json` + `summary.md` | empty sessions or missed hooks; budgets required | +| `rmharness/harness.sh` | `./rmharness/harness.sh --session dumps/real -- rmtool engine list --format=json` | `rm_harness_run` event + stream blobs | read-only only; permission errors reported, never escalated | +| `plugins/mcp_server.py` | `python3 plugins/mcp_server.py --stdio --record` | 24-tool JSON-RPC surface; read/observe default | stdout protocol-only; writes need `--allow-write`/`--allow-inject` | +| `harness/value_demos/run_all.py` | `python3 harness/value_demos/run_all.py` | one `RESULT_JSON` per demo; suite ≤ 180 s | any demo `success: false`; re-run before quoting numbers | + +**Scaffolded scripts (landed, not admitted yet).** The three scripts below +landed with docs 28/29 and are pilots until they have a §2 record; the first +two are dry-run by default (`--run` executes). `--help` is the authoritative +flag list; the outputs and failure readings are in +[`docs/28`](../debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md) and +[`docs/29`](../debug/risc-v-gsp-rm-vm/docs/29-PACKAGING-AND-TRIALING.md). + +| Script | Purpose | Interface | Outputs | Exit codes | +|---|---|---|---|---| +| `harness/port_driver_versions.py` | sweep a version list through `version_diff` + matrix + mutation, one evidence set per version | `--versions <list>` or `--config <json>`, `--baseline <ver>`, `--input-dir <dir>`, `--target <ver>=<path>`, `--symbols <ver>=<path>`, `--ghidra-exports <ver>=<path>`, `--out <dir>`, `--run` (default: dry-run), `--force`, `--extract-runfiles`, `--approval-token` | per-version `version-diff/` + `portability-matrix/` + `mutation-smoke/` + `bundle-manifest.json`, `sweep-summary.json|.md` | 0 complete, 1 step/threshold failure, 2 tool error, 3 RM610 refusal | +| `harness/port_new_device.py` | per-device port: profile, pairing check, first-touch probe, symbols, matrix, checklist | `--device <label>`, `--profile <json>`, `--firmware <path>`, `--reference <path>`, `--symbols <json>`, `--ghidra-exports <tsv>`, `--out <dir>`, `--run` (default: dry-run), `--approval-token` | `device-port.json|.md`, `port-checklist.json|.md`, `symbols/`, sessions, matrix, `bundle-manifest.json` | as above; 1 when pairing is ambiguous or a delta unexplained | +| `harness/package_sim.py` | build (and optionally run) a trial simulation package (§7) | `--target demo\|image`, `--image <path>`, `--service <elf>`, `--device-map <json>`, `--script <py>`, `--slot N`, `--pack-image`, `--tar`, `--run`, `--out <dir>` | package tree + `manifest.json`; `results.json|.md` after `run.sh` | 0 built/run, 1 replay or assertion failure, 2 tool error, 3 gate refusal | + +All three: stdlib-only, simulator/local only, deterministic budgets recorded in +the report, reusing existing harness functions, and running the RM610 label check first. +Their admission evaluation is recorded in +[`future-eng-ops.md`](future-eng-ops.md) F2–F4. + +**Version span and the RM 610 gate.** The automated span is **540 → current** +(anchor 570.144, [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §4.1); +inputs are versioned under `firmware/<SERIES>-GSP/<VER>/` and never overwritten. +**RM 610 requires explicit human approval**: before approval nothing fetches, +opens or analyses a 610 image +([`23-RM610-READINESS.md`](../debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md)). +`version_diff`/`ci_release` refuse any `rm610`-matching label with exit 3 +unless `--approval-token RM610-APPROVED` or `GREENEYE_RM610_APPROVED=1` is +present; the check runs in `--dry-run` and before any target is opened. An +agent may prepare the approval request and run current-anchor rehearsals; only +a named human approves. + +**Evidence bundles.** `ci_release.py` writes `release-report.json`, +`bundle-manifest.json` (sha256 + size of every file) and the step trees +([`docs/21`](../debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md) §4.3). +Exit 0 with all steps `ok` is complete, but read `skipped`; exit 1 means a step +errored (reviewable, not evidence for that section); exit 2 no claim; exit 3 a +gate refusal, not a bug. Verify hashes before sharing and attach the bundle. + +**MCP monitoring.** `plugins/mcp_server.py` exposes the live session +(read/observe by default) and shells out to `probe_firmware`, +`portability_matrix_run`, `mutation_run`, `analyze_current` and +`compare_sessions` ([`docs/20`](../debug/risc-v-gsp-rm-vm/docs/20-MCP-SERVER.md) §5). +Release pattern: matrix → pinned mutation smoke → compare the recorded pair → +read the bundle (`version-diff.json` / `release-report.json` / +`bundle-manifest.json`); the agent does not recompute it. Absent capabilities +return `not_available`, so planning is safe. + +**Cron/agent safety.** Default to read/observe and `--dry-run`; write/inject +flags are explicit and the recording is the audit trail. Physical GPU, mdev +churn, module load/unload and guest access are never unattended. Budgets +everywhere; a wedged target is left stopped. Product changes are proposals — +only humans merge, after physical confirmation of behaviour changes. Local +only: stdio MCP, loopback TCP at most; never expose it to a network. + +## 7. Packaging and trialing + +A trial package lets a user run one **pre-scripted whole-GSP simulation** with +injected code without the lab. It is a developer artifact: local, simulator +only, no hardware, guest or kernel module. It is not a product component and +not a support claim. + +```text +dist/<pkg>/ + README.md scenario, expected result, budgets, L2/L3 caveat + run.sh deterministic runner (greeneye.sh run + settle) + scenario.py fixed stimulus, fixed budgets, records a session + firmware/<image> ELF or .fwimage, hash-pinned + service/<service>.elf injected service (PIC, ABI v1), hash-pinned + device-map/<map>.json greeneye-device-map/1 (bias, entries, fw sha256) + golden/session/ optional reference session for compare + package-manifest.json sha256 + bytes per member (bundle shape, docs/21) +``` + +Runner and scenario follow the injection contract: PIC blob, no libc, no +absolute addresses, one dispatch entry per slot, every write recorded +([`docs/12`](../debug/risc-v-gsp-rm-vm/docs/12-INJECTION-SDK.md) §2–5); the device map +carries provenance so a lenient run is not mistaken for device-modelled behaviour +([`docs/24`](../debug/risc-v-gsp-rm-vm/docs/24-DEVICE-MAP.md) §1). + +```text +python3 harness/package_sim.py --out dist/<pkg> --service <service.elf> \ + --device-map <map.json> --script <scenario.py> [--pack-image] [--tar] [--run] +./dist/<pkg>/run.sh +``` + +Success: the packaged `run.sh` reports `status: ok` (demo target) or the +documented blocked status (real image), and every `manifest.json` component +hash matches. Failure: hash mismatch, nondeterministic replay, unresolved +symbol or missing member. The scaffold has no `--verify` subcommand yet: +verification is re-running `run.sh` and `sha256sum` against the manifest +([`docs/29`](../debug/risc-v-gsp-rm-vm/docs/29-PACKAGING-AND-TRIALING.md) §2–4), +and it stays a pilot until future-eng-ops F4's threshold is met. + +**Sharing and judging results.** Share the package (or its manifest) with the +session it produces. The recipient judges manifest hashes, the session +`summary.md`, the golden comparison and the fidelity statement (L0/L1 only, +L2/L3 absent). Results are evidence about firmware in the simulator; a product +change still requires the accuracy loop and physical confirmation. + +**Packing a modified GSP image.** When a GSP modification (injected service, +shim or patched firmware) is considered ready to ship, `package_sim.py` may +pack the modified image, the tested injected code, device map, scenario and +golden session so the recipient can reproduce the exact run before any product +decision. **Caveat, in every package README: this is a simulation package, not +a signed production image.** It is not a firmware patch, must never be flashed +or injected into a guest, and is not bootable on silicon. Injected code is +development-only ([`docs/12`](../debug/risc-v-gsp-rm-vm/docs/12-INJECTION-SDK.md) §1); +productization goes through `open-iov-vfio` + physical confirmation, never +through a simulator package. + +Local-only, budgeted, session-recorded; never touches `/dev`, a module, an +mdev, a guest or a GPU. No confidentiality, isolation or timing claim derives +from it; distribution follows per-file licences ([`SBOM.md`](../SBOM.md), +[`legal-and-licensing.md`](09-reference/legal-and-licensing.md)). + +## 8. Frontier capabilities + +Everything here is **future** and off the critical path. Each capability enters +the §2 process only when its near-term need is real and its gate is passed; +details are tracked in the companion [`future-eng-ops.md`](future-eng-ops.md) +(same directory) and +[`GREEN-EYE-ROADMAP.md`](../debug/risc-v-gsp-rm-vm/GREEN-EYE-ROADMAP.md) H5/H6. +Deferred context: [`10-post-1.0/experimental-host-acceleration.md`](10-post-1.0/experimental-host-acceleration.md). + +| Frontier capability | Near-term need | Precondition / gate | Evaluation criterion (if funded) | +|---|---|---|---| +| SIMD cores and CUDA kernel execution on device | Execute a real kernel launch path (`vectorAdd`-class) in simulation before the physical backend is finished | H1/H2 stable; injection ABI usable for an engine model | kernel executes within an instruction budget; output matches the native/golden result byte-for-byte; deterministic replay | +| FPGA soft-processor simulation | A cycle-accuracy question worth funding (timing/security/interoperability claim needing silicon-equivalent behaviour) | H3 validated (native parity) and H2 device model proven | the same `Session` scripts run against the RTL backend; execution parity with native; cycle counts recorded as a distinct event class, docs/06 fidelity revisited | +| Accurate Verilog recreation of all components the GSP interacts with, plus the NVRISC environment and intra/inter-micro-controller interconnects | Boot the real image with silicon-equivalent register, interrupt and mailbox semantics; exercise PMU/SEC2/ACR cross-controller boot flows | H2/H5 proven, validated device map, stable `Backend` abstraction | firmware reaches its message loop in the soft environment; M1–M6 improve over the QEMU/native baseline; one cross-controller exchange reproduced and compared with a physical capture | + +## 9. Review discipline + +**Adversarial-review loop.** (1) The author proposes a change with its +revision note (§9.3). (2) A second person or agent attacks it: broken links, +unverifiable claims, thresholds without metrics, tools without a scenario, +missing register rows, scope creep into the manuals. (3) The attacker +reproduces one item end-to-end (link check plus one claimed command); +"looks fine" is not a review. (4) The author resolves findings; the reviewer +signs the note with the evidence used. + +**Proportionate review (no needless overhead).** + +| Change class | Examples | Required review | +|---|---|---| +| A — editorial | wording, formatting, link fixes | `python3 scripts/check-docs-links.py` from the repo root; no design review | +| B — decision | admission, threshold, register, retire changes | one adversarial reviewer + evidence named in the note | +| C — scope/security | out-of-scope changes, gates, packaging caveats | maintainer sign-off + updated admission records | + +Cadence: review at each tool audit +([`27-TOOL-AUDIT.md`](../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md)) and +after any firmware anchor change. Expired pilots must appear in the register; +an empty register of decisions means this document is not being used. + +**Revision notes.** Every change records purpose, scope and benefit; a change +that cannot state a benefit is rejected as overhead. + +| Date | Revision | Purpose / scope | Benefit | Evidence | +|---|---|---|---|---| +| 2026-10-02 | v1 | Establish the admission rule, value axes, decision framework, workflow maps, automation, packaging, frontier gates and review discipline | Tooling decisions become auditable, measurable and reversible; no tool without an evaluation | This document plus linked tool docs and the [`docs/25`](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md) baselines | +| 2026-10-02 | v2 (Part II) | Add the live physical-backend operations: hard rules and scope, build/load order and the full `gsp_phys_*` parameter matrix, the routine supervised cycle with its clean-release checklist, the debugfs inventory and `gsp-phys-state` reference, the physical-backend scenario catalogue and hazard register, the OODA loop and the work-context/porting map | Physical-backend execution becomes auditable and fail-safe: one checklist, a named signal per failure, and no retry of a wedged path | Kernel sources cited per parameter and debugfs file; the current frontier in this document and [`STATE-CURRENT.md`](STATE-CURRENT.md) | + +```bash +python3 scripts/check-docs-links.py # repo root +cd debug/risc-v-gsp-rm-vm && make demo sdk && make test # tool claims +./greeneye.sh info # environment +``` + +--- + +# Part II — Live physical-backend operations + +> **Scope.** This part is the operating manual for cycles against the **live +> physical backend**: the module-owned physical GPU running its real GSP-RM +> session (the chosen execution mechanism in +> [`03-architecture/08`](03-architecture/08-physical-gpu-integration.md), wired +> toward the guest plane in +> [`03-architecture/11`](03-architecture/11-physical-backend-guest-wiring.md)). +> Part I (§§1–9) governs tooling admission, value measurement and retirement. +> Defensive/incident scenarios are owned by [`RUNBOOK.md`](RUNBOOK.md) and are +> cross-linked, not duplicated, here; the current frontier is +> [`STATE-CURRENT.md`](STATE-CURRENT.md); the known unknowns are +> [`04-implementation/01`](04-implementation/01-known-knowns-and-unknowns.md). +> Everything in this part is supervised, bounded and fail-safe-first; nothing +> here authorises unattended hardware work. + +## 10. Live physical-backend operations + +### 10.1 Hard rules and scope + +These apply to every physical cycle; where a step below conflicts with them, +the rules win. + +| # | Rule | Consequence / practice | +|---|---|---| +| 1 | **Supervised only, one GPU cycle at a time.** Never unattended, never parallel with other host work, never as a background agent job. | A named human is present for the cycle; the pre-flight and clean-release checklists (§10.3) are completed for each single attempt. | +| 2 | **Reboot is gated by the written approval policy.** A reboot request MUST first record, in the project's live working record, the reason and the exact post-reboot plan; only then is a named human asked to approve. | No reboot while other work is running; the device is left untouched in the meantime; an unapproved reboot is never initiated by an agent. | +| 3 | **Never enable `gsp_phys_repower`; never re-arm D3cold.** The default 0 latches the pre-claim D0 policy until reboot. | Re-enabling runtime PM/D3cold after a claim can power-gate the board and hard-freeze the host ([`03-architecture/08`](03-architecture/08-physical-gpu-integration.md) §4.2). | +| 4 | **Never retry a failed mutating RPC.** One attempt per staged step; abort the ladder from the first failure. | Retrying has wedged GSP-RM (channel-flag and context-promote incidents); `UNLOADING_GUEST_DRIVER` + `rmmod` and, if needed, an approved reboot remain the only recovery. | +| 5 | **Stop on hard signatures.** `board unresponsive`, `refusing claim`, `do not retry in this boot`, any panic/oops. | Do not write the device again in that boot; keep the locks; request reboot approval per rule 2. | +| 6 | **Verify module identity before and after a cycle.** Record the built module's sha256 (and `srcversion`) before load and re-read it after unload. | A hash mismatch invalidates the cycle's evidence; stop and rebuild from the reviewed source. | +| 7 | **A wedged device is left untouched for the reboot.** No `rmmod` retry loops, no bind/unbind experiments, no power-policy pokes. | The fail-safe (no rebind, D0 locks + runtime reference held, WPR2 up) is the designed resting state. | +| 8 | **Scope.** This part covers the claim/boot, rm-call ladder (host-side real submission), guest-memory forwarding, relay/fault-relay and release paths. Guest-facing emulation debugging is [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md); firmware runtime study is [green-eye](05-operations/03-risc-v-gsp-rm-vm.md). | All other physical work belongs to the scenario owner named in §14. | + +### 10.2 Build and load order, and the module-parameter matrix + +**Build order (physical-backend work needs the kernel module only):** + +1. Build `open-iov-vfio.ko` first; it generates the `Module.symvers` the + optional OpenRM modules link against. `scripts/stack_manager.sh + build-kernel`. +2. Build the optional OpenRM `nvidia.ko` with + `KBUILD_EXTRA_SYMBOLS` only if that management path is in use; the normal + consent path runs the distribution driver for the physical GPU and the + emulation module independently (`scripts/stack_manager.sh build-nvidia`). +3. Build userspace (`libiov`/examples) independently + (`scripts/stack_manager.sh build-userspace`). +4. Stage the GSP boot image into the firmware search path + (`scripts/stack_manager.sh install-firmware`). + +**Load order:** + +1. Prerequisites: `vfio`, `vfio_iommu_type1`, `vfio_pci`, `mdev` + (`scripts/stack_manager.sh load-prereqs`). +2. `open-iov-vfio.ko` with the exact reviewed parameter set + (`scripts/stack_manager.sh load-open-iov`). The backend claim is + **module-owned** at load with `gsp_phys_gpu=1 gsp_phys_hazard_ack=1`, or + **mdev-owned** and deferred to the first mdev create with + `gsp_phys_mdev_owned=1` (fail-open). +3. Record the loaded module identity and the starting state (§10.3 step 0). + +**Module-parameter matrix.** Every current `gsp_phys_*` parameter is listed +below, verified against the kernel sources on the 2026-10-02 tree. The exact +firmware image name used by `gsp_phys_fw` is deliberately not repeated here +(it is the distribution driver's Turing-generation artifact); the source is +authoritative for the literal default. + +| Parameter | Default | Effect | Gate / dependency | +|---|---|---|---| +| `gsp_phys_gpu` | 0 | Claim the physical GPU at module load: bind `open-iov-gsp-phys`, map BARs, stage firmware and run the requested stages; module-owned claim is released only at `rmmod`. | Refuses without `gsp_phys_hazard_ack=1`; refuses if the GPU or its root port is already runtime-suspended. With `gsp_phys_mdev_owned=1` it only arms the deferred path. | +| `gsp_phys_hazard_ack` | 0 | Acknowledges the unbind/rebind host-freeze hazard; second gate for every claim. | Required with `gsp_phys_gpu=1`; without it the claim logs `REFUSED` and returns `-EPERM`. | +| `gsp_phys_boot` | 0 | Full real-GSP boot: FWSEC → Booter Load → RISC-V start → RM-init poll (device writes). | Requires a claim; implies the layout stage; mutually exclusive with `gsp_phys_boot_dryrun`; FWSEC-only is refused because that carve is not host-recoverable. | +| `gsp_phys_boot_dryrun` | 0 | Stage-3d host-memory staging rehearsal (radix3, queues, boot args) with no BAR0 writes and no falcon resets. | Requires a claim; mutually exclusive with `gsp_phys_boot`. | +| `gsp_phys_layout` | 0 | Allocate the boot-argument pages and compute/log the WPR/FB layout; the only device access is a read-only video-memory-size sample. | Requires a claim; implied by the boot and dry-run paths. | +| `gsp_phys_nocat` | 0 | Log every NOCAT journal record (function `0x1020`) individually during the RM-init poll; the unique-record summary is always accumulated and dumped when the poll ends. | Boot path only; diagnostic. | +| `gsp_phys_fw` | distribution GSP image | GSP ELF the backend stages; override with `<name>` relative to the firmware search path. | Requires a claim; a missing/invalid image leaves the backend at ownership rung 1. | +| `gsp_phys_mdev_owned` | 0 | Deferred, refcounted lifecycle: first mdev create claims+boots asynchronously, last remove releases; every failure is fail-open to the emulation path. | Requires `gsp_phys_gpu=1` + `gsp_phys_hazard_ack=1`; a module-owned claim stays up until `rmmod`. | +| `gsp_phys_fwsec` | 0 | Run the FWSEC/FRTS secure-verification carve (device writes). | Requires the layout to have succeeded; **refused unless `gsp_phys_boot=1`**. | +| `gsp_phys_fwsec_dryrun` | 0 | Read-only validation of the VBIOS/BIT/FWSEC parse and the staging/placement math. | Requires a claim; no device writes. | +| `gsp_phys_fwsec_verbose` | 0 | Extra bounded read-only FWSEC kick state dump (falcon regs, scratch, DMEM/IMEM readback) after a successful halt. | Diagnostic; failure path always dumps. | +| `gsp_phys_static_live` | 0 | Overlay the live GPU name/short-name/VBIOS identity from the READY snapshot onto the emulated `GET_GSP_STATIC_INFO` reply (bounded 2 s wait, fail-open). | Requires a READY backend snapshot; other static fields stay emulated. | +| `gsp_phys_ctrl_forward` | 0 | Forward a whitelist of side-effect-free `GSP_RM_CONTROL` queries (bus BAR info, bus info, FB info, GMMU static info) with the adopted server handles; every miss falls back to the emulated handler. | Requires READY; fail-open; per-cmd forwarded/fallback counters. | +| `gsp_phys_work_forward` | 0 | Request one real CE shadow copy per eligible decoded guest push on the ladder channel; the software release path stays authoritative. | Requires a live claim + RM init-done + a built ladder channel; all CE forwarding paths refuse cleanly on a GR ladder. | +| `gsp_phys_guest_mem` | 0 | Let the ladder's guest triggers (6/7) and the forwarding path operate on a pinned guest GPA resolved through the guest-RAM backing file. | Triggers require READY, ladder stage ≥ 3, an identity DMA domain, an active guest-RAM file and a pinned GPA (6/7). | +| `gsp_phys_work_defer_release` | 0 | Defer a forwarded push's guest-visible release until the hardware copy completes (bounded 500 ms), then apply it from the service thread. | Requires `gsp_phys_work_forward=1` + `gsp_phys_guest_mem=1`; with 0 the release applies inline. | +| `gsp_phys_work_hw_primary` | 0 | Skip the software copy for a forwarded guest CE push (hardware authors the data); one software fallback copy on any hardware/queue failure. | Requires `gsp_phys_work_forward=1` + `gsp_phys_guest_mem=1` + `gsp_phys_work_defer_release=1`. | +| `gsp_phys_relay` | 0 | Let the engine author the selected guest-visible semaphore release into the guest page (third PTE on the same fresh line); the hold verifies the word and skips that record's software apply. | Requires `gsp_phys_work_forward=1` + `gsp_phys_guest_mem=1` + `gsp_phys_work_defer_release=1`; default off pending mismatch triage. | +| `gsp_phys_fault_relay` | 0 | On a relay timeout, drain the status queue for the engine fault (`0x1006`, corroborated by `0x1020`), match runlistId/chid and write the guest channel notifier + event. | Requires `gsp_phys_relay=1` and the three parameters above. | +| `gsp_phys_fault_inject` | 0 | Debug: write the relayed pair's source PTE as zero to force an engine fault on the next relay submission; only relayed pairs are affected. | Requires the relay path; default 0 programs the real source HPA. | +| `gsp_phys_rmcall_engine` | 0 | Ladder engine: 0 = COPY0 (CE copy submission), 1 = GR0 (compute object, BIND engineType 1, GR semaphore release). Snapshotted when stage 3 starts; mid-session changes are ignored. | Requires boot + init-done; CE forwarding paths refuse on a GR ladder. | +| `gsp_phys_rmcall_gr_promote` | 0 (clamp 0..3) | GR context-promote bisection: **0 never sends `PROMOTE_CTX`** (the ladder stops after the geometry queries, context buffer/PT allocation, PDB mapping and channel/compute alloc, before BIND); 1 = single MAIN entry, initialize/no-nonmapped, size raw+subctx; 2 = two-phase PA-only then VA-only; 3 = single entry with the exact raw size. Nonzero enables can assert GSP-RM (`0x52a8b04`) and wedge the device. | GR ladder only; value snapshotted at stage 3; **requires confirmed reboot readiness** before enabling. | +| `gsp_phys_repower` | 0 (latched) | Restore the pre-claim runtime-PM/D3cold policy when releasing the GPU; with 0 the D0 locks and runtime reference are deliberately left held until reboot. | **Never enable** (rule 3); re-arming D3cold after a claim can hard-freeze the host. | + +The emulation-side `sema_*` parameters (ring discovery, release recovery, +diagnostic watches, grace windows) are not physical-backend controls; their +canonical description is [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) +§3.2 and [`03-architecture/06`](03-architecture/06-host-module-and-hybrid-model.md). + +### 10.3 The routine supervised cycle + +One cycle is one attempt; there is no second attempt in the same boot after a +hard signature. + +**Step 0 — pre-flight (before anything is loaded):** + +* Host clean: no QEMU process, no mdev under `/sys/bus/mdev/devices/`, the + module not loaded, the physical GPU bound to the distribution driver. +* Record `boot_id`, pstore contents, module sha256/`srcversion`, and the + planned parameter set in the cycle log. +* Confirm supervision (present human), a written recovery plan, reboot + readiness per rule 2, and that other host work is finished. + +**Step 1 — build and identity check.** Rebuild the reviewed source; record the +module hash; never run a cycle against a stale or unknown module. + +**Step 2 — load.** Prerequisites, then `open-iov-vfio.ko` with the reviewed +parameters. Module-owned: the claim and requested stages run at load. +Mdev-owned: load, then create the mdev (the async claim/boot starts) only when +no VM is live. + +**Step 3 — claim/boot prerequisites.** In the cycle log confirm: `claimed`, +`PCI_COMMAND` bus-master on, BAR lengths, `BOOT0`/`BOOT1`, both falcons, WPR2 +down at entry; the D0 settle check passed (claim refuses otherwise); and after +the boot: `stage: primed=1 kicked=1 ran=1 riscv_active=1 rm_init_done=1`, WPR2 +up, `scratch14` hand-off bits advancing. Any claim refusal (`refusing claim`, +`board unresponsive`) ends the cycle (rule 5). + +**Step 4 — ladder (host-side real submission).** Write `gsp-phys-rm-call` +values 1..4 cumulatively; the ladder remembers the highest completed stage: + +| Write | Stage | Green signal | +|---|---|---| +| 1 | root/device/subdevice objects | stage 1 complete, handles logged (`hClient`/`hDevice`/`hSubdevice`). | +| 2 | memory registration | stage 2 complete. | +| 3 | VAS + TSG + channel (COPY0 variant walk; GR0 single attempt) | stage 3 complete (`variant`, `hChannel`, GR: `hGr`, `rm_call_ctx`). | +| 4 | real submission: CE copy or GR semaphore release | `REAL COPY OK` (CE) / `REAL GR SEM OK` (GR); trigger 9 re-runs only the GR release. | +| 5 | adopt static-config handles as stage 1 | stage 5 complete; a later 4 runs stages 2–4 with adopted handles. | +| 6 | guest write direction (W2G) on the pinned GPA | `GUEST W2G COPY OK`. | +| 7 | guest read direction (G2H) on the pinned GPA | `GUEST G2H COPY OK`. | +| 8 | fresh-line PTE self-test (no GPA) | `GUEST PTE SELFTEST OK` / `selftest_ok=1`. | +| 9 | GR semaphore re-run on a built GR ladder | `REAL GR SEM OK` or a decoded failure; no retry. | + +6–8 require `gsp_phys_guest_mem=1`, a READY backend and ladder stage ≥ 3; +6/7 additionally require the GPA pinned via `gsp-phys-guest-gpa`. On the first +failed stage: stop, drain the status queue, and end the ladder (rule 4). + +**Step 5 — guest attach (only if the rung needs it).** Start the VM with a +file-backed shared guest RAM region (`share=yes, prealloc=on`), attach the +mdev, configure `gsp-phys-guest-ram <path> <gpa_base> <offset> <size>`, pin +the GPA, and run the driver-API smoke; the guest must stay green (`ALL OK` +64/64) and any forwarded counters must be explained. Remove the mdev before +`rmmod`; never unload with a live VM. + +**Step 6 — evidence capture.** Keep with the cycle: parameter set, module +identity, boot id, `gsp-phys-state`, ladder traces, `gsp-phys-status-drain` +records, NOCAT/`0x1006` decodes, `gsp-phys-log-dump`, and the clean-release +checklist below. Evidence is property of the change that follows. + +**Step 7 — teardown/unload.** Remove mdevs/stop the VM, then `rmmod` +(`scripts/stack_manager.sh unload`). The release sends +`UNLOADING_GUEST_DRIVER` once (only after a boot), quiesces the falcon, runs +the FWSEC teardown, frees staging, and rebinds the distribution driver only +after a WPR2-down and board-liveness check, with the D0 locks held through a +bounded post-rebind readiness gate. + +**Mandatory clean-release checklist** (every line must hold; any failure stops +the cycle and invokes the fail-safe): + +| Check | Green value | If it fails | +|---|---|---| +| Boot id | unchanged from pre-flight | the device was power-cycled mid-cycle: discard in-window counters and rebuild the plan; expect a fresh scratch pool at the next boot. | +| Module identity | sha256/`srcversion` identical before/after | wrong build ran: stop, rebuild, restart the cycle (rule 6). | +| Protected region | `wpr2: ... state=down`; GSP falcon quiesced | fail-safe: **no rebind**, D0 locks + runtime reference stay held, WPR2 up; reboot approval per rule 2; never retry in this boot. | +| Unload RPC | `rm_unload_submitted=1`, processor suspended (mailbox `0x80000000`) | wedged server; fail-safe as above (the release log says `teardown did not clear WPR2` / `refusing the hand-back`). | +| Driver rebind | `released <bdf>; rebound to nvidia (rc=1)` | rebind failure/timeout: locks stay held; the module logs the manual `modprobe nvidia` path; reboot is the clean recovery. | +| Power-domain locks | exactly 5 `power domain left LOCKED (gsp_phys_repower=0):` lines on this host (GPU + board functions + root port) | wrong count: stop; audit the locks before any further cycle. | +| Rail cycles in window | 0 | any rail cycle rewrote the power risk: treat the cycle as unsafe and stop. | +| pstore / leftovers | pstore empty; no QEMU, no mdev, guest-RAM file released | a crash record or leaked state exists: capture it, clean with `stack_manager.sh cleanup-crash`, and do not start another cycle until the host is clean. | + +Fail-safe summary: on any WPR2/unresponsive/rebind failure the release does +**not** hand the device back, keeps the power-domain locks (and the runtime-PM +reference) held, and the correct next action is an approved reboot with the +written post-reboot plan - never a second attempt in the same boot. + +### 10.4 debugfs inventory + +All files live under the module's debugfs directory +`/sys/kernel/debug/open-iov-vfio/`. **The directory and every file are +root-only on this system**: existence and read checks must run as root (a +permission error can look like an absent file). Every writer is inert unless +its gates hold, and every accepted write performs a real RPC or device +action - log it in the cycle record. + +| File | Mode | Purpose | +|---|---|---| +| `gsp-phys-state` | 0444 | Read-only state snapshot (§10.5): stage/boot state, RM probe/call, GR context, guest copy and relay counters, fault counters, guest-RAM ranges, WPR2/RISCV/falcon/scratch14, queue headers. | +| `gsp-phys-log-dump` | 0444 | Read-only libos task log buffers (`LOGINIT`, `LOGRM`), raw words plus a decoded view. The raw view is authoritative; decode offline where a script is provided by the evidence set. | +| `gsp-phys-status-drain` | 0444 | One bounded status-queue drain per read, printing the decoded `0x1006` engine-fault and `0x1020` NOCAT records; advances the host read pointer (normal consumer protocol). | +| `gsp-phys-unload-rpc` | 0200 | Write `1`: send `UNLOADING_GUEST_DRIVER` once on a live claimed session; every other value is a logged no-op. | +| `gsp-phys-rm-probe` | 0200 | Write `1`: run one `SET_GUEST_SYSTEM_INFO`/`GET_GSP_STATIC_INFO` probe cycle and cache the static-config snapshot. | +| `gsp-phys-rm-call` | 0200 | Write `1`..`9`: the ladder stages, handle adoption, guest triggers and GR re-run of §10.3 step 4. | +| `gsp-phys-guest-gpa` | 0200 | Write `0x<hex>`: pin one 4 KB-aligned guest GPA for triggers 6/7; one GPA per boot, a different GPA is refused. | +| `gsp-phys-guest-ram` | 0200 | Write `<path> <gpa_base> <offset> <size>`: configure the guest RAM backing file and one GPA window (file path fixed for the module cycle); `-` clears. | + +### 10.5 Reading `gsp-phys-state` + +The first lines say whether the device snapshot is meaningful at all +(`inactive: physical GPU not claimed` / `stage-3d staging not primed`); a +snapshot is only read from a primed, claimed session. Then, in order: + +| Line | Fields | Meaning and invariants | +|---|---|---| +| `claimed` / `boot_param` / `dryrun_param` | claim and requested-stage flags | `claimed=0` with a live ladder stage means a release already ran: stop reading device state. | +| `stage:` | `primed`, `kicked`, `ran`, `riscv_active`, `rm_init_done`, `rm_unload_submitted`, `dryrun`, `tx_seq_num` | The boot lifecycle position. READY work needs `primed=1 ran=1 rm_init_done=1`; `rm_unload_submitted=1` means the session is ending and no new forward is legal. | +| `rm_probe:` | `attempted`, `last_ret`, `last_function`, `last_payload_len` | Static-config probe outcome; function 65 with a nonzero payload is the snapshot source. | +| `rm_call:` | `ran`, `stage`, `last_stage`, `last_ret`, `rpcs`, `hClient`/`hDevice`/`hSubdevice`/`hVas`/`hTsg`/`hChannel`, `variant`, `engine`, `hGr`, `token`, `notifier`, `base` | Highest completed ladder stage and the object handles; `engine` is the stage-3 snapshot (COPY0/GR0). A GR `hChannel` with `token` but a faulting push is the context-commit wall, not a transport failure. | +| `rm_call_ctx:` | `prepared`, `valid`, `pa`, `va`, `size`, `pages`, `ptPages`, `gr_promote_param`, `step`, `entrySize` | GR context buffer/PDB; printed only when a GR context was prepared. `gr_promote_param=0` means `PROMOTE_CTX` is never sent; `step` names the last accepted promote step. | +| `rm_call_guest:` | `gpa`, `hpa`, `g2h_ok`, `w2g_ok`, `selftest_ok`, `gpa_valid`, `phys_domain_identity` | The last guest trigger's translation and proof flags; `phys_domain_identity=1` is the prerequisite for programming host PAs into the engine page tables. | +| `work_guest:` | `req`, `fallback`, `ok`, `verify_ok`, `verify_fail`, `pool_exhausted`, `refused`, `cand_none`/`cand_sys_sys`/`cand_mixed`/`cand_vid_vid`, `resolve_fail`, `next_line`, `capacity` | Forwarded guest copies. Green run: `verify_fail=0`, `pool_exhausted=0`, `refused=0`, `resolve_fail=0`; `next_line ≤ capacity` (512 lines per boot, never reused). | +| `work_guard:` | `copy_skipped`, `deferred`, `applied`, `fallback`, `pending`, `overflow`, `engaged`, `timeouts`, `canceled`, `defer_param`, `copy_soft`, `copy_fail`, `hw_primary_param` | Deferred-release hold. **Invariant: `applied + fallback == engaged`.** `pending` is the live hold gauge (should return to 0); `timeouts`/`overflow`/`canceled` must be 0; `copy_soft` counts the designed inline software fallbacks. | +| `work_relay:` | `requested`, `ok`, `fallback`, `skipped_sw`, `verify_fail`, `verify_transient`, `hold_verify_fail`, `resolve_fail`, `timeouts`, `pool_exhausted`, `refused`, `relay_param` | Engine-authored semaphore relay. Green run: `verify_fail=0`, `hold_verify_fail=0`, `timeouts=0`, `pool_exhausted=0`, `refused=0`, `resolve_fail=0`; `verify_transient` is a retry-cleared mismatch and should be rare and explained. | +| `faults:` | `seen`, `stale`, `drained`, `wait_unmatched`, `epoch` | Fault observation at the status queue: records seen vs discarded as stale, records drained, unmatched waits, and the epoch used to match a relay submission to the fault it caused. | +| `fault_relay:` | `matched`, `unmatched`, `delivered`, `no_event`, `late_release`, `relay_param`, `fault_relay_param`, `fault_inject_param` | Guest delivery: `delivered` is a written notifier **and** event; `no_event` means the notifier was written but the auxiliary event post produced no MSI-X; `late_release` means the release landed before the fault could be relayed (no fault is delivered). | +| `guest_ram:` | `active=1 path=... ranges=...` or `active=0` | Backing-file and GPA-window configuration; `active=0` disables every guest-page path. | +| `wpr2:` / `riscv:` / `falcon:` / `scratch14:` | `state=up/down`; active bit; both falcons' CPUCTL/MBOX/FALCON_OS; secure-scratch bit fields | The fail-safe state. A green release ends with `state=down`, both falcons quiesced, and the booter-unload gate bits complete. | +| `cmd_tx` / `stat_tx` | queue header words | Queue producer/consumer cursors; a stalled status read pointer with an advanced command read pointer means "consumed but never replied" (the wedge signature). | + +**Relay data-mismatch semantics (fail-closed).** The relay result is verified +host-side against a source window snapshotted *before* the doorbell. A +mismatch is retried once after 2 ms: if the second read matches, it is a +`verify_transient` (the live guest page was mutated under the first compare) +and the copy stays verified; if it persists it counts `verify_fail`, the +submission returns an error, and the sema service falls back to the software +release. The same fallback covers `hold_verify_fail` (the guest semaphore +re-read did not carry the expected payload). **A software release apply is +never lost or duplicated**, which is why the relay defaults off until +`verify_fail` is triaged. + +## 11. Debugging scenario catalogue + +Physical-backend symptoms, their exact signal, and the bounded response. The +defensive/new-input scenarios (firmware release, OpenRM build, userspace +bump, device/ISA/ROM change, protected state, matrix/CI) are owned by +[`RUNBOOK.md`](RUNBOOK.md) S1–S10 and are not duplicated here. + +| Symptom | Exact signal to look for | Decode / likely cause | First diagnostic | Action | +|---|---|---|---|---| +| Guest GR push faults | `func=0x1006 ... exceptType=0x20 runlistId=0 chid=1 errString="... intr1 80000000"`; NOCAT `recType=4 errorCode=0x20 faultingEngine="GR"` | `0x20` = robust-channel PBDMA error; `intr1` bit 31 = `CTXNOTVALID`; the external-VAS GR engine context was never committed (`ENG_CTX_VALID=0`), so the guest binds via `PROMOTE_CTX` (`0x2080012b`). CE channels are unaffected. | `gsp-phys-status-drain`; `gsp-phys-state` `rm_call`/`rm_call_ctx` | Decode first, no retry. Only a proven promote shape commits the context; keep the client promote path gated by `gsp_phys_rmcall_gr_promote=0` until a variant is proven. | +| `PROMOTE_CTX` wedge | `gsp-nocat: rec seq=N recType=5 errorCode=0x52a8b04 source="ASSERT" faultingEngine=""`, then no fn76 reply (timeout `-110`); status ring stops, server command read pointer advanced | A host/server-RM assert in the promote path - not an engine fault (no `0x1006`). The server consumed the command and wrote no reply. Teardown cannot clear WPR2 (its unlock is fuse-locked on this part). | `gsp-phys-state` (`cmd_tx`/`stat_tx`, `wpr2`), release log | Ladder aborts by design; **do not retry**. Fail-safe holds the device; reboot with written approval (rule 2); bisect exactly one promote variant per cycle after reboot. | +| WPR2 / clean-release failure | `teardown did not clear WPR2 (HI=0x...)`; `release: WPR2 still up after teardown ... refusing the hand-back`; `NOT rebinding ... power domain stays LOCKED` | The booter-unload gate bits were never completed by the wedge, or the board stopped answering; the host-side unlock is fuse-locked. | `gsp-phys-state` `wpr2`, `scratch14`, `stage:` | Fail-safe is the designed state: no rebind, locks held, WPR2 up. Leave the device; reboot with approval. | +| Module unload hang | `rmmod` does not return; `ps` shows the emulation/relay kthreads or work items; mdev still present | Teardown ordering (kthread/workqueue) or an open mdev/VM; the emulation-side cleanup applies. | `ps -e \| grep gsp_emu`; mdev list; `dmesg` tail | Remove mdev/stop VM first; `scripts/stack_manager.sh cleanup-crash`; reboot with approval if it persists (rule 5). | +| Guest `cuInit` wedge / guest flake | guest `cuInit rc=999`, `rmapiControlCacheSet ... NV_ERR_INVALID_PARAMETER`, or a host `gsp-phys-fwsec: unexpected WPR2 already up`; a one-off guest Oops in `free_swap_cache` cleared on retry | On the emulated path this is a known guest-side flake; the WPR2-up line is the FWSEC refusal on a board already in a bad state. | guest dmesg; host `gsp-phys-state` `wpr2`; retry the guest once | If only the guest flaked, restart the guest once. If `unexpected WPR2 already up` appears, stop the cycle and inspect state; never retry FWSEC (rule 4/5). | +| GSP boot failure | `gsp-phys-fwsec: staging not visible before kick (DMEM0 ... IMEM ...)` then `FWSEC/FRTS execution failed: -5`; historical `GSP_INIT_DONE rpc_result=0x59` | Staging DMA did not land (async mdev-owned FWSEC path) or, historically, the pre-boot handshake/identity payload was wrong (`0x59` = operating-system error). | `gsp-phys-state` `stage:`, `gsp-phys-log-dump` | Use the module-owned claim path (proven); the async FWSEC defect is fail-open and off the critical path. Do not repeat a failed boot in the same boot after a hard signature. | +| MMU/PTE faults and the fetch-line rule | engine copy times out with no semaphore; `GUEST CE COPY FAILED: scratch-PT pool exhausted ... capacity=512`; re-translated GPA yields a different HPA | The GMMU caches PTE fetches in 128-byte lines and there is no invalidate: a runtime PTE write into an already-fetched line is invisible. A guest page's HPA can change between translations. | `rm-call: guest HPA changed: gpa=... was ... now ...`; `work_guest` `pool_exhausted`/`next_line` | Always use a fresh never-reused line per pair; pre-build guest-window slots before engine access; pin translated pages; treat pool exhaustion as end-of-budget, not a bug. | +| PBDMA method/class errors | `0x1006` with `intr0` nonzero (a method/class was rejected) versus `intr1` bit 31 (`CTXNOTVALID`); NOCAT `faultingEngine="GR"`; errString `"Channel ID ... intr0 ... intr1 ..."` | Wrong method encoding, class/subchannel or channel state; `intr0 != 0` means the engine rejected the method, so look at the pushed method block, not the context only. | `gsp-phys-status-drain`; compare the method encoding with the engine's class | Fix the method/class encoding; no retry against the same wedged channel; re-run from a fresh ladder. | +| Relay symptoms | `GUEST CE RELAY FAILED: semaphore released ... does not match ... (retried after 2 ms)`; `verify_fail`/`verify_transient`/`hold_verify_fail`; `fallback`/`skipped_sw`; `pool_exhausted`; `guest HPA changed` | One persistent data mismatch (shadow-page/concurrent-guest race or decode offset edge) is the open residual; anything else is the designed fail-closed fallback to the software apply. | `work_relay` line; the relay failure log includes sem target/payload/poll, first differing byte, live/snapshot dumps, `src_mutated`, `snap_vs_dst` | Keep default off until the mismatch is triaged. Abort the feature (`gsp_phys_relay=0`, then `gsp_phys_work_forward=0`) if `timeouts>0` or `verify_fail>0` ever appears. | +| Fault-relay symptoms | `fault_relay delivered=1` versus `no_event=1`; `late_release`; `matched`/`unmatched` by runlistId/chid; `faults epoch` | `delivered` = notifier written and event posted; `no_event` = notifier written but the auxiliary event post found no event object (no MSI-X reaches the guest); `late_release` = release landed first, so no fault is delivered (by design). | `fault_relay` line; `gsp-phys-status-drain` | A `no_event` is an event-path problem, not a fault-decode problem: fix the channel notifier/event wiring. Use `gsp_phys_fault_inject=1` to reproduce the path on demand. | +| Guest driver probe failure | guest sees no PCI device / BAR map fails / probe hangs; host `board unresponsive`, mdev `-EUSERS`; missing `SET_IRQS wired` | Config space, BAR mapping, MSI-X wiring or mdev state on the emulated path - not the physical claim. | guest `lspci -vvv`; host `05-operations/02` §5.1 taxonomy | Follow the taxonomy in [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §5.1; never start a physical claim to debug a guest-side probe failure. | + +## 12. Hazard register (crash causes and debugging side effects) + +| Hazard | Mechanism | Mitigation / practice | +|---|---|---| +| D3cold re-arm / power-gate host freeze | Releasing the D0 policy or re-enabling runtime PM after a claim lets the root port enter D3; the platform can cut the board rail, and the rebind can fail with a host freeze minutes later (no oops, pstore empty). | `gsp_phys_repower=0` (latched) is the default; the D0 locks cover GPU + board functions + root port; the post-rebind readiness gate must pass before anything is released; never unbind/rebind outside the supervised claim; pstore armed each cycle. | +| Failed mutating RPC wedges GSP-RM | One bad mutating operation (channel-flag incident; `PROMOTE_CTX` assert `0x52a8b04`) leaves the server unable to reply or suspend while WPR2 stays up. | One mutating attempt per staged step; never retry a failed mutating RPC; abort the ladder from the first failure; the release fail-safe keeps the locks and refuses the hand-back. | +| debugfs write side effects | Every 0200 file performs a real RPC/device action when its gates hold; a "test" write changes device or guest state. | Root-only; read `gsp-phys-state` and record the intended effect before any write; one write per action; no trigger writes outside a supervised cycle. | +| `pkill -f qemu-system-x86_64` self-match | The pattern also matches the invoking shell's own command line, killing the agent/session instead of the VM. | Use bracket patterns (`pgrep/pkill -f '[q]emu-system-x86_64'`) or explicit PIDs; stop VMs through `scripts/vgpu_vm.py`. | +| Root-only debugfs | The whole `open-iov-vfio` debugfs directory and its files are root-only; a non-root existence check reports absence. | Run existence/read checks under root; treat "missing" as a permission question first. | +| Module reload/unload hazards | Kthread/workqueue teardown can hang; the IOMMU identity-domain property and the guest-RAM file must not outlive the session; unloading with a live mdev/VM corrupts the teardown order. | Remove mdevs and stop VMs first; `gsp_sema_service_work_quiesce()` before staging is freed (release does this); use `scripts/stack_manager.sh unload`/`cleanup-crash`; audit leftovers after every unload. | +| pstore/journal persistence | A freeze leaves no oops but usually a persistent pstore record; the journal stops. | Check `/sys/fs/pstore` before and after every cycle; never clear it; capture it before any reboot or retry - it is the only post-freeze evidence. | +| Guest-RAM shared-file requirement | The engine can reach guest pages only through the QEMU file-backed shared region; a non-shared or non-preallocated RAM backend has no host page cache to translate. | Start QEMU with `memory-backend-file,share=yes,prealloc=on`; configure the file via `gsp-phys-guest-ram`; one file per module cycle; translation is gated on an open mdev and an active range. | +| Release-before-verify race | A hardware copy that times out cannot be canceled, so a late engine write is theoretically possible after the software fallback already applied. | The deferred-release hold withholds the guest release until completion or the bounded window; the software apply is never lost; if `timeouts`/`verify_fail` appears, abort the feature rather than tune around it. | +| Single-inflight limit / deferred-release hold | One forwarded request is in flight at a time; while a hold waits, new forwards are refused and apply their releases inline. | Designed fail-closed: `copy_soft` fallback, 500 ms cap, `pending` returns to 0. Do not deepen the queue without a measured need. | +| Scratch-pool exhaustion | 512 fresh 128-byte PTE lines per boot and no reuse; each copy pair consumes one whole line. | Size a cycle's guest-copy workload to the pool; treat `pool_exhausted` as the designed end-of-budget, then reload/reboot for a fresh pool. | + +## 13. The physical-backend OODA loop + +| Stage | What to do | Exit criterion | +|---|---|---| +| **Observe** | Collect: the `gsp-phys-state` snapshot (§10.5), ladder traces and stage numbers, `gsp-phys-status-drain` records, decoded `0x1006`/`0x1020` faults, `gsp-phys-log-dump`, the clean-release checklist, module sha/boot id, and the evidence set. | The cycle's question can be stated in one discriminating sentence with its parameter set and run identity. | +| **Orient** | Separate static from dynamic deltas. Classify failures: **engine fault** (`0x1006` + NOCAT `faultingEngine` set), **host/server assert** (NOCAT `recType=5`, `faultingEngine=""`, no `0x1006`), **transport** (timeout `-110`, status read pointer stalled while the command pointer advanced), or **data-verify** (`verify_fail`/`hold_verify_fail`). Match relay faults by `epoch` plus runlistId/chid and honour the external-VAS rule (`ENG_CTX_VALID` must be committed by a proven promote before a GR push can complete). | Every delta has exactly one class, or is recorded as an open gap. | +| **Decide** | Choose one: fix / observe more / extend tooling / record an accepted gap. Bisect with exactly one variable per cycle (parameter variant, one promote shape, one method encoding); never retry a wedged path, and never decide "fix" from a single run or a static reading. | The decision names the evidence and the single changed variable. | +| **Act** | Land the code change with the evidence set attached; commit only after a live green result on the same stimulus; fail-safe first (the release must refuse the hand-back before any optimisation). | Same stimulus is green, the clean-release checklist holds, and the change carries its evidence. | +| **Stop rules** | Hard signatures (rule 5), the timebox (three observe/compare iterations without a discriminating delta = stop and write down what is missing), the reboot policy (rule 2), and **no unattended runs** (rule 1). | Nothing physical runs past a stop rule; the state and hypotheses are recorded for the next supervised window. | + +## 14. Work-context map and porting flows + +### 14.1 Which document owns which debugging context + +| Debugging context | Owning document | +|---|---| +| Emulated-driver debugging (guest driver vs the emulated GSP-RM) | [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §1–3, §5 (observability, captures, gdbstub, failure taxonomy). | +| Firmware runtime (GSP/uproc in a RISC-V VM, hooks/watchpoints/sessions) | [`05-operations/03`](05-operations/03-risc-v-gsp-rm-vm.md) and the [green-eye user guide](05-operations/04-green-eye-user-guide.md) (experimental). | +| Live physical backend (claim/boot, ladder, forwarding, relay, release) | **This Part II.** | +| Portability (firmware, hardware, driver version, userspace version) | [`06-portability/*`](06-portability/hardware-software-matrix.md) and [`RUNBOOK.md`](RUNBOOK.md) S1–S10. | +| Performance and correctness validation | [`05-operations/01`](05-operations/01-performance-correctness-security.md) (evaluation ladder, metrics, A/B method). | +| Architecture of the backend and the guest wiring | [`03-architecture/08`](03-architecture/08-physical-gpu-integration.md), [`03-architecture/11`](03-architecture/11-physical-backend-guest-wiring.md). | +| Current frontier and dated status | [`STATE-CURRENT.md`](STATE-CURRENT.md). | + +### 14.2 Triage flow for a CUDA/userspace symptom + +1. **Guest diagnostics first** ([`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §2–3): guest dmesg, PCI config, `nvidia-smi`, the driver-API smoke result, and the exact failing call. +2. **Classify**: emulated reply (guest failed against the emulation plane) vs forwarded reply (`gsp_phys_static_live`/`gsp_phys_ctrl_forward`/`gsp_phys_work_forward` counters moved) vs engine fault (`0x1006`/NOCAT) vs guest-side flake (WPR2 untouched). +3. **Check the owning state/log**: for forwarded paths `work_guest`/`work_guard`/`work_relay`/`fault_relay` in `gsp-phys-state`; for faults `gsp-phys-status-drain`; for boot/claim `stage:`/`wpr2:`; for emulation-only symptoms the emulation trace. +4. **Next step**: emulation handler change under [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) discipline; forwarding parameter/ladder re-run under §10.3; fault decode and no-retry path under §11; guest-flake restart once. A physical claim is never the first response to a guest-side probe failure. + +### 14.3 Porting flows + +| New input | What changes | What must be validated | Procedure | +|---|---|---|---| +| Firmware image | The staged ELF, its signature section and the boot handshake; the static-config snapshot content. | ELF64 RISC-V shape and pairing; FWSEC/booter/RISC-V stages; RM init-done `rpc_result=0`; full static-config reply; one supervised cycle before any emulation overlay change. | [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §4.1; [`06-portability`](06-portability/hardware-software-matrix.md); [`RUNBOOK.md`](RUNBOOK.md) S1. | +| GPU hardware | Device profile (PCI IDs, BAR geometry, real FB size), power-domain topology (lock count), engine inventory, identity overlay. | Claim/D0 lock and clean release on the new board; boot layout; engine capabilities; a supervised cycle at the target rung; matrix entry only once earned. | [`06-portability`](06-portability/hardware-software-matrix.md); [`RUNBOOK.md`](RUNBOOK.md) S5. | +| Kernel driver version | The distribution driver that is unbound/rebound; module build against the kernel; OpenRM build glue if that path is used (never a runtime dependency). | Build order and symbols; claim/release hand-back; no regression on the module-owned session. | [`RUNBOOK.md`](RUNBOOK.md) S2; [`03-architecture/06`](03-architecture/06-host-module-and-hybrid-model.md) §7. | +| CUDA/userspace version | Guest-visible controls/replies and the userspace crash surface; the emulation's reply fidelity. | Driver-API smoke `ALL OK` 64/64; control/ioctl inventory diff explained; A/B on the same anchor. | [`05-operations/02`](05-operations/02-debugging-and-maintenance-workflows.md) §4.2; userspace CUDA playbook; [`RUNBOOK.md`](RUNBOOK.md) S4. | + +### 14.4 Current frontier (2026-10-02) + +Guest CE copies are **hardware-authored with verified data** and the relay and +fault-relay paths are in place, all default off (the deferred-release and +hardware-primary invariants hold; one relay data mismatch is the open +triage residual). The host GR ladder reaches the **context-promote wall**: +the GR0 channel and compute object allocate and the engine path is encoded, +but the external-VAS context is never committed, and the first host promote +attempt asserted GSP-RM (`0x52a8b04`) without a reply and left the protected +region up. The device is therefore parked in the **designed locked +fail-safe** (module unloaded, GPU unbound, D0 locks held, WPR2 up) awaiting a +supervised reboot; the promote path stays gated behind +`gsp_phys_rmcall_gr_promote=0` until a bisected variant is proven. Live detail +and evidence pointers: [`STATE-CURRENT.md`](STATE-CURRENT.md). + +<!-- change (2026-10-02): correct the porting/packaging tool section from +planned interfaces that no longer match the landed scaffolds (docs 28/29) to +the actual flags and outputs, link the device-validation sequence and +future-eng-ops admission evaluations, and mark the scaffolds unadmitted; +scope: eng-ops §3 axis 13, §6, §7, §8 only; benefit: readers no longer build +against non-existent flags or treat unadmitted scripts as routine pipeline. --> diff --git a/docs/future-eng-ops.md b/docs/future-eng-ops.md new file mode 100644 index 0000000..0aa961a --- /dev/null +++ b/docs/future-eng-ops.md @@ -0,0 +1,261 @@ +# Future engineering operations: green-eye capabilities not yet earning their keep + +> Companion to [`eng-ops.md`](eng-ops.md). That document records what the +> project uses now; this one records green-eye capabilities that are built but +> unused, partially used, planned or research, each tied to an impending +> project need, scoped explicitly, and carrying an evaluation that can fail — +> plus a kill criterion if it never pays off. Nothing here is a product +> commitment, and no simulation observation becomes an accuracy claim without +> the physical-confirmation rules of +> [`06-ACCURACY.md`](../debug/risc-v-gsp-rm-vm/docs/06-ACCURACY.md). + +Status snapshot: 2026-10-02. Sources are the green-eye roadmap, tool docs +16–27, the `harness/` entry points and the tool audit (doc 27). The tool tree +is development-only: never shipped, never a dependency of `open-iov-vfio`, +never present in a guest. RM 610 work is gated by human approval +([`docs/21` §3](../debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md), +[`docs/23`](../debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md)). + +## 1. Status vocabulary + +| Status | Meaning | +|---|---| +| `built, not used yet` | Implemented and tested, but no workflow calls it yet; the decision to keep is recorded. | +| `partially used` | Some pieces run in a real workflow; the capability as named is incomplete. | +| `planned` | Scoped in a roadmap/design document, not implemented. | +| `research` | No committed design; needs a funded use case before it becomes `planned`. | + +Priority scale: **P1** do when the gate lands; **P2** next, capacity +permitting; **P3** when a consuming workflow exists; **P4** research or +funded-only. Effort is engineering-time for one developer familiar with the +tree, excluding multi-quarter research. + +Every item below uses the same six fields: status, need, scope, evaluation +(metric / baseline / threshold / kill), gates, effort and priority. + +## 2. Catalogue at a glance + +| # | Capability | Status | Gate | Effort | Priority | Kill when | +|---|---|---|---|---|---|---| +| F1 | MCP automation of full port chains | `partially used` | CI pipeline (done), RM 610 approval | 1–2 wk | P2 | two releases show no MCP-only artifact | +| F2 | Multi-version driver port automation (540→current) | `partially used` | versioned inputs, anchor policy | 2–3 wk | P2 | two versions still need unabsorbed manual steps | +| F3 | Device-port automation scripts | `partially used` | reference device, GB10 availability | 1–2 wk | P2 | capture needs privileged/unsafe operations | +| F4 | Packing injected code and modified GSP images; trial bundles | `partially used` | H1.3, SDK | 2–3 wk | P3 | re-run needs host state or is non-deterministic | +| F5 | Control discovery against the real RM surface | `built, not used yet` | H1.3 + H2.2 | 1–2 wk | P1 | no dispatch table exposed after H2.2 | +| F6 | Value demos extended to rehosted uproc | `planned` | rehost shim (H2.3/R2) | 2–3 wk | P1 | fewer than 4 classes transfer | +| F7 | Native simulator M2+ (map DSL, snapshots, scheduler) | `partially used` | H2.2 exit (hard) | 24–36 wk | P1 | moments cannot restore identically by M3 | +| F8 | Adjacent-microcontroller models | `planned` | H2.2 | 6–10 wk | P2 | PMU/SEC2 semantics unprovable in one milestone | +| F9 | Performance analysis tooling (before/after) | `partially used` | H2.2 for real firmware | 2–3 wk | P3 | reports disagree with physical benchmarks | +| F10 | Guest/userspace friction: RDMA, shared page tables, peers | `research` | funded platform case | 8+ wk | P4 | no funded scenario at review | +| F11 | SIMD cores and CUDA kernel execution on device | `research` | H6/H3, funded reason | unbounded | P4 | first kernel prototype fails bit-exactness | +| F12 | FPGA soft-processor simulation | `planned` | H2/H3 proven, funded timing claim | multi-quarter | P4 | no funded timing claim at review | +| F13 | Verilog recreation of GSP components and NVRISC fabric | `research` | F12 progress, legal review | multi-quarter | P4 | accuracy cannot be bounded after first milestone | +| F14 | Hidden/undocumented/unauthorized-code discovery | `partially used` | H1.3 + H2.2 | 3–5 wk | P2 | two real releases yield no actionable finding | +| F15 | SBOM/architectural understanding from observations | `planned` | H2.2, legal review | 2–3 wk | P3 | observation rows rejected twice by SBOM review | + +Status counts: `built, not used yet` ×1 (F5); `partially used` ×7 (F1, F2, +F3, F4, F7, F9, F14); `planned` ×4 (F6, F8, F12, F15); `research` ×3 (F10, +F11, F13). Each item carries exactly one evaluation criterion. + +## 3. Items + +### F1 — MCP automation of full port chains + +| Field | Detail | +|---|---| +| **Status** | `partially used`. The MCP server is implemented and tested (24 tools, read/observe by default, writes and injection gated; [`20-MCP-SERVER.md`](../debug/risc-v-gsp-rm-vm/docs/20-MCP-SERVER.md)), and it exposes `probe_firmware`, `portability_matrix_run`, `mutation_run` and `compare_sessions`. No agent has driven a complete release chain through it, and `version_diff`/`ci_release` have no MCP tool. | +| **Need** | RM 610 readiness criteria C4/C5 require the MCP path and the release pipeline to be exercised, not merely present ([`23-RM610-READINESS.md`](../debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md)); every future GSP/driver release should be monitorable by an agent while a human keeps the merge gate. Waiting means each release is driven by hand and the 610 evidence pack stays unproven. | +| **Scope** | Does orchestrate the existing MCP tools plus a new `version_diff`/`ci_release` entry into one named, budgeted pipeline ending in a hashed bundle; does not add network transport, does not relax write/inject gating, does not approve product changes. | +| **Evaluation** | Metric: pipeline steps completed through MCP per release with a bundle manifest. Baseline: 0 complete chains (CLI only). Threshold: one 570.144 self-pair rehearsal completes probe → matrix → mutation → compare → bundle through MCP with artifact hashes equal to the CLI chain. Kill: if two releases produce no artifact the CLI chain lacks, retire the pipeline tool and keep MCP session-level only. | +| **Gates** | MCP server (done) and release pipeline (done, [`21`](../debug/risc-v-gsp-rm-vm/docs/21-CI-RELEASE-PIPELINE.md)); human approval before any RM 610 input; recorded session for every write. | +| **Effort / priority** | 1–2 engineering-weeks; **P2** (unblocks RM 610 C4/C5). | + +### F2 — Multi-version driver port automation (540-series → 570.144 → 580/610) + +| Field | Detail | +|---|---| +| **Status** | `partially used`. `cudatool.py info/delta/fdematch`, `diff_corpus.py`, `annotate_corpus.py` and `symbols export` are in production use for the 570.144 anchor, but the docs/22 §2 runbook is ten manual steps. A concurrent `harness/port_driver_versions.py` scaffold is landing as an unadmitted pilot (dry-run default; [`docs/28`](../debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md) §2); no chained multi-version run is admitted or exercised yet and there is no 540-era regression corpus in-tree. | +| **Need** | The anchor is 570.144; deployed guests and the anticipated 580/610 families each repeat the same runbook, and wrapper re-resolution (~1% of wrapper names transfer by address) is a known trap ([`22-PORTING-CUDA-AND-DEVICES.md`](../debug/risc-v-gsp-rm-vm/docs/22-PORTING-CUDA-AND-DEVICES.md)). Each repeat costs days of human time and risks silently moving wrapper names. | +| **Scope** | Does chain the existing CUDA tools into one per-version report (exports, FDE transfer matrix, corpus delta, name map, symbol DB, smoke results) and diff reports across versions; does not decompile unknown binaries automatically, does not change the firmware anchor, does not run a guest smoke without a supervised guest. | +| **Evaluation** | Metric: wall-clock and human minutes per version triage to a complete evidence pack. Baseline: the recorded 570.144 → 570.211 manual runbook. Threshold: one rerun (or 540-era reference) triaged in ≤50% of the baseline human minutes with wrapper re-resolution warnings preserved. Kill: if two consecutive versions need manual steps the chain cannot absorb, drop the chain and keep per-step tools. | +| **Gates** | Versioned parallel directories and anchor policy ([`22`](../debug/risc-v-gsp-rm-vm/docs/22-PORTING-CUDA-AND-DEVICES.md) §1); RM 610 gate; no new acquisition policy. | +| **Effort / priority** | 2–3 engineering-weeks; **P2**. | + +### F3 — Device-port automation scripts + +| Field | Detail | +|---|---| +| **Status** | `partially used`. Probe first-touch, portability matrix and version diff exist for firmware targets, but the device profile fields of docs/22 §3.1 are collected by hand (`lspci`, sysfs, `rmtool` JSON, guest diagnostics) and `config/hardware/*.yaml` is a design target, not read by code. A concurrent `harness/port_new_device.py` scaffold is landing as an unadmitted pilot (dry-run default; [`docs/28`](../debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md) §3); it is not yet exercised and no profile capture has been automated in a workflow. | +| **Need** | TU104 → GB202/GB10-class ports each require the profile table plus a pairing check; a missed field surfaces later as an "emulation bug". Pairing errors are already a known failure mode (570.144 image vs 570.211 driver). | +| **Scope** | Does script the read-only profile capture (PCI IDs, BAR layout, FB geometry, engine inventory, MSI-X topology, ROM quirks, probe-time registers), verify firmware pairing, and run the matrix across reference and candidate to emit a candidate profile plus evidence. Does not bind drivers, does not write to the device, does not claim support. | +| **Evaluation** | Metric: profile fields captured automatically out of the eight in docs/22 §3.1. Baseline: fully manual table. Threshold: ≥7/8 fields plus a pairing verdict and matrix report from one command on the reference TU104. Kill: if any required capture needs privileged writes or is not read-only-safe, retire the script to the manual table. | +| **Gates** | Supervised access to the reference device; GB10 hardware availability; the 610 gate where labels match. | +| **Effort / priority** | 1–2 engineering-weeks; **P2**. | + +### F4 — Packing injected code and modified GSP images; trial bundles + +| Field | Detail | +|---|---| +| **Status** | `partially used`. The PIC service SDK, `greeneye inject`, `elf.write_flat_kernel` and newly tested `write_relocated_elf` exist, and version diff already byte-patches an image ad hoc (docs/25 s07). A concurrent `harness/package_sim.py` scaffold is landing as an unadmitted pilot (trial directory plus `--pack-image`; [`docs/29`](../debug/risc-v-gsp-rm-vm/docs/29-PACKAGING-AND-TRIALING.md)); no packer or trial bundle is admitted, hash-verified in CI, or used by a reviewer workflow yet. | +| **Need** | Injection experiments today are demo-slot recipes; RM-core injection (H1.3) and reviewer replay need a reproducible artifact, the docs/23 C8 evidence bundle needs one-command reproduction, and hand-built byte patches do not carry an ABI/version contract. | +| **Scope** | Does build a deterministic image-pack (input hash, relocation map, injected segments, patched dispatch entries, blob ABI version) and a trial bundle (tool revision, profile, stimulus, expected session); does not sign images, does not bypass firmware checks, does not make an image runnable on hardware, does not ship. | +| **Evaluation** | Metric: manual steps to reproduce a reviewer's injected-service run from the artifact. Baseline: the ad-hoc patch + inject recipe. Threshold: one command yields a bundle whose rerun reproduces a byte-identical session on a clean host. Kill: if a rerun depends on host-specific state or is non-deterministic, retire the pack format and keep sessions as the artifact. | +| **Gates** | H1.3 for RM dispatch patching; `make sdk`; the docs/12 blob contract; native M2 moments optional. | +| **Effort / priority** | 2–3 engineering-weeks; **P3**. | + +### F5 — Control discovery against the real RM surface + +| Field | Detail | +|---|---| +| **Status** | `built, not used yet`. The `Surface` abstraction, sweep, classification, reporting and determinism tests are complete; `--surface rm` exists but exits cleanly because no RM control/method entry point is exposed ([`19-CONTROL-DISCOVERY.md`](../debug/risc-v-gsp-rm-vm/docs/19-CONTROL-DISCOVERY.md) §5). | +| **Need** | Every new GSP image can add or re-point controls silently; static table reading is per-image work and does not reveal triggers or effects. Control inventory (M1) is the input surface the emulation must handle, and the RM 610 port will need it first. | +| **Scope** | Does sweep RM control/method IDs by class with deterministic argument patterns, capture handler entry, selected object, status and effect snapshots, and replay a found route via `reach.py`; does not call hardware or a live RM, does not treat unreachability as proof, does not mutate replies. | +| **Evaluation** | Metric: RM controls classified with handler entry and effect snapshot versus the `rmtool` host-oracle inventory. Baseline: demo-only sweep (189 probes ≈ 36 s, docs/19 §3.4). Threshold: after H1.3/H2.2, one real control space swept with ≥90% of oracle controls attributed or explicitly default-path, plus one previously unknown control reached and replayed with a recorded session. Kill: if H1.3/H2.2 do not expose a dispatch table, retire `--surface rm` and re-scope discovery to rehosted dispatch code. | +| **Gates** | H1.3 + H2.2 (hard); device map ([`24`](../debug/risc-v-gsp-rm-vm/docs/24-DEVICE-MAP.md)); human approval for 610. | +| **Effort / priority** | 1–2 engineering-weeks after the gates; **P1** when H2.2 lands. | + +### F6 — Value demos extended to rehosted uproc + +| Field | Detail | +|---|---| +| **Status** | `planned`. s01–s10 run and are measured on the bundled `uproc_demo`; every real-firmware case in [`25-VALUE-VALIDATION.md`](../debug/risc-v-gsp-rm-vm/docs/25-VALUE-VALIDATION.md) is labelled blocked or projected. No demo runs on a rehosted GSP-derived uproc build. | +| **Need** | The value claim is currently analog-only. Reviewers, the RM 610 C8 evidence pack and the project's "hours, not weeks" success criterion need the same ten question classes answered on NVIDIA-derived code before the physical path is required. | +| **Scope** | Does port the demo classes (reply consumer, counter publication, unknown commands, run-until, snapshot diff, injection, version diff, mutation, Ghidra join, evidence pack) to a rehosted uproc ELF with identical success/failure tables; does not require the full image to boot, does not claim GSP-RM fidelity beyond the rehosted subset. | +| **Evaluation** | Metric: demos passing against a rehosted uproc with the expected outcome. Baseline: 10/10 on the demo analog only. Threshold: ≥6/10 on rehosted uproc, the remainder explicitly device-model-blocked with reasons, per-demo wall clock ≤2× analog. Kill: if fewer than four classes transfer, keep the demo corpus as a tooling regression suite, stop calling it value validation and re-scope docs/25. | +| **Gates** | A rehost shim and a stable symbol DB for the rehosted build (roadmap R2/H2.3). | +| **Effort / priority** | 2–3 engineering-weeks; **P1**. | + +### F7 — Native simulator M2+ (map DSL, snapshots/time-travel, deterministic scheduler) + +| Field | Detail | +|---|---| +| **Status** | `partially used`. M1 (RV64IMAC interpreter + `Session` parity) and M1.5 (native exec/access trace) are implemented and differential-tested; the device map and lenient MMIO mode exist. M2–M6 — map DSL, falcon/CSB/PTIMER devices, moments, deterministic scheduler, injection ABI, performance hardening — are designed but not built ([`16-CUSTOM-SIMULATOR.md`](../debug/risc-v-gsp-rm-vm/docs/16-CUSTOM-SIMULATOR.md)). | +| **Need** | QEMU has no cheap snapshots, deterministic replay or in-process device callbacks, so the fuzzing and discovery harnesses pay a fresh session per probe (measured ~36 s/189 probes) and branch exploration is impossible. H4/H5 work and the R5 fuzzing corpus are gated on cheap forks and replay. | +| **Scope** | Does M2 map DSL + falcon/CSB/PTIMER + moments; M3 deterministic scheduler + event log/replay; M4 device-model parity with H2.2; M5 injection ABI/interception; M6 performance and CLI. Does not claim L2 timing or cycle accuracy, does not replace QEMU before M4, no realtime behaviour. | +| **Evaluation** | Metric: (a) double-run canonical hash equality, (b) moment save/restore latency against the docs/16 §14 targets (≤100 ms save / ≤50 ms restore at 256 MiB sparse), (c) rehosted boot from a profile plus restore reproducing the event-stream hash. Baseline: fresh-session cost and QEMU. Threshold: all M2/M3 exit criteria in docs/16 §13 met on the rehosted uproc. Kill: if moments cannot restore byte-identical sessions by the M3 exit, drop snapshots and keep deterministic replay-only. | +| **Gates** | H2.2 exit (hard, cannot start before); docs/16 §13. | +| **Effort / priority** | 24–36 engineering-weeks for M2–M6; **P1** after H2.2, not before. | + +### F8 — Adjacent-microcontroller models + +| Field | Detail | +|---|---| +| **Status** | `planned`. H5 defines PMU/SEC2/ACR/SOE inventory, per-controller maps and cross-controller scenarios; none is modelled. H2.2 will force partial PMU/SEC2 stubs to pass early GSP init. | +| **Need** | GSP-RM early init reads PMU identity/clock provisioning and SEC2 state; GB10-class and confidential-compute partitions follow the Init → ACR → RM ordering. Cross-controller regressions are invisible until a hardware cycle fails, which is exactly the expensive class this tool exists to remove. | +| **Scope** | Does per-controller memory maps, mailbox protocols and boot state machines as versioned profile data, plus recorded cross-controller sessions; does not fully model the security chain, does not make confidentiality or attestation claims, does not run real ACR/SOE. | +| **Evaluation** | Metric: cross-controller exchanges reproduced and compared against hardware evidence. Baseline: 0. Threshold: one exchange (GSP ↔ PMU identity/clock) reproduced with an A/B session pair reaching the same state as the hardware reference under the docs/06 metric table. Kill: if PMU/SEC2 semantics cannot be established from public evidence within one milestone, keep only documented H2.3 shims and mark adjacent-controller claims out of scope. | +| **Gates** | H2.2; public/authorised source material; a hardware reference capture; docs/23 gate for any 610-family controller. | +| **Effort / priority** | 6–10 engineering-weeks for PMU/SEC2 stubs; **P2**. | + +### F9 — Performance analysis tooling (profiling path, before/after methodology) + +| Field | Detail | +|---|---| +| **Status** | `partially used`. `sim_bench.py`, the trace-overhead measurements (docs/16 §1.1a, docs/25 §6) and the product A/B methodology in [`01-performance-correctness-security.md`](05-operations/01-performance-correctness-security.md) §3 exist. There is no green-eye path that profiles a product run off-hardware or produces an automated before/after report. | +| **Need** | Every emulation change is measured against docs/05 targets (e.g. init within 2× native) using physical cycles and a manual harness; pre-screening structural regressions off-hardware would reduce failed physical cycles. M7 timing is never claimed, so the useful signal is structural (instruction/event counts, session bytes, written pages), not wall clock. | +| **Scope** | Does capture deterministic structural metrics per session and diff them before/after with an explicit fidelity label; does not make timing or wall-clock claims about silicon, does not replace the physical benchmark in docs/05. | +| **Evaluation** | Metric: fraction of perf-sensitive emulation changes that receive a green-eye before/after report before the physical benchmark. Baseline: 0. Threshold: five consecutive changes carry a report, and structural regressions ≥10% are flagged before the hardware run. Kill: if reports agree with the physical benchmark less than 50% of the time, retire them to debug annotations only. | +| **Gates** | H2.2 for real firmware; docs/06 M7 rule; docs/05 methodology is authoritative. | +| **Effort / priority** | 2–3 engineering-weeks; **P3**. | + +### F10 — Guest/userspace compatibility friction: RDMA, shared page tables, simulated peers + +| Field | Detail | +|---|---| +| **Status** | `research`. No peer-device simulation (DPU/NIC/NVMe), no RDMA path; page-table work exists only on the product GMMU/IOMMU side and in the physical-backend wiring. | +| **Need** | Edge-AI platforms place GPU, networking and storage VMs in one trust domain with device-to-device data paths ([`haaeep-and-collaboration.md`](08-ecosystem/haaeep-and-collaboration.md)); guest RDMA and shared-page-table flows are a known friction class, and failures are expensive to debug on hardware. A simulated peer would let the page-table contract be regression-tested early. | +| **Scope** | Does model DPU/NIC/NVMe peers enough to exercise page-table sharing and a GPUDirect-style map → peer DMA → completion path against the simulated GSP; does not implement real network/storage protocols, does not claim performance, does not modify guests. | +| **Evaluation** | Metric: peer-interaction failure classes reproduced off-hardware that matched a hardware capture. Baseline: 0. Threshold: one page-table-sharing interaction reproduced and compared against hardware. Kill: if no funded platform scenario needs it at the next review, keep it as a design note and delete the experiment scaffolding. | +| **Gates** | H2.2/H3; a funded platform use case; no confidential-compute claim; a hardware reference capture. | +| **Effort / priority** | 8+ engineering-weeks; **P4** research. | + +### F11 — SIMD cores and CUDA kernel execution on device + +| Field | Detail | +|---|---| +| **Status** | `research`. The native core is scalar RV64IMAC; there is no SM/SIMT or vector execution model. The product's guest-to-physical execution wiring is in progress ([`11-physical-backend-guest-wiring.md`](03-architecture/11-physical-backend-guest-wiring.md)), so command streams exist but nothing executes kernels off-hardware. | +| **Need** | Executing a real CUDA kernel against a device model would validate the command/memory path beyond copy-engine correctness and enable fail-closed testing of kernel submissions without a physical cycle. Waiting is acceptable while hardware backends exist, which is why this stays research. | +| **Scope** | Does explore a device-side execution model (SIMT/SIMD core plus memory model) able to run a minimal kernel from an emulation-produced command stream; does not claim performance or cycle accuracy, does not replace hardware for CUDA, not on the critical path. | +| **Evaluation** | Metric: vectorAdd-class kernels producing byte-identical results to a hardware reference. Baseline: 0. Threshold: one deterministic kernel correct on the simulator from an emulation-produced command stream. Kill: if the first bounded prototype cannot produce bit-identical results for a 64-element vectorAdd, stop and record why; only a funded cycle-accuracy or thesis use case may restart it. | +| **Gates** | H6/H3; a funded reason; simulator-security posture (docs/16 §16). | +| **Effort / priority** | Unbounded research; **P4**. | + +### F12 — FPGA soft-processor simulation + +| Field | Detail | +|---|---| +| **Status** | `planned`, far later. The H6 annex defines an RTL RV64 core with TCM/MPU/timers, falcon/CSB/ACR windows, DMA and WPR2/FWSEC state, Verilator and FPGA run modes, and a co-simulation bridge. Nothing is built or started. | +| **Need** | When a funded timing/interoperability claim appears (customer-facing timing, hardware validation of an emulated behaviour), functional models are insufficient. Until then, starting would be effort creep against the roadmap's explicit non-critical-path note. | +| **Scope** | Does build/choose an RV64 core and device fabric, pure-RTL simulation for determinism, FPGA for speed, and a `Backend` bridge so the same hooks, sessions and comparisons apply; does not claim confidentiality or isolation, does not ship, never present in a guest. | +| **Evaluation** | Metric: cycle accounting agreement between RTL and a hardware capture on a bounded scenario. Baseline: none. Threshold: one bounded scenario (e.g. mailbox round trip) cycle-accounted within a declared tolerance, or the variance fully explained. Kill: if no funded timing claim exists at the next review, keep the design annex and close the effort without building. | +| **Gates** | H2/H3 proven; validated device map; explicit funded reason; docs/16 §16 H6.3. | +| **Effort / priority** | Multi-quarter; **P4**, funded-only. | + +### F13 — Verilog recreation of GSP-interacting components and the NVRISC fabric + +| Field | Detail | +|---|---| +| **Status** | `research`. H6.1 names the GSP-interacting blocks, but there is no model of the NVRISC environment, interconnect ordering, interrupt fabric or doorbell/mailbox races, and no tracked public design to build one from without a provenance review. | +| **Need** | Deep timing, security and interoperability questions (interconnect ordering, boot-order races, doorbell semantics) cannot be answered by functional models; a documented recreation is the only off-hardware route to them. No funded scenario exists yet, so it is research. | +| **Scope** | Does define and build behavioural/cycle-aware RTL for GSP-interacting blocks and the NVRISC fabric environment with a declared accuracy envelope; does not reproduce secret silicon internals, does not claim confidentiality/attestation, never on the critical path. | +| **Evaluation** | Metric: interconnect-order or race observations reproduced against hardware captures within the stated envelope. Baseline: 0. Threshold: one documented race/ordering case explained and validated on hardware. Kill: if accuracy cannot be stated as a bounded envelope after the first milestone, retire to a paper design and record the decision. | +| **Gates** | F12 progress; a funded reason; legal review (tracked docs describe behaviour only). | +| **Effort / priority** | Multi-quarter research; **P4**. | + +### F14 — Hidden/undocumented/unauthorized-code discovery + +| Field | Detail | +|---|---| +| **Status** | `partially used`. Control discovery, reachability and seeded mutation are built and measured on the demo (injected-slot aliases, default-path `~id`, sentinel-pointer faults; [`19-CONTROL-DISCOVERY.md`](../debug/risc-v-gsp-rm-vm/docs/19-CONTROL-DISCOVERY.md)). Capability inference is designed, decoy/anti-analysis handling is unbuilt, and no real image can be swept before H2.2. | +| **Need** | Firmware is stripped and signed; new images can add handlers, re-route dispatch entries, or insert anti-analysis checks. A missed handler is a silent emulation regression, and the fail-closed rules the product needs come from error paths that only adversarial input reaches. | +| **Scope** | Does sweep inputs, classify unknown/default paths, infer capabilities from watchpoints and trace, catalogue observed anti-analysis techniques and feed fail-closed rules; does not bypass hardware protections, does not attack a live device or guest, does not publish exploit material. | +| **Evaluation** | Metric: hidden/undocumented behaviours found per image that changed a modelling or security decision. Baseline: demo findings only. Threshold: ≥1 new real-image behaviour per release that lands with a replayable probe spec and a recorded emulation/security decision. Kill: if two real-image releases yield no actionable finding, reduce to the fast CI inventory smoke and stop the broader campaign. | +| **Gates** | H1.3/H2.2; docs/14 ethics and safety rules; human review before publication. | +| **Effort / priority** | 3–5 engineering-weeks plus campaign time; **P2**. | + +### F15 — SBOM/architectural understanding from observations + +| Field | Detail | +|---|---| +| **Status** | `planned`. The repository has a tracked SBOM ([`SBOM.md`](../SBOM.md), CycloneDX) and the wiki GPU SBOM is superseded; there is no observation-derived inventory per firmware image that links services, symbols and dispatch tables to the SBOM and architecture docs. | +| **Need** | Version bumps change firmware components and external tooling dependencies; provenance, license and vulnerability response need to know what an image contains and what changed. The architecture docs drift from the binary until a costly re-analysis rediscovers the delta. | +| **Scope** | Does derive an observation-backed component/capability inventory per image (hashes, symbols, services, dispatch tables, external dependencies) and link it to SBOM/architecture updates; does not assert legal conclusions, does not name local sources, does not auto-publish un-reviewed findings. | +| **Evaluation** | Metric: SBOM/architecture deltas per release that are observation-derived and reviewable. Baseline: manual entries. Threshold: one release where every added/removed firmware component row carries a hash plus evidence link from a green-eye session. Kill: if observation rows are rejected by SBOM review twice, drop to a provenance annex inside the session evidence. | +| **Gates** | H2.2 for real images; legal-and-licensing review ([`legal-and-licensing.md`](09-reference/legal-and-licensing.md)); local-provenance rules. | +| **Effort / priority** | 2–3 engineering-weeks; **P3**. | + +## 4. Pruning and retirement + +An item is retired at the review where its evaluation fails or its gate has +not landed for two consecutive reviews. Retirement is a recorded action, not a +silent drift: + +| Outcome | Trigger | Action | Record | +|---|---|---|---| +| **Prune** | Threshold missed, or gate unlanded after two reviews, with no funded exception | Delete the code path and its tests; mark the capability `retired` here; remove it from roadmap commitments | Decision log row with date, evidence link, owner | +| **Demote** | Need is real but not now | Move the item to a research appendix; stop maintaining scaffolding; keep the design pointer | Decision log row | +| **Keep** | Evaluation passes, or a named funded exception exists | Retain with owner and next check date; state the exception's expiry | Decision log row | + +No item has been retired yet (first review pending). The smallest +`built, not used yet` surface found by the tool audit (`Session.symbol_at`, an +alias with no caller) is the first prune candidate: if it still has no caller +at the first review, delete it rather than documenting API symmetry. + +## 5. Review cadence + +| Trigger | Who | What is re-checked | +|---|---|---| +| H2.2 exit | green-eye maintainer + eng-ops owner | F5, F6, F7, F8, F14, F15 gates open; statuses move to `partially used`/`planned` execution | +| Each GSP/driver release | green-eye maintainer | F1/F2/F3 chain cost, F14 findings, F15 SBOM deltas; kill criteria evaluated | +| RM 610 approval request | project lead + maintainer | F1 evidence (C4/C5), bundles for C8; no 610 input before approval | +| Quarterly | eng-ops owner | Re-run or explicitly defer every evaluation; prune/demote/keep decisions recorded in §4 | +| New funded use case | project lead | Admit a new item with the same six fields; research items become `planned` only with funding | + +The doc is reviewed against the tool audit ([`27-TOOL-AUDIT.md`](../debug/risc-v-gsp-rm-vm/docs/27-TOOL-AUDIT.md)) +and the roadmap ([`GREEN-EYE-ROADMAP.md`](../debug/risc-v-gsp-rm-vm/GREEN-EYE-ROADMAP.md)) +so the "what we use now / not yet" split stays true. + +<!-- change (2026-10-02): point F2–F4 at the landed docs 28/29 scaffolds so the +catalogue entries and the tree agree; scope: F2/F3/F4 status rows only; +benefit: a reader can find the pilot code and its evaluation from the catalogue +without treating it as admitted tooling. --> |