From 002e3d7d13c414ea874abbee8d82803dd3a72fe5 Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Fri, 2 Oct 2026 10:58:21 -0400 Subject: debug: green-eye wave 3 - RM harness, local simulator, device map, MCP plugin, value-validation demos, CI pipeline, toolchain docs; add host RM tools rmtool/rmtop --- .../docs/22-PORTING-CUDA-AND-DEVICES.md | 353 +++++++++++++++++++++ 1 file changed, 353 insertions(+) create mode 100644 debug/risc-v-gsp-rm-vm/docs/22-PORTING-CUDA-AND-DEVICES.md (limited to 'debug/risc-v-gsp-rm-vm/docs/22-PORTING-CUDA-AND-DEVICES.md') diff --git a/debug/risc-v-gsp-rm-vm/docs/22-PORTING-CUDA-AND-DEVICES.md b/debug/risc-v-gsp-rm-vm/docs/22-PORTING-CUDA-AND-DEVICES.md new file mode 100644 index 0000000..30f4720 --- /dev/null +++ b/debug/risc-v-gsp-rm-vm/docs/22-PORTING-CUDA-AND-DEVICES.md @@ -0,0 +1,353 @@ +# 22 — Porting with green-eye: CUDA userspace, GSP firmware and new GPU devices + +> **# TOOL IN DEVELOPMENT - NOT READY YET!** See 00-OVERVIEW for status. +> This is the combined porting guide for the three churn axes the project +> faces. It joins the existing CUDA/GSP/firmware procedures with green-eye's +> runtime layer. Items marked "planned (docs/xx)" do not exist yet; do not +> invent commands for them. + +## 1. The three churn axes and the anchor discipline + +NVIDIA ships three things on independent schedules, and any one of them can +break `open-iov-vfio` while the other two stay still: + +| Axis | Examples | Primary evidence path | +|---|---|---| +| **CUDA userspace version** (`libcuda`) | 570.144 → 570.211.01; future 580/610 families | CUDA corpus tooling + guest smoke (`userspace/cuda/docs/07`) | +| **GSP firmware / RM version** | 570.144 → 570.211 GSP image; RM dispatcher and control-set changes | `greeneye probe` + `compare`, host oracle (`docs/05-operations/02` §4.1) | +| **GPU device / architecture** | TU104 → GB202 (RTX 5090), GB10 (Grace-Blackwell), aarch64 hosts | Device profile + pairing check + portability matrix (docs/07 P2/P3, docs/18) | + +They interact: + +* a userspace bump usually implies an emulation-anchor decision (guest driver + and emulated firmware move together) even when the firmware binary is + byte-identical; +* a device change can drag a different firmware family (TU10x vs GB100 vs + `gb10`) and a different partition/boot model (FMC/MP-SK; Init → ACR → RM for + confidential compute) with it; +* an architecture change (aarch64 host, newer NV-RISCV core) can invalidate + environment assumptions that the other two axes assume are stable + (PMU/SOE provisioning, coherent memory, ISA extensions). + +**Anchor discipline.** The lock-step that must stay coherent is the +**guest driver ↔ emulated firmware profile** (the GSP generation whose protocol +the host emulates). The host's distribution driver is **not** part of that +lock-step: it is the oracle used for A/B comparisons, and the optional OpenRM +host modules are only for native vGPU-management workflows. The current anchor +is **570.144**; a version bump moves the guest driver and the emulation, not +the host distro driver (see `docs/05-operations/02-debugging-and-maintenance-workflows.md` +§4.1). Porting rules that follow: + +1. One anchor per port; never evaluate a new `libcuda` against an old emulated + firmware profile and call a mismatch an emulation bug. +2. Every input is versioned and hashed: `firmware/-GSP//`, + `userspace/cuda//`, `config/versions/.yaml`. The `config/` + YAML files are **design targets, not live configuration** until the module + load path reads them (`docs/06-portability/hardware-software-matrix.md` §6). +3. New versions live in parallel directories; the previous anchor stays + untouched so rollback never destroys a working setup. +4. A port is a claim with evidence attached (M1–M6, §4), not a merged patch. + +## 2. Joint workflow A — CUDA userspace version port (`libcuda` bump) + +### 2.1 Decide what kind of port it is (minutes) + +Run the cheap triage before any decompilation or emulation work: + +```bash +cd userspace/cuda +tools/cudatool.py info /run/libcuda.so. +tools/cudatool.py delta /run/libcuda.so. /run/libcuda.so. > /run/delta.txt +``` + +| `delta` reading | Port class | What it implies | +|---|---|---| +| no export moves, small `.text` growth | patch | transfer the name map and spot-check touched subsystems | +| new/removed exports | feature | decode the new API surface before touching the emulation | +| many moves | reshuffle (toolchain/linker) | plan a full re-analysis; a fraction of names will not transfer | + +### 2.2 Static analysis vs dynamic confirmation + +| Step | Existing project tooling | green-eye addition | +|---|---|---| +| Export/interface diff | `tools/cudatool.py delta` | — | +| Function matching / name-transfer measurement | `tools/cudatool.py fdematch --names ` | — | +| Corpus-level diff (added/removed/resized functions) | `tools/diff_corpus.py ` | — | +| Name transfer + corpus annotation | `ApplyNameMap.java` (`--dry-run` first), `tools/annotate_corpus.py` | `./greeneye.sh symbols export` builds a runtime-address symbol DB from the same index/names | +| Driver-API smoke | `tools/test/cuda_drv_test.c` compiled as `/tmp/cdt` (prints `ALL OK`) | `./greeneye.sh symbols resolve --db ... ` spot-checks transferred addresses per call path | +| Runtime workload | `tools/cuda-ab/ab.sh all` (`vector_add.cu`; `RESULT`/`STATUS` must match host) | session-format evidence and `./greeneye.sh compare` when firmware-side sessions exist | +| Guest-side control observation | `tools/guest-diag/ioctl_trace.c`, host module trace (`gsp-resman-vfio:`) | annotation of the firmware-side RPC path with the per-version symbol DB | +| Firmware-side confirmation | — | `./greeneye.sh probe` / `record` / `compare` against the paired firmware | + +**Be honest about the execution boundary.** `cdt` and `vectorAdd` are x86_64 +guest binaries; the RISC-V simulator does not execute them. green-eye's +contribution to a userspace port is the **symbol/evidence layer**: a +runtime-address DB per `libcuda` version, per-address resolution checks for +transferred names, and a recorded comparison format for the GSP-side behaviour +the driver drives. The print-and-verify path (`cdt`, `ab.sh`, ioctl trace) +remains the dynamic oracle. + +### 2.3 Step-by-step runbook + +1. **Acquire and version.** Fetch the new `libcuda.so.` + (`570.144/scripts/fetch_from_guest.sh`, or the driver package with + `--extract-only`). Work in a new `userspace/cuda//` tree copied from + `570.144/`; never overwrite an analysed version. +2. **Cheap triage.** `cudatool.py info` + `delta`; classify the port (§2.1); + record `.text` growth, build-id and ABI note. +3. **Measure transfer before copying.** Run `cudatool.py fdematch` with the + old `names.tsv`. The known result from 570.144 → 570.211.01: all 659 + exports transfer exactly, verified internals 5/7, wrapper-callee internals + 21/29, but only **~1% of generated `wrapper_*` addresses** transfer. Never + move wrapper names by address arithmetic; re-resolve them with + `cudatool.py wrappers` and re-run `annotate_corpus.py`. +4. **Full analysis when needed.** For a feature/reshuffle port run + `ghidra_full_analysis.sh`, `ghidra_export_index.sh`, + `ghidra_batch_decompile.sh`, `ghidra_index_merge.sh` per + `userspace/cuda/docs/07-version-upgrade-playbook.md` §2, then + `diff_corpus.py` for the measured corpus delta. +5. **Transfer names and knowledge.** Apply the map with `ApplyNameMap.java` + `--dry-run` first, then for real; translate the subsystem index; re-validate + entries the delta marked as moved. +6. **Build the green-eye symbol DB** for the new version so runtime addresses + resolve: + + ```bash + cd debug/risc-v-gsp-rm-vm + ./greeneye.sh symbols export \ + --names ../../userspace/cuda//annotations/names.tsv \ + --index ../../userspace/cuda//run/analysis/decompiled/decompiled_index.tsv \ + --callgraph ../../userspace/cuda//run/analysis/elf-index/callgraph.tsv \ + --bias 0x100000 -o symbols-.json + ./greeneye.sh symbols resolve --db symbols-.json 0x + ``` + + `--bias 0x100000` is the Ghidra image base convention for `libcuda` + (docs/02 §6); use `0` for firmware linked at its load address. +7. **Dynamic confirmation on the emulated stack.** Compile and run `cdt` + (`tools/test/cuda_drv_test.c`), with the PTRACE ioctl tracer attached, on + the new userspace version against the **same** emulated firmware profile: + every `CHECK(...)` line and the final `ALL OK` is a per-entry-point + assertion (`cuInit`, `cuDeviceGetCount`, `cuDeviceGet`, `cuCtxCreate`, + `cuMemAlloc`, `cuMemcpyHtoD`, `cuMemcpyDtoH`, `cuMemFree`, `cuCtxDestroy`). + Then run the runtime workload end-to-end: + + ```bash + tools/cuda-ab/ab.sh all # host oracle + guest run + compare + ``` + + `DEVICE` lines are normalised; `RESULT`/`STATUS` must match exactly. Any + new or reordered control in the ioctl trace is the interesting delta. +8. **Compare across versions.** Diff `delta.txt`, the `fdematch` report, the + corpus diff, and the ioctl/control sequences old vs new. Where the firmware + side is recorded, use the session compare primitive: + + ```bash + ./greeneye.sh compare dumps/ dumps/ --symbols + ``` + + For multi-target orchestration use the portability matrix (docs/18); note + it targets firmware ELFs, not `libcuda`. +9. **Only then touch the emulation** — and only if the behaviour diff requires + it, with the docs/07 P4 escalation rule: any change that would change + emulation behaviour is reproduced against the physical/vfio-mdev reference + first. +10. **Update documents**: the per-version decompilation map, `names.tsv` / + `subsystems.tsv`, the version-upgrade playbook notes, and the interface + doc only if controls/capability bits changed. + +### 2.4 Acceptance evidence (CUDA userspace port) + +* [ ] `delta` report (exports added/removed/moved, `.text` growth, build-id); +* [ ] `fdematch` transfer table with counts per source class; +* [ ] wrapper re-resolution evidence; no wrapper transferred by address; +* [ ] `ApplyNameMap.java --dry-run` then real run; corpus diff report; +* [ ] `symbols-.json` with provenance/bias and spot-check resolutions; +* [ ] `cdt` output ending `ALL OK`, with the ioctl trace attached; +* [ ] `ab.sh` host/guest comparison with `RESULT`/`STATUS` matched; +* [ ] ioctl/control diff old vs new explained line by line; +* [ ] firmware anchor unchanged (or explicitly changed in the same PR); +* [ ] M1–M6 table when firmware-side behaviour moved (§4); +* [ ] docs updated; unexplained-delta list empty (empty is a claim). + +## 3. Joint workflow B — new GPU device (RTX 5090 / GB10-class) + +### 3.1 Device profile collection + +| Field | Where it comes from | Artifact | +|---|---|---| +| PCI IDs | `lspci`, vendor docs, `config/hardware/*.yaml` (design target) | device profile | +| BAR layout / sizes | guest `lspci -vvv`, host sysfs | device profile | +| FB geometry (VRAM, BAR1) | host oracle `rmtool mem stats --format=json` | capture in session | +| Engine inventory | host oracle `rmtool engine list --format=json`, guest `guest_engines.c` | inventory table | +| Interrupt topology | MSI-X tables, guest `lspci -vvv` | profile notes | +| ROM/VBIOS quirks | ROM read, laptop/Surface variants | profile overrides | +| Probe-time register values | guest driver init trace (`gsp-resman-vfio:`) | expected-value list | + +The host oracle (`rmharness`, docs/17) is read-only and safe alongside a live +CUDA job; capture it into a session so the numbers carry SHA-256 provenance: + +```bash +cd debug/risc-v-gsp-rm-vm +./rmharness/harness.sh --session dumps/real- --name before \ + -- rmtool engine list --format=json +./rmharness/harness.sh --session dumps/real- -- rmtool mem stats --format=json +``` + +### 3.2 Firmware pairing first + +Verify the image family (TU10x / GH100 / LS10 / `gb10`), the FMC partition +manifest and the version string **before** debugging anything. A mismatched +pairing presents as an emulation bug — the project learned this with 570.144 +vs 570.211. Record firmware hash, family, version string and pairing notes at +the top of the evidence pack. + +### 3.3 Probe first-touch diff and symbol inventory + +```bash +./greeneye.sh symbols export --index --bias 0 -o symbols-.json +./greeneye.sh probe --symbols symbols-.json --record +./greeneye.sh probe --symbols symbols-.json --record +``` + +The probe loads the segments, runs bounded and reports the first divergence +(faulting instruction, access address/size, stopped PC annotated from the +symbol DB). Raw `.fwimage` targets fault early by design — that first-touch +map is the data, not a failure. Changed first-touch addresses across devices +usually point at profile data (register windows, boot args), not logic +(docs/07 P2.2). The symbol inventory is the merged ELF/Ghidra DB per target; +provenance (export file, bias) is recorded in the DB meta. + +### 3.4 Device-model implications (H2/H5) + +What the probe reveals feeds the roadmap, not the product directly: + +* **H2 (platform device model)** — falcon/SEC2 windows, CSB/TRANSCOM mailboxes, + WPR2/FWSEC read-back state, PTIMER, PMU stub, DMA/IOMMU. A device that + touches a different first register set changes the modelled register map + before the image can reach its message loop. +* **H5 (adjacent microcontrollers)** — PMU/SEC2/ACR/SOE mailbox protocols and + boot ordering; a new device family may change identity/clock provisioning or + the Init → ACR → RM boot sequence. +* Entry gate for device-model work is the portability task itself (docs/08 R1); + until then use probe + rehosted uproc paths. + +### 3.5 Folding into the hardware-software matrix + +The project's support ladder is `docs/06-portability/hardware-software-matrix.md` +(Reference / Supported / Experimental / Target / Out of scope). A device is +"tested and documented", not "should work". The fold-in: + +1. Run `harness/portability_matrix.py` across the reference and candidate + device firmware with the same scenario options (docs/18 §1); keep + `/portability-report.json`, per-pair compare reports and sessions. +2. The planned version-diff/CI pipeline (docs/21-CI-RELEASE-PIPELINE.md, + planned) is where those reports become release-gated artifacts; until it + exists, attach them to the matrix process manually. +3. Update the device profile (documentation today; `config/hardware/*.yaml` is + not read by code — do not claim configuration-driven portability). +4. Validate on the physical device with the supervised harness before claiming + support; simulator-only evidence never justifies the claim (docs/06 §6). + +### 3.6 aarch64 host / GB10-class design notes + +* **Host tooling** — green-eye is Python + QEMU; on aarch64 install + `qemu-system-misc` (provides `qemu-system-riscv64`) and the same commands + run. Keep host-specific paths out of session metadata (docs/07 P3.5). + `GREENEYE_QEMU` is the existing escape hatch if the binary lives elsewhere. +* **Firmware family** — GB10-class devices pair a different firmware family + (`gb10`/GB100-era) than TU104's `57X-GSP`; treat the fwimage, FMC manifest + and PMU/SOE images as a new family, not a re-badged TU10x image. +* **Boot/partition model** — expect the newer FMC/manifest partition model and, + for confidential-compute paths, the Init → ACR → RM flow; the rehost shim + needs the matching entry/return convention (docs/09 §4, docs/07 P3). +* **PMU/SOE deltas** — identity and clock provisioning flow through PMU/SOE; + any change there invalidates prior environment assumptions and must be + recorded before probe results are interpreted. +* **ISA/core** — confirm the image is still RV64IMAC or has moved to RV64GC + (`file`/`readelf`); update rehost `-march` flags accordingly. QEMU supports + RV64GC even where the bundled demo uses IMAC. +* **Memory model** — Grace/GB10 coherence and page-size/cache-attribute + differences are memory-model risks (hardware-software-matrix §8); test on + aarch64 early rather than assuming x86 results transfer. + +## 4. Evidence and metrics contract for any port + +The metric contract is `docs/06-ACCURACY.md` §5; it is authoritative for every +port type: + +| # | Metric | How it is measured in a port | +|---|---|---| +| M1 | Inventory coverage | observed controls/RPCs vs emulated handled set; host oracle inventory | +| M2 | Reply agreement | field-by-field reply diff, canonical-size and consumed-field checks | +| M3 | Ordering agreement | sequence alignment of the control/event stream (LCS/edit distance) | +| M4 | State agreement | firmware state transitions vs the emulation state model | +| M5 | Error-path agreement | injected faults take the same status/notifier path | +| M6 | Memory side-effects | read/write map vs shadow state; snapshot hashes at fixed points | +| M7 | Timing | **not claimed**; order-of-events budgets only | + +Rules: at least an A/B pair per conclusion; every claim carries its fidelity +level (L0–L3) and provenance; a simulator observation is evidence about the +firmware, not silicon; any product change is confirmed against a physical +capture first. The evidence pack is docs/07's checklist plus docs/14 §6. + +**Checklist per port type** + +CUDA userspace (A): +* [ ] §2.4 items complete; firmware anchor documented. + +GSP firmware version (P1): +* [ ] firmware hash + version string + pairing notes; +* [ ] symbol DB hash and Ghidra export provenance; +* [ ] probe first-touch old vs new, every changed address explained; +* [ ] portability matrix report + per-pair `compare`; +* [ ] M1–M6 table with reference-capture method and L2/L3 limits; +* [ ] existing guest ladder has no new regression; +* [ ] every new/changed control has a version-keyed handler or explicit + "not used" note. + +Device (P2): +* [ ] profile fields (§3.1) recorded with host-oracle captures; +* [ ] pairing verified before any debugging; +* [ ] probe first-touch diff explained as profile vs logic; +* [ ] engine inventory complete and validated against the host oracle; +* [ ] matrix entry and evidence published; physical validation done before the + support claim. + +Architecture (P3): +* [ ] ISA/image shape confirmed (`file`/`readelf`), rehost flags updated; +* [ ] partition/boot model identified; shim convention matched; +* [ ] PMU/SOE deltas recorded; host portability verified on aarch64; +* [ ] all P2 items. + +## 5. Common failure modes and decided responses + +| Failure mode | Signature | Decided response | Source | +|---|---|---|---| +| Mismatched guest driver ↔ firmware pairing | failures indistinguishable from emulation bugs (e.g. 570.144 image vs 570.211 driver) | verify hash/version string/pairing before debugging; fix the pairing, not the emulation | docs/07 P2.2; docs/05-operations §4.1 | +| Runfile/metapackage version drift | a different major resolves than requested | check the candidate version; prefer the exact-version runfile | docs/05-operations §4.1 pitfalls | +| Silent control-shape change | message layout changes break the emulator with no error | M2 canonical-size + consumed-field checks; per-control probes; decode against the exact-version headers | docs/05-operations §4.1 table | +| Engine-inventory change blocking context creation | `cuCtxCreate` fails because an expected engine (e.g. `COPY14`) is absent | complete the inventory per device/driver profile, validate against `rmtool engine list`; treat inventory as profile data | hardware-software-matrix §6.2 | +| Wrapper re-layout on a patch bump | wrapper names resolve to wrong addresses | transfer exports/internals only; re-resolve wrappers with `cudatool.py wrappers`; never add a delta to old wrapper addresses | userspace/cuda/docs/07 §3 | +| Firmware address drift per build | stale Ghidra addresses produce nonsense | re-derive addresses per firmware build; never carry addresses into a new configuration | docs/05-operations §4.1 pitfalls | +| Host driver assumed to be in lock-step | A/B oracle "updated" and treated as the target | host distro driver is the oracle only; the anchor is guest driver ↔ emulated firmware | docs/05-operations §4.1 | +| Simulator-only divergence | `compare` delta that physical capture cannot reproduce | record as a fidelity gap; do not change the product from it | docs/06 §6; docs/07 P4.5 | +| QEMU missing on an aarch64 host | `qemu-system-riscv64 not found` | install `qemu-system-misc` or set `GREENEYE_QEMU` | docs/02 §9; docs/07 P3.5 | + +## 6. Roadmap hooks + +| Port activity | Roadmap item | What it adds | Gate | +|---|---|---|---| +| First-touch map and boot of real firmware | H2 / R1 | device model so ports boot beyond the first unmapped access | portability/accuracy task requires it | +| Semantic diff of dispatch/handler/control sets | H4.1 | automated portability delta report; per-version symbol corpora | H1+H2 | +| Capability inference across new devices | H4.2 | access map vs capability model; least-privilege findings | watchpoints (done); TCG-session fold (R4) | +| Mutation corpus for new input surfaces | H4.3 / R5 | fail-closed rules validated per version | R1/R2 | +| Differential execution and reimplementation loop | H4.5 | every port lands with A/B sessions and the M1–M6 table | reference-capture pipeline | +| Adjacent-controller work for GB10-class bring-up | H5 | PMU/SEC2/ACR/SOE models, cross-controller sessions | H2.2 | +| Multi-target orchestration | R6 / docs/18 | one report across firmware/device targets | v0.1 stable | +| Version diff and release gating | planned (docs/21) | CI pipeline that gates a port on the evidence bundle | pipeline implemented | +| Control discovery | planned (docs/19) | systematic control/class inventory across versions | document exists | + +Program-level success criterion served here: a new GSP version (or device) can +be triaged — inventory diff + behaviour diff + metric table — in **hours**, +with recorded evidence. -- cgit v1.2.3