# 22 — Porting with risc-v-gsp-rm-debugger: CUDA userspace, GSP firmware and new GPU devices > Status: implemented, documented and tested (audit: docs/27-TOOL-AUDIT.md; value demos: docs/25-VALUE-VALIDATION.md). > This is the combined porting guide for the three churn axes the project > faces. It joins the existing CUDA/GSP/firmware procedures with risc-v-gsp-rm-debugger's > runtime layer. Items marked "planned (docs/xx)" do not exist yet; do not > invent commands for them. ## 1. The three churn axes and the anchor discipline NVIDIA ships three things on independent schedules, and any one of them can break `vgsp` while the other two stay still: | Axis | Examples | Primary evidence path | |---|---|---| | **CUDA userspace version** (`libcuda`) | 570.144 → 570.211.01; future 580/610 families | CUDA corpus tooling + guest smoke (`userspace/cuda/docs/07`) | | **GSP firmware / RM version** | 570.144 → 570.211 GSP image; RM dispatcher and control-set changes | `./risc-v-gsp-rm-debugger.sh probe` + `compare`, host oracle (`docs/05-operations/02` §4.1) | | **GPU device / architecture** | TU104 → GB202 (RTX 5090), GB10 (Grace-Blackwell), aarch64 hosts | Device profile + pairing check + portability matrix (docs/07 P2/P3, docs/18) | They interact: * a userspace bump usually implies an emulation-anchor decision (guest driver and emulated firmware move together) even when the firmware binary is byte-identical; * a device change can drag a different firmware family (TU10x vs GB100 vs `gb10`) and a different partition/boot model (FMC/MP-SK; Init → ACR → RM for confidential compute) with it; * an architecture change (aarch64 host, newer NV-RISCV core) can invalidate environment assumptions that the other two axes assume are stable (PMU/SOE provisioning, coherent memory, ISA extensions). **Anchor discipline.** The lock-step that must stay coherent is the **guest driver ↔ emulated firmware profile** (the GSP generation whose protocol the host emulates). The host's distribution driver is **not** part of that lock-step: it is the oracle used for A/B comparisons, and the optional OpenRM host modules are only for native vGPU-management workflows. The current anchor is **570.144**; a version bump moves the guest driver and the emulation, not the host distro driver (see `docs/05-operations/02-debugging-and-maintenance-workflows.md` §4.1). Porting rules that follow: 1. One anchor per port; never evaluate a new `libcuda` against an old emulated firmware profile and call a mismatch an emulation bug. 2. Every input is versioned and hashed: `firmware/-GSP//`, `userspace/cuda//`, `config/versions/.yaml`. The `config/` YAML files are **design targets, not live configuration** until the module load path reads them (`docs/06-portability/hardware-software-matrix.md` §6). 3. New versions live in parallel directories; the previous anchor stays untouched so rollback never destroys a working setup. 4. A port is a claim with evidence attached (M1–M6, §4), not a merged patch. ## 2. Joint workflow A — CUDA userspace version port (`libcuda` bump) ### 2.1 Decide what kind of port it is (minutes) Run the cheap triage before any decompilation or emulation work: ```bash cd userspace/cuda tools/cudatool.py info /run/libcuda.so. tools/cudatool.py delta /run/libcuda.so. /run/libcuda.so. > /run/delta.txt ``` | `delta` reading | Port class | What it implies | |---|---|---| | no export moves, small `.text` growth | patch | transfer the name map and spot-check touched subsystems | | new/removed exports | feature | decode the new API surface before touching the emulation | | many moves | reshuffle (toolchain/linker) | plan a full re-analysis; a fraction of names will not transfer | ### 2.2 Static analysis vs dynamic confirmation | Step | Existing project tooling | risc-v-gsp-rm-debugger addition | |---|---|---| | Export/interface diff | `tools/cudatool.py delta` | — | | Function matching / name-transfer measurement | `tools/cudatool.py fdematch --names ` | — | | Corpus-level diff (added/removed/resized functions) | `tools/diff_corpus.py ` | — | | Name transfer + corpus annotation | `ApplyNameMap.java` (`--dry-run` first), `tools/annotate_corpus.py` | `./risc-v-gsp-rm-debugger.sh symbols export` builds a runtime-address symbol DB from the same index/names | | Driver-API smoke | `tools/test/cuda_drv_test.c` compiled as `/tmp/cdt` (prints `ALL OK`) | `./risc-v-gsp-rm-debugger.sh symbols resolve --db ... ` spot-checks transferred addresses per call path | | Runtime workload | `tools/cuda-ab/ab.sh all` (`vector_add.cu`; `RESULT`/`STATUS` must match host) | session-format evidence and `./risc-v-gsp-rm-debugger.sh compare` when firmware-side sessions exist | | Guest-side control observation | `tools/guest-diag/ioctl_trace.c`, host module trace (`vgsp:`) | annotation of the firmware-side RPC path with the per-version symbol DB | | Firmware-side confirmation | — | `./risc-v-gsp-rm-debugger.sh probe` / `record` / `compare` against the paired firmware | **Be honest about the execution boundary.** `cdt` and `vectorAdd` are x86_64 guest binaries; the RISC-V simulator does not execute them. risc-v-gsp-rm-debugger's contribution to a userspace port is the **symbol/evidence layer**: a runtime-address DB per `libcuda` version, per-address resolution checks for transferred names, and a recorded comparison format for the GSP-side behaviour the driver drives. The print-and-verify path (`cdt`, `ab.sh`, ioctl trace) remains the dynamic oracle. ### 2.3 Step-by-step runbook 1. **Acquire and version.** Fetch the new `libcuda.so.` (`570.144/scripts/fetch_from_guest.sh`, or the driver package with `--extract-only`). Work in a new `userspace/cuda//` tree copied from `570.144/`; never overwrite an analysed version. 2. **Cheap triage.** `cudatool.py info` + `delta`; classify the port (§2.1); record `.text` growth, build-id and ABI note. 3. **Measure transfer before copying.** Run `cudatool.py fdematch` with the old `names.tsv`. The known result from 570.144 → 570.211.01: all 659 exports transfer exactly, verified internals 5/7, wrapper-callee internals 21/29, but only **~1% of generated `wrapper_*` addresses** transfer. Never move wrapper names by address arithmetic; re-resolve them with `cudatool.py wrappers` and re-run `annotate_corpus.py`. 4. **Full analysis when needed.** For a feature/reshuffle port run `ghidra_full_analysis.sh`, `ghidra_export_index.sh`, `ghidra_batch_decompile.sh`, `ghidra_index_merge.sh` per `userspace/cuda/docs/07-version-upgrade-playbook.md` §2, then `diff_corpus.py` for the measured corpus delta. 5. **Transfer names and knowledge.** Apply the map with `ApplyNameMap.java` `--dry-run` first, then for real; translate the subsystem index; re-validate entries the delta marked as moved. 6. **Build the risc-v-gsp-rm-debugger symbol DB** for the new version so runtime addresses resolve: ```bash cd debug/risc-v-gsp-rm-vm ./risc-v-gsp-rm-debugger.sh symbols export \ --names ../../userspace/cuda//annotations/names.tsv \ --index ../../userspace/cuda//run/analysis/decompiled/decompiled_index.tsv \ --callgraph ../../userspace/cuda//run/analysis/elf-index/callgraph.tsv \ --bias 0x100000 -o symbols-.json ./risc-v-gsp-rm-debugger.sh symbols resolve --db symbols-.json 0x ``` `--bias 0x100000` is the Ghidra image base convention for `libcuda` (docs/02 §6); use `0` for firmware linked at its load address. 7. **Dynamic confirmation on the emulated stack.** Compile and run `cdt` (`tools/test/cuda_drv_test.c`), with the PTRACE ioctl tracer attached, on the new userspace version against the **same** emulated firmware profile: every `CHECK(...)` line and the final `ALL OK` is a per-entry-point assertion (`cuInit`, `cuDeviceGetCount`, `cuDeviceGet`, `cuCtxCreate`, `cuMemAlloc`, `cuMemcpyHtoD`, `cuMemcpyDtoH`, `cuMemFree`, `cuCtxDestroy`). Then run the runtime workload end-to-end: ```bash tools/cuda-ab/ab.sh all # host oracle + guest run + compare ``` `DEVICE` lines are normalised; `RESULT`/`STATUS` must match exactly. Any new or reordered control in the ioctl trace is the interesting delta. 8. **Compare across versions.** Diff `delta.txt`, the `fdematch` report, the corpus diff, and the ioctl/control sequences old vs new. Where the firmware side is recorded, use the session compare primitive: ```bash ./risc-v-gsp-rm-debugger.sh compare dumps/ dumps/ --symbols ``` For multi-target orchestration use the portability matrix (docs/18); note it targets firmware ELFs, not `libcuda`. 9. **Only then touch the emulation** — and only if the behaviour diff requires it, with the docs/07 P4 escalation rule: any change that would change emulation behaviour is reproduced against the physical/vfio-mdev reference first. 10. **Update documents**: the per-version decompilation map, `names.tsv` / `subsystems.tsv`, the version-upgrade playbook notes, and the interface doc only if controls/capability bits changed. ### 2.4 Acceptance evidence (CUDA userspace port) * [ ] `delta` report (exports added/removed/moved, `.text` growth, build-id); * [ ] `fdematch` transfer table with counts per source class; * [ ] wrapper re-resolution evidence; no wrapper transferred by address; * [ ] `ApplyNameMap.java --dry-run` then real run; corpus diff report; * [ ] `symbols-.json` with provenance/bias and spot-check resolutions; * [ ] `cdt` output ending `ALL OK`, with the ioctl trace attached; * [ ] `ab.sh` host/guest comparison with `RESULT`/`STATUS` matched; * [ ] ioctl/control diff old vs new explained line by line; * [ ] firmware anchor unchanged (or explicitly changed in the same PR); * [ ] M1–M6 table when firmware-side behaviour moved (§4); * [ ] docs updated; unexplained-delta list empty (empty is a claim). ## 3. Joint workflow B — new GPU device (RTX 5090 / GB10-class) ### 3.1 Device profile collection | Field | Where it comes from | Artifact | |---|---|---| | PCI IDs | `lspci`, vendor docs, `config/hardware/*.yaml` (design target) | device profile | | BAR layout / sizes | guest `lspci -vvv`, host sysfs | device profile | | FB geometry (VRAM, BAR1) | host oracle `rmtool mem stats --format=json` | capture in session | | Engine inventory | host oracle `rmtool engine list --format=json`, guest `guest_engines.c` | inventory table | | Interrupt topology | MSI-X tables, guest `lspci -vvv` | profile notes | | ROM/VBIOS quirks | ROM read, laptop/Surface variants | profile overrides | | Probe-time register values | guest driver init trace (`vgsp:`) | expected-value list | The host oracle (`rmharness`, docs/17) is read-only and safe alongside a live CUDA job; capture it into a session so the numbers carry SHA-256 provenance: ```bash cd debug/risc-v-gsp-rm-vm ./rmharness/harness.sh --session dumps/real- --name before \ -- rmtool engine list --format=json ./rmharness/harness.sh --session dumps/real- -- rmtool mem stats --format=json ``` ### 3.2 Firmware pairing first Verify the image family (TU10x / GH100 / LS10 / `gb10`), the FMC partition manifest and the version string **before** debugging anything. A mismatched pairing presents as an emulation bug — the project learned this with 570.144 vs 570.211. Record firmware hash, family, version string and pairing notes at the top of the evidence pack. ### 3.3 Probe first-touch diff and symbol inventory ```bash ./risc-v-gsp-rm-debugger.sh symbols export --index --bias 0 -o symbols-.json ./risc-v-gsp-rm-debugger.sh probe --symbols symbols-.json --record ./risc-v-gsp-rm-debugger.sh probe --symbols symbols-.json --record ``` The probe loads the segments, runs bounded and reports the first divergence (faulting instruction, access address/size, stopped PC annotated from the symbol DB). Raw `.fwimage` targets fault early by design — that first-touch map is the data, not a failure. Changed first-touch addresses across devices usually point at profile data (register windows, boot args), not logic (docs/07 P2.2). The symbol inventory is the merged ELF/Ghidra DB per target; provenance (export file, bias) is recorded in the DB meta. ### 3.4 Device-model implications (H2/H5) What the probe reveals feeds the roadmap, not the product directly: * **H2 (platform device model)** — falcon/SEC2 windows, CSB/TRANSCOM mailboxes, WPR2/FWSEC read-back state, PTIMER, PMU stub, DMA/IOMMU. A device that touches a different first register set changes the modelled register map before the image can reach its message loop. * **H5 (adjacent microcontrollers)** — PMU/SEC2/ACR/SOE mailbox protocols and boot ordering; a new device family may change identity/clock provisioning or the Init → ACR → RM boot sequence. * Entry gate for device-model work is the portability task itself (docs/08 R1); until then use probe + rehosted uproc paths. ### 3.5 Folding into the hardware-software matrix The project's support ladder is `docs/06-portability/hardware-software-matrix.md` (Reference / Supported / Experimental / Target / Out of scope). A device is "tested and documented", not "should work". The fold-in: 1. Run `harness/portability_matrix.py` across the reference and candidate device firmware with the same scenario options (docs/18 §1); keep `/portability-report.json`, per-pair compare reports and sessions. 2. The version-diff/CI pipeline ([`docs/21`](21-CI-RELEASE-PIPELINE.md)) packages those reports into release-gated artifacts; the porting scaffolds ([`docs/28`](28-PORTING-AUTOMATION.md)) wrap it per version as unadmitted pilots (evaluation in future-eng-ops F2/F3). 3. Update the device profile (documentation today; `config/hardware/*.yaml` is not read by code — do not claim configuration-driven portability). 4. Validate on the physical device with the supervised harness before claiming support; simulator-only evidence never justifies the claim (docs/06 §6). ### 3.6 aarch64 host / GB10-class design notes * **Host tooling** — risc-v-gsp-rm-debugger is Python + QEMU; on aarch64 install `qemu-system-misc` (provides `qemu-system-riscv64`) and the same commands run. Keep host-specific paths out of session metadata (docs/07 P3.5). `GREENEYE_QEMU` is the existing escape hatch if the binary lives elsewhere. * **Firmware family** — GB10-class devices pair a different firmware family (`gb10`/GB100-era) than TU104's `57X-GSP`; treat the fwimage, FMC manifest and PMU/SOE images as a new family, not a re-badged TU10x image. * **Boot/partition model** — expect the newer FMC/manifest partition model and, for confidential-compute paths, the Init → ACR → RM flow; the rehost shim needs the matching entry/return convention (docs/09 §4, docs/07 P3). * **PMU/SOE deltas** — identity and clock provisioning flow through PMU/SOE; any change there invalidates prior environment assumptions and must be recorded before probe results are interpreted. * **ISA/core** — confirm the image is still RV64IMAC or has moved to RV64GC (`file`/`readelf`); update rehost `-march` flags accordingly. QEMU supports RV64GC even where the bundled demo uses IMAC. * **Memory model** — Grace/GB10 coherence and page-size/cache-attribute differences are memory-model risks (hardware-software-matrix §8); test on aarch64 early rather than assuming x86 results transfer. ## 4. Evidence and metrics contract for any port The metric contract is `docs/06-ACCURACY.md` §5; it is authoritative for every port type: | # | Metric | How it is measured in a port | |---|---|---| | M1 | Inventory coverage | observed controls/RPCs vs emulated handled set; host oracle inventory | | M2 | Reply agreement | field-by-field reply diff, canonical-size and consumed-field checks | | M3 | Ordering agreement | sequence alignment of the control/event stream (LCS/edit distance) | | M4 | State agreement | firmware state transitions vs the emulation state model | | M5 | Error-path agreement | injected faults take the same status/notifier path | | M6 | Memory side-effects | read/write map vs shadow state; snapshot hashes at fixed points | | M7 | Timing | **not claimed**; order-of-events budgets only | Rules: at least an A/B pair per conclusion; every claim carries its fidelity level (L0–L3) and provenance; a simulator observation is evidence about the firmware, not silicon; any product change is confirmed against a physical capture first. The evidence pack is docs/07's checklist plus docs/14 §6. **Checklist per port type** CUDA userspace (A): * [ ] §2.4 items complete; firmware anchor documented. GSP firmware version (P1): * [ ] firmware hash + version string + pairing notes; * [ ] symbol DB hash and Ghidra export provenance; * [ ] probe first-touch old vs new, every changed address explained; * [ ] portability matrix report + per-pair `compare`; * [ ] M1–M6 table with reference-capture method and L2/L3 limits; * [ ] existing guest ladder has no new regression; * [ ] every new/changed control has a version-keyed handler or explicit "not used" note. Device (P2): * [ ] profile fields (§3.1) recorded with host-oracle captures; * [ ] pairing verified before any debugging; * [ ] probe first-touch diff explained as profile vs logic; * [ ] engine inventory complete and validated against the host oracle; * [ ] matrix entry and evidence published; physical validation done before the support claim. Architecture (P3): * [ ] ISA/image shape confirmed (`file`/`readelf`), rehost flags updated; * [ ] partition/boot model identified; shim convention matched; * [ ] PMU/SOE deltas recorded; host portability verified on aarch64; * [ ] all P2 items. ## 5. Common failure modes and decided responses | Failure mode | Signature | Decided response | Source | |---|---|---|---| | Mismatched guest driver ↔ firmware pairing | failures indistinguishable from emulation bugs (e.g. 570.144 image vs 570.211 driver) | verify hash/version string/pairing before debugging; fix the pairing, not the emulation | docs/07 P2.2; docs/05-operations §4.1 | | Runfile/metapackage version drift | a different major resolves than requested | check the candidate version; prefer the exact-version runfile | docs/05-operations §4.1 pitfalls | | Silent control-shape change | message layout changes break the emulator with no error | M2 canonical-size + consumed-field checks; per-control probes; decode against the exact-version headers | docs/05-operations §4.1 table | | Engine-inventory change blocking context creation | `cuCtxCreate` fails because an expected engine (e.g. `COPY14`) is absent | complete the inventory per device/driver profile, validate against `rmtool engine list`; treat inventory as profile data | hardware-software-matrix §6.2 | | Wrapper re-layout on a patch bump | wrapper names resolve to wrong addresses | transfer exports/internals only; re-resolve wrappers with `cudatool.py wrappers`; never add a delta to old wrapper addresses | userspace/cuda/docs/07 §3 | | Firmware address drift per build | stale Ghidra addresses produce nonsense | re-derive addresses per firmware build; never carry addresses into a new configuration | docs/05-operations §4.1 pitfalls | | Host driver assumed to be in lock-step | A/B oracle "updated" and treated as the target | host distro driver is the oracle only; the anchor is guest driver ↔ emulated firmware | docs/05-operations §4.1 | | Simulator-only divergence | `compare` delta that physical capture cannot reproduce | record as a fidelity gap; do not change the product from it | docs/06 §6; docs/07 P4.5 | | QEMU missing on an aarch64 host | `qemu-system-riscv64 not found` | install `qemu-system-misc` or set `GREENEYE_QEMU` | docs/02 §9; docs/07 P3.5 | ## 6. Roadmap hooks | Port activity | Roadmap item | What it adds | Gate | |---|---|---|---| | First-touch map and boot of real firmware | H2 / R1 | device model so ports boot beyond the first unmapped access | portability/accuracy task requires it | | Semantic diff of dispatch/handler/control sets | H4.1 | automated portability delta report; per-version symbol corpora | H1+H2 | | Capability inference across new devices | H4.2 | access map vs capability model; least-privilege findings | watchpoints (done); TCG-session fold (R4) | | Mutation corpus for new input surfaces | H4.3 / R5 | fail-closed rules validated per version | R1/R2 | | Differential execution and reimplementation loop | H4.5 | every port lands with A/B sessions and the M1–M6 table | reference-capture pipeline | | Adjacent-controller work for GB10-class bring-up | H5 | PMU/SEC2/ACR/SOE models, cross-controller sessions | H2.2 | | Multi-target orchestration | R6 / docs/18 | one report across firmware/device targets | v0.1 stable | | Version diff and release gating | implemented (docs/21) | CI pipeline that gates a port on the evidence bundle | pipeline implemented | | Control discovery | implemented (docs/19) | systematic control/class inventory across versions | document exists | Program-level success criterion served here: a new GSP version (or device) can be triaged — inventory diff + behaviour diff + metric table — in **hours**, with recorded evidence.