path: root/debug/risc-v-gsp-rm-vm/docs/22-PORTING-CUDA-AND-DEVICES.md

# 22 — Porting with green-eye: CUDA userspace, GSP firmware and new GPU devices

> **# TOOL IN DEVELOPMENT - NOT READY YET!** See 00-OVERVIEW for status.
> This is the combined porting guide for the three churn axes the project
> faces. It joins the existing CUDA/GSP/firmware procedures with green-eye's
> runtime layer. Items marked "planned (docs/xx)" do not exist yet; do not
> invent commands for them.

## 1. The three churn axes and the anchor discipline

NVIDIA ships three things on independent schedules, and any one of them can
break `open-iov-vfio` while the other two stay still:

| Axis | Examples | Primary evidence path |
|---|---|---|
| **CUDA userspace version** (`libcuda`) | 570.144 → 570.211.01; future 580/610 families | CUDA corpus tooling + guest smoke (`userspace/cuda/docs/07`) |
| **GSP firmware / RM version** | 570.144 → 570.211 GSP image; RM dispatcher and control-set changes | `greeneye probe` + `compare`, host oracle (`docs/05-operations/02` §4.1) |
| **GPU device / architecture** | TU104 → GB202 (RTX 5090), GB10 (Grace-Blackwell), aarch64 hosts | Device profile + pairing check + portability matrix (docs/07 P2/P3, docs/18) |

They interact:

* a userspace bump usually implies an emulation-anchor decision (guest driver
  and emulated firmware move together) even when the firmware binary is
  byte-identical;
* a device change can drag a different firmware family (TU10x vs GB100 vs
  `gb10`) and a different partition/boot model (FMC/MP-SK; Init → ACR → RM for
  confidential compute) with it;
* an architecture change (aarch64 host, newer NV-RISCV core) can invalidate
  environment assumptions that the other two axes assume are stable
  (PMU/SOE provisioning, coherent memory, ISA extensions).

**Anchor discipline.** The lock-step that must stay coherent is the
**guest driver ↔ emulated firmware profile** (the GSP generation whose protocol
the host emulates). The host's distribution driver is **not** part of that
lock-step: it is the oracle used for A/B comparisons, and the optional OpenRM
host modules are only for native vGPU-management workflows. The current anchor
is **570.144**; a version bump moves the guest driver and the emulation, not
the host distro driver (see `docs/05-operations/02-debugging-and-maintenance-workflows.md`
§4.1). Porting rules that follow:

1. One anchor per port; never evaluate a new `libcuda` against an old emulated
   firmware profile and call a mismatch an emulation bug.
2. Every input is versioned and hashed: `firmware/<SERIES>-GSP/<VER>/`,
   `userspace/cuda/<version>/`, `config/versions/<MAJOR>.yaml`. The `config/`
   YAML files are **design targets, not live configuration** until the module
   load path reads them (`docs/06-portability/hardware-software-matrix.md` §6).
3. New versions live in parallel directories; the previous anchor stays
   untouched so rollback never destroys a working setup.
4. A port is a claim with evidence attached (M1–M6, §4), not a merged patch.

## 2. Joint workflow A — CUDA userspace version port (`libcuda` bump)

### 2.1 Decide what kind of port it is (minutes)

Run the cheap triage before any decompilation or emulation work:

```bash
cd userspace/cuda
tools/cudatool.py info    <new>/run/libcuda.so.<new>
tools/cudatool.py delta   <old>/run/libcuda.so.<old> <new>/run/libcuda.so.<new> > <new>/run/delta.txt
```

| `delta` reading | Port class | What it implies |
|---|---|---|
| no export moves, small `.text` growth | patch | transfer the name map and spot-check touched subsystems |
| new/removed exports | feature | decode the new API surface before touching the emulation |
| many moves | reshuffle (toolchain/linker) | plan a full re-analysis; a fraction of names will not transfer |

### 2.2 Static analysis vs dynamic confirmation

| Step | Existing project tooling | green-eye addition |
|---|---|---|
| Export/interface diff | `tools/cudatool.py delta` | — |
| Function matching / name-transfer measurement | `tools/cudatool.py fdematch --names <names.tsv>` | — |
| Corpus-level diff (added/removed/resized functions) | `tools/diff_corpus.py <old_index> <new_index>` | — |
| Name transfer + corpus annotation | `ApplyNameMap.java` (`--dry-run` first), `tools/annotate_corpus.py` | `./greeneye.sh symbols export` builds a runtime-address symbol DB from the same index/names |
| Driver-API smoke | `tools/test/cuda_drv_test.c` compiled as `/tmp/cdt` (prints `ALL OK`) | `./greeneye.sh symbols resolve --db ... <addr>` spot-checks transferred addresses per call path |
| Runtime workload | `tools/cuda-ab/ab.sh all` (`vector_add.cu`; `RESULT`/`STATUS` must match host) | session-format evidence and `./greeneye.sh compare` when firmware-side sessions exist |
| Guest-side control observation | `tools/guest-diag/ioctl_trace.c`, host module trace (`gsp-resman-vfio:`) | annotation of the firmware-side RPC path with the per-version symbol DB |
| Firmware-side confirmation | — | `./greeneye.sh probe` / `record` / `compare` against the paired firmware |

**Be honest about the execution boundary.** `cdt` and `vectorAdd` are x86_64
guest binaries; the RISC-V simulator does not execute them. green-eye's
contribution to a userspace port is the **symbol/evidence layer**: a
runtime-address DB per `libcuda` version, per-address resolution checks for
transferred names, and a recorded comparison format for the GSP-side behaviour
the driver drives. The print-and-verify path (`cdt`, `ab.sh`, ioctl trace)
remains the dynamic oracle.

### 2.3 Step-by-step runbook

1. **Acquire and version.** Fetch the new `libcuda.so.<version>`
   (`570.144/scripts/fetch_from_guest.sh`, or the driver package with
   `--extract-only`). Work in a new `userspace/cuda/<new>/` tree copied from
   `570.144/`; never overwrite an analysed version.
2. **Cheap triage.** `cudatool.py info` + `delta`; classify the port (§2.1);
   record `.text` growth, build-id and ABI note.
3. **Measure transfer before copying.** Run `cudatool.py fdematch` with the
   old `names.tsv`. The known result from 570.144 → 570.211.01: all 659
   exports transfer exactly, verified internals 5/7, wrapper-callee internals
   21/29, but only **~1% of generated `wrapper_*` addresses** transfer. Never
   move wrapper names by address arithmetic; re-resolve them with
   `cudatool.py wrappers` and re-run `annotate_corpus.py`.
4. **Full analysis when needed.** For a feature/reshuffle port run
   `ghidra_full_analysis.sh`, `ghidra_export_index.sh`,
   `ghidra_batch_decompile.sh`, `ghidra_index_merge.sh` per
   `userspace/cuda/docs/07-version-upgrade-playbook.md` §2, then
   `diff_corpus.py` for the measured corpus delta.
5. **Transfer names and knowledge.** Apply the map with `ApplyNameMap.java`
   `--dry-run` first, then for real; translate the subsystem index; re-validate
   entries the delta marked as moved.
6. **Build the green-eye symbol DB** for the new version so runtime addresses
   resolve:

   ```bash
   cd debug/risc-v-gsp-rm-vm
   ./greeneye.sh symbols export \
       --names ../../userspace/cuda/<new>/annotations/names.tsv \
       --index ../../userspace/cuda/<new>/run/analysis/decompiled/decompiled_index.tsv \
       --callgraph ../../userspace/cuda/<new>/run/analysis/elf-index/callgraph.tsv \
       --bias 0x100000 -o symbols-<new>.json
   ./greeneye.sh symbols resolve --db symbols-<new>.json 0x<transferred-addr>
   ```

   `--bias 0x100000` is the Ghidra image base convention for `libcuda`
   (docs/02 §6); use `0` for firmware linked at its load address.
7. **Dynamic confirmation on the emulated stack.** Compile and run `cdt`
   (`tools/test/cuda_drv_test.c`), with the PTRACE ioctl tracer attached, on
   the new userspace version against the **same** emulated firmware profile:
   every `CHECK(...)` line and the final `ALL OK` is a per-entry-point
   assertion (`cuInit`, `cuDeviceGetCount`, `cuDeviceGet`, `cuCtxCreate`,
   `cuMemAlloc`, `cuMemcpyHtoD`, `cuMemcpyDtoH`, `cuMemFree`, `cuCtxDestroy`).
   Then run the runtime workload end-to-end:

   ```bash
   tools/cuda-ab/ab.sh all      # host oracle + guest run + compare
   ```

   `DEVICE` lines are normalised; `RESULT`/`STATUS` must match exactly. Any
   new or reordered control in the ioctl trace is the interesting delta.
8. **Compare across versions.** Diff `delta.txt`, the `fdematch` report, the
   corpus diff, and the ioctl/control sequences old vs new. Where the firmware
   side is recorded, use the session compare primitive:

   ```bash
   ./greeneye.sh compare dumps/<old-fw-run> dumps/<new-fw-run> --symbols
   ```

   For multi-target orchestration use the portability matrix (docs/18); note
   it targets firmware ELFs, not `libcuda`.
9. **Only then touch the emulation** — and only if the behaviour diff requires
   it, with the docs/07 P4 escalation rule: any change that would change
   emulation behaviour is reproduced against the physical/vfio-mdev reference
   first.
10. **Update documents**: the per-version decompilation map, `names.tsv` /
    `subsystems.tsv`, the version-upgrade playbook notes, and the interface
    doc only if controls/capability bits changed.

### 2.4 Acceptance evidence (CUDA userspace port)

* [ ] `delta` report (exports added/removed/moved, `.text` growth, build-id);
* [ ] `fdematch` transfer table with counts per source class;
* [ ] wrapper re-resolution evidence; no wrapper transferred by address;
* [ ] `ApplyNameMap.java --dry-run` then real run; corpus diff report;
* [ ] `symbols-<version>.json` with provenance/bias and spot-check resolutions;
* [ ] `cdt` output ending `ALL OK`, with the ioctl trace attached;
* [ ] `ab.sh` host/guest comparison with `RESULT`/`STATUS` matched;
* [ ] ioctl/control diff old vs new explained line by line;
* [ ] firmware anchor unchanged (or explicitly changed in the same PR);
* [ ] M1–M6 table when firmware-side behaviour moved (§4);
* [ ] docs updated; unexplained-delta list empty (empty is a claim).

## 3. Joint workflow B — new GPU device (RTX 5090 / GB10-class)

### 3.1 Device profile collection

| Field | Where it comes from | Artifact |
|---|---|---|
| PCI IDs | `lspci`, vendor docs, `config/hardware/*.yaml` (design target) | device profile |
| BAR layout / sizes | guest `lspci -vvv`, host sysfs | device profile |
| FB geometry (VRAM, BAR1) | host oracle `rmtool mem stats --format=json` | capture in session |
| Engine inventory | host oracle `rmtool engine list --format=json`, guest `guest_engines.c` | inventory table |
| Interrupt topology | MSI-X tables, guest `lspci -vvv` | profile notes |
| ROM/VBIOS quirks | ROM read, laptop/Surface variants | profile overrides |
| Probe-time register values | guest driver init trace (`gsp-resman-vfio:`) | expected-value list |

The host oracle (`rmharness`, docs/17) is read-only and safe alongside a live
CUDA job; capture it into a session so the numbers carry SHA-256 provenance:

```bash
cd debug/risc-v-gsp-rm-vm
./rmharness/harness.sh --session dumps/real-<device> --name before \
    -- rmtool engine list --format=json
./rmharness/harness.sh --session dumps/real-<device> -- rmtool mem stats --format=json
```

### 3.2 Firmware pairing first

Verify the image family (TU10x / GH100 / LS10 / `gb10`), the FMC partition
manifest and the version string **before** debugging anything. A mismatched
pairing presents as an emulation bug — the project learned this with 570.144
vs 570.211. Record firmware hash, family, version string and pairing notes at
the top of the evidence pack.

### 3.3 Probe first-touch diff and symbol inventory

```bash
./greeneye.sh symbols export --index <new-export.tsv> --bias 0 -o symbols-<new>.json
./greeneye.sh probe <new-fwimage> --symbols symbols-<new>.json --record
./greeneye.sh probe <reference-fwimage> --symbols symbols-<ref>.json --record
```

The probe loads the segments, runs bounded and reports the first divergence
(faulting instruction, access address/size, stopped PC annotated from the
symbol DB). Raw `.fwimage` targets fault early by design — that first-touch
map is the data, not a failure. Changed first-touch addresses across devices
usually point at profile data (register windows, boot args), not logic
(docs/07 P2.2). The symbol inventory is the merged ELF/Ghidra DB per target;
provenance (export file, bias) is recorded in the DB meta.

### 3.4 Device-model implications (H2/H5)

What the probe reveals feeds the roadmap, not the product directly:

* **H2 (platform device model)** — falcon/SEC2 windows, CSB/TRANSCOM mailboxes,
  WPR2/FWSEC read-back state, PTIMER, PMU stub, DMA/IOMMU. A device that
  touches a different first register set changes the modelled register map
  before the image can reach its message loop.
* **H5 (adjacent microcontrollers)** — PMU/SEC2/ACR/SOE mailbox protocols and
  boot ordering; a new device family may change identity/clock provisioning or
  the Init → ACR → RM boot sequence.
* Entry gate for device-model work is the portability task itself (docs/08 R1);
  until then use probe + rehosted uproc paths.

### 3.5 Folding into the hardware-software matrix

The project's support ladder is `docs/06-portability/hardware-software-matrix.md`
(Reference / Supported / Experimental / Target / Out of scope). A device is
"tested and documented", not "should work". The fold-in:

1. Run `harness/portability_matrix.py` across the reference and candidate
   device firmware with the same scenario options (docs/18 §1); keep
   `<out>/portability-report.json`, per-pair compare reports and sessions.
2. The planned version-diff/CI pipeline (docs/21-CI-RELEASE-PIPELINE.md,
   planned) is where those reports become release-gated artifacts; until it
   exists, attach them to the matrix process manually.
3. Update the device profile (documentation today; `config/hardware/*.yaml` is
   not read by code — do not claim configuration-driven portability).
4. Validate on the physical device with the supervised harness before claiming
   support; simulator-only evidence never justifies the claim (docs/06 §6).

### 3.6 aarch64 host / GB10-class design notes

* **Host tooling** — green-eye is Python + QEMU; on aarch64 install
  `qemu-system-misc` (provides `qemu-system-riscv64`) and the same commands
  run. Keep host-specific paths out of session metadata (docs/07 P3.5).
  `GREENEYE_QEMU` is the existing escape hatch if the binary lives elsewhere.
* **Firmware family** — GB10-class devices pair a different firmware family
  (`gb10`/GB100-era) than TU104's `57X-GSP`; treat the fwimage, FMC manifest
  and PMU/SOE images as a new family, not a re-badged TU10x image.
* **Boot/partition model** — expect the newer FMC/manifest partition model and,
  for confidential-compute paths, the Init → ACR → RM flow; the rehost shim
  needs the matching entry/return convention (docs/09 §4, docs/07 P3).
* **PMU/SOE deltas** — identity and clock provisioning flow through PMU/SOE;
  any change there invalidates prior environment assumptions and must be
  recorded before probe results are interpreted.
* **ISA/core** — confirm the image is still RV64IMAC or has moved to RV64GC
  (`file`/`readelf`); update rehost `-march` flags accordingly. QEMU supports
  RV64GC even where the bundled demo uses IMAC.
* **Memory model** — Grace/GB10 coherence and page-size/cache-attribute
  differences are memory-model risks (hardware-software-matrix §8); test on
  aarch64 early rather than assuming x86 results transfer.

## 4. Evidence and metrics contract for any port

The metric contract is `docs/06-ACCURACY.md` §5; it is authoritative for every
port type:

| # | Metric | How it is measured in a port |
|---|---|---|
| M1 | Inventory coverage | observed controls/RPCs vs emulated handled set; host oracle inventory |
| M2 | Reply agreement | field-by-field reply diff, canonical-size and consumed-field checks |
| M3 | Ordering agreement | sequence alignment of the control/event stream (LCS/edit distance) |
| M4 | State agreement | firmware state transitions vs the emulation state model |
| M5 | Error-path agreement | injected faults take the same status/notifier path |
| M6 | Memory side-effects | read/write map vs shadow state; snapshot hashes at fixed points |
| M7 | Timing | **not claimed**; order-of-events budgets only |

Rules: at least an A/B pair per conclusion; every claim carries its fidelity
level (L0–L3) and provenance; a simulator observation is evidence about the
firmware, not silicon; any product change is confirmed against a physical
capture first. The evidence pack is docs/07's checklist plus docs/14 §6.

**Checklist per port type**

CUDA userspace (A):
* [ ] §2.4 items complete; firmware anchor documented.

GSP firmware version (P1):
* [ ] firmware hash + version string + pairing notes;
* [ ] symbol DB hash and Ghidra export provenance;
* [ ] probe first-touch old vs new, every changed address explained;
* [ ] portability matrix report + per-pair `compare`;
* [ ] M1–M6 table with reference-capture method and L2/L3 limits;
* [ ] existing guest ladder has no new regression;
* [ ] every new/changed control has a version-keyed handler or explicit
  "not used" note.

Device (P2):
* [ ] profile fields (§3.1) recorded with host-oracle captures;
* [ ] pairing verified before any debugging;
* [ ] probe first-touch diff explained as profile vs logic;
* [ ] engine inventory complete and validated against the host oracle;
* [ ] matrix entry and evidence published; physical validation done before the
  support claim.

Architecture (P3):
* [ ] ISA/image shape confirmed (`file`/`readelf`), rehost flags updated;
* [ ] partition/boot model identified; shim convention matched;
* [ ] PMU/SOE deltas recorded; host portability verified on aarch64;
* [ ] all P2 items.

## 5. Common failure modes and decided responses

| Failure mode | Signature | Decided response | Source |
|---|---|---|---|
| Mismatched guest driver ↔ firmware pairing | failures indistinguishable from emulation bugs (e.g. 570.144 image vs 570.211 driver) | verify hash/version string/pairing before debugging; fix the pairing, not the emulation | docs/07 P2.2; docs/05-operations §4.1 |
| Runfile/metapackage version drift | a different major resolves than requested | check the candidate version; prefer the exact-version runfile | docs/05-operations §4.1 pitfalls |
| Silent control-shape change | message layout changes break the emulator with no error | M2 canonical-size + consumed-field checks; per-control probes; decode against the exact-version headers | docs/05-operations §4.1 table |
| Engine-inventory change blocking context creation | `cuCtxCreate` fails because an expected engine (e.g. `COPY14`) is absent | complete the inventory per device/driver profile, validate against `rmtool engine list`; treat inventory as profile data | hardware-software-matrix §6.2 |
| Wrapper re-layout on a patch bump | wrapper names resolve to wrong addresses | transfer exports/internals only; re-resolve wrappers with `cudatool.py wrappers`; never add a delta to old wrapper addresses | userspace/cuda/docs/07 §3 |
| Firmware address drift per build | stale Ghidra addresses produce nonsense | re-derive addresses per firmware build; never carry addresses into a new configuration | docs/05-operations §4.1 pitfalls |
| Host driver assumed to be in lock-step | A/B oracle "updated" and treated as the target | host distro driver is the oracle only; the anchor is guest driver ↔ emulated firmware | docs/05-operations §4.1 |
| Simulator-only divergence | `compare` delta that physical capture cannot reproduce | record as a fidelity gap; do not change the product from it | docs/06 §6; docs/07 P4.5 |
| QEMU missing on an aarch64 host | `qemu-system-riscv64 not found` | install `qemu-system-misc` or set `GREENEYE_QEMU` | docs/02 §9; docs/07 P3.5 |

## 6. Roadmap hooks

| Port activity | Roadmap item | What it adds | Gate |
|---|---|---|---|
| First-touch map and boot of real firmware | H2 / R1 | device model so ports boot beyond the first unmapped access | portability/accuracy task requires it |
| Semantic diff of dispatch/handler/control sets | H4.1 | automated portability delta report; per-version symbol corpora | H1+H2 |
| Capability inference across new devices | H4.2 | access map vs capability model; least-privilege findings | watchpoints (done); TCG-session fold (R4) |
| Mutation corpus for new input surfaces | H4.3 / R5 | fail-closed rules validated per version | R1/R2 |
| Differential execution and reimplementation loop | H4.5 | every port lands with A/B sessions and the M1–M6 table | reference-capture pipeline |
| Adjacent-controller work for GB10-class bring-up | H5 | PMU/SEC2/ACR/SOE models, cross-controller sessions | H2.2 |
| Multi-target orchestration | R6 / docs/18 | one report across firmware/device targets | v0.1 stable |
| Version diff and release gating | planned (docs/21) | CI pipeline that gates a port on the evidence bundle | pipeline implemented |
| Control discovery | planned (docs/19) | systematic control/class inventory across versions | document exists |

Program-level success criterion served here: a new GSP version (or device) can
be triaged — inventory diff + behaviour diff + metric table — in **hours**,
with recorded evidence.