summaryrefslogtreecommitdiffstats
path: root/userspace/cuda/docs
diff options
context:
space:
mode:
authorArthur Hanson Rasmusson <arthur@vgpu.io>2026-09-23 13:48:00 -0400
committerArthur Hanson Rasmusson <arthur@vgpu.io>2026-09-23 13:48:00 -0400
commitbfcb856e09207d8edb714373d775537165d06b05 (patch)
tree5404762f261ed0e20bee6a5f484865ae2e18d929 /userspace/cuda/docs
parenta93455bc2e3a10fe070477cad5e098e40b4c069a (diff)
downloadopen-iov-bfcb856e09207d8edb714373d775537165d06b05.tar.gz
open-iov-bfcb856e09207d8edb714373d775537165d06b05.zip
userspace/cuda: complete the analysis pass
* Name recovery: name_candidates.py proposes the hand-written API-layer implementation for each exported wrapper by grouping version variants and intersecting their callees; 180 high-confidence proposals recorded in annotations/name-candidates.tsv, 29 of them body-verified and promoted to names.tsv with evidence in subsystems.tsv. * Engine records: identified the 0x14-byte per-engine descriptor, mapped its fields (kind, engine type, instance, class id, CE caps) and the existence callback, and documented how the missing async copy engine (COPY14) leaves the secondary record zeroed and blocks context creation (docs/05 section 3.7, public-safe). * Targeted type recovery: AnalyzeFunctions.java runs the parameter-ID analyzer over a curated list of 80 functions; 16 improved materially and the typed decompilations are kept under the version run tree. * Cross-version: cudatool.py fdematch measures how names transfer between patch versions (570.144 -> 570.211.01: exports 100%, verified internals ~70%, generated wrappers must be re-resolved by relocation, never by address arithmetic); documented in the upgrade playbook. * Corpus tooling: diff_corpus.py (version-to-version corpus diff) and subsystem_digest.py (per-subsystem call trees). * Deprecate the old per-version helpers whose address handling and analysis timeout produced wrong results. Generated corpora and typed outputs remain untracked under the version run tree.
Diffstat (limited to 'userspace/cuda/docs')
-rw-r--r--userspace/cuda/docs/05-570.144-decompilation.md89
-rw-r--r--userspace/cuda/docs/06-tooling.md103
-rw-r--r--userspace/cuda/docs/07-version-upgrade-playbook.md31
-rw-r--r--userspace/cuda/docs/08-virtualization-approaches.md2
4 files changed, 203 insertions, 22 deletions
diff --git a/userspace/cuda/docs/05-570.144-decompilation.md b/userspace/cuda/docs/05-570.144-decompilation.md
index 84b377e..4e23866 100644
--- a/userspace/cuda/docs/05-570.144-decompilation.md
+++ b/userspace/cuda/docs/05-570.144-decompilation.md
@@ -162,9 +162,18 @@ engine index >= 0x16 -> table + 0x351c + (index-0x16)*0x14
before the helper, the walker calls the callback at `table+0x4e00`
with `(table, desc->[0xc])` — a
-"does this engine/class exist" check — and only proceeds when it returns
-true. The helper then uses `desc->[0]` (engine kind), `desc->[8]` (instance)
-and `desc->[0xc]` (class) for the registration.
+"does this engine/class exist" check (a class-list membership test; the
+callback pointer is installed by the device ABI table setup at
+`libcuda+0x458040`/`libcuda+0x4494d0`, target `libcuda+0x44a170`) — and only
+proceeds when it returns true. The helper then uses `desc->[0]` (engine kind),
+`desc->[8]` (instance) and `desc->[0xc]` (class) for the registration.
+
+The records are built by `libcuda+0x450ff0` from the engine list that the
+`GET_ENGINES_V2` wrapper `libcuda+0x462770` returns (`{kind, engine type,
+instance}` triples at 0x10-byte stride). The record layout and the full field
+map are in §3.7. Note the walker's index ranges cover copy slots `1..0xa` and
+`0xc..0x15`, with `0xb` used for the SEC2 slot and `>=0x16` for decompression
+copy slots; the engine list is a different, dense array.
### 3.4 The engine create/register function
@@ -232,8 +241,8 @@ path on the host's real driver):
* libcuda asks the driver for the GPU's engines (`GET_ENGINES_V2`). The host
oracle's filtered list has **7** engines: GR, CE0..CE4 and `COPY14`, the
async copy engine. The guest list has only **6** — the same minus `COPY14`.
-* `COPY14`'s engine descriptor is `0x793ceb0e`, instance 14, client engine
- type `0x38`, paired with engine class `0xc9b5`. The GSP-RM firmware's
+* `COPY14`'s engine descriptor is `0x793ceb0e`, instance 14, engine type
+ `0x38`, paired with engine class `0xc9b5`. The GSP-RM firmware's
`{class, engDesc}` pair table pairs OBJCE instances `0..7` with class
`0xcab5` and instances `0x0b..0x13` (including `0x0e` = 14) with class
`0xc9b5`.
@@ -256,24 +265,56 @@ static info and add a 13th device-info entry with descriptor `0x793ceb0e`,
instance 14, RM engine type `0x17`, FIFO tag 6, runlist 5) did not change the
guest list, so the missing input is elsewhere: what builds the driver's
classDB entry for descriptor `0x793ceb0e` (class list/static-info class data
-versus the device-info-table fields) is the open question. The record layout
-the walker reads is in §3.3; the per-engine fields it consumes are
-`+0` (kind), `+8` (instance) and `+0xc` (class).
-
-### 3.7 What the emulation should check first
-
-* **Engine-list/classDB consistency — the active blocker (§3.6).** The engine
- records at the offsets in §3.3 must be present and self-consistent for every
- engine index the walker visits, the "does this engine exist" callback
- (`[table+0x4e00]`) must agree with them, and the whole set must agree with
- the class-filtered engine list the driver builds from the GSP static info
- (`GET_ENGINES_V2` + classDB). A missing entry is silent: the walker then
- reads a zeroed record.
+versus the device-info-table fields) is the open question.
+
+### 3.7 The engine descriptor record, and how it explains the blocker
+
+The device state holds a list of 0x10-byte engine entries plus a parallel set
+of 0x14-byte **per-engine descriptor records**; the walker (§3.3) reads the
+records, the helper and create hook consume them. The record was recovered
+from the record builder (`libcuda+0x450ff0`), the `GET_ENGINES_V2` wrapper
+(`libcuda+0x462770`), the create hook disassembly (`libcuda+0x4676f0`) and the
+live guest dump (`run/gdb32b.out`):
+
+| Offset | Field | Meaning | Evidence |
+|---|---|---|---|
+| `+0x00` | engine kind | `1` = graphics (GR), `3` = copy (CE), `4` = SEC2, `5` = decompression copy. The helper normalizes `1` to `0` for the per-context slot index. | builder branches on it; `GET_ENGINES_V2` wrapper classifies each engine type into it; live: GR `t=1`, CE `t=3` |
+| `+0x04` | engine type | raw `NV2080_ENGINE_TYPE` from the engine list (`1` = GR, `9..13` = COPY0..4, `0x38` = COPY14) | builder copies the first 8 bytes of the list entry into `+0/+4`; not read by the walker/helper |
+| `+0x08` | instance | engine instance (`COPY0..4` → `0..4`, `COPY14` → `14`) | helper + create hook read it; create hook copies it into the RM alloc parameters only for kinds 2/3/5; live dump |
+| `+0x0c` | class | RM allocation class for this engine: GR → the device's compute class (live `0xc5c0`), copy → the DMA copy class (live `0xc5b5`), SEC2/decomp copy → their classes | builder stores the per-kind class; the create hook passes `+0xc` straight to `libcuda+0x481360` as the `NV_ESC_RM_ALLOC` class; host capture shows `class=0xc5b5` |
+| `+0x10` | engine caps | for copy records: the CE capability bitmask; **bit 0 (`CE_GRCE`) marks the async/GRCE copy engine**. A copy record with bit 0 set is also copied to the *secondary record* slot and does not take a normal CE slot. | builder tests `+0x10 & 1` and copies to the secondary; live caps `0xbc`/`0xfc` (bit 0 clear) |
+
+Secondary record: the GR context registers a second engine from a dedicated
+0x14-byte slot (same layout) that is filled from the GRCE-capable copy record
+and then normalized to kind `2` and the async copy class. GR's own record is
+larger (a 0x20-byte slot with an unrecovered tail at `+0x10..+0x1f`).
+
+**How this explains the blocker.** The builder derives the records from the
+engine list the driver returns for `GET_ENGINES_V2`; the guest list has 6
+entries (GR + CE0..4) because the class-database filter drops `COPY14`. With
+no `COPY14` entry there is no copy record with the `CE_GRCE` capability bit
+set, so the secondary record slot is never filled and the walker sees class
+`0`. libcuda then allocates the engine object with class `0`, the escape path
+rewrites that to `NV01_ROOT_CLIENT` (`0x41`), the allocation fails with status
+`0x19`, the context rolls back and `cuCtxCreate` returns 999. In other words:
+the guest is not merely missing one entry in a list — it is missing the
+*async copy engine record that seeds the secondary record*.
+
+**What the emulation should check first (revised)**:
+
+* **`COPY14` in `GET_ENGINES_V2`** — the engine's classDB entry must exist so
+ that `gpuUpdateEngineTable`'s class-list filter keeps OBJCE instance 14.
+ The class/advertised-engine-table consistency is the root cause; the
+ secondary-record zeroing is its symptom.
+* **CE capabilities.** The copy records' caps word must carry the `CE_GRCE`
+ bit for the async CE. If `COPY14` is restored but its emulated CE caps still
+ return bit 0 clear, the secondary record stays empty and the class-0
+ allocation returns.
* **Consumed-control reply fidelity.** The controls answered generically (the
host log lists `0x20803039`, `0x20802a12`, `0x20808162`, `0x2080182b`,
`0x20800a70`, `0x20800a93`, …) should be checked for consumed fields, because
the walker's engine table is built from data the driver collected earlier
- (device caps, engine list, class list).
+ (device caps, engine list, class list, CE caps).
* The `NV_ESC_RM_ALLOC` parameter block built by `libcuda+0x481360` for class
`0xc5c0` (`TURING_COMPUTE_A`) must be answered with a coherent object: echo
the request's params back (§3.5) and return a coherent handle; the driver
@@ -307,7 +348,15 @@ citations in docs and gdb sessions use the same numbers.
holds our name map (exports are exact, internals grow over time).
* The decompiler does not recover struct types; field offsets are named
`[obj+0xNN]` and must be interpreted with the descriptions in `docs/01`
- and `docs/03`.
+ and `docs/03`. A targeted Decompiler Parameter ID pass
+ (`tools/ghidra/AnalyzeFunctions.java`, see `docs/06` §2d) re-ran over 80
+ key functions (crash/engine path, resolved wrapper implementations, largest
+ `.text`) and committed their signatures into a project copy
+ (`run/ghidra-typed/`); the re-decompiled bodies are in
+ `run/analysis/typed/` and the before/after report in
+ `run/analysis/typed/paramid_report.tsv`. 16 of 80 bodies changed materially
+ (parameter/type recovery), the rest now carry committed signatures that
+ improve downstream decompilation.
* Indirect calls dominate the API dispatch; the callgraph contains direct
edges only (77,742). Cross-referencing indirect targets requires reading the
wrapper/stub relocations (as in §2) or live traces.
diff --git a/userspace/cuda/docs/06-tooling.md b/userspace/cuda/docs/06-tooling.md
index 85d1c8c..72fc8ac 100644
--- a/userspace/cuda/docs/06-tooling.md
+++ b/userspace/cuda/docs/06-tooling.md
@@ -8,6 +8,9 @@ tools are stdlib-only or Ghidra scripts; nothing needs network access.
userspace/cuda/tools/
├── cudatool.py ELF/interface toolkit (python3, no deps)
├── callwalk.py callee/caller trees and shortest call path
+├── name_candidates.py propose impl names for wrapper callees
+├── subsystem_digest.py per-subsystem call-tree digest
+├── diff_corpus.py diff two decompiled corpus indexes
├── annotate_corpus.py annotated + name-substituted corpus
├── ghidra_full_analysis.sh import + full auto-analysis into a project
├── ghidra_export_index.sh functions/exports/callgraph/strings index
@@ -19,6 +22,7 @@ userspace/cuda/tools/
├── ExportFunctionIndex.java index exporter
├── BatchDecompile.java chunk decompiler
├── DecompileAt.java single function (file-vaddr aware)
+ ├── AnalyzeFunctions.java targeted Decompiler Parameter ID pass
├── ApplyNameMap.java apply/transfer a name map
└── PrintProgramInfo.java address convention + stats
```
@@ -34,12 +38,33 @@ tools/cudatool.py crash <lib> <map_base> <ip> # dmesg -> file vaddr
tools/cudatool.py strings <lib> [--grep RE]
tools/cudatool.py note <lib> # .note.cuda.fwd_compatibility JSON
tools/cudatool.py delta <libA> <libB> # see §4
+tools/cudatool.py fdematch <libA> <libB> [--names names.tsv] [--window 0x100]
```
Useful for: checking what changed between two driver package versions before
spending time on decompilation; converting a guest dmesg crash to a file vaddr
without gdb; getting a function list in a second.
+### 1b. Cross-version function matching — `fdematch`
+
+```
+tools/cudatool.py fdematch <libA> <libB> [--names <names.tsv>] [--window 0x100]
+```
+
+`fdematch` matches function ranges from the two `.eh_frame_hdr` tables. A
+single global address shift is not enough across patch versions, so it uses
+the common exports as **local anchors**: for each A function it looks for a B
+function with the same FDE size within `--window` bytes of
+`old_addr + delta(nearest common export)`. The report gives exact / window /
+unique match counts and the overall match rate, and with `--names` it breaks
+the name map down by its `source` column (exports, wrappers, verified,
+wrapper-callee), which is exactly what you need to know before transferring
+names to a new version. Example result for 570.144 → 570.211.01: 85.5% of
+functions match (58.3% uniquely), **all 659 exports transfer exactly, but
+only ~1% of the generated wrapper addresses do** — the wrapper region is
+regenerated and must be re-resolved with `cudatool.py wrappers`, never by
+address arithmetic.
+
### Address conventions
* `cudatool.py` prints **file virtual addresses** — the `libcuda+0x...` used
@@ -102,6 +127,67 @@ tools/callwalk.py <indexdir>/callgraph.tsv chain 0x31b7f0 0x2cb940
Addresses are file vaddrs; `names.tsv` next to the callgraph is used for
labels.
+## 2d. Targeted type recovery
+
+The full analysis disables the Decompiler Parameter ID analyzer (§2), so
+function signatures stay at the default `FUN_<addr>(...)`. `AnalyzeFunctions.java`
+re-runs that analyzer over a caller-supplied function list instead of the whole
+program: it takes an addresses file (one `0x<file_vaddr>` per line, comments
+allowed), translates through the image base like `DecompileAt.java`, resolves
+each address to its function and runs the analyzer on the resulting address
+set (batched, with a per-function decompiler timeout). Run it against a
+project copy, not the canonical one, because it commits parameters into the
+project:
+
+```
+cp -a <proj_dir> <proj_dir>-typed
+analyzeHeadless <proj_dir>-typed <proj_name> -process <program> -noanalysis \
+ -scriptPath tools/ghidra \
+ -postScript AnalyzeFunctions.java <addresses.txt> \
+ --timeout 120 --batch 20 --out <report.tsv>
+```
+
+Then re-decompile the affected functions with `ghidra_decompile_at.sh` from the
+typed project; the report's `params_before/after` and `GAIN` columns show
+which functions gained structure. `--list` resolves the list without running
+the analyzer (dry run).
+
+## 2d. Subsystem call-tree digest — `subsystem_digest.py`
+
+```
+tools/subsystem_digest.py <subsystems.tsv> <callgraph.tsv> [--out FILE]
+ [--depth 2] [--max-per-node 8] [--subsystem NAME]
+```
+
+For every subsystem in `annotations/subsystems.tsv` it prints the annotated
+members, the subsystem **entry functions** (members not called by another
+member of the same subsystem) and a depth-limited call tree from each entry
+whose direct callees are grouped by subsystem. It closes with a
+cross-subsystem edge count matrix. This is the quickest way to see a
+subsystem's shape and where its boundary with the rest of the driver is,
+without opening Ghidra. Example output for 570.144 is saved under
+`570.144/run/analysis/subsystem-digest.txt` (gitignored).
+
+## 2e. API-implementation candidates — `name_candidates.py`
+
+```
+tools/name_candidates.py <exports.tsv> <callgraph.tsv> <names.tsv> <out.tsv>
+ [--max-outside 2]
+```
+
+Exports are generated stubs → generated wrappers → the hand-written
+implementation of that API. `name_candidates.py` groups the wrappers by base
+name (stripping `_v<N>`, `_ptsz`, `_ptds`), takes the intersection of their
+direct callees and filters out helpers that are also called by wrappers
+outside the group (the trace/init/lock helpers). A single surviving candidate
+is `high` confidence when the group has several variants or the callee is
+unique; otherwise the best-ranked candidate is `medium`. Output rows carry
+the full candidate set and the outside-wrapper count as evidence, plus a
+`status` column that starts as `unverified`. Verification means reading the
+body in the corpus and marking `verified`/`corrected`; only verified entries
+may be copied into `names.tsv` (source `wrapper-callee`). The 570.144 run is
+tracked in `570.144/annotations/name-candidates.tsv`.
+
## 3. Looking things up
```
@@ -131,6 +217,23 @@ first thing to run on a new driver release; it tells you immediately whether
the update is a patch (nothing moved), a feature release (new exports) or a
reshuffle (many moves).
+### 4b. Corpus deltas — `diff_corpus.py`
+
+```
+tools/diff_corpus.py <indexA.tsv> <indexB.tsv> [--label-a A] [--label-b B]
+ [--out FILE] [--max-list 40] [--json]
+```
+
+Diffs two `decompiled_index.tsv` corpora (the merged output of
+`ghidra_batch_decompile.sh` + `ghidra_index_merge.sh`) and reports
+added/removed/moved/resized functions and a summary. Named functions match by
+name; the address shift is estimated from them and anonymous `FUN_*`
+functions match at `old_addr + shift` (a size difference there is a resize).
+This makes a driver upgrade measurable at the function level: run it after
+decompiling the new version to see how much of the previous corpus still
+corresponds. It needs a decompiled index for the other version; if only the
+binary exists, use `cudatool.py fdematch` (no decompilation required).
+
## 5. Name transfer — `ApplyNameMap.java`
```
diff --git a/userspace/cuda/docs/07-version-upgrade-playbook.md b/userspace/cuda/docs/07-version-upgrade-playbook.md
index 337f5eb..23569bd 100644
--- a/userspace/cuda/docs/07-version-upgrade-playbook.md
+++ b/userspace/cuda/docs/07-version-upgrade-playbook.md
@@ -49,6 +49,33 @@ with `PrintProgramInfo.java` before citing addresses.
## 3. Transfer names and knowledge
+Measure the transfer first, before copying anything:
+
+```
+tools/cudatool.py fdematch <previous>/run/libcuda.so.<old> <libcuda.so.<new>> \
+ --names <previous>/annotations/names.tsv
+```
+
+`fdematch` matches function ranges from both `.eh_frame_hdr` tables using the
+common exports as local address anchors (same FDE size within a small window
+of `old_addr + nearest-export delta`), and breaks the previous name map down
+by its `source` column. Measured for **570.144 → 570.211.01**
+(`570.144/run/analysis/fdematch-570.144-570.211.txt`, gitignored):
+
+| Map part | Transferable | How |
+|---|---|---|
+| 659 exports | **100%**, exact | same name, same size, address = old + local delta |
+| verified internals | 5/7 (71%) | anchored same-size window |
+| wrapper-callee internals | 21/29 (72%) | anchored same-size window |
+| generated `wrapper_*` | **1.1%** | **do not transfer by address** |
+| all A functions | 85.5% matched (58.3% unique) | anchored same-size window |
+
+The wrapper result is the important one: the generated wrapper region is
+re-laid-out between patch versions, so wrapper addresses must be re-resolved
+on the new binary with `tools/cudatool.py wrappers` (and then
+`annotate_corpus.py` re-run), never by adding a delta to the old map. Export
+and hand-written internal names do carry over.
+
```
cp <previous>/annotations/names.tsv <new>/annotations/names.tsv
# translate addresses: the map is file-vaddr keyed, so for unchanged
@@ -61,7 +88,9 @@ analyzeHeadless <proj> cuda-full -process <program> -noanalysis \
Also transfer the *subsystem index* (`annotations/subsystems.tsv`: address,
our name, subsystem, evidence) and re-validate the entries that the delta
-marked as moved.
+marked as moved. For corpus-level measurement (added/removed/resized
+functions, not just name transfer) use `tools/diff_corpus.py` on the old and
+new `decompiled_index.tsv` once the new version has been decompiled.
## 4. Validate against the guest
diff --git a/userspace/cuda/docs/08-virtualization-approaches.md b/userspace/cuda/docs/08-virtualization-approaches.md
index 3523ad1..00ca13f 100644
--- a/userspace/cuda/docs/08-virtualization-approaches.md
+++ b/userspace/cuda/docs/08-virtualization-approaches.md
@@ -2,7 +2,7 @@
One-page decision summary for how a guest's CUDA work gets executed in
Open-IOV. Companion to `docs/04-virtualization-and-emulation.md` (design
-principles) and to the project implementation docs (`docs/refactored-architecture.md`,
+principles) and to the project implementation docs (`docs/03-architecture/06-host-module-and-hybrid-model.md`,
`docs/04-implementation/02-practical-challenges-and-adversarial-interop.md`).
## 1. The options