diff options
Diffstat (limited to 'userspace/cuda/docs')
| -rw-r--r-- | userspace/cuda/docs/05-570.144-decompilation.md | 89 | ||||
| -rw-r--r-- | userspace/cuda/docs/06-tooling.md | 103 | ||||
| -rw-r--r-- | userspace/cuda/docs/07-version-upgrade-playbook.md | 31 | ||||
| -rw-r--r-- | userspace/cuda/docs/08-virtualization-approaches.md | 2 |
4 files changed, 203 insertions, 22 deletions
diff --git a/userspace/cuda/docs/05-570.144-decompilation.md b/userspace/cuda/docs/05-570.144-decompilation.md index 84b377e..4e23866 100644 --- a/userspace/cuda/docs/05-570.144-decompilation.md +++ b/userspace/cuda/docs/05-570.144-decompilation.md @@ -162,9 +162,18 @@ engine index >= 0x16 -> table + 0x351c + (index-0x16)*0x14 before the helper, the walker calls the callback at `table+0x4e00` with `(table, desc->[0xc])` — a -"does this engine/class exist" check — and only proceeds when it returns -true. The helper then uses `desc->[0]` (engine kind), `desc->[8]` (instance) -and `desc->[0xc]` (class) for the registration. +"does this engine/class exist" check (a class-list membership test; the +callback pointer is installed by the device ABI table setup at +`libcuda+0x458040`/`libcuda+0x4494d0`, target `libcuda+0x44a170`) — and only +proceeds when it returns true. The helper then uses `desc->[0]` (engine kind), +`desc->[8]` (instance) and `desc->[0xc]` (class) for the registration. + +The records are built by `libcuda+0x450ff0` from the engine list that the +`GET_ENGINES_V2` wrapper `libcuda+0x462770` returns (`{kind, engine type, +instance}` triples at 0x10-byte stride). The record layout and the full field +map are in §3.7. Note the walker's index ranges cover copy slots `1..0xa` and +`0xc..0x15`, with `0xb` used for the SEC2 slot and `>=0x16` for decompression +copy slots; the engine list is a different, dense array. ### 3.4 The engine create/register function @@ -232,8 +241,8 @@ path on the host's real driver): * libcuda asks the driver for the GPU's engines (`GET_ENGINES_V2`). The host oracle's filtered list has **7** engines: GR, CE0..CE4 and `COPY14`, the async copy engine. The guest list has only **6** — the same minus `COPY14`. -* `COPY14`'s engine descriptor is `0x793ceb0e`, instance 14, client engine - type `0x38`, paired with engine class `0xc9b5`. The GSP-RM firmware's +* `COPY14`'s engine descriptor is `0x793ceb0e`, instance 14, engine type + `0x38`, paired with engine class `0xc9b5`. The GSP-RM firmware's `{class, engDesc}` pair table pairs OBJCE instances `0..7` with class `0xcab5` and instances `0x0b..0x13` (including `0x0e` = 14) with class `0xc9b5`. @@ -256,24 +265,56 @@ static info and add a 13th device-info entry with descriptor `0x793ceb0e`, instance 14, RM engine type `0x17`, FIFO tag 6, runlist 5) did not change the guest list, so the missing input is elsewhere: what builds the driver's classDB entry for descriptor `0x793ceb0e` (class list/static-info class data -versus the device-info-table fields) is the open question. The record layout -the walker reads is in §3.3; the per-engine fields it consumes are -`+0` (kind), `+8` (instance) and `+0xc` (class). - -### 3.7 What the emulation should check first - -* **Engine-list/classDB consistency — the active blocker (§3.6).** The engine - records at the offsets in §3.3 must be present and self-consistent for every - engine index the walker visits, the "does this engine exist" callback - (`[table+0x4e00]`) must agree with them, and the whole set must agree with - the class-filtered engine list the driver builds from the GSP static info - (`GET_ENGINES_V2` + classDB). A missing entry is silent: the walker then - reads a zeroed record. +versus the device-info-table fields) is the open question. + +### 3.7 The engine descriptor record, and how it explains the blocker + +The device state holds a list of 0x10-byte engine entries plus a parallel set +of 0x14-byte **per-engine descriptor records**; the walker (§3.3) reads the +records, the helper and create hook consume them. The record was recovered +from the record builder (`libcuda+0x450ff0`), the `GET_ENGINES_V2` wrapper +(`libcuda+0x462770`), the create hook disassembly (`libcuda+0x4676f0`) and the +live guest dump (`run/gdb32b.out`): + +| Offset | Field | Meaning | Evidence | +|---|---|---|---| +| `+0x00` | engine kind | `1` = graphics (GR), `3` = copy (CE), `4` = SEC2, `5` = decompression copy. The helper normalizes `1` to `0` for the per-context slot index. | builder branches on it; `GET_ENGINES_V2` wrapper classifies each engine type into it; live: GR `t=1`, CE `t=3` | +| `+0x04` | engine type | raw `NV2080_ENGINE_TYPE` from the engine list (`1` = GR, `9..13` = COPY0..4, `0x38` = COPY14) | builder copies the first 8 bytes of the list entry into `+0/+4`; not read by the walker/helper | +| `+0x08` | instance | engine instance (`COPY0..4` → `0..4`, `COPY14` → `14`) | helper + create hook read it; create hook copies it into the RM alloc parameters only for kinds 2/3/5; live dump | +| `+0x0c` | class | RM allocation class for this engine: GR → the device's compute class (live `0xc5c0`), copy → the DMA copy class (live `0xc5b5`), SEC2/decomp copy → their classes | builder stores the per-kind class; the create hook passes `+0xc` straight to `libcuda+0x481360` as the `NV_ESC_RM_ALLOC` class; host capture shows `class=0xc5b5` | +| `+0x10` | engine caps | for copy records: the CE capability bitmask; **bit 0 (`CE_GRCE`) marks the async/GRCE copy engine**. A copy record with bit 0 set is also copied to the *secondary record* slot and does not take a normal CE slot. | builder tests `+0x10 & 1` and copies to the secondary; live caps `0xbc`/`0xfc` (bit 0 clear) | + +Secondary record: the GR context registers a second engine from a dedicated +0x14-byte slot (same layout) that is filled from the GRCE-capable copy record +and then normalized to kind `2` and the async copy class. GR's own record is +larger (a 0x20-byte slot with an unrecovered tail at `+0x10..+0x1f`). + +**How this explains the blocker.** The builder derives the records from the +engine list the driver returns for `GET_ENGINES_V2`; the guest list has 6 +entries (GR + CE0..4) because the class-database filter drops `COPY14`. With +no `COPY14` entry there is no copy record with the `CE_GRCE` capability bit +set, so the secondary record slot is never filled and the walker sees class +`0`. libcuda then allocates the engine object with class `0`, the escape path +rewrites that to `NV01_ROOT_CLIENT` (`0x41`), the allocation fails with status +`0x19`, the context rolls back and `cuCtxCreate` returns 999. In other words: +the guest is not merely missing one entry in a list — it is missing the +*async copy engine record that seeds the secondary record*. + +**What the emulation should check first (revised)**: + +* **`COPY14` in `GET_ENGINES_V2`** — the engine's classDB entry must exist so + that `gpuUpdateEngineTable`'s class-list filter keeps OBJCE instance 14. + The class/advertised-engine-table consistency is the root cause; the + secondary-record zeroing is its symptom. +* **CE capabilities.** The copy records' caps word must carry the `CE_GRCE` + bit for the async CE. If `COPY14` is restored but its emulated CE caps still + return bit 0 clear, the secondary record stays empty and the class-0 + allocation returns. * **Consumed-control reply fidelity.** The controls answered generically (the host log lists `0x20803039`, `0x20802a12`, `0x20808162`, `0x2080182b`, `0x20800a70`, `0x20800a93`, …) should be checked for consumed fields, because the walker's engine table is built from data the driver collected earlier - (device caps, engine list, class list). + (device caps, engine list, class list, CE caps). * The `NV_ESC_RM_ALLOC` parameter block built by `libcuda+0x481360` for class `0xc5c0` (`TURING_COMPUTE_A`) must be answered with a coherent object: echo the request's params back (§3.5) and return a coherent handle; the driver @@ -307,7 +348,15 @@ citations in docs and gdb sessions use the same numbers. holds our name map (exports are exact, internals grow over time). * The decompiler does not recover struct types; field offsets are named `[obj+0xNN]` and must be interpreted with the descriptions in `docs/01` - and `docs/03`. + and `docs/03`. A targeted Decompiler Parameter ID pass + (`tools/ghidra/AnalyzeFunctions.java`, see `docs/06` §2d) re-ran over 80 + key functions (crash/engine path, resolved wrapper implementations, largest + `.text`) and committed their signatures into a project copy + (`run/ghidra-typed/`); the re-decompiled bodies are in + `run/analysis/typed/` and the before/after report in + `run/analysis/typed/paramid_report.tsv`. 16 of 80 bodies changed materially + (parameter/type recovery), the rest now carry committed signatures that + improve downstream decompilation. * Indirect calls dominate the API dispatch; the callgraph contains direct edges only (77,742). Cross-referencing indirect targets requires reading the wrapper/stub relocations (as in §2) or live traces. diff --git a/userspace/cuda/docs/06-tooling.md b/userspace/cuda/docs/06-tooling.md index 85d1c8c..72fc8ac 100644 --- a/userspace/cuda/docs/06-tooling.md +++ b/userspace/cuda/docs/06-tooling.md @@ -8,6 +8,9 @@ tools are stdlib-only or Ghidra scripts; nothing needs network access. userspace/cuda/tools/ ├── cudatool.py ELF/interface toolkit (python3, no deps) ├── callwalk.py callee/caller trees and shortest call path +├── name_candidates.py propose impl names for wrapper callees +├── subsystem_digest.py per-subsystem call-tree digest +├── diff_corpus.py diff two decompiled corpus indexes ├── annotate_corpus.py annotated + name-substituted corpus ├── ghidra_full_analysis.sh import + full auto-analysis into a project ├── ghidra_export_index.sh functions/exports/callgraph/strings index @@ -19,6 +22,7 @@ userspace/cuda/tools/ ├── ExportFunctionIndex.java index exporter ├── BatchDecompile.java chunk decompiler ├── DecompileAt.java single function (file-vaddr aware) + ├── AnalyzeFunctions.java targeted Decompiler Parameter ID pass ├── ApplyNameMap.java apply/transfer a name map └── PrintProgramInfo.java address convention + stats ``` @@ -34,12 +38,33 @@ tools/cudatool.py crash <lib> <map_base> <ip> # dmesg -> file vaddr tools/cudatool.py strings <lib> [--grep RE] tools/cudatool.py note <lib> # .note.cuda.fwd_compatibility JSON tools/cudatool.py delta <libA> <libB> # see §4 +tools/cudatool.py fdematch <libA> <libB> [--names names.tsv] [--window 0x100] ``` Useful for: checking what changed between two driver package versions before spending time on decompilation; converting a guest dmesg crash to a file vaddr without gdb; getting a function list in a second. +### 1b. Cross-version function matching — `fdematch` + +``` +tools/cudatool.py fdematch <libA> <libB> [--names <names.tsv>] [--window 0x100] +``` + +`fdematch` matches function ranges from the two `.eh_frame_hdr` tables. A +single global address shift is not enough across patch versions, so it uses +the common exports as **local anchors**: for each A function it looks for a B +function with the same FDE size within `--window` bytes of +`old_addr + delta(nearest common export)`. The report gives exact / window / +unique match counts and the overall match rate, and with `--names` it breaks +the name map down by its `source` column (exports, wrappers, verified, +wrapper-callee), which is exactly what you need to know before transferring +names to a new version. Example result for 570.144 → 570.211.01: 85.5% of +functions match (58.3% uniquely), **all 659 exports transfer exactly, but +only ~1% of the generated wrapper addresses do** — the wrapper region is +regenerated and must be re-resolved with `cudatool.py wrappers`, never by +address arithmetic. + ### Address conventions * `cudatool.py` prints **file virtual addresses** — the `libcuda+0x...` used @@ -102,6 +127,67 @@ tools/callwalk.py <indexdir>/callgraph.tsv chain 0x31b7f0 0x2cb940 Addresses are file vaddrs; `names.tsv` next to the callgraph is used for labels. +## 2d. Targeted type recovery + +The full analysis disables the Decompiler Parameter ID analyzer (§2), so +function signatures stay at the default `FUN_<addr>(...)`. `AnalyzeFunctions.java` +re-runs that analyzer over a caller-supplied function list instead of the whole +program: it takes an addresses file (one `0x<file_vaddr>` per line, comments +allowed), translates through the image base like `DecompileAt.java`, resolves +each address to its function and runs the analyzer on the resulting address +set (batched, with a per-function decompiler timeout). Run it against a +project copy, not the canonical one, because it commits parameters into the +project: + +``` +cp -a <proj_dir> <proj_dir>-typed +analyzeHeadless <proj_dir>-typed <proj_name> -process <program> -noanalysis \ + -scriptPath tools/ghidra \ + -postScript AnalyzeFunctions.java <addresses.txt> \ + --timeout 120 --batch 20 --out <report.tsv> +``` + +Then re-decompile the affected functions with `ghidra_decompile_at.sh` from the +typed project; the report's `params_before/after` and `GAIN` columns show +which functions gained structure. `--list` resolves the list without running +the analyzer (dry run). + +## 2d. Subsystem call-tree digest — `subsystem_digest.py` + +``` +tools/subsystem_digest.py <subsystems.tsv> <callgraph.tsv> [--out FILE] + [--depth 2] [--max-per-node 8] [--subsystem NAME] +``` + +For every subsystem in `annotations/subsystems.tsv` it prints the annotated +members, the subsystem **entry functions** (members not called by another +member of the same subsystem) and a depth-limited call tree from each entry +whose direct callees are grouped by subsystem. It closes with a +cross-subsystem edge count matrix. This is the quickest way to see a +subsystem's shape and where its boundary with the rest of the driver is, +without opening Ghidra. Example output for 570.144 is saved under +`570.144/run/analysis/subsystem-digest.txt` (gitignored). + +## 2e. API-implementation candidates — `name_candidates.py` + +``` +tools/name_candidates.py <exports.tsv> <callgraph.tsv> <names.tsv> <out.tsv> + [--max-outside 2] +``` + +Exports are generated stubs → generated wrappers → the hand-written +implementation of that API. `name_candidates.py` groups the wrappers by base +name (stripping `_v<N>`, `_ptsz`, `_ptds`), takes the intersection of their +direct callees and filters out helpers that are also called by wrappers +outside the group (the trace/init/lock helpers). A single surviving candidate +is `high` confidence when the group has several variants or the callee is +unique; otherwise the best-ranked candidate is `medium`. Output rows carry +the full candidate set and the outside-wrapper count as evidence, plus a +`status` column that starts as `unverified`. Verification means reading the +body in the corpus and marking `verified`/`corrected`; only verified entries +may be copied into `names.tsv` (source `wrapper-callee`). The 570.144 run is +tracked in `570.144/annotations/name-candidates.tsv`. + ## 3. Looking things up ``` @@ -131,6 +217,23 @@ first thing to run on a new driver release; it tells you immediately whether the update is a patch (nothing moved), a feature release (new exports) or a reshuffle (many moves). +### 4b. Corpus deltas — `diff_corpus.py` + +``` +tools/diff_corpus.py <indexA.tsv> <indexB.tsv> [--label-a A] [--label-b B] + [--out FILE] [--max-list 40] [--json] +``` + +Diffs two `decompiled_index.tsv` corpora (the merged output of +`ghidra_batch_decompile.sh` + `ghidra_index_merge.sh`) and reports +added/removed/moved/resized functions and a summary. Named functions match by +name; the address shift is estimated from them and anonymous `FUN_*` +functions match at `old_addr + shift` (a size difference there is a resize). +This makes a driver upgrade measurable at the function level: run it after +decompiling the new version to see how much of the previous corpus still +corresponds. It needs a decompiled index for the other version; if only the +binary exists, use `cudatool.py fdematch` (no decompilation required). + ## 5. Name transfer — `ApplyNameMap.java` ``` diff --git a/userspace/cuda/docs/07-version-upgrade-playbook.md b/userspace/cuda/docs/07-version-upgrade-playbook.md index 337f5eb..23569bd 100644 --- a/userspace/cuda/docs/07-version-upgrade-playbook.md +++ b/userspace/cuda/docs/07-version-upgrade-playbook.md @@ -49,6 +49,33 @@ with `PrintProgramInfo.java` before citing addresses. ## 3. Transfer names and knowledge +Measure the transfer first, before copying anything: + +``` +tools/cudatool.py fdematch <previous>/run/libcuda.so.<old> <libcuda.so.<new>> \ + --names <previous>/annotations/names.tsv +``` + +`fdematch` matches function ranges from both `.eh_frame_hdr` tables using the +common exports as local address anchors (same FDE size within a small window +of `old_addr + nearest-export delta`), and breaks the previous name map down +by its `source` column. Measured for **570.144 → 570.211.01** +(`570.144/run/analysis/fdematch-570.144-570.211.txt`, gitignored): + +| Map part | Transferable | How | +|---|---|---| +| 659 exports | **100%**, exact | same name, same size, address = old + local delta | +| verified internals | 5/7 (71%) | anchored same-size window | +| wrapper-callee internals | 21/29 (72%) | anchored same-size window | +| generated `wrapper_*` | **1.1%** | **do not transfer by address** | +| all A functions | 85.5% matched (58.3% unique) | anchored same-size window | + +The wrapper result is the important one: the generated wrapper region is +re-laid-out between patch versions, so wrapper addresses must be re-resolved +on the new binary with `tools/cudatool.py wrappers` (and then +`annotate_corpus.py` re-run), never by adding a delta to the old map. Export +and hand-written internal names do carry over. + ``` cp <previous>/annotations/names.tsv <new>/annotations/names.tsv # translate addresses: the map is file-vaddr keyed, so for unchanged @@ -61,7 +88,9 @@ analyzeHeadless <proj> cuda-full -process <program> -noanalysis \ Also transfer the *subsystem index* (`annotations/subsystems.tsv`: address, our name, subsystem, evidence) and re-validate the entries that the delta -marked as moved. +marked as moved. For corpus-level measurement (added/removed/resized +functions, not just name transfer) use `tools/diff_corpus.py` on the old and +new `decompiled_index.tsv` once the new version has been decompiled. ## 4. Validate against the guest diff --git a/userspace/cuda/docs/08-virtualization-approaches.md b/userspace/cuda/docs/08-virtualization-approaches.md index 3523ad1..00ca13f 100644 --- a/userspace/cuda/docs/08-virtualization-approaches.md +++ b/userspace/cuda/docs/08-virtualization-approaches.md @@ -2,7 +2,7 @@ One-page decision summary for how a guest's CUDA work gets executed in Open-IOV. Companion to `docs/04-virtualization-and-emulation.md` (design -principles) and to the project implementation docs (`docs/refactored-architecture.md`, +principles) and to the project implementation docs (`docs/03-architecture/06-host-module-and-hybrid-model.md`, `docs/04-implementation/02-practical-challenges-and-adversarial-interop.md`). ## 1. The options |