diff options
| author | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-09-23 12:11:10 -0400 |
|---|---|---|
| committer | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-09-23 12:11:10 -0400 |
| commit | 692c095ae48f2829613873e60e4dbbf14b36323d (patch) | |
| tree | 6e48fc539c7b02b3f30654f7d1703beffa974277 /docs | |
| parent | f874d709e5a45d63c8ee9ab6b0300195aa5d20a5 (diff) | |
| download | open-iov-692c095ae48f2829613873e60e4dbbf14b36323d.tar.gz open-iov-692c095ae48f2829613873e60e4dbbf14b36323d.zip | |
docs: refresh status, supersede stale material, link the CUDA analysis
* Status now matches the live state: guest driver probe and nvidia-smi work;
cuInit/device queries work; cuCtxCreate returns 999 because the emulated
engine list is missing the async copy engine (COPY14). The userspace
libcuda segfault is recorded as solved, with its mechanism (an RM
allocation reply must echo request parameters; zeroing them corrupted the
caller's saved frame pointer).
* Correct the emulation claims: only a subset of RM controls are
special-cased; the rest return a success reply with zeroed parameters whose
consumed fields are still being corrected; allocation replies are coherent
but no server-side object model exists.
* Add the eight-point emulation checklist and reply-fidelity ladder as
explicit gates, documented in userspace/cuda and linked from the
architecture, implementation and operations docs.
* Fix operational errors: build/test commands, module load order, mdev type
names, LFS references, debugfs claims, in-repo firmware claims, CI claims,
portability matrix (guest status, config layout, userspace driver facts).
* Mark historical material: one superseded banner per legacy document with a
pointer to its replacement; STATE.md marked as a 2026-09-21 snapshot.
* Wiki: fix the index, add status banners and in-place factual corrections
(msgq transport vs legacy mailbox, BAR definitions, error statuses,
migration, glossary), and note the archived 2023 imports.
Diffstat (limited to 'docs')
30 files changed, 485 insertions, 99 deletions
diff --git a/docs/01-orientation/01-purpose-and-scope.md b/docs/01-orientation/01-purpose-and-scope.md index eab63bc..c1232f3 100644 --- a/docs/01-orientation/01-purpose-and-scope.md +++ b/docs/01-orientation/01-purpose-and-scope.md @@ -131,13 +131,18 @@ document: * **Working today**: the virtual device, the GSP boot handshake, the RPC transport, the resource-manager emulation surface the guest driver needs to - probe and enumerate, guest `nvidia.ko` and `nvidia-smi`, and the early UVM - initialization path. -* **Needs improvement**: GPU work forwarding (the reason CUDA kernels do not - yet run), the memory/page-table model, Windows guest support, non-KVM - hypervisors, and performance instrumentation. Each of these is tracked with - explicit "known-unknowns" in - [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md). + probe and enumerate, guest `nvidia.ko` and `nvidia-smi`, and CUDA user-space + startup through `cuInit` / `cuDeviceGet`. +* **Needs improvement**: engine topology and reply fidelity first — context + creation returns `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine + list lacks the async copy engine (`COPY14`) and some consumed reply fields + are still generic fallbacks — then GPU work forwarding (the reason CUDA + kernels do not yet run), the memory/page-table model, Windows guest support, + non-KVM hypervisors, and performance instrumentation. Each of these is + tracked with explicit "known-unknowns" in + [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md); + for the CUDA analysis area, start at + [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md). ## 8. Guiding principles diff --git a/docs/02-getting-started/01-install-and-first-run.md b/docs/02-getting-started/01-install-and-first-run.md index e0e71cb..f323dbb 100644 --- a/docs/02-getting-started/01-install-and-first-run.md +++ b/docs/02-getting-started/01-install-and-first-run.md @@ -100,8 +100,12 @@ Inside the guest (the reference guest has SSH and a guest agent configured): ```bash sudo apt update sudo apt install -y linux-headers-generic build-essential -# Install the stock NVIDIA driver matching the firmware/driver profile -sudo apt install -y nvidia-driver-570-open +# Install the stock NVIDIA driver matching the emulation profile. +# The validated combination is the stock 570.144 Linux runfile (same driver +# version as the emulation target). The distro 570.x package can install a +# different 570 build; if you use it, keep the version notes from the +# hardware/software matrix at hand. +sudo sh ./NVIDIA-Linux-x86_64-570.144.run sudo reboot ``` @@ -126,10 +130,12 @@ cd cuda-samples/Samples/1_Utilities/deviceQuery && make && ./deviceQuery ``` **Expectation today:** driver probe, `nvidia-smi`, device enumeration, and the -early CUDA start-up path work; full kernel execution is the active work item. -Read the honest status in +first CUDA calls (`cuInit`, device queries) work. `cuCtxCreate` still fails +with `999` because the emulated engine list is missing the async copy engine; +full kernel execution is not reached yet. Before drawing conclusions from a +CUDA failure, read the honest status in [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) -before drawing conclusions from a CUDA failure. +and the debugger's guide [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md). ## 7. Day-to-day operations @@ -166,7 +172,7 @@ Developer debug knobs (module parameters) are documented in | No mdev type for your GPU | [hardware matrix](../06-portability/hardware-software-matrix.md) | | Guest sees wrong PCIe link speed/width | root-port attachment + config-space mirror (debugging doc) | | `nvidia.ko` probe fails in guest | guest dmesg + `lspci -vvv` + handoff "driver lifecycle" | -| `nvidia-smi` works, CUDA does not | expected today; see the known-knowns doc | +| `nvidia-smi` works, CUDA does not | expected today (`cuCtxCreate` = 999, engine-list gap); see [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) and [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md) | | VM boot fails or the guest hangs | `scripts/stack_manager.sh cleanup-crash`, then reboot | ## 10. Uninstall diff --git a/docs/03-architecture/01-system-overview.md b/docs/03-architecture/01-system-overview.md index ed9b24b..c7a847e 100644 --- a/docs/03-architecture/01-system-overview.md +++ b/docs/03-architecture/01-system-overview.md @@ -32,7 +32,8 @@ Everything therefore falls into two buckets: │ PASS THROUGH (real hardware; for correctness and performance) │ │ • GPU memory: guest VRAM objects backed by real VRAM (goal) │ │ • Page tables / GMMU state used by the GPU (goal) │ -│ • Channel execution: the guest's push buffers run on real channels │ +│ • Channel execution: guest push buffers on real GPU channels (goal; │ +│ today the host software-executes decoded work — a bridge) │ │ • Compute (CUDA) and, later, 3D/video work │ │ • DMA, completion and fault interrupts relayed back to the guest │ └─────────────────────────────────────────────────────────────────────────┘ @@ -59,7 +60,7 @@ HOST MODULE open-iov-vfio.ko ├─ virtual device: config space, BAR0/BAR1/BAR3 windows, MSI-X ├─ emulation core: GSP boot + GSP-RM (RPC msgq, controls, objects, events) ├─ memory model: guest RAM access, VA↔PA maps, VRAM backing - └─ work path: channel/push forwarding to the physical GPU [in progress] + └─ work path: channel/push forwarding to the physical GPU [goal; software bridge today] │ HOST DRIVER nvidia.ko (unmodified) and/or direct GPU interfaces │ @@ -161,9 +162,23 @@ decompiled and mapped (see `wiki/GSP-570.144W-Firmware-Analysis.md` and versions, and performs the resource-manager initialization exchanges. 4. Object creation (clients, devices, subdevices), capability queries, vaspaces, channels, CE caps, GMMU static info — every control the driver - probes — is answered with the correct shape and values. -5. `nvidia-smi` works; CUDA user-space starts; UVM begins channel-manager and - page-tree initialization. + probes — is answered with a well-formed reply. Only a small set (23 control + IDs) is special-cased with real values; the rest fall back to generic + `SUCCESS` with zeroed params, and `GSP_RM_ALLOC` echoes the request params + without server-side object management. Where the driver consumes a reply + value, that value is still being corrected (the reply-fidelity ladder). +5. `nvidia-smi` works; CUDA user-space starts (`cuInit` / `cuDeviceGet` + succeed). Context creation is currently blocked: `cuCtxCreate` returns + `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks the + async copy engine (`COPY14`), so UVM's channel-manager and page-tree + initialization is as far as the guest gets. + +The CUDA-side contract and the reply-fidelity rules are documented in +[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md` §9](../../userspace/cuda/docs/03-driver-and-gpu-interface.md) +and +[`../../userspace/cuda/docs/04-virtualization-and-emulation.md` §6](../../userspace/cuda/docs/04-virtualization-and-emulation.md); +start at +[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md). ## 7. What is next (and why) diff --git a/docs/03-architecture/02-guest-stack-and-cudasw.md b/docs/03-architecture/02-guest-stack-and-cudasw.md index 4ec614b..115f639 100644 --- a/docs/03-architecture/02-guest-stack-and-cudasw.md +++ b/docs/03-architecture/02-guest-stack-and-cudasw.md @@ -88,8 +88,21 @@ open /dev/nvidia* → cuInit → UVM registration (per process) ``` The initialization path is where compatibility is won or lost; the runtime -path is where *execution* is won or lost. Open-IOV's current focus is the -runtime path (forwarding), having already reached the end of initialization. +path is where *execution* is won or lost. Open-IOV's current focus is +pre-forwarding correctness: `cuInit` / `cuDeviceGet` succeed, but context +creation returns `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine +list lacks the async copy engine (`COPY14`); the forwarding decision +(host-RM proxy vs direct submission) is still ahead (see +[`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md)). + +For the CUDA userspace side of this stack — its architecture, how it is built +and executes, and the exact kernel/GPU interface contract we must satisfy — +see [`../../userspace/cuda/docs/01-architecture.md`](../../userspace/cuda/docs/01-architecture.md), +[`../../userspace/cuda/docs/02-build-and-execution.md`](../../userspace/cuda/docs/02-build-and-execution.md) +and +[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../../userspace/cuda/docs/03-driver-and-gpu-interface.md), +starting from +[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md). ## 5. What the guest expects from the device, per phase diff --git a/docs/03-architecture/03-gsp-and-gsp-rm.md b/docs/03-architecture/03-gsp-and-gsp-rm.md index 20d335b..aca908f 100644 --- a/docs/03-architecture/03-gsp-and-gsp-rm.md +++ b/docs/03-architecture/03-gsp-and-gsp-rm.md @@ -185,14 +185,23 @@ is not lost. |---|---|---| | GSP boot state machine | ✅ | stages complete; the driver's poll loop is satisfied | | Message queue transport | ✅ | including the non-contiguous page table for the queue buffer | -| RPC request/reply | ✅ | all functions the driver uses during init are answered | -| Static info tables | ✅ | device/FB/BUS/INTR/GMMU tables with plausible TU104 values | -| Controls during init | ✅ | ~160 distinct control IDs observed per CUDA run; the ones needed to proceed carry real values | -| Object creation | ✅ | correct handle/parameter shapes | +| RPC request/reply | ✅ | all functions the driver uses during init are answered; unknown functions get a success reply with no payload | +| Static info tables | 🟡 | device/FB/BUS/INTR/GMMU tables carry plausible TU104 values; the engine inventory is still incomplete (the async copy engine, COPY14, is missing) | +| Controls during init | 🟡 | ~160 distinct control IDs observed per CUDA run; 23 are special-cased with real values, the rest fall through to a success reply with zeroed parameters - consumed fields of those are still being corrected | +| Object creation | 🟡 | handles/parameters are echoed consistently (the request parameters must be echoed back - zeroing them corrupts the caller), but there is no server-side object model: allocation is acknowledged, not tracked | | Page tables for server-owned ranges | 🟡 | contract received and recorded; leaf population is the forwarding work | | Event/notification delivery | 🟡 | MSI-X works; richer event semantics as forwarding lands | | Firmware *behaviour* (routing, scheduling, real execution) | 🔴 | the forwarding work; see the implementation docs | +The current CUDA blocker is the engine-inventory row: `libcuda` filters the +engine list it receives against the class list and expects the async copy +engine to be present, because the secondary (async) engine record is otherwise +zeroed. See +[`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md) +section 3.7 for the offsets and the consumed fields, and +[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../../userspace/cuda/docs/03-driver-and-gpu-interface.md) +section 9 for the emulation checklist. + ## 8. Where the firmware analysis lives * `wiki/GSP-570.144W-Firmware-Analysis.md` — how the 28.5 MB image was diff --git a/docs/03-architecture/04-uvm.md b/docs/03-architecture/04-uvm.md index 299f46b..05ec80b 100644 --- a/docs/03-architecture/04-uvm.md +++ b/docs/03-architecture/04-uvm.md @@ -50,13 +50,24 @@ Each stage has a distinct failure signature; our debugging docs map them. | Requirement | Detail | Our status | |---|---|---| | GPU identity | must match the uuid from RM | ✅ | -| CE capabilities | per-engine support bits; UVM *requires* `supported && !grce && sysmem` for CPU_TO_GPU/GPU_TO_CPU channels | ✅ (we answer CE_GET_PHYSICAL_CAPS with the correct bits) | +| CE capabilities | per-engine support bits; UVM *requires* `supported && !grce && sysmem` for CPU_TO_GPU/GPU_TO_CPU channels | 🟡 the caps we answer are accepted, but the engine list / CE capabilities are missing the async copy engine (`COPY14`) — the current context-creation blocker | | Channel creation | RM channel objects with GPFIFO/userd/instance parameters | ✅ objects created | -| Pushes executed | UVM's first action on a new channel is an *init push* followed by a semaphore-tracked wait | 🟡 we can complete them (see below) | +| Pushes executed | UVM's first action on a new channel is an *init push* followed by a semaphore-tracked wait | 🟡 the host decodes method streams and executes memsets, copies and semaphore releases in software (a bridge); no kernel is forwarded to the physical GPU yet | | Page trees | UVM initializes page trees with its own pushes; the *server-owned* ranges must be populated by us | 🟡 | | Fault delivery | fault buffer + notification; replay/cancel semantics | 🔴 | | Semantics of completions | tracker values must advance exactly | 🟡 | +**Current blocker (2026-09-23):** `cuCtxCreate` returns `rc=999` +(`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks the async copy +engine (`COPY14`); teardown of the failed context still oopses in the guest +kernel in `uvm_pmm_devmem_page_to_va_space` (twice per boot). The CUDA-side +contract and reply rules are in +[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md` §9](../../userspace/cuda/docs/03-driver-and-gpu-interface.md) +and +[`../../userspace/cuda/docs/04-virtualization-and-emulation.md` §6](../../userspace/cuda/docs/04-virtualization-and-emulation.md); +start at +[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md). + ## 4. The most important discovery: how UVM waits UVM's tracker (`uvm_tracker.c`) is the single most visible symptom of an diff --git a/docs/03-architecture/05-gmmu-amap-hal-gr.md b/docs/03-architecture/05-gmmu-amap-hal-gr.md index dc326ba..d6470b6 100644 --- a/docs/03-architecture/05-gmmu-amap-hal-gr.md +++ b/docs/03-architecture/05-gmmu-amap-hal-gr.md @@ -182,6 +182,17 @@ Engine types are enumerated by the client during init (`FIFO_GET_DEVICE_INFO_TAB and reported in engine/capability queries. A guest's channel is bound to one engine type (e.g. `COPY0 = 0x9`, `GR = 0x1`). +**Engine inventory is a correctness surface, not just data.** The userspace +driver builds its per-engine records from the device/engine information the +kernel driver reports, and filters them against the class list; a missing or +zeroed record is not ignored - it becomes an allocation with class `0` and the +whole context setup rolls back. The current CUDA bring-up blocker is exactly +this: the async copy engine (COPY14, engine descriptor `0x793ceb0e`, instance +14) is absent from the emulated engine list, while a bare-metal oracle reports +it as the seventh engine alongside GR and CE0..4. See +[`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md) +section 3.7 for the record layout the driver's engine walker consumes. + ## 7. Practical reading list inside this repository | Topic | Where | diff --git a/docs/03-architecture/06-host-module-and-hybrid-model.md b/docs/03-architecture/06-host-module-and-hybrid-model.md index fc4bb23..4f9efdb 100644 --- a/docs/03-architecture/06-host-module-and-hybrid-model.md +++ b/docs/03-architecture/06-host-module-and-hybrid-model.md @@ -92,14 +92,39 @@ implementation philosophy: expects, or the driver walks off into uninitialized memory. * **Plausible values second.** Values must be self-consistent (e.g. FB size, BAR sizes, engine counts, page sizes) or later code paths compute nonsense. -* **Never echo garbage.** Queries that pass uninitialized input must receive - zeroed/valid output; echoing the request's stack garbage has caused real - failures. +* **Never invent input.** Queries that pass uninitialized *input* must not read + it; but in/out parameter blocks must be echoed back, because the driver + copies the reply's class parameters over its own buffer at the size the + kernel driver declares. Zeroing an in/out block is destructive: the + `GSP_RM_ALLOC` reply once zeroed a caller's saved frame pointer this way and + faulted `libcuda` (fixed by echoing the request parameters). The handlers currently cover: the boot/static-info exchanges, the device/FB/ BUS/INTR control tables, CE capability and PCE-mask queries, GMMU static info, vaspace reserved-PDE contracts, channel/CE object allocation, and the general -"unknown control returns success with zeroed output" fallback. +"unknown control/function returns success with zeroed output" fallback. The +same fallback now matters for engine inventory: `libcuda` filters the engine +list against the class list, so a missing async copy engine (COPY14) leaves a +record zeroed and blocks context creation (`cuCtxCreate` returns `999`); see +[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md) +and +[`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md) +section 3.7. + +**Current work-execution bridge.** The module also decodes push buffers as +method streams and executes the work it understands in software - memsets, +copies and semaphore releases - and completes the corresponding semaphores. +That is a correctness bridge for memory operations, not the data plane: no +CUDA kernel is forwarded to the physical GPU yet, and the forwarding strategy +(host-RM proxy first, direct submission if needed) is in +[`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) +section 3. + +**Subsystem status note.** Some components are compiled but not currently +wired into the live path: the physical-GSP load/boot path is skipped in favour +of emulation, and the runtime firmware analyzer, version tables, hardware +abstraction tables and dynamic RPC dispatcher are not yet called by the live +flow. Treat their descriptions here as intended design, not active code. ### 3.3 Event channels and interrupts @@ -193,6 +218,8 @@ emulated device inspectable without rebuilding: | `poke_guest=<gpa>,<v>` | write a 32-bit value to guest physical memory | | `poke_fill=<gpa>,<v>,<n>` | fill a range | | `fill_vidmem=<v>` | fill the VRAM shadow | +| `sema_probe=<gpu_va>[,count]` | probe a semaphore VA and nearby state in guest memory | +| `dump_guest=<gpa>,<len>` | dump guest physical memory (length capped) | These are development-grade but production-useful: they are how the memory model was reverse-engineered, and they remain the fastest way to diagnose a diff --git a/docs/03-architecture/07-driver-lifecycle-field-guide.md b/docs/03-architecture/07-driver-lifecycle-field-guide.md index 8d80694..9de5edb 100644 --- a/docs/03-architecture/07-driver-lifecycle-field-guide.md +++ b/docs/03-architecture/07-driver-lifecycle-field-guide.md @@ -88,7 +88,9 @@ gsp-resman-vfio: queue args sharedPA=... ptes=... cmdOff=... statOff=... ## 5. Resource-manager initialization (the control battery) **Guest:** sends `GSP_RM_CONTROL` (76) for every subsystem it initializes. -Representative controls (all observed, all answered with correct shapes): +Of the ~160 observed control IDs, 23 are special-cased and answered with real +values; the rest are answered with a generic `SUCCESS` and zeroed params (the +fallback). Representative controls: | Control | Meaning | What matters | |---|---|---| @@ -103,8 +105,11 @@ Representative controls (all observed, all answered with correct shapes): | `0x20802a02` | CE PCE mask | one bit per engine | | `0x20802a08` | CE fault method buffer size | method buffer sizing | | `0x90f10106` | vaspace reserved-PDE contract | **carries page-table physical addresses** (see GSP-RM doc) | -| `0x20803039` | NVLink masks/VBIOS info | unpopulated but must be shaped | -| `0x20800a70` | BUS flush with sysmembar | UVM issues these frequently; success suffices | +| `0x20800123` / `0x20800170` | `GPU_GET_ENGINES` / `GET_ENGINES_V2` | engine inventory (filtered by the class list / classDB); values are consumed — the guest must see the full engine list, including the async copy engine (`COPY14`), and the current `rc=999` blocker lives here | +| `0x800201` | `GPU_GET_CLASSLIST` | class list (classDB); must agree with the engine list and the CE caps controls | +| `0x2080012b` | GPU_PROMOTE_CTX (560-byte params) | issued immediately before context creation; fallback generic `SUCCESS`, consumed fields not yet coherent | +| `0x20803039` | NVLink masks/VBIOS info | fallback: generic `SUCCESS` with zeroed params | +| `0x20800a70` | BUS flush with sysmembar | fallback: generic `SUCCESS` with zeroed params; UVM issues these frequently | **Log signature:** ``` @@ -125,8 +130,15 @@ CUDA process: 0xc46f TURING_CHANNEL_GPFIFO_A(368 bytes) ← the channel (GPFIFO VA, instance/userd descriptors) 0xc076 GP100_UVM_SW (0 bytes) 0xc5b5 TURING_DMA_COPY_A (8 bytes) ← CE object +0xc5c0 TURING_COMPUTE_A (16 bytes) ← compute object (engine registration path) ``` +The `TURING_COMPUTE_A` (0xc5c0) alloc is where the earlier `libcuda` segfault +was fixed: the caller issues it with `paramsSize=0` against an 8-byte stack +local while the kernel side treats the class params as 16 bytes and copies the +reply params back, so the emulation must echo the request params back in the +alloc reply instead of writing a reply the caller did not ask for. + Channel parameters decode into: `gpFifoOffset` (a GPU VA), `gpFifoEntries`, `engineType` (e.g. COPY0 = 0x9), and `NV_MEMORY_DESC_PARAMS` for instance/userd/ramfc/method buffers with @@ -149,9 +161,18 @@ page-tree init. **Device must:** let UVM's channels initialize (each channel does a first push and waits on a semaphore), and let page-tree initialization proceed. -**Emulation (today):** we can decode the pushes UVM generates and complete -their semaphore releases by translating the semaphore VA to a guest physical -address (see the UVM and host-module docs). This is the live development front. +**Emulation (today):** we can decode the pushes UVM generates and execute +their memsets, copies and semaphore releases in software by translating the +semaphore VA to a guest physical address (see the UVM and host-module docs). +This is a bridge, not forwarding: no work reaches the physical GPU yet. + +**Live development front (2026-09-23):** CUDA context creation — `cuCtxCreate` +returns `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks +the async copy engine (`COPY14`). The engine-record layout and the alloc +reply-fidelity checks are the first things to inspect +([`../../userspace/cuda/docs/05-570.144-decompilation.md` §3.3/§3.6](../../userspace/cuda/docs/05-570.144-decompilation.md); +start at +[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md)). **Log signatures (guest side):** ``` diff --git a/docs/04-implementation/01-known-knowns-and-unknowns.md b/docs/04-implementation/01-known-knowns-and-unknowns.md index 0564ffc..a085211 100644 --- a/docs/04-implementation/01-known-knowns-and-unknowns.md +++ b/docs/04-implementation/01-known-knowns-and-unknowns.md @@ -22,7 +22,10 @@ listed as an unknown-unknown has a *method* by which we will discover it. ### 1.1 Device and platform * A stock 570.144 Linux guest driver probes and initializes against our - emulated TU104 device; `nvidia-smi` works. + emulated TU104 device; `nvidia-smi` works and CUDA user-space starts + (`cuInit`, `cuDeviceGetCount`, `cuDeviceGet`). Context creation does not yet + complete: `cuCtxCreate` returns `rc=999` (`CUDA_ERROR_UNKNOWN`) because the + emulated engine list lacks the async copy engine (`COPY14`). * The same guest **boots unmodified**: no guest code, no guest module parameters are required for the initialization path. * The host is unmodified in the standalone configuration (distribution NVIDIA @@ -77,6 +80,15 @@ it**, **rough cost**. ### 2.1 Execution forwarding +**Immediate blocker (2026-09-23):** the engine topology is not yet right. +`GPU_GET_ENGINES_V2` returns 6 engines (GR + CE0..4) where a bare-metal oracle +returns 7: the async copy engine (`COPY14`; engine descriptor `0x793ceb0e`, +instance 14, client type `0x38`, class pair `0xc9b5`) is missing, so +`cuCtxCreate` returns `rc=999` (`CUDA_ERROR_UNKNOWN`). The zeroed secondary +engine record makes `libcuda` rewrite the channel alloc class to `0x41` +(`NV01_ROOT_CLIENT`) with status `0x19` and roll back. Teardown of the failed +context still oopses in the guest kernel in `uvm_pmm_devmem_page_to_va_space`. + **Unknown:** the correct, isolated way to run a guest's engine work on the physical GPU: which real scheduling primitive to use per engine, how to translate guest pushes to real submissions, and how to relay completion and @@ -129,7 +141,8 @@ forwarded kernel), capture both sides, implement replay; validate with ### 2.4 Version matrix beyond 570.144 **Unknown:** which parts of the observed protocol drift between driver -families (550/560/580/610+), how much is negotiated, and how much must be +families (550/560/570/580 — the families with entries under +`config/versions/`), how much is negotiated, and how much must be configuration. **Why:** durability is a core promise; users will run other versions @@ -269,7 +282,11 @@ tree is the artifact that survives personnel changes. To make a defensible statement, we need all of: 1. **Interfaces:** every guest→device message for the CUDA path is either - answered or forwarded, for the supported driver family. + answered or forwarded, for the supported driver family — and every reply + whose fields the driver consumes is coherent, not just `SUCCESS`-shaped + (the reply-fidelity ladder: status-only → coherent zeroed params → coherent + values for consumed fields → exact values; see + [`../../userspace/cuda/docs/04-virtualization-and-emulation.md` §6](../../userspace/cuda/docs/04-virtualization-and-emulation.md)). 2. **Memory:** every GPU VA used by work is translatable, and every server-owned page-table range is populated correctly. 3. **Execution:** grid launches and copies run on real hardware with correct @@ -283,7 +300,11 @@ To make a defensible statement, we need all of: measured with a published methodology. Until all seven hold, we say "initialization works; execution is in progress", -and we link here. +and we link here. The CUDA-side checklist behind these gates and the current +blocker are in +[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md` §9](../../userspace/cuda/docs/03-driver-and-gpu-interface.md); +start at +[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md). --- diff --git a/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md b/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md index 1deff02..e2769f8 100644 --- a/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md +++ b/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md @@ -18,7 +18,11 @@ the project portable, maintainable and legally clean. Correctness is not "no crash". It is: * the guest observes the same **values** (capabilities, handles, table - contents) it would on real hardware, + contents) it would on real hardware — a control whose reply fields the + driver consumes is only "implemented" once its reply is coherent, not just + `SUCCESS`-shaped (the reply-fidelity ladder: status-only → coherent zeroed + params → coherent values for consumed fields → exact values; see + [`../../userspace/cuda/docs/04-virtualization-and-emulation.md` §6](../../userspace/cuda/docs/04-virtualization-and-emulation.md)), * the guest observes the same **ordering** (completions happen after the work they signal; faults arrive before replays), * the guest observes the same **failure semantics** (errors are surfaced, not @@ -48,8 +52,10 @@ without a methodology is not accepted in this project. * **Management operations** (object creation, capability queries, mappings) are answered from cached/derived state where the hardware has no side effects the guest can observe. -* **Real work** (grid launches, copies) is forwarded. Nothing else is allowed - on the hot path. +* **Real work** (grid launches, copies) is meant to be forwarded to the + physical GPU (see §3). Today only the work the host decoder understands — + memsets, copies and semaphore releases — is executed in software as a + correctness bridge; no CUDA kernel is forwarded yet. ## 3. Forwarding: the three options and how we choose @@ -99,15 +105,25 @@ means maintaining forks for every guest OS and driver version. A patch to the hypervisor means every hypervisor project must carry it. A firmware change is impossible in practice. +In this repository, patch files exist only as build glue for the *optional* +OpenRM submodule build (`legacy-patches/nvidia/`, applied by `install.sh`). +The runtime and the guest stack are never patched, and the deployment path uses +the distribution driver — so nothing patched is required to run the stack. + ### 5.2 How compliance is verified -* **Contract tests**: CI builds run against *stock* drivers obtained from - distribution repositories (or vendor runfiles), and against *unmodified* - hypervisor packages. +* **Contract tests** (goal): CI builds run against *stock* drivers obtained + from distribution repositories (or vendor runfiles), and against + *unmodified* hypervisor packages. Today CI is push/PR-triggered only — builds, + kernel/unit/KUnit/mock tests, static analysis and docs checks; there is no + scheduled guest suite or stock-driver contract test yet, so the guest ladder + is still run by hand (see the operations docs). * **No-patch checks**: our build must not depend on any patch file for third-party components. Where patches exist in the tree for convenience - (e.g. build glue), the contract tests must pass without them; anything that - fails this is a bug. + (e.g. build glue for the optional OpenRM submodule), they are development + scaffolding only: the contract tests must pass without them, and the + deployment path uses the unpatched distribution driver; anything that fails + this is a bug. * **Behavioral equivalence**: where we answer from emulation, we validate the values against traces from real hardware (or from our decompiled study notes) — not against our own previous answers. diff --git a/docs/05-operations/01-performance-correctness-security.md b/docs/05-operations/01-performance-correctness-security.md index c4d8153..0a27709 100644 --- a/docs/05-operations/01-performance-correctness-security.md +++ b/docs/05-operations/01-performance-correctness-security.md @@ -17,6 +17,19 @@ Rung 4 SUSTAINED workloads run for hours; faults/errors recoverable Rung 5 MULTI-TENANT N VMs, isolation + fairness + no cross-talk ``` +**Current rung (2026-09-23):** Rung 1 is green; **Rung 2 is NOT green.** +`cuInit`/`cuDeviceGet` succeed, but `cuCtxCreate` returns `rc=999` +(`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks the async copy +engine (`COPY14`); `cuMemAlloc` is not reached and teardown of the failed +context still oopses in the guest kernel. Before Rung 3 is claimed, the +**reply-fidelity gate** applies: a control whose reply fields the driver +consumes is not implemented until the reply is coherent (status-only → +coherent zeroed params → coherent values for consumed fields → exact values) — +see +[`../../userspace/cuda/docs/04-virtualization-and-emulation.md` §6](../../userspace/cuda/docs/04-virtualization-and-emulation.md). +New to CUDA analysis here: start at +[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md). + ## 2. Correctness ### 2.1 Test corpus @@ -47,9 +60,11 @@ Rung 5 MULTI-TENANT N VMs, isolation + fairness + no cross-talk ### 2.3 Regression gates -`make test` runs the host suite. The CI ladder runs the guest suite on a -scheduled cadence and on every release. A rung regression is a release blocker; -the handoff documents carry the current rung. +`./scripts/stack_manager.sh test` runs the host suite (`make tests` only builds +the test binaries). CI (`.github/workflows/ci.yml`) runs on push and pull +requests only — there is currently no scheduled guest suite and no +stock-driver contract test. A rung regression is a release blocker; the +handoff documents carry the current rung. ## 3. Performance diff --git a/docs/05-operations/02-debugging-and-maintenance-workflows.md b/docs/05-operations/02-debugging-and-maintenance-workflows.md index bc38a4e..331597a 100644 --- a/docs/05-operations/02-debugging-and-maintenance-workflows.md +++ b/docs/05-operations/02-debugging-and-maintenance-workflows.md @@ -88,7 +88,7 @@ Workflow that made the memory model tractable: ### 3.4 Guest-side tracing -* `ptrace` tracer (`tools/guest-diag/ptrace_ioctl.c`): libcuda issues raw +* `ptrace` tracer (`tools/guest-diag/ioctl_trace.c`): libcuda issues raw syscalls, so `LD_PRELOAD` shims see nothing; PTRACE_SYSCALL with `PTRACE_GET_SYSCALL_INFO` does. * UVM debug: `echo 1 > /sys/module/nvidia_uvm/parameters/uvm_debug_prints` @@ -101,11 +101,60 @@ When a failure is understood, write the decision tree into this document tree — the tree is what makes the next failure cheap. The field guide's tables are the result of that discipline so far. +### 3.6 Userspace driver (`libcuda`) crash workflow + +When the guest segfaults inside the userspace driver, the fault is reported as +`lib[base+size] ip`. Resolve it and find the code before reading any +decompilation: + +```bash +# ip/mapping-base decoder: prints the file vaddr, containing function start, +# and nearest export +userspace/cuda/tools/cudatool.py crash <libcuda.so> <mapping_base> <ip> + +# decompile exactly one file vaddr (translates for the Ghidra image base) +userspace/cuda/tools/ghidra_decompile_at.sh <ghidra-project> <program> <libcuda.so> 0xNNNNNN /tmp/fn.c + +# callers/callees of an address in the annotated corpus +userspace/cuda/tools/callwalk.py <elf-index>/callgraph.tsv callers 0xNNNNNN +``` + +* **Address convention**: public offsets are *file virtual addresses* + (`libcuda+0x...`). Ghidra loads the CUDA userspace library at **image base + `0x100000`**, so addresses in old decompilations produced without that + translation are 0x100000 too low (the wrong functions); runtime + observations from those sessions remain valid, but conclusions drawn from + their decompiled code must be re-derived. The current tooling translates + for you. +* The annotated corpus (layout in + [`../../userspace/cuda/READ-ME-FIRST.md` §4](../../userspace/cuda/READ-ME-FIRST.md)) + is searchable by `libcuda+0x...`; names/indexes are our own work product. +* **Engine topology**: `tools/guest-diag/guest_engines.c` dumps the guest's + engine/class list, virtualization mode and CE caps; compare against the + host oracle (the same query on the host driver) before blaming the + emulation. +* **A/B harness**: `tools/cuda-ab/` runs one deterministic CUDA workload on + bare metal and in the guest and diffs the stable result lines. + +**Known current bugs (2026-09-23)**: `cuCtxCreate` returns `rc=999` +(`CUDA_ERROR_UNKNOWN`) because the emulated engine list is missing the async +copy engine `COPY14`; the guest kernel oopses in +`uvm_pmm_devmem_page_to_va_space` on teardown of the failed context; and +control `0x2080012b` (560-byte params, issued immediately before context +creation) is answered with a generic SUCCESS whose consumed fields are not yet +coherent. Start at +[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md), +[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../../userspace/cuda/docs/03-driver-and-gpu-interface.md) +and +[`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md). + ## 4. Workflows for new inputs ### 4.1 New firmware version -1. Add the image under `firmware/<SERIES>-GSP/<VERSION>/` (tracked by LFS). +1. Add the image under `firmware/<SERIES>-GSP/<VERSION>/` (local study + material; firmware blobs are gitignored and not distributed in the + repository). 2. Compute signatures (CRC32/SHA256) with the firmware analyzer. 3. Add a firmware profile entry (addresses/signatures) so runtime analysis can identify it. @@ -118,8 +167,7 @@ the result of that discipline so far. 1. Install the stock driver in a guest (no patches). 2. Capture the full init trace and diff function/control IDs against the previous version. -3. For each difference: decode against our decompiled study notes for that version - (`g_rpc-structures.h`, `rpc_global_enums.h`, control headers). +3. For each difference: decode against the exact-version reference headers. 4. Add a version-keyed handler or a capability table entry. 5. Re-run rungs 1–2; then the CUDA corpus. diff --git a/docs/06-portability/hardware-software-matrix.md b/docs/06-portability/hardware-software-matrix.md index 442e099..b8393f4 100644 --- a/docs/06-portability/hardware-software-matrix.md +++ b/docs/06-portability/hardware-software-matrix.md @@ -33,7 +33,7 @@ require configuration + verification, not new code.** | Guest | Status | |---|---| -| Linux x86_64 (stock `nvidia.ko`, CUDA) | **Reference** (initialization; execution in progress) | +| Linux x86_64 (stock `nvidia.ko`, CUDA) | **Experimental** (bring-up; probe and `nvidia-smi` work, but there is no continuous guest testing and context creation is currently blocked — see §6.1) | | Linux aarch64 | **Target** | | Windows x86_64 (WDDM driver, CUDA, 3D) | **Target** | | Other Unix (FreeBSD, ...) | Out of scope until demand | @@ -82,18 +82,44 @@ tree, add configuration/handlers, re-run the ladder. ## 6. Version matrix (maintained in the repository) -The authoritative matrix lives in `config/`: +The version/hardware configuration lives in `config/`: ``` config/ hardware/<arch>.yaml # PCI IDs, BARs, memory geometry, engine lists versions/<driver>.yaml # RPC/control inventories, capability tables - profiles/<name>.yaml # (hardware, driver, firmware) combinations we test ``` +There is no `profiles/` directory today. Note also that the code currently +hardcodes the TU104 profile, so the YAML surface is the intended interface +rather than the path the module reads at load time — treating it as +authoritative is aspirational until the profiles are wired in. + A profile is the unit of support: "hardware X + driver family Y + firmware Z = profile P, tested at rung R on date D." Releases publish the profile table. +### 6.1 Userspace driver facts + +* **Guest CUDA driver** (`libcuda`) under test: **570.144** (md5 + `ed5b2fc0044df6052df1d5113754c499`). +* **Host oracle**: `libcuda` **570.211.01** with CUDA toolkit **12.0**, used + for A/B comparisons against the guest. +* **Versioned surface**: the RM ioctl / class / control interface the userspace + driver issues — not a stable public API — see + [`../../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../../userspace/cuda/docs/03-driver-and-gpu-interface.md). + +### 6.2 Engine-list portability + +The engine inventory, and how it is filtered by the class list (classDB), are +per-architecture and per-driver data. Today's gap is concrete: the emulated +list lacks the async copy engine (`COPY14`), which blocks context creation +(`cuCtxCreate` returns `rc=999`). Engine inventory belongs in the hardware +profile, validated against the host oracle per combination. For the +re-analysis workflow see +[`../../userspace/cuda/docs/07-version-upgrade-playbook.md`](../../userspace/cuda/docs/07-version-upgrade-playbook.md); +start at +[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md). + ## 7. Architecture properties that enable portability | Property | Why it matters | diff --git a/docs/07-developers/onboarding-and-standards.md b/docs/07-developers/onboarding-and-standards.md index 78503f4..3d7ac78 100644 --- a/docs/07-developers/onboarding-and-standards.md +++ b/docs/07-developers/onboarding-and-standards.md @@ -17,7 +17,9 @@ — where the code lives (30 min). 5. [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) — what to work on and what is hard (30 min). -6. The subsystem doc for whatever you plan to touch. +6. Working on CUDA: [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md) + — CUDA stack analysis, tooling, and the current blockers. +7. The subsystem doc for whatever you plan to touch. ### 1.2 Environment @@ -53,6 +55,8 @@ the VM holding it. The stack manager enforces the order. open-iov/ ├── kernel/ host module (GSP emulation, device model, memory model) ├── userspace/ management library + examples +├── userspace/cuda/ CUDA stack analysis + tooling (start at +│ userspace/cuda/READ-ME-FIRST.md) ├── tools/ debugging and analysis tools (guest-diag, rmq, │ firmware analysis wrappers) ├── scripts/ build/load/test automation, VM harness @@ -96,8 +100,12 @@ open-iov/ the definition of done. * Prefer a table or a decision tree over prose for anything an operator will follow under pressure. -* Protocol facts (IDs, offsets, structures) must cite their source - (the decompiled study notes, or a live trace). +* Protocol facts (IDs, offsets, structures) must cite their source (a live + trace, or an analysis artifact). Userspace (`libcuda`) addresses are *file + virtual addresses* (`libcuda+0x...`); Ghidra loads that library at **image + base `0x100000`**, so old decompilations produced without translating the + offset are 0x100000 too low (the wrong functions) — re-derive any conclusion + drawn from them. ## 4. Tooling @@ -105,7 +113,10 @@ open-iov/ |---|---| | `scripts/stack_manager.sh` | build/load/status/test/unload with correct ordering | | `scripts/vgpu_vm.py` | VM lifecycle incl. guest-agent exec/shell | -| `tools/guest-diag/` | guest-side diagnostics (ptrace tracer, engines, probes, push helper) | +| `tools/guest-diag/` | guest-side diagnostics (ioctl tracer, engines, probes, push helper) | +| `userspace/cuda/tools/` | CUDA binary analysis (crash decoder, callgraph walk, decompile-at, corpus annotation) | +| `tools/cuda-ab/` | host-vs-guest CUDA A/B harness (one deterministic workload, diffed results) | +| `tools/rmq/` | host RM-API oracle for ground-truth GPU/RM data | | `tools/firmware_analyzer.py` + Ghidra scripts | firmware analysis pipeline | | module debug parameters | guest-memory scanning/poking, VRAM inspection | | QEMU gdbstub workflow | guest virtual-memory inspection/writing | @@ -128,9 +139,11 @@ open-iov/ ```bash make kernel # open-iov-vfio.ko (first!) -make nvidia # NVIDIA OpenRM modules with KBUILD_EXTRA_SYMBOLS +./scripts/build_nvidia_openrm.sh # optional OpenRM modules, KBUILD_EXTRA_SYMBOLS from kernel/Module.symvers make userspace # libiov.so, examples make all # in the correct order +make tests # build the host test binaries only (does not run them) +./scripts/stack_manager.sh test # run the host test suite make clean ``` diff --git a/docs/08-ecosystem/haaeep-and-collaboration.md b/docs/08-ecosystem/haaeep-and-collaboration.md index c4fb352..e559840 100644 --- a/docs/08-ecosystem/haaeep-and-collaboration.md +++ b/docs/08-ecosystem/haaeep-and-collaboration.md @@ -92,7 +92,7 @@ licensed and version-durable — so it is the component we are building. | Milestone | Depends on | |---|---| -| CUDA execution in a guest (rung 3) | Open-IOV forwarding work | +| CUDA execution in a guest (rung 3) | Rung 2 (runtime start) being green first — it is not yet: `cuCtxCreate` returns `rc=999` because the emulated engine list lacks the async copy engine (`COPY14`) — and then Open-IOV forwarding work. The interface contract is documented under [`../../userspace/cuda/README.md`](../../userspace/cuda/README.md) (start at [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md)) | | Two-VM coexistence (inference + agent) with isolation | Open-IOV multi-tenancy + platform MAC | | Xen/OpenXT or Qubes Air variant | Open-IOV transport backends | | Attested boot-to-inference chain | TrenchBoot integration | @@ -150,5 +150,6 @@ publish findings and fixes, and we will credit you. us (model server, KV cache, agent harness). * **Everyone**: run the ladder on your hardware and send the evidence. -Contact and participation channels are listed in the repository README. We -would rather have a hundred partners than a hundred forks. +Contact and participation channels are listed in +[`../README.md` §6 "How to get involved"](../README.md#6-how-to-get-involved). +We would rather have a hundred partners than a hundred forks. diff --git a/docs/09-reference/glossary-and-faq.md b/docs/09-reference/glossary-and-faq.md index 255d1a6..2ec4faa 100644 --- a/docs/09-reference/glossary-and-faq.md +++ b/docs/09-reference/glossary-and-faq.md @@ -5,11 +5,15 @@ | Term | Meaning | |---|---| | **AMAPlib** | NVIDIA's shared "address mapping" library: swizzling, bank/channel mapping, aperture rules. Studied for page-table and memory-layout semantics. | +| **Async copy engine (COPY14)** | A copy-engine instance found in the host oracle's engine list but missing from the emulated list — engine descriptor `0x793ceb0e`, instance 14, client type `0x38`, class pair `0xc9b5`. Its absence is the current CUDA context-creation blocker. | | **BAR0 / BAR1 / BAR3** | PCI BAR windows of the GPU: register interface (16 MB), VRAM aperture (256 MB), instance/PRAMIN window. | | **CE** | Copy Engine: DMA engine; also used for UVM's internal work. | +| **Class list (classDB)** | The set of RM classes reported as supported (`GPU_GET_CLASSLIST`); engine queries are filtered against it, so the class list, engine list and caps controls must agree. | | **CUDASW** | The family of NVIDIA user-space software that sits between applications and the driver: `libcuda`, `libcudart`, JIT compiler, fatbinary loader, and friends. | | **Doorbell** | A register/memory location a driver writes to signal "new work is available" for a channel or queue. | | **Engine type** | Numeric identifier for an execution engine (`GR = 0x1`, `COPY0 = 0x9`, ...). | +| **Engine record / engine list** | The per-engine descriptors (kind, instance, class, client type) the driver walks after `GPU_GET_ENGINES`/`GET_ENGINES_V2`; each record must be self-consistent or the driver rewrites the following allocation. | +| **File virtual address (`libcuda+0x...`)** | An address relative to the start of a shared object file (e.g. a `libcuda` offset), as distinct from a runtime mapping or a Ghidra program address; public docs cite these. | | **FWSEC** | Firmware security processor stage of the GSP boot sequence. | | **GNND / GB10** | NVIDIA "Grace-Blackwell" systems-on-chip used in DGX Spark and Surface-class laptops; explicit portability targets. | | **GPFIFO** | GPU FIFO: a ring of entries, each pointing to a push buffer for a channel. | @@ -18,12 +22,15 @@ | **GSP-RM** | The Resource Manager running on the GSP ("self-hosted RM"). | | **GMMU** | GPU Memory Management Unit: page tables, TLB, fault handling. | | **HAL** | Hardware Abstraction Layer: per-generation functions that program registers and formats. | +| **Host oracle / A-B harness** | The same query or deterministic workload run against a bare-metal (host) driver and inside the guest, then diffed — our reference for what the guest should have seen. | | **mdev** | Mediated device: the kernel mechanism that turns a physical device into virtual instances (VFIO). | | **MSGQ** | The GSP message queue transport: shared-memory element rings with a page table, plus doorbells. | | **PBDMA** | Push-buffer DMA engine in the host scheduler that fetches and executes push buffers. | | **PDB** | Page Directory Base: root of a GPU page table. | | **PDE / PTE** | Page-directory entry / page-table entry. | | **Push buffer** | A stream of method calls (engine commands) submitted via a GPFIFO. | +| **QMD** | Queue Meta Data: the per-launch descriptor a GR channel reads to execute a kernel grid. | +| **Reply fidelity** | How completely an emulated reply matches what real hardware returns, as a ladder: status-only → coherent zeroed params → coherent values for consumed fields → exact values. | | **RMCore** | The RM's object system: classes, handles, parents, methods. | | **RM / RM API** | NVIDIA's Resource Manager and the API used to allocate objects and issue controls. | | **resserv / resource server** | The RM component that owns allocation authority, privileges and lifetimes. | @@ -57,10 +64,14 @@ GPU and Open-IOV is an independent module. Building NVIDIA's OpenRM modules from source is only needed for the native-vGPU-management workflows. **Why is `nvidia-smi` working but CUDA not?** -Because CUDA requires *execution*: kernels must run on real hardware. The -current state runs the entire initialization path (including UVM's start-up) -but stops short of executing engine work. See the known-knowns document for -the exact status and plan. +CUDA startup reaches `cuInit` / `cuDeviceGet`, but context creation is blocked: +the emulated engine list lacks the async copy engine (`COPY14`), so +`cuCtxCreate` returns `rc=999` (`CUDA_ERROR_UNKNOWN`) and memory allocation +never runs. (The earlier `libcuda` segfault on this path was fixed by echoing +the request params back in the alloc reply.) Once context creation is green, +CUDA still needs *execution*: kernels must run on real hardware. See the +known-knowns document for the exact status and plan; start at +[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md). **How is this legal / license-clean?** Open-IOV is BSD-3-Clause (GPL-2.0 for kernel modules that link GPL-only kernel @@ -85,9 +96,11 @@ MIG is a hardware partitioning mechanism; Open-IOV's scheduling policy can use it where available. It is part of the multi-tenancy work. **How fast is it?** -Initialization currently completes in the same order of magnitude as native; -execution forwarding is not implemented yet. Performance claims follow the -published methodology in the operations docs. +No published numbers yet: execution forwarding is not implemented, and even +the initialization path has not been measured under the project's methodology. +Any performance claim will come with the methodology and the measurements +together (see +[`../05-operations/01-performance-correctness-security.md`](../05-operations/01-performance-correctness-security.md)). **Where do I report bugs or ask for help?** Open an issue with the standard captures listed in the debugging document. A diff --git a/docs/09-reference/legal-and-licensing.md b/docs/09-reference/legal-and-licensing.md index 74f1363..57aad43 100644 --- a/docs/09-reference/legal-and-licensing.md +++ b/docs/09-reference/legal-and-licensing.md @@ -30,9 +30,12 @@ In this repository you will find: * **GSP firmware images** (`firmware/**/*.fwimage.bin`) — fetched from public distribution channels (e.g. `linux-firmware`); used to understand and - emulate the firmware interface. -* **Decompiled firmware artifacts** (`firmware/**/decompiled/`) — our own - analysis output; kept for study and cited in the wiki. + emulate the firmware interface. They are **not distributed in this + repository**: like all firmware blobs they are gitignored and kept only as + local study material. +* **Decompiled/disassembled artifacts** (`firmware/**/decompiled/`) — our own + analysis output, kept locally for study and cited in the wiki. They are + gitignored and never committed. * **Reformatted interface tables** — IDs, structures and message layouts that we derived, published in `wiki/` and in this documentation tree. @@ -42,13 +45,28 @@ do not point readers at material we cannot redistribute, and do not redistribute third-party binaries as project content beyond what their terms allow. +**CUDA-userspace (`libcuda`) analysis rules:** + +* Decompiled or disassembled output is never committed; it is gitignored study + material. +* Cite userspace locations as `libcuda+0x...` *file virtual addresses*. Note + the Ghidra image-base caveat: that library is loaded at **`0x100000`**, so a + decompilation produced without translating the offset refers to the wrong + function. +* Tracked docs state facts and evidence; conclusions cite numeric IDs, code + offsets and recorded observations, never source locations. +* Name maps and indexes (`userspace/cuda/*/annotations/`) are our own work + product and may be tracked. + ## 3. Adversarial interoperability and the law We implement the *interface contract* so that unmodified NVIDIA software can talk to our device. We do not: * copy NVIDIA driver code into our kernel module, -* patch NVIDIA drivers or firmware, +* patch the guest, runtime or host driver at run time. The only patch series + we carry is build glue for the optional OpenRM submodule (`install.sh`); the + deployment path uses the distribution's NVIDIA driver as shipped, * defeat licensing for NVIDIA's own products (our device is our device; no impersonation of a licensed product identity). @@ -68,8 +86,8 @@ open an issue: we would rather over-comply than argue. contribution does not contain third-party material under incompatible terms. * Write original code. Describe behaviour, cite the evidence, and reimplement interfaces from observation rather than copying anyone's source. -* Do not commit firmware or ISO images unless they are explicitly permitted and - listed in `.gitattributes`/LFS policy. +* Do not commit firmware or ISO images: they are local study/deployment + material and are gitignored (see §2). ## 5. Export, safety and platform notes diff --git a/docs/BUILD_NVIDIA_OPENRM.md b/docs/BUILD_NVIDIA_OPENRM.md index 5699036..309a64f 100644 --- a/docs/BUILD_NVIDIA_OPENRM.md +++ b/docs/BUILD_NVIDIA_OPENRM.md @@ -1,5 +1,10 @@ # Building NVIDIA OpenRM with Open-IOV vGPU VFIO Symbols +> **Status: historical build notes.** These notes predate the current workflow. +> For the current build/load model see +> [03-architecture/06-host-module-and-hybrid-model.md](03-architecture/06-host-module-and-hybrid-model.md) +> and [02-getting-started/01-install-and-first-run.md](02-getting-started/01-install-and-first-run.md). + ## Overview The NVIDIA OpenRM `nvidia.ko` kernel module requires the vGPU VFIO interface symbols exported by `open-iov-vfio.ko`: @@ -18,8 +23,8 @@ These symbols must be resolved at build time via `KBUILD_EXTRA_SYMBOLS` pointing ``` 1. Build open-iov-vfio.ko → Generates kernel/Module.symvers 2. Build NVIDIA OpenRM with KBUILD_EXTRA_SYMBOLS=kernel/Module.symvers -3. Load nvidia.ko first -4. Load open-iov-vfio.ko +3. Load open-iov-vfio.ko first (exports the vGPU VFIO symbols) +4. Optionally load nvidia.ko (imports those symbols) ``` ## Automated Build (Recommended) @@ -76,12 +81,14 @@ modinfo nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko | grep -i vgpu ## Module Loading Order ```bash -# 1. Load NVIDIA OpenRM (provides vGPU VFIO symbols) -sudo insmod nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko - -# 2. Load Open-IOV VFIO driver (registers mdev parent with nvidia.ko) +# 1. Load Open-IOV VFIO driver FIRST (exports the vGPU VFIO symbols and +# registers the mdev parent on the physical GPU) sudo insmod kernel/open-iov-vfio.ko +# 2. Load the optional NVIDIA OpenRM (imports the vGPU VFIO symbols; build it +# with KBUILD_EXTRA_SYMBOLS as above) +sudo insmod nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko + # 3. Copy firmware sudo cp firmware/57X-GSP/570.144/gsp-570.144.fwimage.bin /lib/firmware/ diff --git a/docs/DEBUGGING.md b/docs/DEBUGGING.md index dc95ed7..e2d7fe0 100644 --- a/docs/DEBUGGING.md +++ b/docs/DEBUGGING.md @@ -1,5 +1,11 @@ # Open-IOV Debugging Guide +> **Status: superseded (historical).** Part of the pre-2026-09-23 docs; kept +> as a record. Current entry point: [docs/README.md](README.md). Debugging +> replacement: +> [05-operations/02-debugging-and-maintenance-workflows.md](05-operations/02-debugging-and-maintenance-workflows.md); +> the §3–§4 case studies here remain useful. + > Battle-tested procedures for bringing up the vGPU stack, diagnosing guest > driver failures, and validating a fully working setup. Built from real > debugging sessions (including the 2026-09-21 EPT-livelock hunt). diff --git a/docs/DEVELOPMENT_WORKFLOW.md b/docs/DEVELOPMENT_WORKFLOW.md index 9019ac1..5d43896 100644 --- a/docs/DEVELOPMENT_WORKFLOW.md +++ b/docs/DEVELOPMENT_WORKFLOW.md @@ -1,5 +1,15 @@ # Open-IOV vGPU-IO Development Workflow & Tools Guide +> **Historical document — do not follow as written (2026-09-21).** This guide +> predates the current workflow: it contains the old nvidia-first load order, +> reboot advice, and LFS instructions that no longer apply (LFS was removed; +> firmware and decompiled artifacts are local/gitignored). The current +> workflow is +> [`../scripts/stack_manager.sh`](../scripts/stack_manager.sh), +> [`07-developers/onboarding-and-standards.md`](07-developers/onboarding-and-standards.md) +> and +> [`02-getting-started/01-install-and-first-run.md`](02-getting-started/01-install-and-first-run.md). + ## Overview This document describes the complete development workflow, tooling, and shell integration for working on the Open-IOV vGPU-IO project. It covers everything from firmware reverse engineering to QEMU guest debugging. diff --git a/docs/GSP_EMULATION_ARCHITECTURE.md b/docs/GSP_EMULATION_ARCHITECTURE.md index 8da8f49..70b4b76 100644 --- a/docs/GSP_EMULATION_ARCHITECTURE.md +++ b/docs/GSP_EMULATION_ARCHITECTURE.md @@ -1,5 +1,14 @@ # Open-IOV GSP Emulation Engine — Architecture +> **Status: superseded (historical).** Part of the pre-2026-09-23 docs; kept +> as a record. Current entry point: [docs/README.md](README.md). Architecture +> replacement: +> [03-architecture/01](03-architecture/01-system-overview.md) and +> [03-architecture/06](03-architecture/06-host-module-and-hybrid-model.md); +> the transport section in particular is superseded by the msgq +> documentation in +> [03-architecture/03](03-architecture/03-gsp-and-gsp-rm.md). + > **Audience**: developers working on `kernel/gsp_emulation.c`, `kernel/gsp_rpc_client.c`, > `kernel/bar_emulation.c`, or debugging guest driver bring-up. > diff --git a/docs/ONBOARDING.md b/docs/ONBOARDING.md index bc61a03..615e016 100644 --- a/docs/ONBOARDING.md +++ b/docs/ONBOARDING.md @@ -1,5 +1,10 @@ # Open-IOV Developer Onboarding +> **Status: superseded (historical).** Part of the pre-2026-09-23 docs; kept +> as a record. Current entry point: [docs/README.md](README.md). Onboarding +> replacement: +> [07-developers/onboarding-and-standards.md](07-developers/onboarding-and-standards.md). + > Welcome. This guide gets you from zero to a productive contributor in ~30 > minutes: what the project is, how the repo is laid out, how to build and > test, and the standards we hold the code to. diff --git a/docs/PERFORMANCE.md b/docs/PERFORMANCE.md index 6077920..6ba8d1b 100644 --- a/docs/PERFORMANCE.md +++ b/docs/PERFORMANCE.md @@ -1,5 +1,10 @@ # Open-IOV Performance Testing & A/B Methodology +> **Status: superseded (historical).** Part of the pre-2026-09-23 docs; kept +> as a record. Current entry point: [docs/README.md](README.md). Performance +> replacement: +> [05-operations/01-performance-correctness-security.md](05-operations/01-performance-correctness-security.md). + > How we measure vGPU performance against bare metal, interpret the results, > and keep overhead on the critical path honest. **Run correctness first** > (`docs/DEBUGGING.md` §5) — numbers from a half-working stack are noise. diff --git a/docs/PORTABLE_ARCHITECTURE.md b/docs/PORTABLE_ARCHITECTURE.md index 47d0297..4842407 100644 --- a/docs/PORTABLE_ARCHITECTURE.md +++ b/docs/PORTABLE_ARCHITECTURE.md @@ -1,5 +1,10 @@ # Open-IOV vGPU-IO Portable Architecture Design +> **Status: superseded (historical).** Part of the pre-2026-09-23 docs; kept +> as a record. Current entry point: [docs/README.md](README.md). Portability +> replacement: +> [06-portability/hardware-software-matrix.md](06-portability/hardware-software-matrix.md). + ## Design Goals ### 1. Driver Version Portability diff --git a/docs/README.md b/docs/README.md index fff4f82..cee435e 100644 --- a/docs/README.md +++ b/docs/README.md @@ -97,24 +97,36 @@ and [`01-orientation/02-the-ai-edge-and-ecosystem.md`](01-orientation/02-the-ai- | Emulated vGPU device (PCI, BARs, MSI-X, VFIO mdev) | ✅ Working | | GSP boot emulation (FWSEC/booter/RISCV handshake) | ✅ Working | | GSP-RM message queue + RPC transport | ✅ Working | -| GSP-RM controls: device/fb/bus tables, CE caps, vaspace, GMMU static info | ✅ Working | +| GSP-RM controls: device/fb/bus tables, CE caps, vaspace, GMMU static info | 🟡 Mostly — values are coherent for the special-cased controls; engine inventory/class-list consistency is the current CUDA blocker (COPY14) | | Guest `nvidia.ko` probe, `nvidia-smi` | ✅ Working (TU104) | -| Guest CUDA driver (`libcuda`) startup | 🟡 Partially — UVM advances through channel creation and page-tree init; see [`04-implementation/01-known-knowns-and-unknowns.md`](04-implementation/01-known-knowns-and-unknowns.md) | -| CUDA kernel execution | 🔴 Not yet — requires GPU work forwarding | +| Guest CUDA driver (`libcuda`) startup | 🟡 Partially — `cuInit`/device enumeration work; `cuCtxCreate` returns `999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine list is missing the async copy engine (COPY14). See [`04-implementation/01-known-knowns-and-unknowns.md`](04-implementation/01-known-knowns-and-unknowns.md) and [`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md) | +| CUDA kernel execution | 🔴 Not yet — context creation must succeed first; then GPU work forwarding (today only decoded memsets/copies/semaphore releases execute, in the host) | | OpenGL / Vulkan | 🔴 Not yet | | Windows guest | 🔴 Not yet | | Multi-tenant isolation (VFs, mdev-per-VM) | 🟡 Designed, partially implemented | -We also know exactly *why* CUDA kernels do not run yet, and what the path is: -see the known-knowns/known-unknowns document. The short version: the emulation -is now faithful enough that the guest stack reaches UVM initialization; the -missing piece is forwarding the guest's *work* (push buffers, page trees, -execution) to the physical GPU under our control. +We also know exactly where the CUDA bring-up stands and what the next step is: +the userspace `libcuda` segfault was fixed (an RM allocation reply now echoes +the request parameters instead of returning zeros), and the remaining blocker +is data: the engine list the emulation reports must include the async copy +engine and stay consistent with the class list, because `libcuda` filters +engines against it. The full mechanism, with offsets and the decompiled call +path, is in [`../userspace/cuda/docs/05-570.144-decompilation.md`](../userspace/cuda/docs/05-570.144-decompilation.md). +Beyond that, the missing piece for kernel execution is forwarding the guest's +*work* (push buffers, page trees, execution) to the physical GPU under our +control; see [`04-implementation/02-practical-challenges-and-adversarial-interop.md`](04-implementation/02-practical-challenges-and-adversarial-interop.md). ## 4. How to read these docs (progressive disclosure) Read in order if you are new. Jump directly if you know what you need. +> **Debugging the CUDA bring-up?** Start with +> [`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md) — +> a five-minute guide to the userspace driver analysis, the corrected crash +> path, the current engine-topology blocker, and the lookup recipes for +> `libcuda+0x...` offsets. It links back into this tree for the interface and +> emulation contract. + | # | Document | Audience | Time | |---|----------|----------|------| | 1 | [Purpose and scope](01-orientation/01-purpose-and-scope.md) | everyone | 10 min | @@ -136,11 +148,21 @@ Read in order if you are new. Jump directly if you know what you need. | 17 | [HAAEEP and collaboration](08-ecosystem/haaeep-and-collaboration.md) | stakeholders, partners | 20 min | | 18 | [Glossary and FAQ](09-reference/glossary-and-faq.md) | everyone | reference | -Existing deep-dive material (kept, still valid, being folded into this tree): -the `wiki/` directory (firmware analysis, address reference, wire protocol, -RPC message format) and the `docs/` reference documents -(`GSP_EMULATION_ARCHITECTURE.md`, `DEBUGGING.md`, `PERFORMANCE.md`, -`BUILD_NVIDIA_OPENRM.md`, `VERSION_UPGRADE.md`). +Deep-dive material lives in [`../wiki/`](../wiki/README.md) (firmware +decompilation record, address reference, historical protocol notes). Wiki +pages that predate the current implementation carry a status banner naming +the canonical document in this tree; use the banner, not the old body. + +Historical material is kept but must not be read as current state. The +following carry a **superseded** banner and point at their replacements: +`refactored-architecture.md`, `refactored-implementation-to-do.md`, +`PORTABLE_ARCHITECTURE.md`, `GSP_EMULATION_ARCHITECTURE.md`, `ONBOARDING.md`, +`DEVELOPMENT_WORKFLOW.md`, `PERFORMANCE.md`, `DEBUGGING.md` (sections 3–4 +remain useful case studies), `VERSION_UPGRADE.md`, `BUILD_NVIDIA_OPENRM.md` +(historical build notes; its load order is corrected in place), and +[`../STATE.md`](../STATE.md) (2026-09-21 snapshot). The current versions of +those topics are `03-architecture/01`/`06`, `04-implementation/01`/`02`, +`05-operations/01`/`02`, `06-portability`, and `07-developers`. ## 5. What makes Open-IOV different? diff --git a/docs/VERSION_UPGRADE.md b/docs/VERSION_UPGRADE.md index b9bd387..3fc5de4 100644 --- a/docs/VERSION_UPGRADE.md +++ b/docs/VERSION_UPGRADE.md @@ -1,5 +1,12 @@ # Open-IOV Version Upgrade Guide +> **Status: superseded (historical).** Part of the pre-2026-09-23 docs; kept +> as a record. Current entry point: [docs/README.md](README.md). +> Version-upgrade replacements: +> [06-portability/hardware-software-matrix.md](06-portability/hardware-software-matrix.md) +> and +> [05-operations/02-debugging-and-maintenance-workflows.md](05-operations/02-debugging-and-maintenance-workflows.md). + > How to support a new NVIDIA driver version (e.g. moving from 570.144 to a > 610-series release) end-to-end: firmware acquisition → reverse engineering → > config → host driver → guest validation. The goal is that a version bump is diff --git a/docs/refactored-architecture.md b/docs/refactored-architecture.md index 9a2f401..f45df2f 100644 --- a/docs/refactored-architecture.md +++ b/docs/refactored-architecture.md @@ -1,5 +1,11 @@ # Open-IOV vGPU-IO Architecture +> **Status: superseded (historical).** Part of the pre-2026-09-23 docs; kept +> as a record. Current entry point: [docs/README.md](README.md). Architecture +> replacement: +> [03-architecture/01](03-architecture/01-system-overview.md) and +> [03-architecture/06](03-architecture/06-host-module-and-hybrid-model.md). + > **Status**: Living document. Updated 2026-09-22 post host-stability fix. > **Supersedes**: Previous refactored-architecture.md (v1.0). diff --git a/docs/refactored-implementation-to-do.md b/docs/refactored-implementation-to-do.md index f7e0097..e8ddd3c 100644 --- a/docs/refactored-implementation-to-do.md +++ b/docs/refactored-implementation-to-do.md @@ -1,5 +1,12 @@ # Open-IOV vGPU-IO Refactored Implementation TODO List +> **Status: superseded (historical).** Part of the pre-2026-09-23 docs; kept +> as a record. Current entry point: [docs/README.md](README.md). For the +> current architecture and work list see +> [03-architecture/01](03-architecture/01-system-overview.md), +> [03-architecture/06](03-architecture/06-host-module-and-hybrid-model.md) and +> [04-implementation/01](04-implementation/01-known-knowns-and-unknowns.md). + ## Overview This document tracks the granular implementation steps to refactor Open-IOV from guest-side emulation to **host-side GSP emulation**, enabling unmodified guest NVIDIA drivers to run with full functionality. diff --git a/docs/tools/README.md b/docs/tools/README.md index 232fbe5..c5ebb9f 100644 --- a/docs/tools/README.md +++ b/docs/tools/README.md @@ -17,6 +17,10 @@ from the driver components we studied by reverse engineering and decompilation | `guest_test_agent.py` | `tools/guest_agent/` | Guest-side test agent (vsock/TCP) | | `debug_guest_init.sh` | `scripts/` | One-command debug launch (QEMU + GDB + monitor) | | `windows_guest_automation.sh` | `scripts/` | Windows guest automation via WinRM | +| `rmq` | `tools/rmq/` | Host RM-API oracle: queries the host RM via `/dev/nvidia0` for ground-truth GPU/engine data | +| `cudatool.py`, `callwalk.py`, `ghidra_decompile_at.sh`, `annotate_corpus.py` | `userspace/cuda/tools/` | CUDA userspace analysis: crash decode, callgraph walk, decompile-at, corpus annotation | +| `guest_engines.c`, `ioctl_trace.c` | `tools/guest-diag/` | Guest-side engine-list dump and NVIDIA-ioctl tracer | +| `ab.sh`, `vector_add.cu` | `tools/cuda-ab/` | Host-vs-guest CUDA A/B harness with diffable results | ### RM API Tooling (Planned) | Tool | Documentation | Description | @@ -65,7 +69,11 @@ tools/ ## Integration with Open-IOV Stack Tools will integrate with the host VFIO driver via: - `/dev/nvidia*` character devices (existing nvidia.ko) -- `open-iov-vfio.ko` debugfs interface (`/sys/kernel/debug/open-iov-vfio/`) +- `open-iov-vfio.ko` observability: **there is currently no debugfs interface** — + stats and traces come from `dmesg`, and the module's write-only parameters + (`scan_guest_ram`, `scan_vidmem`, `scan_seq`, `sema_probe`, `scan_pt`, + `scan_masked`, `fill_vidmem`, `poke_guest`, `poke_fill`, `dump_vidmem`, + `dump_guest`) drive guest-memory inspection - VFIO ioctls for guest state introspection - GSP-RM telemetry via firmware analyzer / dynamic dispatcher |
