diff options
Diffstat (limited to 'docs')
30 files changed, 485 insertions, 99 deletions
diff --git a/docs/01-orientation/01-purpose-and-scope.md b/docs/01-orientation/01-purpose-and-scope.md index eab63bc..c1232f3 100644 --- a/docs/01-orientation/01-purpose-and-scope.md +++ b/docs/01-orientation/01-purpose-and-scope.md @@ -131,13 +131,18 @@ document: * **Working today**: the virtual device, the GSP boot handshake, the RPC transport, the resource-manager emulation surface the guest driver needs to - probe and enumerate, guest `nvidia.ko` and `nvidia-smi`, and the early UVM - initialization path. -* **Needs improvement**: GPU work forwarding (the reason CUDA kernels do not - yet run), the memory/page-table model, Windows guest support, non-KVM - hypervisors, and performance instrumentation. Each of these is tracked with - explicit "known-unknowns" in - [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md). + probe and enumerate, guest `nvidia.ko` and `nvidia-smi`, and CUDA user-space + startup through `cuInit` / `cuDeviceGet`. +* **Needs improvement**: engine topology and reply fidelity first — context + creation returns `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine + list lacks the async copy engine (`COPY14`) and some consumed reply fields + are still generic fallbacks — then GPU work forwarding (the reason CUDA + kernels do not yet run), the memory/page-table model, Windows guest support, + non-KVM hypervisors, and performance instrumentation. Each of these is + tracked with explicit "known-unknowns" in + [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md); + for the CUDA analysis area, start at + [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md). ## 8. Guiding principles diff --git a/docs/02-getting-started/01-install-and-first-run.md b/docs/02-getting-started/01-install-and-first-run.md index e0e71cb..f323dbb 100644 --- a/docs/02-getting-started/01-install-and-first-run.md +++ b/docs/02-getting-started/01-install-and-first-run.md @@ -100,8 +100,12 @@ Inside the guest (the reference guest has SSH and a guest agent configured): ```bash sudo apt update sudo apt install -y linux-headers-generic build-essential -# Install the stock NVIDIA driver matching the firmware/driver profile -sudo apt install -y nvidia-driver-570-open +# Install the stock NVIDIA driver matching the emulation profile. +# The validated combination is the stock 570.144 Linux runfile (same driver +# version as the emulation target). The distro 570.x package can install a +# different 570 build; if you use it, keep the version notes from the +# hardware/software matrix at hand. +sudo sh ./NVIDIA-Linux-x86_64-570.144.run sudo reboot ``` @@ -126,10 +130,12 @@ cd cuda-samples/Samples/1_Utilities/deviceQuery && make && ./deviceQuery ``` **Expectation today:** driver probe, `nvidia-smi`, device enumeration, and the -early CUDA start-up path work; full kernel execution is the active work item. -Read the honest status in +first CUDA calls (`cuInit`, device queries) work. `cuCtxCreate` still fails +with `999` because the emulated engine list is missing the async copy engine; +full kernel execution is not reached yet. Before drawing conclusions from a +CUDA failure, read the honest status in [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) -before drawing conclusions from a CUDA failure. +and the debugger's guide [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md). ## 7. Day-to-day operations @@ -166,7 +172,7 @@ Developer debug knobs (module parameters) are documented in | No mdev type for your GPU | [hardware matrix](../06-portability/hardware-software-matrix.md) | | Guest sees wrong PCIe link speed/width | root-port attachment + config-space mirror (debugging doc) | | `nvidia.ko` probe fails in guest | guest dmesg + `lspci -vvv` + handoff "driver lifecycle" | -| `nvidia-smi` works, CUDA does not | expected today; see the known-knowns doc | +| `nvidia-smi` works, CUDA does not | expected today (`cuCtxCreate` = 999, engine-list gap); see [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) and [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md) | | VM boot fails or the guest hangs | `scripts/stack_manager.sh cleanup-crash`, then reboot | ## 10. Uninstall diff --git a/docs/03-architecture/01-system-overview.md b/docs/03-architecture/01-system-overview.md index ed9b24b..c7a847e 100644 --- a/docs/03-architecture/01-system-overview.md +++ b/docs/03-architecture/01-system-overview.md @@ -32,7 +32,8 @@ Everything therefore falls into two buckets: │ PASS THROUGH (real hardware; for correctness and performance) │ │ • GPU memory: guest VRAM objects backed by real VRAM (goal) │ │ • Page tables / GMMU state used by the GPU (goal) │ -│ • Channel execution: the guest's push buffers run on real channels │ +│ • Channel execution: guest push buffers on real GPU channels (goal; │ +│ today the host software-executes decoded work — a bridge) │ │ • Compute (CUDA) and, later, 3D/video work │ │ • DMA, completion and fault interrupts relayed back to the guest │ └─────────────────────────────────────────────────────────────────────────┘ @@ -59,7 +60,7 @@ HOST MODULE open-iov-vfio.ko ├─ virtual device: config space, BAR0/BAR1/BAR3 windows, MSI-X ├─ emulation core: GSP boot + GSP-RM (RPC msgq, controls, objects, events) ├─ memory model: guest RAM access, VA↔PA maps, VRAM backing - └─ work path: channel/push forwarding to the physical GPU [in progress] + └─ work path: channel/push forwarding to the physical GPU [goal; software bridge today] │ HOST DRIVER nvidia.ko (unmodified) and/or direct GPU interfaces │ @@ -161,9 +162,23 @@ decompiled and mapped (see `wiki/GSP-570.144W-Firmware-Analysis.md` and versions, and performs the resource-manager initialization exchanges. 4. Object creation (clients, devices, subdevices), capability queries, vaspaces, channels, CE caps, GMMU static info — every control the driver - probes — is answered with the correct shape and values. -5. `nvidia-smi` works; CUDA user-space starts; UVM begins channel-manager and - page-tree initialization. + probes — is answered with a well-formed reply. Only a small set (23 control + IDs) is special-cased with real values; the rest fall back to generic + `SUCCESS` with zeroed params, and `GSP_RM_ALLOC` echoes the request params + without server-side object management. Where the driver consumes a reply + value, that value is still being corrected (the reply-fidelity ladder). +5. `nvidia-smi` works; CUDA user-space starts (`cuInit` / `cuDeviceGet` + succeed). Context creation is currently blocked: `cuCtxCreate` returns + `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks the + async copy engine (`COPY14`), so UVM's channel-manager and page-tree + initialization is as far as the guest gets. + +The CUDA-side contract and the reply-fidelity rules are documented in +[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md` §9](../../userspace/cuda/docs/03-driver-and-gpu-interface.md) +and +[`../../userspace/cuda/docs/04-virtualization-and-emulation.md` §6](../../userspace/cuda/docs/04-virtualization-and-emulation.md); +start at +[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md). ## 7. What is next (and why) diff --git a/docs/03-architecture/02-guest-stack-and-cudasw.md b/docs/03-architecture/02-guest-stack-and-cudasw.md index 4ec614b..115f639 100644 --- a/docs/03-architecture/02-guest-stack-and-cudasw.md +++ b/docs/03-architecture/02-guest-stack-and-cudasw.md @@ -88,8 +88,21 @@ open /dev/nvidia* → cuInit → UVM registration (per process) ``` The initialization path is where compatibility is won or lost; the runtime -path is where *execution* is won or lost. Open-IOV's current focus is the -runtime path (forwarding), having already reached the end of initialization. +path is where *execution* is won or lost. Open-IOV's current focus is +pre-forwarding correctness: `cuInit` / `cuDeviceGet` succeed, but context +creation returns `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine +list lacks the async copy engine (`COPY14`); the forwarding decision +(host-RM proxy vs direct submission) is still ahead (see +[`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md)). + +For the CUDA userspace side of this stack — its architecture, how it is built +and executes, and the exact kernel/GPU interface contract we must satisfy — +see [`../../userspace/cuda/docs/01-architecture.md`](../../userspace/cuda/docs/01-architecture.md), +[`../../userspace/cuda/docs/02-build-and-execution.md`](../../userspace/cuda/docs/02-build-and-execution.md) +and +[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../../userspace/cuda/docs/03-driver-and-gpu-interface.md), +starting from +[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md). ## 5. What the guest expects from the device, per phase diff --git a/docs/03-architecture/03-gsp-and-gsp-rm.md b/docs/03-architecture/03-gsp-and-gsp-rm.md index 20d335b..aca908f 100644 --- a/docs/03-architecture/03-gsp-and-gsp-rm.md +++ b/docs/03-architecture/03-gsp-and-gsp-rm.md @@ -185,14 +185,23 @@ is not lost. |---|---|---| | GSP boot state machine | ✅ | stages complete; the driver's poll loop is satisfied | | Message queue transport | ✅ | including the non-contiguous page table for the queue buffer | -| RPC request/reply | ✅ | all functions the driver uses during init are answered | -| Static info tables | ✅ | device/FB/BUS/INTR/GMMU tables with plausible TU104 values | -| Controls during init | ✅ | ~160 distinct control IDs observed per CUDA run; the ones needed to proceed carry real values | -| Object creation | ✅ | correct handle/parameter shapes | +| RPC request/reply | ✅ | all functions the driver uses during init are answered; unknown functions get a success reply with no payload | +| Static info tables | 🟡 | device/FB/BUS/INTR/GMMU tables carry plausible TU104 values; the engine inventory is still incomplete (the async copy engine, COPY14, is missing) | +| Controls during init | 🟡 | ~160 distinct control IDs observed per CUDA run; 23 are special-cased with real values, the rest fall through to a success reply with zeroed parameters - consumed fields of those are still being corrected | +| Object creation | 🟡 | handles/parameters are echoed consistently (the request parameters must be echoed back - zeroing them corrupts the caller), but there is no server-side object model: allocation is acknowledged, not tracked | | Page tables for server-owned ranges | 🟡 | contract received and recorded; leaf population is the forwarding work | | Event/notification delivery | 🟡 | MSI-X works; richer event semantics as forwarding lands | | Firmware *behaviour* (routing, scheduling, real execution) | 🔴 | the forwarding work; see the implementation docs | +The current CUDA blocker is the engine-inventory row: `libcuda` filters the +engine list it receives against the class list and expects the async copy +engine to be present, because the secondary (async) engine record is otherwise +zeroed. See +[`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md) +section 3.7 for the offsets and the consumed fields, and +[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../../userspace/cuda/docs/03-driver-and-gpu-interface.md) +section 9 for the emulation checklist. + ## 8. Where the firmware analysis lives * `wiki/GSP-570.144W-Firmware-Analysis.md` — how the 28.5 MB image was diff --git a/docs/03-architecture/04-uvm.md b/docs/03-architecture/04-uvm.md index 299f46b..05ec80b 100644 --- a/docs/03-architecture/04-uvm.md +++ b/docs/03-architecture/04-uvm.md @@ -50,13 +50,24 @@ Each stage has a distinct failure signature; our debugging docs map them. | Requirement | Detail | Our status | |---|---|---| | GPU identity | must match the uuid from RM | ✅ | -| CE capabilities | per-engine support bits; UVM *requires* `supported && !grce && sysmem` for CPU_TO_GPU/GPU_TO_CPU channels | ✅ (we answer CE_GET_PHYSICAL_CAPS with the correct bits) | +| CE capabilities | per-engine support bits; UVM *requires* `supported && !grce && sysmem` for CPU_TO_GPU/GPU_TO_CPU channels | 🟡 the caps we answer are accepted, but the engine list / CE capabilities are missing the async copy engine (`COPY14`) — the current context-creation blocker | | Channel creation | RM channel objects with GPFIFO/userd/instance parameters | ✅ objects created | -| Pushes executed | UVM's first action on a new channel is an *init push* followed by a semaphore-tracked wait | 🟡 we can complete them (see below) | +| Pushes executed | UVM's first action on a new channel is an *init push* followed by a semaphore-tracked wait | 🟡 the host decodes method streams and executes memsets, copies and semaphore releases in software (a bridge); no kernel is forwarded to the physical GPU yet | | Page trees | UVM initializes page trees with its own pushes; the *server-owned* ranges must be populated by us | 🟡 | | Fault delivery | fault buffer + notification; replay/cancel semantics | 🔴 | | Semantics of completions | tracker values must advance exactly | 🟡 | +**Current blocker (2026-09-23):** `cuCtxCreate` returns `rc=999` +(`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks the async copy +engine (`COPY14`); teardown of the failed context still oopses in the guest +kernel in `uvm_pmm_devmem_page_to_va_space` (twice per boot). The CUDA-side +contract and reply rules are in +[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md` §9](../../userspace/cuda/docs/03-driver-and-gpu-interface.md) +and +[`../../userspace/cuda/docs/04-virtualization-and-emulation.md` §6](../../userspace/cuda/docs/04-virtualization-and-emulation.md); +start at +[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md). + ## 4. The most important discovery: how UVM waits UVM's tracker (`uvm_tracker.c`) is the single most visible symptom of an diff --git a/docs/03-architecture/05-gmmu-amap-hal-gr.md b/docs/03-architecture/05-gmmu-amap-hal-gr.md index 6ff9257..6111f54 100644 --- a/docs/03-architecture/05-gmmu-amap-hal-gr.md +++ b/docs/03-architecture/05-gmmu-amap-hal-gr.md @@ -182,6 +182,17 @@ Engine types are enumerated by the client during init (`FIFO_GET_DEVICE_INFO_TAB and reported in engine/capability queries. A guest's channel is bound to one engine type (e.g. `COPY0 = 0x9`, `GR = 0x1`). +**Engine inventory is a correctness surface, not just data.** The userspace +driver builds its per-engine records from the device/engine information the +kernel driver reports, and filters them against the class list; a missing or +zeroed record is not ignored - it becomes an allocation with class `0` and the +whole context setup rolls back. The current CUDA bring-up blocker is exactly +this: the async copy engine (COPY14, engine descriptor `0x793ceb0e`, instance +14) is absent from the emulated engine list, while a bare-metal oracle reports +it as the seventh engine alongside GR and CE0..4. See +[`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md) +section 3.7 for the record layout the driver's engine walker consumes. + ## 7. Practical reading list inside this repository | Topic | Where | diff --git a/docs/03-architecture/06-host-module-and-hybrid-model.md b/docs/03-architecture/06-host-module-and-hybrid-model.md index fc4bb23..4f9efdb 100644 --- a/docs/03-architecture/06-host-module-and-hybrid-model.md +++ b/docs/03-architecture/06-host-module-and-hybrid-model.md @@ -92,14 +92,39 @@ implementation philosophy: expects, or the driver walks off into uninitialized memory. * **Plausible values second.** Values must be self-consistent (e.g. FB size, BAR sizes, engine counts, page sizes) or later code paths compute nonsense. -* **Never echo garbage.** Queries that pass uninitialized input must receive - zeroed/valid output; echoing the request's stack garbage has caused real - failures. +* **Never invent input.** Queries that pass uninitialized *input* must not read + it; but in/out parameter blocks must be echoed back, because the driver + copies the reply's class parameters over its own buffer at the size the + kernel driver declares. Zeroing an in/out block is destructive: the + `GSP_RM_ALLOC` reply once zeroed a caller's saved frame pointer this way and + faulted `libcuda` (fixed by echoing the request parameters). The handlers currently cover: the boot/static-info exchanges, the device/FB/ BUS/INTR control tables, CE capability and PCE-mask queries, GMMU static info, vaspace reserved-PDE contracts, channel/CE object allocation, and the general -"unknown control returns success with zeroed output" fallback. +"unknown control/function returns success with zeroed output" fallback. The +same fallback now matters for engine inventory: `libcuda` filters the engine +list against the class list, so a missing async copy engine (COPY14) leaves a +record zeroed and blocks context creation (`cuCtxCreate` returns `999`); see +[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md) +and +[`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md) +section 3.7. + +**Current work-execution bridge.** The module also decodes push buffers as +method streams and executes the work it understands in software - memsets, +copies and semaphore releases - and completes the corresponding semaphores. +That is a correctness bridge for memory operations, not the data plane: no +CUDA kernel is forwarded to the physical GPU yet, and the forwarding strategy +(host-RM proxy first, direct submission if needed) is in +[`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) +section 3. + +**Subsystem status note.** Some components are compiled but not currently +wired into the live path: the physical-GSP load/boot path is skipped in favour +of emulation, and the runtime firmware analyzer, version tables, hardware +abstraction tables and dynamic RPC dispatcher are not yet called by the live +flow. Treat their descriptions here as intended design, not active code. ### 3.3 Event channels and interrupts @@ -193,6 +218,8 @@ emulated device inspectable without rebuilding: | `poke_guest=<gpa>,<v>` | write a 32-bit value to guest physical memory | | `poke_fill=<gpa>,<v>,<n>` | fill a range | | `fill_vidmem=<v>` | fill the VRAM shadow | +| `sema_probe=<gpu_va>[,count]` | probe a semaphore VA and nearby state in guest memory | +| `dump_guest=<gpa>,<len>` | dump guest physical memory (length capped) | These are development-grade but production-useful: they are how the memory model was reverse-engineered, and they remain the fastest way to diagnose a diff --git a/docs/03-architecture/07-driver-lifecycle-field-guide.md b/docs/03-architecture/07-driver-lifecycle-field-guide.md index 8d80694..9de5edb 100644 --- a/docs/03-architecture/07-driver-lifecycle-field-guide.md +++ b/docs/03-architecture/07-driver-lifecycle-field-guide.md @@ -88,7 +88,9 @@ gsp-resman-vfio: queue args sharedPA=... ptes=... cmdOff=... statOff=... ## 5. Resource-manager initialization (the control battery) **Guest:** sends `GSP_RM_CONTROL` (76) for every subsystem it initializes. -Representative controls (all observed, all answered with correct shapes): +Of the ~160 observed control IDs, 23 are special-cased and answered with real +values; the rest are answered with a generic `SUCCESS` and zeroed params (the +fallback). Representative controls: | Control | Meaning | What matters | |---|---|---| @@ -103,8 +105,11 @@ Representative controls (all observed, all answered with correct shapes): | `0x20802a02` | CE PCE mask | one bit per engine | | `0x20802a08` | CE fault method buffer size | method buffer sizing | | `0x90f10106` | vaspace reserved-PDE contract | **carries page-table physical addresses** (see GSP-RM doc) | -| `0x20803039` | NVLink masks/VBIOS info | unpopulated but must be shaped | -| `0x20800a70` | BUS flush with sysmembar | UVM issues these frequently; success suffices | +| `0x20800123` / `0x20800170` | `GPU_GET_ENGINES` / `GET_ENGINES_V2` | engine inventory (filtered by the class list / classDB); values are consumed — the guest must see the full engine list, including the async copy engine (`COPY14`), and the current `rc=999` blocker lives here | +| `0x800201` | `GPU_GET_CLASSLIST` | class list (classDB); must agree with the engine list and the CE caps controls | +| `0x2080012b` | GPU_PROMOTE_CTX (560-byte params) | issued immediately before context creation; fallback generic `SUCCESS`, consumed fields not yet coherent | +| `0x20803039` | NVLink masks/VBIOS info | fallback: generic `SUCCESS` with zeroed params | +| `0x20800a70` | BUS flush with sysmembar | fallback: generic `SUCCESS` with zeroed params; UVM issues these frequently | **Log signature:** ``` @@ -125,8 +130,15 @@ CUDA process: 0xc46f TURING_CHANNEL_GPFIFO_A(368 bytes) ← the channel (GPFIFO VA, instance/userd descriptors) 0xc076 GP100_UVM_SW (0 bytes) 0xc5b5 TURING_DMA_COPY_A (8 bytes) ← CE object +0xc5c0 TURING_COMPUTE_A (16 bytes) ← compute object (engine registration path) ``` +The `TURING_COMPUTE_A` (0xc5c0) alloc is where the earlier `libcuda` segfault +was fixed: the caller issues it with `paramsSize=0` against an 8-byte stack +local while the kernel side treats the class params as 16 bytes and copies the +reply params back, so the emulation must echo the request params back in the +alloc reply instead of writing a reply the caller did not ask for. + Channel parameters decode into: `gpFifoOffset` (a GPU VA), `gpFifoEntries`, `engineType` (e.g. COPY0 = 0x9), and `NV_MEMORY_DESC_PARAMS` for instance/userd/ramfc/method buffers with @@ -149,9 +161,18 @@ page-tree init. **Device must:** let UVM's channels initialize (each channel does a first push and waits on a semaphore), and let page-tree initialization proceed. -**Emulation (today):** we can decode the pushes UVM generates and complete -their semaphore releases by translating the semaphore VA to a guest physical -address (see the UVM and host-module docs). This is the live development front. +**Emulation (today):** we can decode the pushes UVM generates and execute +their memsets, copies and semaphore releases in software by translating the +semaphore VA to a guest physical address (see the UVM and host-module docs). +This is a bridge, not forwarding: no work reaches the physical GPU yet. + +**Live development front (2026-09-23):** CUDA context creation — `cuCtxCreate` +returns `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks +the async copy engine (`COPY14`). The engine-record layout and the alloc +reply-fidelity checks are the first things to inspect +([`../../userspace/cuda/docs/05-570.144-decompilation.md` §3.3/§3.6](../../userspace/cuda/docs/05-570.144-decompilation.md); +start at +[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md)). **Log signatures (guest side):** ``` diff --git a/docs/04-implementation/01-known-knowns-and-unknowns.md b/docs/04-implementation/01-known-knowns-and-unknowns.md index 0564ffc..a085211 100644 --- a/docs/04-implementation/01-known-knowns-and-unknowns.md +++ b/docs/04-implementation/01-known-knowns-and-unknowns.md @@ -22,7 +22,10 @@ listed as an unknown-unknown has a *method* by which we will discover it. ### 1.1 Device and platform * A stock 570.144 Linux guest driver probes and initializes against our - emulated TU104 device; `nvidia-smi` works. + emulated TU104 device; `nvidia-smi` works and CUDA user-space starts + (`cuInit`, `cuDeviceGetCount`, `cuDeviceGet`). Context creation does not yet + complete: `cuCtxCreate` returns `rc=999` (`CUDA_ERROR_UNKNOWN`) because the + emulated engine list lacks the async copy engine (`COPY14`). * The same guest **boots unmodified**: no guest code, no guest module parameters are required for the initialization path. * The host is unmodified in the standalone configuration (distribution NVIDIA @@ -77,6 +80,15 @@ it**, **rough cost**. ### 2.1 Execution forwarding +**Immediate blocker (2026-09-23):** the engine topology is not yet right. +`GPU_GET_ENGINES_V2` returns 6 engines (GR + CE0..4) where a bare-metal oracle +returns 7: the async copy engine (`COPY14`; engine descriptor `0x793ceb0e`, +instance 14, client type `0x38`, class pair `0xc9b5`) is missing, so +`cuCtxCreate` returns `rc=999` (`CUDA_ERROR_UNKNOWN`). The zeroed secondary +engine record makes `libcuda` rewrite the channel alloc class to `0x41` +(`NV01_ROOT_CLIENT`) with status `0x19` and roll back. Teardown of the failed +context still oopses in the guest kernel in `uvm_pmm_devmem_page_to_va_space`. + **Unknown:** the correct, isolated way to run a guest's engine work on the physical GPU: which real scheduling primitive to use per engine, how to translate guest pushes to real submissions, and how to relay completion and @@ -129,7 +141,8 @@ forwarded kernel), capture both sides, implement replay; validate with ### 2.4 Version matrix beyond 570.144 **Unknown:** which parts of the observed protocol drift between driver -families (550/560/580/610+), how much is negotiated, and how much must be +families (550/560/570/580 — the families with entries under +`config/versions/`), how much is negotiated, and how much must be configuration. **Why:** durability is a core promise; users will run other versions @@ -269,7 +282,11 @@ tree is the artifact that survives personnel changes. To make a defensible statement, we need all of: 1. **Interfaces:** every guest→device message for the CUDA path is either - answered or forwarded, for the supported driver family. + answered or forwarded, for the supported driver family — and every reply + whose fields the driver consumes is coherent, not just `SUCCESS`-shaped + (the reply-fidelity ladder: status-only → coherent zeroed params → coherent + values for consumed fields → exact values; see + [`../../userspace/cuda/docs/04-virtualization-and-emulation.md` §6](../../userspace/cuda/docs/04-virtualization-and-emulation.md)). 2. **Memory:** every GPU VA used by work is translatable, and every server-owned page-table range is populated correctly. 3. **Execution:** grid launches and copies run on real hardware with correct @@ -283,7 +300,11 @@ To make a defensible statement, we need all of: measured with a published methodology. Until all seven hold, we say "initialization works; execution is in progress", -and we link here. +and we link here. The CUDA-side checklist behind these gates and the current +blocker are in +[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md` §9](../../userspace/cuda/docs/03-driver-and-gpu-interface.md); +start at +[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md). --- diff --git a/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md b/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md index 1deff02..e2769f8 100644 --- a/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md +++ b/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md @@ -18,7 +18,11 @@ the project portable, maintainable and legally clean. Correctness is not "no crash". It is: * the guest observes the same **values** (capabilities, handles, table - contents) it would on real hardware, + contents) it would on real hardware — a control whose reply fields the + driver consumes is only "implemented" once its reply is coherent, not just + `SUCCESS`-shaped (the reply-fidelity ladder: status-only → coherent zeroed + params → coherent values for consumed fields → exact values; see + [`../../userspace/cuda/docs/04-virtualization-and-emulation.md` §6](../../userspace/cuda/docs/04-virtualization-and-emulation.md)), * the guest observes the same **ordering** (completions happen after the work they signal; faults arrive before replays), * the guest observes the same **failure semantics** (errors are surfaced, not @@ -48,8 +52,10 @@ without a methodology is not accepted in this project. * **Management operations** (object creation, capability queries, mappings) are answered from cached/derived state where the hardware has no side effects the guest can observe. -* **Real work** (grid launches, copies) is forwarded. Nothing else is allowed - on the hot path. +* **Real work** (grid launches, copies) is meant to be forwarded to the + physical GPU (see §3). Today only the work the host decoder understands — + memsets, copies and semaphore releases — is executed in software as a + correctness bridge; no CUDA kernel is forwarded yet. ## 3. Forwarding: the three options and how we choose @@ -99,15 +105,25 @@ means maintaining forks for every guest OS and driver version. A patch to the hypervisor means every hypervisor project must carry it. A firmware change is impossible in practice. +In this repository, patch files exist only as build glue for the *optional* +OpenRM submodule build (`legacy-patches/nvidia/`, applied by `install.sh`). +The runtime and the guest stack are never patched, and the deployment path uses +the distribution driver — so nothing patched is required to run the stack. + ### 5.2 How compliance is verified -* **Contract tests**: CI builds run against *stock* drivers obtained from - distribution repositories (or vendor runfiles), and against *unmodified* - hypervisor packages. +* **Contract tests** (goal): CI builds run against *stock* drivers obtained + from distribution repositories (or vendor runfiles), and against + *unmodified* hypervisor packages. Today CI is push/PR-triggered only — builds, + kernel/unit/KUnit/mock tests, static analysis and docs checks; there is no + scheduled guest suite or stock-driver contract test yet, so the guest ladder + is still run by hand (see the operations docs). * **No-patch checks**: our build must not depend on any patch file for third-party components. Where patches exist in the tree for convenience - (e.g. build glue), the contract tests must pass without them; anything that - fails this is a bug. + (e.g. build glue for the optional OpenRM submodule), they are development + scaffolding only: the contract tests must pass without them, and the + deployment path uses the unpatched distribution driver; anything that fails + this is a bug. * **Behavioral equivalence**: where we answer from emulation, we validate the values against traces from real hardware (or from our decompiled study notes) — not against our own previous answers. diff --git a/docs/05-operations/01-performance-correctness-security.md b/docs/05-operations/01-performance-correctness-security.md index c4d8153..0a27709 100644 --- a/docs/05-operations/01-performance-correctness-security.md +++ b/docs/05-operations/01-performance-correctness-security.md @@ -17,6 +17,19 @@ Rung 4 SUSTAINED workloads run for hours; faults/errors recoverable Rung 5 MULTI-TENANT N VMs, isolation + fairness + no cross-talk ``` +**Current rung (2026-09-23):** Rung 1 is green; **Rung 2 is NOT green.** +`cuInit`/`cuDeviceGet` succeed, but `cuCtxCreate` returns `rc=999` +(`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks the async copy +engine (`COPY14`); `cuMemAlloc` is not reached and teardown of the failed +context still oopses in the guest kernel. Before Rung 3 is claimed, the +**reply-fidelity gate** applies: a control whose reply fields the driver +consumes is not implemented until the reply is coherent (status-only → +coherent zeroed params → coherent values for consumed fields → exact values) — +see +[`../../userspace/cuda/docs/04-virtualization-and-emulation.md` §6](../../userspace/cuda/docs/04-virtualization-and-emulation.md). +New to CUDA analysis here: start at +[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md). + ## 2. Correctness ### 2.1 Test corpus @@ -47,9 +60,11 @@ Rung 5 MULTI-TENANT N VMs, isolation + fairness + no cross-talk ### 2.3 Regression gates -`make test` runs the host suite. The CI ladder runs the guest suite on a -scheduled cadence and on every release. A rung regression is a release blocker; -the handoff documents carry the current rung. +`./scripts/stack_manager.sh test` runs the host suite (`make tests` only builds +the test binaries). CI (`.github/workflows/ci.yml`) runs on push and pull +requests only — there is currently no scheduled guest suite and no +stock-driver contract test. A rung regression is a release blocker; the +handoff documents carry the current rung. ## 3. Performance diff --git a/docs/05-operations/02-debugging-and-maintenance-workflows.md b/docs/05-operations/02-debugging-and-maintenance-workflows.md index bc38a4e..331597a 100644 --- a/docs/05-operations/02-debugging-and-maintenance-workflows.md +++ b/docs/05-operations/02-debugging-and-maintenance-workflows.md @@ -88,7 +88,7 @@ Workflow that made the memory model tractable: ### 3.4 Guest-side tracing -* `ptrace` tracer (`tools/guest-diag/ptrace_ioctl.c`): libcuda issues raw +* `ptrace` tracer (`tools/guest-diag/ioctl_trace.c`): libcuda issues raw syscalls, so `LD_PRELOAD` shims see nothing; PTRACE_SYSCALL with `PTRACE_GET_SYSCALL_INFO` does. * UVM debug: `echo 1 > /sys/module/nvidia_uvm/parameters/uvm_debug_prints` @@ -101,11 +101,60 @@ When a failure is understood, write the decision tree into this document tree — the tree is what makes the next failure cheap. The field guide's tables are the result of that discipline so far. +### 3.6 Userspace driver (`libcuda`) crash workflow + +When the guest segfaults inside the userspace driver, the fault is reported as +`lib[base+size] ip`. Resolve it and find the code before reading any +decompilation: + +```bash +# ip/mapping-base decoder: prints the file vaddr, containing function start, +# and nearest export +userspace/cuda/tools/cudatool.py crash <libcuda.so> <mapping_base> <ip> + +# decompile exactly one file vaddr (translates for the Ghidra image base) +userspace/cuda/tools/ghidra_decompile_at.sh <ghidra-project> <program> <libcuda.so> 0xNNNNNN /tmp/fn.c + +# callers/callees of an address in the annotated corpus +userspace/cuda/tools/callwalk.py <elf-index>/callgraph.tsv callers 0xNNNNNN +``` + +* **Address convention**: public offsets are *file virtual addresses* + (`libcuda+0x...`). Ghidra loads the CUDA userspace library at **image base + `0x100000`**, so addresses in old decompilations produced without that + translation are 0x100000 too low (the wrong functions); runtime + observations from those sessions remain valid, but conclusions drawn from + their decompiled code must be re-derived. The current tooling translates + for you. +* The annotated corpus (layout in + [`../../userspace/cuda/READ-ME-FIRST.md` §4](../../userspace/cuda/READ-ME-FIRST.md)) + is searchable by `libcuda+0x...`; names/indexes are our own work product. +* **Engine topology**: `tools/guest-diag/guest_engines.c` dumps the guest's + engine/class list, virtualization mode and CE caps; compare against the + host oracle (the same query on the host driver) before blaming the + emulation. +* **A/B harness**: `tools/cuda-ab/` runs one deterministic CUDA workload on + bare metal and in the guest and diffs the stable result lines. + +**Known current bugs (2026-09-23)**: `cuCtxCreate` returns `rc=999` +(`CUDA_ERROR_UNKNOWN`) because the emulated engine list is missing the async +copy engine `COPY14`; the guest kernel oopses in +`uvm_pmm_devmem_page_to_va_space` on teardown of the failed context; and +control `0x2080012b` (560-byte params, issued immediately before context +creation) is answered with a generic SUCCESS whose consumed fields are not yet +coherent. Start at +[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md), +[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../../userspace/cuda/docs/03-driver-and-gpu-interface.md) +and +[`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md). + ## 4. Workflows for new inputs ### 4.1 New firmware version -1. Add the image under `firmware/<SERIES>-GSP/<VERSION>/` (tracked by LFS). +1. Add the image under `firmware/<SERIES>-GSP/<VERSION>/` (local study + material; firmware blobs are gitignored and not distributed in the + repository). 2. Compute signatures (CRC32/SHA256) with the firmware analyzer. 3. Add a firmware profile entry (addresses/signatures) so runtime analysis can identify it. @@ -118,8 +167,7 @@ the result of that discipline so far. 1. Install the stock driver in a guest (no patches). 2. Capture the full init trace and diff function/control IDs against the previous version. -3. For each difference: decode against our decompiled study notes for that version - (`g_rpc-structures.h`, `rpc_global_enums.h`, control headers). +3. For each difference: decode against the exact-version reference headers. 4. Add a version-keyed handler or a capability table entry. 5. Re-run rungs 1–2; then the CUDA corpus. diff --git a/docs/06-portability/hardware-software-matrix.md b/docs/06-portability/hardware-software-matrix.md index 442e099..b8393f4 100644 --- a/docs/06-portability/hardware-software-matrix.md +++ b/docs/06-portability/hardware-software-matrix.md @@ -33,7 +33,7 @@ require configuration + verification, not new code.** | Guest | Status | |---|---| -| Linux x86_64 (stock `nvidia.ko`, CUDA) | **Reference** (initialization; execution in progress) | +| Linux x86_64 (stock `nvidia.ko`, CUDA) | **Experimental** (bring-up; probe and `nvidia-smi` work, but there is no continuous guest testing and context creation is currently blocked — see §6.1) | | Linux aarch64 | **Target** | | Windows x86_64 (WDDM driver, CUDA, 3D) | **Target** | | Other Unix (FreeBSD, ...) | Out of scope until demand | @@ -82,18 +82,44 @@ tree, add configuration/handlers, re-run the ladder. ## 6. Version matrix (maintained in the repository) -The authoritative matrix lives in `config/`: +The version/hardware configuration lives in `config/`: ``` config/ hardware/<arch>.yaml # PCI IDs, BARs, memory geometry, engine lists versions/<driver>.yaml # RPC/control inventories, capability tables - profiles/<name>.yaml # (hardware, driver, firmware) combinations we test ``` +There is no `profiles/` directory today. Note also that the code currently +hardcodes the TU104 profile, so the YAML surface is the intended interface +rather than the path the module reads at load time — treating it as +authoritative is aspirational until the profiles are wired in. + A profile is the unit of support: "hardware X + driver family Y + firmware Z = profile P, tested at rung R on date D." Releases publish the profile table. +### 6.1 Userspace driver facts + +* **Guest CUDA driver** (`libcuda`) under test: **570.144** (md5 + `ed5b2fc0044df6052df1d5113754c499`). +* **Host oracle**: `libcuda` **570.211.01** with CUDA toolkit **12.0**, used + for A/B comparisons against the guest. +* **Versioned surface**: the RM ioctl / class / control interface the userspace + driver issues — not a stable public API — see + [`../../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../../userspace/cuda/docs/03-driver-and-gpu-interface.md). + +### 6.2 Engine-list portability + +The engine inventory, and how it is filtered by the class list (classDB), are +per-architecture and per-driver data. Today's gap is concrete: the emulated +list lacks the async copy engine (`COPY14`), which blocks context creation +(`cuCtxCreate` returns `rc=999`). Engine inventory belongs in the hardware +profile, validated against the host oracle per combination. For the +re-analysis workflow see +[`../../userspace/cuda/docs/07-version-upgrade-playbook.md`](../../userspace/cuda/docs/07-version-upgrade-playbook.md); +start at +[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md). + ## 7. Architecture properties that enable portability | Property | Why it matters | diff --git a/docs/07-developers/onboarding-and-standards.md b/docs/07-developers/onboarding-and-standards.md index 78503f4..3d7ac78 100644 --- a/docs/07-developers/onboarding-and-standards.md +++ b/docs/07-developers/onboarding-and-standards.md @@ -17,7 +17,9 @@ — where the code lives (30 min). 5. [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) — what to work on and what is hard (30 min). -6. The subsystem doc for whatever you plan to touch. +6. Working on CUDA: [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md) + — CUDA stack analysis, tooling, and the current blockers. +7. The subsystem doc for whatever you plan to touch. ### 1.2 Environment @@ -53,6 +55,8 @@ the VM holding it. The stack manager enforces the order. open-iov/ ├── kernel/ host module (GSP emulation, device model, memory model) ├── userspace/ management library + examples +├── userspace/cuda/ CUDA stack analysis + tooling (start at +│ userspace/cuda/READ-ME-FIRST.md) ├── tools/ debugging and analysis tools (guest-diag, rmq, │ firmware analysis wrappers) ├── scripts/ build/load/test automation, VM harness @@ -96,8 +100,12 @@ open-iov/ the definition of done. * Prefer a table or a decision tree over prose for anything an operator will follow under pressure. -* Protocol facts (IDs, offsets, structures) must cite their source - (the decompiled study notes, or a live trace). +* Protocol facts (IDs, offsets, structures) must cite their source (a live + trace, or an analysis artifact). Userspace (`libcuda`) addresses are *file + virtual addresses* (`libcuda+0x...`); Ghidra loads that library at **image + base `0x100000`**, so old decompilations produced without translating the + offset are 0x100000 too low (the wrong functions) — re-derive any conclusion + drawn from them. ## 4. Tooling @@ -105,7 +113,10 @@ open-iov/ |---|---| | `scripts/stack_manager.sh` | build/load/status/test/unload with correct ordering | | `scripts/vgpu_vm.py` | VM lifecycle incl. guest-agent exec/shell | -| `tools/guest-diag/` | guest-side diagnostics (ptrace tracer, engines, probes, push helper) | +| `tools/guest-diag/` | guest-side diagnostics (ioctl tracer, engines, probes, push helper) | +| `userspace/cuda/tools/` | CUDA binary analysis (crash decoder, callgraph walk, decompile-at, corpus annotation) | +| `tools/cuda-ab/` | host-vs-guest CUDA A/B harness (one deterministic workload, diffed results) | +| `tools/rmq/` | host RM-API oracle for ground-truth GPU/RM data | | `tools/firmware_analyzer.py` + Ghidra scripts | firmware analysis pipeline | | module debug parameters | guest-memory scanning/poking, VRAM inspection | | QEMU gdbstub workflow | guest virtual-memory inspection/writing | @@ -128,9 +139,11 @@ open-iov/ ```bash make kernel # open-iov-vfio.ko (first!) -make nvidia # NVIDIA OpenRM modules with KBUILD_EXTRA_SYMBOLS +./scripts/build_nvidia_openrm.sh # optional OpenRM modules, KBUILD_EXTRA_SYMBOLS from kernel/Module.symvers make userspace # libiov.so, examples make all # in the correct order +make tests # build the host test binaries only (does not run them) +./scripts/stack_manager.sh test # run the host test suite make clean ``` diff --git a/docs/08-ecosystem/haaeep-and-collaboration.md b/docs/08-ecosystem/haaeep-and-collaboration.md index c4fb352..e559840 100644 --- a/docs/08-ecosystem/haaeep-and-collaboration.md +++ b/docs/08-ecosystem/haaeep-and-collaboration.md @@ -92,7 +92,7 @@ licensed and version-durable — so it is the component we are building. | Milestone | Depends on | |---|---| -| CUDA execution in a guest (rung 3) | Open-IOV forwarding work | +| CUDA execution in a guest (rung 3) | Rung 2 (runtime start) being green first — it is not yet: `cuCtxCreate` returns `rc=999` because the emulated engine list lacks the async copy engine (`COPY14`) — and then Open-IOV forwarding work. The interface contract is documented under [`../../userspace/cuda/README.md`](../../userspace/cuda/README.md) (start at [`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md)) | | Two-VM coexistence (inference + agent) with isolation | Open-IOV multi-tenancy + platform MAC | | Xen/OpenXT or Qubes Air variant | Open-IOV transport backends | | Attested boot-to-inference chain | TrenchBoot integration | @@ -150,5 +150,6 @@ publish findings and fixes, and we will credit you. us (model server, KV cache, agent harness). * **Everyone**: run the ladder on your hardware and send the evidence. -Contact and participation channels are listed in the repository README. We -would rather have a hundred partners than a hundred forks. +Contact and participation channels are listed in +[`../README.md` §6 "How to get involved"](../README.md#6-how-to-get-involved). +We would rather have a hundred partners than a hundred forks. diff --git a/docs/09-reference/glossary-and-faq.md b/docs/09-reference/glossary-and-faq.md index 255d1a6..2ec4faa 100644 --- a/docs/09-reference/glossary-and-faq.md +++ b/docs/09-reference/glossary-and-faq.md @@ -5,11 +5,15 @@ | Term | Meaning | |---|---| | **AMAPlib** | NVIDIA's shared "address mapping" library: swizzling, bank/channel mapping, aperture rules. Studied for page-table and memory-layout semantics. | +| **Async copy engine (COPY14)** | A copy-engine instance found in the host oracle's engine list but missing from the emulated list — engine descriptor `0x793ceb0e`, instance 14, client type `0x38`, class pair `0xc9b5`. Its absence is the current CUDA context-creation blocker. | | **BAR0 / BAR1 / BAR3** | PCI BAR windows of the GPU: register interface (16 MB), VRAM aperture (256 MB), instance/PRAMIN window. | | **CE** | Copy Engine: DMA engine; also used for UVM's internal work. | +| **Class list (classDB)** | The set of RM classes reported as supported (`GPU_GET_CLASSLIST`); engine queries are filtered against it, so the class list, engine list and caps controls must agree. | | **CUDASW** | The family of NVIDIA user-space software that sits between applications and the driver: `libcuda`, `libcudart`, JIT compiler, fatbinary loader, and friends. | | **Doorbell** | A register/memory location a driver writes to signal "new work is available" for a channel or queue. | | **Engine type** | Numeric identifier for an execution engine (`GR = 0x1`, `COPY0 = 0x9`, ...). | +| **Engine record / engine list** | The per-engine descriptors (kind, instance, class, client type) the driver walks after `GPU_GET_ENGINES`/`GET_ENGINES_V2`; each record must be self-consistent or the driver rewrites the following allocation. | +| **File virtual address (`libcuda+0x...`)** | An address relative to the start of a shared object file (e.g. a `libcuda` offset), as distinct from a runtime mapping or a Ghidra program address; public docs cite these. | | **FWSEC** | Firmware security processor stage of the GSP boot sequence. | | **GNND / GB10** | NVIDIA "Grace-Blackwell" systems-on-chip used in DGX Spark and Surface-class laptops; explicit portability targets. | | **GPFIFO** | GPU FIFO: a ring of entries, each pointing to a push buffer for a channel. | @@ -18,12 +22,15 @@ | **GSP-RM** | The Resource Manager running on the GSP ("self-hosted RM"). | | **GMMU** | GPU Memory Management Unit: page tables, TLB, fault handling. | | **HAL** | Hardware Abstraction Layer: per-generation functions that program registers and formats. | +| **Host oracle / A-B harness** | The same query or deterministic workload run against a bare-metal (host) driver and inside the guest, then diffed — our reference for what the guest should have seen. | | **mdev** | Mediated device: the kernel mechanism that turns a physical device into virtual instances (VFIO). | | **MSGQ** | The GSP message queue transport: shared-memory element rings with a page table, plus doorbells. | | **PBDMA** | Push-buffer DMA engine in the host scheduler that fetches and executes push buffers. | | **PDB** | Page Directory Base: root of a GPU page table. | | **PDE / PTE** | Page-directory entry / page-table entry. | | **Push buffer** | A stream of method calls (engine commands) submitted via a GPFIFO. | +| **QMD** | Queue Meta Data: the per-launch descriptor a GR channel reads to execute a kernel grid. | +| **Reply fidelity** | How completely an emulated reply matches what real hardware returns, as a ladder: status-only → coherent zeroed params → coherent values for consumed fields → exact values. | | **RMCore** | The RM's object system: classes, handles, parents, methods. | | **RM / RM API** | NVIDIA's Resource Manager and the API used to allocate objects and issue controls. | | **resserv / resource server** | The RM component that owns allocation authority, privileges and lifetimes. | @@ -57,10 +64,14 @@ GPU and Open-IOV is an independent module. Building NVIDIA's OpenRM modules from source is only needed for the native-vGPU-management workflows. **Why is `nvidia-smi` working but CUDA not?** -Because CUDA requires *execution*: kernels must run on real hardware. The -current state runs the entire initialization path (including UVM's start-up) -but stops short of executing engine work. See the known-knowns document for -the exact status and plan. +CUDA startup reaches `cuInit` / `cuDeviceGet`, but context creation is blocked: +the emulated engine list lacks the async copy engine (`COPY14`), so +`cuCtxCreate` returns `rc=999` (`CUDA_ERROR_UNKNOWN`) and memory allocation +never runs. (The earlier `libcuda` segfault on this path was fixed by echoing +the request params back in the alloc reply.) Once context creation is green, +CUDA still needs *execution*: kernels must run on real hardware. See the +known-knowns document for the exact status and plan; start at +[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md). **How is this legal / license-clean?** Open-IOV is BSD-3-Clause (GPL-2.0 for kernel modules that link GPL-only kernel @@ -85,9 +96,11 @@ MIG is a hardware partitioning mechanism; Open-IOV's scheduling policy can use it where available. It is part of the multi-tenancy work. **How fast is it?** -Initialization currently completes in the same order of magnitude as native; -execution forwarding is not implemented yet. Performance claims follow the -published methodology in the operations docs. +No published numbers yet: execution forwarding is not implemented, and even +the initialization path has not been measured under the project's methodology. +Any performance claim will come with the methodology and the measurements +together (see +[`../05-operations/01-performance-correctness-security.md`](../05-operations/01-performance-correctness-security.md)). **Where do I report bugs or ask for help?** Open an issue with the standard captures listed in the debugging document. A diff --git a/docs/09-reference/legal-and-licensing.md b/docs/09-reference/legal-and-licensing.md index 74f1363..57aad43 100644 --- a/docs/09-reference/legal-and-licensing.md +++ b/docs/09-reference/legal-and-licensing.md @@ -30,9 +30,12 @@ In this repository you will find: * **GSP firmware images** (`firmware/**/*.fwimage.bin`) — fetched from public distribution channels (e.g. `linux-firmware`); used to understand and - emulate the firmware interface. -* **Decompiled firmware artifacts** (`firmware/**/decompiled/`) — our own - analysis output; kept for study and cited in the wiki. + emulate the firmware interface. They are **not distributed in this + repository**: like all firmware blobs they are gitignored and kept only as + local study material. +* **Decompiled/disassembled artifacts** (`firmware/**/decompiled/`) — our own + analysis output, kept locally for study and cited in the wiki. They are + gitignored and never committed. * **Reformatted interface tables** — IDs, structures and message layouts that we derived, published in `wiki/` and in this documentation tree. @@ -42,13 +45,28 @@ do not point readers at material we cannot redistribute, and do not redistribute third-party binaries as project content beyond what their terms allow. +**CUDA-userspace (`libcuda`) analysis rules:** + +* Decompiled or disassembled output is never committed; it is gitignored study + material. +* Cite userspace locations as `libcuda+0x...` *file virtual addresses*. Note + the Ghidra image-base caveat: that library is loaded at **`0x100000`**, so a + decompilation produced without translating the offset refers to the wrong + function. +* Tracked docs state facts and evidence; conclusions cite numeric IDs, code + offsets and recorded observations, never source locations. +* Name maps and indexes (`userspace/cuda/*/annotations/`) are our own work + product and may be tracked. + ## 3. Adversarial interoperability and the law We implement the *interface contract* so that unmodified NVIDIA software can talk to our device. We do not: * copy NVIDIA driver code into our kernel module, -* patch NVIDIA drivers or firmware, +* patch the guest, runtime or host driver at run time. The only patch series + we carry is build glue for the optional OpenRM submodule (`install.sh`); the + deployment path uses the distribution's NVIDIA driver as shipped, * defeat licensing for NVIDIA's own products (our device is our device; no impersonation of a licensed product identity). @@ -68,8 +86,8 @@ open an issue: we would rather over-comply than argue. contribution does not contain third-party material under incompatible terms. * Write original code. Describe behaviour, cite the evidence, and reimplement interfaces from observation rather than copying anyone's source. -* Do not commit firmware or ISO images unless they are explicitly permitted and - listed in `.gitattributes`/LFS policy. +* Do not commit firmware or ISO images: they are local study/deployment + material and are gitignored (see §2). ## 5. Export, safety and platform notes diff --git a/docs/BUILD_NVIDIA_OPENRM.md b/docs/BUILD_NVIDIA_OPENRM.md index 5699036..309a64f 100644 --- a/docs/BUILD_NVIDIA_OPENRM.md +++ b/docs/BUILD_NVIDIA_OPENRM.md @@ -1,5 +1,10 @@ # Building NVIDIA OpenRM with Open-IOV vGPU VFIO Symbols +> **Status: historical build notes.** These notes predate the current workflow. +> For the current build/load model see +> [03-architecture/06-host-module-and-hybrid-model.md](03-architecture/06-host-module-and-hybrid-model.md) +> and [02-getting-started/01-install-and-first-run.md](02-getting-started/01-install-and-first-run.md). + ## Overview The NVIDIA OpenRM `nvidia.ko` kernel module requires the vGPU VFIO interface symbols exported by `open-iov-vfio.ko`: @@ -18,8 +23,8 @@ These symbols must be resolved at build time via `KBUILD_EXTRA_SYMBOLS` pointing ``` 1. Build open-iov-vfio.ko → Generates kernel/Module.symvers 2. Build NVIDIA OpenRM with KBUILD_EXTRA_SYMBOLS=kernel/Module.symvers -3. Load nvidia.ko first -4. Load open-iov-vfio.ko +3. Load open-iov-vfio.ko first (exports the vGPU VFIO symbols) +4. Optionally load nvidia.ko (imports those symbols) ``` ## Automated Build (Recommended) @@ -76,12 +81,14 @@ modinfo nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko | grep -i vgpu ## Module Loading Order ```bash -# 1. Load NVIDIA OpenRM (provides vGPU VFIO symbols) -sudo insmod nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko - -# 2. Load Open-IOV VFIO driver (registers mdev parent with nvidia.ko) +# 1. Load Open-IOV VFIO driver FIRST (exports the vGPU VFIO symbols and +# registers the mdev parent on the physical GPU) sudo insmod kernel/open-iov-vfio.ko +# 2. Load the optional NVIDIA OpenRM (imports the vGPU VFIO symbols; build it +# with KBUILD_EXTRA_SYMBOLS as above) +sudo insmod nvidia-open-gpu-kernel-modules/kernel-open/nvidia.ko + # 3. Copy firmware sudo cp firmware/57X-GSP/570.144/gsp-570.144.fwimage.bin /lib/firmware/ diff --git a/docs/DEBUGGING.md b/docs/DEBUGGING.md index dc95ed7..e2d7fe0 100644 --- a/docs/DEBUGGING.md +++ b/docs/DEBUGGING.md @@ -1,5 +1,11 @@ # Open-IOV Debugging Guide +> **Status: superseded (historical).** Part of the pre-2026-09-23 docs; kept +> as a record. Current entry point: [docs/README.md](README.md). Debugging +> replacement: +> [05-operations/02-debugging-and-maintenance-workflows.md](05-operations/02-debugging-and-maintenance-workflows.md); +> the §3–§4 case studies here remain useful. + > Battle-tested procedures for bringing up the vGPU stack, diagnosing guest > driver failures, and validating a fully working setup. Built from real > debugging sessions (including the 2026-09-21 EPT-livelock hunt). diff --git a/docs/DEVELOPMENT_WORKFLOW.md b/docs/DEVELOPMENT_WORKFLOW.md index 9019ac1..5d43896 100644 --- a/docs/DEVELOPMENT_WORKFLOW.md +++ b/docs/DEVELOPMENT_WORKFLOW.md @@ -1,5 +1,15 @@ # Open-IOV vGPU-IO Development Workflow & Tools Guide +> **Historical document — do not follow as written (2026-09-21).** This guide +> predates the current workflow: it contains the old nvidia-first load order, +> reboot advice, and LFS instructions that no longer apply (LFS was removed; +> firmware and decompiled artifacts are local/gitignored). The current +> workflow is +> [`../scripts/stack_manager.sh`](../scripts/stack_manager.sh), +> [`07-developers/onboarding-and-standards.md`](07-developers/onboarding-and-standards.md) +> and +> [`02-getting-started/01-install-and-first-run.md`](02-getting-started/01-install-and-first-run.md). + ## Overview This document describes the complete development workflow, tooling, and shell integration for working on the Open-IOV vGPU-IO project. It covers everything from firmware reverse engineering to QEMU guest debugging. diff --git a/docs/GSP_EMULATION_ARCHITECTURE.md b/docs/GSP_EMULATION_ARCHITECTURE.md index 8da8f49..70b4b76 100644 --- a/docs/GSP_EMULATION_ARCHITECTURE.md +++ b/docs/GSP_EMULATION_ARCHITECTURE.md @@ -1,5 +1,14 @@ # Open-IOV GSP Emulation Engine — Architecture +> **Status: superseded (historical).** Part of the pre-2026-09-23 docs; kept +> as a record. Current entry point: [docs/README.md](README.md). Architecture +> replacement: +> [03-architecture/01](03-architecture/01-system-overview.md) and +> [03-architecture/06](03-architecture/06-host-module-and-hybrid-model.md); +> the transport section in particular is superseded by the msgq +> documentation in +> [03-architecture/03](03-architecture/03-gsp-and-gsp-rm.md). + > **Audience**: developers working on `kernel/gsp_emulation.c`, `kernel/gsp_rpc_client.c`, > `kernel/bar_emulation.c`, or debugging guest driver bring-up. > diff --git a/docs/ONBOARDING.md b/docs/ONBOARDING.md index bc61a03..615e016 100644 --- a/docs/ONBOARDING.md +++ b/docs/ONBOARDING.md @@ -1,5 +1,10 @@ # Open-IOV Developer Onboarding +> **Status: superseded (historical).** Part of the pre-2026-09-23 docs; kept +> as a record. Current entry point: [docs/README.md](README.md). Onboarding +> replacement: +> [07-developers/onboarding-and-standards.md](07-developers/onboarding-and-standards.md). + > Welcome. This guide gets you from zero to a productive contributor in ~30 > minutes: what the project is, how the repo is laid out, how to build and > test, and the standards we hold the code to. diff --git a/docs/PERFORMANCE.md b/docs/PERFORMANCE.md index 6077920..6ba8d1b 100644 --- a/docs/PERFORMANCE.md +++ b/docs/PERFORMANCE.md @@ -1,5 +1,10 @@ # Open-IOV Performance Testing & A/B Methodology +> **Status: superseded (historical).** Part of the pre-2026-09-23 docs; kept +> as a record. Current entry point: [docs/README.md](README.md). Performance +> replacement: +> [05-operations/01-performance-correctness-security.md](05-operations/01-performance-correctness-security.md). + > How we measure vGPU performance against bare metal, interpret the results, > and keep overhead on the critical path honest. **Run correctness first** > (`docs/DEBUGGING.md` §5) — numbers from a half-working stack are noise. diff --git a/docs/PORTABLE_ARCHITECTURE.md b/docs/PORTABLE_ARCHITECTURE.md index 47d0297..4842407 100644 --- a/docs/PORTABLE_ARCHITECTURE.md +++ b/docs/PORTABLE_ARCHITECTURE.md @@ -1,5 +1,10 @@ # Open-IOV vGPU-IO Portable Architecture Design +> **Status: superseded (historical).** Part of the pre-2026-09-23 docs; kept +> as a record. Current entry point: [docs/README.md](README.md). Portability +> replacement: +> [06-portability/hardware-software-matrix.md](06-portability/hardware-software-matrix.md). + ## Design Goals ### 1. Driver Version Portability diff --git a/docs/README.md b/docs/README.md index fff4f82..cee435e 100644 --- a/docs/README.md +++ b/docs/README.md @@ -97,24 +97,36 @@ and [`01-orientation/02-the-ai-edge-and-ecosystem.md`](01-orientation/02-the-ai- | Emulated vGPU device (PCI, BARs, MSI-X, VFIO mdev) | ✅ Working | | GSP boot emulation (FWSEC/booter/RISCV handshake) | ✅ Working | | GSP-RM message queue + RPC transport | ✅ Working | -| GSP-RM controls: device/fb/bus tables, CE caps, vaspace, GMMU static info | ✅ Working | +| GSP-RM controls: device/fb/bus tables, CE caps, vaspace, GMMU static info | 🟡 Mostly — values are coherent for the special-cased controls; engine inventory/class-list consistency is the current CUDA blocker (COPY14) | | Guest `nvidia.ko` probe, `nvidia-smi` | ✅ Working (TU104) | -| Guest CUDA driver (`libcuda`) startup | 🟡 Partially — UVM advances through channel creation and page-tree init; see [`04-implementation/01-known-knowns-and-unknowns.md`](04-implementation/01-known-knowns-and-unknowns.md) | -| CUDA kernel execution | 🔴 Not yet — requires GPU work forwarding | +| Guest CUDA driver (`libcuda`) startup | 🟡 Partially — `cuInit`/device enumeration work; `cuCtxCreate` returns `999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine list is missing the async copy engine (COPY14). See [`04-implementation/01-known-knowns-and-unknowns.md`](04-implementation/01-known-knowns-and-unknowns.md) and [`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md) | +| CUDA kernel execution | 🔴 Not yet — context creation must succeed first; then GPU work forwarding (today only decoded memsets/copies/semaphore releases execute, in the host) | | OpenGL / Vulkan | 🔴 Not yet | | Windows guest | 🔴 Not yet | | Multi-tenant isolation (VFs, mdev-per-VM) | 🟡 Designed, partially implemented | -We also know exactly *why* CUDA kernels do not run yet, and what the path is: -see the known-knowns/known-unknowns document. The short version: the emulation -is now faithful enough that the guest stack reaches UVM initialization; the -missing piece is forwarding the guest's *work* (push buffers, page trees, -execution) to the physical GPU under our control. +We also know exactly where the CUDA bring-up stands and what the next step is: +the userspace `libcuda` segfault was fixed (an RM allocation reply now echoes +the request parameters instead of returning zeros), and the remaining blocker +is data: the engine list the emulation reports must include the async copy +engine and stay consistent with the class list, because `libcuda` filters +engines against it. The full mechanism, with offsets and the decompiled call +path, is in [`../userspace/cuda/docs/05-570.144-decompilation.md`](../userspace/cuda/docs/05-570.144-decompilation.md). +Beyond that, the missing piece for kernel execution is forwarding the guest's +*work* (push buffers, page trees, execution) to the physical GPU under our +control; see [`04-implementation/02-practical-challenges-and-adversarial-interop.md`](04-implementation/02-practical-challenges-and-adversarial-interop.md). ## 4. How to read these docs (progressive disclosure) Read in order if you are new. Jump directly if you know what you need. +> **Debugging the CUDA bring-up?** Start with +> [`../userspace/cuda/READ-ME-FIRST.md`](../userspace/cuda/READ-ME-FIRST.md) — +> a five-minute guide to the userspace driver analysis, the corrected crash +> path, the current engine-topology blocker, and the lookup recipes for +> `libcuda+0x...` offsets. It links back into this tree for the interface and +> emulation contract. + | # | Document | Audience | Time | |---|----------|----------|------| | 1 | [Purpose and scope](01-orientation/01-purpose-and-scope.md) | everyone | 10 min | @@ -136,11 +148,21 @@ Read in order if you are new. Jump directly if you know what you need. | 17 | [HAAEEP and collaboration](08-ecosystem/haaeep-and-collaboration.md) | stakeholders, partners | 20 min | | 18 | [Glossary and FAQ](09-reference/glossary-and-faq.md) | everyone | reference | -Existing deep-dive material (kept, still valid, being folded into this tree): -the `wiki/` directory (firmware analysis, address reference, wire protocol, -RPC message format) and the `docs/` reference documents -(`GSP_EMULATION_ARCHITECTURE.md`, `DEBUGGING.md`, `PERFORMANCE.md`, -`BUILD_NVIDIA_OPENRM.md`, `VERSION_UPGRADE.md`). +Deep-dive material lives in [`../wiki/`](../wiki/README.md) (firmware +decompilation record, address reference, historical protocol notes). Wiki +pages that predate the current implementation carry a status banner naming +the canonical document in this tree; use the banner, not the old body. + +Historical material is kept but must not be read as current state. The +following carry a **superseded** banner and point at their replacements: +`refactored-architecture.md`, `refactored-implementation-to-do.md`, +`PORTABLE_ARCHITECTURE.md`, `GSP_EMULATION_ARCHITECTURE.md`, `ONBOARDING.md`, +`DEVELOPMENT_WORKFLOW.md`, `PERFORMANCE.md`, `DEBUGGING.md` (sections 3–4 +remain useful case studies), `VERSION_UPGRADE.md`, `BUILD_NVIDIA_OPENRM.md` +(historical build notes; its load order is corrected in place), and +[`../STATE.md`](../STATE.md) (2026-09-21 snapshot). The current versions of +those topics are `03-architecture/01`/`06`, `04-implementation/01`/`02`, +`05-operations/01`/`02`, `06-portability`, and `07-developers`. ## 5. What makes Open-IOV different? diff --git a/docs/VERSION_UPGRADE.md b/docs/VERSION_UPGRADE.md index b9bd387..3fc5de4 100644 --- a/docs/VERSION_UPGRADE.md +++ b/docs/VERSION_UPGRADE.md @@ -1,5 +1,12 @@ # Open-IOV Version Upgrade Guide +> **Status: superseded (historical).** Part of the pre-2026-09-23 docs; kept +> as a record. Current entry point: [docs/README.md](README.md). +> Version-upgrade replacements: +> [06-portability/hardware-software-matrix.md](06-portability/hardware-software-matrix.md) +> and +> [05-operations/02-debugging-and-maintenance-workflows.md](05-operations/02-debugging-and-maintenance-workflows.md). + > How to support a new NVIDIA driver version (e.g. moving from 570.144 to a > 610-series release) end-to-end: firmware acquisition → reverse engineering → > config → host driver → guest validation. The goal is that a version bump is diff --git a/docs/refactored-architecture.md b/docs/refactored-architecture.md index 9a2f401..f45df2f 100644 --- a/docs/refactored-architecture.md +++ b/docs/refactored-architecture.md @@ -1,5 +1,11 @@ # Open-IOV vGPU-IO Architecture +> **Status: superseded (historical).** Part of the pre-2026-09-23 docs; kept +> as a record. Current entry point: [docs/README.md](README.md). Architecture +> replacement: +> [03-architecture/01](03-architecture/01-system-overview.md) and +> [03-architecture/06](03-architecture/06-host-module-and-hybrid-model.md). + > **Status**: Living document. Updated 2026-09-22 post host-stability fix. > **Supersedes**: Previous refactored-architecture.md (v1.0). diff --git a/docs/refactored-implementation-to-do.md b/docs/refactored-implementation-to-do.md index f7e0097..e8ddd3c 100644 --- a/docs/refactored-implementation-to-do.md +++ b/docs/refactored-implementation-to-do.md @@ -1,5 +1,12 @@ # Open-IOV vGPU-IO Refactored Implementation TODO List +> **Status: superseded (historical).** Part of the pre-2026-09-23 docs; kept +> as a record. Current entry point: [docs/README.md](README.md). For the +> current architecture and work list see +> [03-architecture/01](03-architecture/01-system-overview.md), +> [03-architecture/06](03-architecture/06-host-module-and-hybrid-model.md) and +> [04-implementation/01](04-implementation/01-known-knowns-and-unknowns.md). + ## Overview This document tracks the granular implementation steps to refactor Open-IOV from guest-side emulation to **host-side GSP emulation**, enabling unmodified guest NVIDIA drivers to run with full functionality. diff --git a/docs/tools/README.md b/docs/tools/README.md index 232fbe5..c5ebb9f 100644 --- a/docs/tools/README.md +++ b/docs/tools/README.md @@ -17,6 +17,10 @@ from the driver components we studied by reverse engineering and decompilation | `guest_test_agent.py` | `tools/guest_agent/` | Guest-side test agent (vsock/TCP) | | `debug_guest_init.sh` | `scripts/` | One-command debug launch (QEMU + GDB + monitor) | | `windows_guest_automation.sh` | `scripts/` | Windows guest automation via WinRM | +| `rmq` | `tools/rmq/` | Host RM-API oracle: queries the host RM via `/dev/nvidia0` for ground-truth GPU/engine data | +| `cudatool.py`, `callwalk.py`, `ghidra_decompile_at.sh`, `annotate_corpus.py` | `userspace/cuda/tools/` | CUDA userspace analysis: crash decode, callgraph walk, decompile-at, corpus annotation | +| `guest_engines.c`, `ioctl_trace.c` | `tools/guest-diag/` | Guest-side engine-list dump and NVIDIA-ioctl tracer | +| `ab.sh`, `vector_add.cu` | `tools/cuda-ab/` | Host-vs-guest CUDA A/B harness with diffable results | ### RM API Tooling (Planned) | Tool | Documentation | Description | @@ -65,7 +69,11 @@ tools/ ## Integration with Open-IOV Stack Tools will integrate with the host VFIO driver via: - `/dev/nvidia*` character devices (existing nvidia.ko) -- `open-iov-vfio.ko` debugfs interface (`/sys/kernel/debug/open-iov-vfio/`) +- `open-iov-vfio.ko` observability: **there is currently no debugfs interface** — + stats and traces come from `dmesg`, and the module's write-only parameters + (`scan_guest_ram`, `scan_vidmem`, `scan_seq`, `sema_probe`, `scan_pt`, + `scan_masked`, `fill_vidmem`, `poke_guest`, `poke_fill`, `dump_vidmem`, + `dump_guest`) drive guest-memory inspection - VFIO ioctls for guest state introspection - GSP-RM telemetry via firmware analyzer / dynamic dispatcher |