diff options
Diffstat (limited to 'docs/03-architecture')
| -rw-r--r-- | docs/03-architecture/01-system-overview.md | 25 | ||||
| -rw-r--r-- | docs/03-architecture/02-guest-stack-and-cudasw.md | 17 | ||||
| -rw-r--r-- | docs/03-architecture/03-gsp-and-gsp-rm.md | 17 | ||||
| -rw-r--r-- | docs/03-architecture/04-uvm.md | 15 | ||||
| -rw-r--r-- | docs/03-architecture/05-gmmu-amap-hal-gr.md | 11 | ||||
| -rw-r--r-- | docs/03-architecture/06-host-module-and-hybrid-model.md | 35 | ||||
| -rw-r--r-- | docs/03-architecture/07-driver-lifecycle-field-guide.md | 33 |
7 files changed, 130 insertions, 23 deletions
diff --git a/docs/03-architecture/01-system-overview.md b/docs/03-architecture/01-system-overview.md index ed9b24b..c7a847e 100644 --- a/docs/03-architecture/01-system-overview.md +++ b/docs/03-architecture/01-system-overview.md @@ -32,7 +32,8 @@ Everything therefore falls into two buckets: │ PASS THROUGH (real hardware; for correctness and performance) │ │ • GPU memory: guest VRAM objects backed by real VRAM (goal) │ │ • Page tables / GMMU state used by the GPU (goal) │ -│ • Channel execution: the guest's push buffers run on real channels │ +│ • Channel execution: guest push buffers on real GPU channels (goal; │ +│ today the host software-executes decoded work — a bridge) │ │ • Compute (CUDA) and, later, 3D/video work │ │ • DMA, completion and fault interrupts relayed back to the guest │ └─────────────────────────────────────────────────────────────────────────┘ @@ -59,7 +60,7 @@ HOST MODULE open-iov-vfio.ko ├─ virtual device: config space, BAR0/BAR1/BAR3 windows, MSI-X ├─ emulation core: GSP boot + GSP-RM (RPC msgq, controls, objects, events) ├─ memory model: guest RAM access, VA↔PA maps, VRAM backing - └─ work path: channel/push forwarding to the physical GPU [in progress] + └─ work path: channel/push forwarding to the physical GPU [goal; software bridge today] │ HOST DRIVER nvidia.ko (unmodified) and/or direct GPU interfaces │ @@ -161,9 +162,23 @@ decompiled and mapped (see `wiki/GSP-570.144W-Firmware-Analysis.md` and versions, and performs the resource-manager initialization exchanges. 4. Object creation (clients, devices, subdevices), capability queries, vaspaces, channels, CE caps, GMMU static info — every control the driver - probes — is answered with the correct shape and values. -5. `nvidia-smi` works; CUDA user-space starts; UVM begins channel-manager and - page-tree initialization. + probes — is answered with a well-formed reply. Only a small set (23 control + IDs) is special-cased with real values; the rest fall back to generic + `SUCCESS` with zeroed params, and `GSP_RM_ALLOC` echoes the request params + without server-side object management. Where the driver consumes a reply + value, that value is still being corrected (the reply-fidelity ladder). +5. `nvidia-smi` works; CUDA user-space starts (`cuInit` / `cuDeviceGet` + succeed). Context creation is currently blocked: `cuCtxCreate` returns + `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks the + async copy engine (`COPY14`), so UVM's channel-manager and page-tree + initialization is as far as the guest gets. + +The CUDA-side contract and the reply-fidelity rules are documented in +[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md` §9](../../userspace/cuda/docs/03-driver-and-gpu-interface.md) +and +[`../../userspace/cuda/docs/04-virtualization-and-emulation.md` §6](../../userspace/cuda/docs/04-virtualization-and-emulation.md); +start at +[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md). ## 7. What is next (and why) diff --git a/docs/03-architecture/02-guest-stack-and-cudasw.md b/docs/03-architecture/02-guest-stack-and-cudasw.md index 4ec614b..115f639 100644 --- a/docs/03-architecture/02-guest-stack-and-cudasw.md +++ b/docs/03-architecture/02-guest-stack-and-cudasw.md @@ -88,8 +88,21 @@ open /dev/nvidia* → cuInit → UVM registration (per process) ``` The initialization path is where compatibility is won or lost; the runtime -path is where *execution* is won or lost. Open-IOV's current focus is the -runtime path (forwarding), having already reached the end of initialization. +path is where *execution* is won or lost. Open-IOV's current focus is +pre-forwarding correctness: `cuInit` / `cuDeviceGet` succeed, but context +creation returns `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine +list lacks the async copy engine (`COPY14`); the forwarding decision +(host-RM proxy vs direct submission) is still ahead (see +[`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md)). + +For the CUDA userspace side of this stack — its architecture, how it is built +and executes, and the exact kernel/GPU interface contract we must satisfy — +see [`../../userspace/cuda/docs/01-architecture.md`](../../userspace/cuda/docs/01-architecture.md), +[`../../userspace/cuda/docs/02-build-and-execution.md`](../../userspace/cuda/docs/02-build-and-execution.md) +and +[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../../userspace/cuda/docs/03-driver-and-gpu-interface.md), +starting from +[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md). ## 5. What the guest expects from the device, per phase diff --git a/docs/03-architecture/03-gsp-and-gsp-rm.md b/docs/03-architecture/03-gsp-and-gsp-rm.md index 20d335b..aca908f 100644 --- a/docs/03-architecture/03-gsp-and-gsp-rm.md +++ b/docs/03-architecture/03-gsp-and-gsp-rm.md @@ -185,14 +185,23 @@ is not lost. |---|---|---| | GSP boot state machine | ✅ | stages complete; the driver's poll loop is satisfied | | Message queue transport | ✅ | including the non-contiguous page table for the queue buffer | -| RPC request/reply | ✅ | all functions the driver uses during init are answered | -| Static info tables | ✅ | device/FB/BUS/INTR/GMMU tables with plausible TU104 values | -| Controls during init | ✅ | ~160 distinct control IDs observed per CUDA run; the ones needed to proceed carry real values | -| Object creation | ✅ | correct handle/parameter shapes | +| RPC request/reply | ✅ | all functions the driver uses during init are answered; unknown functions get a success reply with no payload | +| Static info tables | 🟡 | device/FB/BUS/INTR/GMMU tables carry plausible TU104 values; the engine inventory is still incomplete (the async copy engine, COPY14, is missing) | +| Controls during init | 🟡 | ~160 distinct control IDs observed per CUDA run; 23 are special-cased with real values, the rest fall through to a success reply with zeroed parameters - consumed fields of those are still being corrected | +| Object creation | 🟡 | handles/parameters are echoed consistently (the request parameters must be echoed back - zeroing them corrupts the caller), but there is no server-side object model: allocation is acknowledged, not tracked | | Page tables for server-owned ranges | 🟡 | contract received and recorded; leaf population is the forwarding work | | Event/notification delivery | 🟡 | MSI-X works; richer event semantics as forwarding lands | | Firmware *behaviour* (routing, scheduling, real execution) | 🔴 | the forwarding work; see the implementation docs | +The current CUDA blocker is the engine-inventory row: `libcuda` filters the +engine list it receives against the class list and expects the async copy +engine to be present, because the secondary (async) engine record is otherwise +zeroed. See +[`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md) +section 3.7 for the offsets and the consumed fields, and +[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../../userspace/cuda/docs/03-driver-and-gpu-interface.md) +section 9 for the emulation checklist. + ## 8. Where the firmware analysis lives * `wiki/GSP-570.144W-Firmware-Analysis.md` — how the 28.5 MB image was diff --git a/docs/03-architecture/04-uvm.md b/docs/03-architecture/04-uvm.md index 299f46b..05ec80b 100644 --- a/docs/03-architecture/04-uvm.md +++ b/docs/03-architecture/04-uvm.md @@ -50,13 +50,24 @@ Each stage has a distinct failure signature; our debugging docs map them. | Requirement | Detail | Our status | |---|---|---| | GPU identity | must match the uuid from RM | ✅ | -| CE capabilities | per-engine support bits; UVM *requires* `supported && !grce && sysmem` for CPU_TO_GPU/GPU_TO_CPU channels | ✅ (we answer CE_GET_PHYSICAL_CAPS with the correct bits) | +| CE capabilities | per-engine support bits; UVM *requires* `supported && !grce && sysmem` for CPU_TO_GPU/GPU_TO_CPU channels | 🟡 the caps we answer are accepted, but the engine list / CE capabilities are missing the async copy engine (`COPY14`) — the current context-creation blocker | | Channel creation | RM channel objects with GPFIFO/userd/instance parameters | ✅ objects created | -| Pushes executed | UVM's first action on a new channel is an *init push* followed by a semaphore-tracked wait | 🟡 we can complete them (see below) | +| Pushes executed | UVM's first action on a new channel is an *init push* followed by a semaphore-tracked wait | 🟡 the host decodes method streams and executes memsets, copies and semaphore releases in software (a bridge); no kernel is forwarded to the physical GPU yet | | Page trees | UVM initializes page trees with its own pushes; the *server-owned* ranges must be populated by us | 🟡 | | Fault delivery | fault buffer + notification; replay/cancel semantics | 🔴 | | Semantics of completions | tracker values must advance exactly | 🟡 | +**Current blocker (2026-09-23):** `cuCtxCreate` returns `rc=999` +(`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks the async copy +engine (`COPY14`); teardown of the failed context still oopses in the guest +kernel in `uvm_pmm_devmem_page_to_va_space` (twice per boot). The CUDA-side +contract and reply rules are in +[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md` §9](../../userspace/cuda/docs/03-driver-and-gpu-interface.md) +and +[`../../userspace/cuda/docs/04-virtualization-and-emulation.md` §6](../../userspace/cuda/docs/04-virtualization-and-emulation.md); +start at +[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md). + ## 4. The most important discovery: how UVM waits UVM's tracker (`uvm_tracker.c`) is the single most visible symptom of an diff --git a/docs/03-architecture/05-gmmu-amap-hal-gr.md b/docs/03-architecture/05-gmmu-amap-hal-gr.md index 6ff9257..6111f54 100644 --- a/docs/03-architecture/05-gmmu-amap-hal-gr.md +++ b/docs/03-architecture/05-gmmu-amap-hal-gr.md @@ -182,6 +182,17 @@ Engine types are enumerated by the client during init (`FIFO_GET_DEVICE_INFO_TAB and reported in engine/capability queries. A guest's channel is bound to one engine type (e.g. `COPY0 = 0x9`, `GR = 0x1`). +**Engine inventory is a correctness surface, not just data.** The userspace +driver builds its per-engine records from the device/engine information the +kernel driver reports, and filters them against the class list; a missing or +zeroed record is not ignored - it becomes an allocation with class `0` and the +whole context setup rolls back. The current CUDA bring-up blocker is exactly +this: the async copy engine (COPY14, engine descriptor `0x793ceb0e`, instance +14) is absent from the emulated engine list, while a bare-metal oracle reports +it as the seventh engine alongside GR and CE0..4. See +[`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md) +section 3.7 for the record layout the driver's engine walker consumes. + ## 7. Practical reading list inside this repository | Topic | Where | diff --git a/docs/03-architecture/06-host-module-and-hybrid-model.md b/docs/03-architecture/06-host-module-and-hybrid-model.md index fc4bb23..4f9efdb 100644 --- a/docs/03-architecture/06-host-module-and-hybrid-model.md +++ b/docs/03-architecture/06-host-module-and-hybrid-model.md @@ -92,14 +92,39 @@ implementation philosophy: expects, or the driver walks off into uninitialized memory. * **Plausible values second.** Values must be self-consistent (e.g. FB size, BAR sizes, engine counts, page sizes) or later code paths compute nonsense. -* **Never echo garbage.** Queries that pass uninitialized input must receive - zeroed/valid output; echoing the request's stack garbage has caused real - failures. +* **Never invent input.** Queries that pass uninitialized *input* must not read + it; but in/out parameter blocks must be echoed back, because the driver + copies the reply's class parameters over its own buffer at the size the + kernel driver declares. Zeroing an in/out block is destructive: the + `GSP_RM_ALLOC` reply once zeroed a caller's saved frame pointer this way and + faulted `libcuda` (fixed by echoing the request parameters). The handlers currently cover: the boot/static-info exchanges, the device/FB/ BUS/INTR control tables, CE capability and PCE-mask queries, GMMU static info, vaspace reserved-PDE contracts, channel/CE object allocation, and the general -"unknown control returns success with zeroed output" fallback. +"unknown control/function returns success with zeroed output" fallback. The +same fallback now matters for engine inventory: `libcuda` filters the engine +list against the class list, so a missing async copy engine (COPY14) leaves a +record zeroed and blocks context creation (`cuCtxCreate` returns `999`); see +[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md) +and +[`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md) +section 3.7. + +**Current work-execution bridge.** The module also decodes push buffers as +method streams and executes the work it understands in software - memsets, +copies and semaphore releases - and completes the corresponding semaphores. +That is a correctness bridge for memory operations, not the data plane: no +CUDA kernel is forwarded to the physical GPU yet, and the forwarding strategy +(host-RM proxy first, direct submission if needed) is in +[`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md) +section 3. + +**Subsystem status note.** Some components are compiled but not currently +wired into the live path: the physical-GSP load/boot path is skipped in favour +of emulation, and the runtime firmware analyzer, version tables, hardware +abstraction tables and dynamic RPC dispatcher are not yet called by the live +flow. Treat their descriptions here as intended design, not active code. ### 3.3 Event channels and interrupts @@ -193,6 +218,8 @@ emulated device inspectable without rebuilding: | `poke_guest=<gpa>,<v>` | write a 32-bit value to guest physical memory | | `poke_fill=<gpa>,<v>,<n>` | fill a range | | `fill_vidmem=<v>` | fill the VRAM shadow | +| `sema_probe=<gpu_va>[,count]` | probe a semaphore VA and nearby state in guest memory | +| `dump_guest=<gpa>,<len>` | dump guest physical memory (length capped) | These are development-grade but production-useful: they are how the memory model was reverse-engineered, and they remain the fastest way to diagnose a diff --git a/docs/03-architecture/07-driver-lifecycle-field-guide.md b/docs/03-architecture/07-driver-lifecycle-field-guide.md index 8d80694..9de5edb 100644 --- a/docs/03-architecture/07-driver-lifecycle-field-guide.md +++ b/docs/03-architecture/07-driver-lifecycle-field-guide.md @@ -88,7 +88,9 @@ gsp-resman-vfio: queue args sharedPA=... ptes=... cmdOff=... statOff=... ## 5. Resource-manager initialization (the control battery) **Guest:** sends `GSP_RM_CONTROL` (76) for every subsystem it initializes. -Representative controls (all observed, all answered with correct shapes): +Of the ~160 observed control IDs, 23 are special-cased and answered with real +values; the rest are answered with a generic `SUCCESS` and zeroed params (the +fallback). Representative controls: | Control | Meaning | What matters | |---|---|---| @@ -103,8 +105,11 @@ Representative controls (all observed, all answered with correct shapes): | `0x20802a02` | CE PCE mask | one bit per engine | | `0x20802a08` | CE fault method buffer size | method buffer sizing | | `0x90f10106` | vaspace reserved-PDE contract | **carries page-table physical addresses** (see GSP-RM doc) | -| `0x20803039` | NVLink masks/VBIOS info | unpopulated but must be shaped | -| `0x20800a70` | BUS flush with sysmembar | UVM issues these frequently; success suffices | +| `0x20800123` / `0x20800170` | `GPU_GET_ENGINES` / `GET_ENGINES_V2` | engine inventory (filtered by the class list / classDB); values are consumed — the guest must see the full engine list, including the async copy engine (`COPY14`), and the current `rc=999` blocker lives here | +| `0x800201` | `GPU_GET_CLASSLIST` | class list (classDB); must agree with the engine list and the CE caps controls | +| `0x2080012b` | GPU_PROMOTE_CTX (560-byte params) | issued immediately before context creation; fallback generic `SUCCESS`, consumed fields not yet coherent | +| `0x20803039` | NVLink masks/VBIOS info | fallback: generic `SUCCESS` with zeroed params | +| `0x20800a70` | BUS flush with sysmembar | fallback: generic `SUCCESS` with zeroed params; UVM issues these frequently | **Log signature:** ``` @@ -125,8 +130,15 @@ CUDA process: 0xc46f TURING_CHANNEL_GPFIFO_A(368 bytes) ← the channel (GPFIFO VA, instance/userd descriptors) 0xc076 GP100_UVM_SW (0 bytes) 0xc5b5 TURING_DMA_COPY_A (8 bytes) ← CE object +0xc5c0 TURING_COMPUTE_A (16 bytes) ← compute object (engine registration path) ``` +The `TURING_COMPUTE_A` (0xc5c0) alloc is where the earlier `libcuda` segfault +was fixed: the caller issues it with `paramsSize=0` against an 8-byte stack +local while the kernel side treats the class params as 16 bytes and copies the +reply params back, so the emulation must echo the request params back in the +alloc reply instead of writing a reply the caller did not ask for. + Channel parameters decode into: `gpFifoOffset` (a GPU VA), `gpFifoEntries`, `engineType` (e.g. COPY0 = 0x9), and `NV_MEMORY_DESC_PARAMS` for instance/userd/ramfc/method buffers with @@ -149,9 +161,18 @@ page-tree init. **Device must:** let UVM's channels initialize (each channel does a first push and waits on a semaphore), and let page-tree initialization proceed. -**Emulation (today):** we can decode the pushes UVM generates and complete -their semaphore releases by translating the semaphore VA to a guest physical -address (see the UVM and host-module docs). This is the live development front. +**Emulation (today):** we can decode the pushes UVM generates and execute +their memsets, copies and semaphore releases in software by translating the +semaphore VA to a guest physical address (see the UVM and host-module docs). +This is a bridge, not forwarding: no work reaches the physical GPU yet. + +**Live development front (2026-09-23):** CUDA context creation — `cuCtxCreate` +returns `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks +the async copy engine (`COPY14`). The engine-record layout and the alloc +reply-fidelity checks are the first things to inspect +([`../../userspace/cuda/docs/05-570.144-decompilation.md` §3.3/§3.6](../../userspace/cuda/docs/05-570.144-decompilation.md); +start at +[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md)). **Log signatures (guest side):** ``` |
