summaryrefslogtreecommitdiffstats
path: root/docs/03-architecture
diff options
context:
space:
mode:
authorArthur Hanson Rasmusson <arthur@vgpu.io>2026-09-23 12:11:10 -0400
committerArthur Hanson Rasmusson <arthur@vgpu.io>2026-09-23 12:11:10 -0400
commit13f46e60760cc0855484cc69de913ea34f55c24f (patch)
treedf67eb2dfc5afd2c56845e514624a4842152a1f1 /docs/03-architecture
parent21b8ebc9db1d0ae649d6406087ea22b54e3b5775 (diff)
downloadopen-iov-13f46e60760cc0855484cc69de913ea34f55c24f.tar.gz
open-iov-13f46e60760cc0855484cc69de913ea34f55c24f.zip
docs: refresh status, supersede stale material, link the CUDA analysis
* Status now matches the live state: guest driver probe and nvidia-smi work; cuInit/device queries work; cuCtxCreate returns 999 because the emulated engine list is missing the async copy engine (COPY14). The userspace libcuda segfault is recorded as solved, with its mechanism (an RM allocation reply must echo request parameters; zeroing them corrupted the caller's saved frame pointer). * Correct the emulation claims: only a subset of RM controls are special-cased; the rest return a success reply with zeroed parameters whose consumed fields are still being corrected; allocation replies are coherent but no server-side object model exists. * Add the eight-point emulation checklist and reply-fidelity ladder as explicit gates, documented in userspace/cuda and linked from the architecture, implementation and operations docs. * Fix operational errors: build/test commands, module load order, mdev type names, LFS references, debugfs claims, in-repo firmware claims, CI claims, portability matrix (guest status, config layout, userspace driver facts). * Mark historical material: one superseded banner per legacy document with a pointer to its replacement; STATE.md marked as a 2026-09-21 snapshot. * Wiki: fix the index, add status banners and in-place factual corrections (msgq transport vs legacy mailbox, BAR definitions, error statuses, migration, glossary), and note the archived 2023 imports.
Diffstat (limited to 'docs/03-architecture')
-rw-r--r--docs/03-architecture/01-system-overview.md25
-rw-r--r--docs/03-architecture/02-guest-stack-and-cudasw.md17
-rw-r--r--docs/03-architecture/03-gsp-and-gsp-rm.md17
-rw-r--r--docs/03-architecture/04-uvm.md15
-rw-r--r--docs/03-architecture/05-gmmu-amap-hal-gr.md11
-rw-r--r--docs/03-architecture/06-host-module-and-hybrid-model.md35
-rw-r--r--docs/03-architecture/07-driver-lifecycle-field-guide.md33
7 files changed, 130 insertions, 23 deletions
diff --git a/docs/03-architecture/01-system-overview.md b/docs/03-architecture/01-system-overview.md
index ed9b24b..c7a847e 100644
--- a/docs/03-architecture/01-system-overview.md
+++ b/docs/03-architecture/01-system-overview.md
@@ -32,7 +32,8 @@ Everything therefore falls into two buckets:
│ PASS THROUGH (real hardware; for correctness and performance) │
│ • GPU memory: guest VRAM objects backed by real VRAM (goal) │
│ • Page tables / GMMU state used by the GPU (goal) │
-│ • Channel execution: the guest's push buffers run on real channels │
+│ • Channel execution: guest push buffers on real GPU channels (goal; │
+│ today the host software-executes decoded work — a bridge) │
│ • Compute (CUDA) and, later, 3D/video work │
│ • DMA, completion and fault interrupts relayed back to the guest │
└─────────────────────────────────────────────────────────────────────────┘
@@ -59,7 +60,7 @@ HOST MODULE open-iov-vfio.ko
├─ virtual device: config space, BAR0/BAR1/BAR3 windows, MSI-X
├─ emulation core: GSP boot + GSP-RM (RPC msgq, controls, objects, events)
├─ memory model: guest RAM access, VA↔PA maps, VRAM backing
- └─ work path: channel/push forwarding to the physical GPU [in progress]
+ └─ work path: channel/push forwarding to the physical GPU [goal; software bridge today]
│
HOST DRIVER nvidia.ko (unmodified) and/or direct GPU interfaces
│
@@ -161,9 +162,23 @@ decompiled and mapped (see `wiki/GSP-570.144W-Firmware-Analysis.md` and
versions, and performs the resource-manager initialization exchanges.
4. Object creation (clients, devices, subdevices), capability queries,
vaspaces, channels, CE caps, GMMU static info — every control the driver
- probes — is answered with the correct shape and values.
-5. `nvidia-smi` works; CUDA user-space starts; UVM begins channel-manager and
- page-tree initialization.
+ probes — is answered with a well-formed reply. Only a small set (23 control
+ IDs) is special-cased with real values; the rest fall back to generic
+ `SUCCESS` with zeroed params, and `GSP_RM_ALLOC` echoes the request params
+ without server-side object management. Where the driver consumes a reply
+ value, that value is still being corrected (the reply-fidelity ladder).
+5. `nvidia-smi` works; CUDA user-space starts (`cuInit` / `cuDeviceGet`
+ succeed). Context creation is currently blocked: `cuCtxCreate` returns
+ `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks the
+ async copy engine (`COPY14`), so UVM's channel-manager and page-tree
+ initialization is as far as the guest gets.
+
+The CUDA-side contract and the reply-fidelity rules are documented in
+[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md` §9](../../userspace/cuda/docs/03-driver-and-gpu-interface.md)
+and
+[`../../userspace/cuda/docs/04-virtualization-and-emulation.md` §6](../../userspace/cuda/docs/04-virtualization-and-emulation.md);
+start at
+[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md).
## 7. What is next (and why)
diff --git a/docs/03-architecture/02-guest-stack-and-cudasw.md b/docs/03-architecture/02-guest-stack-and-cudasw.md
index 4ec614b..115f639 100644
--- a/docs/03-architecture/02-guest-stack-and-cudasw.md
+++ b/docs/03-architecture/02-guest-stack-and-cudasw.md
@@ -88,8 +88,21 @@ open /dev/nvidia* → cuInit → UVM registration (per process)
```
The initialization path is where compatibility is won or lost; the runtime
-path is where *execution* is won or lost. Open-IOV's current focus is the
-runtime path (forwarding), having already reached the end of initialization.
+path is where *execution* is won or lost. Open-IOV's current focus is
+pre-forwarding correctness: `cuInit` / `cuDeviceGet` succeed, but context
+creation returns `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine
+list lacks the async copy engine (`COPY14`); the forwarding decision
+(host-RM proxy vs direct submission) is still ahead (see
+[`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md)).
+
+For the CUDA userspace side of this stack — its architecture, how it is built
+and executes, and the exact kernel/GPU interface contract we must satisfy —
+see [`../../userspace/cuda/docs/01-architecture.md`](../../userspace/cuda/docs/01-architecture.md),
+[`../../userspace/cuda/docs/02-build-and-execution.md`](../../userspace/cuda/docs/02-build-and-execution.md)
+and
+[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../../userspace/cuda/docs/03-driver-and-gpu-interface.md),
+starting from
+[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md).
## 5. What the guest expects from the device, per phase
diff --git a/docs/03-architecture/03-gsp-and-gsp-rm.md b/docs/03-architecture/03-gsp-and-gsp-rm.md
index 20d335b..aca908f 100644
--- a/docs/03-architecture/03-gsp-and-gsp-rm.md
+++ b/docs/03-architecture/03-gsp-and-gsp-rm.md
@@ -185,14 +185,23 @@ is not lost.
|---|---|---|
| GSP boot state machine | ✅ | stages complete; the driver's poll loop is satisfied |
| Message queue transport | ✅ | including the non-contiguous page table for the queue buffer |
-| RPC request/reply | ✅ | all functions the driver uses during init are answered |
-| Static info tables | ✅ | device/FB/BUS/INTR/GMMU tables with plausible TU104 values |
-| Controls during init | ✅ | ~160 distinct control IDs observed per CUDA run; the ones needed to proceed carry real values |
-| Object creation | ✅ | correct handle/parameter shapes |
+| RPC request/reply | ✅ | all functions the driver uses during init are answered; unknown functions get a success reply with no payload |
+| Static info tables | 🟡 | device/FB/BUS/INTR/GMMU tables carry plausible TU104 values; the engine inventory is still incomplete (the async copy engine, COPY14, is missing) |
+| Controls during init | 🟡 | ~160 distinct control IDs observed per CUDA run; 23 are special-cased with real values, the rest fall through to a success reply with zeroed parameters - consumed fields of those are still being corrected |
+| Object creation | 🟡 | handles/parameters are echoed consistently (the request parameters must be echoed back - zeroing them corrupts the caller), but there is no server-side object model: allocation is acknowledged, not tracked |
| Page tables for server-owned ranges | 🟡 | contract received and recorded; leaf population is the forwarding work |
| Event/notification delivery | 🟡 | MSI-X works; richer event semantics as forwarding lands |
| Firmware *behaviour* (routing, scheduling, real execution) | 🔴 | the forwarding work; see the implementation docs |
+The current CUDA blocker is the engine-inventory row: `libcuda` filters the
+engine list it receives against the class list and expects the async copy
+engine to be present, because the secondary (async) engine record is otherwise
+zeroed. See
+[`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md)
+section 3.7 for the offsets and the consumed fields, and
+[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../../userspace/cuda/docs/03-driver-and-gpu-interface.md)
+section 9 for the emulation checklist.
+
## 8. Where the firmware analysis lives
* `wiki/GSP-570.144W-Firmware-Analysis.md` — how the 28.5 MB image was
diff --git a/docs/03-architecture/04-uvm.md b/docs/03-architecture/04-uvm.md
index 299f46b..05ec80b 100644
--- a/docs/03-architecture/04-uvm.md
+++ b/docs/03-architecture/04-uvm.md
@@ -50,13 +50,24 @@ Each stage has a distinct failure signature; our debugging docs map them.
| Requirement | Detail | Our status |
|---|---|---|
| GPU identity | must match the uuid from RM | ✅ |
-| CE capabilities | per-engine support bits; UVM *requires* `supported && !grce && sysmem` for CPU_TO_GPU/GPU_TO_CPU channels | ✅ (we answer CE_GET_PHYSICAL_CAPS with the correct bits) |
+| CE capabilities | per-engine support bits; UVM *requires* `supported && !grce && sysmem` for CPU_TO_GPU/GPU_TO_CPU channels | 🟡 the caps we answer are accepted, but the engine list / CE capabilities are missing the async copy engine (`COPY14`) — the current context-creation blocker |
| Channel creation | RM channel objects with GPFIFO/userd/instance parameters | ✅ objects created |
-| Pushes executed | UVM's first action on a new channel is an *init push* followed by a semaphore-tracked wait | 🟡 we can complete them (see below) |
+| Pushes executed | UVM's first action on a new channel is an *init push* followed by a semaphore-tracked wait | 🟡 the host decodes method streams and executes memsets, copies and semaphore releases in software (a bridge); no kernel is forwarded to the physical GPU yet |
| Page trees | UVM initializes page trees with its own pushes; the *server-owned* ranges must be populated by us | 🟡 |
| Fault delivery | fault buffer + notification; replay/cancel semantics | 🔴 |
| Semantics of completions | tracker values must advance exactly | 🟡 |
+**Current blocker (2026-09-23):** `cuCtxCreate` returns `rc=999`
+(`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks the async copy
+engine (`COPY14`); teardown of the failed context still oopses in the guest
+kernel in `uvm_pmm_devmem_page_to_va_space` (twice per boot). The CUDA-side
+contract and reply rules are in
+[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md` §9](../../userspace/cuda/docs/03-driver-and-gpu-interface.md)
+and
+[`../../userspace/cuda/docs/04-virtualization-and-emulation.md` §6](../../userspace/cuda/docs/04-virtualization-and-emulation.md);
+start at
+[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md).
+
## 4. The most important discovery: how UVM waits
UVM's tracker (`uvm_tracker.c`) is the single most visible symptom of an
diff --git a/docs/03-architecture/05-gmmu-amap-hal-gr.md b/docs/03-architecture/05-gmmu-amap-hal-gr.md
index 6ff9257..6111f54 100644
--- a/docs/03-architecture/05-gmmu-amap-hal-gr.md
+++ b/docs/03-architecture/05-gmmu-amap-hal-gr.md
@@ -182,6 +182,17 @@ Engine types are enumerated by the client during init (`FIFO_GET_DEVICE_INFO_TAB
and reported in engine/capability queries. A guest's channel is bound to one
engine type (e.g. `COPY0 = 0x9`, `GR = 0x1`).
+**Engine inventory is a correctness surface, not just data.** The userspace
+driver builds its per-engine records from the device/engine information the
+kernel driver reports, and filters them against the class list; a missing or
+zeroed record is not ignored - it becomes an allocation with class `0` and the
+whole context setup rolls back. The current CUDA bring-up blocker is exactly
+this: the async copy engine (COPY14, engine descriptor `0x793ceb0e`, instance
+14) is absent from the emulated engine list, while a bare-metal oracle reports
+it as the seventh engine alongside GR and CE0..4. See
+[`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md)
+section 3.7 for the record layout the driver's engine walker consumes.
+
## 7. Practical reading list inside this repository
| Topic | Where |
diff --git a/docs/03-architecture/06-host-module-and-hybrid-model.md b/docs/03-architecture/06-host-module-and-hybrid-model.md
index fc4bb23..4f9efdb 100644
--- a/docs/03-architecture/06-host-module-and-hybrid-model.md
+++ b/docs/03-architecture/06-host-module-and-hybrid-model.md
@@ -92,14 +92,39 @@ implementation philosophy:
expects, or the driver walks off into uninitialized memory.
* **Plausible values second.** Values must be self-consistent (e.g. FB size,
BAR sizes, engine counts, page sizes) or later code paths compute nonsense.
-* **Never echo garbage.** Queries that pass uninitialized input must receive
- zeroed/valid output; echoing the request's stack garbage has caused real
- failures.
+* **Never invent input.** Queries that pass uninitialized *input* must not read
+ it; but in/out parameter blocks must be echoed back, because the driver
+ copies the reply's class parameters over its own buffer at the size the
+ kernel driver declares. Zeroing an in/out block is destructive: the
+ `GSP_RM_ALLOC` reply once zeroed a caller's saved frame pointer this way and
+ faulted `libcuda` (fixed by echoing the request parameters).
The handlers currently cover: the boot/static-info exchanges, the device/FB/
BUS/INTR control tables, CE capability and PCE-mask queries, GMMU static info,
vaspace reserved-PDE contracts, channel/CE object allocation, and the general
-"unknown control returns success with zeroed output" fallback.
+"unknown control/function returns success with zeroed output" fallback. The
+same fallback now matters for engine inventory: `libcuda` filters the engine
+list against the class list, so a missing async copy engine (COPY14) leaves a
+record zeroed and blocks context creation (`cuCtxCreate` returns `999`); see
+[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md)
+and
+[`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md)
+section 3.7.
+
+**Current work-execution bridge.** The module also decodes push buffers as
+method streams and executes the work it understands in software - memsets,
+copies and semaphore releases - and completes the corresponding semaphores.
+That is a correctness bridge for memory operations, not the data plane: no
+CUDA kernel is forwarded to the physical GPU yet, and the forwarding strategy
+(host-RM proxy first, direct submission if needed) is in
+[`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md)
+section 3.
+
+**Subsystem status note.** Some components are compiled but not currently
+wired into the live path: the physical-GSP load/boot path is skipped in favour
+of emulation, and the runtime firmware analyzer, version tables, hardware
+abstraction tables and dynamic RPC dispatcher are not yet called by the live
+flow. Treat their descriptions here as intended design, not active code.
### 3.3 Event channels and interrupts
@@ -193,6 +218,8 @@ emulated device inspectable without rebuilding:
| `poke_guest=<gpa>,<v>` | write a 32-bit value to guest physical memory |
| `poke_fill=<gpa>,<v>,<n>` | fill a range |
| `fill_vidmem=<v>` | fill the VRAM shadow |
+| `sema_probe=<gpu_va>[,count]` | probe a semaphore VA and nearby state in guest memory |
+| `dump_guest=<gpa>,<len>` | dump guest physical memory (length capped) |
These are development-grade but production-useful: they are how the memory
model was reverse-engineered, and they remain the fastest way to diagnose a
diff --git a/docs/03-architecture/07-driver-lifecycle-field-guide.md b/docs/03-architecture/07-driver-lifecycle-field-guide.md
index 8d80694..9de5edb 100644
--- a/docs/03-architecture/07-driver-lifecycle-field-guide.md
+++ b/docs/03-architecture/07-driver-lifecycle-field-guide.md
@@ -88,7 +88,9 @@ gsp-resman-vfio: queue args sharedPA=... ptes=... cmdOff=... statOff=...
## 5. Resource-manager initialization (the control battery)
**Guest:** sends `GSP_RM_CONTROL` (76) for every subsystem it initializes.
-Representative controls (all observed, all answered with correct shapes):
+Of the ~160 observed control IDs, 23 are special-cased and answered with real
+values; the rest are answered with a generic `SUCCESS` and zeroed params (the
+fallback). Representative controls:
| Control | Meaning | What matters |
|---|---|---|
@@ -103,8 +105,11 @@ Representative controls (all observed, all answered with correct shapes):
| `0x20802a02` | CE PCE mask | one bit per engine |
| `0x20802a08` | CE fault method buffer size | method buffer sizing |
| `0x90f10106` | vaspace reserved-PDE contract | **carries page-table physical addresses** (see GSP-RM doc) |
-| `0x20803039` | NVLink masks/VBIOS info | unpopulated but must be shaped |
-| `0x20800a70` | BUS flush with sysmembar | UVM issues these frequently; success suffices |
+| `0x20800123` / `0x20800170` | `GPU_GET_ENGINES` / `GET_ENGINES_V2` | engine inventory (filtered by the class list / classDB); values are consumed — the guest must see the full engine list, including the async copy engine (`COPY14`), and the current `rc=999` blocker lives here |
+| `0x800201` | `GPU_GET_CLASSLIST` | class list (classDB); must agree with the engine list and the CE caps controls |
+| `0x2080012b` | GPU_PROMOTE_CTX (560-byte params) | issued immediately before context creation; fallback generic `SUCCESS`, consumed fields not yet coherent |
+| `0x20803039` | NVLink masks/VBIOS info | fallback: generic `SUCCESS` with zeroed params |
+| `0x20800a70` | BUS flush with sysmembar | fallback: generic `SUCCESS` with zeroed params; UVM issues these frequently |
**Log signature:**
```
@@ -125,8 +130,15 @@ CUDA process:
0xc46f TURING_CHANNEL_GPFIFO_A(368 bytes) ← the channel (GPFIFO VA, instance/userd descriptors)
0xc076 GP100_UVM_SW (0 bytes)
0xc5b5 TURING_DMA_COPY_A (8 bytes) ← CE object
+0xc5c0 TURING_COMPUTE_A (16 bytes) ← compute object (engine registration path)
```
+The `TURING_COMPUTE_A` (0xc5c0) alloc is where the earlier `libcuda` segfault
+was fixed: the caller issues it with `paramsSize=0` against an 8-byte stack
+local while the kernel side treats the class params as 16 bytes and copies the
+reply params back, so the emulation must echo the request params back in the
+alloc reply instead of writing a reply the caller did not ask for.
+
Channel parameters decode into: `gpFifoOffset` (a GPU VA),
`gpFifoEntries`, `engineType` (e.g. COPY0 = 0x9), and
`NV_MEMORY_DESC_PARAMS` for instance/userd/ramfc/method buffers with
@@ -149,9 +161,18 @@ page-tree init.
**Device must:** let UVM's channels initialize (each channel does a first push
and waits on a semaphore), and let page-tree initialization proceed.
-**Emulation (today):** we can decode the pushes UVM generates and complete
-their semaphore releases by translating the semaphore VA to a guest physical
-address (see the UVM and host-module docs). This is the live development front.
+**Emulation (today):** we can decode the pushes UVM generates and execute
+their memsets, copies and semaphore releases in software by translating the
+semaphore VA to a guest physical address (see the UVM and host-module docs).
+This is a bridge, not forwarding: no work reaches the physical GPU yet.
+
+**Live development front (2026-09-23):** CUDA context creation — `cuCtxCreate`
+returns `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks
+the async copy engine (`COPY14`). The engine-record layout and the alloc
+reply-fidelity checks are the first things to inspect
+([`../../userspace/cuda/docs/05-570.144-decompilation.md` §3.3/§3.6](../../userspace/cuda/docs/05-570.144-decompilation.md);
+start at
+[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md)).
**Log signatures (guest side):**
```