summaryrefslogtreecommitdiffstats
path: root/docs/03-architecture
diff options
context:
space:
mode:
Diffstat (limited to 'docs/03-architecture')
-rw-r--r--docs/03-architecture/01-system-overview.md25
-rw-r--r--docs/03-architecture/02-guest-stack-and-cudasw.md17
-rw-r--r--docs/03-architecture/03-gsp-and-gsp-rm.md17
-rw-r--r--docs/03-architecture/04-uvm.md15
-rw-r--r--docs/03-architecture/05-gmmu-amap-hal-gr.md11
-rw-r--r--docs/03-architecture/06-host-module-and-hybrid-model.md35
-rw-r--r--docs/03-architecture/07-driver-lifecycle-field-guide.md33
7 files changed, 130 insertions, 23 deletions
diff --git a/docs/03-architecture/01-system-overview.md b/docs/03-architecture/01-system-overview.md
index ed9b24b..c7a847e 100644
--- a/docs/03-architecture/01-system-overview.md
+++ b/docs/03-architecture/01-system-overview.md
@@ -32,7 +32,8 @@ Everything therefore falls into two buckets:
│ PASS THROUGH (real hardware; for correctness and performance) │
│ • GPU memory: guest VRAM objects backed by real VRAM (goal) │
│ • Page tables / GMMU state used by the GPU (goal) │
-│ • Channel execution: the guest's push buffers run on real channels │
+│ • Channel execution: guest push buffers on real GPU channels (goal; │
+│ today the host software-executes decoded work — a bridge) │
│ • Compute (CUDA) and, later, 3D/video work │
│ • DMA, completion and fault interrupts relayed back to the guest │
└─────────────────────────────────────────────────────────────────────────┘
@@ -59,7 +60,7 @@ HOST MODULE open-iov-vfio.ko
├─ virtual device: config space, BAR0/BAR1/BAR3 windows, MSI-X
├─ emulation core: GSP boot + GSP-RM (RPC msgq, controls, objects, events)
├─ memory model: guest RAM access, VA↔PA maps, VRAM backing
- └─ work path: channel/push forwarding to the physical GPU [in progress]
+ └─ work path: channel/push forwarding to the physical GPU [goal; software bridge today]
│
HOST DRIVER nvidia.ko (unmodified) and/or direct GPU interfaces
│
@@ -161,9 +162,23 @@ decompiled and mapped (see `wiki/GSP-570.144W-Firmware-Analysis.md` and
versions, and performs the resource-manager initialization exchanges.
4. Object creation (clients, devices, subdevices), capability queries,
vaspaces, channels, CE caps, GMMU static info — every control the driver
- probes — is answered with the correct shape and values.
-5. `nvidia-smi` works; CUDA user-space starts; UVM begins channel-manager and
- page-tree initialization.
+ probes — is answered with a well-formed reply. Only a small set (23 control
+ IDs) is special-cased with real values; the rest fall back to generic
+ `SUCCESS` with zeroed params, and `GSP_RM_ALLOC` echoes the request params
+ without server-side object management. Where the driver consumes a reply
+ value, that value is still being corrected (the reply-fidelity ladder).
+5. `nvidia-smi` works; CUDA user-space starts (`cuInit` / `cuDeviceGet`
+ succeed). Context creation is currently blocked: `cuCtxCreate` returns
+ `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks the
+ async copy engine (`COPY14`), so UVM's channel-manager and page-tree
+ initialization is as far as the guest gets.
+
+The CUDA-side contract and the reply-fidelity rules are documented in
+[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md` §9](../../userspace/cuda/docs/03-driver-and-gpu-interface.md)
+and
+[`../../userspace/cuda/docs/04-virtualization-and-emulation.md` §6](../../userspace/cuda/docs/04-virtualization-and-emulation.md);
+start at
+[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md).
## 7. What is next (and why)
diff --git a/docs/03-architecture/02-guest-stack-and-cudasw.md b/docs/03-architecture/02-guest-stack-and-cudasw.md
index 4ec614b..115f639 100644
--- a/docs/03-architecture/02-guest-stack-and-cudasw.md
+++ b/docs/03-architecture/02-guest-stack-and-cudasw.md
@@ -88,8 +88,21 @@ open /dev/nvidia* → cuInit → UVM registration (per process)
```
The initialization path is where compatibility is won or lost; the runtime
-path is where *execution* is won or lost. Open-IOV's current focus is the
-runtime path (forwarding), having already reached the end of initialization.
+path is where *execution* is won or lost. Open-IOV's current focus is
+pre-forwarding correctness: `cuInit` / `cuDeviceGet` succeed, but context
+creation returns `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine
+list lacks the async copy engine (`COPY14`); the forwarding decision
+(host-RM proxy vs direct submission) is still ahead (see
+[`../../userspace/cuda/docs/08-virtualization-approaches.md`](../../userspace/cuda/docs/08-virtualization-approaches.md)).
+
+For the CUDA userspace side of this stack — its architecture, how it is built
+and executes, and the exact kernel/GPU interface contract we must satisfy —
+see [`../../userspace/cuda/docs/01-architecture.md`](../../userspace/cuda/docs/01-architecture.md),
+[`../../userspace/cuda/docs/02-build-and-execution.md`](../../userspace/cuda/docs/02-build-and-execution.md)
+and
+[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../../userspace/cuda/docs/03-driver-and-gpu-interface.md),
+starting from
+[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md).
## 5. What the guest expects from the device, per phase
diff --git a/docs/03-architecture/03-gsp-and-gsp-rm.md b/docs/03-architecture/03-gsp-and-gsp-rm.md
index 20d335b..aca908f 100644
--- a/docs/03-architecture/03-gsp-and-gsp-rm.md
+++ b/docs/03-architecture/03-gsp-and-gsp-rm.md
@@ -185,14 +185,23 @@ is not lost.
|---|---|---|
| GSP boot state machine | ✅ | stages complete; the driver's poll loop is satisfied |
| Message queue transport | ✅ | including the non-contiguous page table for the queue buffer |
-| RPC request/reply | ✅ | all functions the driver uses during init are answered |
-| Static info tables | ✅ | device/FB/BUS/INTR/GMMU tables with plausible TU104 values |
-| Controls during init | ✅ | ~160 distinct control IDs observed per CUDA run; the ones needed to proceed carry real values |
-| Object creation | ✅ | correct handle/parameter shapes |
+| RPC request/reply | ✅ | all functions the driver uses during init are answered; unknown functions get a success reply with no payload |
+| Static info tables | 🟡 | device/FB/BUS/INTR/GMMU tables carry plausible TU104 values; the engine inventory is still incomplete (the async copy engine, COPY14, is missing) |
+| Controls during init | 🟡 | ~160 distinct control IDs observed per CUDA run; 23 are special-cased with real values, the rest fall through to a success reply with zeroed parameters - consumed fields of those are still being corrected |
+| Object creation | 🟡 | handles/parameters are echoed consistently (the request parameters must be echoed back - zeroing them corrupts the caller), but there is no server-side object model: allocation is acknowledged, not tracked |
| Page tables for server-owned ranges | 🟡 | contract received and recorded; leaf population is the forwarding work |
| Event/notification delivery | 🟡 | MSI-X works; richer event semantics as forwarding lands |
| Firmware *behaviour* (routing, scheduling, real execution) | 🔴 | the forwarding work; see the implementation docs |
+The current CUDA blocker is the engine-inventory row: `libcuda` filters the
+engine list it receives against the class list and expects the async copy
+engine to be present, because the secondary (async) engine record is otherwise
+zeroed. See
+[`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md)
+section 3.7 for the offsets and the consumed fields, and
+[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md`](../../userspace/cuda/docs/03-driver-and-gpu-interface.md)
+section 9 for the emulation checklist.
+
## 8. Where the firmware analysis lives
* `wiki/GSP-570.144W-Firmware-Analysis.md` — how the 28.5 MB image was
diff --git a/docs/03-architecture/04-uvm.md b/docs/03-architecture/04-uvm.md
index 299f46b..05ec80b 100644
--- a/docs/03-architecture/04-uvm.md
+++ b/docs/03-architecture/04-uvm.md
@@ -50,13 +50,24 @@ Each stage has a distinct failure signature; our debugging docs map them.
| Requirement | Detail | Our status |
|---|---|---|
| GPU identity | must match the uuid from RM | ✅ |
-| CE capabilities | per-engine support bits; UVM *requires* `supported && !grce && sysmem` for CPU_TO_GPU/GPU_TO_CPU channels | ✅ (we answer CE_GET_PHYSICAL_CAPS with the correct bits) |
+| CE capabilities | per-engine support bits; UVM *requires* `supported && !grce && sysmem` for CPU_TO_GPU/GPU_TO_CPU channels | 🟡 the caps we answer are accepted, but the engine list / CE capabilities are missing the async copy engine (`COPY14`) — the current context-creation blocker |
| Channel creation | RM channel objects with GPFIFO/userd/instance parameters | ✅ objects created |
-| Pushes executed | UVM's first action on a new channel is an *init push* followed by a semaphore-tracked wait | 🟡 we can complete them (see below) |
+| Pushes executed | UVM's first action on a new channel is an *init push* followed by a semaphore-tracked wait | 🟡 the host decodes method streams and executes memsets, copies and semaphore releases in software (a bridge); no kernel is forwarded to the physical GPU yet |
| Page trees | UVM initializes page trees with its own pushes; the *server-owned* ranges must be populated by us | 🟡 |
| Fault delivery | fault buffer + notification; replay/cancel semantics | 🔴 |
| Semantics of completions | tracker values must advance exactly | 🟡 |
+**Current blocker (2026-09-23):** `cuCtxCreate` returns `rc=999`
+(`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks the async copy
+engine (`COPY14`); teardown of the failed context still oopses in the guest
+kernel in `uvm_pmm_devmem_page_to_va_space` (twice per boot). The CUDA-side
+contract and reply rules are in
+[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md` §9](../../userspace/cuda/docs/03-driver-and-gpu-interface.md)
+and
+[`../../userspace/cuda/docs/04-virtualization-and-emulation.md` §6](../../userspace/cuda/docs/04-virtualization-and-emulation.md);
+start at
+[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md).
+
## 4. The most important discovery: how UVM waits
UVM's tracker (`uvm_tracker.c`) is the single most visible symptom of an
diff --git a/docs/03-architecture/05-gmmu-amap-hal-gr.md b/docs/03-architecture/05-gmmu-amap-hal-gr.md
index 6ff9257..6111f54 100644
--- a/docs/03-architecture/05-gmmu-amap-hal-gr.md
+++ b/docs/03-architecture/05-gmmu-amap-hal-gr.md
@@ -182,6 +182,17 @@ Engine types are enumerated by the client during init (`FIFO_GET_DEVICE_INFO_TAB
and reported in engine/capability queries. A guest's channel is bound to one
engine type (e.g. `COPY0 = 0x9`, `GR = 0x1`).
+**Engine inventory is a correctness surface, not just data.** The userspace
+driver builds its per-engine records from the device/engine information the
+kernel driver reports, and filters them against the class list; a missing or
+zeroed record is not ignored - it becomes an allocation with class `0` and the
+whole context setup rolls back. The current CUDA bring-up blocker is exactly
+this: the async copy engine (COPY14, engine descriptor `0x793ceb0e`, instance
+14) is absent from the emulated engine list, while a bare-metal oracle reports
+it as the seventh engine alongside GR and CE0..4. See
+[`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md)
+section 3.7 for the record layout the driver's engine walker consumes.
+
## 7. Practical reading list inside this repository
| Topic | Where |
diff --git a/docs/03-architecture/06-host-module-and-hybrid-model.md b/docs/03-architecture/06-host-module-and-hybrid-model.md
index fc4bb23..4f9efdb 100644
--- a/docs/03-architecture/06-host-module-and-hybrid-model.md
+++ b/docs/03-architecture/06-host-module-and-hybrid-model.md
@@ -92,14 +92,39 @@ implementation philosophy:
expects, or the driver walks off into uninitialized memory.
* **Plausible values second.** Values must be self-consistent (e.g. FB size,
BAR sizes, engine counts, page sizes) or later code paths compute nonsense.
-* **Never echo garbage.** Queries that pass uninitialized input must receive
- zeroed/valid output; echoing the request's stack garbage has caused real
- failures.
+* **Never invent input.** Queries that pass uninitialized *input* must not read
+ it; but in/out parameter blocks must be echoed back, because the driver
+ copies the reply's class parameters over its own buffer at the size the
+ kernel driver declares. Zeroing an in/out block is destructive: the
+ `GSP_RM_ALLOC` reply once zeroed a caller's saved frame pointer this way and
+ faulted `libcuda` (fixed by echoing the request parameters).
The handlers currently cover: the boot/static-info exchanges, the device/FB/
BUS/INTR control tables, CE capability and PCE-mask queries, GMMU static info,
vaspace reserved-PDE contracts, channel/CE object allocation, and the general
-"unknown control returns success with zeroed output" fallback.
+"unknown control/function returns success with zeroed output" fallback. The
+same fallback now matters for engine inventory: `libcuda` filters the engine
+list against the class list, so a missing async copy engine (COPY14) leaves a
+record zeroed and blocks context creation (`cuCtxCreate` returns `999`); see
+[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md)
+and
+[`../../userspace/cuda/docs/05-570.144-decompilation.md`](../../userspace/cuda/docs/05-570.144-decompilation.md)
+section 3.7.
+
+**Current work-execution bridge.** The module also decodes push buffers as
+method streams and executes the work it understands in software - memsets,
+copies and semaphore releases - and completes the corresponding semaphores.
+That is a correctness bridge for memory operations, not the data plane: no
+CUDA kernel is forwarded to the physical GPU yet, and the forwarding strategy
+(host-RM proxy first, direct submission if needed) is in
+[`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md)
+section 3.
+
+**Subsystem status note.** Some components are compiled but not currently
+wired into the live path: the physical-GSP load/boot path is skipped in favour
+of emulation, and the runtime firmware analyzer, version tables, hardware
+abstraction tables and dynamic RPC dispatcher are not yet called by the live
+flow. Treat their descriptions here as intended design, not active code.
### 3.3 Event channels and interrupts
@@ -193,6 +218,8 @@ emulated device inspectable without rebuilding:
| `poke_guest=<gpa>,<v>` | write a 32-bit value to guest physical memory |
| `poke_fill=<gpa>,<v>,<n>` | fill a range |
| `fill_vidmem=<v>` | fill the VRAM shadow |
+| `sema_probe=<gpu_va>[,count]` | probe a semaphore VA and nearby state in guest memory |
+| `dump_guest=<gpa>,<len>` | dump guest physical memory (length capped) |
These are development-grade but production-useful: they are how the memory
model was reverse-engineered, and they remain the fastest way to diagnose a
diff --git a/docs/03-architecture/07-driver-lifecycle-field-guide.md b/docs/03-architecture/07-driver-lifecycle-field-guide.md
index 8d80694..9de5edb 100644
--- a/docs/03-architecture/07-driver-lifecycle-field-guide.md
+++ b/docs/03-architecture/07-driver-lifecycle-field-guide.md
@@ -88,7 +88,9 @@ gsp-resman-vfio: queue args sharedPA=... ptes=... cmdOff=... statOff=...
## 5. Resource-manager initialization (the control battery)
**Guest:** sends `GSP_RM_CONTROL` (76) for every subsystem it initializes.
-Representative controls (all observed, all answered with correct shapes):
+Of the ~160 observed control IDs, 23 are special-cased and answered with real
+values; the rest are answered with a generic `SUCCESS` and zeroed params (the
+fallback). Representative controls:
| Control | Meaning | What matters |
|---|---|---|
@@ -103,8 +105,11 @@ Representative controls (all observed, all answered with correct shapes):
| `0x20802a02` | CE PCE mask | one bit per engine |
| `0x20802a08` | CE fault method buffer size | method buffer sizing |
| `0x90f10106` | vaspace reserved-PDE contract | **carries page-table physical addresses** (see GSP-RM doc) |
-| `0x20803039` | NVLink masks/VBIOS info | unpopulated but must be shaped |
-| `0x20800a70` | BUS flush with sysmembar | UVM issues these frequently; success suffices |
+| `0x20800123` / `0x20800170` | `GPU_GET_ENGINES` / `GET_ENGINES_V2` | engine inventory (filtered by the class list / classDB); values are consumed — the guest must see the full engine list, including the async copy engine (`COPY14`), and the current `rc=999` blocker lives here |
+| `0x800201` | `GPU_GET_CLASSLIST` | class list (classDB); must agree with the engine list and the CE caps controls |
+| `0x2080012b` | GPU_PROMOTE_CTX (560-byte params) | issued immediately before context creation; fallback generic `SUCCESS`, consumed fields not yet coherent |
+| `0x20803039` | NVLink masks/VBIOS info | fallback: generic `SUCCESS` with zeroed params |
+| `0x20800a70` | BUS flush with sysmembar | fallback: generic `SUCCESS` with zeroed params; UVM issues these frequently |
**Log signature:**
```
@@ -125,8 +130,15 @@ CUDA process:
0xc46f TURING_CHANNEL_GPFIFO_A(368 bytes) ← the channel (GPFIFO VA, instance/userd descriptors)
0xc076 GP100_UVM_SW (0 bytes)
0xc5b5 TURING_DMA_COPY_A (8 bytes) ← CE object
+0xc5c0 TURING_COMPUTE_A (16 bytes) ← compute object (engine registration path)
```
+The `TURING_COMPUTE_A` (0xc5c0) alloc is where the earlier `libcuda` segfault
+was fixed: the caller issues it with `paramsSize=0` against an 8-byte stack
+local while the kernel side treats the class params as 16 bytes and copies the
+reply params back, so the emulation must echo the request params back in the
+alloc reply instead of writing a reply the caller did not ask for.
+
Channel parameters decode into: `gpFifoOffset` (a GPU VA),
`gpFifoEntries`, `engineType` (e.g. COPY0 = 0x9), and
`NV_MEMORY_DESC_PARAMS` for instance/userd/ramfc/method buffers with
@@ -149,9 +161,18 @@ page-tree init.
**Device must:** let UVM's channels initialize (each channel does a first push
and waits on a semaphore), and let page-tree initialization proceed.
-**Emulation (today):** we can decode the pushes UVM generates and complete
-their semaphore releases by translating the semaphore VA to a guest physical
-address (see the UVM and host-module docs). This is the live development front.
+**Emulation (today):** we can decode the pushes UVM generates and execute
+their memsets, copies and semaphore releases in software by translating the
+semaphore VA to a guest physical address (see the UVM and host-module docs).
+This is a bridge, not forwarding: no work reaches the physical GPU yet.
+
+**Live development front (2026-09-23):** CUDA context creation — `cuCtxCreate`
+returns `rc=999` (`CUDA_ERROR_UNKNOWN`) because the emulated engine list lacks
+the async copy engine (`COPY14`). The engine-record layout and the alloc
+reply-fidelity checks are the first things to inspect
+([`../../userspace/cuda/docs/05-570.144-decompilation.md` §3.3/§3.6](../../userspace/cuda/docs/05-570.144-decompilation.md);
+start at
+[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md)).
**Log signatures (guest side):**
```