summaryrefslogtreecommitdiffstats
path: root/wiki/Driver-Lifecycle-Field-Guide.md
diff options
context:
space:
mode:
authorArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-12 00:32:00 -0400
committerArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-12 00:32:00 -0400
commit5df3147130c9254cd5bea34b330a2a42ee51d635 (patch)
tree6a8fdf54bbb291eaccbeef8334e2cdd70af2caa8 /wiki/Driver-Lifecycle-Field-Guide.md
parentf1edf479a167898aaf70dc8f21e9cf18bccbe112 (diff)
downloadopen-iov-5df3147130c9254cd5bea34b330a2a42ee51d635.tar.gz
open-iov-5df3147130c9254cd5bea34b330a2a42ee51d635.zip
site: open-iov.org 2026 documentation update
- tools/site: Markdown-to-MediaWiki converter (tracked-paths-only link mapping, local-marker scan, About-message insert), WikiBot publisher with diff/check modes and publish manifest, live site auditor, image uploader, site-only page sources (landing page, About, index, sidebar, banner templates, About-message template). - wiki: new public deep-dive sources (project status, install, FAQ, system overview, GSP and GSP-RM, execution frontier, host module, guest stack, UVM, physical backend, driver lifecycle, risc-v-gsp-rm-debugger, hardware/software matrix, HAAEEP, glossary); refreshed CUDA userspace status; corrected cross-document links; README rewritten for the new set. - wiki/.publish-manifest.json records each published revision for rollback.
Diffstat (limited to 'wiki/Driver-Lifecycle-Field-Guide.md')
-rw-r--r--wiki/Driver-Lifecycle-Field-Guide.md146
1 files changed, 146 insertions, 0 deletions
diff --git a/wiki/Driver-Lifecycle-Field-Guide.md b/wiki/Driver-Lifecycle-Field-Guide.md
new file mode 100644
index 0000000..1115be9
--- /dev/null
+++ b/wiki/Driver-Lifecycle-Field-Guide.md
@@ -0,0 +1,146 @@
+# Guest Driver Lifecycle: A Field Guide
+
+> **Scope.** A walking tour of what happens between "guest boots" and "CUDA
+> initializes", using message, function and control IDs observed from an
+> unmodified 570.144 guest. Canonical source:
+> [`../docs/03-architecture/07-driver-lifecycle-field-guide.md`](../docs/03-architecture/07-driver-lifecycle-field-guide.md); if this page and the docs tree disagree, the docs tree wins.
+>
+> Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` observed with the stock guest driver; `[design]` stages documented ahead of arrival.
+
+## 1. How to read this page
+
+Each stage lists what the **guest** does, what the **device/firmware** must answer, what host emulation does today, and the log signature to look for (host-side `dmesg` with prefixes `vgsp:` / `vgsp-rm:`; guest-side `dmesg` in the VM). IDs are for 570.144 on the Turing (TU104) profile; other generations add IDs and change versioned values, but the *stages* are stable.
+
+## 2. Stage map
+
+```text
+guest boots
+ │
+ ├─ 1. PCI probe & config space ........ lspci is the reference
+ ├─ 2. GSP boot handshake .............. WPR, FWSEC, booter, RISC-V core
+ ├─ 3. Static configuration ............ GET_GSP_STATIC_INFO (65)
+ ├─ 4. Message queues .................. SET_GUEST_SYSTEM_INFO (1) + init traffic
+ ├─ 5. RM control battery .............. GSP_RM_CONTROL (76) × ~160
+ ├─ 6. Object creation ................. GSP_RM_ALLOC (103) × ~60
+ └─ 7. UVM registration/context ........ cuInit → channel manager → execution frontier
+```
+
+## 3. PCI probe and config space
+
+**Guest:** reads IDs, walks capabilities, sizes BARs, enables Memory Space and Bus Master, programs MSI-X. **Device must:** report a coherent TU104-family GPU with 16 MB BAR0, 256 MB BAR1, an MSI-X capability and a link capability the driver accepts. **Emulation:** config space is synthesized and BAR sizes are enforced. The hypervisor is the usual source of surprises — a device on the root bus can have its PCIe capability rewritten to "root complex integrated endpoint", removing link information the driver wants; attach behind a root port. **Log:** guest-side `lspci -vvv` is the reference.
+
+## 4. GSP boot handshake
+
+**Guest:** writes WPR descriptors, releases FWSEC, then the booter, then the RISC-V core; polls a status mailbox for "GSP init done". **Device must:** complete every stage in the order the driver polls and leave the mailbox in the state the driver expects. **Emulation:** the host module models the stage state machine and auto-clears self-clearing triggers (e.g. MMU invalidate registers) so polls terminate.
+
+```text
+vgsp: GSP-BOOT: ... trigger auto-cleared ...
+```
+
+## 5. Static configuration
+
+**Guest:** `GET_GSP_STATIC_INFO` (65) → a large `GspStaticConfigInfo` structure; reads the VBIOS image from the BAR0 window; queries device/FB/BUS info. **Device must:** provide self-consistent sizes, engine inventories, memory geometry, interrupt topology and a parseable VBIOS blob. **Emulation:** the static structures are synthesized for TU104 (FB size, BAR layout, engine list, vGPU-friendly VF values).
+
+```text
+vgsp-rm: GET_GSP_STATIC_INFO → ...
+```
+
+## 6. Message queues
+
+**Guest:** sends `SET_GUEST_SYSTEM_INFO` (1) with libos regions, a queue buffer description and a page table (the buffer is `NV_MEMORY_NONCONTIGUOUS`); also `GSP_SET_SYSTEM_INFO` (72), `SET_REGISTRY` (73) and `CONTINUATION_RECORD` (71) as init traffic. **Device must:** consume them silently (async) and consume the command queue exactly in the guest's write order. **Emulation:** parses the queue chain, builds the geometry, links the status queue and consumes elements one by one — **element-based, not byte-stream-based**: one message is one element and may span pages.
+
+```text
+vgsp: libos region[i] id8=... pa=... size=...
+vgsp: queue args sharedPA=... ptes=... cmdOff=... statOff=...
+```
+
+## 7. The RM control battery
+
+**Guest:** sends `GSP_RM_CONTROL` (76) for every subsystem it initializes. Of the ~160 observed control IDs, 23 are special-cased with real values; the rest are answered with a generic `SUCCESS` and zeroed params (the fallback).
+
+| Control | Meaning | What matters |
+|---|---|---|
+| `0x20800a40` family | internal device info table | per-index values, entry count |
+| `0x20801112` | FIFO device info table | engine inventory; count must match engine queries |
+| `0x20800a5c` | interrupt kernel table | vector topology the driver maps |
+| `0x20800a59` | GMMU static info | fault buffer sizes; a wrong value breaks UVM early |
+| `0x20800a01` | display static info | display count; headless values are fine |
+| `0x20801823/-a/-b` | BUS/FB info V2 | sysmem connection type and PCIe link caps feed bandwidth math |
+| `0x20802a07` | CE physical caps | per-engine bits; UVM refuses channels without `sysmem` support |
+| `0x20802a0a/0b` | all-CE caps | caps table for all engines |
+| `0x20802a02` | CE PCE mask | one bit per engine |
+| `0x20802a08` | CE fault method buffer size | method buffer sizing |
+| `0x90f10106` | vaspace reserved-PDE contract | carries page-table physical addresses (see [`GSP-RPC-Emulation.md`](GSP-RPC-Emulation.md)) |
+| `0x20800123` / `0x20800170` | `GPU_GET_ENGINES` / `GET_ENGINES_V2` | engine inventory; the guest must see the full list, including async copy (`COPY14`) |
+| `0x800201` | `GPU_GET_CLASSLIST` | class list; must agree with engine list and CE caps |
+| `0x2080012b` | `GPU_PROMOTE_CTX` (560-byte params) | load-bearing GR context-validity operation; a wrong shape is a server wedge handled by the supervised fail-safe |
+| `0x20803039` | NVLink masks/VBIOS info | fallback: generic `SUCCESS` with zeroed params |
+| `0x20800a70` | BUS flush with sysmembar | fallback: generic `SUCCESS` with zeroed params; UVM issues these often |
+
+```text
+vgsp-rm: GSP_RM_CONTROL cmd=0x... hClient=0x... hObject=0x... paramsSize=...
+```
+
+## 8. Object creation
+
+**Guest:** `GSP_RM_ALLOC` (103) creates the object tree. Typical order for a CUDA process:
+
+```text
+0x0 NV01_ROOT (client root, 120-byte params)
+0x80 NV01_DEVICE_0 (56 bytes)
+0x2080 NV20_SUBDEVICE_0 (4 bytes)
+0x90f1 FERMI_VASPACE_A (48 bytes) ← the GPU address space
+0xa06c KEPLER_CHANNEL_GROUP_A (20 bytes) ← CE channel group
+0xc46f TURING_CHANNEL_GPFIFO_A(368 bytes) ← the channel (GPFIFO VA, instance/userd descriptors)
+0xc076 GP100_UVM_SW (0 bytes)
+0xc5b5 TURING_DMA_COPY_A (8 bytes) ← CE object
+0xc5c0 TURING_COMPUTE_A (16 bytes) ← compute object (engine registration path)
+```
+
+A hard-won rule from the `TURING_COMPUTE_A` (0xc5c0) alloc: the caller may issue it with `paramsSize=0` against a small stack local while the kernel side treats the class params as 16 bytes and copies reply params back. The emulation must **echo the request params back in the alloc reply**, or the user-mode client segfaults. Channel parameters decode into `gpFifoOffset` (a GPU VA), `gpFifoEntries`, `engineType` (e.g. `COPY0 = 0x9`), and `NV_MEMORY_DESC_PARAMS` for instance/userd/ramfc/method buffers with **physical** addresses and apertures (`ADDR_SYSMEM = 1`, `ADDR_FBMEM = 2`).
+
+```text
+vgsp-rm: GSP_RM_ALLOC class=0x... paramsSize=... hClient=... hParent=... hObject=...
+```
+
+## 9. UVM registration and the execution frontier
+
+**Guest:** `cuInit` → `UVM_REGISTER_GPU` → channel-manager creation → page-tree initialization; each UVM channel does a first push and waits on a semaphore. **Emulation today:** the host decodes the pushes UVM generates and executes their memsets, copies and semaphore releases in software by translating the semaphore VA to a guest physical address — a bridge, not forwarding: no work reaches the physical GPU through it yet.
+
+**Progress:** context creation now returns (the driver-API smoke completes end to end) and the frontier is execution. The host GR ladder allocates a GR0 channel and a compute-class object; a first semaphore push against an uncommitted GR context faulted `CTXNOTVALID`, and a following `GPU_PROMOTE_CTX` wedged GSP-RM into its designed locked fail-safe. Later, a forwarded guest CUDA launch reached the front-end (ring entry fetched, methods executed) with no observed inline launch-descriptor install and no grid dispatched; context validity at dispatch is ruled out. See [`Execution-Frontier.md`](Execution-Frontier.md). The roadmap from here is:
+
+```text
+UVM registers → cuInit returns → context creation → memory allocation
+→ channel creation for the process → kernel launch → (real execution)
+```
+
+Each remaining step needs the same treatment: satisfy the interface, forward the work, relay the completions.
+
+```text
+nvidia-uvm: ... Tracker entry for value N (sema VA 0x...) channel ID ... CE ...
+nvidia-uvm: ... pending push '' started at ... in channel_init() releasing value N
+```
+
+## 10. Quick reference: function IDs from a stock 570.144 guest
+
+| Function | Name | Frequency per CUDA run (approx.) |
+|---|---|---|
+| 1 | SET_GUEST_SYSTEM_INFO | once |
+| 10 | FREE | ~14 |
+| 47 | UNLOADING_GUEST_DRIVER | on unload |
+| 65 | GET_GSP_STATIC_INFO | once |
+| 70 | UPDATE_BAR_PDE | a few |
+| 71/72/73 | CONTINUATION_RECORD / GSP_SET_SYSTEM_INFO / SET_REGISTRY | once each |
+| 76 | GSP_RM_CONTROL | ~160 |
+| 103 | GSP_RM_ALLOC | ~60 |
+
+If a new driver version produces **new function numbers**, do not guess: add a handler that logs the raw payload, then decode it against that version's observed traffic.
+
+## 11. Related documentation
+
+* [`../docs/03-architecture/07-driver-lifecycle-field-guide.md`](../docs/03-architecture/07-driver-lifecycle-field-guide.md) — canonical field guide.
+* [`GSP-and-GSP-RM.md`](GSP-and-GSP-RM.md) — GSP boot, message queues and RM architecture.
+* [`GSP-RPC-Emulation.md`](GSP-RPC-Emulation.md) — control/alloc emulation on the host side.
+* [`RPC-Message-Format.md`](RPC-Message-Format.md) — wire format of the function 76/103 traffic.
+* [`Address-Reference.md`](Address-Reference.md) and [`GSP-570.144W-Firmware-Analysis.md`](GSP-570.144W-Firmware-Analysis.md) — ID reference and the firmware they belong to.
+* [`UVM-in-the-Guest.md`](UVM-in-the-Guest.md), [`Execution-Frontier.md`](Execution-Frontier.md), [`Host-Module.md`](Host-Module.md), [`Physical-GPU-Backend.md`](Physical-GPU-Backend.md) — registration, the frontier, the host stages and the forwarding backend.