diff options
| author | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-10-12 00:32:00 -0400 |
|---|---|---|
| committer | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-10-12 00:32:00 -0400 |
| commit | 5df3147130c9254cd5bea34b330a2a42ee51d635 (patch) | |
| tree | 6a8fdf54bbb291eaccbeef8334e2cdd70af2caa8 /wiki/Driver-Lifecycle-Field-Guide.md | |
| parent | f1edf479a167898aaf70dc8f21e9cf18bccbe112 (diff) | |
| download | open-iov-5df3147130c9254cd5bea34b330a2a42ee51d635.tar.gz open-iov-5df3147130c9254cd5bea34b330a2a42ee51d635.zip | |
site: open-iov.org 2026 documentation update
- tools/site: Markdown-to-MediaWiki converter (tracked-paths-only link
mapping, local-marker scan, About-message insert), WikiBot publisher
with diff/check modes and publish manifest, live site auditor, image
uploader, site-only page sources (landing page, About, index, sidebar,
banner templates, About-message template).
- wiki: new public deep-dive sources (project status, install, FAQ,
system overview, GSP and GSP-RM, execution frontier, host module,
guest stack, UVM, physical backend, driver lifecycle, risc-v-gsp-rm-debugger,
hardware/software matrix, HAAEEP, glossary); refreshed CUDA userspace
status; corrected cross-document links; README rewritten for the new set.
- wiki/.publish-manifest.json records each published revision for rollback.
Diffstat (limited to 'wiki/Driver-Lifecycle-Field-Guide.md')
| -rw-r--r-- | wiki/Driver-Lifecycle-Field-Guide.md | 146 |
1 files changed, 146 insertions, 0 deletions
diff --git a/wiki/Driver-Lifecycle-Field-Guide.md b/wiki/Driver-Lifecycle-Field-Guide.md new file mode 100644 index 0000000..1115be9 --- /dev/null +++ b/wiki/Driver-Lifecycle-Field-Guide.md @@ -0,0 +1,146 @@ +# Guest Driver Lifecycle: A Field Guide + +> **Scope.** A walking tour of what happens between "guest boots" and "CUDA +> initializes", using message, function and control IDs observed from an +> unmodified 570.144 guest. Canonical source: +> [`../docs/03-architecture/07-driver-lifecycle-field-guide.md`](../docs/03-architecture/07-driver-lifecycle-field-guide.md); if this page and the docs tree disagree, the docs tree wins. +> +> Status: 2026-10-12, tree `master @ 2f066727`. Tags: `[live]` observed with the stock guest driver; `[design]` stages documented ahead of arrival. + +## 1. How to read this page + +Each stage lists what the **guest** does, what the **device/firmware** must answer, what host emulation does today, and the log signature to look for (host-side `dmesg` with prefixes `vgsp:` / `vgsp-rm:`; guest-side `dmesg` in the VM). IDs are for 570.144 on the Turing (TU104) profile; other generations add IDs and change versioned values, but the *stages* are stable. + +## 2. Stage map + +```text +guest boots + │ + ├─ 1. PCI probe & config space ........ lspci is the reference + ├─ 2. GSP boot handshake .............. WPR, FWSEC, booter, RISC-V core + ├─ 3. Static configuration ............ GET_GSP_STATIC_INFO (65) + ├─ 4. Message queues .................. SET_GUEST_SYSTEM_INFO (1) + init traffic + ├─ 5. RM control battery .............. GSP_RM_CONTROL (76) × ~160 + ├─ 6. Object creation ................. GSP_RM_ALLOC (103) × ~60 + └─ 7. UVM registration/context ........ cuInit → channel manager → execution frontier +``` + +## 3. PCI probe and config space + +**Guest:** reads IDs, walks capabilities, sizes BARs, enables Memory Space and Bus Master, programs MSI-X. **Device must:** report a coherent TU104-family GPU with 16 MB BAR0, 256 MB BAR1, an MSI-X capability and a link capability the driver accepts. **Emulation:** config space is synthesized and BAR sizes are enforced. The hypervisor is the usual source of surprises — a device on the root bus can have its PCIe capability rewritten to "root complex integrated endpoint", removing link information the driver wants; attach behind a root port. **Log:** guest-side `lspci -vvv` is the reference. + +## 4. GSP boot handshake + +**Guest:** writes WPR descriptors, releases FWSEC, then the booter, then the RISC-V core; polls a status mailbox for "GSP init done". **Device must:** complete every stage in the order the driver polls and leave the mailbox in the state the driver expects. **Emulation:** the host module models the stage state machine and auto-clears self-clearing triggers (e.g. MMU invalidate registers) so polls terminate. + +```text +vgsp: GSP-BOOT: ... trigger auto-cleared ... +``` + +## 5. Static configuration + +**Guest:** `GET_GSP_STATIC_INFO` (65) → a large `GspStaticConfigInfo` structure; reads the VBIOS image from the BAR0 window; queries device/FB/BUS info. **Device must:** provide self-consistent sizes, engine inventories, memory geometry, interrupt topology and a parseable VBIOS blob. **Emulation:** the static structures are synthesized for TU104 (FB size, BAR layout, engine list, vGPU-friendly VF values). + +```text +vgsp-rm: GET_GSP_STATIC_INFO → ... +``` + +## 6. Message queues + +**Guest:** sends `SET_GUEST_SYSTEM_INFO` (1) with libos regions, a queue buffer description and a page table (the buffer is `NV_MEMORY_NONCONTIGUOUS`); also `GSP_SET_SYSTEM_INFO` (72), `SET_REGISTRY` (73) and `CONTINUATION_RECORD` (71) as init traffic. **Device must:** consume them silently (async) and consume the command queue exactly in the guest's write order. **Emulation:** parses the queue chain, builds the geometry, links the status queue and consumes elements one by one — **element-based, not byte-stream-based**: one message is one element and may span pages. + +```text +vgsp: libos region[i] id8=... pa=... size=... +vgsp: queue args sharedPA=... ptes=... cmdOff=... statOff=... +``` + +## 7. The RM control battery + +**Guest:** sends `GSP_RM_CONTROL` (76) for every subsystem it initializes. Of the ~160 observed control IDs, 23 are special-cased with real values; the rest are answered with a generic `SUCCESS` and zeroed params (the fallback). + +| Control | Meaning | What matters | +|---|---|---| +| `0x20800a40` family | internal device info table | per-index values, entry count | +| `0x20801112` | FIFO device info table | engine inventory; count must match engine queries | +| `0x20800a5c` | interrupt kernel table | vector topology the driver maps | +| `0x20800a59` | GMMU static info | fault buffer sizes; a wrong value breaks UVM early | +| `0x20800a01` | display static info | display count; headless values are fine | +| `0x20801823/-a/-b` | BUS/FB info V2 | sysmem connection type and PCIe link caps feed bandwidth math | +| `0x20802a07` | CE physical caps | per-engine bits; UVM refuses channels without `sysmem` support | +| `0x20802a0a/0b` | all-CE caps | caps table for all engines | +| `0x20802a02` | CE PCE mask | one bit per engine | +| `0x20802a08` | CE fault method buffer size | method buffer sizing | +| `0x90f10106` | vaspace reserved-PDE contract | carries page-table physical addresses (see [`GSP-RPC-Emulation.md`](GSP-RPC-Emulation.md)) | +| `0x20800123` / `0x20800170` | `GPU_GET_ENGINES` / `GET_ENGINES_V2` | engine inventory; the guest must see the full list, including async copy (`COPY14`) | +| `0x800201` | `GPU_GET_CLASSLIST` | class list; must agree with engine list and CE caps | +| `0x2080012b` | `GPU_PROMOTE_CTX` (560-byte params) | load-bearing GR context-validity operation; a wrong shape is a server wedge handled by the supervised fail-safe | +| `0x20803039` | NVLink masks/VBIOS info | fallback: generic `SUCCESS` with zeroed params | +| `0x20800a70` | BUS flush with sysmembar | fallback: generic `SUCCESS` with zeroed params; UVM issues these often | + +```text +vgsp-rm: GSP_RM_CONTROL cmd=0x... hClient=0x... hObject=0x... paramsSize=... +``` + +## 8. Object creation + +**Guest:** `GSP_RM_ALLOC` (103) creates the object tree. Typical order for a CUDA process: + +```text +0x0 NV01_ROOT (client root, 120-byte params) +0x80 NV01_DEVICE_0 (56 bytes) +0x2080 NV20_SUBDEVICE_0 (4 bytes) +0x90f1 FERMI_VASPACE_A (48 bytes) ← the GPU address space +0xa06c KEPLER_CHANNEL_GROUP_A (20 bytes) ← CE channel group +0xc46f TURING_CHANNEL_GPFIFO_A(368 bytes) ← the channel (GPFIFO VA, instance/userd descriptors) +0xc076 GP100_UVM_SW (0 bytes) +0xc5b5 TURING_DMA_COPY_A (8 bytes) ← CE object +0xc5c0 TURING_COMPUTE_A (16 bytes) ← compute object (engine registration path) +``` + +A hard-won rule from the `TURING_COMPUTE_A` (0xc5c0) alloc: the caller may issue it with `paramsSize=0` against a small stack local while the kernel side treats the class params as 16 bytes and copies reply params back. The emulation must **echo the request params back in the alloc reply**, or the user-mode client segfaults. Channel parameters decode into `gpFifoOffset` (a GPU VA), `gpFifoEntries`, `engineType` (e.g. `COPY0 = 0x9`), and `NV_MEMORY_DESC_PARAMS` for instance/userd/ramfc/method buffers with **physical** addresses and apertures (`ADDR_SYSMEM = 1`, `ADDR_FBMEM = 2`). + +```text +vgsp-rm: GSP_RM_ALLOC class=0x... paramsSize=... hClient=... hParent=... hObject=... +``` + +## 9. UVM registration and the execution frontier + +**Guest:** `cuInit` → `UVM_REGISTER_GPU` → channel-manager creation → page-tree initialization; each UVM channel does a first push and waits on a semaphore. **Emulation today:** the host decodes the pushes UVM generates and executes their memsets, copies and semaphore releases in software by translating the semaphore VA to a guest physical address — a bridge, not forwarding: no work reaches the physical GPU through it yet. + +**Progress:** context creation now returns (the driver-API smoke completes end to end) and the frontier is execution. The host GR ladder allocates a GR0 channel and a compute-class object; a first semaphore push against an uncommitted GR context faulted `CTXNOTVALID`, and a following `GPU_PROMOTE_CTX` wedged GSP-RM into its designed locked fail-safe. Later, a forwarded guest CUDA launch reached the front-end (ring entry fetched, methods executed) with no observed inline launch-descriptor install and no grid dispatched; context validity at dispatch is ruled out. See [`Execution-Frontier.md`](Execution-Frontier.md). The roadmap from here is: + +```text +UVM registers → cuInit returns → context creation → memory allocation +→ channel creation for the process → kernel launch → (real execution) +``` + +Each remaining step needs the same treatment: satisfy the interface, forward the work, relay the completions. + +```text +nvidia-uvm: ... Tracker entry for value N (sema VA 0x...) channel ID ... CE ... +nvidia-uvm: ... pending push '' started at ... in channel_init() releasing value N +``` + +## 10. Quick reference: function IDs from a stock 570.144 guest + +| Function | Name | Frequency per CUDA run (approx.) | +|---|---|---| +| 1 | SET_GUEST_SYSTEM_INFO | once | +| 10 | FREE | ~14 | +| 47 | UNLOADING_GUEST_DRIVER | on unload | +| 65 | GET_GSP_STATIC_INFO | once | +| 70 | UPDATE_BAR_PDE | a few | +| 71/72/73 | CONTINUATION_RECORD / GSP_SET_SYSTEM_INFO / SET_REGISTRY | once each | +| 76 | GSP_RM_CONTROL | ~160 | +| 103 | GSP_RM_ALLOC | ~60 | + +If a new driver version produces **new function numbers**, do not guess: add a handler that logs the raw payload, then decode it against that version's observed traffic. + +## 11. Related documentation + +* [`../docs/03-architecture/07-driver-lifecycle-field-guide.md`](../docs/03-architecture/07-driver-lifecycle-field-guide.md) — canonical field guide. +* [`GSP-and-GSP-RM.md`](GSP-and-GSP-RM.md) — GSP boot, message queues and RM architecture. +* [`GSP-RPC-Emulation.md`](GSP-RPC-Emulation.md) — control/alloc emulation on the host side. +* [`RPC-Message-Format.md`](RPC-Message-Format.md) — wire format of the function 76/103 traffic. +* [`Address-Reference.md`](Address-Reference.md) and [`GSP-570.144W-Firmware-Analysis.md`](GSP-570.144W-Firmware-Analysis.md) — ID reference and the firmware they belong to. +* [`UVM-in-the-Guest.md`](UVM-in-the-Guest.md), [`Execution-Frontier.md`](Execution-Frontier.md), [`Host-Module.md`](Host-Module.md), [`Physical-GPU-Backend.md`](Physical-GPU-Backend.md) — registration, the frontier, the host stages and the forwarding backend. |
