diff options
Diffstat (limited to 'docs/03-architecture/07-driver-lifecycle-field-guide.md')
| -rw-r--r-- | docs/03-architecture/07-driver-lifecycle-field-guide.md | 188 |
1 files changed, 188 insertions, 0 deletions
diff --git a/docs/03-architecture/07-driver-lifecycle-field-guide.md b/docs/03-architecture/07-driver-lifecycle-field-guide.md new file mode 100644 index 0000000..8d80694 --- /dev/null +++ b/docs/03-architecture/07-driver-lifecycle-field-guide.md @@ -0,0 +1,188 @@ +# Driver Lifecycle Field Guide + +> A walking tour of what actually happens between "guest boots" and "CUDA +> initializes", with the real message and control IDs we observe from an +> unmodified 570.144 guest. Use this to interpret logs and to place new +> observations. + +## 0. How to use this document + +Each stage lists: what the guest does, what the device/firmware must answer, +what our emulation does today, and the log signature to look for +(`dmesg | grep gsp-resman-vfio` on the host; `dmesg` in the guest). + +The IDs are for the 570.144 generation on the Turing (TU104) profile. Other +generations will have additions and version differences; the *stages* are +stable. + +## 1. PCI probe and config space + +**Guest:** reads vendor/device/class IDs, walks capabilities, sizes BARs, +enables Memory Space and Bus Master, programs MSI-X. + +**Device must:** report a coherent TU104-family GPU with 16 MB BAR0, 256 MB +BAR1, an MSI-X capability, and a link capability the driver accepts. + +**Emulation:** config space is synthesized; BAR sizes are enforced; the +*hypervisor* is the usual source of surprises (a device attached to the root +bus can have its PCIe capability rewritten to "root complex integrated +endpoint", which removes link information the driver wants — attach behind a +root port). + +**Log signature:** none from us; guest-side `lspci -vvv` is the reference. + +## 2. GSP boot handshake + +**Guest:** writes WPR descriptors, releases FWSEC, then the booter, then the +RISC-V core; polls a status mailbox for "GSP init done". + +**Device must:** complete every stage in the order the driver polls, and leave +the mailbox in the state the driver expects. + +**Emulation:** `kernel/gsp_emulation.c` models the stage state machine and +auto-clears self-clearing triggers (e.g. MMU invalidate registers) so polls +terminate. + +**Log signature:** +``` +gsp-resman-vfio: GSP-BOOT: ... trigger auto-cleared ... +``` + +## 3. Static configuration + +**Guest:** `GET_GSP_STATIC_INFO` (65) → receives a large `GspStaticConfigInfo` +structure; reads the VBIOS image from the BAR0 window; queries device/FB/BUS +info. + +**Device must:** provide self-consistent sizes, engine inventories, memory +geometry, interrupt topology, and a VBIOS blob the driver can parse. + +**Emulation:** static structures are synthesized for TU104 (FB size, BAR +layout, engine list, vGPU-friendly VF values). + +**Log signature:** +``` +GSPRM: GET_GSP_STATIC_INFO → ... +``` + +## 4. Message queues + +**Guest:** sends `SET_GUEST_SYSTEM_INFO` (1) with libos regions, a queue +buffer description, and a page table (the queue buffer is +`NV_MEMORY_NONCONTIGUOUS`). It also sends `GSP_SET_SYSTEM_INFO` (72), +`SET_REGISTRY` (73) and `CONTINUATION_RECORD` (71) as init traffic. + +**Device must:** consume them silently (async) and start consuming the command +queue exactly in the order the guest writes elements. + +**Emulation:** parses the queue chain, builds the queue geometry, links the +status queue, and consumes elements one by one. **Element-based, not +byte-stream-based**: each message is one element and may span pages. + +**Log signature:** +``` +gsp-resman-vfio: libos region[i] id8=... pa=... size=... +gsp-resman-vfio: queue args sharedPA=... ptes=... cmdOff=... statOff=... +``` + +## 5. Resource-manager initialization (the control battery) + +**Guest:** sends `GSP_RM_CONTROL` (76) for every subsystem it initializes. +Representative controls (all observed, all answered with correct shapes): + +| Control | Meaning | What matters | +|---|---|---| +| `0x20800a40`-family | internal device info table | per-index values, entry count | +| `0x20801112` | FIFO device info table | engine inventory; count must match engine queries | +| `0x20800a5c` | interrupt kernel table | vector topology the driver maps | +| `0x20800a59` | GMMU static info | **fault buffer sizes** (replayable/legacy) — a wrong value breaks UVM early | +| `0x20800a01` | display static info | display count (headless values are fine) | +| `0x20801823/-a/b` | BUS/FB info V2 | sysmem connection type and PCIe link caps feed bandwidth calculations | +| `0x20802a07` | CE physical caps | per-engine support bits; UVM refuses to make channels without `sysmem` support | +| `0x20802a0a/0b` | all-CE caps | caps table for all engines | +| `0x20802a02` | CE PCE mask | one bit per engine | +| `0x20802a08` | CE fault method buffer size | method buffer sizing | +| `0x90f10106` | vaspace reserved-PDE contract | **carries page-table physical addresses** (see GSP-RM doc) | +| `0x20803039` | NVLink masks/VBIOS info | unpopulated but must be shaped | +| `0x20800a70` | BUS flush with sysmembar | UVM issues these frequently; success suffices | + +**Log signature:** +``` +GSPRM: GSP_RM_CONTROL cmd=0x... hClient=0x... hObject=0x... paramsSize=... +``` + +## 6. Object creation + +**Guest:** `GSP_RM_ALLOC` (103) creates the object tree. Typical order for a +CUDA process: + +``` +0x0 NV01_ROOT (client root, 120-byte params) +0x80 NV01_DEVICE_0 (56 bytes) +0x2080 NV20_SUBDEVICE_0 (4 bytes) +0x90f1 FERMI_VASPACE_A (48 bytes) ← the GPU address space +0xa06c KEPLER_CHANNEL_GROUP_A (20 bytes) ← CE channel group +0xc46f TURING_CHANNEL_GPFIFO_A(368 bytes) ← the channel (GPFIFO VA, instance/userd descriptors) +0xc076 GP100_UVM_SW (0 bytes) +0xc5b5 TURING_DMA_COPY_A (8 bytes) ← CE object +``` + +Channel parameters decode into: `gpFifoOffset` (a GPU VA), +`gpFifoEntries`, `engineType` (e.g. COPY0 = 0x9), and +`NV_MEMORY_DESC_PARAMS` for instance/userd/ramfc/method buffers with +**physical** addresses and apertures (`ADDR_SYSMEM = 1`, `ADDR_FBMEM = 2`). + +**Emulation:** handle management and parameter shapes are validated against +the reference structures; values are logged in a form that makes the object +tree reconstructible from `dmesg`. + +**Log signature:** +``` +GSPRM: GSP_RM_ALLOC class=0x... paramsSize=... hClient=... hParent=... hObject=... +``` + +## 7. UVM registration (the wall we climbed) + +**Guest:** `cuInit` → `UVM_REGISTER_GPU` → channel manager creation → +page-tree init. + +**Device must:** let UVM's channels initialize (each channel does a first push +and waits on a semaphore), and let page-tree initialization proceed. + +**Emulation (today):** we can decode the pushes UVM generates and complete +their semaphore releases by translating the semaphore VA to a guest physical +address (see the UVM and host-module docs). This is the live development front. + +**Log signatures (guest side):** +``` +nvidia-uvm: ... Tracker entry for value N (sema VA 0x...) channel ID ... CE ... +nvidia-uvm: ... pending push '' started at ... in channel_init() releasing value N +``` + +## 8. What comes after registration (roadmap) + +``` +UVM registers → cuInit returns → context creation → memory allocation +→ channel creation for the process → kernel launch → (real execution) +``` + +Each step will need the same treatment: satisfy the interface, forward the +work, relay the completions. The known-unknowns document lists what we expect +to be hard. + +## 9. Quick reference: message/control IDs seen from a stock 570.144 guest + +| Function | Name | Frequency per CUDA run (approx.) | +|---|---|---| +| 1 | SET_GUEST_SYSTEM_INFO | once | +| 10 | FREE | ~14 | +| 47 | UNLOADING_GUEST_DRIVER | on unload | +| 65 | GET_GSP_STATIC_INFO | once | +| 70 | UPDATE_BAR_PDE | a few | +| 71/72/73 | CONTINUATION_RECORD / GSP_SET_SYSTEM_INFO / SET_REGISTRY | once each | +| 76 | GSP_RM_CONTROL | ~160 | +| 103 | GSP_RM_ALLOC | ~60 | + +If a new driver version produces **new function numbers**, do not guess: add a +handler that logs the raw payload, then decode it against the decompiled +study notes for that driver version.
\ No newline at end of file |