# Driver Lifecycle Field Guide

> A walking tour of what actually happens between "guest boots" and "CUDA
> initializes", with the real message and control IDs we observe from an
> unmodified 570.144 guest. Use this to interpret logs and to place new
> observations.

## 0. How to use this document

Each stage lists: what the guest does, what the device/firmware must answer,
what our emulation does today, and the log signature to look for
(`dmesg | grep gsp-resman-vfio` on the host; `dmesg` in the guest).

The IDs are for the 570.144 generation on the Turing (TU104) profile. Other
generations will have additions and version differences; the *stages* are
stable.

## 1. PCI probe and config space

**Guest:** reads vendor/device/class IDs, walks capabilities, sizes BARs,
enables Memory Space and Bus Master, programs MSI-X.

**Device must:** report a coherent TU104-family GPU with 16 MB BAR0, 256 MB
BAR1, an MSI-X capability, and a link capability the driver accepts.

**Emulation:** config space is synthesized; BAR sizes are enforced; the
*hypervisor* is the usual source of surprises (a device attached to the root
bus can have its PCIe capability rewritten to "root complex integrated
endpoint", which removes link information the driver wants — attach behind a
root port).

**Log signature:** none from us; guest-side `lspci -vvv` is the reference.

## 2. GSP boot handshake

**Guest:** writes WPR descriptors, releases FWSEC, then the booter, then the
RISC-V core; polls a status mailbox for "GSP init done".

**Device must:** complete every stage in the order the driver polls, and leave
the mailbox in the state the driver expects.

**Emulation:** `kernel/gsp_emulation.c` models the stage state machine and
auto-clears self-clearing triggers (e.g. MMU invalidate registers) so polls
terminate.

**Log signature:**
```
gsp-resman-vfio: GSP-BOOT: ... trigger auto-cleared ...
```

## 3. Static configuration

**Guest:** `GET_GSP_STATIC_INFO` (65) → receives a large `GspStaticConfigInfo`
structure; reads the VBIOS image from the BAR0 window; queries device/FB/BUS
info.

**Device must:** provide self-consistent sizes, engine inventories, memory
geometry, interrupt topology, and a VBIOS blob the driver can parse.

**Emulation:** static structures are synthesized for TU104 (FB size, BAR
layout, engine list, vGPU-friendly VF values).

**Log signature:**
```
GSPRM: GET_GSP_STATIC_INFO → ...
```

## 4. Message queues

**Guest:** sends `SET_GUEST_SYSTEM_INFO` (1) with libos regions, a queue
buffer description, and a page table (the queue buffer is
`NV_MEMORY_NONCONTIGUOUS`). It also sends `GSP_SET_SYSTEM_INFO` (72),
`SET_REGISTRY` (73) and `CONTINUATION_RECORD` (71) as init traffic.

**Device must:** consume them silently (async) and start consuming the command
queue exactly in the order the guest writes elements.

**Emulation:** parses the queue chain, builds the queue geometry, links the
status queue, and consumes elements one by one. **Element-based, not
byte-stream-based**: each message is one element and may span pages.

**Log signature:**
```
gsp-resman-vfio: libos region[i] id8=... pa=... size=...
gsp-resman-vfio: queue args sharedPA=... ptes=... cmdOff=... statOff=...
```

## 5. Resource-manager initialization (the control battery)

**Guest:** sends `GSP_RM_CONTROL` (76) for every subsystem it initializes.
Of the ~160 observed control IDs, 23 are special-cased and answered with real
values; the rest are answered with a generic `SUCCESS` and zeroed params (the
fallback). Representative controls:

| Control | Meaning | What matters |
|---|---|---|
| `0x20800a40`-family | internal device info table | per-index values, entry count |
| `0x20801112` | FIFO device info table | engine inventory; count must match engine queries |
| `0x20800a5c` | interrupt kernel table | vector topology the driver maps |
| `0x20800a59` | GMMU static info | **fault buffer sizes** (replayable/legacy) — a wrong value breaks UVM early |
| `0x20800a01` | display static info | display count (headless values are fine) |
| `0x20801823/-a/b` | BUS/FB info V2 | sysmem connection type and PCIe link caps feed bandwidth calculations |
| `0x20802a07` | CE physical caps | per-engine support bits; UVM refuses to make channels without `sysmem` support |
| `0x20802a0a/0b` | all-CE caps | caps table for all engines |
| `0x20802a02` | CE PCE mask | one bit per engine |
| `0x20802a08` | CE fault method buffer size | method buffer sizing |
| `0x90f10106` | vaspace reserved-PDE contract | **carries page-table physical addresses** (see GSP-RM doc) |
| `0x20800123` / `0x20800170` | `GPU_GET_ENGINES` / `GET_ENGINES_V2` | engine inventory (filtered by the class list / classDB); values are consumed — the guest must see the full engine list, including the async copy engine (`COPY14`); that inventory is now complete, and the context-creation wait lives later in the completion path |
| `0x800201` | `GPU_GET_CLASSLIST` | class list (classDB); must agree with the engine list and the CE caps controls |
| `0x2080012b` | GPU_PROMOTE_CTX (560-byte params) | historically issued immediately before context creation and answered by the generic fallback. Context creation now returns; in the host GR ladder this promote is the load-bearing GR context-validity operation, and a wrong shape is a known server wedge (no retry; reboot policy - [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md) §7) |
| `0x20803039` | NVLink masks/VBIOS info | fallback: generic `SUCCESS` with zeroed params |
| `0x20800a70` | BUS flush with sysmembar | fallback: generic `SUCCESS` with zeroed params; UVM issues these frequently |

**Log signature:**
```
GSPRM: GSP_RM_CONTROL cmd=0x... hClient=0x... hObject=0x... paramsSize=...
```

## 6. Object creation

**Guest:** `GSP_RM_ALLOC` (103) creates the object tree. Typical order for a
CUDA process:

```
0x0        NV01_ROOT              (client root, 120-byte params)
0x80       NV01_DEVICE_0          (56 bytes)
0x2080     NV20_SUBDEVICE_0       (4 bytes)
0x90f1     FERMI_VASPACE_A        (48 bytes)  ← the GPU address space
0xa06c     KEPLER_CHANNEL_GROUP_A (20 bytes)  ← CE channel group
0xc46f     TURING_CHANNEL_GPFIFO_A(368 bytes) ← the channel (GPFIFO VA, instance/userd descriptors)
0xc076     GP100_UVM_SW           (0 bytes)
0xc5b5     TURING_DMA_COPY_A      (8 bytes)   ← CE object
0xc5c0     TURING_COMPUTE_A       (16 bytes)  ← compute object (engine registration path)
```

The `TURING_COMPUTE_A` (0xc5c0) alloc is where the earlier `libcuda` segfault
was fixed: the caller issues it with `paramsSize=0` against an 8-byte stack
local while the kernel side treats the class params as 16 bytes and copies the
reply params back, so the emulation must echo the request params back in the
alloc reply instead of writing a reply the caller did not ask for.

Channel parameters decode into: `gpFifoOffset` (a GPU VA),
`gpFifoEntries`, `engineType` (e.g. COPY0 = 0x9), and
`NV_MEMORY_DESC_PARAMS` for instance/userd/ramfc/method buffers with
**physical** addresses and apertures (`ADDR_SYSMEM = 1`, `ADDR_FBMEM = 2`).

**Emulation:** handle management and parameter shapes are validated against
the reference structures; values are logged in a form that makes the object
tree reconstructible from `dmesg`.

**Log signature:**
```
GSPRM: GSP_RM_ALLOC class=0x... paramsSize=... hClient=... hParent=... hObject=...
```

## 7. UVM registration (the wall we climbed)

**Guest:** `cuInit` → `UVM_REGISTER_GPU` → channel manager creation →
page-tree init.

**Device must:** let UVM's channels initialize (each channel does a first push
and waits on a semaphore), and let page-tree initialization proceed.

**Emulation (today):** we can decode the pushes UVM generates and execute
their memsets, copies and semaphore releases in software by translating the
semaphore VA to a guest physical address (see the UVM and host-module docs).
This is a bridge, not forwarding: no work reaches the physical GPU yet.

**Live development front (2026-09-29):** CUDA context creation — the client
reaches `cuCtxCreate` and currently ends in a forced `rc=999` on a completion
wait; the engine inventory is complete and the completion/translation path is
the focus. The engine-record layout and the alloc reply-fidelity checks are
documented in
([`../../userspace/cuda/docs/05-570.144-decompilation.md` §3.3/§3.6](../../userspace/cuda/docs/05-570.144-decompilation.md);
start at
[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md));
the execution backend is in
[`08-physical-gpu-integration.md`](08-physical-gpu-integration.md).

**Update (2026-10-02):** context creation now returns (the driver-API smoke
completes end to end) and the frontier is execution. The host GR ladder
allocates a GR0 channel and a compute-class object, but its first semaphore
push faults with an uncommitted GR context (`CTXNOTVALID`), and the
`GPU_PROMOTE_CTX` attempt that followed wedged GSP-RM; the device then sits in
the designed locked fail-safe (see
[`08-physical-gpu-integration.md`](08-physical-gpu-integration.md) §7/§8).

**Log signatures (guest side):**
```
nvidia-uvm: ... Tracker entry for value N (sema VA 0x...) channel ID ... CE ...
nvidia-uvm: ... pending push '' started at ... in channel_init() releasing value N
```

## 8. What comes after registration (roadmap)

```
UVM registers → cuInit returns → context creation → memory allocation
→ channel creation for the process → kernel launch → (real execution)
```

Each step will need the same treatment: satisfy the interface, forward the
work, relay the completions. The known-unknowns document lists what we expect
to be hard.

## 9. Quick reference: message/control IDs seen from a stock 570.144 guest

| Function | Name | Frequency per CUDA run (approx.) |
|---|---|---|
| 1 | SET_GUEST_SYSTEM_INFO | once |
| 10 | FREE | ~14 |
| 47 | UNLOADING_GUEST_DRIVER | on unload |
| 65 | GET_GSP_STATIC_INFO | once |
| 70 | UPDATE_BAR_PDE | a few |
| 71/72/73 | CONTINUATION_RECORD / GSP_SET_SYSTEM_INFO / SET_REGISTRY | once each |
| 76 | GSP_RM_CONTROL | ~160 |
| 103 | GSP_RM_ALLOC | ~60 |

If a new driver version produces **new function numbers**, do not guess: add a
handler that logs the raw payload, then decode it against the decompiled
study notes for that driver version.