# Driver Lifecycle Field Guide

> A walking tour of what actually happens between "guest boots" and "CUDA
> initializes", with the real message and control IDs we observe from an
> unmodified 570.144 guest. Use this to interpret logs and to place new
> observations.

## 0. How to use this document

Each stage lists: what the guest does, what the device/firmware must answer,
what our emulation does today, and the log signature to look for
(`dmesg | grep gsp-resman-vfio` on the host; `dmesg` in the guest).

The IDs are for the 570.144 generation on the Turing (TU104) profile. Other
generations will have additions and version differences; the *stages* are
stable.

## 1. PCI probe and config space

**Guest:** reads vendor/device/class IDs, walks capabilities, sizes BARs,
enables Memory Space and Bus Master, programs MSI-X.

**Device must:** report a coherent TU104-family GPU with 16 MB BAR0, 256 MB
BAR1, an MSI-X capability, and a link capability the driver accepts.

**Emulation:** config space is synthesized; BAR sizes are enforced; the
*hypervisor* is the usual source of surprises (a device attached to the root
bus can have its PCIe capability rewritten to "root complex integrated
endpoint", which removes link information the driver wants — attach behind a
root port).

**Log signature:** none from us; guest-side `lspci -vvv` is the reference.

## 2. GSP boot handshake

**Guest:** writes WPR descriptors, releases FWSEC, then the booter, then the
RISC-V core; polls a status mailbox for "GSP init done".

**Device must:** complete every stage in the order the driver polls, and leave
the mailbox in the state the driver expects.

**Emulation:** `kernel/gsp_emulation.c` models the stage state machine and
auto-clears self-clearing triggers (e.g. MMU invalidate registers) so polls
terminate.

**Log signature:**
```
gsp-resman-vfio: GSP-BOOT: ... trigger auto-cleared ...
```

## 3. Static configuration

**Guest:** `GET_GSP_STATIC_INFO` (65) → receives a large `GspStaticConfigInfo`
structure; reads the VBIOS image from the BAR0 window; queries device/FB/BUS
info.

**Device must:** provide self-consistent sizes, engine inventories, memory
geometry, interrupt topology, and a VBIOS blob the driver can parse.

**Emulation:** static structures are synthesized for TU104 (FB size, BAR
layout, engine list, vGPU-friendly VF values).

**Log signature:**
```
GSPRM: GET_GSP_STATIC_INFO → ...
```

## 4. Message queues

**Guest:** sends `SET_GUEST_SYSTEM_INFO` (1) with libos regions, a queue
buffer description, and a page table (the queue buffer is
`NV_MEMORY_NONCONTIGUOUS`). It also sends `GSP_SET_SYSTEM_INFO` (72),
`SET_REGISTRY` (73) and `CONTINUATION_RECORD` (71) as init traffic.

**Device must:** consume them silently (async) and start consuming the command
queue exactly in the order the guest writes elements.

**Emulation:** parses the queue chain, builds the queue geometry, links the
status queue, and consumes elements one by one. **Element-based, not
byte-stream-based**: each message is one element and may span pages.

**Log signature:**
```
gsp-resman-vfio: libos region[i] id8=... pa=... size=...
gsp-resman-vfio: queue args sharedPA=... ptes=... cmdOff=... statOff=...
```

## 5. Resource-manager initialization (the control battery)

**Guest:** sends `GSP_RM_CONTROL` (76) for every subsystem it initializes.
Representative controls (all observed, all answered with correct shapes):

| Control | Meaning | What matters |
|---|---|---|
| `0x20800a40`-family | internal device info table | per-index values, entry count |
| `0x20801112` | FIFO device info table | engine inventory; count must match engine queries |
| `0x20800a5c` | interrupt kernel table | vector topology the driver maps |
| `0x20800a59` | GMMU static info | **fault buffer sizes** (replayable/legacy) — a wrong value breaks UVM early |
| `0x20800a01` | display static info | display count (headless values are fine) |
| `0x20801823/-a/b` | BUS/FB info V2 | sysmem connection type and PCIe link caps feed bandwidth calculations |
| `0x20802a07` | CE physical caps | per-engine support bits; UVM refuses to make channels without `sysmem` support |
| `0x20802a0a/0b` | all-CE caps | caps table for all engines |
| `0x20802a02` | CE PCE mask | one bit per engine |
| `0x20802a08` | CE fault method buffer size | method buffer sizing |
| `0x90f10106` | vaspace reserved-PDE contract | **carries page-table physical addresses** (see GSP-RM doc) |
| `0x20803039` | NVLink masks/VBIOS info | unpopulated but must be shaped |
| `0x20800a70` | BUS flush with sysmembar | UVM issues these frequently; success suffices |

**Log signature:**
```
GSPRM: GSP_RM_CONTROL cmd=0x... hClient=0x... hObject=0x... paramsSize=...
```

## 6. Object creation

**Guest:** `GSP_RM_ALLOC` (103) creates the object tree. Typical order for a
CUDA process:

```
0x0        NV01_ROOT              (client root, 120-byte params)
0x80       NV01_DEVICE_0          (56 bytes)
0x2080     NV20_SUBDEVICE_0       (4 bytes)
0x90f1     FERMI_VASPACE_A        (48 bytes)  ← the GPU address space
0xa06c     KEPLER_CHANNEL_GROUP_A (20 bytes)  ← CE channel group
0xc46f     TURING_CHANNEL_GPFIFO_A(368 bytes) ← the channel (GPFIFO VA, instance/userd descriptors)
0xc076     GP100_UVM_SW           (0 bytes)
0xc5b5     TURING_DMA_COPY_A      (8 bytes)   ← CE object
```

Channel parameters decode into: `gpFifoOffset` (a GPU VA),
`gpFifoEntries`, `engineType` (e.g. COPY0 = 0x9), and
`NV_MEMORY_DESC_PARAMS` for instance/userd/ramfc/method buffers with
**physical** addresses and apertures (`ADDR_SYSMEM = 1`, `ADDR_FBMEM = 2`).

**Emulation:** handle management and parameter shapes are validated against
the reference structures; values are logged in a form that makes the object
tree reconstructible from `dmesg`.

**Log signature:**
```
GSPRM: GSP_RM_ALLOC class=0x... paramsSize=... hClient=... hParent=... hObject=...
```

## 7. UVM registration (the wall we climbed)

**Guest:** `cuInit` → `UVM_REGISTER_GPU` → channel manager creation →
page-tree init.

**Device must:** let UVM's channels initialize (each channel does a first push
and waits on a semaphore), and let page-tree initialization proceed.

**Emulation (today):** we can decode the pushes UVM generates and complete
their semaphore releases by translating the semaphore VA to a guest physical
address (see the UVM and host-module docs). This is the live development front.

**Log signatures (guest side):**
```
nvidia-uvm: ... Tracker entry for value N (sema VA 0x...) channel ID ... CE ...
nvidia-uvm: ... pending push '' started at ... in channel_init() releasing value N
```

## 8. What comes after registration (roadmap)

```
UVM registers → cuInit returns → context creation → memory allocation
→ channel creation for the process → kernel launch → (real execution)
```

Each step will need the same treatment: satisfy the interface, forward the
work, relay the completions. The known-unknowns document lists what we expect
to be hard.

## 9. Quick reference: message/control IDs seen from a stock 570.144 guest

| Function | Name | Frequency per CUDA run (approx.) |
|---|---|---|
| 1 | SET_GUEST_SYSTEM_INFO | once |
| 10 | FREE | ~14 |
| 47 | UNLOADING_GUEST_DRIVER | on unload |
| 65 | GET_GSP_STATIC_INFO | once |
| 70 | UPDATE_BAR_PDE | a few |
| 71/72/73 | CONTINUATION_RECORD / GSP_SET_SYSTEM_INFO / SET_REGISTRY | once each |
| 76 | GSP_RM_CONTROL | ~160 |
| 103 | GSP_RM_ALLOC | ~60 |

If a new driver version produces **new function numbers**, do not guess: add a
handler that logs the raw payload, then decode it against the decompiled
study notes for that driver version.