# 03 — What the CUDA userspace driver asks of the kernel and the GPU

This document is the emulation contract: everything `libcuda` needs from
below itself. It is the checklist Open-IOV's host-side emulation has to
satisfy for unmodified CUDA to run, and the map used when a guest call fails.

Sources: live ioctl traces from the guest (ptrace tracer), host-side GSP-RM
emulation logs, and decompilation of the RM backend in `libcuda` 570.144. The
ioctl numbers below were observed on the wire; the object classes and control
IDs are quoted from the trace and from public CUDA/RM headers.

---

## 1. Device nodes and the ioctl vocabulary

| Node | Used for |
|---|---|
| `/dev/nvidiactl` | process-wide control: version negotiation, client creation, GPU attach, OS events, exported-object file descriptors |
| `/dev/nvidiaN` | per-GPU: object alloc/free, RM controls, memory alloc/map, event allocation |
| `/dev/nvidia-uvm` | managed memory and page migration (UVM userspace API) |
| `/dev/nvidia-vgpuN` | vGPU plugin node (only inside NVIDIA's own vGPU stack; not used by stock CUDA paths) |

Control-device ioctls (`_IOWR('F', cmd, …)` with magic `'F'`):

| cmd | Name | Purpose |
|---|---|---|
| 200 (`0xc8`) | `NV_ESC_CARD_INFO` | enumerate GPUs and minors |
| 201 (`0xc9`) | `NV_ESC_REGISTER_FD` | bind a per-GPU fd to the control fd |
| 206/207 | `NV_ESC_ALLOC_OS_EVENT` / `FREE_OS_EVENT` | OS event objects (blocking waits, GPU events) |
| 209 (`0xd1`) | `NV_ESC_STATUS_CODE` | driver status on open failure |
| 210 (`0xd2`) | `NV_ESC_CHECK_VERSION_STR` | RM/driver version negotiation |
| 211 (`0xd3`) | `NV_ESC_IOCTL_XFER_CMD` | inter-GPU ioctl forwarding |
| 212 (`0xd4`) | `NV_ESC_ATTACH_GPUS_TO_FD` | attach GPUs to an fd (exported objects) |
| 213 (`0xd5`) | `NV_ESC_QUERY_DEVICE_INTR` | interrupt query |
| 214 (`0xd6`) | `NV_ESC_SYS_PARAMS` | system parameters |

RM escapes (same magic, used on control and per-GPU fds):

| cmd | Name | Purpose |
|---|---|---|
| 0x27 | `NV_ESC_RM_ALLOC_MEMORY` | allocate memory (older path; NVOS02) |
| 0x28 | `NV_ESC_RM_ALLOC_OBJECT` | allocate object (NVOS05) |
| 0x29 | `NV_ESC_RM_FREE` | free object (NVOS00) |
| 0x2a | `NV_ESC_RM_CONTROL` | issue an RM control (NVOS54) |
| 0x2b | `NV_ESC_RM_ALLOC` | allocate object (NVOS21/NVOS64) — **the main alloc path** |
| 0x34 | `NV_ESC_RM_DUP_OBJECT` | duplicate/import an object |
| 0x35 | `NV_ESC_RM_SHARE` | share object |
| 0x41 | `NV_ESC_RM_IDLE_CHANNELS` | idle channels |
| 0x4a | `NV_ESC_RM_VID_HEAP_CONTROL` | heap allocation (NVOS32) |
| 0x4e / 0x4f | `NV_ESC_RM_MAP_MEMORY` / `UNMAP_MEMORY` | map memory into the process (NVOS33/NVOS34) |
| 0x57 / 0x58 | `NV_ESC_RM_MAP_MEMORY_DMA` / `UNMAP_MEMORY_DMA` | map memory into a device VA space (NVOS46/NVOS47) |
| 0x5c / 0x5d | `NV_ESC_RM_EXPORT_OBJECT_TO_FD` / `IMPORT_OBJECT_FROM_FD` | cross-process object sharing |
| 0x52 | `NV_ESC_RM_GET_EVENT_DATA` | read event payload (NVOS41) |
| 0x32/0x33 | `NV_ESC_RM_CONFIG_GET/SET` | registry-style configuration |

In a trace, the ioctl request word encodes the escape and the size; e.g.
`_IOWR('F',0x2b,48)` = `NV_ESC_RM_ALLOC` with a 48-byte payload. Seeing this
request in a crash register dump (`rsi=0xc030462b`) identifies "driver was
allocating an RM object" — that is exactly what happens at the `cuCtxCreate`
crash (see `docs/05`).

## 2. The RM object model as CUDA uses it

Every RM call names a **class**. The classes CUDA allocates, in the order
observed for 570.144 on an emulated TU104:

| Class | Name (public RM header) | When |
|---|---|---|
| `0x0` | `NV01_ROOT` | process client (once, at `cuInit`) |
| `0x80` | `NV01_DEVICE_0` | per GPU, at attach |
| `0x2080` | `NV20_SUBDEVICE_0` | per GPU/subdevice |
| `0x40` | `NV01_EVENT` | events (several) |
| `0xa06c` | `KEPLER_CHANNEL_GROUP_A` | context channel group (TSG) |
| `0x9067` | channel-group/runlist object | context |
| `0x3e` | `NV01_MEMORY_*` | memory objects |
| `0xc46f` | `TURING_CHANNEL_GPFIFO_A` | the channel itself |
| `0xc5c0` | `TURING_COMPUTE_A` | compute engine object bound to the channel |
| `0xc5b5` | (compute class, second engine) | used by the host A/B run for a CE-style object |
| `0xa080` | `KEPLER_DEVICE_VGPU` | vGPU capability query (vGPU only) |
| `0x90f1`-family | VA space (`FERMI_VASPACE_A` etc.) | memory model |

`cuCtxCreate`'s observed ioctl sequence (guest, 570.144):

```
NV_ESC_RM_ALLOC class=0x0      (NV01_ROOT)
NV_ESC_RM_ALLOC class=0x80     (NV01_DEVICE_0)
NV_ESC_RM_ALLOC class=0x2080   (NV20_SUBDEVICE_0)
NV_ESC_RM_ALLOC class=0x40     (events)
NV_ESC_RM_ALLOC class=0xa06c   (KEPLER_CHANNEL_GROUP_A)
NV_ESC_RM_ALLOC class=0x9067   (channel group object)
NV_ESC_RM_ALLOC class=0x3e     (memory)
NV_ESC_RM_ALLOC class=0xc46f   (TURING_CHANNEL_GPFIFO_A)      <- channel
NV_ESC_RM_CONTROL cmd=0x80170d (NV0080_CTRL_FIFO_GET_CHANNELLIST)
NV_ESC_RM_ALLOC class=0xc5c0   (TURING_COMPUTE_A)             <- last alloc
[engine registration attempt; pre-fix this segfaulted — now rc=999, see docs/05 §3.5-3.6]
```

The first two allocs (`NV01_ROOT`, `NV01_DEVICE_0`) actually happen at
`cuInit`/device attach; the trace window shows them repeated per client setup.

## 3. RM controls the CUDA path uses

Controls are `NV_ESC_RM_CONTROL` with a 32-bit command ID and a fixed-size
parameter block. Families relevant to CUDA bring-up (IDs in the low 32 bits
are `(class << 16) | cmd`; the list below is from traces and decompiled call
sites, not exhaustive):

* **Device/GPU info** — `NV0080_CTRL_CMD_GPU_GET_VIRTUALIZATION_MODE`
  (`0x00800089`), `NV2080_CTRL_CMD_GPU_GET_INFO`, `GPU_GET_ENGINES`,
  `GPU_GET_CLASSLIST`, `GPU_GET_SIMULATION_INFO`, `FB_GET_INFO`,
  `BUS_GET_INFO`, `BUS_GET_PCI_INFO`, `BUS_GET_PCI_BAR_INFO`,
  `MC_GET_ARCH_INFO`, `GR_GET_INFO`, `GR_GET_GPC_MASK`, `GR_GET_TPC_MASK`,
  `GR_GET_GLOBAL_SM_ORDER`, `FB_GET_LTC_INFO_FOR_FBP`, `CE_GET_CAPS`,
  `GSP_GET_FEATURES`, `GPU_GET_SW_FEATURES`, `GPU_GET_MEMOP_ENABLE`,
  `FB_GET_CAPS`, `GR_GET_CAPS_V2`.
* **FIFO/channels** — `NV0080_CTRL_CMD_FIFO_GET_CHANNELLIST` (`0x80170d`),
  `NV0080_CTRL_CMD_FIFO_GET_CAPS`, `NVA06F/NVA06C_CTRL_CMD_GPFIFO_SCHEDULE`,
  `SET_ERROR_NOTIFIER`, `NVA06C_CTRL_CMD_SET/GET_TIMESLICE`,
  `NV2080_CTRL_CMD_FIFO_DISABLE_CHANNELS`, per-class
  `GET_CLASS_ENGINEID` (e.g. `NV906F`, `NVA06F`-family),
  `NVC36F_CTRL_CMD_GPFIFO_GET_WORK_SUBMIT_TOKEN`.
* **GR/context** — `GR_CTXSW_PM_MODE`, `GR_CTXSW_SMPC_MODE`,
  `GR_SET_CTXSW_PREEMPTION_MODE`, `GR_PC_SAMPLING_MODE`,
  `GR_GET_CTX_BUFFER_SIZE`, `FLCN_GET_CTX_BUFFER_SIZE`.
* **Memory** — `FB_GET_INFO` (BAR1 info), `NV0000_CTRL_CMD_OS_UNIX_*`
  (flush user cache, exported objects), `NV0041_CTRL_CMD_GET_SURFACE_INFO`,
  `NV0000_CTRL_CMD_CLIENT_GET_ADDR_SPACE_TYPE`.
* **Events/interrupts** — `NV90E6_CTRL_CMD_MASTER_GET_VIRTUAL_FUNCTION_ERROR_CONT_INTR_MASK`
  (interrupt-tree setup), `NV83DE_CTRL_CMD_DEBUG_*` (SM error states, debug
  mode), `RC_ENABLE/DISABLE/SOFT_DISABLE_WATCHDOG`, `RELEASE_WATCHDOG_REQUESTS`.
* **vGPU/licensing** — `NVA080_CTRL_CMD_VGPU_GET_CONFIG` (`0xa0800301`),
  `NVA080_CTRL_CMD_VGPU_SET_LICENSE_INFO`,
  `NV2080_CTRL_CMD_GPU_GET_LICENSABLE_FEATURES`,
  `GPU_GET_GRID_UNLICENSED_STATE_MACHINE_INFO` (`0x2080018c`).
* **Perf/clocks** — `GPU_GET_COMPUTE_POLICY_CONFIG`, boost/sync-boost
  controls, `NV2080_CTRL_CMD_GPU_ACQUIRE/RELEASE_COMPUTE_MODE_RESERVATION`.

Traces show the driver tolerating unknown-but-successful controls: it only
fails when a *specific* reply field is consumed and wrong. This is why the
emulation can start with generic success and then bisect the consumed fields
(documented in `docs/04` as the "reply-fidelity ladder").

## 4. Memory: allocation, mapping, VA

CUDA's memory path uses three layers:

1. **VA reservation** — `NV_ESC_RM_VID_HEAP_CONTROL` with a virtual
   `NVOS32_FUNCTION_ALLOC_SIZE` request (or no reservation at all when UVM
   owns the VA space).
2. **Backing allocation** — vidmem/sysmem `NVOS32` allocation with attributes:
   location (`VIDMEM`/`PCI`/`SYSTEM`), page size, coherency, compression,
   protection, fixed physical address (if requested), or pinning an existing
   host address (`NV01_MEMORY_SYSTEM_OS_DESCRIPTOR`).
3. **Mapping** — `NV_ESC_RM_MAP_MEMORY` (`NVOS33`) to map into the process
   (for host-visible/CPU access, flags include MAP_FIXED, RESERVE_ON_UNMAP,
   DIRECT for incoherent sysmem) and `NV_ESC_RM_MAP_MEMORY_DMA` (`NVOS46`) to
   map into a device VA space (flags include RW/RO, snoop, fixed DMA offset,
   page size, peer flags).

Related classes: `FERMI_VASPACE_A` (VA space; one is shared with UVM, one
legacy, optionally one for fabric-local aperture), `NV01_MEMORY_FABRIC*` /
`NV_MEMORY_FABRIC*` (NVLink/fabric), `NV01_MEMORY_SYSTEM_OS_DESCRIPTOR`
(pinned host memory), `NV01_MEMORY_LOCAL_USER` (vidmem).

For Open-IOV the practical summary is: **all guest memory that CUDA uses is
either an RM allocation backed by emulated VRAM/BAR1, or pinned guest RAM that
the GPU must be able to DMA to.** The project's current model backs VRAM with
host RAM and executes push-buffer work in the host (see the parent handoff).

## 5. Channels, GPFIFO, USERD and doorbells

A CUDA channel is a GPFIFO (array of pushbuffer descriptors) plus one or more
engine objects:

* GPFIFO memory: allocated by CUDA (vidmem or sysmem), passed to the kernel
  driver at channel creation with its offset and entry count, together with
  an error notifier object.
* **USERD**: on Volta and later, CUDA allocates the user-mode doorbell/control
  area itself and passes its handles to the channel allocation; older
  generations map the kernel-provided USERD. On Hopper, a second BAR1-mapped
  usermode region is used for doorbells.
* **Doorbell**: submission writes the GPFIFO put pointer into USERD and rings
  a doorbell token (obtained via `NVC36F_CTRL_CMD_GPFIFO_GET_WORK_SUBMIT_TOKEN`
  on newer ABIs, or a static token on older ones) at a token/offset the
  channel allocation returned.
* **Pushbuffers**: command streams in vidmem or sysmem; a push is a sequence
  of methods (`SET_OBJECT`, engine methods, semaphore releases) terminated by
  a tracking-semaphore release and a new GPFIFO entry.
* **Context share (Volta+)**: subcontexts are created by allocating a
  `FERMI_CONTEXT_SHARE_A` object with the VA space handle and subcontext id;
  channels then reference it instead of the VA space directly.

Open-IOV relevance: the host emulation sees the GPFIFO and pushbuffer memory
the guest writes. The project already decodes push buffers as method streams
and executes memory ops (`kernel/gsp_sema_service.c`); this section defines
the contract those streams follow.

## 6. Synchronization objects below the API

* **Semaphores** — plain 32/64-bit memory values, allocated by CUDA in
  GPU-visible memory. The GPU releases them (semaphore release methods or
  inline memcpy of the payload), the CPU polls or waits. Tracking semaphores
  count completed pushes; QMD semaphores count completed grids.
* **Notifiers** — 4 KB error-notifier memory per channel; the kernel driver
  writes a notification record on channel errors, which CUDA reads when
  synchronizing to map to `CUDA_ERROR_LAUNCH_*`.
* **OS events** — kernel objects (`NV_ESC_ALLOC_OS_EVENT`) used to wake
  blocked threads; GPU events bind an engine event to an OS event so an
  interrupt can signal user space.
* **Interrupts** — the kernel driver owns the GPU interrupt; CUDA's ISR
  thread is woken through the OS event/monitored-fence mechanism and then
  walks channel state, updates blocking-sync accounting and runs DPCs.
* **Trap handler** — for debuggers and `device-side assert`, CUDA installs a
  trap handler (embedded device code) and reads a trace buffer in GPU memory.

## 7. UVM interface

When UVM is available, the userspace driver calls the UVM API (through
`/dev/nvidia-uvm`):

* set the expected kernel driver version (forward compatibility);
* initialize UVM (sharing mode for MPS, HMM/pageable-memory options);
* register each GPU and each VA space;
* register each channel with its platform parameters (the RM handles and
  context-buffer VA/size the kernel driver provided);
* allocate/register managed allocations, prefetch/advice calls;
* semaphores: UVM semaphore acquire/release are pushed into the same streams.

If UVM is unavailable or the vGPU profile lacks UVM/fault capability, CUDA
disables kernel-mode UVM and degrades to zero-copy behavior.

## 8. GSP-RM in the picture

Modern drivers run the resource manager on the GSP (a RISC-V core on the
GPU). From the userspace driver's perspective this changes nothing: it still
issues RM ioctls. Inside the kernel driver, controls become GSP-RM RPCs; in
the Open-IOV emulation, the host answers those RPCs. The host log for the
`cuCtxCreate` window shows, in order:

```
CE_GET_FAULT_METHOD_BUFFER_SIZE -> 12288
GPU_PROMOTE_CTX (0x2080012b, 560 B) -> generic SUCCESS
TURING_CHANNEL_GPFIFO_A alloc, TURING_COMPUTE_A alloc
```

The userspace failure now surfaces after the engine registration attempt: with
the alloc reply fixed (`docs/05` §3.5) the walker reads a zeroed secondary
engine record and `cuCtxCreate` returns `rc=999` (`docs/05` §3.6). The
function-level path is in `docs/05`; the emulation-side question is what input
builds the driver's class-filtered engine list.

## 9. Emulation checklist (derived from the above)

For an unmodified CUDA 12.x userspace driver in a guest to reach a working
context, the host emulation must provide, at minimum:

1. **Control device semantics** — version negotiation, client creation, GPU
   attach, OS events, per-GPU fds.
2. **Object allocation** for the classes in §2, with plausible sizes/handles,
   and successful `NV_ESC_RM_FREE`.
3. **Device/engine topology** — a consistent engine list and class list
   (`GPU_GET_ENGINES`, `GPU_GET_CLASSLIST`, static info) matching what the
   caps/controls claim; per-engine records must be self-consistent (see the
   engine-descriptor discussion in `docs/05`).
4. **Memory** — VA spaces, vidmem/sysmem allocations with the requested
   attributes, device and host mappings, BAR1 info.
5. **Channels** — GPFIFO alloc with notifier and USERD handles, engine object
   alloc, scheduling, channel ids, doorbell tokens/offsets.
6. **Interrupt/event plumbing** — interrupt tree or usermode region, OS
   events, error notifiers.
7. **Reply fidelity for consumed controls** — the small set of controls whose
   *values* (not just status) the driver reads: memory sizes, engine ids,
   class engine ids, BAR1 info, channel ids, version info, UVM caps.
8. **Push-buffer execution** — the GPU work encoded by userspace must actually
   happen (memsets, copies, semaphore releases, kernel grids) or the driver
   will hang in synchronization rather than fail fast.