diff options
Diffstat (limited to 'docs/03-architecture')
| -rw-r--r-- | docs/03-architecture/01-system-overview.md | 183 | ||||
| -rw-r--r-- | docs/03-architecture/02-guest-stack-and-cudasw.md | 139 | ||||
| -rw-r--r-- | docs/03-architecture/03-gsp-and-gsp-rm.md | 204 | ||||
| -rw-r--r-- | docs/03-architecture/04-uvm.md | 172 | ||||
| -rw-r--r-- | docs/03-architecture/05-gmmu-amap-hal-gr.md | 198 | ||||
| -rw-r--r-- | docs/03-architecture/06-host-module-and-hybrid-model.md | 227 | ||||
| -rw-r--r-- | docs/03-architecture/07-driver-lifecycle-field-guide.md | 188 |
7 files changed, 1311 insertions, 0 deletions
diff --git a/docs/03-architecture/01-system-overview.md b/docs/03-architecture/01-system-overview.md new file mode 100644 index 0000000..ed9b24b --- /dev/null +++ b/docs/03-architecture/01-system-overview.md @@ -0,0 +1,183 @@ +# System Overview + +> The one document to read if you want to understand *how Open-IOV works* before +> diving into any subsystem. + +## 1. The core architectural principle + +> **Emulate the interface. Pass through the work.** + +A virtual GPU is a *mediated passthrough* device. The guest must believe it +owns a real NVIDIA GPU with a real GPU System Processor (GSP) running NVIDIA's +firmware. The host must arrange for the guest's work to execute on real +hardware, because emulating compute in software is 100–1000× too slow to be +useful. + +Everything therefore falls into two buckets: + +``` +┌─────────────────────────────────────────────────────────────────────────┐ +│ EMULATE (software; guest expects virtual hardware) │ +│ • PCI configuration space, capability structures, power management │ +│ • BAR0 register interface (shadow, not the physical GPU's BAR0) │ +│ • BAR1 VRAM aperture (guest-visible video memory) │ +│ • Instance/PRAMIN window │ +│ • GSP firmware boot handshake (FWSEC → booter → RISCV → boot complete)│ +│ • GSP-RM message queues, RPC transport, resource-manager controls │ +│ • Event/notification channels │ +│ • MSI-X injection (via eventfd, from the hypervisor's perspective) │ +└─────────────────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────────────────┐ +│ PASS THROUGH (real hardware; for correctness and performance) │ +│ • GPU memory: guest VRAM objects backed by real VRAM (goal) │ +│ • Page tables / GMMU state used by the GPU (goal) │ +│ • Channel execution: the guest's push buffers run on real channels │ +│ • Compute (CUDA) and, later, 3D/video work │ +│ • DMA, completion and fault interrupts relayed back to the guest │ +└─────────────────────────────────────────────────────────────────────────┘ +``` + +The line between the two is the **interface boundary**, and it moves as the +project matures: early work emulates more (everything the guest can observe), +later work forwards more (everything that must be fast). + +## 2. Layer map + +``` +GUEST + application ── CUDA runtime (libcudart) ─┐ + application ── CUDA driver (libcuda) ─┤ userspace: CUDASW stack + OpenGL/Vulkan ── libGLX/libvulkan ──────┘ + │ ioctls / mmaps + nvidia.ko (unmodified) ── RM core, resource server, UVM, modeset/drm + │ MMIO + DMA + MSI-X (a virtual PCIe device) +════════╪════════════════════════════════════════════════════════════════ +HYPERVISOR (KVM/Xen/...) VFIO mdev transport + IOMMU isolation + │ +HOST MODULE open-iov-vfio.ko + ├─ virtual device: config space, BAR0/BAR1/BAR3 windows, MSI-X + ├─ emulation core: GSP boot + GSP-RM (RPC msgq, controls, objects, events) + ├─ memory model: guest RAM access, VA↔PA maps, VRAM backing + └─ work path: channel/push forwarding to the physical GPU [in progress] + │ +HOST DRIVER nvidia.ko (unmodified) and/or direct GPU interfaces + │ +PHYSICAL GPU +``` + +## 3. Host module architecture + +`open-iov-vfio.ko` is a **VFIO mediated-device (mdev) driver**. It registers a +parent on the physical GPU and exposes virtual GPU types; each `mdev` instance +is one virtual GPU for one VM. + +| Subsystem | Files (illustrative) | Responsibility | +|---|---|---| +| mdev/VFIO registration | `kernel/main.c` | parent/type registration, VFIO device, region and IRQ setup | +| Config space | `kernel/main.c`, `kernel/bar_init.c` | identify/command/status, BARs, PCIe caps, MSI-X caps, VF windows | +| BAR0 shadow | `kernel/bar_emulation.c` | 16 MB emulated register space; register reads/writes the guest driver depends on | +| VRAM aperture (BAR1) | `kernel/main.c`, `memory.c` | 256 MB guest-visible video memory (host RAM today; real VRAM later) | +| Instance memory (BAR3) | `kernel/bar_emulation.c` | instance/PRAMIN window behaviour | +| GSP emulation thread | `kernel/gsp_emulation.c` | boot stage completion, doorbell polling, MSI-X injection | +| GSP-RM emulation | `kernel/gsp_rm_emulation.c` | message queues, RPC dispatch, resource-manager controls, objects | +| RPC handlers (firmware-derived) | `kernel/gsp_rpc_client.c` | behaviours decompiled from the GSP firmware | +| Firmware analysis (runtime) | `kernel/firmware_analyzer.c` | CRC/SHA signatures, pattern matching, version identification | +| Version dispatch | `kernel/dynamic_rpc_dispatcher.c`, `kernel/version/` | version-aware handler selection | +| Hardware abstraction | `kernel/architecture/` | per-architecture constants and behaviours | +| Userspace library | `userspace/lib/libvgpu.c` | management API (`vgpu_*`), diagnostics | + +**Data flow for an MMIO access from the guest:** + +``` +guest nvidia.ko writes BAR0 offset X + → KVM exit → QEMU → VFIO region write into the mdev + → nv_vgpu_vfio_dev_write() (or directly into the BAR0 shadow if mmap'd) + → BAR0 shadow state machine (doorbells, VF registers, notify) + → may wake the GSP emulation thread +``` + +**Data flow for an RPC from the guest to "firmware":** + +``` +guest writes a command-queue element into a guest-memory msgq and rings the +GSP doorbell (BAR0) + → emulation thread reads the queue through the hypervisor's guest-RAM + accessor (vfio_dma_rw), parses the element, dispatches the RPC + → handler produces a reply element in the status queue + → emulation injects an MSI-X interrupt to the guest +``` + +## 4. Guest stack summary + +* `nvidia.ko` is the **unmodified** NVIDIA kernel driver. It contains the + resource manager (RM) core, the resource server (`resserv`), the memory + manager, the FIFO/runlist code, the GPU System Processor client path, and + UVM (the unified memory/virtual-memory module, `nvidia_uvm.ko`). +* User space consists of the CUDA driver (`libcuda.so`), the CUDA runtime + (`libcudart.so`), plus graphics/video libraries. The family of NVIDIA + *software* components that mediate between applications and the driver is + referred to in this project as **CUDASW**. +* The guest expects to program a real GPU: it writes registers, allocates + objects, creates channels, submits push buffers and waits on semaphores — + exactly as on bare metal. + +Detailed subsystem documents: + +* [`02-guest-stack-and-cudasw.md`](02-guest-stack-and-cudasw.md) +* [`03-gsp-and-gsp-rm.md`](03-gsp-and-gsp-rm.md) +* [`04-uvm.md`](04-uvm.md) +* [`05-gmmu-amap-hal-gr.md`](05-gmmu-amap-hal-gr.md) +* [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) +* [`07-driver-lifecycle-field-guide.md`](07-driver-lifecycle-field-guide.md) + +## 5. Why a GSP emulation is possible at all + +The guest driver never talks to GPU silicon directly for management: since the +Turing generation, NVIDIA GPUs boot a **RISC-V firmware (the GSP)** that hosts +the Resource Manager. The driver's normal path is to talk to that firmware over +shared memory queues using a small RPC protocol. + +That protocol is: + +* **small** (a few dozen message types at the levels we need), +* **stable** (message layouts are versioned and compiled into the driver), and +* **observable** (everything arrives in guest memory we can read). + +So instead of emulating a GPU, we emulate **the firmware's interface**. The +guest driver is satisfied because the protocol is satisfied; the guest's +*work* is then given to real hardware. + +The firmware itself we study directly: it is a 28 MB RV64IMAC ELF that we have +decompiled and mapped (see `wiki/GSP-570.144W-Firmware-Analysis.md` and +`wiki/Address-Reference.md`). + +## 6. What runs today + +1. Guest boots, enumerates the virtual PCIe device, binds `nvidia.ko`. +2. The driver performs the GSP boot handshake with our emulation and receives + plausible firmware identity/configuration. +3. The driver sets up the RPC message queues (guest sysmem), negotiates + versions, and performs the resource-manager initialization exchanges. +4. Object creation (clients, devices, subdevices), capability queries, + vaspaces, channels, CE caps, GMMU static info — every control the driver + probes — is answered with the correct shape and values. +5. `nvidia-smi` works; CUDA user-space starts; UVM begins channel-manager and + page-tree initialization. + +## 7. What is next (and why) + +CUDA kernel execution requires the guest's *work* to run on the physical GPU. +That means the host module must: + +1. know the memory the guest's GPU work refers to (the VA↔PA model), +2. reproduce or forward the channel/page-table state that the guest expects the + GSP to own, and +3. submit the guest's compute work to real engine(s) and relay completions, + faults and semaphores back. + +The design for this, including what is known and what remains to be learned, is +in: +[`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) +and +[`../04-implementation/02-practical-challenges-and-adversarial-interop.md`](../04-implementation/02-practical-challenges-and-adversarial-interop.md). diff --git a/docs/03-architecture/02-guest-stack-and-cudasw.md b/docs/03-architecture/02-guest-stack-and-cudasw.md new file mode 100644 index 0000000..4ec614b --- /dev/null +++ b/docs/03-architecture/02-guest-stack-and-cudasw.md @@ -0,0 +1,139 @@ +# The Guest Stack and CUDASW + +> What runs inside the guest, how it is layered, and what it expects from the +> device. This is the "what must keep working, unmodified" document. + +## 1. Vocabulary: what "CUDASW" means here + +**CUDASW** is the family of NVIDIA *software* components that sit between an +application and the GPU driver. In this project the term is used deliberately +broadly, because our compatibility target is the whole stack, not one library: + +``` +application + │ CUDA API (runtime) +libcudart.so ──────────────────────────────┐ CUDA runtime: device/stream/ + │ CUDA driver API │ event management, kernel launch +libcuda.so ────────────────────────────────┤ CUDA driver: contexts, modules, + │ ioctls │ memory, channel submission +/dev/nvidia* ──────────────────────────────┘ +nvidia.ko (RM + UVM + modeset) +``` + +Adjacent components in the same family: + +* **libnvidia-ptxjitcompiler** — JIT compilation of PTX to SASS. (No GPU + interaction beyond driver interfaces; usually "just works" once the driver + initializes.) +* **libnvidia-fatbinaryloader / nvrtc** — fatbinary loading. +* **UVM** (`nvidia_uvm.ko`) — unified memory and GPU page-fault-driven memory + management; see [`04-uvm.md`](04-uvm.md). +* **Graphics stack** — `libGLX_nvidia`, `libnvidia-glcore`, `libvulkan` ICD, + plus the kernel-side `nvidia_modeset`/`nvidia_drm` (roadmap). +* **Video stack** — NVENC/NVDEC userspace + kernel interfaces (roadmap). + +## 2. Layering rules the emulation must respect + +1. **The kernel driver is the contract.** Everything above it (libcuda, + libcudart, libnvidia-*) talks to `nvidia.ko` through stable ioctls, memory + mappings and `/dev/nvidia*` nodes. If the kernel driver initializes and + behaves, user space follows. +2. **User space is opaque to us.** We never intercept user-space APIs. All our + hooks are in the device model (host side) or in the *hardware interface* the + kernel driver sees (guest side). +3. **Version skew is normal.** The driver version, the CUDA version and the + firmware version are three independent axes. Our compatibility target is + "the driver family that matches the emulated firmware profile", with + configuration-driven adaptation for others. + +## 3. Kernel driver anatomy (what the guest brings) + +`nvidia.ko` is a large, self-contained stack. The parts that matter to +virtualization: + +| Component | Role | Why it matters to us | +|---|---|---| +| **RM core** | object model, handles, locks, device lifecycle | drives the GSP boot and the entire RPC/control surface | +| **Resource server (`resserv`)** | allocates and tracks resources (objects, memory, channels) | the client-side half of the client/server RM split (see the GSP-RM doc) | +| **Memory manager (`memmgr`)** | VRAM/sysmem objects, mappings, page tables | the guest expects the GSP to own page tables for its VA spaces | +| **FIFO/runlist** | channels, TSGs, GPFIFO, USERD, doorbells | where the guest's work is submitted | +| **GSP client** | message queues + RPC transport + event handling | the interface we emulate | +| **UVM** | unified memory, page-fault handling, channel manager | starts early in CUDA initialization; extremely stateful | +| **`nvidia_modeset` / `nvidia_drm`** | display/DRM | needed later for graphics | +| **CE / GR HALs** | engine programming | needed to *execute* work (forwarded) | + +## 4. Driver initialization vs. runtime + +Understanding the split between these two phases is the key to debugging: + +**Initialization (boot/probe)** — deterministic, mostly one-shot: + +``` +PCI probe → identify GPU from config space and BAR0 registers + → boot the GSP (our emulated handshake) + → read static configuration (device info tables, FB/BUS info, VBIOS, clocks) + → create root client/device/subdevice objects + → initialize RM subsystems (interrupts, FIFO, memory, GMMU, PMU) + → expose /dev/nvidia* and register with the kernel +``` + +**Runtime (per-process, per-context)** — driven by user-space workloads: + +``` +open /dev/nvidia* → cuInit → UVM registration (per process) + → cuCtxCreate → object/allocations → memory maps + → channel creation + push buffers + semaphores + → kernel launches (GR) / copies (CE) / memops + → faults, completions, events +``` + +The initialization path is where compatibility is won or lost; the runtime +path is where *execution* is won or lost. Open-IOV's current focus is the +runtime path (forwarding), having already reached the end of initialization. + +## 5. What the guest expects from the device, per phase + +| Phase | Guest action | Device/firmware expectation | +|---|---|---| +| Probe | config space reads, BAR sizing, MSI-X capability | a plausible TU104-family PCIe GPU | +| Boot | BAR0 register writes, WPR setup, firmware boot sequence | GSP boots, booter completes, "GSP init done" | +| Config | RPC `GET_GSP_STATIC_INFO`, device tables, VBIOS reads | consistent, self-describing tables | +| RM init | `GSP_RM_CONTROL` battery (FB/BUS/CE/INTR/GMMU/...) | correct shapes and plausible values | +| Objects | `GSP_RM_ALLOC` (root, device, subdevice, vaspace, channel, CE objects) | success + correct handles/params | +| Runtime | `GSP_RM_ALLOC`/`GSP_RM_CONTROL` for maps, channels; pushes | executed work, semaphores, faults | + +The field guide [`07-driver-lifecycle-field-guide.md`](07-driver-lifecycle-field-guide.md) +lists the actual message/control IDs observed at each step. + +## 6. Why user-space-only interception does not work + +Two hard-won facts from this project: + +1. **`libcuda` issues raw `syscall` instructions**, so an `LD_PRELOAD` shim + around `ioctl()` sees nothing. If you need to observe CUDA's kernel calls, + use a `ptrace`-based tracer (PTRACE_SYSCALL + PTRACE_GET_SYSCALL_INFO), as + our tools do. +2. **The CUDA driver talks to the *kernel driver*, not to the device.** All + device interaction happens in `nvidia.ko`. This is good news: it means the + emulation boundary is a small number of kernel-side interfaces (MMIO, DMA, + interrupts, and the GSP protocol) rather than a large user-space API + surface. + +## 7. Windows guests + +Windows brings its own `nvlddmkm.sys` driver, which uses the same underlying +device/firmware model (GSP) but different host interfaces (WDDM, MSI-X +semantics, ROM/VBIOS quirks). The emulation target is the same: satisfy the +firmware interface, forward the work. Windows support is a roadmap item (see +the portability matrix), and the design goal is that no Windows-specific code +lives in the guest. + +## 8. Compatibility notes for versions + +* The guest's driver and the emulated firmware profile must agree on the RPC + message versions. The guest negotiates these at start-up + (`SET_GUEST_SYSTEM_INFO` carries the VGX/RPC version); our emulation answers + with a profile that matches the driver family in use. +* CUDA versions ride on driver versions; the test matrix (see + [`../05-operations/01-performance-correctness-security.md`](../05-operations/01-performance-correctness-security.md)) + tracks (driver, CUDA, firmware) triples. diff --git a/docs/03-architecture/03-gsp-and-gsp-rm.md b/docs/03-architecture/03-gsp-and-gsp-rm.md new file mode 100644 index 0000000..20d335b --- /dev/null +++ b/docs/03-architecture/03-gsp-and-gsp-rm.md @@ -0,0 +1,204 @@ +# GSP and GSP-RM + +> The GPU System Processor hosts NVIDIA's Resource Manager in firmware. Since +> Turing, the guest driver's management path is *talking to that firmware*, not +> to silicon. This document explains that architecture and the interface we +> emulate. + +## 1. What the GSP is + +The **GSP** is a small embedded RISC-V processor (RV64IMAC) inside NVIDIA GPUs. +It runs a firmware image (for the 570.144 generation, a ~28 MB ELF) that +contains, among other things, a full port of NVIDIA's **Resource Manager (RM)** +— the component that owns GPU initialization, memory, engines, interrupts and +scheduling. + +Why this matters: a GPU driver as complex as NVIDIA's is not written to +program silicon directly. It is written to talk to an **RM**, and on modern +GPUs a *copy* of the RM runs on the GSP. The driver is then an **RM client**: +it issues requests, receives callbacks, and observes state through a +well-defined protocol. + +``` + ┌──────────────────────────── GUEST ────────────────────────────┐ + │ nvidia.ko │ + │ RM client path ── RPC ──┐ │ + │ GSP boot code │ │ + └────────────────────────────┼──────────────────────────────────┘ + │ shared memory queues + doorbells + ┌────────────────────────────┼──────────────────────────────────┐ + │ GSP (firmware, RISC-V) │ │ + │ RM core (RMCore) ◄─────┘ │ + │ RM API (object model, clients, handles) │ + │ Resource server (resserv): allocation & lifetime │ + │ Memory manager, GMMU, FIFO/runlist, interrupts, PMU │ + └───────────────────────────────────────────────────────────────┘ +``` + +## 2. GSP boot: the handshake we emulate + +The guest driver boots the GSP exactly as it would on bare metal: + +1. **WPR (Write-Protected Region) setup** — carve a protected region of VRAM + for firmware, write descriptors into BAR0 registers. +2. **FWSEC** — the driver verifies firmware signatures (our emulation answers + the microcode/falcon handshake plausibly). +3. **Booter** — the boot ROM/booter is released and consumes the firmware + image (we emulate completion). +4. **RISCV start** — the RISC-V core starts executing the firmware; on real + hardware the firmware then initializes. +5. **GSP init done** — the firmware signals readiness through a status mailbox; + the driver proceeds to initialization RPCs. + +Our emulation implements stage completion rather than the stages themselves — +which is legitimate because the *only observable* to the driver is the state +machine it polls. + +## 3. The transport: message queues and RPC + +GSP-RM's transport is a shared-memory message queue pair: + +``` + guest system memory (allocated by the guest driver, registered with us) + ┌───────────────────────────────┐ + │ command queue (guest→"GSP") │ 512 KB element ring, 4 KB elements + ├───────────────────────────────┤ + │ status queue ("GSP"→guest) │ 512 KB element ring + └───────────────────────────────┘ + + a page table describing the queue pages (the buffer is + NV_MEMORY_NONCONTIGUOUS: each 4 KB page is an independent allocation) +``` + +Each element begins with a transport header (`authTag`, `aad`, checksum, +sequence, element count) followed by an **RPC message**: + +``` +struct rpc_message_header_v { + u32 header_version; u32 signature; u32 length; u32 function; + u32 rpc_result; u32 rpc_result_private; u32 sequence; + union { u32 spare; u32 cpuRmGfid; }; +}; // 32 bytes; payload follows in a union of message structs +``` + +Observations worth knowing (all verified on live hardware): + +* `length` is `sizeof(header) + sizeof(fixed message struct)` — for messages + with variable payloads (e.g. `GSP_RM_ALLOC` with parameters) it does **not** + include the parameters. Do not use it to bound a payload. +* Function numbers are a flat enum shared by several protocols + (GSP-client RPCs, vGPU plugin RPCs). Examples observed from a stock 570.144 + guest: `SET_GUEST_SYSTEM_INFO` (1), `FREE` (10), `UNLOADING_GUEST_DRIVER` + (47), `GET_GSP_STATIC_INFO` (65), `UPDATE_BAR_PDE` (70), + `CONTINUATION_RECORD` (71), `GSP_SET_SYSTEM_INFO` (72), `SET_REGISTRY` (73), + `GSP_RM_CONTROL` (76), `GSP_RM_ALLOC` (103). +* Timing matters: a reply must be matched to the request sequence number; + the driver runs a reply-matching state machine, and a slow reply is + indistinguishable from a hang. + +## 4. RMCore, RM API and the resource server + +Three concepts from the RM show up constantly in traces and in the guest's +driver. Understanding them makes every log readable. + +### 4.1 RMCore + +RMCore is the object system. Everything in the RM is an **object** with: + +* a **class** (a 32-bit class ID, e.g. `0x2080` = `NV20_SUBDEVICE_0`, + `0x906f`-family = channels, `0xc46f` = `TURING_CHANNEL_GPFIFO_A`), +* a **handle** allocated in a per-client handle space (e.g. `0xc1d00002`), +* a **parent** object (forming a tree rooted at a client). + +Traces of `GSP_RM_ALLOC` are therefore traces of object creation, and traces +of `GSP_RM_CONTROL` are method calls on those objects. + +### 4.2 RM API + +The **RM API** is how in-kernel components (and user space via ioctls) allocate +objects and call controls. Two important variants exist in the client: + +* `Alloc` — allocates an object with an implicit handle, +* `AllocWithHandle` — the caller chooses the handle. + +For a GSP client, the *kernel-internal* API table is overridden so that some +calls become RPCs while others are handled locally. Knowing which is which is +the difference between "we see it in the trace" and "we must derive it another +way". Notably, memory objects are allocated **locally by the client RM** (the +pages are guest RAM), while *server-owned* ranges (see §5) are managed by the +GSP. + +### 4.3 Resource server (`resserv`) + +The resource server is the allocation authority: it owns the handle spaces, +enforces privileges, and tracks object lifetimes. Its directives include flags +that matter to us, for example `RMAPI_ALLOC_FLAGS_SERIALIZED` (the parameters +in an RPC are a serialized form that must be deserialized by the receiver) and +`RMAPI_ALLOC_FLAGS_SKIP_RPC` (the allocation is handled locally and never +reaches the server). + +## 5. The client/server split (and why page tables are ours) + +Modern RM has a **client/server** architecture: + +* The **client** (guest `nvidia.ko`) owns per-process state and submits work. +* The **server** (GSP-RM on the GPU, under our emulation) owns GPU-global + state: routing, page tables, scheduling domains, engine configuration. + +For virtualized/partitioned setups, the client tells the server which VA +ranges it wants the *server* to own, and the server manages the deeper page +table levels for those ranges. We receive this contract explicitly in the +`NV90F1_CTRL_CMD_VASPACE_COPY_SERVER_RESERVED_PDES` control, which carries the +**physical addresses of the client's page-table levels** plus the VA range and +page shifts they cover: + +``` +levels[0] phys=0x1297ac000 pageShift=47 (root page directory) +levels[1] phys=0x1297ad000 pageShift=38 +levels[2] phys=0x1297ae000 pageShift=29 +``` + +In a real system, the server then *fills in* the lower levels for +server-owned ranges; the client never writes them. This is a central fact for +the forwarding work: **when the guest's UVM asks the "GPU" to map memory, the +mapping is expected to materialize in page tables that only we can write.** + +## 6. Event channels and notifications + +Beyond request/reply, the guest needs asynchronous events: + +* **Event objects** (`NV01_EVENT_*` classes) are created by the client and + registered with a *notification* configuration: MSI-X vector, or a memory + "semaphore" the GPU can write. +* **Completion events** are delivered when objects change state (channels + finish work, errors occur). +* `cl2080_notification.h` documents the notification layout; the emulation + must deliver exactly the bit/handle/state triple the guest expects, on the + exact vector. + +Our MSI-X path is: emulation thread → `eventfd` → KVM → guest interrupt +vector. The PBA (pending bit array) is emulated so a masked vector's interrupt +is not lost. + +## 7. What we emulate today, precisely + +| Layer | Emulated | Notes | +|---|---|---| +| GSP boot state machine | ✅ | stages complete; the driver's poll loop is satisfied | +| Message queue transport | ✅ | including the non-contiguous page table for the queue buffer | +| RPC request/reply | ✅ | all functions the driver uses during init are answered | +| Static info tables | ✅ | device/FB/BUS/INTR/GMMU tables with plausible TU104 values | +| Controls during init | ✅ | ~160 distinct control IDs observed per CUDA run; the ones needed to proceed carry real values | +| Object creation | ✅ | correct handle/parameter shapes | +| Page tables for server-owned ranges | 🟡 | contract received and recorded; leaf population is the forwarding work | +| Event/notification delivery | 🟡 | MSI-X works; richer event semantics as forwarding lands | +| Firmware *behaviour* (routing, scheduling, real execution) | 🔴 | the forwarding work; see the implementation docs | + +## 8. Where the firmware analysis lives + +* `wiki/GSP-570.144W-Firmware-Analysis.md` — how the 28.5 MB image was + decompiled, RPC dispatchers located, jump tables mapped. +* `wiki/Address-Reference.md` — firmware addresses for structures, functions + and tables. +* `wiki/RPC-Message-Format.md`, `wiki/GSP-RPC-Emulation.md` — protocol details. +* `wiki/WPR-Management.md` — write-protected region semantics during boot. +* `kernel/gsp_rpc_client.c` — the firmware-derived handlers expressed as C. diff --git a/docs/03-architecture/04-uvm.md b/docs/03-architecture/04-uvm.md new file mode 100644 index 0000000..299f46b --- /dev/null +++ b/docs/03-architecture/04-uvm.md @@ -0,0 +1,172 @@ +# UVM — Unified Memory in the Guest + +> UVM (`nvidia_uvm.ko`) is the most demanding consumer of the virtual GPU's +> semantics, and today the best measurement of how faithful our emulation is. +> This document explains what UVM does, what it needs, and what we learned +> from watching it run against our device. + +## 1. What UVM is + +UVM provides unified memory and GPU-driven page-fault handling. In practice it +is a second GPU driver, layered on the RM: + +* **VA spaces** (`uvm_va_space`) — per-process GPU address spaces. +* **Channel managers** — pools of GPU channels (CE, SEC2, GR-as-needed) that + UVM itself uses to execute its own memory-management work. +* **Push buffers / GPFIFO** — UVM builds its own pushes and submits them. +* **Semaphores and trackers** — UVM tracks GPU progress by waiting on + semaphores that the GPU writes when work completes. +* **Page trees** — UVM manages GPU page tables for its VA spaces. +* **Fault handling** — GPU page faults are delivered to UVM (via a fault + buffer and a notification channel), and UVM resolves them by populating page + tables and replaying/cancelling faults. + +For CUDA, UVM is entered early: `cuInit` → driver init → `UVM_REGISTER_GPU`. +If UVM cannot complete GPU registration, CUDA does not start. + +## 2. The registration path (traced end to end) + +``` +libcuda: UVM_REGISTER_GPU ioctl (0x25) + uvm_api_register_gpu + uvm_va_space_register_gpu + uvm_gpu_retain_by_uuid + nvUvmInterfaceRegisterGpu / GetGpuInfo + alloc_parent_gpu / add_gpu + nvUvmInterfaceDeviceCreate (RM object creation) + uvm_hal_init_gpu / get_gpu_caps (CE caps! see below) + uvm_ats_add_gpu / uvm_gpu_init_isr (fault/intr plumbing) + init_channel_manager + create semaphore pool (sysmem) + create channels per CE (each channel: GPFIFO + pushbuffer + semaphores) + channel_init() → first push → semaphore release → wait + uvm_page_tree_init (page tables for the VA space) +``` + +Each stage has a distinct failure signature; our debugging docs map them. + +## 3. What UVM requires from the "GPU" + +| Requirement | Detail | Our status | +|---|---|---| +| GPU identity | must match the uuid from RM | ✅ | +| CE capabilities | per-engine support bits; UVM *requires* `supported && !grce && sysmem` for CPU_TO_GPU/GPU_TO_CPU channels | ✅ (we answer CE_GET_PHYSICAL_CAPS with the correct bits) | +| Channel creation | RM channel objects with GPFIFO/userd/instance parameters | ✅ objects created | +| Pushes executed | UVM's first action on a new channel is an *init push* followed by a semaphore-tracked wait | 🟡 we can complete them (see below) | +| Page trees | UVM initializes page trees with its own pushes; the *server-owned* ranges must be populated by us | 🟡 | +| Fault delivery | fault buffer + notification; replay/cancel semantics | 🔴 | +| Semantics of completions | tracker values must advance exactly | 🟡 | + +## 4. The most important discovery: how UVM waits + +UVM's tracker (`uvm_tracker.c`) is the single most visible symptom of an +unfaithful GPU. It spins, and after 30 seconds it dumps: + +``` +uvm_tracker.c:285 Tracker entry for value 1 (sema VA 0x121000000) + channel ID 1:3 (0x1:0x3) CE 0 +uvm_channel.c:4184 pending push '' started at uvm_channel.c:2535 in channel_init() + releasing value 1 +``` + +What this means, mechanically: + +1. UVM built a push in a push buffer and wrote a **semaphore release** into it: + a method sequence (CE channel class, e.g. `TURING_DMA_COPY_A`) that sets an + address and a payload and then launches a DMA operation that writes the + payload to memory. +2. UVM submitted the push (GPFIFO entry + GPPUT/doorbell) and now waits for the + semaphore payload to reach `value`. +3. On real hardware, the copy engine writes the payload into the semaphore's + memory. On our device, *nobody does* unless we do it. + +The push we found in guest RAM (found by scanning for its distinctive byte +sequence) decodes as: + +``` +20010000 0000c5b5 set-object (TURING_DMA_COPY_A) +2001a000 0000c076 set-object (GP100_UVM_SW) +20038090 00000001 incrementing-method header: count 3, addr 0x240 + + data: upper 32 bits of the semaphore VA +21000000 00000001 lower 32 bits of the VA; payload (1) +200180c0 0000000c LAUNCH_DMA with SEMAPHORE_TYPE=RELEASE_ONE_WORD +``` + +So: **the emulation's job for this push is to write `payload` to the memory at +the semaphore's GPU VA.** Which requires a GPU-VA → guest-PA translation. + +## 5. The memory model UVM assumes (and what we learned) + +* UVM's semaphore pool is **system memory** (`pool->aperture = UVM_APERTURE_SYS`), + one 4 KiB page per channel manager, addressed as 4-byte slots + (`UVM_SEMAPHORE_SIZE = 4`, 1024 slots per page). +* Each semaphore page has a *GPU VA* (chosen by the client's RM) and a *CPU VA* + (a kernel mapping). UVM polls the CPU VA; the GPU writes the GPU VA. +* **Both addresses are recorded in UVM's structures in guest memory.** This is + the key that makes translation possible without any extra help from the + guest (see the host-module doc for the exact structures and offsets). +* The CPU VA is a **direct-map** address, so + `PA = cpu_va - page_offset_base`. Verified live: writing the payload to that + physical address completes the push and UVM advances. +* Values are *per channel* and *monotonic*: a channel's first push releases 1, + the next 2, and so on. UVM has its own sanity check + (`UVM_GPU_SEMAPHORE_MAX_JUMP`) — writing a wrong value is detected and + reported, which is useful for validation. +* The client's **GPU page tables for server-owned ranges are empty chains**: + the client hands us the *physical addresses of its page-table levels* in + `NV90F1_CTRL_CMD_VASPACE_COPY_SERVER_RESERVED_PDES` and expects the server to + own the lower levels. (See the GSP-RM doc, §5.) + +## 6. Page trees and faults + +* `uvm_page_tree_init` initializes the page tables for a VA space using pushes + on UVM's own channels. These pushes are memops / page-table writes; on real + hardware they are executed by the copy engine. +* Fault handling is event-driven: UVM programs a **fault buffer** (a + replayable and a non-replayable buffer, whose sizes come from + `INTERNAL_GMMU_GET_STATIC_INFO` — answered by our emulation) and receives + notifications when the GPU takes a fault. UVM then: + 1. reads the fault entry, + 2. populates the page table (possibly migrating memory), + 3. **replays or cancels the fault** with a `MEM_OP` push. +* For our forwarding design, faults are where "the guest's memory model" and + "the real GPU's memory model" must be reconciled. This is the deepest part + of the remaining work, and is tracked as a known-unknown in + [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md). + +## 7. How to observe UVM + +In the guest: + +```bash +# UVM prints nothing by default. Turn it on: +echo 1 | sudo tee /sys/module/nvidia_uvm/parameters/uvm_debug_prints +# Messages then appear with the "nvidia-uvm:" prefix. +# Other knobs: uvm_release_asserts, uvm_release_asserts_dump_stack, +# uvm_enable_debug_procfs, uvm_debug_enable_push_desc. +``` + +Known-good markers, in order: + +``` +uvm_gpu_retain_by_uuid → ... → channel manager created +→ uvm_page_tree_init pushes +→ (on success) UVM_REGISTER_GPU returns and CUDA proceeds +``` + +Failure signatures we have characterized: + +| Message | Meaning | +|---|---| +| `Failed to initialize the channel manager: NV_ERR_NOT_SUPPORTED` | CE caps were wrong/missing (we fixed this: see the field guide) | +| `stuck waiting` + `Tracker entry ... channel_init()` | the push's semaphore was never written (the emulation must write it) | +| `Assert failed ... unexpected semaphore ... jump from 0x0 to 0x1010101` | we wrote a wrong payload (and UVM caught us — useful!) | + +## 8. Why UVM is the perfect correctness oracle + +UVM exercises in minutes what a driver test suite would test in hours: +memory allocation, page tables, channel setup, push generation, semaphore +semantics, fault handling, error recovery — all with continuous self-checks and +loud diagnostics. When UVM registers cleanly and CUDA workloads pass, the +device is behaving like a real GPU. That is why "UVM advances" is the project's +primary correctness metric on the way to CUDA. diff --git a/docs/03-architecture/05-gmmu-amap-hal-gr.md b/docs/03-architecture/05-gmmu-amap-hal-gr.md new file mode 100644 index 0000000..6ff9257 --- /dev/null +++ b/docs/03-architecture/05-gmmu-amap-hal-gr.md @@ -0,0 +1,198 @@ +# GMMU, AMAPlib, HALs, Runlists, and GR + +> The engine side: how addresses translate, how work is scheduled, and what +> "GR" actually is. Read this when you need to know what the guest's work +> *means*. + +## 1. GMMU — the GPU memory management unit + +The GMMU translates the GPU virtual addresses (VAs) used by engines into +physical addresses (VRAM offsets or system-memory addresses). + +### 1.1 Page-table hierarchy + +NVIDIA's GMMU uses a multi-level page table whose shape depends on the AS +generation. Turing (the reference) uses a four-level hierarchy with entry +coverage: + +| Level | Entries | VA bits consumed | Coverage per entry | +|---|---|---|---| +| Root page directory (PDB) | 4 | top bits | large | +| PDE level 1 | 512 | 9 bits | e.g. 2^38 | +| PDE level 2 | 512 | 9 bits | e.g. 2^29 | +| PTE level | 512+ | 9+ bits | 4 KiB / 64 KiB / 2 MiB pages | + +Verified properties (from the live guest): + +* **PTE encoding: `PA[47:12] | flags`**, where `flags = 0xe` for a valid + system-memory leaf entry in the format the client produces. +* Page sizes in use: 4 KiB (system memory objects) and larger (big pages) for + large allocations. +* **Apertures**: during enumeration the guest uses `ADDR_SYSMEM = 1` and + `ADDR_FBMEM = 2` to describe memory descriptors (e.g. channel instance + memory in VRAM at offset `0xffed000`, method buffers in system memory at a + physical address). The client hands the **physical** address in these + descriptors — a fact we rely on for the memory model. +* The page-table geometry arrives over + `NV90F1_CTRL_CMD_VASPACE_COPY_SERVER_RESERVED_PDES` (level physical + addresses, page shifts, VA range) — see + [`03-gsp-and-gsp-rm.md`](03-gsp-and-gsp-rm.md) §5. + +### 1.2 What "aperture" means for us + +Two address spaces matter: + +* **System memory (sysmem)**: guest RAM. For a virtual device, the "GPU's view" + of guest memory is established by the host (in real vGPU, by the vGPU manager + via the platform IOMMU; here, by our module). +* **Video memory (vidmem / FB)**: the GPU's local memory. In the current + emulation this is a host RAM buffer exposed as the BAR1 aperture; the + forwarding phase replaces it with real GPU memory. + +## 2. AMAPlib and the HALs + +NVIDIA's driver abstracts hardware in layers: + +``` +driver logic (memmgr, FIFO, engines) + │ + ├─ HAL layer: per-generation C functions that program registers and formats + │ e.g. gmmu HAL (page-table formats, invalidations), + │ engine HALs (CE/GR methods), host HAL (doorbells/PUT) + │ + └─ AMAPlib: the "address mapping" library shared by RM and simulations + (DRAM swizzling, bank/channel mapping, physical-to-aperture rules) +``` + +Developers study two component families to understand these layers: + +* **AMAPlib** — the shared "address mapping" component used by the RM and by + hardware simulations: DRAM swizzling, bank/channel mapping, aperture rules. +* **The HALs** — per-generation C code that programs registers and formats + page-table entries and engine commands. + +Our knowledge of them comes from reverse engineering and decompilation of +driver and firmware behaviour, plus measurement of live devices. What is +published here is our observations, numeric IDs and layouts — which is what +another engineer needs. + +Why we care: + +* When we **populate page tables ourselves** (server-owned ranges), we must + produce entries the hardware and the guest both accept — AMAPlib/HAL code is + the specification. +* When we **forward work**, page tables we write must match what real engines + expect, including big-page encodings and invalidation semantics. + +## 3. Channels, GPFIFO, USERD, doorbells + +A **channel** is a hardware execution context. Its components: + +| Piece | Where | Purpose | +|---|---|---| +| **GPFIFO** | GPU memory (sysmem or vidmem) | ring of entries; each points to a push buffer | +| **Push buffer** | GPU memory | the actual method stream the engine executes | +| **USERD** | vidmem (per channel) | channel control block: PUT/GET pointers, doorbell-related state | +| **Instance block** | vidmem or sysmem | the hardware context (engine state, page directory base, etc.) | +| **Runlist / TSG** | scheduled by the host | the list of channels currently eligible to run | + +The submission flow (what a driver does, and what the guest therefore does): + +``` +write methods into a push buffer (CPU, via a mapping) +write a GPFIFO entry pointing at the push (CPU) +update the channel's PUT pointer (CPU → USERD or doorbell register) + ... the scheduler eventually runs the channel; the engine fetches the push + ... on completion the engine writes the GET pointer and/or a semaphore +``` + +Two details that repeatedly matter in practice: + +* **The push buffer format** is a sequence of method/data pairs preceded by + *incrementing-method headers* that encode the method offset (`address >> 2`), + a subchannel and a count. A decoder for this format is the first tool you + need to read a trace (our `scan_seq` module parameter found UVM's pushes by + searching for the decoded data pattern). +* **Engine classes** are the vocabulary of pushes: `TURING_DMA_COPY_A` + (0xc5b5) for copy engines, `TURING_CHANNEL_GPFIFO_A` (0xc46f) for channel + objects, `GP100_UVM_SW` (0xc076) for UVM's software object, etc. + +## 4. Runlist and scheduling + +The GPU schedules channels through a **runlist**: a list of channel +descriptors the scheduler walks, giving each channel a time slice. A **TSG** +(Time-Slice Group) bundles channels that share a scheduling slot — this is the +mechanism for fair sharing and for interleaving contexts. + +For virtualization this is a focal point: + +* In real vGPU, the host vGPU manager owns the runlist and interleaves + guest channels with host channels. +* In our target architecture, *we* own the schedule: guest channels must be + mapped onto real GPU scheduling primitives with correct isolation (a guest + must never observe or influence another guest's channels). +* The guest driver observes scheduling only indirectly: PUT/GET progress, + semaphores, and fault timing. Faithful *semantics* (not exact timing) is the + requirement. + +## 5. GR — the graphics/compute engine + +**GR** is the engine that executes shaders: both graphics pipelines and CUDA +compute kernels. It is the reason a GPU is a GPU. + +What GR needs to run work: + +* **A context** (the "golden context" + per-context state): register state, + memory layout, page directory, shader code. +* **A channel** with a push buffer that issues *method* calls to GR: + loading shader code, binding surfaces, launching grids (`QMD`s in modern + architectures), and signaling completion. +* **Memory** containing the shader binary (SASS), kernel parameters, + descriptors and data. +* **Interrupts/semaphores** to signal completion and errors. + +For CUDA specifically: + +``` +cuLaunchKernel + → CUDA driver builds a GR push (grid config, QMD, parameters) + → submits on the process's GR channel + → GR executes the shader on real hardware + → completion semaphore / event; UVM handles memory effects +``` + +**Consequence for Open-IOV:** CUDA kernel execution is *real hardware +execution*. No emulation of GR can be correct and fast enough; the only viable +path is forwarding the guest's GR work to real hardware (see +[`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) and +the challenges document). This is also why "CUDA works" is a much stronger +statement than "`nvidia-smi` works". + +## 6. Copy engines (CE) and other engines + +* **CE (copy engines)** — DMA between memory regions; also used for UVM's + internal work (page tables, memsets, semaphores). UVM cannot even initialize + without working CE channels, which is why our CE capability reporting was + the first big unblock (`CE_GET_PHYSICAL_CAPS`, `CE_GET_CE_PCE_MASK`). +* **SEC2** — secure copy engine used by UVM for protected operations. +* **NVENC/NVDEC/NVJPG/OFA** — video and image engines; roadmap for graphics + and video phases. + +Engine types are enumerated by the client during init (`FIFO_GET_DEVICE_INFO_TABLE`) +and reported in engine/capability queries. A guest's channel is bound to one +engine type (e.g. `COPY0 = 0x9`, `GR = 0x1`). + +## 7. Practical reading list inside this repository + +| Topic | Where | +|---|---| +| Page-table formats and HAL concepts | this tree's architecture notes plus the decompiled analysis notes | +| Class/method definitions (classes, methods, notifications) | our decompilation notes and reformatted tables in `wiki/` | +| Channel allocation parameter structures | reformatted in the field guide and in the reference wiki pages | +| GMMU controls and static info | `wiki/Address-Reference.md`, `wiki/RPC-Message-Format.md` | +| Notification formats | the reference wiki pages | +| Firmware structure and dispatchers | `wiki/GSP-570.144W-Firmware-Analysis.md` | + +**Rule:** tracked documentation describes components and behaviour, with our +own observations, IDs and layouts. Exact analysis sources live in the +maintainers' study notes. diff --git a/docs/03-architecture/06-host-module-and-hybrid-model.md b/docs/03-architecture/06-host-module-and-hybrid-model.md new file mode 100644 index 0000000..fc4bb23 --- /dev/null +++ b/docs/03-architecture/06-host-module-and-hybrid-model.md @@ -0,0 +1,227 @@ +# The Host Module and the Hybrid Model + +> `open-iov-vfio.ko` in detail: the virtual device, the emulation engine, the +> memory model, and how work will be forwarded. This is the "where the code +> lives" document. + +## 1. Role and position + +``` + guest VM + │ PCIe MMIO / DMA / MSI-X + ┌────┴──────────────────────────────────────────────┐ + │ hypervisor (KVM/QEMU or Xen) │ + │ VFIO mdev transport, IOMMU (per-tenant memory) │ + └────┬──────────────────────────────────────────────┘ + │ + ┌────┴───────────────────────────────────────────────────────────────┐ + │ open-iov-vfio.ko │ + │ virtual device ── emulation core ── memory model ── work path │ + └────┬─────────────────────────────────────┬─────────────────────────┘ + │ host driver interfaces │ (future) vendor-neutral + ┌────┴────────────────────┐ │ + │ unmodified host nvidia.ko│ direct GPU interfaces / VF backends + └──────────────────────────┘ +``` + +The module registers itself as an **mdev parent** on the physical GPU and +exposes one device type per supported profile. Each `mdev` instance is an +independent virtual GPU with its own state, its own BAR windows, its own +interrupts and its own emulation context. + +## 2. Virtual device + +### 2.1 PCI configuration space + +The guest sees a plausible Turing-class GPU: vendor/device IDs, class code, +BARs, capability list (PCIe, MSI-X, power management), and the subsystem +identifiers the driver keys on. Several fields exist only because the driver +reads them for validation: + +* **PCIe link capabilities** — the guest's driver derives bus width/speed and + availability. The emulation must report them; the *hypervisor* can + accidentally rewrite them (see the debugging doc's root-port note). +* **MSI-X table/PBA** — emulated, with per-vector masking and pending bits. +* **VF (virtual function) windows** — registers the SR-IOV-aware driver paths + read during initialization; the emulation fills them with values consistent + with a non-SR-IOV or lightly-SR-IOV configuration as appropriate. + +### 2.2 BAR layout + +| BAR | Size | Backing | Notes | +|---|---|---|---| +| BAR0 | 16 MB | host `vzalloc` shadow | register interface: boot, doorbells, VF registers, PRAMIN window, VBIOS | +| BAR1 | 256 MB | host `vzalloc` vidmem buffer | guest-visible VRAM aperture; replaced by real VRAM backing in the forwarding phase | +| BAR3 | — | instance window | instance/PRAMIN behaviour | + +MMIO reads and writes are served either by the module's VFIO region callbacks +or directly from the shadow when the hypervisor has mapped it into the guest. + +### 2.3 VRAM backing + +Today BAR1 is host RAM. This is deliberate: correctness first, performance and +real VRAM semantics later. Consequences and their mitigations: + +* Guest reads/writes of "VRAM" are cheap and coherent. ✅ +* The real GPU cannot read it. The forwarding phase must either migrate these + objects to real VRAM or expose a DMA-mapped system-memory region to the GPU + (see §5). + +## 3. Emulation core + +### 3.1 GSP emulation thread + +A kernel thread drives the device's "firmware" behaviour: + +1. Watches BAR0 doorbells (and VF registers) for guest actions. +2. Reads message-queue elements from guest memory. +3. Dispatches RPCs to handlers. +4. Writes replies into the status queue. +5. Injects MSI-X interrupts to the guest. + +Guest memory is accessed through the hypervisor's DMA interface +(`vfio_dma_rw()`), which requires calls to originate from a kernel thread +context — a fact the debug tooling codifies. + +### 3.2 RPC handling + +Handlers are organized by function and, inside controls, by control ID. The +implementation philosophy: + +* **Correct shape first.** A reply must have the exact structure the driver + expects, or the driver walks off into uninitialized memory. +* **Plausible values second.** Values must be self-consistent (e.g. FB size, + BAR sizes, engine counts, page sizes) or later code paths compute nonsense. +* **Never echo garbage.** Queries that pass uninitialized input must receive + zeroed/valid output; echoing the request's stack garbage has caused real + failures. + +The handlers currently cover: the boot/static-info exchanges, the device/FB/ +BUS/INTR control tables, CE capability and PCE-mask queries, GMMU static info, +vaspace reserved-PDE contracts, channel/CE object allocation, and the general +"unknown control returns success with zeroed output" fallback. + +### 3.3 Event channels and interrupts + +Guest-visible interrupts (MSI-X) are delivered via `eventfd` to the hypervisor. +The emulation tracks vector masking and pending state so that an interrupt +raised while masked is delivered when unmasked, not lost. + +## 4. Memory model (the heart of the project) + +The guest's GPU work refers to *GPU virtual addresses*. To act on that work the +host module must translate them to memory it can touch. + +### 4.1 What the client gives us + +* **Channel memory descriptors** (`NV_MEMORY_DESC_PARAMS`) with *physical* + addresses: instance memory (VRAM offset), method buffer, userd, ramfc, + error notifiers. These are precise and trustworthy. +* **The page-table contract** for server-owned VA ranges + (`COPY_SERVER_RESERVED_PDES`): page-table level physical addresses, page + shifts, VA range — but with *empty* leaf levels, because the server owns the + leaves. +* **No mapping RPCs for the client's own system-memory objects.** The client + allocates sysmem locally and never tells the server its physical pages. + +### 4.2 What we derive instead + +Because the client does not hand us its sysmem mappings, the module derives +them from the guest's own in-memory bookkeeping: + +1. **VA↔PA pairs live in guest memory.** UVM and the RM keep structures that + record, for every mapped object, both the GPU VA and the kernel CPU VA. + Example (verified): `uvm_rm_mem_t.vas[CPU]` at +0x30 and `vas[GPU]` at + +0x38; channel tracking semaphores at `uvm_channel_t+0x80/0x88`. +2. **CPU VAs are direct-map addresses**, so + `PA = cpu_va - page_offset_base`. The direct-map base can be recovered from + guest memory itself (structures that store their own address), from + `/proc/kallsyms` during development, or from a debug poke session. +3. Therefore a translation table can be built **without any cooperation from + the guest driver** — a crucial property for adversarial interoperability, + and a general technique applicable to future versions. + +### 4.3 Acting on translations + +The module can read and write guest physical memory (via the hypervisor's DMA +interface). That is enough to: + +* complete semaphore releases (write the payload to the mapped slot), +* inspect push buffers and page tables, +* and later, create page tables for server-owned ranges. + +## 5. The forwarding path (design) + +``` +guest push submitted + → module detects submission (doorbell/GPPUT or push-buffer scan) + → decode the push (method stream) + → for each command decide: + (a) EMULATE: pure state/notification (semaphore release, membar, fence) + (b) FORWARD: real engine work (GR grid launch, CE copy, memop) + → for (b): submit equivalent work on a real channel bound to the physical GPU + → relay completions (semaphore/fault/interrupt) back into guest memory +``` + +Three forwarding backends are candidates, in increasing order of capability: + +1. **Host RM/GSP proxy** — use the unmodified host driver's RM to allocate real + memory and create real channels, then replay guest work through them. +2. **Direct engine submission** — build our own channel/push streams to real + engines using the GMMU/HAL definitions, bypassing the host driver. +3. **Hybrid** — proxy for management (memory, channels) and direct submission + for hot-path work (grid launches, copies). + +The decision between them is driven by the practical challenges document; the +short version is that (1) is safest for correctness, (2) is best for latency, +and both must be isolated to be acceptable. + +## 6. Debug tooling in the module + +The module exposes debug parameters (writes to +`/sys/module/open_iov_vfio/parameters/*`) that make the guest's memory and the +emulated device inspectable without rebuilding: + +| Parameter | Purpose | +|---|---| +| `scan_guest_ram=<v>` | find a 64-bit value in guest RAM (0–8 GB), report GPAs | +| `scan_vidmem=<v>` | same for the emulated VRAM shadow | +| `scan_seq=<v1,v2,...>` | find a 32-bit *sequence* (e.g. a method stream) | +| `scan_pt=<pa>` | find page-table entries pointing at a physical address | +| `scan_masked=<v>,<mask>` | find `(x & mask) == v` (e.g. a page-base VA) | +| `dump_vidmem=<off>,<len>` / `summary` | dump/scan the VRAM shadow | +| `poke_guest=<gpa>,<v>` | write a 32-bit value to guest physical memory | +| `poke_fill=<gpa>,<v>,<n>` | fill a range | +| `fill_vidmem=<v>` | fill the VRAM shadow | + +These are development-grade but production-useful: they are how the memory +model was reverse-engineered, and they remain the fastest way to diagnose a +mismatch. Their workflows are documented in +[`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md). + +## 7. Build and symbol contracts + +* `open-iov-vfio.ko` exports the `nvidia_vgpu_vfio_*` interface + (`nvidia_vgpu_vfio_probe/remove`, ops get/set, mdev parent registration) for + hosts that want to pair our device with the NVIDIA OpenRM host modules. +* When NVIDIA's OpenRM modules are built from source, they must be built with + `KBUILD_EXTRA_SYMBOLS=$(pwd)/kernel/Module.symvers`, and **after** the + Open-IOV module, or symbols will not resolve. +* On typical single-GPU developer systems the distribution's NVIDIA driver + manages the physical GPU and the Open-IOV module is used standalone; the + OpenRM submodule path is only needed for the native-vGPU-management + workflows. + +## 8. What is implemented vs. designed + +| Area | Status | +|---|---| +| mdev/VFIO device, config space, BARs, MSI-X | implemented | +| BAR0 register state machine, boot handshake | implemented | +| Message queues + RPC + all init controls observed from stock 570.144 | implemented | +| Guest sysmem access + VA↔PA discovery | implemented (debug-grade, to be productized) | +| Push decoding + semaphore completion | demonstrated live; being productized | +| Page-table population for server-owned ranges | designed, not implemented | +| GPU work forwarding (GR/CE execution) | designed, not implemented | +| Real VRAM backing | designed, not implemented | +| Windows guest, Xen transport, aarch64 hosts | designed, not implemented | diff --git a/docs/03-architecture/07-driver-lifecycle-field-guide.md b/docs/03-architecture/07-driver-lifecycle-field-guide.md new file mode 100644 index 0000000..8d80694 --- /dev/null +++ b/docs/03-architecture/07-driver-lifecycle-field-guide.md @@ -0,0 +1,188 @@ +# Driver Lifecycle Field Guide + +> A walking tour of what actually happens between "guest boots" and "CUDA +> initializes", with the real message and control IDs we observe from an +> unmodified 570.144 guest. Use this to interpret logs and to place new +> observations. + +## 0. How to use this document + +Each stage lists: what the guest does, what the device/firmware must answer, +what our emulation does today, and the log signature to look for +(`dmesg | grep gsp-resman-vfio` on the host; `dmesg` in the guest). + +The IDs are for the 570.144 generation on the Turing (TU104) profile. Other +generations will have additions and version differences; the *stages* are +stable. + +## 1. PCI probe and config space + +**Guest:** reads vendor/device/class IDs, walks capabilities, sizes BARs, +enables Memory Space and Bus Master, programs MSI-X. + +**Device must:** report a coherent TU104-family GPU with 16 MB BAR0, 256 MB +BAR1, an MSI-X capability, and a link capability the driver accepts. + +**Emulation:** config space is synthesized; BAR sizes are enforced; the +*hypervisor* is the usual source of surprises (a device attached to the root +bus can have its PCIe capability rewritten to "root complex integrated +endpoint", which removes link information the driver wants — attach behind a +root port). + +**Log signature:** none from us; guest-side `lspci -vvv` is the reference. + +## 2. GSP boot handshake + +**Guest:** writes WPR descriptors, releases FWSEC, then the booter, then the +RISC-V core; polls a status mailbox for "GSP init done". + +**Device must:** complete every stage in the order the driver polls, and leave +the mailbox in the state the driver expects. + +**Emulation:** `kernel/gsp_emulation.c` models the stage state machine and +auto-clears self-clearing triggers (e.g. MMU invalidate registers) so polls +terminate. + +**Log signature:** +``` +gsp-resman-vfio: GSP-BOOT: ... trigger auto-cleared ... +``` + +## 3. Static configuration + +**Guest:** `GET_GSP_STATIC_INFO` (65) → receives a large `GspStaticConfigInfo` +structure; reads the VBIOS image from the BAR0 window; queries device/FB/BUS +info. + +**Device must:** provide self-consistent sizes, engine inventories, memory +geometry, interrupt topology, and a VBIOS blob the driver can parse. + +**Emulation:** static structures are synthesized for TU104 (FB size, BAR +layout, engine list, vGPU-friendly VF values). + +**Log signature:** +``` +GSPRM: GET_GSP_STATIC_INFO → ... +``` + +## 4. Message queues + +**Guest:** sends `SET_GUEST_SYSTEM_INFO` (1) with libos regions, a queue +buffer description, and a page table (the queue buffer is +`NV_MEMORY_NONCONTIGUOUS`). It also sends `GSP_SET_SYSTEM_INFO` (72), +`SET_REGISTRY` (73) and `CONTINUATION_RECORD` (71) as init traffic. + +**Device must:** consume them silently (async) and start consuming the command +queue exactly in the order the guest writes elements. + +**Emulation:** parses the queue chain, builds the queue geometry, links the +status queue, and consumes elements one by one. **Element-based, not +byte-stream-based**: each message is one element and may span pages. + +**Log signature:** +``` +gsp-resman-vfio: libos region[i] id8=... pa=... size=... +gsp-resman-vfio: queue args sharedPA=... ptes=... cmdOff=... statOff=... +``` + +## 5. Resource-manager initialization (the control battery) + +**Guest:** sends `GSP_RM_CONTROL` (76) for every subsystem it initializes. +Representative controls (all observed, all answered with correct shapes): + +| Control | Meaning | What matters | +|---|---|---| +| `0x20800a40`-family | internal device info table | per-index values, entry count | +| `0x20801112` | FIFO device info table | engine inventory; count must match engine queries | +| `0x20800a5c` | interrupt kernel table | vector topology the driver maps | +| `0x20800a59` | GMMU static info | **fault buffer sizes** (replayable/legacy) — a wrong value breaks UVM early | +| `0x20800a01` | display static info | display count (headless values are fine) | +| `0x20801823/-a/b` | BUS/FB info V2 | sysmem connection type and PCIe link caps feed bandwidth calculations | +| `0x20802a07` | CE physical caps | per-engine support bits; UVM refuses to make channels without `sysmem` support | +| `0x20802a0a/0b` | all-CE caps | caps table for all engines | +| `0x20802a02` | CE PCE mask | one bit per engine | +| `0x20802a08` | CE fault method buffer size | method buffer sizing | +| `0x90f10106` | vaspace reserved-PDE contract | **carries page-table physical addresses** (see GSP-RM doc) | +| `0x20803039` | NVLink masks/VBIOS info | unpopulated but must be shaped | +| `0x20800a70` | BUS flush with sysmembar | UVM issues these frequently; success suffices | + +**Log signature:** +``` +GSPRM: GSP_RM_CONTROL cmd=0x... hClient=0x... hObject=0x... paramsSize=... +``` + +## 6. Object creation + +**Guest:** `GSP_RM_ALLOC` (103) creates the object tree. Typical order for a +CUDA process: + +``` +0x0 NV01_ROOT (client root, 120-byte params) +0x80 NV01_DEVICE_0 (56 bytes) +0x2080 NV20_SUBDEVICE_0 (4 bytes) +0x90f1 FERMI_VASPACE_A (48 bytes) ← the GPU address space +0xa06c KEPLER_CHANNEL_GROUP_A (20 bytes) ← CE channel group +0xc46f TURING_CHANNEL_GPFIFO_A(368 bytes) ← the channel (GPFIFO VA, instance/userd descriptors) +0xc076 GP100_UVM_SW (0 bytes) +0xc5b5 TURING_DMA_COPY_A (8 bytes) ← CE object +``` + +Channel parameters decode into: `gpFifoOffset` (a GPU VA), +`gpFifoEntries`, `engineType` (e.g. COPY0 = 0x9), and +`NV_MEMORY_DESC_PARAMS` for instance/userd/ramfc/method buffers with +**physical** addresses and apertures (`ADDR_SYSMEM = 1`, `ADDR_FBMEM = 2`). + +**Emulation:** handle management and parameter shapes are validated against +the reference structures; values are logged in a form that makes the object +tree reconstructible from `dmesg`. + +**Log signature:** +``` +GSPRM: GSP_RM_ALLOC class=0x... paramsSize=... hClient=... hParent=... hObject=... +``` + +## 7. UVM registration (the wall we climbed) + +**Guest:** `cuInit` → `UVM_REGISTER_GPU` → channel manager creation → +page-tree init. + +**Device must:** let UVM's channels initialize (each channel does a first push +and waits on a semaphore), and let page-tree initialization proceed. + +**Emulation (today):** we can decode the pushes UVM generates and complete +their semaphore releases by translating the semaphore VA to a guest physical +address (see the UVM and host-module docs). This is the live development front. + +**Log signatures (guest side):** +``` +nvidia-uvm: ... Tracker entry for value N (sema VA 0x...) channel ID ... CE ... +nvidia-uvm: ... pending push '' started at ... in channel_init() releasing value N +``` + +## 8. What comes after registration (roadmap) + +``` +UVM registers → cuInit returns → context creation → memory allocation +→ channel creation for the process → kernel launch → (real execution) +``` + +Each step will need the same treatment: satisfy the interface, forward the +work, relay the completions. The known-unknowns document lists what we expect +to be hard. + +## 9. Quick reference: message/control IDs seen from a stock 570.144 guest + +| Function | Name | Frequency per CUDA run (approx.) | +|---|---|---| +| 1 | SET_GUEST_SYSTEM_INFO | once | +| 10 | FREE | ~14 | +| 47 | UNLOADING_GUEST_DRIVER | on unload | +| 65 | GET_GSP_STATIC_INFO | once | +| 70 | UPDATE_BAR_PDE | a few | +| 71/72/73 | CONTINUATION_RECORD / GSP_SET_SYSTEM_INFO / SET_REGISTRY | once each | +| 76 | GSP_RM_CONTROL | ~160 | +| 103 | GSP_RM_ALLOC | ~60 | + +If a new driver version produces **new function numbers**, do not guess: add a +handler that logs the raw payload, then decode it against the decompiled +study notes for that driver version.
\ No newline at end of file |