diff options
Diffstat (limited to 'docs/03-architecture/05-gmmu-amap-hal-gr.md')
| -rw-r--r-- | docs/03-architecture/05-gmmu-amap-hal-gr.md | 198 |
1 files changed, 198 insertions, 0 deletions
diff --git a/docs/03-architecture/05-gmmu-amap-hal-gr.md b/docs/03-architecture/05-gmmu-amap-hal-gr.md new file mode 100644 index 0000000..6ff9257 --- /dev/null +++ b/docs/03-architecture/05-gmmu-amap-hal-gr.md @@ -0,0 +1,198 @@ +# GMMU, AMAPlib, HALs, Runlists, and GR + +> The engine side: how addresses translate, how work is scheduled, and what +> "GR" actually is. Read this when you need to know what the guest's work +> *means*. + +## 1. GMMU — the GPU memory management unit + +The GMMU translates the GPU virtual addresses (VAs) used by engines into +physical addresses (VRAM offsets or system-memory addresses). + +### 1.1 Page-table hierarchy + +NVIDIA's GMMU uses a multi-level page table whose shape depends on the AS +generation. Turing (the reference) uses a four-level hierarchy with entry +coverage: + +| Level | Entries | VA bits consumed | Coverage per entry | +|---|---|---|---| +| Root page directory (PDB) | 4 | top bits | large | +| PDE level 1 | 512 | 9 bits | e.g. 2^38 | +| PDE level 2 | 512 | 9 bits | e.g. 2^29 | +| PTE level | 512+ | 9+ bits | 4 KiB / 64 KiB / 2 MiB pages | + +Verified properties (from the live guest): + +* **PTE encoding: `PA[47:12] | flags`**, where `flags = 0xe` for a valid + system-memory leaf entry in the format the client produces. +* Page sizes in use: 4 KiB (system memory objects) and larger (big pages) for + large allocations. +* **Apertures**: during enumeration the guest uses `ADDR_SYSMEM = 1` and + `ADDR_FBMEM = 2` to describe memory descriptors (e.g. channel instance + memory in VRAM at offset `0xffed000`, method buffers in system memory at a + physical address). The client hands the **physical** address in these + descriptors — a fact we rely on for the memory model. +* The page-table geometry arrives over + `NV90F1_CTRL_CMD_VASPACE_COPY_SERVER_RESERVED_PDES` (level physical + addresses, page shifts, VA range) — see + [`03-gsp-and-gsp-rm.md`](03-gsp-and-gsp-rm.md) §5. + +### 1.2 What "aperture" means for us + +Two address spaces matter: + +* **System memory (sysmem)**: guest RAM. For a virtual device, the "GPU's view" + of guest memory is established by the host (in real vGPU, by the vGPU manager + via the platform IOMMU; here, by our module). +* **Video memory (vidmem / FB)**: the GPU's local memory. In the current + emulation this is a host RAM buffer exposed as the BAR1 aperture; the + forwarding phase replaces it with real GPU memory. + +## 2. AMAPlib and the HALs + +NVIDIA's driver abstracts hardware in layers: + +``` +driver logic (memmgr, FIFO, engines) + │ + ├─ HAL layer: per-generation C functions that program registers and formats + │ e.g. gmmu HAL (page-table formats, invalidations), + │ engine HALs (CE/GR methods), host HAL (doorbells/PUT) + │ + └─ AMAPlib: the "address mapping" library shared by RM and simulations + (DRAM swizzling, bank/channel mapping, physical-to-aperture rules) +``` + +Developers study two component families to understand these layers: + +* **AMAPlib** — the shared "address mapping" component used by the RM and by + hardware simulations: DRAM swizzling, bank/channel mapping, aperture rules. +* **The HALs** — per-generation C code that programs registers and formats + page-table entries and engine commands. + +Our knowledge of them comes from reverse engineering and decompilation of +driver and firmware behaviour, plus measurement of live devices. What is +published here is our observations, numeric IDs and layouts — which is what +another engineer needs. + +Why we care: + +* When we **populate page tables ourselves** (server-owned ranges), we must + produce entries the hardware and the guest both accept — AMAPlib/HAL code is + the specification. +* When we **forward work**, page tables we write must match what real engines + expect, including big-page encodings and invalidation semantics. + +## 3. Channels, GPFIFO, USERD, doorbells + +A **channel** is a hardware execution context. Its components: + +| Piece | Where | Purpose | +|---|---|---| +| **GPFIFO** | GPU memory (sysmem or vidmem) | ring of entries; each points to a push buffer | +| **Push buffer** | GPU memory | the actual method stream the engine executes | +| **USERD** | vidmem (per channel) | channel control block: PUT/GET pointers, doorbell-related state | +| **Instance block** | vidmem or sysmem | the hardware context (engine state, page directory base, etc.) | +| **Runlist / TSG** | scheduled by the host | the list of channels currently eligible to run | + +The submission flow (what a driver does, and what the guest therefore does): + +``` +write methods into a push buffer (CPU, via a mapping) +write a GPFIFO entry pointing at the push (CPU) +update the channel's PUT pointer (CPU → USERD or doorbell register) + ... the scheduler eventually runs the channel; the engine fetches the push + ... on completion the engine writes the GET pointer and/or a semaphore +``` + +Two details that repeatedly matter in practice: + +* **The push buffer format** is a sequence of method/data pairs preceded by + *incrementing-method headers* that encode the method offset (`address >> 2`), + a subchannel and a count. A decoder for this format is the first tool you + need to read a trace (our `scan_seq` module parameter found UVM's pushes by + searching for the decoded data pattern). +* **Engine classes** are the vocabulary of pushes: `TURING_DMA_COPY_A` + (0xc5b5) for copy engines, `TURING_CHANNEL_GPFIFO_A` (0xc46f) for channel + objects, `GP100_UVM_SW` (0xc076) for UVM's software object, etc. + +## 4. Runlist and scheduling + +The GPU schedules channels through a **runlist**: a list of channel +descriptors the scheduler walks, giving each channel a time slice. A **TSG** +(Time-Slice Group) bundles channels that share a scheduling slot — this is the +mechanism for fair sharing and for interleaving contexts. + +For virtualization this is a focal point: + +* In real vGPU, the host vGPU manager owns the runlist and interleaves + guest channels with host channels. +* In our target architecture, *we* own the schedule: guest channels must be + mapped onto real GPU scheduling primitives with correct isolation (a guest + must never observe or influence another guest's channels). +* The guest driver observes scheduling only indirectly: PUT/GET progress, + semaphores, and fault timing. Faithful *semantics* (not exact timing) is the + requirement. + +## 5. GR — the graphics/compute engine + +**GR** is the engine that executes shaders: both graphics pipelines and CUDA +compute kernels. It is the reason a GPU is a GPU. + +What GR needs to run work: + +* **A context** (the "golden context" + per-context state): register state, + memory layout, page directory, shader code. +* **A channel** with a push buffer that issues *method* calls to GR: + loading shader code, binding surfaces, launching grids (`QMD`s in modern + architectures), and signaling completion. +* **Memory** containing the shader binary (SASS), kernel parameters, + descriptors and data. +* **Interrupts/semaphores** to signal completion and errors. + +For CUDA specifically: + +``` +cuLaunchKernel + → CUDA driver builds a GR push (grid config, QMD, parameters) + → submits on the process's GR channel + → GR executes the shader on real hardware + → completion semaphore / event; UVM handles memory effects +``` + +**Consequence for Open-IOV:** CUDA kernel execution is *real hardware +execution*. No emulation of GR can be correct and fast enough; the only viable +path is forwarding the guest's GR work to real hardware (see +[`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) and +the challenges document). This is also why "CUDA works" is a much stronger +statement than "`nvidia-smi` works". + +## 6. Copy engines (CE) and other engines + +* **CE (copy engines)** — DMA between memory regions; also used for UVM's + internal work (page tables, memsets, semaphores). UVM cannot even initialize + without working CE channels, which is why our CE capability reporting was + the first big unblock (`CE_GET_PHYSICAL_CAPS`, `CE_GET_CE_PCE_MASK`). +* **SEC2** — secure copy engine used by UVM for protected operations. +* **NVENC/NVDEC/NVJPG/OFA** — video and image engines; roadmap for graphics + and video phases. + +Engine types are enumerated by the client during init (`FIFO_GET_DEVICE_INFO_TABLE`) +and reported in engine/capability queries. A guest's channel is bound to one +engine type (e.g. `COPY0 = 0x9`, `GR = 0x1`). + +## 7. Practical reading list inside this repository + +| Topic | Where | +|---|---| +| Page-table formats and HAL concepts | this tree's architecture notes plus the decompiled analysis notes | +| Class/method definitions (classes, methods, notifications) | our decompilation notes and reformatted tables in `wiki/` | +| Channel allocation parameter structures | reformatted in the field guide and in the reference wiki pages | +| GMMU controls and static info | `wiki/Address-Reference.md`, `wiki/RPC-Message-Format.md` | +| Notification formats | the reference wiki pages | +| Firmware structure and dispatchers | `wiki/GSP-570.144W-Firmware-Analysis.md` | + +**Rule:** tracked documentation describes components and behaviour, with our +own observations, IDs and layouts. Exact analysis sources live in the +maintainers' study notes. |