diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/03-architecture/03-gsp-and-gsp-rm.md | 23 | ||||
| -rw-r--r-- | docs/03-architecture/06-host-module-and-hybrid-model.md | 22 | ||||
| -rw-r--r-- | docs/03-architecture/08-physical-gpu-integration.md | 27 | ||||
| -rw-r--r-- | docs/03-architecture/09-full-system-picture.md | 666 | ||||
| -rw-r--r-- | docs/03-architecture/10-interaction-and-scheduling.md | 337 | ||||
| -rw-r--r-- | docs/README.md | 97 |
6 files changed, 1131 insertions, 41 deletions
diff --git a/docs/03-architecture/03-gsp-and-gsp-rm.md b/docs/03-architecture/03-gsp-and-gsp-rm.md index 8809bb7..fe51838 100644 --- a/docs/03-architecture/03-gsp-and-gsp-rm.md +++ b/docs/03-architecture/03-gsp-and-gsp-rm.md @@ -54,6 +54,29 @@ Our emulation implements stage completion rather than the stages themselves — which is legitimate because the *only observable* to the driver is the state machine it polls. +```mermaid +sequenceDiagram + autonumber + participant GD as Guest driver + participant R0 as BAR0 shadow + participant GE as GSP emulation thread + participant MQ as Message queues in guest sysmem + participant GRM as GSP-RM emulation + + GD->>R0: WPR descriptors and boot kicks + GE-->>R0: stage status transitions + GD->>R0: poll for boot complete + R0-->>GD: firmware running + GD->>MQ: set up queues and ring the doorbell + GE->>MQ: read and dispatch elements + GRM-->>MQ: write replies + GE-->>GD: MSI-X completion +``` + +The cross-component detail (doorbells, queues, interrupts, latency classes) is +in [`09-full-system-picture.md`](09-full-system-picture.md) and +[`10-interaction-and-scheduling.md`](10-interaction-and-scheduling.md). + ## 3. The transport: message queues and RPC GSP-RM's transport is a shared-memory message queue pair: diff --git a/docs/03-architecture/06-host-module-and-hybrid-model.md b/docs/03-architecture/06-host-module-and-hybrid-model.md index 61be96c..e29951c 100644 --- a/docs/03-architecture/06-host-module-and-hybrid-model.md +++ b/docs/03-architecture/06-host-module-and-hybrid-model.md @@ -31,6 +31,28 @@ exposes one device type per supported profile. Each `mdev` instance is an independent virtual GPU with its own state, its own BAR windows, its own interrupts and its own emulation context. +```mermaid +sequenceDiagram + autonumber + participant ORCH as Orchestrator + participant SYS as Module and mdev sysfs + participant QEMU as QEMU with vfio-pci + participant KVM as KVM + participant GUEST as Guest + + ORCH->>SYS: create mdev for a VM + QEMU->>SYS: open VFIO device, mmap regions, set IRQ eventfds + QEMU->>KVM: map guest RAM, register irqfd + GUEST->>SYS: MMIO and DMA through the emulated device + SYS-->>GUEST: MSI-X via eventfd to irqfd + ORCH->>SYS: destroy mdev after the VM stops +``` + +The full interaction and scheduling detail is in +[`10-interaction-and-scheduling.md`](10-interaction-and-scheduling.md); the +end-to-end maps are in +[`09-full-system-picture.md`](09-full-system-picture.md). + ## 2. Virtual device ### 2.1 PCI configuration space diff --git a/docs/03-architecture/08-physical-gpu-integration.md b/docs/03-architecture/08-physical-gpu-integration.md index f156429..06df22b 100644 --- a/docs/03-architecture/08-physical-gpu-integration.md +++ b/docs/03-architecture/08-physical-gpu-integration.md @@ -167,6 +167,33 @@ per-context scheduling. Tenant policy and fairness are a later layer on top. device-ownership lifecycle; the emulation path remains the control plane and is reused. +## 9. Interaction sequence (end state) + +```mermaid +sequenceDiagram + autonumber + participant GD as Guest driver + participant EMU as GSP-RM emulation - control + participant BE as Backend - real GSP client + participant GSP as Real GSP firmware + participant ENG as Engines + + GD->>EMU: objects, vaspaces, channel creation + EMU->>BE: translate to real objects [planned] + BE->>GSP: allocate real objects and channels + GD->>EMU: work submissions + EMU->>BE: decoded and translated push streams + BE->>ENG: submit on real channels + ENG-->>BE: completions and faults + BE->>EMU: relay into guest memory + EMU-->>GD: trackers and events advance +``` + +The full interaction and scheduling view (queues, interrupts, timing classes, +multi-tenant policy) is in +[`10-interaction-and-scheduling.md`](10-interaction-and-scheduling.md); the +system map is in [`09-full-system-picture.md`](09-full-system-picture.md). + ## See also - [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) - the host module, memory model, build contracts diff --git a/docs/03-architecture/09-full-system-picture.md b/docs/03-architecture/09-full-system-picture.md new file mode 100644 index 0000000..0df0dd2 --- /dev/null +++ b/docs/03-architecture/09-full-system-picture.md @@ -0,0 +1,666 @@ +# The Full System Picture + +> **What this document is.** The visual, end-to-end picture of Open-IOV: every +> component in the finished system, how they interact, where the work actually +> executes, how tenancy is isolated and scheduled, how the design stays portable +> across driver and hardware versions, and how far each piece is from done. +> The other architecture documents explain *why*; this one shows *what* and +> *where*, all the way down. Diagrams are Mermaid and render on GitHub and in +> most Markdown viewers. + +**Status legend used throughout** (the project is honest about maturity): + +| Tag | Meaning | +|---|---| +| `[done]` | implemented and exercised with a stock guest driver | +| `[wip]` | implemented in part; actively being completed | +| `[planned]` | designed, not implemented | +| hardware/firmware | exists in the physical world; referenced, not built by us | +| vendor | NVIDIA components we interoperate with, unmodified | + +## 1. The end-state system map + +All components of the finished system, with the execution backend this project +has chosen (a module-owned GPU driven through its real GSP firmware; see +[`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)). + +```mermaid +flowchart TB + subgraph VM["Guest VM - stock software, no Open-IOV code"] + APP["Applications<br/>CUDA / OpenGL / Vulkan / video"] + USR["Userspace stack<br/>libcuda, libcudart, libGLX, libvulkan"] + GKO["Stock nvidia.ko<br/>RM core + UVM + modeset/DRM"] + APP --> USR --> GKO + end + + subgraph HV["Hypervisor - KVM and QEMU"] + VFIO["VFIO mdev transport<br/>regions, mmap, eventfd / irqfd"] + IOMMU["IOMMU / IOMMUFD<br/>per-VM DMA address spaces"] + end + + subgraph MOD["open-iov-vfio.ko - the only host component"] + VDEV["Virtual device<br/>config space, BAR0/1/3, MSI-X"] + EMU["Emulation core<br/>GSP boot + GSP-RM RPC + objects"] + MEMM["Memory model<br/>VA to PA, pins, VRAM shadow, page tables"] + WORK["Work path<br/>pump, decode, translate, forward"] + BE["Execution backend<br/>real GSP session + RM client"] + VDEV --> EMU + EMU --> MEMM + EMU --> WORK + MEMM --> WORK + WORK --> BE + BE --> MEMM + end + + subgraph HW["Physical GPU"] + GSP["GSP firmware - the real RM server"] + ENG["Engines<br/>GR, CE pair, DEC, ENC"] + GMMU["GMMU and page tables"] + VRAM["VRAM + framebuffer"] + DB["BAR0 / USERD doorbells, MSI-X"] + end + + subgraph HOSTOS["Host OS"] + ND["Distribution nvidia.ko<br/>unmodified; unbound while the service owns the GPU"] + IGPU["Display on another GPU / integrated graphics"] + end + + GKO -->|"MMIO + DMA + MSI-X"| VFIO + VFIO --> VDEV + BE -->|"msgq + doorbells"| GSP + WORK -->|"real push streams on real channels"| ENG + BE --> GMMU + MEMM --> IOMMU + + classDef done fill:#d5f5d5,stroke:#1e7a1e + classDef wip fill:#fff3cd,stroke:#9a7b1a + classDef plan fill:#f8d7da,stroke:#9a1a1a + classDef hw fill:#e2e8f0,stroke:#4a5568 + classDef vendor fill:#e7e0f5,stroke:#5a4a8a + + class VDEV,EMU done + class MEMM,WORK wip + class BE plan + class IOMMU,VFIO done + class GSP,ENG,GMMU,VRAM,DB hw + class ND,USR,GKO,APP vendor +``` + +**Reading the map.** The guest sees only a PCIe device. Everything between +that device and the physical engines is either emulated (control plane) or +translated (work plane) by one kernel module, with no daemon, no host userspace +and no modified driver on either side. The distribution `nvidia.ko` remains +installed and unmodified on the host; it is simply unbound from the GPU while +the service owns it, and the host's display runs on another GPU. + +## 2. The control plane: GSP emulation + +### 2.1 The GSP boot handshake + +The guest driver believes it boots a real GSP. The module completes the +observable boot stages itself and reports the status the driver polls. In the +end state, the *backend* additionally boots the physical GPU's real GSP for +work execution — the two sessions are independent. + +```mermaid +stateDiagram-v2 + direction LR + [*] --> PCIProbe + PCIProbe --> FWSEC: driver kicks boot + FWSEC --> Booter: FRTS status observed + Booter --> RISCV: WPR2 programmed + RISCV --> BootComplete: falcon reports running + BootComplete --> MsgqInit: queue setup in guest sysmem + MsgqInit --> RmInit: version, client, device, subdevice + RmInit --> Runtime: objects, vaspaces, channels + Runtime --> [*]: driver unload / reset + + note right of FWSEC + Emulated today: the module drives the + stage transitions and answers status + reads from the BAR0 shadow. The real + backend boots the physical GSP in + parallel [planned]. + end note +``` + +### 2.2 Guest initialization, end to end + +```mermaid +sequenceDiagram + autonumber + participant GD as Stock nvidia.ko, guest + participant DEV as Virtual device, BAR0 shadow + MSI-X + participant GE as GSP emulation thread + participant GRM as GSP-RM emulation + participant BE as Execution backend + + GD->>DEV: PCI probe, BAR setup, MSI-X enable + DEV-->>GD: config space and capability replies + GD->>DEV: boot kicks and WPR2 writes + DEV->>GE: stage transitions + GE-->>GD: falcon status - boot complete + GD->>GRM: message queues + RPCs - version, client, device + Note over GRM: every reply follows the fidelity ladder -<br/>status only, then coherent zeroed params,<br/>then coherent consumed values, then exact values + GRM-->>GD: coherent replies for consumed fields + GD->>GRM: object allocations, vaspaces, channels, CE caps + GRM-->>GD: handles, engine and class topology + GD->>GRM: UVM register GPU, page trees, context + GRM->>BE: resource needs - real memory, channels [planned] + BE-->>GRM: real objects and channels [planned] +``` + +### 2.3 RPC surface and object lifecycle + +```mermaid +flowchart LR + subgraph GUEST["Guest RM requests"] + A1["version / capability queries"] + A2["client, device, subdevice"] + A3["memory alloc + mmap descriptors"] + A4["vaspace + page-table contracts"] + A5["channels, CE caps, events"] + end + subgraph EMU["GSP-RM emulation"] + B1["dispatcher by function ID"] + B2["special-cased handlers - real values"] + B3["generic fallback - success + zeroed params"] + B4["object table - handles, params echoed"] + B5["reply fidelity checks"] + end + A1 --> B1 + A2 --> B1 + A3 --> B1 + A4 --> B1 + A5 --> B1 + B1 --> B2 + B1 --> B3 + B2 --> B4 + B3 --> B5 + B4 --> B5 + + classDef done fill:#d5f5d5,stroke:#1e7a1e + classDef wip fill:#fff3cd,stroke:#9a7b1a + class B1,B2,B4 done + class B3,B5 wip +``` + +## 3. The memory model + +Guest work refers to GPU virtual addresses. The module must know which physical +pages they denote — for semaphores, push buffers, page trees, and (in the end +state) for programming the real GPU's MMU so hardware walks the same pages. + +```mermaid +flowchart LR + subgraph G["Guest"] + GVA["GPU virtual addresses<br/>client objects and mappings"] + GPT["Guest page tables<br/>incl. server-owned ranges"] + SYS["Guest system memory"] + VRS["VRAM objects - BAR1 aperture view"] + end + + subgraph MM["open-iov-vfio.ko memory model"] + DISC["Discovery<br/>structure walk + content scan + pins"] + PMAP["VA to PA map<br/>page-granular, merge rebuilds"] + SHAD["BAR1 VRAM shadow"] + PIN["Pin tables<br/>self-discovered + authoritative injected"] + end + + subgraph H["Host / hardware"] + DMA["VFIO DMA - read and write guest pages"] + RGMMU["Real GMMU tables<br/>same pages, real hardware walks"] + end + + GVA --> GPT --> SYS + GVA --> VRS + VRS --> SHAD + SYS --> DMA + GPT --> DISC + DISC --> PMAP + PIN --> PMAP + PMAP --> DMA + PMAP --> RGMMU + SYS -.->|"pinned and mapped once"| RGMMU + + classDef done fill:#d5f5d5,stroke:#1e7a1e + classDef wip fill:#fff3cd,stroke:#9a7b1a + classDef plan fill:#f8d7da,stroke:#9a1a1a + class DISC,PMAP,DMA done + class PIN,SHAD wip + class RGMMU plan +``` + +The two contracts that make this possible without guest cooperation: + +* Channel memory descriptors the guest creates already carry **physical + addresses** (instance memory, method buffers, USERD, RAMFC, notifiers). +* The server-owned page-table contract (`COPY_SERVER_RESERVED_PDES`) gives the + page-table level physical addresses; the leaves are ours to populate. + +The remaining work is to derive every mapping from these protocol objects as +they are created, so a clean boot needs no external tooling. + +## 4. The work path + +### 4.1 Today: software-executed bridge + +```mermaid +flowchart LR + A["Guest push appended<br/>GPFIFO entry + doorbell"] --> B["Channel pump reads entry"] + B --> C{"Entry readable?"} + C -->|yes| D["Decode method stream"] + C -->|no| E["Rescue / reconstruct the page<br/>or resolve via pins"] + E --> D + D --> F["Execute understood work in software<br/>memsets, copies, semaphore releases"] + F --> G["Write releases and completions<br/>into guest memory"] + G --> H["Guest trackers advance"] + + D -.->|"planned"| I["Forward to a real engine"] + + classDef done fill:#d5f5d5,stroke:#1e7a1e + classDef wip fill:#fff3cd,stroke:#9a7b1a + classDef plan fill:#f8d7da,stroke:#9a1a1a + class A,B,D done + class E,F,G,H wip + class I plan +``` + +This is a correctness bridge: it advances synchronization and memory +operations, and it is why the guest's initialization walks as far as it does +today. It cannot execute kernels. + +### 4.2 End state: forwarded work + +```mermaid +sequenceDiagram + autonumber + participant GU as Guest GPFIFO + doorbell + participant P as Channel pump + participant D as Decoder + participant M as Memory model + participant B as Execution backend + participant E as GPU engines + participant S as Guest trackers + + GU->>P: doorbell / put pointer advance + P->>D: read push entry + alt entry unreadable + D->>M: resolve or reconstruct the page + end + D->>D: parse the method stream + alt pure state or notification + D->>M: apply release, membar, fence directly + else real engine work + D->>B: translated submission + B->>E: real push stream on a real channel + E-->>B: completion or fault from hardware + B->>M: completion written into guest memory + end + M-->>S: tracker values advance +``` + +Fast paths (semaphores, fences, some memops) stay in the emulation in both +phases: they are microseconds in hardware and lose value by round-tripping +through the real GPU. + +### 4.3 How the GPU reaches runtime and executes work + +```mermaid +flowchart TB + subgraph BOOT["Guest view - emulated"] + A["Probe"] --> B["GSP boot stages"] + B --> C["GSP-RM init"] + C --> D["Objects, vaspaces, channels"] + D --> E["UVM register + page trees"] + E --> F["Context created, allocations mapped"] + F --> G["Pushes submitted"] + end + subgraph REAL["Backend view - real"] + R1["Module binds GPU, boots real GSP"] --> R2["Real RM client: device, vaspaces"] + R2 --> R3["Real channels and page tables"] + R3 --> R4["Guest VA mapped to same pages"] + R4 --> R5["Real submissions scheduled by GSP"] + R5 --> R6["Completion and faults relayed to guest"] + end + G -.->|"translation [planned]"| R4 + + classDef done fill:#d5f5d5,stroke:#1e7a1e + classDef plan fill:#f8d7da,stroke:#9a1a1a + class A,B,C,D,E done + class F,G wip + class R1,R2,R3,R4,R5,R6 plan +``` + +## 5. Scheduling and multi-tenancy + +There is no hardware partitioning (no SR-IOV) on this GPU generation, so +tenancy is a **logical VF model**: each VM holds one mdev instance, and the +module gives each instance its own GSP-RM session, channels, page tables and +scheduling policy. “Per-VF GSP emulation” therefore means per-mdev emulation +context plus a per-mdev real GSP client in the backend. + +```mermaid +flowchart TB + subgraph TEN["Tenants"] + T1["VM A - mdev 1"] + T2["VM B - mdev 2"] + T3["VM C - mdev N"] + end + subgraph MOD["open-iov-vfio.ko"] + Q1["Emulation context 1<br/>BARs, queues, counters"] + Q2["Emulation context 2"] + Q3["Emulation context N"] + POL["Fairness policy [planned]<br/>time slices, quotas, priority, burst"] + C1["Real GSP client 1 [planned]"] + C2["Real GSP client 2 [planned]"] + C3["Real GSP client N [planned]"] + end + subgraph GPU["Physical GPU"] + SCHED["Real GSP scheduler<br/>engine contexts and runlists"] + GRC["GR contexts"] + CEC["CE contexts"] + end + T1 --> Q1 --> C1 + T2 --> Q2 --> C2 + T3 --> Q3 --> C3 + C1 --> POL + C2 --> POL + C3 --> POL + POL --> SCHED + SCHED --> GRC + SCHED --> CEC + + classDef done fill:#d5f5d5,stroke:#1e7a1e + classDef wip fill:#fff3cd,stroke:#9a7b1a + classDef plan fill:#f8d7da,stroke:#9a1a1a + class Q1,Q2,Q3 done + class C1,C2,C3,POL,SCHED plan +``` + +**How multi-tenant performance works (and is intended to):** + +* Each tenant's *state* operations complete in the module on a fast path + (microseconds, no hardware round trip). +* Each tenant's *real work* runs on the physical engines at native speed; the + virtualization overhead is the submission translation plus completion relay, + which amortizes with work size. +* Fairness is a module-side policy on top of the firmware scheduler (time + slices, quotas). It must be measured, not assumed; the measurement harness is + planned with the performance ladder in + [`../05-operations/01-performance-correctness-security.md`](../05-operations/01-performance-correctness-security.md). + +## 6. Isolation: what is guaranteed, at which level + +```mermaid +flowchart TB + H1["Hardware IOMMU / SMMU<br/>per-VM DMA address spaces<br/>guarantee: memory isolation between VMs"] + H2["GPU MMU contexts - per client page tables [planned]<br/>guarantee: one tenant's engine work cannot address another's memory"] + V1["VFIO mdev boundary<br/>all device access mediated; config/BAR filtering<br/>guarantee: guest gets only its own device instance"] + M1["Module state separation<br/>per-mdev emulation context, queues, pins<br/>guarantee: no cross-tenant state pointers"] + P1["Policy and audit [planned]<br/>fairness, rate limits, side-channel review<br/>guarantee: performance isolation, adversarial robustness"] + + H1 --> V1 --> M1 + H2 --> P1 + M1 --> P1 +``` + +| Layer | Isolation strength | Status | +|---|---|---| +| IOMMU / SMMU between VMs | hardware-enforced | available today (hypervisor) | +| KVM memory and vCPU boundary | hypervisor-enforced | available today | +| VFIO mdev mediation (one device per tenant) | hypervisor-enforced | implemented | +| Per-mdev module state and emulation context | software-enforced | implemented | +| Per-client real page tables and GSP clients | software plus GPU MMU | planned | +| Performance isolation and side channels | policy | planned, must be measured | + +The honest statement: **memory isolation is guaranteed by hardware and the +hypervisor today**; *engine-level context separation and performance isolation* +are the parts this project must still build and audit. + +## 7. How the host kernel schedules the module + +```mermaid +flowchart LR + subgraph CTX["Caller contexts"] + QEMU["QEMU / VFIO ioctls and mmap"] + VCPU["vCPU exits on trapped MMIO"] + FW["Physical GSP messages - backend"] + USR["Debug interfaces - dev only"] + end + subgraph KERN["Linux execution contexts in the module"] + VF["VFIO callbacks"] + IRQ["IRQ handlers - physical MSI-X, eventfd signals"] + WQ["Workqueues - RPC, cleanup, bottom halves"] + KT["kthreads - GSP emulation per mdev, channel service"] + TM["Timers and poll loops - doorbells, housekeeping"] + end + QEMU --> VF --> WQ + VCPU --> VF + FW --> IRQ --> WQ + WQ --> KT + KT --> TM + KT --> WQ + + classDef done fill:#d5f5d5,stroke:#1e7a1e + classDef plan fill:#f8d7da,stroke:#9a1a1a + class VF,IRQ,WQ,KT done + class TM plan +``` + +Design rules that keep this healthy: bounded stack frames (audited), no long +scans while holding service locks, blocking work in kthreads not in VFIO +callbacks, and never tearing down the module while a VM holds the device. + +## 8. Hypervisor integration: mdev, VFIO, KVM + +```mermaid +sequenceDiagram + autonumber + participant ORCH as Orchestrator / admin + participant SYS as Module + mdev sysfs + participant QEMU as QEMU with vfio-pci + participant KVM as KVM + participant GUEST as Guest kernel + + ORCH->>SYS: create mdev - type nvidia-nvidia-vgpu-turing, UUID + SYS-->>ORCH: /sys/bus/mdev/devices/uuid ready + QEMU->>SYS: open VFIO device, query regions and IRQ info + QEMU->>SYS: mmap BAR shadows - region index to vaddr + QEMU->>SYS: VFIO_DEVICE_SET_IRQS - eventfd per MSI-X vector + QEMU->>KVM: register irqfd for guest injection + QEMU->>KVM: map guest RAM - IOMMU / IOMMUFD + GUEST->>QEMU: MMIO access to the virtual BAR0 + QEMU->>SYS: region read or write, or direct shadow access + SYS-->>GUEST: MSI-X delivered via eventfd to irqfd to vCPU + ORCH->>SYS: destroy mdev after VM stops - reverse order +``` + +Key properties: the mdev is the *device instance*; VFIO is the *transport* +(region access, IRQ signalling, DMA bridging); KVM is the *delivery* (irqfd, +MMIO exits, vCPU scheduling). The module registers an emulated IOMMU device so +VFIO creates the group; guest memory access flows through the VFIO DMA +interface. Full detail in +[`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md). + +## 9. Component detail: BARs, engines, channels, GMMU + +```mermaid +flowchart LR + subgraph GV["Guest-visible device"] + CFG["PCI config space<br/>TU104 identity, BAR sizes, MSI-X caps"] + B0["BAR0 - 16 MB registers<br/>doorbell, mailbox, PBA"] + B1["BAR1 - 256 MB VRAM aperture"] + B3["BAR3 - 32 MB instance / PRAMIN window"] + MX["MSI-X - 32 vectors"] + end + subgraph MODS["Module shadow state"] + C0["Filtered config writes"] + S0["BAR0 shadow - boot state, doorbells"] + S1["BAR1 shadow - VRAM backing [wip]"] + S3["BAR3 shadow - instance memory"] + IR["Per-vector eventfd injection"] + end + subgraph CH["Channel machinery"] + GPF["GPFIFO - push entries"] + USR["USERD + doorbell tokens"] + RF["RAMFC + instance memory"] + MB["Method buffers"] + NOT["Notifiers and error buffers"] + end + subgraph EN["Engines"] + GR["GR - compute, grid launches"] + CE["CE pair - copies and UVM memops"] + DEC["DEC - video decode [planned]"] + ENC["ENC - video encode [planned]"] + RL["Runlist / TSG scheduling"] + end + subgraph MM["GMMU"] + HI["Page-directory hierarchy"] + LEAF["Leaf PTEs"] + SRV["Server-owned ranges"] + end + CFG --> C0 + B0 --> S0 --> GPF + B1 --> S1 + B3 --> S3 --> RF + MX --> IR + GPF --> USR + GPF --> RF + GPF --> MB + GPF --> NOT + USR --> GR + USR --> CE + CE --> RL + GR --> RL + RL --> ENC + RL --> DEC + S1 --> HI + S3 --> HI + HI --> LEAF + HI --> SRV + + classDef done fill:#d5f5d5,stroke:#1e7a1e + classDef wip fill:#fff3cd,stroke:#9a7b1a + classDef plan fill:#f8d7da,stroke:#9a1a1a + class CFG,C0,B0,S0,B3,S3,MX,IR,GPF,USR,RF,MB,NOT,GR,CE,RL done + class B1,S1,HI,LEAF,SRV wip + class DEC,ENC plan +``` + +## 10. Portability: versions, hardware, hypervisors + +```mermaid +flowchart LR + A["Firmware / driver image"] -->|"CRC32 + SHA-256"| B["Firmware analyzer"] + B --> C["Version tables<br/>and YAML profiles"] + C --> D["Dynamic RPC dispatcher"] + C --> E["Hardware abstraction profiles"] + D --> F["Version-keyed handlers"] + E --> G["Architecture tables<br/>engine inventory, GMMU formats"] + F --> H["GSP-RM emulation"] + G --> H + H --> I["Same guest interface"] + J["New driver version<br/>or new GPU"] --> A + J --> K["Measurement + A/B against host oracle"] + K --> C + + classDef done fill:#d5f5d5,stroke:#1e7a1e + classDef wip fill:#fff3cd,stroke:#9a7b1a + classDef plan fill:#f8d7da,stroke:#9a1a1a + class B done + class C,D,F done + class E,G,H wip + class K plan +``` + +The *interface* is the product: RPC/control inventories are measured, not +guessed; new versions enter by adding a profile and running the A/B workflow +(see [`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md)). +The execution backend is similarly version-keyed: the real GSP protocol is +spoken as a client, with the module carrying the small set of host-side calls +it needs. + +## 11. Performance: distance to bare metal + +```mermaid +flowchart LR + subgraph FAST["Emulated fast path - state only"] + A["Release, fence, membar"] --> B["Applied in the module"] + B --> C["microseconds; no GPU round trip;<br/>faster than hardware would be"] + end + subgraph DATA["Forwarded work path"] + D["Guest submission"] --> E["Decode + translate + submit"] + E --> F["Engine executes at native speed"] + F --> G["Completion relay"] + end + C -.-> H["Cost model: overhead is per submission,<br/>not per unit of work"] + G -.-> H + H -.-> I["Large copies and kernels amortize to near native;<br/>launch-bound workloads pay the translation"] + + classDef done fill:#d5f5d5,stroke:#1e7a1e + classDef plan fill:#f8d7da,stroke:#9a1a1a + class A,B,C done + class D,E,F,G,H,I plan +``` + +| Aspect | Expectation | Status | +|---|---|---| +| State/synchronization latency | low microseconds, no round trip | measured in debug logs | +| Copy bandwidth (large) | near native; DMA is DMA | not measured yet | +| Kernel launch overhead | translation cost per submission; amortized | not measured yet | +| Multi-tenant fairness | policy-controlled, measurable | policy planned | +| Bare-metal parity claim | only after the measurement ladder is green | see operations doc | + +We do not claim parity today, and the docs say so in plain words: the honest +claim is "initialization works; execution is in progress". + +## 12. Implementation and test status + +How far each component is from the finished system. "Tested" means exercised +against a stock guest driver with evidence in the repository. + +| Component | Implemented | Tested | Next step | +|---|---|---|---| +| mdev + VFIO device, config space, BARs, MSI-X | yes | yes | keep regression-free | +| GSP boot emulation (all stages) | yes | yes | unchanged under backend | +| GSP-RM message queues + RPC + controls | yes | yes | reply fidelity for remaining consumed fields | +| Object/alloc model (handles, echoes) | yes | yes | server-side object tracking where consumed | +| Engine/class topology (incl. async copy engine) | yes | yes | version profiles | +| Guest memory access + VA-to-PA discovery | partial | yes | protocol-derived discovery, no external pins | +| Push decode + semaphore completion | partial | yes | complete GR/compute encodings | +| Software execution bridge (memsets, copies, releases) | partial | yes | remains bridge only | +| Page-table population for server-owned ranges | no | no | design + validator, then implement | +| Execution backend (own GPU + real GSP) | no | no | device ownership, real GSP bring-up, RM client | +| Real work forwarding (GR and CE) | no | no | after backend bring-up | +| Fault semantics (buffers, replay, cancel) | no | no | after forwarding | +| Multi-tenant fairness policy | no | no | after multi-tenant forwarding | +| Version matrix (2+ driver families) | partial | partial | per-version profiles + A/B | +| Windows guest | no | no | after Linux execution path | +| Non-KVM transports (Xen, Argo, OpenXT) | no | no | transport interface + backend | +| aarch64 hosts (Grace, GB10) | no | no | architecture-neutral memory model | +| Debug interfaces (dev only) | yes | yes | gate destructive ones; production build | + +## 13. How the unmodified-driver guarantee is achieved + +* **Emulate what the guest observes, forward what it uses.** Every interface + the stock driver touches is answered by the module; every operation that must + be fast goes to real hardware. The guest never learns it is virtualized for + the CUDA/initialization path — it sees a TU104 whose GSP answers correctly. +* **Reply fidelity, not just status.** A reply whose fields the driver consumes + is only considered done when the values are coherent (status-only, then + coherent zeroed params, then coherent consumed fields, then exact values). + This is the mechanism that makes arbitrary graphics and CUDA stacks work + without per-application code. +* **No guest code, no paravirtual enlightenment, no patches.** The guest may + install whatever software it likes; the module presents physical hardware. + The planned execution backend keeps this property: the guest's work is + translated, never the guest's driver. + +## See also + +- [`01-system-overview.md`](01-system-overview.md) - principles and layers in prose +- [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) - the module, memory model, build contracts +- [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md) - the chosen execution backend and its decision record +- [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) - gaps and plans +- [`../05-operations/01-performance-correctness-security.md`](../05-operations/01-performance-correctness-security.md) - measurement ladder, threat model diff --git a/docs/03-architecture/10-interaction-and-scheduling.md b/docs/03-architecture/10-interaction-and-scheduling.md new file mode 100644 index 0000000..3608dc5 --- /dev/null +++ b/docs/03-architecture/10-interaction-and-scheduling.md @@ -0,0 +1,337 @@ +# Interaction and Scheduling: Cross-Component Detail + +> **What this document is.** The zoom lens companion to +> [`09-full-system-picture.md`](09-full-system-picture.md): every +> cross-component interaction in the system — the exact call paths, queues, +> doorbells, interrupts, locks and orderings — plus the multi-tenant scheduling +> model and the timing each interaction is designed to hit. Diagrams are +> Mermaid; status tags follow the same legend as 09. + +Status legend: `[done]` implemented and exercised; `[wip]` in progress; +`[planned]` designed, not built; hardware/firmware exists in the physical +world. + +## 1. Interaction inventory + +Every place two components touch, the mechanism, the latency class it belongs +to, and where it stands. + +| Interaction | Path and mechanism | Latency class | Status | +|---|---|---|---| +| Guest MMIO write to a mapped BAR | PCIe write lands directly in the module's shadow region (no trap) | tens of ns | [done] | +| Guest doorbell signal (boot, mailbox, channel) | write into BAR0 shadow, observed by the polling service | steering only | [done] | +| Guest RPC request | element placed in a guest-sysmem message queue, read by the module over VFIO DMA | 10s of us per round trip | [done] | +| RPC reply + interrupt | module writes reply into the queue, signals eventfd, QEMU raises irqfd into KVM, MSI-X lands in the guest | us | [done] | +| Push-buffer work | guest appends GPFIFO entries; the per-channel pump reads, decodes and applies them | per entry, us to low ms | [wip] | +| Semaphore release to guest memory | module writes the payload word into the resolved guest page | us | [done] | +| VA-to-PA translation discovery | in-module structure walk, content scan, rescue, pins | us to seconds (scan bounded) | [wip] | +| Translation internalization from protocol | derive mappings from RM/UVM objects as they are created | us | [planned] | +| Real channel submission | translated push stream on a real channel, real doorbell | us + hardware | [planned] | +| Physical GPU interrupt | real MSI-X into the module's IRQ handler, relayed to guest trackers | us | [planned] | +| mdev create / destroy | sysfs plus VFIO device registration | ms | [done] | +| Guest RAM mapping | VFIO / IOMMUFD maps guest pages for DMA | ms once | [done] | +| Teardown | ordered stop: VM, device close, mdev remove, module unload | s | [done] | + +## 2. Control-plane call path: guest ioctl to completed reply + +The full path of one resource-manager control during initialization. Boot-stage +register traffic follows the same shape but through the BAR0 shadow instead of +the message queue. + +```mermaid +sequenceDiagram + autonumber + participant GL as Guest libcuda + participant GK as Guest nvidia.ko RM + participant MQ as Message queue in guest sysmem + participant SB as BAR0 shadow doorbell + participant GE as GSP emulation thread + participant GRM as GSP-RM handlers + participant DMA as VFIO DMA bridge + participant EV as eventfd and irqfd + participant KVM as KVM + + GL->>GK: ioctl NV_ESC_RM_CONTROL + GK->>MQ: build RPC element header and payload + GK->>SB: ring doorbell + GE->>SB: poll observes the doorbell + GE->>DMA: read element from guest sysmem + GE->>GRM: dispatch by function ID + GRM->>GRM: special-case handler or fallback, build reply + GRM->>DMA: write reply element into the queue + GE->>EV: signal completion + EV->>KVM: irqfd raises the vector + KVM-->>GK: MSI-X delivered + GK->>MQ: read reply and complete the ioctl + GL-->>GL: continue + + Note over GE,GRM: Reply fidelity ladder applies here -<br/>status, params, consumed values, exact values. + Note over SB,GE: The poll loop is the steering wheel for the whole<br/>control plane; it must never block on long work. +``` + +## 3. Message-queue element lifecycle + +```mermaid +stateDiagram-v2 + direction LR + [*] --> Free + Free --> Filled: guest writes header and payload + Filled --> Signaled: doorbell rung + Signaled --> Read: module reads via DMA + Read --> Validated: sequence, length, bounds checked + Validated --> Dispatched: RPC function ID dispatched + Dispatched --> Replied: handler result written back + Replied --> Marked: sequence and status updated + Marked --> Interrupted: MSI-X signalled + Interrupted --> Consumed: guest reads and frees the slot + Consumed --> Free + + note right of Read + Non-contiguous queue buffers are + translated through the queue page + table before any element is touched. + end note + note right of Validated + A malformed or stale element is + rejected without disturbing the + queue; sequence matching is strict. + end note +``` + +## 4. The channel pump: per-channel work loop + +One instance of this loop runs for every channel of every mdev. It is the +meeting point of guest submissions, translation, decoding and completion. + +```mermaid +stateDiagram-v2 + direction LR + [*] --> Idle + Idle --> EntrySeen: put pointer advanced + EntrySeen --> Reading: fetch the GPFIFO entry + Reading --> Readable: translation exists + Reading --> Unreadable: no translation yet + Unreadable --> Rescue: rescue or resolve the page + Rescue --> Decoding: page reconstructed + Rescue --> Skipped: attempts exhausted or policy skip + Readable --> Decoding: parse method stream + Decoding --> Applying: execute understood operations + Applying --> Completing: releases, notifiers, counters + Completing --> Advancing: consume the entry, advance get + Skipped --> Advancing + Advancing --> Idle: next pass + + note right of Rescue + Bounded scans, hot-page first, with + retry budgets. A skipped entry is a + recorded event, never a silent stall. + end note + note right of Applying + State and notification operations are + applied from the emulation. Real engine + work is forwarded [planned]. + end note +``` + +## 5. Concurrency and ownership + +Which context touches which structure, and under whose discipline. + +```mermaid +flowchart LR + subgraph CTX["Execution contexts"] + ST["Service kthread<br/>pump, releases, injections"] + GT["GSP emulation kthread<br/>per mdev"] + VC["VFIO callbacks<br/>QEMU ioctl and mmap context"] + WQ["Workqueues<br/>RPC, cleanup, bottom halves"] + IR["IRQ handlers<br/>physical MSI-X [planned]"] + end + + subgraph DATA["Shared structures"] + CT["Channel table<br/>handles, rings, notifiers"] + MAP["VA-to-PA map<br/>plus hot-page set"] + PIN["Pin tables<br/>discovered plus injected"] + PEND["Pending release queue"] + CNT["Completion counter table"] + MQS["Message queue shadow"] + CFG["Config and BAR shadows"] + end + + ST --> CT + ST --> MAP + ST --> PIN + ST --> PEND + ST --> CNT + GT --> MQS + GT --> CT + GT --> CFG + VC --> CFG + WQ --> GT + IR --> WQ + + classDef done fill:#d5f5d5,stroke:#1e7a1e + classDef wip fill:#fff3cd,stroke:#9a7b1a + classDef plan fill:#f8d7da,stroke:#9a1a1a + class ST,GT,VC,WQ done + class CT,MAP,PIN,PEND,CNT,MQS,CFG wip + class IR plan +``` + +Rules the design enforces: + +* One writer per structure. Injections are queued and drained by the service; + self-discovery never overwrites authoritative injected data. +* Long scans are sliced and preemptible; the pump and release paths are never + blocked behind a full-RAM pass. +* VFIO callbacks do only bounded work; blocking work lives in kthreads. +* Every bounded budget (stack frames, retry counts, scan slices) is audited at + build time. + +## 6. Timing model and budgets + +| Interaction | Target | Why it matters | +|---|---|---| +| Guest MMIO to shadow | direct memory write | zero trap cost on the data path | +| Doorbell to dispatch | sub-ms | steers every control exchange | +| RPC round trip during init | 10s of us | hundreds of exchanges per boot | +| Release apply | us | trackers poll at high frequency | +| Push entry decode | us per entry | thousands of entries per context | +| Rescue of an unreadable entry | bounded ms | only on translation misses | +| MSI-X to guest | us | completion latency floor | +| Real submission + completion relay | hardware + us | the forwarded-work path [planned] | + +The rule of thumb the architecture is built around: **state operations are +cheaper in the module than they would be on hardware; real work belongs on +hardware and pays only submission and relay.** + +## 7. Multi-tenant scheduling model + +The target model: per-tenant emulation contexts and real GSP clients, with +fairness policy in the module on top of the firmware scheduler. + +```mermaid +sequenceDiagram + autonumber + participant P as Module policy [planned] + participant A as Tenant A client + participant B as Tenant B client + participant S as Real GSP scheduler + participant E as Engines + + Note over P,E: State operations never enter this loop -<br/>they complete in the module on the fast path. + P->>A: grant slice or quota + A->>S: submissions for tenant A + S->>E: tenant A contexts + E-->>S: completions for tenant A + S-->>A: completions relayed + P->>B: switch to tenant B + B->>S: submissions for tenant B + S->>E: tenant B contexts + E-->>S: completions for tenant B + S-->>B: completions relayed + Note over P: Policy inputs - priority, quota, burst,<br/>per-tenant accounting. Measured, not assumed. +``` + +Fairness and performance isolation properties to be measured once forwarding +exists: per-tenant completion accounting, slice enforcement, starvation +freedom, and the overhead each tenant pays for the other's activity. + +## 8. Signal paths: doorbells and interrupts, both directions + +```mermaid +flowchart TB + subgraph IN["Guest to module"] + A["Guest writes doorbell in BAR0 shadow"] --> B["Poll loop observes"] + B --> C["Fetch and dispatch"] + end + subgraph OUT["Module to guest"] + D["Module has a completion"] --> E["eventfd signal"] + E --> F["QEMU irqfd"] + F --> G["KVM injects MSI-X vector"] + G --> H["Guest ISR runs"] + end + subgraph PHY["Module and physical GPU [planned]"] + I["Real channel doorbell"] --> J["GPU executes"] + J --> K["Physical MSI-X"] + K --> L["Module IRQ handler"] + L --> M["Relay to guest or hardware semaphore"] + end + C -.-> I + M -.-> E + + classDef done fill:#d5f5d5,stroke:#1e7a1e + classDef plan fill:#f8d7da,stroke:#9a1a1a + class A,B,C,D,E,F,G,H done + class I,J,K,L,M plan +``` + +## 9. Lifecycle and teardown ordering + +The ordering is not cosmetic: the module must never be unloaded while a VM +holds the device, and the mdev teardown path must not run long scans or +unbounded waits. + +```mermaid +sequenceDiagram + autonumber + participant ORCH as Orchestrator + participant QEMU as QEMU + participant VFIO as VFIO and module + participant SYS as mdev sysfs + + ORCH->>QEMU: stop the VM + QEMU->>VFIO: close device, release regions and IRQs + VFIO-->>QEMU: resources released + ORCH->>SYS: remove mdev + SYS->>VFIO: remove callback - bounded cleanup + VFIO-->>SYS: device gone + ORCH->>VFIO: unload the module + + alt guest hangs or faults + ORCH->>QEMU: reset or destroy the VM first + Note over VFIO: Never unload while a VM holds the device;<br/>forensics come from pstore and bounded logs. + end +``` + +## 10. Version negotiation across components + +```mermaid +sequenceDiagram + autonumber + participant GD as Guest driver + participant EMU as GSP-RM emulation + participant VT as Version profile + participant BE as Execution backend [planned] + participant FW as Real GSP firmware [planned] + + GD->>EMU: version and capability queries + EMU->>VT: select profile for this driver family + VT-->>EMU: RPC surface, controls, caps + EMU-->>GD: negotiated replies + Note over EMU,VT: New driver versions enter as profiles -<br/>measured against the host oracle, never guessed. + BE->>FW: host-side protocol version negotiation + FW-->>BE: firmware capabilities + Note over BE,FW: The backend carries its own small,<br/>version-keyed host-side protocol surface. +``` + +## 11. Data-structure map + +| Structure | Represents | Written by | Consumed by | Status | +|---|---|---|---|---| +| Message queue + shadow | guest RM RPC transport | guest driver; emulation replies | emulation thread; guest RM | [done] | +| BAR0 shadow | register interface (boot, doorbells, PBA) | guest MMIO | emulation poll; pump | [done] | +| Channel table | per-channel ring, get/put, notifier, engine | guest objects; pump | pump; release path | [wip] | +| GPFIFO entries | queued work per channel | guest submissions | pump | [wip] | +| USERD / doorbell tokens | work signalling | guest | module | [wip] | +| VA-to-PA map | translations with hot-page hints | discovery and scans | pump; release path; pins | [wip] | +| Pin tables | authoritative injected translations | debug tooling; later protocol discovery | lookup fast path | [wip] | +| Pending release queue | releases awaiting a translation | decoder | service drain | [done] | +| Completion counters | per-channel progress published to guest pages | service | guest trackers | [done] | +| Real channels and page tables | forwarding state | backend | real GPU | [planned] | + +## See also + +- [`09-full-system-picture.md`](09-full-system-picture.md) - the map this document zooms into +- [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) - module internals and build contracts +- [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md) - the execution backend and decision record +- [`01-system-overview.md`](01-system-overview.md) - principles and layer map diff --git a/docs/README.md b/docs/README.md index 24b2a21..81f02c8 100644 --- a/docs/README.md +++ b/docs/README.md @@ -31,33 +31,41 @@ generations (Turing, Ampere, Hopper, Grace-Hopper, Grace-Blackwell, GB10, ...) and many driver versions — without any of those ecosystems, operating systems, drivers or devices having to change. +```mermaid +flowchart TB + subgraph VM["Guest VM - stock drivers, stock applications"] + A["CUDA / OpenGL / Vulkan applications"] + B["Userspace stack - libcuda, libcudart, libGLX, libvulkan"] + C["Stock nvidia.ko - RM core and UVM"] + A --> B --> C + end + subgraph HYP["Hypervisor - KVM and QEMU"] + D["VFIO mdev transport + IOMMU isolation"] + end + subgraph MOD["open-iov-vfio.ko - the only host component"] + E["Virtual device - BARs, config space, MSI-X"] + F["GSP boot + GSP-RM emulation"] + G["Memory model + work path"] + H["Execution backend - real GSP session"] + E --> F --> G --> H + end + C -->|"emulated PCIe"| D --> E + H --> I["Physical GPU - engines, GMMU, VRAM"] + classDef done fill:#d5f5d5,stroke:#1e7a1e + classDef wip fill:#fff3cd,stroke:#9a7b1a + classDef plan fill:#f8d7da,stroke:#9a1a1a + classDef hw fill:#e2e8f0,stroke:#4a5568 + class E,F done + class G wip + class H plan + class I hw ``` - ┌─────────────────────────────────────────────┐ - │ GUEST VM │ - │ stock nvidia.ko + CUDA / OpenGL / Vulkan │ - └──────────────┬──────────────────────────────┘ - │ PCIe (emulated device) - ┌───────────────────────────────┴───────────────────────────────┐ - │ HYPERVISOR (KVM/Xen) │ - │ VFIO / mdev transport, IOMMU isolation │ - └───────────────────────────────┬───────────────────────────────┘ - │ - ┌───────────────────────────────┴───────────────────────────────┐ - │ HOST: open-iov-vfio.ko (this project) │ - │ • virtual GPU device (BARs, config space, MSI-X) │ - │ • GSP / GSP-RM interface emulation (RPC, msgq, controls) │ - │ • memory model + GPU work forwarding to the physical GPU │ - └───────────────────────────────┬───────────────────────────────┘ - │ - ┌──────────────┴──────────────┐ - │ UNMODIFIED host nvidia.ko │ - │ (or GPU passthrough path) │ - └──────────────┬──────────────┘ - │ - ┌───────┴────────┐ - │ NVIDIA GPU │ - └────────────────┘ -``` + +The full visual map — every component, status tag and interaction — is in +[`03-architecture/09-full-system-picture.md`](03-architecture/09-full-system-picture.md) +(end-to-end) and +[`03-architecture/10-interaction-and-scheduling.md`](03-architecture/10-interaction-and-scheduling.md) +(call paths, queues, interrupts, scheduling). ## 2. Why does this need to exist? @@ -118,6 +126,11 @@ control; the chosen backend — a module-owned GPU running its real GSP firmware as the resource-manager server — is specified in [`03-architecture/08-physical-gpu-integration.md`](03-architecture/08-physical-gpu-integration.md). +For the visual end-state map and the cross-component call paths, see +[`03-architecture/09-full-system-picture.md`](03-architecture/09-full-system-picture.md) +and +[`03-architecture/10-interaction-and-scheduling.md`](03-architecture/10-interaction-and-scheduling.md). + ## 4. How to read these docs (progressive disclosure) Read in order if you are new. Jump directly if you know what you need. @@ -135,21 +148,23 @@ Read in order if you are new. Jump directly if you know what you need. | 2 | [The AI edge and the ecosystem](01-orientation/02-the-ai-edge-and-ecosystem.md) | everyone | 15 min | | 3 | [Install and first run](02-getting-started/01-install-and-first-run.md) | users, evaluators | 20 min | | 4 | [System overview](03-architecture/01-system-overview.md) | everyone technical | 20 min | -| 5 | [Guest stack and CUDASW](03-architecture/02-guest-stack-and-cudasw.md) | developers | 25 min | -| 6 | [GSP and GSP-RM](03-architecture/03-gsp-and-gsp-rm.md) | developers | 30 min | -| 7 | [UVM](03-architecture/04-uvm.md) | developers | 25 min | -| 8 | [GMMU, AMAPlib, HALs, runlist, GR](03-architecture/05-gmmu-amap-hal-gr.md) | developers | 25 min | -| 9 | [Host module and hybrid model](03-architecture/06-host-module-and-hybrid-model.md) | developers | 25 min | -| 10 | [Driver lifecycle field guide](03-architecture/07-driver-lifecycle-field-guide.md) | developers, debuggers | 30 min | -| 11 | [Physical GPU integration](03-architecture/08-physical-gpu-integration.md) | developers, architects | 20 min | -| 12 | [Known-knowns and known-unknowns](04-implementation/01-known-knowns-and-unknowns.md) | everyone technical | 25 min | -| 13 | [Practical challenges and adversarial interoperability](04-implementation/02-practical-challenges-and-adversarial-interop.md) | developers, stakeholders | 20 min | -| 14 | [Performance, correctness, security](05-operations/01-performance-correctness-security.md) | developers, auditors | 25 min | -| 15 | [Debugging and maintenance workflows](05-operations/02-debugging-and-maintenance-workflows.md) | developers, maintainers | 25 min | -| 16 | [Hardware and software matrix](06-portability/hardware-software-matrix.md) | developers, integrators | 20 min | -| 17 | [Developer onboarding and standards](07-developers/onboarding-and-standards.md) | contributors | 20 min | -| 18 | [HAAEEP and collaboration](08-ecosystem/haaeep-and-collaboration.md) | stakeholders, partners | 20 min | -| 19 | [Glossary and FAQ](09-reference/glossary-and-faq.md) | everyone | reference | +| 5 | [The full system picture](03-architecture/09-full-system-picture.md) | everyone technical | 30 min | +| 6 | [Interaction and scheduling](03-architecture/10-interaction-and-scheduling.md) | developers | 30 min | +| 7 | [Guest stack and CUDASW](03-architecture/02-guest-stack-and-cudasw.md) | developers | 25 min | +| 8 | [GSP and GSP-RM](03-architecture/03-gsp-and-gsp-rm.md) | developers | 30 min | +| 9 | [UVM](03-architecture/04-uvm.md) | developers | 25 min | +| 10 | [GMMU, AMAPlib, HALs, runlist, GR](03-architecture/05-gmmu-amap-hal-gr.md) | developers | 25 min | +| 11 | [Host module and hybrid model](03-architecture/06-host-module-and-hybrid-model.md) | developers | 25 min | +| 12 | [Driver lifecycle field guide](03-architecture/07-driver-lifecycle-field-guide.md) | developers, debuggers | 30 min | +| 13 | [Physical GPU integration](03-architecture/08-physical-gpu-integration.md) | developers, architects | 20 min | +| 14 | [Known-knowns and known-unknowns](04-implementation/01-known-knowns-and-unknowns.md) | everyone technical | 25 min | +| 15 | [Practical challenges and adversarial interoperability](04-implementation/02-practical-challenges-and-adversarial-interop.md) | developers, stakeholders | 20 min | +| 16 | [Performance, correctness, security](05-operations/01-performance-correctness-security.md) | developers, auditors | 25 min | +| 17 | [Debugging and maintenance workflows](05-operations/02-debugging-and-maintenance-workflows.md) | developers, maintainers | 25 min | +| 18 | [Hardware and software matrix](06-portability/hardware-software-matrix.md) | developers, integrators | 20 min | +| 19 | [Developer onboarding and standards](07-developers/onboarding-and-standards.md) | contributors | 20 min | +| 20 | [HAAEEP and collaboration](08-ecosystem/haaeep-and-collaboration.md) | stakeholders, partners | 20 min | +| 21 | [Glossary and FAQ](09-reference/glossary-and-faq.md) | everyone | reference | Deep-dive material lives in [`../wiki/`](../wiki/README.md) (firmware decompilation record, address reference, historical protocol notes). Wiki |