summaryrefslogtreecommitdiffstats
path: root/docs
diff options
context:
space:
mode:
authorArthur Hanson Rasmusson <arthur@vgpu.io>2026-09-29 16:11:28 -0400
committerArthur Hanson Rasmusson <arthur@vgpu.io>2026-09-29 16:11:28 -0400
commit78b8f909784676c5ec6f02a6c48d51ec2a589f96 (patch)
tree3bf70f86312b66071ea9a96debcf84fe25ff4ae8 /docs
parentad16069d87c3c852b3f3c68ed41c62d581c213ed (diff)
downloadopen-iov-78b8f909784676c5ec6f02a6c48d51ec2a589f96.tar.gz
open-iov-78b8f909784676c5ec6f02a6c48d51ec2a589f96.zip
docs: add the full-system picture and interaction/scheduling diagrams
- new 03-architecture/09-full-system-picture.md: end-state map, GSP boot, guest init, RPC lifecycle, memory model, work path (bridge and end state), scheduling, isolation levels, KVM/mdev/VFIO, BAR/engine/channel/GMMU, portability, performance model, implementation/test status - new 03-architecture/10-interaction-and-scheduling.md: interaction inventory, control-plane call path, msgq element lifecycle, channel pump, concurrency, timing budgets, multi-tenant scheduling, teardown ordering, versioning - README: mermaid system diagram replaces the ASCII block; reading list extended and renumbered - component docs: GSP boot sequence, mdev/VFIO lifecycle, end-state interaction sequence; STATE-CURRENT links
Diffstat (limited to 'docs')
-rw-r--r--docs/03-architecture/03-gsp-and-gsp-rm.md23
-rw-r--r--docs/03-architecture/06-host-module-and-hybrid-model.md22
-rw-r--r--docs/03-architecture/08-physical-gpu-integration.md27
-rw-r--r--docs/03-architecture/09-full-system-picture.md666
-rw-r--r--docs/03-architecture/10-interaction-and-scheduling.md337
-rw-r--r--docs/README.md97
6 files changed, 1131 insertions, 41 deletions
diff --git a/docs/03-architecture/03-gsp-and-gsp-rm.md b/docs/03-architecture/03-gsp-and-gsp-rm.md
index 8809bb7..fe51838 100644
--- a/docs/03-architecture/03-gsp-and-gsp-rm.md
+++ b/docs/03-architecture/03-gsp-and-gsp-rm.md
@@ -54,6 +54,29 @@ Our emulation implements stage completion rather than the stages themselves —
which is legitimate because the *only observable* to the driver is the state
machine it polls.
+```mermaid
+sequenceDiagram
+ autonumber
+ participant GD as Guest driver
+ participant R0 as BAR0 shadow
+ participant GE as GSP emulation thread
+ participant MQ as Message queues in guest sysmem
+ participant GRM as GSP-RM emulation
+
+ GD->>R0: WPR descriptors and boot kicks
+ GE-->>R0: stage status transitions
+ GD->>R0: poll for boot complete
+ R0-->>GD: firmware running
+ GD->>MQ: set up queues and ring the doorbell
+ GE->>MQ: read and dispatch elements
+ GRM-->>MQ: write replies
+ GE-->>GD: MSI-X completion
+```
+
+The cross-component detail (doorbells, queues, interrupts, latency classes) is
+in [`09-full-system-picture.md`](09-full-system-picture.md) and
+[`10-interaction-and-scheduling.md`](10-interaction-and-scheduling.md).
+
## 3. The transport: message queues and RPC
GSP-RM's transport is a shared-memory message queue pair:
diff --git a/docs/03-architecture/06-host-module-and-hybrid-model.md b/docs/03-architecture/06-host-module-and-hybrid-model.md
index 61be96c..e29951c 100644
--- a/docs/03-architecture/06-host-module-and-hybrid-model.md
+++ b/docs/03-architecture/06-host-module-and-hybrid-model.md
@@ -31,6 +31,28 @@ exposes one device type per supported profile. Each `mdev` instance is an
independent virtual GPU with its own state, its own BAR windows, its own
interrupts and its own emulation context.
+```mermaid
+sequenceDiagram
+ autonumber
+ participant ORCH as Orchestrator
+ participant SYS as Module and mdev sysfs
+ participant QEMU as QEMU with vfio-pci
+ participant KVM as KVM
+ participant GUEST as Guest
+
+ ORCH->>SYS: create mdev for a VM
+ QEMU->>SYS: open VFIO device, mmap regions, set IRQ eventfds
+ QEMU->>KVM: map guest RAM, register irqfd
+ GUEST->>SYS: MMIO and DMA through the emulated device
+ SYS-->>GUEST: MSI-X via eventfd to irqfd
+ ORCH->>SYS: destroy mdev after the VM stops
+```
+
+The full interaction and scheduling detail is in
+[`10-interaction-and-scheduling.md`](10-interaction-and-scheduling.md); the
+end-to-end maps are in
+[`09-full-system-picture.md`](09-full-system-picture.md).
+
## 2. Virtual device
### 2.1 PCI configuration space
diff --git a/docs/03-architecture/08-physical-gpu-integration.md b/docs/03-architecture/08-physical-gpu-integration.md
index f156429..06df22b 100644
--- a/docs/03-architecture/08-physical-gpu-integration.md
+++ b/docs/03-architecture/08-physical-gpu-integration.md
@@ -167,6 +167,33 @@ per-context scheduling. Tenant policy and fairness are a later layer on top.
device-ownership lifecycle; the emulation path remains the control plane and
is reused.
+## 9. Interaction sequence (end state)
+
+```mermaid
+sequenceDiagram
+ autonumber
+ participant GD as Guest driver
+ participant EMU as GSP-RM emulation - control
+ participant BE as Backend - real GSP client
+ participant GSP as Real GSP firmware
+ participant ENG as Engines
+
+ GD->>EMU: objects, vaspaces, channel creation
+ EMU->>BE: translate to real objects [planned]
+ BE->>GSP: allocate real objects and channels
+ GD->>EMU: work submissions
+ EMU->>BE: decoded and translated push streams
+ BE->>ENG: submit on real channels
+ ENG-->>BE: completions and faults
+ BE->>EMU: relay into guest memory
+ EMU-->>GD: trackers and events advance
+```
+
+The full interaction and scheduling view (queues, interrupts, timing classes,
+multi-tenant policy) is in
+[`10-interaction-and-scheduling.md`](10-interaction-and-scheduling.md); the
+system map is in [`09-full-system-picture.md`](09-full-system-picture.md).
+
## See also
- [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) - the host module, memory model, build contracts
diff --git a/docs/03-architecture/09-full-system-picture.md b/docs/03-architecture/09-full-system-picture.md
new file mode 100644
index 0000000..0df0dd2
--- /dev/null
+++ b/docs/03-architecture/09-full-system-picture.md
@@ -0,0 +1,666 @@
+# The Full System Picture
+
+> **What this document is.** The visual, end-to-end picture of Open-IOV: every
+> component in the finished system, how they interact, where the work actually
+> executes, how tenancy is isolated and scheduled, how the design stays portable
+> across driver and hardware versions, and how far each piece is from done.
+> The other architecture documents explain *why*; this one shows *what* and
+> *where*, all the way down. Diagrams are Mermaid and render on GitHub and in
+> most Markdown viewers.
+
+**Status legend used throughout** (the project is honest about maturity):
+
+| Tag | Meaning |
+|---|---|
+| `[done]` | implemented and exercised with a stock guest driver |
+| `[wip]` | implemented in part; actively being completed |
+| `[planned]` | designed, not implemented |
+| hardware/firmware | exists in the physical world; referenced, not built by us |
+| vendor | NVIDIA components we interoperate with, unmodified |
+
+## 1. The end-state system map
+
+All components of the finished system, with the execution backend this project
+has chosen (a module-owned GPU driven through its real GSP firmware; see
+[`08-physical-gpu-integration.md`](08-physical-gpu-integration.md)).
+
+```mermaid
+flowchart TB
+ subgraph VM["Guest VM - stock software, no Open-IOV code"]
+ APP["Applications<br/>CUDA / OpenGL / Vulkan / video"]
+ USR["Userspace stack<br/>libcuda, libcudart, libGLX, libvulkan"]
+ GKO["Stock nvidia.ko<br/>RM core + UVM + modeset/DRM"]
+ APP --> USR --> GKO
+ end
+
+ subgraph HV["Hypervisor - KVM and QEMU"]
+ VFIO["VFIO mdev transport<br/>regions, mmap, eventfd / irqfd"]
+ IOMMU["IOMMU / IOMMUFD<br/>per-VM DMA address spaces"]
+ end
+
+ subgraph MOD["open-iov-vfio.ko - the only host component"]
+ VDEV["Virtual device<br/>config space, BAR0/1/3, MSI-X"]
+ EMU["Emulation core<br/>GSP boot + GSP-RM RPC + objects"]
+ MEMM["Memory model<br/>VA to PA, pins, VRAM shadow, page tables"]
+ WORK["Work path<br/>pump, decode, translate, forward"]
+ BE["Execution backend<br/>real GSP session + RM client"]
+ VDEV --> EMU
+ EMU --> MEMM
+ EMU --> WORK
+ MEMM --> WORK
+ WORK --> BE
+ BE --> MEMM
+ end
+
+ subgraph HW["Physical GPU"]
+ GSP["GSP firmware - the real RM server"]
+ ENG["Engines<br/>GR, CE pair, DEC, ENC"]
+ GMMU["GMMU and page tables"]
+ VRAM["VRAM + framebuffer"]
+ DB["BAR0 / USERD doorbells, MSI-X"]
+ end
+
+ subgraph HOSTOS["Host OS"]
+ ND["Distribution nvidia.ko<br/>unmodified; unbound while the service owns the GPU"]
+ IGPU["Display on another GPU / integrated graphics"]
+ end
+
+ GKO -->|"MMIO + DMA + MSI-X"| VFIO
+ VFIO --> VDEV
+ BE -->|"msgq + doorbells"| GSP
+ WORK -->|"real push streams on real channels"| ENG
+ BE --> GMMU
+ MEMM --> IOMMU
+
+ classDef done fill:#d5f5d5,stroke:#1e7a1e
+ classDef wip fill:#fff3cd,stroke:#9a7b1a
+ classDef plan fill:#f8d7da,stroke:#9a1a1a
+ classDef hw fill:#e2e8f0,stroke:#4a5568
+ classDef vendor fill:#e7e0f5,stroke:#5a4a8a
+
+ class VDEV,EMU done
+ class MEMM,WORK wip
+ class BE plan
+ class IOMMU,VFIO done
+ class GSP,ENG,GMMU,VRAM,DB hw
+ class ND,USR,GKO,APP vendor
+```
+
+**Reading the map.** The guest sees only a PCIe device. Everything between
+that device and the physical engines is either emulated (control plane) or
+translated (work plane) by one kernel module, with no daemon, no host userspace
+and no modified driver on either side. The distribution `nvidia.ko` remains
+installed and unmodified on the host; it is simply unbound from the GPU while
+the service owns it, and the host's display runs on another GPU.
+
+## 2. The control plane: GSP emulation
+
+### 2.1 The GSP boot handshake
+
+The guest driver believes it boots a real GSP. The module completes the
+observable boot stages itself and reports the status the driver polls. In the
+end state, the *backend* additionally boots the physical GPU's real GSP for
+work execution — the two sessions are independent.
+
+```mermaid
+stateDiagram-v2
+ direction LR
+ [*] --> PCIProbe
+ PCIProbe --> FWSEC: driver kicks boot
+ FWSEC --> Booter: FRTS status observed
+ Booter --> RISCV: WPR2 programmed
+ RISCV --> BootComplete: falcon reports running
+ BootComplete --> MsgqInit: queue setup in guest sysmem
+ MsgqInit --> RmInit: version, client, device, subdevice
+ RmInit --> Runtime: objects, vaspaces, channels
+ Runtime --> [*]: driver unload / reset
+
+ note right of FWSEC
+ Emulated today: the module drives the
+ stage transitions and answers status
+ reads from the BAR0 shadow. The real
+ backend boots the physical GSP in
+ parallel [planned].
+ end note
+```
+
+### 2.2 Guest initialization, end to end
+
+```mermaid
+sequenceDiagram
+ autonumber
+ participant GD as Stock nvidia.ko, guest
+ participant DEV as Virtual device, BAR0 shadow + MSI-X
+ participant GE as GSP emulation thread
+ participant GRM as GSP-RM emulation
+ participant BE as Execution backend
+
+ GD->>DEV: PCI probe, BAR setup, MSI-X enable
+ DEV-->>GD: config space and capability replies
+ GD->>DEV: boot kicks and WPR2 writes
+ DEV->>GE: stage transitions
+ GE-->>GD: falcon status - boot complete
+ GD->>GRM: message queues + RPCs - version, client, device
+ Note over GRM: every reply follows the fidelity ladder -<br/>status only, then coherent zeroed params,<br/>then coherent consumed values, then exact values
+ GRM-->>GD: coherent replies for consumed fields
+ GD->>GRM: object allocations, vaspaces, channels, CE caps
+ GRM-->>GD: handles, engine and class topology
+ GD->>GRM: UVM register GPU, page trees, context
+ GRM->>BE: resource needs - real memory, channels [planned]
+ BE-->>GRM: real objects and channels [planned]
+```
+
+### 2.3 RPC surface and object lifecycle
+
+```mermaid
+flowchart LR
+ subgraph GUEST["Guest RM requests"]
+ A1["version / capability queries"]
+ A2["client, device, subdevice"]
+ A3["memory alloc + mmap descriptors"]
+ A4["vaspace + page-table contracts"]
+ A5["channels, CE caps, events"]
+ end
+ subgraph EMU["GSP-RM emulation"]
+ B1["dispatcher by function ID"]
+ B2["special-cased handlers - real values"]
+ B3["generic fallback - success + zeroed params"]
+ B4["object table - handles, params echoed"]
+ B5["reply fidelity checks"]
+ end
+ A1 --> B1
+ A2 --> B1
+ A3 --> B1
+ A4 --> B1
+ A5 --> B1
+ B1 --> B2
+ B1 --> B3
+ B2 --> B4
+ B3 --> B5
+ B4 --> B5
+
+ classDef done fill:#d5f5d5,stroke:#1e7a1e
+ classDef wip fill:#fff3cd,stroke:#9a7b1a
+ class B1,B2,B4 done
+ class B3,B5 wip
+```
+
+## 3. The memory model
+
+Guest work refers to GPU virtual addresses. The module must know which physical
+pages they denote — for semaphores, push buffers, page trees, and (in the end
+state) for programming the real GPU's MMU so hardware walks the same pages.
+
+```mermaid
+flowchart LR
+ subgraph G["Guest"]
+ GVA["GPU virtual addresses<br/>client objects and mappings"]
+ GPT["Guest page tables<br/>incl. server-owned ranges"]
+ SYS["Guest system memory"]
+ VRS["VRAM objects - BAR1 aperture view"]
+ end
+
+ subgraph MM["open-iov-vfio.ko memory model"]
+ DISC["Discovery<br/>structure walk + content scan + pins"]
+ PMAP["VA to PA map<br/>page-granular, merge rebuilds"]
+ SHAD["BAR1 VRAM shadow"]
+ PIN["Pin tables<br/>self-discovered + authoritative injected"]
+ end
+
+ subgraph H["Host / hardware"]
+ DMA["VFIO DMA - read and write guest pages"]
+ RGMMU["Real GMMU tables<br/>same pages, real hardware walks"]
+ end
+
+ GVA --> GPT --> SYS
+ GVA --> VRS
+ VRS --> SHAD
+ SYS --> DMA
+ GPT --> DISC
+ DISC --> PMAP
+ PIN --> PMAP
+ PMAP --> DMA
+ PMAP --> RGMMU
+ SYS -.->|"pinned and mapped once"| RGMMU
+
+ classDef done fill:#d5f5d5,stroke:#1e7a1e
+ classDef wip fill:#fff3cd,stroke:#9a7b1a
+ classDef plan fill:#f8d7da,stroke:#9a1a1a
+ class DISC,PMAP,DMA done
+ class PIN,SHAD wip
+ class RGMMU plan
+```
+
+The two contracts that make this possible without guest cooperation:
+
+* Channel memory descriptors the guest creates already carry **physical
+ addresses** (instance memory, method buffers, USERD, RAMFC, notifiers).
+* The server-owned page-table contract (`COPY_SERVER_RESERVED_PDES`) gives the
+ page-table level physical addresses; the leaves are ours to populate.
+
+The remaining work is to derive every mapping from these protocol objects as
+they are created, so a clean boot needs no external tooling.
+
+## 4. The work path
+
+### 4.1 Today: software-executed bridge
+
+```mermaid
+flowchart LR
+ A["Guest push appended<br/>GPFIFO entry + doorbell"] --> B["Channel pump reads entry"]
+ B --> C{"Entry readable?"}
+ C -->|yes| D["Decode method stream"]
+ C -->|no| E["Rescue / reconstruct the page<br/>or resolve via pins"]
+ E --> D
+ D --> F["Execute understood work in software<br/>memsets, copies, semaphore releases"]
+ F --> G["Write releases and completions<br/>into guest memory"]
+ G --> H["Guest trackers advance"]
+
+ D -.->|"planned"| I["Forward to a real engine"]
+
+ classDef done fill:#d5f5d5,stroke:#1e7a1e
+ classDef wip fill:#fff3cd,stroke:#9a7b1a
+ classDef plan fill:#f8d7da,stroke:#9a1a1a
+ class A,B,D done
+ class E,F,G,H wip
+ class I plan
+```
+
+This is a correctness bridge: it advances synchronization and memory
+operations, and it is why the guest's initialization walks as far as it does
+today. It cannot execute kernels.
+
+### 4.2 End state: forwarded work
+
+```mermaid
+sequenceDiagram
+ autonumber
+ participant GU as Guest GPFIFO + doorbell
+ participant P as Channel pump
+ participant D as Decoder
+ participant M as Memory model
+ participant B as Execution backend
+ participant E as GPU engines
+ participant S as Guest trackers
+
+ GU->>P: doorbell / put pointer advance
+ P->>D: read push entry
+ alt entry unreadable
+ D->>M: resolve or reconstruct the page
+ end
+ D->>D: parse the method stream
+ alt pure state or notification
+ D->>M: apply release, membar, fence directly
+ else real engine work
+ D->>B: translated submission
+ B->>E: real push stream on a real channel
+ E-->>B: completion or fault from hardware
+ B->>M: completion written into guest memory
+ end
+ M-->>S: tracker values advance
+```
+
+Fast paths (semaphores, fences, some memops) stay in the emulation in both
+phases: they are microseconds in hardware and lose value by round-tripping
+through the real GPU.
+
+### 4.3 How the GPU reaches runtime and executes work
+
+```mermaid
+flowchart TB
+ subgraph BOOT["Guest view - emulated"]
+ A["Probe"] --> B["GSP boot stages"]
+ B --> C["GSP-RM init"]
+ C --> D["Objects, vaspaces, channels"]
+ D --> E["UVM register + page trees"]
+ E --> F["Context created, allocations mapped"]
+ F --> G["Pushes submitted"]
+ end
+ subgraph REAL["Backend view - real"]
+ R1["Module binds GPU, boots real GSP"] --> R2["Real RM client: device, vaspaces"]
+ R2 --> R3["Real channels and page tables"]
+ R3 --> R4["Guest VA mapped to same pages"]
+ R4 --> R5["Real submissions scheduled by GSP"]
+ R5 --> R6["Completion and faults relayed to guest"]
+ end
+ G -.->|"translation [planned]"| R4
+
+ classDef done fill:#d5f5d5,stroke:#1e7a1e
+ classDef plan fill:#f8d7da,stroke:#9a1a1a
+ class A,B,C,D,E done
+ class F,G wip
+ class R1,R2,R3,R4,R5,R6 plan
+```
+
+## 5. Scheduling and multi-tenancy
+
+There is no hardware partitioning (no SR-IOV) on this GPU generation, so
+tenancy is a **logical VF model**: each VM holds one mdev instance, and the
+module gives each instance its own GSP-RM session, channels, page tables and
+scheduling policy. “Per-VF GSP emulation” therefore means per-mdev emulation
+context plus a per-mdev real GSP client in the backend.
+
+```mermaid
+flowchart TB
+ subgraph TEN["Tenants"]
+ T1["VM A - mdev 1"]
+ T2["VM B - mdev 2"]
+ T3["VM C - mdev N"]
+ end
+ subgraph MOD["open-iov-vfio.ko"]
+ Q1["Emulation context 1<br/>BARs, queues, counters"]
+ Q2["Emulation context 2"]
+ Q3["Emulation context N"]
+ POL["Fairness policy [planned]<br/>time slices, quotas, priority, burst"]
+ C1["Real GSP client 1 [planned]"]
+ C2["Real GSP client 2 [planned]"]
+ C3["Real GSP client N [planned]"]
+ end
+ subgraph GPU["Physical GPU"]
+ SCHED["Real GSP scheduler<br/>engine contexts and runlists"]
+ GRC["GR contexts"]
+ CEC["CE contexts"]
+ end
+ T1 --> Q1 --> C1
+ T2 --> Q2 --> C2
+ T3 --> Q3 --> C3
+ C1 --> POL
+ C2 --> POL
+ C3 --> POL
+ POL --> SCHED
+ SCHED --> GRC
+ SCHED --> CEC
+
+ classDef done fill:#d5f5d5,stroke:#1e7a1e
+ classDef wip fill:#fff3cd,stroke:#9a7b1a
+ classDef plan fill:#f8d7da,stroke:#9a1a1a
+ class Q1,Q2,Q3 done
+ class C1,C2,C3,POL,SCHED plan
+```
+
+**How multi-tenant performance works (and is intended to):**
+
+* Each tenant's *state* operations complete in the module on a fast path
+ (microseconds, no hardware round trip).
+* Each tenant's *real work* runs on the physical engines at native speed; the
+ virtualization overhead is the submission translation plus completion relay,
+ which amortizes with work size.
+* Fairness is a module-side policy on top of the firmware scheduler (time
+ slices, quotas). It must be measured, not assumed; the measurement harness is
+ planned with the performance ladder in
+ [`../05-operations/01-performance-correctness-security.md`](../05-operations/01-performance-correctness-security.md).
+
+## 6. Isolation: what is guaranteed, at which level
+
+```mermaid
+flowchart TB
+ H1["Hardware IOMMU / SMMU<br/>per-VM DMA address spaces<br/>guarantee: memory isolation between VMs"]
+ H2["GPU MMU contexts - per client page tables [planned]<br/>guarantee: one tenant's engine work cannot address another's memory"]
+ V1["VFIO mdev boundary<br/>all device access mediated; config/BAR filtering<br/>guarantee: guest gets only its own device instance"]
+ M1["Module state separation<br/>per-mdev emulation context, queues, pins<br/>guarantee: no cross-tenant state pointers"]
+ P1["Policy and audit [planned]<br/>fairness, rate limits, side-channel review<br/>guarantee: performance isolation, adversarial robustness"]
+
+ H1 --> V1 --> M1
+ H2 --> P1
+ M1 --> P1
+```
+
+| Layer | Isolation strength | Status |
+|---|---|---|
+| IOMMU / SMMU between VMs | hardware-enforced | available today (hypervisor) |
+| KVM memory and vCPU boundary | hypervisor-enforced | available today |
+| VFIO mdev mediation (one device per tenant) | hypervisor-enforced | implemented |
+| Per-mdev module state and emulation context | software-enforced | implemented |
+| Per-client real page tables and GSP clients | software plus GPU MMU | planned |
+| Performance isolation and side channels | policy | planned, must be measured |
+
+The honest statement: **memory isolation is guaranteed by hardware and the
+hypervisor today**; *engine-level context separation and performance isolation*
+are the parts this project must still build and audit.
+
+## 7. How the host kernel schedules the module
+
+```mermaid
+flowchart LR
+ subgraph CTX["Caller contexts"]
+ QEMU["QEMU / VFIO ioctls and mmap"]
+ VCPU["vCPU exits on trapped MMIO"]
+ FW["Physical GSP messages - backend"]
+ USR["Debug interfaces - dev only"]
+ end
+ subgraph KERN["Linux execution contexts in the module"]
+ VF["VFIO callbacks"]
+ IRQ["IRQ handlers - physical MSI-X, eventfd signals"]
+ WQ["Workqueues - RPC, cleanup, bottom halves"]
+ KT["kthreads - GSP emulation per mdev, channel service"]
+ TM["Timers and poll loops - doorbells, housekeeping"]
+ end
+ QEMU --> VF --> WQ
+ VCPU --> VF
+ FW --> IRQ --> WQ
+ WQ --> KT
+ KT --> TM
+ KT --> WQ
+
+ classDef done fill:#d5f5d5,stroke:#1e7a1e
+ classDef plan fill:#f8d7da,stroke:#9a1a1a
+ class VF,IRQ,WQ,KT done
+ class TM plan
+```
+
+Design rules that keep this healthy: bounded stack frames (audited), no long
+scans while holding service locks, blocking work in kthreads not in VFIO
+callbacks, and never tearing down the module while a VM holds the device.
+
+## 8. Hypervisor integration: mdev, VFIO, KVM
+
+```mermaid
+sequenceDiagram
+ autonumber
+ participant ORCH as Orchestrator / admin
+ participant SYS as Module + mdev sysfs
+ participant QEMU as QEMU with vfio-pci
+ participant KVM as KVM
+ participant GUEST as Guest kernel
+
+ ORCH->>SYS: create mdev - type nvidia-nvidia-vgpu-turing, UUID
+ SYS-->>ORCH: /sys/bus/mdev/devices/uuid ready
+ QEMU->>SYS: open VFIO device, query regions and IRQ info
+ QEMU->>SYS: mmap BAR shadows - region index to vaddr
+ QEMU->>SYS: VFIO_DEVICE_SET_IRQS - eventfd per MSI-X vector
+ QEMU->>KVM: register irqfd for guest injection
+ QEMU->>KVM: map guest RAM - IOMMU / IOMMUFD
+ GUEST->>QEMU: MMIO access to the virtual BAR0
+ QEMU->>SYS: region read or write, or direct shadow access
+ SYS-->>GUEST: MSI-X delivered via eventfd to irqfd to vCPU
+ ORCH->>SYS: destroy mdev after VM stops - reverse order
+```
+
+Key properties: the mdev is the *device instance*; VFIO is the *transport*
+(region access, IRQ signalling, DMA bridging); KVM is the *delivery* (irqfd,
+MMIO exits, vCPU scheduling). The module registers an emulated IOMMU device so
+VFIO creates the group; guest memory access flows through the VFIO DMA
+interface. Full detail in
+[`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md).
+
+## 9. Component detail: BARs, engines, channels, GMMU
+
+```mermaid
+flowchart LR
+ subgraph GV["Guest-visible device"]
+ CFG["PCI config space<br/>TU104 identity, BAR sizes, MSI-X caps"]
+ B0["BAR0 - 16 MB registers<br/>doorbell, mailbox, PBA"]
+ B1["BAR1 - 256 MB VRAM aperture"]
+ B3["BAR3 - 32 MB instance / PRAMIN window"]
+ MX["MSI-X - 32 vectors"]
+ end
+ subgraph MODS["Module shadow state"]
+ C0["Filtered config writes"]
+ S0["BAR0 shadow - boot state, doorbells"]
+ S1["BAR1 shadow - VRAM backing [wip]"]
+ S3["BAR3 shadow - instance memory"]
+ IR["Per-vector eventfd injection"]
+ end
+ subgraph CH["Channel machinery"]
+ GPF["GPFIFO - push entries"]
+ USR["USERD + doorbell tokens"]
+ RF["RAMFC + instance memory"]
+ MB["Method buffers"]
+ NOT["Notifiers and error buffers"]
+ end
+ subgraph EN["Engines"]
+ GR["GR - compute, grid launches"]
+ CE["CE pair - copies and UVM memops"]
+ DEC["DEC - video decode [planned]"]
+ ENC["ENC - video encode [planned]"]
+ RL["Runlist / TSG scheduling"]
+ end
+ subgraph MM["GMMU"]
+ HI["Page-directory hierarchy"]
+ LEAF["Leaf PTEs"]
+ SRV["Server-owned ranges"]
+ end
+ CFG --> C0
+ B0 --> S0 --> GPF
+ B1 --> S1
+ B3 --> S3 --> RF
+ MX --> IR
+ GPF --> USR
+ GPF --> RF
+ GPF --> MB
+ GPF --> NOT
+ USR --> GR
+ USR --> CE
+ CE --> RL
+ GR --> RL
+ RL --> ENC
+ RL --> DEC
+ S1 --> HI
+ S3 --> HI
+ HI --> LEAF
+ HI --> SRV
+
+ classDef done fill:#d5f5d5,stroke:#1e7a1e
+ classDef wip fill:#fff3cd,stroke:#9a7b1a
+ classDef plan fill:#f8d7da,stroke:#9a1a1a
+ class CFG,C0,B0,S0,B3,S3,MX,IR,GPF,USR,RF,MB,NOT,GR,CE,RL done
+ class B1,S1,HI,LEAF,SRV wip
+ class DEC,ENC plan
+```
+
+## 10. Portability: versions, hardware, hypervisors
+
+```mermaid
+flowchart LR
+ A["Firmware / driver image"] -->|"CRC32 + SHA-256"| B["Firmware analyzer"]
+ B --> C["Version tables<br/>and YAML profiles"]
+ C --> D["Dynamic RPC dispatcher"]
+ C --> E["Hardware abstraction profiles"]
+ D --> F["Version-keyed handlers"]
+ E --> G["Architecture tables<br/>engine inventory, GMMU formats"]
+ F --> H["GSP-RM emulation"]
+ G --> H
+ H --> I["Same guest interface"]
+ J["New driver version<br/>or new GPU"] --> A
+ J --> K["Measurement + A/B against host oracle"]
+ K --> C
+
+ classDef done fill:#d5f5d5,stroke:#1e7a1e
+ classDef wip fill:#fff3cd,stroke:#9a7b1a
+ classDef plan fill:#f8d7da,stroke:#9a1a1a
+ class B done
+ class C,D,F done
+ class E,G,H wip
+ class K plan
+```
+
+The *interface* is the product: RPC/control inventories are measured, not
+guessed; new versions enter by adding a profile and running the A/B workflow
+(see [`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md)).
+The execution backend is similarly version-keyed: the real GSP protocol is
+spoken as a client, with the module carrying the small set of host-side calls
+it needs.
+
+## 11. Performance: distance to bare metal
+
+```mermaid
+flowchart LR
+ subgraph FAST["Emulated fast path - state only"]
+ A["Release, fence, membar"] --> B["Applied in the module"]
+ B --> C["microseconds; no GPU round trip;<br/>faster than hardware would be"]
+ end
+ subgraph DATA["Forwarded work path"]
+ D["Guest submission"] --> E["Decode + translate + submit"]
+ E --> F["Engine executes at native speed"]
+ F --> G["Completion relay"]
+ end
+ C -.-> H["Cost model: overhead is per submission,<br/>not per unit of work"]
+ G -.-> H
+ H -.-> I["Large copies and kernels amortize to near native;<br/>launch-bound workloads pay the translation"]
+
+ classDef done fill:#d5f5d5,stroke:#1e7a1e
+ classDef plan fill:#f8d7da,stroke:#9a1a1a
+ class A,B,C done
+ class D,E,F,G,H,I plan
+```
+
+| Aspect | Expectation | Status |
+|---|---|---|
+| State/synchronization latency | low microseconds, no round trip | measured in debug logs |
+| Copy bandwidth (large) | near native; DMA is DMA | not measured yet |
+| Kernel launch overhead | translation cost per submission; amortized | not measured yet |
+| Multi-tenant fairness | policy-controlled, measurable | policy planned |
+| Bare-metal parity claim | only after the measurement ladder is green | see operations doc |
+
+We do not claim parity today, and the docs say so in plain words: the honest
+claim is "initialization works; execution is in progress".
+
+## 12. Implementation and test status
+
+How far each component is from the finished system. "Tested" means exercised
+against a stock guest driver with evidence in the repository.
+
+| Component | Implemented | Tested | Next step |
+|---|---|---|---|
+| mdev + VFIO device, config space, BARs, MSI-X | yes | yes | keep regression-free |
+| GSP boot emulation (all stages) | yes | yes | unchanged under backend |
+| GSP-RM message queues + RPC + controls | yes | yes | reply fidelity for remaining consumed fields |
+| Object/alloc model (handles, echoes) | yes | yes | server-side object tracking where consumed |
+| Engine/class topology (incl. async copy engine) | yes | yes | version profiles |
+| Guest memory access + VA-to-PA discovery | partial | yes | protocol-derived discovery, no external pins |
+| Push decode + semaphore completion | partial | yes | complete GR/compute encodings |
+| Software execution bridge (memsets, copies, releases) | partial | yes | remains bridge only |
+| Page-table population for server-owned ranges | no | no | design + validator, then implement |
+| Execution backend (own GPU + real GSP) | no | no | device ownership, real GSP bring-up, RM client |
+| Real work forwarding (GR and CE) | no | no | after backend bring-up |
+| Fault semantics (buffers, replay, cancel) | no | no | after forwarding |
+| Multi-tenant fairness policy | no | no | after multi-tenant forwarding |
+| Version matrix (2+ driver families) | partial | partial | per-version profiles + A/B |
+| Windows guest | no | no | after Linux execution path |
+| Non-KVM transports (Xen, Argo, OpenXT) | no | no | transport interface + backend |
+| aarch64 hosts (Grace, GB10) | no | no | architecture-neutral memory model |
+| Debug interfaces (dev only) | yes | yes | gate destructive ones; production build |
+
+## 13. How the unmodified-driver guarantee is achieved
+
+* **Emulate what the guest observes, forward what it uses.** Every interface
+ the stock driver touches is answered by the module; every operation that must
+ be fast goes to real hardware. The guest never learns it is virtualized for
+ the CUDA/initialization path — it sees a TU104 whose GSP answers correctly.
+* **Reply fidelity, not just status.** A reply whose fields the driver consumes
+ is only considered done when the values are coherent (status-only, then
+ coherent zeroed params, then coherent consumed fields, then exact values).
+ This is the mechanism that makes arbitrary graphics and CUDA stacks work
+ without per-application code.
+* **No guest code, no paravirtual enlightenment, no patches.** The guest may
+ install whatever software it likes; the module presents physical hardware.
+ The planned execution backend keeps this property: the guest's work is
+ translated, never the guest's driver.
+
+## See also
+
+- [`01-system-overview.md`](01-system-overview.md) - principles and layers in prose
+- [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) - the module, memory model, build contracts
+- [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md) - the chosen execution backend and its decision record
+- [`../04-implementation/01-known-knowns-and-unknowns.md`](../04-implementation/01-known-knowns-and-unknowns.md) - gaps and plans
+- [`../05-operations/01-performance-correctness-security.md`](../05-operations/01-performance-correctness-security.md) - measurement ladder, threat model
diff --git a/docs/03-architecture/10-interaction-and-scheduling.md b/docs/03-architecture/10-interaction-and-scheduling.md
new file mode 100644
index 0000000..3608dc5
--- /dev/null
+++ b/docs/03-architecture/10-interaction-and-scheduling.md
@@ -0,0 +1,337 @@
+# Interaction and Scheduling: Cross-Component Detail
+
+> **What this document is.** The zoom lens companion to
+> [`09-full-system-picture.md`](09-full-system-picture.md): every
+> cross-component interaction in the system — the exact call paths, queues,
+> doorbells, interrupts, locks and orderings — plus the multi-tenant scheduling
+> model and the timing each interaction is designed to hit. Diagrams are
+> Mermaid; status tags follow the same legend as 09.
+
+Status legend: `[done]` implemented and exercised; `[wip]` in progress;
+`[planned]` designed, not built; hardware/firmware exists in the physical
+world.
+
+## 1. Interaction inventory
+
+Every place two components touch, the mechanism, the latency class it belongs
+to, and where it stands.
+
+| Interaction | Path and mechanism | Latency class | Status |
+|---|---|---|---|
+| Guest MMIO write to a mapped BAR | PCIe write lands directly in the module's shadow region (no trap) | tens of ns | [done] |
+| Guest doorbell signal (boot, mailbox, channel) | write into BAR0 shadow, observed by the polling service | steering only | [done] |
+| Guest RPC request | element placed in a guest-sysmem message queue, read by the module over VFIO DMA | 10s of us per round trip | [done] |
+| RPC reply + interrupt | module writes reply into the queue, signals eventfd, QEMU raises irqfd into KVM, MSI-X lands in the guest | us | [done] |
+| Push-buffer work | guest appends GPFIFO entries; the per-channel pump reads, decodes and applies them | per entry, us to low ms | [wip] |
+| Semaphore release to guest memory | module writes the payload word into the resolved guest page | us | [done] |
+| VA-to-PA translation discovery | in-module structure walk, content scan, rescue, pins | us to seconds (scan bounded) | [wip] |
+| Translation internalization from protocol | derive mappings from RM/UVM objects as they are created | us | [planned] |
+| Real channel submission | translated push stream on a real channel, real doorbell | us + hardware | [planned] |
+| Physical GPU interrupt | real MSI-X into the module's IRQ handler, relayed to guest trackers | us | [planned] |
+| mdev create / destroy | sysfs plus VFIO device registration | ms | [done] |
+| Guest RAM mapping | VFIO / IOMMUFD maps guest pages for DMA | ms once | [done] |
+| Teardown | ordered stop: VM, device close, mdev remove, module unload | s | [done] |
+
+## 2. Control-plane call path: guest ioctl to completed reply
+
+The full path of one resource-manager control during initialization. Boot-stage
+register traffic follows the same shape but through the BAR0 shadow instead of
+the message queue.
+
+```mermaid
+sequenceDiagram
+ autonumber
+ participant GL as Guest libcuda
+ participant GK as Guest nvidia.ko RM
+ participant MQ as Message queue in guest sysmem
+ participant SB as BAR0 shadow doorbell
+ participant GE as GSP emulation thread
+ participant GRM as GSP-RM handlers
+ participant DMA as VFIO DMA bridge
+ participant EV as eventfd and irqfd
+ participant KVM as KVM
+
+ GL->>GK: ioctl NV_ESC_RM_CONTROL
+ GK->>MQ: build RPC element header and payload
+ GK->>SB: ring doorbell
+ GE->>SB: poll observes the doorbell
+ GE->>DMA: read element from guest sysmem
+ GE->>GRM: dispatch by function ID
+ GRM->>GRM: special-case handler or fallback, build reply
+ GRM->>DMA: write reply element into the queue
+ GE->>EV: signal completion
+ EV->>KVM: irqfd raises the vector
+ KVM-->>GK: MSI-X delivered
+ GK->>MQ: read reply and complete the ioctl
+ GL-->>GL: continue
+
+ Note over GE,GRM: Reply fidelity ladder applies here -<br/>status, params, consumed values, exact values.
+ Note over SB,GE: The poll loop is the steering wheel for the whole<br/>control plane; it must never block on long work.
+```
+
+## 3. Message-queue element lifecycle
+
+```mermaid
+stateDiagram-v2
+ direction LR
+ [*] --> Free
+ Free --> Filled: guest writes header and payload
+ Filled --> Signaled: doorbell rung
+ Signaled --> Read: module reads via DMA
+ Read --> Validated: sequence, length, bounds checked
+ Validated --> Dispatched: RPC function ID dispatched
+ Dispatched --> Replied: handler result written back
+ Replied --> Marked: sequence and status updated
+ Marked --> Interrupted: MSI-X signalled
+ Interrupted --> Consumed: guest reads and frees the slot
+ Consumed --> Free
+
+ note right of Read
+ Non-contiguous queue buffers are
+ translated through the queue page
+ table before any element is touched.
+ end note
+ note right of Validated
+ A malformed or stale element is
+ rejected without disturbing the
+ queue; sequence matching is strict.
+ end note
+```
+
+## 4. The channel pump: per-channel work loop
+
+One instance of this loop runs for every channel of every mdev. It is the
+meeting point of guest submissions, translation, decoding and completion.
+
+```mermaid
+stateDiagram-v2
+ direction LR
+ [*] --> Idle
+ Idle --> EntrySeen: put pointer advanced
+ EntrySeen --> Reading: fetch the GPFIFO entry
+ Reading --> Readable: translation exists
+ Reading --> Unreadable: no translation yet
+ Unreadable --> Rescue: rescue or resolve the page
+ Rescue --> Decoding: page reconstructed
+ Rescue --> Skipped: attempts exhausted or policy skip
+ Readable --> Decoding: parse method stream
+ Decoding --> Applying: execute understood operations
+ Applying --> Completing: releases, notifiers, counters
+ Completing --> Advancing: consume the entry, advance get
+ Skipped --> Advancing
+ Advancing --> Idle: next pass
+
+ note right of Rescue
+ Bounded scans, hot-page first, with
+ retry budgets. A skipped entry is a
+ recorded event, never a silent stall.
+ end note
+ note right of Applying
+ State and notification operations are
+ applied from the emulation. Real engine
+ work is forwarded [planned].
+ end note
+```
+
+## 5. Concurrency and ownership
+
+Which context touches which structure, and under whose discipline.
+
+```mermaid
+flowchart LR
+ subgraph CTX["Execution contexts"]
+ ST["Service kthread<br/>pump, releases, injections"]
+ GT["GSP emulation kthread<br/>per mdev"]
+ VC["VFIO callbacks<br/>QEMU ioctl and mmap context"]
+ WQ["Workqueues<br/>RPC, cleanup, bottom halves"]
+ IR["IRQ handlers<br/>physical MSI-X [planned]"]
+ end
+
+ subgraph DATA["Shared structures"]
+ CT["Channel table<br/>handles, rings, notifiers"]
+ MAP["VA-to-PA map<br/>plus hot-page set"]
+ PIN["Pin tables<br/>discovered plus injected"]
+ PEND["Pending release queue"]
+ CNT["Completion counter table"]
+ MQS["Message queue shadow"]
+ CFG["Config and BAR shadows"]
+ end
+
+ ST --> CT
+ ST --> MAP
+ ST --> PIN
+ ST --> PEND
+ ST --> CNT
+ GT --> MQS
+ GT --> CT
+ GT --> CFG
+ VC --> CFG
+ WQ --> GT
+ IR --> WQ
+
+ classDef done fill:#d5f5d5,stroke:#1e7a1e
+ classDef wip fill:#fff3cd,stroke:#9a7b1a
+ classDef plan fill:#f8d7da,stroke:#9a1a1a
+ class ST,GT,VC,WQ done
+ class CT,MAP,PIN,PEND,CNT,MQS,CFG wip
+ class IR plan
+```
+
+Rules the design enforces:
+
+* One writer per structure. Injections are queued and drained by the service;
+ self-discovery never overwrites authoritative injected data.
+* Long scans are sliced and preemptible; the pump and release paths are never
+ blocked behind a full-RAM pass.
+* VFIO callbacks do only bounded work; blocking work lives in kthreads.
+* Every bounded budget (stack frames, retry counts, scan slices) is audited at
+ build time.
+
+## 6. Timing model and budgets
+
+| Interaction | Target | Why it matters |
+|---|---|---|
+| Guest MMIO to shadow | direct memory write | zero trap cost on the data path |
+| Doorbell to dispatch | sub-ms | steers every control exchange |
+| RPC round trip during init | 10s of us | hundreds of exchanges per boot |
+| Release apply | us | trackers poll at high frequency |
+| Push entry decode | us per entry | thousands of entries per context |
+| Rescue of an unreadable entry | bounded ms | only on translation misses |
+| MSI-X to guest | us | completion latency floor |
+| Real submission + completion relay | hardware + us | the forwarded-work path [planned] |
+
+The rule of thumb the architecture is built around: **state operations are
+cheaper in the module than they would be on hardware; real work belongs on
+hardware and pays only submission and relay.**
+
+## 7. Multi-tenant scheduling model
+
+The target model: per-tenant emulation contexts and real GSP clients, with
+fairness policy in the module on top of the firmware scheduler.
+
+```mermaid
+sequenceDiagram
+ autonumber
+ participant P as Module policy [planned]
+ participant A as Tenant A client
+ participant B as Tenant B client
+ participant S as Real GSP scheduler
+ participant E as Engines
+
+ Note over P,E: State operations never enter this loop -<br/>they complete in the module on the fast path.
+ P->>A: grant slice or quota
+ A->>S: submissions for tenant A
+ S->>E: tenant A contexts
+ E-->>S: completions for tenant A
+ S-->>A: completions relayed
+ P->>B: switch to tenant B
+ B->>S: submissions for tenant B
+ S->>E: tenant B contexts
+ E-->>S: completions for tenant B
+ S-->>B: completions relayed
+ Note over P: Policy inputs - priority, quota, burst,<br/>per-tenant accounting. Measured, not assumed.
+```
+
+Fairness and performance isolation properties to be measured once forwarding
+exists: per-tenant completion accounting, slice enforcement, starvation
+freedom, and the overhead each tenant pays for the other's activity.
+
+## 8. Signal paths: doorbells and interrupts, both directions
+
+```mermaid
+flowchart TB
+ subgraph IN["Guest to module"]
+ A["Guest writes doorbell in BAR0 shadow"] --> B["Poll loop observes"]
+ B --> C["Fetch and dispatch"]
+ end
+ subgraph OUT["Module to guest"]
+ D["Module has a completion"] --> E["eventfd signal"]
+ E --> F["QEMU irqfd"]
+ F --> G["KVM injects MSI-X vector"]
+ G --> H["Guest ISR runs"]
+ end
+ subgraph PHY["Module and physical GPU [planned]"]
+ I["Real channel doorbell"] --> J["GPU executes"]
+ J --> K["Physical MSI-X"]
+ K --> L["Module IRQ handler"]
+ L --> M["Relay to guest or hardware semaphore"]
+ end
+ C -.-> I
+ M -.-> E
+
+ classDef done fill:#d5f5d5,stroke:#1e7a1e
+ classDef plan fill:#f8d7da,stroke:#9a1a1a
+ class A,B,C,D,E,F,G,H done
+ class I,J,K,L,M plan
+```
+
+## 9. Lifecycle and teardown ordering
+
+The ordering is not cosmetic: the module must never be unloaded while a VM
+holds the device, and the mdev teardown path must not run long scans or
+unbounded waits.
+
+```mermaid
+sequenceDiagram
+ autonumber
+ participant ORCH as Orchestrator
+ participant QEMU as QEMU
+ participant VFIO as VFIO and module
+ participant SYS as mdev sysfs
+
+ ORCH->>QEMU: stop the VM
+ QEMU->>VFIO: close device, release regions and IRQs
+ VFIO-->>QEMU: resources released
+ ORCH->>SYS: remove mdev
+ SYS->>VFIO: remove callback - bounded cleanup
+ VFIO-->>SYS: device gone
+ ORCH->>VFIO: unload the module
+
+ alt guest hangs or faults
+ ORCH->>QEMU: reset or destroy the VM first
+ Note over VFIO: Never unload while a VM holds the device;<br/>forensics come from pstore and bounded logs.
+ end
+```
+
+## 10. Version negotiation across components
+
+```mermaid
+sequenceDiagram
+ autonumber
+ participant GD as Guest driver
+ participant EMU as GSP-RM emulation
+ participant VT as Version profile
+ participant BE as Execution backend [planned]
+ participant FW as Real GSP firmware [planned]
+
+ GD->>EMU: version and capability queries
+ EMU->>VT: select profile for this driver family
+ VT-->>EMU: RPC surface, controls, caps
+ EMU-->>GD: negotiated replies
+ Note over EMU,VT: New driver versions enter as profiles -<br/>measured against the host oracle, never guessed.
+ BE->>FW: host-side protocol version negotiation
+ FW-->>BE: firmware capabilities
+ Note over BE,FW: The backend carries its own small,<br/>version-keyed host-side protocol surface.
+```
+
+## 11. Data-structure map
+
+| Structure | Represents | Written by | Consumed by | Status |
+|---|---|---|---|---|
+| Message queue + shadow | guest RM RPC transport | guest driver; emulation replies | emulation thread; guest RM | [done] |
+| BAR0 shadow | register interface (boot, doorbells, PBA) | guest MMIO | emulation poll; pump | [done] |
+| Channel table | per-channel ring, get/put, notifier, engine | guest objects; pump | pump; release path | [wip] |
+| GPFIFO entries | queued work per channel | guest submissions | pump | [wip] |
+| USERD / doorbell tokens | work signalling | guest | module | [wip] |
+| VA-to-PA map | translations with hot-page hints | discovery and scans | pump; release path; pins | [wip] |
+| Pin tables | authoritative injected translations | debug tooling; later protocol discovery | lookup fast path | [wip] |
+| Pending release queue | releases awaiting a translation | decoder | service drain | [done] |
+| Completion counters | per-channel progress published to guest pages | service | guest trackers | [done] |
+| Real channels and page tables | forwarding state | backend | real GPU | [planned] |
+
+## See also
+
+- [`09-full-system-picture.md`](09-full-system-picture.md) - the map this document zooms into
+- [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) - module internals and build contracts
+- [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md) - the execution backend and decision record
+- [`01-system-overview.md`](01-system-overview.md) - principles and layer map
diff --git a/docs/README.md b/docs/README.md
index 24b2a21..81f02c8 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -31,33 +31,41 @@ generations (Turing, Ampere, Hopper, Grace-Hopper, Grace-Blackwell, GB10, ...)
and many driver versions — without any of those ecosystems, operating systems,
drivers or devices having to change.
+```mermaid
+flowchart TB
+ subgraph VM["Guest VM - stock drivers, stock applications"]
+ A["CUDA / OpenGL / Vulkan applications"]
+ B["Userspace stack - libcuda, libcudart, libGLX, libvulkan"]
+ C["Stock nvidia.ko - RM core and UVM"]
+ A --> B --> C
+ end
+ subgraph HYP["Hypervisor - KVM and QEMU"]
+ D["VFIO mdev transport + IOMMU isolation"]
+ end
+ subgraph MOD["open-iov-vfio.ko - the only host component"]
+ E["Virtual device - BARs, config space, MSI-X"]
+ F["GSP boot + GSP-RM emulation"]
+ G["Memory model + work path"]
+ H["Execution backend - real GSP session"]
+ E --> F --> G --> H
+ end
+ C -->|"emulated PCIe"| D --> E
+ H --> I["Physical GPU - engines, GMMU, VRAM"]
+ classDef done fill:#d5f5d5,stroke:#1e7a1e
+ classDef wip fill:#fff3cd,stroke:#9a7b1a
+ classDef plan fill:#f8d7da,stroke:#9a1a1a
+ classDef hw fill:#e2e8f0,stroke:#4a5568
+ class E,F done
+ class G wip
+ class H plan
+ class I hw
```
- ┌─────────────────────────────────────────────┐
- │ GUEST VM │
- │ stock nvidia.ko + CUDA / OpenGL / Vulkan │
- └──────────────┬──────────────────────────────┘
- │ PCIe (emulated device)
- ┌───────────────────────────────┴───────────────────────────────┐
- │ HYPERVISOR (KVM/Xen) │
- │ VFIO / mdev transport, IOMMU isolation │
- └───────────────────────────────┬───────────────────────────────┘
- │
- ┌───────────────────────────────┴───────────────────────────────┐
- │ HOST: open-iov-vfio.ko (this project) │
- │ • virtual GPU device (BARs, config space, MSI-X) │
- │ • GSP / GSP-RM interface emulation (RPC, msgq, controls) │
- │ • memory model + GPU work forwarding to the physical GPU │
- └───────────────────────────────┬───────────────────────────────┘
- │
- ┌──────────────┴──────────────┐
- │ UNMODIFIED host nvidia.ko │
- │ (or GPU passthrough path) │
- └──────────────┬──────────────┘
- │
- ┌───────┴────────┐
- │ NVIDIA GPU │
- └────────────────┘
-```
+
+The full visual map — every component, status tag and interaction — is in
+[`03-architecture/09-full-system-picture.md`](03-architecture/09-full-system-picture.md)
+(end-to-end) and
+[`03-architecture/10-interaction-and-scheduling.md`](03-architecture/10-interaction-and-scheduling.md)
+(call paths, queues, interrupts, scheduling).
## 2. Why does this need to exist?
@@ -118,6 +126,11 @@ control; the chosen backend — a module-owned GPU running its real GSP firmware
as the resource-manager server — is specified in
[`03-architecture/08-physical-gpu-integration.md`](03-architecture/08-physical-gpu-integration.md).
+For the visual end-state map and the cross-component call paths, see
+[`03-architecture/09-full-system-picture.md`](03-architecture/09-full-system-picture.md)
+and
+[`03-architecture/10-interaction-and-scheduling.md`](03-architecture/10-interaction-and-scheduling.md).
+
## 4. How to read these docs (progressive disclosure)
Read in order if you are new. Jump directly if you know what you need.
@@ -135,21 +148,23 @@ Read in order if you are new. Jump directly if you know what you need.
| 2 | [The AI edge and the ecosystem](01-orientation/02-the-ai-edge-and-ecosystem.md) | everyone | 15 min |
| 3 | [Install and first run](02-getting-started/01-install-and-first-run.md) | users, evaluators | 20 min |
| 4 | [System overview](03-architecture/01-system-overview.md) | everyone technical | 20 min |
-| 5 | [Guest stack and CUDASW](03-architecture/02-guest-stack-and-cudasw.md) | developers | 25 min |
-| 6 | [GSP and GSP-RM](03-architecture/03-gsp-and-gsp-rm.md) | developers | 30 min |
-| 7 | [UVM](03-architecture/04-uvm.md) | developers | 25 min |
-| 8 | [GMMU, AMAPlib, HALs, runlist, GR](03-architecture/05-gmmu-amap-hal-gr.md) | developers | 25 min |
-| 9 | [Host module and hybrid model](03-architecture/06-host-module-and-hybrid-model.md) | developers | 25 min |
-| 10 | [Driver lifecycle field guide](03-architecture/07-driver-lifecycle-field-guide.md) | developers, debuggers | 30 min |
-| 11 | [Physical GPU integration](03-architecture/08-physical-gpu-integration.md) | developers, architects | 20 min |
-| 12 | [Known-knowns and known-unknowns](04-implementation/01-known-knowns-and-unknowns.md) | everyone technical | 25 min |
-| 13 | [Practical challenges and adversarial interoperability](04-implementation/02-practical-challenges-and-adversarial-interop.md) | developers, stakeholders | 20 min |
-| 14 | [Performance, correctness, security](05-operations/01-performance-correctness-security.md) | developers, auditors | 25 min |
-| 15 | [Debugging and maintenance workflows](05-operations/02-debugging-and-maintenance-workflows.md) | developers, maintainers | 25 min |
-| 16 | [Hardware and software matrix](06-portability/hardware-software-matrix.md) | developers, integrators | 20 min |
-| 17 | [Developer onboarding and standards](07-developers/onboarding-and-standards.md) | contributors | 20 min |
-| 18 | [HAAEEP and collaboration](08-ecosystem/haaeep-and-collaboration.md) | stakeholders, partners | 20 min |
-| 19 | [Glossary and FAQ](09-reference/glossary-and-faq.md) | everyone | reference |
+| 5 | [The full system picture](03-architecture/09-full-system-picture.md) | everyone technical | 30 min |
+| 6 | [Interaction and scheduling](03-architecture/10-interaction-and-scheduling.md) | developers | 30 min |
+| 7 | [Guest stack and CUDASW](03-architecture/02-guest-stack-and-cudasw.md) | developers | 25 min |
+| 8 | [GSP and GSP-RM](03-architecture/03-gsp-and-gsp-rm.md) | developers | 30 min |
+| 9 | [UVM](03-architecture/04-uvm.md) | developers | 25 min |
+| 10 | [GMMU, AMAPlib, HALs, runlist, GR](03-architecture/05-gmmu-amap-hal-gr.md) | developers | 25 min |
+| 11 | [Host module and hybrid model](03-architecture/06-host-module-and-hybrid-model.md) | developers | 25 min |
+| 12 | [Driver lifecycle field guide](03-architecture/07-driver-lifecycle-field-guide.md) | developers, debuggers | 30 min |
+| 13 | [Physical GPU integration](03-architecture/08-physical-gpu-integration.md) | developers, architects | 20 min |
+| 14 | [Known-knowns and known-unknowns](04-implementation/01-known-knowns-and-unknowns.md) | everyone technical | 25 min |
+| 15 | [Practical challenges and adversarial interoperability](04-implementation/02-practical-challenges-and-adversarial-interop.md) | developers, stakeholders | 20 min |
+| 16 | [Performance, correctness, security](05-operations/01-performance-correctness-security.md) | developers, auditors | 25 min |
+| 17 | [Debugging and maintenance workflows](05-operations/02-debugging-and-maintenance-workflows.md) | developers, maintainers | 25 min |
+| 18 | [Hardware and software matrix](06-portability/hardware-software-matrix.md) | developers, integrators | 20 min |
+| 19 | [Developer onboarding and standards](07-developers/onboarding-and-standards.md) | contributors | 20 min |
+| 20 | [HAAEEP and collaboration](08-ecosystem/haaeep-and-collaboration.md) | stakeholders, partners | 20 min |
+| 21 | [Glossary and FAQ](09-reference/glossary-and-faq.md) | everyone | reference |
Deep-dive material lives in [`../wiki/`](../wiki/README.md) (firmware
decompilation record, address reference, historical protocol notes). Wiki