diff options
Diffstat (limited to 'docs/03-architecture/10-interaction-and-scheduling.md')
| -rw-r--r-- | docs/03-architecture/10-interaction-and-scheduling.md | 337 |
1 files changed, 337 insertions, 0 deletions
diff --git a/docs/03-architecture/10-interaction-and-scheduling.md b/docs/03-architecture/10-interaction-and-scheduling.md new file mode 100644 index 0000000..3608dc5 --- /dev/null +++ b/docs/03-architecture/10-interaction-and-scheduling.md @@ -0,0 +1,337 @@ +# Interaction and Scheduling: Cross-Component Detail + +> **What this document is.** The zoom lens companion to +> [`09-full-system-picture.md`](09-full-system-picture.md): every +> cross-component interaction in the system — the exact call paths, queues, +> doorbells, interrupts, locks and orderings — plus the multi-tenant scheduling +> model and the timing each interaction is designed to hit. Diagrams are +> Mermaid; status tags follow the same legend as 09. + +Status legend: `[done]` implemented and exercised; `[wip]` in progress; +`[planned]` designed, not built; hardware/firmware exists in the physical +world. + +## 1. Interaction inventory + +Every place two components touch, the mechanism, the latency class it belongs +to, and where it stands. + +| Interaction | Path and mechanism | Latency class | Status | +|---|---|---|---| +| Guest MMIO write to a mapped BAR | PCIe write lands directly in the module's shadow region (no trap) | tens of ns | [done] | +| Guest doorbell signal (boot, mailbox, channel) | write into BAR0 shadow, observed by the polling service | steering only | [done] | +| Guest RPC request | element placed in a guest-sysmem message queue, read by the module over VFIO DMA | 10s of us per round trip | [done] | +| RPC reply + interrupt | module writes reply into the queue, signals eventfd, QEMU raises irqfd into KVM, MSI-X lands in the guest | us | [done] | +| Push-buffer work | guest appends GPFIFO entries; the per-channel pump reads, decodes and applies them | per entry, us to low ms | [wip] | +| Semaphore release to guest memory | module writes the payload word into the resolved guest page | us | [done] | +| VA-to-PA translation discovery | in-module structure walk, content scan, rescue, pins | us to seconds (scan bounded) | [wip] | +| Translation internalization from protocol | derive mappings from RM/UVM objects as they are created | us | [planned] | +| Real channel submission | translated push stream on a real channel, real doorbell | us + hardware | [planned] | +| Physical GPU interrupt | real MSI-X into the module's IRQ handler, relayed to guest trackers | us | [planned] | +| mdev create / destroy | sysfs plus VFIO device registration | ms | [done] | +| Guest RAM mapping | VFIO / IOMMUFD maps guest pages for DMA | ms once | [done] | +| Teardown | ordered stop: VM, device close, mdev remove, module unload | s | [done] | + +## 2. Control-plane call path: guest ioctl to completed reply + +The full path of one resource-manager control during initialization. Boot-stage +register traffic follows the same shape but through the BAR0 shadow instead of +the message queue. + +```mermaid +sequenceDiagram + autonumber + participant GL as Guest libcuda + participant GK as Guest nvidia.ko RM + participant MQ as Message queue in guest sysmem + participant SB as BAR0 shadow doorbell + participant GE as GSP emulation thread + participant GRM as GSP-RM handlers + participant DMA as VFIO DMA bridge + participant EV as eventfd and irqfd + participant KVM as KVM + + GL->>GK: ioctl NV_ESC_RM_CONTROL + GK->>MQ: build RPC element header and payload + GK->>SB: ring doorbell + GE->>SB: poll observes the doorbell + GE->>DMA: read element from guest sysmem + GE->>GRM: dispatch by function ID + GRM->>GRM: special-case handler or fallback, build reply + GRM->>DMA: write reply element into the queue + GE->>EV: signal completion + EV->>KVM: irqfd raises the vector + KVM-->>GK: MSI-X delivered + GK->>MQ: read reply and complete the ioctl + GL-->>GL: continue + + Note over GE,GRM: Reply fidelity ladder applies here -<br/>status, params, consumed values, exact values. + Note over SB,GE: The poll loop is the steering wheel for the whole<br/>control plane; it must never block on long work. +``` + +## 3. Message-queue element lifecycle + +```mermaid +stateDiagram-v2 + direction LR + [*] --> Free + Free --> Filled: guest writes header and payload + Filled --> Signaled: doorbell rung + Signaled --> Read: module reads via DMA + Read --> Validated: sequence, length, bounds checked + Validated --> Dispatched: RPC function ID dispatched + Dispatched --> Replied: handler result written back + Replied --> Marked: sequence and status updated + Marked --> Interrupted: MSI-X signalled + Interrupted --> Consumed: guest reads and frees the slot + Consumed --> Free + + note right of Read + Non-contiguous queue buffers are + translated through the queue page + table before any element is touched. + end note + note right of Validated + A malformed or stale element is + rejected without disturbing the + queue; sequence matching is strict. + end note +``` + +## 4. The channel pump: per-channel work loop + +One instance of this loop runs for every channel of every mdev. It is the +meeting point of guest submissions, translation, decoding and completion. + +```mermaid +stateDiagram-v2 + direction LR + [*] --> Idle + Idle --> EntrySeen: put pointer advanced + EntrySeen --> Reading: fetch the GPFIFO entry + Reading --> Readable: translation exists + Reading --> Unreadable: no translation yet + Unreadable --> Rescue: rescue or resolve the page + Rescue --> Decoding: page reconstructed + Rescue --> Skipped: attempts exhausted or policy skip + Readable --> Decoding: parse method stream + Decoding --> Applying: execute understood operations + Applying --> Completing: releases, notifiers, counters + Completing --> Advancing: consume the entry, advance get + Skipped --> Advancing + Advancing --> Idle: next pass + + note right of Rescue + Bounded scans, hot-page first, with + retry budgets. A skipped entry is a + recorded event, never a silent stall. + end note + note right of Applying + State and notification operations are + applied from the emulation. Real engine + work is forwarded [planned]. + end note +``` + +## 5. Concurrency and ownership + +Which context touches which structure, and under whose discipline. + +```mermaid +flowchart LR + subgraph CTX["Execution contexts"] + ST["Service kthread<br/>pump, releases, injections"] + GT["GSP emulation kthread<br/>per mdev"] + VC["VFIO callbacks<br/>QEMU ioctl and mmap context"] + WQ["Workqueues<br/>RPC, cleanup, bottom halves"] + IR["IRQ handlers<br/>physical MSI-X [planned]"] + end + + subgraph DATA["Shared structures"] + CT["Channel table<br/>handles, rings, notifiers"] + MAP["VA-to-PA map<br/>plus hot-page set"] + PIN["Pin tables<br/>discovered plus injected"] + PEND["Pending release queue"] + CNT["Completion counter table"] + MQS["Message queue shadow"] + CFG["Config and BAR shadows"] + end + + ST --> CT + ST --> MAP + ST --> PIN + ST --> PEND + ST --> CNT + GT --> MQS + GT --> CT + GT --> CFG + VC --> CFG + WQ --> GT + IR --> WQ + + classDef done fill:#d5f5d5,stroke:#1e7a1e + classDef wip fill:#fff3cd,stroke:#9a7b1a + classDef plan fill:#f8d7da,stroke:#9a1a1a + class ST,GT,VC,WQ done + class CT,MAP,PIN,PEND,CNT,MQS,CFG wip + class IR plan +``` + +Rules the design enforces: + +* One writer per structure. Injections are queued and drained by the service; + self-discovery never overwrites authoritative injected data. +* Long scans are sliced and preemptible; the pump and release paths are never + blocked behind a full-RAM pass. +* VFIO callbacks do only bounded work; blocking work lives in kthreads. +* Every bounded budget (stack frames, retry counts, scan slices) is audited at + build time. + +## 6. Timing model and budgets + +| Interaction | Target | Why it matters | +|---|---|---| +| Guest MMIO to shadow | direct memory write | zero trap cost on the data path | +| Doorbell to dispatch | sub-ms | steers every control exchange | +| RPC round trip during init | 10s of us | hundreds of exchanges per boot | +| Release apply | us | trackers poll at high frequency | +| Push entry decode | us per entry | thousands of entries per context | +| Rescue of an unreadable entry | bounded ms | only on translation misses | +| MSI-X to guest | us | completion latency floor | +| Real submission + completion relay | hardware + us | the forwarded-work path [planned] | + +The rule of thumb the architecture is built around: **state operations are +cheaper in the module than they would be on hardware; real work belongs on +hardware and pays only submission and relay.** + +## 7. Multi-tenant scheduling model + +The target model: per-tenant emulation contexts and real GSP clients, with +fairness policy in the module on top of the firmware scheduler. + +```mermaid +sequenceDiagram + autonumber + participant P as Module policy [planned] + participant A as Tenant A client + participant B as Tenant B client + participant S as Real GSP scheduler + participant E as Engines + + Note over P,E: State operations never enter this loop -<br/>they complete in the module on the fast path. + P->>A: grant slice or quota + A->>S: submissions for tenant A + S->>E: tenant A contexts + E-->>S: completions for tenant A + S-->>A: completions relayed + P->>B: switch to tenant B + B->>S: submissions for tenant B + S->>E: tenant B contexts + E-->>S: completions for tenant B + S-->>B: completions relayed + Note over P: Policy inputs - priority, quota, burst,<br/>per-tenant accounting. Measured, not assumed. +``` + +Fairness and performance isolation properties to be measured once forwarding +exists: per-tenant completion accounting, slice enforcement, starvation +freedom, and the overhead each tenant pays for the other's activity. + +## 8. Signal paths: doorbells and interrupts, both directions + +```mermaid +flowchart TB + subgraph IN["Guest to module"] + A["Guest writes doorbell in BAR0 shadow"] --> B["Poll loop observes"] + B --> C["Fetch and dispatch"] + end + subgraph OUT["Module to guest"] + D["Module has a completion"] --> E["eventfd signal"] + E --> F["QEMU irqfd"] + F --> G["KVM injects MSI-X vector"] + G --> H["Guest ISR runs"] + end + subgraph PHY["Module and physical GPU [planned]"] + I["Real channel doorbell"] --> J["GPU executes"] + J --> K["Physical MSI-X"] + K --> L["Module IRQ handler"] + L --> M["Relay to guest or hardware semaphore"] + end + C -.-> I + M -.-> E + + classDef done fill:#d5f5d5,stroke:#1e7a1e + classDef plan fill:#f8d7da,stroke:#9a1a1a + class A,B,C,D,E,F,G,H done + class I,J,K,L,M plan +``` + +## 9. Lifecycle and teardown ordering + +The ordering is not cosmetic: the module must never be unloaded while a VM +holds the device, and the mdev teardown path must not run long scans or +unbounded waits. + +```mermaid +sequenceDiagram + autonumber + participant ORCH as Orchestrator + participant QEMU as QEMU + participant VFIO as VFIO and module + participant SYS as mdev sysfs + + ORCH->>QEMU: stop the VM + QEMU->>VFIO: close device, release regions and IRQs + VFIO-->>QEMU: resources released + ORCH->>SYS: remove mdev + SYS->>VFIO: remove callback - bounded cleanup + VFIO-->>SYS: device gone + ORCH->>VFIO: unload the module + + alt guest hangs or faults + ORCH->>QEMU: reset or destroy the VM first + Note over VFIO: Never unload while a VM holds the device;<br/>forensics come from pstore and bounded logs. + end +``` + +## 10. Version negotiation across components + +```mermaid +sequenceDiagram + autonumber + participant GD as Guest driver + participant EMU as GSP-RM emulation + participant VT as Version profile + participant BE as Execution backend [planned] + participant FW as Real GSP firmware [planned] + + GD->>EMU: version and capability queries + EMU->>VT: select profile for this driver family + VT-->>EMU: RPC surface, controls, caps + EMU-->>GD: negotiated replies + Note over EMU,VT: New driver versions enter as profiles -<br/>measured against the host oracle, never guessed. + BE->>FW: host-side protocol version negotiation + FW-->>BE: firmware capabilities + Note over BE,FW: The backend carries its own small,<br/>version-keyed host-side protocol surface. +``` + +## 11. Data-structure map + +| Structure | Represents | Written by | Consumed by | Status | +|---|---|---|---|---| +| Message queue + shadow | guest RM RPC transport | guest driver; emulation replies | emulation thread; guest RM | [done] | +| BAR0 shadow | register interface (boot, doorbells, PBA) | guest MMIO | emulation poll; pump | [done] | +| Channel table | per-channel ring, get/put, notifier, engine | guest objects; pump | pump; release path | [wip] | +| GPFIFO entries | queued work per channel | guest submissions | pump | [wip] | +| USERD / doorbell tokens | work signalling | guest | module | [wip] | +| VA-to-PA map | translations with hot-page hints | discovery and scans | pump; release path; pins | [wip] | +| Pin tables | authoritative injected translations | debug tooling; later protocol discovery | lookup fast path | [wip] | +| Pending release queue | releases awaiting a translation | decoder | service drain | [done] | +| Completion counters | per-channel progress published to guest pages | service | guest trackers | [done] | +| Real channels and page tables | forwarding state | backend | real GPU | [planned] | + +## See also + +- [`09-full-system-picture.md`](09-full-system-picture.md) - the map this document zooms into +- [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) - module internals and build contracts +- [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md) - the execution backend and decision record +- [`01-system-overview.md`](01-system-overview.md) - principles and layer map |