path: root/docs/03-architecture/10-interaction-and-scheduling.md

# Interaction and Scheduling: Cross-Component Detail

> **What this document is.** The zoom lens companion to
> [`09-full-system-picture.md`](09-full-system-picture.md): every
> cross-component interaction in the system — the exact call paths, queues,
> doorbells, interrupts, locks and orderings — plus the multi-tenant scheduling
> model and the timing each interaction is designed to hit. Diagrams are
> Mermaid; status tags follow the same legend as 09.

Status legend: `[done]` implemented and exercised; `[wip]` in progress;
`[planned]` designed, not built; hardware/firmware exists in the physical
world.

## 1. Interaction inventory

Every place two components touch, the mechanism, the latency class it belongs
to, and where it stands.

| Interaction | Path and mechanism | Latency class | Status |
|---|---|---|---|
| Guest MMIO write to a mapped BAR | PCIe write lands directly in the module's shadow region (no trap) | tens of ns | [done] |
| Guest doorbell signal (boot, mailbox, channel) | write into BAR0 shadow, observed by the polling service | steering only | [done] |
| Guest RPC request | element placed in a guest-sysmem message queue, read by the module over VFIO DMA | 10s of us per round trip | [done] |
| RPC reply + interrupt | module writes reply into the queue, signals eventfd, QEMU raises irqfd into KVM, MSI-X lands in the guest | us | [done] |
| Push-buffer work | guest appends GPFIFO entries; the per-channel pump reads, decodes and applies them | per entry, us to low ms | [wip] (CE copies forward default-off; GR entry fetch live, launch-descriptor install is the frontier — [`../04-implementation/03-execution-frontier.md`](../04-implementation/03-execution-frontier.md)) |
| Semaphore release to guest memory | module writes the payload word into the resolved guest page; a forwarded CE copy can have the engine author it | us | [done] (relay default off) |
| VA-to-PA translation discovery | in-module structure walk, content scan, rescue, pins | us to seconds (scan bounded) | [wip] |
| Translation internalization from protocol | derive mappings from RM/UVM objects as they are created | us | [planned] |
| Real channel submission | translated push stream on a real channel, real doorbell; the ladder's real submission is live | us + hardware | [wip] (guest forwarding default-off) |
| Physical GPU interrupt | real MSI-X into the module's IRQ handler, relayed to guest trackers; forwarded-engine fault observation/relay exists | us | [wip] (default off) |
| mdev create / destroy | sysfs plus VFIO device registration | ms | [done] |
| Guest RAM mapping | VFIO / IOMMUFD maps guest pages for DMA | ms once | [done] |
| Teardown | ordered stop: VM, device close, mdev remove, module unload | s | [done] |

## 2. Control-plane call path: guest ioctl to completed reply

The full path of one resource-manager control during initialization. Boot-stage
register traffic follows the same shape but through the BAR0 shadow instead of
the message queue.

```mermaid
sequenceDiagram
  autonumber
  participant GL as Guest libcuda
  participant GK as Guest nvidia.ko RM
  participant MQ as Message queue in guest sysmem
  participant SB as BAR0 shadow doorbell
  participant GE as GSP emulation thread
  participant GRM as GSP-RM handlers
  participant DMA as VFIO DMA bridge
  participant EV as eventfd and irqfd
  participant KVM as KVM

  GL->>GK: ioctl NV_ESC_RM_CONTROL
  GK->>MQ: build RPC element header and payload
  GK->>SB: ring doorbell
  GE->>SB: poll observes the doorbell
  GE->>DMA: read element from guest sysmem
  GE->>GRM: dispatch by function ID
  GRM->>GRM: special-case handler or fallback, build reply
  GRM->>DMA: write reply element into the queue
  GE->>EV: signal completion
  EV->>KVM: irqfd raises the vector
  KVM-->>GK: MSI-X delivered
  GK->>MQ: read reply and complete the ioctl
  GL-->>GL: continue

  Note over GE,GRM: Reply fidelity ladder applies here -<br/>status, params, consumed values, exact values.
  Note over SB,GE: The poll loop is the steering wheel for the whole<br/>control plane; it must never block on long work.
```

## 3. Message-queue element lifecycle

```mermaid
stateDiagram-v2
  direction LR
  [*] --> Free
  Free --> Filled: guest writes header and payload
  Filled --> Signaled: doorbell rung
  Signaled --> Read: module reads via DMA
  Read --> Validated: sequence, length, bounds checked
  Validated --> Dispatched: RPC function ID dispatched
  Dispatched --> Replied: handler result written back
  Replied --> Marked: sequence and status updated
  Marked --> Interrupted: MSI-X signalled
  Interrupted --> Consumed: guest reads and frees the slot
  Consumed --> Free

  note right of Read
    Non-contiguous queue buffers are
    translated through the queue page
    table before any element is touched.
  end note
  note right of Validated
    A malformed or stale element is
    rejected without disturbing the
    queue; sequence matching is strict.
  end note
```

## 4. The channel pump: per-channel work loop

One instance of this loop runs for every channel of every mdev. It is the
meeting point of guest submissions, translation, decoding and completion.

```mermaid
stateDiagram-v2
  direction LR
  [*] --> Idle
  Idle --> EntrySeen: put pointer advanced
  EntrySeen --> Reading: fetch the GPFIFO entry
  Reading --> Readable: translation exists
  Reading --> Unreadable: no translation yet
  Unreadable --> Rescue: rescue or resolve the page
  Rescue --> Decoding: page reconstructed
  Rescue --> Skipped: attempts exhausted or policy skip
  Readable --> Decoding: parse method stream
  Decoding --> Applying: execute understood operations
  Applying --> Completing: releases, notifiers, counters
  Completing --> Advancing: consume the entry, advance get
  Skipped --> Advancing
  Advancing --> Idle: next pass

  note right of Rescue
    Bounded scans, hot-page first, with
    retry budgets. A skipped entry is a
    recorded event, never a silent stall.
  end note
  note right of Applying
    State and notification operations are
    applied from the emulation. Real engine
    work is forwarded [wip/default-off].
  end note
```

## 5. Concurrency and ownership

Which context touches which structure, and under whose discipline.

```mermaid
flowchart LR
  subgraph CTX["Execution contexts"]
    ST["Service kthread<br/>pump, releases, injections"]
    GT["GSP emulation kthread<br/>per mdev"]
    VC["VFIO callbacks<br/>QEMU ioctl and mmap context"]
    WQ["Workqueues<br/>RPC, cleanup, bottom halves"]
    IR["IRQ handlers<br/>physical MSI-X [planned]"]
  end

  subgraph DATA["Shared structures"]
    CT["Channel table<br/>handles, rings, notifiers"]
    MAP["VA-to-PA map<br/>plus hot-page set"]
    PIN["Pin tables<br/>discovered plus injected"]
    PEND["Pending release queue"]
    CNT["Completion counter table"]
    MQS["Message queue shadow"]
    CFG["Config and BAR shadows"]
  end

  ST --> CT
  ST --> MAP
  ST --> PIN
  ST --> PEND
  ST --> CNT
  GT --> MQS
  GT --> CT
  GT --> CFG
  VC --> CFG
  WQ --> GT
  IR --> WQ

  classDef done fill:#d5f5d5,stroke:#1e7a1e
  classDef wip fill:#fff3cd,stroke:#9a7b1a
  classDef plan fill:#f8d7da,stroke:#9a1a1a
  class ST,GT,VC,WQ done
  class CT,MAP,PIN,PEND,CNT,MQS,CFG wip
  class IR plan
```

Rules the design enforces:

* One writer per structure. Injections are queued and drained by the service;
  self-discovery never overwrites authoritative injected data.
* Long scans are sliced and preemptible; the pump and release paths are never
  blocked behind a full-RAM pass.
* VFIO callbacks do only bounded work; blocking work lives in kthreads.
* Every bounded budget (stack frames, retry counts, scan slices) is audited at
  build time.

## 6. Timing model and budgets

| Interaction | Target | Why it matters |
|---|---|---|
| Guest MMIO to shadow | direct memory write | zero trap cost on the data path |
| Doorbell to dispatch | sub-ms | steers every control exchange |
| RPC round trip during init | 10s of us | hundreds of exchanges per boot |
| Release apply | us | trackers poll at high frequency |
| Push entry decode | us per entry | thousands of entries per context |
| Rescue of an unreadable entry | bounded ms | only on translation misses |
| MSI-X to guest | us | completion latency floor |
| Real submission + completion relay | hardware + us | the forwarded-work path [wip/default-off] |

The rule of thumb the architecture is built around: **state operations are
cheaper in the module than they would be on hardware; real work belongs on
hardware and pays only submission and relay.**

## 7. Multi-tenant scheduling model

The target model: per-tenant emulation contexts and real GSP clients, with
fairness policy in the module on top of the firmware scheduler.

```mermaid
sequenceDiagram
  autonumber
  participant P as Module policy [planned]
  participant A as Tenant A client
  participant B as Tenant B client
  participant S as Real GSP scheduler
  participant E as Engines

  Note over P,E: State operations never enter this loop -<br/>they complete in the module on the fast path.
  P->>A: grant slice or quota
  A->>S: submissions for tenant A
  S->>E: tenant A contexts
  E-->>S: completions for tenant A
  S-->>A: completions relayed
  P->>B: switch to tenant B
  B->>S: submissions for tenant B
  S->>E: tenant B contexts
  E-->>S: completions for tenant B
  S-->>B: completions relayed
  Note over P: Policy inputs - priority, quota, burst,<br/>per-tenant accounting. Measured, not assumed.
```

Fairness and performance isolation properties to be measured once forwarding
exists: per-tenant completion accounting, slice enforcement, starvation
freedom, and the overhead each tenant pays for the other's activity.

## 8. Signal paths: doorbells and interrupts, both directions

```mermaid
flowchart TB
  subgraph IN["Guest to module"]
    A["Guest writes doorbell in BAR0 shadow"] --> B["Poll loop observes"]
    B --> C["Fetch and dispatch"]
  end
  subgraph OUT["Module to guest"]
    D["Module has a completion"] --> E["eventfd signal"]
    E --> F["QEMU irqfd"]
    F --> G["KVM injects MSI-X vector"]
    G --> H["Guest ISR runs"]
  end
  subgraph PHY["Module and physical GPU [wip/default-off]"]
    I["Real channel doorbell"] --> J["GPU executes"]
    J --> K["Physical MSI-X"]
    K --> L["Module IRQ handler"]
    L --> M["Relay to guest or hardware semaphore"]
  end
  C -.-> I
  M -.-> E

  classDef done fill:#d5f5d5,stroke:#1e7a1e
  classDef wip fill:#fff3cd,stroke:#9a7b1a
  class A,B,C,D,E,F,G,H done
  class I,J,K,L,M wip
```

## 9. Lifecycle and teardown ordering

The ordering is not cosmetic: the module must never be unloaded while a VM
holds the device, and the mdev teardown path must not run long scans or
unbounded waits.

```mermaid
sequenceDiagram
  autonumber
  participant ORCH as Orchestrator
  participant QEMU as QEMU
  participant VFIO as VFIO and module
  participant SYS as mdev sysfs

  ORCH->>QEMU: stop the VM
  QEMU->>VFIO: close device, release regions and IRQs
  VFIO-->>QEMU: resources released
  ORCH->>SYS: remove mdev
  SYS->>VFIO: remove callback - bounded cleanup
  VFIO-->>SYS: device gone
  ORCH->>VFIO: unload the module

  alt guest hangs or faults
    ORCH->>QEMU: reset or destroy the VM first
    Note over VFIO: Never unload while a VM holds the device;<br/>forensics come from pstore and bounded logs.
  end
```

## 10. Version negotiation across components

```mermaid
sequenceDiagram
  autonumber
  participant GD as Guest driver
  participant EMU as GSP-RM emulation
  participant VT as Version profile
  participant BE as Execution backend (single live session)
  participant FW as Real GSP firmware (live session)

  GD->>EMU: version and capability queries
  EMU->>VT: select profile for this driver family
  VT-->>EMU: RPC surface, controls, caps
  EMU-->>GD: negotiated replies
  Note over EMU,VT: New driver versions enter as profiles -<br/>measured against the host oracle, never guessed.
  BE->>FW: host-side protocol version negotiation
  FW-->>BE: firmware capabilities
  Note over BE,FW: The backend carries its own small,<br/>version-keyed host-side protocol surface.
```

## 11. Data-structure map

| Structure | Represents | Written by | Consumed by | Status |
|---|---|---|---|---|
| Message queue + shadow | guest RM RPC transport | guest driver; emulation replies | emulation thread; guest RM | [done] |
| BAR0 shadow | register interface (boot, doorbells, PBA) | guest MMIO | emulation poll; pump | [done] |
| Channel table | per-channel ring, get/put, notifier, engine | guest objects; pump | pump; release path | [wip] |
| GPFIFO entries | queued work per channel | guest submissions | pump | [wip] |
| USERD / doorbell tokens | work signalling | guest | module | [wip] |
| VA-to-PA map | translations with hot-page hints | discovery and scans | pump; release path; pins | [wip] |
| Pin tables | authoritative injected translations | debug tooling; later protocol discovery | lookup fast path | [wip] |
| Pending release queue | releases awaiting a translation | decoder | service drain | [done] |
| Completion counters | per-channel progress published to guest pages | service | guest trackers | [done] |
| Real channels and page tables | forwarding state (real channel live; guest page tables open) | backend | real GPU | [wip] |

## See also

- [`09-full-system-picture.md`](09-full-system-picture.md) - the map this document zooms into
- [`06-host-module-and-hybrid-model.md`](06-host-module-and-hybrid-model.md) - module internals and build contracts
- [`08-physical-gpu-integration.md`](08-physical-gpu-integration.md) - the execution backend and decision record
- [`01-system-overview.md`](01-system-overview.md) - principles and layer map