Open-IOV: initial public release

Open-IOV virtualizes NVIDIA GPUs so that unmodified guest drivers run with
real GPU functionality - CUDA first, 3D and video to follow - on unmodified
hosts, without a proprietary vGPU license and without patching any driver,
firmware or hypervisor.

Core principle: emulate the interface, pass through the work.

  kernel/     open-iov-vfio.ko - host VFIO mdev driver: virtual GPU device
              (PCI config space, BAR shadows, MSI-X), GSP boot emulation,
              GSP-RM message-queue/RPC emulation, memory model (GPU-VA to
              guest-PA discovery) and the semaphore-completion service that
              advances guest UVM progress.
  userspace/  libiov.so management API and examples.
  tools/      guest diagnostics, firmware analysis wrappers, RM probes.
  scripts/    stack manager, VM harness, build/load/test automation.
  config/     hardware/version profiles (the portability surface).
  docs/       documentation tree - start at docs/README.md.
  wiki/       long-form firmware and protocol deep dives.

Status: initialization works end to end for a stock 570.144 Linux guest on
Turing (TU104): driver probe, nvidia-smi, the full GSP-RM control battery,
object creation and the early CUDA/UVM start-up path. GPU work forwarding -
the step that makes CUDA kernels execute on real hardware - is the active
work item; see docs/04-implementation/01-known-knowns-and-unknowns.md.

License: BSD-3-Clause for the project; GPL-2.0 for kernel modules that link
GPL-only kernel interfaces. See LICENSE and docs/09-reference/legal-and-licensing.md.

1 files changed, 188 insertions, 0 deletions

diff --git a/docs/03-architecture/07-driver-lifecycle-field-guide.md b/docs/03-architecture/07-driver-lifecycle-field-guide.md
new file mode 100644
index 0000000..8d80694
--- /dev/null
+++ b/docs/03-architecture/07-driver-lifecycle-field-guide.md
@@ -0,0 +1,188 @@
+# Driver Lifecycle Field Guide
+
+> A walking tour of what actually happens between "guest boots" and "CUDA
+> initializes", with the real message and control IDs we observe from an
+> unmodified 570.144 guest. Use this to interpret logs and to place new
+> observations.
+
+## 0. How to use this document
+
+Each stage lists: what the guest does, what the device/firmware must answer,
+what our emulation does today, and the log signature to look for
+(`dmesg | grep gsp-resman-vfio` on the host; `dmesg` in the guest).
+
+The IDs are for the 570.144 generation on the Turing (TU104) profile. Other
+generations will have additions and version differences; the *stages* are
+stable.
+
+## 1. PCI probe and config space
+
+**Guest:** reads vendor/device/class IDs, walks capabilities, sizes BARs,
+enables Memory Space and Bus Master, programs MSI-X.
+
+**Device must:** report a coherent TU104-family GPU with 16 MB BAR0, 256 MB
+BAR1, an MSI-X capability, and a link capability the driver accepts.
+
+**Emulation:** config space is synthesized; BAR sizes are enforced; the
+*hypervisor* is the usual source of surprises (a device attached to the root
+bus can have its PCIe capability rewritten to "root complex integrated
+endpoint", which removes link information the driver wants — attach behind a
+root port).
+
+**Log signature:** none from us; guest-side `lspci -vvv` is the reference.
+
+## 2. GSP boot handshake
+
+**Guest:** writes WPR descriptors, releases FWSEC, then the booter, then the
+RISC-V core; polls a status mailbox for "GSP init done".
+
+**Device must:** complete every stage in the order the driver polls, and leave
+the mailbox in the state the driver expects.
+
+**Emulation:** `kernel/gsp_emulation.c` models the stage state machine and
+auto-clears self-clearing triggers (e.g. MMU invalidate registers) so polls
+terminate.
+
+**Log signature:**
+```
+gsp-resman-vfio: GSP-BOOT: ... trigger auto-cleared ...
+```
+
+## 3. Static configuration
+
+**Guest:** `GET_GSP_STATIC_INFO` (65) → receives a large `GspStaticConfigInfo`
+structure; reads the VBIOS image from the BAR0 window; queries device/FB/BUS
+info.
+
+**Device must:** provide self-consistent sizes, engine inventories, memory
+geometry, interrupt topology, and a VBIOS blob the driver can parse.
+
+**Emulation:** static structures are synthesized for TU104 (FB size, BAR
+layout, engine list, vGPU-friendly VF values).
+
+**Log signature:**
+```
+GSPRM: GET_GSP_STATIC_INFO → ...
+```
+
+## 4. Message queues
+
+**Guest:** sends `SET_GUEST_SYSTEM_INFO` (1) with libos regions, a queue
+buffer description, and a page table (the queue buffer is
+`NV_MEMORY_NONCONTIGUOUS`). It also sends `GSP_SET_SYSTEM_INFO` (72),
+`SET_REGISTRY` (73) and `CONTINUATION_RECORD` (71) as init traffic.
+
+**Device must:** consume them silently (async) and start consuming the command
+queue exactly in the order the guest writes elements.
+
+**Emulation:** parses the queue chain, builds the queue geometry, links the
+status queue, and consumes elements one by one. **Element-based, not
+byte-stream-based**: each message is one element and may span pages.
+
+**Log signature:**
+```
+gsp-resman-vfio: libos region[i] id8=... pa=... size=...
+gsp-resman-vfio: queue args sharedPA=... ptes=... cmdOff=... statOff=...
+```
+
+## 5. Resource-manager initialization (the control battery)
+
+**Guest:** sends `GSP_RM_CONTROL` (76) for every subsystem it initializes.
+Representative controls (all observed, all answered with correct shapes):
+
+| Control | Meaning | What matters |
+|---|---|---|
+| `0x20800a40`-family | internal device info table | per-index values, entry count |
+| `0x20801112` | FIFO device info table | engine inventory; count must match engine queries |
+| `0x20800a5c` | interrupt kernel table | vector topology the driver maps |
+| `0x20800a59` | GMMU static info | **fault buffer sizes** (replayable/legacy) — a wrong value breaks UVM early |
+| `0x20800a01` | display static info | display count (headless values are fine) |
+| `0x20801823/-a/b` | BUS/FB info V2 | sysmem connection type and PCIe link caps feed bandwidth calculations |
+| `0x20802a07` | CE physical caps | per-engine support bits; UVM refuses to make channels without `sysmem` support |
+| `0x20802a0a/0b` | all-CE caps | caps table for all engines |
+| `0x20802a02` | CE PCE mask | one bit per engine |
+| `0x20802a08` | CE fault method buffer size | method buffer sizing |
+| `0x90f10106` | vaspace reserved-PDE contract | **carries page-table physical addresses** (see GSP-RM doc) |
+| `0x20803039` | NVLink masks/VBIOS info | unpopulated but must be shaped |
+| `0x20800a70` | BUS flush with sysmembar | UVM issues these frequently; success suffices |
+
+**Log signature:**
+```
+GSPRM: GSP_RM_CONTROL cmd=0x... hClient=0x... hObject=0x... paramsSize=...
+```
+
+## 6. Object creation
+
+**Guest:** `GSP_RM_ALLOC` (103) creates the object tree. Typical order for a
+CUDA process:
+
+```
+0x0        NV01_ROOT              (client root, 120-byte params)
+0x80       NV01_DEVICE_0          (56 bytes)
+0x2080     NV20_SUBDEVICE_0       (4 bytes)
+0x90f1     FERMI_VASPACE_A        (48 bytes)  ← the GPU address space
+0xa06c     KEPLER_CHANNEL_GROUP_A (20 bytes)  ← CE channel group
+0xc46f     TURING_CHANNEL_GPFIFO_A(368 bytes) ← the channel (GPFIFO VA, instance/userd descriptors)
+0xc076     GP100_UVM_SW           (0 bytes)
+0xc5b5     TURING_DMA_COPY_A      (8 bytes)   ← CE object
+```
+
+Channel parameters decode into: `gpFifoOffset` (a GPU VA),
+`gpFifoEntries`, `engineType` (e.g. COPY0 = 0x9), and
+`NV_MEMORY_DESC_PARAMS` for instance/userd/ramfc/method buffers with
+**physical** addresses and apertures (`ADDR_SYSMEM = 1`, `ADDR_FBMEM = 2`).
+
+**Emulation:** handle management and parameter shapes are validated against
+the reference structures; values are logged in a form that makes the object
+tree reconstructible from `dmesg`.
+
+**Log signature:**
+```
+GSPRM: GSP_RM_ALLOC class=0x... paramsSize=... hClient=... hParent=... hObject=...
+```
+
+## 7. UVM registration (the wall we climbed)
+
+**Guest:** `cuInit` → `UVM_REGISTER_GPU` → channel manager creation →
+page-tree init.
+
+**Device must:** let UVM's channels initialize (each channel does a first push
+and waits on a semaphore), and let page-tree initialization proceed.
+
+**Emulation (today):** we can decode the pushes UVM generates and complete
+their semaphore releases by translating the semaphore VA to a guest physical
+address (see the UVM and host-module docs). This is the live development front.
+
+**Log signatures (guest side):**
+```
+nvidia-uvm: ... Tracker entry for value N (sema VA 0x...) channel ID ... CE ...
+nvidia-uvm: ... pending push '' started at ... in channel_init() releasing value N
+```
+
+## 8. What comes after registration (roadmap)
+
+```
+UVM registers → cuInit returns → context creation → memory allocation
+→ channel creation for the process → kernel launch → (real execution)
+```
+
+Each step will need the same treatment: satisfy the interface, forward the
+work, relay the completions. The known-unknowns document lists what we expect
+to be hard.
+
+## 9. Quick reference: message/control IDs seen from a stock 570.144 guest
+
+| Function | Name | Frequency per CUDA run (approx.) |
+|---|---|---|
+| 1 | SET_GUEST_SYSTEM_INFO | once |
+| 10 | FREE | ~14 |
+| 47 | UNLOADING_GUEST_DRIVER | on unload |
+| 65 | GET_GSP_STATIC_INFO | once |
+| 70 | UPDATE_BAR_PDE | a few |
+| 71/72/73 | CONTINUATION_RECORD / GSP_SET_SYSTEM_INFO / SET_REGISTRY | once each |
+| 76 | GSP_RM_CONTROL | ~160 |
+| 103 | GSP_RM_ALLOC | ~60 |
+
+If a new driver version produces **new function numbers**, do not guess: add a
+handler that logs the raw payload, then decode it against the decompiled
+study notes for that driver version.
\ No newline at end of file