# Open-IOV Current State

> **Snapshot: 2026-10-02 01:25.** This is a dated public status page, not live
> state; it is refreshed on every progress check. Live working notes are
> maintained outside this repository; the public known-knowns/unknowns document
> is authoritative here:
> [`docs/04-implementation/01-known-knowns-and-unknowns.md`](docs/04-implementation/01-known-knowns-and-unknowns.md).
> The former 2026-09-21 snapshot (`STATE.md`) is retired; its content remains
> in the git history.

## Mission

Run **unmodified NVIDIA guest drivers** in VMs with full GPU functionality
(CUDA, graphics, video, AI/ML) on consumer/data-center GPUs without
proprietary GRID licenses - via host-side GSP emulation in
`open-iov-vfio.ko`. The guest stays stock: no guest patches, no guest-side
Open-IOV code.

Architecture stance: *emulate the interface, pass the work through to real
hardware, never patch anyone.* The control plane is emulated; the work is
meant to execute on the physical GPU.

## Milestone ladder (this snapshot)

| Milestone | Status |
|---|---|
| GSP boot handshake, GSP-RM message queues/RPC, BAR/PCI/MSI-X emulation | ✅ |
| Stock 570.144 guest driver probe / `RmInitAdapter` | ✅ |
| `nvidia-smi -L` enumerates the emulated TU104 | ✅ |
| CUDA userspace: `cuInit`, `cuDeviceGetCount`, `cuDeviceGet` | ✅ |
| `cuCtxCreate` returns | ✅ (the whole smoke test now completes) |
| `cuMemAlloc` + `cuMemcpy` roundtrip (`ALL OK` in the smoke test) | ✅ payload roundtrip verified 64/64 in the guest |
| Full `nvidia-smi` output / kernel launch | 🟡 the static runtime now initializes end-to-end with **real releases** (`cudaGetDeviceCount`/`cudaSetDevice`/3×`cudaMalloc`/2×`cudaMemcpyHtoD`/`kernel launch`/`cudaGetLastError` all `rc=0`) and queues a kernel launch; execution awaits the backend |
| CUDA kernel execution (`vectorAdd`) | 🟡 execution backend: physical ownership + firmware staging verified; the unbind/rebind freeze hazard is mitigated by the supervised D0 power-domain lock (3/3 green claim cycles; the 2026-10-01 P1 mdev-owned release path initially ran outside that pre-lock window and re-armed D3cold, freezing the host at 22:32:53 - the G0 fix bundle now latches the D0/D3cold locks by default; see the frontier section), and the real FB size (8 GiB) and boot layout are verified live. The supervised live secure-verification run raised the protected region (WPR2 up) for the first time, then proved a secure-verification-only carve is not host-recoverable on this fused part, so the fail-safe held the device and an approved reboot restored it (distribution driver rebound, GPU healthy). That one-way stage is now retired: the only device-write path is a single full boot lifecycle (secure verification -> bootloader -> RISC-V start -> resource-manager initialization), whose code is implemented in the module sources and has now run live: the first real boot raised WPR2, passed Booter Load and went RISC-V active, and the module received the first real GSP-RM message (event 0x1002); init-done (0x1001) and a clean release are the next steps (see the 2026-10-01 section). A second live boot on 2026-10-01 proved the full boot sequence again (protected region up, bootloader halt, RISC-V active, first GSP-RM message) and reached the resource-manager's inline CPU-sequencer request; the host executor rejected the real request because of an over-strict internal size cap, which was found with a raw read of the message in host memory, fixed, and rebuilt. Cycles 2-3 then ran the real sequencer and received init-done with an RM init failure (`0x59`), which reproduces on a verified matching firmware image; the   server's own shutdown supplied the last handoff bit, so each failed cycle
  completed a **clean release without a reboot**. Cycles 4-10 then captured
  the resource-manager's own error journal and root-caused the failure: the
  boot queue was missing the two pre-boot initialization messages the matching
  driver queues before the kernel kick (a system-information message and a
  packed registry table). That handshake is now implemented and **verified
  live**: the registry transport and packing are proven (a display-disable key
  took effect in the server's journal), the GPU timer is programmed before
  the kick, and the chipset/bridge block of the system information is
  populated from live values. Init now reaches the server's **static-config
  collection** and fails there (`0x59`); the device-id override key was
  exercised with no change, and a display-disable experiment showed display is
  not the first error. A source-level review of the failing collector has now
  closed the host side: it reads none of the remaining host-supplied fields
  (the one registry key on its call graph can only clear an error), so the
  ACPI/workspace/zeroed-field leads are retired. The one hard dependency left
  runtime is enabled only after the collector and the protected region blocks
  host reads). **Cycles 11-13 resolved this wall**: the resource-manager log
  ring had never actually been mapped (its libos init-argument id was
  malformed), which is why the first error was invisible; with the ring live
  the failure traced to the server's chip device-id reading as zero because
  the pre-boot identity fields were sent unpacked instead of as raw PCI
  config-space dwords. Fixing the payload made the server report init-done
  with a zero result and serve the full static-config reply (GPU name, 8 GiB
  FB, internal handles, BAR PDE bases, engine caps), and graceful shutdown
  works through the unload RPC. Each of cycles 4-13 released the device
  cleanly with no reboot.   Since 2026-10-01 17:24 the next rung has both code
  and live results: a host resource-manager client creates memory and
  channel-group objects on the booted server, allocates and programs the
   page-directory base of a virtual-address-space object, and **allocates a real
   channel** on the live server. In cycles 18-25 it went all the way to
   execution: the channel was bound and scheduled, a work-submit token was
   obtained, a copy work item was placed in its GPFIFO and the doorbell rung,
   and after the engine's fetch and method-path faults were diagnosed and fixed
   the physical copy engine ran it - a 4 KiB copy with the semaphore released
   and destination equal to source. Every cycle released the device cleanly
   with no reboot; the next rung is wiring the live physical session into the
   guest-facing mdev/vfio path
   ([`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md)). The guest-wiring rung then advanced: an inert backend-session API was added, and the mediated-device-owned lifecycle (P1, see the frontier section) was implemented and exercised - deferring the physical claim/boot to the first device create, releasing it on the last remove, with fail-open behaviour. The first P1 acceptance cycle exposed two defects - an asynchronous boot marked READY after a failed FWSEC boot, and the P1 release path freezing the host (see the frontier section). The G0 fix bundle (default-latched D0/D3cold locks, post-rebind readiness gate, pre-claim D0 settle, READY gated on a proven boot, pre-kick staging readback) is implemented, built, smoke-tested and now **validated live**: two module-owned supervised cycles (2026-10-02 ~00:57-00:58) were 2/2 green through claim, FWSEC, booter, RISC-V active, init-done (`0x1001`, `rpc_result=0` in ~955 ms) and a latched release with WPR2 down, exactly 5 `power domain left LOCKED` lines, 0 in-window rail cycles, host alive and pstore empty. The retested P1 mdev-owned path - the exact path that hard-froze on 2026-10-01 - now fails open safely (the READY gate releases the claim when no live GSP-RM session exists; no freeze), while the async FWSEC boot defect (`staging not visible before kick` -> `FWSEC/FRTS execution failed: -5`) remains open, quarantined off the critical path; guest wiring proceeds on the module-owned live session. The bundle is still uncommitted. |
| Graphics (3D), video | ❌ |
| Faults, multi-version, Windows, migration, multi-tenancy, performance | ❌ |

Test status is tracked with the CUDA driver-API smoke test
([`tools/test/cuda_drv_test.c`](tools/test/cuda_drv_test.c)): it exercises
`cuInit -> cuDeviceGetCount/Get -> cuCtxCreate -> cuMemAlloc -> cuMemcpyHtoD/
DtoH -> cuMemFree -> cuCtxDestroy` and prints `ALL OK` only when the whole
chain succeeds.

## Right now (frontier)

- **Guest-facing identity wiring (P2) validated live (2026-10-02
  01:11-01:15).** On a module-owned live GSP-RM session the emulation now
  overlays the server's real static configuration into the guest-facing
  identity path: the guest's `nvidia-smi -L` reports `NVIDIA GeForce RTX 2080
  Super with Max-Q Design` (with a synthetic UUID) instead of the generic
  emulated identity, while the full driver-API smoke test still passes `ALL
  OK`. The cycle ran under the supervised harness and released with the
  latched locks (protected region down, distribution driver rebound, 5 power
  locks intentionally left held, 0 in-window rail cycles, host stable, no
  freeze, pstore empty). This is the first live data path of the guest-wiring
  rung; a control run with the live overlay disabled is in progress (outcome
  pending). Guest work is still **not** forwarded to the physical engine -
  the submission/relay wiring is the next rung. The bundle (the default-latched
  D0/D3cold freeze fix plus the identity overlay) is implemented, built and
  exercised live, but not yet committed.
- **G0 fix bundle built after the P1 release-path freeze (2026-10-02 00:55).**
  The first P1 mdev-owned acceptance cycle ended in the known hard freeze
  (journal stops, pstore empty): the P1 release path ran outside the supervised
  D0 pre-lock window and the teardown restored `power/control=auto`, re-arming
  D3cold; the resume ladder ran 22:32:11-22:32:44 and the host froze at
  22:32:53. The same cycle's first async boot failed at FWSEC - the staging DMA
  never landed after an RTD3 rail cycle - yet the backend was marked READY. The
  G0 bundle: a new `gsp_phys_repower` parameter (default 0) makes the power
  unlock latch the D0/D3cold locks instead of releasing them; release gains a
  10 s post-rebind readiness gate; claim gains a 2 s D0 settle/verify before
  the unbind; READY is granted only when the boot proves a live RM session;
  FWSEC reads back the IMEM/DMEM staging before the kick; and the claim
  harness leaves the board latched by default (`--restore-power` opts out).
  Implemented, rebuilt clean, two clean no-param insmod/rmmod smoke cycles at
  00:55, then **validated live** (2026-10-02 ~00:57-00:58): two module-owned
  supervised claim/boot cycles were 2/2 green - claim, FWSEC, booter, RISC-V
  active, init-done (`0x1001`, `rpc_result=0` in ~955 ms), WPR2 down,
  `released 0000:01:00.0; rebound to nvidia (rc=1)`, exactly 5 `power domain
  left LOCKED` lines, 0 in-window rail cycles, host survived, pstore empty;
  module sha256 `dde36d92...`. Uncommitted.
- **Guest-wiring lifecycle (P1) implemented and exercised; retested after G0
  (2026-10-02 ~00:58).** A module parameter selects a lifecycle where the
  physical-GPU claim and boot are deferred to the first mediated device create
  and released on the last remove, run asynchronously and **fail-open**: if the
  backend boot fails the device keeps serving the emulation path. The first
  acceptance cycle verified the deferred claim, the fail-open behaviour (the
  boot failed at the secure verification stage, yet the smoke test still passed
  `ALL OK` payload 64/64 while the backend stayed off), and the release on
  remove; the release/rebind was followed by a host power-management freeze of
  the known kind (the GPU power-gating hazard), which needed an unplanned
  reboot; the release path now latches the D0/D3cold locks by default (the new
  `gsp_phys_repower=0` default in the G0 bundle above), keeping the board
  powered through the driver resume. Retested with the fixed module, the exact
  path that hard-froze now **fails open safely**: the first mdev create
  schedules the async claim, the new READY gate finds no live GSP-RM session
  (`rm_init_done=0 unload_submitted=0`) and releases the claim latched (5
  `power domain left LOCKED`, `released 0000:01:00.0; rebound to nvidia
  (rc=1)`); the host survives, no freeze. The async FWSEC boot itself still
  fails (`staging not visible before kick` - DMEM0/IMEM all zero, the falcon
  is not accepting DMA - aborting before STARTCPU -> `FWSEC/FRTS execution
  failed: -5`), surfaced in milliseconds by the new pre-kick readback instead
  of the old 5 s halt timeout; it is an open, fail-open-safe defect off the
  critical path (an isolation experiment ruled out the obvious hypotheses -
  stopping `nvidia-persistenced` and setting `power/control=on` plus waiting
  runtime active before mdev create gives the identical signature, and the
  module-owned path succeeds with the very same preconditions in the same
  boot). Guest wiring proceeds on the module-owned live session
  (`gsp_phys_gpu=1` claim+boot at load, proven); the P1 async claim stays
  fail-open. No guest work is forwarded to the physical engine yet.
- **Guest-wiring rung opened (2026-10-01 22:00): the first backend-session
  API code is in the tree, inert.** A skeleton that exposes the live physical
  session to the device layer (session request entry points plus a static-info
  snapshot helper) was added behind the existing gates and does not yet alter
  behaviour. In the same window the driver-API baseline was **re-verified green
  on the current build**: a fresh VM cycle ran the smoke test end-to-end
  (`ALL OK`, payload 64/64) with real releases (`faked=0 miss=0 stale=0`), so
  the emulation front is intact ahead of the wiring work. The phased plan for
  connecting the live session to the guest-facing path is
  [`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md).
  Post-single-VM-CUDA research notes (concurrent multi-VM virtual functions,
  3D acceleration, and a native-vs-guest performance program; and a host-side
  acceleration feasibility study) were produced under `backups/` (untracked).
- **Submission-rung milestone (2026-10-01 17:24-21:05): real work now executes
  on the physical engine.** The host resource-manager client allocated a
  virtual-address-space object with a committed page directory, a channel-group
  object and a real copy-engine channel, bound and scheduled it, obtained a
  work-submit token, and filed a copy work item in the channel's GPFIFO. The
  engine's first fetches faulted; draining and decoding the status queue then
  exposed the wrong virtual address in the fetch, and with that fixed the
  physical copy engine executed the block: `REAL COPY OK sem=0xc0ffee01
  src=0x1170d5000 dst=0x11ccbd000 bytes=4096 GPGet=1 GPPut=1 words=26
  token=0x00000001` - a 4 KiB copy with the semaphore released and destination
  equal to source. Every cycle ran under the supervised harness and released the
  device cleanly (server shutdown, protected region down, distribution driver
  rebound); no reboot was needed and the host stayed healthy. The next rung is
  wiring this live session into the guest-facing mdev/vfio path so guest
  submissions execute on the physical engine (phased plan:
  [`docs/03-architecture/11-physical-backend-guest-wiring.md`](docs/03-architecture/11-physical-backend-guest-wiring.md)).
  Evidence: `backups/evidence-20261001-cycle18/` through `...-cycle25/`, with
  per-wall reports in `backups/evidence-20261001-reports/next/`.
- **Physical-backend milestone (2026-10-01 16:45): the host-owned GSP-RM
  completes initialization on the real TU104 and serves the static-config
  RPC.** Three supervised cycles, no reboot, clean release each time:
  1. the resource-manager log ring is now registered correctly (its libos
     init-argument id was malformed), which made the server's previously
     invisible first error readable: a device-name lookup failing because the
     server's chip device-id was zero (cycle 11);
  2. the pre-boot system-information payload now carries the raw PCI
     config-space identity dwords (`device<<16|vendor`) instead of unpacked
     16-bit fields, and the static-config RPC now declares the full reply
     payload; the server then reports init-done with `rpc_result=0`
     (cycle 12);
  3. the static-config RPC returns the full 1656-byte configuration - GPU
     name `NVIDIA GeForce RTX 2080 Super with Max-Q Design`, short `TU104-B`,
     8 GiB FB, internal client/device/subdevice handles, BAR PDE bases,
     engine caps - and shutdown runs through the unload RPC (GSP-RM processor
     suspended in ~220 ms, handoff bits complete) with WPR2 down and the
     distribution driver rebound (cycle 13).

  The physical backend is now a live, queryable GSP-RM: the host can boot it,
  send RM RPCs and receive full replies, and shut it down gracefully. The
  next rung is to drive the guest-facing emulation from this backend.
  Evidence: `backups/evidence-20261001-cycle11/`, `...-cycle12/`,
  `...-cycle13/`.
- **Driver-API baseline re-verified green on the current build** (2026-10-01
  14:20): the smoke test passes end-to-end (`ALL OK`, payload 64/64) with
  real releases (`faked=0`, `miss=0`, `stale=0`); the source-level root-cause
  campaign (cycles 4-10) did not regress it.
- **Physical-backend milestone (2026-10-01)**: the module-owned GPU path
  reached the first successful secure-verification (FWSEC) carve of the
  protected region on the real TU104 under the verified D0 power-domain
  harness. The teardown of a FWSEC-only carve is not host-recoverable on this
  fused part (both the secure-scratch and MMU privilege masks are fuse-locked,
  so the bootloader handoff cannot be seeded and WPR2 cannot be unlocked from
  the host), so the fail-safe deliberately held the device and an **approved
  reboot restored it** (distribution driver rebound, `nvidia-smi -L` green,
  host healthy, no crash record). The one-way stage is now retired: the only
  device-write path is a single supervised full boot lifecycle (secure
  verification -> bootloader -> RISC-V start -> resource-manager
  initialization). That lifecycle is now implemented in the module sources
  (bootloader runner + generated verification/boot image data) and has run
  live under the supervised harness: the first real boot succeeded (see the
  section below), and a second live cycle on the rung-4 build repeated the
  boot and reached the server's run-sequencer request; the executor's size
  cap rejected the real request value (16354), which was diagnosed from a
  raw host-memory read of the message, fixed, and rebuilt. The post-reboot
  recovery is now verified and the zero-timeout REG_POLL default fix
  (commit `98b861d`, build `87309e3a…`) was confirmed live. Cycles 2-3
  (2026-10-01 11:40 and 11:46) then reached **stage 3e on real hardware**:
  the real GSP-RM executed the full 165-command/490-dword sequencer and
  reported RM init failure `0x59` (`NV_ERR_OPERATING_SYSTEM`) on both
  firmware pairings (the default 570.211.01 image and a verified matching
  570.144 image - the image-pairing hypothesis is disproven). Each failed
  cycle now **releases cleanly back to the distribution driver without a
  reboot**: the server's own shutdown sets the remaining secure-scratch
  handoff bit, so all four handoff bits are set and a single `rmmod` clears
   the protected region, rebinds the distribution driver and releases the
   power-domain locks. The error-journal capture has since run on ten supervised
   cycles (each one attempt, each released cleanly with no reboot): the journal
   was read, and the failure was root-caused to two pre-boot initialization
   messages the matching driver queues before the kernel kick (a
   system-information message and a packed registry table) - our boot queue was
   empty. That handshake is
   now implemented and **verified live**: the boot consumes a true
   system-information message and a real registry profile, the registry
   transport and packing are proven (a display-disable key took effect in the
   server's journal), the journal collapsed from 29 unique/851 total records to
   4 unique/5, the pre-kick GPU timer is programmed (the live timer line was
   observed, and the generation-correct zero timer offset was confirmed), and
   the chipset/bridge block of the system information is populated from live
   values. Init now reaches the server's **static-config collection** and fails
   there (`0x59`); the device-id override key was exercised with no change, and
   a display-disable experiment showed display is not the first error. The
   collector's failing step is not visible in the server journal (its log ring
   is enabled only after the collector, and the protected region blocks host
   reads). A source-level review had **closed the host-side input campaign**
   and named a server-PMU dependency as the one hard requirement. **That
   verdict was superseded by cycles 11-13**: the log ring was in fact never
   mapped (its init-argument id was malformed), and once it was registered the
   failure was a zero server-side device-id caused by our unpacked identity
   payload - the PMU path itself was fine. The raw config-space identity fix
   completed initialization (see the milestone bullet above).
- **The full CUDA driver-API smoke test passes with correct data**: the client
  completes `cuCtxCreate -> cuMemAlloc -> cuMemcpyHtoD -> cuMemcpyDtoH ->
  cuMemFree -> cuCtxDestroy` and prints `ALL OK`, with the 64-byte payload
  roundtripping exactly (64/64 verified in the guest).
- **The last two walls**: the GPU page-table walk's address decode (child
  pointers and leaves were 16x too low; fixed with the real entry encoding -
  walks now succeed at scale) and the device-to-host copy (a copy-engine
  class transfer bound on a compute channel whose work was denied by the
  engine-type gate; trusted host-injected pushes may now execute copy-engine
  work as well).
- **Baseline green and quiet**: the payload roundtrip is back to 64/64
  (`ALL OK`), the staging buffer's translation stays resolvable via pins
  streamed from the guest, and high-frequency diagnostics are now gated
  behind a default-off debug switch - a normal boot+run produces a ~520-line
  log instead of a flood (verified twice). The page-directory entry writes
  have since been captured live and validated (the earlier all-zero boots
  were operation-skipped/pre-pin boots); the remaining question is the
  device-address walk gap for the runtime's own buffers.
- Recap of the fixes behind this point: all 20/20 compute-channel releases
  land (three decode paths routed to the exact counter slot); registration
  delivers the incrementing tracking release for every consumed submission;
  the copy-engine push buffer is pinned from guest truth on every boot; and
  the copy-engine decoder consults pinned translations before the scanned
  map, so every push-data operation executes (`memset=19473`, `ringops=10551`
  live).
- **The runtime client now initializes and reaches kernel launch**: with a
  static-runtime `vectorAdd` client, `cudaGetDeviceCount`, `cudaSetDevice`,
  `cudaFree`, three `cudaMalloc`, the HtoD copies and `cudaGetLastError` all
  return success. The client then parks in `cudaDeviceSynchronize` - the
  launch is queued but never **executed**, because real execution requires
  the physical-GPU backend (the next project).
- **What fixed it**: the failing call was never a kernel call - it was the
  clock-domain probe inside libcuda (two device attributes) whose three
  legacy clock controls were answered with zeros; they are now populated
  with exactly the fields the helper checks.
- **Real-release mode now reaches the same rc=0 chain**: the earlier
  per-push 8 s stall was two pump bookkeeping bugs (release detection keyed on
  an unchanged semaphore VA; rewind replays re-entering the CE grace path) -
  both fixed, and the runtime init completes in ~1 min with real releases.
  The content rescue also learned the real method encoding (NON_INC/IMMD/
  ONE_INC/END_PB headers).
- Active work: the owed driver-API smoke-test rerun is now **verified 64/64
  (`ALL OK`)** on the current commits, so the recent pump/parser changes did
  not regress the baseline. The device-claim bring-up verified ownership,
  firmware staging and the boot layout, but unbinding the distribution driver
  can power-gate this laptop's dGPU and hard-freeze the host (two occurrences,
  now root-caused), so that path is disabled behind a two-parameter hazard
  gate. A power-domain lock that keeps the board in D0 across the handoff is
  implemented and **verified across three supervised claim/release cycles on
  2026-10-01** (zero power-domain cycles, zero failed resumes, distribution
  driver rebound cleanly each time, host healthy); the verified launch
  harness holds the GPU and root port in D0 before the distribution driver is
  stopped, and the root-port hold is the load-bearing invariant. The real GSP
  boot has since run live on this path (see the 2026-10-01 section below).
- Observability: a size-keyed copy watch, a VRAM shadow dump/search, and an
  operation log are now module parameters, and the test's output survives
  guest-agent stalls via the serial console.
- Guest health note: agent/SSH timeouts during long runs are the guest-side
  trace reader flooding the ring; they recover on their own, so a first
  timeout is not a reason to restart the VM. Separately, one no-watcher boot
  crashed the guest kernel (a NULL callback in interrupt context) before any
  emulation channel existed - guest-side and untriaged; treat no-watcher runs
  as a known risk. The host stays stable throughout.
- Execution backend (chosen; ownership hazard mitigated and verified): the
  module owns the physical GPU and will run its real GSP firmware as the
  resource-manager server, so real work executes with an unmodified host
  driver and no host daemon; see
  [`docs/03-architecture/08-physical-gpu-integration.md`](docs/03-architecture/08-physical-gpu-integration.md).
  The ownership path was verified live (release the distribution driver from
  the device, bind the module, map the BARs, sample the falcon state, then
  release and rebind the distribution driver); with the device owned, the
  matching GSP image is fetched and validated (ELF64 RISC-V, loadable
  image/version/signature sections identified) and the protected-region
  video-memory/boot layout is computed and logged - all without device writes.
  The unbind/rebind hazard is mitigated by the D0 power-domain lock, verified
  over three supervised cycles; the ownership handoff is now also hardened
  against the distribution driver's udev bind-event power-policy write, and a
  reusable supervised claim-cycle harness exists
  (`tools/guest-diag/phys_claim_cycle.sh`). The real-GSP-boot design is
  complete: exact secure-verification/booter/RISC-V/RM-init sequence and boot
  artifacts (booter/FMC/SEC2 BL bindata; VBIOS BIT FWSEC blob). The real FB
  size (8 GiB, decoded from the local-memory-range register - the BAR1-aperture
  approximation is retired) and the protected-region boot layout have been
  verified live under the supervised claim harness (three green claim/release
  cycles; the root-port D0 hold is the load-bearing invariant), and the
  secure-verification stage (VBIOS security-blob extraction, falcon reset and
  staging) was validated offline and in a read-only dry-run, then run **live
  under the supervised claim harness**: it raised the protected region (WPR2
  up) for the first time on real hardware. Its release teardown, however,
  could not clear WPR2, and the fail-safe deliberately held the device
  (distribution driver unbound, power-domain locks held) rather than hand it
  back; a rescue path is being debugged (or, failing that, an approved
  reboot). A distribution-driver bus-master quirk on the claim path was also
  found and fixed in this run. The bootloader/RISC-V/resource-manager-init
  stages are now implemented in the module sources (bootloader runner plus
  generated boot and verification image data) and have now run live: the
  boot completed through an active RISC-V core and the module received the
  first real GSP-RM message (see the section below); a clean release is
  pending the server shutdown RPC.
- Host stability: the host hang class observed on 2026-09-28 (kernel stack
  overflow in a channel snapshot path) is fixed and frame-audited; repeated
  VM load/attach/teardown cycles run without panics or pstore records. A
  second, distinct freeze class was root-caused on 2026-09-30: the dGPU
  board's power rail is gated by an ACPI power resource behind its root
  port, released through an embedded-controller command. Unbinding the
  distribution driver (as the physical-GPU claim path does) lets the device
  runtime-suspend, the rail is cut, and the re-bind intermittently fails to
  power it back (`not ready ... after resume`), with the host hard-freezing
  about a minute later (no crash record; journal stops, pstore empty; both
  occurrences followed an unbind/rebind, and prior log history has no other
  instance). The claim path refuses to run unless both `gsp_phys_gpu=1` and
  `gsp_phys_hazard_ack=1` are passed, and the power-domain lock (hold the GPU
  and its board functions plus the root port in D0 across the handoff,
  aborting and holding the locks on any power-loss signature) is now
  **implemented and verified across three supervised claim/release cycles**
  (2026-10-01: zero power-domain cycles, zero failed resumes, clean rebind,
  host healthy). The verified launch harness holds the GPU and root port in
  D0 before the distribution driver is stopped; the root-port hold is the
  load-bearing invariant. The path must still not run unattended and every
  cycle is supervised. The device has since been restored by the approved
  reboot (distribution driver rebound, GPU healthy); the secure-verification
  stage is now folded into a single full boot lifecycle, and normal module/VM
  work does not touch the driver binding and is unaffected. See
  [`docs/03-architecture/08-physical-gpu-integration.md`](docs/03-architecture/08-physical-gpu-integration.md).

## First real GSP boot + live GSP-RM session (2026-10-01)

Cycle 09:53:27–09:54:59 on the supervised D0 harness (`--boot --keep`),
`open-iov-vfio.ko` sha256 `7815dacc…`, srcversion `2993AF5C…`. The boot order
observed on the physical TU104:

1. **FWSEC** — WPR2 came up; FRTS error code 0; `WPR2_LO=0x01ffe000`,
   `WPR2_HI=0x01ffee00`.
2. **Booter Load** on SEC2 — halted with `M0=0x0` after 146 ms.
3. **RISC-V active** — `RISCV_STATUS=0x00000035` (active bit 0),
   `FALCON_OS=0`; secure scratch14 raw `0x11000000` (booter-load handoff bit
   24 = 1, booter binary version 1).
4. **First real GSP-RM message** — the staged status queue linked and the
   server's first event was `function=0x1002`
   (`NV_VGPU_MSG_EVENT_GSP_RUN_CPU_SEQUENCER`, `rpc_result=0`), carrying a
   command buffer the host must execute; `GSP_INIT_DONE` (0x1001) follows only
   after that sequencer runs and was not observed in the 60 s poll.

This was the first host↔real-GSP-RM exchange on the physical path. At that
point a clean release was not yet possible: the Booter Unload gate requires
all four secure scratch14 handoff bits (24–27); a host-driven boot set only
bit 24, and bit 27 was expected from the server's shutdown path (host RPC
`UNLOADING_GUEST_DRIVER`, not yet implemented then). The host-side seed/unlock
alternative is fuse-locked on this part. The session was therefore
intentionally left claimed (`--keep`): the GPU was bound to
`open-iov-gsp-phys` and runtime-active; `nvidia` stayed unbound (host
headless; `nvidia-smi` fails by design); host healthy, pstore empty. A fresh
live cycle needed a power cycle at that point. **Status update (cycles 2-3,
same day):** this limitation is retired - the server's shutdown set the
remaining handoff bit and a single `rmmod` released the device without a
reboot (see the frontier section).

**Next steps:** (a) execute the 0x1002 sequencer command buffer in the module
and log 0x1001 as `stage 3e complete`; (b) send `UNLOADING_GUEST_DRIVER` on
the command queue so the server shuts down, sets the remaining handoff bit and
teardown can hand the GPU back without a reboot; (c) prove a request/reply
round trip with the server. All three were implemented, built and have now
run live: cycles 2-3 executed the sequencer and reached init-done, and the
server's own shutdown supplied the missing handoff bit, so each failed cycle
released without a reboot; a request/reply round trip remains to be
exercised. The module sources contain the sequencer executor (BAR0
register-write/modify/poll, delay, store and core-resume opcodes), a
generalized msgq producer (multi-element records,
checksum/sequence/free-space rules, doorbell) with the unload RPC, a
reply-matching status-queue reader, and a debugfs probe that requests the
server's static configuration after init-done; the rung 4a/4b/4c code passed
an adversarial wire-format review plus its fixes.

**Prepared for the next cycle (now history):** an early build (`f61308bd…`)
ran the boot green but the server's run-sequencer request was rejected by an
over-strict host-side cap on the request's own buffer size. The message was
read raw from host memory, the cap corrected (build `87309e3a…`), and the
sequencer then ran live in cycles 2-3 (see the frontier section).

Ending a claimed session: `sudo rmmod open_iov_vfio`, then `echo auto | sudo
tee /sys/bus/pci/devices/0000:01:00.0/power/control` and the same for
`0000:00:01.0`, then `sudo systemctl start nvidia-persistenced`. Since cycles
2-3 this passes the handoff gate (the server's shutdown sets the last handoff
bit) and completes without a power cycle; if the protected region survives
teardown the fail-safe keeps the locks and a power cycle is still required,
with approval.

## Initialization, work and teardown path

Status tags: `[done]`, `[frontier]` (actively worked), `[partial]`,
`[not yet]`.

```
        HOST SIDE                                        GUEST (stock driver, unmodified)
        =========                                        ================================

 [done] build + load open-iov-vfio.ko
        |  (module first: exports the symbols the NVIDIA open modules link against)
        v
 [done] mdev create ------------------------------------>  /
        |  BAR/PCI/MSI-X emulation; GSP emulation thread
        v
 [done] VM attach (QEMU q35 + pcie-root-port) ---------->  config space read; BAR mmap
        |
        v
 [done] GSP boot handshake (emulated) ------------------>  nvidia.ko probe / RmInitAdapter
        |
        v
 [done] GSP-RM message queues / RPC / controls --------->  object + client setup
        |
        v
 [done] engine + class lists --------------------------->  GPU_GET_ENGINES, class DB
        |                                                  (two-GRCE fix; async copy engine)
        v
 [done] ------------------------------------------------>  nvidia-smi -L enumerates TU104
        |
        v
 [done] ------------------------------------------------>  cuInit / DeviceGetCount / DeviceGet
        |
        v
 [done] ------------------------------------------------>  cuCtxCreate
        |                                                  UVM register GPU; channel manager
        |                                                  GR + CE channels; push buffers
        |                                                  semaphore pool; completion counters
        v
 [done] ------------------------------------------------>  cuMemAlloc + cuMemcpy roundtrip
        |                                                  (passes end-to-end; payload 64/64 correct)
 [partial] --------------------------------------------->  full nvidia-smi output / kernel launch
        |                                                  (runtime init rc=0; launch queued;
        |                                                   execution awaits the backend)
```

Work execution (host service vs client synchronization):

```
 [done] host decodes guest pushes (method streams): memsets, copies, releases
           - CE pushes execute at scale; the compute-class inline-DMA HtoD and
             the copy-engine-on-compute DtoH both execute; payload 64/64
           - the runtime client now initializes (rc=0) and queues a kernel
             launch; the queued compute pushes decode but are not executed
 [partial] GPU VA -> guest PA translation: page-table walk fixed and succeeding;
           server-owned page-table population still in progress
 [done]    real work executes on the physical GPU: the host RM client's copy
           engine block runs on the physical CE - 4 KiB copied, semaphore
           released, destination equals source
 [not yet] guest-submitted work relayed to the physical engine (next rung:
           wire the physical session into the mdev/vfio path)
 [not yet] correct results (vectorAdd)
```

Teardown and recovery:

```
 [done]    happy-path teardown exercised end-to-end (cuCtxDestroy OK)
 [done]    VM stop -> mdev remove -> rmmod, in order; never rmmod with a live VM attached
 [done]    crash recovery: pstore inspection; the host hang class is fixed and frame-audited
```

Recent chain of walls removed (2026-09-28 to 2026-09-30):

```
 host hang (stack overflow) -> pump state persistence -> map collapse (merge rebuild)
   -> rescue overhaul -> four-word semaphore releases -> counter injection (all 20)
   -> stale semaphore-pool translation -> fake-only proven insufficient
   -> push buffer found and validated -> per-page pins -> authoritative pin table
   -> deterministic chain solved -> pin-drop and release-push fixes
   -> copy-engine data path executes -> compute-window pinning
   -> registration release delivery -> 20/20 counter slots -> GR-info clock fix
   -> cuCtxCreate OK; payload found (compute-class inline DMA)
   -> GMMU walk decode fixed (child/leaf encoding) -> inline HtoD lands
   -> DtoH copy-engine-on-compute allowed -> payload roundtrip 64/64
   -> runtime client initializes rc=0 (legacy clock-domain fix) -> kernel launch
      queued -> real-release chain rc=0 (pump bookkeeping fixes) -> execution
      backend bring-up is the next wall
```

## Tools, code paths and fixes behind this progress

```
 KERNEL MODULE (open-iov-vfio.ko)                GUEST-SIDE TOOLS (tracked)
 ================================                ==========================
 gsp_emulation.c      boot, doorbells, MSI-X      tools/test/cuda_drv_test.c   smoke test
 gsp_rm_emulation.c   msgq, RPC, controls         tools/guest-diag/vmrestart.sh
   gsp_rm_tu104.h       two-GRCE engine fix       tools/guest-diag/cuda_cycle.sh
 gsp_sema_service.c   THE WORKHORSE               tools/guest-diag/push_guest_tools.sh
   channel pump, method decode, rescue,           tools/guest-diag/inject_counters.sh
   map rebuild, pins, counters, releases          tools/guest-diag/semfix_watch.py
 bar_emulation.c      BAR shadows                 tools/guest-diag/track_waits.sh
 main.c               mdev/VFIO lifecycle         tools/guest-diag/trkinfo.py
   stack-frame audit (-Wframe-larger-than)        tools/guest-diag/bp999.py / scanerr.py
 dynamic_rpc_dispatcher.c  firmware load fix      tools/guest-diag/read_va.py
                                                  tools/guest-diag/guest_kprobes.sh

 DATA STRUCTURES                                 DEBUG PLANES
 ===============                                 ============
 channel table (handle, GPFIFO, USERD,           module stats: applied / miss / memset /
   notifier, ring)                                 ringops / nomap / skip / faked /
 VA->PA map (merge rebuild, hot pages, cache)      pins / ipins
 pin table + authoritative injected pin table    debug params: sema_probe, sema_counter,
 completion counter table (per channel)            sema_map_add, chan_dump, shadow_dump
 semaphore pool page (GPU VA 0x121000000)        QMP gva2gpa + guest gdbstub reads
 push buffer allocation (GPU VA 0x120000000)     guest kprobes; /proc/<pid>/pagemap
 client trackers / RM error record chain         marker-based /dev/kmsg parsing
```

| Wall | Fix |
|---|---|
| Host panic: oversized stack frame in a channel snapshot path | static buffers; frame-size audit on every build |
| Channel pump lost per-pass state (stale entries re-run; rate limit dead) | write-back of rescue/payload state; one-retry skip for compute channels |
| VA-to-PA map collapsed after client teardown | merge rebuild instead of wholesale replace |
| Rescue slow / false candidates | hot-page-first scan; full method-stream validation; retry limits |
| Four-word semaphore releases ignored | decode both release encodings |
| Only 4 of 20 completion counters ever moved | async injection queue; fast injector; per-boot pin, order fixed |
| UVM tracker wait never cleared | stale pool translation found; pool page pinned from guest truth each boot |
| Acknowledging releases without executing work | proven insufficient; pushes must be read and executed |
| Push buffers unreadable | located by CPU VA; per-page translation; authoritative injected pin table |
| Injected pins silently dropped once a pin-table cluster grew | authoritative injected-pin table checked first; larger table, longer probe |
| Real release pushes rejected as implausible (non-zero method start) | decode release-only when the translation came from injected pins |
| Completion release decoded but deferred (counter page untranslated) | pin the compute-channel mapping window from guest truth each boot |

## What "CUDA works" means (open gates)

Per [`docs/04-implementation/01-known-knowns-and-unknowns.md`](docs/04-implementation/01-known-knowns-and-unknowns.md)
section 4, a defensible claim needs **all** of:

1. every guest-to-device message for the CUDA path answered or forwarded,
   with coherent replies for consumed fields;
2. every work VA translatable and server-owned page-table ranges populated;
3. launches of a grid of thread blocks (a CUDA kernel dispatch, e.g.
   `vectorAdd`'s `<<<blocks, threads>>>`) actually executed on the physical
   GPU's compute engine - not merely acknowledged - and copies executing on
   real hardware with correct results;
4. completion (semaphores, events, faults) behaving as the guest expects;
5. at least two driver families passing;
6. tenant isolation;
7. measured performance within an agreed factor of native.

Until then the honest statement is: **initialization works; execution is in
progress.**

## Deferred goals

- Host NVIDIA driver (`nvidia.ko`) over a virtual function — host OS
  acceleration / host display from the open-iov device model — is a
  **post-1.0 stretch goal we are still only considering**. We may work on it at
  some point; we will probably take a crack at it because it is a great
  feature, but **do not block on it, pursue it, or design around it**. It is
  strictly lower priority than every baseline goal (guest CUDA execution,
  multi-VM, 3D, Windows, isolation, performance, portability). It is
  **never pursued automatically**; it requires a human decision once we are
  certain there is no risk — which we are not certain of now — and it may be
  scrapped on architectural grounds. Treat it until further notice as
  **do not implement**.
  - First design (`13-host-vf-acceleration-design.md`) was written and then
    **reverted on 2026-10-02** after independent reviews falsified its
    load-bearing mechanisms; all associated doc edits were reverted.
  - The corrected record, candidate approaches, experimental gating policy and
    revisit criteria are in the post-1.0 document
    [`docs/10-post-1.0/experimental-host-acceleration.md`](docs/10-post-1.0/experimental-host-acceleration.md)
    (intentionally not linked from the docs front door and not part of the
    active reading order; don't index it and don't design toward it).
  - Earlier verdict and options:
    [`docs/03-architecture/12-host-vf-acceleration-future-goal.md`](docs/03-architecture/12-host-vf-acceleration-future-goal.md).

## Environment

- Host GPU: NVIDIA TU104 (RTX 2080 SUPER Mobile), PCI `10de:1ed3`.
- Host OS: Ubuntu, kernel 6.8.0-142; the distribution NVIDIA driver manages
  the physical GPU; `open-iov-vfio.ko` loads as an independent module.
- Guest: Ubuntu 24.04.5 LTS, stock NVIDIA 570.144 driver, QEMU/KVM q35 with a
  `pcie-root-port` for the vGPU, SeaBIOS, mdev passthrough.
- Firmware: TU104 GSP 570.144 (public release).

## Build and load order

Build: host module first (it generates the symbol versions the NVIDIA open
kernel modules link against), then the NVIDIA open kernel modules, then
userspace.

Load: VFIO prerequisites, then `open-iov-vfio.ko`, then create the mdev and
attach it to the VM. Unload in reverse; never `rmmod` while a VM holds the
device.

## See also

- [`docs/README.md`](docs/README.md) - documentation front door
- [`docs/04-implementation/01-known-knowns-and-unknowns.md`](docs/04-implementation/01-known-knowns-and-unknowns.md) - known-knowns, known-unknowns, definition of done
- [`userspace/cuda/READ-ME-FIRST.md`](userspace/cuda/READ-ME-FIRST.md) - CUDA analysis entry point
- [`docs/03-architecture/08-physical-gpu-integration.md`](docs/03-architecture/08-physical-gpu-integration.md) - chosen execution backend (module-owned GPU + real GSP session)
- [`docs/03-architecture/09-full-system-picture.md`](docs/03-architecture/09-full-system-picture.md) - visual end-to-end system map with status tags
- [`docs/03-architecture/10-interaction-and-scheduling.md`](docs/03-architecture/10-interaction-and-scheduling.md) - cross-component interaction, queues, interrupts, scheduling detail

---

*Refresh this page on every progress check (snapshot time in the header);
prefer moving detail into the `docs/` tree and keeping this page short.*