summaryrefslogtreecommitdiffstats
path: root/docs/04-implementation
diff options
context:
space:
mode:
authorArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-02 11:28:49 -0400
committerArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-02 11:28:49 -0400
commit260c73e72674d55bf8f082c0e1011075a0365a78 (patch)
treed1a6de00c2e1b6856012e6ce6c8bea319a7a6b16 /docs/04-implementation
parentf909a5f822c5c4a4e9526b861d4171f8925f5113 (diff)
downloadopen-iov-260c73e72674d55bf8f082c0e1011075a0365a78.tar.gz
open-iov-260c73e72674d55bf8f082c0e1011075a0365a78.zip
docs: engineering-operations playbook and project-wide status accuracy pass
- docs/eng-ops.md gains Part II, live physical-backend engineering operations: hard rules, the full module-parameter matrix, the routine supervised cycle and clean-release checklist, the debugfs inventory, how to read gsp-phys-state, a debugging scenario catalogue (CTXNOTVALID/promote wedge, fail-safe WPR2, unload hangs, guest wedges, MMU/PTE and PBDMA faults, relay symptoms), a hazard register (crash causes and debugging side effects), the physical-backend OODA loop, and the work-context map with porting flows - architecture, implementation, operations, orientation, getting-started, portability, ecosystem and reference docs corrected to the current state: CUDA driver-API smoke green, CE-class work forwarding and engine-authored releases plus fault relay in place (default off), GR execution blocked on engine-context validity, the promote wedge and designed locked fail-safe, and P5 wave-2 subsumed - internal-reference cleanup in tracked docs (no analysis-source or local-archive references); historical architecture document bannered - README reading list and STATE-CURRENT frontier refreshed; new pages (runbook, green-eye user guide, porting sequence, porting/packaging notes) linked from the tree Link check: scripts/check-docs-links.py passes (56 files, no broken links).
Diffstat (limited to 'docs/04-implementation')
-rw-r--r--docs/04-implementation/01-known-knowns-and-unknowns.md64
-rw-r--r--docs/04-implementation/02-practical-challenges-and-adversarial-interop.md8
2 files changed, 40 insertions, 32 deletions
diff --git a/docs/04-implementation/01-known-knowns-and-unknowns.md b/docs/04-implementation/01-known-knowns-and-unknowns.md
index 1a5967b..0c8e50f 100644
--- a/docs/04-implementation/01-known-knowns-and-unknowns.md
+++ b/docs/04-implementation/01-known-knowns-and-unknowns.md
@@ -22,12 +22,14 @@ listed as an unknown-unknown has a *method* by which we will discover it.
### 1.1 Device and platform
* A stock 570.144 Linux guest driver probes and initializes against our
- emulated TU104 device; `nvidia-smi -L` works and CUDA user-space starts
- (`cuInit`, `cuDeviceGetCount`, `cuDeviceGet`). Context creation is in
- progress: the client reaches `cuCtxCreate` and currently ends in a forced
- `rc=999` on a completion wait; the engine inventory is complete (the async
- copy engine is included) and the remaining work is in the completion and
- work-execution path (see §2.1).
+ emulated TU104 device; `nvidia-smi -L` works and the full CUDA driver-API
+ smoke passes (`ALL OK` 64/64, copies matched) — `cuInit`, device queries,
+ context creation, allocation, copies, free and destroy complete, and the
+ copies are verified byte-for-byte. The execution backend is live: CE-class copies are
+ forwarded to the physical copy engine (default off) with engine-authored
+ semaphore releases and a fault relay; the remaining frontier is engine
+ execution — GR kernel execution is blocked on GR engine-context validity
+ (see §2.1).
* The same guest **boots unmodified**: no guest code, no guest module
parameters are required for the initialization path.
* The host is unmodified in the standalone configuration (distribution NVIDIA
@@ -86,29 +88,30 @@ it**, **rough cost**.
inventory including the async copy engine, and the copy-engine capabilities
match the physical device (two GRCEs). The earlier zeroed secondary engine
record that made `libcuda` rewrite the channel alloc class and roll back is
-gone; context creation now progresses past engine selection and stalls later,
-on completion of decoded work (see §1.1).
+gone; context creation completes and the driver-API smoke passes (see §1.1).
-**Unknown (resolved by decision, not yet implemented):** the correct,
-isolated way to run a guest's engine work on the physical GPU. The interface
-analysis in
+**Partially delivered:** the correct, isolated way to run a guest's engine
+work on the physical GPU. The interface analysis in
[`../03-architecture/08-physical-gpu-integration.md`](../03-architecture/08-physical-gpu-integration.md)
shows that proxying through the distribution host driver has no callable entry
point from kernel context, so the **chosen mechanism** is for the module to own
the physical GPU and run its real GSP firmware as the resource-manager server
-(the module becomes the host-side RM client). The emulation path remains the
-control plane.
+(the module becomes the host-side RM client). That backend is now live: CE-class
+copies and their releases are forwarded to the physical copy engine (default
+off) with a fault relay, while the emulation path remains the control plane.
+What is still open is **GR (kernel) execution**, blocked on GR engine-context
+validity.
-**Why:** this is *the* blocker for CUDA kernel execution, and later for
+**Why:** GR execution is *the* blocker for CUDA kernels, and later for
graphics.
-**Plan:** build the chosen mechanism in stages: device ownership + real GSP
-bring-up; host-side RM client for real objects and channels; translation of
-guest work and real submission; completion and fault relay. Measure each stage
-against a scorecard: correctness, isolation, latency, maintenance cost across
-driver versions. Deliverable: a forwarded `vectorAdd` with verified results.
-Alternatives (own scheduling without GSP; vendor vGPU or SR-IOV paths) are
-analyzed and rejected or deployment-gated in the integration doc.
+**Plan:** finish the backend in stages: GR engine-context validity and kernel
+submission; translation of the remaining guest work; completion and fault relay
+for every engine class. Measure each stage against a scorecard: correctness,
+isolation, latency, maintenance cost across driver versions. Deliverable: a
+forwarded `vectorAdd` with verified results. Alternatives (own scheduling
+without GSP; vendor vGPU or SR-IOV paths) are analyzed and rejected or
+deployment-gated in the integration doc.
**Cost:** weeks, not days; the largest single work item in the project.
@@ -146,16 +149,19 @@ remain the top risk; the division-of-labour question above is unchanged.
### 2.3 Fault semantics
-**Unknown:** fault buffer formats and replay/cancel semantics as the *guest*
-expects them in our virtualized setting (we know the buffer sizes and that
-UVM programs them, but not yet the end-to-end contract under forwarding).
+**Partly known:** the backend-side fault observation and relay now exists
+(default off — a notifier write plus an event post; the auxiliary event post
+can fail with `no_event`). **Still unknown:** the replay/cancel contract as the
+*guest* expects it in our virtualized setting (we know the buffer sizes and
+that UVM programs them, but not the end-to-end guest-side contract under
+forwarding).
**Why:** GPU page faults happen constantly in real CUDA workloads (UVM
oversubscription, migrations); correct handling is required for robustness.
**Plan:** reproduce faults deliberately (map a VA to nothing, touch it from a
-forwarded kernel), capture both sides, implement replay; validate with
-`uvm_fault`-heavy tests (e.g. oversubscribed memory).
+forwarded kernel), capture both sides, implement the guest-side replay/cancel
+contract; validate with `uvm_fault`-heavy tests (e.g. oversubscribed memory).
**Cost:** weeks.
@@ -320,9 +326,9 @@ To make a defensible statement, we need all of:
7. **Performance:** within an agreed factor of native for the supported cases,
measured with a published methodology.
-Until all seven hold, we say "initialization works; execution is in progress",
-and we link here. The CUDA-side checklist behind these gates and the current
-blocker are in
+Until all seven hold, we say "initialization and the driver-API path work;
+engine execution is in progress", and we link here. The CUDA-side checklist
+behind these gates and the current execution frontier are in
[`../../userspace/cuda/docs/03-driver-and-gpu-interface.md` §9](../../userspace/cuda/docs/03-driver-and-gpu-interface.md);
start at
[`../../userspace/cuda/READ-ME-FIRST.md`](../../userspace/cuda/READ-ME-FIRST.md).
diff --git a/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md b/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md
index 3265f47..4b886b3 100644
--- a/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md
+++ b/docs/04-implementation/02-practical-challenges-and-adversarial-interop.md
@@ -53,9 +53,11 @@ without a methodology is not accepted in this project.
are answered from cached/derived state where the hardware has no side effects
the guest can observe.
* **Real work** (grid launches, copies) is meant to be forwarded to the
- physical GPU (see §3). Today only the work the host decoder understands —
- memsets, copies and semaphore releases — is executed in software as a
- correctness bridge; no CUDA kernel is forwarded yet.
+ physical GPU (see §3). CE-class copies and their semaphore releases are now
+ forwarded to the physical copy engine (default off) with a host-side fault
+ relay, and memsets/copies/releases the host decoder understands still run in
+ software as a fail-closed correctness bridge. No CUDA kernel is forwarded
+ yet: GR kernel execution is blocked on GR engine-context validity.
## 3. Forwarding: the options and the chosen mechanism