From c17da06d6cd8b9c69449904637bf37729a962989 Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Fri, 2 Oct 2026 11:28:49 -0400 Subject: docs: engineering-operations playbook and project-wide status accuracy pass - docs/eng-ops.md gains Part II, live physical-backend engineering operations: hard rules, the full module-parameter matrix, the routine supervised cycle and clean-release checklist, the debugfs inventory, how to read gsp-phys-state, a debugging scenario catalogue (CTXNOTVALID/promote wedge, fail-safe WPR2, unload hangs, guest wedges, MMU/PTE and PBDMA faults, relay symptoms), a hazard register (crash causes and debugging side effects), the physical-backend OODA loop, and the work-context map with porting flows - architecture, implementation, operations, orientation, getting-started, portability, ecosystem and reference docs corrected to the current state: CUDA driver-API smoke green, CE-class work forwarding and engine-authored releases plus fault relay in place (default off), GR execution blocked on engine-context validity, the promote wedge and designed locked fail-safe, and P5 wave-2 subsumed - internal-reference cleanup in tracked docs (no analysis-source or local-archive references); historical architecture document bannered - README reading list and STATE-CURRENT frontier refreshed; new pages (runbook, green-eye user guide, porting sequence, porting/packaging notes) linked from the tree Link check: scripts/check-docs-links.py passes (56 files, no broken links). --- .../01-performance-correctness-security.md | 23 ++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) (limited to 'docs/05-operations/01-performance-correctness-security.md') diff --git a/docs/05-operations/01-performance-correctness-security.md b/docs/05-operations/01-performance-correctness-security.md index 3a2c6b4..21afd06 100644 --- a/docs/05-operations/01-performance-correctness-security.md +++ b/docs/05-operations/01-performance-correctness-security.md @@ -17,12 +17,14 @@ Rung 4 SUSTAINED workloads run for hours; faults/errors recoverable Rung 5 MULTI-TENANT N VMs, isolation + fairness + no cross-talk ``` -**Current rung (2026-09-29):** Rung 1 is green; **Rung 2 is in progress.** -`cuInit`/`cuDeviceGet` succeed, and the client reaches `cuCtxCreate`, which -currently ends in a forced `rc=999` on a completion wait (engine inventory -complete; completion translation in progress); `cuMemAlloc` is not reached yet -and failed-context teardown has oopsed in the guest kernel in earlier runs. -Before Rung 3 is claimed, the +**Current rung (2026-10-02):** Rungs 1–2 are green: the stock driver probes, +`nvidia-smi` enumerates, and the full CUDA driver-API smoke passes +(`ALL OK` 64/64, copies matched) — `cuInit`, device queries, context creation, +allocation, copies, free and destroy complete, and the copies are verified +byte-for-byte. **Rung 3 is partial:** CE-class copies execute on the physical +copy engine (default off) with engine-authored releases and a fault relay, but +CUDA kernel execution is blocked on GR engine-context validity. Before Rung 3 +is claimed, the **reply-fidelity gate** applies: a control whose reply fields the driver consumes is not implemented until the reply is coherent (status-only → coherent zeroed params → coherent values for consumed fields → exact values) — @@ -64,8 +66,9 @@ New to CUDA analysis here: start at `./scripts/stack_manager.sh test` runs the host suite (`make tests` only builds the test binaries). CI (`.github/workflows/ci.yml`) runs on push and pull requests only — there is currently no scheduled guest suite and no -stock-driver contract test. A rung regression is a release blocker; the -handoff documents carry the current rung. +stock-driver contract test. A rung regression is a release blocker; +[`STATE-CURRENT.md`](../../STATE-CURRENT.md) and this docs tree carry the +current rung. ## 3. Performance @@ -93,6 +96,10 @@ handoff documents carry the current rung. * **Publish the harness**: the benchmark scripts live in `tools/`, and a result is not comparable unless produced by them (or an equivalent, documented harness). +* **Runbooks over methodology**: the live physical-backend runbooks, scenario + scripts, debugfs surfaces and module parameters are collected in + [`eng-ops.md` Part II](../eng-ops.md); repeatable procedures live there, and + this document stays the measurement contract. ### 3.3 Targets -- cgit v1.2.3