diff options
Diffstat (limited to 'docs/06-portability')
| -rw-r--r-- | docs/06-portability/03-device-validation-and-porting-sequence.md | 232 | ||||
| -rw-r--r-- | docs/06-portability/hardware-software-matrix.md | 46 |
2 files changed, 276 insertions, 2 deletions
diff --git a/docs/06-portability/03-device-validation-and-porting-sequence.md b/docs/06-portability/03-device-validation-and-porting-sequence.md new file mode 100644 index 0000000..3c5d1aa --- /dev/null +++ b/docs/06-portability/03-device-validation-and-porting-sequence.md @@ -0,0 +1,232 @@ +# Device Validation and Porting Sequence + +> How Open-IOV takes a new GPU device from "available to test" to a documented +> support decision: which devices can be rented or borrowed, the physical +> preparation, the supervised test-and-port sequence from driver **540** to the +> current anchor, the evidence collected per device, and the go/no-go rules. +> +> Scope note: this document describes the **validation process**; the support +> ladder and the current status of each combination live in +> [`hardware-software-matrix.md`](hardware-software-matrix.md). Nothing here +> moves a device up the ladder by itself. The companion automation is +> [`../../debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md`](../../debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md). + +## 1. Rules that never bend + +1. **No unattended claim cycles.** Every cycle that will be used to claim + support runs with a human supervising the device; the simulator harness may + run unattended, the physical device may not. +2. **Reboots need approval.** Host or device-host reboots, driver + bind/unbind, VM restarts and mdev claim/release for a claim cycle are + approved by the device owner before the cycle starts. One approved cycle at + a time. +3. **The guest never contains Open-IOV code.** No guest agent, no patched + driver, no extra kernel parameters beyond what the stock driver needs. +4. **A claim is evidence, not an intention.** "Tested and documented" only: + a device is Supported after this sequence's evidence exists and the matrix + records it with date, driver family, firmware hash and tested rung. +5. **Simulator evidence is evidence about the firmware, never about silicon.** + Physical capture or explicit "not physically validated" is stated in every + report (see [`../05-operations/01-performance-correctness-security.md`](../05-operations/01-performance-correctness-security.md) + and the accuracy contract in the tool's docs/06). + +The operational steps for a supervised cycle (preflight, capture, recovery) are +owned by the [Open-IOV Defensive Operations Runbook](../RUNBOOK.md) (§P7 and +the physical-cycle response path). This document is the porting sequence that +wraps that cycle. + +## 2. Device classes we can rent or borrow + +Support cost is dominated by *access to hardware*, so the sequence starts by +naming where each class can come from. Eligibility is about PCIe passthrough +and VFIO/mdev, not the marketing tier: + +| Class | Examples | Access path | Known constraints | +|---|---|---|---| +| Turing (RTX 20) | TU104 (RTX 2080-class) | **Owned reference**; further cards bought used | The current Reference profile and development host | +| Ampere consumer (RTX 30) | GA102/GA104 | Lab loan or purchase; workstation rental | Same passthrough path as the reference; engine inventory differs (docs/07 P2) | +| Ada consumer (RTX 40) | AD102/AD103/AD104 | Lab loan or purchase; workstation rental | ROM/VBIOS variants across board partners; inventory is profile data | +| Blackwell consumer (RTX 50) | GB202/GB203 | Vendor eval, partner loan, purchase | Newer driver/firmware family; validate the pairing before debugging | +| GB10-class | DGX Spark, Surface-class RTX laptops | Vendor eval / short loan, retail where available | aarch64 hosts, laptop ROM/VBIOS quirks, possible RV64GC and FMC partition model (docs/07 P3) | +| Datacenter compute | A100, H100/H200, L40S | Cloud bare-metal rental or partner lab; short loans | Rental instances frequently expose MIG/vGPU rather than raw PCIe passthrough — confirm before paying for a cycle | +| Datacenter/rack scale | GB200-class, NVL systems | Partner lab access only | Multi-tenant policy and MIG-aware handling; out of reach for casual rental | + +Procurement rules: + +* Prefer a **bare-metal rental with documented PCIe passthrough and IOMMU** over + a vGPU/mdev instance; the latter cannot exercise the VFIO mdev path. +* A borrowed card arrives with an unknown ROM/VBIOS and firmware pairing: + collect and hash everything in §4 before touching the emulation. +* One device per architecture/family at a time is enough to move the matrix; + do not accumulate hardware before the previous evidence bundle is complete. + +## 3. Physical preparation + +Preflight is a checklist, not a ritual (see the RUNBOOK for the host-specific +commands): + +1. **Host qualification** — x86_64 or aarch64 host, IOMMU enabled + (`intel_iommu=on`/`amd_iommu=on`, `iommu=pt` on aarch64), ACS/isolation + confirmed, kernel with `vfio-pci`; record `uname -a`, kernel, and QEMU + version. +2. **Slot and power** — card seated and powered per vendor spec; auxiliary + power connected; sufficient PSU headroom; thermals monitored for the cycle + (a throttled device invalidates timing-flavoured observations; timing is not + claimed anyway, docs/06 M7). +3. **Isolation** — bind the target GPU to `vfio-pci`; keep the host distro + driver reserved as the **oracle**, never as the emulated anchor (anchor + discipline, docs/22 §1). +4. **Baseline capture** — before any guest runs: host oracle captures + (`rmtool engine list --format=json`, `rmtool mem stats --format=json` + via `rmharness`, docs/17), `lspci -vvv` for the physical and virtual + functions, ROM/VBIOS hash, firmware family and version string. +5. **Guest image** — stock guest driver of the exact version under test, a + CUDA smoke workload and the ioctl/control tracer + (`tools/guest-diag/ioctl_trace.c`). +6. **Supervision plan** — named operator, one-cycle approval, expected + duration, watchdog, and the rollback path (old anchor directories and + config untouched; never reuse old-version addresses). +7. **Simulator rehearsal first** — run the sub-sections below in `--dry-run`, + then in simulator-only mode, before the physical cycle. A plan that cannot + complete in simulation is not ready for silicon. + +## 4. Test-and-port sequence: driver 540 to current + +The sequence is **ascending**: each branch becomes the reference for the next, +so a failure is attributed to one step, not to a jump. The current anchor is +**570.144**; the sweep continues to the newest locally available image, and +610+ stays behind the RM 610 approval gate (docs/23) until that approval +exists. The `harness/` commands below run from +`debug/risc-v-gsp-rm-vm/`; the supervised-cycle commands run on the host and +device under test. + +Per driver/firmware version `N`: + +1. **Acquire and version** — obtain the runfile and matching GSP `.fwimage` + through the project's acquisition rules; lay out + `firmware/<SERIES>-GSP/<N>/` and `config/versions/<N>.yaml` (documentation + until the load path reads it). Never overwrite a working version; hash + every input immediately. +2. **Static triage** — extract the image, build/locate the symbol DB, and run + the release pipeline in dry-run first: + `harness/version_diff.py` and `harness/ci_release.py` (docs/21). The sweep + orchestrator wraps this per version: + `python3 harness/port_driver_versions.py --versions <N> --target <N>=<fwimage> --dry-run` + (docs/28 §2; unadmitted pilot, evaluation in future-eng-ops F2). +3. **Simulator evidence** — execute the sweep against the previous branch and + the anchor: + `python3 harness/port_driver_versions.py --run --baseline <N-1> ...`; + the per-version bundle carries `version-diff/`, `portability-matrix/` and + `mutation-smoke/` results plus the M1–M6 skeleton. +4. **Device profile** — for a new device run + `python3 harness/port_new_device.py --run --device <name> --firmware <N-image> --reference <anchor-image>` + (docs/28 §3; unadmitted pilot, evaluation in future-eng-ops F3); fill the + profile from §4 evidence and reconcile the engine inventory against the + host oracle. +5. **Supervised physical cycle** (approved, attended): + * boot the guest with the stock driver of version `N` on the device bound + to `vfio-pci`; run the CUDA smoke and the ioctl tracer; + * capture host module trace (`gsp-resman-vfio:`), guest dmesg, + `nvidia-smi -q`, `lspci -vvv`, and the control/ioctl sequence; + * compare the physical captures with the simulator sessions + (`greeneye compare`, M1–M6) and resolve every delta as profile, logic or + fidelity gap (docs/07 P4). +6. **No new guest regression** — run the existing guest/CUDA ladder rungs and + confirm no rung regressed before the next version. +7. **Rollback point** — keep the previous version's directories, config and + guest driver untouched until the next version's bundle is complete. + +Only after the current branch's bundle is complete does it become the +reference for the next branch. 610 is not attempted here: see +[`../../debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md`](../../debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md). + +## 5. Sequence diagram + +```mermaid +sequenceDiagram + autonumber + participant O as Operator + participant H as Ops host + participant D as Device under test + participant G as Guest VM (stock driver) + participant S as green-eye harness + participant E as Evidence store + + O->>H: approve one supervised cycle (reboot/bind approval) + O->>H: preflight (IOMMU, vfio-pci bind, baselines) + H->>D: bind target GPU to vfio-pci + O->>S: sweep --dry-run for version N + S-->>E: plan, input hashes + O->>S: sweep --run vs baseline (simulator only) + S-->>E: version-diff + matrix + mutation bundles + O->>G: start guest with stock driver N, run CUDA smoke + G->>D: GSP RPC / control traffic + O->>H: capture host trace, guest dmesg, lspci, oracle + H-->>E: physical captures + O->>S: compare physical vs simulator (M1-M6) + S-->>E: delta report, fidelity gaps + O->>O: go/no-go decision + alt go + O->>E: publish bundle + matrix entry + else hold + O->>E: record gap, keep previous anchor + end + D->>H: rollback bind, restore host driver +``` + +## 6. Evidence collected per device + +| Item | Source | Artifact | +|---|---|---| +| Device profile (PCI IDs, BARs, FB geometry, engines, interrupts, ROM) | `port_new_device` profile + host sysfs | `profiles/<device>.json` | +| Admission/authorisation record | Ops process | cycle approval ID | +| Physical host facts | preflight | `lspci -vvv`, ROM/VBIOS hash, kernel/QEMU versions | +| Host oracle snapshots | `rmharness`/`rmtool` | `engine list`, `mem stats` JSON in a session | +| Firmware pairing | image hash + family + version string | top of the port report | +| Simulator bundle | `port_driver_versions`/`port_new_device` | `bundle-manifest.json` + tree, docs/28 | +| Physical captures | supervised cycle | host module trace, guest dmesg, `nvidia-smi -q`, ioctl sequence | +| Metric table | comparison | M1–M6 with reference-capture method and L2/L3 limits (docs/06 §5) | +| Unexplained-delta list | review | explicit, possibly empty; empty is a claim | +| Support decision | review | matrix entry with rung, date, driver/firmware, evidence hash | + +## 7. Go/no-go criteria + +Proceed to the next driver or publish a matrix entry only when **all** hold: + +* pairing verified and hashed before debugging (family, version string, FMC + manifest where applicable); +* every probe first-touch delta explained as profile data vs logic; +* engine/RPC inventory complete and reconciled against the host oracle; +* M1–M6 do not regress on the reference workload, or each regression has a + decided response; +* no guest-ladder rung regressed; rollback path intact; +* the evidence bundle is complete and hashed, including the physical capture; +* the physical cycle was supervised and approved. + +Hold (keep the previous anchor) when any of: ambiguous pairing, an unexplained +probe or inventory delta, a matrix/M-metric regression without a response, an +incomplete bundle, or a simulator-only claim with no physical capture. A +**fidelity gap** (simulator diverges, physical cannot reproduce) is recorded +as a gap, never "fixed" in the product from simulator evidence alone +(docs/07 P4.5). + +## 8. References + +* Support ladder and per-combination status: + [`hardware-software-matrix.md`](hardware-software-matrix.md) +* Current ops entry point and host procedures: + [`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md), + the [Open-IOV Defensive Operations Runbook](../RUNBOOK.md), and the + [green-eye user guide](../05-operations/04-green-eye-user-guide.md) +* Porting automation (sweep + device scaffold, bundles, M1–M6): + [`../../debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md`](../../debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md) +* Playbooks and porting guide: + [`../../debug/risc-v-gsp-rm-vm/docs/07-PORTABILITY-PLAYBOOKS.md`](../../debug/risc-v-gsp-rm-vm/docs/07-PORTABILITY-PLAYBOOKS.md), + [`../../debug/risc-v-gsp-rm-vm/docs/22-PORTING-CUDA-AND-DEVICES.md`](../../debug/risc-v-gsp-rm-vm/docs/22-PORTING-CUDA-AND-DEVICES.md) +* RM 610 gate: + [`../../debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md`](../../debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md) + +<!-- change (2026-10-02): correct the guest-diag tracer path, state the working +directory for the harness commands, and mark the porting scripts as unadmitted +pilots with their evaluation pointers; scope: §3.1, §4 only; benefit: commands +copy correctly and the scaffolds are not read as routine pipeline steps. --> diff --git a/docs/06-portability/hardware-software-matrix.md b/docs/06-portability/hardware-software-matrix.md index c839578..8fca9e9 100644 --- a/docs/06-portability/hardware-software-matrix.md +++ b/docs/06-portability/hardware-software-matrix.md @@ -35,7 +35,7 @@ yet and selection is compile-time (see §6). | Guest | Status | |---|---| -| Linux x86_64 (stock `nvidia.ko`, CUDA) | **Experimental** (bring-up; probe and `nvidia-smi` work, but there is no continuous guest testing and context creation is currently blocked — see §6.1) | +| Linux x86_64 (stock `nvidia.ko`, CUDA) | **Experimental** (bring-up; probe, `nvidia-smi` and the driver-API smoke pass, and CE copies are forwarded default-off, but kernel execution is blocked on GR engine-context validity and there is no continuous guest testing — see §6.1) | | Linux aarch64 | **Target** | | Windows x86_64 (WDDM driver, CUDA, 3D) | **Target** | | Other Unix (FreeBSD, ...) | Out of scope until demand | @@ -70,11 +70,34 @@ so new transports are a small, isolated implementation. 3. Record the differences; prefer configuration over code. 4. Publish the profile, the evidence, and any workarounds. +**Green-eye portability pipeline (experimental).** + +<!-- change (2026-10-02): add the green-eye portability pipeline to the new-GPU +procedure and point at the RUNBOOK/user guide; additive only. --> + +Before the profile change lands, run the firmware-side pipeline from +`debug/risc-v-gsp-rm-vm/` and attach its output: +`./greeneye.sh probe <image> --symbols <db> --record` for the first-touch map; +`./greeneye.sh symbols export ...` for a provenance-carrying symbol DB; +`python3 harness/portability_matrix.py --target <reference> --target <candidate> +--out dumps/matrix-<candidate>` for per-pair sessions and `compare` reports; +then classify profile-versus-logic deltas with the decision table in +`debug/risc-v-gsp-rm-vm/docs/14-ADVERSARIAL-INTEROP-GUIDE.md` §4.1. The +command-level walkthrough is in the +[green-eye user guide](../05-operations/04-green-eye-user-guide.md); the +incident procedure, evidence expectations and rollback rules are in the +[Open-IOV Defensive Operations Runbook](../RUNBOOK.md) S5. + ### 5.2 New firmware version See [`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md) §4.1. Firmware images are versioned under `firmware/<SERIES>-GSP/<VERSION>/` with -signatures and analysis artifacts. +signatures and analysis artifacts. The defensive procedure, evidence bundle +and rollback rules for this path are in the +[Open-IOV Defensive Operations Runbook](../RUNBOOK.md) S1. + +<!-- change (2026-10-02): cross-link the new firmware-version procedure to the +defensive-operations RUNBOOK S1; additive only. --> ### 5.3 New driver version @@ -118,6 +141,25 @@ config/ A profile is the *unit of support* — what is missing is the code that reads the profile files, not the profile schema. +**Evidence expectations for adding support.** + +<!-- change (2026-10-02): state the evidence required before a combination is +entered in the matrix, and link the RUNBOOK bundle checklist and green-eye +portability pipeline; additive only. --> + +A combination is entered in the matrix only with: the profile fields recorded; +the firmware/device pairing verified by hash and version string before any +debugging; a first-touch diff whose every changed address is explained as +profile or logic; a portability matrix report with per-pair `compare` results +and session hashes; the M1–M6 table with the reference-capture method and the +L2/L3 limits stated; and a supervised physical validation for the support +claim. "Tested and documented" is literal: an untested or simulator-only +combination stays **Target** or **Experimental**. The full bundle checklist +and fault tree are in the +[Open-IOV Defensive Operations Runbook](../RUNBOOK.md) §6–§7; the pipeline +commands are in the +[green-eye user guide](../05-operations/04-green-eye-user-guide.md). + ### 6.1 Userspace driver facts * **Guest CUDA driver** (`libcuda`) under test: **570.144** (md5 |