1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
|
# Device Validation and Porting Sequence
> How Open-IOV takes a new GPU device from "available to test" to a documented
> support decision: which devices can be rented or borrowed, the physical
> preparation, the supervised test-and-port sequence from driver **540** to the
> current anchor, the evidence collected per device, and the go/no-go rules.
>
> Scope note: this document describes the **validation process**; the support
> ladder and the current status of each combination live in
> [`hardware-software-matrix.md`](hardware-software-matrix.md). Nothing here
> moves a device up the ladder by itself. The companion automation is
> [`../../debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md`](../../debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md).
## 1. Rules that never bend
1. **No unattended claim cycles.** Every cycle that will be used to claim
support runs with a human supervising the device; the simulator harness may
run unattended, the physical device may not.
2. **Reboots need approval.** Host or device-host reboots, driver
bind/unbind, VM restarts and mdev claim/release for a claim cycle are
approved by the device owner before the cycle starts. One approved cycle at
a time.
3. **The guest never contains Open-IOV code.** No guest agent, no patched
driver, no extra kernel parameters beyond what the stock driver needs.
4. **A claim is evidence, not an intention.** "Tested and documented" only:
a device is Supported after this sequence's evidence exists and the matrix
records it with date, driver family, firmware hash and tested rung.
5. **Simulator evidence is evidence about the firmware, never about silicon.**
Physical capture or explicit "not physically validated" is stated in every
report (see [`../05-operations/01-performance-correctness-security.md`](../05-operations/01-performance-correctness-security.md)
and the accuracy contract in the tool's docs/06).
The operational steps for a supervised cycle (preflight, capture, recovery) are
owned by the [Open-IOV Defensive Operations Runbook](../RUNBOOK.md) (§P7 and
the physical-cycle response path). This document is the porting sequence that
wraps that cycle.
## 2. Device classes we can rent or borrow
Support cost is dominated by *access to hardware*, so the sequence starts by
naming where each class can come from. Eligibility is about PCIe passthrough
and VFIO/mdev, not the marketing tier:
| Class | Examples | Access path | Known constraints |
|---|---|---|---|
| Turing (RTX 20) | TU104 (RTX 2080-class) | **Owned reference**; further cards bought used | The current Reference profile and development host |
| Ampere consumer (RTX 30) | GA102/GA104 | Lab loan or purchase; workstation rental | Same passthrough path as the reference; engine inventory differs (docs/07 P2) |
| Ada consumer (RTX 40) | AD102/AD103/AD104 | Lab loan or purchase; workstation rental | ROM/VBIOS variants across board partners; inventory is profile data |
| Blackwell consumer (RTX 50) | GB202/GB203 | Vendor eval, partner loan, purchase | Newer driver/firmware family; validate the pairing before debugging |
| GB10-class | DGX Spark, Surface-class RTX laptops | Vendor eval / short loan, retail where available | aarch64 hosts, laptop ROM/VBIOS quirks, possible RV64GC and FMC partition model (docs/07 P3) |
| Datacenter compute | A100, H100/H200, L40S | Cloud bare-metal rental or partner lab; short loans | Rental instances frequently expose MIG/vGPU rather than raw PCIe passthrough — confirm before paying for a cycle |
| Datacenter/rack scale | GB200-class, NVL systems | Partner lab access only | Multi-tenant policy and MIG-aware handling; out of reach for casual rental |
Procurement rules:
* Prefer a **bare-metal rental with documented PCIe passthrough and IOMMU** over
a vGPU/mdev instance; the latter cannot exercise the VFIO mdev path.
* A borrowed card arrives with an unknown ROM/VBIOS and firmware pairing:
collect and hash everything in §4 before touching the emulation.
* One device per architecture/family at a time is enough to move the matrix;
do not accumulate hardware before the previous evidence bundle is complete.
## 3. Physical preparation
Preflight is a checklist, not a ritual (see the RUNBOOK for the host-specific
commands):
1. **Host qualification** — x86_64 or aarch64 host, IOMMU enabled
(`intel_iommu=on`/`amd_iommu=on`, `iommu=pt` on aarch64), ACS/isolation
confirmed, kernel with `vfio-pci`; record `uname -a`, kernel, and QEMU
version.
2. **Slot and power** — card seated and powered per vendor spec; auxiliary
power connected; sufficient PSU headroom; thermals monitored for the cycle
(a throttled device invalidates timing-flavoured observations; timing is not
claimed anyway, docs/06 M7).
3. **Isolation** — bind the target GPU to `vfio-pci`; keep the host distro
driver reserved as the **oracle**, never as the emulated anchor (anchor
discipline, docs/22 §1).
4. **Baseline capture** — before any guest runs: host oracle captures
(`rmtool engine list --format=json`, `rmtool mem stats --format=json`
via `rmharness`, docs/17), `lspci -vvv` for the physical and virtual
functions, ROM/VBIOS hash, firmware family and version string.
5. **Guest image** — stock guest driver of the exact version under test, a
CUDA smoke workload and the ioctl/control tracer
(`tools/guest-diag/ioctl_trace.c`).
6. **Supervision plan** — named operator, one-cycle approval, expected
duration, watchdog, and the rollback path (old anchor directories and
config untouched; never reuse old-version addresses).
7. **Simulator rehearsal first** — run the sub-sections below in `--dry-run`,
then in simulator-only mode, before the physical cycle. A plan that cannot
complete in simulation is not ready for silicon.
## 4. Test-and-port sequence: driver 540 to current
The sequence is **ascending**: each branch becomes the reference for the next,
so a failure is attributed to one step, not to a jump. The current anchor is
**570.144**; the sweep continues to the newest locally available image, and
610+ stays behind the RM 610 approval gate (docs/23) until that approval
exists. The `harness/` commands below run from
`debug/risc-v-gsp-rm-vm/`; the supervised-cycle commands run on the host and
device under test.
Per driver/firmware version `N`:
1. **Acquire and version** — obtain the runfile and matching GSP `.fwimage`
through the project's acquisition rules; lay out
`firmware/<SERIES>-GSP/<N>/` and `config/versions/<N>.yaml` (documentation
until the load path reads it). Never overwrite a working version; hash
every input immediately.
2. **Static triage** — extract the image, build/locate the symbol DB, and run
the release pipeline in dry-run first:
`harness/version_diff.py` and `harness/ci_release.py` (docs/21). The sweep
orchestrator wraps this per version:
`python3 harness/port_driver_versions.py --versions <N> --target <N>=<fwimage> --dry-run`
(docs/28 §2; unadmitted pilot, evaluation in future-eng-ops F2).
3. **Simulator evidence** — execute the sweep against the previous branch and
the anchor:
`python3 harness/port_driver_versions.py --run --baseline <N-1> ...`;
the per-version bundle carries `version-diff/`, `portability-matrix/` and
`mutation-smoke/` results plus the M1–M6 skeleton.
4. **Device profile** — for a new device run
`python3 harness/port_new_device.py --run --device <name> --firmware <N-image> --reference <anchor-image>`
(docs/28 §3; unadmitted pilot, evaluation in future-eng-ops F3); fill the
profile from §4 evidence and reconcile the engine inventory against the
host oracle.
5. **Supervised physical cycle** (approved, attended):
* boot the guest with the stock driver of version `N` on the device bound
to `vfio-pci`; run the CUDA smoke and the ioctl tracer;
* capture host module trace (`gsp-resman-vfio:`), guest dmesg,
`nvidia-smi -q`, `lspci -vvv`, and the control/ioctl sequence;
* compare the physical captures with the simulator sessions
(`greeneye compare`, M1–M6) and resolve every delta as profile, logic or
fidelity gap (docs/07 P4).
6. **No new guest regression** — run the existing guest/CUDA ladder rungs and
confirm no rung regressed before the next version.
7. **Rollback point** — keep the previous version's directories, config and
guest driver untouched until the next version's bundle is complete.
Only after the current branch's bundle is complete does it become the
reference for the next branch. 610 is not attempted here: see
[`../../debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md`](../../debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md).
## 5. Sequence diagram
```mermaid
sequenceDiagram
autonumber
participant O as Operator
participant H as Ops host
participant D as Device under test
participant G as Guest VM (stock driver)
participant S as risc-v-gsp-rm-debugger harness
participant E as Evidence store
O->>H: approve one supervised cycle (reboot/bind approval)
O->>H: preflight (IOMMU, vfio-pci bind, baselines)
H->>D: bind target GPU to vfio-pci
O->>S: sweep --dry-run for version N
S-->>E: plan, input hashes
O->>S: sweep --run vs baseline (simulator only)
S-->>E: version-diff + matrix + mutation bundles
O->>G: start guest with stock driver N, run CUDA smoke
G->>D: GSP RPC / control traffic
O->>H: capture host trace, guest dmesg, lspci, oracle
H-->>E: physical captures
O->>S: compare physical vs simulator (M1-M6)
S-->>E: delta report, fidelity gaps
O->>O: go/no-go decision
alt go
O->>E: publish bundle + matrix entry
else hold
O->>E: record gap, keep previous anchor
end
D->>H: rollback bind, restore host driver
```
## 6. Evidence collected per device
| Item | Source | Artifact |
|---|---|---|
| Device profile (PCI IDs, BARs, FB geometry, engines, interrupts, ROM) | `port_new_device` profile + host sysfs | `profiles/<device>.json` |
| Admission/authorisation record | Ops process | cycle approval ID |
| Physical host facts | preflight | `lspci -vvv`, ROM/VBIOS hash, kernel/QEMU versions |
| Host oracle snapshots | `rmharness`/`rmtool` | `engine list`, `mem stats` JSON in a session |
| Firmware pairing | image hash + family + version string | top of the port report |
| Simulator bundle | `port_driver_versions`/`port_new_device` | `bundle-manifest.json` + tree, docs/28 |
| Physical captures | supervised cycle | host module trace, guest dmesg, `nvidia-smi -q`, ioctl sequence |
| Metric table | comparison | M1–M6 with reference-capture method and L2/L3 limits (docs/06 §5) |
| Unexplained-delta list | review | explicit, possibly empty; empty is a claim |
| Support decision | review | matrix entry with rung, date, driver/firmware, evidence hash |
## 7. Go/no-go criteria
Proceed to the next driver or publish a matrix entry only when **all** hold:
* pairing verified and hashed before debugging (family, version string, FMC
manifest where applicable);
* every probe first-touch delta explained as profile data vs logic;
* engine/RPC inventory complete and reconciled against the host oracle;
* M1–M6 do not regress on the reference workload, or each regression has a
decided response;
* no guest-ladder rung regressed; rollback path intact;
* the evidence bundle is complete and hashed, including the physical capture;
* the physical cycle was supervised and approved.
Hold (keep the previous anchor) when any of: ambiguous pairing, an unexplained
probe or inventory delta, a matrix/M-metric regression without a response, an
incomplete bundle, or a simulator-only claim with no physical capture. A
**fidelity gap** (simulator diverges, physical cannot reproduce) is recorded
as a gap, never "fixed" in the product from simulator evidence alone
(docs/07 P4.5).
## 8. References
* Support ladder and per-combination status:
[`hardware-software-matrix.md`](hardware-software-matrix.md)
* Current ops entry point and host procedures:
[`../05-operations/02-debugging-and-maintenance-workflows.md`](../05-operations/02-debugging-and-maintenance-workflows.md),
the [Open-IOV Defensive Operations Runbook](../RUNBOOK.md), and the
[risc-v-gsp-rm-debugger user guide](../05-operations/04-risc-v-gsp-rm-debugger-user-guide.md)
* Porting automation (sweep + device scaffold, bundles, M1–M6):
[`../../debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md`](../../debug/risc-v-gsp-rm-vm/docs/28-PORTING-AUTOMATION.md)
* Playbooks and porting guide:
[`../../debug/risc-v-gsp-rm-vm/docs/07-PORTABILITY-PLAYBOOKS.md`](../../debug/risc-v-gsp-rm-vm/docs/07-PORTABILITY-PLAYBOOKS.md),
[`../../debug/risc-v-gsp-rm-vm/docs/22-PORTING-CUDA-AND-DEVICES.md`](../../debug/risc-v-gsp-rm-vm/docs/22-PORTING-CUDA-AND-DEVICES.md)
* RM 610 gate:
[`../../debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md`](../../debug/risc-v-gsp-rm-vm/docs/23-RM610-READINESS.md)
<!-- change (2026-10-02): correct the guest-diag tracer path, state the working
directory for the harness commands, and mark the porting scripts as unadmitted
pilots with their evaluation pointers; scope: §3.1, §4 only; benefit: commands
copy correctly and the scaffolds are not read as routine pipeline steps. -->
|
