blob: b9bd3879836b7459180301b5a56c46914c81029e (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
|
# Open-IOV Version Upgrade Guide
> How to support a new NVIDIA driver version (e.g. moving from 570.144 to a
> 610-series release) end-to-end: firmware acquisition → reverse engineering →
> config → host driver → guest validation. The goal is that a version bump is
> **configuration + analysis**, not a rewrite.
---
## 0. Version discipline (read first)
Four components must stay in lock-step; a mismatch between any two produces
failures that are indistinguishable from emulation bugs:
```
guest nvidia driver == host OpenRM nvidia.ko == host GSP firmware == emulation target (decompiled GSP)
```
Today that anchor is **570.144** everywhere. When you bump one, you bump all
four. Verify with `docs/DEBUGGING.md` §7 before any deep debugging.
---
## 1. Acquire the new firmware + matching OpenRM tag
1. Download the Linux x86_64 runfile for the target version:
`https://us.download.nvidia.com/XFree86/Linux-x86_64/<VER>/NVIDIA-Linux-x86_64-<VER>.run`
(the `tesla/` path 404s for consumer releases).
2. Extract the GSP firmware (`gsp_tu10x.bin` / arch-specific) — or fetch the
versioned blob from `linux-firmware.git`:
`nvidia/<VER>/gsp_tu10x.bin`.
3. Extract the `.fwimage` section (host boot image) and record the MD5:
```bash
objcopy --only-section=.fwimage gsp_tu10x.bin gsp-<VER>.fwimage.bin
md5sum gsp-<VER>.fwimage.bin
```
4. Note the matching OpenRM tag at
`github.com/NVIDIA/open-gpu-kernel-modules` (e.g. `570.144`).
## 2. Lay down versioned directories (never overwrite a working version)
```
firmware/<SERIES>-GSP/<VER>/ # e.g. firmware/61X-GSP/610.XX/
gsp-<VER>.fwimage.bin
decompiled/ # ghidra output lands here
config/versions/<MAJOR>.yaml # e.g. config/versions/610.yaml
```
Copy the analysis scripts from the previous version dir
(`firmware/57X-GSP/570.144/*.py`) into the new one — they are script-relative.
## 3. Reverse-engineer the new firmware (Ghidra headless)
```bash
cd tools/ghidra
python3 analyze_fwimage.py \
../../firmware/<SERIES>-GSP/<VER>/gsp-<VER>.fwimage.bin \
/tmp/ghidra_project_<VER> "gsp_<VER>"
cd ../../firmware/<SERIES>-GSP/<VER>
python3 analyze_firmware.py gsp-<VER>.fwimage.bin
```
What to extract (compare against `wiki/Address-Reference.md` for 570.144W):
| Artifact | Why it matters |
|----------|----------------|
| Primary RPC dispatcher + jump table | RPC types/subcommands may shift |
| BAR trap / WPR handlers | BAR emulation behavior |
| Entry/boot sequence | GSP boot RPC ordering the guest driver expects |
| Migration tables | migration protocol (if enabled) |
| RPC message layouts | struct changes break the emulator silently |
| Version string/signature | runtime firmware matching |
Sanity checks: still RV64IMAC? still the same 12 top-level RPC types? new
subcommands? Confidential-computing/TEE additions?
## 4. Produce the version config
Update `config/versions/<MAJOR>.yaml` with:
- `version.name/major/minor` and `firmware.gsp_version`
- discovered dispatcher/handler addresses (if tracked in yaml)
- `features` toggles per what the firmware actually has
- `symbols` (the nvidia.ko vGPU VFIO interface — verify against the new
OpenRM tag; renamed exports break the nvidia build)
## 5. Update the runtime signature database
`kernel/firmware_analyzer.c` matches firmware by CRC32/SHA256 + patterns.
Add the new entry to `known_signatures[]` (version string, hashes, discovered
addresses). This is what lets one `open-iov-vfio.ko` serve multiple firmware
versions safely.
## 6. Rebuild host + guest against the new anchor
1. Point `legacy-patches/nvidia/` at the new OpenRM tag (rebase our delta; verify
`modinfo nvidia.ko | grep depends` still shows `open-iov-vfio`).
2. `./scripts/stack_manager.sh build-all` (kernel → nvidia → userspace).
3. Install the new host firmware to `/lib/firmware/`
(`stack_manager.sh install-firmware`).
4. **Guest**: install the *matching* stock driver in the VM (runfile for the
exact version), replacing the old one. Keep the guest able to boot without
nvidia auto-loading during the swap (temporarily move dkms-built modules
aside — see `docs/DEBUGGING.md`).
## 7. Validate (same checklist every version)
Run the full bring-up checklist in `docs/DEBUGGING.md` §5:
module load → mdev → VM boot → guest probe → `nvidia-smi` → teardown
regression → unit test. Then the workload tests (CUDA vectorAdd, graphics).
Only after the checklist is green do performance comparisons
(`docs/PERFORMANCE.md`).
## 8. Rollback plan
Keep the previous version's directory + config untouched. Rollback =
`git checkout` the repo tag/commit for the old anchor + reinstall the old
guest driver. Because versions live in parallel directories, the working
setup is never destroyed by a forward attempt.
## 9. Multi-version testing matrix (target state)
For each supported anchor version × each supported GPU family
(`config/hardware/*.yaml`), the CI matrix should run the §7 checklist. When
adding a version, extend the matrix rather than replacing the old column —
regressions across versions are the cheapest bugs to catch.
## 10. Common pitfalls (learned the hard way)
- **Runfile version drift**: Ubuntu's `nvidia-driver-XXX-open` metapackages
can resolve to a *newer* major (570-open pulled 580). Always check the
candidate version before installing; prefer the exact-version runfile.
- **GSP firmware filename**: guest driver requests `nvidia/<VER>/gsp_tu10x.bin`;
a wrong-version firmware file = probe failure that looks like an emulation bug.
- **Kernel text addresses differ per build**: re-derive Ghidra addresses per
firmware; never carry 570.144 addresses into a 610 config.
- **Symbol renames across OpenRM tags**: `nvidia_vgpu_vfio_*` renames break
the nvidia build with `Unknown symbol` — check `kernel-open` diffs.
|
