1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
|
# 17 — RM host oracle harness (`rmharness`)
> **# TOOL IN DEVELOPMENT - NOT READY YET!** See 00 for status.
> The RM tools are read-only host diagnostics. They are development tooling:
> never shipped, never present in a guest, never a driver dependency.
## 1. Purpose
The GSP emulation can only be trusted against ground truth. Two of the
metrics in [06-ACCURACY](06-ACCURACY.md) need the real host RM as a reference:
* **M1 — inventory coverage**: which controls/engines the real RM exposes;
* **M6 — memory side-effects / fixed points**: real VRAM/heap/BAR1 numbers
and engine inventory at a moment in time.
`rmharness` packages the host-side oracle tools and wires their output into
risc-v-gsp-rm-debugger sessions, so a simulator/emulation session and a host RM capture
can be recorded, diffed and cited together. It is the host half of the
accuracy loop (H4.5), not a boot path or a replacement for PLAN B traces.
## 2. Tool inventory
| tool | source | role in the loop |
|---|---|---|
| `rmtool` | `tools/rmtool/` (new) | CLI oracle: `gpu list/topology/info`, `engine list/info`, `mem stats`, `nv2080 class list`, allowlisted `nv2080 method call`, `sample timer`; `--format=table\|json\|csv` |
| `rmtop` | `tools/rmtop/` (new) | ncurses dashboard over the same query layer; `--once` renders table/json/csv for capture |
| `rmq` | `tools/rmq/` (pre-existing, unmodified) | FIFO/INTERNAL device info tables for TU104 static data |
Both new tools share one ioctl layer (`tools/rmtool/src/rm_api.c`,
`query.c`): version check, `NV01_ROOT_CLIENT` -> `NV01_DEVICE_0` ->
`NV20_SUBDEVICE_0` alloc ladder, and `NV_ESC_RM_CONTROL` queries. The
device-open protocol follows what `nvidia-smi` does: open
`/dev/nvidia<N>`, `NV_ESC_WAIT_OPEN_COMPLETE`, then allocate on
`/dev/nvidiactl`. The ABI structs match the running driver (the in-tree
userspace SDK snapshot is older and its 40-byte `NVOS64` is rejected).
## 3. Build
```bash
make -C tools/rmtool # builds rmtool + lib/librmapi.a
make -C tools/rmtop # builds rmtop (ncursesw)
make -C tools/rmq # unchanged, still builds
```
Requirements: gcc, ncursesw for rmtop. No external SDK, NVML or
libnvidia-ml linkage. `make clean` removes all artifacts.
## 4. Running
```bash
tools/rmtool/rmtool --help
tools/rmtool/rmtool gpu list --format=json
tools/rmtool/rmtool gpu info -d 0
tools/rmtool/rmtool engine list
tools/rmtool/rmtool mem stats
tools/rmtool/rmtool sample timer --interval=100ms --count=20
tools/rmtop/rmtop --once
tools/rmtop/rmtop -g 0 --interval=250 # interactive (needs a TTY)
```
`gpu list`/`gpu topology` need no device access (they read
`/proc/driver/nvidia/gpus` and PCI sysfs). Everything else opens a
short-lived RM client and issues read-only controls; this is the same
access model as `nvidia-smi` and is safe alongside a running CUDA job.
`--device-spec`/`-d` accepts a minor (`0`) or a bus id (`0000:01:00.0`) and
maps to the RM `deviceId`; `--device PATH` overrides the `/dev/nvidiaN`
node.
## 5. risc-v-gsp-rm-debugger integration
Wrappers `rmharness/rmtool.sh`, `rmharness/rmtop.sh`, `rmharness/rmq.sh`
locate and build the tools and pass arguments through. `rmharness/harness.sh`
adds capture:
```bash
cd debug/risc-v-gsp-rm-vm
./rmharness/harness.sh --session dumps/real-rm-before --name before \
-- rmtool engine list --format=json
./rmharness/harness.sh --session dumps/real-rm-before -- rmtop --once
```
The harness appends one `rm_harness_run` event per run and stores
stdout/stderr under `<session>/blobs/` with sizes and SHA-256. Existing
sessions are appended to, never rewritten; a missing `session.json` is
created with the `greeneye-session/1` format. Because these runs are
external, the event `t` field is `0.0`; `start`/`end` (epoch seconds) carry
the wall clock. `greeneye analyze` tolerates the session; the run count shows
up in the event summary.
`plugins/rmharness_plugin.py` exposes the same wrappers in the REPL
(`rmtool`, `rmtop`, `rmq`), recording through `ge.recorder.blob()` +
`ge.recorder.event()`. Write-looking arguments are refused and recorded as
`rm_harness_skip`. The plugin is stdlib-only and is exercised without QEMU
by `rmharness/tests/test_rmharness_plugin.py` (fake `ge`).
## 6. Safety contract (absolute)
* Host-side read-only queries only. The tools allocate only an RM client /
device / subdevice handle pair; they never create engine, channel or memory
objects.
* Never load/unload kernel modules, bind/unbind the driver, create/destroy
mdevs, claim the physical GPU, reboot, or touch `/sys/fs/pstore`.
* Write paths are enforced in code, not by convention:
* `rmtool` refuses `mem alloc/map/unmap/pin`, `engine allocate/start/stop/
config`, and `channel/context/resserve/cudasw/resman/gsp` with exit 4/3;
* `nv2080 method call` only sends controls on a read-only allowlist
(`GPU_GET_INFO_V2`, `GPU_GET_NAME_STRING`, `GPU_GET_ENGINES_V2`,
`MC_GET_ARCH_INFO`, `FB_GET_INFO_V2`, `BUS_GET_PCI_INFO`,
`TIMER_GET_TIME`, `FIFO_GET_DEVICE_INFO_TABLE`,
`GPU_GET_CLASSLIST_V2`, ...) and refuses anything else before the ioctl;
* `harness.sh` only accepts `rmtool`, `rmtop`, `rmq`;
* the plugin blocks write-looking arguments.
* Another agent may be running a live CUDA/PLAN B job; the tools do not
touch its context and all smoke checks above run without a GPU claim.
## 7. What was and was not run
Smoke checks executed on the development host (driver 570.211.01, TU104
RTX 2080 Super Max-Q, one GPU at `0000:01:00.0`):
* `--help`/`--version` for both tools; `gpu list`, `gpu topology` (proc/sysfs
only);
* `gpu info`, `engine list`, `mem stats`, `sample timer`, `rmtop --once`
(read-only RM controls) — all returned live data (`TU100`, 10 engines GR +
5 CE + NVDEC + NVENC + SW + SEC2, 8 GiB VRAM, BAR1 256 MiB);
* `harness.sh` session recording and the plugin test.
Not run, and why: `rmq` device info tables and `nv2080 class list` return
`NV_ERR_INSUFFICIENT_PERMISSIONS` / `NV_ERR_NOT_SUPPORTED` for an
unprivileged client on this driver (reported, not escalated); interactive
`rmtop` was not driven (no TTY in the build shell); nothing here exercises
migrations, MIG or a second GPU.
## 8. Roadmap tie-ins
* **docs/06-ACCURACY** — recorded `rm_harness_run` blobs are the host
reference for M1/M6; keep the JSON captures in the session directories
cited by the metric table.
* **RISC-V-GSP-RM-DEBUGGER-ROADMAP H3** — the custom simulator's device map and RM-core
interception need real control semantics (IDs, sizes, status values) to
validate against; `rmtool` supplies that inventory before the H2.2 boot is
reached, and the shared `rm_api.c` can become the simulator's host-RM
client for differential tests.
* **RISC-V-GSP-RM-DEBUGGER-ROADMAP H4.5** — the canonical observe -> compare -> implement
loop can now include a host capture (`harness.sh --session`) beside the
simulator session, giving every emulation change an evidence pair.
* **R4 (full-memory trace)** — the session-event schema already carries
external runs; when the TCG trace lands, host `rm_harness_run` events and
firmware `access` events can be interleaved for a complete real-vs-sim
picture.
* **H1.3** — injected services that emulate RM controls should be diffed
field-by-field against the `rmtool nv2080 method call` reply for the same
control id.
## 9. Extending safely
1. Add new read-only controls to the allowlist **and** to `rmtool.h`
structs, with the exact kernel-side size (the userspace SDK snapshot is
not authoritative).
2. Never add an override that bypasses the allowlist/refusals.
3. Record any new query in the session and document the skip reason for
anything privilege-gated.
4. Keep `tools/rmq/` untouched; call it from the harness.
|
