# 17 — RM host oracle harness (`rmharness`) > Status: implemented, documented and tested (audit: docs/27-TOOL-AUDIT.md; value demos: docs/25-VALUE-VALIDATION.md). > The RM tools are read-only host diagnostics. They are development tooling: > never shipped, never present in a guest, never a driver dependency. ## 1. Purpose The GSP emulation can only be trusted against ground truth. Two of the metrics in [06-ACCURACY](06-ACCURACY.md) need the real host RM as a reference: * **M1 — inventory coverage**: which controls/engines the real RM exposes; * **M6 — memory side-effects / fixed points**: real VRAM/heap/BAR1 numbers and engine inventory at a moment in time. `rmharness` packages the host-side oracle tools and wires their output into risc-v-gsp-rm-debugger sessions, so a simulator/emulation session and a host RM capture can be recorded, diffed and cited together. It is the host half of the accuracy loop (H4.5), not a boot path or a replacement for PLAN B traces. ## 2. Tool inventory | tool | source | role in the loop | |---|---|---| | `rmtool` | `tools/rmtool/` (new) | CLI oracle: `gpu list/topology/info`, `engine list/info`, `mem stats`, `nv2080 class list`, allowlisted `nv2080 method call`, `sample timer`; `--format=table\|json\|csv` | | `rmtop` | `tools/rmtop/` (new) | ncurses dashboard over the same query layer; `--once` renders table/json/csv for capture | | `rmq` | `tools/rmq/` (pre-existing, unmodified) | FIFO/INTERNAL device info tables for TU104 static data | Both new tools share one ioctl layer (`tools/rmtool/src/rm_api.c`, `query.c`): version check, `NV01_ROOT_CLIENT` -> `NV01_DEVICE_0` -> `NV20_SUBDEVICE_0` alloc ladder, and `NV_ESC_RM_CONTROL` queries. The device-open protocol follows what `nvidia-smi` does: open `/dev/nvidia`, `NV_ESC_WAIT_OPEN_COMPLETE`, then allocate on `/dev/nvidiactl`. The ABI structs match the running driver (the in-tree userspace SDK snapshot is older and its 40-byte `NVOS64` is rejected). ## 3. Build ```bash make -C tools/rmtool # builds rmtool + lib/librmapi.a make -C tools/rmtop # builds rmtop (ncursesw) make -C tools/rmq # unchanged, still builds ``` Requirements: gcc, ncursesw for rmtop. No external SDK, NVML or libnvidia-ml linkage. `make clean` removes all artifacts. ## 4. Running ```bash tools/rmtool/rmtool --help tools/rmtool/rmtool gpu list --format=json tools/rmtool/rmtool gpu info -d 0 tools/rmtool/rmtool engine list tools/rmtool/rmtool mem stats tools/rmtool/rmtool sample timer --interval=100ms --count=20 tools/rmtop/rmtop --once tools/rmtop/rmtop -g 0 --interval=250 # interactive (needs a TTY) ``` `gpu list`/`gpu topology` need no device access (they read `/proc/driver/nvidia/gpus` and PCI sysfs). Everything else opens a short-lived RM client and issues read-only controls; this is the same access model as `nvidia-smi` and is safe alongside a running CUDA job. `--device-spec`/`-d` accepts a minor (`0`) or a bus id (`0000:01:00.0`) and maps to the RM `deviceId`; `--device PATH` overrides the `/dev/nvidiaN` node. ## 5. risc-v-gsp-rm-debugger integration Wrappers `rmharness/rmtool.sh`, `rmharness/rmtop.sh`, `rmharness/rmq.sh` locate and build the tools and pass arguments through. `rmharness/harness.sh` adds capture: ```bash cd debug/risc-v-gsp-rm-vm ./rmharness/harness.sh --session dumps/real-rm-before --name before \ -- rmtool engine list --format=json ./rmharness/harness.sh --session dumps/real-rm-before -- rmtop --once ``` The harness appends one `rm_harness_run` event per run and stores stdout/stderr under `/blobs/` with sizes and SHA-256. Existing sessions are appended to, never rewritten; a missing `session.json` is created with the `greeneye-session/1` format. Because these runs are external, the event `t` field is `0.0`; `start`/`end` (epoch seconds) carry the wall clock. `./risc-v-gsp-rm-debugger.sh analyze` tolerates the session; the run count shows up in the event summary. `plugins/rmharness_plugin.py` exposes the same wrappers in the REPL (`rmtool`, `rmtop`, `rmq`), recording through `ge.recorder.blob()` + `ge.recorder.event()`. Write-looking arguments are refused and recorded as `rm_harness_skip`. The plugin is stdlib-only and is exercised without QEMU by `rmharness/tests/test_rmharness_plugin.py` (fake `ge`). ## 6. Safety contract (absolute) * Host-side read-only queries only. The tools allocate only an RM client / device / subdevice handle pair; they never create engine, channel or memory objects. * Never load/unload kernel modules, bind/unbind the driver, create/destroy mdevs, claim the physical GPU, reboot, or touch `/sys/fs/pstore`. * Write paths are enforced in code, not by convention: * `rmtool` refuses `mem alloc/map/unmap/pin`, `engine allocate/start/stop/ config`, and `channel/context/resserve/cudasw/resman/gsp` with exit 4/3; * `nv2080 method call` only sends controls on a read-only allowlist (`GPU_GET_INFO_V2`, `GPU_GET_NAME_STRING`, `GPU_GET_ENGINES_V2`, `MC_GET_ARCH_INFO`, `FB_GET_INFO_V2`, `BUS_GET_PCI_INFO`, `TIMER_GET_TIME`, `FIFO_GET_DEVICE_INFO_TABLE`, `GPU_GET_CLASSLIST_V2`, ...) and refuses anything else before the ioctl; * `harness.sh` only accepts `rmtool`, `rmtop`, `rmq`; * the plugin blocks write-looking arguments. * Another agent may be running a live CUDA/PLAN B job; the tools do not touch its context and all smoke checks above run without a GPU claim. ## 7. What was and was not run Smoke checks executed on the development host (driver 570.211.01, TU104 RTX 2080 Super Max-Q, one GPU at `0000:01:00.0`): * `--help`/`--version` for both tools; `gpu list`, `gpu topology` (proc/sysfs only); * `gpu info`, `engine list`, `mem stats`, `sample timer`, `rmtop --once` (read-only RM controls) — all returned live data (`TU100`, 10 engines GR + 5 CE + NVDEC + NVENC + SW + SEC2, 8 GiB VRAM, BAR1 256 MiB); * `harness.sh` session recording and the plugin test. Not run, and why: `rmq` device info tables and `nv2080 class list` return `NV_ERR_INSUFFICIENT_PERMISSIONS` / `NV_ERR_NOT_SUPPORTED` for an unprivileged client on this driver (reported, not escalated); interactive `rmtop` was not driven (no TTY in the build shell); nothing here exercises migrations, MIG or a second GPU. ## 8. Roadmap tie-ins * **docs/06-ACCURACY** — recorded `rm_harness_run` blobs are the host reference for M1/M6; keep the JSON captures in the session directories cited by the metric table. * **RISC-V-GSP-RM-DEBUGGER-ROADMAP H3** — the custom simulator's device map and RM-core interception need real control semantics (IDs, sizes, status values) to validate against; `rmtool` supplies that inventory before the H2.2 boot is reached, and the shared `rm_api.c` can become the simulator's host-RM client for differential tests. * **RISC-V-GSP-RM-DEBUGGER-ROADMAP H4.5** — the canonical observe -> compare -> implement loop can now include a host capture (`harness.sh --session`) beside the simulator session, giving every emulation change an evidence pair. * **R4 (full-memory trace)** — the session-event schema already carries external runs; when the TCG trace lands, host `rm_harness_run` events and firmware `access` events can be interleaved for a complete real-vs-sim picture. * **H1.3** — injected services that emulate RM controls should be diffed field-by-field against the `rmtool nv2080 method call` reply for the same control id. ## 9. Extending safely 1. Add new read-only controls to the allowlist **and** to `rmtool.h` structs, with the exact kernel-side size (the userspace SDK snapshot is not authoritative). 2. Never add an override that bypasses the allowlist/refusals. 3. Record any new query in the session and document the skip reason for anything privilege-gated. 4. Keep `tools/rmq/` untouched; call it from the harness.