1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
|
# Open-IOV Tools Documentation
## Overview
This directory contains documentation for userspace tools built on top of the
Open-IOV vGPU-IO stack. The RM-API tooling reuses behaviour and interfaces
from the driver components we studied by reverse engineering and decompilation
(tracked docs describe components and behaviour).
## Tools
### Debugging & Observability (Implemented)
| Tool | Location | Description |
|------|----------|-------------|
| `guest_init_monitor.py` | `tools/` | Correlates host dmesg with guest serial output |
| `failure_analyzer.py` | `tools/` | Classifies guest init failures with recommendations |
| `benchmark_infrastructure.sh` | `tools/` | VM vs bare metal benchmark runner |
| `guest_test_agent.py` | `tools/guest_agent/` | Guest-side test agent (vsock/TCP) |
| `debug_guest_init.sh` | `scripts/` | One-command debug launch (QEMU + GDB + monitor) |
| `windows_guest_automation.sh` | `scripts/` | Windows guest automation via WinRM |
| `rmq` | `tools/rmq/` | Host RM-API oracle: queries the host RM via `/dev/nvidia0` for ground-truth GPU/engine data |
| `cudatool.py`, `callwalk.py`, `ghidra_decompile_at.sh`, `annotate_corpus.py` | `userspace/cuda/tools/` | CUDA userspace analysis: crash decode, callgraph walk, decompile-at, corpus annotation |
| `guest_engines.c`, `ioctl_trace.c` | `tools/guest-diag/` | Guest-side engine-list dump and NVIDIA-ioctl tracer |
| `ab.sh`, `vector_add.cu` | `tools/cuda-ab/` | Host-vs-guest CUDA A/B harness with diffable results |
### RM API Tooling (Planned)
| Tool | Documentation | Description |
|------|---------------|-------------|
| `rmtool` | [rmtool.md](rmtool.md) | RM API Swiss Army Knife - CLI for state enumeration, engine control, memory ops, NV2080 direct control |
| `rmtop` | [rmtop.md](rmtop.md) | GPU htop equivalent - ncurses real-time dashboard |
## Code Reuse Sources (component level)
All planned tools reuse the driver components we studied. Tracked
documentation refers to them by component only:
| Component | Use Case |
|-----------|----------|
| NVML / nvidia-smi component | CLI framework, GPU enumeration, NV2080 ioctls |
| gpumon / perfmon component | Performance sampling, perfmon counters |
| NV2080 control component | RM control interfaces (classes, methods, events) |
| Resource server component | Resource Server (resserve) API |
| CUDA software (CUDASW) component | CUDA internal layer behaviours |
| Resource manager component | Resman API |
## Build Integration
Tools will be built under `open-iov/tools/`:
```
tools/
├── rmtool/
│ ├── src/
│ ├── lib/
│ └── Makefile
├── rmtop/
│ ├── src/
│ ├── lib/
│ └── Makefile
└── Makefile # Builds all tools
```
## Dependencies
- `libnvidia-rmapi.so` - reused RM API layer (component level)
- `libgpumon.so` - reused perfmon layer (component level)
- `libpciaccess` - PCI configuration access
- `libnuma` - NUMA topology
- `ncursesw` - rmtop UI (wide character support)
- `libjson-c` - JSON output formatting
- `libcurl` - Remote API (future)
## Integration with Open-IOV Stack
Tools will integrate with the host VFIO driver via:
- `/dev/nvidia*` character devices (existing nvidia.ko)
- `vgsp.ko` observability: **there is currently no debugfs interface** —
stats and traces come from `dmesg`, and the module's write-only parameters
(`scan_guest_ram`, `scan_vidmem`, `scan_seq`, `sema_probe`, `scan_pt`,
`scan_masked`, `fill_vidmem`, `poke_guest`, `poke_fill`, `dump_vidmem`,
`dump_guest`) drive guest-memory inspection
- VFIO ioctls for guest state introspection
- GSP-RM telemetry via firmware analyzer / dynamic dispatcher
## Future Tool Ideas
| Tool | Purpose |
|------|---------|
| `rmmigrate` | Live migration coordinator via resserve |
| `rmprofile` | Workload profiler (CUDA/Video/AI correlation) |
| `rmpolicy` | Policy engine for automated engine allocation |
| `rmtelemetry` | Telemetry exporter (Prometheus/OTel/InfluxDB) |
| `rmguest` | Guest introspection via vGPU VFIO interface |
| `rmfirmware` | GSP firmware analysis / update utility |
|
