1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
|
# rmtool - RM API Swiss Army Knife
> **Status: planned / not implemented.** There is no `tools/rmtool/` directory
> or build target today. The implemented RM-API tooling is the `rmq` subset
> (`tools/rmq/`, the host RM-API oracle); treat this page as a design sketch,
> not a user guide. See [tools/README.md](README.md).
## Overview
`rmtool` is a comprehensive userspace utility for interacting with the NVIDIA Resource Manager (RM) API. It provides direct access to RM internal state, engine control, memory management, and performance telemetry. Built on top of driver components studied through reverse engineering and
decompilation (referenced by component).
## Source Code Reuse
| Component | Source Location | Purpose |
|-----------|-----------------|---------|
| nvidia-smi component | CLI framework, NV2080 ioctls, GPU enumeration |
| gpumon / perfmon component | Performance monitoring, sampling infrastructure |
| NV2080 control component | RM control interfaces (classes, methods, events) |
| Resource server component | Resource Server (resserve) API |
| CUDA software (CUDASW) component | CUDA internal layer |
| Resource manager component | Resman API |
## Core Commands
### GPU Enumeration & Topology
```bash
rmtool gpu list # List all GPUs with PCI info
rmtool gpu topology # Show GPU topology (NVLink, PCIe, PCI IDs)
rmtool gpu info -d 0000:65:00.0 # Detailed GPU info (like nvidia-smi -q)
```
### Engine Management
```bash
rmtool engine list # List all engines (GR, CE, NVDEC, NVENC, NVJPG, OFA)
rmtool engine info -d 0 -e gr # Engine details (registers, capabilities)
rmtool engine allocate -d 0 -e gr -c 4 # Allocate 4 GR channels
rmtool engine start -d 0 -e gr -c 0 # Start engine channel
rmtool engine stop -d 0 -e gr -c 0 # Stop engine channel
rmtool engine config -d 0 -e nvenc --preset=h264 --bitrate=10M
```
### Channel & Context Control
```bash
rmtool channel list -d 0 # List all channels on GPU
rmtool channel create -d 0 -e gr -t compute # Create compute channel
rmtool channel submit -d 0 -c 0 --pushbuf=work.bin # Submit work
rmtool channel status -d 0 -c 0 # Channel status (busy, pending, error)
rmtool context list -d 0 # List all contexts
rmtool context info -d 0 -c 0 # Context details (page tables, memory)
```
### Memory Operations
```bash
rmtool mem alloc -d 0 --size=1G --type=video # Allocate VRAM
rmtool mem alloc -d 0 --size=512M --type=system # Allocate sysmem
rmtool mem map -d 0 -h 0x1234 --gpu-va=0x100000 # Map to GPU VA
rmtool mem unmap -d 0 -h 0x1234 # Unmap
rmtool mem pin -d 0 -h 0x1234 # Pin for DMA
rmtool mem info -d 0 -h 0x1234 # Allocation details
rmtool mem stats -d 0 # VRAM/sysmem usage breakdown
```
### Performance Sampling
```bash
rmtool sample engine -d 0 -e gr --interval=100ms --count=100
rmtool sample clocks -d 0 --interval=1s
rmtool sample power -d 0 --interval=1s
rmtool sample thermal -d 0 --interval=1s
rmtool sample pcie -d 0 --interval=1s
```
### NV2080 Direct Control
```bash
rmtool nv2080 class list -d 0 # List all NV2080 classes
rmtool nv2080 method call -d 0 -c 0x906D -m 0x100 --params=...
rmtool nv2080 event subscribe -d 0 -c 0x906D -e 0x100
rmtool nv2080 event wait -d 0 -c 0x906D -e 0x100 --timeout=5000
```
### Resource Server (resserve) Integration
```bash
rmtool resserve status # Resource Server status
rmtool resserve reservation list # List all reservations
rmtool resserve reservation create --gpu=0 --engine=gr --size=4
rmtool resserve reservation release --id=123
```
### CUDA Internal Layer (CUDASW)
```bash
rmtool cudasw context list # CUDA contexts
rmtool cudasw module list # Loaded CUDA modules
rmtool cudasw stream list -c 0 # Streams in context
rmtool cudasw event list -c 0 # Events in context
```
### Resource Manager (Resman)
```bash
rmtool resman client list # Resman clients
rmtool resman resource list -c 0 # Resources owned by client
rmtool resman heap info -d 0 # Heap state
```
### GSP-RM Telemetry
```bash
rmtool gsp rpc-stats -d 0 # RPC queue depths, latency
rmtool gsp doorbell -d 0 # Doorbell rates
rmtool gsp firmware -d 0 # Firmware version, threads
rmtool gsp wpr -d 0 # WPR state
```
## Output Formats
```bash
rmtool gpu list --format=json
rmtool engine list --format=csv
rmtool sample engine --format=prometheus
rmtool mem stats --format=influx
```
## Build
```bash
cd open-iov/tools/rmtool
make
# Links against: libnvidia-rmapi.so (reused RM API component), libpciaccess, libnuma
```
## Architecture
```
rmtool/
├── src/
│ ├── main.c # CLI parsing, command dispatch
│ ├── gpu.c # GPU enumeration, topology
│ ├── engine.c # Engine allocation/control
│ ├── channel.c # Channel/context management
│ ├── memory.c # Memory alloc/map/pin
│ ├── sample.c # Performance sampling
│ ├── nv2080.c # NV2080 direct control
│ ├── resserve.c # Resource Server client
│ ├── cudasw.c # CUDA Internal Layer
│ ├── resman.c # Resource Manager
│ ├── gsp.c # GSP-RM telemetry
│ └── output.c # JSON/CSV/Prometheus formatters
├── include/
│ └── rmtool.h # Internal headers
├── lib/
│ ├── librmapi.a # Recycled RM API code
│ └── libgpumon.a # Recycled perfmon code
└── Makefile
```
## Future Extensions
- **Remote gRPC/REST API** for fleet management
- **Policy Engine** for automated engine allocation
- **Migration Coordination** via resserve
- **Telemetry Export** (Prometheus, OpenTelemetry, InfluxDB)
- **Guest Introspection** via vGPU VFIO interface
- **Workload Profiling** with CUDA/GR engine correlation
|
