1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
|
# rmtool - RM API Swiss Army Knife
## Overview
`rmtool` is a comprehensive userspace utility for interacting with the NVIDIA Resource Manager (RM) API. It provides direct access to RM internal state, engine control, memory management, and performance telemetry. Built by recycling existing code from the NVIDIA driver stack (`integ/gpu_drv/ext-tree/`).
## Source Code Reuse
| Component | Source Location | Purpose |
|-----------|-----------------|---------|
| nvidia-smi core | `ext-tree/drivers/gpgpu/nvidia-smi/` | CLI framework, NV2080 ioctls, GPU enumeration |
| gpumon/perfmon | `ext-tree/drivers/gpgpu/gpumon_perfmon/` | Performance monitoring, sampling infrastructure |
| NV2080 controls | `ext-tree/drivers/gpgpu/nv2080/` | RM control interfaces (classes, methods, events) |
| Resource Server | `ext-tree/drivers/gpgpu/resserve/` | Resource Server (resserve) API |
| CUDA Internal | `ext-tree/drivers/gpgpu/cudasw/` | CUDASW / CUDA Internal Layer |
| Resource Manager | `ext-tree/drivers/gpgpu/resman/` | Resman API |
## Core Commands
### GPU Enumeration & Topology
```bash
rmtool gpu list # List all GPUs with PCI info
rmtool gpu topology # Show GPU topology (NVLink, PCIe, PCI IDs)
rmtool gpu info -d 0000:65:00.0 # Detailed GPU info (like nvidia-smi -q)
```
### Engine Management
```bash
rmtool engine list # List all engines (GR, CE, NVDEC, NVENC, NVJPG, OFA)
rmtool engine info -d 0 -e gr # Engine details (registers, capabilities)
rmtool engine allocate -d 0 -e gr -c 4 # Allocate 4 GR channels
rmtool engine start -d 0 -e gr -c 0 # Start engine channel
rmtool engine stop -d 0 -e gr -c 0 # Stop engine channel
rmtool engine config -d 0 -e nvenc --preset=h264 --bitrate=10M
```
### Channel & Context Control
```bash
rmtool channel list -d 0 # List all channels on GPU
rmtool channel create -d 0 -e gr -t compute # Create compute channel
rmtool channel submit -d 0 -c 0 --pushbuf=work.bin # Submit work
rmtool channel status -d 0 -c 0 # Channel status (busy, pending, error)
rmtool context list -d 0 # List all contexts
rmtool context info -d 0 -c 0 # Context details (page tables, memory)
```
### Memory Operations
```bash
rmtool mem alloc -d 0 --size=1G --type=video # Allocate VRAM
rmtool mem alloc -d 0 --size=512M --type=system # Allocate sysmem
rmtool mem map -d 0 -h 0x1234 --gpu-va=0x100000 # Map to GPU VA
rmtool mem unmap -d 0 -h 0x1234 # Unmap
rmtool mem pin -d 0 -h 0x1234 # Pin for DMA
rmtool mem info -d 0 -h 0x1234 # Allocation details
rmtool mem stats -d 0 # VRAM/sysmem usage breakdown
```
### Performance Sampling
```bash
rmtool sample engine -d 0 -e gr --interval=100ms --count=100
rmtool sample clocks -d 0 --interval=1s
rmtool sample power -d 0 --interval=1s
rmtool sample thermal -d 0 --interval=1s
rmtool sample pcie -d 0 --interval=1s
```
### NV2080 Direct Control
```bash
rmtool nv2080 class list -d 0 # List all NV2080 classes
rmtool nv2080 method call -d 0 -c 0x906D -m 0x100 --params=...
rmtool nv2080 event subscribe -d 0 -c 0x906D -e 0x100
rmtool nv2080 event wait -d 0 -c 0x906D -e 0x100 --timeout=5000
```
### Resource Server (resserve) Integration
```bash
rmtool resserve status # Resource Server status
rmtool resserve reservation list # List all reservations
rmtool resserve reservation create --gpu=0 --engine=gr --size=4
rmtool resserve reservation release --id=123
```
### CUDA Internal Layer (CUDASW)
```bash
rmtool cudasw context list # CUDA contexts
rmtool cudasw module list # Loaded CUDA modules
rmtool cudasw stream list -c 0 # Streams in context
rmtool cudasw event list -c 0 # Events in context
```
### Resource Manager (Resman)
```bash
rmtool resman client list # Resman clients
rmtool resman resource list -c 0 # Resources owned by client
rmtool resman heap info -d 0 # Heap state
```
### GSP-RM Telemetry
```bash
rmtool gsp rpc-stats -d 0 # RPC queue depths, latency
rmtool gsp doorbell -d 0 # Doorbell rates
rmtool gsp firmware -d 0 # Firmware version, threads
rmtool gsp wpr -d 0 # WPR state
```
## Output Formats
```bash
rmtool gpu list --format=json
rmtool engine list --format=csv
rmtool sample engine --format=prometheus
rmtool mem stats --format=influx
```
## Build
```bash
cd open-iov/tools/rmtool
make
# Links against: libnvidia-rmapi.so (from ext-tree), libpciaccess, libnuma
```
## Architecture
```
rmtool/
├── src/
│ ├── main.c # CLI parsing, command dispatch
│ ├── gpu.c # GPU enumeration, topology
│ ├── engine.c # Engine allocation/control
│ ├── channel.c # Channel/context management
│ ├── memory.c # Memory alloc/map/pin
│ ├── sample.c # Performance sampling
│ ├── nv2080.c # NV2080 direct control
│ ├── resserve.c # Resource Server client
│ ├── cudasw.c # CUDA Internal Layer
│ ├── resman.c # Resource Manager
│ ├── gsp.c # GSP-RM telemetry
│ └── output.c # JSON/CSV/Prometheus formatters
├── include/
│ └── rmtool.h # Internal headers
├── lib/
│ ├── librmapi.a # Recycled RM API code
│ └── libgpumon.a # Recycled perfmon code
└── Makefile
```
## Future Extensions
- **Remote gRPC/REST API** for fleet management
- **Policy Engine** for automated engine allocation
- **Migration Coordination** via resserve
- **Telemetry Export** (Prometheus, OpenTelemetry, InfluxDB)
- **Guest Introspection** via vGPU VFIO interface
- **Workload Profiling** with CUDA/GR engine correlation
|
