# Open-IOV Tools Documentation

## Overview
This directory contains documentation for userspace tools built on top of the Open-IOV vGPU-IO stack, leveraging recycled NVIDIA RM API code from `integ/gpu_drv/ext-tree/`.

## Tools

### Debugging & Observability (Implemented)
| Tool | Location | Description |
|------|----------|-------------|
| `guest_init_monitor.py` | `tools/` | Correlates host dmesg with guest serial output |
| `failure_analyzer.py` | `tools/` | Classifies guest init failures with recommendations |
| `benchmark_infrastructure.sh` | `tools/` | VM vs bare metal benchmark runner |
| `guest_test_agent.py` | `tools/guest_agent/` | Guest-side test agent (vsock/TCP) |
| `debug_guest_init.sh` | `scripts/` | One-command debug launch (QEMU + GDB + monitor) |
| `windows_guest_automation.sh` | `scripts/` | Windows guest automation via WinRM |

### RM API Tooling (Planned)
| Tool | Documentation | Description |
|------|---------------|-------------|
| `rmtool` | [rmtool.md](rmtool.md) | RM API Swiss Army Knife - CLI for state enumeration, engine control, memory ops, NV2080 direct control |
| `rmtop` | [rmtop.md](rmtop.md) | GPU htop equivalent - ncurses real-time dashboard |

## Code Reuse Sources
All planned tools recycle existing NVIDIA driver code from `integ/gpu_drv/ext-tree/`:

| Component | Path | Use Case |
|-----------|------|----------|
| nvidia-smi | `drivers/gpgpu/nvidia-smi/` | CLI framework, GPU enumeration, NV2080 ioctls |
| gpumon/perfmon | `drivers/gpgpu/gpumon_perfmon/` | Performance sampling, perfmon counters |
| NV2080 controls | `drivers/gpgpu/nv2080/` | RM control interfaces (classes, methods, events) |
| Resource Server | `drivers/gpgpu/resserve/` | Resource Server (resserve) API |
| CUDA Internal | `drivers/gpgpu/cudasw/` | CUDASW / CUDA Internal Layer |
| Resource Manager | `drivers/gpgpu/resman/` | Resman API |

## Build Integration
Tools will be built under `open-iov/tools/`:
```
tools/
├── rmtool/
│   ├── src/
│   ├── lib/
│   └── Makefile
├── rmtop/
│   ├── src/
│   ├── lib/
│   └── Makefile
└── Makefile          # Builds all tools
```

## Dependencies
- `libnvidia-rmapi.so` - Recycled RM API (from ext-tree)
- `libgpumon.so` - Recycled perfmon (from ext-tree)
- `libpciaccess` - PCI configuration access
- `libnuma` - NUMA topology
- `ncursesw` - rmtop UI (wide character support)
- `libjson-c` - JSON output formatting
- `libcurl` - Remote API (future)

## Integration with Open-IOV Stack
Tools will integrate with the host VFIO driver via:
- `/dev/nvidia*` character devices (existing nvidia.ko)
- `open-iov-vfio.ko` debugfs interface (`/sys/kernel/debug/open-iov-vfio/`)
- VFIO ioctls for guest state introspection
- GSP-RM telemetry via firmware analyzer / dynamic dispatcher

## Future Tool Ideas
| Tool | Purpose |
|------|---------|
| `rmmigrate` | Live migration coordinator via resserve |
| `rmprofile` | Workload profiler (CUDA/Video/AI correlation) |
| `rmpolicy` | Policy engine for automated engine allocation |
| `rmtelemetry` | Telemetry exporter (Prometheus/OTel/InfluxDB) |
| `rmguest` | Guest introspection via vGPU VFIO interface |
| `rmfirmware` | GSP firmware analysis / update utility |