From 5dd8165f1754e156a0f9fb16c7a6bfa90e99ed60 Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Mon, 14 Sep 2026 06:43:11 -0400 Subject: VFIO device registration fix + IOMMUFD callbacks + VFIO ioctl handlers - Removed IOMMU group addition before vfio_register_emulated_iommu_dev() in nv_vgpu_mdev_create() - Added IOMMUFD callbacks (bind_iommufd, unbind_iommufd, attach_ioas, detach_ioas) using kernel's vfio_iommufd_emulated_* functions - Implemented VFIO ioctl handlers: GET_INFO, GET_REGION_INFO, GET_IRQ_INFO, SET_IRQS, MMAP - Added config space read/write handlers at offset 0x70000000000 - Mmap handler for config space, BAR0 (emulation shadow), BAR1-5 (physical VRAM/registers) - Interrupt handler for VFIO_DEVICE_SET_IRQS (DATA_EVENTFD, DATA_NONE, DATA_BOOL) - Added udev rules for vfio group permissions - Created stack_manager.sh unified tool - Fixed mdev remove crash with deferred cleanup in vfio_device_release callback - Kernel 6.8+ IOMMUFD compatibility - QEMU launches successfully with vGPU passthrough --- docs/tools/README.md | 75 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+) create mode 100644 docs/tools/README.md (limited to 'docs/tools/README.md') diff --git a/docs/tools/README.md b/docs/tools/README.md new file mode 100644 index 0000000..1c7c0b5 --- /dev/null +++ b/docs/tools/README.md @@ -0,0 +1,75 @@ +# Open-IOV Tools Documentation + +## Overview +This directory contains documentation for userspace tools built on top of the Open-IOV vGPU-IO stack, leveraging recycled NVIDIA RM API code from `integ/gpu_drv/ext-tree/`. + +## Tools + +### Debugging & Observability (Implemented) +| Tool | Location | Description | +|------|----------|-------------| +| `guest_init_monitor.py` | `tools/` | Correlates host dmesg with guest serial output | +| `failure_analyzer.py` | `tools/` | Classifies guest init failures with recommendations | +| `benchmark_infrastructure.sh` | `tools/` | VM vs bare metal benchmark runner | +| `guest_test_agent.py` | `tools/guest_agent/` | Guest-side test agent (vsock/TCP) | +| `debug_guest_init.sh` | `scripts/` | One-command debug launch (QEMU + GDB + monitor) | +| `windows_guest_automation.sh` | `scripts/` | Windows guest automation via WinRM | + +### RM API Tooling (Planned) +| Tool | Documentation | Description | +|------|---------------|-------------| +| `rmtool` | [rmtool.md](rmtool.md) | RM API Swiss Army Knife - CLI for state enumeration, engine control, memory ops, NV2080 direct control | +| `rmtop` | [rmtop.md](rmtop.md) | GPU htop equivalent - ncurses real-time dashboard | + +## Code Reuse Sources +All planned tools recycle existing NVIDIA driver code from `integ/gpu_drv/ext-tree/`: + +| Component | Path | Use Case | +|-----------|------|----------| +| nvidia-smi | `drivers/gpgpu/nvidia-smi/` | CLI framework, GPU enumeration, NV2080 ioctls | +| gpumon/perfmon | `drivers/gpgpu/gpumon_perfmon/` | Performance sampling, perfmon counters | +| NV2080 controls | `drivers/gpgpu/nv2080/` | RM control interfaces (classes, methods, events) | +| Resource Server | `drivers/gpgpu/resserve/` | Resource Server (resserve) API | +| CUDA Internal | `drivers/gpgpu/cudasw/` | CUDASW / CUDA Internal Layer | +| Resource Manager | `drivers/gpgpu/resman/` | Resman API | + +## Build Integration +Tools will be built under `open-iov/tools/`: +``` +tools/ +├── rmtool/ +│ ├── src/ +│ ├── lib/ +│ └── Makefile +├── rmtop/ +│ ├── src/ +│ ├── lib/ +│ └── Makefile +└── Makefile # Builds all tools +``` + +## Dependencies +- `libnvidia-rmapi.so` - Recycled RM API (from ext-tree) +- `libgpumon.so` - Recycled perfmon (from ext-tree) +- `libpciaccess` - PCI configuration access +- `libnuma` - NUMA topology +- `ncursesw` - rmtop UI (wide character support) +- `libjson-c` - JSON output formatting +- `libcurl` - Remote API (future) + +## Integration with Open-IOV Stack +Tools will integrate with the host VFIO driver via: +- `/dev/nvidia*` character devices (existing nvidia.ko) +- `open-iov-vfio.ko` debugfs interface (`/sys/kernel/debug/open-iov-vfio/`) +- VFIO ioctls for guest state introspection +- GSP-RM telemetry via firmware analyzer / dynamic dispatcher + +## Future Tool Ideas +| Tool | Purpose | +|------|---------| +| `rmmigrate` | Live migration coordinator via resserve | +| `rmprofile` | Workload profiler (CUDA/Video/AI correlation) | +| `rmpolicy` | Policy engine for automated engine allocation | +| `rmtelemetry` | Telemetry exporter (Prometheus/OTel/InfluxDB) | +| `rmguest` | Guest introspection via vGPU VFIO interface | +| `rmfirmware` | GSP firmware analysis / update utility | \ No newline at end of file -- cgit v1.2.3