From cc79b478843f0230fe6aa8ecd822de7ba71c891e Mon Sep 17 00:00:00 2001 From: Arthur Hanson Rasmusson Date: Sat, 3 Oct 2026 14:05:38 -0400 Subject: vgsp: host CUDA bridge + sema release recovery; greeneye simulator wave - kernel: gsp_cuda_bridge (QMD launch capture, debugfs job/out/done, param+output resolution cached from inline writes), sema drain fixes (never hold/starve a channel), GR ring autopin + counter autoreg, per-page push reads, counter-derive gate; module builds clean - tools/cuda-bridge: CUDA driver-API daemon executing mail/vadd kernels on the physical GPU, CUBB2/CBD2 v2 protocol, layout check + self-tests - tools/guest-diag + tools/cuda-ab: scanner/pin fixes, victory push-only mode - debug/risc-v-gsp-rm-vm: Phase S simulator wave (GSP device model, boot ladder, triage/re-demo harnesses, docs 30-41), GPGPU-sim scaffolding - STATE-CURRENT/docs updates --- tools/cuda-bridge/README.md | 104 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 104 insertions(+) create mode 100644 tools/cuda-bridge/README.md (limited to 'tools/cuda-bridge/README.md') diff --git a/tools/cuda-bridge/README.md b/tools/cuda-bridge/README.md new file mode 100644 index 0000000..8ac0b6d --- /dev/null +++ b/tools/cuda-bridge/README.md @@ -0,0 +1,104 @@ +# CUDA Bridge (host-side) + +Host daemon that executes guest CUDA kernels on the real physical GPU for +Open-IOV. It is the execution backend for the endgame path: `vgsp.ko` detects a +guest kernel launch, writes a **CUBB2 v2** job record to the debugfs job file, +this daemon executes it with the CUDA driver API on the physical GPU and writes +a **CBD2 v2** done record, and the module wakes the guest. The v1 +`CUBB`/`CUBD` packed/GPA mode remains for the offline tests. + +Host-only development tooling: it never touches the guest, QEMU, mdev, +`vgsp.ko`, or the GPU reset path. It only maps the guest RAM file and talks to +the NVIDIA driver. + +## Files + +| File | Purpose | +|---|---| +| `cuda_bridge.c` | daemon + v1/v2 protocol structs + bridge/self-test/bench/mutation modes | +| `kernels.cu` | real kernels: `mail_kernel`, `vadd`, `vadd_mutated` | +| `embed_ptx.py` | embeds nvcc-generated PTX into a C header | +| `client_test.py` | external producer check of the v1 job/done transport | +| `bridge_selftest.py` | offline plain-file self-test of `--bridge` (mail, vadd, stale seq, bad magic, failure, bounds) | +| `check_module_layout.py` | mechanical CUBB2/CBD2 struct layout comparison vs `kernel/gsp_cuda_bridge.c` | +| `Makefile` | build, test targets | +| `PROTOCOL.md` | exact wire layout and semantics (v2 frozen; v1 appendix) | +| `INTEGRATION.md` | how `vgsp.ko` uses it (paths, seq semantics, logging) | + +## Build + +``` +make # cuda_bridge +make test # self-test on a fake 4 GiB RAM file +make bench # timings (CUDA events + CPU baseline) +make mutate-test +``` + +Kernel path: `nvcc` was available on the build host (CUDA 12.0, +`/usr/bin/nvcc`), so the Makefile compiles `kernels.cu` to PTX +(`-arch=sm_75`, PTX `.version 8.0`) and `embed_ptx.py` embeds it as a +NUL-terminated byte array; `cuda_bridge` loads it with `cuModuleLoadData`, and +the driver JITs it for Turing (CC 7.5). No PTX is loaded from disk at runtime +and no CUDA toolkit is needed at runtime. If `nvcc` is unavailable on a build +host, a pre-generated `kernels_ptx.h` can be copied in and the `cc` link step +still succeeds. Hand-written PTX was therefore **not** needed. + +Requirements: `libcuda.so` (driver), `cc`, `nvcc` (build only), Python 3 +(embedding + `client_test.py`). + +## Run + +``` +# v2 daemon: poll the debugfs job file, execute, write out + done +./cuda_bridge --bridge \ + --job /sys/kernel/debug/vgsp/cuda_bridge_job \ + --in /sys/kernel/debug/vgsp/cuda_bridge_in \ + --out /sys/kernel/debug/vgsp/cuda_bridge_out \ + --done /sys/kernel/debug/vgsp/cuda_bridge_done + +# v2 offline self-test with plain files (mail, vadd, stale seq, bad magic, +# failure path, bounds) +python3 bridge_selftest.py + +# v1 legacy daemon: poll the job file, execute, write the done file +./cuda_bridge --job /path/job.bin --done /path/done.bin --ram /dev/shm/vgpu-guest-ram + +# v1 one-shot (tests / module bring-up) +./cuda_bridge --job job.bin --done done.bin --ram ram.bin --once + +# fake-RAM self test: mail + vadd low GPA + vadd high GPA, byte-for-byte checks +./cuda_bridge --test [--dir /tmp/opencode/cuda-bridge-test] + +# falsifiability: opcode 2 vs opcode 3 (+1 mutation) must give different results +./cuda_bridge --mutation-test + +# timings and GPU utilization stress +./cuda_bridge --bench [--big-iters N] [--verbose] + +# external transport check (starts/stops the daemon itself) +python3 client_test.py +``` + +Every executed job prints one greppable line: + +``` +CUBB job seq=1000 op=2 n=1048576 gpu="NVIDIA GeForce RTX 2080 Super with Max-Q Design" kernel_ns=41567 status=ok checksum=392344090.500000 bad=0 +CUBB2 seq=42 op=2 n=1048576 gpu="NVIDIA GeForce RTX 2080 Super with Max-Q Design" kernel_ns=41919 status=ok checksum=392344090.500000 bad=0 out_bytes=4194304 +``` + +`--test`, `--mutation-test` and `--bench` use sparse fake RAM files under +`--dir` (default `/tmp/opencode/cuda-bridge-test`); they never touch +`/dev/shm/vgpu-guest-ram`. + +## Evidence (2026-10-03) + +Under `open-iov/backups/evidence-20261003-bridge/`: + +| File | Content | +|---|---| +| `01-self-test.txt` | fake-RAM test: mail exact, vadd low + high GPA windows, bad=0, checksum exact | +| `02-gpu-util.txt` | nvidia-smi @100 ms while 200 x n=2^24 vadd jobs run (0 % idle -> 27 % max, 298/308 samples > 0) | +| `03-timings.txt` | CUDA-event kernel times + CPU baseline (n=2^20: 39.6 us vs 1.84 ms; n=2^24: 620 us vs 28.6 ms) | +| `04-device.txt` | nvidia-smi identity and driver, plus the bridge's own DEVICE line | +| `05-mutation.txt` | opcode 3 proves the real kernel ran: checksum +1048576, bad=1048576, sample c[0]=1.0 | +| `06-daemon-transport.txt` | external producer, two seqs, done records parsed and verified | -- cgit v1.2.3