summaryrefslogtreecommitdiffstats
path: root/tools/cuda-bridge/INTEGRATION.md
diff options
context:
space:
mode:
authorArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-03 14:05:38 -0400
committerArthur Hanson Rasmusson <arthur@vgpu.io>2026-10-03 14:05:38 -0400
commit32b551d43781dc7b8ac4587ca2ba1554b0cbd5e5 (patch)
treed1cab6bcc35e461610cd5dfcf3f51ba679a576fc /tools/cuda-bridge/INTEGRATION.md
parentfa71f1b53b524e5f6c6c1fbff1088d610b9f3236 (diff)
downloadopen-iov-32b551d43781dc7b8ac4587ca2ba1554b0cbd5e5.tar.gz
open-iov-32b551d43781dc7b8ac4587ca2ba1554b0cbd5e5.zip
vgsp: host CUDA bridge + sema release recovery; greeneye simulator wave
- kernel: gsp_cuda_bridge (QMD launch capture, debugfs job/out/done, param+output resolution cached from inline writes), sema drain fixes (never hold/starve a channel), GR ring autopin + counter autoreg, per-page push reads, counter-derive gate; module builds clean - tools/cuda-bridge: CUDA driver-API daemon executing mail/vadd kernels on the physical GPU, CUBB2/CBD2 v2 protocol, layout check + self-tests - tools/guest-diag + tools/cuda-ab: scanner/pin fixes, victory push-only mode - debug/risc-v-gsp-rm-vm: Phase S simulator wave (GSP device model, boot ladder, triage/re-demo harnesses, docs 30-41), GPGPU-sim scaffolding - STATE-CURRENT/docs updates
Diffstat (limited to 'tools/cuda-bridge/INTEGRATION.md')
-rw-r--r--tools/cuda-bridge/INTEGRATION.md115
1 files changed, 115 insertions, 0 deletions
diff --git a/tools/cuda-bridge/INTEGRATION.md b/tools/cuda-bridge/INTEGRATION.md
new file mode 100644
index 0000000..fcdce80
--- /dev/null
+++ b/tools/cuda-bridge/INTEGRATION.md
@@ -0,0 +1,115 @@
+# vgsp.ko integration (v2, live path)
+
+The bridge is a userspace daemon with a fixed-size file protocol so the vGPU
+kernel module drives it without sharing kernel/userspace memory. The v2 records
+are frozen in `PROTOCOL.md` and mirrored by static asserts in
+`kernel/gsp_cuda_bridge.c` and `tools/cuda-bridge/cuda_bridge.c`.
+
+## Expected flow
+
+1. **Detect launch.** `vgsp.ko`'s service thread decodes the guest's inline QMD
+ on the emulated GR compute channel (`gsp_sema_service.c`), classifies the
+ launch shape (mail vs vadd) and reads the parameter bank's pointers/count.
+2. **Marshal + publish.** The module resolves the guest source VAs, copies the
+ inputs into its own buffer (vadd: `a` then `b`; mail: none) and publishes a
+ 256-byte `CUBB2` job with a fresh `seq` in `cuda_bridge_job` while serving
+ the input blob from `cuda_bridge_in`. One job is published at a time
+ (`CUBB2_PEND_MAX` slots, oldest first).
+3. **Daemon executes.** `cuda_bridge --bridge --job
+ /sys/kernel/debug/vgsp/cuda_bridge_job --in
+ /sys/kernel/debug/vgsp/cuda_bridge_in --out
+ /sys/kernel/debug/vgsp/cuda_bridge_out --done
+ /sys/kernel/debug/vgsp/cuda_bridge_done` polls the job file (default 50 ms),
+ reads the inputs, executes the requested kernel on the physical GPU
+ (embedded PTX; `cuMemcpyHtoD` / `cuLaunchKernel` / `cuMemcpyDtoH`), writes
+ the output blob to `cuda_bridge_out`, then always writes a `CBD2` done
+ record to `cuda_bridge_done`.
+4. **Module completes the guest.** `vgsp.ko` pops done records on the service
+ thread, matches `seq`, copies the output blob through the VA translation
+ back into the guest destination, applies the QMD release semaphore, and the
+ guest's `cudaDeviceSynchronize` completes.
+
+Everything is inert until a launch is seen; the path can be switched off with
+the `cuda_bridge` module parameter (default on).
+
+## Daemon invocation (frozen CLI)
+
+```
+cuda_bridge --bridge \
+ --job /sys/kernel/debug/vgsp/cuda_bridge_job \
+ --in /sys/kernel/debug/vgsp/cuda_bridge_in \
+ --out /sys/kernel/debug/vgsp/cuda_bridge_out \
+ --done /sys/kernel/debug/vgsp/cuda_bridge_done
+```
+
+`--poll-ms` overrides the 50 ms default; `--once` exits after one job
+(bring-up only). The legacy v1 `--job J --done D --ram R` GPA mode is separate
+and unchanged.
+
+## Debugfs permissions
+
+`cuda_bridge_job`/`_in` are `0444` and `_out`/`_done` are `0200`, owned by
+root. When the daemon runs unprivileged, the runner opens them first:
+
+```
+for f in job in out done status; do
+ sudo chmod 666 /sys/kernel/debug/vgsp/cuda_bridge_$f 2>/dev/null
+done
+```
+
+## seq semantics
+
+* The module's counter starts at 1 and only increases; a job record is
+ immutable while pending.
+* The daemon records `last_seq` and ignores any valid record with
+ `seq <= last_seq` (duplicate/replay protection). The first valid record
+ after daemon startup always executes, so a daemon restart can re-run at most
+ the single stale job still parked in the interface.
+* A done record whose `seq` matches no pending module job is counted `stale`
+ and dropped; malformed magic/version is counted `malformed`. Both are
+ visible in `cuda_bridge_status`.
+
+## Logging
+
+The daemon logs one line per executed job (stdout only; correctness never
+depends on it):
+
+```
+CUBB2 seq=42 op=2 n=1048576 gpu="NVIDIA GeForce RTX 2080 Super with Max-Q Design" \
+ kernel_ns=41919 status=ok checksum=392344090.500000 bad=0 out_bytes=4194304
+```
+
+`status=err` lines are preceded by a `bridge: job seq=... rejected/failed: ...`
+stderr line naming the cause (bounds, bad opcode, input transport, output
+transport). Every job — success or failure — gets a done record; failures
+never leave the module waiting.
+
+## Limits and error mapping
+
+* `n <= 1<<26`; `in_bytes`, `out_bytes <= 32 MiB`; mail `n <= 4096` (module
+ param-length limit). Violations: `status = 1` (`bad job`), `err_code = 22`
+ (`EINVAL`).
+* Bad opcode: `status = 1`, `err_code = opcode`.
+* Input read failure: `status = 3`, `err_code = 5` (`EIO`); allocation
+ failure: `err_code = 12` (`ENOMEM`).
+* CUDA failure: `status = 2`, `err_code = CUresult`.
+* Output write failure: `status = 3`, `err_code = errno`.
+* `bad` is data quality, not execution failure: `status` stays 0 when the
+ kernel ran and the output was transported.
+
+## Risks / open questions
+
+* **Single writer per job interface.** One job is visible at a time and the
+ module serialises publication; the daemon's `last_seq` only protects
+ duplicates. Multiple daemons against one mdev are unsupported.
+* **Daemon lifetime / restarts.** On restart `last_seq` resets and the one
+ parked job may run again. The module can clear or bump the job when (re)starting
+ the daemon; the re-run is idempotent for mail/vadd.
+* **Timeouts.** A stuck job has no protocol-level timeout; the module bounds
+ `n`/bytes and the runner has bounded waits. A watchdog belongs on the
+ module side.
+* **Device selection.** The daemon always uses `cuDeviceGet(0)`; multi-GPU
+ hosts need a `--device` selector and the module must agree on the physical
+ GPU behind the mdev.
+* **PTX target.** The embedded PTX is `sm_75` (JIT-ed by the driver for the
+ physical GPU); a newer-only GPU needs the build-time `PTX_ARCH` bumped.