diff options
| author | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-10-03 14:05:38 -0400 |
|---|---|---|
| committer | Arthur Hanson Rasmusson <arthur@vgpu.io> | 2026-10-03 14:05:38 -0400 |
| commit | 32b551d43781dc7b8ac4587ca2ba1554b0cbd5e5 (patch) | |
| tree | d1cab6bcc35e461610cd5dfcf3f51ba679a576fc /tools/cuda-bridge/INTEGRATION.md | |
| parent | fa71f1b53b524e5f6c6c1fbff1088d610b9f3236 (diff) | |
| download | open-iov-32b551d43781dc7b8ac4587ca2ba1554b0cbd5e5.tar.gz open-iov-32b551d43781dc7b8ac4587ca2ba1554b0cbd5e5.zip | |
vgsp: host CUDA bridge + sema release recovery; greeneye simulator wave
- kernel: gsp_cuda_bridge (QMD launch capture, debugfs job/out/done, param+output resolution cached from inline writes), sema drain fixes (never hold/starve a channel), GR ring autopin + counter autoreg, per-page push reads, counter-derive gate; module builds clean
- tools/cuda-bridge: CUDA driver-API daemon executing mail/vadd kernels on the physical GPU, CUBB2/CBD2 v2 protocol, layout check + self-tests
- tools/guest-diag + tools/cuda-ab: scanner/pin fixes, victory push-only mode
- debug/risc-v-gsp-rm-vm: Phase S simulator wave (GSP device model, boot ladder, triage/re-demo harnesses, docs 30-41), GPGPU-sim scaffolding
- STATE-CURRENT/docs updates
Diffstat (limited to 'tools/cuda-bridge/INTEGRATION.md')
| -rw-r--r-- | tools/cuda-bridge/INTEGRATION.md | 115 |
1 files changed, 115 insertions, 0 deletions
diff --git a/tools/cuda-bridge/INTEGRATION.md b/tools/cuda-bridge/INTEGRATION.md new file mode 100644 index 0000000..fcdce80 --- /dev/null +++ b/tools/cuda-bridge/INTEGRATION.md @@ -0,0 +1,115 @@ +# vgsp.ko integration (v2, live path) + +The bridge is a userspace daemon with a fixed-size file protocol so the vGPU +kernel module drives it without sharing kernel/userspace memory. The v2 records +are frozen in `PROTOCOL.md` and mirrored by static asserts in +`kernel/gsp_cuda_bridge.c` and `tools/cuda-bridge/cuda_bridge.c`. + +## Expected flow + +1. **Detect launch.** `vgsp.ko`'s service thread decodes the guest's inline QMD + on the emulated GR compute channel (`gsp_sema_service.c`), classifies the + launch shape (mail vs vadd) and reads the parameter bank's pointers/count. +2. **Marshal + publish.** The module resolves the guest source VAs, copies the + inputs into its own buffer (vadd: `a` then `b`; mail: none) and publishes a + 256-byte `CUBB2` job with a fresh `seq` in `cuda_bridge_job` while serving + the input blob from `cuda_bridge_in`. One job is published at a time + (`CUBB2_PEND_MAX` slots, oldest first). +3. **Daemon executes.** `cuda_bridge --bridge --job + /sys/kernel/debug/vgsp/cuda_bridge_job --in + /sys/kernel/debug/vgsp/cuda_bridge_in --out + /sys/kernel/debug/vgsp/cuda_bridge_out --done + /sys/kernel/debug/vgsp/cuda_bridge_done` polls the job file (default 50 ms), + reads the inputs, executes the requested kernel on the physical GPU + (embedded PTX; `cuMemcpyHtoD` / `cuLaunchKernel` / `cuMemcpyDtoH`), writes + the output blob to `cuda_bridge_out`, then always writes a `CBD2` done + record to `cuda_bridge_done`. +4. **Module completes the guest.** `vgsp.ko` pops done records on the service + thread, matches `seq`, copies the output blob through the VA translation + back into the guest destination, applies the QMD release semaphore, and the + guest's `cudaDeviceSynchronize` completes. + +Everything is inert until a launch is seen; the path can be switched off with +the `cuda_bridge` module parameter (default on). + +## Daemon invocation (frozen CLI) + +``` +cuda_bridge --bridge \ + --job /sys/kernel/debug/vgsp/cuda_bridge_job \ + --in /sys/kernel/debug/vgsp/cuda_bridge_in \ + --out /sys/kernel/debug/vgsp/cuda_bridge_out \ + --done /sys/kernel/debug/vgsp/cuda_bridge_done +``` + +`--poll-ms` overrides the 50 ms default; `--once` exits after one job +(bring-up only). The legacy v1 `--job J --done D --ram R` GPA mode is separate +and unchanged. + +## Debugfs permissions + +`cuda_bridge_job`/`_in` are `0444` and `_out`/`_done` are `0200`, owned by +root. When the daemon runs unprivileged, the runner opens them first: + +``` +for f in job in out done status; do + sudo chmod 666 /sys/kernel/debug/vgsp/cuda_bridge_$f 2>/dev/null +done +``` + +## seq semantics + +* The module's counter starts at 1 and only increases; a job record is + immutable while pending. +* The daemon records `last_seq` and ignores any valid record with + `seq <= last_seq` (duplicate/replay protection). The first valid record + after daemon startup always executes, so a daemon restart can re-run at most + the single stale job still parked in the interface. +* A done record whose `seq` matches no pending module job is counted `stale` + and dropped; malformed magic/version is counted `malformed`. Both are + visible in `cuda_bridge_status`. + +## Logging + +The daemon logs one line per executed job (stdout only; correctness never +depends on it): + +``` +CUBB2 seq=42 op=2 n=1048576 gpu="NVIDIA GeForce RTX 2080 Super with Max-Q Design" \ + kernel_ns=41919 status=ok checksum=392344090.500000 bad=0 out_bytes=4194304 +``` + +`status=err` lines are preceded by a `bridge: job seq=... rejected/failed: ...` +stderr line naming the cause (bounds, bad opcode, input transport, output +transport). Every job — success or failure — gets a done record; failures +never leave the module waiting. + +## Limits and error mapping + +* `n <= 1<<26`; `in_bytes`, `out_bytes <= 32 MiB`; mail `n <= 4096` (module + param-length limit). Violations: `status = 1` (`bad job`), `err_code = 22` + (`EINVAL`). +* Bad opcode: `status = 1`, `err_code = opcode`. +* Input read failure: `status = 3`, `err_code = 5` (`EIO`); allocation + failure: `err_code = 12` (`ENOMEM`). +* CUDA failure: `status = 2`, `err_code = CUresult`. +* Output write failure: `status = 3`, `err_code = errno`. +* `bad` is data quality, not execution failure: `status` stays 0 when the + kernel ran and the output was transported. + +## Risks / open questions + +* **Single writer per job interface.** One job is visible at a time and the + module serialises publication; the daemon's `last_seq` only protects + duplicates. Multiple daemons against one mdev are unsupported. +* **Daemon lifetime / restarts.** On restart `last_seq` resets and the one + parked job may run again. The module can clear or bump the job when (re)starting + the daemon; the re-run is idempotent for mail/vadd. +* **Timeouts.** A stuck job has no protocol-level timeout; the module bounds + `n`/bytes and the runner has bounded waits. A watchdog belongs on the + module side. +* **Device selection.** The daemon always uses `cuDeviceGet(0)`; multi-GPU + hosts need a `--device` selector and the module must agree on the physical + GPU behind the mdev. +* **PTX target.** The embedded PTX is `sm_75` (JIT-ed by the driver for the + physical GPU); a newer-only GPU needs the build-time `PTX_ARCH` bumped. |
