summaryrefslogtreecommitdiffstats
path: root/tools/guest-diag
diff options
context:
space:
mode:
Diffstat (limited to 'tools/guest-diag')
-rw-r--r--tools/guest-diag/README.md103
-rwxr-xr-xtools/guest-diag/dbg_seq.sh5
-rwxr-xr-xtools/guest-diag/dbg_seq2.sh5
-rw-r--r--tools/guest-diag/guest_engines.c192
-rwxr-xr-xtools/guest-diag/kret.sh33
-rwxr-xr-xtools/guest-diag/launch_qemu_rootport.sh2
-rw-r--r--tools/guest-diag/ptrace_ioctl.c236
-rwxr-xr-xtools/guest-diag/push_guest_tools.sh31
-rw-r--r--tools/guest-diag/rmq_probe.c132
-rw-r--r--tools/guest-diag/sysc_shim.c73
-rwxr-xr-xtools/guest-diag/vmcycle.sh46
11 files changed, 858 insertions, 0 deletions
diff --git a/tools/guest-diag/README.md b/tools/guest-diag/README.md
new file mode 100644
index 0000000..618663b
--- /dev/null
+++ b/tools/guest-diag/README.md
@@ -0,0 +1,103 @@
+# guest-diag - tracing tools for the emulated vGPU guest
+
+Tools used while driving the guest's stock NVIDIA driver through CUDA bring-up.
+All are tiny and dependency-free (libc only).
+
+## ptrace_ioctl.c - ioctl/result tracer (the important one)
+
+libcuda issues **raw `syscall` instructions**, so LD_PRELOAD shims and ltrace
+never see its ioctls. This ptrace tracer follows a child with
+PTRACE_SYSCALL + PTRACE_GET_SYSCALL_INFO and dumps:
+
+- `req=0x25` (UVM_REGISTER_GPU) buffer on entry and exit
+ (`rmStatus` is the high dword of the 5th qword) - this is how the current
+ failure was found: `rmStatus = 0x56 NV_ERR_NOT_SUPPORTED`
+- `req=0xc020462a` (NV_ESC_RM_CONTROL) cmd + status
+- `req=0xc020462b` / `0x...4629` (RM alloc/free) status when nonzero
+- UVM ioctls 0x1/0x2/0x17/0x19/0x25/0x27
+
+Build/run **in the guest**:
+
+ gcc -O1 -w -o /tmp/ptio ptrace_ioctl.c
+ /tmp/ptio /tmp/cdt 2>&1 | grep -E 'ptio|rmctl|rmio|\^'
+
+Note: `op` in `struct ptrace_syscall_info` is ENTRY=1 / EXIT=2, and the struct
+has `pad[3], arch, instruction_pointer, stack_pointer` before the union.
+
+## sysc_shim.c - known dead end
+
+An `syscall()` LD_PRELOAD interceptor: libcuda bypasses it entirely (inline
+syscall instruction). Kept only as documentation of that fact.
+
+## vmcycle.sh - reload cycle
+
+pkill QEMU -> remove mdevs -> rmmod/insmod open-iov-vfio.ko -> create a fresh
+`nvidia-nvidia-vgpu-turing` mdev -> patch its UUID into
+launch_qemu_rootport.sh -> relaunch. Guest boots in ~60-75 s.
+
+## launch_qemu_rootport.sh - QEMU command line with pcie-root-port
+
+Snapshot of the working launch command (derived from 's argv of a stopped
+VM) plus the root-port fix. **Requires** the mdev UUID to be substituted and:
+
+ mkdir -p /tmp/sockets/<domain-uuid> # else QMP bind fails
+ rm -f /tmp/qemu-guest-agent.sock /tmp/qemu-qmp.sock
+
+Why the root port matters: on the Q35 root bus (`pcie.0`) QEMU's
+`vfio_setup_pcie_cap()` (hw/vfio/pci.c) relabels the mdev as "Root Complex
+Integrated Endpoint" and clears LNK/LNK2 + DeviceCaps, so the guest reads
+`LnkCap = 0` and `getPCIELinkRateMBps()` fails with "Unknown PCIe speed".
+Behind a `pcie-root-port` QEMU leaves the capability alone.
+
+## rmq_probe.c - legacy-ioctl RM control probe (guest or host)
+
+Minimal RM client (client alloc + device alloc + NV_ESC_RM_CONTROL) to query
+device/subdevice controls from userspace, e.g.
+`0x800289 NV0080_CTRL_CMD_GPU_GET_VIRTUALIZATION_MODE`,
+`0x800280 GET_NUM_SUBDEVICES`, `0x20800123 GPU_GET_ENGINES`.
+NOTE: the device alloc currently fails (0x40 in guest, 0x1b on host) - the
+device allocation sequence needs the proper NV0080_ALLOC_PARAMETERS handling
+(or the NVOS64 path that tools/rmq/rmq.c uses); fix before relying on it.
+
+## kret.sh - kretprobe attempt (blocked)
+
+Tries to kretprobe the nvGpuOps*/nvUvmInterface* chain to read return values.
+Blocked on this kernel: "trace_kprobe: Could not probe notrace function
+<name> [nvidia]" - kprobes are not available for nvidia module text.
+Kept for documentation.
+
+## push_guest_tools.sh - rebuild the guest-side binaries from these sources
+
+The guest's /tmp is wiped on boot; this pushes+compiles ptio/ge/rmqp/cdt into
+the guest from the sources kept here. Run it after every VM start.
+
+## guest_engines.c - client RM introspection (used with the ptrace tracer)
+
+Queries the guest client RM for exactly the data UVM's register path uses:
+virtualization mode (0x800289), subdevice count (0x800280), CLASS LIST V2
+(0x800292), ENGINE LIST V2 (0x20800170), CE caps (0x20802a03) and CE->PCE
+masks (0x20802a02). Needs the NVOS64 alloc path; see rmq_probe.c note.
+
+## Guest-side UVM logging (critical for diagnosis!)
+
+UVM's messages are silent until enabled:
+
+ echo 1 | sudo tee /sys/module/nvidia_uvm/parameters/uvm_debug_prints
+
+Then run /tmp/cdt and `dmesg | grep nvidia-uvm`. This produced the current
+blocker's exact message:
+`nvidia-uvm: uvm_gpu.c:1574 init_gpu[pid:..] Failed to initialize the
+channel manager: Call not supported [NV_ERR_NOT_SUPPORTED]`
+(also useful: uvm_enable_debug_procfs, uvm_debug_enable_push_desc,
+uvm_release_asserts, uvm_release_asserts_dump_stack)
+
+## launch_qemu_rootport.sh / .run.sh - QEMU command line
+
+Template (root-port fix) + runtime copy with the live mdev UUID, patched by
+vmcycle.sh. Sockets still live under /tmp because scripts/vgpu_vm.py talks to
+/tmp/qemu-guest-agent.sock by design.
+
+## dbg_seq.sh / dbg_seq2.sh - control-sequence helpers (guest, needs ptio)
+
+Print the RM control sequence immediately before/after the failing
+UVM_REGISTER_GPU ioctl.
diff --git a/tools/guest-diag/dbg_seq.sh b/tools/guest-diag/dbg_seq.sh
new file mode 100755
index 0000000..247bc91
--- /dev/null
+++ b/tools/guest-diag/dbg_seq.sh
@@ -0,0 +1,5 @@
+#!/bin/bash
+/tmp/ptio /tmp/cdt > /dev/null 2>/tmp/tr.log
+n=$(grep -n 'req=0x25' /tmp/tr.log | head -1 | cut -d: -f1)
+echo "marker line: $n"
+sed -n "$((n-28)),$((n+2))p" /tmp/tr.log
diff --git a/tools/guest-diag/dbg_seq2.sh b/tools/guest-diag/dbg_seq2.sh
new file mode 100755
index 0000000..8710fd9
--- /dev/null
+++ b/tools/guest-diag/dbg_seq2.sh
@@ -0,0 +1,5 @@
+#!/bin/bash
+/tmp/ptio /tmp/cdt > /dev/null 2>/tmp/tr.log
+n=$(grep -n 'req=0x25' /tmp/tr.log | head -1 | cut -d: -f1)
+echo "=== controls AFTER register (line $n), first 40: ==="
+sed -n "$((n+1)),$((n+45))p" /tmp/tr.log
diff --git a/tools/guest-diag/guest_engines.c b/tools/guest-diag/guest_engines.c
new file mode 100644
index 0000000..10209e2
--- /dev/null
+++ b/tools/guest-diag/guest_engines.c
@@ -0,0 +1,192 @@
+/* guest_engines.c - query the guest client RM for the data UVM's register
+ * path depends on: virtualization mode, subdevice count, ENGINE LIST,
+ * CLASS LIST, CE caps and CE->PCE masks. Uses the NVOS64 serialized alloc
+ * path (same as tools/rmq/rmq.c) so device allocs succeed.
+ *
+ * Build in guest: gcc -O1 -o /tmp/ge guest_engines.c ; run /tmp/ge
+ */
+#include <stdint.h>
+#include <stdio.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <fcntl.h>
+#include <unistd.h>
+#include <stdlib.h>
+
+typedef uint32_t NvU32;
+typedef uint32_t NvHandle;
+typedef uint64_t NvP64;
+
+#define NV_IOCTL_MAGIC 'F'
+#define NV_ESC_RM_ALLOC 0x2B
+#define NV_ESC_RM_CONTROL 0x2A
+#define NV01_ROOT 0x0
+#define NV01_ROOT_CLIENT 0x41
+#define NV01_DEVICE_0 0x80
+#define NV20_SUBDEVICE_0 0x2080
+
+typedef struct {
+ NvHandle hRoot, hObjectParent, hObjectNew;
+ NvU32 hClass;
+ NvP64 pAllocParms;
+ NvU32 paramsSize, status;
+} NVOS21_PARAMETERS;
+
+typedef struct {
+ NvHandle hRoot, hObjectParent, hObjectNew;
+ NvU32 hClass;
+ NvP64 pAllocParms;
+ NvP64 pRightsRequested;
+ NvU32 paramsSize, flags, status;
+} NVOS64_PARAMETERS;
+
+typedef struct {
+ NvHandle hClient, hObject;
+ NvU32 cmd, flags;
+ NvP64 params;
+ NvU32 paramsSize, status;
+} NVOS54_PARAMETERS;
+
+static int fd;
+
+static NvU32 rm_alloc(NvHandle hRoot, NvHandle hParent, NvU32 hClass,
+ NvHandle *pOut)
+{
+ NVOS21_PARAMETERS p;
+ memset(&p, 0, sizeof(p));
+ p.hRoot = hRoot; p.hObjectParent = hParent; p.hClass = hClass;
+ if (ioctl(fd, _IOWR(NV_IOCTL_MAGIC, NV_ESC_RM_ALLOC, NVOS21_PARAMETERS), &p) < 0) {
+ perror("alloc ioctl"); return 0xffffffff;
+ }
+ *pOut = p.hObjectNew;
+ return p.status;
+}
+
+static NvU32 rm_alloc64(NvHandle hRoot, NvHandle hParent, NvU32 hClass,
+ NvHandle *pOut)
+{
+ NVOS64_PARAMETERS p;
+ memset(&p, 0, sizeof(p));
+ p.hRoot = hRoot; p.hObjectParent = hParent; p.hClass = hClass;
+ p.flags = 1; /* FINN_SERIALIZED */
+ if (ioctl(fd, _IOWR(NV_IOCTL_MAGIC, NV_ESC_RM_ALLOC, NVOS64_PARAMETERS), &p) < 0) {
+ perror("alloc64 ioctl"); return 0xffffffff;
+ }
+ *pOut = p.hObjectNew;
+ return p.status;
+}
+
+static NvU32 rm_control(NvHandle hClient, NvHandle hObject, NvU32 cmd,
+ void *params, NvU32 size)
+{
+ NVOS54_PARAMETERS p;
+ memset(&p, 0, sizeof(p));
+ p.hClient = hClient; p.hObject = hObject; p.cmd = cmd;
+ p.params = (NvP64)params; p.paramsSize = size;
+ if (ioctl(fd, _IOWR(NV_IOCTL_MAGIC, NV_ESC_RM_CONTROL, NVOS54_PARAMETERS), &p) < 0) {
+ perror("control ioctl"); return 0xffffffff;
+ }
+ return p.status;
+}
+
+#define MAX_CLASSES 256
+
+int main(void)
+{
+ NvHandle hClient = 0, hDevice = 0, hSubdevice = 0;
+ NvU32 st;
+
+ fd = open("/dev/nvidiactl", O_RDWR | O_CLOEXEC);
+ if (fd < 0) { perror("open nvidiactl"); return 1; }
+
+ st = rm_alloc(0, 0, getenv("RC") ? NV01_ROOT_CLIENT : NV01_ROOT, &hClient);
+ printf("alloc client: st=0x%x hClient=0x%x\n", st, hClient);
+
+ st = rm_alloc64(hClient, hClient, NV01_DEVICE_0, &hDevice);
+ printf("alloc device: st=0x%x hDevice=0x%x\n", st, hDevice);
+ if (st) return 2;
+
+ st = rm_alloc64(hClient, hDevice, NV20_SUBDEVICE_0, &hSubdevice);
+ printf("alloc subdevice: st=0x%x hSub=0x%x\n", st, hSubdevice);
+
+ {
+ struct { NvU32 mode; NvU32 isGridBuild; } v = {0xdead, 0xdead};
+ st = rm_control(hClient, hDevice, 0x800289, &v, sizeof(v));
+ printf("GET_VIRTUALIZATION_MODE: st=0x%x mode=%u (0=NONE,1=NMOS,2=VGX) gridBuild=%u\n",
+ st, v.mode, v.isGridBuild);
+ }
+ {
+ struct { NvU32 n; } s = {0xdead};
+ st = rm_control(hClient, hDevice, 0x800280, &s, sizeof(s));
+ printf("GET_NUM_SUBDEVICES: st=0x%x n=%u\n", st, s.n);
+ }
+ {
+ struct { NvU32 numClasses; NvU32 classList[MAX_CLASSES]; } c;
+ int i;
+ memset(&c, 0, sizeof(c));
+ st = rm_control(hClient, hDevice, 0x800292, &c, sizeof(c));
+ printf("GET_CLASSLIST_V2: st=0x%x numClasses=%u\n", st, c.numClasses);
+ if (!st && c.numClasses) {
+ printf(" classes:");
+ for (i = 0; i < (int)c.numClasses && i < MAX_CLASSES; i++)
+ printf(" 0x%x", c.classList[i]);
+ printf("\n");
+ }
+ }
+ if (hSubdevice) {
+ struct { NvU32 engineCount; NvU32 engineList[0x54]; } e;
+ int i;
+ memset(&e, 0, sizeof(e));
+ st = rm_control(hClient, hSubdevice, 0x20800170, &e, sizeof(e));
+ printf("GPU_GET_ENGINES_V2: st=0x%x engineCount=%u\n", st, e.engineCount);
+ if (!st && e.engineCount) {
+ printf(" engines:");
+ for (i = 0; i < (int)e.engineCount && i < 0x54; i++)
+ printf(" 0x%x", e.engineList[i]);
+ printf("\n");
+ }
+ for (i = 0; i < (int)e.engineCount && i < 0x54; i++) {
+ struct { NvU32 t, mask; } m;
+ struct { NvU32 t; unsigned char caps[2]; } cc;
+ NvU32 t = e.engineList[i];
+ memset(&m, 0, sizeof(m));
+ st = rm_control(hClient, hSubdevice, 0x20802a02, &m, sizeof(m));
+ printf(" CE_GET_CE_PCE_MASK(type=0x%x): st=0x%x mask=0x%x\n", t, st, m.mask);
+ memset(&cc, 0, sizeof(cc));
+ cc.t = t;
+ st = rm_control(hClient, hSubdevice, 0x20802a03, &cc, sizeof(cc));
+ printf(" CE_GET_CAPS_V2(type=0x%x): st=0x%x caps=%02x %02x\n", t, st, cc.caps[0], cc.caps[1]);
+ }
+ }
+ if (hSubdevice) {
+ struct {
+ NvU32 engineData[16];
+ NvU32 pbdmaIds[2];
+ NvU32 pbdmaFaultIds[2];
+ NvU32 numPbdmas;
+ char engineName[16];
+ } e[32];
+ struct {
+ NvU32 baseIndex, numEntries, bMore;
+ typeof(e) entries;
+ } t;
+ int i, j;
+ NvU32 base = 0;
+ memset(&t, 0, sizeof(t));
+ do {
+ t.baseIndex = base;
+ st = rm_control(hClient, hSubdevice, 0x20801112, &t, sizeof(t));
+ if (st) { printf("FIFO_GET_DEVICE_INFO_TABLE: st=0x%x\n", st); break; }
+ for (i = 0; i < (int)t.numEntries; i++) {
+ printf(" [%2u] %-8s numPbdmas=%u pbdmaIds=[%u,%u] engineData=[",
+ base + i, t.entries[i].engineName, t.entries[i].numPbdmas,
+ t.entries[i].pbdmaIds[0], t.entries[i].pbdmaIds[1]);
+ for (j = 0; j < 16; j++)
+ printf("%s%u", j ? "," : "", t.entries[i].engineData[j]);
+ printf("]\n");
+ }
+ base += t.numEntries;
+ } while (t.bMore);
+ }
+ return 0;
+}
diff --git a/tools/guest-diag/kret.sh b/tools/guest-diag/kret.sh
new file mode 100755
index 0000000..9ed1220
--- /dev/null
+++ b/tools/guest-diag/kret.sh
@@ -0,0 +1,33 @@
+#!/bin/bash
+# kret.sh - kretprobe the nvGpuOps* call chain in the guest's nvidia.ko and
+# capture return values while /tmp/cdt runs. Run with sudo.
+set -u
+
+TR=/sys/kernel/tracing
+SYMS="nvGpuOpsGetGpuInfo nvGpuOpsDeviceCreate nvGpuOpsRmDeviceCreate \
+nvGpuOpsRmSubDeviceCreate queryFbInfo findDeviceClasses queryVirtMode \
+getSysmemLinkInfo getSystemMemoryWindow getNvswitchInfo getEgmInfo getAtsInfo \
+nvGpuOpsQueryGpuConfidentialComputeCaps allocNvlinkStatus"
+
+echo > $TR/kprobe_events
+for f in $SYMS; do
+ if ! grep -q " [tT] $f\b" /proc/kallsyms; then
+ echo "skip $f (not in kallsyms)"
+ continue
+ fi
+ if ! echo "r:ret_$f $f \$retval" >> $TR/kprobe_events 2>/dev/null; then
+ echo "FAILED to register $f"
+ fi
+done
+echo "--- registered:"
+grep ret_ $TR/kprobe_events 2>/dev/null | head -20
+
+if [ -d $TR/events/kprobes ]; then
+ echo 1 > $TR/events/kprobes/enable
+ dmesg -C
+ /tmp/cdt >/dev/null 2>&1
+ echo "--- results (retval per probe):"
+ grep -E "ret_" $TR/trace | awk '{n=$4; sub(/^ret_/,"",n); printf "%s %s\n", n, $NF}' | sort | uniq -c | sort -rn | head -30
+ echo 0 > $TR/events/kprobes/enable
+ echo > $TR/trace
+fi
diff --git a/tools/guest-diag/launch_qemu_rootport.sh b/tools/guest-diag/launch_qemu_rootport.sh
new file mode 100755
index 0000000..52728bd
--- /dev/null
+++ b/tools/guest-diag/launch_qemu_rootport.sh
@@ -0,0 +1,2 @@
+#!/bin/bash
+exec /bin/qemu-system-x86_64 -D /home/rmcore/.local/libvf.io/logs/qemu/e726ba0b-204f-4ec9-91da-983ed7f41d06-session.txt -no-hpet -mem-prealloc -qmp unix:/tmp/qemu-qmp.sock,server,nowait -chardev socket,path=/tmp/qemu-guest-agent.sock,server=on,wait=off,id=qga0 -device virtio-serial-pci -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 -serial file:/tmp/vgpu-guest-serial.log -gdb tcp::1234 -display none -device ivshmem-plain,id=shmem0,memdev=ivshmem_kvmfr -object memory-backend-file,id=ivshmem_kvmfr,mem-path=/dev/shm/kvmfr-e726ba0b-204f-4ec9-91da-983ed7f41d06,size=128M,share=yes -device ivshmem-plain,id=shmem1,memdev=ivshmem_kvmsr -object memory-backend-file,id=ivshmem_kvmsr,mem-path=/dev/shm/kvmsr-e726ba0b-204f-4ec9-91da-983ed7f41d06,size=2M,share=yes -uuid e726ba0b-204f-4ec9-91da-983ed7f41d06 -machine pc-q35-4.2,accel=kvm,usb=off,vmport=off,dump-guest-core=off -cpu host,ss=on,vmx=on,pcid=on,hypervisor=on,arat=on,tsc-adjust=on,umip=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaveopt=on,pdpe1gb=on,ibpb=on,amd-stibp=on,amd-ssbd=on,skip-l1dfl-vmentry=on,pschange-mc-no=on,hv-vapic,hv_time,hv-spinlocks=0x1fff,hv-vendor-id=null,kvm=on,topoext=on -rtc clock=host,base=localtime -m 4096 -smp cores=4,threads=1,sockets=1 -hda /home/rmcore/.local/libvf.io/kernel/linux.qcow2 --enable-kvm -device pcie-root-port,id=rp0,chassis=1,slot=30,bus=pcie.0 -device vfio-pci,id=hostdev0,bus=rp0,sysfsdev=/sys/bus/mdev/devices/<MDEV-UUID>,display=off -device rtl8139,netdev=net0 -netdev user,id=net0,hostfwd=tcp::2222-:22 -net user -net nic -device vhost-vsock-pci,guest-cid=50718 -qmp unix:/tmp/sockets/e726ba0b-204f-4ec9-91da-983ed7f41d06/main.sock,server,nowait -qmp unix:/tmp/sockets/e726ba0b-204f-4ec9-91da-983ed7f41d06/master.sock,server,nowait
diff --git a/tools/guest-diag/ptrace_ioctl.c b/tools/guest-diag/ptrace_ioctl.c
new file mode 100644
index 0000000..5f96b31
--- /dev/null
+++ b/tools/guest-diag/ptrace_ioctl.c
@@ -0,0 +1,236 @@
+/* ptrace_ioctl.c - follow a child with PTRACE_SYSCALL and dump the buffer
+ * results of selected UVM ioctls. Needed because libcuda issues raw syscall
+ * instructions (LD_PRELOAD/ltrace never see these).
+ *
+ * Build (guest): gcc -O1 -w -o /tmp/ptio ptrace_ioctl.c
+ * Run: /tmp/ptio /tmp/cdt
+ */
+#define _GNU_SOURCE
+#include <errno.h>
+#include <signal.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ptrace.h>
+#include <sys/types.h>
+#include <sys/wait.h>
+#include <unistd.h>
+
+#define SYS_IOCTL_X86_64 16
+
+struct syscall_info {
+ unsigned char op;
+ unsigned char pad[3];
+ unsigned int arch;
+ unsigned long long instruction_pointer;
+ unsigned long long stack_pointer;
+ union {
+ struct {
+ unsigned long long nr;
+ unsigned long long args[6];
+ } entry;
+ struct {
+ long long rval;
+ unsigned char is_error;
+ } exit;
+ } u;
+};
+
+static int interesting(unsigned long req)
+{
+ switch (req) {
+ case 0x1: /* UVM_RESERVE_VA */
+ case 0x2: /* UVM_RELEASE_VA */
+ case 0x25: /* UVM_REGISTER_GPU */
+ case 0x27: /* UVM_PAGEABLE_MEM_ACCESS */
+ case 0x17: /* UVM_EVENT_CTRL */
+ case 0x19: /* UVM_REGISTER_MPS_CLIENT */
+ return 1;
+ default:
+ return 0;
+ }
+}
+
+static void dump_words(pid_t pid, unsigned long addr, int nwords)
+{
+ int i;
+
+ fprintf(stderr, " buffer @%#lx:", addr);
+ for (i = 0; i < nwords; i++) {
+ errno = 0;
+ long v = ptrace(PTRACE_PEEKDATA, pid, addr + i * 8, NULL);
+ if (errno)
+ break;
+ fprintf(stderr, " %016lx", (unsigned long)v);
+ }
+ fprintf(stderr, "\n");
+}
+
+struct saved {
+ pid_t pid;
+ int used;
+ unsigned long req, arg;
+} saved[64];
+
+static struct saved *slot(pid_t pid)
+{
+ int i, free_i = -1;
+
+ for (i = 0; i < 64; i++) {
+ if (saved[i].used && saved[i].pid == pid)
+ return &saved[i];
+ if (!saved[i].used && free_i < 0)
+ free_i = i;
+ }
+ if (free_i >= 0) {
+ saved[free_i].used = 1;
+ saved[free_i].pid = pid;
+ saved[free_i].req = 0;
+ return &saved[free_i];
+ }
+ return NULL;
+}
+
+int main(int argc, char **argv)
+{
+ pid_t pid;
+ int nprint = 0;
+
+ if (argc < 2) {
+ fprintf(stderr, "usage: %s prog [args...]\n", argv[0]);
+ return 2;
+ }
+
+ pid = fork();
+ if (pid == 0) {
+ ptrace(PTRACE_TRACEME, 0, 0, 0);
+ raise(SIGSTOP);
+ execvp(argv[1], &argv[1]);
+ perror("execvp");
+ _exit(127);
+ }
+
+ waitpid(pid, NULL, WUNTRACED);
+ ptrace(PTRACE_SETOPTIONS, pid, 0,
+ (void *)(PTRACE_O_TRACESYSGOOD | PTRACE_O_TRACECLONE |
+ PTRACE_O_TRACEFORK | PTRACE_O_TRACEVFORK | PTRACE_O_EXITKILL));
+ ptrace(PTRACE_SYSCALL, pid, 0, 0);
+
+ fprintf(stderr, "[ptio] attached to pid %d\n", pid);
+ for (;;) {
+ int status;
+ pid_t p = waitpid(-1, &status, __WALL);
+
+ if (p < 0) {
+ if (errno == EINTR)
+ continue;
+ break;
+ }
+ if (WIFEXITED(status) || WIFSIGNALED(status))
+ continue;
+ if (!WIFSTOPPED(status))
+ continue;
+
+ {
+ int sig = WSTOPSIG(status);
+ struct syscall_info si;
+
+ memset(&si, 0, sizeof(si));
+ if ((sig == (SIGTRAP | 0x80)) &&
+ ptrace(PTRACE_GET_SYSCALL_INFO, p, sizeof(si), &si) > 0) {
+ struct saved *sv = slot(p);
+
+ if (si.op == 1 /* entry */) {
+ if (sv && si.u.entry.nr == SYS_IOCTL_X86_64) {
+ sv->req = si.u.entry.args[1];
+ sv->arg = si.u.entry.args[2];
+ if (sv->req == 0x25) {
+ fprintf(stderr, "[ptio] ENTER UVM_REGISTER_GPU pid=%d\n", p);
+ dump_words(p, sv->arg, 6);
+ fprintf(stderr, " ^ IN: uuid(2q) | numaEnabled numaNodeId | rmCtrlFd hClient | hSmcPartRef rmStatus\n");
+ }
+ } else if (sv) {
+ sv->req = 0;
+ }
+ } else if (si.op == 2 /* exit */ && sv &&
+ ((sv->req & 0xff) == 0x4629ul ||
+ ((sv->req & 0xffff00fful) == 0x0046002bul) ||
+ ((sv->req & 0xffff00fful) == 0x0046002aul))) {
+ unsigned int own = (unsigned int)(sv->req & 0xff);
+ long q0 = 0, q1 = 0, q2 = 0, q3 = 0;
+ unsigned long st = 0;
+
+ errno = 0; q0 = ptrace(PTRACE_PEEKDATA, p, sv->arg, NULL);
+ errno = 0; q1 = ptrace(PTRACE_PEEKDATA, p, sv->arg + 8, NULL);
+ errno = 0; q2 = ptrace(PTRACE_PEEKDATA, p, sv->arg + 16, NULL);
+ errno = 0; q3 = ptrace(PTRACE_PEEKDATA, p, sv->arg + 24, NULL);
+ if (own == 0x29)
+ st = (unsigned long)((q1 >> 32) & 0xfffffffful);
+ else
+ st = (unsigned long)((q3 >> 32) & 0xfffffffful);
+ if (st != 0)
+ fprintf(stderr,
+ "[rmio] pid=%d nr=0x%x status=0x%lx hClient=0x%lx hObj=0x%lx cmd/class=0x%lx size=%lu\n",
+ p, own,
+ st,
+ (unsigned long)(q0 & 0xfffffffful),
+ (unsigned long)((q0 >> 32) & 0xfffffffful),
+ (unsigned long)(q1 & 0xfffffffful),
+ (unsigned long)((q3 >> 32) & 0xfffffffful));
+ } else if (si.op == 2 /* exit */ && sv &&
+ sv->req == 0xc020462aul) {
+ long q0, q1, q2, q3;
+ errno = 0;
+ q1 = ptrace(PTRACE_PEEKDATA, p, sv->arg + 8, NULL);
+ errno = 0;
+ q2 = ptrace(PTRACE_PEEKDATA, p, sv->arg + 16, NULL);
+ errno = 0;
+ q3 = ptrace(PTRACE_PEEKDATA, p, sv->arg + 24, NULL);
+ q0 = ptrace(PTRACE_PEEKDATA, p, sv->arg, NULL);
+ (void)q0;
+ fprintf(stderr,
+ "[rmctl] pid=%d cmd=0x%lx hClient=0x%lx params=%#lx size=%lu status=0x%lx\n",
+ p, (unsigned long)(q1 & 0xfffffffful),
+ (unsigned long)(q0 & 0xfffffffful),
+ (unsigned long)(q2 & 0xfffffffffffful),
+ (unsigned long)((q2 >> 32) & 0xfffffffful),
+ (unsigned long)((q3 >> 32) & 0xfffffffful));
+ if (((q3 >> 32) & 0xfffffffful) != 0 &&
+ (q2 & 0xfffffffffffful) > 0x10000) {
+ fprintf(stderr, " params:");
+ {
+ int i2;
+ for (i2 = 0; i2 < 6; i2++) {
+ errno = 0;
+ long v = ptrace(PTRACE_PEEKDATA, p,
+ (unsigned long)(q2 & 0xfffffffffffful) + i2 * 8, NULL);
+ if (errno)
+ break;
+ fprintf(stderr, " %016lx",
+ (unsigned long)v);
+ }
+ }
+ fprintf(stderr, "\n");
+ }
+ } else if (si.op == 2 /* exit */ && sv &&
+ interesting(sv->req)) {
+ fprintf(stderr,
+ "[ptio] pid=%d ioctl req=0x%lx arg=%#lx ret=%ld%s\n",
+ p, sv->req, sv->arg,
+ si.u.exit.rval,
+ interesting(sv->req) ? " *" : "");
+ if (sv->req == 0x25) {
+ dump_words(p, sv->arg, 6);
+ fprintf(stderr,
+ " ^ UVM_REGISTER_GPU: uuid(2q) | numaEnabled numaNodeId rmCtrlFd hClient | hSmcPartRef rmStatus\n");
+ } else {
+ dump_words(p, sv->arg, 4);
+ }
+ }
+ }
+ ptrace(PTRACE_SYSCALL, p, 0, sig == (SIGTRAP | 0x80) ? 0 :
+ (sig == SIGTRAP ? 0 : sig));
+ }
+ }
+ return 0;
+}
diff --git a/tools/guest-diag/push_guest_tools.sh b/tools/guest-diag/push_guest_tools.sh
new file mode 100755
index 0000000..a2a3dbb
--- /dev/null
+++ b/tools/guest-diag/push_guest_tools.sh
@@ -0,0 +1,31 @@
+#!/bin/bash
+# push_guest_tools.sh - (re)build the guest-side diagnostic binaries from the
+# sources kept in this directory. The guest's /tmp is wiped on boot, so run
+# this after every VM start; the sources here are the persistent copies.
+#
+# Usage: tools/guest-diag/push_guest_tools.sh
+# (env: VM="vgpu-linux" is not needed; scripts/vgpu_vm.py talks to the
+# running VM's guest agent socket)
+set -e
+
+HERE="$(cd "$(dirname "$0")" && pwd)"
+VMEXEC="$HERE/../../scripts/vgpu_vm.py"
+REPO="$HERE/../.."
+
+push_b64() { # push_b64 <local file> <remote file>
+ local b64
+ b64=$(base64 -w0 "$1")
+ "$VMEXEC" exec "echo '$b64' | base64 -d > '$2'" >/dev/null 2>&1
+}
+
+echo "[*] pushing sources..."
+push_b64 "$HERE/ptrace_ioctl.c" /tmp/ptio.c
+push_b64 "$HERE/guest_engines.c" /tmp/ge.c
+push_b64 "$HERE/rmq_probe.c" /tmp/rmqp.c
+push_b64 "$REPO/tools/test/cuda_drv_test.c" /tmp/cdt.c
+
+echo "[*] compiling in guest..."
+"$VMEXEC" exec "gcc -O1 -w -o /tmp/ptio /tmp/ptio.c && \
+gcc -O1 -w -o /tmp/ge /tmp/ge.c && \
+gcc -O1 -w -o /tmp/rmqp /tmp/rmqp.c && \
+gcc -O1 -o /tmp/cdt /tmp/cdt.c -ldl && echo BUILD_OK"
diff --git a/tools/guest-diag/rmq_probe.c b/tools/guest-diag/rmq_probe.c
new file mode 100644
index 0000000..7554309
--- /dev/null
+++ b/tools/guest-diag/rmq_probe.c
@@ -0,0 +1,132 @@
+/* rmq_probe.c - minimal legacy-ioctl RM client to read device controls:
+ * 0x800289 NV0080_CTRL_CMD_GPU_GET_VIRTUALIZATION_MODE
+ * 0x800280 NV0080_CTRL_CMD_GPU_GET_NUM_SUBDEVICES
+ * 0x800292 NV0080_CTRL_CMD_GPU_GET_CLASSLIST_V2 (count only)
+ * plus (optional) the A080 VGPU config query used by UVM's queryVirtMode.
+ *
+ * Build in guest: gcc -O1 -o /tmp/rmqp rmq_probe.c ; run /tmp/rmqp
+ */
+#include <errno.h>
+#include <fcntl.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <unistd.h>
+
+#define NV_IOCTL_MAGIC 'F'
+#define IOCTL_DIR(nr, sz) _IOC(_IOC_READ | _IOC_WRITE, NV_IOCTL_MAGIC, nr, sz)
+
+#define NV_ESC_RM_ALLOC 0x2b
+#define NV_ESC_RM_CONTROL 0x2a
+#define NV_ESC_RM_FREE 0x29
+
+struct nvos21 {
+ uint32_t hRoot, hObjectParent, hObjectNew, hClass;
+ uint64_t pAllocParms;
+ uint32_t paramsSize, status;
+};
+
+struct nvos54 {
+ uint32_t hClient, hObject, cmd, flags;
+ uint64_t params;
+ uint32_t paramsSize, status;
+};
+
+#define NV01_ROOT_CLIENT 0x00000041
+#define NV01_DEVICE_0 0x00000080
+
+static int fd;
+
+static int rm_alloc(uint32_t hRoot, uint32_t hParent, uint32_t *hNew,
+ uint32_t hClass, void *parms, uint32_t parmsSize)
+{
+ struct nvos21 a;
+
+ memset(&a, 0, sizeof(a));
+ a.hRoot = hRoot;
+ a.hObjectParent = hParent;
+ a.hObjectNew = *hNew;
+ a.hClass = hClass;
+ a.pAllocParms = (uint64_t)(uintptr_t)parms;
+ a.paramsSize = parmsSize;
+ if (ioctl(fd, IOCTL_DIR(NV_ESC_RM_ALLOC, sizeof(a)), &a) < 0) {
+ perror("ioctl alloc");
+ return -1;
+ }
+ *hNew = a.hObjectNew;
+ return (int)a.status;
+}
+
+static int rm_control(uint32_t hClient, uint32_t hObject, uint32_t cmd,
+ void *params, uint32_t paramsSize)
+{
+ struct nvos54 c;
+
+ memset(&c, 0, sizeof(c));
+ c.hClient = hClient;
+ c.hObject = hObject;
+ c.cmd = cmd;
+ c.params = (uint64_t)(uintptr_t)params;
+ c.paramsSize = paramsSize;
+ if (ioctl(fd, IOCTL_DIR(NV_ESC_RM_CONTROL, sizeof(c)), &c) < 0) {
+ perror("ioctl control");
+ return -1;
+ }
+ return (int)c.status;
+}
+
+int main(void)
+{
+ uint32_t hClient = 0, hDevice = 0;
+ struct {
+ uint32_t deviceId;
+ uint32_t hClientShare;
+ uint32_t hTargetClient;
+ uint32_t hTargetDevice;
+ uint32_t flags;
+ uint32_t pad0;
+ uint64_t vaSpaceSize;
+ uint64_t vaStartInternal;
+ uint64_t vaLimitInternal;
+ uint32_t vaMode;
+ uint32_t pad1;
+ } devAlloc;
+ int st;
+ struct {
+ uint32_t virtualizationMode;
+ uint32_t isGridBuild;
+ } virt = { 0xdeadbeef, 0xdeadbeef };
+ struct {
+ uint32_t numSubDevices;
+ } subdev = { 0xdeadbeef };
+
+ fd = open("/dev/nvidiactl", O_RDWR | O_CLOEXEC);
+ if (fd < 0) {
+ perror("open nvidiactl");
+ return 1;
+ }
+
+ st = rm_alloc(0, 0, &hClient, NV01_ROOT_CLIENT, NULL, 0);
+ printf("alloc client: st=0x%x hClient=0x%x\n", st, hClient);
+ if (st != 0)
+ return 2;
+
+ memset(&devAlloc, 0, sizeof(devAlloc));
+ devAlloc.deviceId = 0;
+ st = rm_alloc(hClient, hClient, &hDevice, NV01_DEVICE_0, &devAlloc,
+ sizeof(devAlloc));
+ printf("alloc device: st=0x%x hDevice=0x%x\n", st, hDevice);
+ if (st != 0)
+ return 3;
+
+ st = rm_control(hClient, hDevice, 0x800289, &virt, sizeof(virt));
+ printf("GET_VIRTUALIZATION_MODE: st=0x%x mode=0x%x isGridBuild=%u\n",
+ st, virt.virtualizationMode, virt.isGridBuild);
+
+ st = rm_control(hClient, hDevice, 0x800280, &subdev, sizeof(subdev));
+ printf("GET_NUM_SUBDEVICES: st=0x%x numSubDevices=%u\n",
+ st, subdev.numSubDevices);
+
+ return 0;
+}
diff --git a/tools/guest-diag/sysc_shim.c b/tools/guest-diag/sysc_shim.c
new file mode 100644
index 0000000..9b07e4c
--- /dev/null
+++ b/tools/guest-diag/sysc_shim.c
@@ -0,0 +1,73 @@
+/* sysc_shim.c - intercept syscall() so we can dump ioctl buffer *results*
+ * for UVM ioctls (libcuda calls syscall(SYS_ioctl, ...) directly, so an
+ * ioctl() wrapper is bypassed but syscall() is not).
+ * Build in guest: gcc -shared -fPIC -O1 -o /tmp/sysc.so sysc_shim.c -ldl
+ * Run: LD_PRELOAD=/tmp/sysc.so /tmp/cdt
+ */
+#define _GNU_SOURCE
+#include <dlfcn.h>
+#include <stdarg.h>
+#include <stddef.h>
+#include <sys/syscall.h>
+#include <unistd.h>
+
+static long (*real_syscall)(long, ...);
+
+static void dump(const char *tag, unsigned long req, void *arg, long ret)
+{
+ unsigned int *p = (unsigned int *)arg;
+ char buf[512];
+ int n, i;
+ long *vp = (long *)arg;
+
+ if (!p)
+ return;
+ n = 0;
+ n += __builtin_snprintf(buf + n, sizeof(buf) - n,
+ "SYSC[%s] req=0x%lx ret=%ld dwords: ", tag, req,
+ ret);
+ for (i = 0; i < 16 && n < (int)sizeof(buf) - 16; i++)
+ n += __builtin_snprintf(buf + n, sizeof(buf) - n, "%08x ",
+ p[i]);
+ (void)vp;
+ write(2, buf, n);
+ write(2, "\n", 1);
+}
+
+long syscall(long nr, ...)
+{
+ va_list ap;
+ long a1, a2, a3, a4, a5, a6;
+ long ret;
+
+ if (!real_syscall)
+ real_syscall = dlsym(RTLD_NEXT, "syscall");
+
+ va_start(ap, nr);
+ a1 = va_arg(ap, long);
+ a2 = va_arg(ap, long);
+ a3 = va_arg(ap, long);
+ a4 = va_arg(ap, long);
+ a5 = va_arg(ap, long);
+ a6 = va_arg(ap, long);
+ va_end(ap);
+
+ ret = real_syscall(nr, a1, a2, a3, a4, a5, a6);
+
+ if (nr == SYS_ioctl) {
+ unsigned long req = (unsigned long)a2;
+ switch (req) {
+ case 0x1: /* UVM_RESERVE_VA */
+ case 0x2: /* UVM_RELEASE_VA */
+ case 0x25: /* UVM_REGISTER_GPU */
+ case 0x27: /* UVM_PAGEABLE_MEM_ACCESS */
+ case 0x17: /* UVM_EVENT_CTRL */
+ case 0x19: /* UVM_REGISTER_MPS_CLIENT */
+ dump("uvmioctl", req, (void *)a3, ret);
+ break;
+ default:
+ break;
+ }
+ }
+ return ret;
+}
diff --git a/tools/guest-diag/vmcycle.sh b/tools/guest-diag/vmcycle.sh
new file mode 100755
index 0000000..6faaa75
--- /dev/null
+++ b/tools/guest-diag/vmcycle.sh
@@ -0,0 +1,46 @@
+#!/bin/bash
+# vmcycle.sh - reload cycle for the emulated vGPU stack:
+# stop QEMU -> remove mdevs -> rmmod/insmod open-iov-vfio.ko -> create a
+# fresh mdev -> patch its UUID into the runtime launch script -> relaunch.
+#
+# ALL artifacts stay inside this directory (tools/guest-diag/) so nothing is
+# lost on reboot:
+# launch_qemu_rootport.sh - template QEMU command line (root-port fix)
+# launch_qemu_rootport.run.sh - runtime copy with the current mdev UUID
+# run/qemu.log - QEMU stderr/stdout of the last launch
+#
+# Usage: tools/guest-diag/vmcycle.sh
+# Env: MDEV_PARENT (default: 0000:01:00.0)
+# MDEV_TYPE (default: nvidia-nvidia-vgpu-turing)
+set -e
+
+HERE="$(cd "$(dirname "$0")" && pwd)"
+TEMPLATE="$HERE/launch_qemu_rootport.sh"
+QEMU_SCRIPT="${QEMU_SCRIPT:-$HERE/launch_qemu_rootport.run.sh}"
+MODULE="${MODULE:-$HERE/../../kernel/open-iov-vfio.ko}"
+MDEV_PARENT="${MDEV_PARENT:-0000:01:00.0}"
+MDEV_TYPE="${MDEV_TYPE:-nvidia-nvidia-vgpu-turing}"
+RUNDIR="$HERE/run"
+
+mkdir -p "$RUNDIR"
+[ -f "$QEMU_SCRIPT" ] || cp "$TEMPLATE" "$QEMU_SCRIPT"
+
+pkill -f qemu-system-x86_64 2>/dev/null || true
+sleep 4
+for m in /sys/bus/mdev/devices/*; do
+ [ -d "$m" ] && echo 1 | sudo tee "$m/remove" >/dev/null
+done
+sudo rmmod open_iov_vfio 2>/dev/null || true
+sudo insmod "$MODULE"
+U=$(uuidgen)
+echo "$U" | sudo tee "/sys/class/mdev_bus/$MDEV_PARENT/mdev_supported_types/$MDEV_TYPE/create" >/dev/null
+sed -i "s|/sys/bus/mdev/devices/[0-9a-f-]*|/sys/bus/mdev/devices/$U|" "$QEMU_SCRIPT"
+mkdir -p /tmp/sockets/e726ba0b-204f-4ec9-91da-983ed7f41d06
+rm -f /tmp/qemu-guest-agent.sock /tmp/qemu-qmp.sock
+(setsid nohup "$QEMU_SCRIPT" > "$RUNDIR/qemu.log" 2>&1 &)
+sleep 6
+if pgrep -f qemu-system-x86_64 >/dev/null; then
+ echo "VM up (mdev $U) [qemu log: $RUNDIR/qemu.log]"
+else
+ echo "VM FAILED"; tail -3 "$RUNDIR/qemu.log"; exit 1
+fi
generated by cgit v1.2.3 (git 2.39.1) at 2026-10-12 21:06:31 +0000