# guest-diag - tracing tools for the emulated vGPU guest

Tools used while driving the guest's stock NVIDIA driver through CUDA bring-up.
All are tiny and dependency-free (libc only).

## ptrace_ioctl.c - ioctl/result tracer (the important one)

libcuda issues **raw `syscall` instructions**, so LD_PRELOAD shims and ltrace
never see its ioctls. This ptrace tracer follows a child with
PTRACE_SYSCALL + PTRACE_GET_SYSCALL_INFO and dumps:

- `req=0x25` (UVM_REGISTER_GPU) buffer on entry and exit
  (`rmStatus` is the high dword of the 5th qword) - this is how the current
  failure was found: `rmStatus = 0x56 NV_ERR_NOT_SUPPORTED`
- `req=0xc020462a` (NV_ESC_RM_CONTROL) cmd + status
- `req=0xc020462b` / `0x...4629` (RM alloc/free) status when nonzero
- UVM ioctls 0x1/0x2/0x17/0x19/0x25/0x27

Build/run **in the guest**:

    gcc -O1 -w -o /tmp/ptio ptrace_ioctl.c
    /tmp/ptio /tmp/cdt 2>&1 | grep -E 'ptio|rmctl|rmio|\^'

Note: `op` in `struct ptrace_syscall_info` is ENTRY=1 / EXIT=2, and the struct
has `pad[3], arch, instruction_pointer, stack_pointer` before the union.

## sysc_shim.c - known dead end

An `syscall()` LD_PRELOAD interceptor: libcuda bypasses it entirely (inline
syscall instruction). Kept only as documentation of that fact.

## vmcycle.sh - reload cycle

pkill QEMU -> remove mdevs -> rmmod/insmod vgsp.ko -> create a fresh
`nvidia-nvidia-vgpu-turing` mdev -> patch its UUID into
launch_qemu_rootport.sh -> relaunch. Guest boots in ~60-75 s.

## launch_qemu_rootport.sh - QEMU command line with pcie-root-port

Snapshot of the working launch command (captured from a running VM's QEMU
process) plus the root-port fix. **Requires** the mdev UUID to be substituted and:

    mkdir -p /tmp/sockets/<domain-uuid>      # else QMP bind fails
    rm -f /tmp/qemu-guest-agent.sock /tmp/qemu-qmp.sock

Why the root port matters: on the Q35 root bus (`pcie.0`) QEMU's
`vfio_setup_pcie_cap()` (hw/vfio/pci.c) relabels the mdev as "Root Complex
Integrated Endpoint" and clears LNK/LNK2 + DeviceCaps, so the guest reads
`LnkCap = 0` and `getPCIELinkRateMBps()` fails with "Unknown PCIe speed".
Behind a `pcie-root-port` QEMU leaves the capability alone.

## rmq_probe.c - legacy-ioctl RM control probe (guest or host)

Minimal RM client (client alloc + device alloc + NV_ESC_RM_CONTROL) to query
device/subdevice controls from userspace, e.g.
`0x800289 NV0080_CTRL_CMD_GPU_GET_VIRTUALIZATION_MODE`,
`0x800280 GET_NUM_SUBDEVICES`, `0x20800123 GPU_GET_ENGINES`.
NOTE: the device alloc currently fails (0x40 in guest, 0x1b on host) - the
device allocation sequence needs the proper NV0080_ALLOC_PARAMETERS handling
(or the NVOS64 path that tools/rmq/rmq.c uses); fix before relying on it.

## kret.sh - kretprobe attempt (blocked)

Tries to kretprobe the nvGpuOps*/nvUvmInterface* chain to read return values.
Blocked on this kernel: "trace_kprobe: Could not probe notrace function
<name> [nvidia]" - kprobes are not available for nvidia module text.
Kept for documentation.

## push_guest_tools.sh - rebuild the guest-side binaries from these sources

The guest's /tmp is wiped on boot; this pushes+compiles ptio/ge/rmqp/cdt into
the guest from the sources kept here. Run it after every VM start.

## guest_engines.c - client RM introspection (used with the ptrace tracer)

Queries the guest client RM for exactly the data UVM's register path uses:
virtualization mode (0x800289), subdevice count (0x800280), CLASS LIST V2
(0x800292), ENGINE LIST V2 (0x20800170), CE caps (0x20802a03) and CE->PCE
masks (0x20802a02). Needs the NVOS64 alloc path; see rmq_probe.c note.

## Guest-side UVM logging (critical for diagnosis!)

UVM's messages are silent until enabled:

    echo 1 | sudo tee /sys/module/nvidia_uvm/parameters/uvm_debug_prints

Then run /tmp/cdt and `dmesg | grep nvidia-uvm`. This produced the current
blocker's exact message:
`nvidia-uvm: uvm_gpu.c:1574 init_gpu[pid:..] Failed to initialize the
channel manager: Call not supported [NV_ERR_NOT_SUPPORTED]`
(also useful: uvm_enable_debug_procfs, uvm_debug_enable_push_desc,
uvm_release_asserts, uvm_release_asserts_dump_stack)

## launch_qemu_rootport.sh / .run.sh - QEMU command line

Template (root-port fix) + runtime copy with the live mdev UUID, patched by
vmcycle.sh. Sockets still live under /tmp because scripts/vgpu_vm.py talks to
/tmp/qemu-guest-agent.sock by design.

## dbg_seq.sh / dbg_seq2.sh - control-sequence helpers (guest, needs ptio)

Print the RM control sequence immediately before/after the failing
UVM_REGISTER_GPU ioctl.