Problem: Error opening gds stats, pid: XXXXXX error: No such file or directory

Developer Tools Other Tools System Management and Monitoring (NVML)
1

I am attempting to run gds_stats as is done in the GPUDirect Storage Benchmarking and Configuration Guide. When passing the PID of the gdsio process, I get error: No such file or directory.

(kvikio-venv) [strugf@gpu0300 ~]$ /usr/local/cuda-12.6/gds/tools/gdsio -f /mnt/nvme/strugf/test gdsio -d 0 -w 8 -s 500M -i 1M -x 0 -I 0 -T 120 &
[5] 666370
(kvikio-venv) [strugf@gpu0300 ~]$ ls /proc/666370/
arch_status  coredump_filter     gid_map            mem         oom_score      schedstat     status
attr         cpu_resctrl_groups  io                 mountinfo   oom_score_adj  sessionid     syscall
autogroup    cpuset              ksm_merging_pages  mounts      pagemap        setgroups     task
auxv         cwd                 ksm_stat           mountstats  patch_state    smaps         timens_offsets
cgroup       environ             limits             net         personality    smaps_rollup  timers
clear_refs   exe                 loginuid           ns          projid_map     stack         timerslack_ns
cmdline      fd                  map_files          numa_maps   root           stat          uid_map
comm         fdinfo              maps               oom_adj     sched          statm         wchan
(kvikio-venv) [strugf@gpu0300 ~]$ /usr/local/cuda-12.6/gds/tools/gds_stats -p 666370 -l 3
error opening gds stats, pid: 666370 error: No such file or directory

Can you help me understand why gds_stats is saying there is “No such file or directory”? This also happens when ran as root.

Thanks!

2

gds_stats uses /dev/shm to communicate with running process.
Also please enable gds_stats set to 3 in /etc/cufile.json to open the shared memory and show the stats in gds_stats tool.

3

Yes, it seemed that all I needed to do was enable profiling and then increase the level to 3. This may be an uncommon situation, but the system I am using has CUDA 12.6 and 12.9 installed. It seemed that gds_stats was getting confused by this.

[strugf@gpu0300 gds]$ /usr/local/cuda-12.6/gds/tools/gds_stats -p 759716 -l 3
gds stats version mismatch, expecting: 8 found version: 9

[strugf@gpu0300 gds]$ /usr/local/cuda-12.6/gds/tools/gds_stats
gds_stats version :8
usage :/usr/local/cuda-12.6/gds/tools/gds_stats -p <pid> -l <verbosity(1-3)>

When I Instead called gds_stats from the 12.9 install things worked as expected.

Thanks!