You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
SIGSEGV in cs_sles_solve_ccc_fv when CS_CUDA_ALLOC_DEVICE_UVM is not set (v9.1.0) #164
cs_sles_solve_ccc_fv allocates its extended solver buffers using CS_ALLOC_DEVICE (device-only memory). Those buffers are subsequently passed to the CPU-side convergence check, which dereferences them on the host and causes a SIGSEGV.
Setting CS_CUDA_ALLOC_DEVICE_UVM=1 before launching the solver works around the crash: it remaps cs_alloc_mode_device to CS_ALLOC_HOST_DEVICE_SHARED at initialisation time (see cs_base_cuda.cu:777), making the buffers accessible from both host and device.
Steps to reproduce
Build code_saturne v9.1.0 with CUDA support.
Run any case that exercises the CUDA sparse linear solver (cs_sles_solve_ccc_fv) without setting CS_CUDA_ALLOC_DEVICE_UVM.
Observe SIGSEGV during the convergence check after the first solver call.
Workaround
export CS_CUDA_ALLOC_DEVICE_UVM=1
Set this before invoking the solver. It prevents the crash by ensuring solver buffers are in unified/shared memory.
Expected behaviour
cs_sles_solve_ccc_fv should either allocate convergence-check buffers in host-accessible memory directly, or the convergence check should use a device-to-host copy rather than a direct host dereference.
Do you observe this in the master branch ? Since GPU support is still work in progress, we test mostly on the master branch, not that much on v9.1. A subset of our testsuite runs every night on a few iterations, without forcing CS_CUDA_ALLOC_DEVICE_UVM, and I recall fixing a similar issue a few months (possibly after the v9.1 branching), so I assume the bug concerns v9.1 but not the soon to be released 9.2 (near the end of this month). I am not sure (by the commit messages) which commit actually fixed this.
For your information :
The initial stages of GPU support involved explicit CPU/GPU memory host synchronization, but obviously, when we started to move more kernels to GPU (to avoid memory transfers as mush as obtain faster non-critical kernels), this was difficult to manage, as a same operator (gradient, solver) can be called in different code paths.
So we counted on UVM to help us manage this, and this worked quite well, especially after adding hints such as "read_mostly" (in May 2025, this almost doubled performance).
Since then, we have observed different behavior on different (theoretically similar) machines, where detailed profiling shows that using UVM can add some overhead in some cases. One one cluster, we observed shorter kernel run times, but longer intervals between kernels. Using kernel-only memory when UVM is not needed is faster, because the CUDA runtime does not need to do some mappings on CPU. Using a memory map (another option) is even better, and reduces the performance difference between UVM everywhere and UVM "only where absolutely needed", but this also increases GPU memory use.
For this reason, and to make sure we do not forget to convert some loops to GPU, our approach is to use device-only allocation as much as possible: if this approach is too aggressive, this leads to a memory access error (as in this issue), which id easy to debug, whereas overlooked loops can lead to memory transfer overheads which require very detailed profiling to spot. This does have the disadvantage of causing issues along the way, but they should be less of an issue in future version now that our automated tests include GPU cases.
So I will look to apply your patch/merge request soon for v9.1 after reviewing it. For the master branch, it should not be necessary (and we are currently working on running more multigrid steps on the GPU only, so later this year, I hope that device-only allocations will be fine for most multigrid paths.
You'll have to forgive me, I was getting all giddy about my first ever PR response (:tada:)
ahem
Do you observe this in the master branch?
I haven't seen it on master. I only hit it on v9.1.0 (sm_75, CUDA 13.1, channel-flow, CS_MATRIX_NATIVE).
I did have a dig through master to see how it stands now, though. On master, cs_sles_solve_ccc_fv lives in cs_sles_default.cpp, and the extended _vx / _rhs buffers are no longer allocated device-only unconditionally. amode is pulled from the matrix's own alloc mode and only drops to cs_alloc_mode_device inside a guard, with GPU dispatch switched on explicitly:
So in a non-shared build the buffers stay host-readable, which is the case that crashed on v9.1; in a shared build they go device-only deliberately, with GPU dispatch enabled for it. So the v9.1 crash path looks gone on master.
overlooked loops can lead to memory transfer overheads which require very detailed profiling to spot
Gotcha! I hadn't considered the full cost of UVM beyond the transfer itself, the host-side mappings and, from what you saw, the gaps that can open between kernels. Forcing device-only so a stray host read fails loudly, rather than quietly eating that overhead, makes sense as a default. Honestly, you've provided so much detail and I appreciate it.
Note
I've read the master path rather than run it, so treat anything about master as "from the code" rather than confirmed on a run.
Activity
YvanFournier commentedon Jun 7, 2026
Hello,
Do you observe this in the master branch ? Since GPU support is still work in progress, we test mostly on the master branch, not that much on v9.1. A subset of our testsuite runs every night on a few iterations, without forcing CS_CUDA_ALLOC_DEVICE_UVM, and I recall fixing a similar issue a few months (possibly after the v9.1 branching), so I assume the bug concerns v9.1 but not the soon to be released 9.2 (near the end of this month). I am not sure (by the commit messages) which commit actually fixed this.
For your information :
So I will look to apply your patch/merge request soon for v9.1 after reviewing it. For the master branch, it should not be necessary (and we are currently working on running more multigrid steps on the GPU only, so later this year, I hope that device-only allocations will be fine for most multigrid paths.
diskdog commentedon Jun 7, 2026
Hiya!
You'll have to forgive me, I was getting all giddy about my first ever PR response (:tada:)
ahem
I haven't seen it on master. I only hit it on v9.1.0 (sm_75, CUDA 13.1, channel-flow,
CS_MATRIX_NATIVE).I did have a dig through master to see how it stands now, though. On master,
cs_sles_solve_ccc_fvlives incs_sles_default.cpp, and the extended_vx/_rhsbuffers are no longer allocated device-only unconditionally.amodeis pulled from the matrix's own alloc mode and only drops tocs_alloc_mode_deviceinside a guard, with GPU dispatch switched on explicitly:code_saturne/src/alge/cs_sles_default.cpp
Lines 914 to 921 in 5056ef5
So in a non-shared build the buffers stay host-readable, which is the case that crashed on v9.1; in a shared build they go device-only deliberately, with GPU dispatch enabled for it. So the v9.1 crash path looks gone on master.
Gotcha! I hadn't considered the full cost of UVM beyond the transfer itself, the host-side mappings and, from what you saw, the gaps that can open between kernels. Forcing device-only so a stray host read fails loudly, rather than quietly eating that overhead, makes sense as a default. Honestly, you've provided so much detail and I appreciate it.
Note
I've read the master path rather than run it, so treat anything about master as "from the code" rather than confirmed on a run.
Thanku!!
Keith