Cannot import CUDAQ on an HPC

[talentkeychen]

Hi, maybe this question is more appropriate to ask our HPC admin but since he/she has not yet replied to me for some time, I'll have to ask here:

I’m very new to CUDA-Q. I tried to install CUDA-Q on our school’s HPC. Everything looks fine without any error or warning, but when I try to validate my installation, I cannot even import CUDA-Q and I got a “Segmentation Fault (core dumped)”. Below is what I do. The GPU is H100.

qsub -I -l select=1:mem=5GB:ncpus=1:ngpus=1 -q -P
module load cuda/12.2
module load miniforge/24.3
conda create -n cudaq python=3.10 -y
conda activate cudaq
pip install cudaq

Below is the information for nvcc:

nvcc: NVIDIA (R) Cuda compiler driver

Copyright (c) 2005-2023 NVIDIA Corporation

Built on Tue_Aug_15_22:02:13_PDT_2023

Cuda compilation tools, release 12.2, V12.2.140

Build cuda_12.2.r12.2/compiler.33191640_0

and here is the information for nvidia-smi:

Tue Jun 24 20:13:58 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.144.03             Driver Version: 550.144.03     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 80GB HBM3          On  |   00000000:9D:00.0 Off |                    0 |
| N/A   29C    P0             70W /  700W |       1MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

and this is what I have from python:

Python 3.10.18 | packaged by conda-forge | (main, Jun  4 2025, 14:45:41) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cudaq
Segmentation fault (core dumped)

Any idea on what happend? Thank you!

[bebora]

Hi @talentkeychen, your problem may be related to the one I reported in #2999. Can you run:
PYTHONFAULTHANDLER=1 python -c "import cudaq"
inside your environment and report here the traceback?

[talentkeychen]

sure. My HPC support replied that I should try python 3.11 but apparently it still has this segmentation fault. I pasted the error message here after executing PYTHONFAULTHANDLER=1 python -c "import cudaq":

Fatal Python error: Segmentation fault

Current thread 0x00001555551df740 (most recent call first):
  File "/home/users/astar/ares/chentian/.conda/envs/cudaq/lib/python3.11/site-packages/cupy/cuda/compiler.py", line 329 in _hash_hexdigest
  File "/home/users/astar/ares/chentian/.conda/envs/cudaq/lib/python3.11/site-packages/cupy/cuda/compiler.py", line 332 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1232 in _handle_fromlist
  File "/home/users/astar/ares/chentian/.conda/envs/cudaq/lib/python3.11/site-packages/cupy/cuda/__init__.py", line 9 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1126 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1241 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1232 in _handle_fromlist
  File "/home/users/astar/ares/chentian/.conda/envs/cudaq/lib/python3.11/site-packages/cupy/_core/__init__.py", line 3 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1232 in _handle_fromlist
  File "/home/users/astar/ares/chentian/.conda/envs/cudaq/lib/python3.11/site-packages/cupy/__init__.py", line 16 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1241 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1232 in _handle_fromlist
  File "/home/users/astar/ares/chentian/.conda/envs/cudaq/lib/python3.11/site-packages/cuquantum/bindings/__init__.py", line 5 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1232 in _handle_fromlist
  File "/home/users/astar/ares/chentian/.conda/envs/cudaq/lib/python3.11/site-packages/cuquantum/__init__.py", line 5 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "/home/users/astar/ares/chentian/.conda/envs/cudaq/lib/python3.11/site-packages/cudaq/dynamics/cudm_helpers.py", line 24 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "/home/users/astar/ares/chentian/.conda/envs/cudaq/lib/python3.11/site-packages/cudaq/dynamics/integrators/builtin_integrators.py", line 10 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "/home/users/astar/ares/chentian/.conda/envs/cudaq/lib/python3.11/site-packages/cudaq/dynamics/integrators/__init__.py", line 9 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "/home/users/astar/ares/chentian/.conda/envs/cudaq/lib/python3.11/site-packages/cudaq/__init__.py", line 98 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "<string>", line 1 in <module>

Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, cuquantum.bindings.cudensitymat, cuquantum.bindings._utils, cuquantum.bindings._internal.cudensitymat, cuquantum.bindings.cycudensitymat, cupy._core.core, cupy_backends.cuda._softlink, cupy_backends.cuda.api._runtime_enum, cupy_backends.cuda.api.runtime, cupy._util, cupy.cuda.device, fastrlock.rlock, cupy.cuda.memory_hook, cupy_backends.cuda.stream, cupy.cuda.graph, cupy.cuda.stream, cupy_backends.cuda.api._driver_enum, cupy_backends.cuda.api.driver, cupy.cuda.memory, cupy._core.internal, cupy._core._carray, cupy.cuda.texture, cupy.cuda.function, cupy_backends.cuda.libs.nvrtc (total: 25)
Segmentation fault (core dumped)

[bebora]

Thanks @talentkeychen, it is the same error that I encountered. I can suggest two workarounds:

1. Use a specific Python version while creating the Conda environment. In particular, the most recent versions you can choose are 3.10.15, 3.11.10 and 3.12.7 (as discovered in Cannot import CUDA-Q on newer Python patch versions under Conda #2999). In your case, you can replace conda create -n cudaq python=3.10 -y with conda create -n cudaq python=3.10.15 -y
2. Use miniconda instead of miniforge. Beware: the default channel used by miniconda requires a license and it is not free. Your HPC admins might not approve this workaround.

@sacpis do you believe anything can be done regarding this situation? The problem was independently reported twice. If finding the root cause is too complex, a warning for conda-forge users with recent Python versions might be enough.

[sacpis]

@bebora @talentkeychen Would you be able to update CUDA driver and check it again?

I have this CUDA version on my system.

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 565.57.01              Driver Version: 565.57.01      CUDA Version: 12.7     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX A6000               On  |   00000000:01:00.0 Off |                  Off |
| 30%   34C    P8             16W /  300W |     958MiB /  49140MiB |     49%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

[sacpis]

Have you tried cleaning the conda cache using conda clean --all?

[bebora]

Hi @sacpis, my laptop does not have a discrete GPU, but the crash occurs nonetheless.
Regarding the cache suggestion, the following still crashes on my laptop:

conda clean --all
conda create -n cf_py3_12_8 python=3.12.8 -c conda-forge
conda activate cf_py3_12_8
pip install cudaq
python -c "import cudaq; print(cudaq.__version__)"

I also tried the above commands inside a clean miniconda image on a DGX B200 system, but it crashes as well. Here the details:

$ sudo docker run --rm --gpus=1 -i -t continuumio/miniconda3 /bin/bash
(base) root@ef1b746f8789:/# nvidia-smi
Thu Jun 26 16:15:05 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.124.06             Driver Version: 570.124.06     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA B200                    On  |   00000000:1B:00.0 Off |                    0 |
| N/A   28C    P0            189W / 1000W |       1MiB / 183359MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

There is something wrong with hashlib when using the latest conda-forge Python versions and while importing CUDA-Q. I tried to debug it a few weeks ago but the offending function (_hash_hexdigest) does not crash when called from a simple file that only imports hashlib and sys.

[sacpis]

Thanks @bebora. I was able to repro it with the steps you provided. We will put a warning for conda-forge installation.

[talentkeychen]

Thanks @talentkeychen, it is the same error that I encountered. I can suggest two workarounds:

1. Use a specific Python version while creating the Conda environment. In particular, the most recent versions you can choose are 3.10.15, 3.11.10 and 3.12.7 (as discovered in Cannot import CUDA-Q on newer Python patch versions under Conda #2999). In your case, you can replace conda create -n cudaq python=3.10 -y with conda create -n cudaq python=3.10.15 -y
2. Use miniconda instead of miniforge. Beware: the default channel used by miniconda requires a license and it is not free. Your HPC admins might not approve this workaround.

@sacpis do you believe anything can be done regarding this situation? The problem was independently reported twice. If finding the root cause is too complex, a warning for conda-forge users with recent Python versions might be enough.

Thank you for your reply, and apologize for a slow response, as I was overwhelmed with other urgent duties in my institution over the last few weeks.

Our HPC team has replied saying that they actually followed the instructions on CUDA-Q's official documentation and successfully installed it. Following your suggestions here, I am checking with him which python version he used. Our HPC does not allow miniconda so it has to be from miniforge.

[mitchdz]

I’m taking a look into this. The issue appears to be that some of our .so files statically link against OpenSSL (libcrypto.a), and expose symbols globally in the process, these end up in the global symbol table due to how Python’s dynamic loader handles shared objects.

When Python’s _hashlib module later tries to load its own dynamically linked OpenSSL (libcrypto.so), symbol resolution gets clobbered leading to undefined behavior or a segfault.

This is shown by changing the import order:

# Safe: OpenSSL symbols from _hashlib are resolved first
$ python -c 'import hashlib, cudaq; print(hashlib.sha256(b"foo").hexdigest())'
2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae

# Crash: CUDA-Q's statically linked OpenSSL overrides before _hashlib loads
$ python -c 'import cudaq, hashlib; print(hashlib.sha256(b"foo").hexdigest())'
Segmentation fault (core dumped)

[mitchdz]

Hello, checking in with a update, apologies on the radio silence as I've been busy with getting the next release working.

I have imported cudaq using miniforge and a very basic smoketest is working.

(cudaq-env) bash-4.4# conda info | grep "base environment"
       base environment : /root/miniforge3  (writable)
(cudaq-env) bash-4.4# python3 --version
Python 3.13.5
(cudaq-env) bash-4.4# python -c 'import cudaq, hashlib; print(hashlib.sha256(b"foo").hexdigest())'
2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae

This is a dev build I have for the new python 3.13 wheels where I updated openssl to 3.5.1.

What seems to be happening here is that (as I alluded to above) CUDAQ pollutes the global namespace for openssl with the statically linked version we ship. Issues seem to occur when hashlib has a newer openssl than CUDAQ, thus updating it remedies this issue, at least for now.

This of course will be an issue later when hashlib updates again, so we will be playing wack-a-mole.

I'd like to remove the static linking if possible, but was not comfortable doing a change like this so close to a new release.

I am still testing this updated openssl, but I will try to get it in for this next release (0.12.0) and if so, this might enable miniforge for the people affected by this.