DIA:PRELIMINARY_ANALYSIS cannot find file, DIANN on .d files

[josenimo]

Description of the bug

I am running bigbio/quantms with the current dev branch commit 9b4ea3a.
On the BIGBIO_QUANTMS:QUANTMS:DIA:PRELIMINARY_ANALYSIS step the job starts and ends immediately.
Based on the logs, I think it has to do with pathing, but it might also be my HPC setup.

any clues why would the container fail to find the file?
I tried with both -profile singularity and -profile apptainer

Here is the issue on the nextflow.log

Mar-02 16:50:25.815 [TaskFinalizer-6] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=BIGBIO_QUANTMS:QUANTMS:DIA:PRELIMINARY_ANALYSIS (Olive_20241129_JoN_Evo2_80SPD_P26E17_991_C9_S2-A7_1_8399); work-dir=/data/cephfs-1/home/users/jnimoca_m/scratch/16/5c1cd56768979390291ca2ec153002
  error [nextflow.exception.ProcessFailedException]: Process `BIGBIO_QUANTMS:QUANTMS:DIA:PRELIMINARY_ANALYSIS (Olive_20241129_JoN_Evo2_80SPD_P26E17_991_C9_S2-A7_1_8399)` terminated with an error exit status (141)
Mar-02 16:50:25.888 [TaskFinalizer-6] ERROR nextflow.processor.TaskProcessor - Error executing process > 'BIGBIO_QUANTMS:QUANTMS:DIA:PRELIMINARY_ANALYSIS (Olive_20241129_JoN_Evo2_80SPD_P26E17_991_C9_S2-A7_1_8399)'

Caused by:
  Process `BIGBIO_QUANTMS:QUANTMS:DIA:PRELIMINARY_ANALYSIS (Olive_20241129_JoN_Evo2_80SPD_P26E17_991_C9_S2-A7_1_8399)` terminated with an error exit status (141)


Command executed:

  # Precursor Tolerance value was: 15.0
  # Fragment Tolerance value was: 15.0
  # Precursor Tolerance unit was: ppm
  # Fragment Tolerance unit was: ppm
  
  # Final mass accuracy is ''
  
  # Extract --var-mod and --fixed-mod flags from diann_config.cfg (DIA-NN best practice)
  mod_flags=$(cat diann_config.cfg | grep -oP '(--var-mod\s+\S+|--fixed-mod\s+\S+)' | tr '\n' ' ')
  
  diann   --lib lib.predicted.speclib \
          --f Olive_20241129_JoN_Evo2_80SPD_P26E17_991_C9_S2-A7_1_8399.d \
          --threads 24 \
          --verbose 3 \
           \
          --temp ./ \
           \
          --quick-mass-acc \
          --min-corr 2 --corr-diff 1 --time-corr-only \
          ${mod_flags} \
  
  
  cp report.log.txt Olive_20241129_JoN_Evo2_80SPD_P26E17_991_C9_S2-A7_1_8399_diann.log
  
  cat <<-END_VERSIONS > versions.yml
  "BIGBIO_QUANTMS:QUANTMS:DIA:PRELIMINARY_ANALYSIS":
      DIA-NN: $(diann 2>&1 | grep "DIA-NN" | grep -oP "\d+\.\d+(\.\w+)*(\.[\d]+)?")
  END_VERSIONS

Command exit status:
  141

Command output:
  DIA-NN 1.8.1 (Data-Independent Acquisition by Neural Networks)
  Compiled on Apr 15 2022 08:45:18
  Current date and time: Mon Mar  2 16:49:51 2026
  Logical CPU cores: 32
  Thread number set to 24
  A fast algorithm will be used to select the MS2 mass accuracy setting
  Only peaks with correlation sum exceeding 2 will be considered
  Peaks with correlation sum below 1 from maximum will not be considered
  A single score will be used until RT alignment to save memory; this can potentially lead to slower search
  Modification UniMod:4 with mass delta 57.0215 at C will be considered as fixed
  Modification UniMod:1 with mass delta 42.0106 at n will be considered as variable
  Modification UniMod:35 with mass delta 15.9949 at M will be considered as variable
  DIA-NN will optimise the mass accuracy automatically using the first run in the experiment. This is useful primarily for quick initial analyses, when it is not yet known which mass accuracy setting works best for a particular acquisition scheme.
  
  1 files will be processed
  [0:00] Loading spectral library lib.predicted.speclib
  [0:09] Library annotated with sequence database(s): UP000005640_9606.fasta
  [0:09] Gene names missing for some isoforms
  [0:09] Library contains 20216 proteins, and 19998 genes
  [0:16] Spectral library loaded: 20216 protein isoforms, 24774 protein groups and 8380395 precursors in 3132142 elution groups.
  [0:16] Initialising library
  
  [0:27] File #1/1
  [0:27] Loading run Olive_20241129_JoN_Evo2_80SPD_P26E17_991_C9_S2-A7_1_8399.d
  For most diaPASEF datasets it is better to manually fix both the MS1 and MS2 mass accuracies to values in the range 10-15 ppm.

Command error:
  stat lib: No such file or directory
  stat lib: No such file or directory
  stat lib: No such file or directory
  stat lib: No such file or directory
  stat lib: No such file or directory
  stat lib: No such file or directory
  stat lib: No such file or directory
  stat lib: No such file or directory
  stat lib: No such file or directory
  stat lib: No such file or directory
  stat lib: No such file or directory
  stat lib: No such file or directory
  stat lib: No such file or directory
  stat lib: No such file or directory
  stat lib: No such file or directory
  stat lib: No such file or directory
  stat lib: No such file or directory
  stat lib: No such file or directory

Work dir:
  /data/cephfs-1/home/users/jnimoca_m/scratch/16/5c1cd56768979390291ca2ec153002

Container:
  /data/cephfs-1/home/users/jnimoca_m/scratch/singularity/docker.io-biocontainers-diann-v1.8.1_cv1.img

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

Command used and terminal output

#!/bin/bash
#SBATCH --job-name=nextflow_head
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=4G
#SBATCH --time=24:00:00
#SBATCH --output=nf_out.%j.log
#SBATCH --error=nf_err.%j.err

eval "$(conda shell.bash hook)"     # This exposes conda hook to node
conda activate mcmicro              # This activates environment in node

# Run the pipeline
nextflow run bigbio/quantms \
    -r dev \
    -profile apptainer \
    -process.executor slurm \
    --input '/data/cephfs-1/home/users/jnimoca_m/work/quantms_test/HPC_10_samples.sdrf.tsv' \
    --database '/data/cephfs-1/home/users/jnimoca_m/work/quantms_test/FASTA/UP000005640_9606.fasta' \
    --outdir '/data/cephfs-1/home/users/jnimoca_m/work/quantms_test/output/' \
    -resume

Relevant files

logs.zip

❯ tree logs -a
logs
├── .DS_Store
├── .nextflow.log
├── 6b_2c23f9
│   ├── .command.begin
│   ├── .command.err
│   ├── .command.log
│   ├── .command.out
│   ├── .command.run
│   ├── .command.sh
│   ├── .command.trace
│   └── .exitcode
├── execution_report_2026-03-02_15-25-01.html
├── execution_timeline_2026-03-02_15-25-01.html
├── execution_trace_2026-03-02_15-25-01.txt
├── HPC_10_samples.sdrf.tsv
├── nf_err.23638346.err
├── nf_out.23638346.log
├── params_2026-03-02_15-25-25.json
└── pipeline_dag_2026-03-02_15-25-01.html

System information

No response

[josenimo]

I think my assessment was wrong, some jobs did start, but something killed them all.
Looking into it.

[josenimo]

So, here is my update:

It seems from my 10 files, one ran into an issue at the PRELIMINARY_ANALYSIS step
nextflow retried with double the resources and that did not work.

I am assuming that for some reason that one task fails to find the in silico library, why would this occur with a single job, and not the others...

here is the command.log:

......
1 files will be processed
[0:00] Loading spectral library lib.predicted.speclib
[0:08] Library annotated with sequence database(s): UP000005640_9606.fasta
[0:08] Gene names missing for some isoforms
[0:08] Library contains 20216 proteins, and 19998 genes
[0:13] Spectral library loaded: 20216 protein isoforms, 24774 protein groups and 8380395 precursors in 3132142 elution groups.
[0:13] Initialising library

[0:18] File #1/1
[0:18] Loading run Olive_20241129_JoN_Evo2_80SPD_P26E17_991_C9_S2-A7_1_8399.d
For most diaPASEF datasets it is better to manually fix both the MS1 and MS2 mass accuracies to values in the range 10-15 ppm.
stat lib: No such file or directory
stat lib: No such file or directory
stat lib: No such file or directory
stat lib: No such file or directory
stat lib: No such file or directory
stat lib: No such file or directory
stat lib: No such file or directory
stat lib: No such file or directory

for a good sample the command.log looks like this:

......
1 files will be processed
[0:00] Loading spectral library lib.predicted.speclib
[0:08] Library annotated with sequence database(s): UP000005640_9606.fasta
[0:08] Gene names missing for some isoforms
[0:08] Library contains 20216 proteins, and 19998 genes
[0:12] Spectral library loaded: 20216 protein isoforms, 24774 protein groups and 8380395 precursors in 3132142 elution groups.
[0:12] Initialising library

[0:17] File #1/1
[0:17] Loading run Olive_20241129_JoN_Evo2_80SPD_P26E17_991_C3_S2-A1_1_8393.d
For most diaPASEF datasets it is better to manually fix both the MS1 and MS2 mass accuracies to values in the range 10-15 ppm.
[1:01] Detected MS/MS range: 99.9898 - 1700
[1:03] Run loaded
[2026-03-02T16:50:26.081] error: *** JOB 23638457 ON hpc-cpu-134 CANCELLED AT 2026-03-02T16:50:26 DUE to SIGNAL Terminated ***

[ypriverol]

Can you share the .nextflow.log? @josenimo

[josenimo]

here
nextflow copy.log.txt

However, I think I found the issue, which is not related to quantms.
It seems that the copying one file was interrupted, and led to a file corruption.... 😠

Now that I recopied the problematic file again, it seems to be working 👍

[josenimo]

Sorry for the wall of text and bug noise.
I guess anybody who has the same issue should check their files for a .filepart file ....

[ypriverol]

is it working or not? We can add a step for file consistency check, if is a file corruption issue.

[josenimo]

It is working :)
I think file check is a nice add-on, but not necessary.

Just to be thorough this is how it looked:

Bad file directory had this :

-rw-r--r--  1 jnimoca_m hpc-ag-coscia  2308280832 Feb 23 17:34 analysis.tdf_bin.filepart

Good file directory had this symlink (it was lacking on the bad directory):

lrwxrwxrwx  1 jnimoca_m hpc-ag-coscia         126 Feb 24 17:07 Olive_20241129_JoN_Evo2_80SPD_P26E17_991_C3_S2-A1_1_8393.d -> /data/cephfs-1/work/groups/coscia/users/jnimoca_m/quantms_test/raws/Olive_20241129_JoN_Evo2_80SPD_P26E17_991_C3_S2-A1_1_8393.d

as well as the complete file:
-rw-r--r--  1 jnimoca_m hpc-ag-coscia  3497877504 Nov 29  2024 analysis.tdf_bin