BBNORM: Cannot run program “bgzip”

[BenPonBiobrain]

Description of the bug

I am running metatdenovo (nf-core/metatdenovo vv2.5.1-gd57e1db) with bbnorm using singularity (v3.6.4).
BBNORM fails with the following information in the "all_samples.bbnorm.log"

Set threads to 6
*********** Pass 1 **********
Settings:
threads: 6
k: 31
deterministic: true
toss error reads: false
passes: 1
bits per cell: 16
cells: 73.25B
hashes: 3
base min quality: 5
kmer min prob: 0.5
target depth: 400
min depth: 3
max depth: 500
min good kmers: 15
depth percentile: 64.8
ignore dupe kmers: true
fix spikes: false
Made hash table: hashes = 3 mem = 136.42 GB cells = 73.24B used = 50.244%
Estimated unique kmers: 17040776813
Table creation time: 11794.754 seconds.
java.io.IOException: Cannot run program “bgzip”: /scratch/SlurmTMP/smomw524.8777496/TEMPFILE_BBNORM_P1_R1_40d0ef1f161000ebd9da1ec6b00c71__802.5645338018305.fq.gz (No such file or directory)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1143)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1073)
at fileIO.ReadWrite.getOutputStreamFromProcess(ReadWrite.java:708)
at fileIO.ReadWrite.getBgzipStream(ReadWrite.java:628)
at fileIO.ReadWrite.getGZipOutputStream(ReadWrite.java:533)
at fileIO.ReadWrite.getOutputStream(ReadWrite.java:382)
at fileIO.ReadWrite.getOutputStream(ReadWrite.java:344)
at stream.ReadStreamWriter.(ReadStreamWriter.java:71)
at stream.ReadStreamByteWriter.(ReadStreamByteWriter.java:18)
at stream.ConcurrentGenericReadOutputStream.(ConcurrentGenericReadOutputStream.java:38)
at stream.ConcurrentReadOutputStream.getStream(ConcurrentReadOutputStream.java:71)
at stream.ConcurrentReadOutputStream.getStream(ConcurrentReadOutputStream.java:29)
at jgi.KmerNormalize.count(KmerNormalize.java:1496)
at jgi.KmerNormalize.runPass(KmerNormalize.java:1111)
at jgi.KmerNormalize.main(KmerNormalize.java:673)
Caused by: java.io.FileNotFoundException: /scratch/SlurmTMP/smomw524.8777496/TEMPFILE_BBNORM_P1_R1_40d0ef1f161000ebd9da1ec6b00c71__802.5645338018305.fq.gz (No such file or directory)
at java.base/java.io.FileOutputStream.open0(Native Method)
at java.base/java.io.FileOutputStream.open(FileOutputStream.java:293)
at java.base/java.io.FileOutputStream.(FileOutputStream.java:235)
at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:223)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1110)
... 14 more
Exception in thread “main” java.lang.AssertionError: Could not execute bgzip -c -f -l 4 -@ 6
at fileIO.ReadWrite.getOutputStreamFromProcess(ReadWrite.java:713)
at fileIO.ReadWrite.getBgzipStream(ReadWrite.java:628)
at fileIO.ReadWrite.getGZipOutputStream(ReadWrite.java:533)
at fileIO.ReadWrite.getOutputStream(ReadWrite.java:382)
at fileIO.ReadWrite.getOutputStream(ReadWrite.java:344)
at stream.ReadStreamWriter.(ReadStreamWriter.java:71)
at stream.ReadStreamByteWriter.(ReadStreamByteWriter.java:18)
at stream.ConcurrentGenericReadOutputStream.(ConcurrentGenericReadOutputStream.java:38)
at stream.ConcurrentReadOutputStream.getStream(ConcurrentReadOutputStream.java:71)
at stream.ConcurrentReadOutputStream.getStream(ConcurrentReadOutputStream.java:29)
at jgi.KmerNormalize.count(KmerNormalize.java:1496)
at jgi.KmerNormalize.runPass(KmerNormalize.java:1111)
at jgi.KmerNormalize.main(KmerNormalize.java:673)

Command used and terminal output

No response

Relevant files

No response

System information

No response

[erikrikarddaniel]

I think the key part of the error is this line:

Cannot run program “bgzip”: /scratch/SlurmTMP/smomw524.8777496/TEMPFILE_BBNORM_P1_R1_40d0ef1f161000ebd9da1ec6b00c71__802.5645338018305.fq.gz (No such file or directory)

At first my guess was that the bgzip program was missing from the container, but now I start to wonder if it's the input file that's missing. It can be missing when run in the container either by actually being missing, or by existing on a non-mounted file system. Do you use any parameter or config directive to make Nextflow use /scratch?

I think this thread: https://nfcore.slack.com/archives/CKDDD9JB0/p1688127045331519 might hold the solution, though I'm not sure how to set this up. Maybe your cluster administrator needs to configure Singularity to mount /scratch?

There's also this message: https://nfcore.slack.com/archives/CKDDD9JB0/p1675213873767469. It seems one can add mounts with singularity.runOptions in the nextflow.config in the pipeline execution directory. I'd try something like singularity.runOptions = -B /scratch:/scratch to mount the host /scratch file system as /scratch in the container.

[danilodileo]

There is something that we can do for this issue? or it is a singularity/container issue? if it so, we can close it