Indexlr: spaces are not supported in file path

[mxwang66]

Hello devs,

I ran into this error running Indexlr in python on a linux x64 system. Below is the code to replicate this error:

from btllib import Indexlr, IndexlrFlag

assembly_path = 'genomes/Salmonella bongori/ncbi_dataset/data/GCF_000252995.1/genomic.fna'
kmerlen = 21
windowsize = 200
with Indexlr(assembly_path, kmerlen, windowsize, IndexlrFlag.LONG_MODE, 1) as assembly:
    for record in assembly:
        for m in record.minimizers:
            pass

and the error message:

cat: genomes/Salmonella: No such file or directory
cat: bongori/ncbi_dataset/data/GCF_000252995.1/genomic.fna: No such file or directory
[2024-11-05 01:39:10][WARNING] genomes/Salmonella bongori/ncbi_dataset/data/GCF_000252995.1/genomic.fna is empty.
[2024-11-05 01:39:10][ERROR] A helper process has finished unsuccessfully:
PID: 2313645
Outcome: exited with status 1
[2024-11-05 01:39:10][ERROR] Process pipeline: Communication failure.
[2024-11-05 01:39:10][ERROR] Process pipeline: Spawner process failed.
Segmentation fault (core dumped)

Got the same error when using the indexlr command:

indexlr -k 21 -w 200 --long -o test.out "genomes/Salmonella bongori/ncbi_dataset/data/GCF_000252995.1/genomic.fna"

Both run was successful if the space in the file path is replaced.

Thank you,
Michael

[lcoombe]

Hi Michael,

Thanks for reaching out!

I'm not really surprised that you would see unexpected behaviour when trying to use files with spaces in them - generally, the convention is to avoid using spaces in file/directory names (I found some good documentation of conventions here: https://datamanagement.hms.harvard.edu/plan-design/file-naming-conventions). There would be a number of issues possible there, including that indexlr can take multiple files, and the spaces would make it complicated to distinguish different files vs. a single file with the space(s).

Because of all that, I'd highly recommend removing spaces in your file names. If that isn't a good option (I understand sometimes changing file names after the fact can be messy), then you could make a soft-link to that problem file, and specify that to indexlr instead. Ex:

(btl) [lcoombe@hpce706 tmp]$ ls my\ test/
tmp.fa
(btl) [lcoombe@hpce706 tmp]$ ls -lah tmp1.fa 
lrwxrwxrwx 1 lcoombe btl 14 Nov  5 08:52 tmp1.fa -> my test/tmp.fa
(btl) [lcoombe@hpce706 tmp]$ indexlr -k24 -w5 tmp1.fa 
NC_012920.1_1-50	16371557189216790589 3754088885574348785 9549692006826280120 11376364575436671136 17106421449899264663 2564125396831936452 6856120600046910211 636075098808042691

Thank you for your interest in btllib!
Lauren

[mxwang66]

Thanks for your response Lauren! Yeah the best practice would be having no spaces. But it would nice to make it supported, since python itself supports file path with spaces. But I understand that you might need to spend time on other features. Thanks again for maintaining btllib!

Michael

[vlad0x00]

@lcoombe
I think the issue can be resolved by surrounding path with quotes in https://github.com/bcgsc/btllib/blob/master/src/btllib/data_stream.cpp#L242-L299

At the start of the function, you can do const auto quoted_path = '"' + path + '"'; and then replace all usages of path in that function with quoted_path. Probably also worth checking that the path is not already quoted by checking if the first and last characters are quotes or apostrophes.

[lcoombe]

Thanks @vlad0x00!

That's really helpful - we can take a look at making that tweak (@parham-k or @jwcodee?) (We're a bit short-staffed, and I will be going on leave soon, so might not happen quickly :) )

[parham-k]

I can take care of it.

[lcoombe]

I can take care of it.

Thanks so much Parham!