Problems with fastq-dump (Failed to call external services)

[ialbert]

Edited on Aug 2nd, 2022:

There is now an installer that downloads and installs the native client distributed by NCBI. I have written a bash shell installer for it. Run:

curl http://data.biostarhandbook.com/sratools.sh | bash

After running the above, close and reopen the terminal, then test the results with:

fastq-dump --stdout -X 2 SRR390728

I believe the above to be a solution to the problems people have been experiencing.

---- original message follows ---

When running the check you might see the following:

$ make vcf
mkdir -p reads
fastq-dump -F -X 10000 --split-files -O reads SRR1553425
Failed to call external services.
make: *** [Makefile:110: reads/SRR1553425_1.fastq] Error 64

This appears to be an error with how fastq-dump works with conda.

Basically NCBI is not willing to maintain this software as part of bioconda and volunteers do, but then NCBI keeps altering the software.

A temporary solution is:

Download latest ubuntu version of sratoolkit from ncbi @

http://ftp-trace.ncbi.nlm.nih.gov/sra/sdk/current/sratoolkit.current-ubuntu64.tar.gz

Unpacked and put in home/myusername/bin folder created for doctor.py and added the sra's bin to the path in .bashrc

export PATH=~/bin/sratoolkit.2.11.2-ubuntu64/bin:$PATH

I will attempt to fix this error in the near future.

[sunnysharma84]

Hi, the above solution does not work for me. I am still getting the following error . Please help:
fastq-dump -X 1 -Z SR1553591
2022-06-08T20:28:18 fastq-dump.2.11.0 sys: mbedtls_ssl_get_verify_result returned 0x8 ( !! The certificate is not correctly signed by the trusted CA )
Failed to call external services.

[ialbert]

I suspect that means it is still using the original fastq-dump

which fastq-dump 

will tell you which one gets used, try manually running the new one

~/bin/sratoolkit.2.11.2-ubuntu64/bin/fastq-dump

[cjg2131]

I have had the same issue. Additionally, when I activate bioinfo I get the error message:

xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun

I am assuming this is still using the older fastq-dump file? Manually running the new version was unsuccessful. It say 'No such file or directory'

[ialbert]

make sure to redo the MacOS install again especially when it comes to updating Xcode.

When new MacOS is released things mayneed to be redone.

Then, if a manual installation fails to run, it simply means the program was not placed in the correct folder.

[cjg2131]

Thanks for getting back to me. I tried again and re-installed Xcode but still no luck. After running doctor.py and running the failed command line, I get the following message:

$ fastq-dump -X 1 -Z SRR1553591
2022-06-20T21:14:52 fastq-dump.2.11.0 sys: mbedtls_ssl_get_verify_result returned 0x8 ( !! The certificate is not correctly signed by the trusted CA )
Failed to call external services.

[ialbert]

I found the following recommentation

ncbi/sra-tools#645

[ialbert]

As an alternative you can download the files separately via FTP

bio search  SRR2926782

it will print

[
    {
        "run_accession": "SRR2926782",
        "sample_accession": "SAMN04236697",
        "first_public": "2016-05-27",
        "country": "",
        "sample_alias": "GSM1925712",
        "fastq_bytes": "498649773",
        "read_count": "5344109",
        "library_name": "",
        "library_strategy": "OTHER",
        "library_source": "GENOMIC",
        "library_layout": "SINGLE",
        "instrument_platform": "ILLUMINA",
        "instrument_model": "Illumina HiSeq 2500",
        "study_title": "A Comprehensive Atlas of Arabidopsis Regulatory DNA [DAP-seq]",
        "fastq_ftp": "ftp.sra.ebi.ac.uk/vol1/fastq/SRR292/002/SRR2926782/SRR2926782.fastq.gz"
    }
]

then you can download by the URL:

wget ftp.sra.ebi.ac.uk/vol1/fastq/SRR292/002/SRR2926782/SRR2926782.fastq.gz

It is sad, maddening and tragicomical that NCBI is unable to get its act together to allow us to download data.

[jadab2]

I am having the same problem that was posted back on June 20. I need a bit more detail on how to fix this error. Where are downloaded files supposed to be stored so that there is no error? How is the newest version of the NCBI SRA toolkit supposed to be used? While I am very excited to review/enhance my bioinformatics skills- getting stuck on the initial steps of installing the correct software is somewhat demoralizing.

[ialbert]

alas NCBI does not want to support bioconda, bioconda refuses to directly package sratools etc - as you put right from the get go a newcomer can hit a wall

there is not much I can do, short of reinventing the entire installation, or at least that of fastq-dump,

I thought about out it a bit I am going to modify the installation makefile to also install the latest SRA tools in the correct location ... stay tuned

[ialbert]

I have written a shell script to automate the installation of sratools. Run it like so.

curl http://data.biostarhandbook.com/sratools.sh | bash

After running the above, close and reopen the terminal, then test the results with:

fastq-dump --stdout -X 2 SRR390728

[jadab2]

THANK-YOU! The script worked and doctor.py did not return any errors. I appreciate your responsiveness.

[ngupt123]

Hmm, running the script gives me this error. Not sure what to do. The weird question mark symbols appear to be spaces.

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   898  100   898    0     0   4353      0 --:--:-- --:--:-- --:--:--  4359
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 81.8M  100 81.8M    0     0  20.3M      0  0:00:04  0:00:04 --:--:-- 20.3M
/usr/bin/gzip: 1: �ELF����: not found
/usr/bin/gzip: 3: ���: not found
/usr/bin/gzip: 4: Syntax error: "(" unexpected
tar: Child returned status 2
tar: Error is not recoverable: exiting now
(bioinfo)```

[cjg2131]

Thank you once again. I really appreciate your help. I've downloaded the latest version of sratools. I suspect at this point I just have the wrong version in the directory that I am trying to access. I just don't have the skills necessary to put the correct file in the right place. For example in my home directory here is my printout: $ ls /Users/chrisgraham ASRemlRegister.txt Dropbox bin Applications Library faculty meet today.docx Books Movies miniconda-installer.sh CLC_Data Music miniconda3 CLC_References Pictures sratoolkit.3.0.0-mac64 DL Peas 2015.spv Public sratoolkit.tar.gz Desktop SRR2926782.fastq.gz work Documents Sites Downloads asreml.lic But when I type: which fastq-dump /Users/chrisgraham/miniconda3/envs/bioinfo/bin/fastq-dump (bioinfo) And this version looks like it might be 2.11.0 based on the following call: fastq-dump --stdout SRR390728 | head -n 8 2022-06-20T22:36:21 fastq-dump.2.11.0 sys: mbedtls_ssl_get_verify_result returned 0x8 ( !! The certificate is not correctly signed by the trusted CA ) Failed to call external services. (bioinfo)

…

On Mon, Jun 20, 2022 at 3:40 PM Istvan Albert ***@***.***> wrote: As an alternative you can download the files separately via FTP bio search SRR2926782 it will print [ { "run_accession": "SRR2926782", "sample_accession": "SAMN04236697", "first_public": "2016-05-27", "country": "", "sample_alias": "GSM1925712", "fastq_bytes": "498649773", "read_count": "5344109", "library_name": "", "library_strategy": "OTHER", "library_source": "GENOMIC", "library_layout": "SINGLE", "instrument_platform": "ILLUMINA", "instrument_model": "Illumina HiSeq 2500", "study_title": "A Comprehensive Atlas of Arabidopsis Regulatory DNA [DAP-seq]", "fastq_ftp": "ftp.sra.ebi.ac.uk/vol1/fastq/SRR292/002/SRR2926782/SRR2926782.fastq.gz" } ] then you can download by the URL: wget ftp.sra.ebi.ac.uk/vol1/fastq/SRR292/002/SRR2926782/SRR2926782.fastq.gz It is both sad and maddening that NCBI is unable to get its act together to allow us to download data. — Reply to this email directly, view it on GitHub <#200 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ARVCPG27LW6NDXL3EP3H37DVQDQLZANCNFSM5RLWQE4Q> . You are receiving this because you commented.Message ID: ***@***.*** com>

-- -Chris

[Mike-Guo-2021]

I have the same problem as you, don't know if you have found a solution yet?