Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fetus reference fails to build due to a missing file #16

Open
volkansevim opened this issue Mar 24, 2023 · 0 comments
Open

Fetus reference fails to build due to a missing file #16

volkansevim opened this issue Mar 24, 2023 · 0 comments

Comments

@volkansevim
Copy link

Snakelike fails on rule download_asp because this file on AWS is missing: 'https://md-datasets-cache-zipfiles-prod.s3.eu-west-1.amazonaws.com/mbvhhf8m62-2.zip'

Below is the complete log.

The flag 'directory' used in rule all is only valid for outputs, not inputs.
Building DAG of jobs...
Pulling singularity image docker://satijalab/azimuth-references:vitessce.
Pulling singularity image docker://satijalab/azimuth-references:latest.
Using shell: /usr/bin/bash
Provided cores: 8
Rules claiming more threads will be scaled down.
Job stats:
job                               count    min threads    max threads
------------------------------  -------  -------------  -------------
all                                   1              1              1
download_BBI_individual_organs       14              1              1
download_BBI_metadata                 1              1              1
download_BBI_subsampled               1              1              1
download_asp                          1              1              1
download_enge                         1              1              1
download_lind                         1              1              1
export_zarr                           1              1              1
make_BBI_query                        1              1              1
make_BBI_reference                    1              1              1
preprocess_BBI_reference              1              1              1
setup_asp                             1              1              1
setup_enge                            1              1              1
setup_lind                            1              1              1
total                                27              1              1

Select jobs to execute...

[Fri Mar 17 18:53:04 2023]
rule download_BBI_individual_organs:
    output: data/Stomach_gene_count.RDS
    jobid: 18
    reason: Missing output files: data/Stomach_gene_count.RDS
    wildcards: sample=Stomach
    resources: tmpdir=/tmp

Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg

[Fri Mar 17 18:53:04 2023]
rule download_BBI_individual_organs:
    output: data/Liver_gene_count.RDS
    jobid: 11
    reason: Missing output files: data/Liver_gene_count.RDS
    wildcards: sample=Liver
    resources: tmpdir=/tmp

Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg

[Fri Mar 17 18:53:04 2023]
rule download_BBI_subsampled:
    output: data/gene_count_sampled.RDS
    jobid: 3
    reason: Missing output files: data/gene_count_sampled.RDS
    resources: tmpdir=/tmp

Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg

[Fri Mar 17 18:53:04 2023]
rule download_BBI_individual_organs:
    output: data/Thymus_gene_count.RDS
    jobid: 19
    reason: Missing output files: data/Thymus_gene_count.RDS
    wildcards: sample=Thymus
    resources: tmpdir=/tmp

Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg

[Fri Mar 17 18:53:04 2023]
rule download_BBI_metadata:
    output: data/df_cell.RDS, data/df_gene.RDS
    jobid: 4
    reason: Missing output files: data/df_gene.RDS, data/df_cell.RDS
    resources: tmpdir=/tmp

Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg

[Fri Mar 17 18:53:04 2023]
rule download_lind:
    output: data/lind_data/counts.tsv
    jobid: 23
    reason: Missing output files: data/lind_data/counts.tsv
    resources: tmpdir=/tmp

Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg

[Fri Mar 17 18:53:04 2023]
rule download_enge:
    output: data/enge_data, data/enge_data/enge.tar
    jobid: 25
    reason: Missing output files: data/enge_data/enge.tar
    resources: tmpdir=/tmp

Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg

[Fri Mar 17 18:53:04 2023]
rule download_BBI_individual_organs:
    output: data/Heart_gene_count.RDS
    jobid: 10
    reason: Missing output files: data/Heart_gene_count.RDS
    wildcards: sample=Heart
    resources: tmpdir=/tmp

Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg
[Fri Mar 17 18:53:06 2023]
Finished job 18.
1 of 27 steps (4%) done
Select jobs to execute...

[Fri Mar 17 18:53:06 2023]
rule download_BBI_individual_organs:
    output: data/Lung_gene_count.RDS
    jobid: 12
    reason: Missing output files: data/Lung_gene_count.RDS
    wildcards: sample=Lung
    resources: tmpdir=/tmp

Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg
[Fri Mar 17 18:53:08 2023]
Finished job 19.
2 of 27 steps (7%) done
Select jobs to execute...

[Fri Mar 17 18:53:08 2023]
rule download_BBI_individual_organs:
    output: data/Pancreas_gene_count.RDS
    jobid: 14
    reason: Missing output files: data/Pancreas_gene_count.RDS
    wildcards: sample=Pancreas
    resources: tmpdir=/tmp

Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg
[Fri Mar 17 18:53:11 2023]
Finished job 4.
3 of 27 steps (11%) done
Select jobs to execute...

[Fri Mar 17 18:53:11 2023]
rule download_BBI_individual_organs:
    output: data/Cerebrum_gene_count.RDS
    jobid: 8
    reason: Missing output files: data/Cerebrum_gene_count.RDS
    wildcards: sample=Cerebrum
    resources: tmpdir=/tmp

Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg
[Fri Mar 17 18:53:21 2023]
Finished job 12.
4 of 27 steps (15%) done
Select jobs to execute...

[Fri Mar 17 18:53:21 2023]
rule download_BBI_individual_organs:
    output: data/Adrenal_gene_count.RDS
    jobid: 6
    reason: Missing output files: data/Adrenal_gene_count.RDS
    wildcards: sample=Adrenal
    resources: tmpdir=/tmp

Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg
[Fri Mar 17 18:53:30 2023]
Finished job 11.
5 of 27 steps (19%) done
Select jobs to execute...

[Fri Mar 17 18:53:30 2023]
rule download_asp:
    output: data/asp_data/counts_filtered.tsv, data/asp_data/md_filtered.tsv
    jobid: 21
    reason: Missing output files: data/asp_data/md_filtered.tsv, data/asp_data/counts_filtered.tsv
    resources: tmpdir=/tmp

Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg
[Fri Mar 17 18:53:31 2023]
Error in rule download_asp:
    jobid: 21
    output: data/asp_data/counts_filtered.tsv, data/asp_data/md_filtered.tsv
    shell:
        
        mkdir -p data/asp_data
        cd data/asp_data
        wget 'https://md-datasets-cache-zipfiles-prod.s3.eu-west-1.amazonaws.com/mbvhhf8m62-2.zip'
        unzip mbvhhf8m62-2.zip
        unzip Filtered/'Developmental_heart_filtered_scRNA-seq_and_meta_data.zip'
        gunzip share_files/*
        mv share_files/all_cells_count_matrix_filtered.tsv counts_filtered.tsv
        mv share_files/all_cells_meta_data_filtered.tsv md_filtered.tsv
        echo "ASP data downloaded on: $(date)" > ../../logs/download_asp.log
        
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

[Fri Mar 17 18:53:34 2023]
Finished job 10.
6 of 27 steps (22%) done
[Fri Mar 17 18:53:36 2023]
Finished job 3.
7 of 27 steps (26%) done
[Fri Mar 17 18:53:37 2023]
Finished job 14.
8 of 27 steps (30%) done
[Fri Mar 17 18:53:52 2023]
Finished job 6.
9 of 27 steps (33%) done
[Fri Mar 17 18:55:04 2023]
Finished job 8.
10 of 27 steps (37%) done
[Fri Mar 17 18:55:09 2023]
Finished job 23.
11 of 27 steps (41%) done
[Fri Mar 17 18:55:59 2023]
Finished job 25.
12 of 27 steps (44%) done
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2023-03-17T184850.349820.snakemake.log
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@volkansevim