Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hidden analyses not found when uploading to scout #502

Open
diitaz93 opened this issue Nov 6, 2024 · 3 comments
Open

Hidden analyses not found when uploading to scout #502

diitaz93 opened this issue Nov 6, 2024 · 3 comments
Assignees

Comments

@diitaz93
Copy link
Contributor

diitaz93 commented Nov 6, 2024

Description

Originates from https://github.com/Clinical-Genomics/bug-reports/issues/15
Development case petmako tried to be uploaded using cg upload -r -c petmako but got the error:

TrailblazerAnalysisNotFound: No completed analysis found for case petmako

Trailblazer (Cigrid) showed that the analysis was there.

As the analysis was for development (cust000), it was by default hidden in Trailblazer. When the analysis was manually set to "unhidden" the command worked as expected.

Steps to reproduce

Using taxprofiler development case petmako, run:

cg upload -r -c petmako

The same bug is expected for any development case

Expected behaviour

That development analyses can be uploaded without manually setting them as "unhidden" in trailblazer

Additional context

  • There is a workaround -> This bug is not blocking anyone's job
  • This is only for development cases -> No production cases are affected
@diitaz93 diitaz93 added the Bug label Nov 6, 2024
@Vince-janv
Copy link
Contributor

@diitaz93 assigned to reproduce the bug

@RasmusBurge-CG
Copy link

RasmusBurge-CG commented Nov 13, 2024

In TB: on case petmako perform action Mark as hidden.

Then running command cg upload -r -c petmako:

Output is:

Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
----------------- UPLOAD -----------------
Case petmako exists in Status DB
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
--------------- DELIVERY REPORT ---------------
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Using analysis started at: 2024-10-29 12:01:13
Delivery report already in housekeeper: /home/proj/production/housekeeper-bundles/petmako/2024-10-29/delivery-report.html
Setting log dir to: /home/proj/production/rsync/petmako_241113_09_38_50_010410
Creating folder: /home/proj/production/rsync/petmako_241113_09_38_50_010410
Submit sbatch /home/proj/production/rsync/petmako_241113_09_38_50_010410/petmako_rsync.sh
Running command sbatch /home/proj/production/rsync/petmako_241113_09_38_50_010410/petmako_rsync.sh
Submitted batch job 7600292
Writing slurm jobs to /home/proj/production/rsync/petmako_241113_09_38_50_010410/slurm_job_ids.yaml
Traceback (most recent call last):
  File "/home/proj/production/bin/miniconda3/envs/P_cg/bin/cg", line 8, in <module>
    sys.exit(base())
             ^^^^^^
  File "/home/proj/production/bin/miniconda3/envs/P_cg/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/proj/production/bin/miniconda3/envs/P_cg/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/proj/production/bin/miniconda3/envs/P_cg/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/proj/production/bin/miniconda3/envs/P_cg/lib/python3.11/site-packages/click/core.py", line 1666, in invoke
    rv = super().invoke(ctx)
         ^^^^^^^^^^^^^^^^^^^
  File "/home/proj/production/bin/miniconda3/envs/P_cg/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/proj/production/bin/miniconda3/envs/P_cg/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/proj/production/bin/miniconda3/envs/P_cg/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/proj/production/bin/miniconda3/envs/P_cg/lib/python3.11/site-packages/cg/cli/upload/base.py", line 99, in upload
    upload_api.upload(ctx=context, case=case, restart=restart)
  File "/home/proj/production/bin/miniconda3/envs/P_cg/lib/python3.11/site-packages/cg/meta/upload/nf_analysis.py", line 44, in upload
    self.upload_files_to_customer_inbox(case)
  File "/home/proj/production/bin/miniconda3/envs/P_cg/lib/python3.11/site-packages/cg/meta/upload/error_handling.py", line 14, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/proj/production/bin/miniconda3/envs/P_cg/lib/python3.11/site-packages/cg/meta/upload/upload_api.py", line 103, in upload_files_to_customer_inbox
    delivery_service.deliver_files_for_case(
  File "/home/proj/production/bin/miniconda3/envs/P_cg/lib/python3.11/site-packages/cg/services/deliver_files/deliver_files_service/error_handling.py", line 18, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/proj/production/bin/miniconda3/envs/P_cg/lib/python3.11/site-packages/cg/services/deliver_files/deliver_files_service/deliver_files_service.py", line 82, in deliver_files_for_case
    self._add_trailblazer_tracking(case=case, job_id=job_id, dry_run=dry_run)
  File "/home/proj/production/bin/miniconda3/envs/P_cg/lib/python3.11/site-packages/cg/services/deliver_files/deliver_files_service/deliver_files_service.py", line 148, in _add_trailblazer_tracking
    self.analysis_service.add_upload_job(slurm_id=job_id, case_id=case.internal_id)
  File "/home/proj/production/bin/miniconda3/envs/P_cg/lib/python3.11/site-packages/cg/services/analysis_service/analysis_service.py", line 17, in add_upload_job
    analysis: TrailblazerAnalysis = self.analysis_client.get_latest_completed_analysis(case_id)
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/proj/production/bin/miniconda3/envs/P_cg/lib/python3.11/site-packages/cg/apps/tb/api.py", line 82, in get_latest_completed_analysis
    raise TrailblazerAnalysisNotFound(f"No completed analysis found for case {case_id}")
cg.exc.TrailblazerAnalysisNotFound: No completed analysis found for case petmako

In TB: on case petmako perform action Mark as unhidden

Then: command cg upload -r -c petmako.

Output:

Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
----------------- UPLOAD -----------------
Case petmako exists in Status DB
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
--------------- DELIVERY REPORT ---------------
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Called undefined __pydantic_serializer__ on HousekeeperAPI, please wrap
Called undefined __dataclass_fields__ on HousekeeperAPI, please wrap
Using analysis started at: 2024-10-29 12:01:13
Delivery report already in housekeeper: /home/proj/production/housekeeper-bundles/petmako/2024-10-29/delivery-report.html
/home/proj/production/customers/cust000/inbox/1234/kraken2_k2_pluspf_combined_reports.txt already exists. Overwriting with /home/proj/production/housekeeper-bundles/petmako/2024-10-29/kraken2_k2_pluspf_combined_reports.txt
/home/proj/production/customers/cust000/inbox/1234/kraken2_k2_pluspf.html already exists. Overwriting with /home/proj/production/housekeeper-bundles/petmako/2024-10-29/kraken2_k2_pluspf.html
/home/proj/production/customers/cust000/inbox/1234/centrifuge_p_compressed+h+v.html already exists. Overwriting with /home/proj/production/housekeeper-bundles/petmako/2024-10-29/centrifuge_p_compressed+h+v.html
/home/proj/production/customers/cust000/inbox/1234/bracken_bracken_combined_reports.txt already exists. Overwriting with /home/proj/production/housekeeper-bundles/petmako/2024-10-29/bracken_bracken_combined_reports.txt
/home/proj/production/customers/cust000/inbox/1234/centrifuge_p_compressed+h+v_combined_reports.txt already exists. Overwriting with /home/proj/production/housekeeper-bundles/petmako/2024-10-29/centrifuge_p_compressed+h+v_combined_reports.txt
/home/proj/production/customers/cust000/inbox/1234/multiqc_report.html already exists. Overwriting with /home/proj/production/housekeeper-bundles/petmako/2024-10-29/multiqc_report.html
/home/proj/production/customers/cust000/inbox/1234/nf_core_pipeline_software_mqc_versions.yml already exists. Overwriting with /home/proj/production/housekeeper-bundles/petmako/2024-10-29/nf_core_pipeline_software_mqc_versions.yml
/home/proj/production/customers/cust000/inbox/1234/multiqc_data.json already exists. Overwriting with /home/proj/production/housekeeper-bundles/petmako/2024-10-29/multiqc_data.json
/home/proj/production/customers/cust000/inbox/1234/delivery-report.html already exists. Overwriting with /home/proj/production/housekeeper-bundles/petmako/2024-10-29/delivery-report.html
/home/proj/production/customers/cust000/inbox/1234/IF-04-25_pe_k2_pluspf.kraken2.kraken2.report.txt already exists. Overwriting with /home/proj/production/housekeeper-bundles/petmako/2024-10-29/IF-04-25_pe_k2_pluspf.kraken2.kraken2.report.txt
/home/proj/production/customers/cust000/inbox/1234/IF-04-25_pe_bracken.bracken.tsv already exists. Overwriting with /home/proj/production/housekeeper-bundles/petmako/2024-10-29/IF-04-25_pe_bracken.bracken.tsv
/home/proj/production/customers/cust000/inbox/1234/IF-04-25_pe_p_compressed+h+v.centrifuge.txt already exists. Overwriting with /home/proj/production/housekeeper-bundles/petmako/2024-10-29/IF-04-25_pe_p_compressed+h+v.centrifuge.txt
Setting log dir to: /home/proj/production/rsync/petmako_241113_09_44_46_411326
Creating folder: /home/proj/production/rsync/petmako_241113_09_44_46_411326
Submit sbatch /home/proj/production/rsync/petmako_241113_09_44_46_411326/petmako_rsync.sh
Running command sbatch /home/proj/production/rsync/petmako_241113_09_44_46_411326/petmako_rsync.sh
Submitted batch job 7600294
Writing slurm jobs to /home/proj/production/rsync/petmako_241113_09_44_46_411326/slurm_job_ids.yaml
Transfer of case petmako started with SLURM job id 7600294
Upload of case petmako was successful. Setting uploaded at to 2024-11-13 09:44:48.334048
petmako - uploaded at set to 2024-11-13 09:44:48
petmako analysis has been successfully uploaded

@diitaz93
Copy link
Contributor Author

UPDATE:

The request sent from cg to Trailblazer is based on this:

f"analyses?case_id={case_id}&status[]={AnalysisStatus.COMPLETED}&limit=1"

In Trailblazer, the response is parsed into this model:

class AnalysesRequest(BaseModel):
    workflow: str = ""
    search: str | None = None
    page_size: int = Field(alias="pageSize", default=250)
    page: int = 1
    sort_field: AnalysisSortField = Field(alias="sortField", default=AnalysisSortField.STARTED_AT)
    sort_order: SortOrder = Field(alias="sortOrder", default=SortOrder.DESC)
    status: list[TrailblazerStatus] = []
    priority: list[TrailblazerPriority] = []
    type: list[TrailblazerTypes] = []
    has_comment: bool | None = Field(alias="hasComment", default=None)
    order_id: int | None = Field(alias="orderId", default=None)
    case_id: str | None = None
    delivered: bool | None = None
    include_hidden: bool | None = Field(alias="includeHidden", default=None)

This request is used to parse the TB database using:

    def _filter_analyses(self, request: AnalysesRequest) -> Query:
        filters: list[AnalysisFilter] = [
            AnalysisFilter.BY_WORKFLOW,
            AnalysisFilter.BY_HAS_COMMENT,
            AnalysisFilter.BY_ORDER_ID,
            AnalysisFilter.BY_PRIORITIES,
            AnalysisFilter.BY_STATUSES,
            AnalysisFilter.BY_TYPES,
            AnalysisFilter.BY_CASE_ID,
            AnalysisFilter.BY_SEARCH_TERM,
            AnalysisFilter.BY_IS_VISIBLE,
            AnalysisFilter.BY_DELIVERED,
            AnalysisFilter.SORTING,
        ]
        show_hidden = bool(request.search) or request.include_hidden
        return apply_analysis_filter(
            filter_functions=filters,
            analyses=self.get_query(Analysis),
            has_comment=request.has_comment,
            order_id=request.order_id,
            priorities=request.priority,
            statuses=request.status,
            types=request.type,
            case_id=request.case_id,
            workflow=request.workflow,
            search_term=request.search,
            show_hidden=show_hidden,
            sort_field=request.sort_field,
            sort_order=request.sort_order,
            delivered=request.delivered,
        )

Which assigns False to the attribute show_hidden and excludes hidden analyses from the query.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants