Microsalt jobs not tracked correctly

[seallard]

No jobs are tracked for microsalt in trailblazer, as reported by @Karl-Svard and @karlnyr. This has been an issue for at least a couple of months.

[seallard]

Refinement:

• How are jobs tracked for microsalt cases?

No jobs exist for the cases in the trailblazer database. So the issue is not that the API fails to return the jobs, rather the actual tracking of the jobs in trailblazer is not working somehow.

Automation runs trailblazer scan which fetches jobs from slurm and adds them to the analysis. Looking through the logs, messages like these occur:

2024-01-07 07:22:49 hasta.scilifelab.se trailblazer.store.crud.update[144679] ERROR Error updating analysis for: case - prettystallion : FileNotFoundError

This is the function that fails

    def update_analysis_from_slurm_output(
        self, analysis_id: int, analysis_host: str | None = False
    ) -> None:
        """Query SLURM for entries related to given analysis, and update the analysis in the database."""
        analysis: Analysis | None = self.get_analysis_with_id(analysis_id)
        try:
            self._update_analysis_from_slurm_squeue_output(
                analysis=analysis, analysis_host=analysis_host
            )
        except Exception as exception:
            LOG.error(
                f"Error updating analysis for: case - {analysis.case_id} : {exception.__class__.__name__}"
            )
            analysis.status = TrailblazerStatus.ERROR
            session: Session = get_session()
            session.commit()

[seallard]

These are the errors

[sebastian.allard@hasta:~] [P_trailblazer] 2s 130 $ trailblazer scan
Error updating analysis for: case - grandaphid : [Errno 2] No such file or directory: '/home/proj/production/cancer/cases/grandaphid/analysis/slurm_jobids.yaml'
Error updating analysis for: case - goodbengal : [Errno 2] No such file or directory: '/home/proj/production/nipt/cases/goodbengal/output/sacct/submitted_jobs.yaml'
Error updating analysis for: case - helpedlab : [Errno 2] No such file or directory: '/home/proj/production/nipt/cases/helpedlab/output/sacct/submitted_jobs.yaml'
Error updating analysis for: case - newgopher : [Errno 2] No such file or directory: '/home/proj/production/microbial/results/reports/trailblazer/ACC9662_slurm_ids.yaml'
Error updating analysis for: case - onestag : [Errno 2] No such file or directory: '/home/proj/production/microbial/results/reports/trailblazer/ACC9966_slurm_ids.yaml'
Error updating analysis for: case - funnyaphid : [Errno 2] No such file or directory: '/home/proj/production/mutant/cases/funnyaphid/results/trailblazer_config.yaml'
Error updating analysis for: case - proudfish : [Errno 2] No such file or directory: '/home/proj/production/microbial/results/reports/trailblazer/ACC9337_slurm_ids.yaml'
Error updating analysis for: case - setcolt : [Errno 2] No such file or directory: '/home/proj/production/microbial/results/reports/trailblazer/ACC9963_slurm_ids.yaml'
Error updating analysis for: case - handyshrimp : [Errno 2] No such file or directory: '/home/proj/production/mutant/cases/handyshrimp/results/trailblazer_config.yaml'
Error updating analysis for: case - divinetomcat : [Errno 2] No such file or directory: '/home/proj/production/microbial/results/reports/trailblazer/ACC5119_slurm_ids.yaml'
Error updating analysis for: case - HW2V2DRX2 : [Errno 2] No such file or directory: '/home/proj/production/flowcells/novaseq/runs/230504_A00689_0805_AHW2V2DRX2/slurm_job_ids.yaml'
Error updating analysis for: case - savingstag : [Errno 2] No such file or directory: '/home/proj/production/mutant/cases/savingstag/results/trailblazer_config.yaml'
Error updating analysis for: case - topewe : [Errno 2] No such file or directory: '/home/proj/production/microbial/results/reports/trailblazer/ACC12566_slurm_ids.yaml'
Error updating analysis for: case - fullgoat : [Errno 2] No such file or directory: '/home/proj/production/microbial/results/reports/trailblazer/ACC12567_slurm_ids.yaml'
etc.

So the problem for microsalt seems to be that the slurm_ids.yaml is missing.

This seems to be where the path is mentioned in cg:

    def get_trailblazer_config_path(self, case_id: str) -> Path:
        """Get trailblazer config path."""
        case_obj: Case = self.status_db.get_case_by_internal_id(internal_id=case_id)
        sample_obj: Sample = case_obj.links[0].sample
        project_id: str = self.get_project(sample_obj.internal_id)
        return Path(
            self.root_dir, "results", "reports", "trailblazer", f"{project_id}_slurm_ids.yaml"
        )

However, when looking on Hasta there is no trailblazer sub dir in /home/proj/production/microbial/results/reports. So not sure what is going on here, that directory does not exist at all.

The correct path is, for instance, /home/proj/production/microbial/results/ACC12567_2023.8.30_12.51.22/ACC12567_slurm_ids.yaml

[seallard]

#2896
Microsalt apparently tries to write the slurm ids to two locations, but the trailblazer one does not exist. The other location is only created once the analysis is complete. Will be adressed in the separate issue.