CDM Data Issues Tracker

[callachennault]

Done Condition (What do we need? Why do we need it? Keep this is small as possible!)

• Change CDM sample list to read from data_clinical_cvr file to minimize data lag
• 'UNKNOWN' values for CDM elements in mskarcher_unfiltered - reached out to Chris 11/12
• Create new Airflow connection for server running SLURM - update from Chris on 11/15: there will be no change to the server running the DAGs for this release
• Possibly - Help get new dags set up on AWS Airflow
• Verify if we encounter data lag based on time the databricks query DAG is run each day
• Update CDM metadata

Data issues to discuss

• Unknown values - Discuss with Chris
• How to handle data lag in clinical files - Discuss with Ben/Avery
• archer, hemepact, and access clinical patient and sample files - not being updated by CDM processes. Discuss with Chris
• Should cdm repo be updated? https://github.mskcc.org/cdsi/cdm
• Importer does not read meta_timeline_*.txt files - only meta_timeline.txt

Process improvements

• Only merge in list of predefined CDM files - to prevent extra/unneeded data from being committed/imported
• Check number of 'unknown' values in clinical file and do not merge if above certain threshold
• Store backup of yesterday's data from s3 bucket

Technical Description (How are we going to achieve the above)

Potential Issues

Dependencies

Technical Requirements

Outside People/Teams

Changes