20210708 developer call notes

[tomwhite]

Pre-notes

• Notes from previous meeting

PRs

• sgkit PR#616 (Tom) Xarray consolidate metadata change
• sgkit PR#625 (Tom) Exclude fsspec versions causing intermittent test failures
• sgkit PR#619 (Tom) Support Number=G VCF fields
• sgkit PR#620 (Tom) Issue a warning if the number of alt alleles exceeds the maximum specified

Issues

• sgkit#609 Consider adding mean imputation function
• sgkit#622 PLINK data missing for test_regenie datasets
• sgkit#623 CSV file descriptions for test_regenie results
• sgkit#624 Regenie validation script is broken
• sgkit M#1 0.3.0 release
  • sgkit#319 Make plink and bgen readers more consistent
  • sgkit#463 Replicating Hail's GWAS Tutorial
  • sgkit#607 Deploy distribution artifacts to PyPi on GH release

Discussions

• sgkit#626

Updates on Benchmark

Other topics

Notes

Attendees

• Jeff Hammerbacher
• Rafal Wojdyla
• Tom White
• Eric Czech
• Amit Kumar
• Jerome Kelleher
• Gagan Kaushik
• Liangde Li

Discussion

sgkit PR#620 Issue a warning if the number of alt alleles exceeds the maximum specified

• JK: there’s an implicit contract between genotype array and the alleles. We are currently breaking the contract. Already throwing away some data, so should be consistent. Breaks in an unpredictable way otherwise.
• RW: make it an option?
• TW: ploidy throws an error currently if it overflows. We should make consistent.
• JK: ploidy is usually known in most cases, but #alleles is not, so dn’t necessarily need identical behaviour.
• JK: probably want k most frequent.
• TW: do we need a function to split multi-allelic to biallelic?
• JK: yes
• TW: would be good to get Eric’s opinion too.

sgkit#622 PLINK data missing for test_regenie datasets (and other regenie PRs)

• GK: files will be different
• RW: just use the files there?
• GK: can’t get plink files from zarr
• TW: no export functions to go from zarr to plink.
• TW: run validation automatically to avoid sgkit#624 in future.
• RW: in which case don’t need to commit files to repo.
• RW: blocked on script
• GK: will take a look
• RW: is zip file not enough?
• GK: will check files and see if the same as existing dataset.

sgkit M#1 0.3.0 release

• No further progress this week

sgkit#626 Updates on Benchmark

• LL: more detail in google doc. https://docs.google.com/document/d/1DtLm7GNTiNjsSX337E99uUlPpmbCtQiSLJ6-Ttw-zUY/edit
• JK: just 1000 genomes data or other datasets too? How does data generator work?
• LL: would like a suggestion to create data.
• JK: need some kind of model to create the data. Msprime
• LL: precomputing vars?
• TW: probably no need - just let the functions do it.
• RW: See https://github.com/pystatgen/sgkit/issues/438 for inspiration for generating data

GA4GH future of VCF group

• JK: crisis point for VCF. 150K whole genomes. VCF is breaking everything. PB of data. People quite motivated to come up with something more efficient. JK has been asked to contribute regarding generating data and benchmarking. Zarr as a format? Chunked columnar storage for sure. Can map to sgkit. 6 month’s time.
• TW: spVCF?
• JK: can’t slice by sample. Clinching arg for columnar storage. Can’t really have have separate interchange and computation format.
• TW: appetite for inventing new format or use existing tech?
• JK: strong NIH
• TW: why did it stop last time?
• JK: getting UK biobank data
• RW: mailing list or docs?
• JK: will send on what I have. Slack workspace.
• TW: VRS?
• JK: lots of groups doing different things. Data model for variants (better indels etc). If we could take VCF data model and make it scale that would be fine. Biallelic would be fine. Also there were graph genomes trying to solve the representation problem a while ago.