feat: dataflow decompress prototype #501
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ireneisdoomed
approved these changes
Feb 21, 2024
There was a problem hiding this comment.
This is great, thanks!
How the DAG input/output will look like:
graph LR;
GzippedSuSIEResults["Gzipped SuSIE results"] --> UncompressedSuSIEResults["Uncompressed SuSIE results creation"];
UncompressedSuSIEResults --> eQTLCatalogueCredibleSets["eQTL Catalogue credible sets"];
UncompressedSuSIEResults --> eQTLCatalogueStudyIndex["eQTL Catalogue study index"];
eQTLCatalogueStudyIndex --- UncompressedSuSIEResultsDeletion["Uncompressed SuSIE results deletion"];
eQTLCatalogueCredibleSets --- UncompressedSuSIEResultsDeletion["Uncompressed SuSIE results deletion"];
style eQTLCatalogueCredibleSets fill:#8FBC8F;
style eQTLCatalogueStudyIndex fill:#8FBC8F;
|
because study index and credible sets are co-dependent you might need one single box that generates both outputs |
|
@d0choa Yes, I meant to represent I/O, not actual logic. Comment updated, thanks! |
ireneisdoomed
added a commit
that referenced
this pull request
Mar 4, 2024
…ng results (#514) * feat: dataflow decompress prototype (#501) * chore: commit susie results gist * feat(study_index): add `tissueFromSourceId` to schema and make `traitFromSource` nullable * fix: bug and linting fixes in new eqtl ingestion step * perf: config bugfixes and performance improvements * perf: remove data persistance to avoid executor failure * perf: load susie results for studies of interest only * perf: collect locus for leads only and optimise partitioning cols * feat: parametrise methods to include * feat: run full dag * test: add tests * fix: reorder test inputs * docs: update eqtl catalogue docs * fix: correct typos in tests docstrings * fix: correct typos in tests docstrings * test: fix * revert: revert unwanted change in studyId definition * test: final fix --------- Co-authored-by: David Ochoa <ochoa@ebi.ac.uk>
ireneisdoomed
added a commit
that referenced
this pull request
Mar 6, 2024
…apping results (#518) * feat: dataflow decompress prototype (#501) * chore: commit susie results gist * feat(study_index): add `tissueFromSourceId` to schema and make `traitFromSource` nullable * fix: bug and linting fixes in new eqtl ingestion step * perf: config bugfixes and performance improvements * perf: remove data persistance to avoid executor failure * perf: load susie results for studies of interest only * perf: collect locus for leads only and optimise partitioning cols * feat: parametrise methods to include * feat: run full dag * test: add tests * fix: reorder test inputs * docs: update eqtl catalogue docs * fix: correct typos in tests docstrings * refactor: change mqtl_quantification_methods to mqtl_quantification_methods_blacklist * feat: studyId is based on measured trait and not on gene * feat: credible set lead is the variant with highest pip * feat(studies): change logic in _identify_study_type to extract qtl type based on quantization method * refactor: externalise reading logic to source classes * chore: add mqtl_quantification_methods_blacklist to yaml config * docs: update docs * fix(dag): pass bucket name to GCSDeleteBucketOperator * refactor(coloc): move get_logsum function to common utils * feat(studylocus): add calculate_credible_set_log10bf and use it for eqtlcat credible sets * fix: credible sets dataset is too large and cant be broadcasted * fix(dag): use GCSDeleteObjectsOperator instead of GCSDeleteBucketOperator * fix: correct typo * fix: correct typo
DSuveges
pushed a commit
that referenced
this pull request
Mar 8, 2024
…apping results (#518) * feat: dataflow decompress prototype (#501) * chore: commit susie results gist * feat(study_index): add `tissueFromSourceId` to schema and make `traitFromSource` nullable * fix: bug and linting fixes in new eqtl ingestion step * perf: config bugfixes and performance improvements * perf: remove data persistance to avoid executor failure * perf: load susie results for studies of interest only * perf: collect locus for leads only and optimise partitioning cols * feat: parametrise methods to include * feat: run full dag * test: add tests * fix: reorder test inputs * docs: update eqtl catalogue docs * fix: correct typos in tests docstrings * refactor: change mqtl_quantification_methods to mqtl_quantification_methods_blacklist * feat: studyId is based on measured trait and not on gene * feat: credible set lead is the variant with highest pip * feat(studies): change logic in _identify_study_type to extract qtl type based on quantization method * refactor: externalise reading logic to source classes * chore: add mqtl_quantification_methods_blacklist to yaml config * docs: update docs * fix(dag): pass bucket name to GCSDeleteBucketOperator * refactor(coloc): move get_logsum function to common utils * feat(studylocus): add calculate_credible_set_log10bf and use it for eqtlcat credible sets * fix: credible sets dataset is too large and cant be broadcasted * fix(dag): use GCSDeleteObjectsOperator instead of GCSDeleteBucketOperator * fix: correct typo * fix: correct typo
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Includes working prototype of DAG step to decompress the eQTL catalog
.gzdatasets using DataFlow's template.