feat(datasets): adding new variant annotation model

[DSuveges]

✨ Context

Variant representation replaced from the GnomAD variant annotation to a schema tailored with the Platform variant representation in mind. The data for the new variant model is derived from VEP output. Essentially the idea is that we'll run VEP for every variant we have phenotypic data for every release for the most up to date Ensembl version.

🛠 What does this PR implement

New variant index

• New data model.
• New schema.
• Relevant dataset specific methods migrated to new data model from old variant index and the old variant annotation.
• To be consistent with the rest of the Platform design, instead of variant consequence terms, we are using sequence ontology identifiers. To do the mapping, a mapping file had to be added as a static asset.

Removal of variant annotation - A number of steps were required to eradicate variant annotation from gentropy. There's no need for it anymore, as the variant index data model will contain all required annotation.

• Removing schema.
• Removing dataset module.
• Removing dataset documentation.
• Walked through most of the code and documentation to remove references to variant annotation. This effort might not be complete.

Migrating variant annotation to variant index - As there are multiple dependencies on Variant Annotation dataset, these steps needed to be reviewed and amended to make sure no downstream process would fail.

• GnomAD ingestion step now produces variant index dataset, which can be picked up by the variant index step to bring in cross references to GnomAD and allele frequencies.
• V2G generation now depends on the new variant index.
• The ingestion of GWAS Catalog curated dataset picks up the variant index generated from GnomAD variants.

New data source - Variant annotation is parsed from Ensembl's VEP output. Therefore we can consider Ensembl as a separate datasource, where potentially other parsers can be added (eg. parser for rs id to variant id).

• New datasource added to gentropy with the logic to parse JSON formatted VEP output.
• Documentation was added.
• As the VEP output json can be very big but also can miss all columns where no data is available, it makes sense to load the data with providing the schema. Providing the schema ensures parsing methods to not fail. The schema was put to static assets.

Pipelining

• All relevant configuration was updated to remove variant annotation and use the new variant index wherever it was necessary.
• Step was added to generate variant index based on the Ensembl VEP output.
• All configuration was removed related to variant annotation.
• GWAS Catalog ingestion DAGs were also updated to use the GnomAD variant set.

Tests

• Tests were updated to point to the new classes and new tests were written to test the VEP parser.
• Airflow steps and configs were tested: GnomAD variant ingestion. Variant index generation, V2G generation.

Other refactoring along the way

• Removal of unused sample dataset in the tests folder.
• Updating doc strings when found inaccuracies.
• I have added a line to VSCode config to suppress extension recommendations, as I found them overly annoying.
• The vep_consequences.tsv file got extended with new consequence terms and changed the format slightly to no transformation is required when used.
• As there's no variant annotation, I removed the step, but at the same time, I put together the two GnomAD preprocess step into one single DAG.

🙈 Missing

• !! This PR doesn't include logic or workflow to generate the variant list for the index
• !! The logic and workflow to generate VEP annotation for the variant is also in a separate PR.
• !! The current place for VEP output (vep_output_path: gs://genetics_etl_python_playground/vep/full_variant_index_vcf) is not tidy. This needs to be refactored.

🚦 Before submitting

• Do these changes cover one single feature (one change at a time)? - not really.
• Did you read the contributor guideline?
• Did you make sure to update the documentation with your changes?
• Did you make sure there is no commented out code in this PR?
• Did you follow conventional commits standards in PR title and commit messages?
• Did you make sure the branch is up-to-date with the dev branch?
• Did you write any new necessary tests?
• Did you make sure the changes pass local tests (make test)?
• Did you make sure the changes pass pre-commit rules (e.g poetry run pre-commit run --all-files)?

[DSuveges]

When the score file is read, the schema not right. The scores need to be cast to double.

[ireneisdoomed]

Suggested change

	"nullable": true,
	"nullable": false,

[ireneisdoomed]

See my comments and let me know your thoughts

[DSuveges]

Thanks for the thorough review. I think I could answer most of your questions and implement the suggestions where I could.

[ireneisdoomed]

Thank you for reviewing all my comments.
I'll approve, but please look at my comment about the joining strategy when bringing the frequencies to the index table.