Minor GKS formatting changes and addition of gnomAD flags to annotation #617

theferrit32 · 2023-10-05T15:33:49Z

Modifications:

change popFreqID to popFreqId
change subcohort ids to use a . delimiter instead of , to match the popmax syntax

Additions:

add qcFilters
add lowComplexityRegion (lcr)
add allele balance flagged count (>0.9 in the table) as heterozygousSkewedAlleleCount

One point is that the allele balance code will fail if the number of entries or the bins in the array ht.qual_hists.ab_hist_alt.bin_freq changes. That field is directly used to drive the the UI.

EDIT:
Additional changes:

add hemizygote counts to variant and subcohorts
if no popmax faf exists, sets the popmaxFAF95 object to null instead of setting the freq to 0 and having null or other string in other fields (@matren395)
rename popmaxFAF95 to grpMaxFAF95, popFreqId to grpFreqId (@matren395)

matren395 · 2023-10-16T21:04:19Z

Hey Kyle, could we roll the change to "popFreqId" not returning "None" but instead a string here as well? I had created my own PR for that and the "popFreqID" fix, but it looks like this already contains the latter and would be easy enough to host the former as well

gnomad/utils/annotations.py

theferrit32 · 2023-10-18T15:52:22Z

One more pending change is to add count of hemizygotes. Which needs to be computed, it is not explicitly in a field on the table.

@larrybabb @ahwagner Was there an idea to move some of these fields out of ancillaryResults?

see: ga4gh/va-spec#119 (comment)

I think some of that might have been related to my confusion. I'm not sure the allele balance flagged actually are directly related to the popmax FAF 95. Maybe someone on the gnomad team can clarify.

gnomad/utils/annotations.py

matren395 · 2023-10-20T15:18:47Z

commit to roll in 1) nullable popMaxFAF95 and 2) change all references of 'pop' returned to 'grp' when relevant.

gnomad/utils/annotations.py

theferrit32 · 2023-10-24T15:59:19Z

Waiting for final modifications to upstream schema. There is one more discussion tomorrow. The changes will be relatively easy to implement after that.

See ga4gh/va-spec#121

theferrit32 · 2023-11-01T01:25:28Z

Adding commits and force pushing a branch rebased from main

matren395

might take another run at the actual code in annotations.py later today

gnomad/resources/grch38/gnomad.py

gnomad/utils/annotations.py

theferrit32 · 2023-11-14T20:11:58Z

Latest changeset: https://github.com/theferrit32/gnomad_methods/compare/374112031d6b1178e40cfc23161a19b82b9d5693..74d869e022933612e566755f42d8dd2e0445e114

fixes fafmax for v4 exomes
uses v4 exomes coverage (for both v4 genomes and v4 exomes, is this correct? or should genomes tables not be given any coverage info?)
replaces AF = None (AC/AN = 0/0) with 0. (the jsonschema requires number, can't be null)

matren395

notes! lmk if you wanna chat abt the PR next week

gnomad/utils/annotations.py

gnomad/resources/grch38/gnomad.py

gnomad/utils/annotations.py

theferrit32 · 2024-01-11T17:53:19Z

gnomad/resources/grch38/gnomad.py

-                    coverage("exomes").versions[coverage_version].path
-                )
+            coverage_ht = hl.read_table(
+                coverage(data_type).versions[coverage_version].path


@klaricch this will use genomes version 3.0.1 coverage table for v3 and v4 genomes tables, and exomes version 4.0 coverage table for v4 exomes tables. If the input is a v3 table with data_type=exomes, line 562 will throw an exception KeyError: '3.0.1', which is expected.

gnomad/sample_qc/ancestry.py

theferrit32 · 2024-01-11T18:00:52Z

Rebasing from main to fix lint error from duplicate remaining key #617 (review)

…e_group_dicts, refactor to remove idx as param, construct ids and get freq idx locally

…r and allele balance structures in v3/v4

…axFAF95 if null to conform with latest schema. Move meanDepth to qualityMeasures

…n exomes table

Co-authored-by: Daniel Marten <78616802+matren395@users.noreply.github.com> Co-authored-by: klaricch <kristen@broadinstitute.org>

Co-authored-by: klaricch <kristen@broadinstitute.org>

theferrit32 force-pushed the kf/gks-cg-annotations branch from 84a0f3b to 373c0f2 Compare October 5, 2023 15:54

theferrit32 mentioned this pull request Oct 5, 2023

Add heterozygous allele balance flag to gnomad annotations ga4gh/va-spec#117

Closed

klaricch requested changes Oct 17, 2023

View reviewed changes

theferrit32 force-pushed the kf/gks-cg-annotations branch from 36d3d24 to 011c298 Compare October 18, 2023 15:32

theferrit32 marked this pull request as ready for review October 18, 2023 15:46

klaricch requested changes Oct 18, 2023

View reviewed changes

gnomad/utils/annotations.py Outdated Show resolved Hide resolved

gnomad/utils/annotations.py Outdated Show resolved Hide resolved

gnomad/utils/annotations.py Outdated Show resolved Hide resolved

gnomad/utils/annotations.py Outdated Show resolved Hide resolved

klaricch self-assigned this Oct 19, 2023

matren395 self-requested a review October 20, 2023 15:03

matren395 mentioned this pull request Oct 20, 2023

Change popFreqID to conform to VRS Schema - pending feedback #627

Closed

theferrit32 mentioned this pull request Oct 24, 2023

Add hemizygotes count to gnomad annotations ga4gh/va-spec#120

Closed

matren395 reviewed Oct 24, 2023

View reviewed changes

gnomad/utils/annotations.py Outdated Show resolved Hide resolved

theferrit32 force-pushed the kf/gks-cg-annotations branch from 4a77e59 to e1b8b96 Compare November 1, 2023 01:25

theferrit32 requested review from klaricch and matren395 November 2, 2023 16:32

matren395 reviewed Nov 6, 2023

View reviewed changes

matren395 reviewed Dec 1, 2023

View reviewed changes

gnomad/utils/annotations.py Show resolved Hide resolved

gnomad/resources/grch38/gnomad.py Outdated Show resolved Hide resolved

jkgoodrich added the ga4gh VRS label Dec 7, 2023

mike-w-wilson assigned matren395 Jan 4, 2024

theferrit32 commented Jan 8, 2024

View reviewed changes

gnomad/utils/annotations.py Show resolved Hide resolved

theferrit32 commented Jan 11, 2024

View reviewed changes

gnomad/sample_qc/ancestry.py Outdated Show resolved Hide resolved

theferrit32 force-pushed the kf/gks-cg-annotations branch from 99bad94 to 537a1d9 Compare January 11, 2024 18:01

jkgoodrich mentioned this pull request Jan 16, 2024

Functions to process, filter, annotate and aggregate variants by transcript expression (get the pext scores per variant) #651

Merged

matren395 and others added 26 commits February 27, 2024 15:17

Nullable popMaxFAF95 & pop->grp

bc2cb83

Rename heterozygousAlleleBalanceFlagged to heterozygousSkewedAlleleCount

d5d89c4

Add hemizygote count to overall and by_sex subcohorts

0a48f11

Include hemizygote count for unsexed subcohorts

f402b30

Now that we need to sometimes retrieve other group freq inside _creat…

bd173fe

…e_group_dicts, refactor to remove idx as param, construct ids and get freq idx locally

Remove freq_index_key, not worth it for only a couple values

9c2f6ed

Rename tpl->x

06c172b

Add handling for joint groupmax faf in v4 tables. Add handling for lc…

2b9d2a9

…r and allele balance structures in v3/v4

Add 'remaining' to POP_NAMES and POP_COLORS

6247036

Fix FAF key for v3. Fix when jointGrpMaxFAF95 is null

a07b71f

Add docstrings to add_grpMaxFAF95* functions

35f18ae

Move qualityMeasures to top level of cohort freq statement. Omit grpM…

3c86062

…axFAF95 if null to conform with latest schema. Move meanDepth to qualityMeasures

Add monoallelic flag using AC/AN

0261dbd

Use monoallelic flag from table, which is based on raw AF

9dbd418

Use version field to determine fafmax struct. Handle fafmax subsets i…

13a0ed2

…n exomes table

Replace None alleleFrequency due to 0/0, with 0.0

50797cd

Use v4 exomes coverage if table passed is v4.

6c60dca

Update coverage table determination for gks annotations

ca9cffb

Add fractionCoverage20x annotation to qualityMeasures

91b67f7

Remove duplicate 'remaining' key from ancestry.py

4350015

Apply suggestions from code review

cd66874

Co-authored-by: Daniel Marten <78616802+matren395@users.noreply.github.com> Co-authored-by: klaricch <kristen@broadinstitute.org>

Apply formatting

ed2312c

Delete add_grpMaxFAF95_v3 and gks_compute_seqloc_digest.

b6c5101

lint

2c8c99e

Apply suggestions from code review

0ae91f2

Co-authored-by: klaricch <kristen@broadinstitute.org>

Move high level version check above attempt to read table

caabf24

theferrit32 force-pushed the kf/gks-cg-annotations branch from be25fdb to caabf24 Compare February 27, 2024 20:17

klaricch self-requested a review February 28, 2024 15:39

klaricch approved these changes Feb 28, 2024

View reviewed changes

klaricch merged commit 060fcb8 into broadinstitute:main Feb 28, 2024
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Minor GKS formatting changes and addition of gnomAD flags to annotation #617

Minor GKS formatting changes and addition of gnomAD flags to annotation #617

theferrit32 commented Oct 5, 2023 •

edited

Loading

matren395 commented Oct 16, 2023

theferrit32 commented Oct 18, 2023

matren395 commented Oct 20, 2023

theferrit32 commented Oct 24, 2023

theferrit32 commented Nov 1, 2023

matren395 left a comment

theferrit32 commented Nov 14, 2023 •

edited

Loading

matren395 left a comment

theferrit32 Jan 11, 2024

theferrit32 commented Jan 11, 2024 •

edited

Loading

Minor GKS formatting changes and addition of gnomAD flags to annotation #617

Minor GKS formatting changes and addition of gnomAD flags to annotation #617

Conversation

theferrit32 commented Oct 5, 2023 • edited Loading

matren395 commented Oct 16, 2023

theferrit32 commented Oct 18, 2023

matren395 commented Oct 20, 2023

theferrit32 commented Oct 24, 2023

theferrit32 commented Nov 1, 2023

matren395 left a comment

Choose a reason for hiding this comment

theferrit32 commented Nov 14, 2023 • edited Loading

matren395 left a comment

Choose a reason for hiding this comment

theferrit32 Jan 11, 2024

Choose a reason for hiding this comment

theferrit32 commented Jan 11, 2024 • edited Loading

theferrit32 commented Oct 5, 2023 •

edited

Loading

theferrit32 commented Nov 14, 2023 •

edited

Loading

theferrit32 commented Jan 11, 2024 •

edited

Loading