Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: upgrade clinvar-this for new ClinVar XML format (#72) #77

Merged
merged 9 commits into from
May 29, 2024
Merged

feat: upgrade clinvar-this for new ClinVar XML format (#72) #77

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
1547249
feat: upgrade clinvar-this for new ClinVar XML format (#72)
holtgrewe May 27, 2024
1515ecf
wip
holtgrewe May 27, 2024
ae6904c
wip
holtgrewe May 27, 2024
eebe81c
wip
holtgrewe May 27, 2024
fef27e1
wip
holtgrewe May 27, 2024
f4b2a38
bump clinvar-this
holtgrewe May 27, 2024
8457d73
wip
holtgrewe May 28, 2024
0878108
bump clinvar-this to v0.15.4
holtgrewe May 28, 2024
01cb036
Update -build-artifacts.yml
holtgrewe May 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/actions/acmg-class-by-freq/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ runs:
uses: actions/cache@v3
with:
path: ${{ env.OUTPUT_DIR }}/acmg-class-by-freq
key: acmg-class-by-freq-${{ env.CACHE_SEED }}-${{ steps.get-clinvar.outputs.release-name }}-${{ env.CLINVAR_THIS_VERSION }}-${{ env.MAX_RCVS }}-output
key: acmg-class-by-freq-${{ env.CACHE_SEED }}-${{ steps.get-clinvar.outputs.release-name }}-${{ env.CLINVAR_THIS_VERSION }}-${{ env.MAX_RECORDS }}-output
lookup-only: true

- name: Create requirements.txt
Expand All @@ -50,15 +50,15 @@ runs:
uses: actions/cache@v3
with:
path: ${{ env.OUTPUT_DIR }}/acmg-class-by-freq
key: acmg-class-by-freq-${{ env.CACHE_SEED }}-${{ steps.get-clinvar.outputs.release-name }}-${{ env.CLINVAR_THIS_VERSION }}-${{ env.MAX_RCVS }}-output
key: acmg-class-by-freq-${{ env.CACHE_SEED }}-${{ steps.get-clinvar.outputs.release-name }}-${{ env.CLINVAR_THIS_VERSION }}-${{ env.MAX_RECORDS }}-output

- name: Retrieve cached ClinVar JSONL file
if: |
(steps.check-cache-acmg-class-by-freq-output.outputs.cache-hit != 'true')
uses: actions/cache@v3
with:
path: ${{ env.OUTPUT_DIR }}/convert-clinvar
key: convert-clinvar-${{ env.CACHE_SEED }}-${{ steps.get-clinvar.outputs.release-name }}-${{ env.CLINVAR_THIS_VERSION }}-${{ env.MAX_RCVS }}-output
key: convert-clinvar-${{ env.CACHE_SEED }}-${{ steps.get-clinvar.outputs.release-name }}-${{ env.CLINVAR_THIS_VERSION }}-${{ env.MAX_RECORDS }}-output
fail-on-cache-miss: true

- name: Run the ACMG class by freq. generation
Expand Down
4 changes: 2 additions & 2 deletions .github/actions/convert-clinvar/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ runs:
uses: actions/cache@v3
with:
path: ${{ env.OUTPUT_DIR }}/convert-clinvar
key: convert-clinvar-${{ env.CACHE_SEED }}-${{ steps.get-clinvar.outputs.release-name }}-${{ env.CLINVAR_THIS_VERSION }}-${{ env.MAX_RCVS }}-output
key: convert-clinvar-${{ env.CACHE_SEED }}-${{ steps.get-clinvar.outputs.release-name }}-${{ env.CLINVAR_THIS_VERSION }}-${{ env.MAX_RECORDS }}-output
lookup-only: true

- name: Create requirements.txt
Expand All @@ -50,7 +50,7 @@ runs:
uses: actions/cache@v3
with:
path: ${{ env.OUTPUT_DIR }}/convert-clinvar
key: convert-clinvar-${{ env.CACHE_SEED }}-${{ steps.get-clinvar.outputs.release-name }}-${{ env.CLINVAR_THIS_VERSION }}-${{ env.MAX_RCVS }}-output
key: convert-clinvar-${{ env.CACHE_SEED }}-${{ steps.get-clinvar.outputs.release-name }}-${{ env.CLINVAR_THIS_VERSION }}-${{ env.MAX_RECORDS }}-output

- name: Retrieve cached ClinVar file
if: |
Expand Down
4 changes: 2 additions & 2 deletions .github/actions/convert-clinvar/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@ df -h
clinvar-this data xml-to-jsonl \
${CLINVAR_DIR}/ClinVarFullRelease_00-latest_weekly.xml.gz \
${OUTPUT_DIR}/convert-clinvar/clinvar-full-release.jsonl.gz \
$(if [[ "$MAX_RCVS" != "" ]] && [[ "$MAX_RCVS" != "0" ]]; then \
echo --max-records $MAX_RCVS;
$(if [[ "$MAX_RECORDS" != "" ]] && [[ "$MAX_RECORDS" != "0" ]]; then \
echo --max-records $MAX_RECORDS;
fi)


Expand Down
4 changes: 2 additions & 2 deletions .github/actions/download-clinvar/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ df -h

# Check that the release name corresponds to the date of the current weekly release.

curl https://ftp.ncbi.nlm.nih.gov/pub/clinvar/xml/RCV_xml_old_format/weekly_release/ \
curl https://ftp.ncbi.nlm.nih.gov/pub/clinvar/xml/weekly_release/ \
> /tmp/lst.html

grep 'latest_weekly.xml.gz"' /tmp/lst.html \
Expand All @@ -32,6 +32,6 @@ fi
# Actually download the file

wget -O $CLINVAR_DIR/ClinVarFullRelease_00-latest_weekly.xml.gz \
https://ftp.ncbi.nlm.nih.gov/pub/clinvar/xml/RCV_xml_old_format/weekly_release/ClinVarFullRelease_00-latest_weekly.xml.gz
https://ftp.ncbi.nlm.nih.gov/pub/clinvar/xml/weekly_release/ClinVarVCVRelease_00-latest_weekly.xml.gz

df -h
6 changes: 3 additions & 3 deletions .github/actions/extract-vars/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ runs:
uses: actions/cache@v3
with:
path: ${{ env.OUTPUT_DIR }}/extract-vars
key: extract-vars-${{ env.CACHE_SEED }}-${{ steps.get-clinvar.outputs.release-name }}-${{ env.CLINVAR_THIS_VERSION }}-${{ env.MAX_RCVS }}-output
key: extract-vars-${{ env.CACHE_SEED }}-${{ steps.get-clinvar.outputs.release-name }}-${{ env.CLINVAR_THIS_VERSION }}-${{ env.MAX_RECORDS }}-output
lookup-only: true

- name: Create requirements.txt
Expand All @@ -50,15 +50,15 @@ runs:
uses: actions/cache@v3
with:
path: ${{ env.OUTPUT_DIR }}/extract-vars
key: extract-vars-${{ env.CACHE_SEED }}-${{ steps.get-clinvar.outputs.release-name }}-${{ env.CLINVAR_THIS_VERSION }}-${{ env.MAX_RCVS }}-output
key: extract-vars-${{ env.CACHE_SEED }}-${{ steps.get-clinvar.outputs.release-name }}-${{ env.CLINVAR_THIS_VERSION }}-${{ env.MAX_RECORDS }}-output

- name: Retrieve cached ClinVar JSONL file
if: |
(steps.check-cache-extract-vars-output.outputs.cache-hit != 'true')
uses: actions/cache@v3
with:
path: ${{ env.OUTPUT_DIR }}/convert-clinvar
key: convert-clinvar-${{ env.CACHE_SEED }}-${{ steps.get-clinvar.outputs.release-name }}-${{ env.CLINVAR_THIS_VERSION }}-${{ env.MAX_RCVS }}-output
key: convert-clinvar-${{ env.CACHE_SEED }}-${{ steps.get-clinvar.outputs.release-name }}-${{ env.CLINVAR_THIS_VERSION }}-${{ env.MAX_RECORDS }}-output
fail-on-cache-miss: true

- name: Run the variant extraction generation
Expand Down
6 changes: 3 additions & 3 deletions .github/actions/gene-variant-report/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ runs:
uses: actions/cache@v3
with:
path: ${{ env.OUTPUT_DIR }}/gene-variant-report
key: gene-variant-report-${{ env.CACHE_SEED }}-${{ steps.get-clinvar.outputs.release-name }}-${{ env.CLINVAR_THIS_VERSION }}-${{ env.MAX_RCVS }}-output
key: gene-variant-report-${{ env.CACHE_SEED }}-${{ steps.get-clinvar.outputs.release-name }}-${{ env.CLINVAR_THIS_VERSION }}-${{ env.MAX_RECORDS }}-output
lookup-only: true

- name: Create requirements.txt
Expand All @@ -50,15 +50,15 @@ runs:
uses: actions/cache@v3
with:
path: ${{ env.OUTPUT_DIR }}/gene-variant-report
key: gene-variant-report-${{ env.CACHE_SEED }}-${{ steps.get-clinvar.outputs.release-name }}-${{ env.CLINVAR_THIS_VERSION }}-${{ env.MAX_RCVS }}-output
key: gene-variant-report-${{ env.CACHE_SEED }}-${{ steps.get-clinvar.outputs.release-name }}-${{ env.CLINVAR_THIS_VERSION }}-${{ env.MAX_RECORDS }}-output

- name: Retrieve cached ClinVar JSONL file
if: |
(steps.check-cache-gene-variant-report-output.outputs.cache-hit != 'true')
uses: actions/cache@v3
with:
path: ${{ env.OUTPUT_DIR }}/convert-clinvar
key: convert-clinvar-${{ env.CACHE_SEED }}-${{ steps.get-clinvar.outputs.release-name }}-${{ env.CLINVAR_THIS_VERSION }}-${{ env.MAX_RCVS }}-output
key: convert-clinvar-${{ env.CACHE_SEED }}-${{ steps.get-clinvar.outputs.release-name }}-${{ env.CLINVAR_THIS_VERSION }}-${{ env.MAX_RECORDS }}-output
fail-on-cache-miss: true

- name: Run the gene variant report generation
Expand Down
6 changes: 3 additions & 3 deletions .github/actions/phenotype-links/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ runs:
uses: actions/cache@v3
with:
path: ${{ env.OUTPUT_DIR }}/phenotype-links
key: phenotype-links-${{ env.CACHE_SEED }}-${{ steps.get-clinvar.outputs.release-name }}-${{ env.CLINVAR_THIS_VERSION }}-${{ env.MAX_RCVS }}-output
key: phenotype-links-${{ env.CACHE_SEED }}-${{ steps.get-clinvar.outputs.release-name }}-${{ env.CLINVAR_THIS_VERSION }}-${{ env.MAX_RECORDS }}-output
lookup-only: true

- name: Create requirements.txt
Expand All @@ -50,15 +50,15 @@ runs:
uses: actions/cache@v3
with:
path: ${{ env.OUTPUT_DIR }}/phenotype-links
key: phenotype-links-${{ env.CACHE_SEED }}-${{ steps.get-clinvar.outputs.release-name }}-${{ env.CLINVAR_THIS_VERSION }}-${{ env.MAX_RCVS }}-output
key: phenotype-links-${{ env.CACHE_SEED }}-${{ steps.get-clinvar.outputs.release-name }}-${{ env.CLINVAR_THIS_VERSION }}-${{ env.MAX_RECORDS }}-output

- name: Retrieve cached ClinVar JSONL file
if: |
(steps.check-cache-phenotype-links-output.outputs.cache-hit != 'true')
uses: actions/cache@v3
with:
path: ${{ env.OUTPUT_DIR }}/convert-clinvar
key: convert-clinvar-${{ env.CACHE_SEED }}-${{ steps.get-clinvar.outputs.release-name }}-${{ env.CLINVAR_THIS_VERSION }}-${{ env.MAX_RCVS }}-output
key: convert-clinvar-${{ env.CACHE_SEED }}-${{ steps.get-clinvar.outputs.release-name }}-${{ env.CLINVAR_THIS_VERSION }}-${{ env.MAX_RECORDS }}-output
fail-on-cache-miss: true

- name: Run the phenotype links generation
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/-build-artifacts.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,11 +26,11 @@ env:
# Python version
PYTHON_VERSION: "3.11"
# clinvar-this package version
CLINVAR_THIS_VERSION: "0.14.3"
CLINVAR_THIS_VERSION: "0.15.5"
# Helper to get unique cache keys
CACHE_SEED: "0"
# Maximal number of RCVs to process (0 = no limit).
MAX_RCVS: "0"
# Maximal number of records to process (0 = no limit).
MAX_RECORDS: "0"
# Lower verbosity of TQDM progress bar
TQDM_MININTERVAL: "5"

Expand Down
Loading