Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enhancements to the IDC scripts #201

Merged
merged 24 commits into from
Jan 31, 2024
Merged

Enhancements to the IDC scripts #201

merged 24 commits into from
Jan 31, 2024

Conversation

jmchilton
Copy link
Member

  • More unit tests.
  • Add option of filtering by stage (and other things) in split script.
  • More validation of generated artifacts (in particular tools YAML).
  • More checks in the IDC lint (verify all reference indexers in genomes.yml are keyed in data_managers.yml and all data manager tools are from the main tool shed).

mvdbeek and others added 22 commits January 31, 2024 16:49
@mvdbeek
Add data_manager_mode argument fort IDC use
fa219fd 
Idea is that we build bundles against the existing servers,
then just pull the bundles into an instance that can write to cvmfs.
@mvdbeek
Not sure if that was necessary
ac2103b
@mvdbeek
All setting history name in run_data_managers.py
0b60c00
@natefoo @mvdbeek
Pass args to get_or_create_history()
272d798
@jmchilton @mvdbeek
split genomes for IDC
1f93beb
@natefoo @mvdbeek
Actually return the parser
2f8e3bb
@natefoo @mvdbeek
Fix _idc_split_data_manager_genomes parser
7f115d5
@natefoo @mvdbeek
Include description in fetch
07e0504
@natefoo @mvdbeek
Add logging
8f53527
@natefoo @mvdbeek
Set defaults compatible with run-data-managers
c83ec68
@jmchilton @mvdbeek
Handle empty items and data reload fields in run_data_managers.
230be45
@jmchilton @mvdbeek
Don't serialize unset fields when splitting genomes.yml into run tasks.
ae3a75a
@jmchilton @mvdbeek
Lint fixes & fix for adding __init__ in tests.
45fe43c
@jmchilton @mvdbeek
Improved IDC linting...
76b4519
@jmchilton @mvdbeek
More shed linting...
c0b4cd4
@jmchilton @mvdbeek
Test case for data manager tools YAML generation.
ff2d58f
@jmchilton @mvdbeek
Implement stages and filtering in IDC split script.
2ff111d
@jmchilton @mvdbeek
New defaults... EPHEMERIS_GALAXY and EPHEMERIS_API_KEY
2d9e757
@jmchilton @mvdbeek
Skip broken data table checking logic if no data table reloads are fo…
5356077 
…und...
@jmchilton @mvdbeek
Implement --tool-id-mode=short option to get short IDs for run-data-m…
7b2773c 
…anagers
@natefoo @mvdbeek
Always use "New" dbkeys (even with UCSC builds) in IDC scripts, and
f7ac124 
automatically fetch UCSC description if unset.
@natefoo @mvdbeek
Support checking what needs to be imported/published to CVMFS
d5b92a7
@mvdbeek mvdbeek force-pushed the idc_2 branch 2 times, most recently from 1c1a9a1 to f8460fc Compare January 31, 2024 16:01
@mvdbeek mvdbeek force-pushed the idc_2 branch 2 times, most recently from c4b53a6 to 21ef91c Compare January 31, 2024 16:23
@mvdbeek mvdbeek changed the title Enhancements to the IDC scripts post green branch Nate is using Enhancements to the IDC scripts Jan 31, 2024
mvdbeek
mvdbeek approved these changes Jan 31, 2024
View reviewed changes
Copy link
Member

@mvdbeek mvdbeek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

green!

@mvdbeek mvdbeek merged commit 99154c3 into galaxyproject:master Jan 31, 2024
4 checks passed
This was referenced Jan 31, 2024
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mvdbeek mvdbeek mvdbeek approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@jmchilton @mvdbeek @natefoo